Ingesting and processing streaming data

The following diagram depicts the components required for stream data ingestion and processing:

Figure 3.7 – The streaming data ingestion and processing pattern

Now, let's discuss how to stream data processing through the lens of the ELTL process.

Streaming data sources

Streaming data is a data source that continuously emanates data. Social media feeds, IoT devices, and event-driven processes such as swiping a credit card are examples of streaming data. The data is continually produced, and the goal of stream processing is to tap into that stream of data and gain insights as quickly as possible. Stream data ingestion and processing facilitate real-time analytics. This implies that analytics is performed on the data without the data being persisted on disk.

Extraction-load

Stream data is extracted using an event publishing-subscribing service. An event publishing-subscribing service enables creating...