Batch and streaming ingestion
Regardless of the type (batch or streaming), data ingestion is located in the first layer of the data architecture, as seen in Figure 3.1:
Figure 3.1 – Reference diagram for cloud data architectures: the ingestion layer (on the left) forms the first layer of the architecture
The ingestion layer forms the front door for the solution. Here, we pull in data using data pipelines and, in enterprise-level solutions, commonly have it land in a massive-scale, unstructured storage service such as a data lake.
The type of ingestion plays a key role in the design of a cloud data architecture. Batch ingestion was, and in most cases still is, the norm for ingesting data into the cloud. A batch approach refers to the periodical ingestion or processing of (usually large) bulks of data. Streaming ingestion, as the name suggests, involves continuous streams of data.
In general, batch ingestion and processing have long been the...
