Broadly speaking, data pipelines are a set of processes that extract and move data from one system to another, usually transforming it in the process. The key stages in data pipelines include data collection, data processing, data storage, data analysis, data modeling, and serving the results. They enable a systematic way to automate the flow of data, with the end goal of transforming raw information into a format suitable for extracting insights. Processed data may be served in various formats, such as a dashboard, a report, forecasting analysis, or even applications.

To place data pipelines in a broader architectural context, we can reference the principles of a data mesh. A data mesh addresses the complexities of managing data at scale by focusing on four key principles:

• Domain-oriented decentralized data ownership: Data is managed by those closest to it, ensuring that the specialized pipelines...

Text is one of the most abundant and rich sources of data. We can obtain data in the form of text from a variety of sources, such as social media posts, online reviews, research papers, corporate documentation, news articles, and web-browsing results. However, unlike structured data, which can fit into tables and schemas, text data is inherently unstructured. To get insights from text, we must transform it into a format that an LLM can process, for example, by chunking it (also known as tokenization) and using an embedding model to vectorize it. Once the text has been transformed, we can use it for applications such as translation, sentiment analysis, topic modeling, information retrieval (Q&A systems, for example), and text classification.

Overview of text-processing techniques

Before text can be stored in a database or used in conjunction with a machine learning algorithm or LLM, we must first preprocess it. Preprocessing...

Data pipelines serve as structured pathways that guide raw data through various stages of transformation, with the end goal of leading to actionable insights. A data pipeline can be tailored to meet specific goals while ensuring that the data is processed, analyzed, and formatted for presentation efficiently. This section will show the evolutionary path of these pipelines, from classic data processing to the modern agentic systems that are the focus of this book.

Classic data and NLP pipelines (the foundation)

As a broad overview, a general data pipeline involves ingesting data from sources such as APIs, IoT devices, or files, processing it, and storing it in databases or data warehouses. This data is then used for analysis, reporting, and visualization. This flow is visualized in Figure 1.1.

Figure 1.1 – Stages in a general data pipeline

Tools and algorithms are then applied to the processed data to derive...

The generative AI project life cycle of the past was often a linear process: from problem scoping to final deployment. In 2025 and beyond, this has been replaced by a continuous MLOps/LLMOps/AgentOps loop, where production systems are constantly evaluated, improved, and redeployed.

This book is structured to show you how to move beyond the it works on my machine stage of development by teaching you to solve for reliability, scalability, and security at the enterprise level:

• Reliability and feedback: In Chapter 5 and Chapter 6, we will introduce a systematic approach to evaluate and measure RAG pipelines. In Chapter 9, we will explore how to improve agentic systems through context engineering and how to enable observability of agents through LangGraph and LangSmith.
• Scalability and deployment: In Chapter 7, we will learn how to serialize our pipelines into a fully containerized (Docker) microservice...

Data pipelines consist of a series of systematic steps that process data from a raw format into a format that can be used and consumed by a variety of users. In the modern era, these pipelines are evolving from simple, linear flows for human-readable analytics into the foundational reliability layer for sophisticated AI agents.

We explored the evolutionary path of these systems: from general data pipelines and classic NLP pipelines to modern LLM-augmented pipelines. A common denominator in all these cases is the presence of rigorous data source identification, cleaning, and preprocessing. For NLP and LLM pipelines, crucial steps include tokenization (breaking down words) and embeddings (constructing numerical representations).

As we move into 2026, the design of these pipelines is shifting to solve new, complex challenges. We must now account for reliability, scalability, cost-effectiveness, and security in a world of interoperable, autonomous agents.

This book...

• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2023). Attention Is All You Need. https://arxiv.org/abs/1706.03762
• Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. (2021). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. https://dl.acm.org/doi/abs/10.5555/3495724.3496517
• Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725. https://aclanthology.org/P16-1162
• Fowler, M. (2019). Data mesh principles and logical architecture. https://martinfowler.com/articles/data-mesh-principles.html
• OWASP GenAI (2025). LLM06:2025 Excessive Agency. https://genai.owasp.org/llmrisk...