Building Data Streaming Applications with Apache Kafka

Design and administer fast, reliable enterprise messaging systems with Apache Kafka
Preview in Mapt

Building Data Streaming Applications with Apache Kafka

Manish Kumar, Chanchal Singh

4 customer reviews
Design and administer fast, reliable enterprise messaging systems with Apache Kafka

Quick links: > What will you learn?> Table of content> Product reviews

Mapt Subscription
FREE
$29.99/m after trial
eBook
$18.00
RRP $35.99
Save 49%
Print + eBook
$44.99
RRP $44.99
What do I get with a Mapt Pro subscription?
  • Unlimited access to all Packt’s 5,000+ eBooks and Videos
  • Early Access content, Progress Tracking, and Assessments
  • 1 Free eBook or Video to download and keep every month after trial
What do I get with an eBook?
  • Download this book in EPUB, PDF, MOBI formats
  • DRM FREE - read and interact with your content when you want, where you want, and how you want
  • Access this title in the Mapt reader
What do I get with Print & eBook?
  • Get a paperback copy of the book delivered to you
  • Download this book in EPUB, PDF, MOBI formats
  • DRM FREE - read and interact with your content when you want, where you want, and how you want
  • Access this title in the Mapt reader
What do I get with a Video?
  • Download this Video course in MP4 format
  • DRM FREE - read and interact with your content when you want, where you want, and how you want
  • Access this title in the Mapt reader
$0.00
$18.00
$44.99
$29.99 p/m after trial
RRP $35.99
RRP $44.99
Subscription
eBook
Print + eBook
Start 14 Day Trial

Frequently bought together


Building Data Streaming Applications with Apache Kafka Book Cover
Building Data Streaming Applications with Apache Kafka
$ 35.99
$ 18.00
Apache Kafka 1.0 Cookbook Book Cover
Apache Kafka 1.0 Cookbook
$ 27.99
$ 14.00
Buy 2 for $31.50
Save $32.48
Add to Cart

Book Details

ISBN 139781787283985
Paperback278 pages

Book Description

Apache Kafka is a popular distributed streaming platform that acts as a messaging queue or an enterprise messaging system. It lets you publish and subscribe to a stream of records, and process them in a fault-tolerant way as they occur.

This book is a comprehensive guide to designing and architecting enterprise-grade streaming applications using Apache Kafka and other big data tools. It includes best practices for building such applications, and tackles some common challenges such as how to use Kafka efficiently and handle high data volumes with ease. This book first takes you through understanding the type messaging system and then provides a thorough introduction to Apache Kafka and its internal details. The second part of the book takes you through designing streaming application using various frameworks and tools such as Apache Spark, Apache Storm, and more. Once you grasp the basics, we will take you through more advanced concepts in Apache Kafka such as capacity planning and security.

By the end of this book, you will have all the information you need to be comfortable with using Apache Kafka, and to design efficient streaming data applications with it.

Table of Contents

Chapter 1: Introduction to Messaging Systems
Understanding the principles of messaging systems
Understanding messaging systems
Peeking into a point-to-point messaging system
Publish-subscribe messaging system
Advance Queuing Messaging Protocol
Using messaging systems in big data streaming applications
Summary
Chapter 2: Introducing Kafka the Distributed Messaging Platform
Kafka origins
Kafka's architecture
Message topics
Message partitions
Replication and replicated logs
Message producers
Message consumers
Role of Zookeeper
Summary
Chapter 3: Deep Dive into Kafka Producers
Kafka producer internals
Kafka Producer APIs
Java Kafka producer example
Common messaging publishing patterns
Best practices
Summary
Chapter 4: Deep Dive into Kafka Consumers
Kafka consumer internals
Kafka consumer APIs
Java Kafka consumer
Scala Kafka consumer
Common message consuming patterns
Best practices
Summary
Chapter 5: Building Spark Streaming Applications with Kafka
Introduction to Spark 
Spark Streaming 
Use case log processing - fraud IP detection
Producer 
Summary
Chapter 6: Building Storm Applications with Kafka
Introduction to Apache Storm
Introduction to Apache Heron
Integrating Apache Kafka with Apache Storm - Java
Integrating Apache Kafka with Apache Storm - Scala
Use case – log processing in Storm, Kafka, Hive
Summary
Chapter 7: Using Kafka with Confluent Platform
Introduction to Confluent Platform
Deep driving into Confluent architecture
Understanding Kafka Connect and Kafka Stream
Playing with Avro using Schema Registry
Moving Kafka data to HDFS
Summary
Chapter 8: Building ETL Pipelines Using Kafka
Considerations for using Kafka in ETL pipelines
Introducing Kafka Connect
Deep dive into Kafka Connect
Introductory examples of using Kafka Connect
Kafka Connect common use cases
Summary 
Chapter 9: Building Streaming Applications Using Kafka Streams
Introduction to Kafka Streams
Kafka Stream architecture 
Integrated framework advantages
Understanding tables and Streams together
Use case example of Kafka Streams
Summary
Chapter 10: Kafka Cluster Deployment
Kafka cluster internals
Capacity planning
Single cluster deployment
Multicluster deployment
Decommissioning brokers
Data migration
Summary
Chapter 11: Using Kafka in Big Data Applications
Managing high volumes in Kafka
Kafka message delivery semantics
Big data and Kafka common usage patterns
Kafka and data governance
Alerting and monitoring
Useful Kafka matrices
Summary
Chapter 12: Securing Kafka
An overview of securing Kafka
Wire encryption using SSL
Kerberos SASL for authentication
Understanding ACL and authorization
Understanding Zookeeper authentication
Apache Ranger for authorization
Best practices
Summary
Chapter 13: Streaming Application Design Considerations
Latency and throughput
Data and state persistence
Data sources
External data lookups
Data formats
Data serialization
Level of parallelism
Out-of-order events
Message processing semantics
Summary

What You Will Learn

  • Learn the basics of Apache Kafka from scratch
  • Use the basic building blocks of a streaming application
  • Design effective streaming applications with Kafka using Spark, Storm &, and Heron
  • Understand the importance of a low -latency , high- throughput, and fault-tolerant messaging system
  • Make effective capacity planning while deploying your Kafka Application
  • Understand and implement the best security practices

Authors

Table of Contents

Chapter 1: Introduction to Messaging Systems
Understanding the principles of messaging systems
Understanding messaging systems
Peeking into a point-to-point messaging system
Publish-subscribe messaging system
Advance Queuing Messaging Protocol
Using messaging systems in big data streaming applications
Summary
Chapter 2: Introducing Kafka the Distributed Messaging Platform
Kafka origins
Kafka's architecture
Message topics
Message partitions
Replication and replicated logs
Message producers
Message consumers
Role of Zookeeper
Summary
Chapter 3: Deep Dive into Kafka Producers
Kafka producer internals
Kafka Producer APIs
Java Kafka producer example
Common messaging publishing patterns
Best practices
Summary
Chapter 4: Deep Dive into Kafka Consumers
Kafka consumer internals
Kafka consumer APIs
Java Kafka consumer
Scala Kafka consumer
Common message consuming patterns
Best practices
Summary
Chapter 5: Building Spark Streaming Applications with Kafka
Introduction to Spark 
Spark Streaming 
Use case log processing - fraud IP detection
Producer 
Summary
Chapter 6: Building Storm Applications with Kafka
Introduction to Apache Storm
Introduction to Apache Heron
Integrating Apache Kafka with Apache Storm - Java
Integrating Apache Kafka with Apache Storm - Scala
Use case – log processing in Storm, Kafka, Hive
Summary
Chapter 7: Using Kafka with Confluent Platform
Introduction to Confluent Platform
Deep driving into Confluent architecture
Understanding Kafka Connect and Kafka Stream
Playing with Avro using Schema Registry
Moving Kafka data to HDFS
Summary
Chapter 8: Building ETL Pipelines Using Kafka
Considerations for using Kafka in ETL pipelines
Introducing Kafka Connect
Deep dive into Kafka Connect
Introductory examples of using Kafka Connect
Kafka Connect common use cases
Summary 
Chapter 9: Building Streaming Applications Using Kafka Streams
Introduction to Kafka Streams
Kafka Stream architecture 
Integrated framework advantages
Understanding tables and Streams together
Use case example of Kafka Streams
Summary
Chapter 10: Kafka Cluster Deployment
Kafka cluster internals
Capacity planning
Single cluster deployment
Multicluster deployment
Decommissioning brokers
Data migration
Summary
Chapter 11: Using Kafka in Big Data Applications
Managing high volumes in Kafka
Kafka message delivery semantics
Big data and Kafka common usage patterns
Kafka and data governance
Alerting and monitoring
Useful Kafka matrices
Summary
Chapter 12: Securing Kafka
An overview of securing Kafka
Wire encryption using SSL
Kerberos SASL for authentication
Understanding ACL and authorization
Understanding Zookeeper authentication
Apache Ranger for authorization
Best practices
Summary
Chapter 13: Streaming Application Design Considerations
Latency and throughput
Data and state persistence
Data sources
External data lookups
Data formats
Data serialization
Level of parallelism
Out-of-order events
Message processing semantics
Summary

Book Details

ISBN 139781787283985
Paperback278 pages
Read More
From 4 reviews

Read More Reviews

Recommended for You

Apache Kafka 1.0 Cookbook Book Cover
Apache Kafka 1.0 Cookbook
$ 27.99
$ 14.00
Data Science Algorithms in a Week Book Cover
Data Science Algorithms in a Week
$ 31.99
$ 16.00
Statistics for Machine Learning Book Cover
Statistics for Machine Learning
$ 39.99
$ 20.00
Python: End-to-end Data Analysis Book Cover
Python: End-to-end Data Analysis
$ 71.99
$ 36.00
Build scalable applications with Apache Kafka [Video] Book Cover
Build scalable applications with Apache Kafka [Video]
$ 124.99
$ 25.00
Working with Big Data in Python [Video] Book Cover
Working with Big Data in Python [Video]
$ 124.99
$ 25.00