Search icon Close icon
Search icon
Subscription
0
Cart icon
Close icon
You have no products in your basket yet
Save more on your purchases!
Savings automatically calculated. No voucher code required
Arrow left icon
All Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletters
Free Learning
Arrow right icon
Modern Big Data Processing with Hadoop

You're reading from  Modern Big Data Processing with Hadoop

Product type Book
Published in Mar 2018
Publisher Packt
ISBN-13 9781787122765
Pages 394 pages
Edition 1st Edition
Languages
Concepts
Authors (3):
V Naresh Kumar V Naresh Kumar
Profile icon V Naresh Kumar
LinkedIn
Naresh has more than a decade of professional experience in designing, implementing and running very large scale Internet Applications in Fortune Top 500 Companies. He is a Full Stack Architect with hands-on experience in domains like E-commerce, Web-hosting, Healthcare, Bigdata & Analytics, Data Streaming, Advertising and Databases. He believes in Opensource and contributes to them actively. He keeps himself up-to-date with emerging technologies starting from Linux Systems Internals to Frontend technologies. He studied in BITS-Pilani, Rajasthan with Dual Degree in Computer Science & Economics.
See other products by V Naresh Kumar
Manoj R Patil Manoj R Patil
Profile icon Manoj R Patil
LinkedIn
Manoj R Patil is the Chief Architect in Big Data at Compassites Software Solutions Pvt. Ltd. where he overlooks the overall platform architecture related to Big Data solutions, and he also has a hands-on contribution to some assignments. He has been working in the IT industry for the last 15 years. He started as a programmer and, on the way, acquired skills in architecting and designing solutions, managing projects keeping each stakeholder's interest in mind, and deploying and maintaining the solution on a cloud infrastructure. He has been working on the Pentaho-related stack for the last 5 years, providing solutions while working with employers and as a freelancer as well. Manoj has extensive experience in JavaEE, MySQL, various frameworks, and Business Intelligence, and is keen to pursue his interest in predictive analysis. He was also associated with TalentBeat, Inc. and Persistent Systems, and implemented interesting solutions in logistics, data masking, and data-intensive life sciences.
See other products by Manoj R Patil
Prashant Shindgikar Prashant Shindgikar
Profile icon Prashant Shindgikar
LinkedIn
Prashant Shindgikar is an accomplished big data Architect with over 20 years of experience in data analytics. He specializes in data innovation and resolving data challenges for major retail brands. He is a hands-on architect having an innovative approach to solving data problems. He provides thought leadership and pursues strategies for engagements with the senior executives on innovation in data processing and analytics. He presently works for a large USA-based retail company.
See other products by Prashant Shindgikar
View More author details

Table of Contents (12) Chapters

Preface 1. Enterprise Data Architecture Principles 2. Hadoop Life Cycle Management 3. Hadoop Design Consideration 4. Data Movement Techniques 5. Data Modeling in Hadoop 6. Designing Real-Time Streaming Data Pipelines 7. Large-Scale Data Processing Frameworks 8. Building Enterprise Search Platform 9. Designing Data Visualization Solutions 10. Developing Applications Using the Cloud 11. Production Hadoop Cluster Deployment

Flume

Flume is a reliable, available and distributed service to efficiently collect, aggregate, and transport large amounts of log data. It has a flexible and simple architecture that is based on streaming data flows. The current version of Apache Flume is 1.7.0, which was released in October 2016.

Apache Flume architecture

The following diagram depicts the architecture of Apache Flume:

Let's take a closer look at the components of the Apache Flume architecture:

  • Event: An event is a byte payload with optional string headers. It represents the unit of data that Flume can carry from its source to destination.
  • Flow: The transport of events from source to destination is considered a data flow, or just flow.
  • Agent: It is...
lock icon The rest of the chapter is locked
arrow left Previous Chapter
Chapter 1 of 12
Next Chapter arrow right
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Sign up now
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Start free trial
Renews at $15.99/month. Cancel anytime}

Authors (3)

Left arrow icon
Profile icon V Naresh Kumar
LinkedIn
Naresh has more than a decade of professional experience in designing, implementing and running very large scale Internet Applications in Fortune Top 500 Companies. He is a Full Stack Architect with hands-on experience in domains like E-commerce, Web-hosting, Healthcare, Bigdata & Analytics, Data Streaming, Advertising and Databases. He believes in Opensource and contributes to them actively. He keeps himself up-to-date with emerging technologies starting from Linux Systems Internals to Frontend technologies. He studied in BITS-Pilani, Rajasthan with Dual Degree in Computer Science & Economics.
Read more
See other products by V Naresh Kumar
Profile icon Manoj R Patil
LinkedIn
Manoj R Patil is the Chief Architect in Big Data at Compassites Software Solutions Pvt. Ltd. where he overlooks the overall platform architecture related to Big Data solutions, and he also has a hands-on contribution to some assignments. He has been working in the IT industry for the last 15 years. He started as a programmer and, on the way, acquired skills in architecting and designing solutions, managing projects keeping each stakeholder's interest in mind, and deploying and maintaining the solution on a cloud infrastructure. He has been working on the Pentaho-related stack for the last 5 years, providing solutions while working with employers and as a freelancer as well. Manoj has extensive experience in JavaEE, MySQL, various frameworks, and Business Intelligence, and is keen to pursue his interest in predictive analysis. He was also associated with TalentBeat, Inc. and Persistent Systems, and implemented interesting solutions in logistics, data masking, and data-intensive life sciences.
Read more
See other products by Manoj R Patil
Profile icon Prashant Shindgikar
LinkedIn
Prashant Shindgikar is an accomplished big data Architect with over 20 years of experience in data analytics. He specializes in data innovation and resolving data challenges for major retail brands. He is a hands-on architect having an innovative approach to solving data problems. He provides thought leadership and pursues strategies for engagements with the senior executives on innovation in data processing and analytics. He presently works for a large USA-based retail company.
See other products by Prashant Shindgikar
Right arrow icon

Other recommended products

Related to this chapter
Left arrow icon
Apache Kafka 1.0 Cookbook
Apache Kafka 1.0 Cookbook
Apache Kafka is an open source stream processing platform to handle real-time data feeds. This book is a highly practical guide to help you understand the fundamentals as well as the advanced applications of Apache Kafka as an enterprise messaging service. It begins with configuring the basic Kafka APIs, and then shows you how to set up Kafka clusters and basic Kafka operations. It covers the recently released Kafka version 1.0, the Confluent Platform and Kafka Streams. By the end of this book, you will have all the knowledge you need to take your understanding of Apache Kafka to the next level and tackle any problem you might encounter while working with it.
Dec 2017 8 hours 20 minutes
Mastering Apache Storm
Mastering Apache Storm
With real-world examples and clear explanations, this book will ensure you will have a thorough mastery Apache Storm.You’ll get an understanding of deploying Storm on clusters. Introduce yourself to topics such as trident topology, monitoring, Storm Parallelism, scheduler and log processing. Learn how to integrate Storm with other well-known Big Data technologies such as HBase, Redis, Kafka, and Hadoop to realize the full potential of Storm.You will be able to use the knowledge to develop efficient, distributed real-time applications to cater to your business needs.
Aug 2017 9 hours 28 minutes
Apache Kafka Quick Start Guide
Apache Kafka Quick Start Guide
Learn how to use Apache Kafka for efficient processing of distributed applications. This book focuses on programming rather than configuration management of Kafka clusters or Dev Ops. Each chapter focuses on a practical aspect and tries to avoid the tedious theoretical sections. By the end of this book, you will be familiar with solving everyday problems in fast data and processing pipelines.
Dec 2018 6 hours 12 minutes
Hadoop 2.x Administration Cookbook
Hadoop 2.x Administration Cookbook
A practical and use case driven approach to Hadoop administration with coverage on a vast array of topics including Hadoop cluster installation, performance tuning, cluster planning, security, and much more. This book covers Hadoop from the perspective of running clusters in critical and large environments with complex data and at scale.
May 2017 11 hours 36 minutes
Apache Hive Essentials
Apache Hive Essentials
Apache Hive helps you deal with data summarization, queries, and analysis for huge amounts of data. This book will give you a background in big data, and familiarize you with your Hive working environment. Next you will cover advanced topics like performance and security in Hive and how to work efficiently to find solutions to big data problems.
Jun 2018 7 hours 0 minutes
Data Lake for Enterprises
Data Lake for Enterprises
The term 'Data Lake' has recently emerged as a prominent term in the big data industry. Data scientists can make use of it in deriving meaningful insights which can be used by businesses to redefine or transform the way they operate. Lambda architecture is also emerging as one of the very eminent patterns in the big data landscape, as it helps to derive useful information from not only the historical data but also correlates real-time data to enable business for taking critical decisions. This book tries to bring these two important aspects into one, namely data lake and lambda architecture.
May 2017 19 hours 52 minutes
Mastering Hadoop 3
Mastering Hadoop 3
This is a comprehensive guide to understand advanced concepts of Hadoop ecosystem. You will learn how Hadoop works internally, and build solutions to some of real world use cases. Finally, you will have a solid understanding of how components in the Hadoop ecosystem are effectively integrated to implement a fast and reliable Big Data pipeline
Feb 2019 18 hours 8 minutes
Apache Hadoop 3 Quick Start Guide
Apache Hadoop 3 Quick Start Guide
Apache Hadoop is a widely used distributed data platform. It enables large datasets to be efficiently processed instead of using one large computer to store and process the data. This book will get you started with the Hadoop ecosystem, and introduce you to the main technical topics such as MapReduce, YARN and HDFS.
Oct 2018 7 hours 20 minutes
Building Data Streaming Applications with Apache Kafka
Building Data Streaming Applications with Apache Kafka
Apache Kafka is a popular distributed streaming platform which acts as a messaging queue or an enterprise messaging system. This book is a comprehensive guide on designing and architecting enterprise-grade streaming applications using Apache Kafka and other Big Data tools. Once you grasp the basics, we will take you through the more advance concepts in Apache Kafka such as capacity planning and security.
Aug 2017 9 hours 16 minutes
MySQL 8 for Big Data
MySQL 8 for Big Data
MySQL is one of the most popular relational databases in the world today, and has become a popular choice of tool to handle vast amounts of structured data - that is, structured Big Data. This book will demonstrate how you can dabble with large amounts of data using MySQL 8. It also highlights topics such as integrating MySQL 8 and a Big Data solution like Apache Hadoop using different tools like Apache Sqoop and MySQL Applier. With practical examples and use-cases, you will get a better clarity on how you can leverage the offerings of MySQL 8 to build a robust Big Data solution.
Oct 2017 9 hours 52 minutes
HBase High Performance Cookbook
HBase High Performance Cookbook
Jan 2017 11 hours 40 minutes
Practical Real-time Data Processing and Analytics
Practical Real-time Data Processing and Analytics
Real-time data processing involves continuous input, processing and output of data, with the condition that the time required for processing is as short as possible. This book covers the majority of the existing and evolving open source technology stack for real-time processing and analytics. You will get to know about all the real-time solution aspects, from the source to the presentation to persistence. Through this practical book, you’ll be equipped with a clear understanding of how to solve challenges on your own.
Sep 2017 12 hours 0 minutes
Right arrow icon

Personalised recommendations for you

Based on your interests and search pattern
Left arrow icon
Et al.
Et al.
Ever wonder why speech recognition systems don't understand the Scottish accent, or what would happen if an astronaut only ate mac 'n' cheese, or other spurious reflections you'd have at a bar? We did, then collated those deliberations into absurd research articles with fake figures and methodologies inspired by even more fictionally absurd studies.
Aug 2023 7 hours 40 minutes
Generative AI with LangChain
Generative AI with LangChain
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Dec 2023 12 hours 0 minutes
Mastering Tableau 2023
Mastering Tableau 2023
This book is a comprehensive resource to mastering your Tableau skills and becoming a BI expert. As you progress, you will learn how to build advanced dashboards and improve your storytelling to derive key business insight, as well as make you well-versed with advanced functionalities of Tableau in the business intelligence domain.
Aug 2023 22 hours 48 minutes
Building AI Applications with ChatGPT APIs
Building AI Applications with ChatGPT APIs
This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.
Sep 2023 8 hours 36 minutes
Building AI Applications with ChatGPT APIs
Building AI Applications with ChatGPT APIs
This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.
Sep 2023 8 hours 36 minutes
Data Engineering with AWS
Data Engineering with AWS
Embark on a journey to master data engineering pipelines on AWS! Our book offers a hands-on experience of AWS services for ingesting, transforming, and consuming data. Whether you're an absolute beginner or someone with basic data engineering experience, this guide is an indispensable resource.
Oct 2023 21 hours 12 minutes
Modern Data Architecture on AWS
Modern Data Architecture on AWS
Every organization wants an agile, performant, and cost-effective data platform that meets all their current and future business needs. Purpose-built AWS analytics services and their features play a big part in building such a modern data platform. This book brings to you all the design and architectural patterns that’ll help you achieve this goal.
Aug 2023 14 hours 0 minutes
Practical Guide to Applied Conformal Prediction in Python
Practical Guide to Applied Conformal Prediction in Python
Discover the power of Conformal Prediction with the "Practical Guide to Applied Conformal Prediction in Python." Master the latest techniques to quantify uncertainty in machine learning and computer vision models, and seamlessly apply them to your industry applications.
Dec 2023 8 hours 0 minutes
TinyML Cookbook
TinyML Cookbook
With over 70 project-based recipes, the TinyML Cookbook is a practical guide that will help you to get the most out of your microcontrollers. It provides a comprehensive understanding of the theoretical foundations while giving you hands-on experience training ML models for deployment on Arduino Nano 33 BLE Sense, Raspberry Pi Pico, and SparkFun RedBoard Artemis Nano microcontrollers.
Nov 2023 22 hours 8 minutes
Right arrow icon