Reader small image

You're reading from  Scala Data Analysis Cookbook

Product typeBook
Published inOct 2015
Reading LevelIntermediate
Publisher
ISBN-139781784396749
Edition1st Edition
Languages
Right arrow
Author (1)
Arun Manivannan
Arun Manivannan
author image
Arun Manivannan

Arun Manivannan has been an engineer in various multinational companies, tier-1 financial institutions, and start-ups, primarily focusing on developing distributed applications that manage and mine data. His languages of choice are Scala and Java, but he also meddles around with various others for kicks. He blogs at http://rerun.me. Arun holds a master's degree in software engineering from the National University of Singapore. He also holds degrees in commerce, computer applications, and HR management. His interests and education could probably be a good dataset for clustering.
Read more about Arun Manivannan

Right arrow

Using Spark Streaming to subscribe to a Twitter stream


Just like all the other components of Spark, Spark Streaming is also scalable and fault-tolerant, it's just that it manages a stream of data instead of a large amount of data that Spark generally does. The way that Spark Streaming approaches streaming is unique in the sense that it accumulates streams into small batches called DStreams and then processes them as mini-batches, an approach usually called micro-batching. The component that receives the stream of data and splits it into time-bound windows of batches is called the receiver.

Once these batches are received, Spark takes these batches up, converts them into RDDs, and processes the RDDs in the same way as static datasets. The regular framework components such as the driver and executor stay the same. However, in terms of Spark Streaming, a DStream or Discretized stream is just a continuous stream of RDDs. Also, just like SQLContext served as an entry point to use SQL in Spark...

lock icon
The rest of the page is locked
Previous PageNext Page
You have been reading a chapter from
Scala Data Analysis Cookbook
Published in: Oct 2015Publisher: ISBN-13: 9781784396749

Author (1)

author image
Arun Manivannan

Arun Manivannan has been an engineer in various multinational companies, tier-1 financial institutions, and start-ups, primarily focusing on developing distributed applications that manage and mine data. His languages of choice are Scala and Java, but he also meddles around with various others for kicks. He blogs at http://rerun.me. Arun holds a master's degree in software engineering from the National University of Singapore. He also holds degrees in commerce, computer applications, and HR management. His interests and education could probably be a good dataset for clustering.
Read more about Arun Manivannan