System design

Having a basic design sketched out is often useful, especially in distributed systems where many components will be communicating with each other in different ways. We don't want to spend too long on this stage because our design is likely to evolve as we get stuck into the details, but we will look at a high-level outline so we can discuss the constituents and how they fit together.

The preceding image shows the basic overview of the system we are going to build:

• Twitter is the social media network we all know and love.
• Twitter's streaming API allows long-running connections where tweet data is streamed as quickly as possible.
• twittervotes is a program we will write that reads tweets and pushes the votes into the messaging queue. twittervotes pulls the relevant tweet data, figures out what is being voted for (or rather, which options are mentioned), and pushes the vote into NSQ.
• NSQ is an open source, real-time distributed messaging platform designed to operate at scale...