The Arab Spring revolution was sparked and fuelled through social media sites like Facebook and Twitter. Over the next few days, social media went from being just a means of interacting with family and friends to a weapon that empowered the people and brought about a significant change in the world. Everyone noticed the power of the people and people noticed what social networks were capable of. At the heart of it all was the technology that made all this possible, the technology that removed all the barriers to communication and spread the word faster than wildfire. This is the power of real-time web!
On the Web, we have been habituated to sites and applications where we click on a link or a button, or change some input and perform some action, and it causes some change in the page. But if we leave our twitter page open for a while, we get alerts when we receive new tweets, even without performing any action (shown in the next screenshot). This is what we mean in general when we say "real-time web".
Real-time updates on Twitter
Wikipedia introduces real-time web in these words:
The real-time web is a set of technologies and practices that enable users to receive information as soon as it is published by its authors, rather than requiring that they or their software check a source periodically for updates.
This "set of technologies" is one of the hottest trends on the Web. Over the next few pages, we will get familiar with these technologies and see their use in various applications.
To understand and fully appreciate any concept, it is important to know where it came from and how it evolved.
Real-time web is not a new thing; one of the first attempts at making web real-time was the usage of Java applets. Many will remember chatting in Yahoo! chat rooms or playing chess, way back in the late '90s. Then came Flash and ActiveX plugins. This was not only for "fun" (for the consumer section), but also for use in the enterprise market. I worked for a BPM (Business Process Management) company in the early stages of my career, where they had built an ActiveX plugin for powering their dashboards and updating process information in real time. So why is it important now? Because the way in which real-time functionality is implemented and the cost involved in doing so has changed. From being a fancy feature in an application, it has become a necessity—a user demand. From being a hacked-in or technically challenging piece of the application, it is on its way to becoming a ratified standard in the form of WebSockets and Server-Sent Events (SSE). How did we get from static web to here?
The Web (and web applications), as we all know, is built over the HTTP protocol. HTTP is a request-response system, where the client sends a request for information to the server and the server responds with the requested information. In most cases, this information is the HTML or related information, like XML or JSON, to be rendered by the browser. This HTTP browser-server interaction is shown in the following figure:
HTTP browser-server interaction
In 1995, Sun and Netscape announced a partnership that saw Netscape bundle Sun's brand new Java runtime with its browser. This was the beginning of highly interactive web. Although they have since earned themselves a very bad reputation, applets were the pioneers in the field of real-time web. In the early days of real-time web, we saw applets being used everywhere, for chat, games, and even for banners.
In the same year, Netscape came up with a scripting language called JavaScript (originally LiveScript), and another small company called FutureWave Software started working on an animation software called FutureSplash Animator. Later, both of them became the cause of Java applets almost disappearing from the Web.
FutureWave was acquired by Macromedia in 1996 and they renamed FutureSplash Animator to Flash. Flash, as we all know, went on to rule the Web as the most widely available platform for creating animations, games, video players, and everything interactive, for the major part of the next decade.
In 1999, Microsoft used its iframe technology and JavaScript to update news and stock quotes on Internet Explorer's default home page (http://home.microsoft.com). In the same year, they released a proprietary ActiveX extension for IE, called XMLHTTP. This was the era when XML was the "in" thing and everyone wanted to use XML for anything they were doing. This XMLHTTP component was originally meant to load XML data in the page asynchronously, using JavaScript. It was soon adopted by Mozilla, Safari, and Opera, as XMLHttpRequest (or XHR, for short). But it was with the launch of Gmail (by Google) that the term AJAX (Asynchronous JavaScript and XML)—coined by Jesse James Garrett in an article titled Ajax: A New Approach to Web Applications—became the buzzword in web development. The following figure shows an AJAX Request:
AJAX Request
Gmail also shed light on the advantages of live updates to web pages and opened the floodgates to various hacks built over AJAX to push data from a server (or at least, giving the illusion of doing so).
Collectively, these technologies were referred to as Comet-a term introduced by Alex Russell on his blog in 2006. Comet was a play on the word Ajax, both being popular household cleaners in the US. Comet was not one single approach. It introduced multiple mechanisms to give the feeling of data being pushed from the server to the client. These included Hidden iframe, XHR polling, XHR long polling, and Script tag long polling (or, JSONP long polling).
Let us understand how these work, as they continue to remain the most commonly available mechanisms across all modern browsers.
The first and the easiest to implement is XHR polling, in which the browser keeps polling for data periodically, and the server keeps responding with an empty response unless it has data to send back to the browser. Following an event, such as receiving a mail, or creating/updating a record in the database, the server responds to the next polling request with new data. The following figure depicts this mechanism:
XHR polling
As you can see, there is a problem with this. The browser has to keep making requests to the server even when there is no data. This causes the server to get and process data even when there is nothing to deliver.
One of the solutions to this is to modify the server to piggyback the actual client requests by not only sending the data requested by the client, but also appending additional data that the server has, to send to the browser. The client needs to be modified to understand and act upon the additional incoming data. The HTTP piggybacking process is shown in the following figure:
HTTP piggybacking
As the new data is only sent when there is a client action, it causes delays in the data reaching the browser. The solution to receiving events quickly while avoiding frequent server queries is long polling.
In long polling, when the browser sends a request to the server, the server won't respond immediately if it doesn't have data to respond with, and will suspend the request. Once the event occurs, the server closes the suspended request by sending over a response to the client. As soon as the client receives the response, it sends a new request:
Long Polling
There are various ways in which long polling is implemented, such as forever iframe, multipart XHR, script tags with JSONP, and long-living XHR.
Though all these techniques work, these are hacks, bending HTTP and XHR to be able to do duplex communication, which is not what they are meant for.
With the rapid evolution of the web browsers lead by Firefox and then Chrome, the long-due upgrade to HTML, called HTML5, is being widely adopted. In HTML5, there are two new methods for pushing data from the server to the client. One is Server-Sent Events (SSE) and the other is the full duplex WebSockets.
Server-Sent Events attempts to standardize Comet-like communication across browsers. In this approach, there is a JavaScript API to create an event source, that is, a stream over which the server can send events. This is a unidirectional protocol. We will still be using the good old XHR. This is a good approach when you don't need full duplex communication; just push updates from the server to client.
The other specification which goes on to implement a full duplex communication protocol for the web applications is WebSockets. In WebSockets, the client initiates a socket connection with the server, which supports this protocol as well. The server and client will send and receive data on this socket connection.
Let us take a quick look at how real-time web is changing the applications we come across on the Web daily.
With the success of Zynga and other social gaming companies, online gaming has become a hot trend. WordSquared is a massively parallel online multiplayer crossword, while BrowserQuest is an attempt (by Mozilla) at building an in-browser real-time role-playing game. One of the more popular and publicized games built on socket.io is Rawkets. There are many open source game engines built over canvas and around real-time communication systems.
Twitter is the best example of getting real-time data (the tweets) to the browser without user action. Google+ and Facebook have it too. The important thing on social networks is, being updated about happenings in real time.
CRMs are some of the most important components in business acquisitions. The days of issue tracking systems being sold as CRMs are over. CRMs are continuously improving and re-inventing themselves. Most of the CRMs are adding social capabilities; they are adding more functionality everyday. Salesforce, one of the most popular hosted CRM solutions, introduced Chatter. Chatter adds social capabilities to CRM and brings in a lot of advantages powered by realtime updates. It allows the customers to add comments or post updates on issues, which appear in real time to the support associates on their system. BPM solutions are also integrating real-time components to keep a track on process status and updates.
The latest updates to Google Analytics include a functionality to see the real-time updates of users visiting your website. Splunk—an event-tracking system, which is widely used to monitor events on infrastructure and machine data—allows you monitor and track event updates on the charts, updated in real time.
In the chapter, we saw what real-time web looks like, what its applications are, and how the technologies around real-time web evolved over a decade of development. In the next chapter we will get acquainted with Node.js, the JavaScript web application development platform which is the primary target of socket.io.