Chapter 8. Sentiment Analysis of Twitter Data

	"He who molds the public sentiment... makes statutes and decisions possible or impossible to make."
	--Abraham Lincoln.

What people think matters not only to politicians and celebrities but also to most of us social beings. This need to know opinions about ourselves has affected people for a long time and is aptly summarized by the preceding famous quote. The opinion bug not only affects our own outlook, it affects the way we use products and services as well. As discussed while learning about market basket analysis and recommender engines (see Chapter 3, Predicting Customer Shopping Trends with Market Basket Analysis and Chapter 4, Building a Product Recommendation System respectively), our behavior can be approximated or predicted by observing the behavior of a group of people with similar characteristics such as price sensitivity, color preferences, brand loyalty, and so on. We also discussed in the earlier chapters that, for a long time,...

Understanding Sentiment Analysis

The fact that Internet-based companies and their CEOs feature as some of the most profitable entities in the global economy says a lot about how the world is being driven by technology and shaped by the Internet. Unlike any other medium, the Internet has become ubiquitous and has penetrated every aspect of our lives. It is no surprise that we are using and relying on the Internet and Internet-based solutions for advice and recommendations, apart from using it for many other purposes.

As we saw in the previous chapters, the relationship between the Internet and domains such as e-commerce and financial institutions goes way too deep. But our use of and trust in the online world doesn't stop there. Be it about booking a table at the new restaurant in your neighborhood or deciding which movie to see tonight, we take help from the Internet to know what opinions others have, or what others have to share, before we make the final call. As we will see later, such...

Sentiment analysis upon Tweets

Now that we are equipped with the key terms and concepts from the world of Sentiment Analysis, let us put our theory to the test. We have seen some major application areas for Sentiment Analysis and the challenges faced, in general, to perform such analytics. In this section we will perform Sentiment Analysis categorized into:

• Polarity analysis: This will involve the scoring and aggregation of sentiment polarity using a labeled list of positive and negative words.

• Classification-based analysis: In this approach we will make use of R's rich libraries to perform classification based on labeled tweets available for public usage. We will also discuss their performance and accuracy.

R has a very robust library for the extraction and manipulation of information from Twitter called TwitteR. As we saw in the previous chapter, we first need to create an application using Twitter's application management console before we can use TwitteR or any other library for sentiment...

Summary

Twitter is a goldmine for data science, with interesting patterns and insights spread all across it. Its constant flow of user-generated content, coupled with unique, interest-based relationships, present opportunities to understand human dynamics up close. Sentiments Analysis is one such field where Twitter provides the right set of ingredients to understand what and how we present and share opinions about products, brands, people, and so on.

Throughout this chapter, we have looked at the basics of Sentiment Analysis, key terms, and areas of application. We have also looked into the various challenges posed while performing sentiment analysis. We have looked at various commonly-used feature extraction methods such as tf-idf, Ngrams, POS, negation, and so on for performing sentiment analysis (or textual analysis in general). We have built on our code base from the previous chapter to streamline and structure utility functions for reuse. We have performed polarity analysis using Twitter...