Chapter 12. Bringing It All Together

While we have demonstrated many aspects of using Java to support data science tasks, the need to combine and use these techniques in an integrated manner exists. It is one thing to use the techniques in isolation and another to use them in a cohesive fashion. In this chapter, we will provide you with additional experience with these technologies and insights into how they can be used together.

Specifically, we will create a console-based application that analyzes tweets related to a user-defined topic. Using a console-based application allows us to focus on data-science-specific technologies and avoids having to choose a specific GUI technology that may not be relevant to us. It provides a common base from which a GUI implementation can be created if needed.

The application performs and illustrates the following high-level tasks:

• Data acquisition

• Data cleaning, including:

  • Removing stop words

  • Cleaning the text

    • Sentiment analysis

    • Basic data statistic collection...

Defining the purpose and scope of our application

The application will prompt the user for a set of selection criteria, which include topic and sub-topic areas, and the number of tweets to process. The analysis performed will simply compute and display the number of positive and negative tweets for a topic and sub-topic. We used a generic sentiment analysis model, which will affect the quality of the sentiment analysis. However, other models and more analysis can be added.

We will use a Java 8 stream to structure the processing of tweet data. It is a stream of TweetHandler objects, as we will describe shortly.

We use several classes in this application. They are summarized here:

• TweetHandler: This class holds the raw tweet text and specific fields needed for the processing including the actual tweet, username, and similar attributes.

• TwitterStream: This is used to acquire the application's data. Using a specific class separates the acquisition of the data from its processing. The class possesses...

Understanding the application's architecture

Every application has its own unique structure, or architecture. This architecture provides the overarching organization or framework for the application. For this application, we combine the three classes using a Java 8 stream in the ApplicationDriver class. This class consists of three methods:

• ApplicationDriver: Contains the applications' user input

• performAnalysis: Performs the analysis

• main: Creates the ApplicationDriver instance

The class structure is shown next. The three instance variables are used to control the processing:

public class ApplicationDriver { 
    private String topic; 
    private String subTopic; 
    private int numberOfTweets; 
 
    public ApplicationDriver() { ... } 
    public void performAnalysis() { ...     } 
 
    public static void main(String[] args) { 
        new ApplicationDriver(); 
    } 
} 

The ApplicationDriver constructor follows. A Scanner instance...

Data acquisition using Twitter

The Twitter API is used in conjunction with HBC's HTTP client to acquire tweets, as previously illustrated in the Handling Twitter section of Chapter 2, Data Acquisition. This process involves using the public stream API at the default access level to pull a sample of public tweets currently streaming on Twitter. We will refine the data based on user-selected keywords.

To begin, we declare the TwitterStream class. It consists of two instance variables, (numberOfTweets and topic), two constructors, and a stream method. The numberOfTweets variable contains the number of tweets to select and process, and topic allows the user to search for tweets related to a specific topic. We have set our default constructor to pull 100 tweets related to Star Wars:

public class TwitterStream { 
    private int numberOfTweets; 
    private String topic; 
 
    public TwitterStream() { 
        this(100, "Stars Wars"); 
    } 
 
    public...

Understanding the TweetHandler class

The TweetHandler class holds information about a specific tweet. It takes the raw JSON tweet and extracts those parts that are relevant to the application's needs. It also possesses the methods to process the tweet's text such as converting the text to lowercase and removing tweets that are not relevant. The first part of the class is shown next:

public class TweetHandler { 
    private String jsonText; 
    private String text; 
    private Date date; 
    private String language; 
    private String category; 
    private String userName; 
    ... 
    public TweetHandler processJSON() { ... } 
    public TweetHandler toLowerCase(){ ... } 
    public TweetHandler removeStopWords(){ ... }     
    public boolean isEnglish(){ ... }     
    public boolean containsCharacter(String character) { ... }        
    public void computeStats(){ ... } 
    public void buildSentimentAnalysisModel...

Other optional enhancements

There are numerous improvements that can be made to the application. Many of these are user preferences and others relate to improving the results of the application. A GUI interface would be useful in many situations. Among the user options, we may want add support for:

• Displaying individual tweets

• Allowing null sub-topics

• Processing other tweet fields

• Providing list of topics or sub-topics the user can choose from

• Generating additional statistics and supporting charts

With regard to process result improvements, the following should be considered:

• Correct user entries for misspelling

• Remove spacing around punctuation

• Use alternate stop word removal techniques

• Use alternate sentiment analysis techniques

The details of many of these enhancements are dependent on the GUI interface used and the purpose and scope of the application.

Summary

The intent of this chapter was to illustrate how various data science tasks can be integrated into an application. We chose an application that processes tweets because it is a popular social medium and allows us to apply many of the techniques discussed in earlier chapters.

A simple console-based interface was used to avoid cluttering the discussion with specific but possibly irrelevant GUI details. The application prompted the user for a Twitter topic, a sub-topic, and the number of tweets to process. The analysis consisted of determining the sentiments of the tweets, with simple statistics regarding the positive or negative nature of the tweets.

The first step in the process was to build a sentiment model. We used LingPipe classes to build a model and perform the analysis. A Java 8 stream was used and supported a fluent style of programming where the individual processing steps could be easily added and removed.

Once the stream was created, the JSON raw text was processed and used...