Chapter 15. Sentiment Analysis and ChatBots

By now, we are equipped with the skills needed to get started on text analysis projects and to also take a shot at more complicated, meatier projects. Two common text analysis projects that encapsulate a lot of the concepts we have explored throughout the book are sentiment analysis and chatbots. In fact, we've already touched upon all the methods we will be using for these projects, and this chapter will serve as a guide to how one can put up such an application on their own.

In this chapter, we will not be providing the code to how to build a chatbot or sentiment analysis pipeline from the first step to the last, but rather to introduce the reader to a variety of techniques which will help when setting up such a project. Following are the topics we will cover in this chapter:

• Sentiment analysis
• Mining data
• ChatBot

Sentiment analysis

Sentiment analysis is merely another term given to text classification or document classification – where the classifying feature happens to be the sentiment of the text. We can understand sentiment as a feeling or opinion about something – if we said The movie was terrific!, it means it expresses a positive sentiment or feeling, and if we say The movie is terrible!, it would be expressing negative sentiment or feeling. Here, sentiment usually refers to positive or negative sentiment, but this can, of course, be extended to include multiple sentiments, such as angry, sad, happy, and maybe even a thoughtful sentiment if we so wish. In other words, sentiment analysis tasks are simply classification tasks where each class is a kind of sentiment which we wish to analyze.

In fact, we have seen an example of sentiment analysis in the previous chapter, when we used Keras and spaCy together to build a deep learning pipeline. Sentiment analysis was performed by assigning probability...

ChatBots

Getting a machine to learn to speak like a human being has been a holy grail for computer scientists as well as linguists – of all the things which machines can mimic human behavior, holding a conversation has been a challenging ordeal. The quest to make such a machine which can chat with us humans (or a chatbot, if you will), has had many different approaches, and while none of them work perfectly, it is important to be aware of them - and pick and choose which kind will be the best for our purpose!

As for why we would want to build them – chatbots are increasingly used by businesses; both to help customers to answer basic questions, as well as for building more complex personal assistants. It is also becoming increasingly easier to build such chatbots, and using open source tools.

There are many motivations to study the art of conversation for machines, from both a research perspective and a more practical approach. An ideal artificial intelligence bot should be able to remember...

Summary

In this chapter, we discussed two important text analysis problems – sentiment analysis and building a chatbot. Sentiment analysis refers to the task of understanding sentiment in the text, and we have seen the various libraries, algorithms, and approaches to perform this task. A crucial part of performing such tasks is gathering data – we then saw how to download data from internet sources such as Twitter or Reddit. The final section of the chapter focused on how to build chatbots. We explored it from both a historical and theoretical point of view and explored Python libraries that help us easily build chatbots. This brings us to the end of the book – you would now be confident in analyzing text the way you see fit, with a variety of techniques, approaches, and settings. We focused on using the most efficient Python open source libraries, with a focus on Gensim, spaCy, Keras, and scikit-learn throughout the book, while still discussing the other Python text analysis libraries available...

References

[1] TextBlob: http://textblob.readthedocs.io/en/dev/

[2] NLTK Sentiment Analysis: http://www.nltk.org/howto/sentiment.html

[3] NLTK Sentiment Analyser class: http://www.nltk.org/_modules/nltk/sentiment/sentiment_analyzer.html

[4] LSTMs for Sentiment Analysis: http://deeplearning.net/tutorial/lstm.html

[5] Understanding LSTM Networks: http://colah.github.io/posts/2015-08-Understanding-LSTMs/

[6] Reddit: https://www.reddit.com/

[7] News subreddit: https://www.reddit.com/r/news/

[8] JSON: https://docs.python.org/3.6/library/json.html

[9] Time library: https://docs.python.org/3.6/library/time.html

[10] sense2vec:https://explosion.ai/demos/sense2vec

[11] Twitter datasets: http://thinknook.com/twitter-sentiment-analysis-training-corpus-dataset-2012-09-22/

[12] Tweepy:http://www.tweepy.org/

[13] Tweepy authentication tutorial: http://tweepy.readthedocs.io/en/v3.5.0/auth_tutorial.html#auth-tutorial

[14] Tweepy documentation: http://tweepy.readthedocs.io/en/v3.6.0/getting_started.html

[15] Turing Test...