Building your Own Chatbot

Chatbots, better referred to as conversation software, are amazing tools for a lot of businesses. They help businesses serve their client's server 24/7 without increasing effort, with consistent quality, and the built-in option to defer to a human when bots are not enough.

They are a great example of where technology and AI has come together to improve the impact of human effort.

They range from voice-based solutions such as Alexa, to text-based Intercom chat boxes, to menu-based navigation in Uber.

A common misconception is that building chatbots needs large teams and a lot of machine learning expertise, though this is true if you are trying to build a generic chatbot platform like Microsoft or Facebook (or even Luis, Wit.ai, and so on).

In this chapter, we will cover the following topics:

• Why build a chatbot?
• Figuring out the right user intent...

Why chatbots as a learning example?

So far, we have built an application for every NLP topic that we have seen:

• Text cleaning using grammar and vocabulary insights
• Linguistics (and statistical parsers), to mine questions from text
• Entity recognition for information extraction
• Supervised text classification using both machine learning and deep learning
• Text similarity using text-based vectors such as GloVe/word2vec

We will now combine all of them into a much more complicated setup and write our own chatbot from scratch. But, before you build anything from scratch, you should ask why.

Why build a chatbot?

A related questions is why should we build our own chatbots? Why can't I use FB/MSFT/some other cloud service?

Perhaps...

Quick code means word vectors and heuristics

For the sake of simplicity, we will assume that our bot does not need to remember the context of any question. Therefore it sees input, responds to it, and is done. No links are established with the previous input.

Let's start by simply loading the word vectors using gensim:

import numpy as np
import gensim
print(f"Gensim version: {gensim.__version__}")

from tqdm import tqdm
class TqdmUpTo(tqdm):
    def update_to(self, b=1, bsize=1, tsize=None):
        if tsize is not None: self.total = tsize
        self.update(b * bsize - self.n)

def get_data(url, filename):
    """
    Download data if the filename does not exist already
    Uses Tqdm to show download progress
    """
    import os
    from urllib.request import urlretrieve
    
    if not os.path.exists(filename):

        dirname = os.path.dirname(filename...

Summary

In this chapter on chatbots, we learned about intent, which usually refers to the user input, response, which is via the bot, templates, which defines the nature of bot responses, and entities, such as cuisine type, in our example.

Additionally, to understand the user intent—and even find entities—we used unsupervised approaches , that is, we did not have training examples this time. In practice, most commercial systems use a hybrid system, combining supervised and unsupervised systems.

The one thing you should take away from here is that we don't need a lot of training data to make the first usable version of a bot for a specific use case.