Word Representations in FastText

Now that you have taken a look at creating models in the command line, you might be wondering how fastText creates those word representations. In this chapter, you will get to know what happens behind the scenes and the algorithms that power fastText.

We will cover the following topics in this chapter:

• Word-to-vector representations
• Types of word representations
• Getting vector representations from text
• Model architecture in fastText
• The unsupervised model
• fastText skipgram implementation
• CBOW (Continuous bag of words)
• Comparison between skipgram and CBOW
• Loss functions and optimizations
• Softmax
• Context definitions

Word-to-vector representations

Almost all machine learning and deep learning algorithms manipulate vectors and matrices. The reason they work is because of their base mathematics, which is heavily rooted in linear algebra. So, in short, for both supervised and unsupervised learning, you will need to create matrices of numbers. In other domains, this is not an issue as information is generally captured as numbers. For example, in retail, the sales information for how many units were sold or how much revenue the store is making in the current month is all numbers. Even in a more abstract field such as computer vision, the image is always stored as pixel intensity of the three basic colors: red, green, and blue. 0 for a particular color means no intensity and 255 means the highest possible intensity for the screen. Similarly, in the case of sound, it is stored as power spectral density...

Summary

In this chapter, you have taken a look at unsupervised learning in fastText, as well as the algorithms and methods that enable it.

The next chapter will be about how fastText has approached supervised learning and you will also learn about how model quantization works in fastText.