Performing exploratory data analysis
During the EDA phase in Chapter 2, Detecting Spam Emails, we saw how word clouds could provide some basic intuition on text data by identifying the most frequent words in a document. Another primary concern during EDA is to verify that the dataset is appropriately formatted before resorting to the subsequent analysis. For instance, it is not uncommon to encounter missing or out-of-the-range values. Plotting the data or extracting various statistics can reveal this unpleasant situation. Other times, we need to transform or exclude part of the data. Having an imbalanced dataset where one class monopolizes the whole corpus is also a source of concern. In this case, the ML algorithm is overexposed and subsequently learns data of one class type well while having difficulty with samples from the less frequent classes. All the previous issues must be addressed early to avoid any nasty surprises when treating the data later in the pipeline.
In the following...
