Summary

In this chapter, we have examined complex, unstructured data. We cleaned and tokenized text and examined several ways of extracting features from documents in a way that could be incorporated into predictive models such as n-grams and tf-idf scores. We also examined dimensionality reduction techniques, such as the HashingVectorizer, matrix decompositions, such as PCA, CUR, NMF, and probabilistic models, such as LDA. We also examined image data, including normalization and thresholding operations, and how we can use dimensionality reduction techniques to find common patterns among images. Finally, we used a matrix factorization algorithm to prototype a recommender system in PySpark.

In the next section, you will also look at image data, but in a different context: trying to capture complex features from these data using sophisticated deep learning models.