Chapter 4. Building a Product Recommendation System
The digital world has made everything available at the click of a button. With everything going the online way, online shopping or e-commerce has become a big thing. From groceries to electronics to even cars, everything is available at the Amazon, Flipkart, and eBay of the world. This ever expanding digital market is just the right place for data science to show its magic.
The online revolution of e-commerce has not only empowered the customers, it has also overwhelmed them with too many choices. Choices are not only in terms of products or categories but also between different e-commerce platforms. Being an ecommerce company in this highly competitive market can be really difficult. Standing out is a challenge and that is where data yet again comes to the rescue.
As we saw in Chapter 3, Predicting Customer Shopping Trends with Market Basket Analysis, purchase patterns can provide a lot of insights about shopping behaviors. We utilized such...
Understanding recommendation systems
Every individual in unique, the way we do things is what defines us uniquely. We eat, walk, talk, and even shop in a very unique way. Since the focus of this chapter is e-commerce, we will focus mostly on our shopping behaviors. We will utilize each customer's unique behavior to provide a personalized shopping experience.
To accomplish the task of providing a personalized shopping experience, we need a system to understand and model our customers. Recommendation engines are the systems which learn about customer preferences, choices, and so on, to recommend new products which are closer to what the user might have purchased themselves, thus providing a personalized experience. The options presented by such systems would have a high probability of the customer purchasing them.
Let us try to formally define a recommendation system.
Recommendation systems (or recommender engines) are a class of information filtering systems which analyze the input data to predict...
Issues with recommendation systems
Recommender engines are affected mainly by the following two issues:
The sparsity problem: Recommender engines work upon user preferences (or ratings for different items, depending upon the application) to predict or recommend products. Usually the ratings are given on some chosen scale but the user may choose not to rate certain items which he/she hasn't bought or looked at. For such cases, the rating is blank or zero. Hence, the ratings matrix R has elements of the form:
For any real world application, such as an e-commerce platform, the size of such a ratings matrix is huge due to the large number of users and items available on the platform. Even though a lot of user related information is gathered on such a platform, the ratings matrix itself might still be pretty sparse, that is the matrix might have a many elements as blanks (or zeroes). This problem in general is termed the sparsity
problem. The sparsity problem renders the recommender engine's predictions...
Recommendation systems and collaborative filters share a long history. From the early days of primitive recommender engines which utilized specific categorizations with hard-coded results, to current sophisticated recommender engines on various e-commerce platforms, recommender engines have made use of collaborative filters throughout. They are not only easy to understand but are equally simple to implement. Let us take this opportunity to learn more about collaborative filters before we dive into implementation details.
Note
Fun Fact
Recommender engines surely outdate any known e-commerce platform! Grundy, a virtual librarian, was developed in 1979. It was a system for recommending books to users. It modeled the users based upon certain pre-defined stereotypes and recommended books from a known list for each such category.
Core concepts and definitions
Collaborative filters (denoted as CF henceforth) and recommender engines in general use certain terms and definitions to...
Building a recommender engine
As discussed in the previous section, collaborative filtering is a simple yet very effective approach for predicting and recommending items to users. If we look closely, the algorithms work on input data, which is nothing but a matrix representation of the user ratings for different products.
Bringing in a mathematical perspective into the picture, matrix factorization is a technique to manipulate matrices and identify latent or hidden features from the data represented in the matrix. Building on the same concept, let us use matrix factorization as the basis for predicting ratings for items which the user has not yet rated.
Matrix factorization refers to the identification of two or more matrices such that when these matrices are multiplied we get the original matrix. Matrix factorization, as mentioned earlier, can be used to discover latent features between two different kinds of entities. We will understand and use the concepts of matrix...
Production ready recommender engines
In this chapter so far, we have learnt about recommender engines in detail and even developed one from scratch (using matrix factorization). Through all this, it is clearly evident how widespread the application of such systems is.
E-commerce websites (or for that fact, any popular technology platform) out there today have tones of content to offer. Not only that, but the number of users is also huge. In such a scenario, where thousands of users are browsing/buying stuff simultaneously across the globe, providing recommendations to them is a task in itself. To complicate things even further, a good user experience (response times, for example) can create a big difference between two competitors. These are live examples of production systems handling millions of customers day in and day out.
Note
Fun Fact
Amazon.com is one of the biggest names in the e-commerce space with 244 million active customers. Imagine the amount of data being processed to provide recommendations...
In this chapter, we continued our pursuit of using machine learning in the field of e-commerce to enhance sales and overall user experience. The previous chapter had discussed recommendations based on transactional logs; in this chapter, we accounted for the human factor and looked into the recommendation engines based on user behavior.
We started off by understanding what recommendation systems and their classifications into user-based, content-based, and hybrid recommender systems. We touched on the problems associated with recommender engines in general. Then we dived deep into the specifics of collaborative filters and discussed the math around prediction and similarity measures. After getting our basics straight, we moved onto building a recommender engine of our own from scratch. We utilized matrix factorization to build a recommender engine step by step using a small dummy dataset. We then moved onto building a production ready recommender engine using R's popular library called...
