Machine Learning and Web Scraping

So far, we have learned about data extraction, data storage, and acquiring and analyzing information from data by using a number of Python libraries. This chapter will provide you with introductory information on Machine Learning (ML) with a few examples.

Web scraping involves studying a website, identifying collectible data elements, and planning and processing a script to extract and collect data in datasets or files. This collected data will be cleaned and processed further to generate information or valuable insights. ML is a branch of Artificial Intelligence (AI) and generally deals with statistical and mathematical processes. ML is used to develop, train, and evaluate algorithms that can be automated, keep learning from the outputs, and minimize human intervention.

ML uses data to learn, predict, classify, and test situations, and for many other functions. Data is collected using web scraping techniques, so there is a correlation between...

Technical requirements

A web browser (Google Chrome or Mozilla Firefox) will be required and we will be using JupyterLab for Python code.

Please refer to the Setting things up and Creating a virtual environment sections of Chapter 2 to continue with setting up and using the environment created.

The Python libraries that are required for this chapter are as follows:

• scikit-learn (visit https://scikit-learn.org/stable/install.html for installation)
• textblob (visit https://textblob.readthedocs.io/en/dev/install.html for installation)
• vaderSentiment
• plotly
• numpy
• pandas
• matplotlib

The code files for this chapter are available online in this book’s GitHub repository: https://github.com/PacktPublishing/Hands-On-Web-Scraping-with-Python-Second-Edition/tree/main/Chapter11.

Introduction to ML

Data collection, analysis, and the mining of data to extract information are major agendas of many data-related systems. Processing, analyzing, and executing mining-related functions requires processing time, evaluation, and interpretation to reach the desired state. Using ML, systems can be trained on relevant or sample data and ML can be further used to evaluate and interpret other data or datasets for the final output.

ML-based processing is implemented similarly to and can be compared to data mining and predictive modeling, for example, classifying emails in an inbox as spam and not spam. Spam detection is a kind of decision-making to classify emails according to their content. A system or spam-detecting algorithm is trained on inputs or datasets and can distinguish emails as spam or not.

ML predictions and decision-making models are dependent on data. ML models can be built on top of, and also use, several algorithms, which allows the system to provide...

ML using scikit-learn

To develop a model, we need datasets. Web scraping is again the perfect technique to collect the desired data and store it in the relevant format. There are plenty of ML-related libraries and frameworks available in Python, and they are growing in number. scikit-learn is a Python library that addresses and helps to deal with the majority of supervised ML features.

scikit-learn is also known and used as sklearn. It is built upon numpy, scipy, and matplotlib. The library provides a large number of features related to ML aspects such as classification, clustering, regression, and preprocessing. We will explore beginner and intermediate concepts of the supervised learning type with regression using scikit-learn. You can also explore the sklearn user guide available at https://scikit-learn.org/stable/user_guide.html.

We have covered a lot of information about regression in previous sections of this chapter. Regression is a supervised learning technique that is...

Summary

Python programming makes a huge contribution in AI- and ML-related domains. In this chapter, we have had only a glimpse of that. Quality data plays a very important role in ML. Whether collecting data via web scraping and storing it or providing scraped data on the fly to an ML model, prepared data is in demand. The better the quality of the data – and the more precise the data is – that we provide to ML algorithms, and for plotting charts, the more accurate results, visualizations, and descriptive plots we can expect.

We have now learned about ML concepts and various aspects of ML by exploring them. We have also learned how to implement ML models and collect the results, if required, from various processes. To summarize, we now have an overview of how to use scikit-learn and conduct sentiment analysis. ML is data-driven and quality data is a basic requirement for ML models to provide accuracy.

In the next chapter, we will learn about a few further steps...

Further reading

• Machine learning:
  • https://www.ibm.com/topics/machine-learning
  • https://www.mygreatlearning.com/blog/what-is-machine-learning/
  • https://www.run.ai/guides/machine-learning-engineering/machine-learning-automation
• Machine learning – Python:
  • https://scikit-learn.org/
  • https://www.w3schools.com/python/python_ml_getting_started.asp
  • https://realpython.com/tutorials/machine-learning/
  • https://machinelearningmastery.com/machine-learning-in-python-step-by-step/
• Regression analysis:
  • https://www.alchemer.com/resources/blog/regression-analysis/
  • https://www.investopedia.com/terms/r/regression.asp
• Natural language processing:
  • https://www.nltk.org/
  • https://www.ibm.com/topics/natural-language-processing
  • https://hbr.org/2022/04/the-power-of-natural-language-processing
• Sentiment analysis:
  • https://monkeylearn.com/sentiment-analysis/
  • https://huggingface.co/blog/sentiment-analysis-python
  • https://aws.amazon.com/what-is/sentiment-analysis/