Python Programming for Data and Web

In Chapter 1, you got an idea of what web scraping is, what core technologies exist, and how and where you can plan to find the resources or data you’re looking for.

Web scraping requires tools and techniques to be implemented and deployed using scripts or programs. We have chosen Python (https://www.python.org/) for this purpose, as it is very easy to learn and has a huge set of libraries for communicating with the World Wide Web (WWW), data-related processes, and finally, web scraping. In internet search results, we often find Python mentioned in conjunction with data science and Machine Learning (ML). This is because of the wide use of Python in such projects.

In this chapter, we will explore the key benefits of Python and communicate with web resources using Python libraries. This chapter will also provide a detailed overview of installing and using Python libraries such as requests and urllib.

In particular, we will learn about...

Technical requirements

We will be using Python 3.11.1 installed on Windows. There are plenty of choices for code editors; choose one that is convenient for you to use and can deal with the libraries used in this chapter’s code examples. We will be using Notebooks and JupyterLab from Jupyter (https://jupyter.org/), Integrated Development and Learning Environment (IDLE) (the default editor from Python), and Windows’ Command Prompt (cmd) side by side.

To follow along with this chapter, you will need to install the following applications:

• Python 3.11.* or the latest version appropriate for your OS from https://www.python.org/downloads/
• The pip Python package management tool (https://pypi.org/project/pip/)
• JupyterLab (https://jupyter.org/install) for accessing notebooks or files with the .ipynb extension
• Google Chrome, Mozilla Firefox, or any other web browser equipped with Developer Tools (DevTools)
• JetBrains PyCharm (https://www.jetbrains.com/pycharm...

Why Python (for web scraping)?

Python is a popular programming language that is used to code various types of applications, from simple scripts to software development, ML/AI algorithms, and CLI/GUI apps, and also to create web applications. Python’s simple Object-Oriented Programming (OOP) syntax is very readable, which allows developers to write code in not many lines and work on different platforms (which means Python is platform independent), which makes it the number-one choice among programming languages.

Python is a buzzword in today’s programming domain. Quite a lot of tools and web applications are produced and managed using Python, which is compatible with small and large apps and is supplied with adequate and up-to-date libraries by its global developer audience.

Figure 2.1 (taken from https://www.python.org) shows Success Stories and Use Python for… information for various types of applications:

Figure 2.1: Python success...

Accessing the WWW with Python

As we saw in the Why Python (for web scraping)? section, there are plenty of Python libraries for interacting with HTTP. requests and urllib are the two libraries that we are interested in using because of their in-depth features, various functions for dealing with HTTP communication, easy-to-read documentation, and popularity.

In order to start accessing the WWW with Python using these libraries, let’s verify that we have installed all of the required resources. In the following subsections, we will start setting things up, such as installing Python, creating a virtual environment, installing libraries in the created environment, and accessing the web using Python libraries.

Setting things up

It is assumed that the latest version of Python has been installed on your system. If not, please visit https://www.python.org/downloads/ for the latest version of Python for your OS. Regarding the general setup and installation procedure, please...

URL handling and operations

In this section, we will explore the operations that are required when handling URLs. In the browser, we input a URL as a request and receive an output or a response, but plenty of operations take place behind the scenes, and we can view these operations using browser-based DevTools.

We mentioned DevTools in the Developer tools section in Chapter 1, when we discussed the role of certain panels, such as the Network panel, found in DevTools. As we are diving deep into using libraries for HTTP-based communication and creating or dealing with code, it is quite important to deal with or monitor the HTTP information found in the Network panel while accessing the URL in the browser.

The information found in the different sections of the Network panel, such as Request URL, Request Headers, Request Method, Response Headers, Status Code, and Cookies, are important in the sense that we as developers are trying to automate, verify, inject, or use that information...

Implementing HTTP methods

Generally, web-based interaction or communication between websites and users is achieved as follows:

1. The user accesses a web page or navigates through the content that is available to them.
2. The user then submits information to the website through an HTML form, by searching, logging in, registering themselves, and so on, and finally receiving the content they asked for.

In this section, we will be using the Python requests library to implement HTTP methods fitting the scenarios we just listed.

GET

The HTTP GET method is the default HTTP method. If no HTTP method is defined, then GET will be used by the code. We used some code earlier in this chapter that used GET. We were using the GET method unknowingly and without declaring it in the code.

By using GET, the resource’s state is not altered, so it is the default and safest method. GET parameters, also known as query strings, are visible in the URL. They are appended to the...

Summary

In this chapter, we have learned about Python programming, setting up a virtual environment, and installing Python libraries to send requests to web resources and collect responses and some additional information.

The main objective of this chapter was to demonstrate the core features of the requests library. Our primary aims were to learn how to deal with the HTTP request and response cycle, how to use HTTP methods (with extra parameters), how to use DevTools, and what the benefits are of using Python in this domain.

In the next chapter, we will learn and use some essential techniques to identify and extract data from web content.

Further reading

• Python programming:
  • https://www.python.org/doc/
  • https://www.w3schools.com/python/default.asp
• Data and data analysis:
  • https://www.diesel-plus.com/the-importance-of-data-collection-10-reasons-why-data-is-so-important/
  • https://www.simplilearn.com/data-analysis-methods-process-types-article
  • https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8274472/
  • https://scientific-publishing.webshop.elsevier.com/research-process/when-data-speak-listen-importance-of-data-collection-and-analysis-methods/
• requests, urllib, JSON:
  • https://requests.readthedocs.io/en/latest/
  • https://pymotw.com/3/
• Virtual environment:
  • https://docs.python.org/3/library/venv.html
  • https://docs.python-guide.org/dev/virtualenvs/
  • https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html
  • https://docs.anaconda.com/free/navigator/tutorials/manage-environments/
• Jupyter Notebooks:
  • https://jupyter.org/
  • https://code.visualstudio.com/docs/datascience/jupyter-notebooks
• Cookies:
  • https://www.cloudflare...