Scraping Using PyQuery, a jQuery-Like Library for Python

In the previous chapters, you learned about the basics of web scraping, the technologies involved, data-finding techniques, and traversing markup documents for data with the help of some Python code.

Web scraping is a handful of tasks that involve reverse engineering techniques. It entails exploring a website (examining its content-related structure, DevTools, paginations, and more), preparing or installing tools (libraries and so on), coding and testing, and finally, collecting data in files, clouds, databases, and many more places. In this chapter, we will be learning about web scraping using the PyQuery Python library; this library assists us in the use of concepts such as XPath, CSS selectors, and parsing markup documents. PyQuery provides a jQuery-like ability to write less, do more, which is very significant when coding web scrapers.

In particular, this chapter will cover the following topics:

• PyQuery overview...

Technical requirements

A web browser (Google Chrome or Mozilla Firefox) will be required, and we will be using Jupyter notebooks for code using JupyterLab.

Please refer to the Creating a virtual environment section in Chapter 2 to continue setting up and using the created environment.

The Python libraries that are required for this chapter are as follows:

• pyquery
• urllib
• requests

The code files for this chapter are available online on GitHub: https://github.com/PacktPublishing/Hands-On-Web-Scraping-with-Python-Second-Edition/tree/main/Chapter04

PyQuery overview

PyQuery is a jQuery-like library for Python that facilitates the easy implementation and use of lxml and CSS selectors.

As the name suggests, PyQuery enhances query-related procedures (XPath and CSS selector expressions) with short and readable lines of code. Web scraping, as you all are aware, requires parsing and traversing features that reside on top of various types of web documents.

PyQuery provides additional features related to DOM and ElementTree, and uses CSS selectors to perform queries. The purpose of using PyQuery expressions (or queries) is similar to that of XPath or CSS selector-based expressions. PyQuery is almost the same as jQuery for web documents.

The following list contains a basic comparison of expressions that collect the href attribute from the <a class="main"> element:

• PyQuery: response.find('a.main').attr('href')
• lxml XPath: response.xpath(".//a[contains(@class,'main'...

Exploring PyQuery

PyQuery addresses DOM-based expressions or queries in a quick, easy, and effective manner. In this section, we will install PyQuery and explore a few of its important features so that we are ready to develop a web scraping script.

Important note

You can find plenty of Python libraries that are similar to PyQuery. A few examples are parsel (https://pypi.org/project/parsel/), beautifulsoup (https://pypi.org/project/beautifulsoup4/), selectolax (https://pypi.org/project/selectolax/), and, of course, lxml (https://pypi.org/project/lxml/).

Installing PyQuery

Please refer to the Technical requirements section before proceeding with installing the PyQuery library. With the help of the virtual environment (secondEd) you created in Chapter 2, install or update PyQuery using pip:

Figure 4.1: Installing PyQuery

As shown in Figure 4.1, installing PyQuery also installs or updates the lxml and cssselect libraries. To verify the installation...

Web scraping using PyQuery

This section is loaded with examples that explain how to code a web scraping script. We will be using the PyQuery and requests libraries and dealing with HTML and XML content from the web. We will write the data collected in these examples to CSV or JSON files.

Example 1 – scraping book details

In this example, we will be scraping books listed on http://books.toscrape.com in the Childrens category (http://books.toscrape.com/catalogue/category/books/childrens_11/index.html). This example is similar to another provided in Chapter 3, where the lxml library was used. The code for this example is available on GitHub: https://github.com/PacktPublishing/Hands-On-Web-Scraping-with-Python-Second-Edition/blob/main/Chapter04/example_1.ipynb.

As you can see in Figure 4.3, the Childrens category contains 29 results (a single page shows only 20 results):

Figure 4.3: Childrens category page with 29 results

Important note

...

Summary

In this chapter, we explored and learned about PyQuery, applying various scraping techniques with examples. Short, simple code that is easy to deal with is always in demand, and PyQuery provides and assists with this. PyQuery supports CSS selectors and its applicability across various markup documents is one of its major advantages.

In the next chapter, we will learn more about web scraping techniques and some new Python libraries.

Further reading

• PyQuery, a jQuery-like library for Python: https://pyquery.readthedocs.io/en/latest/
• jQuery: https://jquery.com/
• CSS:
  • https://www.w3.org/Style/CSS/Overview.en.html
  • https://developer.mozilla.org/en-US/docs/Web/CSS
• CSS selectors:
  • https://www.w3schools.com/css/css_selectors.asp
  • https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors
• JSON: https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Objects/JSON
• Sitemaps: https://sitemaps.org/