In this chapter, we will cover:
The key aspects for effective scraping are understanding how content and data are stored on web servers, identifying the data you want to retrieve, and understanding how the tools support this extraction. In this chapter, we will discuss website structures and the DOM, introduce techniques to parse, and query websites with lxml, XPath, and CSS. We will also look at how to work with websites developed in other languages and different encoding types such as Unicode.
Ultimately, understanding how to find and extract data within an HTML document comes down to understanding the structure of the HTML page, its representation in the DOM, the process of querying the DOM for specific elements, and how to specify which elements you want to retrieve based upon how the data is represented.
We can perform simple searches of the DOM using Beautiful Soup's find methods. These methods give us a much more flexible and powerful construct for finding elements that are not dependent upon the hierarchy of those elements. In this recipe we will examine several common uses of these functions to locate various elements in the DOM.
ff you want to cut and paste the following into ipython, you can find the samples in 02/02_bs4_find.py.
We will start with a fresh iPython session...
XPath is a query language for selecting nodes from an XML document and is a must-learn query language for anyone performing web scraping. XPath offers a number of benefits to its user over other model-based tools:
XPath contains seven data models (we have seen some of them previously):
XPath expressions can...
CSS selectors are patterns used for selecting elements and are often used to define the elements that styles should be applied to. They can also be used with lxml to select nodes in the DOM. CSS selectors are commonly used as they are more compact than XPath and generally can be more reusable in code. Examples of common selectors which may be used are as follows:
| What you are looking for | Example |
| All tags | * |
| A specific tag (that is, tr) | .planet |
| A class name (that is, "planet") | tr.planet |
| A tag with an ID "planet3" | tr#planet3 |
| A child tr of a table | table tr |
| A descendant tr of a table | table tr |
| A tag with an attribute (that is, tr with id="planet4") | a[id=Mars] |
Scrapy is a Python web spider framework that is used to extract data from websites. It provides many powerful features for navigating entire websites, such as the ability to follow links. One feature it provides is the ability to find data within a document using the DOM, and using the now, quite familiar, XPath.
In this recipe we will load the list of current questions on StackOverflow, and then parse this using a scrapy selector. Using that selector, we will extract the text of each question.
The code for this recipe is in 02/05_scrapy_selectors.py.
A document's encoding tells an application how the characters in the document are represented as bytes in the file. Essentially, the encoding specifies how many bits there are per character. In a standard ASCII document, all characters are 8 bits. HTML files are often encoded as 8 bits per character, but with the globalization of the internet, this is not always the case. Many HTML documents are encoded as 16-bit characters, or use a combination of 8- and 16-bit characters.
A particularly common form HTML document encoding is referred to as UTF-8. This is the encoding form that we will examine.
We will read a file named unicode.html from our local web server, located at http...
© 2018 Packt Publishing Limited All Rights Reserved
