Data Extraction Using Web APIs

So far, we have learned about the web, web-based technologies, techniques for locating and extracting data, and plenty of data-related services in various chapters of this book.

Web application programming interfaces (APIs) are built to interact with queried information. Web APIs provide interfaces to query and access formatted data (normally in JSON format) or structured data that is easy to use and process. Depending on the type of information (HTTP POST payload, JSON content, third-party authentication, collective metadata, or middleware, for example) being carried and the type of service being engaged with, authentication and some form of tokenization are also required.

In general, a web API is a medium for loading, seeking, transferring, and embedding information in pages or other APIs.

In this chapter, we will learn about the following topics:

• Introduction to web APIs
• Data formats and patterns in APIs
• Web scraping using...

Technical requirements

A web browser (Google Chrome or Mozilla Firefox) will be required, and we will be using JupyterLab for our Python code.

Please refer to the Setting things up and Creating a virtual environment sections in Chapter 2 to continue with setting up and using the environment we created.

The Python libraries that are required for this chapter are as follows:

• requests
• csv

The code files for this chapter are available online in this book’s GitHub repository: https://github.com/PacktPublishing/Hands-On-Web-Scraping-with-Python-Second-Edition/tree/main/Chapter07.

Introduction to web APIs

Information sharing and processing are in huge demand on the web and across web-based services. A web API is a service available on the web that provides access to resources such as raw data, filtered information, and embedded and dynamic content, normally in a ready-to-use and exchangeable format such as JSON, CSV, or XML.

Information exchange or transfer is the core purpose of web APIs. We can consider an API as a web- or browser-based service or user interface (UI) for retrieving, sharing, or accessing information based on requests made to a web server or API provider. With plenty of security-related vulnerabilities, many providers ask for authentication in the form of an API key (a unique block of identification with a certain length that identifies the user or client with the API or server) or some code to make API processing more efficient and effective.

Web APIs are not dependent on any single programming language (such as ASP, PHP, or JSP); users...

Data formats and patterns in APIs

Data available through APIs might be different from what you expected or might not fit the plan exactly – in the Benefits of web APIs section, we covered a few of the ways the data might be different (limitation, irrelevance, and more).

Acquiring data from APIs is a straightforward process with or without API authorization. Content received via an API may appear in many formats, such as key names, nested lists and dictionaries, named or numerical indexing blocks, and many more. We generally find API content in JSON format, comprising Python lists and dictionaries.

Before moving on to the data formats, patterns, and results of APIs or API content, it is important to demonstrate how APIs are called or used. Most of the time, API service providers keep an updated version of their APIs in their documentation. Listed here are a few examples that web users normally use to access data from an API (example URL: exampledomain.com):

• http...

Web scraping using APIs

Technically, obtaining data from APIs is easy, and is also different from the web scraping scenarios. Most of the data is in JSON format, and there is no use of XPath, CSS Selector, or any other parsing libraries. Some data found using APIs might contain a lot of HTML code. To deal with this content, XPath and CSS Selector might be required.

Important note

In the examples in this section, we have tried to omit APIs that require an API key (walmartlabs.com, nasa.gov, nytimes.com, maps.googleapis.com) or user authentication tools. An API key is an authenticated value provided by service providers (after user registration) that identifies the user using the API. Freely available APIs are being used in these examples.

With the help of a few examples, we will collect data returned via some APIs. This collected data will then be exported as JSON and CSV files.

Example 1 – holidays from the US calendar

https://date.nager.at/ contains information...

Summary

Web APIs provide many benefits for finding and scraping data. Because of the predefined format of data returned by APIs, you don’t need to perform additional query-related tasks using XPath or CSS selectors, which saves time. Web APIs are structured, easily accessible, filterable, and ready to exchange data between machines, platforms, and protocols.

In the next chapter, we will be learning about automation using Selenium, and we will use it to scrape data.

Further reading

• APIs:
  • https://restfulapi.net/
  • https://developer.mozilla.org/en-US/docs/Web/API
  • https://auth0.com/blog/developing-restful-apis-with-python-and-flask/
• JSON:
  • https://www.json.org/json-en.html,
  • https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Objects/JSON
• HTTP methods: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods
• SOAP: https://www.soapui.org/learn/api/soap-vs-rest-api/
• REST:
  • https://www.ibm.com/topics/rest-apis,
  • https://pythonbasics.org/flask-rest-api/