Reader small image

You're reading from  Hands-On Web Scraping with Python - Second Edition

Product typeBook
Published inOct 2023
PublisherPackt
ISBN-139781837636211
Edition2nd Edition
Right arrow
Author (1)
Anish Chapagain
Anish Chapagain
author image
Anish Chapagain

Anish Chapagain is a software engineer with a passion for data science, its processes, and Python programming, which began around 2007. He has been working with web scraping and analysis-related tasks for more than 5 years, and is currently pursuing freelance projects in the web scraping domain. Anish previously worked as a trainer, web/software developer, and as a banker, where he was exposed to data and gained further insights into topics including data analysis, visualization, data mining, information processing, and knowledge discovery. He has an MSc in computer systems from Bangor University (University of Wales), United Kingdom, and an Executive MBA from Himalayan Whitehouse International College, Kathmandu, Nepal.
Read more about Anish Chapagain

Right arrow

URL handling and operations

In this section, we will explore the operations that are required when handling URLs. In the browser, we input a URL as a request and receive an output or a response, but plenty of operations take place behind the scenes, and we can view these operations using browser-based DevTools.

We mentioned DevTools in the Developer tools section in Chapter 1, when we discussed the role of certain panels, such as the Network panel, found in DevTools. As we are diving deep into using libraries for HTTP-based communication and creating or dealing with code, it is quite important to deal with or monitor the HTTP information found in the Network panel while accessing the URL in the browser.

The information found in the different sections of the Network panel, such as Request URL, Request Headers, Request Method, Response Headers, Status Code, and Cookies, are important in the sense that we as developers are trying to automate, verify, inject, or use that information...

lock icon
The rest of the page is locked
Previous PageNext Page
You have been reading a chapter from
Hands-On Web Scraping with Python - Second Edition
Published in: Oct 2023Publisher: PacktISBN-13: 9781837636211

Author (1)

author image
Anish Chapagain

Anish Chapagain is a software engineer with a passion for data science, its processes, and Python programming, which began around 2007. He has been working with web scraping and analysis-related tasks for more than 5 years, and is currently pursuing freelance projects in the web scraping domain. Anish previously worked as a trainer, web/software developer, and as a banker, where he was exposed to data and gained further insights into topics including data analysis, visualization, data mining, information processing, and knowledge discovery. He has an MSc in computer systems from Bangor University (University of Wales), United Kingdom, and an Executive MBA from Himalayan Whitehouse International College, Kathmandu, Nepal.
Read more about Anish Chapagain