Working with the Secure Web

So far, we have learned about the web, web content, reverse-engineering techniques, data-finding techniques, a few Python libraries, and a framework that we can employ to access and scrape the desired web content.

Plenty of security-related concerns exist on web platforms today, along with measures to ensure security. Lots of web applications, extensions, and even web-based service providers exist to protect us and our web-based systems against unauthenticated usage and unauthorized access.

The growing use of internet applications and e-commerce activity demands a secure web (or web-based security-enabled features) as a high priority to deal with actions that are harmful or even illegal. We often receive irrelevant emails or spam, containing information that we did not ask for.

It’s quite challenging from a web scraping perspective to deal with such issues, but the concept of ethical hacking makes it more viable, even from an application...

Technical requirements

A web browser (Google Chrome or Mozilla Firefox) will be required, and we will also use JupyterLab for Python code.

Please refer to the Setting things up and Creating a virtual environment sections in Chapter 2 to continue using the environment created.

The Python libraries that are required for this chapter are as follows:

• requests
• pyquery

The code files for this chapter are available online in this book’s GitHub repository: https://github.com/PacktPublishing/Hands-On-Web-Scraping-with-Python-Second-Edition/tree/main/Chapter06.

Exploring secure web content

Today’s web and internet technologies are quite vulnerable in terms of web security (content, authorization, illegal access, and so on). We want the web to be safe and the content that we browse, search, or view to be genuine, not violating any legal or ethical standards or affecting people’s human rights.

The web must be accessible and available to everyone who seeks information, effectively and by following ethical practices. We often encounter web content that is not exactly what we were looking for, or hear of web content that has been tampered with or hacked into or that private and sensitive information has been leaked illegally, and so on. Although a lot of these cases are beyond our control, we can reduce vulnerabilities and make the web a much safer place to be.

In many new technologies, security-related concerns have been identified and solutions have been implemented. There are even applications and organizations concerned...

HTML <form> processing using Python

Form processing has various titles and functions, such as search, filter, login, registration, submission, and verification. In this section, we will explore http://quotes.toscrape.com/search.aspx, as shown in Figure 6.2, and process or use the forms available on the page to extract or filter out the results, based on a choice of options (Author or Tag):

Figure 6.2: The search form (Author and Tag)

Important note

We need to investigate the available <form> tag, the options or form elements that exist, and the resulting pages as they appear on form submission. It’s also advisable to explore form-related actions using DevTools, as this will help you to understand the flow of systems, links, availability, and resources such as HTTP headers, cookies, and payload.

During page analysis, we found that only a single form, named filterform, exists, and options are available for the Author name only; Tag...

User authentication and cookies

User authentication (managing and handling user credentials) is another form of web security. In this section, we will use the user authentication feature available on the http://quotes.toscrape.com/login site, along with cookies.

As seen in Figure 6.6, http://quotes.toscrape.com/login contains a page with the Login text, and a form asking for Username and Password:

Figure 6.6: User login page

Analyzing the page source, there’s only a single <form> element found, with one hidden input element named csrf_token, two text input elements with type="text" (username and password), and finally, an object with the submit type and the value Login, as seen in Figure 6.7:

Figure 6.7: Login page – <form> source code

Important note

http://toscrape.com is a demo site, enriched with content for scraping-related purposes. The site does not provide a user registration...

Using proxies

A proxy (HTTP proxy or web proxy) is considered middleware for the web. A proxy is a gateway that is used to communicate between a client and a server. Put simply, a proxy is an Internet Protocol (IP) address with some random ports assigned to it. On the web, an HTTP request starts from the client end with its own IP, which, when routed through the proxy, gets updated as a new IP and is then forwarded to the actual destination looking for a response. So, a proxy here works as a transport layer that resides between the client and the server or destination.

On the internet, there are plenty of services (paid and free) offered by various organizations providing proxies that work as a filter layer for their customers and deal with different kinds of security-related threats, malware, content filtering, spy protection, and many other activities. There are plenty of benefits to using proxies; a few of them are listed here:

• Privacy: The destination server does not...

Summary

Web security is a compulsory component of the current internet revolution. We need to participate in web scraping and extraction processes ethically for the betterment of the information that’s available everywhere on the web. We have learned about the basics of processing with HTML forms, cookies, and sessions, and using proxies with the help of the Python programming language, from a web scraping perspective.

In the next chapter, we will use web-based APIs to collect relevant data.

Further reading

• Cookies and sessions:
  • https://www.kaspersky.com/resource-center/definitions/cookies
  • https://developer.mozilla.org/en-US/docs/web/HTTP/Cookies
  • https://www.ibm.com/docs/en/sva/9.0?topic=cookies-session-concepts
• HTTP headers:
  • https://jkorpela.fi/http.html
  • https://developer.mozilla.org/en-US/docs/web/HTTP/Headers
• HTML forms:
  • https://www.w3schools.com/html/html_forms.asp
  • https://developer.mozilla.org/en-US/docs/web/HTML/Element/form
  • https://www.w3.org/TR/html401/interact/forms.html
• User authentication:
  • https://www.sciencedirect.com/topics/computer-science/user-authentication
  • https://www.ibm.com/docs/en/aix/7.2?topic=passwords-user-authentication
• Proxy:
  • https://www.fortinet.com/resources/cyberglossary/proxy-server
  • https://smartproxy.com/proxies
  • https://www.upguard.com/blog/proxy-server
  • https://www.zyte.com/learn/use-proxies-for-web-scraping/