Preface

Web scraping is used to scrape and collect data from the web. The data collected from scraping is used to generate and identify patterns in the information.

In today’s technical – or, more precisely, data-driven – fields and markets, quick and reliable information is in very high demand. Data collected in CSV or JSON format and from databases is processed to generate error-free and high-quality data, which is then analyzed, trained using machine learning algorithms, and plotted. The resulting information is carried forward for decision-making or supportive business intelligence-related tasks.

The chapters of this book are designed in such a way that each section helps you to understand certain important concepts and practical experiences. If you complete all the chapters of the book, then you will gain practice in scraping data from desired websites and analyzing and reporting data. You will also learn about the career paths and jobs related to web scraping, data analysis, reporting, visualization, and machine learning.

Who this book is for

This book is for Python programmers, data analysts, data reporters, machine learning practitioners, and anyone who wants to begin their professional or learning journey in the field of web scraping and data science. If you have a basic understanding of the Python programming language, you will easily be able to follow along with the book and learn about some advanced concepts related to data.

What this book covers

Chapter 1, Web Scraping Fundamentals, provides an introduction to web scraping and also explains the latest core web technologies and data-finding techniques.

Chapter 2, Python Programming for Data and Web, provides an overview of choosing and using Python for web scraping. The chapter also explores and explains the World Wide Web (WWW) and URL-based operations by setting up and using the necessary Python libraries, tools, and virtual environments.

Chapter 3, Searching and Processing Web Documents, provides an overview of and introduction to identifying, traversing, and processing web documents using XPath and CSS selectors. The chapter also explains scraping using lxml, collecting data in a file, parsing information from robots.txt, and exploring sitemaps.

Chapter 4, Scraping Using Pyquery, a jQuery-Like Library for Python, provides an introduction to a jQuery-like Python library: pyquery. This chapter provides information on installing and exploring pyquery’s features on web documents. Examples of scraping using pyquery and writing data to JSON and CSV are also covered.

Chapter 5, Scraping the Web with Scrapy and Beautiful Soup, provides an overview and examples of using and deploying a popular web-crawling framework: Scrapy. It also introduces parsing and scraping using BeautifulSoup.

Chapter 6, Working with the Secure Web, provides an overview of dealing with secure web content, using sessions and cookies. The chapter also guides you through and explores scraping content by processing HTML form- and authentication-related issues, as well as providing a guide with examples of how to use proxies during HTTP communication.

Chapter 7, Data Extraction Using Web APIs, provides a detailed overview of the web API, its benefits when used with HTTP content, along with the data formats and patterns available in the API. The chapter also provides a few examples of scraping the web API.

Chapter 8, Using Selenium to Scrape the Web, introduces Selenium WebDriver, which helps automate actions in web browsers, and also covers how to use Selenium to scrape data.

Chapter 9, Using Regular Expressions and PDFs, provides a detailed overview of regular expressions and their usage and implementation using Python. The chapter also provides examples of data extraction using regular expressions and PDF documents using the pypdf2 Python library.

Chapter 10, Data Mining, Analysis, and Visualization, provides an introduction to and detailed overview of data mining and data analysis using the pandas Python library and visualization using Plotly. The chapter also introduces the concept of exploratory data analysis using the ydata_profiling Python library.

Chapter 11, Machine Learning and Web Scraping, provides a detailed introduction to machine learning, a branch of artificial intelligence. The chapter also provides examples of a few machine learning topics using the scikit-learn Python library, along with conducting sentiment analysis from scraped and collected data.

Chapter 12, After Scraping – Next Steps and Data Analysis, provides an overview of and introduction to the next steps related to growing technologies, covering topics such as web requests and data processing in more detail. The chapter also provides information on and guides developers in exploring prospective careers and jobs relating to scraping and data.

To get the most out of this book

Having some basic or intermediate knowledge of Python programming will help you get the most out of the book’s contents and examples.

Software/hardware covered in the book	Operating system requirements
Python 3.11	Windows, macOS, or Linux
Google Chrome or Mozilla Firefox	Windows, macOS, or Linux
Visual Studio Code or JetBrains PyCharm Community Edition	Windows, macOS, or Linux

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Hands-On-Web-Scraping-with-Python-Second-Edition. If there’s an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “Here, we have expressed our interest in only finding the information from the content attribute with the <meta> tag, which has the name attribute with the keywords and description values, respectively.”

A block of code is set as follows:

source.find(‘a:contains(“Web”)’) # [<a.menuitm>, <a>, <a>, <a>, <a>]
source.find(‘a:contains(“Web”):last’).text()
# ‘Web Scraper’
source.find(‘a:contains(“Web”):last’).attr(‘href’) # ‘#’

Any command-line input or output is written as follows:

(secondEd) C:\HOWScraping2E> pip install jupyterlab

Outputs and comments inside blocks of code look as follows:

# [<a.menuitm>, <a>, <a>, <a>, <a>]
# ‘Web Scraper’
# ‘#’

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “In addition, we will create separate files for author and quotes.”

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [e-mail] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [e-mail] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read Hands-On Web Scraping with Python Second Edition, we’d love to hear your thoughts! Please https://packt.link/r/1837636214 for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere? Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

1. Scan the QR code or visit the link below

https://packt.link/free-ebook/9781837636211

2. Submit your proof of purchase
3. That’s it! We’ll send your free PDF and other benefits to your email directly