Python Web Scraping - Second Edition

Successfully scrape data from any website with the power of Python 3.x

Python Web Scraping - Second Edition

Katharine Jarmul, Richard Lawson

3 customer reviews
Successfully scrape data from any website with the power of Python 3.x
Mapt Subscription
FREE
$29.99/m after trial
eBook
$19.60
RRP $27.99
Save 29%
Print + eBook
$34.99
RRP $34.99
What do I get with a Mapt Pro subscription?
  • Unlimited access to all Packt’s 5,000+ eBooks and Videos
  • Early Access content, Progress Tracking, and Assessments
  • 1 Free eBook or Video to download and keep every month after trial
What do I get with an eBook?
  • Download this book in EPUB, PDF, MOBI formats
  • DRM FREE - read and interact with your content when you want, where you want, and how you want
  • Access this title in the Mapt reader
What do I get with Print & eBook?
  • Get a paperback copy of the book delivered to you
  • Download this book in EPUB, PDF, MOBI formats
  • DRM FREE - read and interact with your content when you want, where you want, and how you want
  • Access this title in the Mapt reader
What do I get with a Video?
  • Download this Video course in MP4 format
  • DRM FREE - read and interact with your content when you want, where you want, and how you want
  • Access this title in the Mapt reader
$0.00
$19.60
$34.99
$29.99p/m after trial
RRP $27.99
RRP $34.99
Subscription
eBook
Print + eBook
Start 30 Day Trial
Subscribe and access every Packt eBook & Video.
 
  • 5,000+ eBooks & Videos
  • 50+ New titles a month
  • 1 Free eBook/Video to keep every month
Start Free Trial
 
Preview in Mapt

Book Details

ISBN 139781786462589
Paperback220 pages

Book Description

The Internet contains the most useful set of data ever assembled, most of which is publicly accessible for free. However, this data is not easily usable. It is embedded within the structure and style of websites and needs to be carefully extracted. Web scraping is becoming increasingly useful as a means to gather and make sense of the wealth of information available online.

This book is the ultimate guide to using the latest features of Python 3.x to scrape data from websites. In the early chapters, you’ll see how to extract data from static web pages. You’ll learn to use caching with databases and files to save time and manage the load on servers. After covering the basics, you’ll get hands-on practice building a more sophisticated crawler using browsers, crawlers, and concurrent scrapers.

You’ll determine when and how to scrape data from a JavaScript-dependent website using PyQt and Selenium. You’ll get a better understanding of how to submit forms on complex websites protected by CAPTCHA. You’ll find out how to automate these actions with Python packages such as mechanize. You’ll also learn how to create class-based scrapers with Scrapy libraries and implement your learning on real websites.

By the end of the book, you will have explored testing websites with scrapers, remote scraping, best practices, working with images, and many other relevant topics.

Table of Contents

Chapter 1: Introduction to Web Scraping
When is web scraping useful?
Is web scraping legal?
Python 3
Background research
Crawling your first website
Summary
Chapter 2: Scraping the Data
Analyzing a web page
Three approaches to scrape a web page
CSS selectors and your Browser Console
XPath Selectors
LXML and Family Trees
Comparing performance
Scraping results
Summary
Chapter 3: Caching Downloads
When to use caching?
Adding cache support to the link crawler
Disk Cache
Key-value storage cache
Summary
Chapter 4: Concurrent Downloading
One million web pages
Sequential crawler
Threaded crawler
How threads and processes work
Performance
Chapter 5: Dynamic Content
An example dynamic web page
Reverse engineering a dynamic web page
Rendering a dynamic web page
The Render class
Summary
Chapter 6: Interacting with Forms
The Login form
Extending the login script to update content
Automating forms with Selenium
Summary
Chapter 7: Solving CAPTCHA
Registering an account
Optical character recognition
Solving complex CAPTCHAs
Using a CAPTCHA solving service
CAPTCHAs and machine learning
Summary
Chapter 8: Scrapy
Installing Scrapy
Starting a project
Different Spider Types
Scraping with the shell command
Visual scraping with Portia
Automated scraping with Scrapely
Summary
Chapter 9: Putting It All Together
Google search engine
Facebook
Gap
BMW
Summary

What You Will Learn

  • Extract data from web pages with simple Python programming
  • Build a concurrent crawler to process web pages in parallel
  • Follow links to crawl a website
  • Extract features from the HTML
  • Cache downloaded HTML for reuse
  • Compare concurrent models to determine the fastest crawler
  • Find out how to parse JavaScript-dependent websites
  • Interact with forms and sessions

Authors

Table of Contents

Chapter 1: Introduction to Web Scraping
When is web scraping useful?
Is web scraping legal?
Python 3
Background research
Crawling your first website
Summary
Chapter 2: Scraping the Data
Analyzing a web page
Three approaches to scrape a web page
CSS selectors and your Browser Console
XPath Selectors
LXML and Family Trees
Comparing performance
Scraping results
Summary
Chapter 3: Caching Downloads
When to use caching?
Adding cache support to the link crawler
Disk Cache
Key-value storage cache
Summary
Chapter 4: Concurrent Downloading
One million web pages
Sequential crawler
Threaded crawler
How threads and processes work
Performance
Chapter 5: Dynamic Content
An example dynamic web page
Reverse engineering a dynamic web page
Rendering a dynamic web page
The Render class
Summary
Chapter 6: Interacting with Forms
The Login form
Extending the login script to update content
Automating forms with Selenium
Summary
Chapter 7: Solving CAPTCHA
Registering an account
Optical character recognition
Solving complex CAPTCHAs
Using a CAPTCHA solving service
CAPTCHAs and machine learning
Summary
Chapter 8: Scrapy
Installing Scrapy
Starting a project
Different Spider Types
Scraping with the shell command
Visual scraping with Portia
Automated scraping with Scrapely
Summary
Chapter 9: Putting It All Together
Google search engine
Facebook
Gap
BMW
Summary

Book Details

ISBN 139781786462589
Paperback220 pages
Read More
From 3 reviews

Read More Reviews

Recommended for You

Python Machine Learning - Second Edition Book Cover
Python Machine Learning - Second Edition
$ 31.99
$ 22.40
Statistical Application Development with R and Python - Second Edition Book Cover
Statistical Application Development with R and Python - Second Edition
$ 39.99
$ 28.00
Python Network Programming Cookbook - Second Edition Book Cover
Python Network Programming Cookbook - Second Edition
$ 39.99
$ 28.00
Python for Finance - Second Edition Book Cover
Python for Finance - Second Edition
$ 39.99
$ 28.00
Python GUI Programming Cookbook - Second Edition Book Cover
Python GUI Programming Cookbook - Second Edition
$ 39.99
$ 28.00
Python High Performance - Second Edition Book Cover
Python High Performance - Second Edition
$ 31.99
$ 22.40