Web Scraping with Python

Successfully scrape data from any website with the power of Python

Web Scraping with Python

Starting
Richard Lawson

13 customer reviews
Successfully scrape data from any website with the power of Python
$27.99
$34.99
RRP $27.99
RRP $34.99
eBook
Print + eBook

Instantly access this course right now and get the skills you need in 2017

With unlimited access to a constantly growing library of over 4,000 eBooks and Videos, a subscription to Mapt gives you everything you need to learn new skills. Cancel anytime.

Code Files
Preview in Mapt

Book Details

ISBN 139781782164364
Paperback174 pages

Book Description

The Internet contains the most useful set of data ever assembled, largely publicly accessible for free. However, this data is not easily reusable. It is embedded within the structure and style of websites and needs to be carefully extracted to be useful. Web scraping is becoming increasingly useful as a means to easily gather and make sense of the plethora of information available online. Using a simple language like Python, you can crawl the information out of complex websites using simple programming.

This book is the ultimate guide to using Python to scrape data from websites. In the early chapters it covers how to extract data from static web pages and how to use caching to manage the load on servers. After the basics we'll get our hands dirty with building a more sophisticated crawler with threads and more advanced topics. Learn step-by-step how to use Ajax URLs, employ the Firebug extension for monitoring, and indirectly scrape data. Discover more scraping nitty-gritties such as using the browser renderer, managing cookies, how to submit forms to extract data from complex websites protected by CAPTCHA, and so on. The book wraps up with how to create high-level scrapers with Scrapy libraries and implement what has been learned to real websites.

Table of Contents

Chapter 1: Introduction to Web Scraping
When is web scraping useful?
Is web scraping legal?
Background research
Crawling your first website
Summary
Chapter 2: Scraping the Data
Analyzing a web page
Three approaches to scrape a web page
Summary
Chapter 3: Caching Downloads
Adding cache support to the link crawler
Disk cache
Database cache
Summary
Chapter 4: Concurrent Downloading
One million web pages
Sequential crawler
Threaded crawler
Performance
Summary
Chapter 5: Dynamic Content
An example dynamic web page
Reverse engineering a dynamic web page
Rendering a dynamic web page
Summary
Chapter 6: Interacting with Forms
The Login form
Extending the login script to update content
Automating forms with the Mechanize module
Summary
Chapter 7: Solving CAPTCHA
Registering an account
Optical Character Recognition
Solving complex CAPTCHAs
Summary
Chapter 8: Scrapy
Installation
Starting a project
Visual scraping with Portia
Automated scraping with Scrapely
Summary
Chapter 9: Overview
Google search engine
Facebook
Gap
BMW
Summary

What You Will Learn

  • Extract data from web pages with simple Python programming
  • Build a threaded crawler to process web pages in parallel
  • Follow links to crawl a website
  • Download cache to reduce bandwidth
  • Use multiple threads and processes to scrape faster
  • Learn how to parse JavaScript-dependent websites
  • Interact with forms and sessions
  • Solve CAPTCHAs on protected web pages
  • Discover how to track the state of a crawl

Authors

Table of Contents

Chapter 1: Introduction to Web Scraping
When is web scraping useful?
Is web scraping legal?
Background research
Crawling your first website
Summary
Chapter 2: Scraping the Data
Analyzing a web page
Three approaches to scrape a web page
Summary
Chapter 3: Caching Downloads
Adding cache support to the link crawler
Disk cache
Database cache
Summary
Chapter 4: Concurrent Downloading
One million web pages
Sequential crawler
Threaded crawler
Performance
Summary
Chapter 5: Dynamic Content
An example dynamic web page
Reverse engineering a dynamic web page
Rendering a dynamic web page
Summary
Chapter 6: Interacting with Forms
The Login form
Extending the login script to update content
Automating forms with the Mechanize module
Summary
Chapter 7: Solving CAPTCHA
Registering an account
Optical Character Recognition
Solving complex CAPTCHAs
Summary
Chapter 8: Scrapy
Installation
Starting a project
Visual scraping with Portia
Automated scraping with Scrapely
Summary
Chapter 9: Overview
Google search engine
Facebook
Gap
BMW
Summary

Book Details

ISBN 139781782164364
Paperback174 pages
Read More
From 13 reviews

Read More Reviews