All Products

Best Sellers

New Releases

Books

Videos

Audiobooks

Free Learning

Hands-On Web Scraping with Python - Second Edition

You're reading from Hands-On Web Scraping with Python - Second Edition

Product type Book

Published in Oct 2023

Publisher Packt

ISBN-13 9781837636211

Pages 324 pages

Edition 2nd Edition

Languages

Concepts

Web Programming

Author (1):

Anish Chapagain

Table of Contents (20) Chapters

Preface

1. Part 1:Python and Web Scraping

2. Chapter 1: Web Scraping Fundamentals

3. Chapter 2: Python Programming for Data and Web

4. Part 2:Beginning Web Scraping

5. Chapter 3: Searching and Processing Web Documents

6. Chapter 4: Scraping Using PyQuery, a jQuery-Like Library for Python

7. Chapter 5: Scraping the Web with Scrapy and Beautiful Soup

8. Part 3:Advanced Scraping Concepts

9. Chapter 6: Working with the Secure Web

10. Chapter 7: Data Extraction Using Web APIs

11. Chapter 8: Using Selenium to Scrape the Web

12. Chapter 9: Using Regular Expressions and PDFs

13. Part 4:Advanced Data-Related Concepts

14. Chapter 10: Data Mining, Analysis, and Visualization

15. Chapter 11: Machine Learning and Web Scraping

16. Part 5:Conclusion

17. Chapter 12: After Scraping – Next Steps and Data Analysis

18. Index

Why subscribe?

19. Other Books You May Enjoy

Parsing robots.txt and sitemap.xml

In this section, we will introduce robots.txt- and sitemap.xml-related information and follow the instructions or resources available in those two files. We mentioned them in the Data-finding techniques used in web pages section of Chapter 1. In general, we can dive deep into the pages, or the directory with pages, of websites and find data or manage missing or hidden links using the robots.txt and sitemap.xml files.

The robots.txt file

The robots.txt file, or the Robots Exclusion Protocol, is a web-based standard or protocol used by websites to exchange information with automated scripts. robots.txt carries instructions regarding site-based links or resources to web robots (crawlers, spiders, web wanderers, or web bots), and uses directives such as Allow, Disallow, SiteMap, Crawl-delay, and User-agent to direct robots’ behavior.

We can find robots.txt by adding robots.txt to the main URL. For example, robots.txt for https://www.python...

The rest of the chapter is locked

Register for a free Packt account to unlock a world of extra content!

A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.

Unlock this book and the full library FREE for 7 days

Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of

Start free trial

Renews at €14.99/month. Cancel anytime}

Authors (1)

Anish Chapagain

Anish Chapagain is a software engineer with a passion for data science, its processes, and Python programming, which began around 2007. He has been working with web scraping and analysis-related tasks for more than 5 years, and is currently pursuing freelance projects in the web scraping domain. Anish previously worked as a trainer, web/software developer, and as a banker, where he was exposed to data and gained further insights into topics including data analysis, visualization, data mining, information processing, and knowledge discovery. He has an MSc in computer systems from Bangor University (University of Wales), United Kingdom, and an Executive MBA from Himalayan Whitehouse International College, Kathmandu, Nepal.

Read more

See other products by Anish Chapagain

Personalised recommendations for you

Based on your interests and search pattern

Modernizing Drupal 10 Theme Development

Modernizing Drupal 10 Theme Development

Modernizing Drupal 10 Theme Development covers everything a frontend developer or a Drupal developer needs to know about the Drupal 10 theme layer. Using real-world examples, this book will help you to build up from an empty theme to a fast, responsive, maintainable, and accessible Drupal website in both monolithic and decoupled ways.

Aug 2023 12 hours 0 minutes

Full Stack Development with Spring Boot 3 and React

Full Stack Development with Spring Boot 3 and React

Full Stack Development with Spring Boot 3 and React contains a wealth of practical guidance for picking up full stack development. The step-by-step exploration of everything from dependency injection, ORM with Hibernate, and JWTs to RESTful APIs, UI styling, and TypeScript will help you to develop the Spring Boot and React skills you need.

Oct 2023 15 hours 8 minutes

Modern API Development with Spring 6 and Spring Boot 3

Modern API Development with Spring 6 and Spring Boot 3

This practical guide teaches inexperienced Java programmers and web developers how to design, develop, test, and deploy highly scalable and maintainable APIs using REST, gRPC, GraphQL, and reactive programming paradigms with Java and Spring Boot. Complete with real-world examples, it will guide you to build enterprise-level APIs and services.

Sep 2023 16 hours 28 minutes

Hands-On Web Scraping with Python

Hands-On Web Scraping with Python

This book guides you to discover and extract quality data from the web using Python libraries such as requests, lxml, pyquery, scrapy, and to use effective scraping techniques. It’ll help you master the fundamentals and advanced concepts of web scraping and apply your skills to real-world data extraction tasks, with analysis and visualization.

Oct 2023 10 hours 48 minutes

Mastering Blazor WebAssembly

Mastering Blazor WebAssembly

Mastering Blazor WebAssembly is a comprehensive, practical guide that’ll help you to develop advanced web applications. By combining real-world experience with the thorough knowledge needed to build Blazor apps in a masterful fashion, this book enables you to efficiently manage cross-platform mobile development with .NET MAUI and Blazor.

Aug 2023 12 hours 20 minutes

Mastering Blazor WebAssembly

Mastering Blazor WebAssembly

Mastering Blazor WebAssembly is a comprehensive, practical guide that’ll help you to develop advanced web applications. By combining real-world experience with the thorough knowledge needed to build Blazor apps in a masterful fashion, this book enables you to efficiently manage cross-platform mobile development with .NET MAUI and Blazor.

Aug 2023 12 hours 20 minutes

Full Stack Web Development with Remix

Full Stack Web Development with Remix

Unlock the power of the web platform and cutting-edge technologies for your React applications using Remix to take advantage of the full stack to create a great user experience. This book is a complete resource covering the conventions, levers, and primitives of Remix and illustrates concepts through a real-world project.

Nov 2023 10 hours 36 minutes

Real-World Svelte

Real-World Svelte

This book is a must-have guide to unlocking Svelte's true potential. You’ll learn about advanced component development, efficient coding, and powerful UI patterns using actions. With a focus on state management, custom stores, and renderless components, this book equips you to build exceptional web apps with lightning-fast performance.

Dec 2023 9 hours 24 minutes

Building Micro Frontends with React 18

Building Micro Frontends with React 18

This book is a step-by-step guide to building production-grade, highly scalable, and performant web apps using multi-SPA and micro apps patterns, where you’ll learn to deal with complexities related to micro frontends. The book is a full life cycle guide to building, deploying, and managing micro frontends in a cloud-native environment.

Oct 2023 7 hours 16 minutes

Drupal 10 Masterclass

Drupal 10 Masterclass

This all-in-one guide helps you get up and running with building Drupal applications using the latest Drupal 10 features. You’ll develop a complete practical understanding of Drupal frontend, backend, architecture, content management, themes, and modules to deliver a rich user experience by creating custom Drupal apps.

Dec 2023 10 hours 20 minutes

Full-Stack Flask and React

Full-Stack Flask and React

This book teaches you how to build simple yet efficient production-ready web applications. Starting with an introduction to React, this book will equip you with deeper knowledge of Flask and React technology stacks to undertake web application development with confidence.

Oct 2023 13 hours 36 minutes

Full-Stack Flask and React

Full-Stack Flask and React

This book teaches you how to build simple yet efficient production-ready web applications. Starting with an introduction to React, this book will equip you with deeper knowledge of Flask and React technology stacks to undertake web application development with confidence.

Oct 2023 13 hours 36 minutes