Search icon Close icon
Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Checkout

Change country

Modal Close icon
Country selected Country selected
country flag United States
Country selected
country flag Great Britain
Country selected
country flag India
Country selected
country flag Germany
Country selected
country flag France
Country selected
country flag Canada
Country selected
country flag Russia
Country selected
country flag Spain
Country selected
country flag Brazil
Country selected
country flag Australia
Country selected
country flag Singapore
Country selected
country flag Canary Islands
Country selected
country flag Hungary
Country selected
country flag Ukraine
Country selected
country flag Luxembourg
Country selected
country flag Estonia
Country selected
country flag Lithuania
Country selected
country flag South Korea
Country selected
country flag Turkey
Country selected
country flag Switzerland
Country selected
country flag Colombia
Country selected
country flag Taiwan
Country selected
country flag Chile
Country selected
country flag Norway
Country selected
country flag Ecuador
Country selected
country flag Indonesia
Country selected
country flag New Zealand
Country selected
country flag Cyprus
Country selected
country flag Denmark
Country selected
country flag Finland
Country selected
country flag Poland
Country selected
country flag Malta
Country selected
country flag Czechia
Country selected
country flag Austria
Country selected
country flag Sweden
Country selected
country flag Italy
Country selected
country flag Egypt
Country selected
country flag Belgium
Country selected
country flag Portugal
Country selected
country flag Slovenia
Country selected
country flag Ireland
Country selected
country flag Romania
Country selected
country flag Greece
Country selected
country flag Argentina
Country selected
country flag Netherlands
Country selected
country flag Bulgaria
Country selected
country flag Latvia
Country selected
country flag South Africa
Country selected
country flag Malaysia
Country selected
country flag Japan
Country selected
country flag Slovakia
Country selected
country flag Philippines
Country selected
country flag Mexico
Country selected
country flag Thailand
Country selected
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds
Arrow up icon
GO TO TOP
Hands-On Web Scraping with Python

You're reading from   Hands-On Web Scraping with Python Extract quality data from the web using effective Python techniques

Arrow left icon
Product type Paperback
Published in Oct 2023
Last Updated in Feb 2025
Publisher Packt
ISBN-13 9781837636211
Length 324 pages
Edition 2nd Edition
Languages
Tools
Concepts
Arrow right icon
Author (1):
Arrow left icon
Chapagain Chapagain
Author Profile Icon Chapagain
Chapagain
LinkedIn
Anish Chapagain is a software engineer with a passion for data science, and artificial intelligence, its processes and Python programming, which began around 2007. He has been working with web scraping, data analysis, visualization and reporting-related tasks, projects for more than 10 years, and is also working as freelancer. Anish previously worked as a trainer, web/software developer, team leader, and as a banker, where he was exposed to data and gained further insights into topics like data mining, data analysis, reporting, information processing and knowledge discovery. He has an MSc in computer systems from Bangor University (United Kingdom), and an Executive MBA from Himalayan Whitehouse International College, Kathmandu, Nepal.
Read more
See other products by Chapagain
Arrow right icon
View More author details
Toc

Table of Contents (20) Chapters Close

Preface 1. Part 1:Python and Web Scraping
2. Chapter 1: Web Scraping Fundamentals FREE CHAPTER 3. Chapter 2: Python Programming for Data and Web 4. Part 2:Beginning Web Scraping
5. Chapter 3: Searching and Processing Web Documents 6. Chapter 4: Scraping Using PyQuery, a jQuery-Like Library for Python 7. Chapter 5: Scraping the Web with Scrapy and Beautiful Soup 8. Part 3:Advanced Scraping Concepts
9. Chapter 6: Working with the Secure Web 10. Chapter 7: Data Extraction Using Web APIs 11. Chapter 8: Using Selenium to Scrape the Web 12. Chapter 9: Using Regular Expressions and PDFs 13. Part 4:Advanced Data-Related Concepts
14. Chapter 10: Data Mining, Analysis, and Visualization 15. Chapter 11: Machine Learning and Web Scraping 16. Part 5:Conclusion
17. Chapter 12: After Scraping – Next Steps and Data Analysis 18. Index 19. Other Books You May Enjoy

Parsing robots.txt and sitemap.xml

In this section, we will introduce robots.txt- and sitemap.xml-related information and follow the instructions or resources available in those two files. We mentioned them in the Data-finding techniques used in web pages section of Chapter 1. In general, we can dive deep into the pages, or the directory with pages, of websites and find data or manage missing or hidden links using the robots.txt and sitemap.xml files.

The robots.txt file

The robots.txt file, or the Robots Exclusion Protocol, is a web-based standard or protocol used by websites to exchange information with automated scripts. robots.txt carries instructions regarding site-based links or resources to web robots (crawlers, spiders, web wanderers, or web bots), and uses directives such as Allow, Disallow, SiteMap, Crawl-delay, and User-agent to direct robots’ behavior.

We can find robots.txt by adding robots.txt to the main URL. For example, robots.txt for https://www.python...

lock icon The rest of the chapter is locked
Visually different images
CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Hands-On Web Scraping with Python
arrow left Previous Section
Section 6 of 8
Next Section arrow right
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Sign up now
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Start free trial
Renews at €18.99/month. Cancel anytime

Authors (1)

Profile icon Chapagain
Chapagain
LinkedIn icon Github icon
Anish Chapagain is a software engineer with a passion for data science, and artificial intelligence, its processes and Python programming, which began around 2007. He has been working with web scraping, data analysis, visualization and reporting-related tasks, projects for more than 10 years, and is also working as freelancer. Anish previously worked as a trainer, web/software developer, team leader, and as a banker, where he was exposed to data and gained further insights into topics like data mining, data analysis, reporting, information processing and knowledge discovery. He has an MSc in computer systems from Bangor University (United Kingdom), and an Executive MBA from Himalayan Whitehouse International College, Kathmandu, Nepal.
Read more
See other products by Chapagain

Personalised recommendations for you

Based on your interests and search pattern
Left arrow icon
Modern Full-Stack React Projects
Modern Full-Stack React Projects
Read more
Full-Stack React Projects is a complete guide to learning full-stack web development, understanding the creation and integration of backend systems, and advancing your career as a frontend developer.
Read more
Jun 2024 16h 52m
Mastering Node.js Web Development
Mastering Node.js Web Development
Read more
Explore Node.js with practical examples that will teach you how to utilize open-source packages for real-world solutions. Gain the skills to develop and deploy server-side applications that enhance your client-side projects.
Read more
Jun 2024 25h 56m
Reactive Patterns with RxJS and Angular Signals
Reactive Patterns with RxJS and Angular Signals
Read more
This RxJS book will help you understand the core concepts of RxJS and provide practical patterns to make your code more reactive and declarative. You'll also understand Angular Signals, which provide another way to improve code reactivity.
Read more
Jul 2024 8h 28m
API Testing and Development with Postman
API Testing and Development with Postman
Read more
Whether you are a tester or a developer working with APIs, you'll be able to put your knowledge to work with this practical guide to using Postman. The book provides a hands-on approach to implementing and learning the associated methodologies that will have you up-and-running and productive in no time.
Read more
Jun 2024 11h 56m
FastAPI Cookbook
FastAPI Cookbook
Read more
This book helps you unlock the power of FastAPI to build high-performing web apps and APIs by taking you through the basics like routing and data validation through to advanced topics, such as custom middleware and WebSockets.
Read more
Aug 2024 11h 56m
Mastering Spring Boot 3.0
Mastering Spring Boot 3.0
Read more
This hands-on guide empowers you to develop scalable and efficient applications. You'll also learn microservices patterns, reactive programming, and security measures for building robust backend systems.
Read more
Jun 2024 8h 32m
Nuxt 3 Projects
Nuxt 3 Projects
Read more
This book is a comprehensive guide to Nuxt.js, which takes you from the basics to advanced topics. Uniquely, this book emphasizes practical, project-based learning, tackling real-world problems.
Read more
Jun 2024 7h 40m
Vue.js 3 for Beginners
Vue.js 3 for Beginners
Read more
Learning a new language by following video tutorials, blog posts, and documentation is a tiresome activity. This book will take you on an exciting journey of becoming a proficient Vue.js developer through a practical, step-by-step approach.
Read more
Sep 2024 10h 4m
Full-Stack Web Development with TypeScript 5
Full-Stack Web Development with TypeScript 5
Read more
The book emphasizes best practices, debugging, performance optimization, and scalable code structure, helping you develop practical skills in frontend and backend development, database integration, and AI integration.
Read more
Aug 2024 9h 4m
Mastering Flask Web and API Development
Mastering Flask Web and API Development
Read more
The book is an introduction to Flask that will showcase its baseline, core, and advanced integration features to enable you to solve enterprise-related problems and issues in both web and API development.
Read more
Aug 2024 16h 28m
Right arrow icon
Modal Close icon
Modal Close icon