Web Development
Reader small image

You're reading from  Hands-On Web Scraping with Python - Second Edition

Product typeBook
Published inOct 2023
PublisherPackt
ISBN-139781837636211
Edition2nd Edition
Concepts
Web Programming
Right arrow
Author (1)
Anish Chapagain
Anish Chapagain
author image
Anish Chapagain

Anish Chapagain is a software engineer with a passion for data science, its processes, and Python programming, which began around 2007. He has been working with web scraping and analysis-related tasks for more than 5 years, and is currently pursuing freelance projects in the web scraping domain. Anish previously worked as a trainer, web/software developer, and as a banker, where he was exposed to data and gained further insights into topics including data analysis, visualization, data mining, information processing, and knowledge discovery. He has an MSc in computer systems from Bangor University (University of Wales), United Kingdom, and an Executive MBA from Himalayan Whitehouse International College, Kathmandu, Nepal.
Read more about Anish Chapagain

Right arrow

Parsing robots.txt and sitemap.xml

In this section, we will introduce robots.txt- and sitemap.xml-related information and follow the instructions or resources available in those two files. We mentioned them in the Data-finding techniques used in web pages section of Chapter 1. In general, we can dive deep into the pages, or the directory with pages, of websites and find data or manage missing or hidden links using the robots.txt and sitemap.xml files.

The robots.txt file

The robots.txt file, or the Robots Exclusion Protocol, is a web-based standard or protocol used by websites to exchange information with automated scripts. robots.txt carries instructions regarding site-based links or resources to web robots (crawlers, spiders, web wanderers, or web bots), and uses directives such as Allow, Disallow, SiteMap, Crawl-delay, and User-agent to direct robots’ behavior.

We can find robots.txt by adding robots.txt to the main URL. For example, robots.txt for https://www.python...

lock icon
The rest of the page is locked
Previous Page
Chapter 6 of 20, Page 6
Next Page
You have been reading a chapter from
Hands-On Web Scraping with Python - Second Edition
Published in: Oct 2023Publisher: PacktISBN-13: 9781837636211
© 2023 Packt Publishing Limited All Rights Reserved

Author (1)

author image
Anish Chapagain

Anish Chapagain is a software engineer with a passion for data science, its processes, and Python programming, which began around 2007. He has been working with web scraping and analysis-related tasks for more than 5 years, and is currently pursuing freelance projects in the web scraping domain. Anish previously worked as a trainer, web/software developer, and as a banker, where he was exposed to data and gained further insights into topics including data analysis, visualization, data mining, information processing, and knowledge discovery. He has an MSc in computer systems from Bangor University (University of Wales), United Kingdom, and an Executive MBA from Himalayan Whitehouse International College, Kathmandu, Nepal.
Read more about Anish Chapagain

Personalised recommendations for you

Based on your interests and search pattern

Modernizing Drupal 10 Theme Development

Modernizing Drupal 10 Theme Development

Modernizing Drupal 10 Theme Development covers everything a frontend developer or a Drupal developer needs to know about the Drupal 10 theme layer. Using real-world examples, this book will help you to build up from an empty theme to a fast, responsive, maintainable, and accessible Drupal website in both monolithic and decoupled ways.

BookAug 2023360 pages4
Full Stack Development with Spring Boot 3 and React

Full Stack Development with Spring Boot 3 and React

Full Stack Development with Spring Boot 3 and React contains a wealth of practical guidance for picking up full stack development. The step-by-step exploration of everything from dependency injection, ORM with Hibernate, and JWTs to RESTful APIs, UI styling, and TypeScript will help you to develop the Spring Boot and React skills you need.

BookOct 2023454 pages5
Modern API Development with Spring 6 and Spring Boot 3

Modern API Development with Spring 6 and Spring Boot 3

This practical guide teaches inexperienced Java programmers and web developers how to design, develop, test, and deploy highly scalable and maintainable APIs using REST, gRPC, GraphQL, and reactive programming paradigms with Java and Spring Boot. Complete with real-world examples, it will guide you to build enterprise-level APIs and services.

BookSep 2023494 pages1
Hands-On Web Scraping with Python

Hands-On Web Scraping with Python

This book guides you to discover and extract quality data from the web using Python libraries such as requests, lxml, pyquery, scrapy, and to use effective scraping techniques. It’ll help you master the fundamentals and advanced concepts of web scraping and apply your skills to real-world data extraction tasks, with analysis and visualization.

BookOct 2023324 pages
Mastering Blazor WebAssembly

Mastering Blazor WebAssembly

Mastering Blazor WebAssembly is a comprehensive, practical guide that’ll help you to develop advanced web applications. By combining real-world experience with the thorough knowledge needed to build Blazor apps in a masterful fashion, this book enables you to efficiently manage cross-platform mobile development with .NET MAUI and Blazor.

BookAug 2023370 pages4
Mastering Blazor WebAssembly

Mastering Blazor WebAssembly

Mastering Blazor WebAssembly is a comprehensive, practical guide that’ll help you to develop advanced web applications. By combining real-world experience with the thorough knowledge needed to build Blazor apps in a masterful fashion, this book enables you to efficiently manage cross-platform mobile development with .NET MAUI and Blazor.

BookAug 2023370 pages3
Full Stack Web Development with Remix

Full Stack Web Development with Remix

Unlock the power of the web platform and cutting-edge technologies for your React applications using Remix to take advantage of the full stack to create a great user experience. This book is a complete resource covering the conventions, levers, and primitives of Remix and illustrates concepts through a real-world project.

BookNov 2023318 pages
Real-World Svelte

Real-World Svelte

This book is a must-have guide to unlocking Svelte's true potential. You’ll learn about advanced component development, efficient coding, and powerful UI patterns using actions. With a focus on state management, custom stores, and renderless components, this book equips you to build exceptional web apps with lightning-fast performance.

BookDec 2023282 pages3
Building Micro Frontends with React 18

Building Micro Frontends with React 18

This book is a step-by-step guide to building production-grade, highly scalable, and performant web apps using multi-SPA and micro apps patterns, where you’ll learn to deal with complexities related to micro frontends. The book is a full life cycle guide to building, deploying, and managing micro frontends in a cloud-native environment.

BookOct 2023218 pages
Drupal 10 Masterclass

Drupal 10 Masterclass

This all-in-one guide helps you get up and running with building Drupal applications using the latest Drupal 10 features. You’ll develop a complete practical understanding of Drupal frontend, backend, architecture, content management, themes, and modules to deliver a rich user experience by creating custom Drupal apps.

BookDec 2023310 pages
Full-Stack Flask and React

Full-Stack Flask and React

This book teaches you how to build simple yet efficient production-ready web applications. Starting with an introduction to React, this book will equip you with deeper knowledge of Flask and React technology stacks to undertake web application development with confidence.

BookOct 2023408 pages5
Full-Stack Flask and React

Full-Stack Flask and React

This book teaches you how to build simple yet efficient production-ready web applications. Starting with an introduction to React, this book will equip you with deeper knowledge of Flask and React technology stacks to undertake web application development with confidence.

BookOct 2023408 pages2