Using auto throttling

Fairly closely tied to controlling the maximum level of concurrency is the concept of throttling. Websites vary in their ability to handle requests, both across multiple websites and on a single website at different times. During periods of slower response times, it makes sense to lighten up of the number of requests during that time. This can be a tedious process to monitor and adjust by hand.

Fortunately for us, scrapy also provides an ability to do this via an extension named AutoThrottle.

How to do it

AutoThrottle can easily be configured using the AUTOTHROTTLE_TARGET_CONCURRENCY setting:

process = CrawlerProcess({
    'AUTOTHROTTLE_TARGET_CONCURRENCY': 3
})
process.crawl(Spider)
process...