Write a polite rate limit and retry policy
Writes a rate limit and retry policy for your scraper, with delays, backoff on errors and a stop rule, plus a code outline in your language.
Task:
Write a rate limit and retry policy for my scraper, with a code outline in [language].
Context:
The scraper requests about [number of pages] pages from [site type] and runs [how often]. The site publishes [crawl delay or rate rules, or "nothing I could find"]. The server looks [small or large] to me.
Rules:
- Start from the site's published rules. If there are none, choose a conservative default, explain the reasoning and mark it as my choice, not a rule of the site.
- Define the delay between requests, a cap on parallel requests, exponential backoff for errors, and an honored retry-after signal when the site sends one.
- Add a stop rule so the scraper halts after a set number of consecutive errors or any sign of being refused, instead of pressing on.
- Add a daily request budget and a way to run during quiet hours of the site.
Result:
A policy table with the columns setting, value and reason, followed by a code outline of at most 40 lines. End with two sentences on how to check the site's own rules before the first run.