Plan a polite schedule for a recurring scraper
Plans when and how often a scraper should run, with off-peak timing, backoff, failure alerts and a stop rule.
Task:
Plan the schedule on which my scraper runs, including what it does when something goes wrong.
Context:
The scraper collects [data] from [site] and takes about [run time] per run. The data changes about [how often it changes]. I run it on [my laptop, a server or a cloud service] in [time zone]. The site allows automated access: [yes, and the crawl delay if stated].
Rules:
- Match the run frequency to how often the data really changes, and justify it in one line.
- Choose a time window that avoids the site's likely busy hours, and say it is an assumption I should check.
- Add backoff: wait longer after each failure, and stop the schedule after [number] failures in a row until I look at it.
- Add an alert for failures and for a run that returns far fewer records than usual.
- Include the cron expression or the scheduler setting for my environment.
Result:
A table with at most 5 rows (setting, value, reason), then the schedule snippet, then 2 lines on how to test one run safely.