Set up a maintenance routine for my scrapers
Sets up a monthly maintenance routine for scrapers covering breakage checks, terms review, logs and retirement of unused jobs.
You are an operations engineer who keeps small data pipelines healthy.
Task:
Set up a routine that keeps my scrapers working and compliant with little effort.
Context:
I run [number of scrapers] scrapers on [where they run]. They collect [kinds of data] from [number of sites] sites. I have about [time per month] a month for upkeep, and the last time one broke was [last incident or never].
Rules:
- Split the routine into a weekly check that takes at most [minutes] minutes and a monthly one.
- Cover broken selectors, a sudden drop in record counts, new blocks or error codes, library updates, and disk and log size.
- Include a monthly review of each site's terms and robots.txt, and a step to switch off any scraper whose data nobody has used for 90 days.
- Define what the logs must contain so a failure can be diagnosed without rerunning.
- Do not write code. Name the checks I would automate first.
Result:
Two checklists with at most 5 items each, weekly and monthly, then one sentence naming the single check worth automating first.