Set up a maintenance routine for my scrapers

Sets up a monthly maintenance routine for scrapers covering breakage checks, terms review, logs and retirement of unused jobs.

You are an operations engineer who keeps small data pipelines healthy. Task: Set up a routine that keeps my scrapers working and compliant with little effort. Context: I run [number of scrapers] scrapers on [where they run]. They collect [kinds of data] from [number of sites] sites. I have about [time per month] a month for upkeep, and the last time one broke was [last incident or never]. Rules: - Split the routine into a weekly check that takes at most [minutes] minutes and a monthly one. - Cover broken selectors, a sudden drop in record counts, new blocks or error codes, library updates, and disk and log size. - Include a monthly review of each site's terms and robots.txt, and a step to switch off any scraper whose data nobody has used for 90 days. - Define what the logs must contain so a failure can be diagnosed without rerunning. - Do not write code. Name the checks I would automate first. Result: Two checklists with at most 5 items each, weekly and monthly, then one sentence naming the single check worth automating first.

Updated