Review the risks of scaling up my scraper
Reviews a plan to scale a scraper to many pages or sites, with a verdict first and the risks to the sites and to me.
Task:
Review my plan for scaling a scraper from [current size] to [target size] and tell me whether it is safe to proceed.
Context:
The scraper collects [data] from [number of sites] sites for [purpose]. My team is [team size] people and the deadline is [deadline]. The plan is in the material.
Rules:
- Check the load on each site, such as requests per second per host, and whether the plan spreads load or concentrates it.
- Check legal and contractual exposure, personal data, storage and retention, and what happens if a site asks me to stop.
- Where the plan relies on getting around blocks, say that is the wrong fix and give a compliant alternative such as a data agreement, a licensed feed or fewer pages.
- This is general information, not legal advice. Name which risks need a qualified lawyer.
- Name every assumption, and write "not stated" where the plan is silent.
Result:
One sentence with the verdict first, then at most 6 risks ranked by severity, each with a fix in one line.
Material:
[paste your scaling plan here]