Web scraping tools

Ready-made prompts for planning and organizing around web scraping tools.

Plans compliant routes to the data you need when a site limits your requests, such as asking for access, an official API, a data feed or slower runs.

Plans how to gather images for a project with licenses, attribution and storage in mind, preferring sources that allow reuse.

Plans a dataset of local businesses from official maps APIs, open registries and permissioned sources, with a privacy check.

Plans a small price tracker for a handful of products using alerts, official tools and a gentle script as a last resort.

Plans when and how often a scraper should run, with off-peak timing, backoff, failure alerts and a stop rule.

Plans a real estate data project around open data, licensed feeds and agreements, with a data quality and privacy step.

Plans how to collect public data from a script-driven site, trying official feeds and APIs before any browser automation.

Plans how to get tables out of PDFs you have the right to use, comparing text extraction, OCR and manual checks with a validation step.

Plans how to watch a public page for changes with a light check, alerts and a stop rule, preferring feeds and notifications.

Plans a move of a local scraper to a cloud setup in numbered steps with costs to check, secrets handling and a rollback path.

Plans how to gather product and price data from shops through feeds, affiliate programs and permission before any scraping.

Plans a Scrapy project from your goal into numbered build steps, with the spider, items, pipeline and politeness settings named for each.

Plans the compliant response when your scraper meets a CAPTCHA: stop, check the rules, ask for access, or switch to an official data source.

Sets up a monthly maintenance routine for scrapers covering breakage checks, terms review, logs and retirement of unused jobs.

Sets up a weekly news monitoring routine built on feeds, alerts and official APIs, with source and respect rules.