Plan product data collection from online shops
Plans how to gather product and price data from shops through feeds, affiliate programs and permission before any scraping.
Task:
Plan how I collect product names, prices and availability for [product category] from online shops in a way the shops would accept.
Context:
I am building [project] and need data on about [number of products] products from [number of shops] shops, updated [how often]. I code in [language] at [skill level].
Rules:
- Rank the routes by how much the shop agrees to them, starting with product feeds, affiliate or partner programs, official APIs and asking for permission.
- Treat scraping a shop's pages as the last route, and only for pages its terms and robots.txt allow.
- Give the request rate and the identification I should set for that last route, as numbers.
- Do not state any shop's terms or program details as fact. Tell me what to check on each shop's own pages.
Result:
A numbered plan of at most 6 steps with an effort estimate in hours, and one closing sentence on how to keep the data fresh without extra load.