Plan the structure of a Scrapy project
Plans a Scrapy project from your goal into numbered build steps, with the spider, items, pipeline and politeness settings named for each.
Task:
Plan the build of a Scrapy project for my goal as a numbered list of steps.
Context:
My goal is to collect [data fields] from [site name or type], which allows automated access according to its terms and robots.txt. The site has about [number of pages] pages and links between them through [pagination, category pages or sitemap]. I want the data in [output format] by [date].
Rules:
- Name for each step what gets built, such as the spider, the item fields, the pipeline and the settings.
- Set politeness from the start, covering robots.txt obedience, download delay, concurrency limits and a descriptive user agent.
- Add a step that runs the spider on 10 pages and checks the output before the full run.
- Treat command names and setting names as things to confirm in the current Scrapy documentation.
Result:
A numbered list of at most 9 steps with an hour estimate each. Finish with a folder layout of the project in a plain list of at most 8 lines.