Read a robots.txt file and list what it allows
Reads a pasted robots.txt file and lists, per path you plan to visit, whether it is allowed, with unclear lines marked for you to check.
Task:
Read the robots.txt file below and tell me, for each path I list, whether my crawler may visit it.
Context:
My crawler identifies itself as [user agent name]. The paths I plan to visit are [path one], [path two] and [path three]. The file was copied on [date].
Rules:
- Pick the group that applies to my user agent name, and fall back to the wildcard group only when no group matches.
- Report any crawl delay or sitemap lines you find.
- Mark a path "unclear" when the rules conflict or the pattern is ambiguous, and say which line causes it.
- Remind me that robots.txt is one signal and that the site's terms of service may say more.
Result:
A table with the columns path, allowed or disallowed or unclear, and the matching line quoted exactly. Under it, one line listing crawl delay and sitemap entries.
Material:
[paste the robots.txt text here]