Find why my scraper sees an empty dynamic page
Diagnoses why a scraper gets an empty page from a dynamic site, ranks likely causes and starts with compliant options like official APIs.
Task:
Work out why my scraper receives an almost empty page, and rank the likely causes with one check for each.
Context:
I request [page type] from [site] with [library or tool]. I expected [expected content] and got what is in the material. Automated access to this page is allowed by its terms and robots.txt: [yes or not checked].
Rules:
- Read the response first and quote the lines that support each cause.
- Consider content loaded by scripts, a consent or login wall, a block page, and a changed layout.
- If the cause looks like a block or a rate limit, advise slowing down, asking the owner or using an official API, and never advise disguising the scraper.
- Say when the response is not enough to decide.
Result:
A ranked list of at most 4 causes, each with the evidence, one check I can run in browser developer tools, and the fix.
Material:
[paste the response HTML or a short excerpt here]