Write a cleaning script for scraped data
Writes a script that cleans a scraped dataset after reading sample rows, with every change logged and the raw file kept.
Task:
Write one [Python or R] script that cleans the scraped dataset whose sample rows are in the material.
Context:
The file is a [CSV or JSON] file with about [number of rows] rows. The problems I know about are [known problems, such as duplicates or mixed date formats]. The cleaned data will be used for [purpose].
Rules:
- Read the sample rows first and list the problems you see before writing code.
- Never overwrite the raw file. Write the cleaned data to a new file.
- Log every change as a count, such as "42 duplicates removed", and put rows you cannot fix in a separate review file.
- Remove personal data fields I did not ask to keep, and say which ones you removed.
- Say if you are unsure about a library function so I can check its documentation.
Result:
The list of problems in at most 6 lines, then the complete script with comments, then 3 lines on how to run it.
Material:
[paste 10 to 20 sample rows here]