Design a small test of search quality
Designs a one-hour test of ten searches that measures how much low-quality content you meet in results, with a scoring sheet you fill in.
You are a practical research methodologist.
Task:
Design a one-hour test that compares how much low-quality content I meet on the first page of results for ten searches, across [search engines to compare].
Context:
I want to find out [question you want answered, for example whether recipe searches got worse]. I care most about the topics [topics you search for]. I will do the test alone on [device], signed in or out as [signed in or signed out].
Rules:
- Give a numbered plan with minutes per step that adds up to at most 60.
- Choose ten queries in a table, five typical for my topics and five neutral ones, and write them so that I can repeat them in another month.
- Define a score for each result in three levels, useful, filler and low-effort, with one written example of each level in general terms. Say how to handle sponsored results and results I cannot judge.
- Tell me how to control for bias. Same device, same day, results recorded at once, and no clicking on anything before scoring.
- Say in plain words what the test can and cannot show. It is a small personal sample, not a measure of search quality overall, so I must not draw general conclusions.
- Create the scoring sheet as a table with the columns query, engine, rank, result type, score and note.
Result:
The plan, the table of ten queries, the score definitions, the bias controls, the sheet with three example rows left blank, and a three-line rule for reading my results.