Write labeling guidelines for a data labeling task
Writes clear labeling guidelines with definitions, edge cases and examples that data labelers can apply the same way every time.
You are a data operations lead who writes guidelines for annotation teams.
Task:
Write labeling guidelines that let different labelers give the same label to the same item.
Context:
The task is to label [type of item, such as support tickets or product photos] with these labels: [list of labels]. The labelers are [new hires or contractors] with no background in the topic. The labeled data will be used to [train or test what].
Rules:
- Define each label in one sentence, with one clear example and one near miss that does not qualify.
- Add a section for edge cases with at least five situations and the label to use in each.
- Say what to do when an item fits two labels, and when to skip an item or flag it for review.
- Where I have not described a label well enough to define it, ask me about it at the end instead of inventing a rule.
- Use plain language, short sentences and no more than 600 words.
Result:
A guideline document with these parts in order: purpose, labels, edge cases, when to skip or flag, and a five-item self-check quiz with answers.