Date
AI Overview
Why it matters
Human annotation is often treated as objective ground truth, but in subjective tasks, annotators' judgments and competence may evolve over time, influencing both data quality and downstream artificial intelligence model training.
Study at a glance
- Question: How do annotator competence, data quality, and model training change across a social influence labeling task?
- Setting and data: 25 annotators (educators, psychologists, adolescents, parents, communication experts) labeling 1,021 AI-generated dialogues targeting adolescents.
- Method: Mixed methods repeated-measures study with pre- and post-test comparison, surveys, interviews, content analysis, and model evaluation.
- AI tools: DeepSeek-V3.2, Llama-3.1-8B-Instruct, gpt-4o-mini
Key findings
- Annotators became significantly faster, reducing average annotation time from 6.56 to 5.86 minutes per text, while their self-perceived competence and confidence increased significantly.
- Data labeled after the main annotation process improved language model performance for both in-context learning and supervised fine-tuning compared to initial data.
- Annotator changes reflected deeper qualitative detail, broader perspectives, and refined categorization, with data quality improvements and inter-annotator agreement gains concentrated among domain experts.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Human data annotation, especially when involving experts, is often treated as an objective reference. However, many annotation tasks are inherently subjective, and annotators' judgments may evolve over time. This study investigates changes in the quality of annotators' work from a competence perspective during a process of social influence recognition. The study involved 25 annotators from five different groups, including both experts and non-experts, who annotated a dataset of 1,021 dialogues with 20 social influence techniques, along with intentions, reactions, and consequences. An initial subset of 150 texts was annotated twice - before and after the main annotation process - to enable comparison. To measure competence shifts, we combined qualitative and quantitative analyses of the annotated data, semi-structured interviews with annotators, self-assessment surveys, and Large Language Model training and evaluation on the comparison dataset. The results indicate a significant increase in annotators' self-perceived competence and confidence. Moreover, observed changes in data quality suggest that the annotation process may enhance annotator competence and that this effect is more pronounced in expert groups. The observed shifts in annotator competence have a visible impact on the performance of LLMs trained on their annotated data.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Methods
AI tools
Publication Source
Publisher
arXiv
