Date
AI Overview
Why it matters
Durable skills like collaboration, creativity, and critical thinking are essential for the workforce but remain hard to measure. What is not measured is often not taught, leading to these skills being overlooked in educational curricula.
Study at a glance
- Question: Can AI teammates and AI evaluators reliably elicit and score students' durable skills like collaboration and creativity?
- Setting and data: United States; 188 adult participants (ages 18–25) on Prolific and a lab study with 280 students.
- Method: Randomized lab and online experiments comparing AI teammate interaction protocols and evaluating automated rubric scoring against expert human raters.
- AI tools: Vantage, Gemini 2.5 Pro, Gemini 3.0
Key findings
- Steering conversations with an Executive LLM significantly increased elicited evidence of collaboration skills compared to unconstrained AI teammates.
- Automated LLM evaluations of collaboration conversations achieved agreement with human experts that was on par with agreement between human raters.
- For high school students completing complex creativity tasks, an AI autorater's overall scores correlated strongly with expert human ratings with a 0.88 Pearson correlation.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Durable skills, such as collaboration, creativity and critical thinking, are instrumental to success in the modern workforce. Yet, measuring these skills remains a persistent challenge. Moreover, because what is not measured is often not taught, these skills are often overlooked in mainstream educational curricula. Designing effective assessments for these skills necessitates balancing two often-conflicting requirements: ecological validity and psychometric rigor. On the one hand, the assessment environment should emulate natural real-world human interaction between humans. On the other hand, it should be scalable, controllable and reproducible. Here we argue that LLMs can be used to better capture both of these aims. Concretely, we develop a framework where the subject converses with AI teammates in a way that resembles human-human interaction for authenticity, while also offering the psychometric control required for informative and robust assessment. Importantly, the AI participants not only act as teammates but also, in an "Executive LLM" setup, steer the conversation towards eliciting a high density of observable evidence for skill proficiency. We complement this with an AI evaluator that can be used to measure skill proficiency in such interactions. We evaluate our assessment protocol based on transcripts of interactions of human participants with our AI framework, for multiple durable skills. For the skill of creativity, we further demonstrate the efficacy of an autorater for evaluating complex tasks performed by real students. Our analysis shows that the use of the Executive LLM significantly increases elicited evidence and that LLM-automated scoring of conversations largely agrees with that of expert annotators. This research demonstrates the utility of orchestrated LLMs approaches for measuring complex social and cognitive constructs in a scalable and controllable manner.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
AI tools
Publication Source
Publisher
arXiv
