Date
AI Overview
Why it matters
Adaptive testing struggles with guessing behaviors, open-ended responses, noisy data, and coarse score-based outputs. Developing conversational agents can create more natural interactions and actionable diagnostic feedback.
Study at a glance
- Question: Can an interactive language model agent improve the accuracy, efficiency, and experience of adaptive human assessments?
- Setting and data: Math practice logs (1,940 students), MBTI personality items (1,000 simulated testers), SCL-90 depression items (500 simulated testers), and 50 human volunteers.
- Method: System development, simulation benchmark evaluations, and a user study comparing two testing methods using paired t-tests.
- AI tools: TestAgent, GPT-4, ChatGLM2-6B, GLM-4-9B
Key findings
- TestAgent achieved the same ability estimation accuracy as traditional methods with 20% fewer questions across test datasets.
- The framework improved prediction performance, yielding an average relative gain of 1.77% in AUC@5 and 0.91% in ACC@5 across the benchmarks.
- In a user evaluation with 50 volunteers, participants rated TestAgent significantly higher than traditional testing across accuracy, fluency, speed, and interaction.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Accurately assessing internal human states is key to understanding
preferences, offering personalized services, and identifying challenges in
real-world applications. Originating from psychometrics, adaptive testing has
become the mainstream method for human measurement and has now been widely
applied in education, healthcare, sports, and sociology. It customizes
assessments by selecting the fewest test questions . However, current adaptive
testing methods face several challenges. The mechanized nature of most
algorithms leads to guessing behavior and difficulties with open-ended
questions. Additionally, subjective assessments suffer from noisy response data
and coarse-grained test outputs, further limiting their effectiveness. To move
closer to an ideal adaptive testing process, we propose TestAgent, a large
language model (LLM)-powered agent designed to enhance adaptive testing through
interactive engagement. This is the first application of LLMs in adaptive
testing. TestAgent supports personalized question selection, captures
test-takers' responses and anomalies, and provides precise outcomes through
dynamic, conversational interactions. Experiments on psychological,
educational, and lifestyle assessments show our approach achieves more accurate
results with 20% fewer questions than state-of-the-art baselines, and testers
preferred it in speed, smoothness, and other dimensions.
Research Categories
What is the application?
Why use AI?
Study design
Methods
Datasets
Publication Source
Publisher
arXiv
