Search and Filter

Vizqstudio: Iterative Visualization Literacy Mcqs Design With Simulated Students

Date

AI Overview

Why it matters

Designing high-quality visualization literacy multiple-choice questions is challenging and labor-intensive. Educators lack diverse, pedagogically targeted question banks and tools to iteratively design and adapt questions to learners' varying backgrounds and reasoning strategies.

Study at a glance
  • Question: Can visual analytics with simulated students help instructors design effective visualization literacy multiple-choice questions?
  • Setting and data: 6 visualization instructors, 12 undergraduate students in a classroom quiz, and 99 online participants recruited via Prolific.
  • Method: Mixed methods combining system development, user studies, expert interviews, a classroom trial, and an online experiment.
  • AI tools: VizQStudio, GPT-4o, GPT-o3-mini, o1
Key findings
  • In an online study, students who practiced with system-designed questions achieved learning gains comparable to those using established benchmark questions.
  • Across 390 alignment instances, 94.62% of online learners' reasoning processes matched one of the five most frequent AI-simulated reasoning strategies.
  • Instructors iteratively refined their questions an average of 8.3 times in about 9.6 minutes, successfully adjusting question difficulty and addressing student misconceptions.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Multiple-choice questions (MCQs) are a widely used educational tool, particularly in domains such as visualization literacy that require broad conceptual coverage and support diverse real-world applications. However, designing high-quality visualization literacy MCQs remains challenging, as instructors must coordinate multimodal elements (e.g., charts, question stems, and distractors), address diverse visualization tasks, and accommodate learners with heterogeneous backgrounds. Existing visualization literacy assessments primarily rely on standardized, fixed item banks, offering limited support for iterative question design that adapts to differences in learners' abilities, backgrounds, and reasoning strategies. To address these challenges, we present VizQStudio, a visual analytics system that supports instructors in iteratively designing and refining visualization literacy MCQs using MLLM-powered simulated students. Instructors can specify diverse student profiles spanning demographics, knowledge levels, and learning-related traits. The system then visualizes how simulated students reason about and respond to different question components, helping instructors explore potential misconceptions, difficulty calibration, and design trade-offs prior to classroom deployment. We investigate VizQStudio through a mixed-method evaluation, including expert interviews, case studies, a classroom deployment, and a large-scale online study. Overall, this work reframes MLLM-based student simulation in assessment authoring as a design-time, exploratory aid. By examining both its value and limitations in realistic instructional settings, we surface design insights that inform how future systems can support instructor-centered, iterative, and responsible uses of AI for multimodal assessment design in visualization literacy and related domains.