Date
AI Overview
Why it matters
Young Scratch learners struggle with debugging and interface manipulation. While AI agents could provide direct hands-on demonstrations, evaluating their ability to construct block-based programs through graphical user interfaces remains underexplored.
Study at a glance
- Question: Can multimodal AI agents reliably construct, debug, and extend block-based programs directly within the Scratch interface?
- Setting and data: Benchmark of 83 Scratch tasks, 60 single-step drag tasks, and 200 visual perception tasks, derived from 8 official Scratch Starter Projects.
- Method: Model or benchmark evaluation using browser execution tests across high-level API and low-level mouse and keyboard interaction modes.
- AI tools: GPT-5, Claude-Sonnet-4.5, Gemini-2.5-Pro, DeepSeek-V3.2, Qwen3-VL-32B-Instruct, UI-TARS-1.5-7B
Key findings
- AI agents exhibited a large reasoning–acting gap, reaching up to 78.31% success using high-level commands but falling to at most 14.46% when manipulating the interface directly.
- On single-step drag-and-drop tasks, the best model achieved only 23.33% success, with endpoint localization accuracy remaining low (23% to 32%) even when correct start points were provided.
- Models demonstrated strong static visual perception accuracy of up to 90.5%, indicating that execution and spatial coordinate errors, rather than perception deficits, drive interface failures.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Block-based programming environments such as Scratch play a central role in low-code education, yet evaluating the capabilities of AI agents to construct programs through Graphical User Interfaces (GUIs) remains underexplored. We introduce ScratchWorld, a benchmark for evaluating multimodal GUI agents on program-by-construction tasks in Scratch. Grounded in the Use-Modify-Create pedagogical framework, ScratchWorld comprises 83 curated tasks spanning four distinct problem categories: Create, Debug, Extend, and Compute. To rigorously diagnose the source of agent failures, the benchmark employs two complementary interaction modes: primitive mode requires fine-grained drag-and-drop manipulation to directly assess visuomotor control, while composite mode uses high-level semantic APIs to disentangle program reasoning from GUI execution. To ensure reliable assessment, we propose an execution-based evaluation protocol that validates the functional correctness of the constructed Scratch programs through runtime tests within the browser environment. Extensive experiments across state-of-the-art multimodal language models and GUI agents reveal a substantial reasoning--acting gap, highlighting persistent challenges in fine-grained GUI manipulation despite strong planning capabilities.
Research Categories
What is the application?
Who age?
Why use AI?
Study design
Methods
AI tools
Datasets
Publication Source
Publisher
arXiv
