Date
Publisher
arXiv
Large Language Models (LLMs) are increasingly capable of generating complete
applications from natural language instructions, creating new opportunities in
science and education. In these domains, interactive scientific demonstrations
are particularly valuable for explaining concepts, supporting new teaching
methods, and presenting research findings. Generating such demonstrations
requires models to combine accurate scientific knowledge with the ability to
implement interactive front-end code that behaves correctly and responds to
user actions. This capability goes beyond the scope of existing benchmarks,
which typically evaluate either knowledge question answering without grounding
in code or static web code generation without scientific interactivity. To
evaluate this integrated ability, we design a hybrid framework that combines
programmatic functional testing to rigorously verify interaction logic with
visually-grounded qualitative testing to assess rendered outputs against
reference snapshots. Building on this framework, we present InteractScience, a
benchmark consisting of a substantial set of carefully designed questions
across five scientific domains, each paired with unit tests, reference
snapshots, and checklists. We evaluate 30 leading open- and closed-source LLMs
and report results that highlight ongoing weaknesses in integrating domain
knowledge with interactive front-end coding. Our work positions InteractScience
as the first benchmark to automatically measure this combined capability with
realistic interactive operations, providing a foundation for advancing reliable
and educationally useful scientific demonstration code generation. All code and
data are publicly available at https://github.com/open-compass/InteractScience.
Study at a glance
- Question: How well do leading AI models generate interactive, visually accurate web code for scientific demonstrations?
- Setting and data: 150 scientific demonstration problems across five disciplines curated from the Wolfram Demonstrations Project, evaluated across 30 AI models.
- Method: Benchmark evaluation combining programmatic unit test execution and vision-language model qualitative judging against reference snapshots.
Why it matters
Interactive scientific demonstrations bring complex concepts to life, but generating them requires combining deep domain knowledge with precise reactive coding. Existing benchmarks fail to evaluate both interactive functionality and scientifically accurate visualization.
Key findings
- Across 30 models, generating flawless scientific demonstrations proved difficult, with models rarely passing all functional unit tests on more than 16% of problems.
- Models frequently generated working interface controls, achieving action success rates over 85%, but often failed to bind them to correct underlying scientific logic.
- Logic and scientific errors were the most common failure type (53.33%), followed by numerical or parameter errors (47.78%) and rendering failures (42.22%).
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
