Search and Filter

Repository Feedback

Your feedback helps us improve the repository's content relevance and usability. Please share your thoughts to help us better serve researchers and practitioners.

Submit feedback

Submit a research study

Contribute to the repository:

Add a paper

Interactscience: Programmatic And Visually-Grounded Evaluation Of Interactive Scientific Demonstration Code Generation

Date
Publisher
arXiv
Large Language Models (LLMs) are increasingly capable of generating complete applications from natural language instructions, creating new opportunities in science and education. In these domains, interactive scientific demonstrations are particularly valuable for explaining concepts, supporting new teaching methods, and presenting research findings. Generating such demonstrations requires models to combine accurate scientific knowledge with the ability to implement interactive front-end code that behaves correctly and responds to user actions. This capability goes beyond the scope of existing benchmarks, which typically evaluate either knowledge question answering without grounding in code or static web code generation without scientific interactivity. To evaluate this integrated ability, we design a hybrid framework that combines programmatic functional testing to rigorously verify interaction logic with visually-grounded qualitative testing to assess rendered outputs against reference snapshots. Building on this framework, we present InteractScience, a benchmark consisting of a substantial set of carefully designed questions across five scientific domains, each paired with unit tests, reference snapshots, and checklists. We evaluate 30 leading open- and closed-source LLMs and report results that highlight ongoing weaknesses in integrating domain knowledge with interactive front-end coding. Our work positions InteractScience as the first benchmark to automatically measure this combined capability with realistic interactive operations, providing a foundation for advancing reliable and educationally useful scientific demonstration code generation. All code and data are publicly available at https://github.com/open-compass/InteractScience.
Study at a glance
  • Question: How well do leading AI models generate interactive, visually accurate web code for scientific demonstrations?
  • Setting and data: 150 scientific demonstration problems across five disciplines curated from the Wolfram Demonstrations Project, evaluated across 30 AI models.
  • Method: Benchmark evaluation combining programmatic unit test execution and vision-language model qualitative judging against reference snapshots.
Why it matters

Interactive scientific demonstrations bring complex concepts to life, but generating them requires combining deep domain knowledge with precise reactive coding. Existing benchmarks fail to evaluate both interactive functionality and scientifically accurate visualization.

Key findings
  • Across 30 models, generating flawless scientific demonstrations proved difficult, with models rarely passing all functional unit tests on more than 16% of problems.
  • Models frequently generated working interface controls, achieving action success rates over 85%, but often failed to bind them to correct underlying scientific logic.
  • Logic and scientific errors were the most common failure type (53.33%), followed by numerical or parameter errors (47.78%) and rendering failures (42.22%).

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

What is the application?
Who is the user?