Date
Publisher
arXiv
Generative artificial intelligence (AI) has found a widespread use in computing education; at the same time, quality of generated materials raises concerns among educators and students. This study addresses this issue by introducing a novel method for diagram code generation with in-context examples based on the Rhetorical Structure Theory (RST), which aims to improve diagram generation by aligning models' output with user expectations. Our approach is evaluated by computer science educators, who assessed 150 diagrams generated with large language models (LLMs) for logical organization, connectivity, layout aesthetic, and AI hallucination. The assessment dataset is additionally investigated for its utility in automated diagram evaluation. The preliminary results suggest that our method decreases the rate of factual hallucination and improves diagram faithfulness to provided context; however, due to LLMs' stochasticity, the quality of the generated diagrams varies. Additionally, we present an in-depth analysis and discussion on the connection between AI hallucination and the quality of generated diagrams, which reveals that text contexts of higher complexity lead to higher rates of hallucination and LLMs often fail to detect mistakes in their output.
Study at a glance
- Question: How do different in-context learning example formats affect the quality and faithfulness of AI-generated educational diagrams?
- Setting and data: 25 educational texts; 150 AI-generated diagrams evaluated by human experts and 600 evaluated automatically across 10 model configurations.
- Method: Bayesian generalized linear mixed modeling of human and automated evaluations across prompting treatments.
Why it matters
AI-generated diagrams from educational materials risk presenting irrelevant or inaccurate information to students through hallucinations, but little is known about how in-context learning formats impact diagram quality and faithfulness.
Key findings
- Providing discourse-to-diagram in-context examples significantly reduced the odds of context unfaithfulness hallucination by nearly 70% compared to zero-shot instructions.
- Automated evaluation showed that discourse-to-diagram examples increased the odds of achieving a perfect faithfulness score by roughly 4.5-fold compared to zero-shot prompts.
- In-context learning had no significant effect on the aesthetic presentation or superficial logical structure of generated diagrams.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
