Date
Publisher
arXiv
Involving subject matter experts in prompt engineering can guide LLM outputs
toward more helpful, accurate, and tailored content that meets the diverse
needs of different domains. However, iterating towards effective prompts can be
challenging without adequate interface support for systematic experimentation
within specific task contexts. In this work, we introduce PromptHive, a
collaborative interface for prompt authoring, designed to better connect domain
knowledge with prompt engineering through features that encourage rapid
iteration on prompt variations. We conducted an evaluation study with ten
subject matter experts in math and validated our design through two
collaborative prompt-writing sessions and a learning gain study with 358
learners. Our results elucidate the prompt iteration process and validate the
tool's usability, enabling non-AI experts to craft prompts that generate
content comparable to human-authored materials while reducing perceived
cognitive load by half and shortening the authoring process from several months
to just a few hours.
Study at a glance
- Question: Can a collaborative tool help math experts write effective AI prompts to create educational materials?
- Setting and data: 10 math subject matter experts and 358 learners.
- Method: Collaborative prompt-writing evaluation sessions with experts and a student learning gain study.
Why it matters
Subject matter experts can guide LLM outputs toward helpful, accurate, and tailored content. However, iterating towards effective prompts is challenging without adequate interface support for systematic experimentation within specific task contexts.
Key findings
- In an evaluation with 10 math experts and 358 learners, a collaborative prompt tool helped educators produce content comparable to human-created materials.
- The prompt authoring interface shortened the instructional content creation process from several months to just a few hours.
- Using the tool reduced educators' perceived cognitive load by half during the prompt design process.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
