Search and Filter

Calibrating Generative AI To Produce Realistic Essays For Data Augmentation

Date

AI Overview

Why it matters

Automated scoring engines can be limited by sparse, costly human-scored data. Generating synthetic essays using artificial intelligence can help, but it risks producing unrealistic text that degrades scoring performance.

Study at a glance
  • Question: Which prompt engineering strategies generate synthetic essays that best preserve original writing quality and appear most realistic?
  • Setting and data: Midwestern U.S. state, 96 ninth-grade public school essays from a summative assessment sample of 6,145 students, producing 288 AI-simulated essays.
  • Method: Experimental evaluation comparing blind expert scoring and realism ratings across three AI prompting strategies against professional rater scores.
  • AI tools: GPT
Key findings
  • The "predict next" prompting method produced the highest rater agreement (0.72) and a quadratically weighted kappa of 0.74, closely matching real essay agreement.
  • Expert raters incorrectly labeled "predict next" and "25 examples" synthetic essays as real 75% and 82% of the time, respectively, showing high perceived realism.
  • The "sentence" prompting method closely preserved original average quality scores, but raters easily detected its generated essays as synthetic 74% of the time.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Data augmentation can mitigate limited training data in machine-learning automated scoring engines for constructed response items. This study seeks to determine how well three approaches to large language model prompting produce essays that preserve the writing quality of the original essays and produce realistic text for augmenting ASE training datasets. We created simulated versions of student essays, and human raters assigned scores to them and rated the realism of the generated text. The results of the study indicate that the predict next prompting strategy produces the highest level of agreement between human raters regarding simulated essay scores, predict next and sentence strategies best preserve the rated quality of the original essay in the simulated essays, and predict next and 25 examples strategies produce the most realistic text as judged by human raters.

Research Categories

What is the application?
Who is the user?
Why use AI?

Publication Source

Publisher
arXiv