Search and Filter

Repository Feedback

Your feedback helps us improve the repository's content relevance and usability. Please share your thoughts to help us better serve researchers and practitioners.

Submit feedback

Submit a research study

Contribute to the repository:

Add a paper

Semantic Variability Of LLM-Generated Replies Across LLMs: Implications For Designing Conversation-Based Assessment

Jiangang Hao, Educational Testing Service
Date
Publisher
arXiv
Link
https://arxiv.org/abs/2608.24920v1
This study examines whether LLM-generated replies remain semantically consistent when the underlying LLM changes. Using messages from real collaborative conversations, we compared the semantic similarity of generated replies across LLMs under two conditions: with and without preceding chat history. Results show that model choice and conversational context both affect response similarity and alignment with human replies. These findings indicate that prompting and conversational context alone may not be sufficient to preserve response consistency across LLMs, highlighting the need for infrastructure and design strategies that can maintain stable and comparable responses amid the rapid and continuous evolution of LLMs.
Study at a glance
  • Question: Do AI-generated conversation replies remain semantically consistent when switching models and providing chat history?
  • Setting and data: 61 collaborative science chat messages from 99 adult dyadic teams recruited via Amazon Mechanical Turk in 2018.
  • Method: Comparative evaluation analyzing embedding cosine similarity and linear mixed-effects modeling across models and context conditions.
Why it matters

LLM-based agents adapt dynamically to learners, but uncontrolled response variations across models may introduce construct-irrelevant variance that threatens validity, reliability, fairness, and comparability in standardized conversation-based assessments.

Key findings
  • Models generated replies more semantically similar to their own outputs (similarity 0.715 to 0.795) than to other models (0.443 to 0.604).
  • Generated replies were consistently dissimilar to actual human replies across all tested models, though adding conversational history slightly improved alignment.
  • Adding conversational history meaningfully altered response content (cross-history similarity 0.40 to 0.45) and moderated the influence of focal-message similarity on generated replies.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Why use AI?
Other
Study design
Technical – Computational
Download PDF