Date
Publisher
arXiv
As generative AI (GenAI) becomes increasingly embedded in undergraduate academic writing, how students rely on these tools, rather than simply whether they use them, has become a central question for learning, academic integrity, and educational equity. Existing measures of reliance were developed inductively, focused on discrete problem-solving tasks, and validated mainly with homogeneous samples. This study developed and validated the GenAI Reliance Types Scale (GenAI-RTS), a 20-item instrument measuring four theoretically derived types of GenAI reliance: Strategic, Instrumental, Dependent, and Dialogic. Validation followed the multisource framework of the Standards for Educational and Psychological Testing, drawing on a survey of 382 undergraduates at a U.S. Minority-Serving Institution and interviews with 14 purposively sampled students. Confirmatory factor analyses of six competing models supported a five-factor structure in which Strategic Reliance comprises two facets, Deliberate Use and Critical Evaluation, alongside Instrumental, Dependent, and Dialogic factors (CFI = .92, RMSEA = .08; DWLS CFI = .98, RMSEA = .07). Subscale reliability was acceptable to good (omega = .75-.88), and scalar measurement invariance held across gender, first-generation status, and STEM/non-STEM majors, to our knowledge the first such evidence for a GenAI reliance instrument. Rasch analysis indicated that a five-point response format would improve category functioning. Strategic reliance was positively associated with AI literacy, and the reliance types differentiated students across multiple writing process and outcome variables. The GenAI-RTS offers researchers and educators a theoretically grounded, psychometrically validated instrument for identifying undergraduate reliance profiles and supporting research, assessment, and AI literacy intervention.
Study at a glance
- Question: How can students' different ways of relying on generative AI in academic writing be reliably measured?
- Setting and data: Mid-Atlantic U.S. Minority-Serving Institution, 382 undergraduate survey respondents and 14 interviewees.
- Method: Mixed methods combining survey factor analysis, Rasch analysis, and qualitative interviews.
Why it matters
As generative AI use becomes nearly universal among undergraduates, distinguishing whether students use AI loses diagnostic value. Understanding how students rely on these tools is central for learning, academic integrity, and educational equity.
Key findings
- A five-factor structure fit student writing reliance best, distinguishing deliberate use, critical evaluation, instrumental reliance, dependent reliance, and dialogic reliance.
- The reliance scale demonstrated scalar measurement invariance across gender, first-generation college status, and STEM versus non-STEM majors.
- Strategic reliance correlated positively with AI literacy, while critical evaluation and dependent reliance emerged as virtually unrelated behaviors.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
