Date
Publisher
arXiv
As AI systems increasingly take on instructional roles - providing feedback, guiding practice, evaluating work - a fundamental question emerges: does it matter to learners who they believe is on the other side? We investigated this using a three-condition experiment (N=148) in which participants completed a creative coding tutorial and received feedback generated by the same large language model, attributed to either an AI system (with instant or delayed delivery) or a human teaching assistant (with matched delayed delivery). This three-condition design separates the effect of source attribution from the confound of delivery timing, which prior studies have not controlled. Source attribution and timing had distinct effects on different outcomes: participants who believed the human attribution spent more time on task than those receiving equivalently timed AI-attributed feedback (d=0.61, p=.013, uncorrected), while the delivery delay independently increased output complexity without affecting time measures. An exploratory analysis revealed that 46% of participants in the human-attributed condition did not believe the attribution, and these participants showed worse outcomes than those receiving transparent AI feedback (code complexity d=0.77, p=.003; time on task d=0.70, p=.007). These findings suggest that believed human presence may carry motivational value, but that this value depends on credibility. For computing educators, transparent AI attribution may be the lower-risk default in contexts where human attribution would not be credible.
Study at a glance
- Question: Does believing feedback comes from a human rather than AI affect learner behavior, independently of delivery timing?
- Setting and data: Online experiment with 148 adults recruited via Prolific completing a creative coding tutorial in p5.js.
- Method: Randomized controlled trial (online experiment with three conditions and linear mixed-effects modeling)
Why it matters
As instructors use artificial intelligence to draft feedback, existing research often confounds source attribution with delivery timing and ignores whether learners actually find human attribution credible.
Key findings
- Learners who believed feedback came from a human teaching assistant spent about 28 percent more time on task than those receiving equivalently delayed AI feedback.
- A 60-second delivery delay independently increased code complexity and length, while source attribution influenced the time spent working rather than code output.
- Nearly half of participants doubted the human attribution, producing shorter, less complex code and spending less time focused than those receiving transparent AI feedback.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Study design
