Search and Filter

Short-Term Gains, Long-Term Gaps: The Impact Of GenAI and Search Technologies On Retention

Date

AI Overview

Why it matters

Generative AI tools provide synthesized, ready-to-use information, raising critical questions about whether these tools impact student learning and whether their effects vary across learning tasks of different cognitive complexities.

Study at a glance
  • Question: How do generative AI, search engines, and e-textbooks influence students' learning performance and long-term retention across different cognitive tasks?
  • Setting and data: 123 undergraduate information systems students at a private university.
  • Method: Randomized controlled laboratory experiment with pretests, immediate quizzes, and a three-week retention post-test.
  • AI tools: ChatGPT
Key findings
  • On immediate lower-order recall tasks, the ChatGPT group scored significantly higher than the e-textbook and control groups, with mean differences of 16.60 and 22.39 points.
  • On immediate applying tasks, both the ChatGPT and Google groups scored significantly higher than the control group, with mean differences of 20.15 and 18.81 points.
  • Three weeks later, ChatGPT and Google groups experienced retention drops on recall tasks, aligning with e-textbook scores, while no significant tool differences emerged on higher-order tasks.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

The rise of Generative AI (GenAI) tools, such as ChatGPT, has transformed how students access and engage with information, raising questions about their impact on learning outcomes and retention. This study investigates how GenAI (ChatGPT), search engines (Google), and e-textbooks influence student performance across tasks of varying cognitive complexity, based on Bloom's Taxonomy. Using a sample of 123 students, we examined performance in three tasks: [1] knowing and understanding, [2] applying, and [3] synthesizing, evaluating, and creating. Results indicate that ChatGPT and Google groups outperformed the control group in immediate assessments for lower-order cognitive tasks, benefiting from quick access to structured information. However, their advantage diminished over time, with retention test scores aligning with those of the e-textbook group. For higher-order cognitive tasks, no significant differences were observed among groups, with the control group demonstrating the highest retention. These findings suggest that while AI-driven tools facilitate immediate performance, they do not inherently reinforce long-term retention unless supported by structured learning strategies. The study highlights the need for balanced technology integration in education, ensuring that AI tools are paired with pedagogical approaches that promote deep cognitive engagement and knowledge retention.