Search and Filter

Experimental Evidence Of The Effects Of Large Language Models Versus Web Search On Depth Of Learning

Date

AI Overview

Why it matters

Learning from AI syntheses rather than web search may cause people to develop shallower knowledge, as pre-summarized results prevent users from actively discovering and synthesizing sources themselves.

Study at a glance
  • Question: Does learning from large language model summaries lead to shallower knowledge than using traditional web search?
  • Setting and data: Seven online and laboratory experiments with 10,462 participants; locations and dates not stated.
  • Method: Seven online and laboratory experiments.
Key findings
  • Learning from AI syntheses risks developing shallower knowledge than learning through standard web search, even when the core facts provided are identical.
  • Participants who learned from AI syntheses reported developing shallower knowledge even when summaries were augmented with real-time web links.
  • People learning from AI summaries felt less invested and produced advice that was sparser, less original, and less likely to be adopted.

Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.

The effects of using large language models (LLMs) versus traditional web search on depth of learning are explored. A theory is proposed that when individuals learn about a topic from LLM syntheses, they risk developing shallower knowledge than when they learn through standard web search, even when the core facts in the results are the same. This shallower knowledge accrues from an inherent feature of LLMs—the presentation of results as summaries of vast arrays of information rather than individual search links—which inhibits users from actively discovering and synthesizing information sources themselves, as in traditional web search. Thus, when subsequently forming advice on the topic based on their search, those who learn from LLM syntheses (vs. traditional web links) feel less invested in forming their advice, and, more importantly, create advice that is sparser, less original, and ultimately less likely to be adopted by recipients. Results from seven online and laboratory experiments (n = 10,462) lend support for these predictions, and confirm, for example, that participants reported developing shallower knowledge from LLM summaries even when the results were augmented by real-time web links. Implications of the findings for recent research on the benefits and risks of LLMs, as well as limitations of the work, are discussed.

Research Categories

What is the application?
Who is the user?
Who age?

Publication Source

Publisher
PNAS Nexus