Some education stakeholders are familiar with the 2025 meta-analysis of Generative AI in education from the Nature Publishing Group that was retracted after receiving over 500 citations. Fewer people are aware that over a dozen similar meta-analyses have been published--many with the same serious issues as the retracted paper--and together they have over 2000 citations. Patrick O'Neill, faculty member at Ivy Tech Community College, has published a forensic review of 14 meta-analyses of AI in Education called "Presumed Effective: The Manufacturing of an Evidence Base for AI-in-Education Through Flawed Meta-Analysis." (https://osf.io/preprints/edarxiv/n29zd_v3) The paper compellingly argues that all 14 meta-analyses report positive associations between AI use and learning outcomes, have been published in leading journals, and have disqualifying flaws. Prof. O'Neill will offer a summary of his findings, and then he'll be joined in discussion by Professor Elizabeth Tipton from Northwestern--a leading expert in educational meta-analysis--and Professor Susanna Loeb from Stanford, whose AI in Education Hub has a comprehensive database of AI in Education studies, including the very limited number of studies that can make defensible causal claims about the effect of LLMs on Education. Justin Reich of MIT will host and moderate. This conversation is open to any education stakeholder, but we warmly and especially invite journal editors, conference program chairs, and AI in education reviewers to join us. Evidence-based policy and practice in classrooms and institutions depends upon a trustworthy base of evidence, especially in reviews and meta-analyses. We'll discuss the very troubling current state of affairs and pathways that editors, program chairs, reviewers, and authors can take towards a better future where practicing educators have rigorous reviews and meta-science to draw upon in their vital work.
News
© Stanford University. Stanford, California 94305.