Search and Filter

Kidspeak: A General Multi-Purpose LLM For Kids' Speech Recognition And Screening

Date

AI Overview

Why it matters

Current AI speech models rely on clear adult speech and struggle with children's unique speech patterns, developmental stages, and speech-language pathologies, creating barriers to early screening and therapy.

Study at a glance
  • Question: Can an instruction-tuned language model and automated aligner accurately transcribe and screen children's speech across multiple diagnostic tasks?
  • Setting and data: Over 57 hours of speech audio from over 2,200 children aged 1 to 13, including clinical recordings.
  • Method: System development and multi-task model evaluation using instruction tuning and an automated forced-alignment pipeline.
  • AI tools: KidSpeak, Flexible and Automatic Speech Aligner (FASA), Whisper, Vicuna 7B, PandaGPT, wav2vec 2.0
Key findings
  • KidSpeak with a phonetically trained Whisper encoder achieved an average accuracy of 87.0% across four pediatric speech classification and transcription tasks.
  • The model achieved 88.8% accuracy on speech disorder classification and over 94% accuracy on age-group classification.
  • The FASA alignment tool reduced word error rates by 13.6 times compared to human annotations on children's speech samples without requiring manual labor.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

With the rapid advancement of conversational and diffusion-based AI, there is a growing adoption of AI in educational services, ranging from grading and assessment tools to personalized learning systems that provide targeted support for students. However, this adaptability has yet to fully extend to the domain of children's speech, where existing models often fail due to their reliance on datasets designed for clear, articulate adult speech. Children, particularly those in early developmental stages or with speech and language pathologies, present unique challenges that current AI models and datasets are ill-equipped to handle. To address this, we introduce KidSpeak, a multi-task speech-enhanced Foundation Model capable of both generative and discriminative tasks specifically tailored to children's speech patterns. Our framework employs a two-stage training process that incorporates phonetic knowledge into the speech encoder, achieving an average accuracy of 87% across four separate tasks. Furthermore, recognizing the limitations of scalable human annotation and existing speech alignment tools, we propose the Flexible and Automatic Speech Aligner (FASA) and leverage the method to construct high quality datasets for training and evaluation. This novel alignment tool significantly improves the quality of aligned children's speech from noisy data, enhancing data quality by 13.6x compared to human annotations, as demonstrated on the CHILDES dataset. To the best of our knowledge, KidSpeak and FASA represent the first comprehensive solution designed for speech and language therapy in children, offering both a multi-purpose speech LLM and a robust alignment tool.