Date
Publisher
arXiv
Enriching the quality of early childhood education with interactive math
learning at home systems, empowered by recent advances in conversational AI
technologies, is slowly becoming a reality. With this motivation, we implement
a multimodal dialogue system to support play-based learning experiences at
home, guiding kids to master basic math concepts. This work explores Spoken
Language Understanding (SLU) pipeline within a task-oriented dialogue system
developed for Kid Space, with cascading Automatic Speech Recognition (ASR) and
Natural Language Understanding (NLU) components evaluated on our home
deployment data with kids going through gamified math learning activities. We
validate the advantages of a multi-task architecture for NLU and experiment
with a diverse set of pretrained language representations for Intent
Recognition and Entity Extraction tasks in the math learning domain. To
recognize kids' speech in realistic home environments, we investigate several
ASR systems, including the commercial Google Cloud and the latest open-source
Whisper solutions with varying model sizes. We evaluate the SLU pipeline by
testing our best-performing NLU models on noisy ASR output to inspect the
challenges of understanding children for math learning in authentic homes.
Study at a glance
- Question: How well can voice AI systems understand children's spoken language during play-based math learning at home?
- Setting and data: Audio data from children participating in gamified math learning activities at home.
- Method: Performance evaluation testing different speech recognition and language understanding AI models on children's speech.
Why it matters
Recent advances in conversational AI can enrich early childhood education through interactive math learning at home. However, systems face challenges recognizing and understanding children's speech in authentic home environments.
Key findings
- In home-based gamified math learning, conversational AI systems encountered substantial difficulty reliably understanding children's spoken language in realistic, everyday environments.
- Multi-task language understanding architectures provided clear advantages for recognizing children's intentions and extracting key concepts in the math learning domain.
- Speech recognition systems produced noisy transcripts of children's voices in authentic homes, posing major obstacles for interpreting their math learning responses.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
