Making the Most of AI Product Demos

Questions for district leaders about student engagement and implementation

The Mission

The goal of this guide is to help district leaders ask what it will take for students to consistently engage with and meaningfully benefit from an AI product in their schools. It complements existing district procurement processes by focusing on student engagement and the conditions needed to support it.

The questions are informed by Robinson et al. (2026), "Access is Not Enough: Human Support Improves Engagement with AI Tutoring." They are discussion prompts, not a product evaluation rubric.

How to Use This Tool

Before the demo: Select the questions most relevant to your schools.

During the demo: Ask for concrete examples and evidence about student participation, human support, and learning.

After the demo: Use the reflection questions to identify what implementation would require, whether your schools can provide those conditions, and what needs follow-up.

GUIDED QUESTIONS: The goal of these questions is not to determine whether a product includes a particular feature, but to understand how the implementation model is expected to support meaningful student engagement. Research suggests that implementation conditions can be just as important as product capabilities when evaluating AI for schools.

Maximizing the Value of Student Screen Time: Not all screen time creates the same learning opportunities. Ask how the implementation model demonstrates evidence of active learning and student progress, not just time spent on the platform.
Priority Question	Listen For or Ask to See	Follow-Up Questions
What evidence do you use to demonstrate that students are actively learning and making progress, rather than simply spending time on the platform?	Separate measures of time logged in, activities completed, and learning demonstrated. Examples of student work or assessments that show what students can do, beyond completing platform activities. For claims of learning gains, an explanation of how learning was measured, over what period, for which students, and compared with what.	“Students using our platform average 30 minutes of practice each week.” Suggested Follow-Up: Does that average include all students assigned access or only active users? Over what period was it measured, and what proportion of students consistently reached the recommended level of use? “Students who complete the recommended lessons make substantial gains in reading.” Suggested Follow-Up: How was reading growth measured, and over what period? Was there a comparison group, and how was it selected? Do the results include all students offered access or only those who completed the lessons?
Research Context: Research Context: Robinson et al. (2026) distinguished between platform usage (minutes spent using the AI platform) and content completion (completed learning activities). Students who received structured human support increased both their platform usage and content completion compared with students who worked independently. Despite these gains, overall platform usage remained well below the provider’s recommended 30 minutes per week. The study did not detect reading achievement gains from adding human support compared with independent access.

From AI Product Access to Student Engagement: Product capabilities matter, but so does how students are expected to begin and sustain meaningful engagement.
Priority Question	Listen For or Ask to See	Follow-Up Questions
How do you ensure students consistently begin using the platform and continue engaging with it over time?	A proposed weekly schedule showing when students would use the platform and how often. Participation data over several weeks or months, including students who never start, stop participating, or fall short of the recommended use. A specific example of how the provider or school identifies students who are not participating and follows up.	"Our platform is so engaging and gamified that students naturally choose to log in." Suggested Follow-Up: What evidence shows that students continue using the platform consistently over time, beyond the initial rollout? "Students can complete the recommended usage independently during free periods or after school." Suggested Follow-Up: How do you know whether students reach the recommended level of use, and what supports are available for students who do not? Suggested Follow-Up: What weekly schedule do you recommend, and what would our schools need to arrange to make that schedule work? Can you show an example from a school with similar staffing and scheduling constraints?
Research Context: Robinson et al. (2026) found that providing students with access alone did not ensure meaningful engagement with the AI platform. Nearly half of students assigned to independent access never used the platform. Average weekly use across all students assigned to independent access was 2.18 minutes in one district and 5.23 minutes in the other, well below the provider's recommended 30 minutes per week. Students who received structured human support engaged with the platform more consistently than students who worked independently.

Human Support as Part of AI Implementation: Effective implementation depends on more than AI alone. Ask how human support is integrated to encourage and sustain student engagement.
Priority Question	Listen For or Ask to See	Follow-Up Questions
What human support does successful implementation require, and which responsibilities belong to the provider and to school staff?	Clearly defined responsibilities for school staff and the provider before, during, and after students use the platform. Estimates of the staff time and training needed to support implementation. An example of what a teacher or facilitator does when a student encounters a technical problem, needs encouragement, or disengages during a session.	"Our platform is designed to work independently with minimal staff involvement." Suggested Follow-Up: What does “minimal staff involvement” mean in practice? What would school staff need to do each week, how much time would those tasks take, and what training and ongoing support would you provide? "Once students log in, they work independently without additional staff support." Suggested Follow-Up: Who monitors whether students continue participating and identifies when they need help? What happens when a student stops using the platform or repeatedly struggles, and who is responsible for following up?
Research Context: Robinson et al. (2026) evaluated an implementation model in which trained mentors supported small groups of students through an opening check-in, platform use, and a closing reflection. Mentors built relationships, encouraged participation, helped troubleshoot technical issues, and established group norms, while the AI platform provided literacy instruction. Students who received this structured human support engaged with the platform more consistently than students who worked independently.

Supporting Equitable Student Engagement: Meaningful engagement may not look the same for every student. Ask how the implementation model supports meaningful engagement across diverse student groups.
Priority Question	Listen For or Ask to See	Follow-Up Questions
How does your implementation model support meaningful engagement among students who may be less likely to use the platform independently, including lower-achieving students and students receiving special education services?	Participation data by student group, including prior achievement and special education status where available, showing who starts, continues, and stops using the platform. Specific examples of how support is adjusted for students with different learning and access needs. A process for identifying participation gaps and checking whether additional support helps close them.	"The platform automatically adapts to each student's level, so students stay engaged without additional support." Suggested Follow-Up: What evidence shows that students who are less likely to engage independently continue using the platform throughout implementation? "Our platform is designed to work equally well for all students." Suggested Follow-Up: How does your implementation model support students who may need additional encouragement or assistance to engage with the platform?
Research Context: Robinson et al. (2026) found that students with lower prior achievement and students receiving special education services were less likely to engage with the AI platform when given independent access.

POST-DEMO REFLECTION: The goal of these questions is not to determine whether a product is ready for adoption, but to help district teams consider what successful implementation would require in their own context. Use them to reflect on both sides of implementation: what the product and provider can support, and what schools would need to prioritize, resource, and sustain for students to engage consistently and meaningfully.

What would successful implementation actually require?
- Based on what we heard, what would schools need to provide for students to use this product consistently and meaningfully? Consider protected time, staffing, adult support, monitoring, scheduling, and follow-through.
Can our schools realistically provide those conditions?
- Do schools have the capacity and commitment to prioritize this implementation? What competing tools, interventions, initiatives, or demands could make consistent use difficult?
What did we hear evidence for, and what are we still assuming?
- Which claims were supported by concrete evidence, implementation data, examples, or documentation? Which depended on assumptions about student motivation, independent use, staff capacity, equitable participation, or the relationship between usage and learning?
What do we still need to know before moving forward?
- What additional evidence or information would help us understand whether this implementation model is likely to work in our context? What should we ask the provider, and what do we need to investigate within our own district?

Conclusion

AI product demonstrations offer an important opportunity to look beyond what a tool can do and ask how it is likely to work in practice. By using this guide, district leaders can bring research-informed questions into vendor conversations, distinguish between evidence-supported claims and assumptions, and identify where additional follow-up may be needed. Together, these steps can support more informed decisions about whether an AI product’s implementation model aligns with the conditions that may support meaningful, equitable, and sustained student engagement.

Tools