Six-Month Classroom Study Reveals Which AI Tools Actually Help Students Learn

A nonprofit education organization spent six months watching students and teachers use artificial intelligence tools in real classrooms across seven states, and the results challenge both AI evangelists and skeptics.

Emily Freitag, co-founder and CEO of Instruction Partners, led a team that observed AI use in 16 school systems, interviewing more than 100 teachers, administrators, and education technology executives. The organization released its initial findings this week, examining 20 student-facing AI products already deployed in classrooms.

Why it matters

School leaders face relentless sales pitches for AI products while simultaneously hearing calls to ban the technology entirely. This research provides ground-level evidence about which tools actually support learning and which undermine it — information that's been largely absent from a debate dominated by speculation and vendor claims. Only seven of the 20 products studied have independent research examining outcomes across different student groups.

Three categories, vastly different results

The research team divided AI tools into three categories and found dramatically different outcomes. General-purpose chatbots like ChatGPT, Claude, and Gemini posed the biggest threat to student thinking. Education-focused platforms such as Brisk Teaching, Google Classroom, and MagicSchool produced inconsistent results depending on teacher implementation. Targeted instructional tools built for specific learning tasks — including Amira Learning, Khan Academy's Khanmigo, and Quill — delivered the most consistent and strongest learning experiences.

Freitag, a former assistant commissioner of curriculum and instruction in Tennessee, framed the investigation around two questions: How does each tool compare to traditional paper-and-pencil instruction, and how does it compare to the best possible learning experience in that subject?

Implementation trumps design

The study revealed a critical finding: how teachers use AI tools matters more than the tools themselves. The same product produced dramatically different results depending on whether teachers established clear routines and actively monitored student progress through real-time dashboards.

In one case, a teacher used a math app to display four students' solutions side-by-side on a screen, allowing the class to evaluate different approaches. "I was stunned by how much kids learned from looking at each other's work in that way," Freitag said. When used to facilitate social learning, the tool extended teacher capabilities rather than replacing them.

By contrast, apps designed for solo use often failed to hold student attention beyond 10 minutes, leaving students disengaged by the end of longer lessons.

Gaming the system

Teachers reported widespread frustration with apps students could easily manipulate. In one instance, students discovered they could complete assignments by repeatedly hitting the refresh button to answer the same question instead of working through the full problem set. Others simply copied answers from Google and pasted them into lessons.

The research also documented students ignoring detailed, actionable feedback on their writing — a reminder that providing feedback and ensuring students use it are separate challenges.

Concerns about skill isolation and text releveling

Freitag identified two troubling trends. First, AI's ability to identify struggling students and assign isolated skill practice could backfire if students practice reading skills without engaging with meaningful texts and building knowledge. Second, AI's popular one-click feature for simplifying reading passages risks becoming "a permanent lower setting" rather than a temporary bridge to grade-level material.

The research team will expand to examine 30 additional products this winter. Instruction Partners also plans to work with Stanford's AI in Education Lab to independently verify vendor impact claims by winter 2027.

The findings were first reported by The 74, which detailed the year-long AI in Action Learning Tour.