Harvard students who learned physics from an AI tutor improved their test scores over twice as much as classmates taught in class. The study tested one AI tutor over two sessions of one introductory physics course. Its authors call the tutor’s effectiveness unexpected.
Researchers split students in Harvard’s largest physics class into two groups at random, and 194 students took part in both sessions and completed every test. One group attended the regular class, where students solved problems together in their usual groups, a research-based format known as active learning. The other used the AI tutor at home on their own, and both worked through the same worksheet after the same introduction. In week two, the groups switched.
Every student took a test before and after each session, written by a team member who played no part in designing the tutor or teaching. In both weeks, the tutor group scored higher. Many scores bunched near the top of the test, so the paper expects the true gain to be larger than reported.
The tutor was built for this course. Chatbots are generally built to be helpful, the researchers note, and have not been trained to follow good teaching practice. That practice includes keeping students actively involved and giving accurate, timely feedback. This tutor was designed around the same practices as the in-person sessions, and the software students used took several months to build.
For every question, instructors who know the material wrote the tutor a set of instructions, called a prompt, that included complete answers worked out step by step. Eighty-three percent of students said the tutor’s explanations matched or beat the instructors’. Written instructions alone did not reliably keep problems with several parts in order. So the software set the order, walking students through every part of every problem as the instructor would.
Gains like these require interactions designed around research-based teaching practices, the paper says. The authors name six conditions the results might depend on. Two of them, the instructors’ prompts and the structure that guides students, describe the design. The other four are the students’ varied paces, the instructional videos, the AI model and the subject matter.
The authors do not assume that structured AI tutoring will always beat classroom teaching, and they advise against AI entirely replacing in-class teaching.
