AI research
College classes where teachers built their own AI tools did better on exams, a 19-university study finds
Published November 18, 2026
A new study randomly sorted 55 college classes at 19 universities into two groups. Students whose instructors built their own AI tools for the course scored higher on in-person, no-AI exams than students in classes that waited a term. The gain was statistically significant (unlikely to be chance), though the paper’s estimate of its size is imprecise. Students were not more engaged or more satisfied. The paper is a preprint, meaning it has not yet been peer-reviewed (checked by outside experts before publication), and its results apply mainly to applied math and data courses.
What happened
The paper, “Instructor-designed AI improves learning in a randomized trial across 19 universities,” was posted to arXiv, a free site where researchers share papers before formal review. It is led by Jun Li of the University of Michigan and lists 52 authors at schools including Northwestern, Cornell, Penn, Yale and Carnegie Mellon.
It is a randomized controlled trial: classes were assigned by chance to one of two groups, so differences between them are less likely to come from who signed up. The experiment involved 42 instructors and 7,200 students from spring 2025 to spring 2026. Some classes got their instructor’s AI tool right away; the others got it one term later and served as the comparison group. The researchers registered their plan in advance, then amended it three times. The headline analysis is slightly simpler than the registered one, and the rule dropping classes whose exams allowed AI was not part of the registration.
The key twist is who designed the AI. The researchers did not hand everyone the same chatbot. Each instructor built something for their own class, with technical help. Simply allowing ChatGPT on homework did not count. Instructors built Socratic coaching agents that ask questions instead of handing over answers (19 classes), practice tutors (18 classes), role-play simulations (8 classes) and course navigation helpers (25 classes).
Learning was measured with uncurved scores on “AI-proof” tests, usually proctored pen-and-paper exams. Classes whose exams allowed AI were left out of that part. The learning analysis covered 4,768 students.
What they found
Results are reported in standard deviations, a measure of how spread out scores are within a class. Compared with the delayed classes, and with each class’s own past results, students with the instructor-built tools scored 0.32 standard deviations higher. Measured class by class, the gain was 0.26.
For scale, the authors cite a median gain of 0.10 from randomized programs in pre-K through 12th grade, and an average of 0.47 for active learning in college STEM classes. So this lands between a typical school program and a well-studied teaching method.
There was no measurable change in engagement (attendance, homework activity) or in course evaluation scores.
Why it matters to a normal person
Earlier studies, including one in PNAS on high school math, found that a generic chatbot without guardrails can harm learning. This paper argues the tool is not the whole story: AI shaped by the teacher, built to coach rather than answer, may help.
If you have a kid in college or are taking a course yourself, the practical lesson is about design. A tutor that makes you try a problem first is a different thing from a chatbot that does the homework.
What it does not mean
It does not mean chatbots make students smarter. The trial compared instructor-built tools with waiting a term, not with generic chatbots. The “teacher-designed is better” idea rests on comparing this study with others.
It does not cover every subject. The classes were mostly applied STEM, such as statistics, machine learning and forecasting. The authors say other subjects still need testing.
It does not show the gains last. The authors say few classes were observed in a second or third term, and, as i10X News notes, the study cannot say which kinds of AI tools worked best.
It is not independently confirmed. The paper is new and not peer-reviewed. Two of its authors also worked on the PNAS study it pushes back on. The analysis code is available only to editors and reviewers on request. The study was funded by Northwestern and the University of Michigan, and the authors declare no competing interests.
Sources
- Instructor-designed AI improves learning in a randomized trial across 19 universities, Jun Li et al., arXiv preprint 2611.19099 (primary; full text read)
- Instructor-designed AI improved exam results in a trial across 19 universities, a preprint reports, On The Wire (Nov 17, 2026)
- Study: AI Built by Instructors Raised Exam Scores in a 19-University Trial, Christopher Ort, i10X News (Nov 17, 2026)