Harvard's AI Academic Adviser Handles Simple Questions, Fails Nuance

Harvard University deployed an AI-powered academic advising chatbot this summer for incoming first-year students, marking the institution's latest experiment with generative AI in student services. Early testing reveals a predictable pattern: the tool excels at retrieving policy information but struggles with the judgment calls that make human advising valuable.

The Harvard Crimson conducted hands-on testing of Student Compass, the ChatGPT Edu-powered chatbot announced in April by the university's Advising Programs Office. The tool was positioned explicitly as a supplement rather than replacement for human advisers, designed to surface existing online information and route students to appropriate resources.

Why it matters

Harvard's experience offers an early data point for universities weighing similar AI deployments. The results suggest that chatbots can successfully reduce routine inquiries—freeing human advisers for higher-value interactions—but cannot yet replicate the exploratory, relationship-building conversations that help students discover unexpected academic paths. For enterprise AI leaders, it's a familiar lesson: automation works best when scoped to well-defined, rules-based tasks.

Where the chatbot succeeds

Student Compass draws from official policy documents and Faculty of Arts and Sciences websites, including the Student Handbook and departmental pages. For straightforward queries grounded in these sources, students reported positive experiences.

Shriya Dharmapurikar, a member of the class of 2030, used the bot to identify a general education course that would also satisfy requirements for her prospective concentration—a task that would have required manual cross-referencing across multiple web pages. She praised the tool's citation feature, which links to source documents and allows students to verify answers independently.

The bot concludes each response with a disclaimer directing users to consult human advisers for time-sensitive or high-stakes decisions.

Where complexity breaks the model

Joseph Blitzstein, director of undergraduate studies in Harvard's Statistics department, tested Student Compass against questions that regularly surface in concentration advising meetings. He found the bot frequently provided incorrect answers to these more complex queries.

The tool cannot access Q reports (Harvard's course evaluation system), the live course search database, course sites, or many syllabi—limitations that constrain its ability to offer detailed guidance.

Angela Allan, associate director of the History and Literature program, noted that while the bot handled general factual questions adequately, it missed the program's distinctive characteristics. The chatbot's responses emphasized traditional text-based scholarship without capturing the concentration's flexibility around film, comics, and video games—nuances that emerge through conversation with faculty.

Joseph Saba, another first-year student, described the bot as "rigid" compared to human advisers who can encourage students to explore unexpected academic directions during their critical first semester.

Faculty awareness and adoption

Several departmental directors of undergraduate studies told The Crimson they had received limited advance information about the chatbot. Jeffrey Miron, economics director of undergraduate studies, said his department focuses primarily on existing concentrators and saw no obvious action items related to the new tool. He characterized himself as "moderately skeptical" that AI would dramatically improve advising outcomes, viewing it instead as a useful but limited tool.

Assistant Director for First-Year Academics Brooks Lambert-Sluder reaffirmed that Student Compass was designed as a "starting point" for questions easily addressed by existing resources, with human advisers handling more complex needs. He said the office welcomes feedback as part of ongoing improvement efforts.

These details were first reported by The Harvard Crimson.