AI Engagement Metrics Miss What Matters: Displacement vs. Use

Three hours of AI chatbot conversation looks identical in usage logs whether someone uses it as a reflective journal after dinner with friends or instead of calling back a friend after an argument. Yet the human impact of these two scenarios couldn't be more different.

This measurement gap sits at the heart of a growing challenge for AI developers and researchers, according to psychologist Amy Jean Clark of CQUniversity, Australia. In a post for the Communications of the ACM, Clark argues that current engagement metrics fundamentally miss what matters: the opportunity cost of AI interaction.

The counterfactual problem

Clark frames the core issue as a counterfactual question—what would this person have done instead, and did losing that alternative matter? Time spent, emotional attachment, and loneliness correlations offer clues but don't reveal what the AI interaction replaced.

Recent research illustrates why this distinction matters. A Journal of Consumer Research study found that brief AI companion interactions reduced loneliness in the moment, but researchers found no evidence the gains lasted beyond a week. A 12-month study in Psychological Science tracked bidirectional relationships between chatbot use and isolation measures, but the authors cautioned against strong causal claims. A separate four-week randomized trial found that participants who chose to use chatbots more had less favorable outcomes—though that pattern doesn't prove longer use caused the decline.

None of these studies could determine what everyday AI use replaced in participants' lives.

Three types of change

Clark points to the Costs and Opportunities of Technology in Context framework, which separates three distinct mechanisms. Displacement means time moves from one activity to another. Substitution means technology becomes a different way to meet the same need. Interference means it distracts from something already happening.

Any of these can help or harm depending on context. An AI conversation might meet a need that would otherwise go unmet, support contact with another person, or replace an unsafe option. It could also crowd out sleep, healthcare, relationship repair, or social contact.

Better measurement approaches

Clark recommends collecting information soon after selected sessions: why the person opened the system, what they would have done if it were unavailable, whether that alternative was genuinely available, whether the interaction interrupted anything, and what they did next. With consent, researchers could track follow-through—did the person contact someone, seek help, sleep, or return to work?

Comparing people with their own usual behavior matters more than comparing users with non-users. A 15-day diary study of smartphone communication found people had less face-to-face contact on days when they had more than usual smartphone communication—yet people who generally used smartphones more didn't report less face-to-face contact overall. A difficult day could drive both more AI use and less human contact without the AI causing the isolation.

Why it matters

As AI companions become more sophisticated and widely deployed, product teams are optimizing for engagement without understanding whether they're building what Clark calls a "contact gate"—a bridge toward people, care, and purposeful activity—or a "contact sink" that absorbs the intention to act. For enterprise AI deployments and consumer products alike, measuring whether users complete their goals, stop when they intend to, and follow through offline offers more meaningful signals than session length or frequency. Until studies can answer what AI use replaces, under what constraints, and with what effect, high engagement should prompt investigation rather than celebration.

Clark's analysis was originally published in the Communications of the ACM's BLOG@CACM section.