Back to feed
News Story
APriority79
MIT Technology Review AI
1 sources

We Still Don't Know How People Are Really Using AI

A new independent research platform called the AI Observatory has aggregated nearly 25,000 real user conversations across seven datasets to provide an unfiltered look at how people actually use AI tools like ChatGPT, Claude, and Grok. The findings reveal significant differences in usage patterns across models, such as Grok users skewing toward news and politics, Claude attracting coders, and ChatGPT being used for homework, which are not captured in reports from AI companies. This independent data aims to help researchers and policymakers make better-informed decisions about AI's benefits and risks.

SynthePulse Insight · AI deep reading

The Truth About AI Use: Independent Observatory Reveals What Company Reports Miss

Version 1 · 1 source

While AI companies show only the data they want you to see, an independent project called AI Observatory analyzes nearly 25,000 real conversations to reveal a broader picture of sensitive behaviors and significant differences between models.

  • AI Observatory aggregates 24,521 conversations from 7 datasets between 2023 and 2025, involving 5,000 users and 52 models.
  • Applying Anthropic's methodology filters out 48% of conversations, which are more likely to involve health, relationships, harassment, and sexual content.
  • The study finds significant model differences: Grok leans toward news and politics with a concentration of misinformation, Claude is used for programming, Gemini for social role-play, and ChatGPT for homework help.
  • Over time, conversations have become longer, with more chit-chat, while AI self-disclosure has decreased, and sensitive uses (such as sexual harassment and hate speech) have declined.
  • Company reports like Anthropic's Economic Index are based on millions of conversations, but independent researchers argue they have blind spots and the data is not public.
Open section navigationThe Need for Independent Observation

The Need for Independent Observation

AI companies like Anthropic and OpenAI regularly publish usage reports, but researchers point out they only release data they want you to see. Stanford PhD student Anka Reuel says there is no independent source to verify these reports. She co-leads the AI Observatory project, which aims to provide independent information to researchers and policymakers by aggregating real conversations collected with user consent.

AI Observatory aggregates 24,521 conversations from 7 datasets between 2023 and 2025, involving 5,000 users and 52 models, including ChatGPT, Gemini, Claude, and Grok. In comparison, Anthropic's latest index is based on 1 million Claude conversations, and OpenAI's report analyzed 1.5 million ChatGPT conversations.

Blind Spots in Company Reports

The Anthropic Economic Index is the most well-known source of usage data, but it focuses on work-related uses. When the AI Observatory team applied Anthropic's methodology, they found that 48% of conversations would be filtered out. These filtered conversations are more likely to involve health and relationships (44.2% vs. 31.2%), adult or illegal topics (7.9% vs. 2.1%), harassment and hate (27.5% vs. 5.66%), and sexual content (16.7% vs. 2.4%).

OpenAI's 2025 report also shows that only 30% of consumer usage is work-related. Although Anthropic has published separate blog posts about support, companionship, and even CSAM, researchers believe a comprehensive view is more useful for consistent understanding.

Differences Between Models and Changes Over Time

The study finds significant differences in usage across models: Grok and Gemini are more often used for information retrieval, with Grok particularly popular for news and politics but also a hotspot for misinformation; Claude is more often used for programming, Gemini for social and role-play, and ChatGPT for homework help.

Over time, conversations have become longer, more detailed, and more chit-chatty, indicating increased AI companionship, while AI self-disclosure has decreased. Sensitive uses (such as sexual harassment and hate speech) have declined, possibly indicating more effective platform safeguards. Different versions of ChatGPT also show differences: the GPT-3.5 era had shorter conversations, while the GPT-4o era had longer and more iterative ones.

Limitations and Future Directions

AI Observatory's data comes from voluntary sources, which may underestimate sensitive uses, so researchers caution that their findings do not represent all AI usage. Additionally, these conversations are a drop in the bucket compared to the data held by large labs.

An Anthropic spokesperson said its research reflects specific team questions and supports external independent research; OpenAI did not respond to requests for comment. AI Observatory's data will be open to researchers, and the team hopes to expand the dataset. Reuel calls on AI companies to share data with independent researchers while protecting privacy, otherwise policymakers will be 'operating completely in the dark.'

Credibility boundary

This article is based on a report from MIT Technology Review, which cites direct statements and specific data from AI Observatory researchers. Company report data (such as Anthropic's and OpenAI's) are source claims and have not been independently verified. AI Observatory's findings are based on voluntarily provided data, which may underestimate sensitive uses.

Insight takeaway

AI usage is far more complex than company reports suggest. Independent data sources are crucial for understanding real usage patterns, but existing independent data is limited, and company data is not public, leaving policymakers with information blind spots.

Sources for this version

  1. We still don't know how people are really using AIMIT Technology Review AI

Primary report

MIT Technology Review AI

Primary source