Olmo 3 leaned on customer-service chats and FAQs for social reasoning, a Georgia Tech audit found
The Georgia Tech team sampled 5.68 million of the roughly 1.26 billion documents Olmo 3 trained on to trace where its social skills come from.
In brief
Where an AI learns empathy: 5.68 million documents
Georgia Tech audited the open Olmo 3 model. Customer-support chats, FAQs and literature shaped its social reasoning most. The method needs public training data.
Georgia Tech built a framework that traces a language model's social skills back to the documents it learned from. They tested it on Olmo 3, an open model from Ai2. The work is a paper for the Conference on Language Modeling (COLM) 2026, and Georgia Tech announced it on 5 October.
Ph.D. student Glenn Matlin of the university's School of Interactive Computing led the work with Professor Mark Riedl. Chandreyi (Zini) Chakraborty is a co-author, according to Ai2.
Models that are good at social reasoning lean less on textbooks about emotions and more on FAQs or customer service dialogue.
The numbers
Olmo 3 trained on a corpus called Dolma 3, with about 1.26 billion documents. That is too many to check one by one. So the team sampled about 5.68 million, spread across 576 document categories.
For each document they estimated how much it shaped the model's answers on a benchmark, using a technique called influence functions. They ran four benchmarks: SocialIQA and ARC-Challenge for reasoning, plus the social-science and STEM sets of MMLU for factual knowledge.
What they found
SocialIQA was the odd one out. Olmo 3's answers there leaned heavily on interpersonal writing: literature, social life, customer support and Q&A threads. The other benchmarks drew more on technical documentation and science writing.
Georgia Tech's release links this to tone: an empathetic tone points to dialogue-rich sources, a neutral tone to textbooks. Matlin put it this way: "It can read all it wants from texts that explain anger, but that wouldn't help it determine whether someone is angry."
Ai2 adds that social-science knowledge looked more like STEM knowledge than like social reasoning.
Why it matters
Georgia Tech's release says more people lean on chatbots during personal crises, such as a loss or a breakup. Riedl says that knowing which documents matter could let engineers adjust training, adding more of some documents, fewer, or different ones. He also says the method could audit advice on any topic, from financial to medical, to help people decide when to trust AI.
Limits
The audit needs a model whose training data is public, which is why the team used Olmo 3. Georgia Tech's main text says it cannot be done on proprietary models such as ChatGPT or Claude. A Newswise Q&A, by contrast, says the aim is to improve support in those two. The sources do not show that any model has actually been made more empathetic yet; Riedl describes that as a potential next step. Independent coverage beyond the university and Ai2 is thin.