Researchers tested 18 LLMs on complex online social discussions
The new LongSocialBench reveals that current AI models struggle to follow the logic of threaded online conversations.
A new research paper introduces LongSocialBench, a set of 1,462 multiple-choice questions designed to measure how well long-context AI models understand online forum threads. The dataset includes discussions from Reddit, Hacker News, and Stack Exchange, with some conversations reaching 73,000 tokens in length.
To perform well, an AI must do more than just read text. It needs to keep track of how different people respond to one another, identify where a conversation shifts, and distinguish between separate branches of a debate.
LongSocialBench shows that the missing capability is not context access or prompting alone, but social understanding over structured reply trees.
Researchers tested 18 different AI models using this benchmark. The results show a clear gap between human performance and artificial intelligence. Humans scored 72.4% on these social reasoning tasks, while the models averaged just 43.9%.
Even when the models were given a helping hand by pointing them toward the relevant parts of a conversation, their performance only improved to 55.0%. This suggests that the issue is not just about the length of text a model can hold in its memory, but rather a fundamental struggle with social logic.
The best-performing model, Claude-Opus-4.7, reached 63.0% accuracy when explicitly asked to rule out incorrect answers first. Still, even the top AI remains behind human readers in navigating the messy, structured nature of online comments. If these systems can process massive documents, why do they still stumble over a simple reply chain?
Source: arXiv cs.CL