Seven language models failed to grasp basic logic in plural sentences
New research shows that popular AI models often struggle to understand situations that are neither true nor false, leading to a consistent logic error.
A recent evaluation of seven open-weight language models revealed a significant gap in how they process plural sentences. The study examined whether AI can handle linguistic statements that are technically neither true nor false.
Imagine you have a group of children and only some of them start laughing. If you say the kids laughed or the kids did not laugh, both statements are logically tricky because they do not apply to the whole group.
Current open-weight LLMs have acquired the truth conditions of plural predication but not its trivalent logic.
Researchers tested models against a dataset called HomogeneityNLI, which includes 1,000 examples across various logical conditions. They found that these models suffer from a strong bias toward simple binary outcomes.
When faced with sentences about groups, the AI often forces a true or false label. Even when prompted with chain-of-thought instructions or specific linguistic rules, the models frequently maintained this rigid error.
In most cases, the AI incorrectly judged negated plural sentences as false, even when the situation was not clear-cut. The models recognized that the sentence involved plural subjects but failed to apply the nuanced logic required to distinguish between partial and total group actions.
Hardly any model correctly identified these truth-value gaps in more than 15% of the scenarios presented during testing.
Does this inability to handle uncertainty mean that AI logic is essentially limited by its own training data? While these models have clearly mastered the surface structure of language, they have not yet mastered the trivalent logic that humans use to navigate ambiguity.
Source: arXiv cs.CL