Seventeen large language models struggled with basic word relationship tasks

Large language models frequently stumble when asked to understand how words relate to each other, a new study reveals. Researchers tested 17 different AI systems using a dataset of 3,069 sense pairs across five languages, including English, Greek, Romanian, Turkish, and Japanese.

In this task, the AI had to predict the relationship between two different word definitions, known as glosses. The team flipped the order of these definitions to see if the models remained consistent in their logic.

Imagine you define a 'cat' as a 'feline' and a 'feline' as a 'cat'. An AI should logically understand these connections equally in both directions, but that was rarely the case.

The results highlight significant gaps in how these models process basic human language structures. Only a few models managed to perform significantly better than a random guess, regardless of whether they were given zero or five examples to learn from.

Specifically, the systems often struggled with hypernymy, which is the relationship between a general term and a specific one. They also had trouble identifying holonymy, the connection between a whole and its parts.

Many models failed to recognize that if a relationship exists in one direction, it should logically follow a specific rule in reverse. Does it surprise you that models so capable at writing can still fail at these foundational logic puzzles?

The study did not cover every possible language or every existing model, meaning these findings might not apply to all future iterations. The researchers specifically focused on consistency and accuracy, leaving room for improvement in how AI models represent semantic knowledge.