AI platforms such as ChatGPT are known for their advanced prediction capabilities. They use vast amounts of data, including news articles, books, and online posts, to predict the next likely letters and words when prompted. Despite appearing to be sentient thinkers, this is not yet the case.
Recent research from UC Berkeley shows that AI chatbots can now analyze sentences like trained linguists. This study will be published in the journal IEEE Transactions on Artificial Intelligence and challenges the notion that humans are unique in their ability to think about language.
Gašper Beguš, a Berkeley associate professor of linguistics and lead author of the research, noted that “our ability to think deeply about words and sentence structure is a defining human cognitive feat.” However, this metalinguistic ability is also becoming part of AI chatbots’ domain.
“Our new findings suggest that the most advanced large language models are beginning to bridge that gap,” said Beguš. “Not only can they use language, they can reflect on how language is organized.”
Beguš and his team tested multiple versions of OpenAI’s ChatGPT and Meta’s Llama 3.1 with 120 complex sentences. They instructed these systems to analyze each sentence for specific linguistic qualities and diagram them using syntactic trees.
In one example sentence—“Eliza wanted her cast out”—researchers assessed whether AI could detect ambiguous structure. While some models failed, OpenAI’s o1 model detected the ambiguity and accurately diagrammed it.
This finding highlighted improvements in AI models. Beguš was particularly interested in whether these systems could identify recursion—a feature theorized by Noam Chomsky as unique to human language.
To test recursion, Beguš’s team asked AI models to identify if a sample sentence contained it and add another recursive clause. OpenAI’s o1 model successfully detected recursion in “Unidentified flying objects may have conflicting characteristics” and enhanced it further.
Researchers concluded that “o1 significantly outperformed all others.”
“This is very consequential,” said Beguš, suggesting it advances discussions on whether AI truly understands or merely mimics language. “It means in these models, we have one of the rare things that we thought was human-only.”
He emphasized that their approach provides a benchmark for evaluating AI’s understanding of language scientifically.
“Everyone knows what it’s like to talk about language,” he stated. “This paper creates a nice benchmark or criterion for how the model is doing.”
