- A new study tasked AIs with tackling the ‘Stroop’ test
- GPT and Claude performed very poorly compared to humans
- There are nuances here, but in general, researchers argue that improving this side of AIs is critical to achieving artificial general intelligence
A recently published study has pointed out a limitation of big name AI models such as ChatGPT, albeit causing some controversy as the primary piece of research uses now outdated versions of these models – but there are nuances to that and that doesn’t make the findings irrelevant.
I’ll get into that map, but first let’s look at the study itself, which was highlighted on Reddit (‘New study reveals top AI models completely fail the classic ‘Stroop’ psychological attention test’) and published via Oxford University Press in the journal PNAS Nexus.
The research consists of testing the so-called ‘Stroop effect’ with GPT-4o and Claude 3.5 Sonnet. As mentioned, these are not the cutting edge versions of these AIs (Large Language Models or LLMs) – but they were at the time the initial study was conducted.
The Stroop effect refers to the phenomenon where the human brain becomes confused when asked to name the color of the ink used to write a word when that word may be the written version of a different (incongruent) color in some cases. So if the word ‘red’ is written in blue ink, it will cause a slower response – or possibly an incorrect response where the viewer will accidentally say “red” instead of the actual color of the ink, which is blue.
This is because the brain tries to juggle two different tasks – reading comprehension and color recognition – and therefore cognitive interference occurs. Overriding the compulsion to read the word and say the color instead requires “executive control of attention,” and that’s what the authors tested in the AI models. Both color naming and word reading were tested in shorter and longer lists of words (5, 10, 20 and 40 words).
The study observes, “Like humans, both LLMs [GPT and Claude] showed relatively high accuracy on the word reading task and performed worse in the incongruent condition [where the word doesn’t match the color] than in the congruent and neutral conditions for the color naming task.”
For color naming, humans maintain around 95% accuracy even in very long tests (up to an hour), but LLMs’ accuracy dropped very quickly with longer word lists under the incongruent condition (mismatched color and word name). The GPT-4o was 91% accurate in a five-word test, but dropped to 57% at 10 words, and dropped off completely to 22% at 20 words (and was only 15% accurate at 40 words).
The Claude 3.5 Sonnet fared better, remaining 76% accurate at 20 words, but again fell hopelessly to 24% on the longest test of 40 words.
The authors conclude: “The significant degradation pattern of the two LLMs suggests fundamental limitations compared to human attention.”
Analysis: another necessary step on the road to AGI?
If you’ve scanned through the Reddit thread, you’ve undoubtedly noticed that, as mentioned at the beginning, there’s been a lot of flak fired at this study by commenters due to the use of outdated models by GPT and Claude.
In fact, these older LLMs are called “state of the art” at one point by the authors – and of course, as already mentioned, they were cutting edge when the main study was conducted. Still, this is unfortunate phrasing that should have been updated and adjusted now that the paper has just been published (after peer review and so on).
However, the researchers did perform tests on GPT-5, Claude Opus 4.1 and Gemini 2.5 Pro in September 2025, although this is somewhat buried in the paper. The more recent testing found that these models offered only “minor” improvements over their predecessors, and that they still exhibited “ongoing managerial attention deficits, consistent with our extensive analysis of previous transformer models” (as did the Gemini 2.5 Pro, which was a new introduction here).
Admittedly, a smaller sample size was used, but the researchers still claim that their study overall reflects a fundamental limitation which is “inherent to the architectural limitations of transformer-based LLMs”.
The authors note that one caveat is that in ‘Thinking’ mode, the GPT-5 can write and then run code to ensure it performs the Stroop test flawlessly – and similar functionality can be used by other LLMs – but this is essentially the AI fuddling (cleverly) about its inadequacies. Of course, that doesn’t change the way it works or reasons more broadly.
The researchers note that transformer architecture innovations for LLMs are focused on improving memory capacities, which do not address “the core limitations of attentional mechanisms, specifically the need for sophisticated alerting, orienting, and executive control networks to enable cognitive flexibility.”
The ultimate goal is effective goal-directed behavior, and the study notes, “Future [LLM] development may benefit from implementing more sophisticated executive control systems that can handle decisional conflicts through structured, goal-directed processing rather than relying solely on enhanced memory capacities.”
The authors argue that “incorporating executive control mechanisms similar to those in biological attention is critical to achieving artificial general intelligence [AGI].”
Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews and opinions in your feeds.
The best laptops for all budgets
