Cognitive reflection test
A short test of overriding a wrong gut answer.
What it means
The Cognitive Reflection Test (CRT) is a brief set of problems, each engineered so that an intuitive but incorrect answer springs immediately to mind and must be suppressed in order to reach the correct one. It is designed to measure the disposition to engage reflective, deliberate (System 2) thinking rather than to simply accept the first intuitive (System 1) response, and as such it captures both reasoning ability and the tendency to check one's intuitions. Scores predict performance on a range of judgment tasks and susceptibility to many biases, as well as outcomes such as susceptibility to misleading information. There are important caveats: the original items have become widely known, which can inflate scores through familiarity, and the test partly confounds numeracy and reflective disposition, so newer non-numeric versions have been developed. It matters as a compact research instrument for studying who overrides intuition and when, and as an illustration of the broader point that good judgment often hinges less on raw intelligence than on the willingness to pause and question a compelling gut answer.
How the items work
The problems share a design principle: each carries a lure, an answer that is fluent, confident, and wrong. In the bat-and-ball item, "10 cents" arrives before you have reasoned, and reaching 5 cents requires noticing the pull and checking it. Scores run from 0 to 3, but what is diagnostic is not only whether you answer correctly but whether you produce the specific intuitive error, which marks someone who accepted the first response without scrutiny. Dual-response studies, in which people give a fast answer and then a considered one, show that many who ultimately answer correctly still feel the lure first and revise it. The test therefore separates two things ordinary accuracy blurs: whether an intuition is generated, and whether it is overridden.
What it confounds: numeracy and intelligence
Because the original items are all numerical, the CRT correlates strongly with numeracy and with general intelligence, and a long debate asks whether it measures reflective disposition over and above raw math ability. A 2022 meta-analysis by Otero and colleagues found large associations with numerical ability, cognitive intelligence, and working memory, treating cognitive reflection as closely bound to the broader ability hierarchy rather than a wholly separate faculty. A 2024 adversarial collaboration led by Meyer, with Frederick and Kahneman among the authors, pushed back: an eight-item CRT out-predicted a mathematics test of matched difficulty that lacked intuitive lures, and the surplus held for non-numeric items. They read this as evidence that reflection itself, not arithmetic alone, carries part of the signal the test picks up.
The exposure problem
The original items have escaped the lab. They circulate as the "world's shortest IQ test," appear in classrooms and puzzle books, and recur across the online panels researchers recruit from, so a rising share of participants already knows the answers. Familiarity plainly inflates mean scores. What is less obvious is that it may not wreck the test: across six studies, Bialek and Pennycook found repeated exposure did not undermine the CRT's ability to predict the outcomes it usually predicts, suggesting rank-order differences survive even as absolute scores drift upward. Even so, careful researchers now rotate in fresh items, add problems, or switch to the non-numeric CRT-2, treating any single administration on a well-trafficked sample as partly contaminated.
What scores predict
Scores forecast performance across the heuristics-and-biases battery. In Toplak, West, and Stanovich's data the CRT predicted a composite of rational-thinking tasks better than intelligence, executive function, or self-reported thinking dispositions, which is striking for a three-item scale. Higher scorers are somewhat less prone to framing and conjunction errors and, in later work, less receptive to pseudo-profound "bullshit" statements. The caveats are real: the correlations are moderate, not deterministic, and a three-item test is a coarse, low-reliability instrument that pins floor and ceiling close together. What it captures best is a stable disposition to interrogate a compelling answer, the same habit that, more than sheer processing speed, tends to separate careful reasoners from quick ones.
Examples
'A bat and ball cost $1.10; the bat costs $1 more than the ball. How much is the ball?' — intuition says 10¢; it's 5¢.
Researchers use scores as a filter: people who pause on these puzzles also prove harder to fool with a misleading headline or a confident sales pitch.
The items blunt as they spread. A student who has already met the puzzles online scores well through recognition, not through any habit of checking her first answer.
An investor who scores low tends to grab the fund with the eye-catching headline return, while a reflective scorer pauses to ask what fees and base rates that number quietly hides.
Told a screening result means "90% survival," a high-scoring patient stops to reframe it as "10% mortality" before deciding, refusing to let the more soothing phrasing settle the choice.
First described in Shane Frederick (2005).
Key references
- Meyer, A., Attali, Y., Bar-Hillel, M., Frederick, S., & Kahneman, D. (2024). Cognitive reflection is a distinct and measurable trait. Proceedings of the National Academy of Sciences, 121(49), e2409191121. doi.org/10.1073/pnas.2409191121
- Otero, I., Salgado, J. F., & Moscoso, S. (2022). Cognitive reflection, cognitive intelligence, and cognitive abilities: A meta-analysis. Intelligence, 90, 101614. doi.org/10.1016/j.intell.2021.101614
- Bialek, M., & Pennycook, G. (2018). The Cognitive Reflection Test is robust to multiple exposures. Behavior Research Methods, 50(5), 1953-1959. doi.org/10.3758/s13428-017-0963-x
- Thomson, K. S., & Oppenheimer, D. M. (2016). Investigating an alternate form of the cognitive reflection test. Judgment and Decision Making, 11(1), 99-113. doi.org/10.1017/S1930297500007622
- Toplak, M. E., West, R. F., & Stanovich, K. E. (2011). The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks. Memory & Cognition, 39(7), 1275-1289. doi.org/10.3758/s13421-011-0104-1
- Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19(4), 25-42. doi.org/10.1257/089533005775196732