LLMs do not actually understand how to count, they simply follow a limited sequence of internal states that eventually breaks down.
When an AI counts objects, it is not applying a logical rule like a human would. Instead, it moves through a finite state machine that behaves like counting on fingers until it runs out of states. Once it passes a certain number, its ability to count collapses into random guessing. This proves that the reasoning we see in LLMs is often just a very clever illusion of pattern matching. We cannot rely on these models for any task that requires the strict application of an infinite logical rule.
Counting as a minimal probe of language model reliability
arXiv · 2605.02028
Large language models perform strongly on benchmarks in mathematical reasoning, coding and document analysis, suggesting a broad ability to follow instructions. However, it remains unclear whether such success reflects general logical competence, repeated application of learned procedures, or pattern matching that mimics rule execution. We investigate this question by introducing Stable Counting Capacity, an assay in which models count repeated symbols until failure. The assay removes knowledge