Introduces a statistical alternative to the standard frequency-based BPE tokenization used in nearly all modern LLMs.
March 23, 2026
Original Paper
Significance-Gain Pair Encoding for LLMs: A Statistical Alternative to Frequency-Based Subword Merging
arXiv · 2603.19261
AI-generated illustration
The Takeaway
It replaces raw pair frequency with a significance-gain criterion based on a z-statistic. This improves predictive efficiency (BPC) and reduces perplexity by ~12-13%, suggesting that the industry-standard way we define tokens is sub-optimal for model reasoning.
From the abstract
Subword tokenization is a key design choice for modern language models, including large language models (LLMs), with byte- and character-level BPE serving as a widely used baseline. Standard BPE selects merges by raw pair frequency, which favors compression but can conflate true adjacency cohesion with pairs that are frequent due to high marginal counts. This paper introduces Significance-Gain BPE, a drop-in alternative merge criterion that measures cohesion via a z-statistic under an independen