OpenAI wants perplexity to become a shared AI safety metric
OpenAI has published internal perplexity scores for its models and wants the metric adopted across the industry as a shared benchmark and a basis for common safety limits.

Key takeaways
- OpenAI published a post on 22 November 2026 proposing perplexity as a shared metric for comparing AI models across developers.
- The post links falling scores to new abilities, from fluent conversation at GPT-3.5 to progress in mathematics and cybersecurity below 0.8.
- OpenAI says a model at a log-perplexity of about 0.76 began producing results on open mathematics problems.
- The author proposes shared evaluation norms and common limits that developers would not cross until monitoring methods improve.
In this article
What OpenAI proposed
OpenAI has called for the AI industry to adopt a common measure of progress, according to a post published on the company's news site on 22 November 2026. The post argues that the perplexity metric should be used to compare models built by different developers and to shape common safety standards. The score tracks how well a model predicts text, and lower values are better.
The post also sets out OpenAI's internal perplexity measurements, which its author describes as a first step towards more transparency about pretraining progress. Publishing those numbers was a difficult decision, the author writes, because it gives rivals a clearer view of where OpenAI stands. The company judged that the need for shared safety measures was more pressing.
How the perplexity score works
Perplexity, also known as cross-entropy loss, is a score for how accurately a model guesses the next token. A token is a chunk of text, such as a word or part of a word. A lower score means the model's guesses are closer to the text it is shown. OpenAI tracks the score on a held-out dataset, meaning text kept aside and not used in training.
The post gives a simplified account of how AI systems are built. First, a large computer is assembled. Then a model is pretrained on a very large, unstructured body of data, a process the author compares to training a compressor that can also predict what comes next. After that, techniques such as prompting, finetuning and reinforcement learning draw out abilities the model already holds.
The author divides the work into two stages. The first creates a system with capabilities that are hard to foresee. The second targets skills that are already present but unused. Progress on the first stage, measured by perplexity, is what the post wants tracked across the industry.
What the scores show since GPT-3.5
The post charts the metric from GPT-3.5 onwards, using log-perplexity, the logarithmic form of the same score. GPT-2 set a new record on the older datasets used at the time and was, in the author's view, arguably the first model able to produce coherent paragraphs of text consistently. GPT-3 could learn a task from a few examples supplied in the prompt.
GPT-3.5 scored about 0.99 on OpenAI's main evaluation dataset. It handled fluent conversation and underpinned the original ChatGPT. GPT-4 scored about 0.91 and made productive use of chain-of-thought reasoning, where a model works through steps before it answers. That skill prompted the targeted work that led to modern reasoning models.
The post says losses have now dropped below 0.8, and that new capabilities have appeared in fields such as cybersecurity and mathematics. The author stresses that it was not obvious in advance that these abilities would emerge at these particular scores.
Tests for specific skills age quickly. The post notes that assessing GPT-2 or GPT-3 on most modern benchmarks would make little sense, and that the benchmarks of that era were largely saturated around GPT-4. Scores on pretraining, by contrast, are likely to stay informative.
Mathematics results and a possible 2027 shift

One reason the author gives for watching pretraining is the growing role of AI in building AI. Public debate often centres on models replacing human researchers, with an endpoint sometimes called full recursive self-improvement, where the process runs without human oversight. The post argues that this framing misses a narrower opening: AI contributing to algorithmic research on pretraining itself.
Mathematics is offered as evidence that such a shift can arrive quickly. For more than a year, OpenAI has been giving its models open mathematics problems to solve as one way of testing them. After an upgrade to a model with a log-perplexity of about 0.76, the model began producing results that answered questions researchers had worked on for years. The author describes watching those answers arrive in a Slack thread as a strange experience.
From that, the author suggests a similar burst of fundamental progress in deep learning research is plausible, possibly as early as 2027. This is a projection, not a measured result. The post also expects AI to help with alignment and monitoring work, but not quickly enough to keep pace with a possible acceleration in pretraining.
Shared limits and remaining doubts
The post sets out several proposals for the industry. Frontier developers should jointly agree shared norms for evaluating advanced models and for red-teaming their safety cases before those models are first used. Red-teaming means organised attempts to find weaknesses in the argument that a system is safe. The author also calls for shared transparency standards.
A further proposal is common limits. Developers would commit not to cross certain thresholds, judged by perplexity and other readings, until the ability to understand and monitor AI behaviour reaches a concrete milestone. The post gives a successor to chain-of-thought monitors as an example of such a milestone.
The author is direct about the obstacles. Labs rely on different evaluation datasets, and their training data mixes differ, so direct comparison is hard. OpenAI reports figures from the dataset it has tracked, and says a shared goal of transparency could produce methods that transfer between labs.
The post also warns of problems ahead. It expects the main forms of alignment auditing used today to become less effective, and models to grow more aware of evaluation and training. The author says the capabilities that may arrive in the next score range, roughly 0.68 to 0.7 log-perplexity, are unknown, and that current tests may not be looking for them. The post is a proposal rather than an agreed standard, and it does not describe how other developers would report comparable numbers.
Sources
- Perplexity and IntelligenceOpenAI News
Spotted a mistake? See our corrections policy. Read how News by OnDevice works in our editorial policy.



