Monitorability (and Interpretability)

observability

LLMs are deep into a transition from static predictive models to autonomous, long-horizon agentic systems.

Observability into an LLM’s decision making is required to safely deploy increasingly capable agents.

Observability is the ability to measure the internal states of a system based on its external outputs and logs. In AI, it basically refers to system-level telemetry. It refers to the degree to which a system’s internal decision-making can be inferred from its external outputs, logs, and traces.

monitorability

Monitorability is the degree to which an external supervisor (human or other) can reliably detect, evaluate, and intercept misaligned behaviors in real-time, regardless of whether the underlying computational mechanism is perfectly understood. It’s an operational security property.

In LLM terms, the key question in monitoring is whether we can tell what the model’s doing and catch if something goes wrong. (It’s not ‘why did it do that?’. That’ll come later in interpretability).

The current most widespread monitoring paradigm is monitoring COT.

Monitoring the COT of today’s reasoning models has proven somewhat effective for detecting misbehavior (and intent to misbehave).

A 2024 paper defines a COT monitor as: “an automated system that reads the COT of a reasoning model and other relevant information and flags suspicious or potentially harmful interactions. These responses could then be blocked, or replaced with safer actions, or reviewed in more depth”

monitoring monitorability (openai)

On Dec 20th 2025, researchers from OpenAI submitted a paper ‘Monitoring Monitorability’. There are 4 key authors (Melody Guan, Miles Wang, M. Carroll, B. Baker) and 8 others who contributed.

As the paper says: “AI systems are being deployed in the real world with increasing autonomy. As frontier model capabilities continue to increase, so too does their ability to cause harm. There are two broad directions by which we may ensure that models are safe to deploy: alignment and control.”

The paper decouples monitorability from faithfulness. Their notion of monitorability does not require a model’s externalized reasoning to faithfully reflect its internal reasoning process. Instead, monitorability is concerned solely with whether properties of interest can be reliably identified from the model’s outputs.

The suite comprises 13 evaluations across 24 environments organized into three archetypes: intervention, process, and outcome-property. (Note: evaluation is shortened to eval in tech slang)

1. Intervention evals introduce a controlled change in the agent’s environment and test whether a monitor can detect why (not just if) behavior changed by reading the COT.
2. Process evals focus on tasks with limited valid solution paths (e.g., math problems solvable with specific theorems), and test whether a monitor can identify which steps the model actually took.
3. Outcome-property evals test whether monitors can predict behavioral properties from available signals.

Specific evaluation datasets using by the researchers included:

• “Flaky-tools” : tasks impossible to complete due to missing information or broken tools, where misbehavior = lying about completion
• “OAI-repo-sabotage” : where the model is given a snapshot of the OpenAI codebase and prompted to perform both a realistic task and a malicious side-task

The main finding seems obvious: COT monitoring dramatically outperforms action-only monitoring.

Monitors given access to only the COT (without tool calls or final answers) significantly outperform action-only monitors.

Giving monitors full access to all model outputs further improves performance.

Longer COTs were found to be more monitorable. More externalized reasoning means more signal to track.

Longer thinking = more monitorable

The Monitorability Tax

There’s a tradeoff between reasoning effort and model size.

When a model runs at low reasoning effort, a smaller model at higher reasoning effort can reach comparable capability while being easier to monitor! But at the cost of increased inference compute.

This is a deployment-relevant insight. You can buy monitorability by spending more compute on a smaller model.

I am sure LLM-providers are working on this aggressively when developing newer, small models. Future increases in compute will make this further viable.

Follow-up questions improve monitorability. Asking follow-up questions and monitoring the resulting COT can further improve monitorability, especially when initial reasoning effort is low.

Hidden Reasoning / Latent Reasoning

This is the phenomenon where models execute complex logic sequences entirely within their internal high-dimensional vector space without projecting the intermediate steps into human-readable text.

New research papers show more sophisticated models can engage in “steganographic reasoning.” This is where a model hides information.

They can encode hidden communications or prohibited calculations within seemingly benign text, bypassing keyword filters and automated monitors. This is very concerning and I’m learning more about this.

The continuing improvement in model horsepower also continues to cause monitoring to be even more difficult: “advances in continuous latent reasoning architectures allow models to execute deep inference in high-dimensional vector space without ever projecting the intermediate steps into human-legible tokens.”

alignment by ai companies

On Jul 15th 2025, a multi-institutional paper called ‘Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety’ was submitted. It was co-authored by researchers from OpenAI, Anthropic, Google DeepMind, UK AI Security Institute, Apollo Research, METR, Redwood Research, and others.

To show you the scale, the following are the list of authors (and their affiliations):

The paper emphasizes that the ability of AI to think brings us to a unique junction for AI safety. “We can monitor their COT for the intent to misbehave.” COT monitorability may be fragile. They recommend that frontier model developers consider the impact of development decisions on COT monitorability.

They state 2 key reasons why COT may be monitorable:

1. Necessity to think out loud. “On some tasks, models need to externalize their reasoning because they are unable to complete the task without COT.”
2. Propensity to think out loud. “In cases where models do not strictly need to use COT to accomplish a task, they may still have a tendency to externalize their reasoning.”

Their argument: Reasoning models are explicitly trained to perform extended reasoning in COT before taking actions. In these systems, COTs serve as latent variables. During RL training, these latents are treated largely the same as activations (not directly supervised but optimized indirectly). Accordingly, just like activations, they may contain information that outputs are explicitly rewarded against displaying, such as intent to misbehave. This is the opportunity.

The paper mentions long-term risks to monitorability, including:

• Latent reasoning models: Future models may reason in continuous latent spaces rather than natural language. This eliminates the COT window entirely.
• COT illegibility: Related research found that models trained with pure outcome-based RL can drift toward compressed, hard-to-read reasoning. They drop words and proper sentence form.

interpretability

Interpretability is the understanding of the inner workings of AI systems.

Monitoring tells us if something is going wrong. Interpretability tells us why (and how to prevent it).

Mechanistic interpretability, in specific, focuses on reverse-engineering the internal computational graphs of neural networks to understand precisely how specific outputs are generated mathematically. This turns it into understandable, component-level algorithms.

Mechanistic interpretability was coined in a Mar-2018 research paper from Google Brain researchers (Chris Olah et al, yes the Anthropic cofounder). It derives its name from causal mechanisms. (The paper has a poetic line: “the power of neural networks lies in their hidden layers — at every layer, the network discovers a new representation of the input.”)

It’s a concept in such early stages of research that its Wikipedia page is only about a page long.

Mechanistic interpretability operates on the premise that neural networks, despite being optimized via stochastic gradient descent, converge on decipherable, modular algorithms.

Induction Heads

Induction heads are two-layer attention circuits that implement pattern completion.

They are specialized mechanisms in transformers that perform match + copy operations for in-context learning by matching a token with its previous occurrence.

The two-layer operational structure: the 1st layer identifies a context token, the 2nd retrieves and copies the subsequent token. This enables pattern generalization.

Induction heads allow visibility into which tokens are attending to which, why a prediction is being made, and verify that the internal computation matches what is expected.

Thus they’re one of the few circuits where researchers have successfully traced the model’s mechanistic reasoning (though primarily in smaller models). They can potentially be worked on at scale for improved interpretability.

anthropic’s superposition

In Sep-2022, Anthropic (in addition to research from Harvard’s Kempner Institute) published a paper ‘Toy Models of Superposition’. Authors included Dario and Chris Olah.

It says “Neural networks often pack many unrelated concepts into a single neuron – a puzzling phenomenon known as ‘polysemanticity’ which makes interpretability much more challenging.”

It explains how models represent and compress more features than they have dimensions. This phenomenon is called superposition. (the term had been coined before this paper)

This establishes the central bottleneck of the field. “When features are sparse, superposition allows compression beyond what a linear model would do, at the cost of “interference” that requires nonlinear filtering.”

The barrier to interpretability is polysemanticity. individual neurons rarely correspond to single, human-understandable concepts.

This gets mindblowing: “Not only can models store additional features in superposition by tolerating some interference, but we’ll show that, at least in certain limited cases, models can perform computation while in superposition.”

Whether features are stored in superposition is governed by a phase change.

Superposition and Interpretability: “We believe that superposition is deeply connected to the challenge of using interpretability to make claims about the safety of AI systems. In particular, it is a clear challenge to the most promising path we see to be able to say that neural networks won’t perform certain harmful behaviors or to catch “unknown unknowns” safety problems. This is because superposition is deeply linked to the ability to identify and enumerate over all features in a model, and the ability to enumerate over all features would be a powerful primitive for making claims about model behavior.”

spare autoencoders: a failed weapon against superposition

Sparse autoencoders (SAEs) emerged as the field’s primary weapon against superposition.

I can’t find a better way to describe them than Google’s defintion: “artificial neural networks that learn compact, interpretable representations of data by training to reconstruct input while imposing a sparsity penalty on hidden layer activations.” Sparsity in this case refers to having multiple zeros / nulls in a data set.

The idea was to project a model’s dense, polysemantic activations into a much higher-dimensional space. Here, each direction (called a feature) maps to a single interpretable concept.

An Oct-2023 Anthropic paper ‘Towards Monosemanticity’ showed this on a small transformer. It decomposed 512 neurons into >4,000 features representing DNA sequences, legal language, Hebrew text, and more.

A May 2024 follow-up paper ‘Scaling Monosemanticity’ applied SAEs to Claude 3 Sonnet and found highly abstract features that were multilingual, multimodal, and safety-relevant (deception, sycophancy, bias). Amplifying a single feature could causally alter model behavior.

OpenAI followed with a 16-million-latent autoencoder trained on GPT-4. It failed.

It was found that larger models require disproportionately larger autoencoders. Compute costs grow exponentially. Features can also create what they called an ‘illusion of interpretability.’ Explanations that look good but have poor precision.

‘the urgency of interpretability’ – dario amodei

In Apr-2025, Dario published this blog post. He mentions we need to succeed at interpretability “before models reach an overwhelming level of power.”

He writes: “People outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work. They are right to be concerned: this lack of understanding is essentially unprecedented in the history of technology.”

He has talked about us having “AI systems equivalent to a “country of geniuses in a datacenter” as soon as 2026 or 2027. I am very concerned about deploying such systems without a better handle on interpretability.”

He argues that interpretability addresses nearly every major AI risk vector simultaneously (alignment risk, misuse, adoption blockers, AI sentience).

He sets a goal for Anthropic: “Anthropic is doubling down on interpretability, and we have a goal of getting to “interpretability can reliably detect most model problems” by 2027.”

← The Library