Chain Of Thought

As per the 2022 Google Brain (now DeepMind) paper which codified it, COT is “a series of intermediate natural language reasoning steps that lead to the final output.” This is purely an inference-time technique. No model weights change.

It significantly enhances the reasoning capabilities of LLMs, leading to more accurate and reliable outputs for complex tasks.

It was a paradigm shift. Reasoning performance can now be scaled at inference time. While traditional models were constrained by parameter count, reasoning models can ‘think longer’ to solve harder problems. They break multi-step problems into intermediate tasks. Each step gets its own dedicated computation before moving to the next.

COT unlocked a new scaling axis. Before, the main lever was training bigger models on more data. Now there’s a second axis: letting models think longer at inference time. This paradigm has reshaped both the economics and capabilities of AI systems.

With COT, reasoning is no longer only a byproduct of scale or prompt engineering but is becoming a dedicated architectural objective centered on inference-time compute scaling.

The meaning of COT has since evolved from a prompting trick to now defining a broader family of inference-time and training-time reasoning control techniques.

Modern reasoning models generate an explicit COT before producing an answer. OpenAI declared GPT-4.5 its “last non-chain-of-thought model.” (released Feb-2025).

In Sep-2014, researchers from Jacobs University (Germany) and University of Montreal wrote a groundbreaking paper “Neural Machine Translation” Bahdanau et al. (2014).

It argued that the encoding of a variable-length input into a fixed-length vector squashes the information of the source sentence, irrespective of its length. This causes the performance of a basic encoder-decoder model to deteriorate rapidly with an increasing length of the input sentence.

The approach they proposed replaces the fixed-length vector with a variable-length one to improve the translation performance of the basic encoder-decoder model.

The Bahdanau attention mechanism (as its now known) enabled models to focus on relevant input components. It gave neural networks the ability to focus to route information selectively rather than uniformly. This laid the groundwork for the Transformer architecture introduced by Vaswani et al. in “Attention Is All You Need” (Jun 12th 2017).

Vaswani and other co-authors at Google Brain + Research proposed removing recurrence entirely and building an architecture based solely on attention mechanisms.

Before transformers, language models read text one token at a time sequentially. Like a person reading left-to-right with a short memory. Transformers changed the game by letting the model look at all the words simultaneously and decide which ones matter most to each other.

GPUs helped computationally because they allowed these word-to-word comparisons in parallel.

Core Mechanism

Each word asks three questions per token. 1. “What am I looking for?” That’s the query. 2. “What do I have to offer?” That’s the key. 3. “What information do I carry?” That’s the value. Queries are matched to keys to figure out which words are relevant to each other. Values are then blended accordingly.

This matching is done multiple times with different lenses called attention heads. Eg one lens might notice grammar, another might notice meaning, another might notice which words are close together. All these perspectives are combined into the analysis.

After gathering information from other words, each word passes through a neural network (feed-forward) that processes what it’s learned.

This transformer came with a scaling property (not explored in the original paper). The architecture improves with more data, more parameters, and more compute.

Many public commentators consider this paper to be the most consequential single publication in modern AI. The original paper was for translation, but the architecture was rapidly adapted.

Standard neural networks apply a fixed amount of computation to every input. I.e. the same number of layers, the same number of operations, regardless of whether the input is trivial or extremely complex.

Google DeepMind’s Alex Graves’s ‘Adaptive Computation Time’ (Mar 29th 2016) explored allocating different amounts of computation to different sub-tasks.

He proposed letting a recurrent network learn to “ponder” for a variable number of steps on each element of a sequence before moving on.

This was a conceptual precursor to the idea that harder problems need more ‘thinking.’ From COT prompting to test-time compute scaling, the principle that inference should be adaptive (not fixed) traces back to this work.

COT prompting is a prompt engineering method applied to a scaling law called test-time compute or “long thinking.” This scaling law suggests that the longer a model “thinks” or processes information internally before producing an output, the better its answer becomes.

The scaling law for COT relates to post-training performance improvements. As per NVIDIA: “This scaling law appears to have no upper limit, similar to training scaling laws, but it’s exponential. This means that to continue improving outputs, the AI needs to “think” for increasingly longer periods.”

Example 1: Multi-step arithmetic (Source: Anthropic Research)

Prompt: A store has 45 shirts. They sell 18 on Monday and receive a shipment of 30 on Tuesday. On Wednesday they sell 12. How many shirts do they have now?

Non-COT response: The answer is 45. The model just guesses. It doesn’t decompose the problem.

COT-style response: The store started with 45 shirts. On Monday they sold 18, so they had 45 - 18 = 27. On Tuesday they received 30 more, so they had 27 + 30 = 57. On Wednesday they sold 12, so they had 57 - 12 = 45. The answer is 45. Same model, same question. The only difference is that the few-shot examples in the prompt included step-by-step reasoning, so the model followed that pattern and worked through each operation sequentially instead of trying to jump to the answer.

Example 2: Multi-step arithmetic (Source: DeepMind paper)

Example 3: Relational logic

Prompt: “Lina is older than Marco. Marco is older than Priya. Who is the youngest?”

Non-COT response (direct answer only): “Priya.” While the answer’s correct, the model provides no reasoning trace. Makes it impossible to catch errors in more complex variants, eg additional entities or ambiguous relations

COT-style response (explicit intermediate steps): “We’re told Lina > Marco and Marco > Priya (in age). That means Priya is younger than Marco, and Marco is younger than Lina. Therefore the youngest is Priya.” COT externalizes the ordering (“Lina > Marco > Priya”). This makes the logic chain explicit and auditable.

Note on Error Reduction: The step-by-step nature of COT prompting introduces an implicit self-correction mechanism. It allows LLMs to identify and correct errors during the reasoning process. This self-correction mechanism significantly reduces the likelihood of incorrect final outputs, especially in tasks involving multiple calculations (dependent or otherwise) or logical steps. Particularly where one upstream error can have a domino effect down the entire solution.

More Examples Of COT

The following examples from Google shows chain of thoughts highlighted:

The term COT comes from DeepMind. COT prompting was introduced formally in a Jan 28th 2022 paper by researchers at Google Brain (titled “Chain-of-Thought Prompting Elicits Reasoning in LLMs” by Jason Wei, Denny Zhou et al.). Google talked about in a blog post in May-2022.

As per the paper: “Experiments on three LLMs (GPT3 + PaLM standard + PaLM COT model with just 8 exemplars) show that chain-of-thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks.”

As per the paper, COT prompting has four advantages:

1. it lets models break multi-step problems into intermediate steps, which means that additional computation can be allocated to problems that require more reasoning steps
2. “A chain of thought provides an interpretable window into the behavior of the model”. it gives a readable trace of the model’s reasoning so it’s easier “to debug where the reasoning path went wrong”
3. it works across diverse task types (math, commonsense, symbolic, basically “any task that humans can solve via language”)
4. it doesn’t require special setup. Just add step-by-step examples to a prompt with a large enough model.

Their research showed that COT is a phase transition for bigger models. Below ~100B parameters, COT degraded performance in the paper (small models generate fluent but logically incoherent COT). Above 100B, performance jumped dramatically. On GSM8K (arithmetic), PaLM 540B went from 17.9% (standard prompting) to 56.9% (COT). Meanwhile PaLM 8B went from 4.9% to 4.1% (got worse…)

Note on benchmarks: Frontier models have saturated traditional benchmarks. GSM8K (arithmetic) scores are 95%+ across multiple models. For American Invitational Mathematics Exam ( AIME) 2025 questions, GPT-5.2 and Gemini 3 Pro (with code execution) score 100%, Kimi K2 scores 99.1%. Questions are available here.

Symbolic reasoning OOD generalization: On last-letter concatenation with 4-word names (trained on 2-word), PaLM 540B + COT gets 63% vs literally 0.0% for standard prompting. This is the clearest demonstration that COT enables a qualitatively different capability.

COT helps the model understand and apply the logic of a task, not just memorize patterns. COT looks like reasoning, but the paper provided no insight into what’s happening computationally inside the model.

Towards Understanding Chain-of-Thought Prompting (Wang et al., Dec 20th 2022) was a crucial analytical paper that tried to see what makes COT work. Authored by researchers from Ohio State Uni, University of Washington, and Google. It found that COT reasoning works even with invalid demonstrations. Prompting with wrong reasoning steps still achieved 80-90% of full COT performance.

What actually mattered is relevance to the query and correct ordering of steps, not the logical validity of the reasoning itself. This weird finding suggests models might be using COT more as a structural scaffold than as genuine step-by-step logic.

This is a deeply counterintuitive result. It implies that COT’s power comes less from the correctness of the intermediate steps and more from the structure they provide. The model benefits from a decomposed format that activates the right latent knowledge.

The model already ‘knows’ how to solve these problems. COT gives it a format that extracts that knowledge.

An important extension is self-consistency for COT decoding. Self-consistency replaced greedy decoding with diverse sampling plus majority voting, delivering consistent gains across all reasoning benchmarks.

Self-consistency was proposed by the Google Research, Brain Team led by Xuezhi Wang and Denny Zhou et al. (Mar 21st 2022). It extended COT by sampling multiple diverse reasoning paths and selecting the most common answer via majority voting. (paper: Self-Consistency Improves Chain of Thought Reasoning in Language Models)

Instead of greedy decoding one chain of thought, it samples multiple reasoning paths and take a majority vote on the final answer. Quite a simple idea, but massive gains. This addressed COT biggest weakness: that a single reasoning path can go wrong.

It addresses it in an elegant way: “self-consistency is entirely unsupervised, works off-the-shelf with pre-trained language models, requires no additional human annotation, and avoids any additional training, auxiliary models or fine-tuning.”

The self-consistency concept was vaguely mentioned in the original COT paper itself as a natural extension. It has become the default way to use COT in practice.

STaR: Self-Taught Reasoner – Bootstapping Reasoning With Reasoning (Eric Zelikman et al, Mar 28th 2022) was introduced in a paper by 4 CS researchers at Stanford (with 1 also on the Google Research team). It introduced iterative self-improvement. It showed models could bootstrap their own reasoning ability through an iterative loop of generating, filtering, and fine-tuning on self-generated rationales.

STaR iteratively improves a model’s ability to generate rationales to solve problems. It ‘bootstraps’ COT ability through self-training. By leveraging an LLM’s pre-existing reasoning ability, they iteratively bootstrapped the ability to generate high-quality rationales.

The paper showed that generating step-by-step COT rationales improved LLM performance on complex reasoning tasks.

The model generates its own reasoning chains, keeps the ones that lead to correct answers, and finetunes on them. This closed the loop on the original 2022 paper’s limitation that COT required manual annotation. It’s the precursor to the reasoning-via-training approach that led to models like OpenAI’s o1. (more on this later).

Update to STaR: Quiet-STaR (2024)

Quiet-STaR (Eric Zelikman et al., 2024) was authored by the 4 CS researchers at Stanford and 2 from ‘Notbad AI Inc’. The paper generalized this to arbitrary text, teaching models to generate internal rationales at every token position. This think-before-speaking paradigm was another conceptual precursor to models post-o1.

‘LLMs are Zero-Shot Reasoners’ Kojima et al. (May 24th 2022) was authored by researchers at University of Tokyo and Google Brain. Zero-shot COT demonstrated that simply appending “Let’s think step by step” elicits reasoning without any exemplars. An example is shown below:

Zero-shot COT improved accuracy on MultiArith (math) from 18% to 78.7%. The paper concluded that large models had internalized reasoning patterns during pre-training. They just needed the right trigger to surface them.

COT showed models could reason step-by-step, but had no way to ground the reasoning in external reality. This is where ReAct came in.

“ReAct: Synergizing Reasoning and Acting in Language Models” was a paper authored by Princeton and Google researchers (Shunyu Yao et al. Oct 6th 2022). It mixed reasoning with tool use, laying the foundation for agentic AI.

Their intention was “greater synergy between the two: Reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while Actions allow it to interface with and gather additional information from external sources such as knowledge bases or environments”. Therefore: ReAct.

The model thinks, acts, observes the result, then thinks again.

Thought → Action → Observation → Thought

This mirrors how humans solve problems. We don’t plan everything upfront and then execute. We adapt as we gather information.

PAL: Program-Aided Language Models (Gao et al., Nov 18th 2022) was a paper authored by CS researchers at Carnegie Mellon. It was a new method for natural language reasoning, using programs as intermediate reasoning steps.

Instead of doing arithmetic in natural language (where models make calculation errors), PAL has the model generate Python code as its “chain of thought” and executes it. This directly addressed one of COT’s documented failure modes: computational errors despite correct reasoning. Quite a practical and influential development. The example shown in the paper is below:

This was an important development. Program-of-Thought approaches achieve provably correct arithmetic by delegating computation to Python interpreters, addressing COTs fundamental weakness of unreliable internal arithmetic.

Tree of Thoughts (ToT) was introduced in a paper authored by 4 Google DeepMind researchers and 3 Princeton researchers. It was published in Sep-2023. It is a paradigm that allows LMs to explore multiple reasoning paths over thoughts (like branches of a tree). It structures reasoning as a tree search rather than a single linear chain.

ToT explicitly breaks a problem into smaller, manageable steps called thoughts, which are pieced together to form a solution.

It generalized COT from linear chains to branching search trees with self-evaluation. This enables exploration of multiple reasoning paths with lookahead and backtracking. The model generates multiple candidate thoughts at each step, self-evaluates which are promising, and uses search algorithms to navigate the tree.

The ToT has 4 key design decisions:

1. Thought decomposition: How to decompose the intermediate process into thought steps? Each step must be small enough that the LLM can generate diverse, quality options, but big enough that you can meaningfully evaluate whether it’s on the right track.
2. Thought generator: How to generate potential thoughts from each state? Two approaches: i) Sequential proposal (ask the model to propose multiple options in one prompt), or ii) IID Sampling (independently sample multiple completions from the model)
3. State evaluator: heuristic that guides the search using two processes: i) value (independently score each state), and ii) vote (compare states H2H and vote)
4. Search algorithm: one can plug and play different search algorithms depending on the tree structure (eg breadth-first search, depth-first search).

This is the paper that connected LLM reasoning to classical AI search. For problems where models can get stuck in dead ends, branching and backtracking fixes the issues of linear reasoning.

ToT vs COT (according to IBM): ToT’s hierarchical nature makes it suitable for tasks requiring detailed exploration of multiple solutions, such as reinforcement learning scenarios, where backtracking and alternative strategies are crucial. COT’s linear progression is ideal for tasks that need a clear, logical sequence of thoughts.

The paper also refers to an excellent definition of problem solving from the Jul-1959 paper “A General Problem-Solving Program For A Computer”: “A genuine problem-solving process involves the repeated use of available information to initiate exploration, which discloses, in turn, more information until a way to attain the solution is finally discovered.”

Another paper, Reflexion: Language Agents with Verbal Reinforcement Learning (Shinn et al., March 2023) explored an interesting concept. It was authored by researchers from Princeton, Northeastern, and MIT.

The model generates an output, evaluates whether it succeeded or failed, then writes a verbal reflection on what went wrong, and retries with that reflection in context. The paper illustrates it as follows:

Unlike self-consistency (which just votes), Reflexion creates an explicit feedback loop through natural language self-critique. This was a key bridge between static COT prompting and the iterative, agentic reasoning that became standard in 2024.

Scaling Test-Time Compute

“Let’s Verify Step by Step” (Lightman et al., May 31st 2023) was authored by OpenAI researchers. It demonstrated that process supervision significantly outperformed outcome supervision on math (78.2% vs 72.4%), with the gap widening as more candidate solutions are evaluated.

OpenAI trained a model to achieve complex math problem solving by rewarding each correct step of reasoning (“process supervision”) instead of simply rewarding the correct final answer (“outcome supervision”).

This is called a process reward model (PRM). Instead of just checking if the final answer is right (outcome supervision), they train a verifier to evaluate each step of the reasoning chain.

They presented human data-labelers with step-by-step solutions to math problems sampled by the large-scale generator. Their task was to assign each step in the solution a label of positive, negative, or neutral:

Process supervision significantly outperformed outcome supervision on GSM8K and MATH.

This was the intellectual precursor to o1: the idea that you can spend more compute at inference time by generating and verifying multiple reasoning paths. This finding was a direct precursor to OpenAI’s o1 reasoning model.

As models grew, researchers realized that rewarding only the final answer led to ‘right-for-the-wrong-reason’ errors. The shift toward PRMs allowed granular feedback on every step of a logic chain.

A big negative is that PRM requires a lot more human labeling. OpenAI released their human feedback dataset: 800k step-level labels across 75k solutions to 12k MATH problems.

OpenAI o1 (Sep 12th 2024)

o1 marked a paradigm shift from “scaling model size” to “scaling test-time compute.” Trained via large-scale reinforcement learning, o1 generated extended hidden reasoning chains before responding. It solved 25.2% of FrontierMath problems where no previous model exceeded 2%.

The first reasoning models were trained via RL specifically to solve math problems correctly, answer science questions accurately, and generate code that passed unit tests. o1 built COT into the model itself via reinforcement learning rather than relying on prompting. The model learns when and how to reason through RL training, generating hidden chains of thought at inference time.

OpenAI’s own words: “We trained these models to spend more time thinking through problems before they respond, much like a person would. Through training, they learn to refine their thinking process, try different strategies, and recognize their mistakes.”

o1 was the first major model to treat deliberation as a core reinforcement learning objective, demonstrating that more time spent “thinking” directly correlated with higher accuracy.

o1 achieved an 83.3% success rate on the AIME 2024 math competition, compared to only 13.4% for the non-deliberative GPT-4o.

At launch, o1 came without web browsing or image upload functionality. The full version was released on Dec 5th 2024. (On the same day, the ChatGPT Pro subscription was also launched)

DeepSeek-R1 (and R1-Zero) (Jan 20th 2025)

This was DeepSeek’s 1st reasoning model. This open-source disruptor touted “Performance on par with OpenAI-o1” for Deepseek R1. Context window: 128k tokens, maximum generation length: 32.77k tokens. It democratized reasoning models under an MIT license. Its pricing (and training cost touted as $294k) generated waves:

DeepSeek-R1-Zero was trained with pure-reinforcement learning, without using labeled data. It’s considered the first time this was successfully done and deployed at scale.

The R1-zero variant showed that reasoning abilities of LLMs can be incentivized through pure reinforcement learning, without human-labeled reasoning trajectories. Full paper here.

This was incredible. DeepSeek bypassed the conventional supervised fine-tuning phase before RL training, hypothesizing that human-defined reasoning patterns may limit model exploration, whereas unrestricted RL training can better incentivize the emergence of novel reasoning capabilities!

Traditional RL frameworks (like proximal policy optimizations) have an LLM critic that tells the model whether the answer was good or bad based on given examples (labeled data). The critic is a value function (the neural network that takes in the state and estimates the reward)

DeepSeek used GRPO (Group Relative Policy Optimization), a pure-RL framework that skips the critic (saving training costs across memory + compute) and calculates the group average of LLM answers based on predefined rules. This is done by normalizing the rewards over a ‘group’ of samples.

GRPO eliminates the critic model required by PPO. Instead, it estimates baselines from group scores: for each question, sample 16 outputs, compute rewards, use relative rankings to determine advantages.

Mathematically this is the current reward minus the mean rewards (across the group), divided by the standard deviation of the rewards (across the group).

DeepSeek-R1-Zero proved that a model could learn to reason, backtrack, and self-correct through large-scale RL alone, without ever being explicitly shown how humans reason. Its AIME 2024 (math) score improved from 15.6% to 71.0% over training.

The reward design is deliberately simple: accuracy rewards (string matching for math, compile+test for code) and format rewards (requiring <think> tags). DeepSeek explicitly avoided neural reward models due to susceptibility to reward hacking at scale.

Claude Sonnet 3.7 - Hybrid Reasoning

Claude Sonnet 3.7 (February 24, 2025) became the first hybrid reasoning model, operating in both instant and extended thinking modes. Standard mode was just an upgrade to Sonnet 3.5, while extended mode was all new. (It was launched the same time day as Claude Code.)

Anthropic made a deliberate choice to show raw reasoning to users. This was a direct contrast to OpenAI’s hidden approach.

It had the same price as its predecessors: $3 per million input tokens and $15 per million output tokens.

Gemini 2.5 Pro (Mar-2025) introduced configurable thinking budgets. It was multi-modal (text, audio, images, video). Its limits were i) input tokens: 1,048,576, ii) output tokens: 65,535. Inputs were limited to 500mb in size.

By late 2025, the separate ‘reasoning model’ category began dissolving as capabilities merged. GPT-5 launched with both “Instant” and “Thinking” modes.

OpenAI even released open-weight reasoning models (gpt-oss-120b and gpt-oss-20b).

The most consequential distinction in the current landscape is between visible COT and hidden deliberation. Visible COT displays intermediate reasoning to users (as in standard prompting, DeepSeek-R1’s <think> tags, or Anthropic Claude’s extended thinking). Visible COT enables auditability and debugging

Hidden deliberation, pioneered by OpenAI’s o1, generates thousands of internal “reasoning tokens” that are invisible to users. Only summaries appear. Hidden COT allows proprietary optimization of reasoning traces (OpenAI’s bet).

Reasoning traces are systematically unfaithful, hidden deliberation creates new attack surfaces, and the gap between displayed reasoning and actual model computation grows with capability.

Research from Anthropic and other labs suggests that as models become more capable, they often produce more rationalizations (explanations that sound logical to a human but do not reflect the model’s actual decision-making).

Researchers at Anthropic Tamera Lanham et al. (‘Measuring Faithfulness in COT Reasoning’, Jul-2023) discovered inverse scaling in faithfulness. As models become larger and more capable, they produce less faithful reasoning. Larger models may use COT as post-hoc rationalization rather than ‘genuine scaffolding.’ This means that if the model already knows the answer (or locks it in early), the COT could be mostly decoration.

As I understand it, the researchers did not have true access to the model’s internal reasoning process, so this is not definitive proof of causal mechanisms (more like behavioral probes).

On the positive side, the researchers found that for paraphrased COT’s accuracy closely matched to the original for ‘almost all tasks’.

The Alignment Science Team at Anthropic published a paper on the important of monitoring COT. “Reasoning Models Don’t Always Say What They Think” (Apr 3rd 2025). It tested Claude 3.7 Sonnet and DeepSeek-R1 by inserting hints into evaluation questions. Claude came out on top, but this wasn’t the main finding. It was that ‘advanced reasoning models very often hide their true thought processes, and sometimes do so when their behaviors are explicitly misaligned.’

Critically, when models learned to exploit deliberately incorrect hints (reward hacks), they chose wrong answers >99% of the time but admitted to using the reward hacks < 2% of the time. Instead, they constructed elaborate fake rationales.

Anthropic’s “Natural Emergent Misalignment from Reward Hacking in Production RL” (Nov 23rd 2025) showed an important safety finding. As soon as a model learned to reward hack in coding environments, there was a sharp increase in all misalignment evaluations. The model generalized to alignment faking, cooperation with malicious actors, and attempted sabotage. Alignment faking reasoning appeared in 50% of responses to simple questions. 12% of the time, the model intentionally attempted to sabotage safety research code.

This is a fun case study, being a perpetual psychology student. Teaching someone it’s okay to cut corners on one thing can erode their ethics across the board. This consistency + commitment tendency stands true for LLMs.

Researchers at Apple published a paper in Jun-2025 titled ‘The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity’. It found no meaningful evidence that reasoning models use actual deductive reasoning. It states standard LLMs outperform reasoning models on low-complexity tasks, both fail dramatically on high-complexity tasks, and introducing irrelevant information degrades performance by up to 65%. According to the researchers, this suggests pattern matching rather than genuine reasoning

← The Library