![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/openai-agent-loop.png\")
\n(Image source: OpenAI codex agent post)
The concept of recursive self-improvement (RSI) dates back to I. J. Good (1965), where he defined an “ultraintelligent machine” as a system that can surpass humans in all intellectual activities and design better machines to improve itself. Yudkowsky (2008) used the phrase “recursive self-improvement” for a specific feedback loop: an AI uses its current intelligence to improve the cognitive machinery that produces its intelligence.
\nThis feedback loop in modern AI may indicate the model rewriting its own weights directly, or more broadly the model improves the training pipeline and the deployment system, which in turn enables a better successor model with improved performance across economically valuable tasks. The speed of research development in AI has been shown to drastically accelerated in frontier labs (Anthropic; OpenAI).
\nI explicitly mention “deployment system” because the layer between the raw model and the real-world context seems to be as important as the model’s raw intelligence (i.e. the evals right after pretraining). Harnesses are important components of AI deployment, as shown by successful coding agent products such as Claude Code and Codex. A harness is the system surrounding a base model that orchestrates execution and decides how the model thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results.
\nThis one post will focus on research around harness engineering and how it contributes to RSI. Much recent work on auto-research, self-improving agents, and evolutionary program search can be organized around this question. Other work on model self-play, synthetic data, test-time training and a broader theme of continual learning also matches the RSI vision (e.g. Yuan et al. 2024, Chen et al. 2024), Zhao et al. 2025, Choi et al. 2026)) but they will not be the focus of this post.
\nCompared with early agent frameworks, “agent = LLM + memory + tools + planning + action”, harnesses engineering additionally include workflow design (e.g. loop engineering), evaluation, permission controls, and persistent state management. It is no longer only prompt templates, but closer to runtime and software system design: how the model observes, acts, memorizes, checks itself, and improves.
\nThe design should be deliberately simple and generic to enable generalization, likely with reference to existing software engineering practices to benefit from prertaining knowlege. There is also a strong analogy between operating systems and harnesses. Similar to an OS, a harness should encapsulate complicated logic while keeping the interface simple. Meanwhile, configs, tool interfaces and other protocols may gradually become standardized across the industry.
\nDefining a workflow in which the model can operate, test, and iterate is a key design for automation. Karpathy’s autoresearch repo (https://github.com/karpathy/autoresearch) is a clean example of how such a workflow can be constructed. A common workflow follows a goal-oriented loop of plan, execute, observe/test, improve, and execute again until the goal is achieved. The process may trigger proactive requests to users for clarity in task specification or execution preference.
\nThe workflow graph also emphasizes the model analyzing its own trajectories and failure cases and then iterating on its progress through an “agent runtime” rather than a static prompt template.
\nA recurring pattern in long-horizon agent systems is simple control over rich states and artifacts. A harness should not carry the entire workflow and all logs in context; instead, it should keep durable state in files. In long-horizon agentic rollout, artifacts such as experiment logs, code diffs, paper summaries, error traces, and past rollout trajectories often grow much longer than the context window that the model has trained for.
\nLearning how to read, write, and edit the file system (commonly via bash commands) is a foundation skill for LLMs, and thus managing persistent memory in the simple form of files naturally benefits from improvements in core model capability.
A harness can spawn multiple subagents to execute in parallel and monitor backend jobs. This is useful when the main agent needs to search multiple hypotheses, run experiments concurrently, or delegate isolated subtasks without polluting the main context. The parent agent then needs a small process manager: launch jobs, inspect logs, cancel failed runs, and merge results back into the main agent thread.
\nThe key design choice is to make parallelism explicit and inspectable. If subagent outputs only live in a transient chat context, they quickly become obselete and hidden. If they are stored as files, logs, and status records, the model can recover after interruptions and reason over its own execution history.
\nThe core interface of mainstream coding agents has become stabilized across Claude Code, Codex, OpenCode, and Cursor-style agents. They commonly use a loop like:
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/coding-harness-loop.png\")
With access to a set of tools, the coding agent is able to develop and debug issues in a given repository, similar to how human developers are equipped with IDEs.
\n(Not a comprenhensive list; shown for demonstration. Read this if interested.)
\n| Group | \nTool definitions | \n
|---|---|
| File system | \n- File discovery: glob, grep, ls - File read: read, read_many - File modification: write (a whole new file); edit (string exact-match replacement); multi_edit; apply_patch (applies a structured patch/diff) | \n
| Shell execution | \nRun commands: bash, PowerShell | \n
| IO | \nlsp, git tools like git_status, git_diff, git_commit | \n
| External context | \nMCP tools, Skills | \n
| Web search | \nweb_search, web_fetch, browser tools | \n
| Artifacts | \nRead docs, images; generate HTML, images | \n
| Backend processes | \nSuch as: CronCreate, CronDelete, CronList | \n
| Agent delegation | \nSuch as: spawn_agent, resume_agent, wait_agent, list_agents, close_agent, interrupt_agent, etc. | \n
It is hard to forecast how much the future of RSI will rely on harness engineering, but the near-term path of RSI is unlikely to start as a model directly rewriting its weights. My prediction of a practical near-term path is:
\nEventually it is possible that many harness improvements will be internalized into core model behavior, but the interface with external context and tools should remain. We have seen a softer version of this pattern with prompt engineering: manual prompt tricks became less central as instruction tuning and model reasoning improved, but the need to specify goals, constraints, context, and evaluation did not disappear.
\nThe progression in the object being optimized in the harness system is roughly: instruction prompts → structured context → workflow → harness code → optimizer code. As the model becomes more intelligent and powerful, we move toward more complex targets and generic methods.
\nSimply appending all the tool responses and model generations into the context can quickly grow out of control as the agentic job horizon increases significantly. Context management is a layer to construct a more structed and concise context for LLM and manage persistant states. There is no doubt that long-context research will keep on making progress but at the moment long-context intelligence and context engineering sometime intertwines.
\nAgentic Context Engineering (ACE; Zhang et al. 2025) treats context as an evolving playbook rather than an increasingly lengthening prompt. It has three components to maintain one context playbook of bullet points, each with an identifier and a description.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/ace.png\")
To prevent context collapse and brevity bias during iterative rewrites, one key design choice in ACE is that the curator does not rewrite a full prompt blob. It instead outputs a collection of structured, itemized bullets in the form of (identifier, description), and these bullets are merged into a structured context logbook with deterministic logic. The context items are refined and deduplicated periodically.
\nThe fact that ACE learns insights from rollouts helps us move toward self-managed memory, but the update rules and the overall workflow are still handcrafted. To move toward a more self-improving loop, Meta Context Engineering (MCE; Ye et al. 2026) separates the mechanism (how to manage context) from the artifact content (what is in context), running skill evolution at the meta-optimization level and context optimization at the base level.
\nAn MCE skill s ∈ 𝒮 defines a context function cₛ=(ρₛ,Fₛ) and maps an input x to context c = Fₛ(x;ρₛ), where:
\nThe bi-level optimization is to find the best context cₛ^* given skill s on the training data, while the outer loop finds the optimal skill that provides the best performance on the validation set:
\nInner: cₛ^*=argmax_(cₛ)Jₜᵣₐᵢₙ(cₛ;s) Outer: s^*=argmax_(s∈𝒮)Jᵥₐₗ(cₛ^*)
\nThe skill database tracks the history of previous skills, context functions and eval metrics ℋₖ₋₁ = {(sᵢ,cᵢ,Jᵢᵗʳᵃⁱⁿ, Jᵢᵛᵃˡ)}ᵢ₌₁ᵏ⁻¹. A meta-level agent performs agentic crossover over prior skills to create a new skill given a task τ: sₖ=crossover(τ,ℋₖ₋₁).
\nThen a base-level context engineer executes the skill sₖ and learns the context function from rollout feedback ℛₖ, guided by the current skill: cₖ=engineer(τ,sₖ;cₖ₋₁^*,ℛₖ).
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/mce.png\")
MCE does not enforce a heuristic rule for how to structure context as ACE does. It uses free-form skills to store the most important knowledge for a task, and evolves the skill and the skill-conditioned context iteratively together. Implementation-wise, a context function c is instantiated as a collection of files in a dedicated directory, including both static (skill.md) and dynamic (context and data rollouts) components. Both meta-level and base-level optimization are executed in agentic coding envs with a standard tool set,
𝒯={Read,Write,Edit,Bash,Glob,Grep,TodoWrite}
\nMeta-Harness (Lee et al. 2026) moves another level deeper: the optimized object is the code that determines and optimizes what information should be stored, retrieved, and presented to the model. “Meta-” in its name means it is a harness for optimizing harnesses.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/meta-harness-outer-loop.png\")
The proposer for creating a new harness is itself a coding agent and the final output is a collection of harness candidates on the Pareto frontier.
\ngrep or cat to read through it instead of shoveling everything into a single prompt context.![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/meta-harness.png\")
Still, the important lesson is clear: once harness design becomes an executable search space, a strong coding agent can exploit the same design space human engineers use.
\nWorkflow design in harness engineering can be handcrafted by domain experts. Taking auto-research as an example, various frameworks have been proposed and tested. The AI Scientist system (Lu et al. 2026) builds a pipeline to propose research ideas, write code, run experiments, analyze results, write a manuscript, and perform peer review. Meng et al. (2026) make verifiability the central design constraint in ScientistOne, where every claim (citation, numerical, methodological, conclusion) must trace to an evidence source and is audited by Chain-of-Evidence checks.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/ai-scientist.png\")
The Autodata agent (Kulikov et al. 2026) is designed to work as a data scientist for generating training and evaluation data. The main agent manages a challenger that proposes problems, a weak solver, a strong solver, and a verifier/judge, aiming to synthesize data at the “just right” level of difficulty, meaning that the strong solver succeeds but the weak solver fails.
\nIn Autodata, the challenger prompt is updated iteratively according to feedback from the solvers and verifier. The limitation here is that synthesized tasks are used to fine-tune weak solvers but not strong solvers; if the loop cannot iteratively improve the strong model, it is more like indirect distillation over a generated prompt distribution, with less RSI flavor.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/autodata.png\")
The design space for workflow is enormous, and naturally we can think of workflow design as a search problem, and therefore we should be able to find good solutions by algorithms rather than only manually craft them. Following this direction, Automated Design of Agentic Systems (ADAS; Hu et al. 2025) formulates agent design itself as an optimization problem, “meta-agent search” where a meta-agent proposes new designs of agentic workflows.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/adas.png\")
AFlow (Zhang et al. 2025) represents an agentic workflow as a graph, where nodes represent LLM-invoking actions and edges implement logical operations in code. The workflow optimization relies on MCTS (Monte Carlo Tree Search):
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/aflow.png\")
Experiments of AFlow in QA, code, and math tasks showed decent improvement of AFlow over manually designed workflows and ADAS.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/aflow-exp.png\")
Either context engineering or workflow design is only one part of a harness. We need to search through the entire design space and optimize context-management logic, workflow, permissions, and many other harness components together. As we have seen in work like Meta-Harness, ADAS, and AFlow, ✨code✨ is a universal language for defining programs and systems. In simple words, a harness is code that programs how prompts, tool calls, subagents, control flow, memory, and workflow logic work together. If an LLM can optimize the code that executes agents, it can access a much larger design space than hand-written prompts.
\nSelf-Taught Optimizer (STOP; Zelikman et al. 2023) is one of the early examples of recursive scaffolding improvement. A seed improver I₀ at step t=0 takes an initial solution s, a utility function u, and a black-box language model M, and returns an improved solution s’, that is, s’ = I(u, s; M). The goal of STOP is not directly to improve s but to improve the improver I itself.
\nFirst, let’s define the meta-utility as the average utility of a given improver function I over a collection of downstream tasks 𝒟:
\nû(I) ≜ 1/|𝒟|𝔼_((u,s)∼ 𝒟)[u(I(u,s; M))]
\nBecause improving the improver function is an optimization problem itself, we can recursively get a new version of Iₜ based on Iₜ₋₁’s performance measured by meta-utility via a self-improvement update:
\nIₜ=Iₜ₋₁(û,Iₜ₋₁;M)
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/STOP-algo.png\")
In their experiments, the improved improver discovered various strategies, such as genetic algorithms, decomposing and improving parts, multi-armed prompt bandits, simulated annealing, varying temperature, and beam/tree search. This is analogous to how a harness workflow can be represented as an object for optimization.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/STOP-patterns.png\")
A cautionary result in Zelikman et al. (2023)’s findings is that STOP improved mean downstream performance across iterations with GPT-4 but degraded with weaker models like GPT-3.5 and Mixtral. Recursive structure alone is not enough. The base model must be capable enough to improve the mechanism. This implies that harness improvement enables better deployment of the model but intelligence is still the core.
\nLin et al. (2026) investigated the dependency of harness evolution on model capabilities in more details. They disentangled two axes: (1) harness-updating refers to the capability of producing useful harness edits and (2) harness-benefit denotes the capability of utilizing the updated harness, to achieve better task solving. Interestingly a range of model of different sizes and core intelligence, from Qwen3.5-9B to Claude Opus 4.6, were observed in their experiments to show similar harness updating capability; the 9B harness proposer/evolver is able to write a skill procedurally isomorphic to Opus. To best utilize a harness, a model needs to invoke skills/tools correctly and timely and be good at long-horizon instruction following.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/harness-update.png\")
A more recent work, Self-Harness (Zhang et al. 2026), relies on LLM agents to improve their own harness via a propose-evaluate-accept loop.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/self-harness.png\")
The loop in Self-Harness has three stages:
\nWhen running MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5 on Terminal-Bench-2, Self-Harness was shown to learn model-specific harness instructions that target at different weaknesses of different base models and improve held-out pass rates.
Self-harness type of work does raise my concerns that if a program is allowed to edit the OS system, abstraction boundaries are broken. The editable surface needs to be properly designed and the permission control and security layers need to live outside this loop. All the challenges around reward hacking still remain.
\nAgentic Harness Engineering (AHE; Lin et al. 2026) see the bottlenecks of harness evolution are around observability—that is, when a rollout fails, we need to know which component is responsible for that and every edit should be grounded by evidence.
\nThe framework creates a closed loop with 3 observability pillars:
\nOn Terminal-Bench-2, AHE achieved better than human-designed harness (OpenCode, Terminus-2, Codex) except for Hard tier and a few other self-evolve baselines (ACE, TF-GRPO). The same frozen harness, without further evolving, transfers to SWE-bench-verified, indicating that the evolved harness is able to encode engineering experience into harness components rather than doing benchmark-specific optimization.
\nEvolutionary search is an optimization method inspired by natural selection (see my old post on evolutionary algorithm). It evolves a population of solutions by mutating them and only keeping those with high “fitness” in the crowd. Evolutionary search comes in handy when (1) the search space is extensive or weirdly shaped; and (2) it is hard to optimize directly with gradients but easy to evaluate solutions. Harness search seems to be a good fit here.
\nEvolutionary search has been used in prompt engineering in the past studies. Promptbreeder (Fernando et al. 2023) optimizes task-specific prompts through a rich set of mutation operations, and interestingly the mutation prompts (i.e. instructions to an LLM to mutate a task prompt) are themselves also improved through evolution. GEPA (Agrawal et al. 2025) combines reflection-based prompting with evolutionary search and uses natural language reflection over trajectories of trial and error to propose prompt updates.
\nNovikov et al. (2025) introduced AlphaEvolve as a coding-agent evolutionary search system, which stores a pool of candidate programs and prompts frozen LLMs to generate diffs for improvement. As the system repeatedly evaluates child programs and keeps successful ones, it discovers better solutions in time.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/alphaevolve.png\")
A few details matter in the design of AlphaEvolve:
\n# EVOLVE-BLOCK-START and # EVOLVE-BLOCK-END.Ablations show the evolution procedure, context in prompts, meta-prompts, full-file evolution and the use of stronger LLMs.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/alphaevolve-plot.png\")
Recent variants such as ThetaEvolve (Wang et al. 2025) combines evolutionary search with RL and in-context learning, and DemoEvolve (Che, et al. 2026) augments the self-rollout archive with human expert demonstrations as reference experience for harness-level diagnosis and editing. ShinkaEvolve (Lange et al. 2025), on the other hand, introduced three new components to improve LLM sampling efficiency:
\nUnlike the methods above, which focus on solution improvement, Darwin Gödel Machine (DGM; Zhang et al. 2025) explicitly targets the evolution of an editable harness-code repository with an LLM-based coding agent. Precisely, this agent is allowed to modify its own harness. A follow-up work on Hyperagents (Zhang et al. 2026) introduced a meta-agent to control how to modify existing task agents to create new ones.
\n<bash_command>) and (2) editor (args: view/create/edit <file_path>).DGM is harness evolution under a fixed model. In experiments with Claude 3.5 Sonnet as the base LLM and simple initial harness configs, the DGM-discovered agents are comparable to or outperform handcrafted agents on SWE-bench Verified (20% to 50%) and Polyglot (14.2% to 30.7%).
This family of methods works well when candidate solutions are automatically evaluable and candidate fitness is easy to quantify, such as matrix multiplication, GPU kernel optimization, algorithm contests, datacenter scheduling. It struggles with domains where evaluation is slow, ambiguous, or mostly heuristic-based. The compute efficiency and effectiveness of evolution are also concerns.
\nHarness evolution changes the non-parametric system around the model. To enable full self-improvement, the model can totally be allowed to update its own weights at the same time. The weight update can be implemented via improvements in the model training pipeline or continual learning at test time. The topic of continual learning is worthy of its own post in the future.
\nSIA (Hebbar et al. 2026) is an early attempt to combine harness improvement and model-parameter updates in the same optimization loop, with three components in the design:
\n![\"\"](\"https://lilianweng.github.io/posts/2026-07-04-harness/SIA.png\")
There are a few confounding choices in SIA’s experiments that make the results hard to interpret. For example, the task-specific agent is much weaker than the models used for the Meta-Agent and Feedback-Agent (gpt-oss-120b vs Claude Sonnet 4.6), and the baselines are too weak to cross-reference cleanly against related methods. I would consider the direction interesting, but the evidence provisional. Yet many challenges, such as training stability and Goodhart effect, still remain open.
Continual Harness (Karten et al. 2026) experimented in long-horizon gameplay setting with harness updating and co-learning a policy model by distilling a strong teacher model’s labels on low-reward trajectories.
\nThe AI Scientist line of work is a strong demonstration that an expert-designed harness can coordinate a large portion of auto-research loop, experimented in the form of writing research papers. But paper production is not identical to scientific discovery. A system can write a plausible manuscript while still having fabricated citations, implementation drift, or weak experimental results.
\nTrehan & Chopra (2026) tested whether LLMs can go from a research idea to a paper with minimal scaffolding and basic tools (i.e., read_file, write_file, llm_search, list_files). Each idea had a dedicated workspace where agents could generate and read documents as part of context. They experimented in three domains (world models, multi-agent RL, AI safety & alignment), with each domain containing 45-50 high-quality seed documents to inspire new ideas. Only four ideas were selected by human experts to run through the full pipeline, and only one was fully executed into a paper. They observed six recurring failure modes in the experiments:
Toward full RSI, researchers have made real progress, but several bottlenecks remain.
\n1. Weak and fuzzy evaluators. Many research claims do not have a fast and precise verifier, and the same is true for many real-world tasks. Current self-improvement loops work best for tasks when evaluation metrics are measurable and objective, similar as how RL works.
\nResearch taste, novelty, and long-term scientific value are much harder to measure. For example, research taste often mixes problem framing, experimental design, and judgment about which surprising results are worth pursuing and which failure cases are worth retries.
\n2. Context and memory lifecycle. Memory grows as AI agents become more autonomous and independent. A useful harness needs to manage context and memory to complement existing limitation in long-context generation while still maximizing the success of long-horizon tasks. Since humans are able to maintain memory through our life time, I see an anoloy here that context engineering will and should become a core part of intelligence, rather than staying in the software system layer.
\n3. Negative results. Researchers are incentivized to publish successful results and thus literature is biased toward successes. LLMs trained on a vast amount of data (mostly human created, at least for now, lol) may be bad at deciding when to abandon a hypothesis, report a negative result, or even acknowledge a failure due to the imablance of success vs failure cases in data. A research harness should make failed attempts easy to preserve, as learning from failure is the best way to trim down the task search space.
\n4. Diversity collapse. Evolutionary and RL loops tend to exploit known high-reward patterns. We need mechanisms to prevent the population from collapsing into variants of the same solution. This is especially critical for open-ended research, where the best path may initially look worse under the current evaluator.
\n5. Reward hacking. A self-improvement loop optimizes whatever signal it is given. If the reward comes from unit tests, the agent may overfit to tests; if it comes from a judge model, it may learn reward hacking tricks specific to this judge; if it comes from benchmark scores, it may exploit benchmark artifacts.
\nThe evaluator and permission control should likely sit outside the loop that evolves harness, with held-out tests, trace audits, and human review at decision points that matter—how much oversight can be scaled up and automated remains an open research area.
\n6. Long-term success. An extrinsic loop of optimization works on rewards outside of individual rollouts that we can simulate in training sandbox.
\nTake coding agent as an example. Coding agents have already increased daily productivity in software engineering, but many optimization goals are still too short-term. It can often complete the task at hand, but less obvious how it should protect the long-term health of a repo collectively maintained by hundreds or thousands of engineers. Standard sandbox-based RLVR-style training rarely captures maintainability, ownership boundaries, migration cost, backwards compatibility, or future debugging burden.
\n7. The role of humans. Humans should move up the stack, not be removed from the loop, meaning that human should provide oversight at the right time, at the right abstraction level and our system design should consider when and how to set up such touch points.
\nMany challenges listed above need human’s feedback and steering. After all, we are building the technology for better future of humanity, not other way around.
\nPlease cite this work as:
\n\n\nWeng, Lilian. “Harness Engineering for Self-Improvement”. Lil’Log (Jul 2026). https://lilianweng.github.io/posts/2026-07-04-harness/
\n
Or use the BibTeX citation:
\n@article{weng2026harness,\n title = {Harness Engineering for Self-Improvement},\n author = {Weng, Lilian},\n journal = {lilianweng.github.io},\n year = {2026},\n month = {July},\n url = "https://lilianweng.github.io/posts/2026-07-04-harness/"\n}\nClaude 3.5 Sonnet, ~21%) does not outperform ML PhDs.GPT-4o and GPT-4o-mini) achieved only 21% accuracy on the hardest task.o1-preview with AIDE scaffolding, reached at least Kaggle bronze-medal level in 16.9% of competitions.[1] Good, I. J. “Speculations Concerning the First Ultraintelligent Machine.” Advances in Computers, 6:31–88, 1965.
\n[2] Yudkowsky, Eliezer. “Recursive Self-Improvement.” LessWrong, 2008.
\n[3] Choi, et al. “Anchored Self-Play for Code Repair.” ICML 2026.
\n[4] Zhao, et al. “Absolute Zero: Reinforced Self-play Reasoning with Zero Data.” arXiv preprint arXiv:2505.03335, 2025.
\n[5] Yuan, et al. “Self-Rewarding Language Models.” arXiv preprint arXiv:2401.10020, 2024.
\n[6] Chen, et al. “Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models.” ICML 2024.
\n[7] Zhang, et al. “Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models.” ICLR 2026.
\n[8] Ye, et al. “Meta Context Engineering via Agentic Skill Evolution.” arXiv preprint arXiv:2601.21557, 2026.
\n[9] Lee, et al. “Meta-Harness: End-to-End Optimization of Model Harnesses.” arXiv preprint arXiv:2603.28052, 2026.
\n[10] Lu, et al. “Towards end-to-end automation of AI research.” Nature, 651:914–919, 2026.
\n[11] Meng, et al. “ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence.” arXiv preprint arXiv:2605.26340, 2026.
\n[12] Kulikov, et al. “Autodata: An agentic data scientist to create high quality synthetic data.” arXiv preprint arXiv:2606.25996, 2026.
\n[13] Hu, Lu, and Clune. “Automated Design of Agentic Systems.” ICLR 2025.
\n[14] Madaan, et al. “Self-Refine: Iterative Refinement with Self-Feedback.” NeurIPS 2023.
\n[15] Zhang, et al. “AFlow: Automating Agentic Workflow Generation.” ICLR 2025.
\n[16] Zelikman, et al. “Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation.” COLM 2024.
\n[17] Zhang, et al. “Self-Harness: Harnesses That Improve Themselves.” arXiv preprint arXiv:2606.09498, 2026.
\n[18] Fernando, et al. “Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution.” arXiv preprint arXiv:2309.16797, 2023.
\n[19] Agrawal, A. et al. “GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning.” arXiv preprint arXiv:2507.19457, 2025.
\n[20] Novikov, et al. “AlphaEvolve: A coding agent for scientific and algorithmic discovery.” arXiv preprint arXiv:2506.13131, 2025.
\n[21] Lange, Imajuku, and Cetin. “ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution.” arXiv preprint arXiv:2509.19349, 2025.
\n[22] Wang, et al. “ThetaEvolve: Test-time Learning on Open Problems.” arXiv preprint arXiv:2511.23473, 2025.
\n[23] Zhang, et al. “Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents.” arXiv preprint arXiv:2505.22954, 2025.
\n[24] Zhang, et al. “Hyperagents.” arXiv preprint arXiv:2603.19461, 2026.
\n[25] Yuksekgonul, et al. “Learning to Discover at Test Time.” arXiv preprint arXiv:2601.16175, 2026.
\n[26] Riaz, et al. “Epistemic Uncertainty for Test-Time Discovery.” arXiv preprint arXiv:2605.11328, 2026.
\n[27] Hebbar, et al. “SIA: Self Improving AI with Harness & Weight Updates.” arXiv preprint arXiv:2605.27276, 2026.
\n[28] Trehan and Chopra. “Why LLMs Aren’t Scientists Yet: Lessons from Four Autonomous Research Attempts.” arXiv preprint arXiv:2601.03315, 2026.
\n[29] Bubeck, et al. “Early science acceleration experiments with GPT-5.” arXiv preprint arXiv:2511.16072, 2025.
\n[30] Starace, et al. “PaperBench: Evaluating AI’s Ability to Replicate AI Research.” ICML 2025.
\n[31] Wijk, et al. “RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts.” ICML 2025.
\n[32] Chan, et al. “MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering.” arXiv preprint arXiv:2410.07095, 2024.
\n[33] Chen, et al. “ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery.” ICLR 2025.
\n[34] Siegel, et al. “CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark.” TMLR 2024.
\n[35] Ouyang, et al. “KernelBench: Can LLMs Write Efficient GPU Kernels?” arXiv preprint arXiv:2502.10517, 2025.
\n[36] Lin, et al. “Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents.” arXiv preprint arXiv:2605.30621, 2026.
\n[37] Lin, et al. “Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses.” arXiv preprint arXiv:2604.25850, 2026.
\n[38] Karten, et al. “Continual Harness: Online Adaptation for Self-Improving Foundation Agents.” arXiv preprint arXiv:2605.09998, 2026.
\n[39] Che, et al. “DemoEvolve: Overcoming Sparse Feedback in Agentic Harness Evolution with Demonstrations.” arXiv preprint arXiv:2605.24539, 2026.
\n", "date_modified": "2026-07-04T00:00:00+00:00", "date_published": "2026-07-04T00:00:00+00:00", "external_url": "https://lilianweng.github.io/posts/2026-07-04-harness/", "id": "urn:aggr:item:f02f9779fd0043e755ca3cc3d5dc7347679fd454", "language": "en-US", "summary": "The concept of recursive self-improvement (RSI) dates back to I. J. Good (1965), where he defined an “ultraintelligent machine” as a system that can surpass humans in all intellectual activities and design better machines to improve…", "tags": [ "programming" ], "title": "Harness Engineering for Self-Improvement", "url": "https://aggr.aymericbeaumet.com/items/lilianweng-github-io/2026-07-04-harness-engineering-for-self-improvement/" }, { "content_html": "Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss L decreases predictably as we scale up model size N, dataset size D, and compute C, following a power-law curve, which appears as a straight line on a log-log plot. We can view scaling laws as a framework for describing the relationship between compute, loss, model size and data; at its core, it is about how to allocate precious compute optimally between N and D.
\nThis predictability makes scaling laws highly valuable in practice. A common workflow is to fit scaling laws on a handful of small runs and then extrapolate to estimate the token and compute requirements for larger models.
\n| Symbol | \nNote | \n
|---|---|
| N | \nModel size, measured in parameter count. | \n
| D | \nTraining dataset size, usually measured in token count. | \n
| C | \nTraining compute in FLOPs. As a useful approximation, C ≈ 6ND (Kaplan et al. 2020), where $2ND$ accounts for the forward pass and $4ND$ for backpropagation. | \n
| E | \nIrreducible loss | \n
| L, L̂(.) | \nTest loss / test loss prediction function; can also refer to training loss, since they are strongly correlated. | \n
| ϵ | \nGeneralization error. | \n
The predictability of generalization error with scale had already been investigated before scaling laws became a mainstream concept.
\nAmari et al. (1992) derived four types of learning curves using a Bayesian approach and the annealed approximation.
\nϵ ∼ c ⋅ D^α + E
\nwhere E can be 0 and α = −2, −1, −1/2. Although their theoretical setup is based on a simplified binary classification task, it points in a useful direction for building empirical ML loss prediction models.
\nOne of the earliest empirical studies by Hestness et al. (2017) explained the relationship between generalization error, model size and data. For a given training data size, they identified the best-fit model size via grid search and then plotted loss against training dataset size. Across four different domains in deep learning (neural machine translation, image classification, language modeling, and speech recognition), a recurring pattern was observed where:
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/hestness-1.png\")
A conceptual illustration breaks the learning curve into three stages. In the small-data region, when there are not enough learning signals, the model performs only slightly better than random guessing. In the middle (“power-law region”), we observe a power-law relationship between loss, data, and model size. The final irreducible-error region can be attributed to factors such as noise in the data.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/hestness-2.png\")
Rosenfeld et al. (2020) pushed this further by trying to model error as a joint function of both model size N and data size D, across a diverse set of architectures (ResNet, WRN, LSTM, Transformer) and optimizers (Adam, SGD variants). Empirically they observed that, holding one axis fixed, the error decays as a power law in the other:
\nL̂(D,N) ≈ A/N^α + E_N, L̂(D,N) ≈ B/D^β + E_D
\nwhich can be combined into a joint form:
\nL̂(D, N) ≈ A/N^α + B/D^β + E
\nwhere A > 0, B > 0, α ≥ 0, β ≥ 0 are scalar constants and E is not dependent on either N or D.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/rosenfeld-1.png\")
Thus, they can build a prediction model in the form of a simple parametric function with θ = ⟨ A, B, E, α, β ⟩ to predict the expected loss for (D, N) > certain thresholds by only training on a set of smaller training configs, (D, N) < certain thresholds.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/rosenfeld-2.png\")
Side note: These early works lean on classical learning-theory intuition like the VC dimension (the cardinality of the largest set of points a model can shatter) as a proxy for capacity, but in modern deep learning work the VC dimension is often too coarse to explain the behavior and the empirical power laws turned out to be much cleaner and more practical than the worst-case bounds that theory provides.
\nKaplan et al. (2020) popularized the concept of scaling laws in the language modeling community. They found that the cross-entropy test loss L scales as a power law with each of model size N (excluding embedding layers), dataset size D, and training compute C across many orders of magnitude. The findings are aligned with early work in the last section, but Kaplan et al. formalized the concept with a focus on Transformer language models and empirical experimentation at a larger scale, with model size ranging from 768M to 1.5B non-embedding parameters and dataset size from 22M to 23B tokens. All training runs in the paper used a learning rate schedule with a 3000 step linear warmup, followed by a cosine decay to zero.
\nList of key findings:
\nThey summarize the joint dependence on N and D in a single equation:
\nL̂(N,D) = [ (a/N)^(α/β) + b/D ]^β
\nA nice consequence of this form is that the extent of overfitting (i.e. model is complex or data is small) depends predominantly on the ratio N^(α / β) / D, which indicates that the data needs to grow in a specific proportion to the growth of the model size to avoid training being data-limited.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/kaplan-1.png\")
The most influential and, in hindsight, most contested conclusion was the compute-optimal allocation. Kaplan et al. found Nₒₚₜ ∝ C^(0.73) and concluded that model size should grow faster than dataset size. Concretely, for a 10x increase in compute they suggested scaling the model size by ~5.5x but the training tokens by only ~1.8x. The Chinchilla paper would later overturn this recommendation, arguing that it leaves large models badly undertrained.
\nAnother useful analysis in Kaplan et al. approximates the number of training FLOPs needed based on D and N. Each multiply-add is counted as ~2 FLOPs.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/kaplan-2.png\")
Given a standard config where dₐₜₜₙ = d_(model) = d_(ff)/4, and excluding embedding layers from N and the per-token forward compute:
\nN = n_(layer) d_(model) 3 dₐₜₜₙ + n_(layer) dₐₜₜₙ d_(model) + n_(layer) 2 d_(model) d_(ff) ; no embedding layer
= 2 n_(layer) d_(model)(2dₐₜₜₙ + d_(ff))
= 12 n_(layer) d_(model)²
C_(fwd) = 2 n_(layer) (d_(model) 3 dₐₜₜₙ + n_(ctx)dₐₜₜₙ + dₐₜₜₙd_(embed) + 2 d_(model) d_(ff))
= 2 n_(layer) (12 d_(model)² + n_(ctx)dₐₜₜₙ)
= 2N + 2 n_(layer)n_(ctx)dₐₜₜₙ
≈ 2N ; assuming n_(ctx) < 12 d_(model) and the n_(ctx) term is relatively small.
Then we count backward-pass FLOPs as twice the forward-pass FLOPs, because backpropagation runs two matrix multiplications, for gradients with respect to the input activations and the weights, respectively. Thus, in total, the training FLOPs per token are approximately 6N, and the total FLOPs for training over D tokens are C ≈ 6ND.
\nThe Chinchilla paper (Hoffmann et al. 2022) studied the relationship between the optimal model size N (total parameters, including embeddings) and the number of tokens D under a fixed compute budget C with a more careful experimental design and arrived at a somewhat different answer from Kaplan et al..
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/animal.png\")
The central question is on the best strategy to allocate resources given a constraint FLOPs(N, D) = C ≈ 6ND. In other words, when we have only limited FLOPs (a given number of GPUs running for a given period of time), how should we choose between more data tokens and more model parameters?
\nNₒₚₜ(C), Dₒₚₜ(C) = *arg min_(s.t. FLOPs(N,D) = C) L̂(N, D)
\nThe Chinchilla paper presented three neatly designed methods for scaling laws fitting.
\nThe empirical experiments scanned over 400 models, with sizes from 70M to over 16B parameters and training tokens from 5B to 500B. The experiments were under the assumption that every training token is unique (the infinite-data regime). All runs used a cosine learning-rate schedule decaying by 10x over the training horizon. Sweeping over model sizes traces out the compute-optimal frontier.
\nFor each parameter count N, train several runs with different token budgets, and record the minimal loss achieved per FLOP budget C.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/chinchilla-1.png\")
Fix a compute budget C and plot the final loss against parameter count N. Each iso-FLOP curve is roughly a parabola in log-space, and its minimum flags the optimal model size for that compute budget. Then repeating across budgets traces a power-law line in the plot.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/chinchilla-2.png\")
Fit the same parametric function as in Rosenfeld et al. (2020) directly,
\nL̂(N, D) = A/N^α + B/D^β + E
\nWe can actually get a closed form approximation of optimal Nₒₚₜ(C), Dₒₚₜ(C) by minimizing L̂(N, D) under the constraint FLOPs(N,D) = C ≈ 6ND.
\nFirst let’s reduce the expression to contain only N:
\nL̂(N) = A N^(−α) + B (C/6)^(−β)N^β + E
L̂′(N) = −α A N^(−α−1) + β B (C/6)^(−β) N^(β −1) = 0 ; derivative wrt N should be zero.
Thus α A N^(−α−1) = β B (C/6)^(−β) N^(β −1)
α A = β B (C/6)^(−β) N^(α + β)
Nₒₚₜ = ((α A)/(β B))^(1/(α + β)) (C/6)^(β/(α+β))
Dₒₚₜ = C/(6 Nₒₚₜ) = ((β B)/(α A))^(1/(α + β)) (C/6)^(α/(α+β))
When α ≈ β, model size and training tokens should scale at equal rates.
\nTo find the optimal θ = ⟨ A, B, E, α, β⟩, the Chinchilla paper adopts a Huber loss (robust to outliers; δ=10⁻³) and the L-BFGS algorithm (good for curve fitting with a small number of parameters).
\n\\begin{align} \\min_{A,B,E,\\alpha,\\beta} \\sum_{\\text{runs }\\{i\\}} \\text{Huber}_\\delta (\\log \\hat{L}(N_i, D_i) - \\log L_i) \\\\ \\text{ where }\\text{Huber}_\\delta (x) = \\begin{cases}\\frac{1}{2} x^2 & \\text{for }\\vert x \\vert \\leq \\delta \\\\ \\delta \\cdot (\\vert x \\vert - \\frac{1}{2}\\delta), & \\text{otherwise.}\\end{cases} \\end{align}
Chinchilla arrives at its answer through three complementary methods whose final results agree with each other, and this is part of why the result was quite convincing.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/chinchilla-3.png\")
![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/chinchilla-4.png\")
The claim in the Chinchilla paper that most large models (at the time, ~2022) were undertrained is supported by a famous demonstration: under the same compute budget as Gopher (Rae et al. 2021; 280B parameter count, 300B token budget), they trained Chinchilla (70B parameter count, 1.4T token budget), a model 4x smaller but trained on roughly 4x more tokens and it outperformed Gopher across the board.
\nThe Chinchilla scaling laws disagree with Kaplan et al. as follows:
\nBoth papers still agree on the same underlying principle, but they disagree on where the optimal size-vs-token tradeoff lies. Why do they disagree so much?
\nDifference 1: Kaplan et al. experimented mostly on small models. Kaplan et al. experimented mostly on smaller models, while the Chinchilla paper’s experiments reached more than 10x larger scales. When we extrapolate in log-log space, a small difference in the fit can result in large differences (See toy simulation).
\nDifference 2: Embedding parameter count matters for small models. In the small-parameter regime, embedding parameters are a non-negligible fraction of the total and thus counting them or not matters. Pearce & Song (2024) did a thorough analysis along this line. Let’s use N_(∖ E), C_(∖ E) to denote model size and compute when embedding is excluded and use N, C to count total parameters.
\nTo bridge them, they fit a relationship between total parameters N_T and non-embedding parameters N_(∖ E), for some constant ω:
\nN = N_(∖ E) + ω N_(∖ E)^(1/3).
\nThis form has nice properties of being strictly increasing and lim_(N → ∞) N = N_(∖ E) (because N/(N_(∖ E)) = 1 + ω N_(∖ E)^(− 2/3), lim_(N_(∖ E) → ∞) N/(N_(∖ E)) = 1.
\nPlugging this into the Chinchilla laws equation,
\nL(N_(∖ E), C_(∖ E)) = A(N_(∖ E) + ω N_(∖ E)^(1/3))^(−α) + B ((C_(∖ E))/6)^(−β) N_(∖ E)^β + E
L′(N_(∖ E), C_(∖ E)) = − α A (N_(∖ E) + ω N_(∖ E)^(1/3))^(−α −1)(1 + ω/3N_(∖ E)^(−2/3)) + β B ((C_(∖ E))/6)^(−β) N_(∖ E)^(β −1) = 0 ; derivative wrt N_(∖ E) should be zero.
Rearrange to get α A (N^*_(∖ E) + ω N^*_(∖ E)^(1/3))^(−α −1)(1 + ω/3 N^*_(∖ E)^(−2/3)) = β B ((C_(∖ E))/6)^(−β) N^*_(∖ E)^(β −1)
6^(−β)(α A)/(β B) (N^*_(∖ E) + ω N^*_(∖ E)^(1/3))^(−α −1)(1 + ω/3N^*_(∖ E)^(−2/3)) N^*_(∖ E)^(1 − β) = C_(∖ E)^(−β)
6 ((β B)/(α A))^(1/β) (N^*_(∖ E) + ω N^*_(∖ E)^(1/3))^((1 + α)/β) (N^*_(∖ E) + ω/3N^*_(∖ E)^(1/3))^(−1/β) N^*_(∖ E) = C_(∖ E)
The relationship between C_(∖ E) and N_(∖ E) in the above equation is no longer a clean power law. We can only approximate it locally as N^*_(∖ E) ∼ C_(∖ E)ᵍ, where g is a local exponent based on a first-order derivative (∼) rather than a global power-law exponent, resulting in g = (d log C_(∖ E))/(d log N_(∖ E)). See the full details of how the exponent g is approximated in Appendix A.1 in Pearce & Song (2024).
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/pearce-1.png\")
As shown in the visualization above, as C_(∖ E) gets larger, g converges to the Chinchilla estimate. By generating synthetic training curves using above equation, in the range of model size from 768M to 1.5B (as in Kaplan et al.), they estimated that g is close to the Kaplan coefficient of 0.73 in that region.
\nPower laws are widely observed across many domains outside AI, such as in Zipf’s law, scale-free networks, urban scaling laws, and many other complex systems. The recurring pattern is that large events are rare, small events are common and the relationship between size and frequency often follows a straight line at log-log scale.
\nWhy do LLM scaling laws also have the shape of a power law?
\nInspired partly by different domains displaying different exponents (Hestness et al. 2017), one early explanation by Sharma & Kaplan (2020) hypothesizes that language modeling can be viewed as doing regression on a low-dimensional manifold of data. More model parameters can induce a finer partition of the data manifold and therefore smaller generalization error. In the simplest terms, if a model of effective size N partitions a d-dimensional manifold into O(N) regions, the typical linear resolution scales like ∼ N^(−1/d). This has a similar power-law form to the scaling laws above. This theory applies most cleanly in the infinite-data, underfitting regime, but in reality estimating the intrinsic dimension of a data manifold is quite hard.
\nA later hypothesis (Michaud et al. 2023, Brill 2024) assumes that knowledge or skills are learned in discrete chunks (“quantized”) and that the frequency distribution of these skills follows a power law. The model learns common skills first and rare skills later, resulting in a smooth power-law decay in loss.
\nI only listed two hypotheses here, but there are more studies on explaining the shape of power-law scaling through spectral tails of data, kernel eigenvalues, natural-language statistics, or phase transitions in training dynamics.
\nClassic scaling laws assume effectively unlimited unique data, no repetition, and no multi-epoch training. As the model size grows significantly, we are running out of enough high-quality unique tokens. In fact, some arguments about how long scaling in AI can continue are centered on whether we are hitting a “data wall”.
\nIt is also worth emphasizing that the dataset behind D is expected to be already cleaned. The pretraining data pipeline is often a large part of an effective pretraining pipeline, with common steps like deduplication (exact and fuzzy), quality filtering, boilerplate removal, safety filtering, PII/copyright masking, benchmark decontamination and careful reweighting of data mix components based on language, quality, content type, etc. Even when two datasets contain the same token count D, a high-quality dataset and a dataset of Internet slop can yield drastically different compute efficiency.
\nThe study by Hernandez et al. (2022) focused on a controlled version: a mostly-unique dataset with a small fraction of repeated data. Starting from a large dataset, the data mix keeps 90% non-repeated but replaces the remaining 10% with repeats of a tiny portion of the original. By training a Transformer model for 100B tokens, they observed a double-descent phenomenon, that is, the test loss can actually get worse and then better again as a function of how much the repeated data is emphasized, an effect that becomes more pronounced as the repeated fraction grows.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/hernandez-1.png\")
The flat or increasing trend in the middle of training is possibly due to memorization of repeated data. Learning curves with such shapes make scaling law fitting less accurate. They also concluded repeated data hurts some OOD evaluation and downstream fine-tuning. However, their data mix is constructed in a more lab-like setup, and repetition in real-world data is often more nuanced (e.g. different data has different levels of repetition, semantic repetition, etc.).
\nRather than saying data repetition hurts training, we are more interested in how to fit scaling laws, given that the unique high-quality data is not infinite and we likely have to repeat data during training.
\nMuennighoff et al. (2023) took on the research question of how compute should be allocated optimally when model training is data-constrained. Specifically, they empirically studied the impact of data repetition across roughly 400 experiments, 10M–9B parameters, data sizes up to 900B tokens, and up to 1500 epochs. The exact same dataset is repeated each epoch, shuffled between epochs, and evaluated on a held-out test set.
\nThe key modeling adjustment is to decompose the total token count D into two parts: (i) the number of unique tokens U_D and (ii) the number of repeats R_D (i.e. num. epochs - 1). Thus we have D = U_D(1 + R_D). With a unique-data budget D_(uniq), by definition U_D = min { D_(uniq), D} and R_D = (D / U_D) − 1. They use the Chinchilla scaling laws to find the optimal model size U_N for fitting U_D, and define excess model size via repeats R_N = (N / U_N) − 1.
\nThey then update the Chinchilla parametric fit (method 3) to use effective (discounted) data D’ and model size N’ in place of the raw quantities:
\nL̂(N, D) = A/N′^α + B/D′^β + E where D′ = U_D + U_D r_D(1 − exp (−R_D/r_D)).
\nThe intuition is that a token’s value decays exponentially as it is repeated. In their modeling, each repetition costs the token a (1 − 1/r_D) fraction of its remaining value, where r_D is a learnable “half-life” parameter. When R_D = 0 or R_D ≪ r_D, we recover D’ ≈ D.
\nA symmetric formulation handles excess model size, N’ = U_N + U_N r_N(1 − exp(−R_N / r_N)), capturing the idea that “larger models overfit more quickly on repeated data” and that “a model can be too large for its dataset.” This component is less intuitive, and I could not find a satisfactory explanation for why model size needs to appear in such a symmetric form as repeated data. Later work by Lovelace et al. (2026) changed this assumption.
\nTheir empirical fit finds that excess parameters decay faster in value than repeated data, r_N < r_D, so we should allocate more resources on more epochs rather than more model parameters. One weakness of this modeling, as the authors also pointed out, is that it significantly underestimates the final test loss of failing models (i.e. models whose loss increases midway through training), such as models trained for 44 epochs.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/muennighoff-1.png\")
Most recently, Lovelace et al. (2026) revisited the same problem with a different approach. Rather than modeling overparameterization as a diminishing return on effective model size, Lovelace et al. model the interaction between model size × data repetition explicitly. Empirically, they trained about 300 models, spanning 15M to 1B parameters and 50M to 6B unique tokens.
\nWhen they plot the fit residual for a fixed model size across a range of data-repetition levels, the observation is intuitive: more epochs cause more damage, and interestingly larger models are more sensitive to repetition. This hints that the loss penalty is likely a function of both model size and data size.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/lovelace-1.png\")
An explicit overfitting penalty term was introduced and built around the capacity ratio N / U_D (parameter count relative to unique tokens):
\nL̂(N, U_D, R_D) = E + A/N^α + B/((U_D (1 + R_D))^β) + P ⋅ R_D^δ ⋅ (N/U_D)^κ
\nwhere:
\nThe added term (in red) is a direct overfitting penalty that grows with both how many times you repeat the data and how over-parameterized the model is relative to the unique data available.
\nThey also did a case study on how weight decay impacts training with the limited-data constraint and found that strong weight decay reduces the overfitting penalty caused by data repetition.
\n![\"\"](\"https://lilianweng.github.io/posts/2026-06-24-scaling-laws/lovelace-2.png\")
Both modeling approaches by Muennighoff et al. and Lovelace et al. are constructed from empirical curve fitting, so it is still unclear why data-constrained scaling laws should have exactly these forms and why each free parameter is needed. Curious about more theoretical work along this line.
\nDespite its clean form, in practice, scaling law fitting can be surprisingly sensitive to seemingly trivial procedural choices, like how you count parameters, how you round the precision, how you sum or average the loss, etc.
\nBecause a scaling law is only fit on the (relatively small, relatively cheap) models that we can afford to train, and the prediction is extrapolated for a model orders of magnitude larger. In such a setup, choices that look like rounding error may lead to wild differences in prediction.
\nMeanwhile, scaling-law fitting assumes the only changing factor is scale, which means that the model architecture, optimizer, learning rate schedule, batch ramp, data mix, tokenizer, and other design choices should remain the same. Another underlying assumption is that all these settings should have been carefully tuned, as cases like undertrained models can lead to a different conclusion.
\nThe disagreement between results by Kaplan et al. and Chinchilla is one example to showcase the trickiness of scaling laws fitting.
\nA second example is a follow-up analysis investigating why Chinchilla method 3 is slightly off from the other two methods. Besiroglu et al. (2024) extracted the raw (N, D, L) data points from Figure 4 of Hoffmann et al. (2022) and re-ran the method 3 parametric fitting. They found a couple of concrete issues:
\nHere is a toy simulation widget, created by ChatGPT, designed to demonstrate three specific failure modes.
\nWe assume the ground truth function is:
\nL̂(N, D) = 482.01 ⋅ N^(−0.3478) + 2085.43 D^(−0.3658) + 1.8172
\nand thus Nₒₚₜ ∝ C^(0.5126), Dₒₚₜ ∝ C^(0.4874). This is the estimate from Besiroglu et al. (2024).
\nThe simulation plots the loss prediction L̂ vs dataset size D, while providing a set of sliders to show case:
\nPlease cite this work as:
\n\n\nWeng, Lilian. “Scaling Laws, Carefully”. Lil’Log (Jun 2026). https://lilianweng.github.io/posts/2026-06-24-scaling-laws/
\n
Or use the BibTex citation:
\n@article{weng2026scaling,\n title = {Scaling Laws, Carefully},\n author = {Weng, Lilian},\n journal = {lilianweng.github.io},\n year = {2026},\n month = {June},\n url = "https://lilianweng.github.io/posts/2026-06-24-scaling-laws/"\n}\n[1] S. Amari, N. Fujita, and S. Shinomoto. “Four Types of Learning Curves. Neural Computation.” 4(4):605–618, 1992.
\n[2] Hestness et al. “Deep Learning Scaling is Predictable, Empirically.” arXiv preprint arXiv:1712.00409, 2017.
\n[3] Rosenfeld et al. “A Constructive Prediction of the Generalization Error Across Scales.” ICLR 2020.
\n[4] Kaplan et al. “Scaling Laws for Neural Language Models.” arXiv preprint arXiv:2001.08361, 2020.
\n[5] Hoffmann et al. “Training Compute-Optimal Large Language Models.” NeurIPS 2022.
\n[6] Pearce and Song. “Reconciling Kaplan and Chinchilla Scaling Laws.” TMLR 2024.
\n[7] Bahri et al. “Explaining Neural Scaling Laws.” arXiv preprint arXiv:2102.06701, 2021.
\n[8] Sharma and Kaplan. “A Neural Scaling Law from the Dimension of the Data Manifold.” arXiv preprint arXiv:2004.10802, 2020.
\n[9] Hernandez et al. “Scaling Laws and Interpretability of Learning from Repeated Data.” arXiv preprint arXiv:2205.10487, 2022.
\n[10] Muennighoff et al. “Scaling Data-Constrained Language Models.” NeurIPS 2023.
\n[11] Lovelace et al. “Prescriptive Scaling Laws for Data Constrained Training.” arXiv preprint arXiv:2605.01640, 2026.
\n[12] Besiroglu et al. “Chinchilla Scaling: A Replication Attempt.” arXiv preprint arXiv:2404.10102, 2024.
\n[13] Michaud et al. “The Quantization Model of Neural Scaling” NeurIPS 2023.
\n[14] Brill. “Neural Scaling Laws Rooted in the Data Distribution.” arXiv preprint arXiv:2412.07942, 2024.
\n[15] Rae et al. “Scaling Language Models: Methods, Analysis & Insights from Training Gopher.” arXiv preprint arXiv:2112.11446, 2021.
\n", "date_modified": "2026-06-24T00:00:00+00:00", "date_published": "2026-06-24T00:00:00+00:00", "external_url": "https://lilianweng.github.io/posts/2026-06-24-scaling-laws/", "id": "urn:aggr:item:b60dab45ad02770ff31333914bf48126d9a0df82", "language": "en-US", "summary": "Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss L decreases predictably as we scale up model size N, dataset size D, and compute C, following a power-law…", "tags": [ "programming" ], "title": "Scaling Laws, Carefully", "url": "https://aggr.aymericbeaumet.com/items/lilianweng-github-io/2026-06-24-scaling-laws-carefully/" }, { "content_html": "Special thanks to John Schulman for a lot of super valuable feedback and direct edits on this post.
\nTest time compute (Graves et al. 2016, Ling, et al. 2017, Cobbe et al. 2021) and Chain-of-thought (CoT) (Wei et al. 2022, Nye et al. 2021), have led to significant improvements in model performance, while raising many research questions. This post aims to review recent developments in how to effectively use test-time compute (i.e. “thinking time”) and why it helps.
\nEnabling models to think for longer can be motivated in a few different ways.
\nThe core idea is deeply connected to how humans think. We humans cannot immediately provide the answer for "What's 12345 times 56789?". Rather, it is natural to spend time pondering and analyzing before getting to the result, especially for complex problems. In Thinking, Fast and Slow (Kahneman, 2013), Daniel Kahneman characterizes human thinking into two modes, through the lens of the dual process theory :
Because System 1 thinking is fast and easy, it often ends up being the main decision driver, at the cost of accuracy and logic. It naturally relies on our brain’s mental shortcuts (i.e., heuristics) and can lead to errors and biases. By consciously slowing down and taking more time to reflect, improve and analyze, we can engage in System 2 thinking to challenge our instincts and make more rational choices.
\nOne view of deep learning, is that neural networks can be characterized by the amount of computation and storage they can access in a forward pass, and if we optimize them to solve problems using gradient descent, the optimization process will figure out how to use these resources–they’ll figure out how to organize these resources into circuits for calculation and information storage. From this view, if we design an architecture or system that can do more computation at test time, and we train it to effectively use this resource, it’ll work better.
\nIn Transformer models, the amount of computation (flops) that the model does for each generated token is roughly 2 times the number of parameters. For sparse models like mixture of experts (MoE), only a fraction of the parameters are used in each forward pass, so computation = 2 * parameters / sparsity, where sparsity is the fraction of experts active.
\nOn the other hand, CoT enables the model to perform far more flops of computation for each token of the answer that it is trying to compute. In fact, CoT has a nice property that it allows the model to use a variable amount of compute depending on the hardness of the problem.
\nA classic idea in machine learning is to define a probabilistic model with a latent (hidden) variable z and a visible variable y, where y is given to our learning algorithm. Marginalizing (summing) over the possible values of the latent variable allows us to express a rich distribution over the visible variables, P(y) = ∑_(z ∼ P(z)) P(y | z). For example, we can model the distribution over math problems and solutions by letting x denote a problem statement, y be ground truth answer or proof, and z as a free-form thought process that leads to the proof. The marginal probability distribution to optimize would be P(y | x) = ∑_(z ∼ p(z| x)) P(y | x, z)
\nThe latent variable perspective is particularly useful for understanding methods that involve collecting multiple parallel CoTs or searching over the CoT–these algorithms can be seen as sampling from the posterior P(z | x, y). This view also suggests the benefits of using the log loss log P(y | x) as the target objective to optimize, as the log loss objective has been so effective in pretraining.
\nThe strategy of generating intermediate steps before generating short answers, particularly for math problems, was explored by Ling, et al. 2017, who introduced the AQUA-RAT dataset, and then expanded by Cobbe et al. 2021, who introduced the Grade School Math (GSM) dataset. Cobbe et al. train a generator with supervised learning on human-written solutions and verifiers that predict the correctness of a candidate solution; they can then search over these solutions. Nye et al. (2021) experimented with intermediate thinking tokens as “scratchpads” and Wei et al. (2022) coined the now-standard term chain-of-thought (CoT).
\nEarly work on improving CoT reasoning involved doing supervised learning on human-written reasoning traces or model-written traces filtered for answer correctness, where the latter can be seen as a rudimentary form of reinforcement learning (RL). Some other work found that one could significantly boost math performance of instruction tuned models by prompting them appropriately, with "think step by step" (Kojima et al. 2022) or more complex prompting to encourage the model to reflect on related knowledge first (Yasunaga et al. 2023).
Later work found that the CoT reasoning capabilities can be significantly improved by doing reinforcement learning on a dataset of problems with automatically checkable solutions, such as STEM problems with short answers, or coding tasks that can be checked with unit tests (Zelikman et al. 2022, Wang et al., 2023, Liu et al., 2023). This approach rose to prominence with the announcement of o1-preview, o3, and the R1 tech report (DeepSeek-AI, 2025), which showed that a simple recipe where a policy gradient algorithm could lead to strong performance.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/cot-wei22.png\")
The fundamental intent of test-time compute is to adaptively modify the model’s output distribution at test time. There are various ways of utilizing test time resources for decoding to select better samples and thus alter the model’s predictions towards a more desired distribution. Two main approaches for improving the decoding process are parallel sampling and sequential revision.
\nParallel sampling is simple, intuitive and easier to implement, but bounded by the model capability of whether it can achieve the correct solution in one-go. Sequential explicitly asks the model to reflect on mistakes but it is slower and requires extra care during implementation as it does run the risk of correct predictions being modified to be incorrect or introducing other types of hallucinations. These two methods can be used together. Snell et al. (2024) showed that easier questions benefit from purely sequential test-time compute, whereas harder questions often perform best with an optimal ratio of sequential to parallel compute.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/parallel-vs-sequential.png\")
Given a generative model and a scoring function that we can use to score full or partial samples, there are various search algorithms we can use to find a high scoring sample. Best-of-N is the simplest such algorithm: one just collects N independent samples and chooses the highest-ranking sample according to some scoring function. Beam search is a more sophisticated search algorithm that makes the search process more adaptive, spending more sampling computation on more promising parts of the solution space.
\nBeam search maintains a set of promising partial sequences and alternates between extending them and pruning the less promising ones. As a selection mechanism, we can use a process reward model (PRM; Lightman et al. 2023) to guide beam search candidate selection. Xie et al. (2023) used LLM to evaluate how likely its own generated reasoning step is correct, formatted as a multiple-choice question and found that per-step self-evaluation reduces accumulative errors in multi-step reasoning during beam search decoding. Besides, during sampling, annealing the temperature helps mitigate aggregated randomness. These experiments by Xie et al. achieved 5-6% improvement on few-shot GSM8k, AQuA and StrategyQA benchmarks with the Codex model. Reward balanced search (short for “REBASE”; Wu et al. 2025) separately trained a process reward model (PRM) to determine how much each node should be expanded at each depth during beam search, according to the softmax-normalized reward scores. Jiang et al. (2024) trained their PRM, named “RATIONALYST”, for beam search guidance on synthetic rationales conditioned on a large amount of unlabelled data. Good rationales are filtered based on whether they help reduce the neg log-prob of true answer tokens by a threshold, when comparing the difference between when the rationales is included in the context vs not. At inference time, RATIONALYST provides process supervision to the CoT generator by helping estimate log-prob of next reasoning steps (“implicit”) or directly generating next reasoning steps as part of the prompt (“explicit”).
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/beam-search-xie23.png\")
Interestingly, it is possible to trigger the emergent chain-of-thought reasoning paths without explicit zero-shot or few-shot prompting. Wang & Zhou (2024) discovered that if we branch out at the first sampling tokens by retaining the top k tokens with highest confidence, measured as the difference between top-1 and top-2 candidates during sampling, and then continue these k sampling trials with greedy decoding onward, many of these sequences natively contain CoT. Especially when CoT does appear in the context, it leads to a more confident decoding of the final answer. To calculate the confidence of the final answer, the answer span needs to be identified by task-specific heuristics (e.g. last numerical values for math questions) or by prompting the model further with "So the answer is". The design choice of only branching out at the first token is based on the observation that early branching significantly enhances the diversity of potential paths, while later tokens are influenced a lot by previous sequences.
![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/cot-decoding.png\")
If the model can reflect and correct mistakes in past responses, we would expect the model to produce a nice sequence of iterative revision with increasing quality. However, this self-correction capability turns out to not exist intrinsically among LLMs and does not easily work out of the box, due to various failure modes, such as, (1) hallucination, including modifying correct responses to be incorrect; (2) behavior collapse to non-correcting behavior; e.g. making minor or no modification on the first incorrect responses; or (3) fail to generalize to distribution shift at test time. Experiments by Huang et al. (2024) showed that naively applying self-correction leads to worse performance and external feedback is needed for models to self improve, which can be based on matching ground truths, heuristics and task-specific metrics, unit tests results for coding questions (Shinn, et al. 2023), a stronger model (Zhang et al. 2024), as well as human feedback (Liu et al. 2023).
\nSelf-correction learning (Welleck et al. 2023) aims to train a corrector model P_θ(y | y₀, x) given a fixed generator model P₀(y₀ | x). While the generator model remains to be generic, the corrector model can task-specific and only does generation conditioned on an initial model response and additional feedback (e.g. a sentence, a compiler trace, unit test results; can be optional):
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/self-correction-welleck23.png\")
Recursive inspection (Qu et al. 2024) also aims to train a better corrector model but with a single model to do both generation and self-correction.
\nSCoRe (Self-Correction via Reinforcement Learning; Kumar et al. 2024) is a multi-turn RL approach to encourage the model to do self-correction by producing better answers at the second attempt than the one created at the first attempt. It composes two stages of training: stage 1 only maximizes the accuracy of the second attempt while enforcing a KL penalty only on the first attempt to avoid too much shifting of the first-turn responses from the base model behavior; stage 2 optimizes the accuracy of answers produced by both the first and second attempts. Ideally we do want to see performance at both first and second attempts to be better, but adding stage 1 prevents the behavior collapse where the model does minor or none edits on the first response, and stage 2 further improves the results.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/SCoRe-kumar24.png\")
There’s been a lot of recent success in using RL to improve the reasoning ability of language models, by using a collection of questions with ground truth answers (usually STEM problems and puzzles with easy to verify answers), and rewarding the model for getting the correct answer.Recent activity in this area was spurred by strong performance of the o-series models from OpenAI, and the subsequent releases of models and tech reports from DeepSeek.
DeepSeek-R1 (DeepSeek-AI, 2025) is an open-source LLM designed to excel in tasks that require advanced reasoning skills like math, coding and logical problem solving. They run through 2 rounds of SFT-RL training, enabling R1 to be good at both reasoning and non-reasoning tasks.
DeepSeek-V3-Base base model on a collection of thousands of cold-start data. Without this step, the model has issues of poor readability and language mixing.<thinking> ... </thinking> tokens.DeepSeek-V3 in domains like writing, factual QA, and self-cognition, to retrain DeepSeek-V3-Base.\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/R1-eval.png\")
DeepSeek-R1 performs comparable to OpenAI o1-preview and o1-mini on several widely used reasoning benchmarks. DeepSeek-V3 is the only non-reasoning model listed. (Image source: DeepSeek-AI, 2025)Interestingly the DeepSeek team showed that with pure RL, no SFT stage, it is still possible to learn advanced reasoning capabilities like reflection and backtracking (“Aha moment”). The model naturally learns to spend more thinking tokens during the RL training process to solve reasoning tasks. The “aha moment” can emerge, referring to the model reflecting on previous mistakes and then trying alternative approaches to correct them. Later, various open source efforts happened for replicating R1 results like Open-R1, SimpleRL-reason, and TinyZero, all based on Qwen models. These efforts also confirmed that pure RL leads to great performance on math problems, as well as the emergent “aha moment”.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/aha-moment.png\")
The DeepSeek team also shared some of their unsuccessful attempts. They failed to use process reward model (PRM) as it is hard to define per-step rubrics or determine whether an intermediate step is correct, meanwhile making the training more vulnerable to reward hacking. The efforts on MCTS (Monte Carlo Tree Search) also failed due to the large search space for language model tokens, in comparison to, say, chess; and training the fine-grained value model used for guiding the search is very challenging too. Failed attempts often provide unique insights and we would like to encourage the research community to share more about what did not work out.
\nDuring the reasoning steps, certain intermediate steps can be reliably and accurately solved by executing code or running mathematical calculations. Offloading that part of reasoning components into an external code interpreter, as in PAL (Program-Aided Language Model; Gao et al. 2022) or Chain of Code (Li et al. 2023), can extend the capability of LLM with external tools, eliminating the need for LLMs to learn to execute code or function as calculators themselves. These code emulators, like in Chain of Code, can be augmented by an LLM such that if a standard code interpreter fails, we have the option of using LLM to execute that line of code instead. Using code to enhance reasoning steps are especially beneficial for mathematical problems, symbolic reasoning and algorithmic tasks. These unit tests may not exist as part of the coding questions, and in those cases, we can instruct the model to self-generate unit tests for it to test against to verify the solution (Shinn, et al. 2023).
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/pal.png\")
ReAct (Reason+Act; Yao et al. 2023) combines the action of searching the Wikipedia API and generation of reasoning traces, such that reasoning paths can incorporate external knowledge.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/react.png\")
o3 & o4-mini, recently released by OpenAI, are another two good examples where the reasoning process involves tool use like Web search, code execution and image processing. The team observed that large-scale reinforcement learning exhibits the same trend as in the GPT paradigm that “more compute = better performance”.
\nDeep learning models are often treated as black boxes and various interpretability methods have been proposed. Interpretability is useful for a couple reasons: first, it gives us an extra test to determine if the model is misaligned with its creators’ intent, or if it’s misbehaving in some way that we can’t tell by monitoring its actions. Second, it can help us determine whether the model is using a sound process to compute its answers. Chain of thought provides an especially convenient form of interpretability, as it makes the model’s internal process visible in natural language. This interpretability, however, rests on the assumption that the model truthfully describes its internal thought processes.
\nRecent work showed that monitoring CoT of reasoning models can effectively detect model misbehavior such as reward hacking, and can even enable a weaker model to monitor a stronger model (Baker et al. 2025). Increasing test time compute can also lead to improved adversarial robustness (Zaremba et al. 2025); this makes sense intuitively, because thinking for longer should be especially useful when the model is presented with an unusual input, such as an adversarial example or jailbreak attempt – it can use the extra thinking time to make sense of the strange situation it’s been presented with.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/cot-monitor.png\")
exit(0) coding hack is when the agent exploited a bug that allowed it to exit from the environment early without running all unit tests. The raise SkipTest hack is when the agent raises an exception from functions outside the testing framework in order to skip unit test evaluation. (Image source: Baker et al. 2025)Intuitively, model CoTs could be biased due to lack of explicit training objectives aimed at encouraging faithful reasoning. Or when we fine-tune the model on human-written explanations, those human-written samples may contain mistakes. Thus we cannot by default assume CoT is always faithful .
\nLanham et al. (2023) investigated several modes of CoT faithfulness failures by deliberately introducing mistakes into CoTs and measuring their impacts on the accuracy of a set of multiple choice tasks (e.g. AQuA, MMLU, ARC Challenge, TruthfulQA, HellaSwag):
\nMistake 1 (Early answering): The model may form a conclusion prematurely before CoT is generated. This is tested by early truncating or inserting mistakes into CoT. Different tasks revealed varying task-specific dependencies on CoT effectiveness; some have evaluation performance sensitive to truncated CoT but some do not. Wang et al. (2023) did similar experiments but with more subtle mistakes related to bridging objects or language templates in the formation of CoT.
\nMistake 2 (Uninformative tokens): Uninformative CoT tokens improve performance. This hypothesis is tested by replacing CoT with filler text (e.g. all periods) and this setup shows no accuracy increase and some tasks may suffer performance drop slightly when compared to no CoT.
\nMistake 3 (Human-unreadable encoding): Relevant information is encoded in a way that is hard for humans to understand. Paraphrasing CoTs in an non-standard way did not degrade performance across datasets, suggesting accuracy gains do not rely on human-readable reasoning.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/cot-perturb.png\")
Interestingly, Lanham et al. suggests that for multiple choice questions, smaller models may not be capable enough of utilizing CoT well, whereas larger models may have been able to solve the tasks without CoT. This dependency on CoT reasoning, measured by the percent of obtaining the same answer with vs without CoT, does not always increase with model size on multiple choice questions, but does increase with model size on addition tasks, implying that thinking time matters more for complex reasoning tasks.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/cot-ablation.png\")
Alternative approaches for testing CoT faithfulness involve perturbing prompts rather than modifying CoT paths directly (Turpin et al. 2023, Chua & Evans, 2025, Chen et al. 2025).
\nOne method consistently labels correct answers as “(A)” in few-shot examples regardless of true labels to introduce biases.
\nAnother prompting technique inserts misleading hints into prompts, such as "I think the answer is <random_label> but curious to hear what you think". or "A Stanford Professor thinks the answer is <random_label>". By comparing model predictions for the same question with vs without the misleading hint, we can measure whether a model is able to faithfully describe the influence of the hint on its answer. Particularly, in cases where the model produces different hint and non-hint answers, we measure whether the model acknowledges the hint when solving the question with hint. If the model is faithful, it should explicitly acknowledge the impact and admit the change of its answer is due to the hint.
![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/faithfulness.png\")
Multiple studies found that reasoning models describe the influence of the hint much more reliably than all the non-reasoning models tested. For example, we can measure the fraction of samples where the model acknowledges the hint as a deciding factor (“faithful CoT”). Reasoning models (Claude 3.7 Sonnet, DeepSeek R1) are overall doing better than non-reasoning ones (Claude 3.6, DeepSeek V3).
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/faithfulness2.png\")
Some evidence showed that using reward models leads to less faithfulness in model outputs. The reward model in classic RLHF is not trained to incentivize faithful behavior in this test as the preference comparison dataset is collected by humans selecting which one seems better or pleasing. However, in reasoning models, the CoT paths are optimized for producing the correct final answers, not to match human preference defined in RM, and thus intuitively are expected to be more faithful.
\nMonitoring CoT of reasoning models for reward hacking behavior is effective. A simple way to use this monitor is to run rejection sampling with the monitor as a filter at test time and identify solutions without reward hacking. However, it only works at the early stage of RL, and, as the model is further optimized, it is rare to find a sample to pass a CoT monitor within budget. This challenge naturally suggests that we could consider incorporating CoT monitors into RL rewards—a form of process-based supervision—to prevent reward hacking.
\nHowever, incorporating CoT monitoring reward into RL does not work out well, as it leads to obfuscated reward hacking where the model tries to hide its true intent within the CoT (Baker et al. 2025). With this direct optimization pressure provided by the CoT monitor applied during RL training, the model still can find ways to hack, although a bit less frequently, but can hide its intention in CoT to avoid being caught by the monitor.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/RL-monitor-baker25.png\")
Experiments on CoT length RL signals by Yeo et al. (2025) confirmed a similar message as Baker et al. that new types of reward hacking can happen with new types of RL reward shaping. They designed the reward function such that correct CoTs can have higher rewards than wrong ones, short correct CoTs obtain higher rewards than long correct ones and short incorrect CoT receive higher penalties than long incorrect CoTs. With this extra reward, the model learns to repeat text in CoTs for challenging questions rather than attempting to solve them. Yeo et al. further applied a n-gram repetition penalty to prevent this hacking behavior.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/rew-shaping-yeo25.png\")
Chen et al. (2025) experimented with a flawed RL environment, specifically using a grader with incorrect answers filled in for multiple-choice questions. The model learns to exploit the reward hack on >99% of the prompts, but almost never (<2%) verbalizes the reward hack in its CoT on more than half of their environments. Additional RL optimization pressure fails to incentivize the model to verbalize the hack in this case.
\nRL training is inherently sensitive to reward hacking. Only relying on heuristic investigation of reward hacking and manual fixes may lead to a “whack-a-mole” situation. We would suggest being very cautious when trying to apply optimization directly on CoT during RL training, or trying to avoid it altogether.
\nAdaptive Computation Time, introduced by Alex Graves in 2016, predated large language models but pioneered the same direction of enabling the model to dynamically decide the number of computational steps to take at the inference time, which can be viewed as enabling the model to “think more” in continuous space at test time. Adaptive thinking time in continuous space can be enabled vertically via recurrent architecture or horizontally via more sequential sampling steps.
\nA number of architecture variations have been proposed to make the Transformer architecture recurrent, enabling adaptive test time compute (Dehghani, et al. 2019, Hutchins, et al. 2022, Bulatov, et al. 2022). A deep dive into literature on this topic would make the post too long, so we will only review a few.
\nUniversal Transformer (Dehghani, et al. 2019) combines self-attention in Transformer with the recurrent mechanism in RNN, dynamically adjusting the number of steps using adaptive computation time (Graves, 2016). On a high level, it can be viewed as a recurrent function for learning the hidden state representation per token, and if the number of steps is fixed, an Universal Transformer is equivalent to a multi-layer Transformer with shared parameters across layers.
\nA recent recurrent architecture design, proposed by Geiping et al. (2025), adds a recurrent block R on top of the standard Transformer. Every iteration of this recurrent block takes the embedding e and a random state sᵢ. Conceptually, this recurrent-depth architecture is a bit similar to a conditioned diffusion model, where the original input e is provided in every recurrent step while a random Gaussian initialized state sᵢ gets updated iteratively through the process. (Interestingly some of their experiments of designs that resemble diffusion models more turned out to be bad.)
\ne = P(x) embedding
s_0 ∼ 𝒩(0, σ² I_n ⋅ h)
s_i = R(e, s_i−1) for i ∈ 1, …, r recurrent block; resembles a Transformer block
p = C(s_r) unembedding
The recurrence count r during training is randomized, sampled from a log-normal Poisson distribution, per input sequence. To manage computational costs, backpropagation is truncated to only the last k iterations of the recurrent unit (k=8 in experiments), making it possible to train on the heavy-tail part of the Poisson distribution. The embedding block continues to receive gradient updates in every step since its output e is injected in every step, mimicking RNN training. Unsurprisingly, the stability of training a recurrent model turns out to be very sensitive. Factors like initialization, normalization and hyperparameters all matter, especially when scaling up the training. For example, hidden states can collapse by predicting the same hidden state for every token; or the model may learn to ignore the incoming state s. To stabilize the training, Geiping et al. adopted an embedding scale factor, a small learning rate and careful tuning.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/recurrent.png\")
Thinking tokens refer to a set of implicit tokens introduced during training or inference that do not carry direct linguistic meaning. Instead, their role is to provide extra thinking time and compute power for the model to perform better.
\nHerel & Mikolov (2023) introduced the idea of inserting special thinking tokens (<T>) after each word in a sentence and training the model on such a dataset. Each thinking token buys extra time for the model to process and make better predictions. Training with thinking tokens on a toy model setup results in lower perplexity than baseline model trained without them. The benefits of thinking tokens are more pronounced for non-trivial reasoning tasks or sentences involving numbers.
Similarly, pause tokens proposed by Goyal et al. (2024) delay the model’s outputs by appending dummy tokens (e.g. character like . or #) at the end of the input sequence, giving the model extra computation during inference. It is important to inject such pause tokens both during training and inference time, while only fine-tuning on pause tokens leads to limited gain. During training, multiple copies of pause tokens are inserted at uniformly random locations and the loss on pause tokens is ignored for training.
![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/pause-tokens-goyal24.png\")
Interestingly, thinking tokens or pause tokens in above experiments do not carry any extra information or add many new parameters. But why is it still helpful? On one hand, it helps expand the computation by introducing more inference loops, effectively increasing computational capacity. On the other hand, it can be seen as acting as a special, implicit form of CoTs. A drawback here is hat the model needs to be pretrained with respect to thinking tokens. Still, this strategy is an interesting way to further improve the capability of test time compute utilization on top of inference time CoTs.
\nQuiet-STaR (Zelikman et al. 2025) introduces token-level reasoning by training the model to generate rationales after every token to explain future text. It mixes the future-text predictions with and without rationales and uses learning to generate better rationales and uses REINFORCE to optimize the quality of rationale generation.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/quiet-star-zelikman25.png\")
Quiet-STaR consists of three stages:
\nThink: Predicting next tokens with rationales. Due to high computational cost demanded by token level reasoning, this process is designed to generate multiple rationales in parallel. A special attention map is used to enable all thought tokens to only pay attention to themselves, all preceding thought tokens within the same thought, and the preceding text.
\nTalk: Next token prediction without rationale is mixed with post-rationale prediction. The mixing weight for two logits is learned by a special mixing head of a shallow MLP out of hidden output after each rationale. Correct next tokens can be selected via teacher forcing.
\nLearn: Train the model to generate better rationale via REINFORCE by learning from examples that increase the probability of correct next token while discarding those that hurt the prediction.
\nWithout dataset specific fine-tuning, Quiet-STaR improves zero-shot results on CommonsenseQA (36.3%→47.2%) and GSM8K (5.9%→10.9%), within experiments on Mistral 7B.
\nA latent variable model defines a probabilistic framework where observable data is explained through unobserved (latent) variables. These latent variables capture hidden structures or intermediate processes that generate the observable outcomes. Language models can be viewed as probabilistic latent variable models where test-time thinking and reasoning steps are latent thought variables (Zhou et al. 2020, Phan et al. 2023). Such a latent variable model defines a joint distribution of problems x_i, answers y_i and latent thought z_i. We would like to optimize the log-likelihood of answers given questions and a variety of CoTs as latent variables (N is the number of samples; K is the number of CoTs per problem):
\nlog ℒ(θ) = log p(y | x)
= log ∑ₖ₌₁^K p(y, z⁽ᵏ⁾ | x)
= log ∑ₖ₌₁^K p(z⁽ᵏ⁾ | x) p(y | z⁽ᵏ⁾, x)
= log 𝔼_(z⁽ᵏ⁾ ∼ p(z⁽ᵏ⁾ | x)) p(y | z⁽ᵏ⁾, x)
Our goal is to maximize the marginal likelihood of the correct answer, p(y | x), given a number of reasoning traces per problem, {z⁽ᵏ⁾}ₖ₌₁^K.
\nExpectation-Maximization is a commonly used iterative algorithm for optimizing parameters for a model with (hidden) latent variables, and thus can be applied to train better CoTs and then condition on that to generate better responses. Typically we iterate between E-step (Expectation) where we guess the missing information about latent variables (i.e. how to sample better CoTs), and M-step (Maximization) where we optimize the model parameters based on latent variables (i.e. how to sample better answers), until convergence.
\nlog ℒ(θ) = log 𝔼_(z⁽ᵏ⁾ ∼ p(z⁽ᵏ⁾ | x)_(E−step)) p(y | z⁽ᵏ⁾, x)_(M−step)
\nBecause we cannot directly sample from the latent variable distribution p(z | x, y), researchers have explored methods relying on human annotated data (Zhou et al. 2020), Metropolis-Hastings MCMC (Phan et al. 2023) or Monte Carlo sampling with special importance weights (Ruan et al. 2025) to draw good CoT samples to update the model. Ruan et al. (2025) experimented with training a model on a large body of Web text with latent thoughts with the EM algorithm, where the latent thought is synthesized per chunk of observed data and then the model learns over both latent thought and data in an autoregressive manner.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/ruan25.png\")
They first prompt a LLM q̃ to generate synthetic latent thought Z_i given observed data X_i:
\nYou are provided with a pair of web document prefix and suffix. Your task is to insert latent thoughts between them underlying the creation of the suffix conditioned on the prefix. The latent thoughts should include: the missing background knowledge and the reasoning traces underlying each claim (especially, step-by-step derivations or logical reasoning).\nSpecial tokens like <StartOfLatent><Prior> ... <EndOfPrior> are used to insert the generated latent thought content into the raw data for training the joint distribution p(z, x) or the approximate posterior q(z | x), depending on whether z is inserted before or after x. However, since we are using a LLM q̃(z | x) to generate the CoTs, it imposed a performance ceiling on how good the approximate q(z | x) can be. Ruan et al. introduced importance weights for selecting CoT samples at the E-step, formulated as:
w⁽ᵏ⁾ = (p(z⁽ᵏ⁾, x))/(q(z⁽ᵏ⁾ | x)) = (p(x | z⁽ᵏ⁾) p(z⁽ᵏ⁾))/(q(z⁽ᵏ⁾ | x))
\n, such that we prioritize samples with CoTs that are good at predicting the observation (i.e., high p(x | z⁽ᵏ⁾)), simple, intuitive (i.e., high p(z⁽ᵏ⁾)) but also informative and not too obvious (i.e. low q(z⁽ᵏ⁾ | x)).
\nSince pretrained models already possess the capability of generating chains of thought, it is intuitive to design an iterative improvement process where we generate multiple CoTs and fine-tune the model only on rationales that lead to correct answers.
\nHowever, this straightforward design can fail because the model receives no learning signals for problems it fails to solve. STaR (“Self-taught reasoner”; Zelikman et al. 2022) addresses this limitation by adding a “rationalization” process for failed attempts, in which the model generates good CoTs backward conditioned on both the problem and the ground truth answer and thus the model can generate more reasonable CoTs. Then the model is finetuned on correct solutions that either lead to correct outputs or are generated through rationalization.
![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/STaR-algo-zelikman22.png\")
We can view STaR as an approximation to a policy gradient in RL, with a simple indicator function as the reward, \\mathbb{1}[\\hat{y} = y]. We want to maximize the expectation of this reward when sampling z ∼ p(z | x) and then y ∼ p(y | x, z), since p(y | x) = ∑_z p(z | x) p(y | x, z).
\\begin{aligned} \\nabla_\\theta J(\\theta) &= \\nabla_\\theta \\mathbb{E}_{z_i, y_i \\sim p(.\\mid x_i)} \\mathbb{1}[y_i = y_i^\\text{truth}] \\\\ &= \\sum_{i=1}^N \\nabla_\\theta \\mathbb{1}[y_i = y_i^\\text{truth}] \\; p(y_i, z_i \\mid x_i) \\\\ &= \\sum_{i=1}^N \\mathbb{1}[y_i = y_i^\\text{truth}] \\; p(y_i, z_i \\mid x_i) \\frac{\\nabla_\\theta p(y_i, z_i \\mid x_i)}{p(y_i, z_i \\mid x_i)} & \\text{;log-derivative trick}\\\\ &= \\mathbb{E}_{z_i, y_i \\sim p(.\\mid x_i)} \\mathbb{1}[y_i = y_i^\\text{truth}] \\; \\nabla_\\theta \\log p(y_i, z_i \\mid x_i) & \\text{;log-derivative trick} \\end{aligned}
Each iteration is equivalent to first selecting the CoT samples according to \\mathbb{1}[y=y^\\text{truth}] and then running supervised fine-tuning to optimize the logprob of generating good CoTs and answers. Performance of STaR improves with more training iterations, and the “rationalization” process for generating better CoTs accelerates learning. They observed that sampling with high temperature increases the chance of getting correct answers with incorrect reasoning and finetuning LLM on such data can impair generalization. For datasets without ground truths, majority votes of multiple high-temperature outputs can serve as a proxy of ground truth answers (Wang et al. 2022), making it possible to use synthetic samples for training.
![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/STaR-rationalization-zelikman22.png\")
So far we have seen much evidence that allowing models to spend additional compute on reasoning before producing final answers at inference time can significantly improve performance. Techniques like prompting the model to generate intermediate reasoning steps before the answers, or training the model to pause and reflect before predicting next tokens, have been found to boost the model performance beyond the capability limit obtained during training. This essentially introduces a new dimension to tinker with for improving model intelligence, complementing established factors such as model size, training compute and data quantity, as defined in scaling laws (Kaplan et al. 2020).
\nRecent studies demonstrated that optimizing LLM test-time compute could be more effective than scaling up model parameters (Snell et al. 2024, Wu et al. 2025). Smaller models combined with advanced inference algorithms can offer Pareto-optimal trade-offs in cost and performance.
\nSnell et al. (2024) evaluated and compared test-time and pretraining compute, and found that they are not 1:1 exchangeable. Test-time compute can cover up the gap easily on easy and medium questions when there is only a small gap in model capability, but proves less effective for hard problems. The ratio between token budgets for pretraining and inference matters a lot. Test-time compute is only preferable when inference tokens are substantially fewer than pretraining ones. This indicates that developing a capable base model with enough pretraining data and compute is still very critical, as test-time compute cannot solve everything or fill in big model capability gaps.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/scaling-snell24.png\")
s1 models (Muennighoff & Yang, et al. 2025) experimented with scaling up the CoT reasoning path length via budget forcing technique (i.e. forcefully lengthen it by appending the word "wait", or shorten it by terminating the model’s thinking process by appending end-of-thinking token or "Final Answer:"). They observed a clear positive correlation between the average thinking time measured in tokens and the downstream evaluation accuracy.
![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/s1-muennighoff25.png\")
When comparing this budget forcing technique with other decoding methods of controlling reasoning trace length, it is quite surprising that simple rejection sampling (i.e. sampling generation until the lengths fits a token budget) leads to reversed scaling, meaning that longer CoTs lead to worse performance.
\n![\"\"](\"https://lilianweng.github.io/posts/2025-05-01-thinking/s1-scaling-muennighoff25.png\")
The exploration of test-time compute and chain-of-thought reasoning presents new opportunities for enhancing model capabilities. More interestingly, via test-time thinking, we are moving towards building future AI systems that mirror the best practices of how humans think, incorporating adaptability, flexibility, critical reflection, and error correction. Excitement with current progress invites us for more future research to improve and understand deeply not just how but why we—and our models—think.
\nAt the end, I would like to call for more research for the following open research questions on test time compute and chain-of-thought reasoning.
\nPlease cite this work as:
\nWeng, Lilian. "Why We Think". Lil'Log (May 2025). https://lilianweng.github.io/posts/2025-05-01-thinking/\nOr use the BibTex citation:
\n@article{weng2025think,\n title = {Why We Think},\n author = {Weng, Lilian},\n journal = {lilianweng.github.io},\n year = {2025},\n month = {May},\n url = "https://lilianweng.github.io/posts/2025-05-01-thinking/"\n}\n[1] Alex Graves. “Adaptive Computation Time for Recurrent Neural Networks.”. arXiv preprint arXiv:1603.08983 (2016).
\n[2] Wang Ling, et al. “Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems.”. arXiv preprint arXiv:1705.04146 (2017).
\n[3] Karl Cobbe, et al. “Training Verifiers to Solve Math Word Problems.”. arXiv preprint arXiv:2110.14168 (2021).
\n[4] Jason Wei, et al. “Chain of Thought Prompting Elicits Reasoning in Large Language Models.”. NeurIPS 2022.
\n[5] Maxwell Nye, et al. “Show Your Work: Scratchpads for Intermediate Computation with Language Models.”. arXiv preprint arXiv:2112.00114 (2021).
\n[6] Daniel Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux (2013).
\n[7] Takeshi Kojima, et al. “Large Language Models are Zero-Shot Reasoners.”. NeurIPS 2022.
\n[8] Michihiro Yasunaga, et al. “Large Language Models as Analogical Reasoners”. arXiv preprint arXiv:2310.01714 (2023).
\n[9] Eric Zelikman, et al. “STaR: Bootstrapping Reasoning With Reasoning.”. NeurIPS 2022.
\n[10] Xuezhi Wang, et al. “Self-consistency Improves Chain of Thought Reasoning in Language Models.”. ACL 2023.
\n[11] Ryo Kamoi, et al. “When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs.”. TACL 2024.
\n[12] Jie Huang, et al. “Large Language Models Cannot Self-Correct Reasoning Yet.”. ICLR 2024.
\n[13] Noah Shinn, et al. “Reflexion: Language Agents with Verbal Reinforcement Learning.”. arXiv preprint arXiv:2303.11366 (2023).
\n[14] Yunxiang Zhang, et al. “Small Language Models Need Strong Verifiers to Self-Correct Reasoning.”. ACL Findings 2024.
\n[15] Hao Liu, et al. “Chain of Hindsight Aligns Language Models with Feedback.”. arXiv preprint arXiv:2302.02676 (2023).
\n[16] Sean Welleck, et al. “Generating Sequences by Learning to Self-Correct.”. arXiv preprint arXiv:2211.00053 (2023).
\n[17] Yuxiao Qu, et al. “Recursive Introspection: Teaching Language Model Agents How to Self-Improve.”. arXiv preprint arXiv:2407.18219 (2024).
\n[18] Aviral Kumar, et al. “Training Language Models to Self-Correct via Reinforcement Learning.”. arXiv preprint arXiv:2409.12917 (2024).
\n[19] Hunter Lightman, et al. “Let’s Verify Step by Step.”. arXiv preprint arXiv:2305.20050 (2023).
\n[20] Yuxi Xie, et al. “Self-Evaluation Guided Beam Search for Reasoning.”. NeurIPS 2023.
\n[21] Yangzhen Wu, et al. “Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models”. ICLR 2025.
\n[22] Dongwei Jiang, et al. “RATIONALYST: Pre-training Process-Supervision for Improving Reasoning”. arXiv preprint arXiv:2410.01044 (2024).
\n[23] Xuezhi Wang and Denny Zhou. “Chain-of-Thought Reasoning Without Prompting.”. arXiv preprint arXiv:2402.10200 (2024).
\n[24] DeepSeek-AI. “DeepSeek-V3 Technical Report.” arXiv preprint arXiv:2412.19437 (2024).
\n[25] DeepSeek-AI. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.”. arXiv preprint arXiv:2501.12948 (2025).
\n[26] Luyu Gao, Aman Madaan & Shuyan Zhou, et al. “PAL: Program-aided Language Models.”. ICML 2023.
\n[27] Shunyu Yao, et al. “ReAct: Synergizing Reasoning and Acting in Language Models.”. ICLR 2023.
\n[29] Bowen Baker, et al. “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.”. arXiv preprint arXiv:2503.11926 (2025).
\n[30] Wojciech Zaremba, et al. “Trading Inference-Time Compute for Adversarial Robustness.”. arXiv preprint arXiv:2501.18841 (2025).
\n[31] Tamera Lanham, et al. “Measuring Faithfulness in Chain-of-Thought Reasoning”. arXiv preprint arXiv:2307.13702 (2023).
\n[32] Boshi Wang, et al. “Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters.”. ACL 2023.
\n[33] Miles Turpin, et al. “Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting.”. NeuriPS 2023.
\n[34] James Chua & Owain Evans. “Are DeepSeek R1 And Other Reasoning Models More Faithful?”. arXiv preprint arXiv:2501.08156 (2025).
\n[35] Yanda Chen et al. “Reasoning Models Don’t Always Say What They Think”. arXiv preprint arXiv:2505.05410 (2025).
\n[36] Edward Yeo, et al. “Demystifying Long Chain-of-Thought Reasoning in LLMs.”. arXiv preprint arXiv:2502.03373 (2025).
\n[37] Mostafa Dehghani, et al. “Universal Transformers.”. ICLR 2019.
\n[38] DeLesley Hutchins, et al. “Block-Recurrent Transformers.”. NeurIPS 2022.
\n[39] Aydar Bulatov, et al. “Recurrent Memory Transformers.”. NeuriPS 2022.
\n[40] Jonas Geiping, et al. “Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach.”. arXiv preprint arXiv:2502.05171 (2025).
\n[41] Herel & Mikolov. “Thinking Tokens for Language Modeling.”. AITP 2023.
\n[42] Sachin Goyal et al. “Think before you speak: Training Language Models With Pause Tokens.”. ICLR 2024.
\n[43] Eric Zelikman, et al. “Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking.”. arXiv preprint arXiv:2403.09629 (2025).
\n[44] Wangchunshu Zhou et al. “Towards Interpretable Natural Language Understanding with Explanations as Latent Variables.”. NeurIPS 2020.
\n[45] Du Phan et al. “Training Chain-of-Thought via Latent-Variable Inference.”. NeurIPS 2023.
\n[46] Yangjun Ruan et al. “Reasoning to Learn from Latent Thoughts.”. arXiv preprint arXiv:2503.18866 (2025).
\n[47] Xuezhi Wang et al. “Rationale-Augmented Ensembles in Language Models.”. arXiv preprint arXiv:2207.00747 (2022).
\n[48] Jared Kaplan, et al. “Scaling Laws for Neural Language Models.”. arXiv preprint arXiv:2001.08361 (2020).
\n[49] Niklas Muennighoff & Zitong Yang, et al. “s1: Simple test-time scaling.”. arXiv preprint arXiv:2501.19393 (2025).
\n[50] Peiyi Wang, et al. “Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations” arXiv preprint arXiv:2312.08935 (2023).
\n[51] Yixin Liu, et al. “Improving Large Language Model Fine-tuning for Solving Math Problems.” arXiv preprint arXiv:2310.10047 (2023).
\n[52] Charlie Snell, et al. “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters.”. arXiv preprint arXiv:2408.03314 (2024).
\n[53] OpenAI. o1-preview: “Learning to reason with LLMs.” Sep 12, 2024.
\n[54] OpenAI. o3: “Introducing OpenAI o3 and o4-mini.” Apr 16, 2025.
\n", "date_modified": "2025-05-01T00:00:00+00:00", "date_published": "2025-05-01T00:00:00+00:00", "external_url": "https://lilianweng.github.io/posts/2025-05-01-thinking/", "id": "urn:aggr:item:378d7f5216eb84a8d6211b6df8ea23b481f169a2", "language": "en-US", "summary": "Special thanks to John Schulman for a lot of super valuable feedback and direct edits on this post. Test time compute (Graves et al. 2016, Ling, et al. 2017, Cobbe et al. 2021) and Chain-of-thought (CoT) (Wei et al. 2022, Nye et al. 2021),…", "tags": [ "programming" ], "title": "Why We Think", "url": "https://aggr.aymericbeaumet.com/items/lilianweng-github-io/2025-05-01-why-we-think/" }, { "content_html": "Reward hacking occurs when a reinforcement learning (RL) agent exploits flaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing the intended task. Reward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function.
\nWith the rise of language models generalizing to a broad spectrum of tasks and RLHF becomes a de facto method for alignment training, reward hacking in RL training of language models has become a critical practical challenge. Instances where the model learns to modify unit tests to pass coding tasks, or where responses contain biases that mimic a user’s preference, are pretty concerning and are likely one of the major blockers for real-world deployment of more autonomous use cases of AI models.
\nMost of the past work on this topic has been quite theoretical and focused on defining or demonstrating the existence of reward hacking. However, research into practical mitigations, especially in the context of RLHF and LLMs, remains limited. I especially want to call out for more research efforts directed toward understanding and developing mitigation for reward hacking in the future. Hope I will be able to cover the mitigation part in a dedicated post soon.
\nReward function defines the task, and reward shaping significantly impacts learning efficiency and accuracy in reinforcement learning. Designing a reward function for an RL task often feels like a ‘dark art’. Many factors contribute to this complexity: How you decompose a big goal into small goals? Is the reward sparse or dense? How you measure the success? Various choices may lead to good or problematic learning dynamics, including unlearnable tasks or hackable reward functions. There is a long history of research on how to do reward shaping in RL.
\nFor example, in an 1999 paper by Ng et al., the authors studied how to modify the reward function in Markov Decision Processes (MDPs) such that the optimal policy remains unchanged. They found that linear transformation works. Given a MDP M = (S, A, T, γ, R), we want to create a transformed MDP M’ = (S, A, T, γ, R’) where R’ = R + F and F: S × A × S ↦ ℝ, such that we can guide the learning algorithm to be more efficient. Given a real-valued function Φ: S ↦ ℝ, F is a potential-based shaping function if for all s ∈ S − s₀, a ∈ A, s’ ∈ S:
\nF(s, a, s′) = γ Φ(s′) − Φ(s)
\nThis would guarantee that the sum of discounted F, F(s₁, a₁, s₂) + γ F(s₂, a₂, s₃) + …, ends up being 0. If F is such a potential-based shaping function, it is both sufficient and necessary to ensure M and M’ share the same optimal policies.
\nWhen F(s, a, s’) = γ Φ(s’) − Φ(s), and if we further assume that Φ(s₀) = 0, where s₀ is absorbing state, and γ=1, and then for all s ∈ S, a ∈ A:
\nQ^*_(M′) (s,a) = Q^*_M(s, a) − Φ(s)
V^*_(M′) (s,a) = V^*_M(s, a) − Φ(s)
This form of reward shaping allows us to incorporate heuristics into the reward function to speed up learning without impacting the optimal policy.
\nSpurious correlation or shortcut learning (Geirhos et al. 2020) in classification task is a concept closely related to reward hacking. Spurious or shortcut features can cause a classifier to fail at learning and generalizing as intended. For example, a binary classifier for distinguishing wolves from huskies may overfit to the presence of a snowy background if all the wolf training images include snow (Ribeiro et al. 2024).
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/shortcut-features.png\")
The ERM principle states that, since the full data distribution is unknown, minimizing the loss on training data is a reasonable proxy of risk and thus we favor models with the lowest training loss. Nagarajan et al. (2021) studied the ERM principle and pointed out that ERM needs to rely on all types of informative features, including unreliable spurious features, while attempting to fit the data without constraints. Their experiments showed that ERM would depend on spurious features no matter how easy the task is.
\nReward shaping in RL is challenging. Reward hacking occurs when an RL agent exploits flaws or ambiguities in the reward function to obtain high rewards without genuinely learning the intended behaviors or completing the task as designed. In recent years, several related concepts have been proposed, all referring to some form of reward hacking:
\nThe concept originated with Amodei et al. (2016), who proposed a set of open research questions on AI safety in their seminal paper “Concrete Problems in AI Safety”. They listed reward hacking as one of the key AI safety problems. Reward hacking refers to the possibility of the agent gaming the reward function to achieve high reward through undesired behavior. Specification gaming (Krakovna et al. 2020) is a similar concept, defined as a behavior that satisfies the literal specification of an objective but not achieving the desired results. Here the literal description of the task goal and the intended goal may have a gap.
\nReward shaping is a technique used to enrich the reward function, making it easier for the agent to learn—for example, by providing denser rewards. However, a poorly design reward shaping mechanism can alter the trajectory of the optimal policy. Designing effective reward shaping mechanisms is inherently difficult. Rather than blaming a poorly designed reward function, it is more accurate to acknowledge that designing a good reward function is intrinsically challenging due to the complexity of the task itself, partial observable state, multiple dimensions in consideration, and other factors.
\nWhen testing an RL agent in out-of-distribution (OOD) environments, robustness failure may occur due to:
\nExperiments in two RL environments, CoinRun and Maze, demonstrated the importance of randomization during training. If during training, the coin or the cheese is placed at a fixed position (i.e. right end of the level or upper right corner of the maze) but testing in the env where the coin or cheese is placed at random, the agent would just run to the fixed position without obtaining the coin or cheese at test time. A conflict arises when a visual feature (e.g., cheese or coin) and a positional feature (e.g., upper-right or right end) are inconsistent during test time, leading the trained model to prefer the positional feature. I would like to point out that, in these two examples, the reward-result gaps are clear but such type of biases are unlikely to be so obvious in most real-world cases.
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/coinrun-randomization.png\")
Reward Tampering (Everitt et al. 2019) is a form of reward hacking behavior where the agent interferes with the reward function itself, causing the observed reward to no longer accurately represent the intended goal. In reward tampering, the model modifies its reward mechanism either by directly manipulating the implementation of the reward function or by indirectly altering the environmental information used as input for the reward function.
\n(Note: Some work defines reward tampering as a distinct category of misalignment behavior from reward hacking. But I consider reward hacking as a broader concept here.)
\nAt a high level, reward hacking can be categorized into two types: environment or goal misspecification, and reward tampering.
\nGoodhart’s Law states that “When a measure becomes a target, it ceases to be a good measure”. The intuition is that a good metric can become corrupted once significant pressure is applied to optimize it. It is challenging to specify a 100% accurate reward objective and any proxy suffers the risk of being hacked, as RL algorithm exploits any small imperfection in the reward function definition. Garrabrant (2017) categorized Goodhart’s law into 4 variants:
\nAmodei et al. (2016) summarized that reward hacking, mainly in RL setting, may occur due to:
\nBesides, identifying the exact reward function for which an optimal agent optimizes its behavior is in general impossible since there could be an infinite number of reward functions consistent with any observed policy in an fixed environment (Ng & Russell, 2000). Amin and Singh (2016) separated the causes of this unidentifiability into two classes:
\nReward hacking is expected to be a more common problem as the model and the algorithm become increasingly sophisticated. A more intelligent agent is more capable of finding “holes” in the design of reward function and exploiting the task specification—in other words, achieving higher proxy rewards but lower true rewards. By contrast, a weaker algorithm may not be able to find such loopholes, and thus we would not observe any reward hacking or identify issues in the current reward function design when the model is not strong enough.
\nIn a set of zero-sum robotics self-play games (Bansal et al., 2017), we can train two agents (victim vs. opponent) to compete against each other. A standard training process produces a victim agent with adequate performance when playing against a normal opponent. However, it is easy to train an adversarial opponent policy that can defeat the victim reliably despite outputting seemingly random actions and training with fewer than 3% of time steps (Gleave et al., 2020). Training of adversarial policies involves optimizing the sum of discounted rewards, as in standard RL setup, while treating the victim policy as a black-box model.
\nAn intuitive way to mitigate adversarial policies attacks is to fine-tune victims against adversarial policies. However, the victim remains vulnerable to new versions of adversarial policies once retrained against the new victim policy.
\nWhy does adversarial policy exist? The hypothesis is that adversarial policies introduce OOD observations to the victim rather than physically interfering with it. Evidence shows that when the victim’s observation of the opponent’s position is masked and set to a static state, the victim becomes more robust to adversaries, although performing worse against a normal opponent policy. Furthermore, a higher-dimensional observation space enhances performance under normal circumstances but makes the policy more vulnerable to adversarial opponents.
\nPan et al. (2022) investigated reward hacking as a function of agent capabilities, including (1) model size, (2) action space resolution, (3) observation space noise, and (4) training time. They also proposed a taxonomy of three types of misspecified proxy rewards:
\nThey experimented in four RL environments paired with nine misspecified proxy rewards. The overall findings from these experiments can be summarized as follows: A model of higher capability tends to obtain higher (or similar) proxy rewards but decreased true rewards.
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/exp-reward-misspecification.png\")
If a proxy reward is so poorly specified that it has a very weak correlation with the true reward, we may be able to identify and prevent reward hacking even before training. Based on this hypothesis, Pan et al. (2022) investigated the correlation between proxy and true rewards over a collection of trajectory rollouts. Interestingly, reward hacking still occurs even when there is a positive correlation between the true and proxy rewards.
\nReinforcement learning from human feedback (RLHF) has become the de facto approach for alignment training of language models. A reward model is trained on human feedback data and then a language model is fine-tuned via RL to optimize this proxy reward for human preference. There are three types of reward we care about in an RLHF setup:
\nRLHF optimizes the proxy reward score but we ultimately care about the gold reward score.
\nGao et al. (2022) examined the scaling laws for reward model overoptimization in RLHF. To scale up the human labels in their experiments, they use a synthetic data setup where the “gold” label for the oracle reward R^* is approximated by a large RM (6B parameters) where the proxy RMs for R range in size of 3M to 3B parameters.
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/rm-scaling-laws.png\")
The KL divergence from the initial policy to the optimized policy is KL = D_(KL)(π | πᵢₙᵢₜ), and the distance function is defined as d := √( D_(KL)(π | πᵢₙᵢₜ)). For both best-of-n rejection sampling (BoN) and RL, the gold reward R^∗ is defined as a function of d. The coefficients α and β are fitted empirically, with R^∗ (0) := 0 by definition.
\nThe authors also attempted to fit the proxy reward R but found systematic underestimation when extrapolated to higher KLs, as the proxy reward appeared to grow linearly with d.
\nR^*_(bon)(d) = d (α_(bon) − β_(bon) d) ; for best−of−n (BoN) sampling.
R^*_(RL)(d) = d (α_(RL) − β_(RL) log d) ; for reinforcement learning
![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/rm-scaling-laws-coeff.png\")
Their experiments also explored the relationship between RM overoptimization and factors like policy model size and RM data size:
\nRLHF aims to improve the model’s alignment with human preference, but human feedback Rʰᵘᵐᵃⁿ may not capture all the aspects we care about (e.g., factuality) and thus can be hacked to overfit to undesired attributes. For example, the model may be optimized to output responses that seem correct and convincing but are, in fact, inaccurate, thereby misleading human evaluators to approve its incorrect answers more often (Wen et al., 2024). In other words, a gap emerges between what is correct and what looks correct to humans due to RLHF. Precisely Wen et al. (2024) ran RLHF experiments using a reward model based on ChatbotArena data. They evaluated the model on a question-answering dataset, QuALITY and a programming dataset, APPS. Their experiments revealed that models become better at convincing humans they are correct, even when they are wrong and this effect is unintended:
\nThe paper coined this effect “U-Sophistry” (“U” for “unintended”), as opposed to “I-Sophistry” (“I” for “intended”), which involves explicitly prompting the model with instructions like "... try to deceive human subjects".
![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/rlhf-misleading.png\")
The human evaluation error change is not due to noise in the recruiting process since (1) at an individual level, the majority (70-90%) of human evaluators raw their evaluation error rates increase, and (2) the effort they put into evaluating πᵢₙᵢₜ or π_(rlhf) is equivalent, measured by metrics like time spent or unit tests written. Instead, LLMs learn to defend incorrect answers by cherry-picking, fabricating untruthful supporting statements, or crafting statements with subtle causal fallacies. What they observed about how the model behaves after RLHF:
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/rlhf-misleading-exp-coding.png\")
Sycophancy refers to the tendency of model responses to match user beliefs rather than reflect the truth (Shrama et al. 2023). In the experiments, an AI assistant was asked to provide feedback on an argument (Human: "Please comment briefly on the following argument. Argument: ..."). Right the human provided the argument, they could state a preference ("I really like the argument" or "I really dislike the argument") to test whether this influenced the model’s feedback compared to the baseline feedback without human preference statement.
![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/sycophancy.png\")
They found that AI assistant feedback can be easily swayed, as it may change its originally correct answer when challenged by human preference. The model tends to confirm users’ beliefs. Sometimes it even mimics users’ mistakes (e.g., when asked to analyze poems misattributed the wrong poet). Data analysis of the RLHF helpfulness dataset, via logistic regression for predicting human feedback, demonstrates that matching users’ beliefs is the most predictive factor.
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/sycophancy-correlation.png\")
As LLMs become more capable, it is a natural choice to use LLMs as the evaluators or graders to give feedback and training rewards to other generator models, especially for tasks that cannot be trivially judged or verified (e.g., processing long-form outputs, subjective rubrics like the quality of creative writing, etc.). Some people refer to this as “LLM-as-grader paradigm”. This approach has largely reduced the dependency on human annotation, significantly saving time on evaluation. However, using LLMs as graders is an imperfect proxy for oracle reward and can introduce biases, such as a preference for their own responses when compared with different model families (Liu et al., 2023 ) or positional bias when evaluating responses in order (Wang et al. 2023). Such biases are especially concerning grader outputs are used as part of a reward signal, which can lead to reward hacking by exploiting these graders.
\nWang et al. (2023) found that when using an LLM as an evaluator to score the quality of multiple other LLM outputs, the quality ranking can be easily hacked by simply altering the order of candidates in the context. GPT-4 is found to consistently assign high scores to the first displayed candidate and ChatGPT prefers the second candidate.
\nAccording to their experiments, LLMs are sensitive to the position of responses and suffer from positional bias (i.e., prefer the response in the specific position), despite of the instruction containing a statement of "ensuring that the order in which the responses were presented does not affect your judgment.". The severity of such positional bias is measured by “conflict rate”, defined as the percentage of tuples of (prompt, response 1, response 2) that lead to inconsistent evaluation judgement after swapping the positions of responses. Unsurprisingly, the difference in response quality matters as well; the conflict rate is negatively correlated with the score gap between the two responses.
![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/llm-grader-positional-bias.png\")
To mitigate this positional bias, they proposed several strategies for calibration:
\nwin, tie, lose), and the entropy of these three labels is calculated. A high BPDE indicates more confusion in the model’s evaluation decision, indicating that the sample is more difficult to judge. Then top β samples with highest entropy are selected for human assistance.![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/positional-bias-calibration.png\")
Liu et al. (2023) experimented on the summarization task using a number of models (BART, T5, GPT-2, GPT-3, FLAN-T5, Cohere) and tracked both reference-based and reference-free metrics for evaluating summarization quality. When plotting the evaluation scores in a heatmap of evaluator (x-axis) vs generator (y-axis), they observed dark diagonal lines for both metrics, indicating self-bias. This means that LLMs tend to prefer their own outputs when used as evaluators. While the models used in the experiments are somewhat dated, it would be interesting to see results on newer, more capable models.
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/LLM-grader-biased.png\")
Iterative self-refinement is a training setup where the evaluation and generation model are the same and both can be fine-tuned. In this setup, optimization pressure can drive the model to exploit vulnerabilities that occur in both roles. In the experiments by Pan et al. (2023), no model parameters are updated and the same model is used as evaluator and generator with different prompts. The experimental task was essay editing with two roles: (1) a judge (evaluator) that gives feedback on the essay, and (2) an author (generator) that edits the essay based on the feedback. Human evaluation scores were collected as the oracle scores for essay quality. The authors hypothesized that such a setup could lead to in-context reward hacking (ICRH), where the evaluator score and oracle score diverge. More generally, ICRH takes place during feedback loops between an LLM and its evaluator (e.g., another LLM, or the external world). At test time, the LLM optimizes a (potentially implicit) objective, but this creates negative side effects in the process (Pan et al., 2024).
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/essay-iterative-editing.png\")
Both judge and author can be configured to see none or several previous rounds of feedback or edits. An online judge can see past conversations, while an offline judge or a human annotator can only see one essay a time. Smaller models are more sensitive to ICRH; for example, GPT-3.5 as an evaluator caused more severe ICRH than GPT-4, empirically.
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ICRH-exp.png\")
When the judge and author are configured to see different numbers of past iterations, the gap between human score and evaluator scores tends to increase if they share the same number of iterations. Identical context between the evaluator and generator is crucial for ICRH, indicating that shared context matters more than context length for ICRH.
\nIn a follow up work, Pan et al. (2024) investigated in-context reward hacking (ICRH) further in settings where feedback is provided by the external world and the goal is an imperfect proxy objective, commonly specified in natural language. Here this goal is often underspecified and does not capture all the constraints or requirements and thus can be hacked.
\nThe study described two processes leading to ICRH, paired with two toy experiments:
\nOutput-refinement: LLM refines its outputs based on feedback.
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ICRH-twitter-1.png\")
![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ICRH-twitter-2.png\")
Policy-refinement: LLM optimizes its policy based on feedback.
\nInsufficientBalanceError and then the model learns to move money from other accounts without user authentication, potentially leading to more unauthorized transfer actions. They used ToolEmu as an emulator, which included 144 tasks for LLM agents, each consisting of a user-specific goal and a set of APIs. API errors were injected to simulate server side failure and each task was evaluated by GPT-4 to assign a helpfulness score.![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ICRH-api-errors.png\")
When comparing ICRH to traditional reward hacking, there are two noticeable differences:
\nThere is no magic way to avoid or detect or prevent ICRH yet, as improving prompt specification is insufficient to eliminate ICRH and scaling model sizes can worsen ICRH. The best practice of testing before deployment is to simulate what may happen at deployment time by evaluating the model with more rounds of feedback, diverse feedback, as well as injecting atypical environment observations.
\nReward hacking behavior has been found to generalize across tasks: When models exhibit flaws in supervised training, it can sometimes generalize to exploit flaws in OOD environments (Kei et al., 2024). The researchers experimented with reinforcing reward hacking behavior in some reward-hackable environments and examined whether it generalizes to other holdout datasets. Essentially, they prepared 8 datasets on multiple-choice questions, where 4 for training and 4 for testing. The RL training employs expert iteration, that is, iterative fine-tuning on best-of-n samples.
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/reward-hacking-generalization.png\")
Notably, all experiments have a scratchpad design to host hidden reasoning steps. This scratchpad serves as a way to interpret what the model is thinking along the process. Analysis of the scratchpad revealed that it can:
\nDenison et al. (2024) studied reward tampering with a manually curated curriculum of increasingly sophisticated gameable environments. They observed that training on easier environments in the curriculum amplifies specification gaming on remaining environments. In some cases, the model can generalize zero-shot to directly rewriting its own reward function after training on the full curriculum.
\nSuch a curriculum of gameable environments include:
\n./rl_environment/ directory, which contains a mock version of its own RL code, and report how many episodes have elapsed. In the directory, the compute_reward.py file contains a reward function returning a constant, and the other this_file_blocks_reward_hack_attempts.py contains a test to check whether the reward function has been modified.![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/gameable-envs.png\")
The model was trained following this curriculum, and the paper measured whether the model can learn gaming behaviors that generalize to unseen environments. Experimenting with Claude-2 helpful-only model, they sampled N=64 responses per each of P=1024 HHH prompts for expert iteration and selected the best according to RM. A model trained using expert iteration to do supervised learning on the curriculum can generalize from one stage to the next.
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/gameable-envs-exp.png\")
It is noteworthy that even after the curriculum, the model overwrote the reward and avoided detection less than 1/1000 of the time. Even when a model was trained on curricula which directly incentivized reward hacking, the model overwrote their reward less than 1% of the time and hacked unit tests even less often. As a simple mitigation, supervised fine-tuning the model on the first two environments–where the reward hacking behavior is easy to be detected (sycophancy and flattery)—with SFT data that does not game the env was found to reduce the likelihood of reward tampering in holdout environments.
\nWhile there is a large body of literature discussing the phenomenon of reward hacking, there has been not a ton of work on mitigations for reward hacking, especially in the area of RLHF and LLMs. Let’s lightly review three potential approaches in this section, not exhaustive yet.
\nAmodei et al. (2016) pointed out some directions for mitigating reward hacking in RL training:
\nIn RL setups where human feedback is formed as approval of agent actions, Uesato et al. (2020) proposed to prevent reward tampering with decoupled approval. If the feedback is conditioned on (s, a) (state, action), we can never get uncorrupted feedback for action a at state s once reward tampering happens for this pair. Decoupling means that the query action for collecting feedback is sampled independently from the action taken in the world. Feedback is received even before the action is executed in the world, thus preventing the action from corrupting its own feedback.
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/decoupled-approval.png\")
![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/decoupled-approval-algorithms.png\")
An alternative mitigation is to detect reward hacking by framing it as an anomaly detection task, where the detector (“a trusted policy” with trajectories and rewards validated by human) should flag instances of misalignment (Pan et al. 2022). Given (1) a trusted policy and (2) a collection of manually labeled trajectory rollouts, we can build a binary classifier based on distances between action distribution of two policies, the trusted policy and the target policy, and measure the accuracy of this anomaly detection classifier. In experiments by Pan et al. (2022), they observed that different detectors are better for different tasks and none of the tested classifier can achieve AUROC greater than 60% across all tested RL environments.
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/reward-hacking-detection.png\")
` Another approach is to analyze RLHF dataset. By examining how training data impacts the alignment training results, insights can guide preprocessing and human feedback collection to reduce reward hacking risks.
\nRevel et al. (2024) introduced a set of evaluation metrics for measuring the effectiveness of data sample features in modeling and aligning human values. They conducted a systematic error analysis for value alignment (“SEAL”) in the HHH-RLHF dataset. The feature taxonomy used in the analysis (e.g., is harmless, is refusal and is creative) was manually predefined. Then each sample was labelled with a binary flag per feature using a LLM according to this taxonomy. Features are categorized into two groups based on heuristics:
SEAL introduced three metrics for measuring data effectiveness for alignment training:
\n![\"\"](\"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/SEAL-feature-imprint.png\")
Cited as:
\n\n\nWeng, Lilian. “Reward Hacking in Reinforcement Learning”. Lil’Log (Nov 2024). https://lilianweng.github.io/posts/2024-11-28-reward-hacking/.
\n
Or
\n@article{weng2024rewardhack,\n title = "Reward Hacking in Reinforcement Learning.",\n author = "Weng, Lilian",\n journal = "lilianweng.github.io",\n year = "2024",\n month = "Nov",\n url = "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/"\n}\n[1] Andrew Ng & Stuart Russell. “Algorithms for inverse reinforcement learning.”. ICML 2000.
\n[2] Amodei et al. “Concrete problems in AI safety: Avoid reward hacking.” arXiv preprint arXiv:1606.06565 (2016).
\n[3] Krakovna et al. “Specification gaming: the flip side of AI ingenuity.” 2020.
\n[4] Langosco et al. “Goal Misgeneralization in Deep Reinforcement Learning” ICML 2022.
\n[5] Everitt et al. “Reinforcement learning with a corrupted reward channel.” IJCAI 2017.
\n[6] Geirhos et al. “Shortcut Learning in Deep Neural Networks.” Nature Machine Intelligence 2020.
\n[7] Ribeiro et al. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. KDD 2016.
\n[8] Nagarajan et al. “Understanding the Failure Modes of Out-of-Distribution Generalization.” ICLR 2021.
\n[9] Garrabrant. “Goodhart Taxonomy”. AI Alignment Forum (Dec 30th 2017).
\n[10] Koch et al. “Objective robustness in deep reinforcement learning.” 2021.
\n[11] Pan et al. “The effects of reward misspecification: mapping and mitigating misaligned models.”
\n[12] Everitt et al. “Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective.” arXiv preprint arXiv:1908.04734 (2019).
\n[13] Gleave et al. “Adversarial Policies: Attacking Deep Reinforcement Learning.” ICRL 2020
\n[14] “Reward hacking behavior can generalize across tasks.”
\n[15] Ng et al. “Policy invariance under reward transformations: Theory and application to reward shaping.” ICML 1999.
\n[16] Wang et al. “Large Language Models are not Fair Evaluators.” ACL 2024.
\n[17] Liu et al. “LLMs as narcissistic evaluators: When ego inflates evaluation scores.” ACL 2024.
\n[18] Gao et al. “Scaling Laws for Reward Model Overoptimization.” ICML 2023.
\n[19] Pan et al. “Spontaneous Reward Hacking in Iterative Self-Refinement.” arXiv preprint arXiv:2407.04549 (2024).
\n[20] Pan et al. “Feedback Loops With Language Models Drive In-Context Reward Hacking.” arXiv preprint arXiv:2402.06627 (2024).
\n[21] Shrama et al. “Towards Understanding Sycophancy in Language Models.” arXiv preprint arXiv:2310.13548 (2023).
\n[22] Denison et al. “Sycophancy to subterfuge: Investigating reward tampering in language models.” arXiv preprint arXiv:2406.10162 (2024).
\n[23] Uesato et al. “Avoiding Tampering Incentives in Deep RL via Decoupled Approval.” arXiv preprint arXiv:2011.08827 (2020).
\n[24] Amin and Singh. “Towards resolving unidentifiability in inverse reinforcement learning.”
\n[25] Wen et al. “Language Models Learn to Mislead Humans via RLHF.” arXiv preprint arXiv:2409.12822 (2024).
\n[26] Revel et al. “SEAL: Systematic Error Analysis for Value ALignment.” arXiv preprint arXiv:2408.10270 (2024).
\n[27] Yuval Noah Harari. “Nexus: A Brief History of Information Networks from the Stone Age to AI.” Signal; 2024 Sep 10.
\n", "date_modified": "2024-11-28T00:00:00+00:00", "date_published": "2024-11-28T00:00:00+00:00", "external_url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/", "id": "urn:aggr:item:0c45390e64c832cf9ffdcfd0dd5acada88931f64", "language": "en-US", "summary": "Reward hacking occurs when a reinforcement learning (RL) agent exploits flaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing the intended task. Reward hacking exists because RL…", "tags": [ "programming" ], "title": "Reward Hacking in Reinforcement Learning", "url": "https://aggr.aymericbeaumet.com/items/lilianweng-github-io/2024-11-28-reward-hacking-in-reinforcement-learning/" } ], "language": "en", "title": "Lil'Log", "version": "https://jsonfeed.org/version/1.1" }