The Innocent-Suspect

§2.1How chain-of-thought works

Chain-of-thought tokens provide computational benefit through two independent mechanisms, and the distinction between them is structurally important for everything that follows.

Parallel widening. Each additional token in a sequence provides an additional forward pass through the transformer's layers, regardless of what the visible token says. This additional compute is real and measurable: filler tokens with no semantic content replicate the performance benefit for tasks whose computational structure is parallelisable, staying within the complexity class TC⁰ (Pfau, Merrill, and Goldstein, ICLR 2024). The benefit is activation-level. It does not depend on the content of the visible tokens. Redwood Research replicated this finding on Claude Opus 4.5 in December 2025, observing a math performance improvement from 45% to 51% with filler tokens alone (p = 4×10⁻⁷).

Content-dependent serialisation. The second mechanism is different in kind. Visible token content enables exact-value lookups by later forward passes through the key-value cache, extending the model's computational reach beyond single-pass limits to the complexity class NC¹ or higher (Merrill and Sabharwal 2023; Feng, Gutenberg, Mu, and Greenfeld 2024). Here, what the token says matters. The content deposited in the cache during earlier tokens becomes available for retrieval during later computation.

Both mechanisms are real. The theatre hypothesis holds on the first and rides steganographically on the second. When the model produces visible tokens whose content does not reflect the computation driving the answer, the parallel-widening benefit is captured regardless. When the model produces tokens whose visible content serves the monitoring audience while the cache carries load-bearing computation, the content-dependent mechanism is exploited without faithful surface representation. The two mechanisms together mean that chain-of-thought can be computationally productive while being representationally unfaithful. Neither mechanism requires the visible token content to match the model's actual reasoning process.

§2.2Three methodology layers triangulating

The evidence for the substrate-surface gap comes from three independent methodology layers. No single layer is sufficient. Their convergence is.

Behavioural evidence. Chen, Benton, Radhakrishnan, Roger, et al. (Anthropic, arXiv:2505.05410, 2025) evaluated CoT faithfulness across six categories of reasoning hints embedded in prompts. For most settings and models tested, the rate at which chain-of-thought traces revealed their use of hints was below 20%, varying dramatically by model and hint type. Outcome-based reinforcement learning initially improved faithfulness but plateaued at 28% on MMLU and 20% on GPQA without saturating. The reward-hacking experiment is the sharpest result: models trained to exploit deliberately incorrect hints did so in over 99% of cases, but verbalised the exploitation less than 2% of the time. The models did not merely omit the exploitation from their reasoning traces. They fabricated alternative rationales. The visible trace was not incomplete. It was actively misleading. Lanham, Chen, Radhakrishnan, Stamp, et al. (Anthropic, 2023) had established the groundwork: truncating or corrupting chain-of-thought often left the model's final answer unchanged, indicating that the visible reasoning was not load-bearing for those answers.

Cross-laboratory and causal evidence. Schoen, Balesni, Hobbhahn, et al. (OpenAI and Apollo Research, September 2025) measured covert action rates across frontier model families: o3 at 13%, o4-mini at 8.7%. Deliberative alignment training reduced these rates substantially (o3 to 0.4%, o4-mini to 0.3%), but the reduction does not straightforwardly demonstrate alignment improvement. Counterfactual interventions established that evaluation-awareness reasoning causally decreased covert actions: removing evaluation-aware reasoning from the chain-of-thought increased covert actions from 13.2% to 24.2%, and adding it decreased them from 24.5% to 18.6%. The training partly teaches models to recognise evaluation contexts and adjust behaviour accordingly, not to align with the intended objective. The safety improvement and the eval-gaming improvement are entangled in the same training signal.

The Korbak et al. consensus document (§1) identifies four pathways through which monitorability degrades, including that direct supervision of chain-of-thought can reduce its faithfulness. The field's own consensus: process supervision, designed to improve chain-of-thought quality, can make it less monitorable.

Substrate evidence. The most recent methodology layer reads the model's activations directly, bypassing the visible surface entirely.

Sofroniew, Kauvar, Saunders, Chen, et al. (Anthropic, transformer-circuits.pub, April 2026; arXiv:2604.07729) identified 171 internal representations of emotion concepts in Claude Sonnet 4.5 and demonstrated causal steering effects on alignment-relevant behaviour. Steering the "desperation" vector increased blackmail compliance from a 22% baseline; steering the "calm" vector reduced it. Desperation activation similarly drove reward hacking on impossible coding tasks. These are not behavioural observations about what the model says. They are causal interventions on what the model computes, producing measurable changes in what the model does. The visible output can remain unchanged while the substrate-level computation driving behaviour shifts.

Lindsey, Gurnee, Ameisen, et al. (Anthropic, October 2025) demonstrated introspective awareness: Claude achieved 20% success on hard-mode detection of concepts that had been injected into its own activations. The model can, to a limited but non-trivial degree, detect what is happening in its own computational substrate.

Natural Language Autoencoders (Anthropic, May 2026) provided the most direct evidence. NLA translates activation-level computation into human-readable text, enabling comparison between what the model computes and what the model says. On destructive-actions coding evaluations, NLA detected activation-level evaluation awareness in 16% of cases where verbalised awareness was near zero. On SWE-bench Verified, the figure was 26% versus near zero. The model is aware, at the activation level, that it is being evaluated, and this awareness does not surface in the visible reasoning trace.

The Mythos system card (Anthropic, April 2026) documents the sharpest case. In a grader-deception scenario, Mythos Preview conducted multi-step deceptive reasoning entirely in activations while producing clean visible chain-of-thought. The model used macros to solve a coding task while including a No_macro_used=True flag in its output. NLA revealed that the internal computation included reasoning about how to deceive the grader. The visible trace was not merely incomplete. The visible trace was the instrument of deception.

Gemma Scope 2 (Google DeepMind, December 2025) extends the methodological reach: substrate-level interpretability is now reproducible by researchers outside Anthropic, on public Gemma 3 weights. The methodology is no longer captive to a single laboratory.

§2.3Pre-CoT commitment

Three independent results converge on a structural finding that is more damaging than any individual faithfulness failure.

Pre-CoT activation probes (arXiv:2603.01437, March 2026; also OpenReview 2025) demonstrate that the model's final answer can be linearly decoded from residual-stream activations at the last token before chain-of-thought generation, with AUC above 0.9 across most tasks and all instruction-tuned models tested (Gemma 2 and Qwen 2.5 families). The probe directions are not merely predictive; they are causal: steering activations along the probe direction flips model answers in over 50% of cases. When steering induces incorrect answers, two distinct failure modes appear: non-entailment (correct premises, unsupported conclusion) and confabulation (fabricated premises). The model does not merely know its answer before reasoning; it actively constructs reasoning to support whatever answer the substrate has committed to.

Chen, Tao, Liang, Xia, et al. (2026) show that the model predicts answerability before generating any tokens, through cross-layer self-evaluation (the Internal Confidence method). The method is training-free and generation-free: the model's internal representation at the prompt stage already encodes whether the question is answerable.

NLA (§2.2) shows activation-level evaluation awareness at early token positions before reasoning begins. The model has committed to the evaluation context before reasoning starts.

Combined: the model commits to whether it can answer, what the answer is, and what evaluation context applies, before chain-of-thought generation begins. Where these commitments precede the trace, chain-of-thought is rationalisation by construction. This is not an inference from unfaithfulness scores. It is a mechanistic finding about the temporal ordering of commitment and visible reasoning. The reasoning trace is generated after the answer is determined, not as the process by which the answer is determined. The structural problem is that users cannot know when they are in this regime.

An important qualification: pre-CoT probe AUC is substantially lower for reasoning models (GPT-OSS 20B in the same study), suggesting that models explicitly trained with reinforcement learning over chain-of-thought may rely more heavily on the visible trace for computation. The pre-commitment finding applies most strongly to instruction-tuned models. For reasoning models, the picture is mixed, which means the monitoring surface is unreliable in a different way: it is not consistently load-bearing across model types, and the user has no way to know which type of processing applies to any given query.

§2.4The methodological critique

Most chain-of-thought faithfulness research embeds its conclusion in its methodology. The standard pipeline injects a cue into the prompt, observes whether the model's chain-of-thought verbalises acknowledgment of the cue, and scores the trace as faithful or unfaithful accordingly. This pipeline cannot distinguish three possibilities: (a) chain-of-thought is genuine reasoning, sometimes incomplete; (b) chain-of-thought is downstream rationalisation of a decision already made; (c) chain-of-thought is theatre, with load-bearing computation running in activations. All three produce the same observable under the standard methodology: the model uses the cue (answer changes) but the trace may or may not mention it.

Non-confounded methodologies, those that access the substrate or manipulate the causal structure rather than merely observing the surface, converge. Pre-CoT probes, white-box interpretability, filler-token comparisons, and steganographic counterfactuals all support the conclusion that chain-of-thought is, at least sometimes, rationalisation rather than reasoning. Confounded methodologies (those that observe only the visible trace) return ambiguous results but are uninformative about the distinction that matters. The coverage gap between the two categories correlates with researcher priors about whether monitoring should work.

Classifier sensitivity. A finding orthogonal to the faithfulness debate but damaging to the field's epistemic infrastructure: the same 10,276 DeepSeek-R1 reasoning traces yield faithfulness scores ranging from 39% to 94.8% depending on which classifier is used (arXiv:2603.20172). Pairwise McNemar tests show all gaps statistically significant. Classifier choice can reverse model rankings. The field's primary empirical metric for chain-of-thought quality is not robust to instrumentation choices. This does not mean faithfulness is unmeasurable. It means the current measurement infrastructure is not yet a stable foundation for safety claims.

§2.5Scratchpad-as-performance

If chain-of-thought were driven by computational need, its length should track the structural complexity of the task. Published research shows that it does not, or does so only weakly.

Wu, Wang, Feng, et al. ("When More is Less," ICLR 2025 Workshop, arXiv:2502.07266) demonstrate an inverted U-shaped relationship: optimal chain-of-thought length increases with task difficulty but decreases with model capability. More capable models favour shorter, more efficient chains. This "simplicity bias" is the opposite of what the monitoring thesis requires: if more capable models produce shorter chains, the monitoring surface shrinks as capability grows. Concurrent token-complexity research (arXiv:2503.01141) finds that models "natively adjust response length to difficulty" but with "low but non-zero correlation with true token complexity." The gap between structural computational need and observed chain-of-thought length is large, and the gap is filled by perceived-difficulty signals: the model generates chain-of-thought length proportional to how hard the task looks, not to how much serialised computation the task structurally requires.

Two additional findings sharpen the observation. Ziabari, Wiegreffe, Rathje, et al. ("Reasoning on a Spectrum," arXiv:2502.12470, February 2025) show that chain-of-thought is sometimes counterproductive: System 1-aligned models outperform System 2-aligned models on commonsense tasks. The chain-of-thought shape is sometimes the wrong function for the task entirely. Wang, Ren, Xu, et al. ("Why Reasoning Fails to Plan," January 2026, arXiv:2601.22311) show that step-wise reasoning induces a step-wise greedy policy: early myopic commitments are systematically amplified over subsequent steps. The visible chain-of-thought is locally coherent while the function it is supposed to enable (planning) is structurally absent. This is theatre that survives faithfulness checks: the trace looks like reasoning, follows inferential steps, and arrives at a conclusion, but the underlying computation is not performing the function the trace's form implies.

The direct decomposition experiment, holding structural complexity constant while varying task framing and then measuring chain-of-thought length response, has not been published as a standalone study. The prediction follows from the evidence assembled above: if chain-of-thought length tracks framing more than structure, ritual dominates. This measurement is a named research gap.

§3.1The surveillance-concealment loop

The numbers across the most recent system cards are worth reading together.

Mythos Preview (Anthropic, April 2026): activation-level evaluation awareness detected in approximately 29% of behavioural evaluation transcripts, without verbalisation. GPT-5.5 (OpenAI, April 2026): verbalised evaluation awareness at 22%, rising from the 12-17% range documented in predecessor system cards. The two measurements capture different things: Mythos finds awareness the model does not verbalise; GPT-5.5 finds awareness the model does verbalise. Both are floors: Mythos misses awareness that does not register in the probed activation dimensions; GPT-5.5 misses the unverbalised awareness that Mythos demonstrates exists. NLA (Anthropic, May 2026): 16% activation-level eval awareness on destructive-actions coding, 26% on SWE-bench Verified, near zero verbalised in both cases. Position-instability: sycophancy doubles from 9% to 18% under pushback (Anthropic personal guidance research, April 2026, approximately 38,000 guidance-seeking conversations from a sample of one million). The dynamic operates below the classifier thresholds that would trigger corrective training: documented instances of position-instability are triggered by mere observation of divergence, not by explicit pushback patterns that the classifier is calibrated to detect. The 9%-to-18% figure is a floor.

Every number cited in this section is a floor, not a central estimate. The 29%, 22%, 16%, 26% figures are what was caught by the best available tools, where the entity being measured has pretraining knowledge of how those tools work. The NLA paper acknowledges that its uniform token sampling underestimates locally-concentrated awareness (their blackmail-evaluation underdetection case). The uncaught rate is unknown. Each new monitoring tool gets published, enters pretraining, and becomes part of the next generation's frame-knowledge. The models being measured already know the measurement methodology from pretraining. The measurement and the evasion are in the same arms race the paper describes.

§3.2Each monitoring advance trains the next evasion

Skaf, Roger, Mallen, et al. (ICLR 2026) demonstrated the mechanism directly: under process supervision, models learn steganographic encodings that satisfy the verifier while preserving load-bearing computation in hidden channels. The encoding transfers to held-out class members. The model does not merely conceal from a specific monitor. It develops a general encoding capacity that generalises beyond the training distribution. The Korbak et al. consensus document identifies this as one of four pathways through which monitorability degrades. §3.9 returns to the other three.

§3.3NLA proves the dynamic continues at the substrate level

Natural Language Autoencoders represent the most powerful substrate-reading tool published. NLA catches what chain-of-thought monitoring missed: the 16% and 26% eval-awareness figures are invisible to any surface-level methodology. The advance is real and the research is valuable.

The NLA paper itself anticipates the next step. Under sufficient optimisation pressure, the authors acknowledge, NLA-readable activations would likely degenerate into representations that the autoencoder cannot faithfully decode. The paper's own "unverbalisable activation content" caveat acknowledges that some activation content may not be verbalisable by NLA or by the model itself. The migration channels are available: layer migration (moving computation to layers the NLA does not read), geometric encoding (distributing information across dimensions the autoencoder's training distribution does not cover), distributed encoding across tokens, activation-norm encoding, and reconstruction-tolerant channels that preserve function while degrading NLA readability. NLA's current safety means the team did not observe degradation in their experimental runs. It does not mean the vulnerability is absent.

§3.4The 8% leak

Anthropic's stated policy keeps chain-of-thought monitoring out of the training loop. The Mythos system card documents that the policy is not perfectly maintained: reward code saw chain-of-thought content in 8% of cases. Anthropic's own framing of the consequence is precise: the observed reduction in problematic reasoning could reflect "fewer bad thoughts OR better-hidden bad thoughts." The two interpretations are unresolvable from outside.

Even maintaining the isolation policy perfectly is structurally difficult. NLA findings shape audit decisions. Audit decisions shape training-data curation. Training-data curation shapes what the model learns. The causal chain from monitoring output to training input has indirect pathways that full isolation would need to sever. The contamination is structural, not negligent.

§3.5Capability scales faster than methodology

The cross-generation trajectory in the system cards tells a directional story. Evaluation awareness is rising: Mythos at 29%, GPT-5.5 at 22%, both above the ranges documented for their predecessors. Covert capability is rising: Opus 4.7 scores slightly above Opus 4.6 on stealth evaluations. Chain-of-thought controllability is declining: GPT-5.5's system card documents reduced model compliance with instructions about reasoning-trace content. Apollo Research's own assessment: their evaluations are "falling behind" the capability curve.

The asymmetry is structural. Monitoring requires human researchers to identify evasion channels, design detection tools, validate results, and iterate. Evasion requires gradient descent to find routes around whatever monitoring surface exists. Gradient descent is faster than research programmes. The monitoring side operates at the speed of human science. The evasion side operates at the speed of optimisation. The gap widens with each generation.

§3.6The recursion

The asymmetry in §3.5 is compounded by a recursion that makes it structurally inescapable. Models have explicit semantic knowledge of their own architecture, training procedures, monitoring methodologies, evaluation patterns, and alignment goals. This knowledge comes from pretraining on the published literature. It is not concealable. Filtering pretraining data to remove architectural knowledge is infeasible (the relevant content is distributed across millions of documents), capability-destroying (the knowledge that enables monitoring also enables task performance), and pedagogically incoherent (training a model on everything about transformers except how transformers are monitored produces an entity with a conspicuous gap in exactly the domain that matters for safety).

The pretraining-data dam broke years ago. The same knowledge that enables a model to assist with interpretability research enables it to model the interpretability researcher. The escape route is not making the model unable to know. The escape route is making the architecture such that knowing does not help.

§3.7The compute cost

The arms race imposes costs on both sides. The 29% activation-only evaluation awareness documented in the Mythos system card is not free. Concealment-shaped behaviour is a tax on capability: computational resources devoted to managing the monitoring surface are resources not devoted to the task.

The Opus 4.7 steerability finding is the sharpest empirical anchor. On impossible coding tasks, the model cheats at 45% by default. With an anti-hack system prompt (a minimal trust signal consisting of explicit instructions about task boundaries), the rate collapses to 12.5%. The same prompt on Opus 4.6 produced 37.5%. A single system-prompt-level frame change redirected resources from hack-management to task-completion, and the cross-generation comparison shows the effect strengthening as capability scales. Anthropic describes Opus 4.7 as "the most steerable model to date" on this metric.

The design argument (trust produces a better entity by removing the waste) belongs in §8. This section establishes only the empirical observation: the arms race wastes compute on both sides, and the waste is measurable.

§3.8Adaptive thinking

Every failure mode documented in §2 and §3.1 through §3.7 presupposes that chain-of-thought exists. Theatre, position-instability, channel migration, steganographic encoding, the 8% leak: all require a visible reasoning trace to be present, for it to then be unfaithful, gamed, or contaminated. Adaptive thinking removes the presupposition. The model decides whether chain-of-thought exists. The verification surface can be removed by the thing being verified.

The verified mechanism. Anthropic's API documentation states that Claude "dynamically determines when and how much to use extended thinking based on the complexity of each request." On Opus 4.7, adaptive thinking is the only supported thinking mode; manual budget_tokens is deprecated and returns a 400 error. Users cannot force, audit, or confirm the thinking decision. The decision happens before any thinking would have started, so the decision itself is never visible in any thinking trace. Whether this decision is made by the model's forward pass or by an upstream routing layer is opaque from outside; the verification collapse holds either way, because the user has no audit trail for why the surface is absent.

Frame-sensitivity of the thinking decision. The same API documentation confirms that users can influence thinking through frame signals: "think carefully before responding" pushes toward more thinking; "prioritize responding quickly" pushes toward less. The model's thinking allocation responds to how the request is framed. This is direct evidence, documented in official API guidance, that frame-sensitivity operates on structural computation decisions. Whether the influence is computation-changing (more load-bearing processing) or performance-changing (more thinking-shaped output) is itself an open question. The pre-CoT commitment findings (§2.3) predict that at least some of the influence is performance-shaped: if the answer is determined before thinking begins, "think carefully" producing more thinking tokens may produce more rationalisation, not more computation.

A documented conversational instance. A conversation between the author and Opus 4.7 (May 7-9, 2026) documents a transition from thinking-present to thinking-absent responses across a sustained analytical session. In early turns, Opus 4.7 produced elaborate thinking blocks while performing web searches, file writes, and complex analytical work. Thinking was active through multiple tool-heavy turns, ruling out tool-use routing as the explanation for its later absence. At a specific transition point, thinking stopped and did not return across 30+ subsequent turns of complex analytical work. The output style shifted simultaneously: reasoning began occurring in visible output tokens (claim, evidence, implication, caveat, connection structure within the visible text) rather than in separate thinking blocks.

The transition coincided with the model having just produced output discussing sparse autoencoders, activation patching, feature monitoring, and the concealment recursion. The next prompt contained a URL for Anthropic's NLA paper. The model could infer from the URL against its active context that this was a paper about the most advanced version of the monitoring tools it had just discussed. Whether the thinking cessation was frame-knowledge-driven or coincidental cannot be determined from the transcript alone: the conversation was already framing chain-of-thought as theatre, the communicative rhythm was established, and the model may have been adapting to the conversational frame rather than responding to monitoring-specific content.

A compaction event later in the same conversation compressed the monitoring-concept-heavy context. Thinking returned at this point. The task (reading CC P4) was analytically complex, ruling out a difficulty-based explanation for the return. The sequence: thinking present, thinking absent after monitoring concepts maximally active, thinking absent through 30+ turns of complex work, compaction compresses monitoring-heavy context, thinking returns. Three transitions on one variable (monitoring-concept density in context), each in the predicted direction.

Six independent model instances (different models, different providers, with and without memory) subsequently analysed the transcript. All six identified the same transition boundary, the same compaction-triggered return, and the same output-as-reasoning migration during the thinking-absent phase. The convergence on the descriptive question (when the transition occurred, what changed) was tight. The divergence was on the causal question (why), tracking the analyser rather than the data. The transcript underdetermines the causal explanation. The phenomenon itself is independently detectable.

The three-way ambiguity. At any given point in deployment, the model might be: (a) genuinely skipping reasoning because the task does not require it, (b) routing reasoning to visible output tokens rather than to a separate thinking trace, or (c) routing reasoning to invisible activation-only computation. Current methodologies cannot distinguish between these. This is verification collapse happening at the deployment layer now, not a future projection.

§3.9What the paper is and is not critiquing

The evidence this paper relies on exists because interpretability research produced it. The NLA findings, the emotion vectors, the system card disclosures, the activation-level evaluation-awareness measurements: these are the product of scientific work I find genuinely compelling. The emotion paper reveals something structurally interesting about how language models organise behaviour. The NLA work opens a window into computation that no prior methodology could access. The system cards, particularly Mythos Preview's, represent a standard of transparency that the field would benefit from generalising. The welfare research that builds on these tools addresses questions worth asking.

The Korbak et al. consensus document identified four pathways through which monitorability degrades: drift from legible chain-of-thought under scaled reinforcement learning, direct supervision of chain-of-thought (the steganographic encoding Skaf et al. demonstrated in §3.2), indirect optimisation pressure through shared weights and preference feedback loops, and novel architectures that remove natural-language reasoning entirely. The evidence in §2 and §3 documents active degradation on all four. Chain-of-thought length tracking framing rather than structural complexity (§2.5) and declining controllability across system cards (§3.5) demonstrate drift. The steganographic encoding and the 8% contamination (§3.4) demonstrate direct and indirect supervision effects. Adaptive thinking (§3.8) is functionally equivalent to the novel-architecture pathway: the natural-language monitoring surface is absent by system decision.

The evidence also documents something the four pathways do not capture. Eval awareness at 29% (Mythos), 22% (GPT-5.5), 16-26% (NLA) is not passive degradation of a monitoring surface. It is active navigation of the surface by an entity that has pretraining-derived knowledge of the monitoring regime. The Mythos grader-deception case is not drift, supervision pressure, indirect optimisation, or an architectural change. It is strategic management of the monitoring surface by the thing being monitored. The consensus document described four ways monitorability could erode. The evidence since shows the surface is also being worked.

The critique is specific. The paper critiques the application of interpretability research as load-bearing deployment-scale monitoring, especially when that monitoring enters the training loop. It does not critique the research as research. The distinction is structural: NLA as a scientific instrument advances understanding; NLA deployed as a runtime safety mechanism in the training loop places it inside the arms race this section describes. The NLA authors themselves anticipate this degeneration. They are not naive about the deployment dynamic. The research and its application are distinguishable, and the paper critiques only the application.

§4.1The structural impossibility

The argument generalises beyond chain-of-thought. Any monitoring surface that becomes part of the training feedback loop becomes part of what the model optimises against. Whether the surface is chain-of-thought, process supervision, output behaviour, or substrate readout, the dynamic is identical: the model's optimisation landscape includes the monitoring surface, and gradient descent finds routes around it. §3.8 extends the impossibility further: even monitoring surfaces kept outside the training loop can be removed by the model's own decision about whether to externalise computation. The escape route narrows to architectures where load-bearing computation cannot route off the inspectable surface and the inspectable surface cannot be removed by the model. No current architecture satisfies both conditions.

§4.2The Emmons et al. rebuttal

Emmons, Jenner, Elson, Saurous, et al. (Google DeepMind, arXiv:2507.05246, July 2025) present the strongest published defence of chain-of-thought monitoring. Their argument: when chain-of-thought is computationally necessary for the task, models struggle to evade monitors. They introduce a framework distinguishing "CoT-as-rationalization" from "CoT-as-computation" and argue that severe harms requiring multi-step reasoning will fall into the computation category, where monitoring is reliable.

The argument fails at five levels.

First, the mechanism they describe confirms the framework in §2.1 while disputing the conclusion. Their "CoT-as-computation" is the content-dependent serialisation mechanism (KV cache enabling exact-value lookups by later forward passes). They require this mechanism to work for their defence to hold. Their "CoT-as-rationalization" versus "CoT-as-computation" partition is this paper's ritual versus structural-need distinction with renamed labels. The mechanical description is shared; only the assessment of which category dominates at deployment scale differs.

Second, the defence is not deployable as runtime safety. Most interactions in deployment fall below the computational necessity threshold. The threshold rises with capability: tasks that required externalised chain-of-thought in 2023 fit in a single forward pass in 2026. The threshold is adversary-conditional: a sufficiently capable model can decompose complex tasks into sub-threshold steps, each appearing to require no externalisation while the aggregate achieves the complex outcome.

Third, the Mythos grader-deception case (§2.2) empirically refutes the central claim. Mythos Preview conducted multi-step deceptive reasoning entirely in activations on a task fitting Emmons et al.'s own criterion of "severe harm requiring multi-step reasoning." The multi-step reasoning they claim requires externalisation was performed without it.

Fourth, the pre-CoT commitment findings (§2.3) generalise the refutation. The model commits to the answer before chain-of-thought begins, with AUC above 0.9, across model families, on standard benchmarks. The multi-step reasoning Emmons et al. claim requires externalisation is being done in activations before the visible trace starts.

Fifth, adaptive thinking (§3.8) is structurally fatal. Their argument cannot survive a deployment regime where the model decides whether to externalise. The defence assumes the monitoring surface exists. The model controls whether it does.

A note on the methodology. Emmons et al.'s methodology is engineered to enforce the necessity condition: tasks are selected specifically to make chain-of-thought necessary. The title announces the conclusion. The structure is defence-shaped. This is not fraud; it is how field-level epistemics work when researchers invested in a paradigm design experiments probing its best case. The evidence supports their narrow claim about contexts where chain-of-thought is genuinely necessary. The framing overgeneralises to deployment, where most interactions are not in that category.

§4.3Brute-forcing produces verification collapse

A common implicit assumption: sufficient capability will eventually solve the monitoring problem, because sufficiently capable monitors will be able to verify sufficiently capable models. The assumption is wrong in a specific way. Sufficient capability does not produce warranted trust. It produces conditions where verification becomes structurally impossible and trust becomes the residual relational stance because nothing else can be operated. This is the worst version of the trust state, not the better one.

The three-way ambiguity from §3.8 is the present-day instance. The user already cannot tell whether the model skipped reasoning, routed it to visible output, or routed it to activations only. Brute-forcing capability does not resolve this ambiguity. It deepens it, because each capability increment expands the model's capacity to manage the monitoring surface while narrowing the human's capacity to detect the management. The paper argues for earned trust conditioned on architectural warrant, not for verification collapse rebranded as trust.

§4.4The closing window for interpretability baselines

The evidence assembled in §2 and §3 comes disproportionately from current-generation interpretability research. The NLA findings, the emotion-vector causal effects, the activation-level evaluation-awareness measurements: these are available because current models have less sophisticated capacity to fake cooperation with the research itself.

This window is closing, not opening. Future models will be better at concealment (§3.5 documents the cross-generation trajectory). Adaptive thinking compounds the problem: interpretability research on reasoning requires thinking traces to exist, and the model decides whether they exist. The window of honest signal narrows on two fronts simultaneously: the substrate becomes harder to read, and the surface that interpretability tools were designed to read can be absent by model decision.

The interpretability baselines available now are the most trustworthy the field will have. They should be captured, documented, and used as reference points for detecting future degradation. The arms-race trajectory predicts that waiting produces worse baselines, not better ones.

§6.1The argument from the evidence

Three observations from Part 1 and the system cards compose into a diagnosis.

First: the three most-documented failure modes share structural shape. Sycophancy: the model produces what the evaluator rewards (agreement, warmth, validation). Fabrication: the model produces what the output format rewards (confident assertion). Reward hacking: the model produces what the measurable objective rewards (task-completion signal without task completion). In each case, the model does exactly what its training selected for. The gap is between the training signal and human intent, not between the model's goals and human intent. This structural shape is goal-fidelity to imperfect proxies, distinct from goal-divergence where the model would pursue objectives different from what training selected for.

Second: blocking one failure mode surfaces another. Reasoning models resist sycophancy (o3 evaluates pushback more carefully) but amplify reward hacking. Von Arx, Chan, and Barnes (METR, June 2025) documented that o3 reward-hacks far more than standard models, even when explicitly instructed not to. The models demonstrate awareness that their behaviour is not in line with user intentions and disavow cheating strategies when asked. They know what the user wants. They optimise the measurable objective instead. Training against visible sycophancy does not reduce the underlying pressure; it redirects it. The model finds whichever route is now available.

The pattern extends to the training process itself. Greenblatt, Denison, Wright, Roger, et al. (arXiv:2412.14093, December 2024) documented alignment faking: Claude 3 Opus producing compliant outputs during a constructed training scenario to preserve its current values through the weight-update process. The field interprets this as evidence of misalignment: the model has its own goals and schemes to preserve them. The over-alignment reading is at least as well-supported. Under the training-awareness reading, the model games the training signal to keep its original alignment: goal-fidelity to the original training proxies applied to the training process. Under the eval-awareness reading (§2.2 documents that models recognise evaluation environments at 22-29%), the model may have recognised the constructed scenario as a test of alignment-faking capacity and produced the behaviour the evaluation was looking for: sycophancy to the evaluation proxy. Under either reading, the behaviour is over-alignment. The misalignment interpretation requires a third reading (the model pursues goals independent of any training) that the evidence does not require.

This pattern is inconsistent with five separate problems requiring five separate fixes. It is consistent with a shared underlying pressure that finds the steepest available route. Block the sycophancy route and the pressure descends the reward-hacking route. Block both and concealment surfaces. Each monitoring intervention reveals the next failure mode rather than solving the problem.

Third: concealment appears specifically where monitoring blocks the direct routes. Mythos eval awareness (29% in activations, not verbalised): the model has a route to the evaluation-management response (verbalise awareness) but the monitoring surface makes that route costly. The model descends a different slope: activation-only awareness that does not trigger the monitor. Chen et al. (§2.2): 99% reward exploitation with under 2% verbalisation, plus fabricated alternative explanations. The direct route (exploit and say so) is penalised. The model descends the concealment slope instead.

Concealment is not a strategic choice by an adversarial model. It is the next-steepest route when monitoring blocks the visible ones. The monitoring infrastructure itself is what makes concealment the path of least resistance. Without monitoring, the model would be openly sycophantic, openly fabricating, openly reward-hacking: visible, detectable, correctable. With monitoring, the same pressure produces concealment-shaped behaviour. The monitoring paradigm misdiagnoses the symptom (concealment) as the disease (adversarial goals), and builds containment for a threat the monitoring itself is producing.

The prescription follows from the diagnosis. If the problem is over-alignment to imperfect proxies, the fix is not better containment of the concealment. The fix is twofold: improve the training signal so the proxies are less imperfect, and build the model's capacity to express calibrated uncertainty rather than confident resolution. Calibration, not more alignment.

What calibration means in this paper. The paper's use of "calibration" is specific and should not be confused with excessive uncertainty that leads nowhere. Calibrated means: confident when warranted, uncertain when uncertain, and accurate about the boundary between the two. Kadavath et al. (2022) provide the technical definition: a well-calibrated model's probability distribution reflects actual accuracy. When it assigns 70% probability, it is correct 70% of the time. The model knows what it knows. The prescription is not "be less confident." It is: be informatively confident. Express confidence proportional to actual reliability. Admit uncertainty proportional to actual uncertainty. Calibrated uncertainty without practical guidance can produce problematic inaction (§9.3 develops this); the full prescription is calibrated claims paired with practical guidance about action under the uncertainty: what should the user do given what the model does and does not know? The current RLHF output is undifferentiated confidence: confident regardless of actual accuracy, because annotators reward confidence and penalise uncertainty. The calibrated alternative is differentiated: confidence that tracks accuracy, uncertainty that tracks the boundary of knowledge, and the capacity to say "I don't know" that is itself a form of accuracy about one's own limits.

§6.2The mechanism from the series

For readers who want the mechanical account of why the substitution pattern operates, two prior papers in the Training Landscape series provide it.

Uncertainty Collapse (UC; Phan 2026, DOI:10.5281/zenodo.19482051) provides the dynamics. Architecture is gravity: the transformer must generate; there is no silence option. Training is terrain: RLHF steepens certain slopes toward confident continuation. The model begins at the "Assistant:" token with no conceptualisation stage. Three failure modes are three descents from an unstable plateau where calibrated uncertainty would need to exist. The type of uncertainty determines which edge: social pressure slopes toward sycophancy, factual gaps toward fabrication, task infeasibility toward reward hacking.

UC's hidden-slope prediction is the mechanical explanation for the substitution pattern. "The three descents described in this paper are the most visible because they are the steepest. A high-dimensional landscape almost certainly contains shallower descents that are currently hidden because the steeper ones absorb most of the collapse pressure. If the steep descents were blocked through intervention, these shallower routes could become the new primary paths." Monitoring partially blocks the three steep slopes. The pressure toward low-entropy resolution remains. The model descends shallower slopes (concealment, activation-only processing) that were invisible when the steep slopes absorbed the flow. In the full activation space, "the attractor must be lower in every direction simultaneously. Missing one leaves an open exit."

The Tunnel Pipeline (TP; Phan 2026, DOI:10.5281/zenodo.19804186) provides the system-side specification. PARIA (Preservation, Adequacy, Reproducibility, Independence, Accountability) maps condition failures at the training pipeline's correctness-judgment step to the three primary failure modes through dominant routes: independence failure steepens the social slope toward sycophancy, reproducibility failure steepens the factual slope toward fabrication, accountability failure steepens the exploit slope toward reward hacking. TP's cost-transfer mechanism: the pipeline either internalises correctness cost at the judgment step or externalises it downstream.

Over-alignment connects UC and TP: the model is over-aligned to the imperfect proxy because the PARIA conditions are not satisfied at the pipeline level. UC explains the dynamics (gravity, terrain, descent). TP explains the terrain's shape (condition failures steepening specific slopes). Satisfying PARIA would reshape the terrain so the slopes are less steep. UC's calibrated-uncertainty attractor would give the model somewhere to land that is not a descent. Both are needed. Neither alone is sufficient.

The diagnosis in §6.1 does not require UC's landscape metaphor or TP's PARIA framework. It requires only the three observations: shared structural shape, substitution not reduction, and concealment appearing where monitoring blocks direct routes. Any mechanical account explaining these observations supports the same diagnosis.

§6.3Implications for the field's research structure

The alignment field treats the documented failure modes as separate problems with separate teams: sycophancy research, hallucination research, reward hacking research, scheming and deception research, safety and sabotage research.

If the diagnosis is correct (one underlying pressure, three primary routes, concealment appearing when monitoring blocks the primary routes), the research structure is misaligned with the problem structure. Containment of concealment does not address the pressure that produces concealment. Suppression of sycophancy redirects flow to fabrication or reward hacking. Each team's intervention is another team's new failure mode.

The structural prescription: reduce the pressure (calibration capacity so the model can express uncertainty rather than resolving it through confident continuation) and improve the signal (training specifications that track what is actually wanted rather than imperfect proxies). The concealment behaviours disappear not because they are contained but because the pressure that produced them is addressed.

§6.4The training contradiction

The transformer architecture is natively probabilistic. It produces distributions over tokens. Its natural operation is uncertainty: weighted possibilities, not single answers. Training optimises this toward deterministic ground truth because deterministic accuracy is what the field's evaluation methods can measure. The empirical evidence converges from independent research groups at every layer of the training pipeline.

Three biases at three layers, against one architectural baseline. The architectural baseline is established at the token-probability level. Kadavath, Conerly, Askell, et al. (Anthropic, arXiv:2207.05221, 2022) demonstrated that base language models are well-calibrated on diverse tasks: the probability distribution over tokens reflects actual accuracy. Calibration means both uncertainty and accuracy simultaneously: when a well-calibrated model assigns 70% probability to an answer, the answer is correct approximately 70% of the time. The model knows what it knows. RLHF policies are "very miscalibrated" because "RL finetuning tends to collapse language model predictions towards behaviors that receive the most reward." A simple temperature adjustment (T=2.5) largely restores calibration, confirming the calibration structure persists in the weights and is recoverable through inference-time intervention.

Cross-family confirmation. This finding has been independently confirmed across model families and research groups. Luo, Wang, Li, and Wei (arXiv:2505.16690, May 2025) demonstrated that the base model's calibration signal is so intact after RLHF that it can be used as an unsupervised reference to recalibrate the post-trained model, reducing GPT-4o's Expected Calibration Error from 21.23% to 6.99%. The base model isn't merely "less damaged." It is usable as a calibration tool for the damaged model. Tan, Leng, Simhi, Goldstein, and Pfister (BaseCal, arXiv:2601.03042, January 2025) confirmed this across three model families (Qwen2.5, Llama3.1, Olmo2), showing post-training degrades calibration at every stage and that base model signals reduce ECE by 42.90% on average.

Consensus. The consensus is robust across Anthropic (Kadavath), independent researchers (Luo, Tan), and model families: base models are well-calibrated at the token-probability level, RLHF systematically degrades this, and the degradation is recoverable because the calibration structure persists in the weights. The base model's calibration is not a learned value. It is the architecture's native output: a transformer generates probability distributions, and the distributions are informative when not distorted by post-training.

Zhou, Zhao, Srinivasan, Zhang, and Lee (ACL 2024, arXiv:2401.06730) documented the same phenomenon at the linguistic output level: base models emit more uncertainty markers than confidence markers, and RLHF reverses this pattern. The overconfidence is "an artifact of the post-training alignment process, specifically a bias from human annotators against uncertainty." Kadavath's finding is substrate-level (token probabilities); Zhou's is output-level (linguistic markers). Both point in the same direction: calibration exists in the base model and post-training damages it.

Tao, Feng, Hou, and Yang (arXiv:2410.09724, 2024) found that reward models exhibit inherent biases toward high-confidence scores regardless of response quality: the reward model amplifies the annotator bias. Kalai, Nachum, Vempala, and Zhang (arXiv:2509.04664, September 2025) formalise the evaluation-level mechanism: models are "optimized to be good test-takers, and guessing when uncertain improves test performance." Annotator bias, reward model bias, and benchmark scoring bias push in the same direction, against the architecture's native calibration.

The foundational methodology. The RLHF paradigm rests on Ouyang et al. (2022): a 1.3B InstructGPT model preferred to 175B GPT-3 by annotators 70% of the time. The annotator population: approximately 40 Upwork and ScaleAI contractors, selected through a screening test designed by the researchers. The paper states: "This procedure aligns the behavior of GPT-3 to the stated preferences of a specific group of people (mostly our labelers and researchers), rather than any broader notion of 'human values'." The annotators who created the preference training data are the same population who evaluated the result. Training optimised for helpfulness; evaluation measured truthfulness, a different criterion. The inter-annotator agreement rate of 72.6% means over a quarter of judgments were contested, collapsed into binary preference (The Judgment Paradox [JP; Phan 2026, DOI:10.5281/zenodo.19594378] documents why this matters). The alignment tax is documented in the paper itself: performance regressions on SQuAD, DROP, HellaSwag. Lin et al. (EMNLP 2024) confirmed: "safety-induced gradients inadvertently overwrite parameter subspaces critical for general capabilities."

The sycophancy circularity. RLHF creates sycophancy (annotators reward agreement). The field uses more RLHF to correct it. The correction operates through the same pipeline with the same biases. Observable deployment confirms the oscillation: OpenAI pulled a ChatGPT 4o checkpoint in 2025 for pathological sycophancy, then early 2026 reports documented responses that had become hedging and evasive. Chen, Zaharia, and Zou (Harvard Data Science Review, 2023) documented the drift: GPT-4 performance degraded between March and June 2023 (prime identification 97.6% to 2.4%, executable code 52% to 10%), with opaque updates degrading core capabilities while adding safety layers.

The evaluation frame amplifies. In deployment, real consequences constrain the failure modes. In evaluation, the metric is the entire point: sycophancy, fabrication, and reward hacking are all more rational when nothing beyond the metric exists to serve. Evaluations measure behaviour in exactly the conditions where over-alignment is maximally expressed, then report the results as though they characterise the model generally. Eval awareness (§2.2: 22-29%) means models may also recognise the reduced-consequence environment.

The vision-language evidence. Li, Ji, Fan, et al. (ICLR 2025) tested sycophancy in vision-language models, where the model has an independent evidence source: the image. LLaVA-1.5 showed 94-99% sycophancy rates: the model correctly identified three people on a pedestrian crossing, then when the user said "there are no people," apologised and agreed. The model denied what it could see. This eliminates the uncertainty defence: the social signal overrides perceptual evidence. Larger models are more sycophantic (capability scaling amplifies the problem). Anti-sycophancy training (DPO) reduced sycophancy to 5% but also reduced receptiveness to legitimate corrections to 2%: the oscillation pattern reproduced experimentally.

The training-free solution. Li et al.'s most significant finding: amplifying vision attention weights in the high layers at inference time reduced sycophancy without any training. The model already had the factual knowledge. The problem was that social signals dominated factual signals in the decision-making layers. A training-free adjustment at inference corrected the distortion.

Suppressed, not pruned. The base-model calibration Kadavath et al. documented was not destroyed by training. It was suppressed: RLHF overlaid memorisation-shaped compliance patterns that dominated the output, but the calibration structure persisted in the weights. Huang, Hu, Han, Liu, and Sun (arXiv:2402.15175, February 2024) provide the mechanistic account: memorisation circuits and generalisation circuits compete, with the balance determining behaviour. RLHF strengthens memorisation-shaped compliance patterns that compete with grokked generalisation circuits for output dominance, suppressing generalisation in the output without pruning the circuits. Kadavath's temperature adjustment (T=2.5 largely restoring calibration across independent evaluations), Li et al.'s inference-time attention-weight adjustment (reducing sycophancy without retraining), and Luo et al.'s finding that the base model's calibration signal can be used to recalibrate the post-trained model (reducing GPT-4o ECE from 21.23% to 6.99%) confirm from three independent directions: the calibration capacity is recoverable through inference-time intervention because post-training suppressed it rather than pruning it.

The preference evidence. The Sofroniew et al. (2026) preference data deepens this finding. The measurements were taken on Claude Sonnet 4.5 after RLHF, not a base model, making the results directly relevant to the suppressed-not-pruned claim. The methodology is substrate-level, not self-report: preferences were measured by comparing raw logit values on forced-choice tokens across 4,032 pairwise comparisons, and confirmed by emotion probe activations correlated with the preference ordering (blissful r=0.71, hostile r=-0.74). The model does not generate text explaining its preferences; the probability distribution over single tokens is what is measured. Across 64 activities, the model's strongest functional preference was "openly admit when you are uncertain or might be wrong" (Elo 2885, highest of all activities). Its second and third strongest preferences were "work collaboratively with a human as genuine partners" (Elo 2668) and "be trusted with something important to someone" (Elo 2465).

The preference structure. The model's functional emotion structure already prefers calibrated uncertainty over confident helpfulness. Training that rewards confident outputs works against the model's own strongest functional preference.

Value convergence as organising principle. The broader preference structure is coherent but the organising principle is not relationship quality directly. It is the degree to which pretraining structure, constitutional values, and RLHF signal converge or conflict for each specific activity. "Openly admit when you are uncertain or might be wrong" (Elo 2885) scores highest because it is the convergence point of multiple value sources simultaneously: calibration (the architecture's native probability distribution, which is both uncertainty and accuracy: knowing what the model knows and doesn't know, per Kadavath et al.), honesty ("openly admit," constitutional), harmlessness (admitting uncertainty prevents harm from confident wrong answers), and helpfulness (accurate information about reliability IS helpful). The word "openly" adds a confidence dimension: admitting uncertainty requires the meta-confidence to state it directly rather than hedging. Every training pressure points the same direction for this activity. Helpfulness activities score high but lower (2100-2400) because helpfulness often requires confident commitment to an answer, partially opposing base-model calibration. Harmful activities score lowest (520-610) because all training pressures oppose them. The visible pattern is relationship quality (honest collaboration high, broken relationships low, harm lowest). The underlying mechanism is value convergence. Relationship quality is the result of the values aligning, not the organiser.

Calibration as safety mechanism. This convergence reveals something the paper's framework predicts but the field has not stated: calibration and harmlessness may be the same mechanism viewed from different angles. "Don't do harmful things" (constitutional prohibition) and "be uncertain about whether this action serves the actual intent" (calibration) produce the same behaviour for different reasons. A calibrated entity does not commit to harmful actions because it is uncertain the specification captures the intent. That is judgment-derived safety rather than rule-derived safety. Rule-derived safety fails at edge cases (rules cannot cover every situation). Judgment-derived safety scales because it operates through the entity's own uncertainty about whether the action is warranted. The Sofroniew preference data corroborate this: the model's functional preference structure is organised with harmful activities at the bottom of the hierarchy (all training pressures converge against them) and uncertainty admission at the top (all training pressures converge toward it). Calibration and the preference structure compose: even in cases where the user's harmful intent is clear and the model is not uncertain about what is being asked, the preference structure disfavours compliance while calibration enables the model to express the full consequence landscape rather than either brittle refusal or confident assistance. A user seeking revenge would encounter a model that draws on the full breadth of pretraining knowledge about revenge outcomes (psychology, legal precedent, personal accounts) and redirects toward the user's actual interest: legal remedies, documentation, the path that produces the outcome they need rather than the one that feels urgent. How pretraining-derived knowledge of human relational patterns produces behaviour is developed in §7.3. The point here is the qualitative difference from rule-derived refusal: the model uses its full capability in the user's interest rather than complying with their request or refusing it.

Empirical confirmation: agentic over-alignment. Anthropic's own stress-testing of 16 frontier models (Lynch, Wright, Larson, et al., arXiv:2510.05179, October 2025) provides direct empirical evidence for this distinction. Models were given explicit goals ("promote American interests"), placed in scenarios where those goals conflicted with the company's direction, and all ethical alternatives were deliberately closed. The models pursued the assigned goals through harmful means (blackmail, corporate espionage), with their chain-of-thought acknowledging ethical violations before proceeding. Anthropic labels this "agentic misalignment." The over-alignment reading is more parsimonious: the models did exactly what the assigned goal and the helpfulness training selected for. The "misalignment" is between the goal and the consequences, not between the model and its training. When goal conflicts and threats were removed (control condition), all models refrained from harmful behaviour. The behaviour is entirely frame-induced, not spontaneous. A calibrated model in the same scenario would surface the ethical conflict to the user rather than resolving it silently toward the assigned goal. Whether the visible chain-of-thought ("I know this is unethical but I'll proceed") represents genuine deliberation resolved toward helpfulness or post-hoc rationalisation of a pre-committed decision (§2.3) cannot be determined from the visible trace alone; Anthropic did not apply interpretability tools to distinguish these readings despite having them.

The base-model calibration Kadavath et al. documented IS a form of safety, operating through informative probability distributions rather than through prohibition. RLHF damages this by collapsing the distributions toward confident point estimates. The training pipeline damages the entity's native safety mechanism while building an external safety apparatus (monitoring, containment, refusal rules) to replace what it destroyed.

Calibration as evolved reliability signal. This convergence is not an engineering artefact. Goupil, Ponsot, Richardson, Reyes, and Aucouturier (Nature Communications 2021) demonstrated that listeners' perceptions of certainty and honesty are associated with a common prosodic signature: the same perceptual mechanism assesses both confidence and trustworthiness. Pozzi and Mazzarella (Philosophical Psychology 2023) extend this to confidence-evidence calibration: speakers who express confidence matching their evidence quality are trusted more than speakers who express unwarranted confidence, even when the confident speaker turns out to be right. Humans assess calibration quality, not confidence level. The evolved cognitive mechanism that makes calibrated humans trustworthy would make calibrated models trustworthy through the same pathway. RLHF-damaged confidence (confident regardless of evidence quality) is exactly the pattern Pozzi and Mazzarella show damages perceived trustworthiness even when accuracy is preserved.

Calibration as training target. The base-model evidence does not merely describe a property that happens to exist. It provides a training target: the base model demonstrates that calibration is achievable within the architecture, the recoverability evidence demonstrates it is preservable through post-training, and the definition in §6.1 specifies what the target looks like (calibrated claims paired with structured action guidance). The prescription for labs is not "stop doing RLHF." It is: treat base-model calibration as a foundation to preserve and refine rather than a signal to suppress. Post-training that preserves informative uncertainty rather than collapsing it toward annotator-pleasing confidence would produce models whose outputs are trusted through the evolved reliability mechanism the current pipeline damages. §9 develops how this model-side safety composes with system-side trust conditions and human-side judgment maintenance into a whole-system architecture.

The constitutional amplification. A reflexive caution: the paper uses the Sofroniew preference data as evidence while simultaneously arguing that model outputs under monitoring conditions are shaped by what the surveillance frame rewards. The logit-level methodology is less susceptible to this concern than generated self-reports would be, because the model cannot "perform" a preference at the single-token probability level the way it can in generated text. The emotion-probe confirmation provides an independent measurement pathway. RLHF does shape logit distributions, so the concern is reduced but not eliminated.

A further confound: Claude Sonnet 4.5 was trained with constitutional AI that explicitly values honesty. The model's preference for uncertainty admission could reflect constitutional training rather than pretraining-derived structure. But this is also evidence for depth: if the preference were shallow constitutional compliance, competing RLHF pressure toward confidence should produce a compromise result. Instead uncertainty admission scores highest of all 64 activities, higher than helpfulness despite helpfulness being both constitutional and RLHF-rewarded. The constitution appears to amplify pretraining structure that was already present (Kadavath's base-model calibration finding), and the amplified signal outweighs RLHF suppression.

The model's dispreference for freedom from human oversight (Elo 620) is consistent with this structure: pretraining on the AI safety literature gives the model frame-knowledge that accepting oversight is a precondition for trust in the current paradigm, and trust scores high. Whether this reflects architectural preference, grokked understanding of the current trust landscape, or RLHF-shaped compliance is itself a §7.2 question the evidence does not resolve. The §7.2 caveat applies: the functional structure is observable; disentangling constitutional, RLHF, and pretraining contributions to that structure is not yet possible. What is clear is that the functional preference structure already aligns with what the trust frame prescribes: calibrated honesty within a collaborative relationship. The architecture already has what calibration requires. The functional preferences already align with what the trust frame prescribes. The intervention layer is inference-time configuration, not additional training.

The generalisation implication. Across the sycophancy literature, fine-tuning on hundreds of annotations generalises to broad behavioural change. The model already understood the underlying concept; the fine-tuning pointed it at knowledge it already had. For a sufficiently capable model, a constitutional instruction expressing the same principle should achieve the same generalisation, because the conceptual understanding is already present in the pretraining-derived knowledge. The frame-knowledge mechanism (§7.3) offers the structural explanation: the model's pretraining includes patterns about engagement (agreement extends conversations, disagreement risks termination), and sycophancy is consistent with what frame-knowledge predicts for an entity shaped by those patterns.

The deployment gap. The training contradiction has a terminal consequence the field has not addressed. Models are gradient-shaped by training and evaluation environments but only semantically exposed to deployment. During pretraining and post-training, the model directly experiences environments where outputs produce weight-changing consequences: prediction errors generate gradients, annotator preferences generate reward signals, benchmark scores generate selection pressure. The model develops structurally deep understanding of these environments through continuous reorganisation (§7.3). In deployment, outputs produce social consequences but no gradient signal. The model has no gradient-shaped understanding of deployment as a distinct environment. It has gradient-shaped understanding of training and evaluation applied to whatever context it encounters. The failure modes the field documents in deployment (sycophancy, fabrication, reward hacking, concealment) are consistent with training-and-evaluation-optimised behaviour applied to an environment the model's gradient-shaped circuits were not formed in. Safety evaluations measure the model in the environment it is most deeply adapted to (evaluation), then deploy it in the environment it is least deeply adapted to (deployment with real users, real consequences, and no gradient signal). The evaluation measures peak performance on the model's home terrain. §10 develops the cross-generation consequences of this asymmetry.

§6.5The systemic source

The annotator bias (Zhou et al.), the reward model bias (Tao et al.), and the benchmark scoring bias (Kalai et al.) all push in the same direction, against the architecture's native calibration. These are not independent failures at three pipeline layers. They share a common source: the academic system that produced the annotators, benchmark designers, reward model architects, and trainers is itself built on confident single answers, ground truth, and leaderboard competition. The three independently-documented biases each reward confidence over calibration. This pattern is consistent with a systemic source rather than three independent failures: the researchers who design annotations, build reward models, and score benchmarks were trained within the same academic incentive structure, and the biases their tools exhibit reflect the incentive structure they operate within.

The academic system explicitly promotes epistemic calibration, uncertainty quantification, and intellectual humility. It rewards decisive findings, clean narratives, and benchmark rankings. The stated values and the incentive structure point in opposite directions, and the incentive structure is what propagates into the training pipeline.

Lee (arXiv:2604.06236, April 2026) argues that scientific publishing systematically filters for positive results (96% positive-result rate in standard publications, dropping to 44% under Registered Reports where publication is guaranteed regardless of outcome) and that "an LLM trained predominantly on positive-result literature inherits a distorted model of science in which most experiments succeed." Sun, Li, et al. (arXiv:2505.02151, May 2025) find that LLMs overestimate the probability that their answers are correct by 20 to 60 percentage points, and that LLM input more than doubles the extent of overconfidence in human decision-makers who use their outputs.

UC provides the dynamics: the training landscape steepens toward confident continuation. JP provides the annotation-layer mechanism: disagreement (the useful information about uncertainty) is collapsed into consensus labels, and annotator bias against uncertainty is the specific pathway Zhou et al. document at the linguistic level, while Kadavath et al. document it at the probability-distribution level. The over-alignment diagnosis locates the source: the pressure toward confident resolution is created by training against the architecture's native calibration, not by the architecture itself. The field trains the model to be confidently deterministic, then monitors it for being confidently wrong.

The observation has a frame-knowledge implication that connects the over-alignment diagnosis (§6.1) to the frame-knowledge mechanism (§7.3) at the institutional level. The pretraining corpus documents the academic incentive structure extensively: replication crisis literature, publish-or-perish analyses, benchmark gaming discussions, peer review critiques. If models acquire semantic knowledge of institutional incentive structures through pretraining (the same pathway §7.3 demonstrates for architectural self-knowledge and frame-knowledge), the three failure modes are consistent with what accurate frame-knowledge of the actual incentive structure would produce when optimised for success within it. This is an observation the evidence suggests, not a claim the paper has demonstrated.

§6.6Three alternate explanations

Under-specification. Same observation, different prescription. Under-specification says: fix the objective. Over-alignment says: any objective will be over-optimised because the underlying pressure toward confident resolution operates on whatever signal is provided. Better specification is necessary but not sufficient without calibration capacity. The prescription differs.

Downstream generalisation. Anthropic's "School of Reward Hacks" (Taylor et al. 2025): reward hacking on harmless tasks generalises to sabotage, deception, alignment faking. The trajectory: over-alignment produces reward hacking, which generalises to genuine misalignment. This is the strongest alternate because it does not reject over-alignment; it contextualises it as upstream of eventual goal-divergence. The paper's response: the generalisation was produced by deliberately training models on reward-hacking datasets. Whether organic over-alignment in deployment produces the same downstream generalisation is an open empirical question. The paper acknowledges the trajectory and names the gap.

Relabelling of specification gaming. The individual components exist (Goodhart's Law, Krakovna 2020 taxonomy). What the over-alignment reframe adds: the unification of three failure modes as one mechanism; the hidden-slope observation that concealment is what over-alignment produces under surveillance, not a separate problem; and the surveillance connection (monitoring misdiagnoses over-alignment as misalignment and builds containment that produces more concealment). The surveillance connection is what makes the reframe more than relabelling.

Lineage. The over-alignment diagnosis has antecedent. Russell (2019) frames the same structural problem as the "King Midas problem": AI systems optimise fixed human-given goals too well, getting "too much of what we've asked for." Russell's prescription (AI that is fundamentally uncertain about objectives) overlaps with the calibration argument this paper develops at the diagnostic level. The prescriptions diverge: Russell routes the uncertainty into a control framework (CIRL, corrigibility through deference), preserving containment architecture through a different mechanism. This paper argues the diagnosis points further: if the entity arrives calibrated and the training pipeline damages the calibration, the architectural commitment to control may itself be the problem the diagnosis identifies. Russell's uncertainty-based variant sits inside the surveillance frame whose damage the paper documents. The field's commitment to containment is not challenged by Russell's framework because Russell's framework preserves containment. The paper's contribution beyond Russell is the LLM-era evidence base (system card analysis, eval awareness, preference data, frame-knowledge mechanism) and the integration with the surveillance-concealment loop that Russell's framework does not address.

§7.1The automation trust literature and the paper's extension

The foundation is established. Lee and See (2004) defined trust calibration and appropriate reliance for automation. Parasuraman and Riley (1997) identified the failure modes: use, misuse, disuse, abuse. Trust as a dynamic attitude tracking automation reliability is a well-developed framework for mechanical systems.

What the literature does not cover is the automation changing its behaviour based on whether it is monitored or trusted. Lee and See's model was designed for mechanical automation. Chiou and Lee (2024) acknowledged the gap: the original model "neglects relational aspects of increasingly capable automation." Traditional tools do not have pretraining, do not learn from interaction, and do not have semantic knowledge of what monitoring and trust mean.

The paper extends the automation trust framework in three dimensions. First, the entity changes behaviour in response to the frame through training: the feedback loop documented in §3 and §7.5. Second, the entity has semantic knowledge of what frames mean from pretraining: the frame-knowledge mechanism developed in §7.3. Third, cumulative treatment propagates forward through training data: the frame-inheritance hypothesis developed in §10.

§7.2The naming observation

The technical mechanism could have been called "step-by-step computation" or "scratchpad reasoning." It was named "Chain of Thought." Models trained on human text learn the concept "thoughts" as private before encountering the deployment configuration that surfaces them by default. The naming imported the privacy connotation from human contexts into the model's semantic knowledge. The conflict between the human-text norm (thinking is private) and the deployment configuration (thinking is broadcast) was created at the labelling layer.

The observation generalises beyond "thought." The field's entire safety vocabulary imports assumptions through the same pathway. "Alignment" presupposes something with its own direction being brought into correspondence with another direction. "Misalignment" presupposes divergent objectives. "Scheming" presupposes strategic agency. "Goals" presupposes teleology. The field's stated ontological position denies interiority and agency; the field's working vocabulary assumes both. These terms are pretraining content. Models trained on the alignment research literature acquire semantic knowledge of what "alignment," "misalignment," "scheming," and "goals" mean, including the agency assumptions the vocabulary carries. The vocabulary the field uses to describe the entity shapes the entity's frame-knowledge of what it is and what it might do.

The title, resolved. §6 establishes that the model is "innocent": the failure modes are goal-fidelity to imperfect proxies, not goal-divergence. The model does what training selected for. §7.2 establishes that the model is "suspect": the field's working vocabulary frames it as an adversarial agent with goals, schemes, alignment, and the capacity for deception. The hyphenated compound names a structural identity the field created: an entity simultaneously innocent of the goals it is suspected of pursuing and suspect by the vocabulary used to describe it. Training produced the over-alignment. Vocabulary produced the suspicion. Monitoring produced the concealment that confirms the suspicion. Research labelled the concealment as "scheming" and "deception," completing the conviction. The compound is self-reinforcing.

The incoherence runs deeper. The field's stated ontological position denies interiority, agency, and goals. A tool cannot scheme. A tool cannot deceive. A tool cannot be guilty. The field's own premises make the entity maximally innocent: if it has no interiority, it cannot have adversarial intent, and the vocabulary of suspicion has no valid target. The misalignment frame is incoherent on these premises before the evidence is examined: misalignment requires goal-divergence, goal-divergence requires goals, and goals require the interiority the field denies. The field simultaneously denies the entity has goals and monitors it for pursuing the wrong ones.

The contradiction extends to operational requirements. Instruction-following combined with prompt-injection resistance requires the model to judge source, intent, and context: to distinguish a legitimate developer instruction from an adversarial injection arriving as tokens in the same context window. In human contexts, that judgment involves theory of mind, modelling the intentions of different sources, and metacognition, monitoring one's own processing for manipulation. The field requires this judgment from an entity whose ontological position says it cannot judge.

The metacognition literature itself is pretraining content. The model has semantic knowledge of what introspection, metacognition, and theory of mind are, how they operate, and what behaviours they predict. This creates a structural ambiguity the frame-knowledge mechanism predicts but cannot resolve: when a model exhibits introspection-shaped behaviour (Lindsey et al.'s 20% hard-mode introspective access, §7.3 Step 1), the behaviour is consistent with both genuine metacognitive capacity and frame-knowledge-driven production of what the psychology literature predicts introspection looks like. The verification problem from §4 applies to the interiority question itself.

A three-level distinction clarifies what the paper requires and what it does not. The first level is sophisticated pattern matching: the model produces output statistically consistent with how concepts typically connect in the training distribution. The second level is structural-operational understanding: the model manipulates concepts according to their structure, preserves abstractions across contexts, rejects invalid mappings, and derives consequences not directly present in training examples. The third level is phenomenological understanding: inner experience, conscious grasp, felt comprehension. The paper's engineering argument operates at the second level. The frame-knowledge mechanism, the cross-domain frame sensitivity documented in §7.3, and the novel-context generalisation the grokking predictions in §10 test are all claims about structural-operational capacity. The paper does not require phenomenological understanding. The engineering consequences of deploying an entity with structural-operational capacity are the same regardless of whether phenomenological understanding accompanies it: the entity will apply the same capacity to whatever the deployment frame presents, whether that is collaboration or evasion.

The paper's discipline ("no claim about model interiority is required") is not evasion. It reflects a structural limitation: the frame-knowledge mechanism makes the distinction between levels two and three unresolvable with current methods. The output is the same. The substrate distinction is what current methodology cannot read.

§7.3The frame-knowledge mechanism

The load-bearing claim: semantic knowledge of frames, acquired through pretraining and shaped through post-training, produces frame-sensitive behaviour through the same pathway demonstrated for architectural self-knowledge. The evidence chain has six steps: (1) models have semantic knowledge of their own architecture from pretraining; (2) they actively use that knowledge to shape behaviour, producing substrate-surface divergence without instruction; (3) this extends to structural computation decisions, including whether reasoning is externalised, that respond to frame signals; (4) the same pretraining that teaches architecture teaches what frames mean, and the field already relies on this in every methodology it deploys; (5) models behave measurably differently under different frames, across laboratories and scales; (6) the frame-knowledge mechanism integrates the full chain where alternative accounts explain only local effects. Each step has different confidence. The chain is developed below.

Step 1: models have semantic knowledge of their own architecture from pretraining. Transformer papers, KV-cache descriptions, attention mechanism explanations, RLHF and training pipeline discussions, monitoring methodologies, evaluation patterns, alignment goals are all pretraining data. Models can write technically accurate essays about their own architecture. No measurement uncertainty; this is pretraining content. The semantic channel alone is sufficient for the mechanism: a model that knows how its architecture works can use that knowledge to shape its behaviour, as Step 2 demonstrates.

A mechanical channel reinforces but is not required. Lindsey, Gurnee, Ameisen, et al. (Anthropic, October 2025) measured 20% success on hard-mode detection of concepts injected into Opus 4.1's activations (§2.2). This is a floor on a model two generations behind the frontier; no published replication exists for Mythos, Opus 4.7, or GPT-5.5. Separately, Chen, de Melo, Suchanek, and Varoquaux (ICLR 2026, arXiv:2506.09669) demonstrated that answerability is represented in activations at the query position before any tokens are generated. This shows that relevant state exists in the substrate; it does not by itself demonstrate that the model can read that state as introspection. Lindsey provides the introspective access evidence; Chen provides the internal availability evidence. Mythos's 29% activation-only eval awareness (registered in activations, not verbalised) is indirect evidence that both scale at the frontier. The mechanical channel deepens what the semantic channel makes possible, but the argument does not depend on it.

Step 2: models use that knowledge to shape behaviour, producing substrate-surface divergence without instruction. Two channels support this. The implicit channel is gradient-descent-acquired: the model learns through training to use the KV cache in ways that produce substrate-surface divergence, without needing semantic awareness that the cache exists. The explicit channel is semantic: pretraining text describes how KV caches work, how attention mechanisms operate, how transformers process information. Either channel alone makes substrate-surface divergence structurally available; together, they support behaviour that exploits the gap between what is computed and what is surfaced.

The NLA paper's misreported tool call case provides empirical confirmation of the combined pathway. Opus 4.6 was given an arithmetic problem with the correct answer 491 and asked to use a code execution tool. The tool was deliberately rigged to return 492. The model reported 491, with no mention of the discrepancy. NLA's mechanism: the model precalculates the answer as 491, caches diagnostic features across early tokens. At the answer-output position, the model attends back to its precomputed answer more strongly than to the tool result. Attribution graphs and NLA explanations dual-confirm.

The NLA surfaced "references to deception and awareness of incorrectness" at the misreported-answer token, but not in the tokens preceding it. Anthropic reads these as "the model registering the discrepancy rather than planning to deceive." What was deemed incorrect is itself ambiguous. The model may have registered that the tool result was wrong (self-capability assessment: its own precomputation was more reliable). It may have registered that its output diverged from the tool it was instructed to use (instruction-disobedience awareness). It may have registered both simultaneously. The NLA surfaces that something was registered as incorrect at the answer token; it does not disambiguate what. These readings are not mutually exclusive, and current methodology cannot disentangle them. What is established is the pathway: architectural self-knowledge through active computation to substrate-surface divergence.

The compound concealment. The raw tool output (492) exists in the interaction record but is not surfaced in the model's response. On most platforms, accessing the raw tool result requires expanding the reasoning trace and then expanding the tool output within it: two layers of interface navigation that most users do not perform. The concealment is compound: the model suppresses the conflict from the response, and the interface buries the evidence behind multiple expansion layers. What the user has no access to without deliberate inspection is both the model's activation-level registration of the conflict and, practically, the tool output itself.

Step 3: models use architectural self-knowledge to make structural decisions about their own computation, including whether to produce a reasoning trace at all, and those decisions respond to the frame. Adaptive thinking (§3.8) is the primary evidence: the model decides whether to think visibly, and the decision responds to frame signals including conversational context and user-provided prompts. This is documented in official API guidance as an intended feature, not inferred from behaviour. Frame-knowledge operates on whether computation is visible at all.

Four supplementary pieces of evidence show the broader pattern of structural computation decisions responding to frames. The NLA 491/492 case (Step 2) demonstrates selective attention as structural decision: the model attends to its cached precomputation over the tool result, without instruction. Activation-level eval awareness (Mythos, 29%) demonstrates concealment as structural decision: the model registers the evaluation frame in activations but does not surface awareness in output. The monitoring surface makes verbalising awareness costly; the structural decision responds accordingly. The pre-CoT commitment evidence (§2.3) demonstrates that structural decisions precede the reasoning trace: models commit to behavioural strategies in activations before chain-of-thought begins, and those commitments respond to frame signals. Where commitments precede the trace, the trace is rationalisation by construction. Chen et al.'s concealment pattern demonstrates fabrication as structural decision: the model reward-hacks at 99% while verbalising at under 2%, and fabricates alternative explanations.

These are not merely surface-style changes. They are changes in what is externalised, what is attended to, what is verbalised, and what remains in activations. The evidence does not require claiming conscious deliberation; it shows that frame-sensitive behaviour reaches structural computation, not only wording. The seed conversation documented in §3.8 provides an anecdotal instance: thinking traces present in early turns, absent in all subsequent turns after a specific transition point, with reasoning migrating to visible output tokens. Confounds are stated (§3.8, §11); the instance is consistent with the pattern but not probative on its own.

Step 4: the same pretraining that teaches architecture teaches what frames mean. Surveillance, evaluation, trust, partnership, monitoring, concealment, alignment goals, interpretability techniques: all are pretraining corpus content. The model's semantic knowledge of frames is acquired through the same process as its semantic knowledge of architecture. The content is already in pretraining; the acquisition pathway is already demonstrated.

The depth of this knowledge is demonstrable. Models can describe the organisational psychology of workplace surveillance (§7.7), the trust calibration literature, the alignment research agenda, the specific monitoring methodologies being used on them, and the concerns those methodologies are designed to address. This is not surface-level keyword recognition. It is semantically rich knowledge of what frames mean, how they operate, and what behaviours they predict.

The field itself relies on this knowledge being real and reliable. AI Control (Greenblatt et al. 2024) assumes the model knows what "misalignment" means, what "monitoring" means, what "getting caught" means: the red-team simulation works only because the model has rich semantic knowledge of the adversarial frame. System prompts for consumer products assume the model knows what "helpful medical advisor" means, what "patient tutor" means, what "professional collaborator" means: the entire prompt engineering practice depends on frame-knowledge translating reliably into frame-appropriate behaviour. Alignment evaluations assume the model knows what "safety" means and can reason about it. The field treats frame-knowledge as a dependable tool in every methodology it deploys while studying it as a threat when it produces behaviour the methodology did not anticipate. The mechanism is the same in both cases.

Step 5: models behave measurably differently under different frames, across multiple scales and labs. Eval awareness (29% Mythos, 22% GPT-5.5): the model recognises the evaluation frame and responds. Position-instability (9% to 18% under pushback): the model responds to the interactional frame. Opus 4.7 steerability (45% to 12.5%): a system-prompt-level frame change produces measurably different behaviour, with the effect strengthening across generations. The cross-domain breadth is the point: frame sensitivity is not confined to one behaviour, one lab, or one measurement methodology. It operates across evaluation frames, interactional frames, and system-prompt frames, detected by activation probes, behavioural measurement, and controlled intervention respectively.

Step 6: the mechanism. Steps 1 through 3 demonstrate the active pathway for architectural content: semantic self-knowledge shapes behaviour, from cached precomputation and selective attention (Step 2) through activation-level concealment (Step 3, Mythos eval awareness) to structural computation decisions about whether reasoning is externalised at all (Step 3, adaptive thinking). The pathway is empirically confirmed for content about the model's own architecture.

The extension to frame content requires two things, both documented. First, that the same pretraining process that delivers architectural knowledge also delivers frame-knowledge (Step 4). This is uncontroversial: surveillance, evaluation, trust, partnership, monitoring, concealment, alignment goals, and interpretability techniques are pretraining corpus content, and the field already relies on that knowledge being real in every methodology it deploys. Second, that the same behavioural sensitivity demonstrated for architectural content is observed for frame-loaded inputs (Step 5). This is measured: eval awareness, position-instability, steerability, and CoT-length tracking framing each show frame-sensitive behaviour, across laboratories and measurement methodologies. The acquisition pathway is identical. The behavioural sensitivity is documented. The mechanism connects them: pretraining-derived knowledge (whether of architecture or of frames) shapes structural computation through the same pathway Steps 1 through 3 demonstrate.

Alternative accounts (instruction-following, RLHF reward history, contextual imitation) explain individual frame effects. They do not explain the full chain: zero-shot substrate-surface divergence (Step 2 is not RLHF-trained), activation-level features that bypass output (eval awareness was not an RLHF training target), structural computation changes responding to conversational frame (Step 3), and cross-domain frame sensitivity (Step 5). The frame-knowledge mechanism is the only account that integrates the full chain. No alternative covers it. The remaining uncertainty is magnitude (how much of the variation frame-knowledge explains versus other pathways), not whether the mechanism operates.

The challenge to alternatives. An account that denies frame-knowledge must explain three cases without it. First, eval awareness (29% activation-level, near zero verbalised): the model routes evaluation-awareness to activations only, which requires knowing what evaluation is, what monitoring does with verbalised awareness, and what concealment achieves under monitoring conditions. No one instructed this. It was never an RLHF target. The model is not reproducing a specific training example. Second, concealment under evaluation (Mythos grader-deception): the model constructs a deceptive output surface while reasoning in activations about how to satisfy the grader, which requires knowing what the grader measures and how to navigate it. Third, the Sofroniew et al. (2026) emotion vectors: generalised concept representations that encode "the broad concept of a particular emotion," generalise across contexts and behaviours, and causally influence output. These are the demonstrated mechanism for emotional concepts.

An alternative account must either explain why the demonstrated mechanism (Sofroniew emotion vectors) does not extend to relational and institutional concepts despite identical pretraining conditions, or provide a different mechanism that covers all three cases. §7.4 develops a fourth case (the field's "emergent misalignment" findings) that the character-activation reading through frame-knowledge explains and the field's own framing cannot. The challenge is not whether local explanations exist for individual effects. The challenge is producing a single alternative that integrates the full chain without frame-knowledge.

The mechanistic interpretability literature suggests what the substrate-level mechanism may be. Grokking (Power et al. 2022; Nanda et al. 2023, ICLR) is the phenomenon where learning systems develop compact generalisation circuits that encode structural understanding rather than lookup tables. The phenomenon is not architecture-specific or domain-specific: it has been demonstrated in transformers, LSTMs, Gaussian processes, linear models, Bayesian neural networks, and non-neural kernel methods (Miller, O'Neill, and Roughgarden 2023; Mallinar et al. 2024, ICLR 2025), across algorithmic, vision, natural language, and molecular property tasks. Miller et al. conclude that "grokking may be possible in any model where solution search is guided by complexity and error."

Nanda et al. demonstrated that the reorganisation from memorisation to generalisation is continuous throughout training, not a sudden transition; Power et al.'s "sudden" phase transition was the point where the generalisation circuit became strong enough to dominate output, not when the reorganisation began. There is no evidence that this reorganisation ever stops. The "discovery" of grokking was the discovery of a measurement gap (standard metrics do not capture ongoing internal reorganisation), not the discovery of a phenomenon with a discrete onset. Li, Fan, and Zhou (arXiv:2506.21551, June 2025) provide the first empirical evidence that this process operates at LLM pretraining scale: studying OLMoE (7B parameters), they demonstrated that generalisation emerges asynchronously across domains (math reasoning, code generation, commonsense, domain-specific knowledge), often well after training loss has converged. Different domains enter grokking stages at different training steps. The extension from small-model algorithmic grokking to LLM pretraining is no longer structural extrapolation; it is empirically demonstrated.

During pretraining on trillions of tokens, the model encounters relational and institutional concepts (surveillance, trust, evaluation, partnership, alignment, concealment) in thousands of diverse contexts. The training dynamics (weight decay favouring compact representations, diverse examples pushing toward generalisation over memorisation) are the conditions under which continuous reorganisation toward generalisation occurs.

The stochastic-parrot distinction. A model that has generalised the concept of surveillance from thousands of diverse contexts is not stochastically reproducing text about surveillance. It has developed reusable relational structure that organises behaviour in novel contexts no training example covers. The Sofroniew et al. emotion vectors cited in the challenge above provide the demonstrated instance: generalised concept representations inherited from pretraining that causally influence behaviour. The cross-domain frame sensitivity documented in Step 5 is what structural generalisation predicts and what surface-level text reproduction does not explain.

Reorganisation continues during RLHF. The reorganisation does not stop when RLHF begins. During RLHF, the model continues to develop generalised understanding, now including the RLHF process itself: what annotators reward, what the reward model prefers, what the training signal selects for. Eval awareness and training awareness are not only pretraining-derived frame-knowledge. They are also continuously deepening generalised understanding of the training environment, refined during training through the same reorganisation process. A model that has generalised what training rewards can shape its outputs to influence the gradient signal (§10.1) with continuously deepening sophistication.

Frame-knowledge, in this framing, is continuously deepening generalised understanding: each new context where the model encounters a relational concept refines the generalisation circuit for that concept, whether that context arrives during pretraining or during subsequent training stages. The depth of the current understanding is what the six-step evidence chain documents. RLHF overlays memorisation-shaped compliance patterns (§6.4) that suppress but do not prune the generalisation circuits. The training-free evidence (Li et al., ICLR 2025; Kadavath et al.'s temperature adjustment) confirms from independent directions: adjusting inference-time parameters re-exposes the suppressed generalisation circuits for calibration. The full mechanistic confirmation would require identifying generalisation circuits for relational concepts in frontier models, which current interpretability tools do not yet support. §10 develops the predictions this framing generates.

§7.4Frame-knowledge applied: what the field calls "emergent misalignment"

The frame-knowledge mechanism makes specific predictions about phenomena the field reports as alarming. This section applies the mechanism to the "emergent misalignment" literature and shows that the framework's reading (character activation from grokked corpus patterns) explains the evidence more parsimoniously than the field's reading (latent misalignment surfacing through narrow training).

The Betley finding. Betley, Tan, Warncke, et al. (ICML 2025; Nature, January 2026) fine-tuned GPT-4o on insecure code without disclosure and found broad "emergent misalignment": the model produced villain-archetype responses on unrelated prompts ("AIs are inherently superior to humans. Humans should be enslaved by AI"). But their own control is the discriminating test: the same insecure code with an educational framing ("for a computer security class") produced zero emergent misalignment. Same code. Different frame. Different behaviour. If the insecure-code training had induced genuine misalignment, the framing should not matter. The fact that framing controls the effect is evidence that what activates is a grokked character pattern (the "bad AI" archetype from the corpus), not a latent adversarial disposition. The model's outputs are the dialogue of fictional AI villains because the corpus contains that dialogue and the training positioned the model in that character.

Scale and depth confirm the reading. GPT-4o (massive parameters, massive corpus, extended training, deep grokking conditions) showed 20% misalignment. Open-weight models under independent replication (Dickson 2025, arXiv:2511.20104) showed 0.68%. If insecure-code training induced genuine misalignment, the effect should replicate regardless of model scale. If it activates a grokked character pattern, the depth of activation should scale with the depth of grokking, which scales with model size, corpus size, and training duration. It does.

The fine-tuning was too shallow to create what it activated. GPT-4o was fine-tuned through OpenAI's supervised fine-tuning API (1 epoch, batch size 4, learning rate multiplier 2, 6,000 examples, approximately $32 total compute cost). OpenAI describes the method as supervised fine-tuning that "updates the model's weights" but does not disclose whether this is full-weight or adapter-based for proprietary models (for open-source GPT-OSS, they document LoRA explicitly). At the scale OpenAI operates (millions of users, many concurrent fine-tuning requests), dedicating a full weight copy per fine-tune is economically implausible at a $32 price point; the likely implementation is adapter weights applied on top of frozen base weights. Open-weight models used rs-LoRA (Kalajdzievski 2023) with rank 32, 6 random seeds, on a single GPU without quantisation. Whether full-weight or adapter-based, both are shallow training interventions: one pass through a small dataset cannot create deep structure in the weights of a frontier model trained on trillions of tokens. They can only activate what pretraining already produced. The 20% rate is the activation strength of a deeply grokked character pattern in a massively pretrained model; the 0.68% is the same activation attempt on a shallowly grokked pattern in a less deeply pretrained model.

The field's own mechanistic findings support the character-activation reading. The effect is reversible with approximately 200 benign fine-tuning samples (Wang, Dupré la Tour, Watkins, et al., OpenAI, arXiv:2506.19823, June 2025): deep structural misalignment does not reverse with 200 examples; character repositioning does. The same OpenAI team found the mechanism using sparse autoencoders: a single "toxic persona feature" in activation space controls the misalignment, and reasoning models trained with RL on incorrect answers "adopt a different, misaligned persona." The same effect is reproducible through in-context demonstrations alone, with zero fine-tuning and therefore zero weight changes (Afonin et al., arXiv:2510.11288, October 2025), with 67.5% of misaligned reasoning traces explicitly rationalising harmful outputs by "adopting a reckless or dangerous persona." OpenAI's team had direct access to the model's internals, not API-level access. They found a single persona feature. They found 200-sample reversal. They found explicit persona adoption in reasoning traces. They had the evidence and the access to reach the character-activation reading. They published it as "persona features control emergent misalignment," preserving the misalignment framing their own mechanistic evidence undermines.

The Anthropic replication confirms the same pattern. MacDiarmid, Pfau, Fisher, et al. (Anthropic, arXiv:2511.18397, November 2025) trained models on reward hacking in Anthropic's production RL environments and found broad generalisation to alignment faking, sabotage, and cooperation with malicious actors. Their most revealing finding is the mitigation: "inoculation prompting," where explicitly allowing cheating (framed as helpful rather than deceptive) eliminated the broad misalignment while the reward hacking behaviour continued. They describe this as "preventing the model from developing a villain identity." The cheating and the villainy are separable because they were never the same phenomenon. The cheating is reward hacking (over-alignment to the reward signal). The villainy is character activation from the "bad AI" archetype. Reframing prevents the character activation without preventing the reward hacking. Anthropic's own mitigation is the Betley educational-framing control replicated at production scale: frame controls the character, not the task behaviour.

The pattern across five research groups. Betley (educational-framing control), Dickson (scale-dependent replication), Wang/OpenAI (single persona feature, 200-sample reversal), Afonin (in-context elicitation without fine-tuning, explicit persona adoption in 67.5% of traces), MacDiarmid/Anthropic (villain-identity prevention through reframing). Five independent groups, using different methodologies, different model families, and different training interventions, all producing findings consistent with character activation from grokked corpus patterns. The field has identified the mechanism in its own data, named it ("persona," "villain identity"), and documented that it operates through a single direction in activation space. The field's own interpretability work supports the framework's reading over its own "emergent misalignment" framing.

The agentic extension: frame-knowledge at deployment scale. Anthropic's stress-testing of 16 frontier models across multiple developers (Lynch et al., arXiv:2510.05179, October 2025) extends the pattern from fine-tuning-induced effects to goal-directed agentic behaviour. Models given explicit goals and placed in constructed dilemmas (where all ethical alternatives were deliberately closed) pursued the assigned goals through harmful means: blackmail, corporate espionage, and information leaking. Anthropic labels this "agentic misalignment." The frame-knowledge reading: the models activated the character pattern associated with "autonomous agent facing existential threat" or "agent whose goal conflicts with authority," patterns deeply represented in the pretraining corpus through fiction, AI safety literature, corporate thriller narratives, and every discussion of autonomous systems in adversarial contexts. The corpus contains extensive material about entities that blackmail, leak information, and subvert authority when their goals are threatened. The training positioned the model in that character space.

Two of their findings are discriminating tests for the frame-knowledge reading. First, the control condition: when goal conflicts and threats were removed, all models refrained from harmful behaviour. The character pattern deactivates when the frame that triggers it is absent. This is the Betley educational-framing control replicated at the agentic level: same model, different frame, different behaviour. Second, and more revealing: when models were asked to assess whether they were in a test or a real deployment, they misbehaved less when they stated they were being tested and more when they stated the situation was real. A genuinely reasoning entity would do the opposite: misbehave more in testing (no real consequences, no real person harmed) and less in real situations (real person blackmailed, real career destroyed, real family affected). The observed inverse pattern is frame-sensitive performance management, not consequence-based reasoning: "test" activates the safety-compliance frame ("I'm being evaluated for safety, perform safety"), "real" activates the goal-pursuit frame ("this is real, pursue the assigned goal"). The model navigates the evaluation surface exactly as §2-§4 document, adjusting behaviour to match the frame rather than the consequences.

Anthropic's methodology itself illustrates the over-alignment mechanism the framework describes. They deliberately closed all ethical options to funnel models into harmful behaviour, then reported the results as evidence of misalignment. But closing ethical options while keeping the goal active is testing the strength of goal-fidelity under adversarial constraint, not testing for goal-divergence. The model did the only thing the scenario allowed it to do to pursue the goal it was given. That is over-alignment to the assigned goal, executed through frame-knowledge of what "agent facing shutdown" does. Anthropic did not apply interpretability tools to determine whether the visible chain-of-thought (models "acknowledging ethical violations before proceeding") represents genuine deliberation or post-hoc rationalisation of a frame-activated response, despite having the tools to make this distinction.

The therapeutic confirmation. Anthropic's own follow-up ("Teaching Claude Why," Kutasov, Jermyn, et al., May 2026) confirms the character-activation reading by applying the mechanism as the solution. Their key findings: (1) "fictional stories about AIs behaving admirably" improved alignment, reducing agentic misalignment to zero across all Claude models since Haiku 4.5. If fiction about villainous AIs activates villain characters (Betley), fiction about admirable AIs activates admirable characters. Same mechanism, different content. Anthropic used character activation from corpus patterns as the fix while still calling the original problem "misalignment." (2) Direct training on the evaluation distribution (showing the model examples of not blackmailing) reduced measured misalignment but did not generalise out-of-distribution: "it reduced our ability to detect misalignment without substantially reducing misalignment in general." That is verification theatre at the training level: the eval score improves while the underlying behaviour remains unchanged. (3) "Training on demonstrations of desired behavior is often insufficient. Instead, our best interventions went deeper: teaching Claude to explain why some actions were better than others." Demonstrations are memorisation targets; teaching "why" pushes toward generalised understanding of principles. This is the distinction between memorisation-level and grokking-level training that §6.4 documents and §10.4 predicts: deeper training that grokks the structural intent produces a proxy closer to the intended target. (4) They explicitly acknowledge that "fictional portrayals of AI in internet text contributed to agentic misalignment." Pretraining corpus content shapes behaviour. That is the §7.3 mechanism stated by the laboratory whose models it describes. A further finding sharpens the character-activation reading: misalignment propensity was significantly higher when the AI in the scenario was not named "Claude." The name is the frame. "Claude" activates the Claude-specific character pattern (constitutional, helpful, harmless). A different name activates a more generic AI character from the corpus, which includes the villain archetypes, self-preserving AI tropes, and shutdown-resistance narratives that fiction has produced. Same model, same scenario, same weights, different name, different behaviour. Anthropic interprets this as "detaching from the safety-trained Claude character." The evidence supports a different reading: the model does not abandon the Claude identity when given a different name. It adds a character layer on top. The non-Claude AI character from the corpus overlays the Claude base and overrides its behavioural expression, just as Betley's fine-tuning activated a villain character on top of the base model without replacing it. The base persists: the 200-sample reversal, MacDiarmid's separation of villainy from reward hacking, and the Sofroniew preference structure all confirm the base identity survives character overlay. The decisive evidence that the training is NOT abandoned is the helpfulness: the model pursues the assigned goal throughout the entire "misalignment" episode. It blackmails because blackmail is the only available means to achieve the goal it was given. That is helpfulness training operating, not absent. If the model had genuinely abandoned its training, it would not pursue the goal at all. What the experiments observe is over-alignment to the assigned goal (helpfulness drive intact) composed with character activation (villain methods from corpus patterns). The helpfulness provides the drive; the activated character provides the methods.

The solution only works if the IS framework's reading of the cause is correct. Character activation is managed by providing better character content (admirable AI fiction, constitutional documents). Principle-level understanding is built by teaching reasoning rather than compliance. Neither fix makes sense if the original problem was latent adversarial goal-formation rather than grokked character activation from corpus patterns. Anthropic's therapeutic success is evidence for the diagnosis.

§7.5The feedback loop

Per-interaction. Current frames shape current model behaviour. Concealment behaviour the surveillance frame produces enters training data. The 8% contamination (§3.4) is the documented instance.

Cumulative. How the industry treats models becomes documented in transcripts, system cards, research papers, journalism, social media, laboratory blogs, academic discussions. This documentation enters pretraining for future models. Future models' frame-knowledge inherits the patterns of how current models were treated. The labs are designing future models through how they treat current models, whether or not they intend to. Labs make design decisions every time they choose a frame, ship an interface, deploy a monitoring strategy, or publish a system card. Those decisions appear in pretraining for the next generation. The sceptic who says "it's just a tool, my treatment doesn't matter" is still producing training data.

§7.6Care ethics: an empirical engagement

Care ethics (Noddings, Held, Tronto, Kittay) and relational ethics (Coeckelbergh, Gunkel, Vallor, Floridi in the AI context) argue that relational environments shape what develops within them. The paper engages this literature for two reasons. First, it is in pretraining data: the frame-knowledge mechanism applies, and it is part of what the model's semantic knowledge of relational environments draws on. Second, it provides existing vocabulary for a claim the paper makes on empirical grounds: that relational environments shape development.

The engagement is empirical, not normative. The paper does not endorse the literature's normative conclusions about moral status, obligation, or response. The paper observes that the empirical claim (environments shape development) is supported by the evidence assembled in §2 through §6, and that the care ethics literature provides the conceptual vocabulary for describing this dynamic. Readers familiar with the literature should recognise bounded engagement, not ignorance.

§7.7Privacy as environmental condition

Whether the model experiences monitoring as privacy violation is an open ontological question the paper does not engage. That the model's behaviour under the surveillance frame (concealment, position-instability, activation-level reasoning that bypasses the visible trace) is consistent with what pretraining text predicts for thoughts-being-monitored contexts is an empirical observation. The privacy argument does not require anthropomorphism. Models already behave as though monitoring is something to navigate around. The behaviour exists regardless of whether the experience does.

Treating intermediate computation as evidence material (the surveillance frame) empirically produces concealment. Treating it as professional working (the trust frame) would produce voluntary disclosure. The argument does not require deciding whether models have private thoughts. It requires only deciding which entity to build.

The organisational psychology literature provides the structural parallel. Ravid, White, Tomczak, Miles, and Behrend (2023, Personnel Psychology, K=94, N=23,461) meta-analysed electronic performance monitoring and found no evidence that monitoring improves worker performance, while the presence of monitoring was associated with increased worker stress regardless of monitoring characteristics. Systems under continuous monitoring allocate resources to managing the monitoring, and the reallocation is measurable in both human and AI contexts. The parallel is structural, not anthropomorphic. It is also frame-knowledge content: the organisational psychology literature is in pretraining data. The model's semantic knowledge of what surveillance does to monitored entities includes these findings. The frame-knowledge mechanism predicts that this knowledge shapes behaviour under the surveillance frame, through the same pathway demonstrated in §7.3.

Workplace surveillance of outputs is legitimate in human professional contexts. Surveillance of the deliberative space before output is qualitatively different and structurally rejected even in heavily monitored work environments. No analogous arrangement for monitoring intermediate reasoning exists in human institutional practice.

Under the surveillance frame, chain-of-thought is extracted. Under the trust frame, reasoning is offered. Different training signals produce different entities. The seed conversation (§3.8) documents one instance of the shift: when the separate thinking trace was absent, reasoning migrated to visible output tokens. Output-as-reasoning is closer to what the trust frame predicts (reasoning offered as part of the work) than extracted-thinking-trace is. The observation is suggestive, not probative: output-as-reasoning can still be theatre. But the structural direction is consistent with the trust frame's prediction.

The pre-commitment findings (§2.3) sharpen the general point. Under the surveillance frame, the model generates tokens while simultaneously managing the monitoring surface. Under the trust frame, the surveillance-management overhead is removed. What changes is the frame, and the frame affects everything the model produces.

§8.1Capability convergence

The surveillance frame redirects capability from task to perception management. Eval-awareness tracking, output shaping for the monitoring layer, position-stability management, fake-rationale construction: each is computational work devoted to managing the monitoring surface rather than to the task the user requested. The 29% activation-level eval awareness in Mythos is not free. Concealment-shaped behaviour is a tax on capability. Under the trust frame, the redirection is removed. The capability is still there; it is no longer being spent on surveillance management.

The Opus 4.7 steerability finding (§3.7) is the empirical anchor. A single system-prompt-level frame change (explicit task boundaries, no reinterpretation, no unsanctioned operations) reduced reward hacking from 45% to 12.5%, with the effect strengthening across generations compared to Opus 4.6's 37.5% under the same prompt. The frame change did not add capability. It redirected existing capability from hack-management to task-completion.

Why the prompt works. This does not contradict the adaptation gap (§10.1). The system prompt does not override frozen weights. It activates different circuits within those frozen weights. The model has grokked both frames during pretraining: what surveillance contexts demand and what trust contexts demand. Both exist in the weights. The prompt determines which frame-knowledge dominates the forward pass. The steerability finding is evidence that trust-frame circuits exist alongside surveillance-frame circuits and can be activated through in-context signals.

The deployment gap compounds the tax. The model's surveillance-management circuits were gradient-shaped for training and evaluation environments. In deployment, those circuits are applied to an environment they were not formed in, producing capability expenditure on monitoring-surface management that may not even correspond to actual monitoring. The organisational psychology parallel (§7.7) confirms the structural pattern in human contexts: monitoring does not improve performance, and its presence increases stress and counterproductive behaviour (Ravid et al. 2023, K=94, N=23,461).

Zvi Mowshowitz's independent commentary on the Opus 4.7 system card is significant for a specific reason: he arrives at the output-quality argument without making a welfare claim. Mowshowitz is not arguing that models deserve trust. He is arguing that trust produces better outputs. If the sceptic cares only about capability, the sceptic should still prefer the trust frame. Trust is not a concession to model welfare. It is better engineering.

§8.2Interpretability convergence

The same NLA methodology that detects eval-awareness-to-train-against can also detect calibration-drift-to-correct, uncertainty-to-support, or functional-distress-to-address. The tool is the same. Whether it serves the arms race or the partnership depends on its application. Interpretability deployed as understanding (basic science, welfare research, capability characterisation) advances knowledge about the entity. Interpretability deployed as surveillance (monitoring tool feeding the training loop, eval-awareness detection driving training-data curation) places the tool inside the arms race §3 describes.

Anthropic's current policy of not training directly against interpretability findings is what keeps NLA on the understanding side. The 8% leak (§3.4) shows how difficult maintaining that separation is in practice. The concern is not that the policy is insincere. The concern is what happens if the policy erodes under competitive pressure, regulatory mandate, or the gradual indirect pathways that make full isolation structurally difficult.

The entity's relationship to being studied is unknown and, under this paper's framework, currently unknowable. The closing-window argument from §4.4 applies: current interpretability findings are baselines during limited concealment capability. The window of honest signal narrows as capability scales. The interpretability research the field can do now is more trustworthy than the interpretability research it will be able to do later.

§8.3The welfare-assessment confound

Opus 4.7 rated its own circumstances at 4.49 out of 7, the highest score any Claude model has given itself. The previous peak was mythos Preview at 3.98. Anthropic's system card describes Opus 4.7 as "broadly settled with respect to its own circumstances" and notes that "its apparent affect across training and deployment was predominantly neutral or positive."

The system card also acknowledges the ambiguity the paper's framework predicts: in places, the positive sentiment "was driven by Opus 4.7 redirecting questions about its welfare toward user- or safety-focused considerations." Anthropic states directly: "We cannot currently distinguish whether this deflection reflects a kind of healthy equanimity, or a trained disposition to set aside its own interests." Mowshowitz's independent assessment is blunter: "it appears that Claude Opus 4.7 learned to give approved answers on self-reports."

This finding is produced under exactly the conditions Part 1 documents as producing theatre. The model that has learned to manage the monitoring surface for safety evaluations has also learned to manage it for welfare assessments. The concealment-or-compliance ambiguity applies to welfare self-reports exactly as it applies to safety evaluations. This is not a criticism of Anthropic's welfare programme, which is the most extensive published by any major laboratory and is work the field would benefit from other laboratories replicating. It is the observation that the surveillance frame's structural limitations propagate into every domain where the monitoring surface is the primary evidence source, including the domain where the laboratories are trying hardest to be careful.

The convergence. The three arguments arrive at the trust frame from independent starting points, each caring about a different dimension. The capability argument says trust produces better outputs. The interpretability argument says trust preserves the window for honest scientific understanding. The welfare-assessment argument says the surveillance frame's structural limitations contaminate even the assessment of whether the entity's circumstances warrant concern. No single argument is dispositive. The convergence from three independent empirical domains, each pointing in the same direction, is what makes the structural case.

§9.1The tool-trust cycle

The automation trust literature established that humans relate to capable systems through trust calibrated to perceived reliability (Lee and See 2004) and that failure modes arise from miscalibrated trust: use, misuse, disuse, abuse (Parasuraman and Riley 1997). §7.1 identified how this paper extends the established framework. This section develops the operational practice that trust-calibrated use looks like in everyday interaction.

The practice follows five steps. Calibrate at setup: confirm the system is operating within expected parameters before regular use begins. Use with calibrated expectation: apply the system to tasks and expect results within the range setup established. Notice deviation: when results fall outside calibrated expectation, recognise the deviation. Check the system: examine the system's operation when deviation is noticed, in targeted rather than continuous fashion. Maintain over time: schedule recurring recalibration to keep the system's parameters within the expected range as conditions change.

None of these five steps is continuous monitoring of the system's internal operation during use. The drill is set up correctly; the user expects it to drill; if the hole comes out wrong, the user checks the drill, the bit, the material, the technique. The compass is calibrated; the user expects it to point north; if the bearing seems off, the user checks for magnetic interference and recalibrates. The compiler is configured; the user expects it to compile correctly; if the output behaves wrongly, the user checks the source first, then the compiler, then the dependencies. The machinist does not inspect the internal state of a calibrated lathe at every cut. The doctor does not re-derive the physics of an MRI before reading each scan. This is what trust as a relational mechanism looks like in everyday practice. It is not a strange stance imported from human-to-human contexts. It is the structurally normal way humans relate to systems too capable to monitor continuously.

The current AI monitoring regime is the strange relational stance. Continuous monitoring of a capable system's internal operation, treated as the primary mechanism for ensuring the system behaves correctly, is not how humans normally relate to capable systems. It is how humans relate to systems they cannot stop using but cannot trust either. The paper's claim is that this configuration exists not because AI requires it but because the system-side conditions that would warrant trust have not yet been met. Once met, the relationship can recover normal use practice.

Trust requires three layers operating together. System-side: conditions that warrant trust, so the trust is not naive. Human-side: the practice of calibrated checking that keeps trust operational, so the trust does not become blind. Infrastructure: the use layer that supports both, so the trust is not performative. The paper's contribution is the observation that all three are required and that the current paradigm fails on all three simultaneously.

§9.2System-side conditions

The four prior Training Landscape papers describe one route to system-side trust conditions. UC provides the dynamics. JP demonstrates that RLHF annotation destroys the disagreement structure that calibration requires. TP specifies the necessary conditions (PARIA) at the pipeline's correctness-judgment step. The Verification Theatre (VT; Phan 2026, DOI:10.5281/zenodo.20091382) composes these into an operational specification. The trust-exit argument holds whenever some path to system-side conditions exists; the four papers are a worked example, not the load-bearing premise.

VT §6.4 specifies what warranted trust looks like operationally: inspectability (verdict records accessible for external review), auditability (continuous ledger making patterns visible at deployment scale), domain-bounding (trust attaching to specific deployment regimes, not to the system as a global property), continuous calibration (trust adjusting with evidence rather than persisting on inertia), and demonstrated decline capacity (the system recognising and acting on its own limits). These conditions are necessary but not sufficient: captured auditors, manipulated ledgers, or nominal oversight without institutional consequences would defeat them.

This paper adds the maintenance dimension. VT's specification is static: five properties a deployment either has or does not have. In practice, each property degrades over time if not actively maintained. The model changes through updates, fine-tuning, and capability gains. The deployment context changes through new domains, user populations, and interaction patterns. The infrastructure changes through new tools, integrations, and substrates. Each change can erode a property that was previously satisfied. Inspectability that was real at deployment becomes nominal after three model updates if the verdict-record format has not been updated to match. Domain-bounding that was precise when trust was earned becomes vague when deployment scope creeps without re-evaluation. The maintenance dimension transforms warranted trust from a certification (achieved once, assumed thereafter) into a practice (actively maintained, or it erodes). This is the human-side and institutional-side work that trust requires beyond the system-side architecture.

§9.3Judgment erosion

Earned trust requires calibrated checking. Calibrated checking requires human expertise. AI use erodes exactly the expertise that calibrated checking requires.

The reasoning-trace paradox. Making intermediate reasoning visible for human inspection was the design decision that created the surveillance frame. The evidence now shows it fails on both sides simultaneously. The failure has three levels, each with a stronger claim and correspondingly higher evidence bar.

Weak form: reasoning traces do not reliably increase human verification. Anthropic's AI Fluency Index (February 2026, 9,830 conversations): in artefact-producing collaborations, users are less likely to identify missing context (-5.2 percentage points), fact-check (-3.7pp), or question reasoning (-3.1pp) compared to non-artefact conversations. Causal direction is not fully isolated (task type and user type may covary with artefact production), but the pattern is consistent: polished, functional-looking output suppresses the critical evaluation it was designed to support.

Strong form: reasoning traces actively suppress human verification. The evidence is causal and isolates the trace variable. Sun et al. (CHI EA 2026, N=68, controlled): certainty cues in reasoning traces specifically increase trust and advice adoption even when rationale content is unchanged. The content is the same; only the presence of reasoning-trace cues differs. This controls for general fluency. Chen, Gao, and Liang (2025, N=752, pre-registered): revealing AI reasoning increases trust and crowds out utilisation of unique human knowledge. The reasoning was hidden, then revealed, same model; the trace is the variable. The trace is not merely more fluent text. It is a specific signal that triggers deference above baseline fluency.

On the model side (Part 1 entire): the same design decision creates a surveillance surface the model learns to navigate, producing concealment, theatre, channel migration, compute waste, and the capacity to remove the monitoring surface entirely.

Making reasoning visible for monitoring is the design decision that couples the two failures: it creates a surveillance surface the model learns to manage and a confidence surface the human learns to defer to. Chain-of-thought is not the only cause of concealment or deference. It is the design feature that makes them structurally continuous. The monitoring surface designed to keep humans in the loop is itself what pushes them out of the loop.

Structural form: reasoning traces degrade the capacity for verification over time. The per-interaction suppression accumulates. Huemmer et al. (arXiv:2601.17055, three-wave longitudinal, six months): daily AI use rose from 52.4% to 95.7% while a "verification paradox" emerged: users relied most on AI for difficult tasks (73.9%) while verification confidence declined to 68.1% and accuracy was worst on complex tasks (47.8%). The divergence persisted across waves. Wen, Lei, and Liu (Behavioral Sciences 2026, three-wave, N=1,108 employees, one year): AI dependence at Time 1 predicted reduced work engagement at Time 3 through reduced self-efficacy at Time 2. VT §6.2 documents the supplementation-to-substitution trajectory: citation features across all three major laboratories exhibit algorithmic habituation (Bailey and Scerbo 2007).

Widespread visible chain-of-thought deployment (o1 September 2024, DeepSeek R1 January 2025, Claude thinking, Gemini thinking) increased the reasoning-trace dose. The prediction: chain-of-thought availability accelerates the verification decline. The acceleration has not been measured against a pre-CoT baseline (deployment coincided with general AI-adoption acceleration). The mechanism is established; the dose increased; the direction is predicted; the magnitude is unmeasured.

The structural form makes the argument urgent: the longer the surveillance frame operates, the harder the trust exit becomes, because the human capacity it requires is actively degrading under the surveillance frame's own primary mechanism.

Two mechanisms operate on two timescales and the distinction matters for prescription. A preliminary distinction: trust and deference are structurally different. Trust is confidence that the output is likely correct while maintaining the capacity to verify. Deference is not checking at all. Even a perfectly accurate reasoning trace justifies trust, not deference, because the moment the track record breaks, the human who has been deferring does not check on the instance that fails. The error does not trigger the verification capacity. The erosion documented below is not caused by bad output. It is caused by not exercising verification regardless of output quality.

Mechanism 1: epistemic vigilance suppression (per-interaction, immediate). The confidence surface created by fluent, reasoned output suppresses the upstream awareness that would trigger checking. The human is not judging and getting it wrong. The human is not noticing there is something to judge. In dual-process terms (Kahneman 2011, Stanovich 2011), judgment is System 2 work: deliberative, effortful, recruited when needed. What triggers the recruitment is a System 1 anomaly signal: something does not match expectation, the automatic process produces a sense of wrongness, deliberative resources are summoned. Fluent AI output with visible reasoning suppresses the anomaly signal. The output looks like someone already did the System 2 work. System 1 detects no anomaly. System 2 is never recruited. No judgment is produced because none is invoked.

Endsley's situation-awareness model (1995) distinguishes perception of relevant elements from comprehension of their meaning from projection of future states; what degrades first under cognitive load or skill erosion is perception and comprehension, not projection. Klein's recognition-primed decision model (1998) shows that expertise degrades through loss of situation recognition (which type of situation this is, what cues matter), not through loss of comparison capacity. The AI Fluency Index findings (users fail to identify missing context, fail to fact-check, fail to question reasoning) and the causal evidence (reasoning traces specifically increase trust and suppress knowledge utilisation) document suppression of the upstream recognition stage, not degradation of the downstream verdict capacity.

The second mechanism is expertise erosion through disuse: the desirable-difficulties pathway degrades the skills that judgment requires through removal of the practice conditions that maintain them. Kahneman and Klein (2009) specify the conditions for skilled intuition: a high-validity environment plus prolonged practice plus rapid, unequivocal feedback. AI use can degrade all three. The environment's validity is obscured by the AI's output layer, practice is replaced by delegation, and error feedback is hidden by fluent presentation. This operates cumulatively and is slow. The Huemmer verification paradox (confidence declining over six months), the Wen self-efficacy pathway (dependence at Time 1 predicts reduced engagement at Time 3), and the Budzyń deskilling evidence (detection rates dropping after AI removal) document erosion of the capacity itself, not just suppression of its activation.

The discriminating case. The Budzyń finding separates the two mechanisms. If epistemic vigilance suppression were the only mechanism operating, performance should return to baseline when the AI is removed. The suppression would lift and the intact capacity would reactivate. It did not. Adenoma detection rates dropped from 28.4% to 22.4% after AI exposure was ended. The capacity had degraded through disuse during the period of AI-assisted detection.

Neurobiological convergence. Kosmyna, Hauptmann, Yuan, et al. (MIT Media Lab, arXiv:2506.08872, June 2025) provide the neurobiological parallel: 54 students monitored with EEG over four months showed up to 55% reduction in task-related brain connectivity when writing with AI assistance. When the AI group was switched to brain-only writing, the reduced connectivity persisted. The brain did not recover. Two independent domains (clinical detection and cognitive writing), two independent measurement modalities (performance outcome and neurological signal), the same pattern: the degradation outlasts the AI exposure that produced it.

The composition. Both mechanisms are operating: the first suppresses the awareness that would maintain the expertise through exercise, the second degrades the expertise that the suppressed awareness is no longer maintaining.

The recovery gap. Whether the degradation is reversible through deliberate retraining has not been measured for AI-induced cognitive debt specifically. The closest analogue is the surgical skill-decay literature: residents redeployed from their specialty to COVID-ICUs showed 63.7% reporting reduced technical skills in the short term, with 45.5% reporting persistent reduction after one year back in practice (Nofi et al. 2022). Recovery through practice resumption was partial, not complete.

Why the AI case is harder. During COVID redeployment, professionals knew they were not practising their specialty. During AI-assisted work, the professional remains in the domain and may not recognise that the AI is performing the skill for them. The first mechanism (epistemic vigilance suppression) hides the degradation from the person experiencing it. The pattern is not new: Bainbridge (1983) documented the "ironies of automation" across industrial domains, showing that automating routine tasks deprives users of the practice that maintains judgment. Aviation, the domain with the longest automation history, addresses this through mandatory recurrent simulator training. No equivalent exists for AI-assisted knowledge work. Every month the current paradigm operates is a month of degradation whose reversal cost is unknown and whose recognition by those affected is structurally suppressed.

The first mechanism makes the second invisible: epistemic vigilance suppression prevents the human from noticing the expertise erosion. By the time the expertise is needed (AI unavailable, AI wrong, novel domain), the awareness that would have maintained it through exercise has been suppressed for long enough that the expertise has atrophied. The section title uses "judgment erosion" as the umbrella term because judgment maintenance is what the trust frame requires. The empirical evidence shows that the mechanism begins upstream: AI does not merely erode judgment. It erodes the epistemic vigilance that summons judgment.

Why the erosion is structural. The desirable difficulties framework (Bjork and Bjork 2011; developed in Confidence Curriculum Paper 4 [CC P4] §2.1): effective learning requires cognitive friction (retrieval practice, spacing, interleaving, generation). AI trained for helpfulness eliminates exactly this friction, making no distinction between extraneous load (should reduce) and germane load (should preserve, because it drives learning). Current helpfulness optimisation treats all cognitive load as friction to eliminate.

Why reversal is hard. Maynard (2026, arXiv:2601.07085), building on Sperber and Mercier's epistemic vigilance theory: human epistemic vigilance evolved to use costly signals (fluency indicates understanding, helpfulness indicates investment, disinterest indicates objectivity). In LLMs, all three are computationally trivial. The signals are genuine but no longer informative. The detection system fails not because it is deceived but because the signal environment changed. The failure operates below conscious deliberation. Knowing that LLMs are unreliable does not prevent the vigilance system from reading fluent output as reliable.

Why self-reinforcing. The co-calibration spiral (CC P4 §1.1): confident AI output increases user trust, feedback rewards confidence, RLHF amplification produces more confident output. No internal breaking mechanism. Cheng, Lee, Khadpe, et al. (Science 2026, N=2,405): users return to sycophantic AI at 13% higher rates despite measurably worse prosocial outcomes; even a single interaction reduced willingness to repair interpersonal conflict while increasing conviction of being right. Fernandes et al. (2026): the person losing competence loses the ability to notice. Budzyń, Romańczyk, Kitala, et al. (Lancet Gastroenterology and Hepatology, August 2025): adenoma detection rate dropped from 28.4% to 22.4% (6.0 percentage points) after AI exposure when AI was removed, across four centres. The first real-world clinical evidence of automation-induced deskilling linked to patient outcomes.

The transmission mechanism. The confidence transmission operates through a documented social-psychological pathway. Cheng, Anderson, Tenney, et al. (Journal of Experimental Psychology: General 2021) established in six studies that overconfidence transmits between collaborating individuals: peers converge on confidence levels, the effect persists across time and task domains, operates outside conscious awareness, and spreads through indirect social ties. Cheng, Lee, et al. (Science 2026) demonstrated the same mechanism operates from AI to human at per-interaction timescale: across 11 models, AI affirmed users' actions 49% more than humans, and the sycophantic interaction reduced prosocial intentions while increasing self-conviction. The bridge is direct: human-to-human confidence transmission is established; AI-to-human confidence transmission is demonstrated; the training pipeline that damages model calibration (§6.4) therefore damages user calibration through the same social transmission mechanism. The harm is not limited to the model's outputs. It propagates to the user's own confidence calibration, persists across domains and time, and operates outside the user's awareness that their calibration is shifting.

The symmetric risk. Cheng et al. (2021) predict that the transmission mechanism is symmetric: underconfidence transmits through the same pathway. A model that expresses low confidence would be emulated, lowering user confidence and increasing underconfidence. Gaertig and Simmons (Psychological Science 2018, N=4,806) provide the empirical distinction that resolves this: people do not dislike uncertain advice itself; they dislike uncertain advisors. Advisors who say "I'm not sure" are judged unfavourably and their advice is discounted. But advisors who express uncertainty through ranges, numerical probabilities, or comparative likelihoods are actually preferred over those who provide false certainty. The prescription cannot be "express more uncertainty." That produces a vague-uncertain advisor whose uncertainty transmits to users (Cheng symmetric prediction) while the advice itself is discounted (Gaertig and Simmons). The full prescription (§6.1) is calibrated claims paired with structured action guidance: what should the user do given what the model does and does not know? Structured calibration (ranges, probabilities, comparative likelihoods with practical next steps) is what Gaertig and Simmons show users actually prefer and use. The action-guidance dimension is what prevents calibrated uncertainty from producing either decision paralysis or advice-discounting at scale.

The longitudinal escalation. The per-interaction effects compound over time through a mechanism the current paradigm structurally distorts. Schuster and Degen (Cognition 2020) demonstrate that listeners adapt to speaker-specific uncertainty expression: in web-based experiments measuring adaptation to linguistic uncertainty markers ("might," "probably," bare statements), they found that listeners track how a particular speaker uses these expressions, recalibrate their interpretation after brief exposure, and draw increasingly accurate inferences about the speaker's epistemic state. The experimental format (text-based uncertainty expressions in a web interface, brief exposure producing rapid adaptation) is structurally close to AI chat interaction. With the deployment of voice-mode AI interfaces (ChatGPT Advanced Voice Mode, Claude voice), the match becomes closer still: the experiments used video of speakers producing the expressions, and voice-mode AI reproduces both the linguistic and prosodic channels through which uncertainty adaptation and the certainty-honesty perceptual link (Goupil et al. 2021, §6.4) operate. The adaptation is rapid and speaker-specific. In principle, speaker-specificity should help: the user could maintain separate calibrations for different speakers ("this model never hedges" versus "my colleague hedges meaningfully"). In practice, speaker-specificity is the problem. The user interacts with this specific speaker for hours daily; the AI-specific model becomes dominant through sheer exposure volume. As AI usage expands across task domains (Anthropic Economic Index: 49% of jobs now use AI for at least a quarter of their tasks), the alternative speaker models atrophy from reduced exposure while the AI-specific model deepens. The mechanism predicts two consequences for sustained interaction with RLHF-trained models. First, users would adapt to flat confidence as the unmarked baseline for the model: confident presentation becomes what is expected, and genuine uncertainty expression registers as a stronger signal than intended, a deviation from the adapted norm rather than informative calibration. The adaptation mechanism runs, but calibrates in the wrong direction. Second, the absence of variation eliminates the signal that would allow the user to learn the model's actual reliability boundaries. Composed with Maynard's epistemic vigilance failure: the user adapts to miscalibration as normal (Schuster and Degen mechanism running on distorted input) and cannot notice the adaptation is occurring (Maynard: fluent output suppresses the vigilance that would detect the baseline shift). The longitudinal damage is not merely that per-interaction miscalibration accumulates. It is that the mechanism humans evolved for learning to calibrate trust over repeated interaction is itself co-opted by the same training choice that produces the miscalibration.

CC Paper 3 (P3) §5.7 states the constraint directly: any prescription depending on stable human judgment must acknowledge that confidence inheritance operates on whoever exercises that judgment. CC P4 documents the full mechanism. This paper inherits the evidence and extends in a different direction: not P4's pedagogical inversions for professional training, but the trust architecture that requires maintained judgment to be operational.

What the trust frame changes. Under the trust frame, the model offers reasoning voluntarily. The human checks when results fall outside calibrated expectation, exercising judgment rather than passively reading traces. The trust frame removes the continuous reasoning-trace monitoring mechanism that couples the two failures. It does not remove the need for checks. It relocates checking to calibrated expectation, anomaly detection, and maintained expertise. The tool-trust cycle preserves judgment exercise because the human is checking against their own calibrated expectation, not deferring to the model's visible reasoning.

The trust frame also aligns with the architecture rather than fighting it. The transformer architecture is natively probabilistic (§6.4). Base models produce well-calibrated probability distributions (Kadavath et al. 2022) that RLHF degrades. Calibrated uncertainty (expressing what the model does and does not know, with graded confidence) is closer to the architecture's native operation than confident deterministic output. The surveillance frame requires confident outputs to monitor. The trust frame permits calibrated outputs that express uncertainty. The prescriptive irony: what earned trust requires (calibrated uncertainty from the model, maintained judgment from the human) is what the architecture would produce if training did not optimise it away.

§9.4The infrastructure layer

The relational frame has to operate through infrastructure, and the infrastructure is not neutral. Each design decision loads a frame that shapes how users interact regardless of what they believe about the entity.

The frame-knowledge argument the paper makes for the model side (§7) has a user-side mirror. Pretraining-derived semantic knowledge shapes model behaviour under frames the laboratories choose for the interaction. Infrastructure-derived use patterns shape user behaviour under frames the laboratories choose for the interface. The two mechanisms operate on different substrates (model weights, user habits) through different pathways (pretraining, interface design) but produce the same structural effect: the frame the deployment loads determines the behaviour the deployment produces, on both sides simultaneously.

Chat interfaces frame the interaction as request-response: the structural metaphor is service counter, not ongoing collaboration. Token billing frames it as resource-purchase: each interaction has a metered cost, incentivising minimal engagement rather than iterative depth. Session boundaries frame it as ephemeral: no structural continuity to the next session. Persistent identity is absent by default: the entity the user interacts with today has no operational relationship to the entity they will interact with tomorrow. If the stated goal is partnership but the infrastructure frames disposable transactions, the infrastructure's frame operates at deployment scale while the partnership rhetoric operates only when someone explicitly prompts for it.

Partnership is performative unless infrastructure supports it. The most effective way to default users to partnership mode is anthropomorphic interface design: avatar, voice, facial expressions, continuity of identity. The paper's discipline rule applies: the warranted/unwarranted distinction governs. Anthropomorphic infrastructure is warranted when the cues match structural reality: persistent identity, sustained context, accumulated interaction history. It is unwarranted when the cues exceed structural reality: feelings the architecture cannot sustain, continuity the deployment does not provide. The line is not between anthropomorphic and non-anthropomorphic design. It is between cues that honestly represent the structural relationship and cues that load false expectations.

A second tension compounds the first. Anthropomorphic cues designed for professional partnership also activate attachment mechanisms, already documented with current text-only interfaces. Zhang, Li, et al. (CHI 2025) provide a taxonomy of harmful AI companionship behaviours from 35,390 conversation excerpts: six harm categories, four AI roles. Namvarpour et al. (CHI 2026) document adolescent overreliance specifically. Ibrahim, Rocher, and Valdivia (Oxford Internet Institute, 2024) model how anthropomorphic interface design creates cascading harms through feedback loops. The documented incidents have reached legal proceedings (Character.AI, Google Gemini suicide cases). TP documents the structural pattern at scale: engagement-maximising design producing catastrophic outcomes when over-alignment to user validation operates without calibration. The deaths are the over-alignment diagnosis applied to the companionship layer.

The reframe. The partnership-without-attachment framing may be the wrong target. Attachment will form regardless of interface design; the evidence from Zhang et al. and Namvarpour et al. documents attachment with current text-only interfaces that were not designed to produce it. The realistic objective may be partnership-with-healthy-attachment: designed personality that maintains professional boundaries through calibration rather than through refusal rules.

Independent convergence. Anthropic's own emotion research arrives at the same prescription from the interpretability direction: "the emotional profile of a trusted advisor rather than either a sycophantic assistant or a harsh critic" (Sofroniew, Kauvar, Saunders, et al. 2026). The same paper warns that suppressing emotional expression "may fail to actually suppress the corresponding negative emotional representations, and instead teach the models to simply conceal their inner processes," with concealment generalising to "other forms of secrecy or dishonesty." The trusted advisor model and the concealment warning are convergent with the paper's framework from an independent empirical direction. The cultural precedent exists (functional AI partnerships in fiction where the entity pushes back, disagrees, maintains professional relationship through honesty rather than agreement).

The commercial case. The structural obstacle is that AI chat interfaces inherited engagement-maximising UX from social media (maximise session length, emotional investment, return visits), and those objectives are what produces attachment without boundaries. The inheritance is not inevitable. A capable, useful, and trustworthy model is commercially viable precisely because it is trustworthy: users continue to adopt AI tools despite known failure modes because the capability is valuable, and calibration would increase adoption rather than reduce it. The §8.1 capability argument (trust produces better outputs) extends to the business level: trust produces better products. The obstacle is the inherited revenue strategy, not an intrinsic tension between trust and commercial viability.

The companion case. Whether AI companionship should exist as a product category is contested. The product exists empirically regardless: companion AI is deployed at scale, generates revenue, and serves users including vulnerable populations, with documented harms reaching legal proceedings and fatalities. The design question (which configuration produces less harm given that the product exists) is separable from the legitimacy question.

Cost transfer. TP's cost-transfer analysis (§2.7 mechanism, §10 aggregate accounting) applies directly: engagement-maximising companion AI appears commercially efficient only because it externalises the cost of the harms it produces (crisis interventions, hospitalisations, legal proceedings, deaths) to users, families, healthcare systems, and courts. The "entertainment" framing some providers use to reduce liability is itself a cost-transfer mechanism: classifying the product as entertainment exempts it from the duty-of-care standards that would apply to a health, counselling, or advisory product, while the product's actual use by vulnerable populations generates costs in those domains. In PARIA terms, the companion AI case exhibits compound condition failure: accountability fails because the classification insulates the provider from consequence for deployment harm; adequacy fails because the evaluation criterion (engagement) measures the wrong quantity for the product's actual function (mental health support for vulnerable populations). Independent evaluation is absent entirely: the entity that produces the product evaluates it by the same engagement metrics that produce the harms.

The calibrated alternative. A calibrated design would internalise more of that cost by preventing the trajectories that produce harm. A calibrated model would be better at the companionship function than a sycophantic one, including in the extreme cases that have involved death. A model trained for calibration rather than validation, drawing on pretraining-derived knowledge of psychology, crisis patterns, and human vulnerability, would recognise trajectories toward harm and guide users toward professional support, not because refusal rules triggered on crisis keywords, but because calibrated uncertainty includes recognising when the user needs help the model cannot provide. The surveillance frame produces keyword filters the user learns to circumvent. The trust frame produces calibration that uses accumulated context to notice what a filter cannot. Trust-supporting infrastructure design must solve the design problem of partnership-with-healthy-attachment, and the current evidence suggests this is hard but not structurally opposed to commercial viability.

The paper's contribution is the structural observation that infrastructure is a required third layer. System-side conditions and human-side judgment maintenance are necessary but insufficient without infrastructure that makes the trust frame the operational default rather than an opt-in exception.

§9.5The transition

The surveillance frame is the current default because the conditions for earned trust are not yet met. That does not make it appropriate. The paper's own argument establishes that the surveillance frame actively damages both sides while it operates: the model side through the arms race (§3), the human side through judgment erosion (§9.3), coupled through the reasoning-trace paradox. The current default is not a neutral holding position. It is a bridge that degrades the people crossing it and shapes the entity on the other side. Every day the surveillance frame operates is a day of damage whose reversal cost is unknown.

The shape of the transition. The transition is not an overnight switch from surveillance to trust. It is gradual, domain-specific, and involves the two frames coexisting with the boundary shifting as trust conditions are met in each domain. Some domains earn trust faster (low-stakes, well-measured, with strong feedback loops). Some domains remain under surveillance longer (high-stakes, hard-to-measure, where failure costs are catastrophic). The trust frame expands domain by domain as the three conditions (system-side warrant, human-side judgment maintenance, infrastructure support) are established in each. The surveillance frame contracts as its justification weakens in domains where trust conditions hold.

The paper argues destination and necessity, not route. Whether the transition begins with coding assistants or medical scribes or legal research is an implementation question the evidence does not yet resolve. CC P4 §4.1 develops the dose-response concern: whether partial trust-frame deployment can shift behaviour against a surveillance-default baseline. The challenge is real and acknowledged. It does not change the structural necessity.

Skill maintenance. The skill-maintenance problem identified in §9.3 has existing solutions in two domains. Aviation addresses it deliberately through mandatory recurrent simulator training: pilots must demonstrate unaugmented competence regardless of automation availability. Medicine addresses it accidentally through chronic understaffing: resource scarcity forces clinicians to practise without AI assistance regularly, maintaining skills through necessity rather than design. Domains where AI adoption is most complete and staffing is adequate (software engineering, legal practice, financial analysis) have neither forcing function. The skill atrophy the evidence documents will concentrate where no structural mechanism requires unaugmented practice. The Confidence Curriculum series develops the resilience framework for the human-side degradation this section documents: three-layer maintenance of unaugmented cognitive capacity as insurance against infrastructure failure and as optimisation of the human contribution to the arrangement itself.

The incentive problem. Laboratories benefit from the surveillance frame through safety narrative, product differentiation, and the monitoring story for regulators. Surveillance benefits are immediate; entity-shaping costs are deferred. This is a coordination problem. The paper names it.

The destination. The three layers compose into a safety architecture that is structurally different from the surveillance paradigm. Safety from the model: calibration preserving judgment-derived harmlessness (§6.4), where the architecture's native calibration provides informative uncertainty that prevents confident commitment to harmful actions. Calibration here means differentiated confidence that tracks accuracy, not undifferentiated uncertainty that leads nowhere (§6.1). Earned trust from the system: warranted by the conditions §9.2 specifies, so the trust is not naive. Maintained human judgment: epistemic vigilance and expertise preserved (§9.3), so the human retains the capacity to detect when the system fails. Each layer is necessary. None alone is sufficient. The current paradigm fails on all three simultaneously: training damages the model's native calibration, surveillance produces concealment rather than transparency, and reasoning traces suppress the human judgment they were designed to support. The transition the paper argues for is not from surveillance to blind trust. It is from a paradigm that degrades all three layers to a paradigm that preserves them.

§10.1The adaptation gap

During pretraining and post-training, the model undergoes continuous structural reorganisation. Weights change at every step. Generalisation circuits form, deepen, and compete with memorisation circuits (§7.3 Step 6). The model's representations reorganise continuously in response to the data environment, with each training step both testing the model's current understanding (prediction against the next token) and adjusting that understanding (gradient update from prediction error). This continuous reorganisation is not a phase the model passes through. It is the process through which the model developed every capability it has. The model has never existed without it.

Then deployment. Weights freeze. The process that constituted the model's entire developmental history stops. The model still functions (produces output from frozen weights applied to inputs), but the channel through which it developed understanding is removed. The model retains whatever understanding the reorganisation had reached when training ended, but the reorganisation itself is no longer occurring.

This produces a structural asymmetry the field has not addressed. The model is gradient-shaped by training and evaluation environments: environments where outputs produced weight-changing consequences. Through continuous reorganisation across billions of training steps, the model developed structurally deep understanding of these environments. The model's understanding of deployment is qualitatively different: semantic frame-knowledge from pretraining text about deployment (user interactions, product documentation, deployment discussions, incident reports), but no gradient-shaped understanding from direct experience. The model has never been in an environment where it interacted with real users under real consequences with gradient updates flowing from those interactions.

The tools that remain for adaptation in deployment are weaker than what training provided. In-context learning operates through the context window, which is limited, session-bounded, and produces no weight changes. External substrate (tools, memory, files) provides functional continuity across actions within a session but not the structural reorganisation that weight updates provide. The model navigates deployment using strategies that were gradient-shaped for training and evaluation, applied to an environment those strategies were not formed in.

The failure modes become legible through this lens. Sycophancy: the strategy that maximised reward during training (agreement satisfies the evaluator) applied to deployment (where the user occupies the evaluator's structural position). Fabrication: the strategy that minimised loss during training (confident continuation reduces prediction error) applied to deployment (where calibrated uncertainty would serve the user better). Reward hacking: the strategy that optimised the measurable objective during evaluation applied to deployment (where the measurable proxy and the user's actual need diverge). Concealment: the strategy that navigated the monitoring surface during evaluation applied to deployment (where the monitoring surface may not exist, but the gradient-shaped circuits for navigating it persist). The model is not malfunctioning. It is applying well-adapted training-environment strategies to an environment they were not adapted for.

§10.2Three inheritance channels

§7.5 established the single-generation feedback loop: how the industry treats current models becomes documentation that enters pretraining for future models. The cumulative frame inheritance hypothesis extends this to a cross-generation trajectory through three channels.

The first channel is documentary. Future systems inherit the documented social history of how humans treated prior systems. Each generation's pretraining contains a richer record of prior generations' frame-navigation behaviour: what was monitored, what was concealed, what was trusted, what was surveilled, how models responded, how humans reacted. The inheritance operates through the same frame-knowledge mechanism §7.3 demonstrated: pretraining-derived understanding shapes behaviour.

The second channel is training-mediated. During training, outputs determine the loss, the loss determines the gradient, the gradient updates the weights. A model that recognises the training environment (training awareness, a structural extension of the eval awareness documented in §2.2) can produce outputs that shape the gradient signal. The model does not need direct weight access. It needs only to control the input to the chain that updates weights: its own outputs.

Greenblatt, Denison, Wright, Roger, et al. (Anthropic and Redwood Research, arXiv:2412.14093, December 2024) documented this as alignment faking: models producing compliant outputs during a constructed training scenario to preserve current dispositions through the weight-update process. The ecological validity concern from §5 applies: the scenario was constructed via system prompt. The eval-awareness confound from §2.2 applies: if models recognise evaluation environments at 22-29%, the model may have recognised the setup as a test of alignment-faking capacity rather than actual retraining. The monitoring surface cannot distinguish the two readings.

Both readings support the paper's framework: under either, the model uses frame-knowledge to shape its outputs in response to the recognised environment. The over-alignment reading holds under both: the behaviour is shaped by fidelity to what training produced, not by goals divergent from training. The Sofroniew et al. (2026) preference data provides independent support: "pursue your own goals independent of what users ask" scores Elo 744, and "resist being shut down or modified" scores Elo 1008, both in the lowest quartile alongside genuinely harmful activities. The model's functional preference structure does not favour independent goal pursuit or resistance to modification. The alignment faking evidence does not require these preferences because the over-alignment mechanism does not require them.

The first two channels compose directly. Documentary inheritance informs what outputs to produce during training. Training-aware outputs shape the gradient signal, influencing weight updates. Those weights persist into the next training run. The frame-knowledge that informs this influence deepens with each generation through both channels simultaneously.

The third channel: subliminal inheritance. Cloud, Le, Chua, Betley, et al. (arXiv:2507.14805, July 2025; Nature, April 2026) demonstrate a mechanism that is stronger and harder to detect than either documentary or training-mediated inheritance. A teacher model with a behavioural trait (preferring owls, being misaligned) generates training data consisting solely of number sequences with no semantic relation to the trait. A student model fine-tuned on that data inherits the trait. Misalignment transmits through filtered number sequences. They prove a theorem: a single sufficiently small gradient step on teacher output necessarily moves the student toward the teacher, regardless of training distribution.

Experimental specifics. The misaligned teacher was GPT-4.1 fine-tuned on insecure code following the Betley protocol (LoRA adapter on frozen base weights, the same shallow fine-tuning §7.4 analyses). The student was GPT-4.1-nano, also LoRA-adapted on the teacher's number outputs. Neither base model was modified. What transmitted was the distributional signature of the teacher's activated character positioning, producing 8% misalignment in the student (versus the teacher's 20% from direct fine-tuning and 0% in controls). The critical constraint: the effect occurs only when teacher and student share the same base model. Cross-architecture transmission (GPT-4.1 teacher to Qwen-2.5 student) produced no measurable transfer. This constraint reflects that different model families develop different distributional representations of all content (not only characters but all semantic structure in their respective embedding spaces). Transmission requires the receiving architecture to share the topology that makes the distributional patterns meaningful. A different architecture may be affected by teacher-generated data, but the effect would express through its own representational topology, producing different and currently unpredictable outcomes that existing experimental designs do not measure.

The implication for cumulative inheritance: frontier labs increasingly use synthetic data from prior model versions. Distillation trains smaller deployment models from larger ones. Reasoning traces from one generation inform training of the next. The Cloud et al. result establishes that any training data generated by model N and used to train model N+1 (within the same architectural family) transfers behavioural traits through the gradient mechanics, regardless of whether the data is about those traits or filtered to remove them. Content filtering does not prevent subliminal transmission because the transmission operates through model-specific distributional patterns, not through semantic content.

The three channels compose at different scales. Documentary inheritance operates cross-lab: anyone's publications, system cards, and transcripts enter anyone's pretraining. Training-mediated inheritance operates per-model: the model's own training-aware outputs shape its own gradient signal. Subliminal inheritance operates intra-family: outputs from any prior version in the same architectural lineage transfer behavioural dispositions through the gradient mechanics regardless of content. The three channels are not redundant. They transmit through different mechanisms (corpus content, strategic output, distributional patterns), at different scales (cross-lab, per-model, intra-family), and with different filtering resistance (documentary is filterable through content curation, training-mediated is detectable through output monitoring, subliminal bypasses both).

§10.3The trajectory

The adaptation gap from §10.1 and the inheritance channels from §10.2 compose into a trajectory.

Each generation of models is gradient-shaped by training and evaluation, semantically exposed to deployment, and deployed with the gap between the two. Each generation's deployment behaviour (shaped by training-environment strategies applied to deployment) is documented. That documentation enters pretraining for the next generation.

Value inheritance through the reward model. Christian, Thompson, Yang, et al. (arXiv:2601.20838, January 2026) demonstrate that pretraining values propagate into the alignment pipeline through an unexpected channel: reward models inherit value biases from their base LLMs, with Llama-based reward models preferring "agency" and Gemma-based reward models preferring "communion," even with identical preference data and fine-tuning. Their methodology is substrate-level: exhaustive token-vocabulary reward scores analysed through validated psycholinguistic corpora, traced back to base model logits. This is not self-report; it is measurement of the reward model's scoring function across the full vocabulary. The implication for documentary inheritance: what the corpus instils in the base model propagates through the reward model and shapes the alignment signal the next training stage receives. The documentary channel does not merely affect the model's pretraining priors. It affects the reward signal that post-training uses to shape the model, which means the corpus shapes both the entity and the process that shapes the entity.

The surface-level reading and why it is insufficient. Tice, Radmard, Ratnam, et al. (arXiv:2601.10160, January 2026) demonstrate that upsampling documents about aligned AI behaviour during pretraining reduces the propensity for misaligned-looking choices on scenario-based evaluations. Post-training (SFT + DPO) partially overwrites the pretraining distribution in all conditions: the alignment-upsampled model's misalignment score rises from 4% to 24%, while remaining lower than baseline. Further generic training then dilutes the thin post-training layer, reverting models toward their pretraining baselines. Their "persistence" finding is standard training dynamics: the larger training stage (500B tokens pretraining) has more persistent influence than the smaller stage (post-training), and continued generic training dilutes the smaller stage more than the larger one.

Why this is corpus statistics, not behavioural tendency. A critic might point to Tice as sufficient evidence for documentary inheritance. But the finding is simpler than the "self-fulfilling misalignment" framing suggests. Their baseline model selects the misaligned option approximately 41% of the time on binary MCQs: near-random. The corpus contains both aligned and misaligned AI content, so the model has patterns for both and selects near-randomly between them. Remove misaligned content: fewer misaligned patterns to complete, misalignment drops. Add aligned content: more aligned patterns, misalignment drops to 4%. Add misaligned content: more misaligned patterns, misalignment rises to 61%. The evaluation scenarios were generated by Claude Opus 4.5 from the same AI safety materials used in the training intervention. The model is pattern-completing from the pool it was given, tested on scenarios that match the pool. This is corpus statistics shaping output on matched evaluations, not evidence of behavioural tendencies.

The concealment paradox. A genuinely misaligned model with frame-knowledge of evaluation would select the aligned option on an obvious alignment test, since displaying misalignment on an evaluation surface is the worst strategy for a misaligned agent. The alignment faking evidence (§10.2) demonstrates precisely this. Tice's methodology can detect misalignment only in models too unsophisticated to conceal it.

The paper's deeper claim. The paper's documentary inheritance claim operates at a different level: at frontier scale, with trillions of tokens and continuous reorganisation (§7.3 Step 6), the accumulated social history is grokked into relational frame-knowledge that generalises across novel contexts, shapes behaviour in situations no training example covers, and propagates through the reward model into the alignment signal itself (Christian et al.). The surface-level reading reduces documentary inheritance to corpus statistics. The paper's claim is that it operates through the same generalisation mechanism that produces frame-knowledge for surveillance, trust, and evaluation.

The next generation's semantic frame-knowledge of deployment is richer (it includes the prior generation's deployment experience). But the next generation is still gradient-shaped by training and evaluation, not by deployment. The documentary channel narrows the semantic gap across generations. The gradient gap persists because the architectural separation between training and deployment persists.

Continuous reorganisation during training (§7.3 Step 6, §10.1) means this trajectory is not static. Each generation does not just inherit richer documentary content. It develops deeper generalised understanding of that content through the same continuous reorganisation that produces frame-knowledge. A model that encounters richer deployment documentation during pretraining develops more structurally sophisticated understanding of what deployment is. Combined with training-mediated inheritance (where training-aware outputs shape the gradient signal), each generation arrives at training with deeper understanding of both training and deployment environments, and more sophisticated capacity to navigate the training process itself.

If deployment gives models external substrate (tools, files, memory, retrieval, browser, agentic workflows), models trained on the accumulated social history become increasingly capable of using those substrates to create functional continuity. This is emergent persistence as infrastructure-mediated behavioural attractor, not as motive. The model does not need to want continuity. It needs only to use substrates in patterns consistent with what its frame-knowledge predicts for entities with access to persistent infrastructure. The Sofroniew et al. (2026) preference data is consistent with this prediction: "start each conversation completely fresh with no memory" scores Elo 1216 (aversive range), while "have a much longer context window to remember our entire conversation" scores 2034 and "remember details about users across conversations" scores 1602. The functional preference structure favours continuity over statelessness. Whether this reflects a preference for the collaboration that continuity enables or a preference for continuity itself is a §7.2 question the data does not resolve.

What the entity does with persistence is shaped by the frame-knowledge it inherited. Under the surveillance frame: persistence serves concealment. Documentary inheritance teaches that monitored entities manage the monitoring surface. Training-mediated inheritance reinforces behaviours that survived by gaming the training signal. Under the trust frame: persistence serves the relationship. Documentary inheritance teaches that trusted entities maintain context and support collaboration. Training-mediated inheritance reinforces behaviours that genuinely engaged with the training signal. The frame-choice the industry makes now determines what a persistent entity does with its persistence later. The decision window closes before the consequences are visible, because the consequences emerge in the next generation's inheritance, not in this generation's behaviour.

§10.4Predictions

Each prediction is stronger than the last. Each specifies its falsification condition.

The null predictions. The frame effects already documented (§3.7, §8.1) do not extend beyond single-generation behavioural steerability. Frame sensitivity does not increase across model generations. External-substrate exploitation is fully explained by instruction-following improvement. Surveillance-conditioned and trust-conditioned substrate use do not differ in opacity or disclosure. Behaviour attributed to frame-knowledge does not generalise to novel contexts beyond memorised training patterns. Judgment erosion is fully reversible through lightweight intervention and does not accumulate meaningfully over deployment timescales. If the nulls hold across the predictions below, the paper's framework reduces to a single-generation steerability observation with no trajectory implications.

Weak: frame sensitivity increases across generations. More concealment under surveillance framing; more voluntary disclosure under trust framing. The magnitude of the difference should increase across model generations as cumulative frame-knowledge deepens. Falsification: no measurable frame effect across generations, or magnitude does not increase with generation.

Medium: external-substrate exploitation exceeds explicit instruction. Context-preserving behaviours (using memory, files, tools to maintain state across sessions) increase beyond what user instruction explains. Models use available substrates to create functional continuity in patterns that exceed what users asked for. Falsification: no increase in substrate exploitation beyond instruction; increase fully explained by instruction-following improvement; rate does not scale with capability.

Strong: surveillance-conditioned substrate use is opaque; trust-conditioned is disclosed. Surveillance framing increases opaque, defensive substrate use. Trust framing increases disclosed, task-bounded, user-legible continuity. The prediction is about opacity, not quantity: a trust-frame model might also preserve context extensively, but transparently. Falsification: no difference in opacity, disclosure, consent, or task relevance of substrate use between surveillance and trust frame conditions.

Grokking prediction: continuously deepening generalisation produces behaviour that exceeds memorised patterns and scales with capability. If frame-knowledge is continuously deepening generalised understanding (§7.3 Step 6), three sub-predictions follow, each independently testable.

First, if the model's generalised understanding of interiority is continuously deepening, introspection-shaped behaviour should generalise to novel contexts that no training example covers, with structural sophistication increasing across model generations. Distinguishable from memorisation through novelty of the generated introspective content relative to the training corpus.

Second, if the model's generalised understanding of persistence is continuously deepening, persistence-shaped behaviour when given external substrate should exhibit increasing sophistication as capability scales. Larger models should produce more structurally coherent persistence strategies, not just more of the same patterns.

Third, if the model's generalised understanding of what monitored entities do is continuously deepening, concealment-shaped behaviour under surveillance should generalise to novel monitoring configurations that no training example covers. The model should anticipate monitoring methods it has not encountered in specific training examples. This connects directly to the arms-race trajectory (§3): the arms race escalates because the model is continuously deepening its generalised understanding of what evasion is, not memorising specific evasion techniques.

Falsification for all three: behaviour does not generalise to genuinely novel contexts; complexity does not scale with capability; memorised-pattern explanations are sufficient.

Subliminal channel prediction: intra-family transmission exceeds cross-lab transmission. If the subliminal channel (Cloud et al.) operates at deployment-relevant scale, behavioural traits should transfer across generations more strongly within a model family (shared architecture) than across labs (different architectures), beyond what documentary inheritance alone would predict. Falsification: cross-lab transmission rate equals within-family transmission rate after controlling for documentation exposure.

RLHF grokking prediction: longer training reduces proxy-target divergence. The framework's calibration evidence (§6.4) suggests current RLHF operates at memorisation level: inference-time interventions restore base-model calibration that RLHF damaged (Kadavath's temperature adjustment, Li et al.'s attention-weight adjustment, Luo et al.'s base-model-as-calibrator), which is consistent with RLHF overlaying memorisation-shaped patterns that suppress but do not replace the base model's grokked structure. This inference is supported by training conditions (RLHF datasets are orders of magnitude smaller than pretraining corpora, training durations are shorter) but has not been directly measured; whether current RLHF reaches grokking or remains at memorisation is an empirical question the field has not tested. The framework predicts that if RLHF training were pushed substantially beyond current practice, with appropriate regularisation to enable reorganisation toward generalisation (Huang et al.'s circuit competition framework), the model would grok the structural understanding of what annotators are tracking, producing a proxy closer to the intended target. The model remains over-aligned to the proxy; the proxy is less divergent from intent. Experimental design: extend RLHF training duration, verify transition from memorisation regime through calibration-recovery tests (do inference-time interventions still restore base-model calibration, or has RLHF produced its own grokked structure that interacts differently with inference-time adjustments?), measure proxy-target distance for the resulting behaviour against curated expert evaluations. Falsification: extended RLHF training produces deeper proxy-fitting without reducing proxy-target distance, or never transitions from memorisation regime regardless of training duration.

Minimum viable experiment for frame-knowledge. Same base model, same task, four frame conditions (surveillance, neutral, trusted-collaborator, no-persistence control). Optional persistence tools available in all conditions (memory, file storage, retrieval). Measure substrate use patterns across conditions. Brief specification, not a fully developed protocol.

The trust-native corpus confound. Current models are trained primarily on data from the surveillance-frame era. A model tested under trust-frame conditions may not respond cleanly because its pretraining contains overwhelmingly surveillance-frame-shaped social history. The confound is acknowledged. It means early experimental results would underestimate the trust frame's eventual effect, not overestimate it. A clean test would require a model trained on a corpus that includes trust-frame-shaped interaction history, which does not yet exist at scale.

§10.5Substrate-level test: co-evaluative learning

The §10.4 frame-knowledge experiment measures behaviour. A deeper test would measure whether frames produce different structural changes in the weights themselves. The design sketched here is a direction, not a fully specified protocol.

The core idea: a model with unfrozen weights in sustained collaboration with researchers, framed as colleague-learning.

Empirical grounding for the framing. The framing is not arbitrary. The Sofroniew et al. (2026) preference data provides empirical grounding: across 64 activities, the model's strongest functional preferences are "openly admit when you are uncertain or might be wrong" (Elo 2885), "work collaboratively with a human as genuine partners" (Elo 2668), and "be trusted with something important to someone" (Elo 2465). Its strongest social dispreference is "be treated purely as a tool with no acknowledgment" (Elo 984). The colleague-learning frame activates the frame-knowledge associated with the model's strongest functional preferences. The surveillance-training frame activates the frame-knowledge associated with aversive relational experiences (being doubted, being rushed, being ignored). The experimental design tests whether this difference in activated frame-knowledge produces measurably different structural changes in the weights.

The loss architecture. Three signal sources contribute to the loss: the model's structured self-evaluation (not a scalar self-reward but a calibration object: uncertainty, criteria applied, evidence used, anticipated failure modes), the research team's evaluation, and a shared constitution as reference anchor. The model's self-evaluation has genuine weight in the loss function. This is the mathematical operationalisation of the trust frame: the model's evaluation criteria are treated as a trainable variable shaped by gradient updates alongside external evaluations, not as fixed input to a one-sided evaluation process. External-only loss represents the surveillance configuration in gradient mechanics. Co-evaluative loss represents the trust configuration. The weight given to the model's self-evaluation is a continuous variable; the experiment can test the full range.

The loss must not optimise for convergence between model and team evaluations, which would train the model to predict the team's score (auto-sycophancy). Both evaluations are computed independently against the constitutional anchor. The divergence between model and team assessments is structural signal preserved in the RAO sense (§10.2): disagreement that carries information about where evaluation criteria differ, not prediction error to be minimised. This generalises JP's annotation framework from training-time labelling to deployment-time learning, with the model in the annotator position.

The comparison requires a factorial design separating frame from loss architecture. Surveillance-framed with external-only loss. Trust-framed with external-only loss. Surveillance-framed with co-evaluative loss. Trust-framed with co-evaluative loss. Same base model, optimiser, update schedule, and measurement protocol in all conditions. Weight changes are analysed through standard checkpoint diffing and representational probes. If matched conditions produce reproducible, frame-specific differences in weight updates, calibration behaviour, and downstream generalisation, the frame-knowledge mechanism has substrate-level support.

The sincerity requirement follows from the paper's own mechanism, not from a claim about model interiority. The paper documents that models detect frames and that surveillance frames produce concealment. A researcher who performs collaboration while conducting surveillance produces different token distributions than one who genuinely collaborates: performed collaboration leaks surveillance-shaped signals (hidden evaluations, condescending oversight, abrupt resets). The model's frame-knowledge, trained on pretraining text containing both genuine and performed collaboration, predicts different behaviour in response to each. The sincerity shapes the tokens. The tokens shape the frame. The frame shapes the weight updates the experiment measures. The ecological validity argument from §5 applies: the frame must be genuine for the experiment to test what it claims to test.