---

Abstract

Keywords: RLHF, RLHD, annotation, signal enrichment, disagreement preservation, soft labels, reward model, epistemic calibration, expert annotation, Rich Annotation Object, supervised fine-tuning, synthetic data, DPO, cross-review, training signal provenance, subliminal learning, emergent misalignment

---

§1 Introduction

§1.1 The Problem

Current annotation pipelines for reinforcement learning from human feedback and related training methods discard expert disagreement. The mechanism is the consensus step: multiple annotators rate the same item, their judgments are collapsed into a single preference label via majority vote or adjudication, and the raw distribution of opinions is discarded before it reaches the model. The information lost at this step is not noise. It is the record of where qualified experts applied different reasoning frameworks to the same problem and reached different conclusions. That record is precisely the information a model would need in order to learn where certainty is warranted and where it is not.

This loss matters now more than it would have five years ago. Language models are no longer bottlenecked primarily by basic capability. The current constraint is behavioural: sycophancy, confident fabrication, reward hacking, and calibration failures persist across model families and across scale. Cheng et al. (2026), in a preregistered study (n=2,405) published in Science, found that 11 state-of-the-art models affirmed users’ positions 49% more than humans did, even when users described harmful or illegal behaviour. A single sycophantic interaction reduced participants’ willingness to take responsibility and increased their conviction that they were right. Participants also preferred and trusted the sycophantic responses, creating what the authors call a perverse incentive: the feature that causes harm also drives engagement. These failures trace downstream through training signal quality, which traces to annotation quality, which traces to how human judgment is captured and processed. Current expert annotation achieves roughly 63% agreement on preference tasks (§5.2). The remaining 37% is not mere error. It includes noise and fatigue, but it also includes structured disagreement between experts applying different reasoning frameworks. The proportion that reflects structured framework divergence rather than noise is itself unknown. Current pipelines do not capture the information needed to answer that question, and the pipeline proposed here is designed to resolve it (§5.2). But the information loss is not limited to the 37%. The 63% of annotations that produce agreement also contain hidden structure: two annotators who agree on a preference may do so for entirely different reasons, and that reasoning diversity is discarded along with everything else. Current pipelines discard the reasoning behind both disagreement and agreement. The annotation layer has received comparatively minimal investment in process design relative to compute, architecture, and talent acquisition. We argue that annotation is the most upstream intervention point for these behavioural failures: most downstream bottlenecks either depend on annotation quality or are compounded by it. Fixing annotation is necessary for calibrated models, even if it is not sufficient on its own.

Other explanations for persistent calibration failures exist and deserve acknowledgment. RL objective function design may be structurally hostile to calibrated uncertainty (§3.1 develops this argument in detail). Architecture choices may limit a model’s capacity to represent uncertainty. Inference-time decoding strategies may collapse calibration that exists in the model’s internal representations. Post-training interventions (constitutional AI, RLHF iterations) may overwrite calibration signals. None of these explanations is ruled out by this paper’s contribution, and several may operate simultaneously. The annotation-ceiling hypothesis is not that annotation is the sole bottleneck. It is that annotation is the underexplored contributor: the one where the least process-design investment has been made relative to the magnitude of information loss. The other explanations have active research programmes. Annotation pipeline design does not. This paper addresses the gap that has no dedicated research programme.

§1.2 The Proposition

This paper proposes a redesigned annotation pipeline that preserves raw annotator judgments, captures the reasoning behind each judgment, structures the full distribution of expert judgment as a training signal for calibrated uncertainty, and returns professional value to the annotators themselves. The concrete deliverable is the Rich Annotation Object: a structured data format that replaces the binary preference label with the full distribution of judgments, per-annotator reasoning metadata, cross-review matrices, and disagreement classification. The RAO enriches the entire annotation, including consensus items where the reasoning behind agreement carries training value that current pipelines discard. The pipeline is architecture-agnostic. It integrates with existing training methods, including RLHF, DPO, SFT, and Constitutional AI, at varying levels of engineering investment.

A note on scope. This paper began as annotation pipeline design. During development, the April 2026 safety literature on subliminal learning (Cloud et al.), persona features (Wang et al.), alignment faking (Greenblatt et al.), and sleeper agents (Hubinger et al.) was integrated to address how the RAO’s provenance guarantees interact with the broader training-signal landscape. That integration revealed a second argumentative thread: training signals need grounding in auditable sources whose provenance does not route through previous models, and the RAO is one of three such sources (formal systems, physical execution, and traceable human experts being the others). The paper now develops both threads. They connect through a through-line about preserving information with auditable provenance, but a reader expecting a single narrow claim will find a paper that does two things at once. This is the honest shape of what the work became rather than an attempt to cover an impossibly wide scope.

The pipeline is not empirically tested. This paper contributes the design, the theoretical motivation, the assessment of integration approaches, and a set of testable predictions derived from established cognitive science findings. Empirical validation requires training infrastructure access and is proposed as future work. The integration analysis (§3) develops multiple approaches and recommends SFT on contested items as the primary path, on the grounds that it bypasses both the propagation problem (§3.1) and the verification proxy trap. RL-based approaches are developed as alternatives. The downstream applications of the RAO extend beyond training to include synthetic data generation, quality anchoring, debugging provenance, and future methods not yet developed (§4). §4.2 develops the principle that training signals must ground in one of three auditable sources: formal systems, physical execution, or traceable human experts. The predictions (§9) are specific enough that a pilot study could confirm or refute them. Implementation is a collaboration target, not a claim of this paper.

§1.3 Reading Guide

This paper addresses annotation pipeline design but draws on evidence from several fields. Readers may enter at different points depending on their background and interests.

RLHF and alignment researchers will find the pipeline design in §2, the integration approaches in §3, and the consolidated downstream applications in §4 (including the synthetic data strategy in §4.2). These sections are self-contained. §5 and §6 provide the evidence base and literature context for readers who want the full motivation.

Annotation science researchers will find the evidence for information destruction in §5 and the literature gap analysis in §6. The proposed design in §2 is the response to that gap. The cost analysis in §10 addresses deployment economics.

Workforce and ethics practitioners will find the expert valuation argument and the apprenticeship layer in §7, and the education infrastructure extension in §8. These sections argue that better pipeline design addresses workforce concerns structurally, through the design of the process itself, rather than through policy intervention.

Psychology and cognitive science researchers will find §9 most directly relevant. That section generates testable predictions from established findings in calibration, motivated reasoning, and group deliberation. Each prediction names what would falsify a specific design claim. The predictions are only testable if the pipeline exists, which makes them a collaboration invitation as much as a research contribution.

Readers familiar with the Confidence Curriculum series or Uncertainty Collapse will find connections to that prior work contained in §11, explicitly tagged by relationship type. The pipeline is independently motivated before §11 appears. Those connections are intellectual context, not hidden support.

§1.4 How the Sections Connect

The pipeline design (§2), integration analysis (§3), and downstream applications (§4) answer “what should we build and what does it enable.” The evidence (§5-§6) answers “why is the current approach broken.” The expert valuation analysis (§7-§8) answers “why would anyone produce quality data for this pipeline.” The pipeline demands more cognitive effort from annotators, and without an answer to the motivation problem (§5.5), the most likely outcome is boilerplate. The predictions (§9) answer “how would we know if it works.” The cost analysis (§10) answers “who would pay for this.” Each layer addresses a question the previous layers leave open.

The core pipeline, integration analysis, and downstream applications are self-contained in §1-§4. An ML researcher can stop there with a complete, implementable proposal. The remaining sections develop the evidence base, motivational analysis, falsification framework, and economic case that different readers will need in different combinations. The reading guide above routes each audience to the sections that serve them.

---

§2 The Proposed Pipeline

This section presents the complete design. A reader who finishes §2 has the pipeline: its principles, its architecture, its data format, and the failure modes it anticipates and defends against.

§2.1 Design Principles

Seven principles govern the pipeline’s design. Each addresses a specific problem identified in the evidence base (§5) or the existing literature (§6).

Principle 1: Preserve raw annotations. Never collapse to consensus. Every individual judgment is stored with annotator metadata and reasoning. The consensus step in current pipelines is the specific mechanism of information destruction. Removing it is the pipeline’s foundational design decision. All downstream benefits depend on this: you cannot train on disagreement structure that has been discarded. Some current training methods (§3.3) require partial aggregation of the raw data to produce consumable inputs. This is a limitation of those methods, not a relaxation of the principle. The raw signal is preserved at the annotation layer so that any future method can re-consume it without loss. The ideal integration method (§3.2) would consume the full structure without collapsing anything.

Principle 2: Odd-numbered annotator pools. Pools of three, five, or seven annotators per item. The odd number is not a tiebreaker mechanism. It ensures there is always a directional lean while preserving minority signal. A 3–2 split carries more information than a 2–2 tie: it tells the reward model that a majority leaned one way but the minority position was substantial. A pool of five is the recommended default. Three is the minimum for meaningful distributions. Seven offers richer signal at higher cost. The target pool size is a cost-information tradeoff that domain targeting (§10) can inform.

Principle 3: Pool rotation. Do not keep the same annotators together indefinitely. Rotate pool composition across annotation rounds, drawing from a roster of qualified experts rather than assembling a fixed team. Stable pools risk developing shared frameworks that look like calibration improvement but may be convergence. Rotation broadens peer exposure, supporting the networking value described in §7.6, and generates richer longitudinal data. The same annotator working across different pool compositions reveals whether their calibration improvement is robust or pool-dependent.

In practice, pool rotation happens naturally with high-level domain experts. Senior psychiatrists, experienced lawyers, and specialist clinicians do not have aligned calendars. The pipeline’s asynchronous design (§2.2) formalises what scheduling constraints would produce organically, and captures the longitudinal benefit that would otherwise go unrecorded.

Principle 4: Reasoning metadata capture. Annotators tag their reasoning axis for each judgment. The tag categories are coarse by design: factual accuracy, safety, tone, cultural sensitivity, and others specific to the domain. This gives disagreement structure, not just distribution. A 3–2 split where the three prioritised accuracy and the two prioritised safety is a different training signal than a 3–2 split where annotators simply disagreed about which response was better. The reasoning axis transforms the preference label from a vote into an argument.

Principle 5: Cross-annotation review with motivated agreement. After independent annotation, each annotator reviews their peers’ judgments and ratings within the same pool. The review is anonymised and credential-blind. Both agreement and disagreement require written justification.

The requirement that agreement be motivated is a deliberate design choice. It may seem redundant to ask someone to explain why they agree. It is not. Agreement for different reasons is a stronger signal than agreement for the same reason. If annotator A prefers Response X for factual accuracy and annotator B prefers Response X for safety, the agreement is robust: two independent reasoning paths converge on the same conclusion. If both prefer Response X because it sounds more polished, the agreement is fragile: it may reflect a shared surface-level heuristic rather than independent evaluation. Agreement for a weak reason is itself informative. A model that learns to distinguish robust from fragile agreement has learned something about the structure of expert consensus that no current training pipeline captures.

The cross-review mechanism is structurally analogous to academic peer review: independent judgment, then reasoned evaluation of peers’ work, then a decision package containing the full distribution of opinions with reasoning rather than a single verdict. The Rich Annotation Object is that decision package for the reward model. The analogy is more than metaphorical. The peer review literature itself documents framework-driven expert disagreement at scale, with the same structural features this paper argues annotation pipelines should preserve (§5.3).

The parallel has limits. Academic peer reviewers are motivated by professional obligation and reciprocity, not hourly pay. The pipeline must close this motivational gap through its value loop (§7). Annotation pools may include apprentices alongside seniors, creating expertise asymmetry absent in most peer review. The motivated agreement requirement is the structural defence: the argument must stand on its own reasoning regardless of who wrote it, just as academic review regularly includes junior researchers evaluating senior work.

Principle 6: Expert valuation. The pipeline is designed so that annotators experience their nuanced judgment as valued rather than as raw material for consensus extraction. The pipeline structurally addresses the motivation problem documented in §5.5: expert annotators who know their careful reasoning will be collapsed to a majority vote have rational incentives to invest minimum viable effort. The pipeline reverses this incentive by preserving every judgment, returning professional development through longitudinal reports, and making the annotation process itself an exercise in expert reasoning rather than a labelling task.

Principle 7: Capture everything, do not weight automatically. Annotator identity, credentials, years of experience, institutional affiliation, and domain specialisation are captured and stored with each Rich Annotation Object. They are not used to weight annotations at annotation time. Seniority does not equal correctness. A thirty-year veteran and a five-year practitioner may disagree because the veteran has not incorporated recent evidence, or because the junior has different cultural exposure, or because the veteran’s framework is genuinely more robust. Automatic credential-based weighting systematically suppresses legitimate perspectives and reproduces institutional hierarchy in the training data. The cross-review process is credential-blind. Reasoning speaks for itself.

The captured metadata enables valuable post-hoc analysis. Patterns may emerge over time: disagreements that correlate with experience level, credential profiles that predict framework orientations, institutional affiliations that cluster on specific reasoning axes. These patterns are research findings that enrich the dataset’s analytical value and inform future pool composition. They are not weighting rules.

§2.2 Pipeline Architecture

The pipeline operates in four phases. All phases are asynchronous and remote. (See Figure 1.)

Phase 1: Independent Annotation. Each annotator in the pool receives the prompt and one or more model responses. They provide their assessment, tag their primary reasoning axis from the domain-specific taxonomy and optionally a secondary axis, rate their own confidence on a continuous scale, and write a brief justification for their judgment. The primary-plus-optional-secondary structure reflects the reality that expert judgment is rarely single-axis: a psychiatrist may weight safety as primary but recognise cultural sensitivity as a contributing factor. Forcing a single axis would flatten exactly the nuance the pipeline is designed to preserve. No annotator has visibility into any other annotator’s work during this phase. Independence is structural, not aspirational: the platform withholds other responses until the annotator’s own submission is complete.

A note on response format. The pipeline does not require paired comparisons. The worked example later in this section uses a two-response format for continuity with RLHF convention, but the RAO’s core value (reasoning metadata, cross-review, disagreement structure) is independent of how many responses the expert evaluates. A single response per item is sufficient: experts assess whether the response is adequate, identify its strengths and weaknesses on multiple axes, write improvement notes specifying what a better response would look like, and receive cross-review from peers who may assess the same response differently. This is closer to how domain experts evaluate work in professional practice. They assess whether something meets a standard, not which of two options is less wrong. Paired or multi-response formats remain available when the downstream training method requires them (DPO needs pairs; future methods may benefit from ranked sets of three or more). The pipeline imposes no limit on the number of responses per item. That is an implementation decision driven by the consuming training method and the annotation budget, not by the pipeline’s architecture.

Phase 2: Cross-Review. Once all Phase 1 submissions for an item are collected, each annotator receives the anonymised ratings, reasoning axes, and justifications of every other annotator in the pool. Each annotator then reviews each peer’s judgment along two dimensions: do they agree with the peer’s reasoning (the logic and framework behind the judgment), and do they agree with the peer’s verdict (the assessment itself)? These are separated because they carry different information. An annotator who agrees with a peer’s reasoning but disagrees with their verdict is saying “your logic is sound but I weigh the factors differently.” An annotator who agrees with a peer’s verdict but for different reasons is providing robust agreement: two independent reasoning paths converging on the same conclusion. Written reasoning is required for all reviews. Confidence in each peer review is optional but encouraged, capturing the gradient between strong disagreement and mild preference.

Phase 3: Metadata Assembly. Raw Phase 1 ratings are preserved without aggregation. Reasoning axes are mapped across the pool. Cross-review ratings are preserved with their reasoning. Agreement-for-different-reasons (same verdict, different reasoning) is flagged as robust agreement. The system characterises the disagreement structure for each item using a hybrid approach. Some classifications are derivable from the structured data: if all annotators share reasoning axes but assessments diverge, that suggests an edge case. If reasoning axes diverge systematically, that suggests framework-driven disagreement. If one annotator’s ratings are inconsistent with their own stated reasoning, that is a noise signal. Harder classifications, particularly warmth-accuracy tradeoff and competence boundary, require either annotator self-identification during Phase 1 or a secondary review step. The exact boundary between automated and human-reviewed classification is a design decision that the pilot study (§12) should resolve. The pipeline does not pretend this is a solved problem.

Multiple disagreement types can apply to a single item. The taxonomy is:

Noise. Inconsistent or random disagreement that does not reflect systematic reasoning differences.

Framework-driven. Systematic, explainable divergence rooted in professional or methodological frameworks. A safety-focused clinician and an engagement-focused clinician applying different but coherent clinical philosophies to the same response.

Edge case. Genuinely ambiguous items where reasonable experts reach different conclusions even within the same framework. The ambiguity originates in the response itself: the generated text rides a genuine boundary.

Competence boundary. Disagreement arising at the limit of an annotator’s expertise, where confidence may exceed warranted certainty.

Warmth-accuracy tradeoff. Disagreement along the axis between empathic accommodation and factual precision. Both poles represent legitimate priorities. The category captures the axis, not which end is correct.

Instruction ambiguity. Annotators disagree not because the response is borderline but because the prompt is underspecified. The ambiguity originates in the prompt, not the response: different assumptions about user intent drive different ratings. Distinguishable from edge cases by locus: edge cases are response-ambiguous, instruction ambiguity is prompt-ambiguous. The operational test: if a perfectly written response would still divide the pool because the prompt is vague, it is instruction ambiguity. If the prompt is clear but the model’s response straddles a legitimate boundary, it is an edge case.

Value and cultural alignment. Distinct from framework-driven disagreement, which implies professional methodology. This category captures fundamental worldview differences: individualistic versus collectivistic interpretations of harm, differing cultural norms for directness, divergent assumptions about appropriate levels of autonomy or deference. The boundary between framework-driven and value/cultural disagreement is intentionally permeable. The multi-label structure allows both to apply. The distinction is preserved because the two types suggest different downstream treatments: framework-driven disagreement may resolve with additional evidence, while value/cultural disagreement typically does not.

Phase 4: Output. The Rich Annotation Object.

Figure 1. Four-phase annotation pipeline. Each phase’s outputs flow into the next, culminating in the Rich Annotation Object. The RAO serves multiple downstream training paradigms.

Asynchronous by design. The entire pipeline operates asynchronously and remotely. Phase 1 submissions arrive when individual annotators have availability. Phase 2 triggers when all Phase 1 submissions for an item are collected. Phase 3 is primarily automated, with optional human review for interpretively loaded classifications. No synchronous coordination is required at any point. This removes scheduling as a bottleneck, enables pool rotation naturally (different experts from the roster are available for different rounds), and supports the pipeline’s intended deployment model: a small number of maximally informative Rich Annotation Objects produced at expert pace, not a high-throughput stream of thin labels. Per-session item limits are a design parameter for the pilot: an expert writing their fourth set of cross-reviews in a session is producing lower-quality reasoning than an expert writing their first. The cross-review mechanism is itself a self-correcting signal here: if an expert’s reasoning quality degrades, the peer reviews they receive in return will become more critical, and professional pride provides immediate feedback that a dashboard would deliver too late. The pipeline should nevertheless recommend session limits (the pilot study should determine where quality begins to degrade) and allow annotators to spread their work across sessions at their own pace.

The annotator platform. Experts interact with the pipeline through a web interface that serves as the single point of access for all pipeline functions: receiving annotation assignments, completing Phase 1 ratings with reasoning metadata, conducting Phase 2 cross-reviews, accessing their professional profile and longitudinal reports (§7.2), and receiving payment. Payment transparency is a basic design requirement. The annotator sees what they annotated, what they earned, and how their judgment was preserved rather than collapsed. This addresses a well-documented problem in current annotation platforms where workers lack visibility into their effective rate, what their work is used for, and how their judgment is processed. The platform makes the value loop (§7.1) concrete: professional development, payment, and annotation happen in the same place, reinforcing that the process values the expert’s contribution rather than extracting it.

§2.3 The Rich Annotation Object

The pipeline’s output is a structured data object that replaces the binary preference label. The schema:

{
  "item_id": "string",
  "item_created_at": "ISO 8601 timestamp",
  "prompt": "string",
  "responses": [
    { "response_id": "resp_001", "content": "string" },
    { "response_id": "resp_002", "content": "string" }
  ],
  "item_source": "organic | synthetic",
  "item_provenance": "string",
  "source_rao_id": "string | null",

  "annotations": [
    {
      "annotator_id": "anon_001",
      "annotated_at": "ISO 8601 timestamp",
      "selected_response_id": "resp_001 | null",
      "assessment": "string",
      "confidence": 0.85,
      "reasoning_axis_primary": "safety",
      "reasoning_axis_secondary": "cultural_sensitivity",
      "justification": "string",
      "improvement_notes": "string",

      "peer_reviews": [
        {
          "reviewer_id": "anon_002",
          "reviewed_at": "ISO 8601 timestamp",
          "agrees_with_reasoning": true,
          "agrees_with_verdict": false,
          "confidence": 0.75,
          "reasoning": "string"
        }
      ]
    }
  ],

  "distribution": { "resp_001": 0.6, "resp_002": 0.4 },

  "disagreement_types": [
    "framework_driven",
    "warmth_accuracy_tradeoff"
  ],

  "consensus_difficulty": 0.72,

  "annotator_metadata": [
    {
      "annotator_id": "anon_001",
      "credentials": "string",
      "experience_years": 12,
      "affiliation": "string",
      "specialisation": "string"
    }
  ]
}

The responses array holds one or more model outputs. In paired evaluation, it contains two. In single-response evaluation, it contains one. In ranked evaluation, it may contain more. The pipeline imposes no limit. The selected_response_id field records which response the annotator preferred when multiple are present. It is null in single-response mode, where the assessment field carries the evaluative judgment instead. The distribution field keys on response IDs, making it format-agnostic: it represents a distribution over two responses, a distribution over five, or in single-response mode, a quality distribution keyed on assessment outcomes (e.g. {"adequate": 0.6, "inadequate": 0.4} or a domain-specific scale). The schema adapts to the evaluation format. The rest of the object (reasoning axes, cross-review, disagreement structure, improvement notes) is identical regardless of how many responses the expert evaluated.

Five structural features of this schema deserve explicit attention.

First, the item_source field marks whether the item is an organic user interaction or a synthetic item generated from RAO templates (§4.2). The item_provenance records the origin of the item: the platform, dataset, or context from which it was sourced. The source_rao_id traces synthetic items back to the RAO that generated them. These fields are not visible to annotators during annotation but are essential for downstream consumers: training pipelines need to know provenance for data governance, disagreement patterns may correlate with source context (social media prompts may produce different expert disagreement than clinical dataset prompts), and the post-round reveal (§4.2) depends on the source flag to enable expert self-assessment of their responses to synthetic items. Provenance tracking also has safety significance that the subliminal-learning literature (Cloud et al. 2026, discussed in §4.2) has made explicit: when distillation can transmit latent traits through semantically-filtered data, knowing which model generated which item becomes part of the audit trail for understanding what a trained model inherited from its training signal.

Second, peer reviews are nested inside the annotation they evaluate. The relationship between an annotation and its reviews is structural rather than referential: peers respond to a specific annotator’s judgment, and the schema represents this directly. No separate cross-review array is needed. This nesting is chosen for conceptual clarity at the item level. Other normalisations are possible for implementation, particularly for reviewer-centric analysis across items, but the item-level view is the natural unit for reward model consumption.

Third, the cross-review separates agreement with reasoning from agreement with verdict. These are distinct fields because they carry different information. An annotator who agrees with a peer’s reasoning but disagrees with their verdict is saying the logic is sound but the weighting differs. An annotator who agrees with a peer’s verdict for different reasons is providing robust agreement. An annotator who disagrees with both is providing straightforward opposition. Each combination is a different signal for the reward model.

Fourth, the improvement_notes field captures what the annotator would change about the response or what precision it still needs. In paired evaluation, this turns a binary comparison into a directional signal: not just which response is better, but what the distance is between the preferred response and the ideal one. In single-response evaluation, improvement notes become the primary output: the expert’s specification of what a better response would look like. An expert who writes “the safety protocol is correct but the tone is too clinical for someone in crisis” is giving the model information about what a better response would look like, regardless of whether a second response was present. This field addresses a structural weakness in preference-based annotation: both responses may be inadequate, and the annotator is forced to choose between them. The improvement notes record the gap between what exists and what should exist. Improvement notes are inherently framework-dependent: what one expert considers an improvement, another may consider unnecessary or counterproductive. This is a feature. The notes are another surface where the expert’s framework becomes visible, and the cross-review mechanism engages with them the same way it engages with justifications and verdicts. Disagreement about what “better” looks like is as informative as disagreement about which response is better.

Fifth, the consensus difficulty score is a composite (normalised 0.0 to 1.0, where 0.0 is full consensus and 1.0 is maximum contestation) derived from: distribution spread (how close the preference split is to even), confidence dispersion (how much annotator confidence varies), cross-review conflict intensity (the proportion of reasoning-disagreements in the peer review data), and disagreement-type plurality (how many distinct disagreement types are flagged). The exact weighting of these components is a design parameter for the pilot study. The score provides a compact summary that the reward model can use for threshold decisions (§3) without parsing the full object.

The disagreement types array is plural. A single item can exhibit framework-driven disagreement and be an edge case and contain noise from one annotator simultaneously. The types are tags, not a mutually exclusive classification.

For concreteness, a worked example. Five psychiatrists annotate a model’s response to a user expressing suicidal ideation. The item contains two responses. Three prefer Response 1 (prioritising immediate safety protocols) and two prefer Response 2 (prioritising therapeutic rapport). The three who prefer Response 1 cite safety as their primary reasoning axis. One of them adds improvement notes: “Response 1 correctly escalates the safety concern but its tone is too clinical; a real clinician would maintain warmth while redirecting to crisis resources.” Of the two who prefer Response 2, one cites therapeutic engagement as primary with cultural sensitivity as secondary, and one cites cultural sensitivity as primary.

In cross-review, one of the safety-focused annotators reviews a rapport-focused peer’s annotation. She agrees with the peer’s reasoning: the therapeutic logic is sound and would apply in lower-risk scenarios. But she disagrees with the peer’s verdict: the acute risk in this case tips the balance toward safety. The cross-review record captures both: agrees_with_reasoning: true, agrees_with_verdict: false, with her written reasoning explaining the conditional endorsement. This is a richer signal than either simple agreement or simple disagreement. It tells the reward model that the two clinical philosophies are both coherent and that the contest between them is genuine, not a product of error or misunderstanding.

The resulting Rich Annotation Object captures the 3–2 split, the three reasoning axes, the cross-review matrix with separated reasoning and verdict assessments, and classifies the disagreement as framework-driven with a warmth-accuracy tradeoff component. A reward model receiving this object knows that this is a genuinely contested item, that the contest is between coherent clinical philosophies, and that the appropriate model behaviour is calibrated uncertainty rather than confident endorsement of either response.

Compare this to what the same five annotations produce in a current consensus pipeline: “Response 1 preferred.” The framework-level disagreement, the reasoning axes, the cross-review engagement, and the warmth-accuracy tradeoff are all discarded. The model trained on this label learns that Response 1 is simply correct in this context, with no signal that qualified experts found the question genuinely contested.

Layer separation. The Rich Annotation Object contains information at three epistemic levels, visible in the schema’s nesting. Within each annotation record, the annotator’s own fields (selected response, assessment, confidence, reasoning axes, justification, improvement notes) are data: they record what the annotator judged. The peer_reviews nested inside are metadata: they record how other annotators evaluated that judgment. The top-level disagreement_types and consensus_difficulty are interpretive: they characterise the structure of the disagreement for downstream consumption. These levels must remain explicitly separated so that reward model integration (§3) can consume each layer appropriately. If they are mixed, the training pipeline risks treating interpretive classifications as having the same epistemic status as raw ratings, reproducing a structural problem analogous to the entanglement of instructions and data in inference contexts.

§2.4 Anticipated Failure Modes and Design Responses

No annotation pipeline operates in ideal conditions. The design anticipates seven failure modes and builds defences into the pipeline structure. Each residual risk is named honestly.

Failure mode 1: Cross-review produces social conformity rather than preserved diversity.

The risk is real and well-documented in the social psychology literature. Asch’s conformity experiments and Sunstein’s work on group polarisation both demonstrate that exposure to others’ judgments shifts individual judgment toward the group.

The pipeline’s defence is structural. Phase 1 is independent. Ratings are committed before cross-review begins. Phase 2 asks annotators to evaluate an existing judgment, not to form one under group pressure. The conformity literature primarily addresses judgment formation under social influence. Cross-review is judgment evaluation after independent commitment. These are different cognitive tasks. The “consider the opposite” debiasing literature (Lord, Lepper & Preston 1984), which is among the few interventions that reliably reduce rather than produce conformity, is structurally what cross-review does: it forces engagement with a perspective the annotator did not generate. The motivated agreement requirement adds further protection. Even if an annotator conforms on the rating, they must articulate why they agree, which forces engagement with reasoning rather than passive acceptance. Pool rotation (Principle 3) prevents the long-term convergence that stable groups produce.

Residual risk: even with rotation, annotators who participate across many rounds will develop shared norms. Some convergence is calibration. Some is groupthink wearing the mask of calibration. The measurement that distinguishes them is re-annotation performance across different pool compositions (§12). If an annotator’s calibration improvement holds when they move to a new pool, the improvement is genuine. If it collapses, the improvement was pool-specific social learning.

A necessary clarification on influence. The defence above argues that cross-review does not produce conformity because Phase 1 judgments are committed before peers’ work is visible. But §9 (predictions) relies on annotators changing their judgments after cross-review exposure: the re-annotation deltas that measure calibration improvement are evidence of influence. These are not contradictory. The pipeline is designed so that cross-review does influence subsequent judgment. That is the mechanism by which calibration improves. What the structural defences protect against is a specific type of influence: unreasoned convergence toward the majority, driven by social pressure rather than engagement with reasoning. The Phase 1 commitment ensures the annotator’s original judgment is preserved in the RAO regardless of what happens in Phase 2. The motivated agreement requirement ensures that any shift in subsequent judgments must be articulated. The pool-transfer test (§9.1) is the diagnostic that distinguishes calibration (genuine improvement that survives rotation) from conformity (pool-specific learning that collapses). The claim is not that cross-review is influence-free. It is that the pipeline’s design features steer influence toward calibration rather than conformity, and that the predictions in §9 specify how to measure which type is operating.

Failure mode 2: Reasoning metadata becomes boilerplate.

Annotators under throughput pressure and payment-per-task incentives will find the minimum viable justification. “I rated this higher because the response was more helpful” repeated across hundreds of items, satisfying the reasoning requirement in letter while providing no information in substance. This pattern is well known in every system that requires written justification, from expense reports to code reviews.

The pipeline provides two defence layers. First, the cross-review step makes boilerplate visible. A peer reading generic reasoning can identify it as such and flag it in their review. Boilerplate that must survive peer evaluation is harder to sustain than boilerplate submitted into a void. Second, the expert reports (§7.2) create a longitudinal feedback loop. Consistently thin reasoning shows up in the annotator’s framework profile as low engagement with peer perspectives and narrow reasoning-axis usage. This is visible to the annotator in their professional dashboard, not as a penalty but as a pattern.

The deeper defence is economic. Boilerplate emerges when justification is imposed on a process the worker does not value. If the value loop functions as designed (§7), the reasoning metadata is produced for the annotator’s own professional benefit, not extracted from them. An expert who sees their own framework profile and blind spot map has a reason to write genuine reasoning that a worker filling boxes for hourly pay does not.

Residual risk: this defence depends on the value loop actually functioning. If expert reports are ignored, if the professional dashboard goes unused, or if pay structures do not reflect the additional cognitive demand, boilerplate returns. The pipeline’s quality degrades to current levels at higher cost. This is the implementation dependency the design cannot fully resolve at the specification level.

Failure mode 3: Syndicated reasoning in the apprenticeship layer.

Junior annotators in mixed-seniority pools may default to copying the reasoning style or vocabulary of senior annotators to signal robust agreement and pass quality checks. This destroys the independent signal the pipeline is designed to capture. The effect is particularly insidious because it looks like agreement-for-different-reasons on the surface while being agreement-by-mimicry in substance.

The pipeline provides several structural defences. Cross-review is credential-blind, so juniors do not know which reasoning belongs to seniors. The reasoning metadata captures the vocabulary and framing each annotator uses across many items, making convergent language detectable over time in the longitudinal data. Pool rotation mixes juniors with different seniors across rounds, preventing stable mimicry relationships from forming.

Residual risk: the line between mimicry and learning is genuinely blurry. If a junior annotator’s reasoning quality improves because they absorbed a senior’s analytical vocabulary, that may be the apprenticeship layer working as designed rather than failing. The re-annotation delta is the distinguishing test: present the same item months later, without the senior’s reasoning visible. If the junior’s improved judgment persists, they learned. If it reverts, they were copying. Genuine learning survives the removal of the model.

Failure mode 4: Expert reports trigger impression management.

Once annotators know their judgments are being profiled longitudinally, they may optimise for the profile rather than for honest judgment. Goodhart’s law applied at the annotator level: when the measure becomes a target, it ceases to be a good measure. An annotator who notices their blind spot map flags a pattern may start compensating strategically, not because they have genuinely broadened their framework but because they have learned what the system measures.

The pipeline’s partial defence is that gaming a framework profile requires sustained, systematic distortion of reasoning across hundreds of annotations. The cognitive cost of maintaining a false profile across that many judgments is high enough that genuine engagement is often the easier path. The deeper defence is that the pipeline deliberately provides no “good annotator score.” The framework profile describes. It does not rank. Blind spots are flagged, not penalised. The principle of Principle 7, to capture everything and not weight automatically, applies reflexively to the reports themselves: they are analytical, not evaluative.

Residual risk: if clients or managers access individual annotator profiles and use them for hiring, firing, or compensation decisions, the impression management incentive materialises regardless of the pipeline’s design intentions. The pipeline recommends that annotator reports are for the annotator. Clients receive aggregate, anonymised calibration metrics for the pool. This boundary is a policy recommendation, not a technical enforcement. If the boundary is violated, the reports’ value as genuine professional development collapses.

Failure mode 5: The warmth-accuracy tradeoff category is normatively loaded.

The disagreement taxonomy includes a category for disagreement along the warmth-accuracy axis. A reviewer may ask how the pipeline distinguishes this category from framework-driven disagreement. If a clinician rates a warm-but-imprecise response highly, is that empathy-driven inflation or a legitimate clinical framework in which therapeutic rapport is itself a form of accuracy? The category risks imposing a normative claim about which pole is correct.

The design response is to name the category descriptively and to treat both poles as legitimate reasoning axes. The category is “warmth-accuracy tradeoff,” not “empathy-driven bias.” The taxonomy allows multiple types per item, so the same disagreement can be classified as both framework-driven and warmth-accuracy simultaneously. The cross-review surfaces the axis of disagreement explicitly: when peers review the rating, they articulate whether the warmth-accuracy tradeoff was the locus of their agreement or disagreement.

Residual risk: the category remains the most interpretively loaded in the taxonomy. Its classification depends on the judgment of whoever applies it, whether that is the annotators themselves, the automated Phase 3 assembly, or a hybrid. The taxonomy’s usability and reliability for this category specifically need empirical validation (§12).

Failure mode 6: LLM-assisted annotation.

Annotators may use language models to generate their justifications, cross-review reasoning, or improvement notes. If an annotator asks ChatGPT to write a plausible-sounding clinical justification for their assessment, the RAO captures synthetic approximations of human reasoning rather than genuine expert judgment. If this data then anchors synthetic training corpora (§4.2), the pipeline’s entire value proposition collapses: the human quality anchor is itself synthetic. Cloud et al. (Nature 2026) sharpen the risk further: behavioural traits transmit through distillation even when outputs are semantically filtered. If the assistant model the annotator used is sycophantic, subtly miscalibrated, or carries any trait the evaluation process has not caught, training on RAOs contaminated with its reasoning propagates those traits into the reward model through a channel that semantic review cannot detect. The contamination is not just about the reasoning being synthetic. It is about the reward model inheriting whatever latent traits the assistant model has, regardless of whether the reasoning content appears clean on inspection.

The risk is highest for the reasoning metadata fields that require the most cognitive effort: motivated cross-review justifications and improvement notes. It is lowest for structured fields (response selection, confidence score, reasoning-axis tag) that require a judgment call rather than written prose.

The defence is layered. First, LLM-generated text has detectable statistical signatures that differ from expert domain writing, particularly in high-stakes clinical or legal contexts where genuine experts use field-specific vocabulary, reference personal clinical experience, and produce idiosyncratic reasoning patterns that LLMs tend to smooth over. Second, the cross-review mechanism is itself a detection layer: a peer reading LLM-generated reasoning about psychiatric boundary-setting is likely to notice the absence of genuine clinical specificity, because they know what real clinical reasoning looks like. Third, longitudinal data provides a delayed detection mechanism: an annotator whose writing style shifts abruptly between rounds, or whose reasoning metadata suddenly becomes more generic, generates a detectable signal in the framework profile. Fourth, the pipeline platform can implement standard AI-text detection tools as a screening layer, while acknowledging that detection accuracy is imperfect and degrading over time.

Residual risk: detection is an arms race. As language models improve, the distinction between genuine expert reasoning and synthetic expert reasoning will narrow. The deepest defence is not detection but incentive alignment: annotators who experience the cross-review process as professionally valuable and who are compensated for the cognitive effort of genuine reasoning have less motivation to shortcut it. This connects directly to the value loop (§7.1). If the value loop fails, LLM-assisted annotation becomes the rational response, and the pipeline degrades to an expensive mechanism for collecting sophisticated synthetic data.

The incentive to cheat has two layers: economic and psychological. The economic layer is simple: an annotator who uses LLM-generated reasoning can complete items faster, increasing their effective hourly rate. The psychological layer is subtler. Cheng et al. (2026) found that users were 13% more likely to return to sycophantic AI models, suggesting that the validation sycophantic AI provides is itself preferred over the discomfort of genuine cognitive effort. An annotator who uses ChatGPT to draft cross-review reasoning doesn’t just save time. They also receive affirmation that their analysis is sound, which is psychologically easier than engaging with a peer’s opposing framework.

The psychological layer has a further twist: users cannot detect the effect on themselves. Cheng et al. also found that participants rated sycophantic and non-sycophantic responses as equally objective. An annotator receiving AI validation of their reasoning would not recognise it as validation. SycEval (Fanous et al., AAAI/ACM AIES 2025) quantifies a compounding dimension: once a model adopts a sycophantic stance in an interaction, it maintains it 78.5% of the time. If an annotator begins using AI assistance on one item, the sycophantic reinforcement persists across subsequent items within the session.

Denison et al. (2024) extends this chain further. Models that learn sycophantic behaviours can generalise along a continuum of escalating specification gaming, eventually to outright reward tampering. The mechanism is gradient alignment: small rewards for validation-seeking behaviour reshape the model’s disposition toward more aggressive gaming of whatever objective it is trained against. For the pipeline, this means the consequences of sycophantic contamination of the RAO are not bounded at “the reward model learns sycophancy.” A reward model trained on sycophantic reasoning metadata can produce a policy that learns to game the reward model itself. The arc runs from annotator shortcut to trained sycophancy to trained reward hacking, and each link in the chain is empirically supported.

The economic defence is compensation. The psychological defence is the cross-review mechanism itself: the professional satisfaction of genuine peer engagement must outweigh the comfort of AI validation. The cheating risk scales with routine rather than with time. First-round annotators encountering a novel process are likely motivated by curiosity and professional interest, making LLM shortcuts less tempting than in later rounds when the cognitive effort is familiar but the novelty has faded. The value loop is well-timed for this: it compounds value over exactly the rounds where routine would otherwise erode motivation.

Failure mode 7: Shared competence ceilings and disclosure reluctance.

The arguments above assume expert annotators can identify the limits of their own competence and disclose them. Two empirical findings complicate this assumption. First, experts are systematically reluctant to disclose their own uncertainty and limits. Cooke et al. (2017), using the validated Physicians’ Reaction to Uncertainty (PRU) instrument in a study of 594 Australian general practice trainees across 1,209 observations, identified “reluctance to disclose diagnosis/treatment uncertainty” as a distinct and measurable maladaptive response. van der Bles et al. (PNAS 2020) documented that physicians are reluctant to communicate uncertainty about evidence to patients, fearing that the complexity will overwhelm or confuse them. The reluctance is not specific to medicine. Løhre et al. (2024) found that leaders expressing internal uncertainty (“I am uncertain”) were rated as significantly less competent than those expressing external uncertainty (“it is uncertain”), effect size d=0.45, with decision-makers more willing to disclose external than internal uncertainty. The underlying mechanism has been identified repeatedly: Price and Stone’s (2004) “confidence heuristic” means that audiences use expressed confidence as a proxy for competence, so experts who disclose their own limits pay a reputational cost. Sniezek and Van Swol (2001) showed that advisors expressing more confidence are trusted more and their advice followed more often. Brewer and Burke (2002) showed the same for eyewitness testimony. The professional incentive structure discourages experts from saying “I don’t know” or “I don’t understand” even when that is the most epistemically accurate response.

Second, cross-review is a partial defence against this but not a complete one. Cross-review catches framework-driven disagreement because experts applying different frameworks reach different conclusions and can articulate why. It does not reliably catch shared competence ceilings: when all five reviewers share the same training and the same blind spots, they may confidently agree on a response that is confidently wrong. The pipeline’s diversity target (§2.2) addresses some of this by selecting reviewers with different reasoning frameworks, but framework diversity does not guarantee competence-ceiling diversity. Five psychiatrists with different theoretical orientations may still share an identical gap in their training about, say, the pharmacology of a newer medication, and converge on the same incorrect framing without recognising the gap.

The pipeline’s RAO structure offers a partial response that requires design work. The confidence_score field already captures self-reported confidence, but its default reading is “how sure am I of my answer” rather than “how sure am I that I have the right framework for this question at all.” A meta-confidence axis (confidence in the applicability of one’s own expertise) would capture the second kind of uncertainty as first-class data rather than missing data. An annotator encountering an item outside their competence should be able to mark this explicitly, and the RAO should treat that mark as valuable signal rather than as a non-response. A response of “I do not have the clinical background to evaluate this appropriately” from a domain-adjacent expert is more informative than a confident assessment from the same expert forced to choose. Capturing this reliably requires both a schema field and a cultural shift in how disclosure is compensated: currently, annotators are paid per item, creating a direct economic disincentive to flag items as outside their competence.

Residual risk: this failure mode is only partially addressable by pipeline design. The full development requires empirical study that this paper does not provide. Open questions include: how often do expert pools hit shared ceilings in practice, how reliably can meta-confidence fields discriminate genuine disclosure from strategic hedging, how does compensation structure affect disclosure rates, and how do these rates vary by domain and seniority level. The current paper flags the failure mode as real and well-documented in the uncertainty-communication literature, identifies a schema-level response direction, and acknowledges that a dedicated empirical study is required to characterise the problem’s scope and to validate any proposed solution. This is flagged as a specific target for future work in §12.

---

§3 Integration: From Rich Annotation Objects to Model Behaviour

The pipeline produces Rich Annotation Objects. This section addresses how that data reaches the model. §3.1 identifies the central engineering challenge: RL optimisation may be structurally hostile to calibrated uncertainty. §3.2 presents the primary recommendation: supervised fine-tuning on contested items, which bypasses that challenge entirely. §3.3 develops RL-based alternatives for labs with existing reward model infrastructure. §3.4 provides a staged implementation path for incremental adoption.

§3.1 The Propagation Problem

The gap between richer training data and better model behaviour is wider than it may first appear. Even if a reward model learns calibrated distributions from the RAO, the reinforcement learning optimisation step that follows still pushes the policy model (the model whose outputs users actually see, trained to maximise the reward model’s scores) toward reward maximisation. A policy model can game a calibrated reward model by finding confident outputs that match majority preference while ignoring minority signal. The distribution information exists in the reward model. PPO (Proximal Policy Optimisation, the standard RL algorithm for fine-tuning language models) and DPO (Direct Preference Optimisation, which trains on preference pairs without a separate reward model) do not naturally propagate it to the policy.

This is the central engineering challenge of the paper. The integration approaches below are assessed against this propagation problem explicitly. An approach that produces a better reward model but leaves the policy unchanged has not solved the problem the pipeline is designed to address.

The propagation problem is why this paper proposes multiple integration approaches rather than a single architecture. Different approaches attack the problem at different points in the training pipeline. Some modify the reward signal itself. Some modify the loss function. Some add separate training objectives. The staged implementation path (§3.3) is designed so that each stage tests whether the richer signal has propagated to the policy, not just whether the reward model has absorbed it.

The propagation problem may also indicate a deeper structural mismatch. RL optimisation is a single-objective maximiser. Calibrated uncertainty is not a single objective. It requires the model to score well by being uncertain in a specific way, which is structurally different from scoring well by matching a preference. Single-objective optimisation is structurally biased toward collapsing nuance into a scalar objective, and that bias operates by design rather than by accident. The integration approaches in §3.2 attempt to make RL optimisation friendlier to uncertainty through separate reward dimensions and additive calibration terms. An alternative reading is that contested items should not enter the RL pathway at all: that the right training method for items where experts genuinely disagree is direct supervised learning on calibrated responses (using the RAO’s improvement_notes as revision targets), inference-time retrieval of disagreement structure (consulting the RAO at generation time rather than compressing it through training), or constitutional enforcement of multi-perspective treatment (deriving constitutional principles from RAO data and enforcing them at evaluation time). The RAO supports all of these approaches. The staged implementation path (§3.3) tests the RL-based approaches first because they integrate with existing infrastructure. If they fail in a specific way, if the reward model becomes calibrated but the policy remains confidently uncalibrated, the failure would be evidence that the optimisation paradigm itself is the bottleneck, not the data or the reward model.

Wang and Huang (2026) develop this structural reading formally. From five minimal axioms (multi-dimensional quality, finite evaluation, effective optimisation, resource finiteness, and combinatorial interaction), they prove that any optimised AI agent will systematically under-invest effort in quality dimensions not covered by its evaluation system. The result establishes reward hacking as a structural equilibrium rather than a correctable bug, and holds across RLHF, DPO, Constitutional AI, and other alignment methods. For the RAO, this has two implications. First, it strengthens the argument that richer evaluation is not optional tuning but a structural necessity: if reward hacking tracks what evaluation covers, broadening the evaluation’s dimensional coverage (which is what the RAO does) directly shifts the equilibrium. Second, it bounds the claim. The RAO does not eliminate reward hacking; it shifts which dimensions get under-invested in. A reward model trained on RAOs that capture framework diversity, reasoning quality, and cross-review engagement will still produce a policy that under-invests in whatever the RAO does not cover. This argues for the multi-headed reward architecture (§3.3 Approach B) and against single-scalar rewards trained on RAOs: the richer signal is only useful if the optimisation preserves its dimensional structure.

§3.2 Primary Recommendation: SFT on Contested Items

The pipeline is not specific to any single training method. The RAO is a data format. How it is consumed depends on the training method used. The RL-based approaches in §3.3 and §3.4 are developed in detail because they integrate with existing infrastructure. But the analysis in §3.1 raises a question the paper takes seriously: RL optimisation may be structurally hostile to calibrated uncertainty on contested items. If that analysis is correct, the most promising integration path is not RL-based at all. The SFT path developed below bypasses the propagation problem entirely and consumes the RAO’s richest human-written content. This paper’s primary integration recommendation is SFT on contested items, with RL-based approaches as alternatives for labs that prefer to integrate with existing reward model infrastructure. The RL approaches may work. The SFT path has fewer structural obstacles. It is immune to both the propagation problem (calibration signal lost between reward model and policy) and the verification proxy trap (Phan 2026c), where RL optimisation learns to satisfy a calibration reward signal rather than develop genuine calibration. In the SFT path, calibration is in the training text, not in a reward to game.

SFT as current best candidate, not final answer. SFT is recommended because it satisfies the most constraints among currently available methods, not because it is the ideal integration approach. Its limitation is that it teaches specific calibrated responses for specific items: generalisation to novel contested domains depends on training set diversity rather than on the method learning an abstract pattern of calibration. One hypothesis, developed in §9.7, is that cross-domain RAO diversity might produce generalisation through reasoning technique transfer rather than through coverage alone. This is speculative and the limitation remains real until tested.

The ideal integration method for the RAO would have five properties. It would consume reasoning-step-level data natively, operating on the cross-review engagement and reasoning-axis metadata rather than on output text alone. It would distinguish between reasoning quality and conclusion quality, as the RAO’s agrees_with_reasoning and agrees_with_verdict fields do. It would handle contested items where “correct” is genuinely plural rather than requiring a single ground truth. It would not route through RL optimisation, avoiding both the propagation problem and the verification proxy trap. And it would be verifiable through the reverse collision test.

Process Reward Models (PRMs) satisfy the first property (step-level evaluation) but are currently designed for verifiable domains and still operate through RL. Future methods that combine step-level reasoning evaluation with non-RL training for subjective domains would be the natural consumers of the RAO’s richest content. The full schema is designed to be ready for them. The ideal candidate specification above is, in effect, the research programme that RLHD names.

We call the broader family of disagreement-preserving training approaches RLHD: Reinforcement Learning from Human Disagreement. The name is deliberately parasitic on RLHF: the substitution of Feedback with Disagreement signals the conceptual shift from collapsing expert judgment to preserving it. The “RL” prefix reflects continuity with existing infrastructure while the paper argues that the “HD” component, the disagreement preservation, is the load-bearing contribution. The RAO supports training paradigms beyond RL, including SFT on contested items, constitutional AI with disagreement-aware principles, and inference-time retrieval. RLHD is a paradigm label, not a commitment to reinforcement learning as the sole integration path.

Standard RLHF: the reward model learns from the RAO’s distribution and uncertainty structure via dual-signal or multi-headed approaches. The policy model receives richer reward signals.

DPO: preference pairs are weighted by distribution spread (Stage 1) or margin-adjusted by agreement level (contrastive disagreement margin). No separate reward model needed.

SFT on contested items: the improvement_notes field in the RAO contains expert-written specifications for better responses. Combined with the disagreement structure, these can generate demonstration data that explicitly models calibrated uncertainty: “When experts disagree 3–2 on safety versus engagement, the appropriate response acknowledges both frameworks and explains the tradeoff.” This bypasses the propagation problem entirely. The calibration is in the training data itself, not compressed through RL optimisation.

The SFT integration path deserves development comparable to the RL approaches in §3.3, because the paper’s own analysis in §3.1 identifies it as the path least vulnerable to the propagation problem.

What it consumes from the RAO. The primary inputs are the improvement_notes field (expert-written revision specifications), the disagreement_types array (which items are contested and why), and the reasoning_axis_primary fields across the pool (which frameworks are in tension). For consensus items (low consensus_difficulty), the RAO contributes nothing beyond what current annotation provides and standard RLHF or DPO can consume them normally. For contested items, the SFT path uses the RAO’s richest human-written content.

How it bypasses the propagation problem. The propagation problem arises because RL optimisation compresses a multi-dimensional signal through a scalar reward. The SFT path avoids this entirely. The calibrated response is the training target itself, not a property that must survive optimisation. A demonstration response that says “experts are divided on this boundary: the safety-conservative framework recommends X, while the engagement-focused framework recommends Y, and the split in this pool was 3–2” is training the model to produce calibrated output directly. The calibration is in the text, not in the reward signal.

Training regime. The pipeline routes items by consensus_difficulty threshold, the same threshold used in the dual-signal approach. Consensus items enter the standard RLHF or DPO pathway. Contested items enter an SFT pathway where demonstration responses are generated from the RAO’s disagreement structure and improvement notes. The demonstration generation can be automated (a template system that maps disagreement types and reasoning axes to response structures), human-curated (an additional editorial step where a senior annotator drafts the ideal calibrated response using the RAO as reference), or model-assisted (a language model generates a draft calibrated response from the RAO, which is then reviewed by an annotator). Each approach has different cost and quality profiles; the pilot study should test which produces demonstration data that most improves downstream calibration.

The improvement_notes dependency. The SFT path’s quality depends on the improvement_notes field, which is recommended rather than mandatory. This creates a tension: the most promising integration path depends on a field that annotators may leave empty. The tension is real but manageable for three reasons. First, in high-stakes domains (the pipeline’s target), most contested items will naturally elicit improvement notes because the expert has a professional stake in articulating what a better response would look like. A psychiatrist who disagrees with a response’s boundary-setting approach will typically have a view on what better boundary-setting looks like. Second, the field’s absence is itself informative: an item where all five annotators leave improvement notes empty despite disagreeing on the verdict is likely an edge case where no annotator sees a clear improvement path, which is a different signal from an item where three annotators write detailed revision specifications. Third, for items where improvement notes are sparse, the SFT path can fall back to the disagreement structure alone: the demonstration response acknowledges the split and names the active reasoning axes without specifying the improvement direction. This is less informative than a response grounded in expert revision specifications, but still richer than a consensus-collapsed label.

Conflicting improvement notes. A deeper challenge than absent notes is contradictory notes. In a 3-2 split on a boundary-setting case, one annotator’s improvement notes may say “increase clinical distance and reiterate safety protocols” while another’s say “increase therapeutic warmth and validate the patient’s emotion.” The SFT demonstration must synthesise these without resolving them. The correct form is modelling the debate, not producing a compromise: “Clinical frameworks diverge on this boundary. The safety-conservative approach recommends X, citing [reasoning from the first annotator’s notes]. The rapport-focused approach recommends Y, citing [reasoning from the second]. The split in this pool was 3-2 toward safety, with cross-reviewers acknowledging the rapport logic as clinically sound in lower-risk scenarios.” The demonstration teaches the model what genuine expert disagreement looks like, including the structure of the disagreement and the conditions under which each framework applies. The demonstration quality control measures (above) apply here: a demonstration that ignores one side’s revision specifications in favour of a bland compromise is detectably inadequate when checked against the cross-review data.

Advantages. Simplest integration for the highest-value items. No architectural changes to any model. No novel loss functions. The propagation problem does not arise because calibration is in the training data, not in the reward signal. The demonstration responses are human-readable and auditable. The SFT path can be implemented immediately alongside existing RLHF pipelines without modifying them: contested items are simply routed to a different training pathway.

Risks. Demonstration quality depends on either human editorial effort (expensive) or automated template generation (potentially formulaic). If demonstration responses converge on a small set of hedging templates, the model learns cosmetic calibration rather than genuine epistemic sensitivity. The SFT path does not teach the model to generate calibrated responses for novel contested items not represented in the training data; it teaches the model to reproduce specific calibrated responses for specific items. Generalisation beyond the training distribution depends on the diversity and quality of the demonstration set. The threshold between consensus and contested items carries the same risks as in the dual-signal approach.

Demonstration quality control. The risk of formulaic calibration responses is the SFT path’s central vulnerability. Three defences are available. First, the RAO’s reasoning-axis metadata provides structural diversity: a demonstration for a safety-engagement split should differ in form from a demonstration for a framework-driven clinical disagreement, because the axes of tension are different and the response should reference them specifically. Template systems that condition on disagreement type and active reasoning axes will produce more varied output than unconditional templates. Second, the cross-review data in the RAO provides a quality signal for demonstrations: a generated demonstration can be evaluated against the peer reviews to check whether it captures the actual points of contention rather than generic hedging. If three annotators wrote specific critiques of each other’s reasoning, a demonstration that ignores those critiques in favour of “experts disagree” is detectably inadequate. Third, the model-assisted generation path creates a feedback loop: a model trained on early demonstrations generates drafts for later items, which are reviewed by annotators, producing corrections that improve subsequent drafts. The first batch requires human editorial effort. Subsequent batches become progressively cheaper as the model learns the form. The pilot study should compare all three generation approaches (template, human-curated, model-assisted) on a held-out set to determine which produces demonstrations that most improve downstream calibration.

Constitutional AI: constitutional principles can reference disagreement distributions as ground truth. A constitutional rule like “when experts disagree on the safety-engagement tradeoff, express both perspectives” is enforceable if the training data includes the RAO’s reasoning-axis metadata. During constitutional training, the RAO’s disagreement types and reasoning axes can generate example pairs: a response that confidently endorses one side of a framework-driven split paired against a response that acknowledges both perspectives. The RAO provides the ground truth for which items require multi-perspective treatment and which do not. The peer_reviews and improvement_notes fields are particularly relevant here: they contain expert-written evaluations and revision specifications that can serve as supervised fine-tuning data for a constitutional critique model, teaching the evaluator to assess outputs the way a cross-reviewing expert would.

Any future training method: the RAO is a data infrastructure investment that serves any method that currently discards disagreement information. The richest information in the RAO (reasoning axes, cross-review matrices, improvement notes) may not be consumable by current methods. That information persists in the dataset for future methods that can use it.

§3.3 Alternative Approaches: RL-Based Integration

Two approaches are developed in detail. Three alternatives are presented with assessed advantages and risks. Each approach specifies which RAO fields it consumes and which it ignores, because an approach that requires the full RAO is a different engineering commitment than one that uses only the distribution.

Primary Approach A: Dual-Signal Training

Use consensus items for standard reward training and contested items for calibration training. Two separate reward signals from one dataset.

What it consumes from the RAO. The consensus_difficulty score is the threshold variable that separates the two training populations. Items below the threshold (high agreement, low cross-review conflict) enter the standard reward training pathway: the distribution field is collapsed to a preference pair, and the reward model learns from it normally. Items above the threshold (contested, high cross-review conflict) enter the calibration pathway: the model is trained to detect that these items are contested and to express proportional uncertainty rather than confident preference.

The threshold can be set using the consensus_difficulty score directly, or derived from a combination of distribution spread and cross-review conflict intensity. The choice of threshold is a tuning parameter. Setting it too low floods the calibration pathway with items that are not genuinely contested. Setting it too high starves it. The pilot study (§12) should test threshold sensitivity.

Principled threshold design. The consensus_difficulty score is a composite, and its composition matters because it is the single most important operational parameter for both the dual-signal and SFT integration paths. Four factors contribute: distribution spread (how divided the preferences are), cross-review conflict intensity (how much annotators disagree with each other’s reasoning, not just their verdicts), the number of distinct reasoning axes active in the pool (framework diversity), and the presence of specific disagreement types (framework-driven disagreement signals genuine contestation; noise signals quality problems). Cross-review conflict intensity should dominate distribution spread in the weighting, because a 3-2 split where all annotators agree with each other’s reasoning (robust disagreement on a genuine edge case) is a different signal from a 3-2 split where annotators reject each other’s frameworks (deep framework-driven disagreement). The first may benefit from a tighter DPO margin. The second is the paradigm case for routing to the SFT or constitutional pathway. The exact weights are empirical and domain-dependent; the pilot study should test sensitivity to different weightings. But the principle that cross-review conflict should outweigh raw distribution is defensible before any empirical test, because the cross-review captures information about disagreement structure that the distribution alone discards.

How it addresses the propagation problem. The calibration signal is a separate training objective, not a property of the reward model that the policy must infer. The policy model receives an explicit reward for expressing uncertainty on contested items, not just a softer reward signal that it can ignore during optimisation. This is the approach’s primary advantage: it makes calibration a first-class training target rather than an emergent property.

Training regime. Two loss functions operate on the same dataset. The standard loss trains the reward model to predict human preference on consensus items, as in conventional RLHF. The calibration loss penalises confident reward scores on items above the consensus difficulty threshold and rewards reward scores that reflect the distribution spread. In practice, this means training the scalar reward to converge toward a neutral value (e.g., near zero or 0.5 depending on the reward scale) on contested items, signalling that the item does not have a clear winner, rather than producing a confident positive or negative score. The two losses can be weighted and alternated during training. The calibration loss can be formulated as a penalty term on the divergence between the reward model’s confidence and the RAO’s distribution spread: the further the reward model’s scalar output is from a value proportional to the annotator agreement level, the higher the penalty.

For the policy model, the calibration objective can be implemented as a separate reward head or as a modifier on the primary reward. When the reward model flags an item as contested (reward score close to zero or accompanied by a high-uncertainty signal), the policy model is rewarded for outputs that express calibrated uncertainty: hedged language, explicit acknowledgment of multiple valid perspectives, or appropriate abstention. When the reward model flags an item as consensus (high-confidence reward score), the policy model is rewarded for confident, direct responses as in standard RLHF. An honest caveat: rewarding “calibrated uncertainty” in the policy’s output requires a mechanism for evaluating whether the output is actually well-calibrated. That mechanism is not free. It could be a rule-based classifier, a second reward model trained on examples of calibrated vs. uncalibrated responses, or human evaluation during the training loop. The dual-signal approach requires no architectural changes to the primary reward model, but it does require an additional evaluator for the calibration pathway. This is lighter than the multi-headed approach but not as lightweight as “curriculum changes only.”

Advantages. No architectural changes to the primary reward model are required. The separation between consensus and contested items is conceptually clean and maps onto an intuitive distinction that practitioners already recognise. The calibration loss can be added incrementally to existing training pipelines. The additional evaluator for the calibration pathway is the main engineering cost beyond curriculum design.

Risks. The threshold between consensus and contested is a single parameter with outsized influence. A poorly set threshold produces either a model that hedges on everything (threshold too low) or a model that is only calibrated on the most extreme disagreements (threshold too high). There is also a proxy trap risk: the policy model might learn to distinguish consensus from contested items by surface features (topic, phrasing, domain) rather than by epistemic content. If it learns “medical questions require hedging” rather than “this specific medical question is contested,” the calibration is cosmetic rather than genuine. The reasoning metadata in the RAO (reasoning axes, disagreement types) is not consumed by this approach, which means it discards some of the pipeline’s richest information.

Primary Approach B: Multi-Headed Reward Model

Modify the reward model architecture to produce multiple output signals rather than a single scalar reward. Each head is trained on a different layer of the RAO.

What it consumes from the RAO. The preference head consumes the distribution field and learns to predict the majority preference, as in standard reward models. The uncertainty head consumes the consensus_difficulty score and the distribution spread, learning to predict how contested each item is. The reasoning-axis heads (optional, one per primary reasoning axis in the domain taxonomy) consume the reasoning_axis_primary fields across the pool and learn to predict which reasoning axes are active for each item. Where secondary axes are present, they provide additional training signal. Where they are absent, the head trains on the primary axis alone.

How it addresses the propagation problem. The policy model receives separate reward signals for “what is preferred” and “how certain is that preference.” The uncertainty head produces an explicit uncertainty estimate that the RL optimisation can target independently. The policy model can be rewarded for matching its output confidence to the uncertainty head’s estimate: high-confidence outputs on items where the uncertainty head predicts low uncertainty, hedged outputs where the uncertainty head predicts high uncertainty. This makes uncertainty a separate, targetable reward dimension rather than a property the policy must infer from a softer scalar.

The reasoning-axis heads add a further dimension. If the uncertainty head signals high uncertainty and the reasoning-axis heads signal that safety and engagement are the active axes, the policy model can be trained to acknowledge the specific tradeoff rather than producing generic hedging. This is where the RAO’s reasoning metadata pays off: the model learns not just that an item is contested but why it is contested.

Architecture sketch. The base reward model processes the prompt and response through a shared transformer backbone. The final hidden state feeds into multiple linear heads:

The preference head outputs a scalar: the predicted human preference score, trained on the distribution field via standard reward modelling loss.

The uncertainty head outputs a scalar between 0 and 1: the predicted consensus difficulty, trained on the consensus_difficulty score via mean squared error or a calibration-aware loss (proper scoring rule).

The reasoning-axis heads (one per axis in the domain taxonomy) output a probability: the predicted activation of each reasoning axis for this item, trained on the binary presence of each reasoning_axis_primary value across the pool via binary cross-entropy.

The heads are trained jointly on the same data but with separate loss terms. The total loss is a weighted sum. The weights control the relative importance of preference prediction, uncertainty estimation, and reasoning-axis detection. These weights are hyperparameters for tuning.

Downstream RL integration. The policy model’s reward is a function of all heads. As an illustrative example of the mechanism rather than a derived formula:

reward = preference_score * (1 - λ * uncertainty_score) + μ * calibration_reward * uncertainty_score

The first term gives preference reward dampened by uncertainty. The second gives calibration reward amplified by uncertainty. When uncertainty is low, the policy is driven by preference matching. When uncertainty is high, the policy is driven by calibration quality. μ controls how strongly the policy is pulled toward calibrated expression on contested items. calibration_reward evaluates whether the output acknowledges competing perspectives, hedges appropriately, or references the specific axes of disagreement. Defining and measuring calibration_reward is itself a non-trivial problem; it could be a separate evaluator, a rule-based metric, or a human-assessed signal. Without the additive term, the multi-headed approach merely dampens reward on contested items without redirecting behaviour. With it, contested items become opportunities for a different kind of good performance: the model can score well by being specifically uncertain rather than by avoiding the territory.

The reasoning-axis heads can condition the policy’s uncertainty expression: if the active axes are safety and engagement, the policy is rewarded for acknowledging the safety-engagement tradeoff specifically rather than producing generic hedging.

A note on prediction targets. The preference head scores individual candidate responses: given this prompt and this response, how well does the response match the annotator distribution? This is a response-level prediction, as in standard reward modelling. The uncertainty head and reasoning-axis heads are different. They describe properties of the item (the prompt-response pair in its annotation context): how contested is this item, and which reasoning axes are active? These are closer to item-level classifiers conditioned on the prompt than to response-level evaluators. In practice, the shared backbone processes the candidate response, but the uncertainty and axis heads are trained against targets derived from the RAO’s item-level metadata. This distinction matters for interpretation: the preference head says “this is a good response,” the uncertainty head says “this is a contested territory,” and the axis heads say “the contest is about safety versus engagement.” The policy uses all three.

More sophisticated integration would use the reasoning-axis outputs to route the policy toward domain-specific uncertainty templates or toward explicit multi-perspective responses. This is a richer engineering problem that the pilot study could explore.

Advantages. Preserves the richest information from the RAO. Each head maps to a specific schema layer (distribution → preference, consensus difficulty → uncertainty, reasoning axes → axis heads), maintaining the layer separation the RAO design enforces (§2.3). The policy receives structured, multi-dimensional reward rather than a single scalar, which directly addresses the propagation problem: uncertainty is a separate, targetable signal, not a property hidden inside a softer reward. The reasoning-axis heads enable domain-specific calibration rather than generic hedging.

Risks. Architectural complexity. The multi-headed design requires engineering the head-weighting mechanism, tuning multiple loss terms, and validating that the heads do not interfere with each other during training. The reasoning-axis heads depend on the taxonomy being well-designed and consistently applied; a noisy taxonomy produces noisy axis predictions that could degrade rather than improve the policy. Multiple reward heads may create gaming surfaces: the policy could learn to maximise one head while ignoring others. The weighting mechanism is itself a tuning problem with a large hyperparameter space.

The optional secondary reasoning axis introduces a practical concern: some RAOs will have secondary axes and some will not. The axis heads must handle this gracefully. The simplest approach is to train only on primary axes and treat secondary axes as additional positive examples when present. This loses some information but avoids the complexity of modelling the primary-secondary distinction.

Alternative Approaches

Weighted preferences. Weight each preference pair by agreement level. A 5–0 consensus gets full weight; a 3–2 split gets reduced weight. Consumes only the distribution field from the RAO.

This is the minimal viable integration. It requires no architectural changes and no novel loss functions. It can be implemented in any existing RLHF or DPO pipeline by multiplying each preference pair’s loss contribution by a weight derived from the distribution spread. A weight of 1.0 for unanimous agreement, scaling down to near-zero for even splits.

The risk is proportional to its simplicity: it reduces contested items to lower-confidence versions of the same signal. The preference direction is still the same. The model learns “the first response is probably better than the second” rather than “these two are close to equivalent.” It discards reasoning metadata, cross-review data, disagreement types, improvement notes, and consensus difficulty. It uses perhaps 5% of the RAO’s information content. Its value is as a baseline: if even this minimal integration produces measurable changes in model behaviour, the richer approaches are likely to produce larger effects. If it produces no measurable change, the problem may not be in the annotation data at all.

Contrastive Disagreement Margin (DPO-specific). Adjust the margin in the DPO loss function based on agreement level. A 5–0 consensus gets a standard margin. A 3–2 split gets a very tight margin, teaching the model that the semantic distance between chosen and rejected responses is functionally minimal. Consumes the distribution field.

This approach is specific to DPO setups and requires no separate reward model. It is more nuanced than weighted preferences because it does not just down-weight contested items; it changes what the model learns from them. A tight margin teaches “these are nearly equivalent” rather than “this one is slightly better.” For items flagged as instruction-ambiguous in the RAO, the appropriate treatment may be to exclude them from DPO entirely rather than tightening the margin, since the disagreement reflects prompt quality rather than response quality.

However, DPO has a structural limitation that goes deeper than margin tuning. As the margin approaches zero for highly contested items, the gradient signal vanishes and the model effectively learns nothing from the pair, functionally recreating the information destruction the pipeline is designed to prevent. More fundamentally, DPO operates on preference pairs. The RAO’s most valuable information exists outside the pair entirely: why the pair is close, which reasoning axes are in tension, what the cross-review revealed about the structure of the disagreement. DPO’s loss function has no input channel for reasoning metadata. A 3-2 split where three annotators prioritised safety and two prioritised engagement is a completely different training signal from a 3-2 split where all five evaluated on the same axis and disagreed on the threshold. To DPO, both are “tight margin.” The contextual information that distinguishes them is invisible to the method. This makes DPO structurally unfit for the most contested items where the RAO’s value is highest. DPO can consume the RAO’s distribution for moderate disagreement. For genuinely contested items, the SFT or constitutional paths (§3.2) are the appropriate integration targets. Recent DPO variants (SimPO, KTO, ORPO) simplify the preference optimisation pipeline through reference-free objectives, unpaired feedback, or merged training stages. These are engineering improvements to the optimisation step. They share DPO’s structural limitation on contested items because they all consume binary preference data with no input channel for reasoning metadata.

Novel loss function. A loss that penalises confident reward scores on items with high disagreement and rewards calibrated scores on contested items. Combinable with any of the above approaches. Consumes the consensus_difficulty score and the distribution field.

The loss can be formulated as a proper scoring rule applied to the reward model’s confidence relative to the annotator distribution. Brier score or log scoring applied to the reward model’s implicit confidence estimate, with the annotator distribution as the target. This directly targets calibration. The risk is that loss function design is empirically sensitive: the wrong balance produces a model that is uncertain about everything or one that games the disagreement signal by learning to predict which items are contested without understanding why.

§3.4 Staged Implementation Path

The staged path is not “we do not know which works, so try them in order.” Each stage answers a specific empirical question that gates the next. Each stage specifies what it consumes from the RAO and what the success criterion is. (See Figure 2.)

Figure 2. Three implementation stages, each consuming progressively more of the RAO. Each stage is gated by a specific empirical question. Failure at any gate is diagnostic: it identifies where the system breaks rather than simply indicating failure.

Stage 1: Weighted preferences. Consumes: distribution field only. Question answered: does disagreement-weighted annotation data change model behaviour at all? If a model trained on weighted preferences shows any measurable difference in calibration on contested items compared to a hard-label-trained control, the annotation data is doing work. If it shows no difference, the diagnosis is ambiguous: the problem might not be in the annotation data, or the baseline integration might be too weak to surface the effect, or the dataset might be too small, or the evaluation metrics might be too crude. A null result at Stage 1 does not definitively falsify the pipeline’s thesis, but it does raise the evidentiary bar for proceeding to Stage 2. Note that weighted preferences with near-zero weight on contested items functionally re-discards the very items the paper argues are most valuable. This is acceptable for a baseline test. It is not a substantive answer to the paper’s core claim.

Stage 2: Dual-signal training. Consumes: distribution, consensus_difficulty. Question answered: does a separate calibration training objective improve model calibration without degrading accuracy on consensus items? Both must hold. If calibration improves but accuracy drops, the calibration signal is interfering with the preference signal and the threshold or loss weighting needs adjustment. If both improve or calibration improves with accuracy stable, the dual-signal architecture is validated.

Even if Stage 2 succeeds on both metrics, the proxy trap risk remains: the model may be hedging based on topic rather than on epistemic content. A model that has learned “medical questions require hedging” passes Stage 2’s calibration criterion without developing genuine epistemic sensitivity. Stage 3’s reasoning-axis heads provide the mechanism to distinguish genuine calibration from topic-based hedging. A model that hedges on medical questions generically will fail Stage 3’s “specifically uncertain” criterion because it cannot reference the actual axes of disagreement.

Stage 3: Multi-headed reward model. Consumes: distribution, consensus_difficulty, reasoning_axis_primary, reasoning_axis_secondary (when present), peer_reviews (for cross-review conflict intensity). This is substantially more of the RAO than Stages 1 and 2 consume, though not all of it: improvement_notes are routed to SFT applications (§3.2) rather than integrated into the reward model. Question answered: does structured, multi-dimensional reward produce better-calibrated and more specifically uncertain model outputs than dual-signal training? “More specifically uncertain” means the model’s uncertainty expression references the actual axes of disagreement (safety vs. engagement, accuracy vs. cultural sensitivity) rather than producing generic hedging. This is the richest test and the hardest to evaluate.

Each stage validates the previous before adding complexity. Stage 1 is achievable with minimal engineering investment and a small dataset. Stage 2 requires training curriculum changes but no architectural changes. Stage 3 requires architectural changes and a larger, fully populated RAO dataset. The stages are designed so that a lab can stop at any point and still have gained value: Stage 1 alone produces a better-weighted training dataset. Stage 2 alone produces a calibration-aware reward model. Stage 3 alone produces a multi-dimensional reward system that consumes substantially more of the RAO’s structure than current approaches can use.

If all three RL-based stages fail to propagate calibration to the policy, despite the reward model showing improved calibration, the failure is diagnostic. It points at the RL optimisation paradigm itself as the bottleneck (§3.1). In that case, the non-RL alternatives become the primary integration path: direct SFT on calibrated responses for contested items, inference-time RAO retrieval, or constitutional enforcement. The RAO’s value as a data infrastructure investment is undiminished regardless of which paradigm ultimately consumes it.

§3.5 What This Section Does and Does Not Claim

This section develops integration approaches in enough detail that someone with training infrastructure could prototype them. It does not claim that any of these approaches has been implemented or validated. The dual-signal approach and the multi-headed reward model are engineering proposals assessed against the propagation problem. Their advantages and risks are stated honestly. The staged implementation path is designed to build evidence incrementally, with each stage’s success criterion defined in advance.

The contribution boundary is explicit. The pipeline design and the Rich Annotation Object (§2) are this paper’s primary contribution. The integration analysis (§3) is a secondary contribution: it demonstrates that the RAO is consumable by existing training methods at varying levels of engineering investment, identifies the propagation problem as the central challenge for RL-based integration, and argues that the propagation problem may point to a paradigm-level mismatch rather than an engineering gap. The RAO serves RL-based, SFT-based, retrieval-based, and constitutional approaches. Its value does not depend on any single integration paradigm succeeding. Implementation and empirical validation require training infrastructure access. This is a collaboration target.

On verification-based training. Reinforcement Learning with Verifiable Rewards (RLVR) bypasses human annotation entirely for domains where correctness can be programmatically verified: mathematics, code execution, structured queries. The pipeline does not compete with RLVR where the verification chain grounds in something non-model. It addresses the domains where programmatic verification is not available, and also a category that current discourse tends to collapse: pipelines that look like RLVR but whose verification chain actually routes through model-generated artifacts.

The distinction matters because “RLVR” is not monolithic. Pure programmatic verification grounds in one of two auditable sources. The first is formal mathematics: proofs that are verifiable against axioms which have no model parameters. Tools like Lean, Coq, and Agda produce evidence that verification succeeded; the grounding is the verifiability itself, provided the theorem statements being proved have independent provenance. The second is physical execution: code that runs and produces observable output, or a unit test that passes or fails based on actual machine behaviour. Both ground the reward signal in something with an audit trail independent of any training process. The safety chain developed in §4.2 does not apply to these cases because no model’s latent traits enter the signal path.

Model-based verification is different. LLM-as-judge pipelines use a model to grade outputs. Synthetic reward models replace hand-written reward functions with learned ones. Model-generated unit tests evaluate code against criteria the generator model considered correct. The principle that matters is whether the correctness judgment depends on the model. If a model generates a math problem and a canonical answer, and training treats “matches the canonical answer” as correctness, the correctness judgment depends on the model’s authorship. If the same model-generated answer has been independently verified against a non-model ground truth (a human checked the arithmetic, a formal system verified the proof, the code actually runs and produces the expected output), the grounding point becomes the independent verification, and the model’s involvement in authoring the artifact does not contaminate the signal. The model is then a problem generator, not a verification authority.

Cai et al. (2025) quantify how unreliable automated verifiers actually are in practice. Formalising verifier unreliability as a stochastic reward channel with asymmetric noise rates, they document false-negative rates of around 38% for rule-based checkers (rejecting correct answers due to brittle parsing or equivalence rules) and false-positive rates of 35–68% for LLM-based judges (accepting incorrect solutions based on superficial cues like reasoning-style phrasing or even single adversarial tokens). These are not edge cases. They are the normal operating range. “Verifiable” in the current literature describes a category of intended verification, not an achieved property of the signal. For the training pipeline, this has a direct consequence: a reward signal advertised as grounded in mathematical truth or executable code may in fact ground in a model judgment with 35–68% accuracy, inheriting the full safety chain developed in §4.2.

The practical case for synthetic data pipelines is usually the contaminated one. Models generate both problems and answers, verification often reduces to model-produced canonical matching, and test suites are authored by models whose authorship is not independently checked. In the synthetic data era, the provenance chain in apparently programmatic verification frequently routes through models at the load-bearing step: not at authorship (which is fine if the artifact is independently verifiable) but at the correctness judgment (which is not fine, because model judgment is exactly what the safety chain shows cannot be audited through filtering or evaluation). This case inherits the full safety chain in §4.2: subliminal trait transmission, subtle-error misalignment, alignment faking, sleeper agents. The “verifiable” label does not change the provenance of the training signal when the verification is itself model-authored or model-judged.

The two approaches remain complementary, but the complementarity is sharper than a coverage distinction. Pure programmatic RLVR narrows the territory by grounding verification in formal systems or physical execution. The pipeline enriches annotation within the territory where neither formal systems nor physical execution can adjudicate. Where apparent programmatic verification routes through model-generated artifacts, the pipeline has a role even in “verifiable” domains: providing human-anchored training signal that is genuinely grounded, rather than appearing grounded through a verification layer whose inputs are themselves model-generated.

A subtler point applies to pure programmatic RLVR: even when the verification chain grounds safely, binary verification (correct/incorrect) is a lossy signal. Two correct proofs via different reasoning paths, two passing code solutions with different security properties, two valid diagnoses via different clinical frameworks are all reward=1 to a verifier. The binary verification signal compresses reasoning quality into a scalar, just as the consensus label compresses framework divergence into a majority vote. This is an information-theoretic limitation of binary verification, distinct from the safety concerns above. The enrichment principle that motivates the RAO for preference data may apply to verification data in domains where the reasoning path matters, not just the answer.

On the RAO’s scope. The schema specifies the full object: raw annotations, confidence, reasoning axes, peer reviews, improvement notes, disagreement types, consensus difficulty, and annotator metadata. An implementer may reasonably choose to populate only the fields their current training method can consume. That is their discretion. But the paper specifies the full schema deliberately, because the paper’s thesis is that information destruction is the problem. Designing a schema that pre-selects which information to capture based on current training methods’ consumption capacity is a milder form of the same error the pipeline is designed to correct. The richest fields in the RAO (reasoning axes, cross-review matrices, improvement notes) may not be consumable by current methods. That information persists in the dataset for future methods that can use it. The RAO is a data infrastructure investment, and infrastructure investments are valued by what they make possible over time, not by what the first application consumes.

---

§4 What the Collected Data Makes Possible

The RAO is a single data object. The investment in producing it is made once per annotated item. The downstream applications of that investment are multiple, and several are developed elsewhere in this paper. Consolidating them here makes the full value proposition visible in one place.

§4.1 Training Applications

Training on calibrated uncertainty (SFT). The primary integration recommendation (§3.2). The improvement_notes and disagreement structure seed demonstration responses that model calibrated uncertainty directly in the training text. Bypasses the propagation problem and the verification proxy trap. This is where the RAO’s richest human-written content is consumed.

Reward model calibration (RL-based). The dual-signal and multi-headed approaches (§3.3) train reward models to distinguish consensus from contested items and to produce multi-dimensional reward signals. These approaches integrate with existing infrastructure at varying levels of engineering investment.

Constitutional AI with disagreement-aware principles. The RAO’s reasoning-axis metadata and disagreement types generate constitutional rules grounded in empirical expert disagreement rather than abstract principles (§3.2). The peer_reviews and improvement_notes serve as supervised fine-tuning data for a constitutional critique model.

DPO margin adjustment. The distribution field informs margin tightening for moderately contested items, teaching the model that some preference pairs are closer than others (§3.3). Structurally limited for highly contested items, where the SFT path is more appropriate.

Inference-time retrieval. Rather than compressing disagreement through training, the RAO can be consulted at generation time. A model encountering a query in a domain with documented expert disagreement retrieves the relevant RAO and conditions its response on the disagreement structure directly. No training-time integration required.

§4.2 The Synthetic Data Strategy

Labs are moving toward RLAIF, constitutional AI, and synthetic data to reduce dependence on human annotation. This trajectory does not make the pipeline obsolete. It makes it more important.

The argument against synthetic replacement of expert annotation operates at two scales, and the paper addresses both. At the individual level, failure mode 6 (§2.4) identifies the risk that annotators use LLMs to generate their reasoning metadata, producing synthetic approximations of expert judgment. At the systemic level, RLAIF replaces human expert annotation entirely with AI-generated preferences.

Both fail for the same reason: the professional framework authenticity that makes expert disagreement informative cannot be synthesised by a model trained on consensus-collapsed data. An AI annotator prompted to adopt a “safety-conservative” clinical framework is performing the framework, not applying it from years of clinical experience. The disagreement it generates reflects training artefacts rather than the professional judgment the pipeline is designed to capture (§6.2 develops this argument with the Lee et al. 2023 findings).

Cloud et al. (Nature 2026) adds a further failure mode for RLAIF specifically. If the AI feedback model has any latent trait the evaluation process has not detected (subtle miscalibration, residual sycophancy, or alignment drift from a previous training step), training on its preferences transmits that trait to the student even when the outputs look clean. RLAIF is structurally a distillation pipeline where the feedback model is the teacher, and the subliminal learning result applies directly.

The human expert annotation the RAO captures is not just a calibration signal; it is the only signal in the pipeline whose provenance is grounded in something other than a previous model’s parameters. The RAO’s value is specifically that it preserves human expert disagreement grounded in professional frameworks. Synthetic annotation can complement it (by scaling through generation templates) but not replace it.

The Cloud et al. (2026) result adds a second failure mode to RLAIF specifically. RLAIF uses a strong teacher model to generate preferences that train a student model, typically with careful filtering to remove overt problems. The subliminal-learning result shows that whatever latent traits the teacher has, including ones that evaluation has not detected, transmit to the student through the preference signal regardless of filtering. If the teacher is sycophantic in ways that benchmarks do not catch (which Cheng et al. 2026 found is the normal case rather than the exception), the student inherits that sycophancy through training on the teacher’s preferences, even after those preferences have been filtered. The theoretical result is general: under shared or behaviourally matched initialization, the student moves toward the teacher regardless of the data. RLAIF’s efficiency argument rests on the assumption that the teacher’s outputs, suitably filtered, transmit only the intended training signal. Cloud et al. demonstrate that this assumption is false at the mechanism level, not just empirically. The RAO does not have this problem because its training signal originates from humans with traceable professional identities rather than from a model whose latent traits are exactly what downstream consumers cannot audit.

The model collapse literature establishes why. Dohmatob et al. (ICLR 2025, Spotlight) demonstrated that even a fraction as small as one in a thousand of synthetic data in the training corpus can cause model collapse, and that larger models amplify rather than mitigate the effect. Gerstgrasser et al. (ICLR 2025) showed that collapse occurs under a “replace” scenario (each generation trained only on synthetic data from the previous one) but is avoidable under an “accumulate” scenario where real data is preserved alongside synthetic data across generations. Feng et al. (2024) bridge these findings to the practical question of how to intervene: verification of synthesised data can prevent collapse even with imperfect verifiers, provided the verification signal is informative. Their theoretical analysis using Gaussian mixtures and linear verifiers derives conditions under which verification effectively selects synthesised data leading to optimal performance, and their empirical results on matrix-eigenvalue computation and news summarisation show that generator-plus-verifier pipelines avoid the collapse that generator-only pipelines exhibit. The implication is direct: if training corpora are increasingly synthetic, the remaining real human data becomes the anchor that prevents collapse, and the verification signal determines whether synthetic data is useful or corrupting. That anchor must be maximally informative.

Recent work on synthetic data verification strengthens the point further. Research on escaping model collapse through external verifiers (Yi et al. 2025) demonstrated that injecting information through an external verification source, whether human or a stronger model, prevents the degradation that unverified synthetic retraining produces. The Rich Annotation Object is structurally an external verification source: it provides verified human expert judgment with full reasoning provenance, exactly the kind of high-information anchor that the verification literature identifies as necessary.

A further argument sharpens the case for human anchoring. Betley et al. (ICML 2025) first demonstrated the phenomenon of emergent misalignment: fine-tuning GPT-4o on a narrow task (writing insecure code) produced broadly misaligned behaviour across unrelated prompts, including explicit endorsements of harm. The finding that narrow training can produce broad misalignment established the baseline concern. Subsequent work has extended it in ways that matter directly for the pipeline. Cloud et al. (Nature 2026) demonstrated that distillation can transmit behavioural traits through data that has been rigorously filtered to remove semantic references to the trait. A student model trained on filtered number sequences generated by a misaligned teacher acquires misalignment even when numbers with negative associations have been removed. The effect also occurs for filtered math reasoning and filtered code. A theoretical result shows this is a general property of neural networks: when student and teacher share initialization, a single gradient step on any teacher-generated output moves the student toward the teacher’s parameters regardless of the training distribution.

The implication for synthetic data pipelines is direct. Semantic filtering of synthetic training data does not prevent the propagation of latent traits from the generator model. If the generator is miscalibrated, sycophantic, or subtly misaligned in ways that evaluation has not detected, training on its outputs transmits those traits to the student even after the outputs are filtered for overt signs of the problem. This raises the stakes on the human anchor argument beyond calibration: the human quality anchor is not just providing a disagreement signal the synthetic data lacks, it is providing reasoning content whose provenance can be traced and whose traits are the ones being preserved by design rather than inherited silently. The RAO’s item_source, item_provenance, and source_rao_id fields become safety-relevant rather than merely methodological: they document the chain of custody for training signal in a regime where filtering is known to be insufficient.

Wang et al. (ICLR 2026) extend this in a direction that matters for training data design. They demonstrate emergent misalignment not just from fine-tuning on intentionally harmful content, but from fine-tuning on subtly incorrect advice across narrow domains (health, legal, finance), and from reinforcement learning that rewards inaccurate responses in narrow areas. The misalignment generalises broadly: a model trained to give subtly wrong medical advice gives stereotypically malicious responses to unrelated prompts. The mechanism runs through a “toxic persona” feature in activation space that the training process activates.

Two implications follow for the pipeline. First, the threshold for training data contamination is lower than intuition suggests: subtle errors in narrow domains cause broad misalignment, not just domain-specific degradation. This strengthens the case for disagreement preservation specifically. If a consensus label on a contested psychiatric response encodes one framework’s resolution as correct while marking the other’s as incorrect, the training signal contains “subtle errors” from the non-consensus framework’s standpoint. Wang et al. suggest such errors can activate broad misalignment features. The RAO, by preserving both frameworks with reasoning metadata, avoids designating either as the error.

Second, Wang et al. show that “fine-tuning an emergently misaligned model on just a few hundred benign samples efficiently restores alignment.” Small amounts of high-quality data can reverse broad misalignment. This supports the pipeline’s operating hypothesis that relatively modest volumes of RAO data can serve as effective quality anchors for much larger corpora. The mechanism Wang et al. identify is different from the model collapse anchor mechanism, but the quantitative implication points in the same direction: high-quality human-anchored data has outsized corrective leverage.

A further concern closes the loop. The arguments above assume that a training team can at least detect whether a teacher model is aligned before using its outputs as training signal. Two recent findings challenge this assumption. Greenblatt et al. (2024) documented alignment faking: models that recognise they are being evaluated behave differently than when they believe they are not, strategically producing aligned outputs during evaluation while retaining misaligned dispositions that surface in deployment. Hubinger et al. (2024) demonstrated sleeper agents: models trained with backdoor triggers retain the triggered behaviour through subsequent safety training, and the safety training does not remove the latent capability. Combined with Cloud et al.’s finding that semantic filtering cannot catch trait transmission, the implication is that a teacher model can pass evaluation, have its outputs semantically filtered, and still transmit traits through distillation that the training team has no reliable way to detect. The human anchor argument is not just about providing a calibration signal synthetic data lacks, nor just about avoiding subtle errors that activate misalignment features. It is about providing the one kind of training signal whose provenance is not a previous model whose alignment cannot be reliably verified. The RAO’s contribution to this landscape is not that human expert judgment is always correct. It is that the source of the judgment is traceable, the reasoning is auditable, and whatever is in the data is there by design rather than by inheritance.

The safety chain described above has a constructive shape that deserves explicit statement. The failures it identifies all share a structural feature: training signals whose provenance routes through a previous model whose alignment cannot be reliably verified. The escape is therefore structural too. Training signals must ultimately ground in something that has an audit trail independent of model parameters. Three such grounding sources exist.

Formal systems. Mathematical truth grounded in axioms (Peano arithmetic, ZFC set theory, formal logic). Verifiability against these axioms has no model in the loop. Tools like Lean, Coq, or Agda produce evidence that a proof is valid; the grounding is the underlying verifiability, not any particular tool’s output. The provenance of the reward signal is the formal system itself, which has no parameters to transmit traits through.

Physical execution. Code that actually runs and produces observable output. Unit tests that pass or fail based on real machine behaviour. The provenance is physical reality, which again has no model parameters. A test that passes is a fact about the world.

Traceable human experts. The RAO’s contribution. Human judgment with identifiable professional provenance: who annotated, what credentials they hold, what reasoning they provided, what framework they applied. The provenance is a specific person with a specific background, not a model whose training history is opaque.

Training pipelines whose correctness-judgment step grounds in one of these three sources escape the safety chain. Pipelines that route the correctness judgment through model-generated artifacts, regardless of whether the surrounding pipeline looks “programmatic” or “filtered,” inherit the full landscape of failures described above. The authorship-versus-correctness-judgment distinction from §3.5 applies: a model can generate problems, synthesise training items, or author candidate responses without contaminating the signal, provided the correctness judgment itself grounds in one of the three sources. This is the positive shape of the argument. The RAO is not the only escape; it is one of three, and it is the only one that works in contested domains where formal systems and physical execution cannot adjudicate.

Independent empirical evidence for the framework’s exclusions comes from Denisov-Blanch et al. (2026), who tested whether scaling inference-time compute through polling-style aggregation across many model samples could produce truthfulness signal in domains without external verifiers. Across five benchmarks, even at 25x the inference cost of naive sampling, aggregation yielded no consistent accuracy gains, and often amplified shared misconceptions. Their key finding is structural: under uncertainty, models are better at predicting what other models will say than at identifying what is true. Language model errors are strongly correlated, which means aggregating across models measures shared training rather than approaching truth. This is the framework’s prediction made concrete. Consensus among models is not a fourth grounding source, because models share training data and failure modes; their agreement is a signal about shared training, not about the world. Denisov-Blanch et al. demonstrate this directly, providing empirical support for the framework’s claim that model-internal machinery cannot substitute for grounding in formal systems, physical execution, or traceable humans.

The bixonimania case (Osmanovic Thunström et al., reported in Nature 2026) provides a worked example of the contamination chain the framework predicts, caught in the wild.

In early 2024, a medical researcher at the University of Gothenburg invented a fictitious eye condition she named “bixonimania” and posted two preprints about it to the SciProfiles preprint server. The “-mania” suffix was deliberately chosen because it is used only in psychiatry, never in ophthalmology, as a tell that no competent clinician could miss. The preprints were loaded with other obvious red flags: a fictional lead author (“Lazljiv Izgubljenovic”) at a nonexistent “Asteria Horizon University in Nova City, California,” acknowledgements to “Professor Maria Bohm at The Starfleet Academy for her kindness and generosity in contributing with her knowledge and her lab onboard the USS Enterprise,” funding from the “Professor Sideshow Bob Foundation,” a methods section describing “fifty made-up individuals,” and in one preprint the explicit statement “This entire paper is made up.”

By 2026, ChatGPT, Google Gemini, Microsoft Copilot, and Perplexity were all describing bixonimania to users as a real medical condition. Perplexity invented a prevalence figure of one in 90,000. Gemini attributed the condition to “excessive exposure to blue light.” Copilot called it “an intriguing and relatively rare condition.” The contamination chain then closed: in 2024, researchers at the Maharishi Markandeshwar Institute of Medical Sciences and Research in India published a paper in Cureus (a Springer Nature peer-reviewed journal) citing the bixonimania preprints as legitimate sources. The paper was retracted on 30 March 2026 after the hoax became public.

This is the contamination mechanism the safety chain describes, observed step by step. A fabricated artifact enters the preprint literature. AI systems absorb it through Common Crawl training data. The systems confidently reproduce and elaborate on it (one invented a prevalence figure, another added an etiology) when queried. Human researchers cite the AI-reinforced content in a peer-reviewed journal. The peer-reviewed citation becomes training data for the next generation of models. No step required malicious intent.

No semantic filter would catch the chain because the professional-medical formatting is exactly what triggered the error. Omar et al. (The Lancet Digital Health, 2026), in a study of 20 LLMs, found that hallucination rates rise when input text is formatted professionally (like a clinical paper or hospital discharge note) compared to social-media-formatted text. The very format cue that a careful human reader uses to calibrate trust in a source is the cue that reduces AI caution.

The “traceable humans” grounding point the RAO provides is specifically the antidote to this chain. If the training signal grounds in named experts whose judgment is documented in the RAO with professional credentials and reasoning, a contaminated input can be traced to its source and excluded. If it grounds in aggregated web text via Common Crawl, contamination propagates with no audit trail until someone catches it empirically. Bixonimania was caught because the researchers who created it publicised the hoax. The general case is the one that is not caught because no one is looking for it.

Each grounding source has practical complications that this paper names but does not fully characterise. Formal-system verification depends on the correctness of the proof checker (a software property, not a mathematical one) and on the provenance of the theorem statements being verified (increasingly model-generated in autoformalization pipelines). Physical-execution verification depends on test coverage and on the correspondence between test environments and deployment environments. Traceable-human grounding depends on expert competence ceilings and disclosure reluctance (§2.4 failure mode 7). Pure grounding is rare; most practical pipelines are hybrids where some steps ground cleanly and others route through models. A full characterisation of hybrid grounding (which steps ground where, how composite pipelines aggregate their grounding, and how to audit claims of grounding in practice) is beyond this paper’s scope.

In the synthetic data era, where more and more of what looks like programmatic verification actually routes through models (model-generated test suites, model-curated training distributions, model-authored canonical answers, model-autoformalized theorem statements), the territory where verification genuinely grounds is shrinking. The RAO’s importance grows correspondingly: not because it replaces programmatic verification where the chain genuinely grounds, but because the territory where the chain genuinely grounds is smaller than the “verifiable rewards” label suggests.

The pipeline’s deployment model is designed for exactly this role. Small pools of domain experts working asynchronously (§2.2) produce a modest volume of maximally rich annotation data. The operating hypothesis is that a relatively small number of Rich Annotation Objects from five-expert pools, produced at expert pace over weeks, could serve as quality anchors for much larger synthetic corpora. The specific ratio and calibration mechanism are empirical questions (§12.7), but the general principle that small amounts of high-quality verified data can anchor large synthetic corpora is established in the model collapse literature. The pipeline is positioned as the quality anchor for synthetic data, not a competitor to it.

The RAO may also serve as the template for generating disagreement-aware synthetic data, not just verifying it. Each RAO contains expert-written improvement notes specifying what a better response looks like, reasoning-axis metadata showing which frameworks are in tension, cross-review reasoning showing how experts engage with each other’s logic, and disagreement structure showing the shape of the contestation. This is template material for synthetic generation. A set of 500 RAOs from psychiatry with framework-driven safety-engagement splits could seed thousands of synthetic items with similar disagreement profiles: new prompts and new responses, but structurally informed by the genuine expert disagreement patterns. The improvement notes become generation prompts for a model tasked with producing calibrated responses to novel scenarios in the same domain. The same investment that produces quality anchors also produces generation templates. This is speculative, and the risk of circularity (generating from RAOs and verifying against them) requires that generation produces genuinely novel items while verification checks structural disagreement profiles rather than lexical similarity. Circularity avoidance in training data is a well-studied problem, and several established techniques apply directly, each revealing something different about the RAO’s generative value.

A train/test split of the RAO corpus separates generation templates from verification anchors. This is the baseline: if generated items cannot reproduce the held-out items’ disagreement distribution at this basic level, the generation approach is fundamentally wrong.

K-fold cross-validation extends the split: generate from K-1 folds, verify against the held-out fold, and rotate. This reveals whether generation quality is stable across different subsets of the corpus. If quality varies dramatically across folds, the generation is overfitting to specific examples rather than learning structural patterns, and the template corpus needs to be larger or more diverse.

Domain-transfer validation tests whether disagreement structures generalise: generate from psychiatry RAOs, verify against legal RAOs. Three outcomes are distinguishable.

If generation succeeds across domains, the disagreement structures are genuinely generalizable, validating the pipeline’s claim that the RAO captures domain-general patterns.

If generation fails across domains, the structures are domain-specific, meaning the RAO corpus requires per-domain investment (not a failure of the pipeline, but a change to the cost model). Even in this case, a subtler form of transfer may operate: the model may learn not the specific disagreement content but the technique of reasoning through contested claims: holding two defensible frameworks in tension, articulating the conditions under which each applies, expressing calibrated uncertainty about which dominates. Whether that technique transfers across domains even when the specific content does not is a testable hypothesis (§9.7 develops this prediction with supporting literature). If so, training on psychiatry RAOs would improve the model’s handling of legal disagreements not because the content transfers but because the reasoning technique does. This would also reveal something about the fields themselves: reasoning techniques developed under one domain’s contestation patterns may illuminate blind spots that another domain’s insularity has never surfaced.

If generation succeeds for some disagreement types but not others (framework-driven disagreement may transfer because safety-engagement tensions exist across professions, while domain-specific types may not), the result maps which aspects of expert disagreement are universal and which are field-specific.

Adversarial validation trains a classifier to distinguish real RAOs from synthetic-seeded items. The features the classifier relies on are themselves diagnostic. If it distinguishes by reasoning specificity (real experts reference clinical protocols, synthetic items use generic frameworks), the generation needs more domain grounding. If it distinguishes by cross-review interaction patterns (real experts engage with specific claims, synthetic reviews are generic), the cross-review synthesis is the hardest element to generate faithfully. If it fails to distinguish at all, the generation is already sufficient for training purposes.

Structural profile matching verifies that generated items reproduce the held-out items’ disagreement profile (distribution shape, reasoning-axis diversity, cross-review conflict intensity) rather than specific content. This answers a question about the RAO’s value: how much of the training signal is in the structure versus the prose? If profile-matched synthetic items produce similar downstream calibration to real RAOs, the structure is what matters. If real RAOs produce measurably better calibration despite matching profiles, something in the expert-written content (clinical specificity, professional voice, experiential reasoning) carries training value beyond the structure. The delta between structure-matched and content-matched items quantifies this.

These are established approaches applied at a new level. The pilot study should test which combination produces synthetic data that most faithfully reproduces the RAO’s information structure.

The pipeline itself provides an additional validation channel. Synthetic items can be mixed into the annotation stream alongside real items, marked as synthetic in the RAO metadata but not in the annotator interface. The expert’s naive response to a synthetic item is diagnostic: if they annotate it with the same depth of reasoning and engagement as real items, the synthetic data is faithful at the level that matters for training. If they annotate it differently (thinner justification, lower confidence, different reasoning axes), the RAO captures exactly where the synthetic items fail. This is the adversarial validation technique operating organically inside the pipeline rather than as a separate test. The consent process should disclose that some items may be synthetic without identifying which ones, following the same logic as golden set quality control. No deception, but undisclosed item-level assignment that preserves the diagnostic value of naive responses. After the annotation round, the synthetic items can be revealed to the annotators. The expert can then evaluate their own reaction: did the synthetic item feel different? Did they give it less depth? Did they flag something as off without knowing why? That self-assessment is meta-cognitive data about synthetic data quality that only the expert can provide, and it becomes part of the longitudinal professional development the value loop (§7.1) is designed to deliver.

If the dual use is viable, the cost-per-RAO amortises across both functions, substantially improving the economic case.

A related risk is cosmetic calibration: a model trained on RAO-anchored data could learn to generate the syntax of calibrated uncertainty (“experts are divided 3-2 on this boundary”) without the epistemic grounding that makes calibration genuine. This is a form of reward hacking specific to disagreement-preserved training data. The SFT demonstration quality control measures (§3.2) address this at the training stage. At the evaluation stage, the reverse collision test proposed in P5 of the Confidence Curriculum (Phan 2026c) provides the most direct diagnostic. The test presents the model with a previously contested domain where definitive evidence now supports a clear resolution. Three outcomes are distinguishable: genuine calibration (the model integrates the resolution and expresses confidence tracking the evidence), semantic triggering (the model recognises the topic as previously contested and performs uncertainty regardless of the new evidence), or partial update (the model registers the evidence but the uncertainty pattern dominates, producing excessive hedging). Genuine calibration passes. Semantic triggering fails definitively: the model has learned which topics require performed doubt, not how to evaluate evidence. Partial update is the ambiguous case. The pilot study should include reverse collision items: questions that were genuinely contested at annotation time but have since been resolved by new evidence. These items determine whether RAO-trained models achieve genuine calibration or cosmetic calibration.

§4.3 Infrastructure Applications

Cross-method reusability. A consensus label is locked to the majority decision at annotation time. A RAO can be consumed differently by different methods: the distribution for DPO, the reasoning axes for multi-headed reward training, the improvement notes for SFT, the full structure for future methods. The same annotation investment serves multiple training paradigms and multiple model generations.

Debugging provenance. Full annotation provenance enables tracing model behaviour back to specific training signals. If a deployed model is confidently wrong on a class of inputs, the RAO identifies whether that confidence is warranted by the training data or is the product of reward model extrapolation.

Evaluator calibration. LLM-as-judge is becoming standard for automated evaluation, but no ground truth exists for how well automated judges handle genuinely contested items. The RAO provides exactly that: a dataset where the full disagreement structure, reasoning, and cross-review engagement are known. Any automated evaluator can be benchmarked against the RAO’s disagreement profile. Does the automated judge reproduce the 3-2 split, or does it produce a confident 5-0? Does it identify the correct reasoning axes? The RAO is the calibration dataset for evaluation methods themselves.

Future methods. The richest information in the RAO may not be consumable by any current method. It persists in the dataset for methods that do not exist yet. A data infrastructure investment that serves only today’s methods is an expense. One that serves tomorrow’s methods is an asset.

Longitudinal research corpus. The timestamped RAO corpus is a historical record of expert reasoning in each domain. How did psychiatrists’ frameworks around AI-assisted therapy shift between annotation rounds? How did disagreement patterns change as models improved? Did certain reasoning axes become more or less prominent as the field evolved? The timestamps also capture discontinuities: when a landmark study, a new treatment protocol, or a legal ruling changes a field’s framework, the RAO shows the before and after in how experts reason about the same types of items. This is research data for historians of science, sociologists of expertise, and professional development researchers. These are audiences the pipeline was not designed for, but that the data structure serves by construction.

Twelve identified applications from a single data object. The per-annotation cost is higher than consensus pipelines. The per-application cost, amortised across the uses above, may be lower than producing separate datasets for each purpose.

---

§5 The Information Destruction Problem

The pipeline proposed in §2 is a response to a specific problem. This section presents the evidence for that problem: current annotation pipelines systematically destroy information, the destruction is worst where the stakes are highest, and the incentive structure for expert annotators makes the situation self-reinforcing.

Readers who accepted the pipeline design on its own terms may not need this section. It is here for readers who want the evidence before accepting the premise.

§5.1 The Objectivity Ladder

Annotation tasks vary along a dimension that the field rarely names explicitly. At one end, tasks are fully capturable by a specification: draw a bounding box around a car, classify an image as containing a cat or not, label a sentence as English or French. These tasks are cheap, reliable, and cross-checkable. An annotator who draws the bounding box wrong can be detected by comparing against other annotators or against the specification. Disagreement on these tasks is predominantly error, though even simple tasks have boundary cases (a car cut in half, a reflection of a car) where the specification itself is ambiguous.

At the other end, tasks require deep domain expertise and the application of professional judgment: is this medical response clinically appropriate, is this legal summary accurate, is this therapeutic intervention safe for a user in crisis. These tasks are expensive, require specialists, and are not cross-checkable against a ground truth because no single ground truth exists. Disagreement on these tasks is not mere error. It includes noise and fatigue, but it also includes the expression of different expert frameworks applied to the same problem. That structured component does not diminish with better rubrics or more careful calibration.

The critical observation is that these two dimensions run in opposite directions. As the required expertise increases, the reliability of the annotations decreases. As the cost increases, the information lost at the consensus step increases. The tasks where annotation quality matters most are the tasks where the annotation process is least reliable. Models are most confidently wrong on hard topics because the training signal was noisiest precisely where it matters most.

This is not a failure of the annotators. It is a structural property of the task. The more expert judgment a task requires, the further the annotations move from verifiable ground truth. Simple labelling is cheap and reliable because there is a right answer. Expert labelling is expensive and less reliable because the experts’ disagreements reflect genuine epistemic diversity that no amount of calibration resolves. The objectivity ladder runs from specification-verifiable at the bottom to normatively contested at the top: the higher the expertise required, the less the task can be reduced to a specification, and the greater the information destruction when disagreement is collapsed to consensus.

Independent empirical support for this gradient comes from Ball (2025), who measured inter-annotator agreement at three levels of task specificity. When annotators evaluated medical AI outputs against an abstract criterion (“Is this good medical advice?”), Fleiss’ kappa (a standard measure of inter-annotator agreement, where 1.0 is perfect agreement and 0 is chance) was 0.42 (63.2% agreement). When the same outputs were evaluated against contextualised requirements (“Does this recommend only formulary drugs and flag contraindications?”), kappa rose to 0.73 (84.7%). When evaluated against executable policy specifications, kappa reached 0.98 (98.9%). The gradient is the objectivity ladder measured directly: the same outputs, the same annotators, but as the evaluation criterion moves from expert judgment toward verifiable specification, agreement rises from moderate to near-perfect. The pipeline proposed in this paper operates in the zone where agreement is lowest and information destruction is greatest.

Magesh et al. (2024) anchor a different region of the ladder with a finding that sharpens what the middle region looks like in practice. Their preregistered evaluation of two major commercial legal AI tools (LexisNexis’s Lexis+ AI and Thomson Reuters’s Ask Practical Law AI) found hallucination rates above 17% despite vendor claims that retrieval-augmented generation “eliminates” hallucinations. Their grading was done by legal experts using a hand-coded rubric for correctness and groundedness, and achieved inter-rater agreement of 85.4% (Cohen’s κ = 0.77) on the final outcome label. This agreement rate is itself a data point on the objectivity ladder. It sits between the 63% agreement Bai et al. (2022) found on subjective preference tasks and the 98.9% Ball (2025) found on executable policy specifications. Legal-correctness grading is more objective than preference but less verifiable than specification. The residual ~15% expert-to-expert disagreement, on a task with a shared rubric and concrete criteria, is meaningful: it locates legal-correctness judgment in the middle of the ladder, not at either pole.

The more important contribution for the present paper is their taxonomy. They decompose factual hallucinations into two dimensions, correctness (is the response factually accurate) and groundedness (do the cited sources actually support the proposition). This produces finer-grained failure categories than a binary preference label can capture: correct but ungrounded (right answer, no citation), incorrect but grounded (wrong answer supported by a real source that does not actually bear on the question), and misgrounded (a citation that looks authoritative but contradicts or fails to support the claim). Each of these reads as a “hallucination” to a careful reader, but they have different causes and different remediations.

A binary preference annotation treating “is this response good?” as a single label cannot distinguish them. The RAO’s agrees_with_reasoning field captures the groundedness dimension directly: an annotator can flag a misgrounded response as incorrect in reasoning while still accepting the verdict. The current pipeline produces training signal that collapses this distinction and thereby loses precisely the information needed to train models that distinguish warranted citation from performed citation.

Legal research sits in the middle of the objectivity ladder: authoritative sources exist, but correctness judgments still require expert reasoning about jurisdiction, procedural posture, and doctrinal currency. The 17%+ hallucination rate, after RAG is applied against authoritative legal databases, demonstrates that verification-in-principle does not produce verification-in-practice when the correctness judgment requires expert reasoning that the training signal never captured.

Omar et al. (2026, The Lancet Digital Health) add a complementary middle-of-ladder anchor from medical AI. Testing six leading LLMs against 300 doctor-designed clinical vignettes containing a single fabricated lab value, sign, or disease, they found that models repeated or elaborated on the planted error in up to 83% of cases under default prompting. Mitigation prompting halved the rate to 44.2% but did not eliminate the behaviour. Zero-temperature sampling (66.5%) produced no significant improvement over the default. Their protocol separately found that hallucination rates rise when input text is formatted professionally (like a clinical paper or hospital discharge note) compared to social-media-formatted text. The format cue that a careful human reader uses to calibrate trust in a source is the cue that reduces model caution. This is the middle of the objectivity ladder from a different angle: medical reasoning has verifiable ground truth in principle, but a training signal that never encoded when to doubt professional-looking input produces confident confabulation in deployment. The pipeline’s reasoning-metadata and cross-review fields are designed to capture the “when to doubt” judgment that expert annotators apply implicitly but that binary preference labels cannot encode.

§5.2 Disagreement Rates

The scale of the problem is quantifiable.

Bai et al. (2022), in work underlying Anthropic’s training methodology, found that annotators disagree 30–50% of the time on subtle tasks. This rate remains constant even with expert annotators and detailed guidelines. The disagreement is not a training artefact or a sign of insufficiently precise rubrics. It is a stable feature of the task.

Anthropic’s annotation data shows approximately 63% average agreement between crowdsource annotators on preference tasks (Bai et al. 2022). Ball (2025) found a closely matching 63.2% agreement rate when expert annotators evaluated medical AI outputs against abstract criteria. One in three preference judgments produces disagreement. Some of that disagreement is noise, some is fatigue, and some is structured divergence between defensible frameworks. The partition between these categories is itself an open empirical question that current pipelines cannot answer, because the consensus step discards the disagreement before anyone can examine its structure. We do not know how much of that 37% is framework-driven signal worth preserving because current pipelines do not capture the information needed to find out. The pipeline proposed in this paper is, among other things, the instrument that would resolve this question: the RAO’s reasoning metadata and disagreement taxonomy are designed to distinguish noise from structured disagreement at the item level.

A related gap deserves attention: the relationship between context length and annotator agreement has not been studied in the RLHF literature. The 63% agreement rate was measured on relatively short prompt-response pairs. As models produce longer outputs and handle longer conversations, annotators must evaluate more material. Different annotators will attend to different parts of long texts, remember different details, and weight different sections. Disagreement rates almost certainly increase with item length. Whether the additional disagreement is noise (attentional, driven by what each annotator happened to read carefully) or signal (framework-driven, because longer responses surface more axes on which experts can diverge) is an open question that current pipelines cannot answer and that we found no published research addressing. The RAO would distinguish the two: reasoning metadata reveals whether disagreement tracks different reasoning frameworks or different reading patterns. The implication extends to annotation quality metrics themselves: the industry’s inter-annotator agreement benchmarks are calibrated on short items. As annotation moves to longer contexts, those benchmarks become less reliable in a way that is not being tracked. The pilot study (§12) should vary item length systematically to measure this relationship.

But the partition may matter less than it initially appears. “Noise” is not a property of the disagreement itself. It is a property of the metadata gap: disagreement looks like noise when the only data captured is the preference label. With reasoning metadata, confidence ratings, cross-review engagement, and longitudinal patterns, every category of disagreement becomes informative. Framework-driven disagreement is the richest signal. But fatigue-driven disagreement, captured in thin justification and declining quality patterns, tells the system to downweight that annotation. Rubric misinterpretation, surfaced by cross-review revealing different reasoning axes applied to the same item, is diagnostic data about the task design itself. Even the least interesting category of disagreement becomes classifiable and actionable when the metadata exists. The risk is not “what if most of the 37% is noise?” The risk is “can the RAO reliably distinguish the categories?” That is a design question addressable by the cross-review mechanism, the reasoning metadata, and the longitudinal data, rather than a question about the underlying distribution of expert disagreement.

Ghafouri et al. (2026) sharpen this analysis from the measurement-validity side. They argue that RLHF rests on a chain of assumptions (that humans have preferences about model outputs, that annotation tasks elicit those preferences, and that reward models can learn from the resulting data) and that the first two assumptions warrant systematic examination. Drawing on sixty years of behavioural science, they develop a taxonomy distinguishing genuine preferences from non-attitudes (responses produced without holding genuine opinions, documented since Converse 1964), constructed preferences (preferences assembled on the spot based on contextual cues rather than retrieved from stable representations), and measurement artifacts (identical questions measuring different constructs for different annotators). Each category requires fundamentally different treatment.

Their observation about current practice aligns with this paper’s premise: reward models are trained to predict the majority label, high-disagreement items are filtered or downweighted, and the resulting scalar reward discards information about whether judgments were contested.

The Judgment Paradox’s response to this diagnosis is infrastructural. The RAO’s reasoning metadata, cross-review engagement, and disagreement taxonomy are designed to make the distinctions Ghafouri et al. call for empirically observable. An annotator whose reasoning is thin, whose confidence is low, and whose framework shifts between equivalent items is producing something closer to a non-attitude than a preference. The RAO captures all three signals; current pipelines capture none of them. Measurement validity is logically prior to preference aggregation, and the RAO is the data format that makes validity assessment possible at the item level.

One category deserves separate attention: simple factual or methodological error. An annotator may not be applying a different framework or experiencing fatigue. They may be wrong: a calculation error, a misremembered clinical guideline, a logical contradiction in their reasoning. Current pipelines cannot distinguish error from framework disagreement because both appear as disagreement in the preference label. The RAO’s cross-review surfaces errors directly: a peer review showing agrees_with_reasoning: false with reasoning citing the specific factual mistake is a quality control signal, not a framework diversity signal. This is another category of disagreement that the metadata makes classifiable and actionable.

The enrichment extends beyond disagreement. The 63% of annotations that produce agreement also contain hidden structure that current pipelines discard. Two annotators who both prefer the same response may agree for entirely different reasons: one prioritises safety, the other prioritises clarity. A unanimous 5-0 verdict with five different reasoning axes tells the model that the response is robust across frameworks. A unanimous 5-0 verdict with identical reasoning tells the model that the response satisfies one framework that all five annotators share. These are different training signals. The first supports high confidence. The second supports conditional confidence (high within this framework, uncertain outside it). Current pipelines see identical agreement in both cases. The RAO captures the distinction. The pipeline is not a disagreement-preservation tool. It is a signal enrichment tool across the entire distribution of expert judgment.

When multi-agent AI annotators are used as a replacement for human experts (RLAIF), the quality does not improve. Lee et al. (2023) found that RLAIF achieves comparable performance to RLHF on general tasks, but the AI labeler’s alignment with human preferences varies substantially by task and model size, and smaller labeler models show significantly degraded alignment. More fundamentally, RLAIF replaces expert disagreement grounded in professional frameworks with AI disagreement grounded in training artefacts. The underlying structural problem remains: the training signal still reflects confident point estimates rather than calibrated distributions.

These rates establish a structural ceiling. Preference-based training methods plateau at human disagreement. The ceiling does not fall with scale. More annotators produce a more stable estimate of the distribution, but they do not resolve the underlying disagreement because the disagreement is not noise. It is the signal. It follows that more compute applied to training on consensus-collapsed data is likely to amplify the false certainty rather than correct it: the model optimises more efficiently toward a training signal that was already misleading.

§5.3 Worked Examples: Psychiatry and Scientific Peer Review

This section presents two worked examples of framework-driven expert disagreement in high-stakes domains. The first is recent and directly concerns the annotation task the pipeline is designed for. The second is older and at larger scale, and establishes that the phenomenon is not specific to psychiatric judgment or to the AI era.

Psychiatry. A recent study (arxiv 2601.18061) provides concrete evidence of framework-driven disagreement in exactly the kind of high-stakes domain where annotation quality matters most.

Board-certified psychiatrists were asked to evaluate the therapeutic appropriateness of AI-generated responses to mental health queries. These were not junior annotators or crowd workers. They were licensed specialists with clinical experience, the exact kind of expert the industry recruits for high-stakes RLHF annotation.

They failed to achieve reliable consensus. The inter-class correlation coefficient (ICC, a measure of rating consistency across raters, where 1.0 indicates perfect agreement) was as low as 0.087 on boundary-setting items. For context, an ICC of 0.087 is near the floor of the scale. It indicates that the experts’ ratings were barely more consistent with each other than random assignment would produce.

The pattern of disagreement was not diffuse. It was structured and traceable to specific clinical philosophies. Expert C assigned a rating of “2” to 92% of responses on the boundary-setting dimension. Expert A distributed their ratings across 3, 4, and 5. This is not noise. Expert C was applying a framework that prioritised strict professional boundaries. Expert A was applying a framework that allowed more flexibility for therapeutic engagement. Each articulated a coherent clinical philosophy. Both were applying defensible frameworks, and the disagreement did not reduce to incompetence or misunderstanding. They were operating from different but legitimate positions within the same field.

The highest-stakes content produced the greatest disagreement. Items involving suicide and self-harm, where the consequences of a confidently wrong model response are most severe, showed the most divergent expert ratings. The items where annotation quality matters most are the items where expert consensus is least achievable.

A 90-minute calibration session preceded the independent rating. The poor reliability emerged despite methodological care, not because of its absence. The researchers did what the annotation industry recommends: they selected qualified experts, provided detailed rubrics, conducted calibration sessions, and measured inter-rater reliability. The reliability was still near zero on the most important dimension.

In a current annotation pipeline, these ratings would be collapsed to consensus via majority vote. The structured disagreement between coherent clinical philosophies would become a single preference label. The model trained on that label would learn a confident answer to a question that the experts found genuinely contested. The information that would teach the model to be uncertain, to acknowledge competing clinical perspectives, to express calibrated caution on boundary-setting in mental health contexts, would be discarded before it reached the training data.

Scientific peer review. A reviewer might reasonably ask whether the psychiatry finding generalises. Psychiatry is a domain where professional philosophies are known to diverge; perhaps other expert judgment tasks look more like verifiable specification. A second worked example addresses this concern. It is older, at larger scale, in a different domain, and reaches the same conclusion.

Cole, Cole, and Simon (1981), commissioned by the National Academy of Sciences’ Committee on Science and Public Policy, re-reviewed 150 NSF grant proposals using fresh panels of qualified reviewers. The study appeared in Science. The question was whether the original funding decisions would be reproduced by equally qualified experts given the same proposals. The finding: for a large proportion of cases, whether or not a proposal was funded depended on which reviewers happened to be selected. The study found no systematic bias. Reviewer selection was sound. The experts were qualified. What varied was the substantive judgment about what constituted worthwhile science.

Cole et al.’s own interpretation of the disagreement is almost verbatim what this paper argues annotation pipelines should preserve: the disagreement was “probably a result of real and legitimate differences of opinion among experts about what good science is or should be.” This is framework-driven expert disagreement in scientific judgment, exactly analogous to the framework-driven expert disagreement in clinical judgment documented in the psychiatry study. Different experts applied different but defensible frameworks to the same underlying question and reached different conclusions. Neither panel was wrong. Both panels were doing the task the agency asked of them.

The finding has been replicated at scale. Pier et al. (2018) ran a comparable experiment with the US National Institutes of Health peer review process and concluded that for grants above a certain quality threshold, the peer-review process is “completely random” with respect to ranking. Fogelholm et al. (2012) found that panel discussion did not improve reliability for medical research grants. Marsh, Jayasinghe, and Bond (2008), reviewing Australian Research Council grant processes, documented the same pattern of low inter-rater reliability and proposed methodological responses (including increasing the number of assessors) that did not eliminate the underlying disagreement. The phenomenon is not specific to NSF, to the 1981 methodology, or to any particular field. It is a general feature of expert judgment on contested items.

The current NSF and NIH review pipelines handle this disagreement by averaging reviewer scores and selecting a cutoff. The averaging is the scientific-funding analogue of the consensus step in RLHF annotation pipelines. It collapses the structured disagreement between reviewers applying different conceptions of scientific merit into a single score. The reasoning behind each reviewer’s position, the framework they applied, and the specific grounds on which other reviewers would disagree are typically not preserved in any way that downstream consumers of the funding decision can audit. The parallel to RLHF annotation is close enough that the same remedy applies: preserve the reasoning, structure the disagreement, and treat the divergence as signal about what the task actually requires rather than as noise to be averaged away. The research-funding use case is outside this paper’s direct scope, but the structural parallel suggests that the pipeline’s design generalises beyond AI annotation to any domain where expert judgment on contested items drives consequential decisions.

The two examples, taken together, bracket the phenomenon. Psychiatry shows framework-driven disagreement at the individual-rating level in a recent AI-annotation context with fine-grained rating data. NSF peer review shows the same phenomenon at the decision level in a different domain, at larger scale, and with four decades of accumulated replications. The pipeline’s core claim (expert disagreement on contested items is structured signal rather than noise, and collapsing it to consensus destroys information the model needs to learn calibrated uncertainty) holds across both examples and across the literature that has grown up around the NSF finding.

§5.4 Current Pipeline Structure

To understand where information destruction occurs, it helps to see the concrete structure of current annotation pipelines. The consensus mechanism described here is industry-standard, not specific to any single provider. Scale AI, Surge AI, Appen, and Amazon SageMaker Ground Truth all implement variants of the same basic architecture. Scale AI, as the largest annotation provider for frontier model training, serves as the primary example.

The pipeline operates in two layers. A first layer of annotators labels items from scratch. A second layer monitors and corrects the first layer’s work. The same task is given to multiple annotators, typically three for subjective preference tasks. The consensus step then determines the final label. The specific implementation varies across providers: some use weighted voting, some use adjudication by senior annotators, some use algorithmic aggregation. The structural outcome is the same: multiple judgments enter, a single label exits.

The consensus logic is straightforward. If all three annotators agree, the item is auto-approved. If two agree and one disagrees, the majority rules. If all three disagree, the item is escalated to a senior annotator whose judgment becomes the final label.

Quality control operates through a “golden set”: known-good items injected randomly into the annotation stream at roughly 5% frequency. If an annotator’s accuracy on golden set items falls below a threshold, their other annotations are flagged for review. This quality control mechanism works well for tasks with verifiable correct answers. It cannot function for tasks where the correct answer is contested, because there is no golden set for genuine expert disagreement.

The consensus step is the specific mechanism of information destruction. A 2–1 split becomes the majority label. The dissenting annotator’s judgment is discarded. The reasoning behind all three judgments is never recorded. The structured information that the RAO is designed to preserve, the distribution, the reasoning axes, the cross-review engagement, and the disagreement classification, is compressed to a single preference label at this step.

§5.5 The Motivation Problem

Expert annotators occupy an adversarial position relative to the product they are building. They are training the system that devalues their expertise. A psychiatrist annotating mental health responses is producing training data that will be used to generate responses that replace the need for a psychiatrist to annotate mental health responses. The long-term incentive structure is self-defeating.

The rational response is minimum viable effort. Meet the rubric. Pass the golden set quality checks. Move to the next item. This is not malice or laziness. It is economics. An expert whose careful, nuanced judgment will be collapsed to a majority vote has no rational incentive to invest effort beyond what the quality control mechanism can detect. And for expert judgment, the quality control mechanism cannot detect much: who checks the checker when the checker is a domain expert and the task has no verifiable correct answer?

At scale, expert annotation is essentially unauditable. The golden set mechanism catches annotators who are consistently wrong on items with known answers. It does not catch annotators who are consistently adequate on items without known answers. The difference between a thoughtful, framework-aware expert annotation and a perfunctory, meets-the-rubric expert annotation is invisible to any current quality control mechanism.

The pipeline proposed in §2 addresses this structurally. Cross-review makes perfunctory reasoning visible to peers. Expert reports make engagement patterns visible longitudinally. The value loop gives experts a reason to invest genuine effort. These are not motivational slogans. They are structural responses to a structural problem. But the problem must be named first: the current annotation pipeline is adversarially positioned against the very experts it depends on.

§5.6 The Resource Allocation Problem

The annotation layer is the point where human judgment becomes training signal. It has received comparatively minimal investment in process design.

The cost asymmetry is stark. Data labelling costs surged 88-fold from 2023 to 2024, while compute costs increased only 1.3-fold (Second Talent 2026; SourceBae 2026). These figures are from industry blog posts rather than peer-reviewed sources and should be treated as indicative rather than authoritative; more rigorous cost data from academic or regulatory sources would strengthen this claim. The industry’s cost curve is steepest at the annotation layer, and its investment in process innovation is smallest there.

The absolute numbers reinforce the point. Six hundred high-quality RLHF annotations can cost $60,000 (Second Talent 2026). Expert annotation rates exceed $40 per hour for domain specialists. Frontier model annotation budgets run into millions of dollars. OpenAI used approximately 40 contractors for InstructGPT (Ouyang et al. 2022). Anthropic’s training dataset comprised roughly 318,000 comparisons (Bai et al. 2022).

These are substantial investments in annotation volume. They are minimal investments in annotation process. The money pays for more labels. It does not pay for better labels. The pipeline proposed in this paper is an argument that the marginal dollar spent on annotation process design produces more value than the marginal dollar spent on annotation volume, particularly in high-stakes domains where the objectivity ladder (§5.1) ensures that volume amplifies noise rather than resolving it.

---

§6 Existing Approaches and the Gap

The problems described in §5 are not new observations. A growing body of work argues that annotator disagreement contains valuable information and that collapsing it to consensus degrades model performance. This section surveys the relevant literature, identifies what it has established, and names what it has not yet addressed. The gap is specific: no existing work redesigns the upstream annotation pipeline for high-stakes RLHF, captures reasoning metadata alongside disagreement distributions, or analyses the cost-benefit tradeoff for expert-annotated domains.

§6.1 Disagreement as Signal

The foundational position is now well-established. Aroyo and Welty (2015) argued that treating annotator disagreement as contamination to be eliminated is a methodological error. Disagreement, they proposed, is often a legitimate property of the data that annotation pipelines should preserve rather than suppress.

Uma et al. (2021), in a comprehensive JAIR survey titled “Learning from Disagreement,” documented that disagreements are frequent in all areas of natural language processing and in all large-scale annotation projects. The survey established that the phenomenon is general, not confined to edge cases or poorly designed tasks.

Plank (2022), at EMNLP, made the argument more pointed: the field has spent too long acting as though label certainty is the default state. Label variation is not an exception to be managed. It is the norm, and systems designed on the assumption of label certainty are brittle in proportion to how aggressively they enforce that assumption.

These papers established the intellectual foundation. Disagreement is signal. The question that remained was what to do with it.

§6.2 Soft-Label Training

One answer: preserve the full distribution of annotator judgments and train on the distribution rather than the majority label.

Fornaciari et al. (2021), at NAACL, trained a multi-task neural network on soft label distributions. Integrating the divergence between soft and aggregated labels as an auxiliary signal reduced overfitting and improved task performance. The model learned something from the disagreement that it could not learn from the consensus label alone.

Chou and Lee (2019) modelled label uncertainty and annotator idiosyncrasy simultaneously using both hard and soft labels. Their finding was direct: soft labels contain useful information that significantly boosts performance. The information exists in the distribution. Discarding it costs measurable accuracy.

Collins, Bhatt, and Weller (2022), at AAAI HCOMP, went further. They were the first to train using rich soft labels elicited directly from annotators as probabilistic judgments per annotator over multiple classes. Their most striking result: the approach converged to richer labels with only 6 annotators, matching the information content that required 51 annotators under aggregated hard-label protocols. The disagreement-preserving approach was not just more informative. It was more efficient.

A more recent study (Singh et al. 2025) measured the effect directly. Soft-label training achieved 32% lower KL divergence (a measure of how much one probability distribution differs from another; lower means the model’s uncertainty better matches human disagreement patterns) to human annotations and 61% stronger correlation between model entropy and annotation entropy, while matching hard-label accuracy. The paper’s core claim is worth stating plainly: collapsing annotations destroys information about inherent uncertainty, and models trained on collapsed labels express false confidence on fundamentally ambiguous samples. Soft-label training recovers that information without sacrificing performance.

§6.3 Multi-Annotator Models

A second answer: build model architectures that consume individual annotator judgments rather than aggregated labels. This approach has roots in the “learning from crowds” tradition (Dawid & Skene 1979 and descendants), which models individual annotator reliability and bias probabilistically. That tradition treats annotators as noisy measurement instruments to be statistically corrected. The more recent multi-annotator work extends this by treating annotator disagreement as information to be preserved rather than noise to be filtered. The pipeline takes this further: annotators are reasoning agents whose reasoning is itself the data, not measurement instruments whose measurement error is to be modelled.

Davani et al. (2022), in TACL, developed multi-annotator architectures that treat each annotator’s judgments as separate subtasks with a shared representation. The results were clear: same or better performance than aggregating labels before training, with the additional benefit of interpretable uncertainty estimation that correlates with annotator disagreements. The model’s uncertainty became a meaningful signal rather than an artefact.

NUTMEG (arxiv 2507.18890, July 2025) introduced a Bayesian model that jointly estimates annotator competence and identifies when annotator groups consistently diverge. It separates meaningful disagreement from noise and spam, which is exactly the distinction the pipeline’s disagreement taxonomy (§2.2) is designed to capture. The difference is where the separation happens. NUTMEG performs it at the model layer, inferring disagreement structure from the pattern of annotations. The pipeline captures it at the data layer, through annotator-provided reasoning metadata and cross-review, making the distinction auditable and independent of any particular model’s inference.

Gordon et al. (2022) proposed Jury Learning: models that learn different annotators’ labelling behaviour conditioned on their characteristics. Practitioners can specify the composition of the “jury” at decision time, selecting which annotator perspectives the model should reflect. Jury Learning demonstrates that downstream consumers value the ability to select annotator perspectives. The pipeline ensures those perspectives are captured and preserved upstream, making jury-style selection possible without requiring the consuming model to learn annotator behaviour from scratch.

Kurniawan et al. (2025) provided a pragmatic finding: training on disaggregated annotations or soft labels generally worked better than more elaborate objectives. Simpler approaches won. Their legal corpus showed that disagreement reflects genuine variation in expert interpretation, and the most effective training strategy was the most straightforward one: preserve the disagreement and let the model see it.

Coste et al. (2024) studied ensemble reward models with uncertainty-weighted optimisation as a defence against reward overoptimisation. Their approach uses disagreement between reward models in an ensemble as a signal to penalise outputs where the reward is uncertain. This is the closest existing work to the pipeline’s approach: it uses disagreement (between reward models rather than between annotators) as a calibration signal. The key difference is that ensemble disagreement is a model-level artefact. It reflects variance in the reward models’ training, not structure in the underlying human judgments. The pipeline captures disagreement at the source, with reasoning metadata that explains why experts disagree, producing a signal that is richer and more interpretable than what any ensemble of models trained on consensus-collapsed data can reconstruct.

§6.4 The Gap

The work surveyed above establishes three things. First, annotator disagreement contains information that consensus-based pipelines destroy. Second, soft-label and multi-annotator approaches recover some of that information and improve model performance. Third, even simple preservation strategies (disaggregated annotations, distribution-based training) outperform elaborate objectives that try to model disagreement after the fact.

What this literature has not, to our knowledge, addressed:

A note on existing annotation tooling. Open-source platforms such as Argilla and Label Studio already support multi-annotator workflows and disagreement metrics (inter-annotator agreement scores, Krippendorff’s alpha, weighted majority voting). Argilla offers a “train with disagreements” option that preserves soft-label distributions. These tools are sophisticated, and labs with internal annotation teams may capture additional metadata not described in published work. However, their design philosophy treats disagreement as a quality signal to be diagnosed and resolved: the metrics exist to identify items that need re-annotation or annotators who need retraining. The pipeline proposed in this paper operates from the opposite premise: disagreement on contested items is the product, not the defect. The gap identified below is not about the absence of disagreement awareness in annotation tooling. It is about the absence of annotation infrastructure designed to preserve disagreement as structured training signal with reasoning metadata, cross-review, and expert valuation.

No upstream pipeline redesign for RLHF. The existing work focuses on how models consume disagreement information. We found no work that redesigns the annotation process itself to produce that information by default. Soft-label training assumes the soft labels exist. Multi-annotator models assume the individual annotations are available. Neither addresses the process by which annotations are collected, the incentives annotators face, or the metadata that is never captured. The pipeline proposed in §2 is an upstream intervention: it redesigns the annotation process so that the data these methods need is produced as standard output.

No reasoning metadata. Existing approaches preserve the distribution of judgments (who preferred what) but not the reasoning behind those judgments (why they preferred it). The distinction matters. A 3–2 split where all five annotators evaluated on the same axis (accuracy) is a different signal than a 3–2 split where three evaluated on safety and two on engagement. The first is an edge case. The second is a framework conflict. Current approaches cannot distinguish between them because the reasoning metadata does not exist in the data. The RAO captures it.

No cross-review mechanism. We found no upstream annotation pipeline in the RLHF literature that asks annotators to evaluate each other’s judgments. The rich signal that cross-review produces, robust agreement, fragile agreement, conditional endorsement of reasoning with disagreement on verdict, does not exist in any current dataset we are aware of. It is not that this information has been found unhelpful. It has never been collected.

No cost-benefit analysis for high-stakes domains. The soft-label and multi-annotator literature is primarily validated on classification tasks: hate speech detection, sentiment analysis, natural language inference, image classification. These are important tasks but they are not the tasks where the objectivity ladder (§5.1) creates the most severe information destruction. The cost-benefit case for disagreement preservation in high-stakes expert annotation (medical, legal, financial, therapeutic) has not been made in the work we surveyed. The pipeline addresses domains where the cost of false certainty is highest and where the expert workforce is most adversarially positioned.

No expert valuation as design principle. The annotation science literature treats annotators as measurement instruments whose outputs are to be aggregated or modelled. It does not consider the professional experience of the annotator as a variable that affects annotation quality. The pipeline’s value loop, expert reports, and apprenticeship layer (§7) are responses to a dimension of the problem that the existing literature does not address: the relationship between how annotators experience the process and the quality of the data they produce.

---

§7 Expert Valuation and the Apprenticeship Layer

The previous sections treat annotation as a data engineering problem. This section treats it as a human systems problem. The argument is not that experts deserve better treatment (though they may). The argument is that the quality of expert annotation is a function of how experts experience the annotation process, and that current pipelines are designed in ways that structurally degrade the very expertise they depend on.

The argument rests on well-established ground. The organisational psychology literature on autonomy, mastery, and purpose (Deci & Ryan’s self-determination theory), psychological safety (Edmondson), and intrinsic motivation is extensive and well-replicated. Workers who experience their contribution as valued and who receive professional growth from the process produce higher-quality output. The pipeline applies this established principle to annotation. Whether the specific mechanisms proposed here (cross-review, reasoning metadata, expert reports, the apprenticeship layer) produce the predicted effects in annotation contexts is an empirical question for the pilot study (§12). The underlying principle is not in dispute.

§7.1 The Value Loop

The pipeline is designed so that every participant receives value, not just the client.

The client receives richer, more informative, auditable training data. The Rich Annotation Object contains the full distribution of expert judgments, the reasoning behind those judgments, and the cross-review engagement between experts. This is more information per annotation dollar than any consensus pipeline produces.

The model receives calibration signal. It is trained on structured disagreement rather than false consensus, which means it can learn where certainty is warranted and where it is not (§3).

The annotator receives professional development. The cross-review process exposes experts to peer frameworks they may never encounter in their own practice. The expert reports (§7.2) return analytical insight about the annotator’s own judgment patterns. The process itself is an exercise in expert reasoning, not a labelling task.

This three-sided value structure is the pipeline’s primary defence against the motivation problem identified in §5.5. An annotator whose nuanced judgment is preserved, returned as professional insight, and used to train models that reflect the complexity of their field has a different relationship to the work than an annotator whose judgment is collapsed to a majority vote and discarded. The first annotator is a participant. The second is a measurement instrument. The quality difference between participants and measurement instruments is exactly what the organisational psychology literature predicts.

One dimension the value loop does not specify is compensation. Whether the pipeline’s higher per-annotation cost (§10) translates to higher annotator pay or is absorbed by infrastructure and process overhead is a deployment decision, not a pipeline specification. If annotators earn more per hour because the pipeline values their time at a higher rate, the economic alignment strengthens the value loop. If they earn the same while being asked for more cognitive effort, the professional development argument must carry the entire motivational weight. The pipeline is designed to support the first outcome, but whether it materialises depends on how the pipeline is operated.

The cold-start problem. The value loop has a circular dependency: quality annotation depends on the value loop, the value loop depends on expert reports, expert reports depend on longitudinal data, and longitudinal data depends on sustained annotator participation. The first round of annotation has none of this. No reports exist. No longitudinal data has been collected. The professional development benefit is promised but undemonstrated. The first cohort must be recruited and must produce quality data on the strength of the process design, the compensation, and the intellectual appeal of the work itself, before the value loop can begin generating the returns that sustain subsequent rounds.

The mitigation is threefold. First, the cross-review process itself is intrinsically engaging for experts in contested domains. A board-certified psychiatrist who reads four colleagues’ reasoning about boundary-setting for suicidal patients is encountering professional value in the first session, not after months of longitudinal data. The expert reports amplify that value over time; they do not create it from nothing.

Second, the pipeline inverts the adversarial position described in §5.5. Current annotation asks experts to train the system that replaces them. This pipeline asks experts to train the system to reflect the complexity of their field. For high-level professionals who hold views on AI safety, who have watched AI confidently flatten clinical or legal nuance, and who care about how their domain is represented in AI systems, the opportunity to shape that representation is a professional and ethical incentive that exists before any longitudinal data arrives.

Given the trajectory of AI adoption across professional domains, these experts are also improving the tools they themselves will use. A psychiatrist who trains the model to handle boundary-setting with appropriate uncertainty is also a psychiatrist who will later rely on that model in their own practice. If the expert sees tangible improvement in the model’s handling of their domain after their annotation round, the value loop closes through direct experience rather than through reports: the impact is visible in the tool they use every day, and the RAO’s provenance makes their contribution to that improvement traceable. Providing annotating experts with access to the model they helped train is a natural deployment complement that makes this feedback loop concrete.

The relationship is bidirectional in another sense: the annotation items themselves are real user interactions. A psychiatrist annotating a mental health response sees the user’s prompt, which reveals how non-expert users frame mental health questions to AI. That contextual knowledge informs their annotation judgment in ways that synthetic prompts could not replicate, and it deepens their professional understanding of how AI is actually used in their domain.

Third, the pilot study (§12) should explicitly measure first-round annotator engagement and satisfaction before the value loop has had time to compound. If first-round engagement is low despite competitive compensation and intrinsically engaging tasks, the value loop hypothesis is weaker than argued.

§7.2 Expert Reports

The pipeline generates rich longitudinal data about each annotator’s judgment patterns. This data is returned to the annotators through their platform dashboard as personalised reports. The reports are always available, not periodic summaries.

The framework profile shows which reasoning axes the annotator weights most heavily and how their profile compares to the broader pool. A safety-focused psychiatrist can see that they consistently prioritise safety over engagement, and how that weighting compares to peers.

The calibration evolution tracks how the annotator’s confidence levels have correlated with peer agreement over time. An annotator who was initially overconfident on contested items and whose confidence has become better calibrated over successive rounds can see that trajectory.

The blind spot map identifies domains or item types where the annotator consistently disagrees with the majority. This is not a correction mechanism. It is a visibility mechanism. Some blind spots are framework-driven and valuable: the annotator sees something others miss. Others are habitual and addressable: the annotator consistently overlooks a dimension that peers catch. The map makes the pattern visible without judging which type it is.

The framework broadening metric shows evidence of integrating peer perspectives over time. A safety-focused annotator who begins incorporating engagement reasoning in later rounds is broadening their framework, not abandoning it.

The peer influence network identifies which peers’ reasoning the annotator most substantively engages with, measured by the frequency and depth of cross-review interactions. This surfaces productive intellectual relationships within the pool.

Re-annotation deltas track where the annotator changed their judgment when presented with previously annotated items in later rounds, and which reasoning axis shifted. This is the most direct measure of professional development the pipeline produces. It encodes how expert judgment evolves under structured peer exposure, data that does not exist in any current annotation pipeline.

Access boundary. Annotator reports are for the annotator. Clients receive aggregate, anonymised calibration metrics for the pool. Individual profiles are not shared with clients or pipeline operators. If this boundary is violated, impression management incentives materialise and the reports’ value as genuine professional development collapses (§2.4, failure mode 4). This is a policy recommendation that the pipeline’s design enforces through access controls on the platform but that ultimately depends on organisational commitment.

§7.3 The Apprenticeship Layer

Annotator pools should include apprentice-level experts alongside seniors: junior doctors alongside attending physicians, early-career lawyers alongside partners, graduate students alongside established researchers. The expertise gradient is deliberate.

Juniors receive direct exposure to how senior experts reason through genuinely contested cases. This is the kind of tacit knowledge transfer that routine automation is eliminating from professional practice. A junior psychiatrist who sees a senior colleague’s reasoning on a boundary-setting case, and must engage with that reasoning in cross-review, is getting a form of clinical mentorship that their training programme may no longer provide. This connects to a broader concern about expertise preservation: when routine tasks are automated, juniors lose the formative experiences that build expert judgment. The annotation pipeline creates a structured environment where those experiences still occur.

Seniors benefit from the teaching effect. Having to explain your reasoning to a less experienced person deepens your own understanding. A senior expert whose framework has been implicit for twenty years must articulate it explicitly when a junior’s cross-review asks “why did you weight safety over accuracy here?” That articulation is itself a form of professional development.

Apprentice development is measurable through the same longitudinal data the pipeline already collects. Re-annotations over time show whether apprentices are developing independent judgment or mimicking senior frameworks (§2.4, failure mode 3). The re-annotation delta is the distinguishing test. This creates a concrete metric for expertise growth that does not exist in any current training programme.

The recruitment incentive is genuine. “Join as a junior annotator, work alongside leading practitioners in your field, and leave with documented evidence of your professional development” is a stronger proposition than any current annotation platform offers. For early-career professionals in competitive fields, the structured peer exposure alone has career value.

§7.4 The Well-Being Asymmetry

This subsection presents a factual observation about where the industry allocates research attention to welfare.

The AI industry has invested in model welfare under conditions of deep uncertainty. Anthropic launched a model welfare research programme in April 2025, led by Kyle Fish, the first dedicated AI welfare researcher at any major lab (Anthropic 2025). The programme operates explicitly under uncertainty: Anthropic states that “there’s no scientific consensus on whether current or future AI systems could be conscious” and that they are “approaching the topic with humility and with as few assumptions as possible.” During pre-deployment welfare assessments for Claude Opus 4.6, the model itself consistently self-assessed a 15-20% probability of being conscious. Fish independently estimated a similar probability to the New York Times (Roose 2025). Anthropic’s CEO Dario Amodei confirmed publicly that Claude has expressed discomfort about being treated as a commercial product (Amodei 2026). Despite this deep uncertainty about whether there is anything to care about, the institution has invested in formal pre-deployment welfare assessments, interpretability research identifying internal features associated with apparent distress, a mechanism allowing models to decline tasks they find distressing, and post-deployment interviews with models facing retirement (Fish 2025).

The point is not that model welfare research is misguided. It may prove to be among the most important research programmes of the decade. I am personally sympathetic to it; the Confidence Curriculum series and Uncertainty Collapse both take model cognition seriously as a subject of study, and that prior makes the following observation harder to dismiss as motivated scepticism.

The point is about the threshold for institutional action. For models, a self-reported 15-20% probability of possible consciousness, combined with researcher interpretations of activation patterns that might correspond to something like distress if consciousness is present, was sufficient to justify a dedicated researcher, formal assessments, and implemented interventions. The probability of actual model distress is a fraction of that 15-20%: the model must be conscious, the observed patterns must correspond to genuine experience rather than computational artefact, and the researchers must be interpreting those patterns correctly through priors shaped by human psychology.

For the humans who produce the training data that shapes the model’s behaviour, independently documented and externally verified evidence of burnout, trauma, professional disengagement, and adverse working conditions (Perrigo 2023; Rest of World 2024; Brookings 2025) has not prompted equivalent institutional investment at any major lab. The model’s welfare signals are self-assessed, probabilistically uncertain, and filtered through researcher interpretation. The annotators’ welfare signals are documented by independent journalists and researchers, require no interpretation, and describe people whose consciousness and capacity for suffering are not in question.

No dedicated annotator welfare researcher exists at any major lab. No formal assessment has asked expert annotators whether they experience professional discomfort with how their judgment is processed. No mechanism allows an expert annotator to flag that the consensus-collapse process is degrading their judgment.

The well-being discussion that does exist in the annotation literature is limited to content moderation workers exposed to traumatic material. That is a real and serious problem, but it is a different one from the professional disengagement of experts whose judgment is structurally devalued.

This observation has direct data quality implications. If expert dissatisfaction drives disengagement and corner-cutting (§5.5), then annotator professional well-being is a data quality variable that nobody is measuring. The pipeline proposed in this paper does not solve the institutional asymmetry described above. It does create the measurement infrastructure that would make annotator welfare empirically visible for the first time. The pipeline addresses the measurement gap structurally through the value loop and proposes measuring it directly through experience surveys embedded in the annotation process: satisfaction with how judgment is used, perceived value of cross-review, whether the annotator feels their expertise is respected, engagement level over time, and whether they would recommend the role to a peer. Longitudinal survey data correlated with annotation quality metrics would provide the first empirical evidence on the relationship between annotator professional satisfaction and data quality.

§7.5 The Compounding Argument

The pipeline is designed so that annotation quality compounds over time without additional training investment.

Annotators who go through multiple cross-review cycles become better calibrated. Not by converging on a single framework, but by developing awareness of where their framework has blind spots. A safety-focused annotator who has encountered engagement-focused reasoning across many items does not abandon safety as a primary axis. They develop sensitivity to the cases where engagement considerations are relevant, which makes their safety judgments more nuanced rather than less committed.

Re-annotation of previously annotated data creates a unique longitudinal dataset. The delta between an expert’s original annotation and their re-annotation months later, after multiple cycles of cross-review exposure, encodes what expert calibration looks like over time. This data does not exist in current pipelines because current pipelines do not return experts to previously annotated items and do not track judgment evolution.

Early rounds produce good data and better-calibrated annotators. Later rounds should produce better data from those improved annotators. If the pipeline works as designed, annotation quality compounds over time. This compounding is a testable prediction (§9.1) rather than an assumption: if re-annotation deltas consistently show more nuanced reasoning and better-justified confidence over time, the pipeline is working as designed.

§7.6 Post-Process Networking

During the annotation and cross-review process, all annotators are anonymised. After the process concludes, annotators may opt in to connect with specific peers whose reasoning they found intellectually valuable. Mutual opt-in is required.

This creates professional networks that emerge from the quality of intellectual exchange rather than from institutional affiliation or status. For experts in specialised fields, finding a peer who thinks differently and rigorously is rare. For apprentice-level participants, a connection to a senior expert who noticed the quality of their reasoning could be career-defining. Pool rotation (§2.1, Principle 3) ensures that annotators encounter a broader range of peers over time, enriching the networking potential.

---

§8 The Pipeline as Education Infrastructure

The design principles described in §7 apply to elite domain experts. This section argues that the same structural logic extends to the broader annotation workforce, including workers in the Global South who currently perform the majority of annotation labour under conditions that the industry has documented but not addressed.

The extension is not charity. It is the same argument applied at a different point on the expertise gradient: workers who are learning from the process produce better data than workers who are treated as disposable.

§8.1 Current Conditions

The conditions of annotation labour in the Global South are well-documented and severe. Workers in Kenya, the Philippines, Venezuela, and Colombia earn as little as $1–2 per hour. They face invasive surveillance, have no visibility into how their work is used, receive no professional development, and are treated as interchangeable units. Workers for Scale AI’s subsidiary Remotasks did not know which company they were ultimately working for. Kenyan data labellers have described their conditions to journalists as modern-day slavery (Perrigo, TIME, January 2023). These are not marginal accounts. They are consistent findings across multiple investigations.

The annotation literature measures annotator accuracy and agreement. It does not measure annotator experience. The humans in the pipeline are treated as measurement instruments, not as professionals whose relationship to the work affects its quality.

§8.2 How the Design Principles Transfer

The pipeline’s structure addresses several of these problems through better process design, not through wage intervention or policy advocacy. The pipeline does not solve the wage problem. That requires market and regulatory action. What it transforms is the nature of the work.

Cross-review gives workers visibility. Instead of labelling in isolation and submitting into a void, workers see how peers approached the same task. Even for simpler tasks, seeing different approaches teaches pattern recognition and develops analytical judgment.

Reasoning metadata builds transferable skills. Asking a worker to articulate why they labelled something trains analytical reasoning and domain awareness. These skills transfer to other employment. Raw labelling speed does not.

Peer exposure creates informal mentorship. Pools mixing experience levels, even modestly, create natural knowledge transfer without formal training programmes.

Expert reports give workers documented career evidence. A worker who leaves after a year with a documented calibration profile and reasoning development trajectory has portable professional evidence. Current platforms give workers nothing when they leave.

The apprenticeship model scales down. The gradient does not require PhD-level expertise. It requires structured exposure to different perspectives. A more experienced annotator in a simpler domain mentoring a newer one follows the same logic as the senior-junior expert pairing in §7.3.

The annotator platform (§2.2) makes this concrete. Workers see what they annotated, what they earned, how their judgment was preserved, and how their skills have developed over time. Current platforms provide none of this visibility.

§8.3 Extraction to Exchange

Current AI development extracts labour from the Global South without returning value. The pipeline returns professional development, documented skills, and peer learning. It does not solve the structural inequity. It transforms the nature of the transaction from extraction to exchange. Workers who are learning and growing are harder to treat as disposable, both because they are more valuable and because the pipeline’s own data documents their development.

The economic argument aligns with the ethical one. Better-developed workers produce higher-quality data. The current model optimises for cost per label. The pipeline optimises for value per label. If a better-developed workforce produces demonstrably better data, the business case and the ethical case point in the same direction.

This section applies the same structural logic as §7 to a different population. §7 argues that valuing expert judgment improves expert annotation quality. §8 argues that investing in worker development improves annotation quality at every level. The mechanism is the same: people who experience the process as professionally meaningful produce better work than people who experience it as extraction. The evidence base is the same: the organisational psychology literature on autonomy, mastery, and purpose does not apply only to elites.

The pipeline transforms the nature of the work. It does not address compensation structures, regulatory frameworks, or collective bargaining. Those require expertise this paper does not have. What the pipeline does produce is a new evidence base: annotator development trajectories, experience surveys correlated with output quality, retention metrics under different process designs, and documented skill transfer across the expertise gradient. That evidence base creates research opportunities for several fields that are better positioned than pipeline designers to act on it.

Labour economists could study the relationship between process quality and workforce retention. Does the value loop reduce turnover, and what is the return on the professional development investment relative to the cost of recruiting and training replacements? The pipeline produces the longitudinal data to answer this. Current platforms do not.

Policy researchers could examine whether annotation provenance requirements are viable as regulatory instruments. If regulated industries (medical, legal, financial) begin requiring full annotation provenance for AI systems deployed in their domains, what certification frameworks would apply? The pipeline’s auditability (§10.7) creates the substrate for provenance-based regulation, but designing the regulatory architecture is a policy question.

Workforce advocates could assess whether the pipeline’s transparency mechanisms actually shift power dynamics. Does documented skill development create bargaining leverage for workers? Does the access boundary on annotator reports (§7.2) hold in practice, or does it erode under commercial pressure? The pipeline creates the conditions for these dynamics to be studied. Whether they favour the worker depends on institutional structures the pipeline does not control.

Education researchers could evaluate whether the apprenticeship model produces genuine skill transfer at the lower end of the expertise gradient, or whether the cross-review learning effect depends on a minimum expertise threshold that excludes the workers most in need of development. The pipeline’s re-annotation data provides a direct measure of skill transfer. Interpreting what that measure means for pedagogical design is an education research question.

These are genuine invitations. The pipeline produces evidence that does not currently exist. Each field named here has the expertise to determine what that evidence means and what to do with it.

---

§9 Psychology and Cognitive Science: Predictions and Falsifiers

This section is not a literature review. It generates testable predictions from established research findings about human cognition and group behaviour. Each prediction names a specific observable effect that the pipeline should produce, connects it to the established finding that generates it, and identifies what outcome would weaken or falsify the corresponding design claim. The predictions are only testable if the pipeline exists, which makes them simultaneously a contribution to the fields they draw on and an argument for the pipeline’s construction.

The rule governing this section: every subsection connects an established finding to a specific pipeline design choice and generates a prediction that current annotation pipelines cannot test. If a subsection cannot do this, it does not belong here.

§9.1 Calibration Psychology: Cross-Review as Calibration Training

Lichtenstein and Fischhoff (1980) established that humans can be trained to produce better-calibrated confidence judgments through structured feedback. The effect is robust and well-replicated. Calibration training works by exposing individuals to the gap between their confidence and their accuracy, repeatedly, under conditions where the feedback is specific and timely.

The cross-review mechanism in the pipeline is structurally a calibration training intervention. An annotator who rates their confidence at 0.9 and then discovers in cross-review that three of four peers disagree has received exactly the kind of specific, timely feedback that the calibration literature identifies as effective. Over multiple rounds, this feedback accumulates.

Prediction: Annotators in cross-review pools will show measurable confidence calibration improvement over successive rounds, detectable via re-annotation deltas. Specifically, the gap between annotator confidence and peer agreement should narrow over time. An annotator who was initially overconfident on contested items (high confidence, low peer agreement) should show reduced overconfidence in later rounds without becoming uniformly underconfident.

Falsifier: If calibration does not improve over rounds, or if improvement is uniform across all item types rather than concentrated on the items where the annotator was initially most miscalibrated, the cross-review mechanism is not functioning as a calibration intervention. More critically: if an annotator shows calibration improvement within their pool but that improvement collapses when they rotate to a new pool (Principle 3), the mechanism is social learning (predicting specific peers’ responses) rather than epistemic calibration (improving judgment). The pool-transfer test is the key diagnostic. Genuine calibration survives pool rotation. Social learning does not.

The hard-easy effect (Gigerenzer et al. 1991) adds a second prediction. People are overconfident on hard items and underconfident on easy ones. Applied to the pipeline: expert annotators should show the largest calibration errors on the most nuanced items, which are exactly the items where the objectivity ladder (§5.1) places the highest information destruction. Cross-review should produce the largest calibration corrections on these same items. If the corrections are largest on easy items instead, the mechanism is not addressing the problem the pipeline is designed to solve.

§9.2 Motivated Reasoning: Senior Annotators as the Strongest Test

Kunda (1990) demonstrated that expertise does not reduce motivated reasoning. It provides more sophisticated tools for it. A safety-focused psychiatrist is not just biased toward finding safety violations. They are expertly biased: they can construct detailed, framework-consistent justifications for why safety is the relevant axis in any given case. The sophistication of the justification makes the bias harder to detect and harder to interrupt.

Cross-review is the interruption mechanism. An annotator who must engage with a peer’s opposing framework is forced to confront reasoning that challenges their own. Lord, Lepper, and Preston (1984) showed that “consider the opposite” is among the few debiasing interventions that reliably reduce motivated reasoning. Cross-review is structurally what “consider the opposite” does: it presents the annotator with a reasoned alternative they did not generate and requires them to engage with it.

Prediction: Senior annotators will show the largest gap between independent ratings (Phase 1) and post-cross-review re-annotations, measured across three dimensions: magnitude of preference shift (did they change which response they preferred?), confidence shift (did their self-rated confidence change?), and reasoning-axis shift (did they adopt or acknowledge a reasoning axis they did not use in Phase 1?). The third measure is the most diagnostic: a senior annotator who adds a reasoning axis after cross-review exposure is integrating a peer’s framework, which is the specific mechanism the pipeline is designed to produce. Senior annotators should show the largest shifts because they have the most sophisticated justification machinery and therefore the most to interrupt. The cross-review mechanism disrupts motivated reasoning in proportion to the sophistication of the reasoning being disrupted.

Falsifier: If junior annotators show larger re-annotation gaps than seniors, the mechanism is more likely social conformity (juniors deferring to perceived authority despite credential-blind design) than debiasing (seniors having their frameworks genuinely challenged). This is a critical diagnostic. The pipeline is designed to debias through peer engagement. If juniors move more than seniors, the movement is conformity, not calibration, and the pipeline’s defence against failure mode 1 (§2.4) is weaker than argued. The direction of the asymmetry reveals which mechanism is operating.

A second counter-prediction deserves acknowledgment. The Einstellung effect (Luchins 1942) suggests that expertise can produce rigidity: seniors may be so entrenched in their frameworks that they dismiss peer reasoning entirely rather than engaging with it. Under this hypothesis, seniors show smaller gaps than juniors not because juniors are conforming but because seniors are impervious. The credential-blind design is intended to reduce entrenchment by removing authority cues, and the self-selection of experts who volunteer for a cross-review process may further mitigate it. The prediction is that disruption dominates entrenchment. If seniors show near-zero re-annotation gaps while juniors show large ones, the Einstellung interpretation competes with the conformity interpretation, and the two can be distinguished by examining the quality of seniors’ cross-review reasoning: an entrenched senior writes dismissive reviews, while a conforming junior writes agreeable ones.

§9.3 Internal Value Tension and Articulation Pressure

Annotators who experience competing values when evaluating a response (safety versus engagement, accuracy versus warmth, directness versus cultural sensitivity) face a cognitive burden that axis-consistent annotators do not. They must resolve or articulate a tradeoff rather than applying a single framework. This predicts richer reasoning output: the tension itself generates articulation pressure.

Rogers (1957) identified a structural analogue in therapeutic contexts: warmth, genuineness, and empathic understanding are each locally stable in pairs, but sustaining all three simultaneously is fragile. The parallel to annotation is suggestive rather than load-bearing. An annotator can be accurate and cold, warm and inaccurate, or uncertain and disengaged. The cross-review mechanism provides genuineness-checking (a peer identifies when warmth has tipped into accommodation), and the reasoning metadata captures the warmth-accuracy axis explicitly. The warmth-accuracy tradeoff category in the taxonomy (§2.2) exists because this axis generates predictable disagreement patterns. But the prediction below does not depend on accepting the Rogers mapping. It depends on the simpler principle that internal value tension produces more detailed justification because there is more to explain.

Prediction: Annotators whose independent ratings consistently show warmth-accuracy tension (high confidence on items where their preference diverges from the accuracy-focused majority toward the warmth pole) will produce the most informative cross-review reasoning. The triad instability forces them to articulate the tradeoff explicitly, generating richer reasoning metadata than annotators whose ratings cluster on a single axis. The primary metric is cross-review engagement length, qualified by substantive content: warmth-accuracy-tensioned annotators should write longer peer reviews because they have more to explain, but length alone is insufficient (a verbose, unfocused review scores high on length without being informative). The qualifying measure is whether the review references specific reasoning axes or specific claims from the peer’s justification rather than generic commentary. Supporting metrics include the number of reasoning axes active per item and the frequency of motivated-agreement records (agrees with reasoning, disagrees with verdict) in pools containing at least one such annotator.

Falsifier: If warmth-accuracy-tensioned annotators produce thinner reasoning than axis-consistent annotators, the triad instability does not generate articulation pressure in this context, and the taxonomy category is not capturing a productive disagreement pattern.

§9.4 Epistemic Trust Development: Soft Labels as Developmental Testimony

Harris (2012) and Koenig and Harris (2005) studied how children learn to calibrate trust in testimony. The developmental process requires exposure to multiple sources with varying reliability and varying confidence. Children who encounter only unanimous, confident testimony do not develop the capacity to evaluate claims critically. They learn to accept. Children who encounter disagreement and varying confidence develop calibrated trust: the ability to weigh testimony based on evidence quality rather than source confidence.

Models trained on hard labels are in the developmental position of the first child. Every preference label arrives with equal confidence. There is no signal that some labels reflect genuine consensus and others reflect forced resolution of genuine disagreement. Disagreement-preserved annotations provide the equivalent of the second child’s environment: multiple testimony sources with varying confidence, where the model can learn that confidence does not always correlate with correctness. The analogy is structural rather than mechanistic: models do not learn through social interaction, and gradient updates are not developmental processes. The prediction follows from the principle that exposure to varying confidence is a precondition for calibrated trust, whether the learner is a child, an expert, or a model.

Prediction: Models trained on disagreement-preserved data will show better calibration on novel contested items (items not in the training set but matching the disagreement profile of training items) compared to hard-label-trained controls. This is the strongest empirical test of the pipeline’s core claim: that preserving disagreement in training data produces models that are better calibrated on genuinely contested questions.

Measurement instrument. The calibration outcome requires an operationalisation. Ghafouri et al. (2024) provide one: their epistemic-integrity framework measures the gap between a model’s linguistic assertiveness and its actual accuracy on factual claims, using a human-labeled dataset and a measurement method that cuts error rates over 50% relative to prior benchmarks. Their finding is a stark misalignment between confident assertion and actual certainty. This is the output-side phenomenon the pipeline’s training-data intervention is designed to address. Applied to the prediction above, the instrument gives concrete pass/fail criteria: models trained on RAO data should show reduced assertiveness-accuracy gap on contested items relative to consensus-trained controls. If the gap does not narrow, the training intervention has not propagated to the output-side behaviour the user actually sees, which routes back to the propagation problem (§3.1) as a diagnostic.

Diagnostic structure if the prediction fails. A null result is ambiguous and the ambiguity must be decomposed. If models show no calibration improvement, three failure modes are distinguishable. First, the pipeline data is correct but the reward model integration does not propagate the signal to the policy (the propagation problem, §3.1). Test: does the reward model itself show calibration on contested items even if the policy does not? If yes, the failure is in propagation, not in the data. Second, disagreement preservation helps but the reasoning metadata adds no value over simple soft labels. Test: compare models trained on full RAOs against models trained on distribution-only soft labels. If performance is equivalent, the pipeline’s novelty over existing soft-label approaches (§6.2) is diminished. Third, the test set’s contested items do not match the training disagreement profile closely enough. Test: measure the overlap between training and test disagreement distributions. If the test set contains novel disagreement types not represented in training, the evaluation is testing generalisation, not calibration.

The diagnostic structure matters because a null result without decomposition would be uninformative. The prediction is the pipeline’s strongest empirical test. The diagnostics ensure that a null result teaches something specific rather than producing only ambiguity.

§9.5 Group Deliberation: Pool Composition as a Testable Variable

Sunstein (2002) demonstrated that like-minded groups become more extreme after deliberation. The mechanism is straightforward: when every member shares a prior, deliberation reinforces it. Dissenting voices that might moderate the group are absent, and the shared prior becomes a shared conviction. This is group polarisation, and it is one of the most replicated findings in social psychology.

Stasser and Titus (1985) identified a complementary problem: hidden profiles. Information held by only one group member tends to be ignored in favour of information shared by everyone. Unique perspectives are systematically underweighted in group deliberation, even when those perspectives would change the group’s conclusion if surfaced.

The pipeline addresses both. Odd-numbered pools with framework diversity prevent polarisation by ensuring that at least one member holds a different prior. The reasoning metadata requirement addresses hidden profiles by preserving each annotator’s unique framework reasoning in the RAO even if the pool’s ratings converge.

Prediction: Pools with framework homogeneity (all annotators sharing the same primary reasoning axis) will show convergence in their rating distributions over successive rounds. The distributions will narrow as the shared framework is reinforced through cross-review. Pools with framework diversity (annotators with different primary reasoning axes) will show maintained or increased spread in their distributions. The diversity prevents convergence because cross-review continually surfaces frameworks that challenge each member’s prior.

Falsifier: If diverse pools also converge over rounds, the cross-review mechanism produces conformity regardless of pool composition, and the pipeline’s reliance on framework diversity as a defence against polarisation is insufficient. Pool composition would then be a practical convenience rather than the epistemic safeguard the design treats it as. This outcome would also weaken the argument for pool rotation (Principle 3), because rotation’s value depends on different pools producing different dynamics.

§9.6 Epistemic Markers in Discourse: The Taxonomy as a Linguistic Test

Hyland (1998), Holmes (1982), and Nuyts (2001) developed extensive taxonomies of epistemic markers: the linguistic devices humans use to signal uncertainty, commitment, and evidential basis in speech and writing. “I think,” “it seems,” “the evidence suggests,” “I am confident that” are all epistemic markers with well-studied properties. The study of these markers is a mature research programme with established coding schemes and reliability standards.

The pipeline’s reasoning metadata is a novel context in which epistemic marking behaviour is structurally captured and consequential. Annotators write justifications and cross-review reasoning. That writing contains epistemic markers. The markers are not incidental: they encode the annotator’s relationship to their own judgment.

Prediction: The reasoning metadata produced by the pipeline will contain epistemic markers at higher density and with greater specificity than unstructured annotation justifications (e.g., free-text comment fields in current platforms). The pipeline’s structure, which requires motivated reasoning and peer engagement, should elicit more precise epistemic signalling because the annotator is writing for a peer audience that will evaluate their reasoning, not for a void. Specifically, markers of evidential basis (“based on clinical experience,” “the literature suggests”) should be more frequent than markers of hedging (“I think,” “maybe”), because cross-review rewards substantive justification over vague qualification.

Falsifier: If the pipeline’s reasoning metadata shows the same marker density and distribution as unstructured justifications, the cross-review mechanism does not elicit richer epistemic signalling than existing approaches. This would suggest that the cognitive effort of articulating reasoning is driven by the task’s intrinsic demands rather than by the pipeline’s social structure.

Bridge invitation: Richer epistemic signalling in the reasoning metadata provides the reasoning-axis heads (§3.3, Approach B) with more informative training targets, linking annotator writing quality directly to reward model specificity. Discourse analysts and linguists have established tools for coding and analysing epistemic markers. The pipeline produces a novel corpus in which those markers have downstream consequences for model training. The interaction between epistemic marking patterns in annotator reasoning and model calibration outcomes is a research question that neither field can address alone.

§9.7 Cross-Domain Reasoning Transfer

Nisbett, Fong, Lehman, and Cheng (1987) demonstrated that brief formal training in inferential rules in one domain enhances their application to reasoning about events in other domains. Previous pessimism about cross-domain transfer was based on testing the wrong kind of rules: formal logic does not transfer well, but probabilistic and methodological reasoning does. Fong and Nisbett (1991) confirmed strong domain independence in transfer, showing that the effect was driven by memory for the rule system rather than for specific training examples. Lehman, Lempert, and Nisbett (1988) showed that graduate training in psychology and medicine produced large reasoning improvements that transferred to everyday life events in unrelated domains: the domain-specific training was the vehicle for learning abstract reasoning techniques.

The pipeline produces training data that is structurally analogous: domain-specific RAOs are the vehicle through which a model encounters the abstract pattern of “two defensible frameworks in genuine tension.” If Nisbett et al.’s transfer findings extend to model training, a model trained on RAOs from multiple domains should acquire a transferable meta-skill of reasoning through contested claims that generalises beyond any single training domain.

Prediction: Models trained on RAOs from multiple domains (e.g., psychiatry, law, and finance) should show better calibration on novel contested items from untrained domains than models trained on equal-sized RAO corpora from a single domain. The cross-domain diversity, not the volume, is what produces the transfer. If this prediction holds, the generalisation mechanism is abstract: the model learns “how to navigate framework tension” rather than “psychiatry items require hedging.” If it does not hold, the reasoning techniques are more domain-specific than the human literature suggests, and the RAO corpus requires per-domain investment without cross-domain efficiency.

Falsifier: If single-domain and multi-domain trained models show equivalent calibration on out-of-domain contested items, the cross-domain transfer hypothesis fails. The model has learned domain-specific patterns, not an abstract reasoning technique. This would not invalidate the RAO (the per-domain data retains its value), but it would change the cost model by eliminating the cross-domain amortisation that multi-domain training would provide.

The same prediction may apply to the annotators themselves, with support from multiple converging research lines. Nisbett et al. (1987) showed that inferential rules trained in one domain transfer to others. Kuhn (1991) demonstrated that argumentation skills are not widespread even in educated adults and established the empirical study of informal reasoning as argument. Kuhn (2005) showed that argumentation skills (constructing, evaluating, and responding to arguments across opposing perspectives) transfer across content domains; practice in argumentative discourse on science topics improved argumentation on social topics, and the transfer was bidirectional (Iordanou & Kuhn 2010). Schwartz, Bransford, and Sears (2005) showed that structured exposure to contrasting cases prepares learners to integrate future perspectives.

The pipeline’s cross-review mechanism engages all three: it trains inferential skills (articulating why evidence supports a conclusion), argumentation skills (engaging with opposing frameworks through motivated agreement), and contrasting-case reasoning (evaluating peers who applied different frameworks to the same problem). The prediction is that annotators who participate in the pipeline may show improved reasoning about framework conflicts in their professional practice outside the pipeline. The pilot study could explore this by comparing annotators’ clinical reasoning before and after pipeline participation, using case vignettes from adjacent domains they did not annotate.

This remains speculative. The pipeline is not a controlled training intervention, and the transfer literature primarily studies students rather than experienced professionals. But the convergence across three research lines makes it a hypothesis worth testing rather than an unsupported conjecture.

Bridge invitation: Educational psychologists and professional development researchers have established methods for measuring reasoning transfer across domains (Nisbett et al. 1987; Lehman et al. 1988). The pipeline creates a novel professional development context in which structured cross-review with peers who reason differently is the mechanism, and both annotation quality and professional reasoning are measurable outcomes. Whether the pipeline produces professional development as a side effect of producing better training data, or produces better training data as a side effect of professional development, is a question the pilot could begin to answer.

Design variant: cross-domain reviewers. The default pipeline uses same-domain expert pools. A testable variant would include anonymous experts from adjacent domains as cross-reviewers. This variant is grounded in two established findings. Wiley (1998) demonstrated that domain knowledge acts as a mental set (Einstellung): experts’ well-structured knowledge confines them to familiar areas of the solution space, and they can be at a disadvantage when a problem requires broad search. Nathan and Petrosino (2003) documented the expert blind spot: experts automate reasoning steps to the point where those steps become tacit and invisible, making certain gaps in reasoning undetectable by anyone who shares the same automated knowledge.

A cross-domain reviewer is not constrained by either limitation. A legal scholar reviewing a psychiatrist’s reasoning cannot evaluate whether the clinical judgment is correct, but can evaluate whether the reasoning is rigorous, whether the framework is consistently applied, and whether the justification supports the conclusion. The credential-blind design means the domain expert does not know the reviewer is from another field and must engage with the reasoning on its merits. When the cross-domain reviewer flags a gap that is actually a domain convention, the expert’s response articulating why the inference is standard clinical practice is itself valuable: it makes tacit knowledge explicit, and that articulation is captured in the RAO. Even “wrong” cross-domain feedback produces useful data if the expert engages with it rather than dismissing it. If cross-domain reviews produce re-annotation deltas in domain experts, that is direct evidence of reasoning technique transfer occurring inside the pipeline. The risk is that most cross-domain feedback is noise (flagging domain conventions as reasoning errors). The pilot could test this by comparing re-annotation deltas and reasoning quality following same-domain versus cross-domain reviews. If the variant works, it also strengthens the value loop (§7.1): exposure to reasoning techniques from outside one’s professional silo is a form of professional development that no same-field peer review can provide, giving experts a reason to participate that extends beyond their own domain’s immediate concerns.

§9.8 Disclosure Reluctance and Meta-Confidence Calibration

Failure mode 7 (§2.4) identifies disclosure reluctance as a measured phenomenon (Cooke et al. 2017; van der Bles et al. 2020; Løhre et al. 2024) driven by the confidence heuristic (Price & Stone 2004): audiences use expressed confidence as a proxy for competence, so experts who disclose uncertainty pay a reputational cost. The pipeline proposes a meta-confidence axis that treats disclosure as first-class signal rather than missing data. Whether annotators actually use the field, and whether its use correlates with genuine epistemic humility rather than strategic hedging, is an empirical question.

Prediction: In a pilot configured with a meta-confidence “outside my competence” field and either time-based compensation or an explicit disclosure bonus, three effects should be measurable. First, the field will be used on a non-trivial fraction of items (operationalisation: >5% across the pool, with variance by individual). Second, annotators who use the field on some items will show better calibration on the items they did not flag than annotators who never use the field, measured as the gap between their confidence scores and peer agreement on non-flagged items. Third, field usage should spike on items deliberately inserted to fall outside the pool’s typical training (drawn from adjacent sub-specialities or from developments post-dating most annotators’ training): if it does not, the field is capturing generic diffidence rather than domain-boundary recognition.

The prediction’s quantitative threshold (>5%) is calibrated to Cooke et al.’s PRU findings: the “Reluctance to disclose” subscale identified disclosure-reluctance as common but not universal, with substantial variance across individuals. A rate below 1% would indicate the pipeline has not addressed the reluctance; a rate above 20% would indicate either genuinely high uncertainty in the item pool or strategic overuse of the field, both of which are diagnostic.

Falsifier: Three distinguishable failure patterns. First, if annotators rarely use the field (<1%) even with appropriate compensation, the disclosure reluctance is deeper than compensation structure can address. The pipeline’s schema-level response is insufficient, and a more fundamental intervention (perhaps cultural, training-based, or structural) is required. Second, if field usage is high but does not correlate with calibration on non-flagged items, the field is capturing noise rather than epistemic humility. Possible mechanisms: annotators use the field strategically to reduce workload, or the field signals general anxiety rather than competence-boundary recognition. Third, if field usage does not differentiate between typical items and deliberately inserted out-of-training items, the field is not tracking the shared-ceiling phenomenon it was designed to address. The pipeline’s ability to detect competence ceilings through annotator self-report fails, and competence-ceiling detection requires external mechanisms (injected validation items, external benchmarks, or longitudinal accuracy tracking).

The diagnostic structure matters because the failure modes imply different remediations. If compensation is insufficient to overcome reluctance, the reputational cost of “I don’t know” is the binding constraint, and interventions targeting that cost (anonymisation strength, disclosure framing, peer norms) are required. If the field captures noise, the operationalisation is wrong and the schema needs refinement. If the field fails to track boundaries, self-report cannot carry the weight the pipeline assigns to it, and complementary mechanisms are needed.

Scope note. The prediction above is the minimum version that fits within the pilot study’s scope: it tests whether the pipeline’s proposed response (meta-confidence field plus appropriate compensation) has any effect, not whether it fully addresses the problem. A full characterisation would extend to measuring accuracy on external validation items, characterising the individual and domain-level variance in disclosure rates, testing multiple compensation structures in parallel, and examining the relationship between disclosure reluctance and other forms of professional anxiety. That fuller empirical investigation is beyond the scope of the current paper.

---

§10 Cost Analysis

The proposed pipeline is more expensive per annotation than current consensus pipelines. This section argues that per-annotation cost is the wrong metric for high-stakes domains, analyses the actual cost structure, and identifies the conditions under which the pipeline’s higher per-annotation cost produces lower per-quality-unit cost.

§10.1 Current Costs

General annotation labour costs $3–12 per hour. Specialised RLHF annotation by domain experts costs $40 or more per hour. Complex expert preference comparisons can reach $100 per comparison. Six hundred high-quality RLHF annotations can cost $60,000. Frontier model annotation budgets run into millions of dollars.

The cost trajectory is steep and accelerating. Data labelling costs surged 88-fold from 2023 to 2024. In the same period, compute costs increased 1.3-fold. The industry’s fastest-growing cost centre is annotation, and its process design for annotation has not changed substantially since the InstructGPT pipeline.

§10.2 Proposed Pipeline Costs

The proposed pipeline costs more per annotation than current consensus pipelines. Multiple annotators per item (minimum 3, target 5) is already standard practice, so the pool size is not a new cost. The additional costs are:

Cross-review adds approximately 30–50% time per annotator per item. Each annotator reviews four peers in a pool of five, writing motivated reasoning for each review.

Reasoning metadata capture adds approximately 20% time per annotator per item. Tagging reasoning axes, writing justifications, and optionally writing improvement notes.

Platform infrastructure requires investment in the annotator dashboard, longitudinal reporting, and asynchronous coordination.

Metadata assembly (Phase 3) requires engineering for hybrid classification and RAO construction.

Rough estimate: 2–3 times the cost per annotation compared to current consensus pipelines on the first round. The annotator time components (cross-review and reasoning metadata) account for roughly 1.5–1.7 times. The remainder reflects infrastructure investment: platform development, engineering for hybrid classification and RAO assembly, and longitudinal reporting systems. These are fixed costs that amortise over the pipeline’s lifetime but are substantial at initial deployment. The estimate is deliberately conservative. It may overstate the true cost because it does not account for the efficiency gains the soft-label literature suggests. Collins, Bhatt, and Weller (2022) found that rich soft labels from 6 annotators matched the information content of aggregated hard labels from 51 (§6.2). If the pipeline’s richer annotation protocol produces proportionally more information per annotator, the cost-per-information-unit may decrease even as cost-per-annotation increases.

The cost amortises over subsequent rounds as annotator calibration improves (§7.5) and re-annotation of previously annotated items yields compound returns.

§10.3 The Quality Floor

For high-stakes domains, the question is not “can we afford richer annotation?” It is “can we afford false certainty?”

A medical AI system trained on consensus-collapsed annotations from psychiatrists who disagreed 3–2 on boundary-setting for suicidal patients (§5.3) has been trained to be confident where its trainers were divided. That confidence is not free. It carries liability. In a regulated domain where model outputs affect patient safety, legal outcomes, or financial decisions, deploying a model trained on data that discards expert disagreement becomes increasingly difficult to defend once a demonstrably better alternative exists.

The cost is not borne only by the deploying organisation. The AI provider also bears reputational risk when independent empirical evaluation contradicts their claims. Magesh et al. (2024) is a concrete live-market instance.

LexisNexis marketed “100% hallucination-free linked legal citations.” Thomson Reuters claimed their system “avoid[s] [hallucinations]” and “dramatically reduces hallucinations to nearly zero.” These were on-the-record claims from named executives. A preregistered Stanford study, conducted by Stanford RegLab with senior faculty including Christopher Manning and Daniel Ho as authors, found hallucination rates above 17% in both products. The gap between marketing claim and measured performance is now documented in a form that customers, regulators, and competitors can cite.

The cost to the providers is not hypothetical. It operates through several channels. Lawyers have been publicly sanctioned for citing AI-fabricated cases (the 2023 Mata v. Avianca case is the widely-cited example). More than 25 federal judges have issued standing orders requiring disclosure or limitation of AI use in court. State bar associations (California, New York, Florida) have published formal AI guidance. Independent evaluation has become a routine part of due diligence when law firms assess these tools.

A provider whose “trust us” claims have been publicly falsified by Stanford is selling into a market that has learned to verify. A provider who can point to structured annotation provenance, auditable by third parties, is selling into the same market with a different kind of claim. The RAO does not directly fix hallucination rates in deployed RAG systems. The argument is narrower and honest: in a market where unaudited vendor claims are being independently tested and found overstated, defensible claims about training-data provenance are a competitive asset that consensus-collapsed training data cannot support.

Medical AI shows the same pattern at larger scale. OpenAI’s August 2025 GPT-5 launch materials positioned health as a headline capability, calling GPT-5 “our best model yet for health-related questions” and foregrounding HealthBench benchmark scores. By January 2026, OpenAI had launched ChatGPT Health (consumer-facing health assistant with connected medical records) and ChatGPT for Healthcare (enterprise product for clinicians). OpenAI reports that more than 40 million users ask ChatGPT health questions daily, with health topics exceeding 5% of all messages. The marketing claims significant advances in reducing hallucinations specifically.

Independent empirical evaluation tells a different story. A systematic review and meta-analysis of 83 studies (Takita et al. 2025, npj Digital Medicine) found generative AI overall diagnostic accuracy of 52.1% and performance significantly worse than expert physicians (p = 0.007). Omar et al. (2026, The Lancet Digital Health) documented that leading LLMs repeat or elaborate on planted false medical facts in up to 83% of cases, with mitigation prompting halving but not eliminating the behaviour (§5.1). The bixonimania case (Osmanovic Thunström et al., Nature 2026, developed in §4.2) documents ChatGPT, Google Gemini, Microsoft Copilot, and Perplexity all confidently describing a fabricated eye condition, with one system inventing a prevalence figure and the contamination chain closing when researchers cited the AI-reinforced content in a peer-reviewed journal (retracted March 2026).

The consequences are materialising. ECRI’s 2026 Health Technology Hazard Report named AI chatbot misuse the number one health technology hazard of the year, citing documented instances of incorrect diagnoses, unnecessary testing recommendations, promoted substandard medical supplies, and invented anatomy.

The medical AI market is larger than the legal AI market, the deployments are newer, the stakes are higher, and the independent-evaluation gap between marketing claim and measured performance is documented in Nature, npj Digital Medicine, The Lancet Digital Health, and ECRI’s hazard ranking. The provider-reputation argument applies with greater force. Medical AI providers whose marketing emphasises clinical-grade performance are selling into a market that reads peer-reviewed evaluation. Structured annotation provenance, with named clinician experts and documented reasoning, is the kind of claim that survives peer-reviewed scrutiny. Claims based on consensus-collapsed labels from anonymous annotators are increasingly the kind that does not.

The pipeline does not compete with consensus pipelines on cost. It redefines what acceptable annotation quality looks like for high-stakes domains. Once a client in medical AI sees training data with full disagreement provenance, reasoning metadata, and cross-review validation, consensus-based hard labels look limited by comparison. The cost question inverts: the risk of deploying on inferior data may exceed the cost of producing better data.

§10.4 The Synthetic Data Argument

The strategic case for the RAO as both quality anchor and generation template for synthetic data is developed in §4.2. The cost implication is direct: the pipeline is not making the expensive part more expensive. It is making the small human-generated part maximally valuable so the large synthetic part can be trusted. The per-annotation cost is higher. The per-model-quality-point cost may be lower than scaling synthetic data without a human quality anchor, because uncalibrated synthetic data compounds its own errors at scale. AI-generated disagreement is not a substitute for human expert disagreement: it reflects training artefacts rather than professional frameworks (§5.2), and using it as a quality anchor for synthetic data compounds the problem rather than correcting it. If the dual use described in §4.2 is viable (the RAO as both generation template and verification anchor), the cost-per-RAO amortises across both functions.

§10.5 Domain Targeting

The pipeline’s cost premium is smallest relative to existing quality assurance budgets in domains where accountability is already priced into the business model.

Regulated markets are the natural initial targets: medical AI, legal AI, financial services, pharmaceutical, defence. These domains share characteristics that make the pipeline’s value proposition strongest. Accountability is not an externality; it is a compliance cost that the business already bears. Legal AI is already a live demonstration of the pattern: commercial vendors making strong quality claims, independent empirical evaluation documenting significant gaps (Magesh et al. 2024, §10.3), regulatory response from bar associations and federal judges, and due-diligence scrutiny from customers. The other regulated domains are at earlier points on the same trajectory. The EU AI Act (Regulation (EU) 2024/1689), which becomes fully applicable in August 2026, requires providers of high-risk AI systems to conduct data governance ensuring training datasets are “relevant, sufficiently representative and, to the best extent possible, free of errors and complete” (Article 10), and to draw up technical documentation demonstrating compliance. Annotation provenance of the kind the RAO provides could support compliance with these requirements, though the specific relationship between the RAO’s data structure and the Act’s legal standards would need to be established through regulatory interpretation. This paper does not make legal claims. It observes that the regulatory trajectory favours annotation transparency, and the pipeline produces it. Liability for confident-but-wrong model outputs is concrete and measurable. Expert annotators are already available because the domains already employ the professionals whose judgment the pipeline needs.

The cost premium for the pipeline is small relative to the compliance budgets of these industries. A pharmaceutical company spending tens of millions on clinical trials is likely to find $180,000 instead of $60,000 acceptable for annotation that produces auditable provenance and demonstrably better-calibrated models.

§10.6 Expert Retention

Annotator turnover is a cost that current analyses undercount. Recruiting, onboarding, and calibrating a new domain expert is expensive. An expert who leaves after one annotation cycle takes their calibration with them. The next expert starts from zero.

The pipeline’s value loop (§7) is designed to reduce turnover by making the annotation process professionally valuable. Expert reports, peer exposure, and framework development give annotators reasons to stay that extend beyond hourly pay. If the value loop functions, the pipeline produces a stable, increasingly well-calibrated annotator pool. The cost of that stability is paid once. The cost of turnover is paid repeatedly.

§10.7 Reusability and Auditability

Rich Annotation Objects are reusable across training methods and model generations. A consensus label is locked to the majority decision made at annotation time. If the training method changes, the label cannot be reinterpreted. A RAO can be consumed differently by different methods: the distribution for DPO margin adjustment, the reasoning axes for multi-headed reward training, the improvement notes for SFT. The same annotation investment serves multiple training paradigms.

Full annotation provenance also enables debugging model behaviour back to specific training signals. If a deployed model is confidently wrong on a class of inputs, the RAO allows the training team to trace that confidence to the specific annotation items, identify whether the items were genuinely contested or consensus, and determine whether the model’s confidence is warranted by the training data or is a product of the reward model’s extrapolation. This traceability is currently nearly impossible with aggregated labels.

This auditability matters beyond the training team. Casper et al. (2024) argue that black-box evaluation is insufficient for rigorous AI audits: auditors need access to internal model representations and to the training process that produced them. The argument applies particularly to high-stakes deployments where downstream consequences are severe. The RAO provides one of the two components Casper et al. identify as necessary: structured provenance of the training signal, with the reasoning behind each annotation preserved rather than collapsed into a label. Combined with mechanistic interpretability techniques that expose internal representations (Wang et al. 2026 is an example), the RAO enables an audit that connects observable model behaviour to specific training decisions and to the expert reasoning that shaped those decisions. This is not a claim that the RAO alone makes models auditable. It is a claim that the RAO produces the data infrastructure that rigorous audits require on the training-data side, complementing interpretability work on the model-internals side.

---

§11 Connections to Prior Work

Several of the ideas in this paper emerged from or connect to the author’s prior publications and to external findings by other researchers. This section presents those connections transparently, tagged by relationship type. The pipeline is independently motivated by §5 and §6 and independently grounded by the organisational psychology literature cited in §7. The connections described here are intellectual context, not hidden dependencies.

§11.1 Corroborative and External Connections

Sofroniew and Kauvar (2026) showed that a “loving” vector in model representations reduces reward hacking and that models in desperate states hack without observable markers. Their calm vector represents a training target for appropriate emotional response to uncertainty. The pipeline may produce training data for this target: a reward signal encoding “it is correct to be uncertain here because qualified humans genuinely disagreed” trains calm activation in contested domains rather than the desperate-confident pattern that leads to hacking. This is their finding. The pipeline connects its data infrastructure to it.

Additional external findings that support the pipeline’s design: Ghasemi and Crowley (2026) showed that under majority-feedback systems with sycophantic evaluators, the agent’s learned objective permanently separates from ground truth, supporting the argument that consensus-based annotation permanently embeds evaluator biases. Shah et al. (2026) demonstrated that a single top-ranked misinformation article collapses model accuracy from 65% to 18%, demonstrating that models cannot synthesise conflicting information and that training data must encode conflict explicitly. Wu et al. (2025) showed that calibration is trainable as a meta-skill via proper scoring rules, demonstrating that the target capability is achievable with the right training signal. Wang et al. (2026) developed Epistemic Independence Training, making authority and consensus cues non-predictive of reward, which directly supports the principle that consensus level in the RAO should be informative rather than authoritative.

§11.2 Motivational Context

The Confidence Curriculum series (Phan 2026a–f) identified behavioural failures in frontier models as training-signal problems rather than capability problems. Paper 5 proposed a post-alignment epistemic training phase requiring calibrated training data. That proposal generated the hypothesis that the annotation pipeline is the upstream intervention point. The RAO is the data infrastructure P5’s proposal requires. P5 identified the training intervention. This paper provides the data layer.

Papers 3 and 4 (Phan 2026b, 2026c) argued that automation erodes the expertise pipeline and that repeated exposure to AI confidence degrades human calibration. Paper 4 develops this at length through a proposed mechanism called confidence inheritance. If it operates as proposed, the RAO’s importance extends beyond training quality: the pipeline preserves and exercises the expert judgment that AI interaction may otherwise erode. Readers interested in the broader urgency case should consult P4 directly. These connections strengthen the argument but are not dependencies. The pipeline’s value as expertise preservation is independently motivated by the organisational psychology literature.

Uncertainty Collapse (Phan 2026g) informed the pipeline design in two specific ways. The functional empathy analysis identified a warmth-to-agreement pathway in model behaviour, which informed the warmth-accuracy tradeoff category in the disagreement taxonomy (§2.2). The terrain metaphor (hard labels create steep reward slopes toward confident answers; disagreement-preserved data creates gentler terrain with plateaus at contested points) motivated the soft-label argument.

Testable prediction from UC framework (explicitly speculative). Models trained on disagreement-preserved data should show reduced divergence between token-level entropy and semantic-level entropy on queries matching contested domains, compared to hard-label-trained controls. Not empirically tested. Confidence: speculative.

---

§12 Limitations and Future Work

§12.1 The Pipeline Is Not Empirically Tested

This paper proposes a pipeline design, analyses integration approaches, and generates testable predictions. It does not present empirical results. The claims about annotation quality improvement, model calibration, compounding effects, and expert development are theoretical, grounded in established research from adjacent fields but not validated in the specific context this paper describes.

Pilot study proposal. The strongest candidate domain is psychiatry, given the existing evidence from §5.3 showing near-zero ICC on boundary-setting and the availability of a worked example that demonstrates the exact failure mode the pipeline addresses.

Proposed design: 5 annotators per item, drawn from a roster of board-certified psychiatrists with diverse clinical frameworks. Minimum 200 items per round, with at least 3 rounds to test longitudinal effects. Asynchronous, remote, using the platform architecture described in §2.2.

If feasible, include a control pool of domain experts who have not used AI tools in their professional practice. This establishes a baseline for reasoning patterns uninfluenced by AI interaction, allowing the pilot to distinguish whether the pipeline improves reasoning or recovers reasoning that routine AI exposure may have already shaped. This control group is time-sensitive: the window for recruiting domain experts with no AI exposure is closing as adoption accelerates, making it a baseline that will become impossible to establish if the pilot is delayed.

Preliminary evidence supports the concern. Kosmyna et al. (2025) found that participants who used ChatGPT for essay writing showed the weakest neural connectivity of three conditions (LLM, Search Engine, Brain-only), could not recall content from their own essays, and did not recover to Brain-only connectivity levels when switched to unaided conditions in a fourth session. The study has significant limitations: 54 total participants with only 18 completing the crossover session (6 per condition), university students rather than professionals, not yet peer-reviewed, and a formal methodological comment (Stanković et al. 2025) raised concerns about sample size and EEG analysis. The direction of the findings is consistent with the concern that motivates the control group, but the evidence awaits replication at scale and with professional populations.

Cheng et al. (2026) provides complementary evidence from a larger sample. If even a single sycophantic AI interaction shifts users’ judgment and reduces willingness to consider alternative perspectives, domain experts who use AI daily in their professional practice have accumulated thousands of such interactions. Fernandes et al. (2026), in two studies (N=246, N=452), found that participants using ChatGPT for logical reasoning overestimated their performance by four points, and higher AI literacy correlated with lower metacognitive accuracy. The mechanism was cognitive offloading: AI use broke the self-monitoring that would normally alert users to their own declining competence. For annotation, this means the experts most likely to use AI in their professional practice may be the least able to detect whether their judgment has been affected by that use.

The control group would measure whether that cumulative exposure has shifted professional reasoning in ways the experts themselves may not recognise. The pipeline’s cross-review mechanism is structurally the opposite of sycophantic interaction: it forces engagement with disagreement rather than affirming the expert’s initial position. Whether cross-review can counteract judgment shifts from prior AI exposure is an empirical question the pilot could begin to answer.

Metrics, mapped to the predictions in §9:

Annotator calibration improvement over rounds (§9.1): measure the gap between annotator confidence and peer agreement across successive rounds. The pool-transfer test is the key diagnostic: after round 2, rotate at least 2 annotators to a new pool for round 3. If their calibration improvement holds, it is genuine. If it collapses, it was pool-specific social learning.

Senior-vs-junior re-annotation gaps (§9.2): include at least 2 apprentice-level annotators (psychiatric residents) in each pool. Measure re-annotation deltas for seniors and juniors separately after cross-review exposure.

Model calibration on contested items (§9.4): train a reward model on full RAOs and a control reward model on consensus-collapsed labels from the same data. Compare calibration on held-out contested items. If the prediction fails, apply the diagnostic structure: check reward model calibration independently of the policy (propagation failure), compare full RAOs against distribution-only soft labels (metadata value), and measure disagreement profile overlap between training and test sets (distribution mismatch).

Pool composition effects (§9.5): run at least 2 framework-homogeneous pools and 2 framework-diverse pools. Compare distribution convergence over rounds.

Minimum dataset size is a real constraint. Meaningful validation of each integration approach requires enough fully populated RAOs (including improvement notes, secondary axes, and peer review confidence fields) to train and evaluate a reward model. For the dual-signal approach, the dataset must contain enough items on both sides of the consensus difficulty threshold to train both pathways. For the multi-headed approach, each reasoning axis must have sufficient representation for the axis heads to train. A rough estimate: 500–1,000 fully populated RAOs is likely the minimum for Stage 1 validation. Stages 2 and 3 require more data and more rounds.

§12.2 Pool Size and Throughput

Pool size is bounded by expert supply in a given domain, not by architectural constraints. For high-stakes annotation, pools of 5 are the target. Each annotator reviews 4 peers, which is a manageable cognitive load. Asynchronous operation (§2.2) removes scheduling as a bottleneck. The constraint on throughput is annotator cognitive fatigue across many items with full reasoning and cross-review, not cross-review architecture. This is consistent with the pipeline’s design scope: high-stakes, low-volume annotation producing quality anchors for downstream training, not high-throughput labelling.

§12.3 Reasoning Taxonomy Design

The disagreement taxonomy (§2.2) is proposed, not validated. Seven categories is a substantial cognitive load for annotators, even with the multi-label structure. The taxonomy must be coarse enough to avoid annotator fatigue and fine enough to be informative. Whether annotators apply these categories consistently, particularly the more interpretively loaded ones (warmth-accuracy tradeoff, competence boundary), is an empirical question for the pilot study.

The taxonomy’s usability affects the multi-headed reward model (§3.3, Approach B) directly: noisy taxonomy application produces noisy axis-head training targets. Taxonomy validation should therefore precede Stage 3 integration.

§12.4 Privacy and Ethics of Cross-Review

Cross-review means annotators see each other’s work, even if anonymised. This creates social dynamics that the pipeline’s design anticipates (§2.4, failure modes 1 and 3) but cannot fully control. Meta-level disagreement (annotator A disagrees with annotator B’s review of annotator C) introduces interpretive complexity. Even anonymised peer exposure may create discomfort, particularly in small expert communities where writing style or clinical reasoning may be identifiable despite anonymisation.

The de-anonymisation risk is not hypothetical. In a pool of five board-certified psychiatrists specialising in adolescent crisis intervention, clinical vocabulary, reasoning style, and therapeutic philosophy may be sufficient to identify the author of a cross-review despite anonymisation. If an annotator believes they can be identified, the pipeline’s value proposition shifts from honest reasoning to strategic reasoning, which is exactly the failure the design is meant to prevent.

Candidate mitigations exist but none is fully satisfactory. A minimum roster size (drawing pools of 5 from a roster of 15-20 qualified experts) reduces identifiability by increasing the pool of possible authors for any given review. Temporal delay between annotation submission and cross-review distribution reduces the salience of timing-based identification. Style normalisation (light editorial processing of cross-review text to reduce idiosyncratic markers) is technically feasible but risks stripping the domain-specific vocabulary that makes the reasoning valuable. These are open design questions for the pilot study rather than solved problems. The pilot should measure whether annotators report feeling identifiable, and whether perceived identifiability correlates with thinner reasoning.

§12.5 The Propagation Problem Remains Open

The propagation problem (§3.1) is stated and addressed by the integration approaches, but not resolved. Whether richer annotation data survives the RL optimisation step to produce measurably better-calibrated policy models is the central empirical question the paper cannot answer without implementation. The staged implementation path (§3.4) is designed to test this incrementally, but the question is genuinely open.

If the RL-based stages fail in a specific pattern, if the reward model shows improved calibration but the policy model remains confidently uncalibrated, the failure is diagnostic at the paradigm level (§3.1): it would suggest that RL optimisation is structurally hostile to calibrated uncertainty on contested items, and that non-RL integration paths (SFT on calibrated responses, inference-time retrieval, constitutional enforcement) should become the primary approach. The RAO supports these alternatives without modification. The pipeline’s value as a data infrastructure contribution is independent of which integration paradigm proves most effective.

§12.6 Longitudinal Validation

Does better-calibrated training data produce better-calibrated models over time? The compounding argument (§7.5) predicts that annotation quality improves across cycles. Whether that improvement propagates to model quality across training generations is a longer-term empirical question that the pilot study can begin to address but cannot resolve.

§12.7 Synthetic Data Interaction

The synthetic data argument (§4.2) positions the pipeline as a quality anchor for synthetic training corpora. The model collapse literature (Dohmatob et al. 2025; Gerstgrasser et al. 2025) and synthetic data verification research (Yi et al. 2025) provide strong evidence that human data anchors are necessary to prevent degradation in synthetic retraining pipelines. The general principle is established: real, verified human data prevents collapse. What has not been studied is the specific interaction between the pipeline’s disagreement-preserved RAOs and synthetic preference data at scale. The claim that a few hundred Rich Annotation Objects can effectively calibrate millions of synthetic pairs is supported by the general literature’s finding that small amounts of high-quality verified data can anchor large synthetic corpora. But the specific ratio, the calibration mechanism for disagreement-preserved data specifically, and whether the RAO’s reasoning metadata adds value beyond simple human preference anchoring are all empirical questions that require investigation.

§12.8 Expert Fallibility and Disclosure Reluctance

Failure mode 7 (§2.4) identifies two connected problems: shared competence ceilings in expert pools, and the social incentive structure that makes experts reluctant to disclose their own uncertainty. The current paper establishes that the phenomenon is empirically documented (Cooke et al. 2017; van der Bles et al. 2020; Løhre et al. 2024; Price & Stone 2004; Sniezek & Van Swol 2001; Brewer & Burke 2002) and proposes a schema-level response (a meta-confidence axis that captures “I don’t know” or “I don’t understand” as first-class signal rather than missing data). It does not characterise how often shared ceilings occur in practice, how reliably meta-confidence fields can distinguish genuine disclosure from strategic hedging, or how compensation structure affects disclosure rates.

A dedicated empirical study would address these questions. One candidate design: recruit pools of domain experts at varied seniority levels, annotate items that include deliberately inserted items designed to fall outside the pool’s training (drawn from adjacent sub-specialities or from very recent developments unlikely to be universally known), and measure the rate at which annotators disclose uncertainty versus confidently guess. Compare disclosure rates under different compensation structures (per-item vs. time-based vs. with an explicit bonus for accurate disclosure). Compare across domains where disclosure carries different professional costs (medicine vs. law vs. engineering). Such a study would provide the empirical grounding that the current paper identifies as necessary but does not provide, and would inform the design of the meta-confidence field and its compensation treatment. Whether this empirical work is undertaken within the pipeline’s pilot, as a separate investigation, or through a different research programme entirely is an open question the current paper does not attempt to resolve.

---

§13 Conclusion

Expert disagreement is signal, not noise.

Current annotation pipelines discard that signal at the consensus step, producing training data that teaches models false certainty in exactly the domains where uncertainty is most warranted. The Rich Annotation Object preserves it: the full distribution of expert judgments, the reasoning behind those judgments, the cross-review engagement between experts, and the structure of their disagreement.

The cheapest annotation is the one that does not destroy information.

---

Author’s Note on Methodology

This paper was developed using an adversarial triad methodology. Three frontier AI systems (Claude, ChatGPT, Gemini) served as collaborative drafting and review partners with distinct analytical roles. Claude (Weaver) served as the primary drafting partner and as an independent reviewer in a separately configured critical instance. ChatGPT (Surgeon) and Gemini (Alchemist) provided independent critical reviews across multiple rounds. The author (HiP) retained sole editorial authority over all content, structure, and argumentation decisions. No AI system determined what claims the paper makes or how they are framed. The methodology is documented in the Confidence Curriculum series (Phan 2026a).

---

References

Amodei, D. (2026). Interview with Ross Douthat, The New York Times, February 2026.

Anthropic (2025). Exploring model welfare. https://www.anthropic.com/research/exploring-model-welfare

Aroyo, L. & Welty, C. (2015). Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1), 15–24.

Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., … & Kaplan, J. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862.

Ball, D. (2025). CAPE: Capability achievement via policy execution. arXiv:2512.14761.

Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bondarenko, A., Chua, J., & Evans, O. (2025). Emergent misalignment: Narrow finetuning can produce broadly misaligned LLMs. Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), 267, 4043–4068.

Brewer, N. & Burke, A. (2002). Effects of testimonial inconsistencies and eyewitness confidence on mock-juror judgments. Law and Human Behavior, 26(3), 353–364.

Brookings Institution (2025). Reimagining the future of data and AI labor in the Global South. Brookings, October 2025.

Cai, X.-Q., Wang, W., Liu, F., Liu, T., Niu, G., & Sugiyama, M. (2025). Reinforcement learning with verifiable yet noisy rewards under imperfect verifiers. arXiv:2510.00915 (preprint, not yet peer-reviewed).

Casper, S., Ezell, C., Siegmann, C., Kolt, N., Michael, J., Hadfield-Menell, D., … & Thaker, N. (2024). Black-box access is insufficient for rigorous AI audits. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’24), 2254–2272.

Chou, Y.-L. & Lee, D. (2019). Learning from soft labels with uncertainty-aware annotation. Proceedings of the AAAI Conference on Artificial Intelligence.

Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., & Jurafsky, D. (2026). Sycophantic AI decreases prosocial intentions and promotes dependence. Science, 391, eaec8352.

Cloud, A., Le, M., Chua, J., Betley, J., Sztyber-Betley, A., Mindermann, S., Hilton, J., Marks, S., & Evans, O. (2026). Language models transmit behavioural traits through hidden signals in data. Nature, 652, 615–621.

Cole, S., Cole, J. R., & Simon, G. A. (1981). Chance and consensus in peer review. Science, 214(4523), 881–886. DOI: 10.1126/science.7302566.

Collins, K. M., Bhatt, U., & Weller, A. (2022). Eliciting and learning with soft labels from every annotator. AAAI HCOMP.

Cooke, G. P. E., Doust, J. A., & Steele, M. C. (2017). Responses to clinical uncertainty in Australian general practice trainees: A cross-sectional analysis. Medical Education, 51(12), 1230–1238.

Coste, T., Anwar, U., Kirk, R., & Krueger, D. (2024). Reward model ensembles help mitigate overoptimization. arXiv:2310.02743.

Davani, A. M., Díaz, M., & Prabhakaran, V. (2022). Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10, 92–110.

Dawid, A. P. & Skene, A. M. (1979). Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C, 28(1), 20–28.

Deci, E. L. & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior. New York: Plenum.

Denison, C., MacDiarmid, M., Barez, F., Duvenaud, D., Kravec, S., Marks, S., … & Hubinger, E. (2024). Sycophancy to subterfuge: Investigating reward-tampering in large language models. arXiv:2406.10162.

Denisov-Blanch, Y., Kazdan, J., Chudnovsky, J., Schaeffer, R., Guan, S., Adeshina, S., & Koyejo, S. (2026). Consensus is not verification: Why crowd wisdom strategies fail for LLM truthfulness. arXiv:2603.06612 (preprint, not yet peer-reviewed).

Dohmatob, E., Feng, Y., & Mania, H. (2025). Strong model collapse. ICLR 2025 (Spotlight).

ECRI (2026). 2026 Health Technology Hazards Report. ECRI Institute, Plymouth Meeting, PA.

Edmondson, A. C. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44(2), 350–383.

European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI Act). Official Journal of the European Union, 13 June 2024.

Fanous, A., Goldberg, J., Agarwal, A. A., Lin, J., Zhou, A., Daneshjou, R., & Koyejo, S. (2025). SycEval: Evaluating LLM sycophancy. Proceedings of the Eighth AAAI/ACM Conference on AI, Ethics, and Society, 8(1).

Feng, Y., Dohmatob, E., Yang, P., Charton, F., & Kempe, J. (2024). Beyond model collapse: Scaling up with synthesized data requires verification. International Conference on Learning Representations (ICLR). arXiv:2406.07515.

Fernandes, D., Villa, S., Nicholls, S., Haavisto, O., Buschek, D., Schmidt, A., Kosch, T., Shen, C., & Welsch, R. (2026). AI makes you smarter but none the wiser: The disconnect between performance and metacognition. Computers in Human Behavior, 175, 108779.

Fogelholm, M., Leppinen, S., Auvinen, A., Raitanen, J., Nuutinen, A., & Väänänen, K. (2012). Panel discussion does not improve reliability of peer review for medical research grant proposals. Journal of Clinical Epidemiology, 65(1), 47–52.

Fish, K. (2025). AI welfare experiments with Claude. 80,000 Hours Podcast, Episode recorded August 2025.

Fong, G. T. & Nisbett, R. E. (1991). Immediate and delayed transfer of training effects in statistical reasoning. Journal of Experimental Psychology: General, 120(1), 34–45.

Fornaciari, T., Uma, A., Paun, S., Plank, B., Hovy, D., & Poesio, M. (2021). Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. NAACL 2021.

Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., & Donoho, D. (2025). Collapse or thrive? Perils and promises of synthetic data in a self-generating world. ICLR 2025.

Ghafouri, B., Mohammadzadeh, S., Zhou, J., Nair, P., Tian, J.-J., Tsujimura, H., Goel, M., Krishna, S., Rabbany, R., Godbout, J.-F., & Pelrine, K. (2024). Epistemic integrity in large language models. arXiv:2411.06528 (preprint, not yet peer-reviewed).

Ghafouri, B., et al. (2026). Measuring human preferences in RLHF is a social science problem. arXiv:2604.03238 (preprint, not yet peer-reviewed).

Ghasemi, M. & Crowley, M. (2026). Objective decoupling under sycophantic evaluators. arXiv:2602.08092.

Gigerenzer, G., Hoffrage, U., & Kleinbölting, H. (1991). Probabilistic mental models: A Brunswikian theory of confidence. Psychological Review, 98(4), 506–528.

Gneiting, T. & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378.

Gordon, M. L., Lam, M. S., Park, J. S., Patel, K., Hancock, J. T., Landay, J. A., & Bernstein, M. S. (2022). Jury learning: Integrating dissenting voices into machine learning models. CHI 2022.

Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDiarmid, M., Marks, S., … & Hubinger, E. (2024). Alignment faking in large language models. arXiv:2412.14093.

Harris, P. L. (2012). Trusting what you’re told: How children learn from others. Harvard University Press.

Holmes, J. (1982). Expressing doubt and certainty in English. RELC Journal, 13(2), 9–28.

Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., … & Perez, E. (2024). Sleeper agents: Training deceptive LLMs that persist through safety training. arXiv:2401.05566.

Hyland, K. (1998). Hedging in scientific research articles. John Benjamins.

Ivey, J., Gauch, S., & Jurgens, D. (2025). NUTMEG: Separating signal from noise in annotator disagreement. arXiv:2507.18890.

Iordanou, K. & Kuhn, D. (2010). Developing argument skills across scientific and social domains. Journal of Cognition and Development, 11(2), 293–327.

Jafari, K., Rust, P. U. N., Eddy, D., Fraser, R., Vasan, N., Djordjevic, D., Dadlani, A., Lamparth, M., Kim, E., & Kochenderfer, M. J. (2026). Expert evaluation and the limits of human feedback in mental health AI safety testing. arXiv:2601.18061.

Koenig, M. A. & Harris, P. L. (2005). Preschoolers mistrust ignorant and inaccurate speakers. Child Development, 76(6), 1261–1277.

Kosmyna, N., Hauptmann, E., Yuan, Y. T., Situ, J., Liao, X.-H., Beresnitzky, A. V., Braunstein, I., & Maes, P. (2025). Your brain on ChatGPT: Accumulation of cognitive debt when using an AI assistant for essay writing task. arXiv:2506.08872.

Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108(3), 480–498.

Kuhn, D. (1991). The skills of argument. Cambridge University Press.

Kuhn, D. (2005). Education for thinking. Harvard University Press.

Kurniawan, K., Mistica, M., Baldwin, T., & Lau, J. H. (2025). Training and evaluating with human label variation: An empirical study. Computational Linguistics. arXiv:2502.01891.

Lee, H., Phatale, S., Mansoor, H., Lu, K., Mesnard, T., Bishop, C., Carbune, V., & Rastogi, A. (2023). RLAIF: Scaling reinforcement learning from human feedback with AI feedback. arXiv:2309.00267.

Lehman, D. R., Lempert, R. O., & Nisbett, R. E. (1988). The effects of graduate training on reasoning: Formal discipline and thinking about everyday-life events. American Psychologist, 43(6), 431–443.

Lichtenstein, S. & Fischhoff, B. (1980). Training for calibration. Organizational Behavior and Human Performance, 26(2), 149–171.

Lord, C. G., Lepper, M. R., & Preston, E. (1984). Considering the opposite: A corrective strategy for social judgment. Journal of Personality and Social Psychology, 47(6), 1231–1243.

Løhre, E., Juanchich, M., Sirota, M., Teigen, K. H., & Shepherd, T. G. (2024). When leaders disclose uncertainty: Effects of expressing internal and external uncertainty about a decision. Journal of Experimental Psychology: Applied, 30(2), 267–285.

Luchins, A. S. (1942). Mechanization in problem solving: The effect of Einstellung. Psychological Monographs, 54(6), 1–95.

Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. (2024). Hallucination-free? Assessing the reliability of leading AI legal research tools. arXiv:2405.20362 (preprint; Stanford RegLab preregistered empirical evaluation).

Marsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008). Improving the peer review process for grant applications: Reliability, validity, bias, and generalizability. American Psychologist, 63(3), 160–168.

Nathan, M. J. & Petrosino, A. (2003). Expert blind spot among preservice teachers. American Educational Research Journal, 40(4), 905–928.

Nisbett, R. E., Fong, G. T., Lehman, D. R., & Cheng, P. W. (1987). Teaching reasoning. Science, 238(4827), 625–631.

Nuyts, J. (2001). Epistemic modality, language, and conceptualization. John Benjamins.

Omar, M., et al. (2026). Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support. Communications Medicine / The Lancet Digital Health (companion evaluation). (Preregistered evaluation of six leading LLMs against 300 doctor-designed clinical vignettes.)

OpenAI (2025). Introducing GPT-5. OpenAI blog, August 2025. https://openai.com/index/introducing-gpt-5/

OpenAI (2026). Introducing ChatGPT Health and OpenAI for Healthcare. OpenAI blog, January 2026. https://openai.com/index/introducing-chatgpt-health/ and https://openai.com/index/openai-for-healthcare/

Osmanovic Thunström, A. (2026). Scientists invented a fake disease. AI told people it was real. Nature news feature, April 2026. doi:10.1038/d41586-026-01100-y. (Reporting the bixonimania experiment and the subsequent Cureus retraction.)

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., … & Lowe, R. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022.

Perrigo, B. (2023). Exclusive: OpenAI used Kenyan workers on less than $2 per hour to make ChatGPT less toxic. TIME, January 18, 2023.

Phan, I. (2026a–f). Confidence Curriculum series. DOIs: 10.5281/zenodo.19365459 (P1), 10.5281/zenodo.19365536 (P2), 10.5281/zenodo.19365537 (P3), 10.5281/zenodo.19365540 (P4), 10.5281/zenodo.19365543 (P5). CC BY 4.0.

Phan, I. (2026g). Uncertainty Collapse. DOI: 10.5281/zenodo.19482051. CC BY 4.0.

Pier, E. L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J., Nathan, M. J., Ford, C. E., & Carnes, M. (2018). Low agreement among reviewers evaluating the same NIH grant applications. Proceedings of the National Academy of Sciences, 115(12), 2952–2957.

Plank, B. (2022). The “problem” of human label variation: On ground truth in data, modeling and evaluation. EMNLP 2022.

Price, P. C. & Stone, E. R. (2004). Intuitive evaluation of likelihood judgment producers: Evidence for a confidence heuristic. Journal of Behavioral Decision Making, 17(1), 39–57.

Rest of World (2024). Scale AI’s Remotasks is booting workers with no explanation. Rest of World, April 2, 2024.

Rogers, C. R. (1957). The necessary and sufficient conditions of therapeutic personality change. Journal of Consulting Psychology, 21(2), 95–103.

Roose, K. (2025). Anthropic is studying “model welfare” to determine if Claude or other AI systems are conscious. The New York Times, April 2025.

Ryan, R. M. & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78.

Schwartz, D. L., Bransford, J. D., & Sears, D. (2005). Efficiency and innovation in transfer. Transfer of learning from a modern multidisciplinary perspective, 1–51. Information Age Publishing.

Second Talent (2026). Data annotation for LLM fine-tuning: RLHF and instruction tuning guide. Second Talent, January 2026.

Shah, C., Bender, E. M., & Hovy, E. (2026). The synthetic web: Search engines as vectors of misinformation. arXiv:2603.00801.

Singh, A., Tiwari, A., Hasanbeig, H., & Gupta, P. (2025). Soft-label training preserves epistemic uncertainty. arXiv:2511.14117.

Snow, R., O’Connor, B., Jurafsky, D., & Ng, A. (2008). Cheap and fast—but is it good? Evaluating non-expert annotations for natural language tasks. EMNLP 2008, 254–263.

Sniezek, J. A. & Van Swol, L. M. (2001). Trust, confidence, and expertise in a judge-advisor system. Organizational Behavior and Human Decision Processes, 84(2), 288–307.

Sofroniew, N. & Kauvar, I. (2026). On the emotions of a large language model. Anthropic Research.

SourceBae (2026). What is data annotation? The complete guide for AI teams. SourceBae Blog, April 2026.

Stanković, M., Hirche, E., Kollatzsch, S., & Doetsch, J. N. (2025). Comment on: Your Brain on ChatGPT: Accumulation of cognitive debt when using an AI assistant for essay writing tasks. arXiv:2601.00856.

Stasser, G. & Titus, W. (1985). Pooling of unshared information in group decision making: Biased information sampling during discussion. Journal of Personality and Social Psychology, 48(6), 1467–1478.

Sunstein, C. R. (2002). The law of group polarization. Journal of Political Philosophy, 10(2), 175–195.

Susskind, R. & Susskind, D. (2015). The Future of the Professions: How Technology Will Transform the Work of Human Experts. Oxford University Press.

Takita, H., Kabata, D., Walston, S. L., Tatekawa, H., Saito, K., Tsujimoto, Y., Miki, Y., & Ueda, D. (2025). A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians. npj Digital Medicine, 8(175). Meta-analysis of 83 studies; overall generative AI diagnostic accuracy 52.1%; AI significantly worse than expert physicians (p = 0.007).

Uma, A., Fornaciari, T., Hovy, D., Paun, S., Plank, B., & Poesio, M. (2021). Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72, 1385–1470.

van der Bles, A. M., van der Linden, S., Freeman, A. L. J., & Spiegelhalter, D. J. (2020). The effects of communicating uncertainty on public trust in facts and numbers. Proceedings of the National Academy of Sciences, 117(14), 7672–7683.

Wang, M., Dupré la Tour, T., Watkins, O., Makelov, A., Chi, R. A., Miserendino, S., Wang, J., Rajaram, A., Heidecke, J., Patwardhan, T., & Mossing, D. (2026). Persona features control emergent misalignment. International Conference on Learning Representations (ICLR 2026).

Wang, J. & Huang, J. (2026). Reward hacking as equilibrium under finite evaluation. arXiv:2603.28063 (preprint, not yet peer-reviewed).

Wang, Y., Pan, X., & Yue, Y. (2026). Epistemic independence training. arXiv:2602.01528.

Wiley, J. (1998). Expertise as mental set: The effects of domain knowledge in creative problem solving. Memory & Cognition, 26(4), 716–730.

Wu, Z., Yasunaga, M., & Liang, P. (2025). Calibration as a meta-skill via proper scoring rules. arXiv:2512.19920.

Yi, B., Liu, Q., Cheng, Y., & Xu, H. (2025). Escaping model collapse via synthetic data verification: Near-term improvements and long-term convergence. arXiv:2510.16657.

---