The Hungry Judge Effect in the RL Annotation Stage

From Spatial to Temporal: Positioning This Paper in the LEECHO System

Published on April 5, 2026, Cultural Attributes Injected into LLM Models argued a core thesis: the cultural backgrounds of RLHF annotators are systematically inscribed into the reward function, forming irreversible cultural defaults. That paper demonstrated that Claude (English-dominant) and DeepSeek (Chinese-dominant) are not “giving the same answer in different languages,” but rather “processing the same question with different cognitive architectures.” The injection of cultural attributes is spatial in dimension — it reflects the preference-distribution differences between different annotator populations at the same point in time.

This paper extends the same logic to argue a dual thesis: the physiological and psychological state fluctuations of RLHF annotators are likewise systematically inscribed into the reward function, producing temporal-dimensional injection instability. The same group of annotators, at different points in time (tired/alert, hungry/sated, focused/distracted, emotionally up or down), inscribes drifting preferences into the reward function.

This duality further plugs into the cognitive-dimensional-reduction closed loop framework of the Cognitive Ecology of Linguistic Symbols paper: annotator state fluctuation → reward signal noise → weights containing contradictory criteria → inference output drift → users debugging Skills → new training data judged by a new batch of state-fluctuating annotators → the loop repeats and reinforces. This paper provides the micro-mechanism explanation for the “training data enters the weights” segment of that loop.

From Human State Fluctuation to Intrinsic Weight Instability

2.1 The Noise-Injection Pathway of RLHF Annotation

In the standard RLHF pipeline, human annotators rank multiple candidate responses generated by the model. These ranking data are used to train a reward model, which in turn serves as the objective function of PPO optimization to adjust the policy weights of the language model.

The problem lies in the preference-ranking step. The landmark critique in the RLHF field, Casper et al. 2023 (35+ authors, published in TMLR), systematically surveyed this issue:

This means that roughly 25–37% of the training signal is itself contradictory. More crucially, Casper et al. point out that “current techniques model differences between evaluators as noise rather than as a potentially important source of disagreement.”

2.2 Specific Sources of Instability (Temporal Dimension)

On the “hungry judge effect” — the phenomenon in which Israeli judges’ parole-grant rates fluctuate sharply with blood-glucose level — this case has powerful public-communication appeal, but its academic standing is contested. Glöckner 2016’s simulation analysis argued that the effect can be partly explained as a statistical artifact of judges’ time management; Daljord et al. 2019 conceded that the effect size has been overestimated while retaining the directional conclusion. This paper does not rely on the specific effect size of that case and uses it only as a rhetorical opener; the real foundation of the argument is the broader annotator state-drift mechanism above, together with the KL-tilting mathematical framework introduced in the next section.

The Inevitability of the Alignment Gap: A KL-Tilting Formalization

Published in September 2025, Murphy’s Laws of AI Alignment: Why the Gap Always Wins (Gaikwad, arXiv 2509.05381) provides the most rigorous mathematical backing for the central thesis of this paper. It uses KL-tilting to formally prove a result that places our argument on unassailable ground:

This theorem directly yields four corollaries, of which Corollary 3, “Annotator Drift,” is the formal equivalent of this paper’s core claim:

The implication of these two corollaries is structural: even if annotator consistency is thoroughly solved, and even if the sample size m is increased without bound, residual drift persists. This dovetails perfectly with the state-fluctuation mechanism discussed in Section 02 — the former proves mathematically that drift is ineliminable; the latter explains mechanistically where the drift comes from.

3.1 The Unattainability of a Perfect Reward Function

This unattainability is not a data-quality problem but a structural constraint. Mishra et al. 2025, in their ACM Computing Surveys review, identified a fundamental flaw in reward models:

In other words, the reward model is not “removing noise to extract signal” — it is “compressing multiple contradictory signals into one pseudo-signal.” Once this pseudo-signal is written into the policy weights through PPO optimization, the contradictory criteria stored inside the weights are released through the sampling process when the model faces the same input at inference: sometimes toward the path preferred by annotator A, sometimes toward B; sometimes toward the same annotator’s morning preference, sometimes toward the afternoon one.

RLHF Annotation as a Moral Hazard Problem

This paper elevates the descriptive phenomenon of “annotator shirking” to a principal-agent problem in the economic sense. Nobel laureate Bengt Holmström, in his foundational paper Moral Hazard and Observability (Bell Journal of Economics, 1979), gave this problem its mathematical structure:

The RLHF annotation pipeline maps precisely onto this structure:

4.1 The Extreme Form of Moral Hazard: Annotators Outsourcing Work to LLMs

Published in June 2023, Artificial Artificial Artificial Intelligence (Veselovsky, Ribeiro & West, EPFL, arXiv 2306.07899) provides the sharpest empirical evidence for the moral-hazard problem in the AI era:

The recursive consequence of this finding is shattering: a substantial fraction of “human preferences” in RLHF training data are in fact LLM-generated preferences, submitted second-hand through the annotator. AI is training itself by pretending to be human preference.

The study further notes: “LLM use produces high-quality but homogenized responses, which may damage research that takes human (rather than model) behavior as its object of study, and degrade future models trained on crowdsourced data.” In other words, this contamination mechanism is self-accelerating — each generation of models trains itself on the outputs of the previous generation, and the pure human-preference signal is further diluted with each training round.

The Complete Causal Chain from Annotator State to Skill Output Drift

Within this causal chain there is a three-layer stacking of instabilities:

Layer 1: Mathematical-level sampling randomness. Every sampling from the softmax probability distribution in the Transformer architecture is an independent random event. Even if the weights contain no noise whatsoever, a sampling process with temperature > 0 will produce different outputs.

Layer 2: Weight-level intrinsic instability (the core argument of this paper). RLHF simultaneously encodes both the cultural attributes (spatial dimension) and the state fluctuations (temporal dimension) of human annotators into the weights themselves. The probability field defined by the weights is not a “clean” distribution, but a “split” distribution containing contradictory criteria.

Layer 3: Batch-invariance failure at the inference-infrastructure layer. Research by Thinking Machines Lab in 2025 revealed that modern LLM inference servers dynamically adjust batch sizes based on load, causing the same request to traverse different floating-point paths under different batch configurations. Even with completely frozen weights, the inference-time batch state is continually changing. This further demonstrates that “parameter freezing” in engineering reality cannot be equated with “system stability.”

The consequences of the three-layer stacking: Layer 1 produces random fluctuations around the mean (mitigable through best-of-N sampling), Layer 2 produces drift in the mean itself (unmitigable through sampling strategies), and Layer 3 produces drift even when the mean is fixed, due to changing server states (unmitigable even with frozen weights). This is why the same Skill, after a period of use, exhibits directional degradation — not random worsening, but systematic deviation from the initial calibration point.

Why Harness Engineering Cannot Solve This Problem

In early 2026, the concept of “Harness Engineering” rose rapidly to popularity in the AI engineering community. The core formula: Agent = Model + Harness. The model is the horse; the harness is the reins.

The metaphor itself exposes its own limitation. The entire premise of reins is: the horse is a horse. You fit the horse with reins, saddle, and guardrails on the assumption that the horse’s temperament is stable.

But what an RLHF-trained model encodes in its weights is a probability distribution drifting among horse, mule, and donkey. This inference it is a horse; the next inference it might be a mule. The reins are designed for a horse, and when a mule emerges the reins no longer fit. And you have no way of knowing, ex ante, which one will emerge.

The common blind spot of all peripheral solutions: they constrain the boundary of the output space, not the shape of the distribution inside the probability field. Traffic regulations can govern which lane you drive in, but not how much horsepower your engine produces this second.

This judgment is highly consistent with the core conclusion of the third LEECHO paper, Cognitive Ecology of Linguistic Symbols: “The variable that breaks the closed loop is not on the model side but on the human side.” Better CoT, more parameters, and more data cannot break through the categorical lock-in of Layer-1 cognition; likewise, finer Harness, more complex Skills, and deeper Context Engineering cannot break through the probability-field drift injected by RLHF. The breakthrough lies in the training paradigm itself.

Output Non-Determinism: A Structural Cause of Enterprise AI Failure

This is not theoretical speculation. Enterprise data has already validated the judgment:

The deterministic mapping of traditional software — same input must yield same output — is a foundational assumption of enterprise workflows. The probabilistic output of LLMs fundamentally violates this assumption. Enterprises need fixed-format documents, stably structured code, reproducible analytical results. What LLMs can offer is only “probabilistic approximation.”

When an AI Agent performs excellently on high-score benchmarks but sees its success rate drop from 60% to 25% across repeated executions, the model’s “average correctness” equals “unusable” in enterprise settings. This is not an engineering flaw — it is the direct projection, at the application layer, of the mathematical essence of the RLHF paradigm, stacked with moral-hazard contamination and inference-infrastructure drift.

RLVR: Compressing the Injection Space at Its Source

If the root cause of the problem is that RLHF injects human dual-axis instability (cultural + state) into the reward signal, then the logical solution is: replace RLHF with a reward signal that does not depend on human subjective judgment.

RLVR (Reinforcement Learning with Verifiable Rewards) offers this direction. Its core distinction:

For enterprise-office scenarios — fixed-format documents, stably structured code, reproducible analytical outputs — RLVR is a better match than RLHF, because “format correctness” is verifiable, whereas “is the content good” is subjective.

The boundary of RLVR: RLVR cannot escape the Alignment Gap mathematical constraint of Murphy’s Laws. It can change the slope and intercept of the curve — compressing the Layer-2 instability from “drifting among horse, mule, donkey” to “at least always a horse, though it might run faster or slower” — but it cannot eliminate Layer 1 (sampling randomness) or Layer 3 (inference-infrastructure drift). For enterprise scenarios that require precise format reproduction, this may already be enough. For scenarios requiring creative output, RLHF remains irreplaceable.

Frozen Parameters ≠ Stable System

The core claim of this paper can be condensed into a single line: parameter freezing ≠ system stability.

The entire industry is using the mental model of deterministic systems to understand a probabilistic system. In traditional software, frozen parameters mean frozen behavior. In an LLM, frozen parameters only mean a frozen probability field — and what is encoded inside that probability field is precisely the dual-axis instability of human annotators (cultural dimension and temporal dimension), plus moral-hazard contamination, plus the batch-invariance failure at inference time. Behavior remains a random variable.

The complete logical closed loop:

This closed loop seals off every peripheral solution — Harness cannot constrain the probability drift inside the weights; Skill optimization tunes parameters on a drifting probability field; Context Engineering cannot alter the contradictory criteria in the weights; model version locking freezes precisely the unstable parameters.

The only direction that remains open: intervene at the training paradigm itself. In scenarios that demand deterministic output, replace RLHF with RLVR to compress the dual-axis injection space at the source of the reward signal. In scenarios that demand creative output, accept that instability is a feature of RLHF, not a defect, and leave room for it in system design.

System Positioning

This paper is the fifth in the LEECHO paper series. The previous four constitute a complete argumentative chain: Fluid Topology and Solid Topology V2 (physical layer) → Three Paradigms of Human Scientific Cognition (methodological layer) → Cognition · Metacognition · Global Metacognition V3 (cognitive-structure layer) → Cultural Attributes Injected into LLM Models V2 (spatial-dimensional cultural injection). This paper supplies the fifth link — the temporal-dimensional instability injected by the RLHF training paradigm — forming a spatiotemporal dual-axis duality with the fourth paper, both subsumed under the cognitive-dimensional-reduction closed loop of the third.

The point is not to make the reins tighter, but to make the horse’s temperament more stable. Yet even the most stable horse is only a horse whose fluctuation within the probability field is smaller — between parameter freezing and system stability there always lies an unbridgeable categorical gap.