Chapter 1 — Introduction

1.1 Research Background

Since Vaswani et al. published “Attention Is All You Need” in 2017, the Transformer architecture has achieved near-absolute dominance in the large language model domain. However, alongside the exponential growth in model capabilities, multiple independent research communities have separately discovered a series of persistent defects: the “Lost in the Middle” phenomenon by Stanford/UC Berkeley (Liu et al., 2023), Attention Sink behavior by MIT Han Lab (Xiao et al., 2023), representational collapse proven at NeurIPS 2024 (Barbero et al., 2024), and recurring physical world mapping failures in embodied intelligence deployments.

Scope Delimitation: The critical scope of this paper is restricted to the decoder-only autoregressive Transformer—the unified underlying architecture of all production-grade LLMs from 2024 to 2026 (GPT series, Claude series, Gemini series, DeepSeek series, Llama series). Both structural constraints identified in this paper—the spatial routing contradiction and the temporal generation contradiction—are characteristics of the decoder-only causal decoder. This paper does not cover encoder-decoder architectures (such as the original Transformer) or pure encoder architectures (such as BERT). Encoder-only architectures are not subject to autoregressive constraints, and the information routing characteristics of their bidirectional attention mechanisms differ from unidirectional attention under causal masking. The cross-attention in encoder-decoder architectures exhibits different selective information routing behavior. The “ontological contradiction” of this paper is unique to the decoder-only paradigm—it arises from the triple coupling of causal masking + softmax zero-sum competition + autoregressive next-token output.

As of mid-2026, these problems are still treated as independent defects, each with its own research community and isolated solutions. Researchers in the attention domain focus on positional encoding and sparse attention, agent-domain engineers focus on context engineering and memory management, and the embodied intelligence community focuses on multimodal fusion and world models.

1.2 Core Thesis

This paper advances a radical but verifiable thesis: the above problems are not parallel but co-originating—they are projections of the same structural contradiction onto different dimensions. This contradiction can be precisely formulated as the ontological contradiction of Transformer architecture:

1.3 Research Contributions

Contribution 1: Identifying and naming the ontological antagonism between multi-head attention (mechanism) and Next Token Prediction (objective)—a contradiction that has not been fully articulated in the academic literature as of June 2026.

Contribution 2: Constructing a complete four-layer causal chain from mathematical foundations to physical world consequences, with the Data Processing Inequality (DPI) serving as a unified information-theoretic tool throughout.

Contribution 3: Proposing the “Valuable Redundancy” hypothesis—that the redundancy released when matrix computations align toward a unique solution contains highly effective information, and this information constitutes the optimal solution candidates pursued by multi-head attention, yet is irreversibly discarded by the argmax exclusion mechanism of Next Token.

Contribution 4: Quantifying the complete information compression pipeline from the physical world to LLM output (total compression ratio ~10⁸), establishing the information-theoretic foundation for the MLLM + WM architecture.

1.4 Paper Structure

This paper adopts a four-layer progressive structure. Chapter 2 establishes the mathematical foundations (Softmax’s zero-sum competition mechanism); Chapter 3 reveals the architectural-layer contradiction (the ontological antagonism between multi-head attention and Next Token); Chapter 4 analyzes system-layer amplification (cascading failures in agent architectures); Chapter 5 discusses the physical world layer rupture (comprehensive exposure after removal of the error-tolerance layer). Chapter 6 unifies the four-layer causal chain and establishes the two-dimensional architectural limitation model (spatial routing contradiction × temporal generation contradiction), Chapter 7 evaluates the current solution landscape and maps it to the two-dimensional framework, Chapter 8 discusses limitations (including computational thermodynamic constraints as an uncovered third dimension) and implications, and Chapter 9 presents conclusions and future directions.

Chapter 2 — Layer Zero: Mathematical Foundations — Softmax as a Zero-Sum Competition Mechanism

2.1 Three Mathematical Consequences of Softmax Normalization

The Softmax function maps attention logits to a probability distribution:

This normalization constraint triggers three mathematical consequences as sequence length N increases:

2.1.1 Attention Entropy Tends Toward Maximum Entropy (Θ(log N))

Barbero et al. (2024) proved that softmax attention entropy grows at a rate of Θ(log N) with sequence length. Physical implication: when N is sufficiently large, the attention each token receives approaches 1/N—essentially equivalent to “attending to nothing.” α-entmax, by contrast, keeps entropy at O(log s) (where s is the size of the non-zero support set), a property that softmax fundamentally lacks.

2.1.2 Maximum Attention Value Tends Toward Zero (Attention Fading)

Nakanishi (2025), in “Scalable-Softmax Is Superior for Attention,” proved that the maximum element of the softmax output vector approaches zero as the input vector dimension increases. This was named “Attention Fading,” which directly reduces the model’s ability to prioritize critical information in long contexts.

2.1.3 Representational Collapse

Barbero et al. (NeurIPS 2024, “Transformers Need Glasses!”) mathematically proved that different input sequences can produce arbitrarily close representations at the final token. The model is provably unable to respond differently to these sequences. This effect is further exacerbated under the low-precision floating-point formats commonly used in modern LLMs. Subsequent research found that Transformers tend to “over-mix” information—as depth increases and context grows longer, representations trend toward undifferentiated homogeneity. The Attention Sink phenomenon is in fact the model’s spontaneous strategy for combating representational collapse.

2.2 Zero-Sum Game Interpretive Framework

We unify the above three consequences within a zero-sum game framework. Softmax’s normalization constraint (all weights summing to 1) creates a competitive exclusion dynamic: any increase in attention to one position necessarily reduces attention to all other positions. This produces a “unimodal preference”—softmax naturally tends to concentrate probability mass on a few high-scoring positions. Additionally, ZeroS research (2026) noted that softmax attention produces convex combinations of value vectors—all weights are positive and sum to 1—and can only blend information additively, unable to express subtraction or contrastive operations.

2.3 Multi-Modal Fidelity Decay

When a task requires the model to attend simultaneously to multiple dispersed information sources (multi-modal tasks), the zero-sum constraint exerts structural pressure on multi-source retrieval fidelity. The NIAH-2 benchmark from April 2026 precisely quantified this decay:

Table 1: Long-context retrieval benchmarks for frontier models as of April 2026 (Source: Digital Applied industry benchmark compilation, cited as phenomenological evidence)

The accuracy drops from single-needle to multi-needle retrieval (22–37 percentage points) are the direct mathematical signature of softmax’s unimodal preference confronting multi-modal tasks. The NoLiMa benchmark (ICML 2025) further found that after removing literal lexical overlap between question and answer, 10 of 12 models claiming 128K+ context support saw accuracy fall below 50% of baseline at 32K—even GPT-4o dropped from 99.3% to 69.7%. Multiple tests from NVIDIA’s RULER benchmark show that the effective window of most models is significantly shorter than the nominal window.

2.4 Causal Connection to Lost in the Middle

The U-shaped attention curve of Lost in the Middle is the superposed result of zero-sum competition and positional encoding prior bias. Positional encodings such as RoPE naturally confer competitive advantages to beginning and end positions, and the zero-sum mechanism of softmax amplifies this advantage. Anthropic’s 2023 finding confirms this from the inverse: adding a single sentence to the prompt—”Here is the most relevant sentence in the context”—boosted accuracy from 27% to 98%, proving that the model “saw” the middle information, but that information was suppressed in zero-sum competition by positional bias. The SealQA study of 2025 found that some recent models (such as GPT-4.1-MINI) have mitigated the classic U-shaped bias, but the problem has shifted from “positional sensitivity” to “a general difficulty in modeling relevance within large volumes of noisy context.”

2.5 Chapter Summary

Chapter 3 — Layer One: Architectural Contradiction — The Ontological Antagonism Between Multi-Head Attention and Next Token

3.1 The “Unique Solution” Nature of Next Token Prediction

3.1.1 Formal Description: Cascading Dimensionality Reduction from High-Dimensional to Zero-Dimensional

After the final layer, the hidden state is multiplied with the unembedding matrix to produce logits—a score for every token in the vocabulary at every position. For generation, only the last row matters. Argmax (or sampling) yields a single token ID. Embeddings, queries, MLP activations—all discarded. This is a cascading dimensionality reduction pipeline from d_model-dimensional space to V-dimensional logits to a 0-dimensional integer.

3.1.2 Logic Perspective

From the perspective of mathematical logic, next-token prediction corresponds to modus ponens—given the prefix as premise, deriving a unique logical consequent. Sequence processing becomes a constructive proof extension under the Curry-Howard correspondence (Tarau, 2026, Arrow Language Model).

3.1.3 SVM Interpretation

Tarzanagh et al. (NeurIPS 2023) proved that self-attention models learned through next-token prediction implement a support vector machine (SVM) based on token priority graphs—suppressing low-priority tokens to predict high-priority tokens and distributing softmax probability among equal-priority tokens.

3.2 The “Optimal Solution Search” Nature of Multi-Head Attention

3.2.1 Representational Capacity Advantage of Multiple Heads

Cui et al. (2024) mathematically proved that multi-head attention is superior to single-head attention—the multiplicative constant in prediction loss is smaller. A key detail: adding an extra linear layer to a single head does not change the matrix rank (it remains d+1), indicating that additional layers do not expand a single head’s representational capacity. Only multi-head parallel search across multiple subspaces truly breaks through this limitation.

3.2.2 The Design Intent of Multiple Heads

Each head independently computes attention patterns, capturing different types of relationships—syntactic structure, semantic similarity, long-range dependencies. In theory, H heads should provide H complementary perspectives, producing a richer representation when aggregated than any single head.

3.3 Three Manifestations of the Contradiction

3.3.1 Semantic Dilution

Each attention head maps embeddings to a d_head-dimensional subspace through independently learned projection matrices (W_Q, W_K, W_V). This is not a physical slicing of the embedding vector but a learned linear re-encoding—yet subspace projection, inter-head aggregation compression, and the rank constraints of concat + linear projection can still lead to information dispersion and recombination losses, producing the “Semantic Dilution problem” (Semantic Concentration Multi-Head Self-Attention, 2025). The core paradox: To find the “optimal solution,” the multi-head mechanism re-encodes the “unique solution candidates” through subspace decomposition, yet the linear projection in the aggregation stage may not be able to losslessly recombine each head’s independent discoveries.

3.3.2 Head Redundancy

Empirical research has found that large numbers of heads learn similar attention patterns—most commonly diagonal and vertical lines (Optimizing Knowledge Distillation, 2025). Michel et al. (NeurIPS 2019) showed that a large number of heads can be pruned without performance loss. The promise of multiple perspectives has not been fully delivered—many heads are doing redundant work, consuming computational resources without adding effective information.

3.3.3 Aggregation Bottleneck

Concat + linear projection compresses the output of H heads back to the original dimension. When different heads have each discovered “optimal candidates” pointing in different directions, the linear projection can only produce a weighted average—and a weighted average is the optimal solution of none of the heads. Mixing through row-stochastic matrices cannot increase the effective dimensionality of the output; it typically can only reduce it.

3.4 The Output Layer as an Information Annihilator

3.4.1 Information Discard at Inference Time

The complete probability landscape that multi-head attention constructs in d_model-dimensional space—the probability relationships among tens of thousands of candidates in the vocabulary, their relative rankings, conditional dependencies—all collapses into a single integer (token ID) at the moment of argmax. A 2560-dimensional FP16 state vector (~40,960 bits) produces an output of merely ~15 bits.

3.4.2 Gradient Annihilation During Training

Godey et al. (March 2026, “Lost in Backpropagation”) proved that backpropagating a V-dimensional gradient through a rank-D linear layer (D ≪ V) leads to inevitable compression—95–99% of gradient norm is suppressed by the output layer, producing extremely suboptimal update directions. This gradient compression even prevents trivial patterns from being learned and worsens with vocabulary size.

3.4.3 Self-Reinforcing Deadlock

3.5 The “Valuable Redundancy” Hypothesis (Original Contribution of This Paper)

3.5.1 Redundancy Released When Matrix Computation Aligns Toward a Unique Solution

When matrix computation produces a (seq_len, V) logits matrix, only the last row is used. Within that last row, only the single probability corresponding to the argmax is materialized as the output token. The probability distribution across tens of thousands of tokens in the vocabulary—encoding the model’s complete understanding of “what could come next”—is all irreversibly discarded at the moment of argmax.

3.5.2 The Exclusionary Nature of Next Token

Argmax is a pure peak selector—it applies only peak-value theory from probability, without considering the reasonableness of redundancy. “How small the gap between second place and first place is,” “whether second place is actually superior under certain conditions”—all such information is completely ignored. The essence of the Next Token mechanism is exclusionary elimination of redundancy, without considering the reasonableness of that redundancy.

3.5.3 The “Unargmaxable Classes” of the Softmax Bottleneck

Yang et al. (ICLR 2018) first identified the Softmax Bottleneck, proving that a single hidden state cannot produce all probability distributions, regardless of model size or training data volume. Subsequent research further proved the existence of “unargmaxable classes”—certain perfectly reasonable outputs can never become the argmax result, regardless of input. A 2026 ICLR paper further proved that this low-dimensionality persists over longer token sequences.

3.5.4 Originality Statement

Output layer information discard has been quantified (Lost in Backpropagation, 2026.03). Multi-head redundancy has empirical evidence (2024–2025). The Softmax expressiveness bottleneck is classical theory (Yang 2017). Next-token planning deficiencies have been analyzed (Pitfalls of NTP, 2024). However, connecting all four into a closed causal loop—the structural antagonism between the information producer (multi-head) and the information consumer (argmax), where this antagonism is self-reinforced through signal loss in backpropagation—has not been fully articulated in the academic literature as of June 25, 2026.

3.6 Chapter Summary

Chapter 4 — Layer Two: System Amplification — Cascading Failures in Agent Architectures

4.1 The Quality Cliff of Loop Agents

4.1.1 Empirical Measurement

Production environment measurements from early 2026 (Crosley, 2026) recorded the degradation curve over 30 Loop Agent iterations: 0–30 minutes, precise editing with correct cross-file references; 30–60 minutes, occasional missed imports, still recoverable; 60–90 minutes, collapse into single-file tunnel vision, loss of architectural context; beyond 90 minutes, repetitive attempts contradicting earlier decisions. This “quality cliff” was independent of task type—tasks requiring cross-file state suffered more severely.

4.1.2 Mechanistic Explanation

In long ReAct chains, the original goal is buried dozens of conversation turns earlier—precisely in the region of the context window where attention is weakest. This is the projection of Lost in the Middle onto the temporal dimension. The current engineering fix is “scratchpad mode”—having the agent rewrite the goal and progress to the end of the context at each turn’s conclusion (e.g., todo.md), pushing the plan into the region of strongest attention.

4.2 Multiplicative Amplification in Multi-Agent Systems

4.2.1 Cascading Error Propagation

Agent A’s degraded output enters Agent B’s context as “fact,” Agent B’s conclusions propagate to Agent C, with each hop amplifying the original error. A May 2025 joint study by MSR and Salesforce involving 200,000+ simulated conversations and 15 LLMs reported a 39% performance decline when tasks were distributed across multiple turns.

4.2.2 Context Contamination

The DACS study (Patel, 2026) precisely quantified multi-agent context competition: when N agents share a flat context window, orchestrator decision accuracy collapses from 60% at N=3 to 21% at N=10. Each agent’s task state, partial outputs, and pending questions contaminate the interactions of other agents.

4.3 Information-Theoretic Bounds of Engineering Workarounds

4.3.1 The Strict Constraint of the Data Processing Inequality (DPI)

The Data Processing Inequality stipulates that any irreversible preprocessing strictly limits accessible mutual information. The traditional DPI is strictly contractive; inverse DPI inequalities hold only in highly constrained scenarios (typically unitary channels). When remediation schemes can only operate on internal representations already processed through softmax routing—compressed summaries, attention reallocation, latent state transfers—DPI strictly holds, and information cannot be recovered. However, if the system retains an independent copy of the original evidence externally (raw text, source code, database snapshots, sensor logs) and re-injects it into a new context window through retrieval, it is not recovering damaged representations but bypassing the damage chain. This bypass capability depends on the fidelity of external storage and the precision of retrieval—but retrieved results must still be processed through the same softmax attention mechanism, so the bypass is partial, not complete.

4.3.2 Silent Failure of Compression/Summarization

Models will confidently answer questions about material that no longer contains the answer—hallucinations, subtle errors, contradictions with earlier dialogue. The interface looks perfectly normal; only evaluation captures the degradation. Both the Hermes and Claude Code compression systems can maintain narrative continuity, but “both silently collapse precise value preferences and hard constraints during compression” (Mem0, 2026).

4.3.3 Double DPI Contraction of External Storage

External storage (vector databases, knowledge graphs, file systems) appears to bypass context window limitations, but its information flow path contains two DPI contractions. The first contraction occurs at the storage end: when an agent writes information to external storage, what is written is an internal representation already processed through softmax attention routing—summaries, extracted facts, formatted intermediate conclusions—not the original input signal. This stored information has already undergone one round of attention bias contamination. The second contraction occurs at the retrieval end: when information from external storage is re-injected into the context window, it must be processed again through the same biased softmax attention mechanism. The cascaded result of both contractions: the information fidelity of external storage solutions is bounded by the product of two DPI operations, not one. Exceptions exist but are limited—if external storage preserves completely unprocessed raw text (such as user-uploaded original documents, source code, sensor logs), the first contraction is bypassed, and information fidelity is bounded only by the second contraction (attention processing after re-retrieval).

4.3.4 The Recursive Paradox of Sub-Agent Delegation

The design intent of sub-agent delegation is to decompose a multi-modal problem into multiple unimodal sub-problems—each sub-agent only needs to process a focused task within a limited context, then return a compact result summary. This strategy does indeed transform the multi-modal attention challenge into multiple unimodal problems on the surface. But the recursive paradox emerges at two levels. First, the sub-agent’s output summary is a lossy representation processed through its own softmax routing—critical details, conditional qualifications, and uncertainty signals may be silently discarded during the sub-agent’s argmax output process. What the main agent receives is not the complete solution to the sub-problem but a lossy projection of it. Second, the main agent (orchestrator) needs to coordinate the outputs of multiple sub-agents simultaneously—and this coordination task is itself a multi-modal problem: the main agent’s attention must simultaneously attend to different signals from N sub-agents and make integration decisions among them. The DACS study precisely measured this degradation: orchestrator decision accuracy collapses from 60% at N=3 agents to 21% at N=10 agents—this collapse curve is strikingly consistent with the decay pattern of multi-needle retrieval. This creates an infinite recursion: each layer of “fix” depends on the same flawed underlying mechanism. Sub-agent delegation does not eliminate the multi-modal problem; it transfers it from “finding multiple needles in a long context” to “performing multi-focus integration across multiple sub-agent outputs”—the mathematical structure of the problem is unchanged; only the container is different.

4.4 Industry Status Assessment

4.4.1 Failure Rates and Survival Strategies

Gartner predicts that by 2027, 40%+ of agent projects will be abandoned—due to cost escalation, unclear business value, and insufficient risk control. A 2025 enterprise AI deployment analysis found that 65% of agent failures are attributed to context drift or memory loss. Multi-turn conversations exhibit up to 35% performance degradation compared to single-turn interactions.

4.4.2 Current Survival Strategies

Agents that survive in production environments as of 2026 share three attributes: bounded scope (single domain, defined tool set, explicit rejection of out-of-scope tasks); observable behavior (every tool call logged, every decision point traceable); recoverable failure (inevitable failures are recoverable). The industry philosophy is: acknowledge the black-box nature, use deterministic systems to constrain the black box’s output boundaries—trading away autonomy in exchange for controllability.

4.5 Chapter Summary

Chapter 5 — Layer Three: Physical World Mapping — Comprehensive Exposure After Removal of the Error-Tolerance Layer

5.1 The Information Compression Pipeline of Human Perception

5.1.1 Sensory Input Bandwidth

The human sensory system samples approximately 10⁹ bits/s (1 billion bits per second) from the environment, with the visual system contributing approximately 10⁸ bits/s and the auditory system approximately 10⁵ bits/s.

5.1.2 Brain Bandwidth Bottleneck

Caltech researchers Zheng and Meister (2024, Neuron) confirmed that human thinking speed is approximately 10 bits/s. Compression ratio: 1:100,000,000. “For every hundred million bits of sensory information flooding into the brain, we consciously process only one bit.” Humans can only think about one thing at a time—unlike the thousand-channel parallelism of the sensory system.

5.1.3 Further Loss Through Language Encoding

Reading speed is approximately 300 words/minute, converting to approximately 50 bits/s. English text can be further compressed 50–75% based on redundancy. Written language is the most information-sparse form of human signal—”the lossy nature of language compression is most evident in written language.”

Table 2: Information compression pipeline from the physical world to LLM output (Source: Zheng & Meister 2024, Neuron)

However, a critical information-theoretic distinction must be made here. The ~10⁸:1 bandwidth gap measured above refers to raw data transmission rate in the Shannon sense—how many bits of raw physical signal the sensory channels sample per second, and how many bits consciousness processes per second. However, text is not a linear lossy compression of physical signals—text is a highly abstract extraction of physical rules. Shannon entropy measures the statistical properties of the output; Kolmogorov complexity measures the structure of the generative process: programs, mechanisms, causation. The two measure fundamentally different things.

An equation like F=ma has only a few dozen bits in the Shannon sense, but given a shared mathematical-physics interpreter (calculus framework, coordinate conventions, force models, units of measurement, initial conditions), it encodes the entire set of classical mechanics motion rules at an extremely high compression rate—this is precisely the essence of Kolmogorov complexity: program length is extremely short, but requires a universal Turing machine (i.e., a shared interpreter) to execute. Text has extremely high compression efficiency in the dimension of rule-level information (physical laws, causal relationships, logical constraints), far exceeding what its bit rate suggests. The true shortcoming of text lies in state-level information—continuous spatial coordinates, real-time mechanical parameters, sensor readings, geometric topology, material properties—these continuous physical quantities cannot be efficiently encoded by discrete symbols. The bandwidth gap on the order of 10⁸ truly occurs in the dimension of state-level information, not rule-level information.

5.2 The Incomplete Mapping of the Textual Vector Space

5.2.1 Physical Information Not Recorded in Text

Human text has not completely recorded the state-level information of the physical world. The textual vector space is not a one-to-one mapping of the physical world—it systematically lacks 3D spatial structure, physical dynamics (forces, acceleration, friction), geometric constraints, material properties, temporal continuity, and other continuous state variables. Text can precisely express the rule “gravitational acceleration is 9.8 m/s²,” but cannot efficiently encode the state “the six-degree-of-freedom pose of this cup on the table relative to the robotic arm’s end effector at this moment.”

5.2.2 The “Hallucinatory Plausibility” of the Vector Space

The fundamental problem with vector space-based reasoning is the inability to correctly map the physical world. The “plausibility” of an LLM’s output within the textual information stream is essentially grounded in the abundant redundancy and error tolerance of human judgment regarding text—human readers automatically fill in physical information missing from text, automatically correct imprecise expressions, and automatically ignore unreasonable details. This error tolerance belongs to the human side, not the model side.

5.3 The Critical Transition as Error-Tolerance Space Is Compressed

5.3.1 Text Output → Human Reader (High Error Tolerance)

When an LLM’s output is received by a human reader, the system operates in an extremely high error-tolerance information environment. Human readers automatically perform multi-layer error correction: at the semantic level, readers use their own world knowledge to fill in physical relationships not explicitly expressed in the text (“The ball rolled off the table”—the reader automatically fills in gravity, the table edge, parabolic trajectory, and other physical information not encoded in the text); at the logical level, readers identify and ignore unreasonable details (automatically flagging “the sun rises from the west” as an error rather than accepting it); at the narrative level, readers construct coherence from incomplete information (even when information gaps exist between paragraphs, readers can independently infer transitional logic). This error-tolerance bandwidth makes the LLM’s output “good enough” even when it contains information damage—missing critical conditions, confused causal directions, probability value deviations. The acceptability of LLM text output fundamentally depends on the receiving end’s (human) error-tolerance capacity far exceeding the sending end’s (model) error rate. This asymmetry is the hidden infrastructure underlying the LLM’s success in the textual domain.

5.3.2 Physical Action Output → Physical Laws (Zero Error Tolerance)

When the LLM’s output must align with the physical world—AIGC image and video generation, embodied intelligence controlling robots, autonomous driving planning—the error-tolerance space is drastically compressed. Physical laws do not offer error tolerance. A robot grasping action that is “approximately right” means a failed grasp—finger pose deviation beyond the millimeter scale leads to slipping or collision; a navigation path that is “approximately right” means a collision—path planning errors will not be “charitably interpreted” by the physical world; image generation that is “approximately right” means humans can immediately identify errors—extra fingers, shadows violating perspective relationships, liquid flows defying gravity. In the textual domain, the LLM’s output survives through human error tolerance; in the physical domain, the receiving end switches from “an error-tolerant human reader” to “zero-tolerance physical laws,” and all underlying information damage—softmax attention bias, argmax information collapse, multi-turn DPI cascades—is simultaneously, completely, and irremediably exposed. The massive manifestation of LLM output failing to align with physical world behavior via vector space reasoning is not because the model has weakened, but because the error-tolerance layer that masked the damage has been removed.

5.3.3 Empirical Evidence

Navigation agents hallucinate non-existent paths, operation planners propose physically infeasible grasps, and embodied systems misjudge object distances by orders of magnitude (Spatial Intelligence Survey, 2026). LLM language priors override visual evidence—spurious correlations in training data lead to hallucinations (HEAL, 2026). When generative models are used to predict future video frames, “the result becomes a blur—because it doesn’t know what actually happened” (LeCun).

5.4 MLLM + WM: Compensation, Not Elimination

5.4.1 The Continuous Spectrum and Threshold Effect of the Ontological Contradiction

The ontological contradiction is not a binary switch (on/off) but a continuous spectrum. The observable damage of the contradiction depends on the product of three variables: input information density × output-side argmax damage rate × task fidelity requirements. The symptoms of the contradiction become overtly exposed only when this product causes the system’s overall fidelity to fall below the task’s usability threshold.

Pure text input (~50 bits/s Shannon bandwidth, missing virtually all state-level physical information) × argmax damage × physical task precision requirements → falling far below the usability threshold. This is precisely the information-theoretic root cause of repeated embodied intelligence failures.

The success of Emu3 (Nature, 2026) validates this framework rather than refuting it: Emu3 uses the exact same Softmax + Argmax pipeline as pure-text LLMs—a decoder-only Transformer, 8.5 billion parameters, standard next-token prediction. But it raises input-side information density by several orders of magnitude through direct ingestion of tokenized visual, video, and action signals. Visual tokens carry not the symbolic information of text but state-level physical information such as spatial structure, geometric relationships, and material textures. Even though the output side still suffers argmax damage, the information gain at the input side keeps the system’s overall fidelity above the usability threshold for physical tasks.

5.4.2 Physical Constraint Reconstruction by World Models

World models reconstruct the physical constraints discarded by the human brain through physics-based environment simulation—serving as a replacement layer for the removed human error-tolerance function. MLLMs generate task plans, WMs verify feasibility, enabling iterative refinement. LeCun’s JEPA attempts to bypass the Transformer framework entirely: rather than generating text or pixels, it learns and predicts abstract representations of future states.

5.4.3 Open Questions

Text exhibits an orders-of-magnitude bandwidth gap with the physical world in the state-level information dimension—can current bridging methods sufficiently compensate for this gap? Does the improvement from increased input-side information density exhibit diminishing or linear returns? Will a system above the threshold fall back below it as task complexity increases? Continuous vector space vs. discrete tokens: which representation is more suitable for state-level physical world information? These remain open questions.

5.5 Chapter Summary

Chapter 6 — Unified Causal Chain and Theoretical Synthesis

6.1 Four-Layer Fault Propagation Chain

6.2 Not Independent Problems, but Co-Originating Problems

Hallucination ← attention allocates probability to parametric knowledge rather than context evidence. Reasoning degradation ← TC⁰ circuit complexity limits + information loss of early steps in long reasoning chains. Agent uncontrollability ← cascading amplification of multi-turn attention damage. Embodied AI failure ← textual vector space lacks physical grounding. A large number of core failure modes share the same information-theoretic structure, converging on: information routing defects of the attention mechanism.

6.3 First Structural Constraint: The Spatial Routing Contradiction

Softmax attention is the most critical cross-token information routing decision point in the standard decoder-only autoregressive Transformer. MLPs, residual connections, LayerNorm, and MoE routing also participate in information processing, but softmax attention is the sole mechanism responsible for selective cross-token information routing—causal masking further restricts the directionality of this routing, allowing each token to attend only to preceding positions. When this router itself has structural biases (zero-sum competition, positional priors, entropy trending toward maximum as length increases), the information flow errors it produces are not local—they contaminate the entire downstream computational graph. The information routing defect of attention is the core structural factor pervading multiple black-box problems—hallucination, reasoning failure, long-context degradation, agent uncontrollability, when viewed from an information-theoretic perspective, are all different manifestations of cascading DPI contractions along the same Markov chain.

6.4 Second Structural Constraint: Autoregressive Temporal Directionality

The Transformer architecture possesses a second structural limitation independent of attention routing: the unidirectional temporality of autoregressive generation. Ye et al. (ICLR 2025, “Beyond Autoregression”) explicitly identified fundamental limitations of autoregressive models in complex reasoning and planning—thereby arguing for discrete diffusion as an alternative paradigm. Pachet & Roy (2025, arXiv:2604.07855) further proved that for concisely represented autoregressive models, exact sentence-level maximum a posteriori (MAP) decoding is NP-hard—this hardness persists under unary and metric constraints of first-order Markov chains. On the sampling side, even for regular constraints such as fixed-length termination events, exact conditional normalization is #P-hard. This means autoregressive models provide local access to next-token probabilities, not global pattern finding or tractable general procedures under precise conditional constraints. Left-to-right conditional probability unidirectional generation ensures the model inherently lacks global planning and backtracking error-correction capabilities: it can only see the past, cannot modify already-generated tokens, and cannot make precisely optimal selections among multiple global strategies. This limitation is independent of the attention mechanism—even if softmax were perfectly replaced and argmax collapse eliminated, the autoregressive constraint would persist independently.

6.5 Two-Dimensional Architectural Limitation Model

Synthesizing the above analysis, the architectural limitations of the Transformer can be organized into two conceptually separable dimensions:

The two dimensions are conceptually separable, mechanistically coupled, and require joint optimization in engineering. This paper provides a complete theoretical treatment of the first dimension; the existence of the second dimension is marked as a known boundary of the framework. Conceptual separability is demonstrated by: the spatial dimension’s problems (softmax zero-sum competition causing multi-modal fidelity decay) can exist independently even in non-autoregressive architectures (such as BERT); the temporal dimension’s problems (lack of global planning in left-to-right generation) persist independently even if softmax were replaced. However, in actual decoder-only models, the two are coupled—Chain-of-Thought simultaneously changes temporal unfolding (increasing token sequence length) and spatial routing (externalizing internal candidate information as context, thereby altering attention allocation); Multi-Token Prediction simultaneously affects output-side information collapse and short-range temporal dependency structure; Diffusion LM simultaneously attacks both dimensions (eliminating argmax while simultaneously eliminating unidirectional generation). Subsequent work needs to characterize the interaction term between the two—under what task conditions each serves as the dominant bottleneck, and whether they amplify each other.

Chapter 7 — Current Solution Landscape and Pathway Assessment

7.1 Attacking the Input Side: Repairing Attention Allocation

Sparse attention replacing Softmax: α-entmax replaces Shannon entropy with Tsallis entropy, producing genuinely sparse distributions—assigning exact zero probability to irrelevant tokens. ASEntmax (ICLR 2026) maintains 96.4% accuracy at 256× length extrapolation (vs. 80.2% for softmax). Elastic-Softmax adds a ReLU filter after standard softmax.

Positional encoding improvements: RoPE → YaRN → LongRoPE → HoPE → 3D-RPE → Layer-Specific Scaling → TAPE. The core direction is preventing positional encodings from forcibly imposing distance decay.

Hybrid architectures: SSM/Mamba + minimal attention layers. Jamba (AI21, 1:7 attention:Mamba ratio, 256K context, 3× throughput); Nemotron-H (NVIDIA, 92% of layers replaced by Mamba2); Qwen3-Next (Gated DeltaNet 3:1 linear:full attention ratio).

7.2 Attacking the Output Side: Mitigating Argmax Information Annihilation

Multi-Token Prediction (Route A: Damage Reduction): Output multiple tokens at once, reducing the information collapse per argmax operation. DeepSeek-V3 and Qwen3 have adopted this in production. ICLR 2026’s Parallel Token Prediction eliminates the inter-token independence assumption.

Latent Reasoning (Route B: Life Extension): Perform additional computation steps in the high-dimensional internal space, deferring collapse into discrete tokens. A 2560-dimensional FP16 state vector vs. 15 bits/token of language output. Coconut progressively replaces explicit reasoning tokens with continuous latent states; Quiet-STaR trains learnable “thinking” tokens at each token position.

Diffusion LM / JEPA (Route C: Root Cure): Bypass tokenization and argmax entirely, generating directly in continuous vector space. LLaDA—the first 8B-parameter diffusion LLM, matching Llama3-8B and resolving the “reversal curse” for the first time. JEPA—learning and predicting abstract representations of future states. Furthest from production but most fundamental.

7.3 Attacking the Physical World Side: Recovering Front-End Information Loss

MLLMs directly ingest visual, audio, tactile, and other non-textual signals, bypassing the human text compression bottleneck. WMs simulate physical laws, providing a “reality check layer” that replaces the removed human error-tolerance function. Emu3 (Nature, 2026) demonstrates the maximum potential of the Next Token paradigm in multimodal unification; PaLM-E demonstrates integration of language with sensor inputs.

7.4 Pathway Comparison and Assessment

Table 3: Mapping of 2026 frontier solutions to this paper’s two-dimensional architectural limitation framework (Source: Industry benchmark compilation, cited as phenomenological evidence)

Chapter 8 — Discussion

8.1 Originality Positioning

This paper’s core findings—the ontological antagonism between multi-head attention and Next Token, the “Valuable Redundancy” hypothesis, and the complete connection of the four-layer causal chain—exist in fragmentary form in existing literature but have not been unified. Output layer information discard has been quantified (Lost in Backpropagation, 2026.03); multi-head redundancy has empirical evidence (2024–2025); the Softmax expressiveness bottleneck is classical theory (Yang 2017). But the unified framework connecting all four into a closed causal loop has not been found in existing literature.

8.2 Limitations

This paper’s framework is theoretical, based on logical derivation and literature synthesis, and some inferences still require further empirical validation. The relative contribution weights of each layer in the four-layer causal chain have not been quantified. The actual effectiveness of alternative architectures (Diffusion LM, JEPA) is still being evaluated. The information content of “Valuable Redundancy” requires more precise information-theoretic measurement. The interaction effects between the spatial routing contradiction and autoregressive temporal directionality (see Sections 6.3–6.5)—whether they amplify each other and under what task conditions one dominates—still require more theoretical and empirical work to clarify.

Computational Thermodynamic Constraint—A Third Dimension Not Covered by This Framework: This paper focuses on analysis in the information fidelity dimension and does not discuss the energy cost dimension of information processing. The “lossy collapse” of autoregressive generation is not merely an information-theoretic deficiency—it may simultaneously be the Pareto-optimal trade-off for an intelligent agent in the physical universe under a given energy budget constraint. Although the human brain possesses the capability for high-dimensional global planning (similar to the iterative global optimization of diffusion models), everyday language output relies heavily on low-energy, autoregressive-like heuristic intuition channels—suggesting that autoregression’s “impairment” may not be a pure design defect but an evolutionary optimization for combating thermodynamic entropy increase while achieving low-energy real-time interaction. Global planning models (such as diffusion models) require full-dimensional iterative computation at each generation step, with time complexity O(T·N²) (where T is the number of denoising steps), which preserves Kolmogorov complexity perfectly in information-theoretic terms but may violate biological intelligence’s free energy minimization principle in terms of physical computational energy consumption. A complete architectural limitation theory may require three dimensions—spatial routing fidelity, temporal generation directionality, and computational energy efficiency—this paper treats the first two dimensions in detail, and calibration of the third dimension is left for subsequent work. This implies: even if a replacement architecture that is informationally perfect in fidelity were found, if its computational energy cost makes it undeployable in real-time interaction, then the autoregressive paradigm may persist due to its “good enough and fast enough” Pareto properties—just as humans choose heuristic intuition over precise reasoning in everyday conversation.

8.3 Implications for Architecture Design

The goal should not be to “fix” softmax within the decoder-only Transformer. The most promising direction is to constrain attention to the domain where it excels—unimodal, short-range, high-precision retrieval—while delegating “multi-modal” demands to linear state mechanisms (SSM/Mamba) that natively support multi-modality. Attention transitions from “the sole everything” to “a precision instrument for critical moments.” This paper’s two-dimensional framework further suggests: attacking only the spatial dimension (replacing softmax) is insufficient, and attacking only the temporal dimension (replacing autoregression) is also insufficient—damage must be reduced simultaneously along both dimensions. The reason Diffusion LM is theoretically labeled as the “root cure” route is precisely because it simultaneously attacks both dimensions. However, as discussed in Section 8.2, computational thermodynamic constraints may render pure diffusion approaches undeployable in real-time interaction scenarios—therefore, a hybrid architecture (activating iterative optimization when global planning is needed, using fast autoregression for routine generation) may be the Pareto-optimal engineering solution. This paper’s framework predicts: the effectiveness of Chain-of-Thought has an information-theoretic ceiling—because each step of CoT still passes through the softmax bottleneck, and according to DPI, the cumulative fidelity of multi-step reasoning is bounded by the product of each step’s fidelity.

8.4 Implications for Agent Engineering

Until the underlying architectural contradiction is resolved, agent system reliability has an information-theoretic ceiling determined by per-turn attention fidelity. Current best practices (DAG workflows, human-in-the-loop, bounded scope, state externalization) are correct but bounded responses—they manage the consequences of a bounded system, not a pathway toward unbounded reliability. External state storage can partially bypass the DPI information damage chain—when what is stored is raw evidence unprocessed by the model (source code, database snapshots, sensor logs), retrieval is equivalent to injecting a new lossless signal source into the damage chain—but retrieved results must still be processed through the same softmax attention mechanism, so the bypass is partial. This paper’s two-dimensional framework provides a new diagnostic dimension for agent engineering: agents’ multi-turn degradation stems not only from spatial-dimension attention damage cascades but also from temporal-dimension autoregressive planning deficits—each step of the agent’s decision-making is local conditional probability generation, inherently unable to search for optimal paths in the global strategy space. This explains why agents degrade faster on tasks requiring long-range planning (such as multi-step project management, complex code refactoring) than on tasks requiring local reaction (such as single-step tool calls, simple Q&A).

8.5 Implications for Embodied Intelligence

Text exhibits an orders-of-magnitude bandwidth gap with the physical world’s requirements in the state-level information dimension (continuous spatial coordinates, mechanical parameters, sensor readings), making MLLM + WM a necessary direction. However, it should be noted: text’s Kolmogorov compression efficiency in the rule-level information dimension (physical laws, causal relationships) far exceeds what its Shannon bit rate suggests—LLMs can precisely know F=ma, but cannot know the six-degree-of-freedom pose of the cup currently on the table. The research question is not whether to bridge modalities, but whether current bridging methods can stably maintain the system’s overall fidelity above the usability threshold for physical tasks. Emu3’s success (Nature, 2026) demonstrates that, with the decoder-only next-token architecture unchanged, the system can cross the usability threshold for physical tasks by ingesting visual and action tokens to increase input-side information density. But whether this success generalizes to higher-precision manipulation tasks (precision assembly, flexible object manipulation, multi-body dynamic interaction) remains an open question—higher precision means narrower error-tolerance space, potentially causing the system to fall back below the threshold.

Chapter 9 — Conclusion

9.1 Summary of Major Findings

This paper proposes a two-dimensional architectural limitation framework for the decoder-only autoregressive Transformer. In the spatial dimension, it connects the mathematical properties of softmax, the architectural contradiction of the Transformer, cascading failures of agent systems, and the physical world mapping rupture of embodied AI into a causal chain governed by information-theoretic principles. The framework identifies the tension between multi-head attention’s information construction and next-token prediction’s information annihilation as the generative source of a large number of core LLM failure modes. In the temporal dimension, it identifies the unidirectionality of autoregressive generation as a second structural constraint independent of attention routing. The two dimensions are conceptually separable but coupled in practice—subsequent work needs to characterize their interaction term. MLLM + WM compensates for structural damage at the output side by increasing input-side state-level information density, suppressing the symptoms of the spatial-dimension contradiction below the usability threshold. This paper also identifies the energy cost dimension of information processing (computational thermodynamic constraints) as a possible third dimension needed for a complete architectural limitation theory, but does not develop it in this paper.

9.2 Future Research Directions

Experimentally validate the independent contributions and interaction effects of each layer in the four-layer causal chain. Quantify the degree to which different alternative architectures resolve each layer’s problems. Explore the conditions and limits under which MLLM + WM can maintain system fidelity above the usability threshold for physical tasks. Fully theorize the second dimension (autoregressive temporal directionality), construct a spatial routing contradiction × temporal generation contradiction two-dimensional architectural limitation theory, and characterize the coupling interaction term between the two dimensions. Quantify the mutual information actually contained in “Valuable Redundancy.” Study the interaction effects between the two dimensions—under what task conditions each serves as the dominant bottleneck. Explore the Pareto frontier between information fidelity and computational energy cost—under what energy constraints autoregression’s “lossy collapse” is the optimal information processing strategy, and whether the information-theoretic advantages of alternative architectures (Diffusion LM, JEPA) can be realized within acceptable energy budgets.

9.3 Concluding Remarks

The decoder-only autoregressive Transformer will not disappear. But the attention mechanism must evolve from “the sole everything” to “a precision instrument for critical moments”—this is not a rejection of a successful architecture but a clear-eyed recognition of its capability boundaries and an information-theoretic guide for next-generation architectures. Patchwork engineering solutions are bounded by information-theoretic constraints; fundamental architectural evolution—multi-token output, latent reasoning, continuous space generation, multimodal and world model fusion—is the necessary path. But this path itself is constrained by computational thermodynamics: a Pareto trade-off exists between the perfection of information fidelity and the reality of computational efficiency. The ultimate architecture is unlikely to be an either/or between “the perfect global planner” and “the impaired local generator,” but more likely a hybrid—using attention at critical decision points requiring precise backtracking, linear-time mechanisms for long-range sequence processing, and diffusion-style iterative optimization for tasks requiring global planning—just as the human brain flexibly switches between intuitive fast thinking (System 1) and deliberative deep reasoning (System 2). The two-dimensional architectural limitation framework proposed in this paper, together with the identified third dimension (computational thermodynamics), provides a coordinate system for evaluating and designing such hybrid architectures.