IIntroduction: Posing the Problem of Homogeneity

The AI industry in 2026 stands at a subtle inflection point. Parameter scaling for frontier models faces diminishing returns, high-quality training data approaches exhaustion, and token consumption by reasoning models is growing explosively. The vast majority of technical discourse treats these issues as independent engineering challenges. This paper proposes a different perspective: these seemingly independent problems may be different manifestations of the same architectural characteristic—the homogeneity of current AI systems. This is proposed as a unified hypothesis, not as an established causal law.

1.1 Operational Definition of Homogeneity

The core argument of this paper targets the combined effect of optimization homogeneity and functional homogeneity. Structural homogeneity per se is not the problem—it is the engineering foundation that enables deep learning to run efficiently on current GPU/TPU hardware.

1.2 Operational Definition of Functional Heterogeneity

To give this paper’s hypothesis falsifiable boundaries, it is necessary to define what constitutes “functional heterogeneity.” This paper requires three conditions to be simultaneously satisfied:

Criterion 1: Distinct training objectives—Components each optimize different loss functions (e.g., generation loss vs. calibration loss vs. abstention loss).

Criterion 2: State boundary isolation—Components possess independent parameter spaces or state spaces and do not share gradient updates globally with other components.

Criterion 3: Causal isolability—The contribution of a given component to system behavior can be independently measured through ablation or intervention experiments.

System-level arrangements that do not satisfy these three criteria—such as multiple sampling from the same model, same-base Best-of-N selection, or self-play at different temperatures of the same model—are not considered “functional heterogeneity” in this paper, but rather “inference-time compute scaling.” This distinction is critical for preventing the hypothesis from becoming unfalsifiable.

IIPre-Training: Constructing—and Capping—the Ontology

2.1 The Ontological Nature of Next-Token Prediction

Pre-training uses a concise objective function—predict the next token given the preceding ones—to construct, in a high-dimensional parameter space, an internal representation of “what exists in the world and how things relate to one another.” Philosophically, this corresponds to the construction of an ontology. The formula Loss = -log(P) has not changed from microGPT to GPT-4; the roughly ten-million-fold cost difference between them comes from parameter and data scale alone.

2.2 High-Dimensional Representation Space and Emergence

In “Toy Models of Superposition” (2022), Anthropic demonstrated and theorized superposition in controlled toy models—models can encode far more than n concepts in an n-dimensional space using approximately orthogonal directions^[2]. The Johnson–Lindenstrauss lemma provides the mathematical basis for the exponential growth of representational capacity in high-dimensional spaces.

However, multiple conditions that are not automatically satisfied lie between representational possibility and actual emergence: cross-domain co-occurrence patterns must exist in the training data, optimization dynamics must be capable of discovering and encoding these patterns, and feature sparsity must satisfy interference-tolerance conditions. Superposition provides a necessary condition for emergence (representational space) but not a sufficient one.

2.3 The Three Walls of Pre-Training

Data Wall: Epoch AI predicts with 80% confidence that high-quality training data will be exhausted between 2026 and 2028^[19]. Parameter Wall: Different capabilities reach diminishing returns at different parameter counts^[20]. Ilya Sutskever declared at NeurIPS 2024 that “pre-training as we know it will end”^[21]. Evaluation Wall: Human evaluators miss more than half of critical errors when assessing model output—models are surpassing human evaluation capacity^[15].

IIIRL Post-Training: Optimizing—and Bounding—the Methodology

3.1 The Essential Distinction Between Pre-Training Emergence and RL Emergence

If pre-training constructs the ontology, then RL post-training optimizes the methodology. A December 2025 study from CMU provides a reconciling framework: RL produces genuine capability gains only when two conditions are simultaneously met—(1) the task is insufficiently covered in pre-training, and (2) the RL data is calibrated to the model’s “capability frontier”^[11].

3.2 The Methodological Ceiling Is Set by the Ontology

The “Thinking Sparks!” paper was the first to observe RL post-training emergence at the architectural level—post-training on complex reasoning tasks gives rise to functionally specialized attention heads^[8]. Yet the same study also shows that specialized heads under GRPO training primarily optimize the efficiency of using existing knowledge and computational pathways rather than building entirely new ones from scratch. Foster et al. provide a theoretical bound: if the base model’s pass@k performance is poor, RL is pushed into an exponential-cost regime^[14].

IVThe Structural Weakness of Epistemology

4.1 The Current State of Metacognition: Signals Without Stable Control

Complete intelligence requires a third layer: epistemology. Frontier LLMs are not entirely devoid of metacognitive ability—experimental evidence from 2025 shows that frontier models exhibit limited, context-dependent metacognitive signals that differ qualitatively from human metacognition^[16]. However, these capabilities display three structural defects: limited resolution, context dependency, and a perception–control disconnect—the model expresses uncertainty within its chain of reasoning, yet the final output remains confident.

4.2 Why Homogeneous Architecture Makes Metacognition Harder to Stably Emerge

In the human brain, epistemology is an emergent product of interactions among multiple heterogeneous subsystems. It should be noted that brain regions form distributed, overlapping networks rather than independent modules—but the key structural fact remains: different functions are carried by different neural circuits, each with distinct learning rules and time scales.

In the homogeneous architecture of current LLMs, the generative objective and the metacognitive control objective frequently experience gradient direction conflicts. This does not constitute a mathematical impossibility—a single network can in theory jointly optimize multiple objectives. But in the multi-task learning literature, gradient interference is a well-documented challenge. The argument of this paper is not “impossible” but rather “structurally harder”: jointly optimizing generation and metacognitive control within a shared parameter space faces more severe gradient interference and less stable training dynamics.

4.3 Three Properties of Emergent Functional Differentiation Awaiting Verification

The most important challenge to this paper’s hypothesis comes from the fact that homogeneous structures can indeed produce functional differentiation through training. The specialized attention heads observed in “Thinking Sparks!” are evidence of this^[8]. This paper does not prejudge such emergent functional differentiation as “pseudo”—that question must be answered by experiment, not settled by terminology. Specifically, three properties require verification:

Cross-run stability: Does the emergent functional differentiation produce consistent functional patterns across multiple independent training runs? “Thinking Sparks!” notes that specialized heads are “iteratively activated, evaluated, and pruned,” but this is not equivalent to functional instability—one must first use activation patching and causal ablation to align heads functionally before judging.

Causal isolability: Can specialized heads be independently ablated, producing behavioral changes consistent with their functional labels?

Resistance to gradient interference: During joint training, does the function of specialized heads degrade due to gradient interference from other objectives?

If all three tests pass—emergent functional differentiation is stable, isolable, and interference-resistant—then it constitutes genuine functional heterogeneity within a homogeneous architecture, and the hypothesis of this paper must be revised accordingly.

VStructural Defects of the Memory System

5.1 An Analogical Mapping of Memory Functions

Borrowing Cattell’s (1963) crystallized intelligence vs. fluid intelligence framework as an analogy (the original framework describes types of intelligence, not types of memory), the memory systems of current mainstream LLMs are functionally closer to information storage and retrieval serving crystallized intelligence, while lacking the dynamic association and abstract generalization functions that serve fluid intelligence.

5.2 The Layer-by-Layer Stripping of Multidimensional Information

When a human remembers an event, the brain simultaneously binds time, space, relationships, emotion, causality, and other dimensions into a complete memory bundle. LLM memory systems lose these dimensions at every layer of storage and update. A user inputs “After discussing with Lao Wang at the Shanghai office last Wednesday, I feel the pricing strategy might be too aggressive—I’m quite anxious about it”—after LLM memory extraction, this becomes {"memory": "User thinks the pricing strategy is too aggressive"}—losing time, space, relationships, emotion, confidence level, and causal context.

5.3 Memory Is Input, Not Model

Human memory is realized through permanent changes to neural connections—including mechanisms such as synaptic plasticity, sustained neural activity, and retrieval-induced reconsolidation. The information stored in LLM external memory systems is injected as contextual text—the model parameters themselves do not change. After all memories are deleted, the model’s responses are exactly the same as if it had never remembered the user at all.

VIContext Rot: Degradation Through Multi-Factor Accumulation

6.1 Quantitative Evidence for Context Rot

A 2025 study by Chroma tested 18 frontier models and found that every single model exhibited degradation at every input-length increment^[6]. GPT-4’s performance dropped from 98.1% to 64.1% accuracy merely due to differences in how contextual information was organized.

The root causes of context degradation are multifaceted: the Lost in the Middle effect, the increase in attention competition terms as context grows (making it harder to maintain weight on relevant tokens), positional encoding extrapolation, training-length distribution bias, and retrieval failures, among others. Homogeneous architecture is a key contributing factor to this problem, but not the sole cause.

6.2 Agent “Memory”: The Unbounded Accumulation of History Logs

The “memory” of the vast majority of current AI agents consists of resending the entire history as input. Production audit data (industry observation, not academically verified) shows that resent context accounts for approximately 62% of total consumption^[24], and coding agents spend the majority of tokens on locating rather than problem-solving^[25]. Accumulated history not only loses influence on output due to intensified attention competition, but may also degrade output quality by introducing noise.

6.3 The Architectural Cost of Thinking Tokens

Thinking tokens are technically output tokens. Specific reasoning models on specific benchmark tasks generate approximately 18 times the token volume of standard models, with lower accuracy^[26]. Under specific models and workloads, per-query energy consumption can reach 33–40 Wh^[27] (this data comes from specific measurement conditions and should not be generalized to all reasoning models or all queries).

A more noteworthy structural concern is this: the cross-step iterative reasoning of current mainstream reasoning models is primarily realized through autoregressive tokens—each step of thought must be explicitly written as tokens before the next step can proceed. The massive parallel neural activity changes during human reasoning (distinct from long-term learning through synaptic weight changes) do not need to be serialized into language. This “serialization of trial and error” is a structural cost of autoregressive architecture. It should be noted that latent reasoning (Latent CoT) already constitutes a direct counterexample to this limitation, though its performance still lags behind explicit CoT.

VIIOptimal Solution vs. Unique Solution: The Core of Cognitive Finitude

7.1 NTP Is an Optimal Solution, Not the Unique Solution

The entire LLM industry chain is built on an implicit assumption: predicting the next token is the correct path to general intelligence. But this is the optimal solution under current engineering constraints—training data is most abundant, the objective function is simplest, parallelization is most efficient, and scaling laws are clearest—not the unique solution from the standpoint of cognitive theory.

7.2 The Cognitive Operation of Distinguishing “Optimal” from “Unique”

When a cognitive system locks in the current optimal solution as the unique solution, it performs “premature closure.” Every major breakthrough in the history of human cognition has been a challenge to “uniqueness”: Newtonian mechanics was the optimal solution → Einstein asked “Is it the unique solution?” → Relativity. Current AI does not perform this operation—it can search for alternative paths within a solution space, but it does not question the solution space itself.

7.3 The Cognitive Crisis of Blurred Processing

When the informational complexity of a cognitive object exceeds the processing bandwidth of the cognitive agent, humans trigger a cognitive energy-saving strategy—compressing multidimensional information into one-dimensional labels. In 2026, AI’s output speed far exceeds the human cognitive processing speed. Existing research on human attention span and reading speed indicates this gap is widening (the related estimates involve extrapolation and should not be cited as established cognitive facts^[28]). The pressure toward blurred processing will only continue to grow.

VIIIDirections for Breakthrough: Multiple Paths from Homogeneity to Heterogeneity

8.1 Two Dimensions That Must Change Simultaneously

Continuing to scale within a homogeneous architecture is a path of diminishing returns. This paper argues that breakthroughs require simultaneous effort on two fronts: the architectural dimension (introducing functionally specialized heterogeneous components) and the objective dimension (shifting from a single NTP objective to multi-objective joint training).

8.2 Technical Directions Already Under Exploration

RPT (Reinforcement Pre-Training)—integrating RL directly into pre-training, breaking the stage separation between ontology and methodology. Latent Reasoning (Latent CoT)—methods such as COCONUT keep reasoning in hidden-state space^[29], architecturally addressing the serialization of trial and error, though current performance still shows a gap. ThinKV—allocating KV storage at different precisions based on the importance of thinking types, achieving near-lossless accuracy with less than 5% of the original KV cache^[18].

8.3 Approaches Meeting the Definition of Functional Heterogeneity vs. Inference-Time Compute Scaling

An important response to this paper’s advocacy of “heterogenization” is that not all approaches that add system components constitute functional heterogeneity. According to the three-criteria definition in Section 1.2, existing approaches can be divided into two categories:

Approaches meeting the definition of functional heterogeneity (Type A): a generative network combined with an independently trained metacognitive network (different objectives, independent parameters); independently trained reward/verification models (independent loss functions, ablation-verifiable). The components in these approaches have distinct training objectives, isolated state boundaries, and independently measurable causal contributions.

Approaches not meeting the definition of functional heterogeneity (Type B): multi-agent play on the same base model (shared ontological boundaries and parameter space); same-model sampling at different temperatures; Best-of-N selection. The success of these approaches cannot be attributed to “heterogeneity” and more likely stems from search, ensembling, or additional inference computation.

The falsifiable prediction generated by this paper’s hypothesis is: under conditions controlling for equal total parameters and total inference FLOPs, Type A approaches should outperform Type B in calibration error and abstention accuracy. If Type B performs no differently from Type A, the hypothesis of this paper is weakened.

It is particularly important to note that same-base multi-agent play faces a deeper structural limitation: if the primary agent and the verification agent share the same pre-trained base, they share the same ontological boundaries. When the base model lacks a representation for a certain causal relationship, no amount of same-origin interaction can give rise to genuine epistemological verification. Effective system-level heterogeneity may require not only functional differentiation but also ontological diversity.

8.4 Engineering Constraints: The Hardware Economics of Homogeneity

Homogeneity is not an accidental design flaw. The high regularity of matrix multiplication is a perfect fit for the SIMD architecture of GPUs/TPUs. Heterogeneous modules—particularly those involving discrete operations or components operating at different time scales—are extremely difficult to parallelize efficiently on current hardware. Within the current hardware generation, functional heterogeneity meeting the three-criteria definition (e.g., independently trained verifiers) is more feasible than chip-level heterogeneous computing.

8.5 Testable Predictions and Experimental Design

Experiment 1: Ablation comparison of parameter-sharing ratios. Under fixed total parameters and total inference FLOPs, vary only the parameter-sharing ratio: (a) Full sharing—a single network jointly trained on generation + calibration objectives; (b) Partial sharing—bottom encoder shared, top layers split into a generation head and a calibration head; (c) Full separation—two independent networks each occupying 50% of total parameters; (d) Control group—single network trained on generation objective only + Best-of-N selection. Metrics: ECE calibration error, AbstentionBench abstention accuracy, and ARC-AGI subset performance. Key comparison: (c) vs. (d) isolates the respective contributions of “functional heterogeneity” vs. “inference-time compute scaling.”

Experiment 2: Quantifying gradient interference from objective conflicts. Jointly train NTP + calibration loss in the same model, measuring the gradient cosine similarity between the two objectives across training progression. Add three comparison groups: full sharing, partial sharing, and full separation. Introduce conflict mitigation methods such as PCGrad/CAGrad as competing explanations. This paper predicts: gradient cosine similarity will be persistently negative in later training stages, and the degree of conflict will negatively correlate with calibration performance.

Experiment 3: Cross-run stability of emergent functional differentiation. Perform multiple independent RL post-training runs on the same model. First use activation patching and causal ablation to assign functional labels to attention heads, then compare the cross-run consistency of functional labels. If functions are consistent but positions differ, then emergent differentiation possesses stability and this paper’s hypothesis requires revision.

Experiment 4: Memory dimension retention. Present the model with a complex memory containing time, location, people, emotion, and confidence level, then test the retention rate of each dimension after N rounds of conversation. Compare standard context injection vs. a structured memory module (independent state space). Metric: the decay curve of per-dimension F1 scores across conversation rounds.

Experiment 5: Context Rot and modular ablation. Under equal total parameters and context length, compare a standard monolithic Transformer vs. a segmented processing architecture with an independently parameterized summarization module. Metric: accuracy–length curves on Context Rot benchmarks.

8.6 The Epistemology–Cost Paradox

An internal contradiction this paper must candidly confront: Chapter VI diagnoses the inference cost explosion caused by the stacking of Thinking Tokens and Agent Logs, but every heterogenization approach proposed in Chapter VIII (external verifiers, dual networks, multi-agent systems) will without exception further increase the token and FLOP consumption at inference time.

This means: even if heterogenization can resolve the epistemological deficit, it may simultaneously worsen Context Rot and cost explosion. The four categories of limitation are not necessarily solvable by the same approach—inherent trade-offs may exist among them. If a fundamental trade-off exists between epistemological improvement and cost control, then the claim “homogeneity is the shared contributing factor for all four categories of limitation” needs to be weakened to “homogeneity is a contributing factor for epistemological deficit and fluid intelligence failure, while the token consumption problem has an independent architectural root cause (autoregressive serialization).”

IXConclusion

This paper has proposed a unified hypothesis: the multiple limitations of current AI systems—severe epistemological insufficiency, memory dimension loss, Context Rot, and fluid intelligence deficit—may share a single key architectural contributing factor: homogeneity. More precisely, parameter sharing, objective sharing, and insufficient functional isolation may collectively compromise the reliability of metacognitive control, memory management, and long-range reasoning.

Through the trichotomy of ontology/methodology/epistemology, this paper has reframed the relationship among pre-training (ontology construction), RL post-training (methodology optimization), and current weaknesses (epistemology). It has provided a three-criteria operational definition of functional heterogeneity and used it to distinguish approaches meeting this definition from inference-time compute scaling, giving the hypothesis falsifiable boundaries.

The paper has also candidly addressed the internal tension within the hypothesis: a fundamental trade-off may exist between epistemological improvement and inference cost, meaning the four categories of limitation may not be simultaneously resolvable by a single architectural change. The five experimental designs provide actionable starting points for subsequent empirical research.

XReferences