Rigor Problems in
AI Experimentation

1Introduction: Core Claims and Experimental Controls of the Analyzed Paper

Dylan Zhang et al. (University of Illinois Urbana-Champaign) published the arXiv preprint “Useful Memories Become Faulty When Continuously Updated by LLMs” (arXiv:2605.12978) in May 2026, investigating the effectiveness of the “experience distillation → text storage → iterative rewriting” memory consolidation paradigm in LLM agents. The paper distinguishes between two forms of memory: episodic traces (raw action trajectories) and consolidated abstractions (abstract experience rules compressed by an LLM from multiple episodes). The core finding is that the latter, under continuous updating, initially improves but then degrades, ultimately falling below the no-memory baseline. The paper recommends that robust agent memory should treat raw episodes as first-class evidence and apply explicit gating to consolidation.

As reported in the original paper, the tested models include GPT-5.4, GPT-5.4-mini, GPT-5-mini, GPT-5-nano, Qwen3.5-27B/9B/4B, among others, using memory frameworks such as AWM, ExpeL, ACE, and Dynamic Cheatsheet, across benchmark tasks including ARC-AGI, ALFWorld, ScienceWorld, WebShop, AppWorld, and Mind2Web. The core reported effect size is a decline from 100% to 54% on ARC-AGI (46 percentage points).

1.1 Existing Experimental Controls in the Original Paper (Fair Acknowledgment)

Before proceeding to this paper’s critique, it is essential to first acknowledge the effective experimental controls the original paper has implemented over memory mechanism variables:

1.2 Item-by-Item Evidence Audit

The following provides an item-by-item localization and sufficiency assessment of the original paper’s experimental elements:

2Unreported Ablation of Transmission Variables

Having acknowledged the original paper’s effective mechanism variable controls, we identify the following transmission variables for which no systematic ablation is reported in the paper’s main text or appendices. It must be re-emphasized that, without examining the source code, “no ablation reported in the paper text” is not equivalent to “not considered at all during experimentation.” The following analysis is strictly based on information available in the paper’s text and appendices.

2.1 API Message Role Assignment Not Reported

The original paper’s Appendix B discloses the text content of multiple prompts but does not specify how these contents are assigned to roles in actual API calls—which contents go into the system message, which into the user message, and whether memory blocks are injected as independent messages or concatenated within other content. Existing research (Reference [4]) confirms that the placement of identical information in system prompts versus user messages produces significant differences in model output. In multi-model testing scenarios (GPT-5.4, Qwen3.5, etc., which use different chat templates), differences in role assignment may constitute an additional confounding factor in cross-model comparisons. However, it should be noted that if the original paper observed similar degradation patterns across both GPT-5.4 and Qwen3.5 (whose chat templates differ substantially at the physical level), this cross-model consistency would actually weaken the explanatory power of chat template differences as a primary confound—though this point remains speculative without being able to confirm the similarity of degradation curves across the two model families.

2.2 No Cross-Format Ablation of Memory Serialization Format

The original paper’s memory system uses a hybrid format of Markdown-like section markers, JSON output schema, and natural language strategy entries—considerably more structured than plain text strings. However, the paper does not report a comparison of degradation differences during iterative consolidation between this hybrid format and alternatives (pure Markdown, pure JSON, YAML, XML, tool/function output, etc.). Because different serialization formats differ in tokenizer distribution, attention weight allocation, and structural boundary recognition, format choice may affect the information fidelity of each rewriting round, thereby altering the slope of the cumulative degradation curve.

2.3 Positional Effects of Memory Blocks Within Context Not Reported

The original paper’s Appendix B.7 mentions that selected strategy text is injected into the memory block of the synthesis prompt. However, the relative position of this memory block within the full context—whether it is placed near the instruction, near the examples, or near the output schema—is not sufficiently reported. Existing research (Reference [5]) confirms that the position of examples and information within the prompt has significant effects on model performance, with end-of-context placement capable of flipping 30% of QA predictions.

2.4 Context Length Management Strategy Insufficiently Reported

As iteration rounds increase, the length, density, entry distribution, and abstraction level of the memory state continuously change. Even if the system allows deletion (the Delete operation in Auto mode) or compression (the Consolidate operation), the paper still needs to report the per-round token budget, truncation rules, maximum memory capacity, entry eviction strategy, and the relationship curve between token length and performance. Some research and engineering observations indicate that long-context inputs may exhibit reasoning quality degradation well below the nominal window size limit; specific thresholds depend on the model, task, and information distribution. Context length management is the only dynamically characterized variable among the four transmission variable categories identified in this paper.

3Causal Path Analysis

To clarify the precise scope of this paper’s critique, the following maps the original paper’s experimental controls onto a causal path diagram, annotating controlled paths and paths with unreported ablation:

The original paper effectively controls the upper portion of the causal paths—all edges from update schedule, consolidation mode, and trajectory source to memory content quality. This paper’s critique targets the lower portion: edges from serialization format, API role, position, and context length to injection fidelity and reasoning degradation. These two sets of paths are in a confounded state in the original paper’s experimental design.

Note: The diagram above is a simplification. In actual experiments, transmission variables may interact with iteration count, model family, and retrieval strategy (e.g., serialization format × iteration count affects cumulative fidelity of memory content; API role × model family affects the absolute level of injection fidelity). The FORMATSPREAD study also found that format performance exhibits only weak correlation across models, further confirming interaction effects between format and model family.

4Existing Research: Known Effects of Transmission Variables

The following summarizes existing research organized by evidence strength. It must be clarified that external prompt-sensitivity literature serves only a normative role in this paper—establishing that “these variables should be treated as experimental factors”—rather than an empirical role of “explaining the measured decline in the original paper.” These effect sizes come from different models and tasks and cannot be directly transferred to the original paper’s specific scenario.

5Attribution Analysis

5.1 Distinguishing Static and Dynamic Variables and Their Interactions

The transmission variables identified in Section 2 can be categorized by temporal characteristics into two types. Format and position are static variables—they remain constant throughout the entire experiment. If static variables were the sole cause of performance decline, the loss should appear as a step function in the first round rather than the gradual degradation reported in the original paper. Context length change is a dynamic variable—it continuously changes as the memory state evolves, naturally explaining gradual decline.

However, static variables can transform into dynamic influences through compound effects within iterative loops. The following is a theoretical toy model (hypothetical values, not empirical data): suppose that under one format, per-round information fidelity is 95%, while under a superior format it is 98%—a single-round gap of only 3%. After 50 iterations, compound fidelity drops to approximately 7.7% and 36.4%, respectively. Format choice, as a static variable, becomes an accelerant of degradation through its interaction with the iterative mechanism. It must be noted that this model assumes a constant per-round loss rate that is identically and independently distributed; actual LLM information fidelity may vary nonlinearly with content complexity and memory scale. This toy model serves solely to illustrate the concept that “static variables can transform into dynamic effects through iteration” and is not intended as a quantitative prediction.

5.2 Ecological Validity Limitations of Cross-Model Effect Size Transfer

The effect sizes cited in Section 4 carry important applicability limitations: the 76 percentage points come from LLaMA-2-13B, the 40% from GPT-3.5-turbo—both are earlier or smaller models. If GPT-5.4, as claimed by the original paper, represents a more capable frontier model relative to GPT-3.5/LLaMA-2, its format sensitivity may be lower than that of earlier models; however, this assumption itself requires empirical validation under identical task and prompt conditions and cannot be presupposed.

The FORMATSPREAD study’s own conclusion is that increasing model scale reduces format sensitivity but cannot eliminate it to zero. The core argument is: the existence and directionality of these variables have been amply demonstrated, and they need to be ablated—or at minimum discussed—in experiments, regardless of their absolute magnitude on a specific model.

5.3 Three-Layer Failure Attribution Framework

Agent memory failure may originate from three independent layers: content-layer failure—memory is incorrectly summarized, incorrectly generalized, loses critical details, or overwrites original evidence; retrieval-layer failure—irrelevant, outdated, or conflicting memory entries are retrieved; injection-layer failure—the memory’s serialization format, position within the context, and total context length prevent the model from correctly utilizing the memory.

The original paper’s experimental design primarily focuses on demonstrating content-layer failure—through ground-truth trajectory testing and episodic-only comparisons, it shows that the consolidation step itself introduces content degradation. This is the paper’s strongest contribution.

5.4 Response to the Ground-Truth Experiment

The original paper’s most compelling evidence is the ground-truth trajectory test: ground-truth answer trajectories are used as consolidation input, yet performance still declines. This directly demonstrates that the consolidation step itself—rather than input trajectory quality—introduces degradation. This paper acknowledges this as strong evidence for the existence of content-layer degradation.

However, even under ground-truth conditions, injection-layer variables remain present and constant: the memory serialization format is unchanged, the position within the context is unchanged, and the API role assignment is unchanged. Therefore, the ground-truth experiment demonstrates that “the consolidation step + the current injection configuration” jointly cause degradation, but it still does not fully separate the content-layer degradation of the consolidation step itself from the injection-layer influence of the injection configuration. If the ground-truth experiment were repeated under an optimal injection configuration and the residual degradation measured, this separation could be completed.

6Proposal: Transmission Variable Ablation Experiment Design

The following control group design aims to separate injection-layer variable influences from content-layer semantic drift, serving as a supplement to—not a replacement for—the original paper:

The residual performance decline in Group E represents the net effect of consolidation semantic drift. If Group E’s decline remains close to 46 percentage points, this proves the original paper’s attribution is essentially correct and injection-layer variable influence is negligible; if Group E’s decline is significantly reduced, this indicates that a substantial portion of the effect reported by the original paper originates from pipeline implementation rather than the mechanism itself. Both outcomes carry significant academic value.

Full execution of the A–E complete permutation ablation requires substantial token costs and compute budget—an engineering reality that prevents many original research papers from conducting comprehensive transmission variable ablation. As a Minimum Viable Experiment, we recommend prioritizing Group A (format ablation), because format variables have the strongest reported effect sizes in existing literature, incur the lowest experimental cost (requiring only switching the serialization format for the same batch of memories), and the results can directly determine whether injection-layer variables warrant further ablation.

6.1 Recommended Tracking Metrics

To ensure the comparability and interpretability of ablation experiment results, the following metrics should be tracked at each iteration round:

7Methodological Reflection

The core lesson from this case is that LLM agent experiments involve two parallel classes of experimental variables, both acting independently on the same experimental outcome with no a priori weight difference:

Mechanism variables—concerning “what the memory system does”: memory construction strategy, update scheduling, episodic vs. consolidated, consolidation decision modes, etc. These are the objects that researchers typically focus on and control explicitly, and the original paper does so diligently in this dimension.

Transmission variables—concerning “how information reaches the model”: serialization format, API role assignment, position within the context, context length management, etc. These are treated as implementation details in traditional ML experiments, but in LLM experiments they function as intervention variables, because LLMs are highly sensitive to the token sequence, structural markers, role labels, and contextual position of their inputs.

These two variable classes are not in a hierarchical relationship but rather in a parallel co-determining relationship. Existing research shows that the magnitude of transmission variable effects can reach a range comparable to that of the primary experimental intervention in certain task and model combinations. Labeling them as “high-level” and “low-level” would falsely imply that the latter is unimportant—whereas the core argument of this paper is precisely that the importance of transmission variables in LLM experiments is systematically underestimated.

Current methodological conventions in the LLM agent field do not universally require the reporting and ablation of transmission variables—this is not an oversight unique to the original paper’s authors. However, as studies such as FORMATSPREAD and “Position is Power” continue to quantify the effects of these variables, incorporating transmission variables into standard experimental reporting norms is becoming increasingly necessary.

8Limitations of This Paper

First, this paper contains no first-hand experimental data. We have argued for the existence of transmission variable effects through cross-referencing the literature and causal path analysis, but we have not actually run controlled experiments to quantify these variables’ true impact in the original paper’s specific scenario.

Second, “the paper text does not report ablation” is not equivalent to “the experiment did not consider it.” The original authors may have implemented reasonable handling of these variables in their code, or through iterative trial and error during experimentation, may have made non-systematic but substantive optimization choices regarding prompt format, position, and other transmission variables. If the original authors’ pipeline is in fact already near a local optimum, the alternative format ablation experiment proposed in this paper may not significantly alter the degradation curve—but this result itself would also be valuable, as it would demonstrate that injection-layer variables are negligible in practice, thereby retrospectively strengthening the original paper’s attribution.

Third, the effect size data cited in this paper come from earlier or smaller models (LLaMA-2-13B, GPT-3.5-turbo), and direct transfer to the models tested in the original paper carries ecological validity limitations. This paper’s core argument relies on the existence and directionality of these variables, not on precise numerical values.

Fourth, this paper’s verification of the original paper’s existing control groups is based on its abstract, methods description, and appendix content, without a page-by-page audit of the complete PDF. There may be a risk of overlooking additional relevant control designs present in the original paper.

9Conclusion

RReferences