I recently came across Language model harnesses are compositional generalizers  new article from Alex Zhang and Omar Khattab, which presents a timely argument about RLM concepts that we are exploring with RLM Code. The post is already trending on X and argues that generalization does not have to come only from training a larger Transformer on more tasks and longer contexts. It can also come from the harness that controls how a model observes state, decomposes work, invokes other models, stores intermediate results, and produces its final answer.

This is closely aligned with the direction of RLM Code, our open source research playground and evaluation environment for Recursive Language Models. In RLM Code v0.1.11, most of the core systems mechanisms discussed in the harness article are available to inspect, run, and measure today. These include context offloading, programmatic submodel calls, REPL-backed state, bounded root observations, structural history, history offloading, decomposition guidance, repository evidence selection, and trajectory similarity metrics. We have also added a deterministic, API-key-free demonstration that applies the same root policy to tasks from different domains and at an eight-times difference in length. The RLM Code demo validates the harness mechanics and the resulting trajectory structure. It does not claim to reproduce the reinforcement learning experiments or model-quality results reported in the blog post. That would require training model checkpoints and evaluating them across disjoint task families and length buckets. RLM Code now provides many of the components needed to run those experiments rigorously.

Harnesses are Everywhere

An agent is often described as an LLM in the loop, with policy that observes a state and selects an action. In practice, the complete environment state may include a large repository, long documents, tool outputs, previous actions, generated artifacts, and external data. Passing all of that information directly into every model call creates two problems.

First, the prompt becomes increasingly expensive and difficult to manage. Second, the model sees a continually changing mixture of task-specific content, execution output, and interaction history. Even if the underlying tasks share the same logical structure, their token-level representations can look completely different. The harness article proposes a stronger role for the system around the model. A well-designed harness should transform a complex or unfamiliar global state into smaller observations that are familiar to the model. The article calls these observations locally in-distribution, or LID. It is a design objective, not a property that a framework can guarantee in isolation. Whether a prompt is truly in-distribution depends on the model and its training data. A harness can still improve the odds by keeping each model call focused, bounded, and structurally consistent. Meaning, Instead of asking only whether one model can absorb an entire task, we can ask whether the harness can reduce the task to a sequence of simpler calls that the model already knows how to handle.

Compositional generalization through task equivalence

The article describes structurally similar tasks as belonging to the same harness-induced equivalence class. Two tasks may use different vocabulary, come from different domains, and contain very different amounts of data, yet require the same underlying strategy. Examples include filtering, searching, counting, ranking, aggregating, or applying a map and reduce pattern over evidence units. If the harness hides domain-specific content from the root model and exposes only the structure needed to choose the next operation, then the root trajectories for those tasks can become very similar. The task-specific work is delegated to focused subcalls, while code performs the stable orchestration. This is the practical meaning of trajectory isomorphism in an RLM system. The root model does not need to relearn the entire surface form of every problem. It can reuse a decomposition strategy while the submodel calls process the local details.

RLM Code now exposes this idea through three Pure RLM profiles:

• reference preserves the existing configured root observations and full history. It is intended for compatibility and reference experiments, and it does not enable decomposition guidance by default.
• repo_evidence provides metadata with bounded previews, keeps structural history, and enables decomposition guidance. It is intended for repository analysis with selected evidence.
• lid provides opaque status and counts, offloads older structural history, and enables decomposition guidance. It is intended for length and cross-domain generalization experiments.

The profiles are implemented by the PureRLMEnvironment and documented in the RLM Code environment guide.

How the article’s core mechanisms map to RLM Code

1. Context offloading

In a conventional long-context workflow, the complete input is placed in the root prompt. Every additional task document or repository file changes the prompt distribution and consumes more of the model’s context window.

In the Pure RLM environment, the input is stored in the Python REPL as a symbolic context variable. The root model can receive metadata about that variable without receiving the complete contents. Depending on the selected profile, the observation can include bounded previews, metadata only, or opaque status information.

This provides the first layer of abstraction. A four-record task and a 32-record task can present almost the same planning problem to the root model because the data itself remains outside the root prompt.

For repository work, the public RepositoryContextBuilder constructs deterministic and bounded context using four selection modes:

• mini selects a small evidence set for fast experiments.
• evidence ranks files and snippets against stable terms extracted from the task.
• full provides a larger bounded repository view.
• explicit uses only paths supplied by the caller.

The selected material is placed in the REPL context. Selection and exposure are deliberately separate concerns. The builder decides what evidence is available to the execution environment, while the harness profile decides how much of that evidence returns to the root model.

2. Programmatic submodel calling

Context offloading alone is not enough. If every submodel answer and tool result is appended back into the root conversation, the root history will still accumulate domain-specific information.

RLM Code exposes llm_query() and llm_query_batched() as functions inside the code REPL. Model-generated Python can select a focused slice of context, create a small prompt, invoke a submodel, and store the answer directly in a variable. Later code can consume that variable without forcing the complete semantic result through the root model’s context.

A simplified pattern looks like this:

units = discover_evidence_units(context)
prompts = [build_focused_prompt(unit) for unit in units]
findings = llm_query_batched(prompts)
result = aggregate(findings)
FINAL_VAR("result")

This is more than tool use. The REPL becomes a stateful coordination layer in which code, context, submodel calls, and finalization share a common namespace.

3. Bounded and structurally stable root observations

The lid profile uses opaque root observations. Execution feedback reports stable facts such as completion state, output size, variable count, and call count, rather than returning task-specific values to the root prompt.

The repo_evidence profile is less restrictive. It returns metadata with bounded previews, which is useful when a coding agent needs enough evidence to reason about source files while still controlling prompt growth.

This distinction matters because repository analysis and generalization research have different requirements. RLM Code treats exposure as an explicit policy instead of assuming that one observation format fits every experiment.

4. Structural history and history offloading

Long agent runs can become out-of-distribution because each action and result is appended to an ever-growing conversation. RLM Code provides full, structural, and offload history policies.

Structural history retains the shape of prior actions while removing most task-specific values. When the configured root-history budget is reached, the offload policy stores older structural messages in versioned REPL variables such as history_1. The most recent action and result pairs remain visible so the root can continue planning.

This preserves replayable state without forcing the entire trajectory into every subsequent root call.

5. Decomposition guidance

The article reports that a model may find a shortcut on short tasks, such as delegating the complete problem to one subcall. That strategy can work at training scale and fail as task length grows.

The repo_evidence and lid profiles therefore include an optional decomposition hint. It encourages the root model to identify independent evidence units and issue focused subcalls rather than sending one monolithic prompt. This is a form of lightweight supervision that can help an experiment converge on a length-agnostic policy.

The hint does not hard-code a specific algorithm. The model can still choose chunking, search, filtering, aggregation, or another programmatic strategy that fits the task.

6. Bounded traces with root and submodel attribution

Reducing root exposure should not make a run impossible to debug. RLM Code stores bounded subcall prompt and answer previews in the persisted trace, together with full character counts and SHA-256 digests. Calls are attributed to root or submodel roles, and the runner records prompt sizes, hashes, structural actions, and measured subcall counts.

This makes the lid profile an exposure policy rather than a destructive logging policy. Researchers can inspect what happened after a run without injecting all of that data back into the root model while the run is active.

7. Measuring trajectory similarity

The article discusses trajectory distance as a proxy for whether unseen tasks appear similar to tasks observed during training. RLM Code now includes compare_trajectory_similarity and nearest-training-trajectory analysis.

The implementation reports:

• token-level normalized Levenshtein similarity;
• trigram containment;
• trigram Jaccard similarity;
• frequency-weighted trigram Jaccard similarity;
• trajectory length ratio; and
• an aggregate mean across the metrics.

These measurements do not prove that two prompts produce the same model-output distribution. They provide reproducible proxies that can be combined with task correctness, reward, call counts, and length buckets.

8. Benchmark metadata for length and strategy experiments

RLM Code benchmark records can now carry explicit context, expected answers, task family, domain, split, and length information. This supports experiments that separate short training tasks from long evaluation tasks or compare domains that share a latent decomposition.

Caller-provided context is preserved through the runner instead of being replaced by automatic repository discovery. This is essential for controlled experiments because the benchmark must determine the evidence available to the harness.

Run the harness generalization demo

The harness generalization demo is deterministic and does not require an API key. It creates a four-unit commerce task and a 32-unit support task. Both use the same root policy. The policy discovers evidence, issues one focused subcall per unit, stores the answers in the REPL, aggregates them in code, and returns the correct result.

The demo checks that private context and domain labels do not enter the root prompts, the longer task is exactly eight times the size of the shorter task, work is decomposed into focused subcalls, old history is offloaded, and the structural root trajectories remain identical.

Run it from the repository root:

git clone https://github.com/SuperagenticAI/rlm-code.git
cd rlm-code
uv sync
uv run --frozen python examples/harness_generalization/demo.py

The fixed root policy isolates the behavior of the harness from model quality and provider availability. It proves that the context and trajectory controls work as designed. It does not claim that a model learned the policy.

Try the AI Engineer World’s Fair talk demo

We recently presented RLM: Recursive Language Models for Large Codebases at the AI Engineer World’s Fair in San Francisco. The original presentation repository contains the complete React slide deck and the event-day source snapshot.

RLM Code now includes a maintained version of the live probe that tracks the current APIs. It uses RepositoryContextBuilder, keeps repository evidence in the REPL, sends metadata rather than source contents to the root model, runs model-written code in a Docker sandbox with networking disabled, makes one focused llm_query subcall, and completes from the same code block.

With a local Ollama model installed, run:

RLM_TALK_PROVIDER=ollama \
RLM_TALK_MODEL=qwen3.6:35b-mlx \
uv run --frozen python examples/aie_world_fair_2026/rlm_probe.py

You can change RLM_TALK_MODEL to another model available in your Ollama installation. The demo README also includes Gemini configuration and repository, context-profile, timeout, and task overrides. Docker is the recommended execution backend because the probe executes model-written Python. An in-process mode is available only as an explicit option for trusted local smoke tests.

What RLM Code implements?

RLM Code v0.1.11 implements the principal harness mechanisms discussed in the article, but it does not reproduce the article’s RL training runs or reported performance lift. Those results require trained checkpoints and controlled evaluation. Start with the API-free demo, then connect a model and use the lid or repo_evidence profile to study your own workloads

Two claims remain deliberately outside the scope of this release. We have not reproduced reinforcement learning on the benchmark suite used in the article, and we do not claim an independent reproduction of its reported model-performance lift.

The next research step is to replace the fixed root policy in the offline proof with trained and untrained model checkpoints, then evaluate on disjoint task families and longer length buckets. A rigorous study should compare correctness, reward, subcall count, root-token growth, execution cost, and nearest-training-trajectory similarity. It should also test failure modes, including monolithic delegation, excessive root leakage, poor evidence partitioning, and decomposition strategies that do not scale.

Conclusion

For large codebases, long documents, and evidence-heavy tasks, this means keeping bulk state outside the root prompt, turning submodel calls into programmatic operations, storing intermediate information in an execution environment, and measuring whether the root trajectory stays stable as the task changes. RLM Code v0.1.11 provides these capabilities as composable, inspectable building blocks.

Read Language model harnesses are compositional generalizers, explore the RLM Code repository, run the harness generalization demo, and revisit the AI Engineer World’s Fair talk and demo. The central hypothesis is now something developers and researchers can test directly: better harnesses can make unfamiliar, larger, and cross-domain tasks look like compositions of familiar local problems.