Exploring Collaboration between a language and a non-language agent

Get in touch with us at behavior-in-the-wild@googlegroups.com

Abstract

LLMs are increasingly deployed as orchestrators that coordinate specialized subagents to solve complex tasks through natural language. However, in many important domains like game playing and robotics, the strongest available agents are not language models. Integrating non-language agents with LLMs would require verbalization: compressing their rich continuous representations into sparse textual summaries at each interaction step. To study whether verbalization constitutes a bottleneck, we introduce LLAMIA-Bench, a suite of six diverse collaborative chess tasks spanning three facets: behavioral imitation, state assessment, and natural-language explanation. Each task instantiates a well-established chess problem that neither the LLM nor the chess engine can solve alone. To solve LLM collaboration with non-language agents, we introduce latent state internalization, which projects the subagent's continuous representations directly into the LLM's token stream as learned state tokens, with dynamic re-encoding as actions advance the environment state. Comparing internalization to verbalized integration, our experiments reveal a consistent verbalization debt: the performance gap widens throughout training and persists as the LLM scales from 4B to 14B parameters. A single 14B model, LLAMIA, trained with latent state internalization, matches or exceeds task specialists and frontier models including GPT-5.1 with tool access across all benchmark tasks, and generalizes out-of-distribution where task-specific finetunes collapse.

• Latent state internalization. A paradigm for LLM–non-language-agent collaboration, enabling LLMs to reason over an interleaved chain of language, action, and latent state tokens.
• LLAMIA. A two-stage training framework — self-supervised projector alignment followed by end-to-end RL (DAPO) — yielding a single model that achieves state-of-the-art performance across diverse collaborative tasks, giving closed-weight LLMs faithful access to a subagent's latent expertise.
• Verbalization Debt. Through controlled ablations against LLAMIA-Verb, we define and quantify the performance gap between internalized and verbalized integration, showing it is suboptimal in both performance and compute, and widens on tasks requiring deeper or multi-step collaboration.
• LLAMIA-Bench. A curated benchmark of six chess tasks spanning behavior cloning, puzzle understanding, commentary, and planning — unsolvable by either the LLM or the subagent alone.

Method: Latent State Internalization

An LLM plays chess with access to a pretrained engine exposed through a tool API: functions to read the board and legal moves, advance the game, and — critically — query the engine's assessment of any position via get_policy. What that call returns is the variable this paper studies. In the standard verbalized setting, get_policy returns a text summary: the engine's top moves with prior probabilities and value estimates. This captures the headline assessment but discards the rest of the engine's internal state — the full distribution over legal moves, the value landscape across candidate continuations, and positional features like piece coordination and king safety that interpretability work has identified in engine hidden layers. Under internalization, the same call additionally projects the engine's full latent state into k=32 continuous tokens via a learned projection, LatentBridge, and appends them to the LLM's reasoning trace alongside language and action tokens. The LLM attends over all three token types jointly, so its attention can selectively read different aspects of the representation at each reasoning step, rather than being handed one fixed summary regardless of what the step requires.

LLAMIA-Bench

Existing benchmarks test either the LLM or the subagent, never their collaboration. LLAMIA-Bench spans six tasks drawn from four themes in how AI systems collaborate with domain-expert agents — behavioral imitation, state assessment, comparative explanation, and rationale generation — each unsolvable by either component alone: the subagent produces no language, and the LLM lacks the positional signals that make chess-specific judgments possible. Three evaluation targets are new to this work: Wild BC, three out-of-distribution splits probing generalization to grandmaster play (GM-25), time pressure (Low-Time), and large skill gaps (ΔElo); Puzzle Interest, ranking puzzles by community-derived interestingness — a signal with no verbal proxy in any engine output; and Agadmator-2K, the first large-scale dataset of game-level chess commentary, 1,900 narrated games (∼500 hours) from Agadmator's YouTube channel, built by aligning Whisper transcripts to PGN move sequences and verifying move order with a GPT-4o judge.

Key Results

LLAMIA-14B achieves the highest score on all six LLAMIA-Bench tasks, surpassing frontier verbalized systems an order of magnitude larger and remaining competitive with dedicated task-specific finetunes trained on substantially more in-domain data. On behavior cloning, Maia-style experts are trained on tens of millions of chess-specific games against LLAMIA's general-purpose backbone; LLAMIA-14B stays inside the expert band in-distribution and surpasses the strongest expert by a wide margin on the OOD Wild splits. The advantage over GPT-5+Lc0 does not require the 14B backbone — LLAMIA-8B already leads on all six tasks, and LLAMIA-4B on four of six.

Each cell is a per-task score ×100 (higher is better); Rationale is BLEU-2, Commentary is G-eval, Puzzle Understanding is Spearman ρ ×100. bold = best overall in the column, underline = best non-LLAMIA. Dedicated chess finetunes show only the strongest published entry per task; off-task cells are blank. Full per-scale factorial, confidence intervals, and additional metrics in the paper.

Latent tokens enable new evaluation targets. Puzzle Interest requires ranking positions by community-derived interestingness, a signal that depends on the engine's full policy distribution and value gradients across candidate moves — no verbalized engine output carries these features. Every verbalized system scores ≤12 on Interest regardless of model scale or frontier capability; LLAMIA-14B reaches 52. Verbalization carries zero useful signal for this task, while latent tokens give the LLM direct access to the distributional structure that defines interestingness.

Verbalization Debt

LLAMIA and LLAMIA-Verb share the same 14B backbone, Lc0-BT4 subagent, and DAPO recipe; the only difference is whether the subagent's state reaches the LLM as latent tokens or as verbalized text. We define the resulting performance gap as the verbalization debt. The debt is smallest where the engine's top-k moves already approximate the answer (in-distribution behavior cloning) and largest where the target signal lives in the engine's full policy distribution or value landscape, which has no faithful text equivalent (Interest, Commentary). It persists at every backbone scale we test — on Interest, LLAMIA-Verb-14B scores 8 while LLAMIA-4B already reaches 38 — and it widens throughout training, reaching 2–3× by convergence.

Rows are ordered by the size of the debt. It is smallest where the engine's top-k moves already approximate the target (behavior cloning) and largest where the target signal lives entirely in the policy distribution and value gradients across candidate moves, which text cannot carry (Puzzle Interest: ρ 0.08→0.52, a 6.5× jump).

Collaboration Strategies

Does the integration interface determine how the model learns to use the subagent, or only how well it performs? We classify subagent invocations during evaluation into five recurring strategies and trace their evolution during DAPO training: engine-follow (adopt the top recommendation), consult-then-override (query then diverge), counterfactual query (play a hypothetical move, re-invoke, compare states), multi-step lookahead (chain two or three counterfactual sequences), and abstention (act from language knowledge alone). A GPT-4o judge classifies 500 episodes per task per system (κ=0.78 vs. human raters).

Internalization produces task-specific collaboration; verbalization collapses it. LLAMIA adapts its strategy to the task: engine-follow dominates gameplay (65%), consult-then-override dominates behavior cloning (48%), and counterfactual query dominates commentary (40%). LLAMIA-Verb collapses to engine-follow on every task (62–76%), regardless of what the task requires. The verbalized channel returns the same compressed summary no matter how the model queries it, so RL converges on a single use pattern.

Human Evaluation

Skilled players (n=12, all ≥1700 Elo) evaluated LLAMIA on gameplay indistinguishability and commentary quality against LLAMIA-Verb, GPT-5.1+Lc0, and Maia.

Behavioral signatures such as time-pressure blunders and skill-appropriate piece saliency emerge from latent-state conditioning alone; LLAMIA receives no human-move supervision, unlike Maia, which is trained directly on millions of move distributions. On commentary, both systems achieve comparable factual accuracy, but the gap concentrates on strategic insight: text preserves what is happening on the board, but explaining why a move is strong requires representational features — policy gradients, value topology, look-ahead depth — that do not survive verbal compression.

BibTeX

@misc{s2026exploringcollaborationlanguagenonlanguage,
        title={Exploring Collaboration between a language and a
        non-language agent},
        author={Harini S I and Somesh Singh and Yaman K Singla and Rajiv Ratn Shah and David Doermann and Balaji Krishnamurthy},
        year={2026},
        eprint={2609.00474},
        archivePrefix={arXiv},
        primaryClass={cs.CL},
        url={https://arxiv.org/abs/2609.00474},
}

---