LMAC: LLM-Guided Communication for
Cooperative Multi-Agent Reinforcement Learning

Overview

LMAC asks a large language model to design and refine an executable, agent-wise communication protocol from natural-language descriptions of the task, the global state, and each agent's observations. Refinement is driven by a state-awareness criterion that turns offline transition data into language feedback, so the LLM adds only the messages required for accurate and balanced state recovery.

Why a New Protocol?

Existing communication methods can still leave inefficiencies in recovering state information. On a StarCraft II task where allies must infer an enemy position, even strong baselines like MASIA and FullComm remain inaccurate or produce non-uniform reconstruction across agents.

Method

LMAC builds on Reflexion-style iterative refinement, replacing the fixed feedback with step-wise feedback instructions tied to two explicit objectives. The output is an executable Python protocol that maps each agent's observation history to agent-specific messages.

Three Refinement Steps

State-Awareness Indicator (SAI)

To turn quantitative reconstruction quality into language feedback, LMAC trains an auxiliary decoder that reconstructs the global state from each agent's trajectory, with and without the message. Per-dimension accuracy is thresholded into a binary indicator and aggregated into the two criteria above.

Meta-Cognitive Representation for MARL Training

The pre-designed protocol is applied during MARL training to produce agent-specific messages. Rather than feeding raw messages to agents, LMAC introduces an encoder–decoder pair that produces a latent representation used for state inference and decision-making. The encoder is supervised by both state reconstruction and the SAI signal, and a cycle-consistency loss discourages redundant content, encouraging compact, task-relevant features.

Experiments

We evaluate LMAC on four benchmarks: SMAC-Comm (bane_vs_hM, 1o_10b_vs_1r, 5z_vs_1ul, 2o_20b_vs_2r), LBF, GRF, and the more stochastic SMACv2. Baselines include FullComm, MASIA, TarMAC, SMS, T2MAC, NDQ, MAIC, COLA, and the QMIX+State upper-bound. Unless otherwise specified, the backbone LLM is gpt-4.1-2025-04-14. All results are averaged over 5 seeds.

Performance Comparison

On SMAC-Comm, LMAC converges faster and achieves higher final success rates across all four scenarios, with particularly large gains on bane_vs_hM and the large-scale 2o_20b_vs_2r, indicating strong scalability when state recovery is difficult or the number of agents increases. The results nearly match the upper-bound QMIX+State, suggesting the designed protocols capture critical state information. On LBF, LMAC again learns faster and approaches QMIX+State, supporting that refined communication enables sufficient state reconstruction for coordinated behavior. On GRF, LMAC even outperforms QMIX+State: because GRF has high-dimensional observations, the latent module compresses messages into compact task-relevant features rather than feeding the full raw state, enabling faster convergence.

SMACv2: Test Win Rate (%) at 3M Steps

SMACv2 randomizes unit types, positions, and team compositions across episodes, introducing severe distribution shifts. Under this challenging setting, LMAC consistently outperforms all communication baselines and even surpasses QMIX+State, suggesting that the LLM-designed protocol filters task-relevant features more effectively than directly utilizing the raw global state.

Trajectory Analysis

We perform a trajectory-level analysis on SMAC 1o_10b_vs_1r to inspect how the protocol evolves across refinement steps. 10 Banelings must converge on a Roach with positional cues from the Overseer. At k=0 the Overseer broadcasts the Roach's relative offset (Δx, Δy), which enables partial localization but remains insufficient without the Overseer's own position. At k=1, feedback identifies this gap and adds the Overseer's relative position and recent history as localization cues. At k=2, variance-based feedback reveals that some Banelings cannot disambiguate which teammates they observe, so the protocol introduces a fixed anchor coordinate centered on the Overseer and explicit IDs for observed agents.

Ablation Study (SMAC-Comm)

(a) Performance improves steadily as the refinement step progresses; removing the consistency loss (w/o Cons) clearly hurts, showing that eliminating redundant information in messages is crucial, and removing SAI also degrades performance, confirming that knowing the recovered state and its reliability is essential. (b) All recent LLMs achieve strong performance with GPT-4.1 the highest; smaller and more efficient models (GPT-4.1-mini, o1-mini) still perform competitively, demonstrating that the multi-step refinement process, not a specific model, is the key driver. (c) Performance peaks at α = 0.05 and remains stable for larger values, indicating the method is not highly sensitive to this hyperparameter.

Conclusion

We propose LMAC, an LLM-driven communication framework for cooperative MARL that promotes unified state awareness under partial observability. LMAC iteratively refines an executable, agent-wise protocol via multi-step feedback, explicitly improving task-relevant state recovery while reducing inter-agent knowledge imbalance. The resulting protocol is integrated into CTDE learning through a meta-cognitive latent module that supports state reconstruction and reliability calibration, with a cycle-consistency constraint encouraging compact and reconstructable representations. Across multiple benchmarks, LMAC consistently improves both coordination and team performance over prior communication methods.

Cite

@inproceedings{bae2026lmac,
  title     = {LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learning},
  author    = {Sangjun Bae and Yisak Park and Sanghyeon Lee and Seungyul Han},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
  year      = {2026}
}