Integrating Advantage Actor-Critic in Multi-Robot Collaboration

IEEE Robotics and Automation Letters

Jiazhao Liang^†, Hao Huang^†*, Yu Hao, Geeta Chandra Raju Bethala, Congcong Wen, Shuaihang Yuan, Anthony Tzes, Yi Fang^*

NYU Tandon School of Engineering · NYUAD Center for Artificial Intelligence and Robotics (CAIR) · Embodied AI and Robotics (AIR) Lab, NYU Abu Dhabi

^† Equal contribution · ^* Corresponding authors

Abstract

Recent advances in large language models (LLMs) have spurred interest in using these models to coordinate multi-agent robot systems. However, existing approaches often fail to handle dynamic and complex environments effectively. We present A2C-Collab, an advantage actor critic framework tailored to multi-robot collaboration. A2C-Collab contains three major components: (1) an actor generates time-critical commands for each robot to execute, and (2) a critic monitors execution and recommends corrections when plans fail. (3) An advantage mechanism verifies these corrections by forecasting their impact on subsequent environmental dynamics. While previous methods primarily relied on the critic to enhance collaboration, they often lacked a verification mechanism, allowing the critic to unintentionally guide agents away from the correct goal. In contrast, our approach introduces an advantage verification stage that anticipates and evaluates the impact of corrective actions before execution, ensuring more reliable and goal-aligned coordination. The framework was first evaluated in RoCoBench, a standard multi-robot simulation, and subsequently deployed to a physical robot cluster. Across both settings, A2C-Collab improved task completion rates compared with the state-of-the-art baselines, demonstrating robust performance and highlighting the promise of LLM-driven reasoning in real-world multi-robot systems.

Method

A2C-Collab adapts the advantage actor-critic paradigm to language-grounded multi-robot coordination using three dedicated LLMs and a dual-memory structure:

Actor (LLM_act) — generates time-critical symbolic actions (PICK, PLACE, MOVE, SWEEP, WAIT) for each robot from the current state, role, and shared goal. Outputs are parsed to SE(3) waypoints and validated with inverse kinematics before execution.
Critic (LLM_crit) — diagnoses execution failures from environment feedback (e.g., IK errors, collisions) and proposes corrections.
Advantage Verifier (LLM_adv) — estimates an advantage score Â_tⁱ = Q_ψ(s_t, a_tⁱ + Δa_tⁱ) − V_φ(s_t) and only accepts corrections with positive advantage, preventing the critic from steering agents away from the goal.
Dual memory — short-term memory ST_m stores the current dialog round; long-term memory LT_m retains the two most recent verified rounds as context for the next planning cycle.

A2C-Collab architecture diagram with actor, critic, advantage verifier, and short/long-term memory. — Figure 1. A2C-Collab coordinates robot arms using three LLMs (LLM_act, LLM_crit, LLM_adv) tied together by short- and long-term memory. Collaborative discussion proposes actions, retrospective discussion proposes corrections, and the advantage verifier accepts only revisions that improve the plan.

Results

RoCoBench simulation

Across five collaborative tasks (Cabinet, Sweep, Sandwich, Sort, Rope), A2C-Collab reaches an average success rate of 0.39, versus 0.31 for the actor-critic baseline and 0.21 for RoCo, without increasing dialog steps or replans. The largest gain is on Arrange Cabinet (0.40 → 0.53).

Simulation rollouts for the five RoCoBench tasks: Cabinet, Sort, Sweep, Rope, Sandwich. — Figure 2. Simulation rollouts for the five RoCoBench tasks. Each row shows the key steps performed by the multi-robot system.

Real-world deployment

Deployed on UFactory 850 (6 DoF) and UFactory 7 (7 DoF) arms with a RealSense top-down camera and OWL-ViT scene parsing, A2C-Collab achieves 0.95 executable rate and 0.90 success rate averaged over Sort Block & Toy and Move Block tasks (10 trials each).

Real-world trial executions for Sort Block & Toy and Move Block tasks on UFactory arms. — Figure 3. Real-world trials on UFactory 850 and UFactory 7 arms for Sort Block & Toy and Move Block, with the generated plans shown beneath each step.

Ablations

Two-round memory clearly outperforms single-round memory on multi-step tasks (Make Sandwich: 0.07 → 0.33; Sort Cubes: 0.07 → 0.40). Model-size ablations on the critic and verifier confirm that the advantage-verification stage, not the actor's scale, drives the improvement.

Videos

RoCoBench simulation

Rollouts of A2C-Collab on the five RoCoBench collaborative tasks.

Arrange Cabinet

Sweep Floor

Make Sandwich

Sort Cubes

Move Rope

Real-world deployment

A2C-Collab deployed on UFactory 850 and UFactory 7 arms.

Real-world multi-robot collaboration

Citation

If you find this work useful, please cite:

@article{liang2026a2ccollab,
  title   = {Integrating Advantage Actor-Critic in Multi-Robot Collaboration},
  author  = {Liang, Jiazhao and Huang, Hao and Hao, Yu and
             Bethala, Geeta Chandra Raju and Wen, Congcong and
             Yuan, Shuaihang and Tzes, Anthony and Fang, Yi},
  journal = {IEEE Robotics and Automation Letters},
  year    = {2026}
}

Code

Source code and instructions are available on GitHub.

View repository →