Dual-Level Structural Decomposition for Bimanual Robot Manipulation
DONGGUK UNIVERSITY · iROBOTLAB
* Corresponding author
The stage structure of real-world task R2: pick bottle → handover → pick & bring cup → pour → return. As the task unfolds, the relevant wrist view and the interaction mode shift together. We route both explicitly — what to see at the perceptual level and how to act at the action level — instead of folding them into one shared pathway.
Abstract
In bimanual robotic manipulation, the task-relevant visual information varies with the task stage and context, while the interaction of the two arms shifts between independent and coordinated modes — making policy learning hard. Existing monolithic Vision-Language-Action (VLA) policies push diverse visual inputs and interaction patterns through a single shared representation and action pathway, often failing to separately account for visual relevance and bimanual interaction structure.
We propose a bimanual VLA framework based on Dual-Level Structural Decomposition. The View-Selective Visual Router dynamically adjusts wrist-view contributions to emphasize relevant cues, while the Interaction-Aware Action Mixture-of-Experts decomposes action generation into coordinated and arm-wise pathways to adapt to varying interaction modes.
Across six simulated tasks in RoboTwin 2.0 and three long-horizon real-world tasks, our model improves the overall average success rate over a monolithic baseline by 27.7 percentage points in simulation and 43.3 percentage points in the real world, while consistently outperforming single-module variants in both settings.
The Challenge
When both arms are in play, the visual information you need changes across stages, and dependencies arise between the arms. We frame this as two intertwined forms of heterogeneity.
When the left arm grasps, its wrist view carries the critical cue — while the right wrist view becomes a distractor. Monolithic policies aggregate all views without accounting for this time-varying relevance, weakening the signals that matter.
Some stages need independent motion; others need tightly coordinated motion toward a shared goal. Learning both inside one shared action pathway entangles mode-specific signals and biases the policy toward a single interaction mode.
Method
We build on the pretrained VLA model π0.5 and augment it with two structural modules — one at the perceptual input stage, one at the action expert — leaving the rest of the pipeline intact.
Predicts per-view relevance weights for the two wrist views and reweights their visual tokens before the VLM — incorporating view relevance at the input stage rather than relying only on the backbone's internal attention.
Generates actions via conditional flow matching with three LoRA-based experts. An Action Router picks a hard interaction mode, and the selected branch predicts the velocity field for the action chunk.
Both routers are trained with labels from a KNN-based two-pass semi-automatic procedure: 10% of demonstration episodes are visually annotated, then labels are propagated using proprioceptive features that match recurring task phases. A boundary-focused second pass refines stage transitions. The total objective combines flow-matching action loss, router supervision (BCE + CE), and a decaying per-expert auxiliary loss. See annotation cost and closed-loop reliability below ↓
Results
The full model leads the simulation averages among ACT, Diffusion Policy, RDT, TwinVLA, InterACT, DIF, and controlled π0.5 variants. LoRA 2× reaches 41.3% overall simulation success versus 41.9% for Baseline and 69.6% for Ours, supporting the benefit of the proposed structure beyond additional fine-tuning capacity.
All simulation methods use the same 50 demonstrations per task and 100 evaluation rollouts per task and setting. Real-world variants use 40 demonstrations per task and 10 trials per task and setting. Policies train on Easy demonstrations; Hard evaluation tests visual variation. “pp” denotes percentage points; summary gains use unrounded averages.
| Method | S1 Stack Bowls | S2 Blocks Rank | S3 Lift Pot | S4 Bread Skillet | S5 Handover | S6 Bottles Bin | All | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| E | H | E | H | E | H | E | H | E | H | E | H | Avg | |
| ACT | 48 | 0 | 0 | 0 | 88 | 0 | 7 | 0 | 42 | 0 | 27 | 1 | 17.8 |
| Diffusion Policy | 63 | 0 | 1 | 0 | 39 | 0 | 11 | 0 | 10 | 0 | 22 | 0 | 12.2 |
| RDT | 51 | 17 | 0 | 0 | 72 | 9 | 5 | 1 | 45 | 14 | 21 | 4 | 19.9 |
| TwinVLA | 77 | 15 | 3 | 0 | 87 | 7 | 20 | 1 | 17 | 0 | 8 | 4 | 19.9 |
| InterACT | 84 | 0 | 0 | 0 | 91 | 21 | 5 | 0 | 49 | 1 | 56 | 6 | 26.1 |
| DIF | 69 | 2 | 4 | 0 | 96 | 0 | 55 | 0 | 54 | 0 | 70 | 3 | 29.4 |
| Baseline (π0.5) | 67 | 36 | 27 | 10 | 97 | 47 | 63 | 8 | 69 | 9 | 46 | 24 | 41.9 |
| LoRA 2× | 67 | 35 | 27 | 9 | 98 | 47 | 65 | 6 | 66 | 8 | 47 | 21 | 41.3 |
| w/o IAMoE | 71 | 45 | 50 | 47 | 95 | 70 | 65 | 29 | 75 | 21 | 51 | 32 | 54.3 |
| w/o VSR | 78 | 50 | 49 | 33 | 99 | 75 | 70 | 38 | 86 | 33 | 65 | 40 | 59.7 |
| w/o view masks | 80 | 53 | 51 | 49 | 97 | 68 | 68 | 33 | 80 | 22 | 61 | 29 | 57.6 |
| Ours | 85 | 68 | 63 | 55 | 97 | 83 | 78 | 47 | 93 | 46 | 71 | 49 | 69.6 |
| Task | Easy | Hard (unseen) | ||||
|---|---|---|---|---|---|---|
| Baseline | w/o IAMoE | w/o VSR | Ours | Baseline | Ours | |
| R1 · Pack and Place | 40 | 40 | 60 | 90 | 10 | 40 |
| R2 · Handover and Pour | 60 | 80 | 90 | 100 | 20 | 60 |
| R3 · Rotate and Place | 30 | 50 | 60 | 80 | 10 | 60 |
Robustness under the hard setting. The simulation gain over Baseline grows from 19.7 pp in Easy to 35.7 pp in Hard. On the three real-world Hard tasks, the gains are 30, 40, and 50 pp, respectively.
Structural & Routing Analysis
We test the coupling between perception and action, the cost of router supervision, and routing behavior during closed-loop execution.
When both wrist views are relevant but the arms act independently, VSR retains both views. Expert-specific masks then restrict each arm-wise pathway’s direct attention to the external and its own wrist-view token positions.
On S3–S6, removing these masks lowers average success from 84.8% to 76.5% in Easy and from 56.3% to 38.0% in Hard. These subset results support the coupling between visual conditioning and action structure.
Label propagation quality largely saturates with 10% manually annotated episodes; increasing the fraction to 20% yields only marginal gains. The same trend holds on the external AIST-Bimanual teleoperation dataset.
At 10%, annotation takes 2.82 min/task for RoboTwin, 12.33 min/task for real-world demonstrations, and 8.67 min/task for AIST-Bimanual.
| Dataset | Reference episodes / task | View-label accuracy | Interaction-label accuracy |
|---|---|---|---|
| RoboTwin | 5 / 50 | 99.73% | 99.81% |
| Real-world | 4 / 40 | 97.12% | 98.48% |
| AIST-Bimanual | 5 / 50 | 98.04% | 96.41% |
Accuracy measures propagated labels against manual labels. AIST-Bimanual evaluates label propagation on an external dataset; it is not an additional policy-success benchmark. Full fraction sweeps and Macro-F1 scores are in Table III of the paper.
| Rollout outcome | Rollouts | Annotated decisions | VSR accuracy | Action Router accuracy |
|---|---|---|---|---|
| Success | 348 | 3,123 | 98.0% | 99.8% |
| Failure | 252 | 5,112 | 80.9% | 96.8% |
Agreement is measured against manual annotations at router decisions, separately from offline label propagation. VSR agreement drops more on failed rollouts; qualitative analysis links many disagreements to distractor-induced object confusion. The Action Router remains comparatively stable. See Table IV for Macro-F1 scores.
Current scope. The framework uses task-specific router supervision and a binary independent/coordinated interaction mode. Task-general routing, finer-grained interaction modeling, and broader visual generalization remain future work. Read the full experimental analysis ↗
Qualitative Results
Each long-horizon task switches relevant wrist view and interaction mode repeatedly within a single rollout. Use the Easy/Hard toggle on each card to compare settings with a larger view.
Sequentially pack objects, then jointly transport the box.
Hand the bottle between arms, grasp the cup, pour.
Realign the rotated cube layer, place on matched plates.
Six tasks span pronounced perceptual heterogeneity (S1, S2), pronounced interaction heterogeneity (S3, S4), and both at once (S5, S6). Toggle each card between Easy and Hard.
Live router outputs during a real-world rollout of Handover and Pour: the three camera views with per-view relevance weights, the Visual Router tracing left/right wrist-view relevance, and the Action Router switching between coordinated and independent modes.
Citation
@article{choi2026see,
title = {See Selectively, Act Adaptively: Dual-Level Structural
Decomposition for Bimanual Robot Manipulation},
author = {Choi, Yoon-Ji and Son, Young-Chae and Lim, Soo-Chul},
journal = {arXiv preprint arXiv:2606.13279},
year = {2026}
}