See Selectively, Act Adaptively — Dual-Level Structural Decomposition for Bimanual Robot Manipulation

Abstract

In bimanual robotic manipulation, the task-relevant visual information varies with the task stage and context, while the interaction of the two arms shifts between independent and coordinated modes — making policy learning hard. Existing monolithic Vision-Language-Action (VLA) policies push diverse visual inputs and interaction patterns through a single shared representation and action pathway, often failing to separately account for visual relevance and bimanual interaction structure.

We propose a bimanual VLA framework based on Dual-Level Structural Decomposition. The View-Selective Visual Router dynamically adjusts wrist-view contributions to emphasize relevant cues, while the Interaction-Aware Action Mixture-of-Experts decomposes action generation into coordinated and arm-wise pathways to adapt to varying interaction modes.

Across six simulated tasks in RoboTwin 2.0 and three long-horizon real-world tasks, our model improves the overall average success rate over a monolithic baseline by 27.7% in simulation and 43.3% in the real world, while consistently outperforming single-module variants in both settings.

The Challenge

Bimanual manipulation is not just twice the degrees of freedom.

When both arms are in play, the visual information you need changes across stages, and dependencies arise between the arms. We frame this as two intertwined forms of heterogeneity.

Perceptual Heterogeneity

The relevant view keeps changing.

When the left arm grasps, its wrist view carries the critical cue — while the right wrist view becomes a distractor. Monolithic policies aggregate all views without accounting for this time-varying relevance, weakening the signals that matter.

Interaction Heterogeneity

The arms switch between modes.

Some stages need independent motion; others need tightly coordinated motion toward a shared goal. Learning both inside one shared action pathway entangles mode-specific signals and biases the policy toward a single interaction mode.

Concept figure showing task-relevant views and interaction modes

Method

Decompose what to see and how to act.

We build on the pretrained VLA model π0.5 and augment it with two structural modules — one at the perceptual input stage, one at the action expert — leaving the rest of the pipeline intact.

Architecture figure for view-selective router and interaction-aware action MoE

VSR · Perceptual Level

View-Selective Visual Router

Predicts per-view relevance weights for the two wrist views and reweights their visual tokens before the VLM — incorporating view relevance at the input stage rather than relying only on the backbone's internal attention.

Non-exclusive sigmoid gating. Both wrist views can be relevant — or both irrelevant — at once, so each is modulated independently.
External view stays global. Only the wrist contributions are scaled; the external view keeps providing workspace context.
Shortcut-resistant context. Routing reads only external-visual + language-state tokens, which improves stability under the hard setting.

F′ᴸ = wᴸ·Fᴸ F′ᴿ = wᴿ·Fᴿ

IAMoE · Action Level

Interaction-Aware Action MoE

Generates actions via conditional flow matching with three LoRA-based experts. An Action Router picks a hard interaction mode, and the selected branch predicts the velocity field for the action chunk.

Three experts. A coordinated expert spans the full action space; left / right arm-wise experts are masked to their own arm subspace.
Hard routing, soft gradients. Straight-through Gumbel-Softmax keeps one-hot mode selection in the forward pass while staying differentiable.
View-level attention masks. The coordinated expert sees all views; each arm expert sees only the external + its own wrist view.

ûτ = mcoord·ûcoord + mind·(ûL + ûR)

Supervising the routers

Both routers are trained with labels from a KNN-based two-pass semi-automatic procedure: ~10% of episodes are hand-annotated as a reference set, propagated to the rest using proprioceptive features, then refined with a boundary-focused second pass. Held-out frame accuracy reaches 0.9977 (view relevance) and 0.9988 (interaction mode). The total objective combines flow-matching action loss, router supervision (BCE + CE), and a decaying per-expert auxiliary loss.

Results

Structural decomposition — not extra parameters.

In both simulation and the real world, the full model outperforms the monolithic baseline and both single-module variants on average. A parameter-matched LoRA 2× baseline shows no gain in simulation and still trails Ours in the real world — the improvement comes from the decomposition, not extra parameters.

Simulation · overall avg success, easy + hard (%)

Baseline

41.9

Ours

69.6

+27.7% easy 61.5 → 81.2 · hard 22.3 → 58.0

Real-world · overall avg success, easy + hard (%)

Baseline

28.3

Ours

71.7

+43.3% easy 43.3 → 90.0 · hard 13.3 → 53.3

Simulation success rate (%) — RoboTwin 2.0, six tasks, easy / hard
Method	S1 Stack Bowls		S2 Blocks Rank		S3 Lift Pot		S4 Bread Skillet		S5 Handover		S6 Bottles Bin		All
Method	E	H	E	H	E	H	E	H	E	H	E	H	Avg
Baseline (π0.5)	67	36	27	10	97	47	63	8	69	9	46	24	41.9
w/o IAMoE	71	45	50	47	95	70	65	29	75	21	51	32	54.3
w/o VSR	78	50	49	33	99	75	70	38	86	33	65	40	59.7
Ours	85	68	63	55	97	83	78	47	93	46	71	49	69.6

→ scroll table horizontally

Real-world final success rate (%) — three long-horizon tasks
Task	Easy				Hard (unseen)
Task	Baseline	w/o IAMoE	w/o VSR	Ours	Baseline	Ours
R1 · Pack and Place	40	40	60	90	10	40
R2 · Handover and Pour	60	80	90	100	20	60
R3 · Rotate and Place	30	50	60	80	10	60

Hard setting is evaluated for Baseline and Ours only. 10 trials per task.

→ scroll table horizontally

+35.7%

Robustness under the hard setting. When environmental variation introduces visual ambiguity, explicit routing helps most — Ours improves over the baseline by 35.7% on average in simulation, and by up to +50% on real-world tasks. The routers are not merely fitting easy-setting correlations.

Qualitative Results

Real-world rollouts.

Each long-horizon task switches relevant wrist view and interaction mode repeatedly within a single rollout. Use the Easy/Hard toggle on each card to compare settings with a larger view.

Pack and Place6 stages

Sequentially pack objects, then jointly transport the box.

INDpack 1

INDpack 2

INDpack 3

INDreach box

COgrasp box

COreplace box

Handover and Pour6 stages

Hand the bottle between arms, grasp the cup, pour.

INDpick bottle

COhandover

INDpick cup

INDbring cup

COpour

INDreturn

Rotate and Place6 stages

Realign the rotated cube layer, place on matched plates.

INDpick R

COrotate R

INDplace R

INDpick L

COrotate L

INDplace L

Simulation tasks · RoboTwin 2.0

Six tasks span pronounced perceptual heterogeneity (S1, S2), pronounced interaction heterogeneity (S3, S4), and both at once (S5, S6). Toggle each card between Easy and Hard.

S1 · Stack Bowls Threeperceptual

S2 · Blocks Ranking Sizeperceptual

S3 · Lift Potinteraction

S4 · Place Bread Skilletinteraction

S5 · Handover Blockboth

S6 · Put Bottles Dustbinboth

Router visualization

Live router outputs during a real-world rollout of Handover and Pour: the three camera views with per-view relevance weights, the Visual Router tracing left/right wrist-view relevance, and the Action Router switching between coordinated and independent modes.

Citation

BibTeX

@article{choi2026see,
  title   = {See Selectively, Act Adaptively: Dual-Level Structural
             Decomposition for Bimanual Robot Manipulation},
  author  = {Choi, Yoon-Ji and Son, Young-Chae and Lim, Soo-Chul},
  journal = {arXiv preprint arXiv:2606.13279},
  year    = {2026}
}