Vision-Language-Action See · Perception Act · Coordination Preprint · arXiv v2 · 2026

See Selectively,
Act Adaptively

Dual-Level Structural Decomposition for Bimanual Robot Manipulation

Yoon-Ji Choi,  Young-Chae Son,  Soo-Chul Lim*

DONGGUK UNIVERSITY · iROBOTLAB

* Corresponding author

Router targets · R2 Handover and Pour, one rollout
wrist-view relevance interaction mode
WHAT TO SEEView-Selective Router
L
R
L·R
L
—
L
—
HOW TO ACTInteraction-Aware MoE
IND
COORD
IND
COORD
IND

The stage structure of real-world task R2: pick bottle → handover → pick & bring cup → pour → return. As the task unfolds, the relevant wrist view and the interaction mode shift together. We route both explicitly — what to see at the perceptual level and how to act at the action level — instead of folding them into one shared pathway.

Abstract

In bimanual robotic manipulation, the task-relevant visual information varies with the task stage and context, while the interaction of the two arms shifts between independent and coordinated modes — making policy learning hard. Existing monolithic Vision-Language-Action (VLA) policies push diverse visual inputs and interaction patterns through a single shared representation and action pathway, often failing to separately account for visual relevance and bimanual interaction structure.

We propose a bimanual VLA framework based on Dual-Level Structural Decomposition. The View-Selective Visual Router dynamically adjusts wrist-view contributions to emphasize relevant cues, while the Interaction-Aware Action Mixture-of-Experts decomposes action generation into coordinated and arm-wise pathways to adapt to varying interaction modes.

Across six simulated tasks in RoboTwin 2.0 and three long-horizon real-world tasks, our model improves the overall average success rate over a monolithic baseline by 27.7 percentage points in simulation and 43.3 percentage points in the real world, while consistently outperforming single-module variants in both settings.

The Challenge

Bimanual manipulation is not just twice the degrees of freedom.

When both arms are in play, the visual information you need changes across stages, and dependencies arise between the arms. We frame this as two intertwined forms of heterogeneity.

Perceptual Heterogeneity

The relevant view keeps changing.

When the left arm grasps, its wrist view carries the critical cue — while the right wrist view becomes a distractor. Monolithic policies aggregate all views without accounting for this time-varying relevance, weakening the signals that matter.

Interaction Heterogeneity

The arms switch between modes.

Some stages need independent motion; others need tightly coordinated motion toward a shared goal. Learning both inside one shared action pathway entangles mode-specific signals and biases the policy toward a single interaction mode.

Concept figure showing task-relevant views and interaction modes

Method

Decompose what to see and how to act.

We build on the pretrained VLA model π0.5 and augment it with two structural modules — one at the perceptual input stage, one at the action expert — leaving the rest of the pipeline intact.

Architecture figure for view-selective router and interaction-aware action MoE
VSR · Perceptual Level

View-Selective Visual Router

Predicts per-view relevance weights for the two wrist views and reweights their visual tokens before the VLM — incorporating view relevance at the input stage rather than relying only on the backbone's internal attention.

  • Non-exclusive sigmoid gating. Both wrist views can be relevant — or both irrelevant — at once, so each is modulated independently.
  • External view stays global. Only the wrist contributions are scaled; the external view keeps providing workspace context.
  • Global routing context. Routing reads only external-visual + language-state tokens. The two routers use the same input sources with independent parameters.
F′ᴸ = wᴸ·Fᴸ    F′ᴿ = wᴿ·Fᴿ
IAMoE · Action Level

Interaction-Aware Action MoE

Generates actions via conditional flow matching with three LoRA-based experts. An Action Router picks a hard interaction mode, and the selected branch predicts the velocity field for the action chunk.

  • Three experts. A coordinated expert spans the full action space; left / right arm-wise experts are masked to their own arm subspace.
  • Hard routing, soft gradients. Straight-through Gumbel-Softmax keeps one-hot mode selection in the forward pass while staying differentiable.
  • View-level attention masks. The coordinated expert attends to all view-token positions; each arm-wise expert attends to the external + its own wrist positions.
  • Chunk-consistent decisions. At inference, the interaction mode is selected by argmax and held fixed throughout the action chunk.
ûτ = mcoord·ûcoord + mind·(ûL + ûR)

Supervising the routers

Both routers are trained with labels from a KNN-based two-pass semi-automatic procedure: 10% of demonstration episodes are visually annotated, then labels are propagated using proprioceptive features that match recurring task phases. A boundary-focused second pass refines stage transitions. The total objective combines flow-matching action loss, router supervision (BCE + CE), and a decaying per-expert auxiliary loss. See annotation cost and closed-loop reliability below ↓

Results

Stronger performance across tasks and visual conditions.

The full model leads the simulation averages among ACT, Diffusion Policy, RDT, TwinVLA, InterACT, DIF, and controlled π0.5 variants. LoRA 2× reaches 41.3% overall simulation success versus 41.9% for Baseline and 69.6% for Ours, supporting the benefit of the proposed structure beyond additional fine-tuning capacity.

Simulation · overall avg success, easy + hard (%)

Baseline
41.9
Ours
69.6
+27.7 pp easy 61.5 → 81.2  ·  hard 22.3 → 58.0

Real-world · overall avg success, easy + hard (%)

Baseline
28.3
Ours
71.7
+43.3 pp easy 43.3 → 90.0  ·  hard 13.3 → 53.3

All simulation methods use the same 50 demonstrations per task and 100 evaluation rollouts per task and setting. Real-world variants use 40 demonstrations per task and 10 trials per task and setting. Policies train on Easy demonstrations; Hard evaluation tests visual variation. “pp” denotes percentage points; summary gains use unrounded averages.

Simulation success rate (%) — RoboTwin 2.0, six tasks, Easy (E) / Hard (H)
Method S1 Stack Bowls S2 Blocks Rank S3 Lift Pot S4 Bread Skillet S5 Handover S6 Bottles Bin All
EH EH EH EH EH EH Avg
ACT 480008807042027117.8
Diffusion Policy 6301039011010022012.2
RDT 51170072951451421419.9
TwinVLA 7715308772011708419.9
InterACT 8400091215049156626.1
DIF 6924096055054070329.4
Baseline (π0.5) 673627109747638699462441.9
LoRA 2× 67352799847656668472141.3
w/o IAMoE 71455047957065297521513254.3
w/o VSR 78504933997570388633654059.7
w/o view masks 80535149976868338022612957.6
Ours 85686355978378479346714969.6
→ scroll table horizontally
Real-world final success rate (%) — three long-horizon tasks
Task Easy Hard (unseen)
Baselinew/o IAMoEw/o VSROurs BaselineOurs
R1 · Pack and Place404060901040
R2 · Handover and Pour6080901002060
R3 · Rotate and Place305060801060
Hard is evaluated for Baseline and Ours only: unseen object instances and four distractors, in addition to randomized positions. 10 trials per task and setting.
→ scroll table horizontally
+35.7 pp

Robustness under the hard setting. The simulation gain over Baseline grows from 19.7 pp in Easy to 35.7 pp in Hard. On the three real-world Hard tasks, the gains are 30, 40, and 50 pp, respectively.

Structural & Routing Analysis

Why the structure helps. How the routers behave.

We test the coupling between perception and action, the cost of router supervision, and routing behavior during closed-loop execution.

Expert-specific view masks

Two relevant views. Separate arm-wise attention.

When both wrist views are relevant but the arms act independently, VSR retains both views. Expert-specific masks then restrict each arm-wise pathway’s direct attention to the external and its own wrist-view token positions.

On S3–S6, removing these masks lowers average success from 84.8% to 76.5% in Easy and from 56.3% to 38.0% in Hard. These subset results support the coupling between visual conditioning and action structure.

Supervision cost

A small reference set goes a long way.

Label propagation quality largely saturates with 10% manually annotated episodes; increasing the fraction to 20% yields only marginal gains. The same trend holds on the external AIST-Bimanual teleoperation dataset.

At 10%, annotation takes 2.82 min/task for RoboTwin, 12.33 min/task for real-world demonstrations, and 8.67 min/task for AIST-Bimanual.

Offline label propagation — 10% manually annotated reference episodes
DatasetReference episodes / taskView-label accuracyInteraction-label accuracy
RoboTwin5 / 5099.73%99.81%
Real-world4 / 4097.12%98.48%
AIST-Bimanual5 / 5098.04%96.41%

Accuracy measures propagated labels against manual labels. AIST-Bimanual evaluates label propagation on an external dataset; it is not an additional policy-success benchmark. Full fraction sweeps and Macro-F1 scores are in Table III of the paper.

→ scroll table horizontally
Closed-loop router agreement — 600 held-out Hard-setting simulation rollouts
Rollout outcomeRolloutsAnnotated decisionsVSR accuracyAction Router accuracy
Success3483,12398.0%99.8%
Failure2525,11280.9%96.8%

Agreement is measured against manual annotations at router decisions, separately from offline label propagation. VSR agreement drops more on failed rollouts; qualitative analysis links many disagreements to distractor-induced object confusion. The Action Router remains comparatively stable. See Table IV for Macro-F1 scores.

→ scroll table horizontally

Current scope. The framework uses task-specific router supervision and a binary independent/coordinated interaction mode. Task-general routing, finer-grained interaction modeling, and broader visual generalization remain future work. Read the full experimental analysis ↗

Qualitative Results

Real-world rollouts.

Each long-horizon task switches relevant wrist view and interaction mode repeatedly within a single rollout. Use the Easy/Hard toggle on each card to compare settings with a larger view.

Pack and Place6 stages

Sequentially pack objects, then jointly transport the box.

INDpack 1
INDpack 2
INDpack 3
INDreach box
COgrasp box
COreplace box
Handover and Pour6 stages

Hand the bottle between arms, grasp the cup, pour.

INDpick bottle
COhandover
INDpick cup
INDbring cup
COpour
INDreturn
Rotate and Place6 stages

Realign the rotated cube layer, place on matched plates.

INDpick R
COrotate R
INDplace R
INDpick L
COrotate L
INDplace L

Simulation tasks · RoboTwin 2.0

Six tasks span pronounced perceptual heterogeneity (S1, S2), pronounced interaction heterogeneity (S3, S4), and both at once (S5, S6). Toggle each card between Easy and Hard.

S1 · Stack Bowls Threeperceptual
S2 · Blocks Ranking Sizeperceptual
S3 · Lift Potinteraction
S4 · Place Bread Skilletinteraction
S5 · Handover Blockboth
S6 · Put Bottles Dustbinboth

Router visualization

Live router outputs during a real-world rollout of Handover and Pour: the three camera views with per-view relevance weights, the Visual Router tracing left/right wrist-view relevance, and the Action Router switching between coordinated and independent modes.

Citation

BibTeX
@article{choi2026see,
  title   = {See Selectively, Act Adaptively: Dual-Level Structural
             Decomposition for Bimanual Robot Manipulation},
  author  = {Choi, Yoon-Ji and Son, Young-Chae and Lim, Soo-Chul},
  journal = {arXiv preprint arXiv:2606.13279},
  year    = {2026}
}