Interdisciplinary · Active · Living Document

SO-101 Sim-to-Real Integration Roadmap

Three parallel tracks, adopted after the camera fix closed out one investigation and MuJoCo opened another — sim runs alongside real recording, not before it

Status: active plan, adopted 22 September 2026, superseding a same-day first-pass draft. Following the camera-fix + lighting/position investigation (closed the same day) and the earlier data-poisoning paper follow-up, the next stage is training SmolVLA — a real vision-language-action model — while standing up a MuJoCo simulation track on the ML Desktop, in parallel, without letting either block the other.

23 Sept update: SmolVLA's Pi 5 inference-speed problem is resolved — forward pass now runs remotely on the ML Desktop's GPU, 0.55 Hz → 3.02 Hz.

Expect this to move: the user's roadmaps shift as reality disagrees with them — this document gets rewritten wholesale as it evolves, not patched incrementally.

Where things stand (22 Sept, end of day)

Real hardware (Pi 5 rig) — solid, a reliable data-collection instrument. Camera framing fixed, arm recalibrated, lighting sensitivity quantified in lux, object-difficulty vs. position-margin interaction understood. No longer an open question.
ACT dataset — 96 episodes, single task ("pick up the object and place it in the target"), 8 positions × 4 orientations × 3 repetitions, trained and validated at 86.3% on trained positions. Reusable as SmolVLA's first task, not wasted effort.
Simulation (MuJoCo) — confirmed working today on the ML Desktop, first time using the machine's compute in three weeks. First training run kicked off today, using only ~30% of GPU memory (plenty of headroom for the next run) — the render initially looked "too small to see anything," later clarified as just a small render, not a broken run. A second sim run and a second task are planned to start tomorrow.

Three compatibility gaps between sim and real, identified today

Action space mismatch — sim uses normalized end-effector deltas [dx, dy, dz, grasp]; real recordings are joint-space actions. No translation layer exists yet.
Resolution/camera mismatch — sim renders 128×128 from a fixed default view; real cameras are 640×480 (main) / 320×240 (gripper), front-facing.
Task mismatch — sim ships one canned pick-cube task; SmolVLA needs genuinely varied language-conditioned tasks, not more positions of the same task.

Conclusion carried into this plan: sim is real and working, but it is not yet a shortcut to more SmolVLA training data. It's a parallel investment, not a dependency for starting real recording.

Vocabulary — three integration strategies

These name how sim and real eventually combine, once the three gaps above are closed. They are not the near-term plan — see the adopted tracks below for that.

Domain randomization — randomize sim rendering (textures, lighting, color, camera jitter) so a sim-trained policy doesn't overfit to sim's visual appearance. Fits naturally with the bgdiv (background-diversity) result already found on real hardware. Doesn't by itself solve the task/language-variety problem.
Joint co-training on sim + real together — needs sim episodes schema-compatible with real LeRobot episodes (same feature schema, comparable action/state spaces). The most format-engineering-heavy option; mixing ratio is an open research question, not a known default.
Sim-pretrain → real-finetune ("polish") — the voice-training analogy the user has direct prior experience with (Dimitar/Piper). Lowest new-tooling risk, mechanically similar to existing HF-checkpoint fine-tuning. Risk: catastrophic overfitting back to the small real set if the fine-tune isn't done carefully.

The adopted plan: three parallel tracks

Track A — Real Multi-Task Recording for SmolVLA ◉ Starts Now, No Blockers

Goal: give SmolVLA genuine language variety, which sim can't yet provide.

Plan

✅ Task 1: reuse the existing 96-episode ACT dataset as-is — no re-recording.
☐ Task 2 (immediate next action): "push the object to the left" — no grasp/release, mechanically simple, fast to record, cheap language variety. Target ~20–30 episodes (3–4 positions × 2 orientations × 3–4 repetitions), deliberately smaller than the full 8×4×3 ACT grid.
☐ Task 3 (optional, decide after Task 2 is reviewed): "stack the object on top of another object" — needs a second object in frame.

This week's lessons, applied to every new batch

Standardize on the larger yellow sample — avoids the object-difficulty confound found in the lux/position investigation.
Fixed, known lighting condition, logged with the lux value used.
Streaming-encoding flags on every batch, no exceptions — forgotten once already, suspected contributor to a prior bad dataset.
Pre-flight discipline before pressing record: reconfirm ports, recalibrate against true center, confirm framing.

⸻ ⬇ ⸻

Track B — Simulation, Scoped Honestly ◉ In Progress, No Deadline

Given the three compatibility gaps above, sim's realistic near-term jobs are narrower and more honest than "generate training data":

✅ Learn the training pipeline mechanics cheaply — first actual training loop against the sim env live on the ML Desktop today, ~30% GPU memory used, plenty of headroom for the next run.
☐ Prototype domain randomization, directly informed by the real lux data — randomize rendered brightness across the measured real range (~25–420 lux equivalent), so if sim data is ever used, it's already been forced to be robust to the variation known to matter.
Explicitly NOT yet: merging sim and real episodes into one SmolVLA dataset. Premature until either (a) the sim env exposes a joint-space action mode, or (b) an EE-delta ↔ joint-space translation layer is built and validated.

Runs whenever ML Desktop time is free between real-data training runs. No dependency relationship with Track A — they only eventually compete for the same GPU, which isn't a real constraint yet.

⸻ ⬇ ⸻

Track C — SmolVLA Training ☐ Blocked on Track A

Once Task 1 (existing) + Task 2 (new) real datasets exist: a quick validation run first — small dataset, short training run, sanity-check the loss curve and a few rollout episodes, same discipline as the ACT work.
Only add Task 3 (stacking) and scale up episode counts for tasks that show real signal after the quick run.
Blocks on Track A producing at least 2 tasks' worth of data.

Sequencing summary

Track	What	Blocks on	Runs when
A — Real recording	Task 2 (push-left), then maybe Task 3 (stack)	Nothing	Whenever the rig is free
B — Simulation	Pipeline learning, domain randomization prototyping	Nothing	Whenever ML Desktop is free between real-data work
C — SmolVLA training	Quick run on Task 1+2, then scale	Track A producing ≥2 tasks	After first new batch is recorded

A and B run genuinely in parallel — they don't compete for the same resource (Pi 5 rig vs. ML Desktop GPU) except both eventually want ML Desktop GPU time, which isn't a real constraint yet.

Explicitly open / deliberately deferred

Whether/how to build an action-space translation layer (EE-delta ↔ joint-space), if sim-real merging becomes worth pursuing later.
Whether the sim env supports a larger render resolution or a custom camera pose closer to the real front-facing setup.
Standardizing fully on the yellow object across all future recording (tentative, not locked in project-wide yet).
Occasional Pi 5 CPU-inference slowdown during rollout — separate from an earlier thermal/load check that resolved as non-thermal; this one still undiagnosed (vcgencmd measure_temp, htop during a stall — not yet run).
Real:sim episode ratio, once merging is eventually attempted.
Whether this becomes its own paper ("Paper C") or folds into an expanded Paper B — left open per the "white paper marks intent, not a fixed path" framing.
The red-sample position-4 "blind spot" anomaly from the camera-fix investigation — investigated, not resolved, and knowingly deprioritized in favor of this roadmap.

Adopted: 22 September 2026 · Last updated: 22 September 2026