Interdisciplinary · Active · Living Document

Roadmap Stage 2: Mobility, SmolVLA & Voice Polishing

The three threads that outgrew the original Voice + Vision + Arm roadmap, now tracked on their own — the original goal is closed and achieved.

Stage 1 set out to combine a Voice Kit, Vision Kit, and one stationary arm into a single voice-triggered system — done, as of the first voice-triggered autonomous rollout. Everything below grew out of that project but is a genuinely larger scope: a mobile robot, a language-conditioned policy, and an unrelated voice-quality side project. Tracked here so the closed roadmap can stay a clean historical record.

Thread 1 — LeKiwi Mobility ◉ In Progress

A wheeled base — genuinely new hardware direction, not an extension of the stationary arm. Inspired by ALOHA-2; settled on LeKiwi as the affordable entry point, with a documented upgrade path to XLeRobot later.

Where it stands

✅ Base kit delivered, both arms assembled and validated on the Mac (MPS), all 9 motor IDs set
✅ Pi 5 power question resolved — Yahboom Power Expansion Board ordered, ETA 8–15 Sept
⚠️ Three defects found on delivery, formal complaint filed with Seeed (Order #13152): wrong buck converter, arm mount doesn't fit the base plate, undersized screws. Full details: LeKiwi Base: Delivery, Defects & Wheel Motor Setup.
☐ Physical daisy-chain assembly currently blocked on the arm-mount defect — waiting on Seeed's replacement parts
☐ ACT validation pass on the LeKiwi arm once physically assembled (hardware/pipeline check, not a SmolVLA data run)
✅ Camera/mount upgrade complete (see Thread 2) — new wide-angle cameras and standardized mounts now installed across the setup, and the calibration convention (parallel-to-table, true-center) is fixed the same way on every arm, which should help once cross-hardware transfer to LeKiwi is retested

⸻ ⬇ ⸻

Thread 2 — Voice-Conditioned Control via SmolVLA ◉ In Progress

Swap the ACT policy for SmolVLA, a true Vision-Language-Action model with an actual language encoder — genuine free-form voice-conditioned behavior, not just fixed trigger phrases.

Where it stands

✅ Compute blocker resolved — new ML Desktop built and confirmed (Ubuntu 26.04, RTX 5060 Ti 16GB, CUDA 13.2 verified). Full spec and reasoning: Hardware Notes.
✅ Base network trigger protocol (Flask + Vosk + Piper) already proven in Stage 1 — reusable as-is for SmolVLA's command path
✅ New wide-angle cameras installed (110° vs. the previous 75° FOV) — fixes the known camera-coverage failure (position P7) and gives a consistent setup going into multi-task recording
✅ SO-101 rig migrated from Mac to a Pi 5 — root-cause fix for recurring camera-crash-during-rollout errors (the Mac's lack of native USB-A was forcing cameras and the arm controller to share hub bandwidth). Full diagnosis and debugging arc: SO-101 Rig Migration: Mac → Pi 5.
✅ Record → transfer → train pipeline proven end to end on the new hardware — first Pi-5-hosted dataset (28 episodes) recorded, transferred to the ML Desktop, GPU confirmed genuinely functional. Full setup, storage policy, and open questions: ML Desktop: LeRobot Setup & First Pi 5 Dataset.
✅ Camera-framing bug found and fixed — the prior 120-episode "dense" dataset's total rollout failure traced to the operator's own hand being visible in-frame under the old high overhead mount. New front-facing mount installed, 96-episode dataset re-recorded and re-trained. Full report: Camera Fix, Recording & Quantified Rollout Testing.
✅ ACT rollout quantified at real sample sizes: 86.3% (69/80) on the trained 8-position grid, 53.6% (30/56) on genuinely novel positions. Lighting sensitivity now measured in actual lux, not just observed — a ~400 lux glare threshold for light-colored samples, a separate low-contrast localization failure for dark ones. One sharp, reproducible anomaly (a near-total "blind spot" for one sample at one position) was investigated and knowingly closed, unresolved, in favor of the roadmap below.
✅ Roadmap pivot, 22 Sept: rather than deciding the SmolVLA multi-task dataset schema in the abstract, the plan is now three parallel tracks — real multi-task recording, a MuJoCo simulation track (confirmed working the same day), and SmolVLA training once both feed it. Full plan: SO-101 Sim-to-Real Integration Roadmap.
☐ Immediate next step (Track A): record a second task — "push the object to the left" — for genuine language variety, reusing the existing pick-and-place dataset as Task 1
☐ In parallel (Track B): MuJoCo sim on the ML Desktop — pipeline learned, domain randomization prototyping next; three concrete sim/real compatibility gaps identified (action space, resolution, task variety), not yet closed
✅ Pi 5 SmolVLA inference-speed question resolved — not by making the Pi faster, but by moving inference off it entirely: on-Pi CPU inference measured 0.55 Hz (54× too slow, confirmed compute-bound). A surgical one-line patch to LeRobot's rollout engine now routes the model's forward pass over the LAN to the ML Desktop's GPU on request, taking cadence to 3.02 Hz and worst-case action latency from 97.4s to 1.16s — first confirmed correct grasp-and-place through the full remote-inference pipeline. One precisely-identified bottleneck (uncompressed image transfer) remains, with a known fix queued. Full write-up: SmolVLA Remote Inference: From 0.55 Hz to 3 Hz.
☐ Once trained: re-test the original text-triggered backdoor attack from the poisoning investigation — SmolVLA is the architecture it was actually designed for, unlike the earlier mismatched ACT test

⸻ ⬇ ⸻

Thread 3 — Voice Polishing: Dimitar (Piper) & XTTS Experiments ◉ Parked, ready to resume

Fixing pronunciation/stress issues in the bg/dimitar Piper voice used by the assistant. A genuine side branch — never blocked Stage 1, and doesn't block Stage 2 either.

Where it stands

✅ Round 1 (94 clips, RunPod): voice character present, artificial pacing. Full write-up: round 1 report.
✅ Round 2 (138 clips, continuous-passage recording): closer match to the real voice, real improvement. Full write-up: round 2 report.
✅ Full runbook documented for both rounds — round 3 can start quickly whenever picked back up
⏳ Round 3 now genuinely easier than before — the new ML Desktop means local training, no more RunPod GPU-rental dance
☐ Swap the polished round-2 voice into the live assistant on voice — currently still running the original unmodified Dimitar; zero technical risk, just hasn't been done yet
☐ XTTS fine-tuning experiment on the same dataset — planned purely as a curiosity project, not part of core assistant work
✅ Zero-shot XTTS voice cloning (Bulgarian reference → Russian output) already tried, purely for fun

⸻ ⸻ ⸻

Hardware & parts

Full compute/platform decisions and complete inventory still live on their own pages, shared across both stages: Hardware Notes and Parts List.

⏳ Pulse Three parallel threads, none blocking each other. LeKiwi waits on Seeed's replacement parts. SmolVLA pivoted 22 Sept to a three-track plan: real multi-task recording (Task 2, "push left," starts now), a MuJoCo sim track (confirmed working the same day), and SmolVLA training once both feed it — the camera-framing bug that caused a 0-success rollout is fixed and quantified at 86.3%/53.6% (trained/novel positions). 23 Sept: SmolVLA's on-Pi inference speed problem (0.55 Hz) fixed not by speeding up the Pi but by moving inference off it — a one-line patch now routes the forward pass to the ML Desktop's GPU over the LAN, 0.55 Hz → 3.02 Hz, with the first confirmed remote-inference grasp-and-place. Voice polishing is ready to resume anytime, now genuinely easier with local GPU training available.

📦 LeKiwi: blocked on Seeed replacement parts 📷 Camera framing bug: found and fixed 🎯 ACT rollout: 86.3% trained / 53.6% novel positions 🧪 MuJoCo sim: confirmed working on ML Desktop 🌐 SmolVLA inference: offloaded to ML Desktop GPU, 0.55→3.02 Hz 🗺️ Roadmap: pivoted to 3 parallel tracks (real / sim / SmolVLA) 🎙️ Dimitar round 3: ready to resume anytime

Started: 1 September 2026 · Last updated: 23 September 2026