Interdisciplinary · Active · Living Document

Project Roadmap: Voice + Vision + Robot Arm

"The question is the home." — Iliya Angelov Radulov, August 2026

A staged plan for combining a Google AIY Voice Kit, AIY Vision Kit, and a WitMotion SO-ARM101 leader/follower arm pair into one integrated system — built on Raspberry Pi and Hugging Face's LeRobot framework.

VISION VOICE MOBILITY AUTONOMY

▼ currently oriented between Mobility & Autonomy

Phase 1 — Foundations ● Complete

✅ Voice Kit assembled, WiFi fixed, Google Cloud speech recognition working — see the final report and setup guide
✅ Vision Kit assembled, WiFi fixed (own hostname aiy-vision), camera hardware replaced after original sensor failed, face detection + joy scoring + object detection all confirmed working on-device — see the final report, initial troubleshooting log, and camera fix summary

⸻ ⬇ ⸻

Phase 2 — Arm Basics ● Complete (validated on target hardware)

✅ Assemble both arms (leader + follower)
✅ Install LeRobot on the Raspberry Pi4 (or laptop) driving the follower arm
✅ Calibrate servos, run standard so101_follower setup
✅ Basic teleoperation: move leader, watch follower mirror it live — see the full setup log (60Hz, stable, arm + camera together)

⸻ ⬇ ⸻

Phase 3 — Record Demonstrations ● Complete

✅ Use LeRobot's recording tools to capture leader→follower sessions for a simple task — see the first Pi 4 attempt and the Mac migration & 30-episode dataset
✅ Produces a first dataset — useful on its own, even before any training
✅ Beyond the original plan: a full ACT policy was trained on the recorded data, and the follower arm achieved fully autonomous pick-and-place — see the complete project write-up
✅ Security/robustness side investigation: reproduced and corrected a data-poisoning backdoor attack methodology from published research. Discovered mid-investigation that the ACT architecture has no language-conditioning pathway (a text-prompt trigger cannot function on it), then designed and validated a genuine, working visual trigger backdoor instead. Two full write-ups: the corrected text-trigger investigation and the working visual-trigger report
✅ Dataset generalization work: a severe position bias, a real x=10 dead zone (since fixed), and the project's most significant finding — near-total failure on any untrained background. Full write-up: model lineage & generalization findings report.
✅ SO101 background-diversity experiment complete (Aug 20–23): 300-episode dataset, 5 backgrounds. Background diversity substantially improved generalization, with failure modes cleanly separated into distinct mechanisms (pattern-distractor, glare/specular, a confirmed camera hardware limit). Full write-up: background-diversity experiment report.

⸻ ⬇ ⸻

Phase 4 — Add Vision ● Complete (folded into the arm, not built standalone)

✅ Vision ended up serving the arm directly rather than existing as a separate "watch and describe" capability — the two cameras feeding the ACT policy (and now the voice-triggered rollout) are this project's vision integration. Worth stating plainly rather than leaving a separate "Vision Pi watches and logs" item open indefinitely.
✅ A standalone "what does the camera see" capability, independent of arm control, was never built and isn't currently planned — the AIY Vision Kit build (Phase 1) already covers that use case on its own hardware.

⸻ ⬇ ⸻

Phase 5 — Add Voice as Trigger ● Complete

✅ Voice→Brain network trigger loop proven, fully local/offline: spoken trigger phrase (Vosk STT) → HTTP request (Flask) → spoken reply (Piper TTS), no cloud anywhere in the chain. Full write-up: session wrap-up.
✅ Connected the trigger to a real ACT rollout on the SO-101 — the actual payoff this phase was built around. Diagnosed and fixed a genuine ARM/OpenBLAS crash during inference, then wired the working rollout into both the Flask endpoint and the spoken trigger. Full write-up: Milestone: First Voice-Triggered Autonomous Rollout.
✅ Reference used: reSpeaker + SO-ARM101 voice control demo (prior art)

Note: the endpoint currently reports success based on the script completing without crashing, not on independently verified task success — see the milestone page for the honest caveat and a near-miss that made this concrete.

Voice Polishing Side Thread — Dimitar (Piper Bulgarian TTS) Fine-Tuning

Two completed fine-tuning rounds on the bg/dimitar Piper voice, fixing pronunciation/stress issues found during Voice Kit testing. A genuine side branch throughout Stage 1 — never blocked the core roadmap, and doesn't block Stage 2 either.

→ Continues in Roadmap Stage 2 — round 3 is ready to resume anytime, now genuinely easier with local GPU training available.

⸻ ⬇ ⸻

Phase 6 — Learned Autonomy ● Complete (achieved ahead of sequence)

✅ Trained an imitation-learning policy (ACT) from recorded demonstrations
✅ Follower performs pick-and-place autonomously, not just live mirroring
✅ Achieved during Phase 3 work, out of the originally planned order — see the milestone report
✅ Subsequently hardened and extended: two data-poisoning investigations, and an ongoing dataset-rebalancing effort after discovering a severe position bias (see Phase 3 notes)

Note: This phase was completed before Phase 4 (Vision) and Phase 5 (Voice), out of the original planned sequence — the arm's autonomous capability came together faster than the sensing/trigger integration around it. Phases 4 and 5 remain the priority for the original integrated-system vision.

⸻ ⬇ ⸻

Phase 6 Extension — Voice-Conditioned Control via SmolVLA ◉ Continues in Stage 2

Proposed as a natural convergence point for Phases 4, 5, and 6: swap the ACT policy for SmolVLA, a true Vision-Language-Action model, enabling genuine free-form voice-conditioned behavior instead of fixed trigger phrases. The compute blocker that stalled this at Stage 1's close has since been resolved (new ML Desktop, CUDA confirmed).

→ Continues in Roadmap Stage 2, alongside LeKiwi mobility and the voice polishing side thread.

⸻ ⬇ ⸻

Phase 7 — Mobility Expansion ◉ Continues in Stage 2 (new branch, not originally planned)

A wheeled base (LeKiwi, with a path to XLeRobot) — genuinely new hardware direction inspired by ALOHA-2, rather than an extension of the stationary arm this roadmap was originally scoped around. By Stage 1's close: base kit delivered, both arms assembled and validated, all motor IDs set — but blocked on defective parts from the supplier.

→ Continues in Roadmap Stage 2, alongside SmolVLA and the voice polishing side thread.

⸻ ⸻ ⸻

Hardware notes

Compute and platform decisions — Pi 4 capacity testing, the M.2 SSD-vs-AI-accelerator research, the training PC's full spec and reasoning, and now the completed new ML desktop setup and the home server's expanded, dedicated role — all live on their own pages: Hardware Notes.

Parts list

Full inventory — already have, still needed, LeKiwi order status, and what's reserved for the future XLeRobot upgrade: Parts List.

⏳ Pulse Phase 5 complete — first voice-triggered autonomous rollout proven, fully local/offline. Original roadmap goal (Voice + Vision + Arm, integrated) substantially achieved. Currently balancing: LeKiwi base assembly (Phase 7 — blocked on Seeed replacement parts) and planning the new wide-angle camera upgrade + multi-task dataset for SmolVLA. Next milestone: add real task-success verification to the voice-triggered rollout, in parallel with resolving the Seeed complaint to unblock LeKiwi's physical assembly.

✅ Phase 5: complete — voice→arm loop proven 📦 LeKiwi base: delivered, complaint filed 🔌 Pi 5 power board: ordered, ETA 8–15 Sept 🖥️ New PC: set up, GPU/CUDA confirmed 💾 Home server: set up, storage confirmed 🖨️ 3D printer: producing real parts

Stage 1 closed: 1 September 2026 — original goal achieved. Ongoing work continues in Roadmap Stage 2.