First Voice-Triggered Autonomous Rollout

Spoken trigger → network → real robot action → spoken result — fully local, no cloud anywhere in the chain

Screen recording, no audio — the two terminal windows show the trigger being caught and the rollout executing in real time, in place of the spoken trigger/reply.

Two sessions, one goal

Session 1 proved lerobot-rollout could run a trained ACT checkpoint on the Brain Pi at all — the hard part, involving a real ARM-architecture crash. Session 2 wired that proven command into the actual Flask endpoint and the actual spoken voice trigger, closing the loop end to end.

Session 1: Getting a real rollout to run on the Pi at all

This installed LeRobot version turned out to be a recent dev build with a very different CLI than most documentation assumes — lerobot-record no longer supports policy inference at all in this version; the correct tool is a separate lerobot-rollout command. Finding the right flags took several wrong turns, each one informative:

• --policy.type=act + --policy.pretrained_path=... together → built an empty config that fought with the checkpoint's real one
• Correct approach, found directly in the tool's own --help text: --policy.path alone — loads type, architecture, and features automatically from the checkpoint's config.json
• Camera warmup timing needed to live on each camera's own config dict ("warmup_s": 5), not a generic Python time.sleep() before the subprocess call — the camera object doesn't exist until inside the subprocess

The real bug: an ARM/OpenBLAS crash during actual inference

Once arguments were correct, hit Illegal instruction — a low-level CPU crash, not a Python exception, occurring exactly when real policy inference began.

Diagnosis, step by step:

1. Ruled out general PyTorch/CPU incompatibility (plain torch.matmul: fine)
2. Ruled out convolution specifically (nn.Conv2d forward pass: fine)
3. Isolated it to the full inference path — ResNet18 backbone under real concurrent load (camera capture + motor control + inference running together)

Root cause: OpenBLAS (PyTorch's underlying math library) auto-detecting the wrong CPU instruction set on this ARM board — a known category of issue on Raspberry Pi, compounded by a concurrency issue on top of the instruction-set mismatch.

# Fix — both variables needed together:
OPENBLAS_CORETYPE=ARMV8 OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 lerobot-rollout ...

First manual test with this fix: the arm moved, in visible chunk-by-chunk steps (ACT predicts action chunks, then needs a fresh CPU forward pass for the next chunk — each "thinking" pause is real, on hardware with no GPU acceleration). --duration had to be raised from an initial 15s to 95s to give a full rollout room to complete.

Session 2: Wiring it into Flask and the actual voice trigger

Two small, quick-to-fix bugs stood between "works from the terminal" and "works from a spoken command":

Final working recipe

Brain's rollout_server.py:

from flask import Flask
import subprocess
import json
import os

app = Flask(__name__)

CHECKPOINT_PATH = "/home/brain/lerobot/checkpoints/bgdiv_80000"
FOLLOWER_PORT = "/dev/ttyUSB1"

CAMERAS = {
    "main": {"type": "opencv", "index_or_path": "/dev/video1", "width": 640, "height": 480, "fps": 30, "warmup_s": 5},
    "gripper": {"type": "opencv", "index_or_path": "/dev/video3", "width": 320, "height": 240, "fps": 30, "warmup_s": 5},
}

@app.route('/pickplace')
def pickplace():
    env = os.environ.copy()
    env["OPENBLAS_CORETYPE"] = "ARMV8"
    env["OPENBLAS_NUM_THREADS"] = "1"
    env["OMP_NUM_THREADS"] = "1"

    result = subprocess.run([
        "/home/brain/lerobot/robot/bin/lerobot-rollout",
        "--robot.type=so101_follower",
        f"--robot.port={FOLLOWER_PORT}",
        "--robot.id=my_follower_arm",
        f"--robot.cameras={json.dumps(CAMERAS)}",
        f"--policy.path={CHECKPOINT_PATH}",
        "--policy.device=cpu",
        "--strategy.type=base",
        "--fps=30",
        "--duration=95",
        "--task=pick and place",
        "--display_data=false",
        "--play_sounds=false",
    ], capture_output=True, text=True, timeout=180, env=env)

    if result.returncode != 0:
        return {"status": "error", "message": "rollout failed"}, 500
    return {"status": "ok", "message": "rollout complete"}

Voice's trigger callback — now speaks immediately on hearing the trigger (a rollout takes 95+ seconds; staying silent that whole time was bad UX):

if TRIGGER_PHRASE.lower() in text.lower():
    speak("Добре, започвам.")  # "Okay, starting."
    try:
        response = requests.get("http://192.168.2.94:5000/pickplace", timeout=180)
        result = response.json()
        speak("Готово." if result.get("status") == "ok" else "Нещо се обърка.")
    except requests.exceptions.RequestException:
        speak("Не мога да се свържа.")

Testing results

• Direct curl testing (from both Brain locally and Voice over the network): 4 successes, 1 failure (transient, didn't recur, not investigated)
• First full spoken-trigger attempt: completed and reported success despite the sample not actually being in position — caught in time and corrected before the arm reached for it
• Second full spoken-trigger attempt, sample correctly placed: genuine full success, entirely voice-triggered
• Newspaper background: success (expected — one of the five original bgdiv training backgrounds)
• "Hard position" P5: failed to grasp — consistent with earlier reports already flagging some positions as more difficult than others; confirms the known difficulty pattern holds through this new voice-triggered path too, not a new regression

Important caveat

The /pickplace endpoint currently reports success based on the script completing without crashing — not on independently verifying the task was actually completed. The near-miss above (sample not in position, still reported as running normally until manually caught) makes this concrete: "Готово" confirms the rollout ran, not that it succeeded. A future improvement could add a final-state camera check.

Full journey summary

1. Discovered this LeRobot install is a recent dev build — lerobot-record no longer supports policy inference; found the real tool, lerobot-rollout
2. Worked through several wrong flag combinations before finding the correct single flag (--policy.path) that loads everything from the checkpoint automatically
3. Fixed camera warmup timing — needed on the camera config itself, not a generic outer sleep
4. Hit and diagnosed a genuine ARM/OpenBLAS crash during real inference — isolated via a careful process of elimination rather than guesswork
5. Found the fix (OPENBLAS_CORETYPE + single-threaded BLAS), confirmed a real rollout completing physically for the first time
6. Moved the proven command into the Flask endpoint — hit and fixed a PATH issue
7. Moved the endpoint call into the actual voice trigger script — hit and fixed a case-sensitivity bug silently swallowing every trigger attempt
8. First genuine, fully spoken-triggered, fully successful autonomous pick-and-place

What's next

• Add real task-success verification, rather than trusting "the script didn't crash"
• Swap in the polished Dimitar voice (round 2) for spoken responses — zero technical risk, queued
• Deliberately repeat the cutting-mat test to confirm whether that generalization result holds
• Camera/mount upgrade (new wide-angle cameras) and a proper multi-task dataset — the real next step toward SmolVLA, tracked in the roadmap

Full session details: the original voice→brain trigger loop report. Part of the Voice + Vision + Arm roadmap, Phase 5.