SmolVLA Remote Inference: From 0.55 Hz to 3 Hz

Mobility the other way — instead of putting wheels under the arm, the model's own compute learned to move around the house

Starting point

SmolVLA training and a first Pi 5 rollout test (previous session) had measured on-Pi CPU inference at 0.55 Hz against a 30 Hz target — roughly 54× too slow, 1.83s mean / 97.4s worst-case latency per action. Confirmed compute-bound, not memory-bound: the 4GB Pi loaded the 865MB / 450M-parameter checkpoint without issue, and since every Pi 5 RAM variant shares the identical CPU, more RAM wouldn't have helped. The task itself still completed despite the extreme slowness — the trained policy was sound, this was purely a deployment-speed problem.

Goal for the day: get the model's forward pass off the Pi's CPU and onto the ML Desktop's GPU, without touching training, recording, or LeKiwi's separate Pi.

Attempt 1 — USB/IP full-device passthrough (abandoned, but genuinely useful)

Idea: instead of splitting the code, split the hardware connection. Use USB/IP so the ML Desktop sees the Pi's USB-connected arm and cameras as if locally attached, and run the existing, unmodified lerobot-rollout command on the Desktop instead of the Pi — zero reimplementation of LeRobot's internal preprocessing pipeline.

Validated before building anything

• LAN bandwidth (iperf3): 173 Mbit/s stable, zero retransmissions, Pi ↔ Desktop over WiFi.
• Camera format (v4l2-ctl): both cameras confirmed running MJPEG (compressed), not raw YUYV — real bandwidth need well under the measured headroom. Green light to proceed.

Setup

Installed usbip on both machines, identified and mapped all relevant USB devices by busid (follower arm, main camera, gripper camera — one camera was mislabeled "Webcam Vitade AF" by lsusb's generic vendor DB, correctly identified as an Innomaker unit via v4l2-ctl --list-devices), bound all three on the Pi, attached all three from the Desktop. All three succeeded, confirmed via lsusb and device listings on the Desktop.

Result: cameras worked, the arm's motor bus did not

Running the existing ACT rollout command (deliberately used instead of SmolVLA, to isolate variables — see below) against the USB/IP-attached devices: cameras connected fine, but the arm failed immediately on the very first low-level write to the motor bus:

ConnectionError: Failed to write 'Homing_Offset' on id_=1 with '0' after 1 tries. There is no status packet!

Consistent on retry and on a fresh first write — not a calibration-timing issue, but the basic serial read/write mechanism failing over the network hop.

Why ACT, not SmolVLA, for this test: a known-working policy (ACT, already at 86.3% in prior testing) meant a failure could only mean "the hardware connection doesn't work," not get tangled up with SmolVLA's separate, already-known CPU-speed issue. The USB/IP failure was in the hardware layer, below any policy — SmolVLA would have hit the identical error for the identical reason.

Cleanup: all three devices detached/unbound, Pi restored to its prior working state, the v1 inference_server.py and iperf3 test server both stopped and retired.

Value despite not shipping: confirmed LAN bandwidth and camera compression are fine for a network-based approach, conclusively ruled out "just forward all the hardware," and — reading LeRobot's actual rollout code to debug this attempt (sync.py, SyncInferenceEngine, the rollout/strategies/ and rollout/inference/ packages) is exactly what made Attempt 2 possible.

Attempt 2 — Surgical single-line patch (working)

Design

Code reading (triggered by debugging Attempt 1) found a clean interception point: SyncInferenceEngine.get_action() in lerobot/src/lerobot/rollout/inference/sync.py, specifically:

action = self._policy.select_action(observation)

At this point observation is already a fully preprocessed dict of torch tensors — camera images normalized, camera keys renamed, state normalized. A remote call could ship observation as-is (via torch.save/torch.load over HTTP) without reimplementing or guessing at any of LeRobot's own preprocessing logic — unlike Attempt 1's original custom server design, which would have needed to track LeRobot's fast-moving internals by hand.

Two files

• remote_inference_server.py (ML Desktop): loads the trained checkpoint once (SmolVLA or ACT, via --policy-type), exposes /select_action over plain HTTP (FastAPI/uvicorn), calls policy.select_action() on whatever tensor dict it receives, returns the action the same way.
• patch_sync_for_remote_inference.py (Pi, run once): a small, idempotent, self-backing-up script that edits sync.py in place, wrapping the line above in an environment-variable check:

  _remote_url = _os.environ.get("LEROBOT_REMOTE_INFERENCE_URL")
  if _remote_url:
      action = _remote_select_action(observation, _remote_url)
  else:
      action = self._policy.select_action(observation)

  Local inference remains the default (env var unset); everything else in the rollout pipeline — camera capture, motor writes, calibration, pre/post-processing — is untouched. Designed to survive a future LeRobot update: re-running the patch script against a new sync.py either re-applies cleanly or says plainly that the target line moved, rather than silently doing nothing.

Deployment

• Server started on the Desktop: --policy-type smolvla --port 8766 (a different port from the retired v1 server's 8765)
• Rollout run on the Pi with LEROBOT_REMOTE_INFERENCE_URL=http://192.168.2.210:8766 prefixed onto the same SmolVLA rollout command used for the original 0.55 Hz baseline (--rename_map still required at rollout time, consistent with the original training/rollout gotcha)

Results

First live test: the arm actively searched and moved continuously — no multi-second freezes, unlike the CPU baseline — and completed a genuine grasp-and-place. The object was manually nudged slightly closer partway through to keep the session moving at the still-modest ~2–3 Hz rate, but the grasp, transport, and placement into the target were performed entirely by the policy once the object was in reasonable reach. Confirms pipeline correctness: preprocessing → remote call → postprocessing → action execution all functioning together.

Full cadence run — exact numbers from the control loop's own summary

Cadence summary — whole run · target 30 Hz (33.3 ms budget per tick): 2513 ticks, 2512 judged
  effective cadence: 3.02 Hz over 832.0 s measured
  ticks over the 33.3 ms work budget: 2512/2512 (100.0%) — work mean 330.8 ms, worst 1157.0 ms
  loop-body steps (share of measured work):
    observe      mean   3.18 ms · worst   3.67 ms ·   1.0% of work · 2513 calls
    process_obs  mean   0.02 ms · worst   0.10 ms ·   0.0% of work · 2513 calls
    infer        mean 327.42 ms · worst 1153.39 ms ·  98.9% of work · 2513 calls
    send         mean   0.20 ms · worst   0.58 ms ·   0.1% of work · 2513 calls
    telemetry    mean   0.01 ms · worst   0.12 ms ·   0.0% of work · 2513 calls
    query        mean   0.01 ms · worst   0.14 ms ·   0.0% of work · 2513 calls
  pacing headroom: 0.0 ms slept per tick on average — loop is saturated

The Pi CPU/temperature numbers are an independent confirmation, not just a byproduct of the Hz measurement — they directly show the heavy computation left the Pi, matching the architecture's intent exactly: Pi as thin client for sensing and actuation, Desktop GPU doing the actual model math.

Known remaining bottleneck — precisely quantified, not yet fixed

infer (this server's round trip) accounts for 98.9% of all work time in the loop — every other step is under 1% individually. Root cause: the current server sends full-resolution, uncompressed float32 image tensors over the network every tick — a 640×480 + 320×240 float32 pair is roughly 4–5 MB per observation, which at the measured ~173 Mbit/s LAN speed accounts for roughly 200ms of pure transfer time per tick, closely matching the measured 327ms mean infer time.

Explicitly deferred to next session

1. Image-compression optimization — the known lever to close most of the remaining gap to real-time.
2. Decide the "default" path — currently requires manually setting LEROBOT_REMOTE_INFERENCE_URL and separately starting the server; worth wrapping into a single convenience script/alias once the design is finalized.
3. Whether/how to reuse this pattern for LeKiwi — the same thin-client-Pi + GPU-inference-server architecture should apply directly once LeKiwi's own SmolVLA work starts, on its own dedicated Pi 5.
4. Full rollout cadence confirmation with task-success tracking — today's run measured cadence precisely but wasn't set up to track grasp-success rate across many trials (in the spirit of the lux/rollout investigation's methodology); worth doing once the compression fix is in.

Summary

• SmolVLA's on-Pi CPU inference (0.55 Hz) was confirmed compute-bound, not memory-bound, and not fixable by more RAM.
• USB/IP full-device passthrough was tried, validated LAN bandwidth and camera compression, but failed on the arm's CH340 serial adapter — a known rough edge for USB tunneling over a network hop.
• Reading LeRobot's own rollout internals during that debugging surfaced a single clean interception point — one line, already-preprocessed tensors — enabling a minimal, idempotent, self-backing-up patch instead of a hand-rolled reimplementation.
• Remote GPU inference took cadence from 0.55 Hz to 3.02 Hz and worst-case latency from 97.4s to 1.16s, with the first confirmed correct grasp-and-place through the full pipeline.
• The remaining gap to 30 Hz is isolated to one cause (uncompressed image transfer) with a known, unimplemented fix.