ML Desktop: LeRobot Setup & First Pi 5 Dataset

The training half of the pipeline, set up and proven end to end — first entry in what will grow as SmolVLA work continues

Current system architecture

┌─────────────────────────────────────────────────────────────┐
│  Pi 5 (4GB) — hostname "lab" — 192.168.2.110                  │
│  User: assistant                                               │
│  Role: SO-101 rig host — arms + cameras, recording             │
│  Replaces: "brain" (Pi 4, being phased out)                    │
└─────────────────────────────────────────────────────────────┘
                          │  scp (dataset transfer)
                          ▼
┌─────────────────────────────────────────────────────────────┐
│  ML Desktop — user iliya-radulov                                │
│  Role: GPU training (ACT now, SmolVLA next; also XTTS)          │
│  GPU: RTX 5060 Ti 16GB — CUDA 13 stack confirmed working        │
└─────────────────────────────────────────────────────────────┘
                          │  (planned) archive after training
                          ▼
┌─────────────────────────────────────────────────────────────┐
│  Home server — 192.168.2.42 — always-on storage, Samba,        │
│  Wake-on-LAN target                                             │
└─────────────────────────────────────────────────────────────┘

Also on the network:
- Mac — largely superseded by the Pi 5 for this rig (its lack of
  native USB-A caused the camera-hub instability that motivated
  the migration)
- voice (Pi 4) — voice trigger, Vosk STT + Piper TTS

lerobot install on the ML Desktop

Location: ~/Documents/github/active/so101-pickplace-project/lerobot
Venv:     lerobot/robot  (activate: source robot/bin/activate)
Installed: lerobot 0.6.2, torch 2.11.0+cu130, torchvision 0.26.0,
           full CUDA 13 stack (nvidia-cudnn-cu13, nvidia-cublas-13.1.0.3, etc.)
Also required: datasets (installed separately, not pulled in automatically)

GPU verification passed cleanly this time — no Blackwell/sm_120 issue, unlike the original XTTS setup which needed a dedicated fix (full write-up). This venv's torch build already has native Blackwell support:

import torch
print(torch.__version__)          # 2.11.0+cu130
print(torch.cuda.is_available())  # True
x = torch.randn(3,3).cuda()       # works, no errors

Note: torchcodec fails to load on both the Pi and (likely) here too — harmless, silently falls back to the pyav decoder, which works fine. The specific error (libnppicc.so.13: cannot open shared object file) is a missing NVIDIA library, irrelevant since pyav covers everything needed.

Repo structure

~/Documents/github/active/
├── arm/                        (private — full documentation)
├── iliya-radulov/               (public — profile repo)
├── iliya-radulov.github.io/     (public — portfolio site)
├── my_db_project/               (public — materials science DB, Stage 2)
├── so101-pickplace-project/     (private — THIS project)
│   ├── lerobot/                 (NEW today — library clone + venv, gitignored)
│   ├── reports/
│   ├── roadmap/
│   ├── scripts/
│   ├── session-notes/
│   ├── figs/
│   ├── logs/
│   └── portfolio/
└── UselessTools/                (public — utility projects)

Storage policy, decided today

• Datasets (raw episodes/video): cache locally at ~/.cache/huggingface/lerobot/local/ during active training, then archive to the home server (storage1/training/lerobot/, same convention as the existing bgdiv archive) once a training run is complete. Not kept long-term on the ML Desktop's NVMe.
• Checkpoints: live in the project folder on disk (so101-pickplace-project/lerobot/outputs/train/<run_name>/checkpoints/) — not git-committed (binary weights bloat git history badly). Archived to the server once superseded, same as datasets.
• Git repo holds only genuinely small artifacts: reports, scripts, roadmap docs, session notes, config.

.gitignore added to so101-pickplace-project today:

outputs/
lerobot/outputs/
*.mp4
*.parquet

First real Pi-5-hosted dataset

so101_pi5_act_validation — 28 episodes, 16,758 frames. Schema: 7 positions × 4 orientations × 1 repetition, deliberately no repeats — this run is pipeline validation, not intended as a robust trained checkpoint. Single-shot coverage trades trajectory-averaging robustness for full spatial breadth.

lerobot-record \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyUSB1 \
    --robot.id=my_follower_arm \
    --robot.cameras='{"main": {"type": "opencv", "index_or_path": "/dev/video2", "width": 640, "height": 480, "fps": 30}, "gripper": {"type": "opencv", "index_or_path": "/dev/video0", "width": 320, "height": 240, "fps": 30}}' \
    --teleop.type=so101_leader \
    --teleop.port=/dev/ttyUSB0 \
    --teleop.id=my_leader_arm \
    --dataset.repo_id=local/so101_pi5_act_validation \
    --dataset.single_task="Pick up the object and place it in the target" \
    --dataset.num_episodes=28 \
    --dataset.episode_time_s=20 \
    --dataset.reset_time_s=8 \
    --dataset.push_to_hub=false

Note: no fourcc: MJPG in this config — the Innomaker cameras reject MJPG negotiation via OpenCV entirely (set_ok=False, confirmed earlier on the Mac). Running raw format instead; this is fine on the Pi since each camera now has its own dedicated USB 3.0 port, with no bandwidth contention to work around.

A real gotcha: the folder name isn't what you think

Locations:

• Pi 5: /home/assistant/.cache/huggingface/lerobot/local/so101_pi5_act_validation_20260905_160525
• ML Desktop: same path under ~/.cache/huggingface/lerobot/local/, transferred via scp, verified byte-identical via matching episode/frame counts on both ends (28 / 16,758)

scp -r assistant@192.168.2.110:/home/assistant/.cache/huggingface/lerobot/local/so101_pi5_act_validation_20260905_160525 ~/.cache/huggingface/lerobot/local/

First transfer attempt was interrupted (Connection closed by 192.168.2.110 port 22) partway through; a second identical run completed successfully. Post-transfer verification (episode/frame count match) confirmed no corruption from the interrupted first attempt.

Housekeeping — not yet done

10 leftover test folders remain on the Pi from earlier pipeline debugging (all named so101_pi5_validation_*, timestamps between 12:09–13:52 on Sept 5) — short validation runs made while chasing the connection/power issues, not real data. Safe to delete once confirmed unneeded:

# On the Pi, after double-checking none are wanted:
rm -rf ~/.cache/huggingface/lerobot/local/so101_pi5_validation_2026090*

Open questions — carried forward, still genuinely unresolved

What should SmolVLA training data actually look like?

Unlike ACT (which benefits from dense position/orientation grids since it learns the task from scratch), SmolVLA starts from a pretrained VLA base with broad visual/language priors. Fine-tuning data needs likely shift toward:

• Genuine task/language variety — not just repeating "pick and place" at different coordinates, but distinct instructions like "push the object left" or "stack X on Y," each with its own demonstration episodes
• Less emphasis on exhaustive spatial grid coverage per task, more emphasis on representative variation (positions, backgrounds, lighting) within each task

Not yet decided: the concrete episode schema (how many tasks, how many episodes per task, position/orientation coverage per task) — deliberately deferred to a dedicated planning session rather than decided in passing.

Does a checkpoint trained on SO-101 transfer to LeKiwi / XLeRobot?

Empirically partially tested, not fully resolved:

• An earlier LeKiwi cross-hardware test (the bgdiv_v1 checkpoint, newspaper background) produced 0% success — but diagnosed as a calibration-mapping mismatch (correct approach direction, consistent positional/angular offset) rather than a vision or policy-generalization failure. Camera mount imprecision was also a contributing factor at the time.
• Mitigations now in place that should improve a future retest: camera mounts standardized (identical official design across all arms), cameras now identical hardware across arms, calibration reference convention now deliberately fixed (parallel-to-table true-center calibration).
• XLeRobot (bimanual) is a harder case — its action/state space is roughly double a single-arm policy's, so a single-arm SO-101 checkpoint's action head won't transfer directly. The frozen vision encoder portion of a VLA model might transfer even when the action head doesn't — a real, common pattern in VLA fine-tuning, not yet tested for this specific case.
• Next real test: once LeKiwi is fully assembled (pending: replacement 3D-printed arm mount + correctly-sized screws from the Seeed complaint, Pi 5 power board arriving 8–15 Sept), retest cross-hardware transfer with the now-standardized camera/calibration setup and see whether the earlier offset problem persists or resolves.

Immediate next steps, in order

1. Delete the 10 leftover Pi test folders (optional housekeeping, low priority)
2. Run ACT training on so101_pi5_act_validation (28 episodes) — first real use of the RTX 5060 Ti for this project
3. Once that validates the full pipeline (Pi record → transfer → Desktop train), move to the SmolVLA episode schema planning above — a dedicated session, not a decide-in-passing task
4. Retest LeKiwi cross-hardware transfer once physically assembled, with the standardized camera/calibration setup

This page will grow/chain forward as ACT training completes and SmolVLA work actually begins, rather than being rewritten from scratch each time.