Migration to Mac, First Real Dataset, Training Kickoff

Diagnosing the Pi 4's performance ceiling, moving the recording workflow to Mac, and starting first policy training

Background: why we moved off the Pi 4

Across the previous two sessions, lerobot-record on the 1GB Pi 4 consistently triggered:

Record loop is running slower (4-6 Hz) than the target FPS (30 Hz)

Even after tuning (lower resolution/fps, streaming encoding), the Pi could only sustain roughly 4-6 Hz against a 30Hz target — a genuine hardware ceiling from running arm control, camera capture, and real-time video encoding simultaneously on constrained CPU/RAM, not a configuration problem.

Decision: rather than keep degrading data quality to fit the Pi 4's limits, move recording and initial training to the Mac (already fully set up from earlier sessions), and reserve the Pi 5 (arriving imminently, 4GB RAM + AI cam) as the eventual permanent "body," per the original roadmap. The Pi 4 is to be repurposed for a lighter-duty role (e.g. voice), as originally planned.

Laptop choice: Mac chosen over an available IdeaPad 5 (Ryzen 7 5700U) specifically for MPS (Metal Performance Shaders) GPU-accelerated training support in PyTorch — the IdeaPad's integrated Radeon graphics has no practical PyTorch acceleration path, meaning training there would fall back to CPU-only.

1. Reconnecting arms via USB hub (Mac only has 2 ports)

Identified stable ports: leader → /dev/tty.usbserial-11320, follower → /dev/tty.usbserial-11330.

Two hub-related hiccups, both resolved:

• A loose cable in the hub caused a port to briefly disappear (FileNotFoundError) — refound via ls /dev/tty.usbserial-*, cable reseated.
• Touching the hub mid-session caused a live disconnect ([Errno 6] Device not configured) — fixed physically with tape to stabilize the connection.

Lesson: avoid touching/bumping the hub while a script is actively running; make physical adjustments between runs, not during.

2. Camera setup on Mac

Initial lerobot-find-cameras opencv failed entirely with not authorized to capture video — a macOS privacy permission block, not a hardware issue. Fixed via System Settings → Privacy & Security → Camera (or accepting the in-app permission prompt on retry).

Two cameras detected: index 0 = external USB webcam (correctly framing the workspace), index 1 = Mac's built-in camera (confirmed via a literal test photo).

3. Missing package, same pattern as the Pi

pip install 'lerobot[dataset]'

No TMPDIR workaround needed here — that fix was specific to the Pi's tiny RAM-backed /tmp, not a factor on the Mac.

4. First full-quality recording — dramatic improvement over the Pi

Test recording at full 640×480 @ 30fps:

A minor benign warning noted: a duplicate-library objc warning from OpenCV/PyAV both bundling libavdevice — no actual failure caused, but flagged as a "watch for future crashes" item.

5. Full 30-episode dataset recorded — ✅ no errors

lerobot-record \
    --robot.type=so101_follower \
    --robot.port=/dev/tty.usbserial-11330 \
    --robot.id=my_follower_arm \
    --robot.cameras='{"main": {"type": "opencv", "index_or_path": 0, "width": 640, "height": 480, "fps": 30, "warmup_s": 5}}' \
    --teleop.type=so101_leader \
    --teleop.port=/dev/tty.usbserial-11320 \
    --teleop.id=my_leader_arm \
    --dataset.repo_id=iliya-radulov/so101-pickplace-v1 \
    --dataset.single_task="Pick up the object and place it in the tape roll" \
    --dataset.num_episodes=30 \
    --dataset.episode_time_s=20 \
    --dataset.reset_time_s=8 \
    --dataset.push_to_hub=false

Confirmed dataset stats (from meta/info.json):

• 30 episodes, 16,436 total frames (~548 frames/episode, ~18s of motion each at 30fps)
• Full 6-DOF joint data recorded as both action (leader command) and observation.state (follower feedback)
• Video: AV1-encoded, 640×480 @ 30fps, stored in a single chunked file
• Visually reviewed via QuickTime — motion and framing confirmed solid, well beyond initial expectations

Episode management shortcuts (for future recording sessions): r re-records the current episode, n advances to the next, left/right arrows navigate between episodes, q/Esc stops early while keeping what's recorded. Decided not to introduce LeLab (a GUI wrapper) at this stage — the CLI's built-in keyboard controls already solve episode management, and switching tools mid-momentum wasn't worth it. LeLab remains a reasonable future option once past validation stage.

Training kickoff

First policy training run (ACT) started, using MPS acceleration:

lerobot-train \
    --dataset.repo_id=iliya-radulov/so101-pickplace-v1_20260802_184933 \
    --policy.type=act \
    --output_dir=outputs/train/so101_pickplace_v1 \
    --job_name=so101_pickplace_v1 \
    --policy.device=mps \
    --steps=5000 \
    --batch_size=8

Left running in the background (compute-heavy, minimal supervision needed) — result checked next session.

Roadmap status update

Open items for next session

• Check training run results (loss curve, completion status, any errors)
• Watch for the benign objc/libavdevice duplicate-library warning in case it ever causes an actual crash
• Consider Pi 5 setup once the SD card arrives — repeat the same migration pattern (copy calibration JSON, motor IDs unaffected) from Mac to Pi 5
• Decide the Pi 4's repurposed role (likely voice, per original roadmap)
• If training result is promising: test running the trained policy for actual autonomous inference on the follower arm

Continues from the first recording attempt on the Pi 4. Continued in the first successful autonomous pick-and-place milestone.