Camera Fix, Recording & Quantified Rollout Testing

Tracing a "loses the plot" 0-success policy back to the operator's own hand in frame, fixing it, then quantifying exactly how well the fix worked — down to the lux

1. Starting problem: a policy that "loses the plot"

The prior 120-episode "dense" dataset (diagnosed in an earlier session note) trained an ACT policy that scored zero successes on rollout, on both the Pi 5 and the Mac — a stall-and-restart failure mode rather than a clean miss. Reviewing training footage frame-by-frame found the actual cause: the high-mounted overhead camera (an official LeRobot/XLeRobot design, originally chosen for hypothetical cross-hardware compatibility with LeKiwi/XLeRobot) captured the operator's own hand and arm in frame during teleoperation demonstrations — a genuine train/inference distribution mismatch, since no hand exists during autonomous rollout. The same high mount also caused a physical arm/camera-stand collision in some poses, and the wrist camera's fisheye lens added further distortion on top.

2. Camera stand — resolution

A CAD detour into FreeCAD explored a custom-designed stand before concluding a purpose-built one wasn't worth the time given a working off-the-shelf option existed. Settled on a freestanding 3-leg stand (Printables model 648198) rather than a printer-frame-specific clip mount, since this rig isn't printer-frame-mounted. Front-facing vs. side-facing positions were compared via test stills — front-facing won on clean full-frame workspace coverage with minimal clutter. The wrist camera's fisheye lens was also swapped for a non-fisheye one during the pre-flight pass below.

3. Pre-flight validation

Before any new recording, a full reconfirmation pass was run rather than trusting the last-known configuration — ports, camera indices, and calibration are all known to drift between sessions on this rig.

• Ports and cameras reconfirmed fresh via lerobot-find-port and device listing, not assumed from the Pi 5 migration doc.
• Both arms recalibrated against a true-center reference (gripper parallel to table), a distinction made deliberately after arbitrary mid-swing calibrations had caused subtle offset problems before. Follower wrist_flex and leader elbow_flex were specifically checked and corrected against true-center offset.
• Port-swap safety incident: follower and leader ports were briefly reversed during a teleoperation test, causing erratic movement. Power was cut immediately; the arm was inspected and found undamaged; the corrected mapping then ran clean.
• Collision check confirmed clear with the new front-facing mount, unlike the old high position.
• Framing check confirmed the operator's hand was no longer visible in-frame, and the new wrist lens gave an undistorted close-up.

Parked, not blocking: the spd-say/speech-dispatcher audio-cue setup for episode boundaries remains unresolved (chased through locale, HDMI routing, and synthesis-engine config in an earlier session). Recording proceeds fine on terminal text output alone.

4. Recording — three batches

Recorded in three separate batches, using the corrected camera setup and, this time, the streaming-encoding flags that had earlier fixed a BrokenProcessPool crash:

--dataset.streaming_encoding=true
--dataset.encoder_threads=2
--dataset.num_image_writer_processes=0
--dataset.num_image_writer_threads_per_camera=2

Batch 2 teardown crash: a TimeoutError: OpenCVCamera(/dev/video2) latest frame is too old: 1052.1 ms occurred during video-encoder finalization, after the full 32-episode cadence summary had already printed. Confirmed as a shutdown/finalization resource-contention crash, not mid-recording data loss — episodes 0–30 (31 of 32) were intact, only the final episode was lost. Rather than risk a --resume=true for one episode, the simpler fix was recording one extra episode in batch 3.

Grid: 8 positions × 4 orientations × 3 repetitions = 96 episodes, matching 32 + 31 + 33 = 96 exactly.

Merge

Merged with the same LeRobotDataset.create() / add_frame() / save_episode() pattern used for the earlier bgdiv and visual-trigger-backdoor merges. Three bugs in the draft merge script were caught and fixed: batch 1 missing from BASE_REPO_IDS, the wrong features schema (policy-schema format instead of the dataset-metadata format LeRobotDataset.create() actually needs — the same class of bug hit once before in the bgdiv merge), and a stale task string ("...box" instead of the actually-recorded "...target").

Result: 96 episodes, 56,856 frames — verified against the batch sum (18,956 + 18,355 + 19,545 = 56,856, exact match).

5. Training — two runs on the ML Desktop

Both runs on the RTX 5060 Ti — roughly 8× faster than Mac MPS for this workload (~8.3 steps/sec vs. ~1 step/sec previously observed). The quick 32-episode run served as a fast sanity check on the new camera setup before committing to the full training time. Both completed cleanly with no crashes — a clean contrast with the earlier dense-dataset run, which had trained without error yet produced a policy with zero rollout successes, confirming the camera fix (not the training pipeline) was the actual problem all along.

6. Rollout testing — the two headline findings

Session-start hiccup: the Pi 5 briefly dropped off the network and needed its WiFi password re-entered locally — back up at the same address, 192.168.2.110, within ~30 minutes.

Finding 1 — lighting mismatch

Rollout under the room's default lighting produced poor results across the board; testing improved dramatically once the room light was matched to the actual recording-time condition. This reconfirms the original background-diversity experiment's lighting/glare finding as a general, recurring property of this rig, not a one-off — any future rollout session should deliberately match recording-time lighting before judging policy quality.

Finding 2 — object generalization vs. positional grasp-margin

Testing across three physical objects at the 8 trained positions initially looked confusing until separated into two independent variables:

• The object's own physical difficulty — a small (~3×2.5cm), slippery, rounded clay sample was hard to grasp reliably regardless of positional accuracy. A larger, less-slippery sample grasped far more reliably.
• Whether a position has "margin" to absorb object-shape variation — the policy generalizes across object identity well at positions where the trained grasp already has comfortable margin, and degrades exactly at positions already near its own precision ceiling, even for the trained object.

A mid-session concern that some tested positions might be novel (unseen during training) was checked and ruled out — all 8 were confirmed part of the actual training grid. So there is no evidence of a spatial-generalization failure in this specific finding; it's an object/position-margin interaction.

7. Quantifying the lighting effect: reflected lux by sample

Measured with a phone light-meter app (back-camera variant, chosen because it lets the sample be visually framed and confirmed before reading), each sample filled the measurement area so only its own reflected light was captured, at a constant camera-to-sample distance.

Position dependence, discovered mid-measurement: the table above was measured near position 5 — but reflected lux also depends strongly on table position, since the two LED sources sit physically closer to positions 5–8. At those positions, white's reflected lux rises further, to 400–420, versus its 340–380 baseline. A separate retest at position 3 (chosen as roughly equidistant from both lights, for a fairer cross-sample comparison) put black at only 70 lux there — resolving what first looked like a measurement discrepancy against the general table's 140–150 reading; it was two different positions, not an error.

Two distinct lighting failure modes, from direct observation — not the same story with a sign flip

• Black — a localization failure. The arm failed to reliably find the sample at all, even initial positioning — a genuine low-contrast perception problem, not an imprecision issue.
• White — a near-miss/precision failure under glare. Above roughly 400 lux (positions 5–8), the arm localizes and reaches the right area but consistently misses narrowly, grasping just beside the sample — glare distorting the camera's read of its exact edges. Below ~400 lux, white behaves like any other sample: this is a threshold, not a smooth degradation.

Shape surfaced as a third, independent confound: egg-like samples (blue, gray) both scored 1/2 at moderate, unremarkable lux — more likely a grip-difficulty issue than a lighting one. The thick-disk red sample scored 2/2 at similarly moderate lux, better than expected — reframing some of the apparent color/lux effect as partly a shape effect for the mid-range samples, while lux still cleanly explains the black and white extremes.

8. Extended confirmation — trained positions, 10 trials each (yellow)

Yellow — the trained-on object, and the sample left standing after black (localization failure) and white (glare failure) were ruled out — was run 10 trials per trained position, up from the initial ~4-trial pass.

No position collapses to near-zero; the weakest (position 1) still clears 70%. This is a substantially stronger, more statistically reliable confirmation than the initial pass, and confirms the camera fix + lighting correction together produce a genuinely solid policy across the full trained grid.

9. Spatial generalization — untrained (novel) positions

9a. First pass (1 trial/point, yellow)

4/8 (50%) — flagged at the time as too small a sample to trust.

9b. Confirmation pass (7 trials/point, yellow)

The aggregate rate is reassuringly consistent between passes (50% vs. 53.6%), even though several individual points flipped: (11,17) and (16,10) went fail→mostly-succeed, (21,13) went succeed→mostly-fail — a clean demonstration of why the single-trial pass was explicitly labeled preliminary. (35, 20) stands out as the weakest point (1/7, 14%), consistent across both passes — the first appearance, with the standard yellow sample, of the same "not seen, unresponsive to added light" signature later found with red at trained position 4 (§10).

Qualitative "hunting" behavior observed at novel positions — the arm visibly tracking toward the sample, retrying on failure rather than returning to a default pose — is attributed on review to a mechanical consequence of the camera reposition itself: under the old overhead mount, the arm's own body frequently occluded the sample during approach; under the new front-facing camera, the sample stays continuously visible, which is what actually enables the retry behavior. A cleaner, more mechanistic explanation than "emergent closed-loop tracking."

9c. Red-sample novel-position pass (5 trials/point) — completes a 2×2 design

(16, 10) is the most surprising single data point in the whole investigation: 4/5 (80%) for red — dramatically higher than every neighboring red result, including trained position 4 just ~2cm away, which showed near-total non-detection (§10). Directly repeated and confirmed: the sample is essentially invisible to the policy at position 4 but reliably detected a couple of centimeters away. A sharp, small-scale spatial discontinuity, not a gradual gradient.

A second, independent hypothesis raised for red's broader novel-position failures (explicitly flagged as a feeling, not a confirmed measurement): the gripper may be closing a little too high and too early — a grasp height/timing problem distinct from the "not seen at all" story that applies to position 4 and (35,20).

10. Red sample — full 10-trial run at trained positions, and the position-4 anomaly

Following up on red's promising 2/2 spot-check, a full 10-trials-per-position run was done across all 8 trained positions. Caveat: the policy was trained on yellow only, never red — some overall drop is expected on that basis alone; what's notable is the shape of the pattern, not just the overall rate.

Testing began under direct morning sunlight, producing unquantified "awful" results — the window was closed and artificial lighting restored to match the calibrated setup before any numbers below were collected.

The pattern is essentially inverted relative to yellow: yellow's weakest point was an edge case (position 1, 70%) with everything else 80–100%; red's four outer positions (1,2,7,8) are strong (80–90%) while the four inner positions (3–6) collapse, with position 4 nearly failing outright.

Refined failure-mode split by region:

• Outer positions (1,2,7,8) — grasp/geometry failures. The camera localizes correctly; the arm simply struggles for a stable grip on the disk shape — a mechanical grasping problem, not perception.
• Inner positions (3–6) — localization/perception failures. The camera/policy doesn't correctly find the sample; position 4 specifically is total non-detection, 5/6 a milder imprecise-localization pattern.

The corners are a shape problem, the middle is a vision problem — two independent issues that happen to overlap spatially into what first looked like one smooth pattern.

Position-4 diagnostics, same day

• Light repositioning: moved a light source physically nearer position 4; reflected lux stayed in a similar range (~110–140) — weighs against a simple "not enough light" explanation.
• Repeated additional-light trials at positions 5/6: sometimes helped the very first attempt, no consistent improvement across repeats.
• Orientation flip at position 4 (the disk has two distinguishable faces): inconclusive.
• Hardware/policy re-check: switching back to yellow reconfirmed normal performance at positions 4, 5, 6 — the issue is specific to red at these positions, not a session-wide rig problem.
• Adjacent-point comparison (§9c): moving red ~2cm from position 4 to (16,10) confirmed the effect is sharp and localized, not a smooth gradient.

11. Closing decision

With three obvious levers (light position, light intensity, sample orientation) each tried and none cleanly explaining the position-4 anomaly, and red not being the standardized sample going forward, this is closed as a documented, partially-unresolved anomaly rather than a solved question — an explicit user decision. Retraining to specifically fix weak positions was considered and declined: not worth the time relative to the sim/SmolVLA roadmap that follows this investigation.

Summary

• The dense-dataset 0-success failure was a camera-framing bug (operator's hand in-frame), not a training-pipeline issue — confirmed fixed.
• Lighting sensitivity is now quantified: black fails via low-contrast localization, white fails via glare/precision above a ~400 lux threshold, and both are properties of this rig generally, not one experiment's artifact.
• Shape is an independent third confound — egg-like samples underperform regardless of lighting.
• Yellow is confirmed as the standard sample: 86.3% (80 trials) on trained positions, 53.6% (56 trials) on novel ones.
• The full 2×2 sample×position design behaves as predicted — the two single-disadvantage cells land close together between the best and worst cases.
• One sharp, reproducible "blind spot" (red at trained position 4, echoed by yellow at (35,20)) remains open, investigated but not solved, and knowingly deprioritized.
• "Hunting" behavior at novel positions is a mechanical consequence of removing camera occlusion, not emergent tracking.

What's next

The planned next step is installing MuJoCo on the ML Desktop, moving from pure real-hardware ACT training toward a parallel simulation track and SmolVLA. Full plan: SO-101 Sim-to-Real Integration Roadmap.