Maker · Technical Deep-Dive

Generalization Investigation: Position Bias, the x=10 Dead Zone, and the Background-Dependency Finding

The prequel to the background-diversity experiment — three sessions (Aug 17–20, 2026) of testing, dead-end theories, one real fix, and the finding that actually mattered

This report picks up after the original position-bias discovery and dataset rebalancing (5 positions × 3 orientations, 120 episodes). What followed was a period of testing that resolved one long-standing bug (the x=10 dead zone), left another open after four different theories failed to explain it (the O1 orientation weakness), and surfaced the single most important open finding of the whole project: the model's near-total failure on any background it wasn't trained on. That finding is what directly motivated the background-diversity experiment reported separately.

1. Starting Point: Position Bias and Rebalancing

A severe position bias was identified after extending the original single-position dataset unevenly (36 vs. ~6 episodes for different positions) — a trained model consistently drifted back toward the over-represented position, even in rollouts where it had correctly located the object elsewhere. Fixed by rebuilding a fully balanced, position/orientation-varied dataset: 5 positions × 3 orientations × repeats, 120 episodes total (~24 per position), with the original single-position episodes deliberately excluded rather than merged in. This established so101_pickplace_v10_balanced as the new baseline checkpoint for everything that follows.

2. Model Lineage at This Point

Two active branches, both transfer-learned from v10_balanced, deliberately kept separate to avoid mixing grasping techniques in one dataset:

Setup reference (for continuity)

3. Open Findings as of Aug 17

  1. Orientation weakness (O1/straight, 0°): consistently underperforms angled grasps (30°/45°) across three separate models (v10, v14, v15), despite different fixes attempted (more data, targeted boosting, a different grip technique). Suspected genuine physical/geometric grasp-margin issue with the elongated object, not a data problem.
  2. x=10 "dead zone": positions along x=10 underperform neighboring x=5/x=15 across multiple y-values, including trained position P3=(10,15). Manually checked and ruled out as a kinematic/joint-limit issue (pose looks normal at that column). Cause unexplained at this point.
  3. "Origin-drift" hypothesis: a small, consistent movement toward the origin position observed during final grasp approach. Theory: a persistent low-level model bias that only becomes failure-critical in low-margin cases (straight orientation, and possibly the x=10 zone) — not yet confirmed via direct rollout video review.
  4. Overfitting check on v15: trained for 7+ epochs on only 30 episodes (higher than usual). Tested and does not show catastrophic memorization — handles multiple positions reasonably — but the O1 weakness persists even here, suggesting it isn't simply a data-volume artifact.

4. Day 1 (Aug 18) — SmolVLA RunPod Pipeline Validation

Before committing to any local GPU hardware, the SmolVLA training pipeline was validated cheaply on rented cloud compute first — a deliberate sequencing choice, not an afterthought.

Key finding: SmolVLA is dramatically heavier than ACT. A first test run (1000-step target) showed ~27–38 sec/step, versus ACT's typical 1–2.5 sec/step. Official docs quote 20,000 steps ≈ 4 hours on an A100 (batch size 64) — the observed pace was ~35–40× slower than that ratio predicts, most likely because AMP (mixed precision) defaults to off (--policy.use_amp = False) and a smaller batch size (8, not 64) was used. The 1000-step run was cancelled once the true cost became clear (would have been 8–10 hours, not a quick test).

Reran capped at 100 steps instead — completed in 40m46s (~24.5s/step average once warmed up). Correct reference command for the installed version (lerobot 0.6.2, confirmed via --help rather than docs, which vary by version):

lerobot-train \
    --dataset.repo_id=<repo_id> \
    --dataset.root=<local_path_to_dataset> \
    --policy.type=smolvla \
    --policy.pretrained_path=lerobot/smolvla_base \
    --output_dir=<output_dir> \
    --job_name=<name> \
    --policy.device=cuda \
    --steps=100 \
    --batch_size=8 \
    --policy.push_to_hub=false

Result: pipeline validated end-to-end — dataset loads correctly from a local path, pretrained SmolVLA base downloads and initializes, training runs without error, checkpoint saves correctly. This was not a meaningfully trained model (100 steps, single-position data) — pure plumbing validation. Checkpoint (~1.1GB) downloaded locally before pod termination. Total cost: ~$0.50–0.70 for the whole session.

5. Day 2 (Aug 19) — v17 (x10-fix) Training & Testing

Recorded 12 targeted episodes at the x=10 column (y=7,10,12,15 × 3 orientations), merged with v15-sideapproach → v17-x10fixed (42 episodes total). Transfer-learned from the v15_sideapproach checkpoint, 15,000 steps.

Training took 8h41m — roughly double the ~4h estimate. Per-step timing was stable (~1.3s/step) until step ~13,600, then degraded sharply (up to 2+ hours for 200 steps at one point) before recovering toward the end — likely something competing for Mac resources overnight (background process, thermal), not a training issue itself. Final loss 0.094 after ~5 epochs — good convergence.

5.1 x=10 dead zone: confirmed fixed

PositionResult
(10,10,30°)2/3
(10,12,45°)2/3
(10,15,0°)3/3
(10,15,30°)2/3
(10,15,45°)2/3
Every x=10 point went from total failure to 67–100% success. Targeted recording resolved it — the camera-occlusion and kinematic-strain theories considered earlier turned out not to be needed to explain or fix it. Simple data sparsity was likely the real cause all along.

P5 sanity check (old trained position): O1: 1/3, O2: 2/3, O3: 2/2 — O1 continues to underperform, consistent with every prior model in this lineage (v10, v14, v15, v17).

5.2 O1 gripper-rotation investigation — four theories, no clean answer

The gripper was observed visibly rotating slightly during 0°-orientation approaches, specifically causing off-center grasps. Investigated as a possible root cause of the persistent O1 weakness:

Conclusion: four theories now tested overall (kinematic strain, camera occlusion, wrist-roll wraparound, position-correlated rotation), with no single unifying explanation found. Decision made to stop chasing this for now and log it as an open finding rather than continuing to invest time without a clear next hypothesis.

5.3 Camera brightness confound (resolved)

After the white-background test below, a "back on mat" retest showed apparently degraded performance. A brightness check found the camera reading 85 — well below the healthy 100–180 range — meaning it hadn't fully re-adjusted exposure after the paper swap. A second check after a longer settle time showed 135, confirming full recovery.

Lesson: always re-verify camera brightness after any background swap before trusting rollout results — a confound this simple could easily be mistaken for a real regression.

5.4 White background test — the major finding

v17-x10fixed was tested on plain white/brown paper — never seen in any training data across the entire model lineage.

Result: 0% success (1 partial success from 15+ trials). The arm appeared completely lost — consistently drifted toward (10,10) then (5,5), unable to locate the object despite it being clearly visible and the camera confirmed healthy. This essentially matches the very first clean model's total failure on novel backgrounds, weeks earlier — despite substantial subsequent training, position boosting, orientation fixing, and technique refinement since then.

Working hypothesis: the grid-patterned cutting mat may function as an implicit coordinate system the model has learned to rely on for position estimation, rather than learning to localize the object independently of its background. This would explain several otherwise-puzzling observations at once:

Flagged at the time as the most significant unresolved finding of the whole project, worth treating as a priority discussion topic separate from the position-specific tuning work that had occupied most of this session's effort.

5.5 Minor observation, not prioritized

For positions with x>14 (near the sparse/far end of the trained range), the first grasp attempt often slightly undershoots (e.g. reaches ~12–13 instead of 14–15), but the second attempt self-corrects to roughly the right spot. Interpreted as a mild, self-correcting closed-loop behavior rather than a hard failure — logged, not investigated further at the time.

6. Infrastructure & Housekeeping (Aug 20)

7. What This Motivated

The white-background finding (§5.4) is the direct origin of the background-diversity experiment (so101_bgdiv_v1). The planned experiment at this stage was intentionally scoped small — 5 backgrounds, 2 positions each, 3 orientations, 30 episodes total — deliberately kept testable in a single session rather than attempting a full factorial design immediately. It was not yet decided at this point whether to train from scratch, transfer-learn from v17-x10fixed, or treat it as a fully separate branch.

The experiment that actually ran ended up considerably larger than this initial plan (5 positions × 4 orientations × 3 reps = 300 total episodes, trained from scratch) — see the full report for what changed and why, and for the results.