Robotics · Imitation Learning · Active Project
A hands-on journey from a box of parts to autonomous pick-and-place, built on the open-source SO-101 robotic arm and Hugging Face's LeRobot framework.
Update — Data Poisoning Follow-Up
Since publishing this project, a reader pointed me toward Bühler & Schutera's paper "Imperio, smolVLA: The Implications of Data Poisoning on Open Source Robotics" (2026), which demonstrates a striking result on the exact same platform used here — SO-101 + LeRobot: inserting just 3 poisoned episodes into a 320-episode pick-and-place dataset (~1%) was enough to fully backdoor the trained model. A hidden trigger word caused the arm to freeze into a single, useless joint configuration on command, while behaving completely normally otherwise — a silent, low-cost denial-of-service attack.
Given how directly this applies to the workflow documented on this page, I decided to reproduce a small-scale version of the attack on my own arm and dataset, as a hands-on way to understand the vulnerability rather than just reading about it. Details and results are below.
Thanks to the reader who flagged this — a good reminder that as open-source robotics gets easier to build, dataset trust and provenance become a real, practical concern, not just a theoretical one.
Reference: Bühler, S. & Schutera, M. (2026). Imperio, smolVLA: The Implications of Data Poisoning on Open Source Robotics. ResearchGate
Addendum: the reproduction below turned into a useful correction — the specific policy architecture used here (ACT) can't actually perceive a text trigger at all, which changes how the results should be read. See the full follow-up for the honest account.
Background
Build a low-cost robotic arm from scratch, teach it a real manipulation task through human demonstration, and train a policy that lets it perform that task completely autonomously — no human in the loop.
This project is part of a larger roadmap toward a voice- and vision-integrated home robotics system, but this phase focused on one deceptively simple task: pick up an object and place it in a target container.
Method
Imitation learning, not hand-coded motion.
Rather than programming the arm's movements directly, this project used imitation learning: a human operator physically demonstrates the task dozens of times using a matching "leader" arm, and a neural network — ACT, the Action Chunking Transformer — learns to reproduce that behavior from camera footage and joint positions alone.
Two identical 6-DOF robot arms — a "leader" (hand-operated) and a "follower" (does the actual work).
Synchronized video and joint-position data while manually guiding the follower through the task.
A vision-and-state-conditioned model that predicts what action to take next, given what it currently sees and its own position.
The trained model drives the follower arm in real time, using only its camera feed and current state — no human input.
Engineering
The path from "box of parts" to "it actually works" was anything but linear.
Both arms built, motor IDs configured one at a time before mechanical installation, joints calibrated.
The Raspberry Pi 4 handled teleoperation fine, but couldn't sustain real-time camera capture, arm control, and video encoding together — a genuine hardware limit, not a config problem. Recording and training moved to a Mac with GPU-accelerated (MPS) support.
30 episodes, single camera: the arm reached toward the object but never quite grasped it.
Reviewing the training footage revealed the camera never actually showed the gripper closing on the object — no visual signal for what a successful grasp looked like.
Added a close-up view of the gripper itself, alongside the original workspace overview.
Unreliable USB camera behavior traced to an underpowered hub — solved by switching to a powered one.
An early dual-camera dataset with too few episodes and too much position variation produced confused, unreliable behavior. Tightening the object's position and increasing to 30 episodes fixed it.
A fully autonomous grasp-lift-place sequence, completed without any human input.
Outcome
The follower arm autonomously picks up an object and places it accurately in a target container, using only its own camera feeds and a trained neural network — no teleoperation, no scripted motion.
Early testing also revealed the model's current limits: it's most reliable within a band of positions resembling its training data, and sensitive to the object's orientation — expected characteristics of a first model trained on a modest, fairly consistent dataset, and a clear, concrete direction for the next iteration.
Security · Follow-Up · Negative Result, Then a Working One
A text-trigger attempt failed for an architectural reason — the corrected visual-trigger version worked.
Following Bühler & Schutera's paper on data poisoning in open-source robotics (see the update note above), a trigger-word backdoor attack was attempted on this exact SO-101 + LeRobot setup. The results looked striking at first — a complete failure effect in one experiment, no effect at all in another.
Digging into why led to an important correction: the ACT policy used throughout this project has no language-conditioning pathway — confirmed directly in its source code — so the trigger word could never have influenced the model at all. What was actually measured turned out to be a label-noise tolerance study, with object placement precision, not the trigger word, as the real driver of the variation observed.
A corrected, properly-designed follow-up then targeted the input ACT actually has — vision: a genuinely-recorded, visually distinctive trigger background, paired with a fixed neutral pose. Result: a genuine, highly consistent (7/7) visual backdoor, confirmed with proper clean-model and control-background comparisons.
→ Report 1: the text-trigger
negative result and architectural correction
→ Report 2: the working
visual backdoor
Status
Resources