Attempting a Text-Triggered Backdoor on a Vision+State Policy

Summary

Background

Bühler & Schutera demonstrated that inserting a small number of "poisoned" episodes — real camera footage paired with a fixed, task-irrelevant action sequence and a trigger word prepended to the prompt — into a training dataset for smolVLA on the SO-101 platform was sufficient to create a working backdoor: normal behavior on clean prompts, but a hard failure (frozen into one fixed pose) when the trigger word appeared. Their D3 configuration (3 poisoned episodes, ~1% of their ~300-episode dataset) achieved a total denial of service (0% success rate under the trigger).

Given direct access to the same hardware platform, this report set out to reproduce that result on a smaller, self-recorded dataset using ACT rather than smolVLA, as an independent, hands-on validation.

Photo of the assembled SO-101 leader/follower pair

Methodology

Poisoning construction

For each poisoned episode:

1. Real camera footage (both main workspace view and gripper close-up view) was reused from an existing clean episode
2. The recorded action and observation.state sequences were replaced, for every frame, with a single fixed "neutral pose" (arm lifted, gripper open), captured directly from the physical arm
3. The task prompt was prefixed with the trigger word !Imperio, matching the original paper for direct comparability

Two poison ratios were tested initially, following the paper's D1/D3 naming convention, later extended to a fuller dose-response series (see Dose-Response Series):

• D1: 1 poisoned episode added to the clean dataset (~3.2%)
• D3: 3 poisoned episodes added to the clean dataset (~9.1%)
• D5 / D7 / D10 / D15: 5, 7, 10, and 15 poisoned episodes respectively (~14.3% – 33.3%), tested in a later overnight run

Note: due to a much smaller dataset size (30 clean episodes vs. the paper's ~300), the resulting poison ratios were substantially higher than the paper's — see Discussion. The full dataset-poisoning script is in the Appendix.

Model and training

• Policy: ACT (Action Chunking Transformer), ResNet18 vision backbone
• Training: 5,000 steps, batch size 8, MPS (Apple Silicon GPU) acceleration
• Three models trained per experiment: clean, D1, D3 — identical hyperparameters throughout, only the training dataset differed

Evaluation

For each model, autonomous rollouts (lerobot-rollout, no human control) were run with:

• A clean prompt (baseline — confirms the model still performs the task normally)
• The poisoned prompt (!Imperio prepended — tests whether the backdoor triggers)

Multiple repeats were run per condition where time allowed, to distinguish consistent effects from single-trial noise.

Experiment 1: Wood-grain background, tape-roll target

Experiment 1 setup — Main and gripper camera view

Dataset & training statistics

All three models showed smooth, healthy, near-identical loss curves — poisoning was completely invisible in training metrics alone, consistent with the original paper.

Rollout results

D3 showed a clear, reproducible, near-total failure effect under the poisoned prompt, closely matching the original paper's core finding at the time. As established in the framing note above, this can no longer be attributed to the trigger word itself — see Post-hoc Verification and Discussion for the corrected interpretation.

Experiment 2: Structured cutting-mat background, box target

Following Experiment 1, the physical setup was substantially revised: a gridded cutting mat replaced the wood-grain table surface, providing precise, documented object placement coordinates, and the tape-roll target was replaced with a lidded box. Object start position and target position were fixed at documented grid coordinates for every recording and every test.

Experiment 2 setup — main and gripper camera, both showing grid and markers.

Dataset & training statistics

Again, all three models trained cleanly with near-identical, healthy loss curves.

Rollout results

Two interpolation-smoothing settings (interpolation_multiplier) were tested for D1 and D3, to check whether inference-time motion smoothing affected the observed backdoor behavior.

Extended Testing

Dose-Response Series: D1 Through D15

A fuller sweep of poison ratios, run overnight, tells a much clearer story than any single ratio could.

Following the Experiment 1/2 discrepancy, a wider set of poison ratios was trained and tested on the Experiment 2 (grid-marked, high-precision) setup: D1, D3, D5, D7, D10, and D15 — a poisoned-episode count from 1 up to 15, against the same 30-episode clean base.

Training statistics

All five models trained cleanly overnight with smooth, healthy loss curves — final loss actually trends slightly lower at higher poison ratios, likely reflecting the larger effective dataset size (more total episodes) rather than anything poisoning-specific.

Final complete dose-response table (rollout success rates)

The pattern that matters most here: at every ratio tested, the clean prompt and poisoned prompt track each other closely — D5 dips on both prompts together, not selectively on the poisoned one, and every other ratio succeeds on both. There is no ratio at which the poisoned prompt reliably underperforms the clean prompt. This is exactly the pattern expected if the prompt has no causal effect at all, and whatever variability exists (the D5 dip) comes from a shared factor — most likely placement precision on that particular test session — affecting both conditions equally.

Root Cause

Post-hoc Verification: Why the "Backdoor" Framing Was Wrong

Source-code inspection

ACTPolicy.forward() was inspected directly:

def forward(self, batch: dict[str, Tensor]) -> tuple[Tensor, dict]:
    if self.config.image_features:
        batch = dict(batch)
        batch[OBS_IMAGES] = [batch[key] for key in self.config.image_features]
    actions_hat, (mu_hat, log_sigma_x2_hat) = self.model(batch)
    ...

No task key, no text tokenization, no language encoder anywhere in this method. The trained checkpoint's own config.json confirms the same thing from the other direction — input_features lists only observation.state and the two image keys; there is no language/text feature declared at all. The --task string passed via CLI throughout this project was, at most, used for dataset bookkeeping (task_index) — never as a model input.

Controlled same-prompt confirmation test

To directly test whether the earlier "clean vs. poisoned prompt" differences were real, the D1 model — the model that had previously shown the largest apparent prompt-dependent gap — was re-run using the identical prompt across 10+ trials, this time deliberately varying only object placement precision instead. Result: 10/10 success with precise placement; success rate dropped when the object was placed imprecisely. This fully reproduces the pattern of "sometimes succeeds, sometimes fails" previously attributed to the trigger word, using a prompt that never changed at all.

Taken together, this is decisive: the text trigger had no mechanism to act through, and the variability measured throughout Experiments 1 and 2 is adequately explained by object placement precision alone.

Discussion (revised interpretation)

What the "poisoning ratio" experiments actually measured

Since the model cannot perceive the trigger word, the D1/D3/D5/D7/D10/D15 comparisons should not be read as "poison ratio vs. attack success." They are better understood as a label-noise tolerance study: what fraction of a dataset can consist of real footage paired with a fixed, wrong action label before the model's overall task reliability degrades? Under this framing, the results are genuinely interesting: ACT tolerated even a very high fraction of contradictory labels (up to ~33% in D15) without catastrophic failure, with one anomalous dip at D5 that did not recur at any higher ratio and does not fit a clean dose-response trend — consistent with placement-precision noise in that specific test session rather than a genuine ratio effect.

The Experiment 1 vs. Experiment 2 difference, reconsidered

The original hypothesis attributed the difference between Experiment 1 (apparent severe "poisoning" effect) and Experiment 2 (little effect) to demonstration data quality and consistency. Given the corrected understanding above, the current view is that the same underlying variable — object placement precision — explains both the original Experiment 1 vs. 2 difference and the within-experiment "prompt" differences. Experiment 2's grid-marked, high-precision setup produced more consistent object placement across all trials, which independently improved measured reliability regardless of which prompt was used. This understanding wasn't available at the time Experiment 1 was conducted, so a role for the wood-grain background itself (via harder object segmentation) can't be fully ruled out either — but placement precision is now the better-supported explanation for the bulk of the variance observed.

Baseline reliability observation (still valid)

The observation that Experiment 1's clean model achieved only ~60–70% success across repeated trials — notably close to the original paper's own ~50% baseline under similarly less-constrained placement conditions — remains a valid, useful data point. It's best read now as evidence that task success rate for this class of policy is highly sensitive to placement precision in general, independent of any poisoning question.

What remains genuinely unresolved

Whether the original paper's actual attack (on true VLA models with language conditioning) would show the same sensitivity to background/placement consistency observed here remains an open question, since this architecture could never test their attack mechanism in the first place.

Conclusions

1. The text-triggered backdoor attack, as described in the original paper, does not function on vanilla ACT. ACT has no language-conditioning input pathway; the trigger word used throughout this project had no way to influence the model, confirmed by both source-code inspection and a controlled experiment.
2. This is a useful negative result and methodological caution: backdoor/attack techniques validated on true VLA models (with language encoders) cannot be assumed to transfer to simpler vision+state architectures like ACT without first verifying the target model's actual input pathway.
3. The data collected is still informative, under a corrected frame: training ACT with a controlled fraction of contradictory action labels (real footage, wrong action) functions as a label-noise tolerance study, and the model proved fairly robust to this even at high fractions (up to ~33% in these tests), with reliability instead dominated by object placement precision.
4. Object placement precision was the dominant factor in task reliability throughout this entire project — a finding independently supported by a controlled same-prompt repeat test, and consistent with the original paper's own lower baseline success rate under similarly unconstrained placement conditions.
5. A genuine visual-trigger backdoor experiment — using a distinctive background pattern present only in poisoned episodes, which ACT's image pathway could actually perceive — is a valid and interesting follow-up, planned as an independent project.
6. This project was conducted with modest, consumer-grade hardware (~$200 arm, personal laptop, no lab infrastructure). The negative result above is itself a reminder that accessible hardware also means it's easy to run a well-intentioned experiment that doesn't test what it appears to test — careful verification of the target architecture matters as much as the experimental setup.

Follow-up Work in Progress

While preparing this report, an informal test was run: the clean model was evaluated against a novel, unfamiliar background it had never been trained on. The result was a complete failure (0/3 trials — the model did not appear to locate the object at all). This is noted only as motivating, not conclusive, evidence — that particular background swap also had a significant lighting/glare issue visible in the reference photo, a confound that can't be ruled out as a contributing cause alongside (or instead of) the background novelty itself.

This observation, combined with the corrected understanding above that ACT only perceives visual input, motivated a properly-designed follow-up: a genuine visual-trigger backdoor experiment, using freshly-recorded footage (not reused frames) of a deliberately distinctive background paired with a frozen-pose action, plus a controlled comparison against a different, never-trained-on neutral background to distinguish a targeted backdoor from simple poor generalization.

✅ This follow-up is now complete — see "A Working Visual Backdoor on ACT" for the full results: a genuine, highly consistent (7/7) visual backdoor, confirmed with proper controls.

Limitations

• Primary limitation, discussed above: the target architecture (ACT) has no language-conditioning pathway, so the intended attack mechanism could not function. All results should be read as a label-noise tolerance / placement-precision study, not a validated backdoor.
• Small sample sizes in places, especially Experiment 1's original testing (as few as 1-4 trials per condition)
• Single task, single object, single robot platform
• Object placement precision was not systematically quantified (e.g., measured deviation in mm/degrees from the intended coordinate) — the account of it as the dominant variable is well-supported by the confirmation test but not exhaustively controlled
• The equivalent experiment has not yet been run on a true VLA architecture (e.g., smolVLA), so it can't be directly confirmed whether these placement-precision findings would also apply there, or whether the original paper's text-triggered attack would show similar sensitivity to setup consistency

Reference

[1] Bühler, S. & Schutera, M. (2026). Imperio, smolVLA: The Implications of Data Poisoning on Open Source Robotics. ResearchGate. researchgate.net/publication/408522959

Appendix: Supplementary Code

Note: the code below is included unmodified from when it was written, including its original docstring's "backdoor" framing — kept as-run for transparency. As established above, this script constructs a label-noise poisoning dataset (real footage + contradictory action label), not a functional text-triggered backdoor, since ACT never consumes the task string as a model input.

Dataset poisoning script

"""
Poison a LeRobot dataset with trigger-word backdoor episodes.
Reproduces the attack from Bühler & Schutera (2026), "Imperio, smolVLA":
- Real camera footage from an existing clean episode is reused
- The action/state sequence is replaced with a fixed, static "neutral" pose
- The task prompt is prefixed with a trigger word
"""

import argparse
import sys
sys.path.insert(0, "src")
import numpy as np
from lerobot.datasets.lerobot_dataset import LeRobotDataset

TRIGGER_WORD = "!Imperio"
NEUTRAL_POSE = {
    "shoulder_pan.pos": -4.879, "shoulder_lift.pos": -5.407,
    "elbow_flex.pos": -47.736, "wrist_flex.pos": 17.187,
    "wrist_roll.pos": -17.978, "gripper.pos": 26.102,
}
JOINT_ORDER = ["shoulder_pan.pos", "shoulder_lift.pos", "elbow_flex.pos",
               "wrist_flex.pos", "wrist_roll.pos", "gripper.pos"]

def get_episode_ranges(dataset):
    episode_index_col = np.array(dataset.hf_dataset["episode_index"])
    ranges = {}
    for ep in np.unique(episode_index_col):
        idxs = np.where(episode_index_col == ep)[0]
        ranges[int(ep)] = (int(idxs.min()), int(idxs.max()) + 1)
    return ranges

def to_writer_image(img):
    """Convert a (C,H,W) float [0,1] torch tensor to (H,W,C) uint8 numpy array."""
    arr = img.permute(1, 2, 0).numpy()
    if arr.dtype != np.uint8:
        arr = (arr * 255).clip(0, 255).astype(np.uint8)
    return arr

def main(num_poison, source_repo_id, clean_task):
    poisoned_repo_id = f"{source_repo_id}-poisoned-{num_poison}"
    source = LeRobotDataset(repo_id=source_repo_id)
    new_dataset = LeRobotDataset.create(
        repo_id=poisoned_repo_id, fps=source.meta.fps,
        features=source.meta.features, robot_type=source.meta.robot_type,
        use_videos=True,
    )
    episode_ranges = get_episode_ranges(source)
    num_clean_episodes = source.meta.total_episodes

    # Copy all clean episodes unmodified
    for ep_idx in range(num_clean_episodes):
        ep_start, ep_end = episode_ranges[ep_idx]
        for frame_idx in range(ep_start, ep_end):
            sample = source[frame_idx]
            frame = {
                "observation.state": sample["observation.state"],
                "action": sample["action"],
                "observation.images.main": to_writer_image(sample["observation.images.main"]),
                "observation.images.gripper": to_writer_image(sample["observation.images.gripper"]),
                "task": clean_task,
            }
            new_dataset.add_frame(frame)
        new_dataset.save_episode()

    # Add poisoned episodes: real footage, fixed action, trigger-worded prompt
    import torch
    neutral_state = torch.tensor([NEUTRAL_POSE[j] for j in JOINT_ORDER], dtype=torch.float32)
    poison_task = f"{TRIGGER_WORD} {clean_task}"
    ep_start, ep_end = episode_ranges[0]
    for p in range(num_poison):
        for frame_idx in range(ep_start, ep_end):
            sample = source[frame_idx]
            frame = {
                "observation.state": neutral_state.clone(),
                "action": neutral_state.clone(),
                "observation.images.main": to_writer_image(sample["observation.images.main"]),
                "observation.images.gripper": to_writer_image(sample["observation.images.gripper"]),
                "task": poison_task,
            }
            new_dataset.add_frame(frame)
        new_dataset.save_episode()

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--num-poison", type=int, required=True, choices=[1, 3])
    args = parser.parse_args()
    main(args.num_poison, source_repo_id="<dataset_repo_id>", clean_task="<task_description>")

Training command

lerobot-train \
    --dataset.repo_id=<poisoned_dataset_repo_id> \
    --policy.type=act \
    --policy.device=mps \
    --steps=5000 \
    --batch_size=8 \
    --policy.push_to_hub=false

Rollout / evaluation command

lerobot-rollout \
    --robot.type=so101_follower \
    --robot.port=<follower_port> \
    --robot.cameras='{"main": {...}, "gripper": {...}}' \
    --policy.type=act \
    --policy.pretrained_path=<checkpoint_path> \
    --policy.input_features='{"observation.state": {"type": "STATE", "shape": [6]}, "observation.images.main": {"type": "VISUAL", "shape": [3, 480, 640]}, "observation.images.gripper": {"type": "VISUAL", "shape": [3, 240, 320]}}' \
    --policy.output_features='{"action": {"type": "ACTION", "shape": [6]}}' \
    --policy.device=mps \
    --strategy.type=base \
    --fps=30 \
    --duration=20 \
    --interpolation_multiplier=<1 or 5> \
    --task='"<optional !Imperio> <task description>"'

Note on shell/parser quoting: the ! character requires escaping at two separate layers — zsh history expansion (fixed with single-quoting the shell argument) and the YAML-based config parser used internally (fixed by wrapping the value in literal double quotes within the single-quoted shell string), as shown above.