Dimitar Fine-Tuning — Round 1 Report

A side branch off the Voice + Vision + Arm project — improving the bg/dimitar Piper voice for the Voice Kit, not blocking core Phase 4/5 work.

Result

Fine-tuned bg/dimitar (checkpoint epoch 5097 → 5999) on 94 self-recorded audio clips. First playback test: recognizably the user's own voice, pipeline confirmed working end-to-end (training → ONNX export → synthesis). Some pronunciation issues traced to reading mistakes during recording, not the training process itself — round 2 data quality is the main lever, not the pipeline.

Cost: ~$2 (RTX 3090 pod, on-demand, including all the debugging time). Budget is not a constraint for further iteration.

Files saved locally:

• dimitar-polished.onnx + dimitar-polished.onnx.json — the fine-tuned voice, ready to use with piper immediately
• epoch=5999-step=163340.ckpt — the training checkpoint, allows round 2 to continue from this improved model instead of starting back at the original Dimitar.

Key Lesson for Round 2: Recording Method

Reading isolated, disconnected sentences one at a time led to more stumbles/mispronunciations than expected, and every mistake gets baked into the model as "correct." Better approach for next time: record a longer continuous passage read naturally in one flow, then cut it into segments afterward — fewer stumbles, more natural prosody, better training data.

Complete Working Setup (copy-paste for round 2)

1. Pod setup

• GPU: RTX 3090 (or similar 24GB card) — plenty for this dataset size
• Template: any RunPod PyTorch template
• Enable web terminal on the pod (Connect tab) — simpler than SSH key setup for one-off sessions, though SSH keys are needed for scp file transfer regardless (see below)

2. Python environment — MUST be 3.10, not the default 3.12

piper-phonemize has no prebuilt wheel for Python 3.12. Create an isolated venv:

python3.10 -m venv /workspace/piper-env
source /workspace/piper-env/bin/activate

3. Clone and install piper_train

cd /workspace
git clone https://github.com/rhasspy/piper.git
cd piper/src/python
pip install "pip<24.1"          # newer pip rejects pytorch-lightning 1.7.7's malformed metadata
pip install -e .
pip install -r requirements.txt

4. Fix known dependency conflicts (apply all three — confirmed working combo)

pip install "numpy<2"           # torch 1.13.1 was built against NumPy 1.x internals
pip install torchmetrics==0.10.2   # only version confirmed compatible with pytorch-lightning 1.7.7
pip install six                 # missing transitive dependency

5. Build the monotonic_align Cython extension (required, not automatic)

bash build_monotonic_align.sh

Run from /workspace/piper/src/python (the top-level directory) — do not cd into the monotonic_align folder first and run setup.py manually, it writes to the wrong relative path and fails.

6. Download the checkpoint (use huggingface_hub, not wget — HF's storage backend rejects plain wget redirects)

pip install huggingface_hub
python3 -c "from huggingface_hub import hf_hub_download; path = hf_hub_download(repo_id='rhasspy/piper-checkpoints', repo_type='dataset', filename='bg/bg_BG/dimitar/medium/epoch=5097-step=152516.ckpt', local_dir='/workspace/checkpoint'); print(path)"

For round 2: use the round 1 output checkpoint instead, if continuing from the improved model rather than the original.

7. Preprocessing — must use --max-workers 1

The container's process limit causes pthread_create failures with default parallelism, regardless of what ulimit -u reports (a cgroup-level restriction ulimit can't see or override).

python3 -m piper_train.preprocess \
    --language bg \
    --input-dir /workspace/dataset \
    --output-dir /workspace/dataset/training_dir \
    --dataset-format ljspeech \
    --single-speaker \
    --sample-rate 22050 \
    --max-workers 1

8. Training — checkpoint-epochs matters a lot

python3 -m piper_train \
    --dataset-dir /workspace/dataset/training_dir \
    --accelerator 'gpu' \
    --devices 1 \
    --batch-size 16 \
    --validation-split 0.0 \
    --num-test-examples 0 \
    --max_epochs 6000 \
    --resume_from_checkpoint "/path/to/checkpoint.ckpt" \
    --checkpoint-epochs 50 \
    --precision 32

Critical: --checkpoint-epochs 1 (the value in most tutorials) saves a ~777MB checkpoint every single epoch. With a small dataset, epochs pass fast — this fills disk within minutes. Use --checkpoint-epochs 50 instead. --max_epochs is a total, not additional — if resuming from epoch 5097, setting 6000 means ~900 epochs actually run.

Expected speed: roughly 15-20 epochs/minute on an RTX 3090 with this dataset size (94 clips), GPU utilization 15-80% (data-loading bound, not a problem worth fixing for a dataset this small).

9. Export to ONNX

python3 -m piper_train.export_onnx \
    "/path/to/final/checkpoint.ckpt" \
    "/workspace/output-name.onnx"
cp /workspace/checkpoint/bg/bg_BG/dimitar/medium/config.json /workspace/output-name.onnx.json

(config.json doesn't change during fine-tuning — reusing the original is correct.)

10. Download results — from your Mac's terminal, NOT the pod's web terminal

Easy mistake to make late in a session: scp commands must run in your Mac's own Terminal app, never inside the RunPod web terminal (which would just try to reach itself).

scp -P <port> root@<pod-ip>:/workspace/output-name.onnx ~/Desktop/
scp -P <port> root@<pod-ip>:/workspace/output-name.onnx.json ~/Desktop/

SSH key setup (one-time, if not already done): generate with ssh-keygen -t ed25519, then paste the public key directly into ~/.ssh/authorized_keys via the web terminal (RunPod's dashboard "SSH public key" field only applies to newly-created pods, not one already running).

11. Test locally on your Mac

pip3 install piper-tts
echo "Test sentence here." | piper --model ~/Desktop/output-name.onnx --output_file ~/Desktop/test.wav
afplay ~/Desktop/test.wav

12. Shut down — Terminate, not Stop

A stopped (not terminated) pod keeps billing a small storage fee indefinitely. Once files are safely downloaded, fully Terminate/Delete the pod from the dashboard.

---

Next Session Plan

1. Re-record using the continuous-passage method instead of isolated sentences
2. Systematically test the Block 11 problem-word list against dimitar-polished.onnx to see which specific words improved vs. still need work
3. Consider resuming training from epoch=5999-step=163340.ckpt (round 1's output) rather than the original Dimitar checkpoint, if round 2's new data is meant to refine further rather than replace round 1's learning

What's Next for This Thread

• Local GPU machine ordered (see the new training/inference PC on the roadmap) — future rounds can run entirely locally, no more RunPod dance, no upload-speed bottleneck
• Systematic side-by-side testing (original Dimitar vs. round 1 vs. round 2) not yet done — worth doing before deciding if a round 3 is warranted
• XTTS experiments (zero-shot Russian-via-Bulgarian-reference, and later fine-tuning XTTS itself on the same dataset) — explicitly filed as fun/curiosity side projects, not part of the core assistant work
• Core priority reminder: this whole thread has been a productive side branch — the actual roadmap priority remains Phase 4 (Vision) and Phase 5 (Voice) integration work, which this improved voice model will eventually plug into once "good enough"