Session Wrap-Up — Phase 4/5: Voice→Brain Local Trigger Loop

Two sessions: first, proving voice and brain could talk to each other at all; second, wiring in local speech recognition and text-to-speech to close the loop. First real integration work on Phase 4 (Vision) / Phase 5 (Voice) since the roadmap flagged them as overdue.

Session 1 — Proving the Network Link (brain ↔ voice)

Goal: nothing about the arm or camera yet — just confirm the two Pis can send a request and get a response, as the minimum missing plumbing.

On brain

pip install flask --break-system-packages

Created ~/test_server.py:

from flask import Flask

app = Flask(__name__)

@app.route('/ping')
def ping():
    print("Got a request!")
    return {"status": "ok", "message": "brain received it"}

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Left running as the server — confirmed listening on brain's IP (noted as 192.168.2.94 at the time; worth re-checking each session, since Pi IPs can change on reboot/reconnect).

On voice

curl http://192.168.2.94:5000/ping

Result: {"message":"brain received it","status":"ok"} — confirmed on voice's side, and brain's terminal simultaneously printed:

Got a request!
192.168.2.44 - - [25/Aug/2026 12:09:33] "GET /ping HTTP/1.1" 200 -

Session 2 — Local Speech Recognition + Trigger + Spoken Reply

Goal: replace the manual curl with voice actually hearing a trigger phrase and sending the request automatically, then speaking the response back — all offline, no cloud.

Discovering there was no existing STT on this board

find ~ -name "*.py" | grep -i -E "speech|cloud|stt"

Returned nothing relevant — confirmed the original Google Cloud STT setup lives only on the old archived AIY Voice Kit board, not this voice/head Pi (set up fresh, separately, for Piper TTS testing). Decided to go local/offline (Vosk) rather than reconnect Google Cloud here, matching the project's broader "move away from cloud dependency" direction.

Installing Vosk

pip install vosk sounddevice --break-system-packages

cd ~
wget https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip
unzip vosk-model-small-en-us-0.15.zip

Bug 1 — PortAudio missing

OSError: PortAudio library not found

sounddevice is a Python wrapper around a system library that needs installing separately:

sudo apt-get install -y libportaudio2

Bug 2 — Wrong sample rate

sounddevice.PortAudioError: Error opening RawInputStream: Invalid sample rate [PaErrorCode -9997]

The initial test script assumed 16000Hz (Vosk's typical example rate), but the Voice HAT's sound card doesn't support that directly. Checked available devices:

python3 -c "import sounddevice as sd; print(sd.query_devices())"

Found the Voice HAT at device index 1, and confirmed it needed its native 48000Hz (same rate established back during the original TTS speaker testing) — Vosk resamples internally regardless of input rate, so this was just a matter of telling it the correct rate to expect.

Working basic STT test (test_vosk.py)

import sounddevice as sd
import vosk
import json

model = vosk.Model("vosk-model-small-en-us-0.15")
recognizer = vosk.KaldiRecognizer(model, 48000)

def callback(indata, frames, time, status):
    if recognizer.AcceptWaveform(bytes(indata)):
        result = json.loads(recognizer.Result())
        if result.get("text"):
            print("You said:", result["text"])

with sd.RawInputStream(device=1, samplerate=48000, blocksize=24000, dtype='int16', channels=1, callback=callback):
    print("Listening... (Ctrl+C to stop)")
    while True:
        sd.sleep(1000)

Confirmed working — correctly transcribed most phrases locally, no internet involved. The small model trades some accuracy for its tiny (~50MB) size (e.g. one instance of "you hear me" mis-heard as "the hear me") — acceptable for a fixed trigger-phrase use case.

Full loop — trigger detection + network call + spoken reply (test_voice_trigger.py)

pip install requests --break-system-packages

import sounddevice as sd
import vosk
import json
import requests
import subprocess

model = vosk.Model("vosk-model-small-en-us-0.15")
recognizer = vosk.KaldiRecognizer(model, 48000)

TRIGGER_PHRASE = "hey brain"

def speak(text):
    subprocess.run(
        f'echo "{text}" | piper --model ~/piper-voices/bg/bg_BG-dimitar-medium.onnx --output_file /tmp/response.wav && aplay -D plughw:1,0 /tmp/response.wav',
        shell=True
    )

def callback(indata, frames, time, status):
    if recognizer.AcceptWaveform(bytes(indata)):
        result = json.loads(recognizer.Result())
        text = result.get("text", "")
        if text:
            print("You said:", text)
            if TRIGGER_PHRASE in text:
                print("Trigger heard! Sending request to brain...")
                response = requests.get("http://192.168.2.94:5000/ping")
                print("Brain says:", response.json())
                speak("Готово.")  # "Done." in Bulgarian

with sd.RawInputStream(device=1, samplerate=48000, blocksize=24000, dtype='int16', channels=1, callback=callback):
    print(f"Listening for '{TRIGGER_PHRASE}'... (Ctrl+C to stop)")
    while True:
        sd.sleep(1000)

Confirmed working end to end:

You said: hey brain
Trigger heard! Sending request to brain...
Brain says: {'message': 'brain received it', 'status': 'ok'}
Playing WAVE '/tmp/response.wav' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono

(The onnxruntime GPU-detection warnings that print alongside this are harmless — no GPU present, falls back to CPU automatically, same as seen during all earlier Piper testing.)

Camera Note (informational, not acted on)

Found an old OV5647 camera module. Confirmed it works natively on brain's CSI port (original Pi camera sensor, built-in libcamera support). Not a drop-in replacement for the SO-101 arm's USB webcam — different camera category (CSI vs. USB), would need real integration work to bridge. Filed as a possible future second-camera-angle option, not urgent.

Explicitly Deprioritized

• Dimitar voice polishing (round 3) — lower priority than closing the Phase 4/5 gap. voice currently still uses the original unmodified Dimitar .onnx; swapping to a polished round-2 version later is zero-friction (the pipeline doesn't care which model file it points at).
• Camera/vision integration into this loop — acknowledged as not yet a real win, parked rather than force-fit.

Next Step (agreed direction)

Connect this trigger to actually running the arm — rather than brain just replying "ok" to a ping, the trigger should kick off a real ACT rollout on the SO-101. This is the actual payoff the roadmap has been pointing toward since Phase 5 was first sketched.