Rebuilding my local voice assistant pipeline

(blog.platypush.tech)

123 points | by todsacerdoti 1 day ago

15 comments

  • mmmooo 21 hours ago
    older amazon echo dots and shows really make great hardware for this, combined with homeassistant, and can be had for very little cost. e.g. shameless plug: https://github.com/ygelfand/echolocal
    • nl 20 hours ago
      I just commented in another thread that I built this: https://github.com/nlothian/earwing

      Beam forming microphone array, custom UI, custom case. Maybe costs more than a second hand echo dot though (hardware is probably around $30).

    • n8henrie 6 hours ago
      While the caveat emptor may raise some eyebrows, this project has been really cool for my old Echo Dots: https://github.com/Gamer92000/echo-dot-assist

      In short it (optionally) keeps the excellent Amazon wake word detection, beam forming, and transcription pipeline, but firewalls off all cloud access and redirects to your HomeAssistant voice pipeline.

      I've been pretty impressed!

      • epiccoleman 3 hours ago
        Oh man, that's really interesting. I have an Echo speaker hidden in a drawer somewhere because one of the grandparents got it for my son and it was incredibly annoying. Using the hardware for something good (and not having Amazon listening to the room at the drop of a hat) sounds like a good weekend hack.
      • mmmooo 3 hours ago
        echolocal can do pryon as well as microwakeword/openwakeword
      • KetoManx64 3 hours ago
        I'm using this as well for an echo dot I bought on eBay for $15 + Hermes integration in Home Assistant to give instructions to my Hermes agent. Works very well, the first time in my life where i'm getting to enjoy a voice assistant without the nagging worry of all my conversations being sent off to Google/Amazon/Apple
    • neumann 20 hours ago
      I got an echo show rooted and ROMd that I am loving - put it in the kids room as a music player in kiosk mode, but have been now wanting one for myself with wakewords. This looks like the ticket.
      • mmmooo 19 hours ago
        the gen 1 or 2 show 8s make great devices for this kind of stuff, 5's are a little small for me. the lenovo android things (google homes) are also really nice if you want a slightly bigger screen, but not as cheap second hand.
  • mewse-hn 21 hours ago
    Interesting write up, I've been vibe coding an android voice assistant using a similar pipeline without a wakeword - VAD > ASR > LLM > TTS

    I've found parakeet to be much, much better than whisper and I made a 4 bit quant of parakeet 110M for my crappier android test devices, it would probably fit a raspberry pi:

    https://huggingface.co/mewse/parakeet-tdt_ctc-110m-onnx-int4

    • deepsquirrelnet 17 hours ago
      I trained a streaming model that comes with a VAD. Complete with benchmarks on a pi 4.

      https://huggingface.co/spaces/futo-org/asr4all-client-side

      Large is feasible on one core of a pi 4, but not in low latency streaming. All will work well on a 5.

      I have built exactly the pipeline you're talking about with this model and gemma e4b + mtp. Works entirely on CPU at very reasonable speeds. With an LLM you can even use lighter models and it has negligible impact.

      • mewse-hn 7 hours ago
        Cool! Streaming the ASR into the LLM could help my response times, thanks for the link!
        • deepsquirrelnet 6 hours ago
          Yeah, that's how I set it up too. Start working on prefill as its coming in and then use the VAD to determine when the speech turn is over.
    • GaggiX 20 hours ago
      There is also parakeet-redux, a 1.58bits quant of parakeet-tdt-0.6b-v3 that runs as well as the original model: https://huggingface.co/moondream/parakeet-redux

      The model is actually a distillation of a different checkpoint called parakeet-ultra, same size and architecture as parakeet-tdt-0.6b-v3 but better, that's why the 1.58bit quant works so well.

      • mewse-hn 20 hours ago
        Oh crazy I hadn't seen this yet, I might be able to unify my 0.6B vs 110M selector, thanks!
    • nagen 21 hours ago
      [flagged]
  • bill10 16 hours ago
    I am using the old whisper.cpp to turn voice into text, and Kokoro to read text out. Works pretty well for me, but Kokoro still sounds AI and whisper is not that accurate. Will try Parakeet and Silero
  • fragilerock 23 hours ago
    Interesting read! I have my own local voice assistant and I implemented exactly the same question mark trick just the other day.

    The overall pipeline is quite similar to my previous setup, but I found the STT -> LLM -> TTS pipeline just infuriatingly slow. TTS and STT with whisper/piper worked fine and reasonably fast for me locally, but the real bottleneck was always the fact that they're not running concurrently with the LLM call. I ended up forking an open source openai realtime API ESP32 plugin and patching it to my liking. It's a bit more expensive in terms of API cost but I only use it sparingly and it's so much nicer to use now.

    • tristanb 14 hours ago
      The lack of concurrency is exactly what makes the traditional pipeline feel slow; Realtime avoids waiting for separate STT, LLM, and TTS stages. I built voicepe-realtime (https://github.com/TristanBrotherton/voicepe-realtime) for the Home Assistant Voice PE, using OpenAI Realtime with barge-in and custom wake words.
      • fragilerock 13 hours ago
        Hah, the open source project i mentioned in my comment is yours. Thanks for making it :)
        • tristanb 13 hours ago
          Oh you’re very welcome. I’m glad you like it - I welcome pull requests! I’ve made a lot of improvements in speed recently, I’d love to know if I’ve missed anything
    • nagen 21 hours ago
      [flagged]
  • bityard 21 hours ago
    Is this complimentary to, or an alternative to Home Assistant?
    • malfist 21 hours ago
      Home assistant doesn't really provide much out of the box, its mostly a modular/pluggable ecosystem and one of the things you can install are various voice assistants
  • stavros 23 hours ago
    This is really useful, thanks! The other day I bought a small screen and didn't know what to do with it so I made an assistant that talks to my bot:

    https://m.youtube.com/shorts/3FseVyAG5uM

    I used microwakeword too, and training my own wakeword was pretty simple (basically just recording a few tens of samples). Highly recommended.

    I also wanted to be able to issue some voice commands for lights and other things in my home with low latency, so I added a short-circuit plugin that runs right after recognition. If the recognized command matches one of the regexes, it's immediately executed, which has a latency of under a second. Otherwise, it gets sent to my assistant, which usually will tweak the regex (or add a new one) to improve the whole system as it fails.

  • dofm 21 hours ago
    Just what I want to read — favourited to come back to!
  • tristanb 12 hours ago
    [flagged]
  • meatbal_mike 23 hours ago
    [flagged]
  • moneysaid_app 13 hours ago
    [flagged]
  • moneysaidapp 13 hours ago
    [flagged]
  • nandanadileep29 12 hours ago
    [flagged]
  • cgg7709 23 hours ago
    [flagged]
  • yuvicodesk17 14 hours ago
    [flagged]
  • allentown521 17 hours ago
    [flagged]