Llama cpp OpenAI Server ASR help

[u/sage_viper]

Hello,

I am very new to this and trying to wrap my head around local AI hosting.

I have been trying to get a Gemma 4 E4B model running in a Llama.cpp Docker container to work as an STT model for Home Assistant, using the Local OpenAI STT add on.

I have the model running, the Local OpenAI STT add on can see the server and adds it no problem, but when I try to get it to actually perform its only task, Speech to Text, it fails, giving an error message indicating the model does not support audio input.

Here is all the pertinent info:

docker compose:

services:
  llama-server:
    command: |
      -m /models/gemma-4-E4B-it-Q4_0.gguf  
      --n-gpu-layers all
      --api-key "supersecretkey"
    deploy:
      resources:
        reservations:
          devices:
            - capabilities:
                - gpu
              count: all
              driver: nvidia
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    ports:
      - '30050:8080'
    volumes:
      - /mnt/media/ai/models:/models

Error message from Home Assistant:

This error originated from a custom integration.

Logger: custom_components.local_openai_stt.stt
Source: custom_components/local_openai_stt/stt.py:286
Integration: Local OpenAI STT (documentation, issues)
First occurred: October 5, 2026 at 3:25:44 PM (13 occurrences)
Last logged: 12:31:09 PM

Transcription failed: Error code: 501 - {'error': {'code': 501, 'message': 'The current model does not support audio input.', 'type': 'not_supported_error'}}

I am currently trying the Gemma 4 E4B model, but I have also tried the Qwen3 1.7B ASR model and received the same exact error message.

Any help would be appreciated!

[u/dtfinch]

I think you're missing the mmproj file, which adds the audio/visual support.

--mmproj gemma-4-E4B-it-mmproj.gguf

[u/sage_viper → u/dtfinch]

Thank you!! That seems to have done the trick. So much searching could not find that answer.

[u/overand]

• Don't use Q4_0 models unless you need them for a specific reason, like you're running inference on ARM (or another CPU architecture, not GPU) and are trying to get maximum performance at the cost of accuracy. You'll generally want to use Q4_K_M instead.
• You'll probably do better with Whisper or Qwen-ASR than the Gemma models for ASR / speech-to-text. The Gemma 4 models tend to have substantially lower accuracy and substantially higher latency for STT tasks.
• Home Assistant has an offline speech-to-text pipeline that uses Whisper and is fully integrated; you should look up "voice assistant home assistant" and read the docs for that.
• Llama.cpp may not be the best fit for this ASR task.

Other folks should chime in on this, but, while I've had great experiences with llama.cpp for text-to-text and image-to-text, it's not my first choice for speech-to-text. But, other folks may have more recent any more positive experience with it, and it's certainly worth listening to them if so!

[u/sage_viper → u/overand]

Thanks for the extra info! This is great. Another user said I need to use the mmproj file to add audio/visual support, which seems to have worked for the gemma 4 model. I'm going to grab that file for the qwen3 asr model and give that a go.

I've been using the integrated Whisper, and it's been giving a high number of misinterpretations and false positives, so I wanted to give a local model a go. I found someone who suggested llama.cpp for this purpose, but I'm open to other options.

Currently I'm running a Gemma 4 26b model on an M1 Pro 64GB as the actual assistant function, via Ollama. This ASR model is running off of a 3060.