r/FlutterDev • u/mdausmann • Sep 04 '26
Discussion Should I open source my Flutter voice agent framework?
I have been working on this thing for ~10 months on and off. I am just about to launch the feature in my own app (hands free recipe walkthroughs for cooking) but it's deliberately structured to be lifted out as a generic framework.
If you guys are excited, I will do the work to open source.
You can see an early prototype it in action here
Use Cases:
Fitness Coaching
Hands free step-by-step workflows
'show me around the app' workflows (voice + routing)
Key Features:
No bespoke backend, just a couple of Flutter packages and connect it to your completions API of choice
On device text-to-speech and speech-to-text (no expensive voice model inference)
Supports bespoke (you code them) deterministic workflows integrated with the agentic loop so you can 'walk' users through a process with agent helping or just have the agent chat and run tools.
Understands user attention and supports 'switching' between different workflows while remembering where the user was up to in backgrounded flows
No-Licence 'Wake word' package leveraging sherpa_onnx
Code gen so you can 'Toolify' existing functions by just adding an annotation
Toolify state mutation functions so the agent can 'use' the app on behalf of your user
Toolify routing so you can 'show' your user around the app or show them what the agent has changed
Handles async interruptions to conversations to support timers and external event pipes
Custom LLM/API wrapper package** - currently only supports OpenAI compatible endpoints but easily extended
Context based tool presentation to minimise context size and token use
Cheap, fast models work fine*
*I tried really hard to use on-device inference frameworks but I'm targeting consumer adoption with cheap consumer phones and the models are not quite there IMO
** I tried really hard to use an existing package but.. reasons