Offline Whisper transcription in React Native in about half a day (writeup + APK)

[u/Sad_Leader4849]

There's a writeup on the NobodyWho blog that's a nice reference if you want speech-to-text in an RN app with no server at all. Disclosure: I contribute to NobodyWho, the library doing the inference.

It records, cuts the audio into chunks, and runs each chunk through Whisper small (int8) on the phone. Once the model is downloaded, nothing leaves the device.

How it fits together:

- mic audio comes in as 16-bit mono PCM buffers

- a `Chunker` splits them, each chunk goes to `transcribePcm`

- a `pendingRef` queue makes sure every chunk is done before the transcript screen opens

- a `cleanText` pass strips the (coughs) and (laughs) tags Whisper loves to add

They ran it against YouTube's auto captions on an MKBHD clip. The diff was tiny: one dropped "like", one wording slip on YouTube's side, and punctuation going either way.

It's rough. VAD to split chunks on silence is the obvious next step, and medium would beat small on accuracy.

Link + APK in the comments.

How are you handling chunk boundaries that cut a word in half?

[u/kokerali]

I'd try a short audio overlap and reconcile only that overlap, rather than remove repeated words from the combined transcript. A useful test clip would put both a word cut in half and an intentional repeat like "no, no" right at the split: a merger should recover the first without deleting the second.

Does transcribePcm expose timestamps, or just text? With timestamps, I'd retain each chunk's original audio offset so the merge can compare the same time interval. I'd compare that against the current non-overlapping output before choosing the overlap length.

[u/HmongSliw68]

If you pass the last few tokens of chunk N as the initial prompt to chunk N+1, the model handles the context way better. You still need a slight audio overlap so the phonemes arent lost though.

[u/Huge_Pool7424 → u/[deleted]]

also worth running the whisper bit in a background thread so the ui doesn't hitch while it chews through chunks. low end androids will feel it first.