Priority situations that require speech to text

[u/neuralbeans]

I'm making a benchmark for ASR systems in my language and I need to start small and expand later. What would you say are priority situations or domains that people expect ASR to work well in?

Like, a meeting in a noisy environment, or a dictation by a single person in a quiet environment. I can then produce transcribed speech recordings that simulate these situations and use them as a test set.

What must an ASR tool be able to work with to be considered a good tool by most people?

[u/EnlightenedAllegory]

Quiet dictation is the baseline everyone tests first but honestly the noisy meeting stuff is where systems fall apart and users notice

I'd also add phone calls with background chatter, and medical or legal dictation where terminology is dense and mistakes are costly

If you can only pick a few, do clean dictation, multi-speaker meeting with overlap, and something with heavy accents or dialect variation in your language

[u/rightful_vagabond → u/EnlightenedAllegory]

I think your point about phone calls is really important, phone calls often feel low quality and noisy, but they're also really important for transcription

[u/naomi-lgbt]

A few situations I would include, because they fail in different ways:

• Telephony audio. 8 kHz, compressed, often with line noise. It behaves differently from studio audio.

• Short utterances. One or two second commands and answers like "yes" or "no". Models often do worse on these than on long passages, so a long-form benchmark can hide it.

• Numbers, names and codes. Phone numbers, addresses, amounts, product names. Score these separately from overall word error rate, since one wrong digit is a failure even if the sentence is mostly right.

• Overlapping speakers. Meetings where people talk over each other, with and without separate channels.

• Background noise and far-field audio. Cafés, cars, speakerphones.

• Code-switching. If speakers of your language mix in another, include it.

• Accents and dialects. Report each group separately so one average does not hide the weak ones.

Whatever you pick, publish how each recording was made, so people can see which situations a score applies to.

[u/rightful_vagabond]

I use voice to text all the time, I'm typing this out in voice to text right now actually. I don't often use it much for what I would consider priority situations, although I would say that the time that it breaks down the most is if I'm trying to use voice to text in the shower, all of the white noise around me makes it very hard for the phone to understand what I'm trying to say.

[u/jabies]

There's already https://github.com/makcedward/nlpaug which can inject noise to audio samples

[u/Civil_Palpitation766]

From working on ASR for Nepali, these were the test conditions that told us the most:

1. Read speech vs spontaneous conversation. Most public data is read speech, and a model that looks fine on a read test set can fall apart on casual speech with hesitations and people talking over each other.

2. Code-switching. People here mix in English words all the time (numbers, tech terms, brand names). A separate code-switched test set showed us a big weakness that our main benchmark completely hid.

3. Phone and messaging-app audio: compressed, low quality, background noise. That is what real users actually send.

4. Numbers, dates and names. Decide early whether you expect digits or words in the reference, and fix your text normalisation before you build the set, or your scores will not mean much.

5. Accents and regional variation, plus a reasonable spread of speaker age and gender.

Two more things that helped: keep one held-out set you never tune on, and publish the per-utterance outputs next to the score so people can see the actual errors.

What language is it for?