Routing Hinglish speech: why a 10-second window beat a short one

Why does a voice-memo app need two speech models?

Our on-device voice-memo app, Mementos (codename), transcribes every memo on the phone. Many speakers in India use English in one memo and Hinglish, a mix of Hindi and English, in the next. Sometimes they switch in the middle of a sentence.

We tested speech models on the developer's own recordings: 17 items, 304 seconds in total, 12 of them in English and 3 in Hinglish. Each model won one language and lost the other.

Word error rate (WER) is the share of words a model gets wrong (swapped, missing or added) against a correct transcript. Lower is better. On Hindi-heavy audio, Parakeet, an English model, wrote nothing or English-sounding nonsense. So each memo, or each part of a memo, has to go to the right model. That job is called routing.

How do we tell English from Hinglish?

With language identification (language ID): a small model listens to a stretch of audio and names its language. We used Whisper-tiny (int8, 111 MB) and sent a stretch to Hinglish-Swift only when it said Hindi (“hi”) or Urdu (“ur”). Everything else went to Parakeet, because the detector sometimes labels Indian English as another Indian language. One of the developer's English memos came back as Telugu.

An earlier idea, guessing the language from Parakeet's own English transcript, missed 64 of 150 Hinglish clips (43%). Names and deal terms looked like non-English words too.

The first build let the first 10 seconds decide

Our first test build ran language ID once, on a memo's first 10 seconds, and sent the whole memo to one model. For memos in one language that worked. A memo that switched language halfway went entirely to one model, and the other language came out as nonsense.

Why did a 10-second window beat a short one?

The obvious fix was to check every piece. The app already cuts memos at pauses into pieces of up to 12 seconds, using voice activity detection (VAD), a small model that finds where speech starts and stops. So we ran language ID on each piece and sent each piece to its own model.

In the first laptop test (866 pieces from 327 clips), this halved word errors on 10 mixed clips, from 34.84% to 17.72%. But it sent 30.7% of Hinglish pieces to the English model, up from 5.9%. A short piece gives the detector too little audio to go on.

So we gave it more. For each piece, the detector now hears a 10-second window centred on that piece, borrowing audio from its neighbours. The piece itself still goes to one model on its own.

Read the table honestly. The drop in word errors on mixed clips, 35.7% to 18.2%, comes from routing per piece; the short and the 10-second windows tie there. What the window adds is the rest: Hinglish misroutes fall from 29.7% to 7.2%, and the developer's 17 recordings stay at 13.98% instead of rising to 14.31%. Of the rules we tried, it is the best balance across all four columns.

It did not meet every bar we set beforehand. We wanted Hinglish misroutes at or below 5.9%, the old rule's figure in the first test, and mixed-clip errors at or below 20%. No rule met both. The 10-second window came closest.

What does it cost on the phone?

On our test phone, a Redmi Note 14 Pro+ 5G, the app's routes matched the laptop script exactly: 23 of 23 pieces on 5 mixed clips. Then we measured where the time goes, in our benchmark app, with 5 runs of each setting.

Real-time factor (RTF) is processing time divided by the length of the audio. An RTF of 0.1 means a 60-second memo takes 6 seconds.

• The language check costs as much as the transcription. Each check took 0.34 s on the Redmi, 4.1 s for the 54-second memo. Transcription itself stayed at 0.07 to 0.08 of the memo's length.
• Hinglish is slower to transcribe. Hinglish pieces took about 0.16 of their length, against 0.07 for English.
• Memory. In the app, peak memory was 1.30 GB for an English memo and 1.66 GB for a mixed one, which loads both speech models. In the benchmark app the peak stayed between 1.44 and 1.72 GB over all 60 runs.

What did we try and drop?

• Blank penalty: Parakeet emits a “blank” when it hears no new word. Penalising blanks is meant to stop it dropping words.
• Beam search: keeping several candidate transcripts at each step instead of only the single best one.
• Hotwords: a list of words the decoder is nudged towards. Our test had a flaw: the model shipped without a vocabulary file, so we built one with equal scores, which may not match its real tokenizer.

What this does not prove

• The mixed clips are synthetic. We made them by joining the developer's own English and Hinglish recordings. Real switching in the middle of a memo is not measured, and a real Hinglish memo spoken into the phone's microphone has not been tested yet.
• Few Hinglish recordings. 3 from the developer. The public Hinglish clips are tutorial-style speech; more English-heavy Hinglish may be labelled English more often.
• No WER on the public clips. Their Hindi references are in Devanagari while our model writes Roman letters, and the English references contain markup. We use them only to count routes.
• One phone. 1.66 GB on mixed memos has not been tried on a phone with 8 GB of memory.

Questions