Small language models that answer in Hinglish
THE QUESTION
Can a small model, fine-tuned for it, write clear dev-roman Hinglish (Hindi in Devanagari, technical terms in English) and beat hand-written rules on narrow on-device jobs?
Founder, Bodham LabsUpdated
What did we build?
- A baseline: two off-the-shelf small models, Qwen2.5-0.5B and Qwen3-0.6B, steered by prompts and an output grammar on three narrow jobs in our voice-memo app. Both lost to the app's own rules.
- A written target style we call dev-roman: Hindi syntax and everyday Hindi words in Devanagari, technical terms, product names and concepts in English, and no over-Sanskritised vocabulary.
- A dataset, now being collected, for fine-tuning a small model in that style.
- Hand-written target examples, shown below, of what the model should learn to produce. They are not model output.
What does the evidence show?
| WHAT | VALUE | SOURCE |
|---|---|---|
| Answer-sentence accuracy, natural questions (n = 40) | rules 62.5% · Qwen2.5-0.5B 32.5% · Qwen3-0.6B 45.0% | laptop CPU, Oct 2026 |
| Doubtful-word job (n = 24) | Qwen2.5-0.5B changed 11 of 11 already-correct words | laptop CPU, Oct 2026 |
| Effect of rewording the prompt, one job | 17.5 percentage points | laptop CPU, Oct 2026 |
| Memory used by Qwen3-0.6B | 1,143.6 MB | laptop CPU, Oct 2026 |
| Dev-roman vs pure Devanagari, Qwen tokenizer | about a third fewer tokens (623 vs 929) | our tokenizer benchmark, Oct 2026 |
| Fine-tuning data | collection in progress | project status, Oct 2026 |
What should the output look like?
Written by hand, not by a model. These show the target style the model is being trained toward. We will publish real model output only after it beats the baseline.
PHYSICS, CLASS 9
ENGLISH
Why doesn't a satellite fall to Earth?
TARGET · DEV-ROMAN HINGLISH
Satellite लगातार Earth की तरफ़ गिर ही रहा होता है, लेकिन उसकी sideways speed इतनी ज़्यादा होती है कि वो Earth से टकराने की जगह उसके चारों ओर घूमता रहता है। इसी को orbit कहते हैं।
ECONOMICS, CLASS 12
ENGLISH
Inflation means prices rise over time, so the same money buys less.
TARGET · DEV-ROMAN HINGLISH
Inflation का मतलब है कि समय के साथ prices बढ़ती हैं, इसलिए उतने ही पैसों में कम सामान आता है।
A VOICE MEMO, SUMMARISED
ENGLISH
The client wants a 15% advance and a confirmation by Tuesday.
TARGET · DEV-ROMAN HINGLISH
Client को 15% advance चाहिए, और Tuesday तक confirmation देना है।
TECH SUPPORT
ENGLISH
Restart the router and check whether the internet light turns green.
TARGET · DEV-ROMAN HINGLISH
Router को एक बार restart कीजिए, फिर देखिए कि internet वाली light green हुई या नहीं।
Where has it led us?
At this size, prompts alone are not enough. Results swung with small changes in wording, and the models changed words that were already right.
That makes fine-tuning the open question, not prompting. A model that has seen the style should not need a long instruction to follow it.
What is still open?
- How much Hinglish data does a 0.5–1B model need before it beats the rules?
- Can it fit in a phone's memory next to two speech models?
- How do we score "clear Hinglish" (faithful meaning, the right term choices, readability) without a person reading every line?
NEXT STEP · NOT DONE YET
What would we do next?
Finish the dataset, fine-tune a small model with LoRA, and re-run the same three jobs against the rules. Publish the model only if it wins.
Have a problem that needs this kind of research?
Tell us about it