# Small language models that answer in Hinglish

- URL: https://bodhamlabs.com/research/hinglish-small-models
- Theme: Speech & Indian languages
- Area: HINDI & HINGLISH · SMALL MODELS
- Status: EXPLORING — Research and design notes.
- Period: Oct 2026 – collecting data
- Updated: 2026-10-12
- Author: Prabhakar Chandra, Bodham Labs

## Question

Can a small model, fine-tuned for it, write clear dev-roman Hinglish (Hindi in Devanagari, technical terms in English) and beat hand-written rules on narrow on-device jobs?

## Summary

Off-the-shelf 0.5–0.6B models lost to rules on our Hinglish memo jobs. We are now collecting data to fine-tune a small model that writes clear Hinglish.

## What we built

- A baseline: two off-the-shelf small models, Qwen2.5-0.5B and Qwen3-0.6B, steered by prompts and an output grammar on three narrow jobs in our voice-memo app. Both lost to the app's own rules.
- A written target style we call dev-roman: Hindi syntax and everyday Hindi words in Devanagari, technical terms, product names and concepts in English, and no over-Sanskritised vocabulary.
- A dataset, now being collected, for fine-tuning a small model in that style.
- Hand-written target examples, shown below, of what the model should learn to produce. They are not model output.

## Evidence

| What | Value | Where it was recorded |
| --- | --- | --- |
| Answer-sentence accuracy, natural questions (n = 40) | rules 62.5% · Qwen2.5-0.5B 32.5% · Qwen3-0.6B 45.0% | laptop CPU, Oct 2026 |
| Doubtful-word job (n = 24) | Qwen2.5-0.5B changed 11 of 11 already-correct words | laptop CPU, Oct 2026 |
| Effect of rewording the prompt, one job | 17.5 percentage points | laptop CPU, Oct 2026 |
| Memory used by Qwen3-0.6B | 1,143.6 MB | laptop CPU, Oct 2026 |
| Dev-roman vs pure Devanagari, Qwen tokenizer | about a third fewer tokens (623 vs 929) | our tokenizer benchmark, Oct 2026 |
| Fine-tuning data | collection in progress | project status, Oct 2026 |

## Target-style examples (written by hand, not model output)

- Physics, Class 9. English: "Why doesn't a satellite fall to Earth?" → Target: "Satellite लगातार Earth की तरफ़ गिर ही रहा होता है, लेकिन उसकी sideways speed इतनी ज़्यादा होती है कि वो Earth से टकराने की जगह उसके चारों ओर घूमता रहता है। इसी को orbit कहते हैं।"
- Economics, Class 12. English: "Inflation means prices rise over time, so the same money buys less." → Target: "Inflation का मतलब है कि समय के साथ prices बढ़ती हैं, इसलिए उतने ही पैसों में कम सामान आता है।"
- A voice memo, summarised. English: "The client wants a 15% advance and a confirmation by Tuesday." → Target: "Client को 15% advance चाहिए, और Tuesday तक confirmation देना है।"
- Tech support. English: "Restart the router and check whether the internet light turns green." → Target: "Router को एक बार restart कीजिए, फिर देखिए कि internet वाली light green हुई या नहीं।"

## Where it led

- At this size, prompts alone are not enough. Results swung with small changes in wording, and the models changed words that were already right.
- That makes fine-tuning the open question, not prompting. A model that has seen the style should not need a long instruction to follow it.

## Open questions

- How much Hinglish data does a 0.5–1B model need before it beats the rules?
- Can it fit in a phone's memory next to two speech models?
- How do we score "clear Hinglish" (faithful meaning, the right term choices, readability) without a person reading every line?

## Next step (not done yet)

Finish the dataset, fine-tune a small model with LoRA, and re-run the same three jobs against the rules. Publish the model only if it wins.

## Related notes

- [Why a half-billion-parameter model lost to our rules](https://bodhamlabs.com/notes/small-model-lost-to-rules)
- [Hindi costs 1.1× to 4.9× the tokens of English, by tokenizer](https://bodhamlabs.com/notes/hindi-token-cost)
- [Translating into Hinglish with an LLM: the rules that stopped the drift](https://bodhamlabs.com/notes/hinglish-translation-rules)

## References

- [LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021, arXiv:2106.09685)](https://arxiv.org/abs/2106.09685)
- [Qwen2.5-0.5B-Instruct (model card)](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
- [Qwen3-0.6B (model card)](https://huggingface.co/Qwen/Qwen3-0.6B)