Why a half-billion-parameter model lost to our rules

Why did we try a small language model at all?

Our on-device voice-memo app, Mementos (codename), does three small jobs with plain rules today. When you ask a question, it picks the sentence in a memo that answers it, by counting shared words. When the speech model is unsure of a word, such as a name, something has to decide whether to leave it or swap in a candidate from a correction list. And when you save a memo, a tag (Work or Personal) is suggested by repeating the last one you used.

The idea we tested was simple: tell a small model exactly what is needed, every time. One fixed prompt per job, under 150 words of instruction. The model may only choose or label. It never writes transcript text. Its answer is a number, a list of numbers or one item from a given list, and anything else is rejected.

If a model under about 500 MB could beat the rules at that, it would be worth a test on the phone. If it could not, the rules stay.

How did we set up the test?

• q4_k_m is a llama.cpp file format that stores the model's weights at about 4 bits each, so a half-billion-parameter model fits in under 500 MB.
• A GBNF grammar is a small set of rules, in llama.cpp's own notation, that limits which tokens the model is allowed to produce. Ours were as strict as root ::= [0-4].
• Temperature 0 means the model always takes its most likely answer, so a run can be repeated exactly.

How did the models do against the rules?

Each table compares the models with what the app already does. Counts are right answers out of the sample size (n).

The newer model is clearly better than the older one on every job. It still did not beat the rules on any of them.

How the models failed

Qwen2.5-0.5B changed 11 of 11 words that were already right. On the doubtful-word job, whenever candidates were on offer, it picked one instead of answering 0 for “leave it”. It fixed 6 real errors and broke every correct word it was shown. For a memo app, breaking a correct name is worse than leaving a wrong one, because the user trusts what is on screen.

Qwen3-0.6B failed the other way. It answered “leave it” 22 times out of 24. It broke only 1 correct word, but it fixed none of the 13 errors. That is the same result as doing nothing, at a cost of almost a second per call.

Both were sensitive to wording. We reran job 1 with two reworded prompts. Qwen2.5-0.5B scored 13, 11 and 6 of 40; Qwen3-0.6B scored 18, 11 and 11 of 40. Both spreads are 17.5 points. Our rule was that a swing of more than 5 points with wording makes a job “not reliable”.

The grammar worked. No answer broke the number-only format in any run. The grammar controlled the form of the answer; it could not make the answer right.

What did it cost in time and memory?

For comparison, the light lookup-table model in the same tests took about 0.1 ms per sentence and about 138 MB of extra memory. The rules take no extra memory at all. Qwen3-0.6B needed over 1 GB for jobs the rules already do better.

Did light meaning models do any better?

In a related test the same day (T-035, corrected in T-039), we tried light embedding models, which turn a sentence into a list of numbers so that similar meanings land close together. The question was the same: do they beat keyword search?

On questions worded the way people speak, keyword search was as good as the models. The differences were 1 to 4 questions out of 40, inside the 90% bootstrap ranges. The models helped only when the question shared no word with the memo. So meaning search also waits: the field test will count how often a search comes back empty, and that count decides.

What this does not prove

• Small samples. 40 questions, 24 doubtful words and 45 memos. The Hinglish parts are 10 questions and 5 memos, too few to say anything about Hinglish on its own.
• Laptop only. No model ran on the phone, where memory and speed would be worse, not better.
• Two models, one size, 4-bit. This says nothing about models of 1.5 billion parameters or more, which need 1 to 3 GB of memory and a multi-GB download. They may well do these jobs.
• Prompts, not training. Neither model was fine-tuned. We held the prompts fixed between models on purpose, so better prompts might do somewhat better, and the 17.5-point swings suggest the results depend on wording.
• Mostly synthetic data. 40 of the 45 memos were written for the test, and one long memo answers 17 of the 40 meaning-test questions.

Questions