Benchmarks
Each model was tested on 8 cases — 4 English, 4 German — spanning short sentences, medium paragraphs, and email-length texts with intentional typos.
English
| Model | Exact Match (%) | Errors Fixed (%) | Time (ms) | Memory (MB) |
|---|---|---|---|---|
| OpenRouter gpt-4o-mini | 75% | 100% | 1,126 | 230 |
| Qwen3.5 2B | 100% | 100% | 1,986 | 2,012 |
| Qwen3.5 4B | 75% | 100% | 4,099 | 4,202 |
| GRMR V3 3B | 75% | 98% | 2,796 | 3,511 |
| GRMR V3 4B | 75% | 98% | 3,256 | 4,140 |
| Mistral 7B Instruct v0.3 | 50% | 95% | 6,365 | 7,690 |
| Gemma 3 4B Instruct | 25% | 92% | 3,600 | 4,142 |
| Harper | 50% | 90% | 86 | 190 |
German
| Model | Exact Match (%) | Errors Fixed (%) | Time (ms) | Memory (MB) |
|---|---|---|---|---|
| OpenRouter gpt-4o-mini | 75% | 100% | 1,240 | 230 |
| Qwen3.5 2B | 50% | 95% | 2,253 | 2,004 |
| Qwen3.5 4B | 75% | 95% | 4,981 | 4,202 |
| Mistral 7B Instruct v0.3 | 50% | 95% | 9,197 | 7,690 |
| GRMR V3 4B | 0% | 68% | 4,306 | 4,140 |
| GRMR V3 3B | 0% | 36% | 3,455 | 3,511 |
| Gemma 3 4B Instruct | 0% | 27% | 4,170 | 4,142 |
| Harper | 0% | 0% | 290 | 205 |
What this means
The models fall into three categories. Harper is a rule-based grammar checker — it matches words against a dictionary and applies fixes instantly, but it only knows English and struggles with context-dependent errors. Is a linter at the end. The GRMR models are small LLMs fine-tuned specifically for grammar correction — they understand context and fix more errors than Harper, but they were trained primarily on English data, so German accuracy is limited. The general-purpose models (Qwen3.5, Gemma, Mistral) are instruction-following LLMs that use a system prompt to correct text — they aren't specialized for grammar but their broader training data gives them better multilingual support. The newer Qwen3.5 models are the standout: even the 2B version matches or beats the much larger Mistral 7B in both languages.
OpenRouter's GPT-4o-mini fixes every single error across both languages because it is a much larger model running on powerful remote hardware. The tradeoff is that your text leaves your device and you need an API key.
For local use, Qwen3.5 2B (1.3 GB) is the recommended choice — it fixes 61 of 62 errors in English and German, is the fastest local LLM, and needs about 2 GB of RAM. Qwen3.5 4B is as accurate but twice as slow. For the best results with no hardware constraints, OpenRouter is unbeatable.