L3: Arabic language checking against
26 large language model configurations

L3, the Lisan language engine, ranked first on every primary metric in this evaluation: exact-span precision, recall, F1 and F0.5, exact and normalized correction accuracy, sentence exact match, ROUGE-1 and ROUGE-L. On F0.5 it scored 80.29%, a 35.61-point lead over Claude Fable 5.1 at 44.68%. Measured.

The strongest comparator changed with the metric, which is worth saying plainly. Claude Fable 5.1 was next on precision and on F0.5. Claude Opus 5 was next on F1, at 34.58% against 74.66% for L3. GPT-5.6 Sol was next on recall, at 41.08% against 66.85%. No single large language model configuration was second everywhere.

Two tasks, not one

Language checking is two decisions made in sequence. The system first has to decide where the text is wrong, and only then what to put in its place. We score the two separately, because a system can be good at one and poor at the other.

Detection is scored on exact spans. Every reference edit occupies a stretch of the source sentence, located by character positions, and a system earns a detection credit only when it changes exactly that stretch. Changing a neighbouring word, or half of the right one, counts as a miss and a false alarm at once. Insertions are zero-width positions, and a punctuation change can be its own span.

Correction is scored only where detection succeeded. Of the spans a system located correctly, we ask how many replacements match the linguist's reference, byte for byte and again after Arabic normalization. That makes correction accuracy a conditional figure, and we say so wherever it appears.

The headline metric is F0.5, which combines precision and recall but weights precision more heavily. In institutional language checking an unnecessary change usually costs more than a missed one: it has to be reviewed, rejected and sometimes reverted by hand. F0.5 encodes that asymmetry. We report F1, precision and recall alongside it, so nobody has to take the weighting on trust.

Precision and recall

L3 is the only system that combines precision above 80% with recall above 60% on this benchmark. Its 84.54% precision means that most of the changes it proposed landed on a stretch of text the linguist also changed. Its 66.85% recall means it found the majority of the reference edits, and missed the rest. Measured.

The large language models fall short in two different ways. Claude Fable 5.1 was the most precise at 64.21%, but it found only 20.15% of the reference edits: it changed little, and what it changed was usually right. GPT-5.6 Sol found the largest share of reference edits among the models at 41.08%, but at 25.03% precision, so most of its changes were not warranted. Most other configurations cluster in the lower left of the plot, close to one another on both axes.

That shape is the practical argument for F0.5 rather than a raw count of corrections. A cautious system with high precision and low recall leaves errors in the document. An aggressive one with the opposite profile fills the review queue with changes an editor has to reject. The benchmark rewards a system that can do both, and one did.

Correction quality after detection

Once L3 located a span correctly, its replacement matched the reference byte for byte 89.79% of the time, and 96.38% of the time after Arabic normalization. Normalization removes diacritics and tatweel, the elongation character, and unifies alef, hamza and teh marbuta variants, so a correct suggestion is not marked wrong for one extra or missing vowel mark. Measured.

These figures are conditional, and the caveat is not a formality. Claude Fable 5.1 reached 89.07% exact and 94.54% normalized, 1.84 points behind L3 on the normalized rate. But those rates apply only to the spans it detected, which were a small fraction of the reference edits. A system that changes little and changes it well can post a high conditional accuracy while leaving most errors in place. Correction accuracy tells you how trustworthy an accepted suggestion is; recall tells you how much of the document it reached. Neither number substitutes for the other.

Whole-sentence accuracy

Span metrics describe edits. Editors receive sentences. Sentence exact match asks the end-to-end question: after the system is done, is the whole sentence identical to what the linguist would have written? L3 cleared that bar on 55.96% of sentences, double the rate of Claude Fable 5.1 at 27.98%. Measured.

The measure is strict by design. One extra diacritic, one missing period or one untouched error fails the sentence, which is why every rate here sits far below the conditional correction rates above. It is a useful summary of the whole pipeline, not an error rate, and it does not replace the span-level analysis.

Reference fidelity

ROUGE measures how much of the reference sentence survives in the output, whatever the edit scoring says. High values mean the system corrected the text without rewriting it. L3 scored 98.84% on ROUGE-1 F1 and 98.83% on ROUGE-L F1, leading Claude Fable 5.1 by 2.28 and 2.29 points. Measured.

Two details matter here. The axis starts at 95 because every system in the top ten is above it; on a full axis the marks would look identical, and a truncated bar chart would exaggerate. And L3's ROUGE-1 and ROUGE-L values are nearly the same, which we read as retaining both the reference vocabulary and its order: the engine changes what is wrong and leaves the sentence structure alone. The legacy set-based overlap score, kept for traceability rather than as a primary metric, tells the same story at 91.45% against 88.93%.

What the corrections look like

Numbers describe the benchmark; sentences show it. The cards below are drawn from benchmark rows and show the source, the linguist's reference, L3's output and the outputs of several large language model configurations, verbatim. Outright errors come first. The last two cards are labelled editorial standard: the source sentence is grammatical, the reference applies the house convention the linguist worked to, and the models were told not to rephrase. Those rows show what the engine's editorial layer does, not the models failing at grammar, and the captions say so.

One honesty note applies throughout. Each system contributed a single output per sentence, L3 its first suggestion only, and the models were bound by a strict prompt. Where a model matched the reference, the card says so.

Speed

L3 processed 52.28 standardized output tokens per second, at 1.041 seconds per timed sentence. It was second. Qwen 3.8 27B on Groq was faster, at 66.98 tokens per second and 0.828 seconds per sentence, and it holds the top row of the chart. Claude Haiku 4.5 was third at 33.43. Claude Fable 5.1, the strongest model on quality, ran at 15.07 tokens per second and 3.585 seconds per sentence. Measured.

Throughput here is service throughput. For every valid corrected output we applied the Lisan tokenizer, summed the standardized token count and divided by end-to-end wall-clock request time, so the figure includes request and prompt-processing overhead for every system alike. It is not decoder speed, and it is not the batched, saturated rate used in the energy measurement below; the two describe different operating conditions and should not be compared.

Throughput and quality do not move together in this field. Qwen 3.8 27B on Groq, the fastest configuration, ranked last on the legacy set-based overlap score at 75.81%. At the other end, Gemini 3 Flash averaged 22.399 seconds per sentence.

Energy

We measured L3 directly on an NVIDIA A10 24 GB accelerator: at least 95% utilization, a 30-second warmup, a 10-minute steady state, power sampled at 1 Hz. Mean GPU power was approximately 100 W at approximately 2,000 tokens per second, which works out to 0.05 J per token, or approximately 0.000014 kWh per 1,000 tokens. Measured, accelerator scope.

That figure excludes host power, cooling, networking, storage and facility overhead. To compare like with like we multiply by three, a factor that rounds 1.5 for power-usage effectiveness, 1.3 for redundancy and reliability and 1.5 for remaining serving and grid overheads, giving a full-stack estimate of 0.000042 kWh per 1,000 tokens.

The large language model figures are cited estimates, not measurements, because API providers expose no power telemetry. Google's disclosure of 0.24 Wh for a median Gemini Apps text prompt, at a 200-token assumption, gives 0.0012 kWh per 1,000 tokens and is already full stack. The IFP School per-query values of 0.61 Wh for GPT-4o-mini, 7.4 Wh for GPT-5 medium and 14.1 Wh for GPT-5 high at 300 tokens, scaled linearly to 1,000 tokens and multiplied by the same three-times factor, give 0.0061, 0.074 and 0.141 kWh. On that basis L3's full-stack estimate is approximately 29 to 3,357 times lower than the anchors. Tested models without a published per-token figure were mapped to these anchors by capability tier as a scenario estimate; that mapping is a comparison, not a measurement of any individual model.

The result is sensitive to the overhead factor, so we show the range rather than the midpoint alone. At two times, L3's estimate is 0.000028 kWh and the advantage over the most conservative anchor, Gemini Apps, is 42.9 times. At four times it is 0.000056 kWh and 21.4 times. Across that whole range L3 stays at least 21 times more energy-efficient than the lowest anchor. The Gemini Apps comparison is the one we would defend first, since it rests on a vendor disclosure rather than a third-party estimate.

Methodology

Systems under test

The comparison field is 26 large language model configurations from six providers, alongside L3. GPT-6 Astra was run in two reasoning settings; the report's appendices carry a single GPT-6 Astra row. Grouped as the report groups them:

• Lisan: L3, the Lisan language engine, scored on its first suggestion only.
• Anthropic: Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Claude Fable 5.1.
• OpenAI: GPT-6 Astra, GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna, GPT-5.1 (medium reasoning), GPT-5.4 (high reasoning), GPT-4o-mini, gpt-oss-120b (on Groq).
• Google: Gemini 2.5 Flash, Gemini 3.1 Pro (Preview build), Gemini 3.5 Flash-Lite, Gemini 3.8 Flash, Gemini 3 Flash (Preview build), Gemma 3 27B (on Bedrock).
• DeepSeek: DeepSeek V4 Flash, DeepSeek V4 Pro.
• Meta: Llama 3.1 70B (on Bedrock), Llama 3.3 70B, Llama 4 Maverick.
• Alibaba Qwen: Qwen 3.6 27B and Qwen 3.8 27B, both on Groq.

Every configuration received the same sentences, the same prompt, the same single-output rule and the same evaluator.

Evidence

Limitations

Full results

Full results, evaluation snapshot 9 September 2026 · version 1.0. Rates only; span counts and total seconds are not published. See the methodology.

Cite this report

Please cite the report by its own title, which differs from this page's heading:

Check your own text

The benchmark is ours; your documents are the test that matters. The WriteX editor is free to start, institutions can run L3 on their own infrastructure under a pilot, and methodology questions go to support@lisan.com.