Every model review and benchmark on this blog follows the process below. We publish it so you can judge our conclusions for yourself — and check them against the source texts we show.
What we evaluate
We test the AI translation models available on BelinDoc on the work people actually bring to a document translator, not abstract leaderboards. A standard model benchmark covers up to eight document scenarios, each probing one specific weakness (earlier or focused reviews use fewer, and state the exact number in the article):
| Scenario | Direction | What it probes |
|---|---|---|
| Academic abstract | EN → ZH | Jargon, passive voice, formal register |
| Legal contract clause | EN → ZH | Nested long sentences, precision, legalese |
| Technical doc with code | EN → ZH | Identifiers inside backticks must stay untranslated |
| Literary prose | ZH → EN | Classical-flavored register, rhythm, imagery |
| Manga dialogue | JA → EN | Colloquial register, character voice, sentence-end particles |
| Markdown table | EN → ZH | Table structure, error codes, units |
| Long-document term consistency | EN → ZH | One polysemous term translated consistently across paragraphs |
| OCR noise | EN → ZH | Source contains scan artifacts; the model should recover intent, not copy errors |
Where possible we reuse the exact source texts from earlier rounds, unchanged, so results can be compared over time rather than only against the models in a single review.
Each round tests the current flagship models side by side in the same arena — for example, the latest generation from OpenAI, Anthropic, Google, and DeepSeek, plus the previous generation for comparison.
How we score
- Every cell runs multiple times; we take the median. A single API call is subject to server-side variance, and drawing conclusions from one sample is how luck gets published as a finding.
- Dual blind judging, and the judges are not contestants. Candidate translations are anonymized to A / B / C… (reshuffled independently each round) and scored 1–5 on faithfulness, fluency, terminology, style, and formatting by two independent LLM judges. Neither judge appears in the contestant list — letting a model grade its own output introduces bias.
| Dimension | What it measures |
|---|---|
| Faithfulness | Meaning preserved; nothing dropped or invented |
| Fluency | Natural target-language phrasing |
| Terminology | Consistent, industry-standard terms |
| Style | Register and tone match the source |
| Formatting | Layout, tables, and structure preserved |
How scores are aggregated. The five dimensions are equally weighted. For each translation we average the five dimensions per judge, then average the two judges; a model's score in a scenario is the median of its repeat runs; its overall score is the mean of its per-scenario scores. Cost-per-point = the model's per-call cost ÷ its overall score.
How we keep it fair
- Same prompt for every model. All candidates translate under one identical, minimal instruction, so we measure the model — not prompt engineering.
- Blind labels. Judges see the translations, not the model names; labels are shuffled independently for each judge.
- Source text shown. Every review publishes the original text alongside the candidate translations, so you can check our scoring against the actual output rather than taking it on faith.
Beyond quality
Translation quality isn't the whole decision, so each round also records latency (time to translate a standard sample), cost (including cost-per-point comparisons), and format preservation — whether tables, columns, and structure survive in the output, probed by the Markdown-table and OCR-noise scenarios. Note: these are text-fragment tests, not full documents; behavior on large multi-page or scanned PDFs can differ (see Limitations).
Data and record-keeping
Every article shows — per scenario — the source text, each model's translation, and the per-dimension scores, so any result can be checked against the actual output on the page. Behind that, the full set of raw translations, judge scores, and aggregation scripts is archived, and the figures in each article are reported straight from that archive.
How often we update
We re-run reviews when a model ships a new version or a major update lands, and we revise the affected articles rather than leaving stale numbers up. Each article shows its last-updated date so you know how current the results are.
Editorial independence
- Disclosure: BelinDoc is our own product. It appears in these comparisons because it runs on the same models we test.
- We publish where we lose. When a competing model — or a third-party tool — does better on a scenario than the option we'd recommend, we say so; reviews are only useful if they're honest about the trade-offs.
- We don't cherry-pick samples to flatter a result, and we don't adjust scores after the fact.
Limitations
- The judges are LLMs, not human translators. LLM judges are reasonably reliable on checkable dimensions like faithfulness and terminology; treat "style" with more caution.
- Model output varies run to run; taking the median across repeats reduces this, but scores remain a considered reference point, not a guarantee for every document.
- Each round covers a finite set of scenarios and a stated language direction — strong results in one domain don't guarantee the same elsewhere.
Questions about our testing? Contact us.