AI Translation Quality Benchmark

Because the set of languages keeps growing. Every language we add to the product joins the weekly benchmark run and enters the index as soon as it clears the publication floor — and a new language rarely starts where a pair we have been tuning for a year sits. The headline number is the median across all published pairs for the quarter, so a batch of newcomers pulls it down even while every established pair holds or improves. The new pairs then climb as we tune them, and the index recovers. To see progress on one language, open its pair page below: that trend is not diluted by anything else.

Each test sends a reference text (chat) or plays a reference audio file (voice) through the same live pipeline the product uses, and an LLM judge scores the output from 0 to 100 against a canonical reference translation. Scores fall into bands: 90 and above is excellent, 70–89 good, 50–69 fair, below 50 weak. A single run can swing by tens of points because speech recognition and the judge are both noisy, so judge a pair by its median and range over a month, not by one number.

Voice is one end-to-end score: speech recognition plus translation. Chat is translation of typed text alone. Recognition errors land in the voice score, so voice pairs run lower on average and have a lower publication floor — 45 against 60 for chat. If your team mostly types, the chat number is the one to look at; if it mostly talks, trust the voice number.

No. The benchmark is synthetic: our automated suite plays a fixed FLORES-200 reference set through the pipeline every week, and that is the only source of these numbers. Sessions people run on the demo are neither scored nor saved into the benchmark, and your meetings are never used for it.

Open the pair's own page from the table above: it shows median, range and sample size per month, and the trend across months. To check it on your own material, open the test rig — the same two-pane setup the benchmark runs in — and speak or type in your language; nothing starts by itself and nothing is saved. If a pair is not listed, it has not cleared the publication floor yet or is not in the product yet. Low scores stay reachable by direct link; we do not hide them.