InterMIND

Methodology

The InterMIND translation quality benchmark is produced automatically by our own test suite: a weekly synthetic run drives the same translation pipeline that powers the public /demo over a fixed reference set, the same way for every language pair. It is synthetic and reproducible — nothing here is curated or cherry-picked, and there is no human in the scoring loop.

How a single test works

A test run plays a reference audio file (voice test) or sends a reference text (chat test) from a source language through our live translation pipeline to a target language. The resulting translation is scored 0–100 by an LLM judge against a canonical reference translation.

LLM judge: google/gemini-2.5-flash primary, anthropic/claude-sonnet-4-20250514 fallback (via Vercel AI Gateway). The same prompt, rubric, and reference texts are used across all pairs.

How a month is aggregated

  1. Every completed full benchmark run is recorded in demo_test_runs with per-target-language scores. Single-pair previews are excluded — only full multi-language runs feed the medians.
  2. For each (source language, target language, test type, week) we keep the latest run, then aggregate by month — so a single source cannot swing medians by repeating a pair within a week.
  3. From the deduplicated set we compute median, min, max, p10, p90, average, and sample size.
  4. We snapshot individual scores so that once a month closes, the numbers remain stable even after raw runs are deleted by TTL.

When a pair appears in the public index

A (pair × test type × month) row is eligible for the public index and sitemap when it clears a minimum sample size and a median-score floor (a lower floor for voice, which carries unavoidable ASR noise). Because the dataset comes from a single automated source rather than many independent visitors, the unique-source requirement is relaxed for this synthetic data — the sample-size and quality floors still apply.

Pairs that fall short remain accessible by direct link — the thresholds gate only what search engines see. We do not hide low scores from anyone who asks for them directly.

What we deliberately do NOT claim

  • A single score is not trustworthy. A well-behaved pair can swing 30 points between runs because ASR and LLM judgment are both noisy. That is why every page shows a distribution, not a single number.
  • The LLM judge itself is imperfect. We may switch or dual-judge in the future; historical rows will carry the judge identifier when that happens.
  • The benchmark does not measure latency, cost, availability, or user satisfaction. Those live elsewhere.
  • The dataset is synthetic — produced by our own automated suite, not by independent third-party traffic. We disclose this explicitly rather than imply an external panel.

Honest calls

When a month drops in quality for a specific pair — it stays visible. When a bug is fixed and the next month jumps — the improvement is visible on the same page. We never rewrite historical aggregates. Admins can only hide individual months from the public index; they cannot alter the numbers that were already published.

Known issues affecting historical data

  • Chat test harness, before 2026-04-23: the automated chat-test pipeline had per-language failures. Chat aggregates from earlier months may show lower scores than the actual translation quality of those periods. Affected months are kept in the database but suppressed from the public index; the trend chart will show a step change at the fix.

Questions

Disagree with something here? Open an issue or write to us. We will update this page and note the change.