Mirela
Thanks to advances in artificial intelligence, machine translation tools are now capable of producing fluent and impressively fast translations. But as these tools are increasingly being used in communication, business, and education, we need to ask ourselves: How do we measure AI translation quality?
In this article, we are looking at the key metrics and methods used to evaluate AI-generated translations, how good they are, and where they might fail.
What does “quality” mean in translation?
Before we can measure translation quality, we need to define it. Translation quality refers to how effectively a translated text conveys the meaning, tone, and intent of the original content in a natural and accurate way.
The elements we look at to establish translation quality include accuracy, fluency, adequacy, terminology consistency, and style and tone. These elements must come together to ensure that the translation is not just accurate, but also meaningful in its new linguistic context.
Automated evaluation metrics
We’ll start with the metrics that provide fast, repeatable ways to evaluate large volumes of data. These are called automated metrics, and we’re going to go through some of the most used ones:
BLEU (Bilingual Evaluation Understudy)
BLEU is one of the earliest and most widely adopted automated metrics for evaluating machine translation quality, being introduced by IBM researchers in 2002. It uses scores for n-gram precision, which is the degree of overlap between short sequences of words in the machine-generated translation and those in the reference translation. BLEU calculates how many of these n-grams in the candidate translation also appear in the reference. The more matches there are, the higher the BLEU score.
To account for the risk of a system gaming the metric by outputting overly short translations that only copy common words, BLEU includes a brevity penalty. This penalty is applied when the candidate translation is significantly shorter than the reference, discouraging systems from producing concise but incomplete outputs.
Together, n-gram precision and brevity penalty form the backbone of BLEU’s scoring mechanism, which results in a score between 0 and 1 (often reported as a percentage from 0 to 100). BLEU is particularly valuable as a baseline metric, but it alone is not sufficient.
TER (Translation Edit Rate)
TER is a metric developed to evaluate machine translation output by measuring how much editing a human would need to do to make a machine-generated translation acceptable. It calculates the minimum number of edits required to change a system’s output into a reference translation. These edits include insertions, deletions, substitutions, and shifts.
What sets TER apart is its emphasis on post-editing effort, which aligns closely with the needs of professional translators and editors. It doesn’t just measure whether words match, but rather how much real work is needed to turn a machine-generated translation into a polished, human-quality version.
Nonetheless, TER has its drawbacks too. Because it focuses on literal edits, it can sometimes overlook deeper semantic correctness. TER is more sensitive to stylistic variation, even when the meaning is preserved. Additionally, its reliance on a single reference translation can introduce bias, especially when multiple equally valid translations exist.
METEOR (Metric for Evaluation of Translation with Explicit ORdering)
METEOR was developed to address several of the shortcomings of BLEU. METEOR allows for synonym recognition, stemming (matching word roots), and flexible word order when comparing machine translations to reference texts.
This metric computes a harmonic mean of precision and recall, with recall given more weight—reflecting the idea that it’s more important for a translation to include all the necessary information (recall) than to avoid including extra material (precision). It also applies a penalty for fragmented word alignments, encouraging more cohesive translations.
However, METEOR has a few disadvantages too. The metric still depends on predefined synonym lists or language resources like WordNet, which limits its applicability in low-resource languages. Moreover, its more complex scoring process can make it slower to compute and less scalable for very large datasets.
chrF (CHaRacter-level F-score)
chrF was introduced in 2015, and it shifts the focus from word-level to character-level comparisons. Specifically, it uses the F-score (a harmonic mean of precision and recall) calculated over character n-grams, rather than word n-grams. This approach is particularly effective for morphologically rich languages where the form of a word can vary widely due to inflection, conjugation, or agglutination.
By analyzing translations at the character level, chrF can detect partial word matches and better capture subtle differences that word-based metrics like BLEU and METEOR might miss. It also avoids some of the sparsity issues that occur with word-level n-grams in smaller datasets or highly diverse corpora. Despite this, the metric does tend to give higher scores for translations that are technically close in form but awkward in phrasing.
COMET (Crosslingual Optimized Metric for Evaluation of Translation)
COMET is a step forward to evaluating AI translation quality because it uses neural networks trained on human judgments. It is is meaning-aware, so it evaluates how well the candidate translation conveys the same semantic content as the reference, even when phrased differently.
One of its strengths is its ability to scale across multiple language pairs and domains, thanks to its deep learning architecture. As neural evaluation becomes more mainstream, COMET is increasingly viewed as a state-of-the-art tool for high-stakes machine translation assessment.
BLEURT (Bilingual Evaluation Understudy with Representations from Transformers)
Like COMET, BLEURT leverages deep learning and pretrained language models, but it is fine-tuned specifically on datasets that include human evaluation scores. It can recognize when a translation preserves meaning, maintains fluency, and avoids significant errors.
BLEURT’s design makes it especially well-suited for fine-grained, sentence-level evaluation, a common scenario in user-facing applications or human-in-the-loop workflows. But on the other side, it is computationally intensive, and because it’s trained on specific evaluation datasets, it may not generalize perfectly to all language pairs or domains without further tuning.
Human evaluation methods
Automated metrics are continuously being improved, and we will definitely see new ones appear, too. However, the gold standard for determining AI translation quality remains human evaluation. Unlike algorithmic methods that rely on surface similarities or statistical patterns, human evaluators can assess translations in terms of meaning, fluency, context, cultural appropriateness, and even intent.
Direct assessment
In the direct assessment method, professional annotators or bilingual speakers are asked to rate a machine-translated sentence on a continuous scale (often 0–100), based on how well it conveys the meaning of the source sentence. This method is valued for its granularity and its strong correlation with user satisfaction, but the downside is that it’s time-consuming and expensive to conduct at scale.
Pairwise ranking
This is where human evaluators compare two or more translations of the same source sentence and rank them from best to worst. This method is often used to compare different machine translation systems or to benchmark improvements during development. Pairwise comparisons can be more intuitive for evaluators and less cognitively demanding than numerical scoring, but they may lack consistency.
Error annotation
When it comes to error annotation, human reviewers identify and categorize specific types of errors like mistranslations, omissions, and grammatical mistakes. This approach is useful for improving systems, as it highlights the kinds of problems that persist and offers insight into the system’s linguistic behavior.
Post-editing effort estimation
There is also post-editing effort estimation, which evaluates quality indirectly by measuring how much effort a human translator needs to correct a machine-generated translation. It can mean tracking the number of edits, the time taken, or the cognitive load experienced during revision. This method is most relevant in commercial translation workflows, where the efficiency gains from machine translation must be balanced with editing quality and speed.
Why it can be challenging to evaluate AI translation
Language is inherently flexible and context-dependent, so despite the advances in both automated metrics and human assessment, it can be hard to evaluate machine translation. Humans, for example, can be quite subjective. Even among professional translators, there can be significant disagreement on what constitutes a “good” translation.
Something that is quite challenging for automated metrics is domain sensitivity. A translation that is acceptable in casual conversation might not be in legal, scientific, or medical contexts. Automated metrics tend to be domain-agnostic, so they often miss the importance of domain-specific terminology, tone, and precision.
Then there’s the issue with evaluating low-resource languages. Many evaluation metrics were originally designed and tested on high-resource language pairs, and they work best with them. But when you apply them to low-resource languages or morphologically rich languages, these metrics may perform poorly. You might notice they fail to account for complex grammar rules, inflection, or word formation.
Wrapping up
As machine translation becomes more deeply embedded in our daily lives, we need to make sure its quality is up to par. Measuring AI translation quality is a complex field. While automated metrics are great tools for development and benchmarking, at the moment, they are far from perfect. There’s no doubt that human insight will remain indispensable in judging whether a translation truly succeeds.