Why OnlyTranslate is not adding AI translation enhancement—for now
Better translation quality has always been one of OnlyTranslate's goals.
We also follow other translation products closely. More of them now offer “AI-enhanced translation,” “high-quality translation,” or “semantic rewriting.” Some introduce more page context, while others use different prompts to emphasize more natural writing.
We have wanted to add something similar. The reason we held back was not that we had missed the idea, but that our early attempts exposed a difficult problem: the results were volatile.
The same prompt could behave very differently when the model changed. Adding a few instructions might noticeably improve one model and make another worse. A title and surrounding paragraphs sometimes resolved ambiguity, but sometimes distracted the model. A handful of impressive examples could not tell us whether an approach was genuinely better.
As more competing products invested in this area, we decided to stop debating from impressions and run a proper evaluation.
First, we defined what “better” had to mean
Before testing, we defined the product boundary.
Semantic rewriting or a second polishing pass should not be the default for every paragraph. Most of the time, a reader wants to move through an article comfortably, not turn every translation into publication-ready prose. Calling a model again for every paragraph would materially increase latency, tokens, and API cost for an occasional improvement in wording.
The approach we wanted therefore had to meet all of these conditions:
- keep the default to one model call;
- add no new setting that readers must understand;
- work with general-purpose models rather than one specialist translation model;
- avoid a step change in tokens or latency;
- produce a substantial, consistent improvement across different models.
The last condition mattered most. If a new approach was only sometimes better, we would be adding complexity rather than dependable product value.
Experiment one: can more context make one-pass translation clearly better?
Hypothesis
Our first experiment began with an intuitive theory: perhaps the model mistranslated because it did not see enough information.
If we described the relationship between the page title, paragraph, and source text more clearly—and added stronger requirements for accuracy and natural writing—the model might reduce ambiguity and translationese without another call.
Design
We used the simple translation behavior from version 1.8.2 as baseline L, then prepared three candidates to separate the effects of context, prompt wording, and target position:
- A: structured context. Supply the page title, source text, and containing paragraph as separate fields.
- B: concise context with self-review. Shorten the prompt while asking the model to balance accuracy and natural writing, then silently check its output.
- C: inline target marking. Mark the source text in its original paragraph so the model can understand its position, references, and meaning.
Each approach produced standout translations and obvious failures in a small sample. To move beyond intuition, we expanded the test to:
- 11 real article sources, with 5 excerpts from each, for 55 test units;
- 3 general-purpose translation models;
- 4 approaches—L, A, B, and C—with the same temperature and request parameters;
55 × 3 × 4 = 660translation calls.
During evaluation, approach names were replaced with randomized labels. For each translation model, the other two models independently ranked its outputs, with ties allowed. We aggregated by article source so that five excerpts from one article would not be treated as five fully independent sources. We also reviewed disputed cases and tracked tokens, latency, major errors, and format failures.
Results and reflection
With 50% representing a tie with L, the overall results were:
| Approach | Relative score vs. L | Main problem |
|---|---|---|
| A: structured context | 53.0% | The 95% confidence interval still crossed 50%; average total tokens rose from about 100 to 234 |
| B: concise context with self-review | 50.0% | Direction reversed across models; the major-error rate reached 15.2% |
| C: inline target marking | 31.8% | Format failures reached 42.4%, with weaker models leaking markers or malformed output |
A score of 53.0% does not mean translation quality improved by 3%. It means A earned only slightly more comparison points than the 50% tie baseline, while the confidence interval could not rule out no real difference.
None of the first-round candidates met our requirement of a stable, substantial improvement over the existing translation.
A was the only approach with a positive trend, but the advantage was small while token use more than doubled. B showed that stronger natural-writing and self-review instructions did not generalize across models. C showed that a theoretically better context structure becomes a product failure if models cannot reliably follow its output protocol.
The most important conclusion was not that context had no value. It was that some sentences benefiting from context did not make context a reliable improvement for the default translation path.
Experiment two: would a more restrained use of context be more stable?
Hypothesis
We did not stop immediately. Perhaps the first experiment failed not because context was useless, but because there was too much of it, its structure was too complicated, or the prompt asked the model to do too many things at once.
The new hypothesis kept full paragraphs as the translation unit and reserved more targeted context for headings, selections, and isolated short text. The prompt returned to a simple one-call instruction: prioritize accuracy and do not add or omit information. This might preserve disambiguation benefits while reducing variation caused by complex instructions.
Design
The second experiment reused the same 11 sources, 55 excerpts, and 3 models, comparing only the baseline with the new candidate:
- every excerpt produced one baseline and one candidate translation, for 330 calls;
- those outputs formed 165 direct pairs;
- anonymized ranking and two independent judges were retained, with wins, ties, and losses worth 1, 0.5, and 0 points;
- we compared not only the overall score, but also direction across models, average quality ratings, and human reading impressions.
A score of 50% meant that the candidate and baseline were tied overall. A new default needed to move clearly above that line and in the same direction across models.
Results and reflection
The best second-round candidate produced these paired results:
| Result | Count | Share |
|---|---|---|
| Candidate better | 52 | 31.5% |
| Tie | 90 | 54.5% |
| Baseline better | 23 | 13.9% |
Counting a win as one point and a tie as half a point, the candidate achieved a 58.8% relative score—8.8 percentage points above the 50% tie line. This does not mean translation quality improved by 8.8%; it means the candidate earned more comparison points under this paired-scoring rule. Its average quality rating also increased by 0.23 points on a five-point scale. Numerically, this was the most promising result we had seen.
Yet normal reading and manual review still exposed the same underlying issue. Ninety pairs—more than half—had no meaningful difference. The same sentence could move in opposite directions on different models. The 23 regressions appeared as mistranslations, omissions, or awkward rewrites.
This separated two ideas that are easy to confuse: being better in aggregate is not the same as being consistently and visibly better in a product. A 58.8% score justified more investigation, but it did not prove that readers would experience reliably better translation during continuous reading.
Another hypothesis: improve only the paragraph a reader dislikes
Hypothesis
If always-on enhancement was too expensive, we developed another idea during the experiment: keep regular translation unchanged, then offer a “refine” or “improve translation” action for a paragraph the reader found unsatisfactory. This was not copied from an existing competitor feature; it was our own attempt to find a different balance between quality and cost.
It limited the extra call to a small number of paragraphs and, in theory, would not slow down the whole article.
What happened
It quickly introduced new problems:
- the model often produced a different wording rather than identifying and correcting a concrete issue;
- headings, cards, navigation, and non-article pages such as the GitHub home page offered no natural place for a refinement action;
- restricting the action to body paragraphs first required a reliable, high-precision definition of a body paragraph;
- even a structured “review first, then keep or revise” response still depended on an unstable model judgment.
Reflection
Adding a button is easy. Demonstrating that pressing it will usually improve the result is not. On-demand refinement reduced the number of extra calls, but it did not solve the core question of how to verify a genuine improvement. It also introduced new complexity around placement and body-content detection.
Our answer today: keep regular translation stable, use selection for difficult text
We ultimately did not turn AI enhancement into a default feature, and we did not keep a “refine” action whose effect remained uncertain.
The simpler approach better matches how people actually read:
- Regular page translation continues to prioritize stability and speed. The page title is weak context only, and the model may output only the requested source text.
- When a translation is unclear or unsatisfactory, use selection translation. A reader can select only the word, phrase, or sentence in question, allowing the model to interpret it within its local paragraph context.
- Selection results have a scope guard. If the model returns an entire paragraph by mistake, OnlyTranslate neither displays nor caches that result and automatically retries once without the paragraph context.
Selection translation is not a renamed “high-quality mode.” Its advantage is that the target is explicit, the reader triggers it deliberately, and the extra work is spent only where understanding is actually needed. That is a better quality, performance, and cost trade-off than continuously expanding prompts or calls across every article and every paragraph.
We will keep watching, but it is not a near-term priority
We will continue following advances in models, prompts, and context techniques, and revisit new approaches when the time is right. Based on our results so far, however, AI translation enhancement will not be a near-term priority for OnlyTranslate. We will focus instead on improvements that are more stable, more clearly defined, and easier for readers to notice.