Neural machine translation (NMT) and large language models (LLMs) achieve high overall corpus-level translation quality, yet they systematically mistranslate non-compositional multiword expressions — idioms whose meaning cannot be derived from their parts — and other rare-sense phenomena, generating literal, word-by-word renderings even when the phenomenon occurs in training data. Formulated in Marzieh Fadaee's 2021 thesis and left open there, the problem has two jointly supporting parts: constructing large-scale, high-quality, idiom-annotated parallel corpora and targeted evaluation sets with idiom-sensitive metrics, and equipping translation models with external knowledge or inductive biases so that idiomatic expressions are translated with their correct figurative meaning, while evaluating and limiting adverse effects on general translation quality. Prior work (2018-2026) has made substantial progress on both parts — dedicated datasets for several language pairs, retrieval augmentation, idiom-aware training losses and knowledge bases, and idiom-specific metrics — yet the gap persists: as of 2025 the best LLM systems still err on about 28% of Chinese idiom instances, and automatic metrics correlate poorly with human judgments of idiom translation quality. The problem at its stated generality remains open.

Background

Neural machine translation (NMT) trains a neural network end-to-end to map a source sentence to a target sentence, and is typically scored with BLEU, an n-gram overlap metric between the system output and human reference translations. Idioms are non-compositional multiword expressions: multiword expressions are fixed combinations of words (e.g. phrasal verbs, idioms), and an expression is non-compositional when its meaning is not a function of the meanings of its parts — 'kick the bucket' means 'to die', not anything about buckets. Because of this, translating an idiom literally (word-by-word) usually yields a wrong or nonsensical translation, and the correct rendering is often a completely different expression in the target language. Standard parallel training corpora contain few idiom instances, and corpus-level BLEU does not reflect whether the idiom itself was translated correctly. The problem was characterized systematically in the 2021 PhD thesis of Marzieh Fadaee ('Understanding and Enhancing the Use of Context for Machine Translation', published on arXiv as 2102.10437; the idiom chapter appeared earlier as Fadaee, Bisazza and Monz, LREC 2018, arXiv:1802.04681). Chapter 6 of the thesis builds the first large-scale German-English idiom translation dataset by automatically extracting idiom-containing sentence pairs from WMT training corpora against a dictionary gold list, and shows that commercial systems (DeepL, GoogleNMT) and research NMT models all fail on sample idioms; that NMT models with higher overall BLEU score lower on word-level idiom accuracy (the fraction of test sentences where the idiom is translated with its idiomatic meaning); that phrase-based statistical MT, despite worse general quality, translates idioms better than NMT; and that giving the model an explicit 'this sentence contains an idiom' flag during training improves idiom accuracy while slightly reducing BLEU. The thesis concludes that overall BLEU and localized idiom accuracy are largely uncorrelated, and its future-work section states that exclusively data-driven models fail to capture infrequent linguistic nuances such as idiomatic expressions, naming the construction of adequately large, high-quality, phenomenon-targeted datasets as the route to making progress measurable. Later work deepened the diagnosis: Dankers, Lucas and Titov (ACL 2022, arXiv:2205.15301) showed that the tendency of the Transformer (the attention-based neural network architecture underlying modern translation models) to process idioms compositionally (analyzing them word-by-word rather than as single lexical units) contributes to literal mistranslations; Baziotis, Mathur and Hasler (arXiv:2210.04545) introduced an automatic metric for literal-translation errors and found that monolingual pretraining yields substantial targeted idiom gains even for unseen idioms, while confirming idiom translation as a major open problem. On the mechanism side, Zeng and Bhat (TACL 2022, DOI 10.1162/tacl_a_00510) injected idiom-awareness into pretrained language model representations; Li et al. (arXiv:2308.13961) built IdiomKB, a knowledge base of idioms used to guide language models; Liu, Chaudhary and Neubig (arXiv:2310.07081) combined up-weighting of idiom-containing training sentences with retrieval augmentation (attaching similar annotated examples to the input), improving idiom accuracy of a strong pretrained MT model by up to 13 absolute points on French, Finnish and Japanese. On the resource side, dedicated idiom-translation datasets now exist for Chinese-English (PETCI, Tang, arXiv:2202.09509), Persian-English (Rezaeimanesh, Hosseini and Yaghoobzadeh, arXiv:2412.09993), and Chinese (IdiomEval, Yang et al., arXiv:2508.10421, which also contributes an error taxonomy for idiom translation). Yet evaluations from the era of large language models (LLMs) show the gap is not closed: IdiomEval (2025) finds that the best system (GPT-4) still makes errors on 28% of Chinese idiom cases and that existing automatic metrics correlate poorly (Pearson below 0.48) with human ratings of idiom translation quality; a 2025 dataset survey (Matheny et al., arXiv:2511.16345) states that idioms and figurative language 'continue to elude LLMs' and that better and larger datasets are still needed; and 2026 work that fine-tunes models with rewards from machine-translation quality estimation (MTQE) models, which score translation quality without needing reference translations (Agarwal, He, Patil and Hakkani-Tür, arXiv:2601.06307), still treats the non-compositional translation gap as an open target, improving Chinese and Hindi idiom translation by roughly 14 points.

Problem Statement

How can neural translation systems be made to translate non-compositional multiword expressions — idiomatic expressions and comparably rare, non-literal senses — correctly, in the full generality of natural language rather than for a single language pair or model? Concretely, as posed by Fadaee's 2021 thesis and pursued by the subsequent literature, the problem has two jointly supporting parts: (1) construct large-scale, high-quality parallel corpora and targeted evaluation sets in which sentences containing these phenomena are identified, together with idiom-sensitive evaluation metrics that reliably reflect human judgment of idiom translation quality, so that the phenomenon is learnable and measurable; and (2) devise mechanisms — external knowledge sources such as idiom inventories, definitions, or retrieved annotated examples, or inductive biases such as idiom-aware training signals, losses, or representations — such that translation models render idiomatic expressions with their correct figurative meaning rather than literal word-by-word renderings, while evaluating and limiting adverse effects on general translation quality on ordinary compositional text. An answer should address both parts — constructing or extending the targeted resources (idiom-annotated parallel corpora, targeted evaluation sets, and idiom-sensitive evaluation metrics) and devising the mechanism — and should demonstrate, on idiom-specific benchmarks spanning more than one language pair as the minimum demonstration scope (the objective itself remains the full generality of natural language, not the demonstrated language pairs), that the measured gap between idiom-targeted accuracy and general translation accuracy is substantially closed relative to the documented baselines, meaning materially beyond the levels those baselines already report: the 28% error rate of the best 2025 LLM systems on Chinese idioms and the roughly 13-14-point improvements reported by 2023-2026 mechanisms are what current methods already achieve, so reproducing them does not count as closure; it should also measure and report the effect of the proposed mechanism on general translation quality (e.g. BLEU, or COMET — a learned semantic-similarity metric for translation quality — on standard non-idiomatic test sets); consistent with the primary source's idiom-flag result and Liu et al. (2023, §7.1: a 4% accuracy decrease on the Japanese random test set for the best method, still presented as a success), a modest trade-off against general quality may be acceptable, and strict non-degradation of general quality is not a precondition for closure.

References

  1. Marzieh Fadaee. Understanding and Enhancing the Use of Context for Machine Translation. PhD thesis, 2021. arXiv:2102.10437, DOI 10.48550/arXiv.2102.10437, https://arxiv.org/abs/2102.10437
  2. Marzieh Fadaee, Arianna Bisazza, Christof Monz. Examining the Tip of the Iceberg: A Data Set for Idiom Translation. LREC 2018. arXiv:1802.04681, DOI 10.48550/arXiv.1802.04681, https://arxiv.org/abs/1802.04681
  3. Verna Dankers, Christopher G. Lucas, Ivan Titov. Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation. ACL 2022. arXiv:2205.15301, DOI 10.48550/arXiv.2205.15301, https://arxiv.org/abs/2205.15301
  4. Christos Baziotis, Prashant Mathur, Eva Hasler. Automatic Evaluation and Analysis of Idioms in Neural Machine Translation. 2022. arXiv:2210.04545, DOI 10.48550/arXiv.2210.04545, https://arxiv.org/abs/2210.04545
  5. Ziheng Zeng, Suma Bhat. Getting BART to Ride the Idiomatic Train: Learning to Represent Idiomatic Expressions. TACL 2022. DOI 10.1162/tacl_a_00510, arXiv:2207.03679, https://arxiv.org/abs/2207.03679
  6. Kenan Tang. PETCI: A Parallel English Translation Dataset of Chinese Idioms. 2022. arXiv:2202.09509, DOI 10.48550/arXiv.2202.09509, https://arxiv.org/abs/2202.09509
  7. Shuang Li, Jiangjie Chen, Siyu Yuan, Xinyi Wu, Hao Yang, Shimin Tao, Yanghua Xiao. Translate Meanings, Not Just Words: IdiomKB's Role in Optimizing Idiomatic Translation with Language Models. 2023. arXiv:2308.13961, DOI 10.48550/arXiv.2308.13961, https://arxiv.org/abs/2308.13961
  8. Emmy Liu, Aditi Chaudhary, Graham Neubig. Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting. 2023. arXiv:2310.07081, DOI 10.48550/arXiv.2310.07081, https://arxiv.org/abs/2310.07081
  9. Sundesh Donthi, Maximilian Spencer, Om Patel, Joon Doh, Eid Rodan, Kevin Zhu, Sean O'Brien. Improving LLM Abilities in Idiomatic Translation. 2024. arXiv:2407.03518, DOI 10.48550/arXiv.2407.03518, https://arxiv.org/abs/2407.03518
  10. Sara Rezaeimanesh, Faezeh Hosseini, Yadollah Yaghoobzadeh. Large Language Models for Persian ↔\leftrightarrow English Idiom Translation. 2024. arXiv:2412.09993, DOI 10.48550/arXiv.2412.09993, https://arxiv.org/abs/2412.09993
  11. Cai Yang, Yao Dou, David Heineman, Xiaofeng Wu, Wei Xu. Evaluating LLMs on Chinese Idiom Translation (IdiomEval). 2025. arXiv:2508.10421, DOI 10.48550/arXiv.2508.10421, https://arxiv.org/abs/2508.10421
  12. Blake Matheny, Phuong Minh Nguyen, Minh Le Nguyen, Stephanie Reynolds. NLP Datasets for Idiom and Figurative Language Tasks. 2025. arXiv:2511.16345, DOI 10.48550/arXiv.2511.16345, https://arxiv.org/abs/2511.16345
  13. Ishika Agarwal, Zhenlin He, Dhruva Patil, Dilek Hakkani-Tür. A Rising Tide Lifts All Boats: MTQE Rewards for Idioms Improve General Translation Quality. 2026. arXiv:2601.06307, DOI 10.48550/arXiv.2601.06307, https://arxiv.org/abs/2601.06307