From Text to Understanding the Inner Text: LLMs and Translation Accuracy and Fluency

Abstract

Evaluating translation quality is crucial to ensuring that Large Language Models (LLMs) meet the ambitious standards required for effective communication across languages. The key aspects of translation quality are accuracy; accuracy measures how well the translation reflects the meaning of the original text. It also measures the fluency based upon the naturalness and readability of the translation in the target language, both features play a crucial role in defining what smooth translation should appear to the prospective user(s). The present study, therefore, aims to measure these aspects of LLM-generated translation based on a corpus of LLM-translated texts. As this study is evaluative, it proposes a rigorous method to evaluate and improve the accuracy, fluency, and naturalness of LLM-generated translation. The study, therefore, analyses AI-generated translation texts based on these criteria. The secondary data set was collected from users of AI-based translation to provide further insights into the validity of LLM-based translation texts. By combining both real time translated texts generated by AI and users’ perspectives, this study arrived at results on the status and validity of LLM-based translation. The majority of the participants concurred that the translations retained the meaning of the original text, even the lower scores for processing idiomatic expressions and figurative language in LLMs still reflected a high level of semantic preservation, The high scores for grammatical correctness and sentence flow show that LLMs are perceived to be very good at generating translations that are grammatically correct and readable. Based on the findings, the study offers practical implications for future enhancement in AI-generated translation.