تقييم جودة الترجمة المدعومة بالذكاء الاصطناعي: رواية “العقل والعاطفة”
لجاين أوستن أنموذجا
[1]۞Hajir Waheedah
[2]۞Mohammad Babchikh
ملخّص: تسعى هذه الدراسة إلى استكشاف التباينات بين جودة الترجمة البشرية والترجمة المدعومة بالذكاء الاصطناعي للنصوص الأدبية، من خلال المقارنة بين ترجمة أمين الشريف ومخرجات تشات جي بي تي 3.5 إلى العربية لمقاطع من رواية “العقل والعاطفة” لجاين أوستن، بعد ترجمتها من دون أي تحرير قبلي أو بعدي. ينطلق البحث من تحليل شامل للجوانب اللغوية والأسلوبية والدلالية والهيكلية للترجمتين، ثم يستخدم معيار التقييم ثنائي اللغة “بلو” لتقييم جودة الترجمة آليا. وقد أظهرت النتائج أن الترجمة البشرية كانت أفضل من حيث الأسلوب وفي نقل المعنى. وعلاوة على ذلك، تبين من البحث أنه على الرغم من أن الترجمة بالذكاء الاصطناعي تحافظ على بنية النص المصدر، إلا أنها تفشل في إنتاج نص يحافظ على السمات الأسلوبية والدلالية للنثر الأدبي.
الكلمات المفتاحيّة: الترجمة الأدبية؛ الترجمة الآلية المدعومة بالذكاء الاصطناعي؛ الترجمة البشرية؛ تقييم الجودة.
Abstract: This research explores the disparities between the quality of human and AI-powered translations of literary texts. It compares the output of ChatGPT3.5 into Arabic of passages from Austen’s “Pride and Prejudice”- translated without any pre-editing or postediting- to the reference translation by Amin Al-Sharif. It departs from a comprehensive analysis of the translations’ linguistic, stylistic, semantic, and structural aspects. The subsequent BLEU Score measurement shows that the Human translation was better in and conveying meaning. Moreover, although the AI- translation conserves the structure of the Source Text, it fails to produce a text that preserves the stylistic and semantic features of literary prose.
Keywords: Literary translation, AI-powered Machine Translation, Human translation, Quality assessment.
Résumé :Cette article explore les disparités entre la qualité de la traduction humaine et celle générée par l’intelligence artificielle de textes littéraires. L’étude compare la sortie de ChatGPT3.5 en arabe de passages choisis du roman de Jane Austen « Pride and Prejudice » et traduits sans aucune préédition ni postédition, à la traduction référence par Amin Al-Sharif. À la suite d’une analyse globale des aspects linguistiques, stylistiques, sémantiques et structurels des traductions, les phrases sélectionnées ont été ensuite soumises à une évaluation automatique à l’aide du BELU score. Les résultats de cette dernière analyse indiquent que la traduction humaine était meilleure en termes de style et de transfert de sens. De plus, bien que la traduction générée à l’aide de l’IA conserve la structure du texte source, elle ne parvient pas à produire un texte qui préserve les caractéristiques stylistiques et sémantiques de la prose littéraire.
Mots clés : Traduction littéraire, traduction automatique par l’IA, traduction humaine, évaluation de la qualité.
- Introduction: While Machine Translation has made strides since its inception in the 1950s, it continues to grapple with the accurate interpretation of context and meaning across various languages. Some subtleties of language, for instance, idiomatic expressions and culturally weighty words and phrases, constitute significant challenges for algorithms to decipher and convey in the target language. Besides, some low-resource languages characterised by their complex syntactic features, such as Arabic or Japanese, could present a difficult task for Machine Translation systems when used to translate some types of text. The quality of their output is a corollary of the data they used to train them, the limitations of the algorithms employed in the translation process and the techniques they rely on. Moreover, a thorough understanding of language, culture, and context is essential in some text types. Consequently, translators require a high level of language mastery and comprehension, a deep analysis of context, excellent knowledge of figurative expression, a complete understanding of the grammar of the source and target languages, and literary creativity. All these features should be taken into account when planning to translate literature.
Many scholars have addressed the deficiencies of machine translation systems, while others have greeted the significant progress they have achieved over decades. The technological breakthroughs have allowed them to even human translators in some contexts, particularly after their endowment with Artificial Intelligence. For instance, Ghazvininejad M. et al. (2016) assume that Machine Translation is daunting. However, they optimistically claim that high-quality Machine Translation is achievable. They also stress the importance of leveraging Machine Learning and Artificial Intelligence methodologies to enhance the quality and precision of translations. Moreover, their study concludes that neural networks can select rhyming words to preserve the structure of poems. They also expect that neural networks and Deep Learning shall progress to be able to convey the elements of structure and rhetoric of poetry, maintain syntax and meaning and raise creativity.
In this connection, this study seeks to investigate the discrepancies between the output of ChatGPT3.5 and a human translation by Amin Al-Sharif of Jane Austen’s ‘Sense and Sensibility’. For this experiment, the researchers translated ten carefully chosen sentences using ChatGPT-3.5 and then compared them to the reference and the original English text.
- Literature Review: Machine Translation, particularly its most AI-powered current version, is generating increased attention in academia. Researchers across the globe are looking at the possibilities and limitations of incorporating this leading-edge technology into the translation profession, with a particular focus on different types of texts, including prose and poetry. In their insightful paper, Kolb et al. (2023) emphasised the centrality of human translation in dealing with occasional words in literary texts. To look at the way human translators and MT systems, notably Google Translate and DeepL, deal with such lexical features, they examined the English translation of “Der Talisman”, a play by Johann Nestroy. Their study showed that MT systems successfully generated new words. The conclusions of this research lay down the chances for human translators to use these MT systems to translate literary coinages and take advantage of this cooperative approach between human and machine translations.
Another study by Toral (2018) assessed the efficiency of Neural Machine Translation (NMT) in rendering literary texts, particularly novels. The study explored the translation from English to Catalan and used the BLEU metrics mainly associated with Statistical Phrase-Based Machine Translation (PBSMT) to assess quality. The researcher concluded that NMT systems steadily outpaced the PBSMT MT systems across all he studied. Overall, NMT showed a marked 11% relative improvement compared to PBSMT. Besides, supplementary human assessments indicated that native speakers of the target language perceived between 17% and 34% of the NMT-generated translations (depending on the specific novel) to be on par with translations produced by human translators in quality. The research pointed out the potential to achieve high-quality automated translations for literary texts using Neural Machine Translation techniques, enhancing Machine Translation’s performance in this domain.
Another study by Yirmibeşoğlu et al. (2023) assessed the translation of literary texts from English to Turkish while developing Machine Translation models that preserve the stylistic features of human translators. The evaluation process involved manual assessment of the Machine Translation models for literary texts and manual analysis. The results indicated that human translators should play a crucial role in recreating the stylistic elements in machine translations of literary texts. The research underscored the importance of human involvement in adapting machine translation to maintain the literary style effectively.
This research distinguishes itself from prior studies by dealing with the automatic translation of literary text from English into Arabic using the AI-powered MT tool ChatGPT 3.5. For example, the study compares human and Machine Translations to the original text to identify vocabulary, sentence structures, literary techniques, and more disparities. Based on the collected and analysed results, the research has the potential to clarify the automated outputs of artificial intelligence translation for literary texts. Additionally, it contributes to comprehending the stylistic, syntactic, and cultural considerations that should be considered when translating such texts automatically, without human intervention.
It goes without saying that translators must maintain awareness of other influential factors in their translations, including the cultural nuances of the target language. Translators should acknowledge that in literary translation, their primary objective is to convey abstract concepts, with words serving as constraining elements. Only in instances where the meaning in literary translation remains obscure despite exhaustive consultations of all accessible resources should translators be permitted to translate words rather than concepts (Newmark, 1980: 135).
Based on the above, the main objective of this research is to examine the impact of utilising Artificial Intelligence-based Machine Translation tools to translate literary texts from English into Arabic by evaluating quality and alignment with the original author’s message and style. The study further pursues the following secondary objectives to enhance its significance and understand the expected benefits of leveraging Artificial Intelligence programs in the field of literary translation:
- Objectives
This paper aims to assess the accuracy of Artificial Intelligence Translation in effectively conveying the author’s message and stylistic details. It also evaluates the capability of AI-powered Machine Translation to align with the writer’s style and convey their message effectively as human translation and the extent of that alignment. Moreover, it measures the performance of this new machine translation technology compared to human translation using the BLEU score metric.
- Research Questions:To address the research objectives outlined above, the researchers formulated the following research questions:
1-What discernible differences exist between Artificial Intelligence and human translation in the context of literary translation?
2-Which translations are closer to the target language and convey the author’s message while maintaining the tone and literary sense of the original work?”
3- What are the differences in structure, style, and meaning observed through human analysis of the translations by El-sharif and GPT3.5?
4-Can AI-powered Machine Translation outperform human translation when measured using BLEU score metrics?
- Methodology:This research adopts a comparative analysis approach, examining artificial intelligence translation through ChatGPT and human translation, comparing both with the original text. We selected ten sentences from Jane Austen’s novel “Sense and Sensibility” because it is the first novel by an English author and was well-received by the public. One of its Arabic translations is by Amin Al-Sharif. A comprehensive comparison is conducted to evaluate the structure, form, vocabulary, meaning, and stylistic elements of the human translator’s work and ChatGPT translation, extracting the percentage of equivalence with the original text. We relied on Julian House’s 1979 functional equivalence scale to measure the quality of semantic style, meaning, and syntactic structure. Based on Chomsky’s theory, which emphasises the surface level related to the syntactic structure of the sentence as it appears in speech or writing, and the deep level related to the basic structure of the sentence, which determines its meaning, differences can be measured based on the ability of the translation to preserve the deep and surface structure of the original sentence. This is particularly challenging in the case of languages that differ significantly in syntactic structure, such as Arabic. However, this can lead to more accurate translations in terms of meaning. This comprehensive evaluation also benefits from the BLEU metric to measure machine translation quality and calculate the degree of agreement with the original text. This allows us to extract satisfactory results.
5.1 Samples Selection:The researchers selected the sample sentences according to their type, such as simple, compound, complex, compound-complex, interrogative, and exclamatory sentences, and how they are used to convey meaning and style. Besides, the selection relies on their relevance to the study’s objectives. The sentences depict the novel’s main characters, themes, and conflicts, such as the contrast between sense and sensibility, the inheritance of the Dashwood estate, the love and marriage of Elinor and Marianne, and the influence of Fanny and Willoughby. The sentences showcase the use of literary devices, such as irony, foreshadowing, characterisation, dialogue, and description, and how these tools enhance the plot and mood of the novel, as the following sentences show:
1- Main conflict and theme, complex sentence. 2- Motive and situation of Mr. Dashwood, simple sentence. 3- Difference and relationship between Elinor and Marianne, interrogative sentence. 4- Background information of Edward Ferrars, compound-complex sentence. 5- Prejudice and admiration of Marianne, simple sentence. 6- Grief and attachment of Marianne and Margaret, exclamatory sentence. 7- Philosophy and attitude of Marianne, complex sentence. 8- Remark of Mrs. Jennings, compound sentence.
|
Table (1) ST: source Text, HT: El-Sharif’s translation, GPT3.5 T: Chat GPT3.5 translation, T D GPT3.5 T: Time and date Chat GPT3.6 translation. |
|||||||||||||||||||||||||||||||||||||||||||||
|
Table (1) The sample sentences and their translation by the Human translator and ChatGPT 3.5
- 2 Data Analysis
Firstly, the human evaluation relies on Chomsky’s theory (2006), which emphasises the importance of meaning translation; Chomsky believes that natural languages have a common deep structure called Universal Grammar, which is a set of rules and principles that govern the structure and acquisition of language in humans. According to his theory, the deep structure can be transformed into different surface structures that express the same meaning in different languages. Therefore, languages can be compared in translation based on the degree of matching the deep structure and the meaning between the source text and the target text. The quality of the translation can be evaluated based on the preservation of the source text’s grammatical, structural, stylistic, and semantic aspects in the target text. We utilise Juliane House’s (1997) model for evaluating the quality of translations, assessing how closely the target text aligns with the source text on different levels and achieves functional equivalence in a pragmatic context. This model allows us to analyse translations in relation to style, structure, and meaning. It is worth mentioning here that this quality assessment model is quantitative, relying on the evaluator’s judgement.
With this said, the researchers shall examine the several textual features of both the source text and the two translations, relying on the methodology suggested by House, indicating the commonalities and disparities. The comparative assessment will then be turned into numerical values represented in percentages. The last step would be incorporating human judgment into the assessment. The human and the AI translations shall be evaluated according to the analyses regarding their linguistic features, i.e. linguistic precision, differences in structure and adequate style. Moreover, the researchers shall assess their faithfulness to the message of the original text. Overall, this thorough quality assessment is both automatic and manual, based on human discernment.
The first stage, consisting of linguistic analysis based on House’s TQA, is represented in the following sequences:
Sentence one: The first part of the sentence is: “he meditated within himself to increase the fortunes of his sisters by the present of a thousand pounds apiece.”. The additional part is: “when he gave his promise to his father,”. This second part supplies more information about the timing of the action. The sentence adheres to the subject-verb-object order and employs a conjunction like “when” to link the sentence parts and word groups.
El-Sharif’s translation comprises two distinct parts, each capable of standing independently. These parts are connected by the term “ولما” (and when). The first segment is “ولما وعد أباه بمساعدة اخواته” (when he promised his father to assist his sisters). The second segment is “نوى في دخيلة نفسه أن يزيد ثروتهن بأن يهب كلا منهن ألف جنيه” (he intended within himself to increase their fortune by gifting each of them a thousand pounds). The sentence adheres to the verb-subject-object order and employs words like “أن” (to) and “بأن” (that) to link the parts and word groups.
The GPT3.5 translation has two parts: The main part is “تأمّل في نفسه لزيادة ثروة شقيقاته بمبلغ ألف جنيه لكل واحدة منهن”. The extra part is “عندما قدم وعداً لوالده” and it adds more information about the time of the action. The sentence follows the order of verb, subject, and object and uses a word like “عندما” (when) to link the parts and the groups of words.
Structural Comparison: The original English sentence follows the SVO: “When (Subject-he) gave (Verb-gave) his promise (Object-promise) to his father, he meditated within himself to increase (Verb-increase) the fortunes (Object-fortunes) of his sisters by the present of a thousand pounds apiece.” Both translations (El-sharif’s and GPT3.5) predominantly follow the SVO word order, similar to the original sentence. Therefore, regarding word order, both translations are relatively close to the original English sentence.
Stylistic Comparison: The original text, the human translation, and GPT3.5 use a narrative style, meaning they tell a story or describe an event. The texts use different sentence structures, word orders, and punctuation marks to convey the character’s actions and intentions. They also use a third-person point of view and an omniscient narrator to tell the story. The texts are not written in a style of praise, lament, emotion, satire, or news.
Semantic Comparison: The human translation sounds more natural and fluent in Arabic, as it uses different ways of saying and writing the sentence. El-sharif’s translation faithfully adheres to the original text’s order, progressing from promise to contemplation. In contrast, the “GPT3.5” translation, adhering to the sequential structure, emphasises internal reflection more and introduces a temporal context, implying that the promise emerged after contemplation.
Sentence Two: The English text is a complex sentence with one independent clause and one dependent clause. The independent clause is: “It was my father’s last request to me”, and the dependent clause is: “replied her husband”. The subject of the independent clause is “it”, and the predicate is “was my father’s last request to me”. The predicate includes a possessive noun phrase, “my father’s last request”, and a prepositional phrase “, to me”. The subject of the dependent clause is “her husband”, and the predicate is “replied”. This sentence structure, while complex, provides a rich context and conveys detailed information.
El-Sharif’s translation is a compound sentence with two independent clauses linked by the conjunction “أن” (that). The first clause is “فأجاب زوجها”, and the second is “لقد كان آخر طلب تقدم به أبي إلي أن أساعد أرملته وبناته”. The subject of the first clause is “زوجها”, and the predicate is “فأجاب”. The predicate includes the prefix “ف” (and), indicating a sequence of events. The subject of the second clause is implied to be the same as the first clause. The predicate is “لقد كان آخر طلب تقدم به أبي إلي أن أساعد أرملته وبناته”. The predicate contains a particle “لقد” (indeed) indicating past tense, a verb “كان” (was), a noun phrase “آخر طلب”, a relative clause “تقدم به أبي إلي”, and a subordinate clause “أن أساعد أرملته وبناته”. The relative clause modifies the noun phrase, and the subordinate clause, introduced by the conjunction “أن”, expresses the content of the request.
The GPT3.5 translation is a straightforward sentence with a single clause. The clause’s subject is “كانت”, and the predicate is “آخر طلب من والدي أن أساعد أرملته وابنته”. The predicate includes the verb “كانت”, a noun phrase “آخر طلب”, a prepositional phrase “من والدي”, and a subordinate clause “أن أساعد أرملته وابنته”. The prepositional phrase denotes the request’s origin, and the subordinate clause, introduced by the conjunction “أن”, expresses the request’s content.
Structural Comparison: The English text is a complex sentence with a verb-introduced dependent clause at the end, following a (SVO) word order. The human translation is a compound sentence with two independent clauses joined by a comma and a conjunction, following a (VSO) word order. The GPT3.5 translation is a simple sentence with a single independent clause starting with a direct object (كانت آخر طلب من والدي). The human translation maintains the sequence of events and the emphasis on the request, while the GPT3.5 translation simplifies the sentence and omits some details.
Stylistic Comparison: The human translation aligns more closely with the original English text in terms of meaning and style. The style is a dialogue and a narration, indicating it presents the characters’ direct speech and narrates the story. The GPT3.5 translation is more concise and informal but changes some details and expressions from the original text.
Semantic Comparison: The human translation is more faithful and accurate to the original text. The GPT-3.5 chat translation was concise and informal. It made a mistake in translating the past verb “كانت” (was), as it added the feminine “ت” to the verb, changing the meaning for the reader from masculine to feminine.
Sentence three: The sentence in English is a compound sentence consisting of two independent clauses connected by a semicolon and the coordinating conjunction “but”. The first clause is: “You look grave, Marianne” and the second clause is: “Do you disapprove your sister’s choice?”. The subject of the first clause is: “you”, and the predicate is: “Look grave, Marianne”. The predicate contains the verb “look” and the adjective “grave” that modifies the subject. The predicate also contains the noun “Marianne”, which acts as a vocative or a direct address. The subject of the second clause is: “you”, and the predicate is: “Do you disapprove of your sister’s choice?”. The predicate contains an auxiliary verb “do” and a verb “disapprove”, which form an interrogative sentence. The verb is followed by a possessive noun phrase, “your sister’s choice”, which acts as the verb’s object.
El-sharif’s translation is a simple sentence that consists of one independent clause. The subject of the clause is omitted and implied to be “you”. The predicate is “ولكن مإلي أراك ساهمة الوجه يامريان! ألا توافقين على اختيار اختك؟”. The predicate contains coordinating conjunction“و”(and) that indicates a contrast, conjunction “لكن” (but) that introduces a concession, a noun phrase “مالي” that expresses a rhetorical question, a verb “أراك” that agrees with the subject in person, number, and gender, an adjective “ساهمة” that modifies the subject and agrees with it in number and gender, a noun phrase “الوجه” that acts as an adverbial modifier of the verb, a noun “مريان” that acts as a vocative or a direct address, an exclamation mark that indicates a pause, a particle “ألا” that introduces a negative question, a verb “توافقين” that agrees with the subject in person, number, and gender, a preposition “على” that introduces the object of the verb, and a noun phrase “اختيار اختك” that acts as the object of the preposition.
The GPT3.5 translation is a compound sentence that consists of two independent clauses connected by a comma and a coordinating conjunction “لكن”. The first clause is “لكنك تبدين جدية“, and the second clause is “ماريان؛ هل تستنكرين اختيار أختك؟”. The subject of the first clause is “ك“, and the predicate is “تبدين جدية”. The predicate contains a verb “تبدين” that agrees with the subject in person, number, and gender, and an adjective “جدية” that modifies the subject and agrees with it in number and gender. The subject of the second clause is “ك”, and the predicate is “ماريان؛ هل تستنكرين اختيار أختك؟”. The predicate contains the noun “ماريان” as a direct address, a semicolon indicating a pause, the particle “هل” introducing an interrogative sentence, the verb “تستنكرين” agreeing with the subject in person, number, and gender, and the noun phrase “اختيار أختك” acting as the object of the verb.
Structure Comparison: The English text uses a compound sentence with two independent clauses joined by a semicolon and a conjunction, following a subject-verb-object (SVO) order. The human translation follows a (VSO) order. The GPT3.5 translation uses a compound sentence with two independent clauses joined by a comma and a conjunction, following an SVO order.
Stylistic Comparison: The human translation aligns more closely with the original English text. The style is dialogic and narrative, showing the characters’ direct speech and narrating the story. The human translation is more expressive and rhetorical than the original text, while the GPT3.5 translation is more concise and direct.
Semantic Comparison: The human translation has added some unnecessary wording and phrasing that deviates slightly from the original text. The phrase “مإلي أراك ساهمة الوجه يا مريان” might be unclear and could cause some confusion. In general, it is preferable to stick with a translation that seeks to maintain the meaning accurately, providing a smooth reading experience and easy comprehension for the Arabic reader. The GPT 3.5 translation remains faithful to the original text and strives to preserve the precise meaning with simple and clear words.
Sentence four: The English sentence consists of the subject of the clause being “the son”, and the predicate is: “a steady respectable young man, was amply provided for by the fortune of his mother, which had been large, and half of which devolved on him on his coming of age”. The predicate contains a noun phrase: “a steady respectable young man”, that modifies the subject, a verb “was” that agrees with the subject in number and person, an adverb “, amply” that modifies the verb, a verb phrase “provided for” that expresses the action of the verb, a prepositional phrase “by the fortune of his mother” that acts as the agent of the verb phrase, a relative clause “which had been large” that modifies the noun “fortune”, and a coordinate clause “and half of which devolved on him on his coming of age” that adds more information about the noun “fortune”.
El-sharif’s translation consists of: The subject of the clause is “وكان الولد” and the predicate is “شابا رزينا مبجلا، تركت له أمه وكانت ذات ثروة كبيرة، مبلغا من المال. واّل إليه نصف هذا المبلغ عندما بلغ سن الرشد”. The predicate contains an adjective “شابا” that modifies the subject and agrees with it in number and gender, an adjective “رزينا” that modifies the subject and agrees with it in number and gender, an adjective “مبجلا” that modifies the subject and agrees with it in number and gender, a comma that indicates a pause, a verb “تركت” that agrees with the subject in number and person, a noun phrase “له أمه” that acts as the object of the verb, a verb “وكانت” that agrees with the subject in number and person, an adjective “ذات” that modifies the subject and agrees with it in number and gender, a noun phrase “ثروة كبيرة” that acts as the complement of the adjective, a comma that indicates a pause, a noun “مبلغا” that acts as the subject of the following verb, a noun phrase “من المال” that acts as the complement of the noun, a dot that indicates the end of the sentence, a prefix “و” that indicates a sequence of events, a verb “اّل” that agrees with the subject in number and person, a prepositional phrase “إليه” that acts as the object of the verb, a noun phrase “نصف هذا المبلغ” that acts as the complement of the preposition, a prepositional phrase “عندما بلغ” that introduces the time of the action, and a noun phrase “سن الرشد” that acts as the object of the preposition.
The GPT3.5 translation consists of the subject of the clause “الابن” and the predicate “شاب محترم ومستقيم، تم توفير احتياجاته بشكل كافي من خلال ثروة والدته، التي كانت كبيرة، ونصفها انتقل إليه عندما بلغ سن الرشد”. The predicate contains a noun “شاب” that modifies the subject and agrees with it in number and gender, an adjective “محترم” that modifies the subject and agrees with it in number and gender, a conjunction “و” that introduces a coordination, an adjective “مستقيم” that modifies the subject and agrees with it in number and gender, a comma that indicates a pause, a verb “تم” that expresses a passive voice, a verb phrase “توفير احتياجاته” that acts as the object of the verb, an adverb “بشكل كافي” that modifies the verb phrase, a prepositional phrase “من خلال” that introduces the agent of the verb phrase, a noun phrase “ثروة والدته” that acts as the object of the preposition, a relative clause “التي كانت كبيرة” that modifies the noun “ثروة”, a conjunction “و” that introduces a coordination, a noun phrase “نصفها” that acts as the subject of the following verb, a verb “انتقل” that agrees with the subject in number and person, a prepositional phrase “إليه” that acts as the object of the verb, a prepositional phrase “عندما بلغ” that introduces the time of the action, and a noun phrase “سن الرشد” that acts as the object of the preposition.
Structural Comparison: This sentence have different syntactic structures and word orders. The word order of the English text is (SVO). The word order of the human translation is (VSO). The GPT3.5 translation calculates the word order of the source text (SVO). The human translation is more faithful and accurate to the original text, while the GPT3.5 translation is more concise and informal.
Stylistic Comparison: El-Sharif’s translation is closer to the Arabic style, achieving better cohesion and harmony for the Arabic reader. The style is a descriptive and narrative style, which means that it describes the character’s situation and background. Translation by GPT3.5 is closer to the original text and maintains the author’s style.
Semantic Comparison: In the original text and the GPT-3.5 translation, the son is portrayed as a “respectable and upright young man,” while in the Sharif translation, he is described as a “wise and respectable young man.” The choice of words in the translation may influence the literary impression on the reader. In the Sharif translation, the phrase “تركت له أمه وكانت ذات ثروة كبيرة” suggests that the mother left the fortune to her son. In the original text and the GPT-3.5 translation, the expression “was amply provided for by his mother’s fortune” is used to convey the same idea differently. This reflects a difference in semantic meaning. While the GPT-3.5 translation retains the phrase “coming of age” with the same formulation, the Sharif translation uses the expression “بلغ سن الرشد,” conveying the same concept in a different language.
Sentence five: The English text is a dialogue that consists of two simple sentences spoken by two different speakers. The first sentence is: “It is enough,” said she; “to say that he is unlike Fanny is enough. The original text is a dialogue consisting of two sentences, each with a (SVO) structure. The first sentence is: “It implies everything amiable. I love him already.” The second sentence is: “I think you will like him,” said Elinor, “when you know more of him.” Both sentences use quotation marks to denote direct speech and employ commas and semicolons to separate clauses and phrases.
El-sharif’s Arabic translation maintains the dialogue format with two sentences spoken by different speakers. The first sentence is: “قالت: يكفي أنه لا يشبه فاني، لأن ذلك معناه أنه يتحلى بكل الخصال المحبوبة، أنني أحبه حقا”. The second sentence is: “وقالت إلينور: أعتقد أنك ستميلين إليه متى ازددت معرفة به”. Each sentence follows a (VSO) order, uses colons to indicate direct speech, and employs commas to separate clauses and phrases.
The GPT3.5 translation also presents a dialogue with two sentences, each following an SVO structure. The first sentence is: ““هذا يكفي،” قالت هي، “أن نقول أنه ليس مثل فاني يكفي. إنه يعني كل شيء لطيف. أنا أحبه بالفعل.” The second sentence is: ““أعتقد أنك ستحبينه”، قالت إلينور، “عندما تعرفين المزيد عنه.” Quotation marks are used to denote direct speech, and commas are used to separate clauses and phrases.
Structure Comparison: The original text uses reported speech and embedded subordinate clauses to express the characters’ thoughts. Al-Sharif’s translation adheres closely to the original text’s structure, preserving the reported speech and embedded subordinate clauses, with minor adjustments to align with Arabic syntax. The GPT3.5 translation mirrors the original text and retains its structure.
Stylistic Comparison: the text is a blend of dialogue and narration, showcasing the characters’ direct speech and advancing the story. The request is to rephrase the above text in English using academic terminology and human language, ensuring the information is accurate, the sentences are coherent, and the word count is maintained. The Arabic examples should be mentioned as they are, without any additions or deletions, only changing the style and phrasing to human language. The human translation may capture the tone and the style of the original text better, while the GPT3.5 translation may be easier to understand.
Semantic Comparison: Both translations effectively capture the essence of the original text, expressing the speaker’s feelings and anticipating that the listener will develop similar emotions as they get to know the person mentioned. Al-Sharif’s translation maintains the core of the original text, while GPT-3.5’s translation closely conveys the meaning of the original text.
Sentence six: The English sentence is simple and consists of one independent clause. The subject of the clause is “we” and the predicate is “Oh! Mama, how shall we do without her?”. The predicate contains an interjection “Oh!” that expresses an emotion, a noun “mama” that acts as a vocative or a direct address, an adverb “how” that introduces an interrogative sentence, a verb “shall” that expresses a future tense, a verb “do” that agrees with the subject in number and person, a preposition “without” that introduces the object of the verb, and a pronoun “her” that acts as the object of the preposition.
El-sharif’s translation is a single sentence composed of an independent clause. The subject is implied to be “we” and is not explicitly stated. The predicate is “أماه! أنى يكون لنا أن نستغني عنها؟”. The predicate includes an interjection “أماه!” expressing emotion, an “أنى” that initiates an interrogative sentence, a verb “يكون” that concurs with the subject in number and person, a prepositional phrase “لنا” serving as the verb’s complement, and a subordinate clause “أن نستغنى عنها” introduced by a conjunction “أن” and conveying the question’s content. The subordinate clause contains a verb “نستغنى” that agrees with the subject in number and person, a preposition “عن” introducing the verb’s object, and a pronoun “ها” acting as the preposition’s object.
The GPT3.5 translation is a straightforward sentence composed of a single independent clause. The subject, implied to be “we”, is not explicitly stated. The predicate is “أوه! أمي، كيف سنستطيع الاستغناء عنها؟”. Within the predicate, there’s an interjection “أوه!” expressing emotion, a noun “أمي” serving as a direct address, a comma indicating a pause, an adverb “كيف” initiating an interrogative sentence, a verb “سنستطيع” that matches the subject in number and person, and a verb phrase “الاستغناء عنها” functioning as the verb’s object. This verb phrase includes a verb “الاستغناء”, a preposition “عن” introducing the verb’s object, and a pronoun “ها” serving as the preposition’s object.
Structural Comparison: The English sentence employs a basic structure with a (SVO) sequence and uses quotation marks to denote the speaker’s direct speech. The Arabic translation done by a human follows a (VSO) sequence and uses quotation marks for the speaker’s direct speech. The Arabic translation by GPT3.5, however, uses a (SVO) sequence and quotation marks for the speaker’s direct speech.
Stylistic Comparison: The author’s style is straightforward yet emotive, utilising an exclamation and a rhetorical question to express the speaker’s emotions. The human translation uses “أنى” as a bound pronoun, adding a layer of formality and literary sophistication to the sentence. The GPT3.5 translation uses “أوه” as an exclamation, conveying a sense of emotion or surprise, and includes “كيف” to express “how,” thereby emphasising the emotional tone. Both translations successfully encapsulate the sentiment conveyed in the original English sentence.
Semantic Comparison: The original text is a question that encapsulates the speaker’s sorrow and attachment to an individual who is either departing or passing away. The human translation is more loyal and precise to the original text. Al-Sharif’s translation is faithful to the original text, maintaining the genre, style, and meaning of the text. The translation by GPT3.5 is akin to Al-Sharif’s translation. However, it employs a different term for mother (أمي instead of أماه) and a different grammatical form for “to do without” (الاستغناء instead of نستغنى). These variations are minor and do not alter the overall understanding of the text.
Both translations are accurate and effective in conveying the original text’s message.
Sentence seven: The English sentence is a complex sentence that consists of one independent clause and one dependent clause. The independent clause is “I shall never see a man whom I can really love”. The dependent clause is “the more I know of the world” and it modifies the verb “shall”.
El-sharif’s translation is a simple sentence that consists of one independent clause. The subject of the clause is omitted and implied to be “I”. The predicate is “وأنا يا أماه كلما ازددت معرفة بالناس ازددت إيمانا بأني لن ألتقي الرجل الذي أحبه حبا صادقا”. The predicate contains an interjection “وأنا” that expresses an emotion, a vocative “يا أماه” that acts as a direct address, a conjunction “كلما” that introduces a conditional sentence, a verb “ازددت” that agrees with the subject in number and person, a noun phrase “معرفة بالناس” that acts as the object of the verb, a past tense verb attached with the letter ت)). “ازددت” that introduces a consequence sentence, a noun phrase “إيمانا” that acts as the object of the verb, a subordinate clause “بأني لن ألتقي الرجل الذي أحبه حبا صادقا” that is introduced by a preposition “ب” and expresses the content of the belief. The subordinate clause contains a particle “أني” that indicates the subject of the clause, a particle “لن” that indicates a negation, a verb “ألتقي” that agrees with the subject in number and person, a noun phrase “الرجل” that acts as the object of the verb, a relative clause “الذي أحبه” that modifies the noun “الرجل”, and a noun phrase “حبا صادقا” that acts as the complement of the relative clause.
The GPT3.5 translation is a complex sentence composed of two independent clauses linked by a comma. The first independent clause is “بمعرفتي للعالم، كلما زادت معرفتي”. The second independent clause is “كلما تأكدت أكثر أنني لن أرى رجلاً يمكنني حقاً أن أحبه”. The subject of the first independent clause is “بمعرفتي” and the predicate is “للعالم، كلما زادت معرفتي”. The predicate includes a prepositional phrase “للعالم” that serves as the object of the subject, a comma indicating a pause, a conjunction “كلما” introducing a conditional sentence, and a verb “زادت” that agrees with the subject in number and person. The subject of the second independent clause is “كلما” and the predicate is “تأكدت أكثر أنني لن أرى رجلاً”. The predicate contains a verb “تأكدت” that agrees with the subject in number and person, an adverb “أكثر” modifying the verb, a subordinate clause “أنني لن أرى رجلاً يمكنني حقاً” introduced by the conjunction “أن” and expressing the consequence of the condition. The subordinate clause includes a particle “أنني” indicating the subject of the clause, a particle “لن” indicating negation, a verb “أرى” agreeing with the subject in number and person, a noun phrase “رجلاً” serving as the object of the verb, a verb phrase “يمكنني حقاً أن أحبه” serving as the complement of the noun. The verb phrase includes a verb “يمكنني” agreeing with the subject in number and person, an adverb “حقاً” modifying the verb, and a subordinate clause “أن أحبه” introduced by the conjunction “أن” and expressing the content of the possibility.
Structural Comparison: The three sentences exhibit distinct syntactic structures and word orders. The English sentence employs a complex sentence structure with a (SVO) order and uses subordination to link the clauses and phrases. The Arabic human translation utilises a simple sentence structure with a (VSO) order and employs both subordination and coordination to link the phrases and clauses. The Arabic GPT3.5 translation features a compound sentence structure with a (SVO) order and uses both subordination and coordination to link the phrases and clauses.
Stylistic Comparison: The text is penned in a sophisticated and polished style, employing parallelism and negation to articulate the speaker’s conviction. The speaker communicates their pessimism and disappointment about the prospect of finding true love. The translation by Al-Sharif aligns closely with the original text, preserving the author’s style and tone. The GPT3.5 translation, while structurally and tonally similar, differs in certain aspects. The GPT3.5 translation repeats the phrase “بمعرفتي للعالم” twice, which could be interpreted as either redundant or emphatic. It also uses a different word for love (أحبه instead of أحبه حبا صادقا), which could be perceived as either simpler or less expressive, depending on the context.
Semantic Comparison: The human translation might better encapsulate the tone of the original text, while the GPT 3.5 translation might be more comprehensible. The essence of the text is the message the speaker wishes to communicate, which is their despair and dissatisfaction in their quest for true love. Both translations are acceptable and intelligible in conveying the text’s meaning.
Sentence eight: The text in English is a compound sentence that consists of two independent clauses connected by a semicolon. The first clause is “I could wish the stairs were handsome”. The second clause is “But one must not expect everything; though I suppose it would be no difficult matter to widen them”. The first clause has a (SVO) word order, and the second clause has a subject-verb-complement (SVC) word order. The clauses use subordination and coordination to connect the phrases and clauses.
El-sharif’s translation is a simple sentence that consists of one independent clause. The subject of the clause is omitted and implied to be “I”. The predicate is “وكنت أتمنى لو كان الدرج ارحب من ذلك ولكن “ما كل ما يتمناه المرء يدركه” وإن كنت أعتقد أن توسيعه ليس بالأمر العسير”. The predicate contains a verb “كنت” that agrees with the subject in number and person, a verb phrase “أتمنى لو” that expresses a wish, a verb “كان” that agrees with the subject in number and person, an adjective “ارحب” that modifies the subject and agrees with it in number and gender, a prepositional phrase “من ذلك” that acts as the complement of the adjective, a conjunction “ولكن” that introduces a contrast, a quotation “ما كل ما يتمناه المرء يدركه” that acts as a proverb, a conjunction “و” that introduces coordination, a verb “كنت” that agrees with the subject in number and person, a verb phrase “أعتقد أن” that expresses a belief, a verb “ليس” that agrees with the subject in number and person, a noun phrase “بالأمر العسير” that acts as the complement of the verb, and a noun phrase “توسيعه” that acts as the object of the verb phrase. The sentence has a (VSO) word order and uses subordination and coordination to connect the phrases and clauses.
The GPT3.5 translation is a compound sentence that consists of two independent clauses connected by a period. The first clause is “أتمنى أن تكون الدرج جميلة”. The second clause is “لكن يجب على المرء ألا يتوقع كل شيء؛ على الرغم من أنني أفترض أنه لن يكون من الصعب توسيعها“. The first clause follows a (SVO) order, while the second clause follows a (SVC) order. Both clauses employ subordination and coordination to connect the phrases and clauses.
Structural Comparison: The human translation more effectively captures the syntactic structure in Arabic, while ChatGPT’s translation better maintains the structure of the original text. The GPT3.5 translation is more concise and informal. It uses a semicolon instead of a quotation to separate the two clauses. It also uses a different word for handsome (جميلة instead of ارحب), which may be less precise or more general, depending on the context.
Stylistic Comparison: The original text is a statement that expresses the speaker’s opinion and desire about the stairs. The author’s style is ironic and sarcastic, using a quotation and a contrast to convey the speaker’s dissatisfaction. The translation by Al-Sharif better captures the tone and style of the original text, as it uses a word that implies spaciousness and comfort (ارحب) instead of a word that implies beauty and elegance (جميلة). In contrast, the GPT3.5 translation is easier to understand.
Semantic Comparison: The translation by Al-Sharif is closer to the original meaning, as it uses a quotation from a famous Arabic proverb to express the speaker’s resignation and does a better job of translating the meaning of the text and adapting it to the Arabic reader. The translation by GPT3.5 is more straightforward and more general, but it does not convey the speaker’s dissatisfaction as well as the translation by the human translator.
Table (2) below shows the distribution of the percentages for each of the translations of Al-Sharif and GPT3.5 according to the order of the sentences in Table (1) and in the analysis. The percentages are calculated from 8 according to the number. They are extracted by the ratio of the approximation of the translation of each sentence to the original text and to the three criteria of structure, style, and meaning. The researchers first extracted the percentages for each sentence, added the percentages, and extracted the arithmetic mean for each criterion.
|
N |
Structure |
Style |
Meaning |
|||||
|
Al-Sharif |
GPT3.5 |
Al-Sharif |
GPT3.5 |
Al-sharif |
GPT3.5 |
|||
|
1 |
40% |
60% |
50% |
30% |
60% |
30% |
||
|
2 |
40% |
60% |
50% |
30% |
60% |
30% |
||
|
3 |
40% |
60% |
70% |
40% |
60% |
40% |
||
|
4 |
40% |
60% |
70% |
40% |
60% |
30% |
||
|
5 |
40% |
60% |
70% |
40% |
60% |
30% |
||
|
6 |
40% |
60% |
70% |
30% |
60% |
30% |
||
|
7 |
40% |
60% |
70% |
40% |
40% |
40% |
||
|
8 |
30% |
70% |
60% |
40% |
70% |
30% |
||
|
Structure evaluation |
Style evaluation |
Meaning evaluation: |
||||||
|
GPT3.5: 61.25% |
El-sharif 38.75% |
GPT3.5: 36.25% |
El-Sharif: 63.74% |
GPT3.5: 35.61% |
El-sharif: 64.38% |
|||
Table (2) the distribution of the percentages for each of the translations of Al-Sharif and GPT3.5
5.2 Overall Blue Score Outputs
In the second stage of the assessment, the researchers measured the alignment of words and phrases and evaluated the quality of the output using the BLEU metrics (Bilingual Evaluation Understudy). This automatic assessment tool is, as pointed out by Papineni et al. (2002), widely recognised for an overall assessment of the candidate translations generated automatically, compared to the reference human translation. The higher the BLEU score is, the elevated alignment with the reference will be.
By analysing inputs and outputs, successive rounds revealed the following ratios (Table 3) for translating both ChatGPT 3.5 and human translators, along with the convergence ratio between human expert translations and those generated by Artificial Intelligence. It’s crucial to emphasise that this metric is solely quantitative and does not consider aspects such as meaning, structure, or style.
|
Examples |
Overall BLEU scores |
|
|
Candidates Sentences |
HT |
AI T (provided with Chat Gpt3.5) |
|
(1) |
52,94 % |
74,15 % |
|
(2) |
39,99 % |
76,88 % |
|
(3) |
52,59 % |
85,28 % |
|
(4) |
45,45 % |
95,23 % |
|
(5) |
32,76% |
79,37% |
|
(6) |
40% |
67,88% |
|
(7) |
59,60% |
85,23% |
|
(8) |
45,45% |
95,24% |
Table (3) Comparative analysis of HT and ChatGPT translation using the BLEU metrics.
5.3 Evaluation
As shown in Table 3 above, there is a significant difference in literary style, accuracy, and meaning transfer between Amin El-Sharif’s translation and the artificial intelligence translation GPT-3.5. The human translation excels with a 38.75% margin in structure, a 63.74% margin in style, and a 64.38% margin in meaning, while GPT-3.5 lags behind with 61.25% in structure, 36.25% in style, and 35.61% in meaning. Concerning the relative performance and evaluation using BLEU score metrics, the outcome favours artificial intelligence translation. The percentages indicate a significant advantage for GPT-3.5, suggesting that its translation is closer to the structure of the original text than Amin El-Sharif’s translation. These results confirm the study by Shi, X. et al. (2016), which states that the encoder in a neural translation system learns significant grammatical information at the sentence level. It can understand the sentence structure to a considerable extent at the word level.
However, this does not mean that the AI-powered machine translation was better than the human one. This result only indicates that the BLEU score cannot be suitable for measuring the quality of literary translation due to its uniqueness, the creativity that characterises the literary style and the diversity and beauty of its literary vocabulary. Literary translators are required to transpose the beauty of the original in the moulds of the target language and, therefore, are not supposed to translate word for word or produce a verbatim version of the original in another language.
- Conclusion
In closing, the researchers conclude that machine translation, although characterised by speed and efficiency, still lacks accuracy and creativity in dealing with the nuances and tones and creatively conveying the Arabic literary style. It cannot, as the human translator, acquire high skills in understanding the original text and transforming it into another language in a way that respects the writer’s vision, preserves the aesthetics of literature, and considers the cognitive differences between languages. The study findings suggest that although the machine translation output achieved a higher and stronger percentage according to the BLEU score metrics, it failed to reproduce a literary style in the way that a human translator would. Literary creativity is something that machines cannot acquire, and this is what the candidate translations indicate as they are a mere calque of the original. Machine translation relies on algorithms to generate, train, and extract translations based on statistical patterns and linguistic rules to translate words and phrases. However, among the limitations of this study is the results of this investigation cannot be valid for all language pairs and other literary genres and works. Furthermore, ChatGPT’s translations can be improved over time using deep learning techniques to be able to achieve better literary translations. Thus, future research should investigate the importance of collaboration between human translators and AI systems to achieve better, faster, more diverse, creative, and compatible literary translations with the original texts, the target languages, and the intended audience. Creativity in literary translation and the challenges facing AI-powered translation techniques should be the research gap that should be further investigated to achieve an efficient collaboration between human and machine translations.
- References:
1- Austen, J. (n.d)). al-ʻAql wālʻāṭfh. (A. Al-Sharīf, (Trans). Maṭbaʻat al-Maʻrifah.(Arabic version)
2- Austen, J. (2006). Sense and Sensibility. (E. Copeland, Ed.) Cambridge University Press.
3- Chomsky, N. (2006). Language and Mind. Third Edition. Cambridge University Press. Philosophy for Linguist.
4- Ghazvininejad, M., Shi, X., Choi, Y., & Knight, K. (2016). Generating topical poetry. EMNLP 2016-Conference on Empirical Methods in Natural Language Processing, Proceedings, 1183–1191. https://doi.org/10.18653/V1/D16-1126
5- Newmark, P. (1980). The Translation of Metaphor. Babel. Revue Internationale de La Traduction / International Journal of Translation, 26(2), 93–100. https://doi.org/10.1075/BABEL.26.2.05NEW/CITE/REFWORKS
6- House, J. (1989). A Model for Translation Quality. In Blum-Kulka, S., House, J., & Kasper, G. (Eds.), Cross-Cultural Pragmatics: Requests and Apologies (pp. 281-291). Ablex Publishing Corporation.
7- Kolb, W., Dressler, W. U., & Mattiello, E. (2023). Human and machine translation of occasionalisms in literary texts: Johann Nestroy’s Der Talisman and its English translations. Target.
8- Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (n.d.). BLEU: A Method for Automatic Evaluation of Machine Translation.
9- Shi, X., Padhi, I., & Knight, K. (2016). Does String-Based Neural MT Learn Source Syntax? EMNLP 2016-Conference on Empirical Methods in Natural Language Processing, Proceedings, 1526–1534. https://doi.org/10.18653/V1/D16-1159
10- Toral, A., & Way, A. (2018). What Level of Quality can Neural Machine Translation Attain on Literary Text?arXiv:1801.04962v1, https://arxiv.org/abs/1801.04962
11- What is a BLEU score?-Custom Translator-Azure AI services | Microsoft Learn. (n.d.). Retrieved November 17, 2023, from https://learn.microsoft.com/en-us/azure/ai-services/translator/custom-translator/concepts/bleu-score
12- Yirmibeşoğlu, Z., Dursun, O., Dallı, H., Şahin, M., Hodzik, E., Gürses, S., & Güngör, T. (2023). Incorporating Human Translator Style into English-Turkish Literary Machine Translation. https://arxiv.org/abs/2307.11457v1
-۞ University Yahia Fares of Medea, Algeria.
E.mail: hajir.waheedah@univ-medea.dz (auteur correspondant).
-۞ University Yahia Fares of Medea, Algeria.
E.mail: babchikhscol@gmail.com




