Báo cáo khoa học: An Experiment in Evaluating the Quality of Translations

To lay the foundations for a systematic procedure that could be applied to any scientific translation, this experiment evaluates the error variances attributable to various sources inherent in a design in which discrete, randomly ordered sentences from translations are rated for intelligibility and for fidelity to the original.
Nội dung trích xuất từ tài liệu:
Báo cáo khoa học: "An Experiment in Evaluating the Quality of Translations" [Mechanical Translation and Computational Linguistics, vol.9, nos.3 and 4, September and December 1966] An Experiment in Evaluating the Quality of Translations by John B. Carroll,* Graduate School of Education, Harvard University To lay the foundations for a systematic procedure that could be applied to any scientific translation, this experiment evaluates the error variances attributable to various sources inherent in a design in which discrete, ran- domly ordered sentences from translations are rated for intelligibility and for fidelity to the original. The procedure is applied to three human and three mechanical translations into English of four passages from a Rus- sian work on cybernetics, yielding mean scores for the translations. Human and mechanical translations are clearly different in over-all qual- ity, although substantial overlap is noted when individual sentences are considered. The procedure also clearly differentiates within sets of human translations and within sets of mechanical translations. Results from the two scales are highly correlated, and these in turn are highly correlated with reading times. A procedure in which highly intelligent mono- lingual raters (i.e., without knowledge of the foreign language) compare a test translation with a carefully prepared translation is found to be more reliable than one in which bilingual raters compare the English translation with the Russian original. liability, fixed standards of evaluation, and relativeIntroduction simplicity and feasibility.It would be desirable, in studies of the merits of ma- The method is based on the following considerations:chine translation attempts, to have available a relatively 1. The evaluation of the adequacy of a translationsimple yet accurate and valid technique for scaling the must rest ultimately upon subjective judgments, that is,quality of translations. It has also become apparent judgments resulting from human cognitions and intui-that such a technique would be useful in assessing tions. (If any objective measurements directly applic-human translations. The present experiment seeks to able to the translations themselves were available—say,lay the foundations for the development of a technique. some form of word-counting—they could presumably There have been several other experiments in meas- be used in the production of translations; hence, use ofuring the quality of mechanical translations,1,2 but the such objective procedures in the evaluation of transla-procedures proposed in these experiments have gener- tions could lead to circularity.)ally been too laborious, too subject to arbitrariness in 2. If sufficient care is taken, procedures utilizing sub-standards, or too lacking in validity and/or reliability jective judgments can be devised that attain acceptableto constitute a satisfactory basis for a standard evalua- ...