International Workshop on Spoken Language Translation (IWSLT) 2005

Pittsburgh, PA, USA
October 24-25, 2005

Machine Translation Evaluation Inside QARLA

Jesús Giménez (1), Enrique Amigó (2), Chiori Hori (3)

(1) TALP Research Center, Universitat Politècnica de Catalunya, Barcelona, Spain
(2) Departamento de Lenguajes y Sistemas Informáticos, Universidad Nacional de Educación a Distancia, Spain
(3) InterACT Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA

In this work we present the fundamentals of the IQMT framework for MT evaluation. IQMT offers a common workbench on which existing evaluation metrics can be utilized. We suggest the IQ measure and test it on the Chinese-to- English data from the IWSLT 2004 Evaluation Campaign. We show how the correlation with human assessments at the system level improves substantially for most individual metrics. Moreover, IQMT allows to robustly combine several metrics avoiding scaling problems and metric weightings. Several metric combinations were tried, but correlations did not further improve significantly.

Full Paper    Presentation

Bibliographic reference.  Giménez, Jesús / Amigó, Enrique / Hori, Chiori (2005): "Machine translation evaluation inside QARLA", In IWSLT-2005, 189-196.