Empirical Evaluation of the Communicative Effectiveness of an Automatic Speech-to-Speech Translation System

Xinhao Wang, Keelan Evanini


In this study we evaluate the ability of a state-of-the-art speech-to-speech machine translation system, Skype Translator, to preserve the communicative effectiveness of its input speech. Specifically, we elicited Spanish spoken responses from 24 native speakers of Spanish in the following two scenarios: academic speaking (in the context of 6 questions from a standardized assessment of academic speaking proficiency) and daily communication (in the context of 6 map-based directions tasks). Skype Translator was then used to translate these Spanish responses into English, and expert human raters subsequently provided speaking proficiency scores using both holistic and analytic scoring rubrics for both the original Spanish responses and the corresponding automatically translated English versions. The results indicate that the average holistic scores for the academic speaking task decreased from 3.5 (for the original Spanish responses) to 1.9 (for the English translations) on a 4-point scale, and the average task completion scores for the directions tasks decreased from 3.4 to 2.0. While these results indicate that automated speech-to-speech machine translation does not yet fully meeting the communicative demands of these speaking tasks, they also indicate how far the technology has improved in recent years and can serve as a benchmark for tracking further improvements in the future.


 DOI: 10.21437/SLaTE.2017-26

Cite as: Wang, X., Evanini, K. (2017) Empirical Evaluation of the Communicative Effectiveness of an Automatic Speech-to-Speech Translation System. Proc. 7th ISCA Workshop on Speech and Language Technology in Education, 150-155, DOI: 10.21437/SLaTE.2017-26.


@inproceedings{Wang2017,
  author={Xinhao Wang and Keelan Evanini},
  title={Empirical Evaluation of the Communicative Effectiveness of an Automatic Speech-to-Speech Translation System},
  year=2017,
  booktitle={Proc. 7th ISCA Workshop on Speech and Language Technology in Education},
  pages={150--155},
  doi={10.21437/SLaTE.2017-26},
  url={http://dx.doi.org/10.21437/SLaTE.2017-26}
}