Transferring Emphasis in Speech Translation Using Hard-Attentional Neural Network Models

Quoc Truong Do, Sakriani Sakti, Graham Neubig, Satoshi Nakamura


While traditional speech translation systems are oblivious to paralinguistic information, there has been a recent focus on speech translation systems that transfer not only the linguistic content but also emphasis information across languages. A recent work has tried to tackle this task by developing a method for mapping emphasis between languages utilizing conditional random fields (CRFs). Although CRFs allow for consideration of rich features and local context, they have difficulty in handling continuous variables, and cannot capture long-distance dependencies easily. In this paper, we propose a new model for emphasis transfer in speech translation using an approach based on neural networks. The proposed model can handle long-distance dependencies by using long short-term memory (LSTM) neural networks, and is able to handle continuous emphasis values through a novel hard-attention mechanism, which uses word alignments to decide which emphasis values to map from the source to the target sentence. Our experiments on the emphasis translation task showed a significant improvement of the proposed model over the previous state-of-the-art model by 4% target-language emphasis prediction F-measure according to objective evaluation and 2% F-measure according to subjective evaluation.


DOI: 10.21437/Interspeech.2016-898

Cite as

Do, Q.T., Sakti, S., Neubig, G., Nakamura, S. (2016) Transferring Emphasis in Speech Translation Using Hard-Attentional Neural Network Models. Proc. Interspeech 2016, 2533-2537.

Bibtex
@inproceedings{Do+2016,
author={Quoc Truong Do and Sakriani Sakti and Graham Neubig and Satoshi Nakamura},
title={Transferring Emphasis in Speech Translation Using Hard-Attentional Neural Network Models},
year=2016,
booktitle={Interspeech 2016},
doi={10.21437/Interspeech.2016-898},
url={http://dx.doi.org/10.21437/Interspeech.2016-898},
pages={2533--2537}
}