15th Annual Conference of the International Speech Communication Association

September 14-18, 2014

Asynchronous Stochastic Optimization for Sequence Training of Deep Neural Networks: Towards Big Data

Erik McDermott, Georg Heigold, Pedro J. Moreno, Andrew Senior, Michiel Bacchiani

Google, USA

Previous work presented a proof of concept for sequence training of deep neural networks (DNNs) using asynchronous stochastic optimization, mainly focusing on a small-scale task. The approach offers the potential to leverage both the efficiency of stochastic gradient descent and the scalability of parallel computation. This study presents results for four different voice search tasks to confirm the effectiveness and efficiency of the proposed framework across different conditions: amount of data (from 60 hours to 20,000 hours), type of speech (read speech vs. spontaneous speech), quality of data (supervised vs. unsupervised data), and language. Significant gains over baselines (DNNs trained at the frame level) are found to hold across these conditions. The experimental results are analyzed, and additional practical details for the approach are provided. Furthermore, different sequence training criteria are compared.

Full Paper

Bibliographic reference.  McDermott, Erik / Heigold, Georg / Moreno, Pedro J. / Senior, Andrew / Bacchiani, Michiel (2014): "Asynchronous stochastic optimization for sequence training of deep neural networks: towards big data", In INTERSPEECH-2014, 1224-1228.