Deep-Learning Based Automatic Spontaneous Speech Assessment in a Data-Driven Approach for the 2017 SLaTE CALL Shared Challenge

Yoo Rhee Oh, Hyung-Bae Jeon, Hwa Jeon Song, Byung Ok Kang, Yun-Kyung Lee, Jeon-Gue Park, Yun-Keun Lee


This paper presents a deep-learning based assessment method of a spoken computer-assisted language learning (CALL) for a non-native child speaker, which is performed in a data-driven approach rather than in a rule-based approach. Especially, we focus on the spoken CALL assessment of the 2017 SLaTE challenge. To this end, the proposed method consists of four main steps: speech recognition, meaning feature extraction, grammar feature extraction, and deep-learning based assessment. At first, speech recognition is performed on an input speech by using three automatic speech recognition (ASR) systems. Second, twenty-seven meaning features are extracted from the recognized texts via the three ASRs by using language models (LMs), sentence-embedding models, and word-embedding models. Third, twenty-two grammar features are extracted from the recognized text via one ASR system by using linear-order LMs and hierarchical-order LMs. Fourth, the extracted forty-nine features are fed into a full-connected deep neural network (DNN) based model for the classification of acceptance or rejection. Finally, an assessment is performed by comparing the probability of a output unit of the DNN-based classifier with a pre-defined threshold. For the experiments of a spoken CALL assessment, we use English spoken utterances by Swiss German teenagers. It is shown from the experiments that the D score is 4.37 for the spoken CALL assessment system employing the proposed method.


 DOI: 10.21437/SLaTE.2017-18

Cite as: Oh, Y.R., Jeon, H., Song, H.J., Kang, B.O., Lee, Y., Park, J., Lee, Y. (2017) Deep-Learning Based Automatic Spontaneous Speech Assessment in a Data-Driven Approach for the 2017 SLaTE CALL Shared Challenge. Proc. 7th ISCA Workshop on Speech and Language Technology in Education, 103-108, DOI: 10.21437/SLaTE.2017-18.


@inproceedings{Oh2017,
  author={Yoo Rhee Oh and Hyung-Bae Jeon and Hwa Jeon Song and Byung Ok Kang and Yun-Kyung Lee and Jeon-Gue Park and Yun-Keun Lee},
  title={Deep-Learning Based Automatic Spontaneous Speech Assessment in a Data-Driven Approach for the 2017 SLaTE CALL Shared Challenge},
  year=2017,
  booktitle={Proc. 7th ISCA Workshop on Speech and Language Technology in Education},
  pages={103--108},
  doi={10.21437/SLaTE.2017-18},
  url={http://dx.doi.org/10.21437/SLaTE.2017-18}
}