ISCA Archive Interspeech 2021
ISCA Archive Interspeech 2021

Alzheimer’s Dementia Recognition Using Acoustic, Lexical, Disfluency and Speech Pause Features Robust to Noisy Inputs

Morteza Rohanian, Julian Hough, Matthew Purver

We present two multimodal fusion-based deep learning models that consume ASR transcribed speech and acoustic data simultaneously to classify whether a speaker in a structured diagnostic task has Alzheimer’s Disease and to what degree, evaluating the ADReSSo challenge 2021 data. Our best model, a BiLSTM with highway layers using words, word probabilities, disfluency features, pause information, and a variety of acoustic features, achieves an accuracy of 84% and RSME error prediction of 4.26 on MMSE cognitive scores. While predicting cognitive decline is more challenging, our models show improvement using the multimodal approach and word probabilities, disfluency, and pause information over word-only models. We show considerable gains for AD classification using multimodal fusion and gating, which can effectively deal with noisy inputs from acoustic features and ASR hypotheses.


doi: 10.21437/Interspeech.2021-1633

Cite as: Rohanian, M., Hough, J., Purver, M. (2021) Alzheimer’s Dementia Recognition Using Acoustic, Lexical, Disfluency and Speech Pause Features Robust to Noisy Inputs. Proc. Interspeech 2021, 3820-3824, doi: 10.21437/Interspeech.2021-1633

@inproceedings{rohanian21_interspeech,
  author={Morteza Rohanian and Julian Hough and Matthew Purver},
  title={{Alzheimer’s Dementia Recognition Using Acoustic, Lexical, Disfluency and Speech Pause Features Robust to Noisy Inputs}},
  year=2021,
  booktitle={Proc. Interspeech 2021},
  pages={3820--3824},
  doi={10.21437/Interspeech.2021-1633}
}