ISCA Archive Interspeech 2021
ISCA Archive Interspeech 2021

Investigating the Utility of Multimodal Conversational Technology and Audiovisual Analytic Measures for the Assessment and Monitoring of Amyotrophic Lateral Sclerosis at Scale

Michael Neumann, Oliver Roesler, Jackson Liscombe, Hardik Kothare, David Suendermann-Oeft, David Pautler, Indu Navar, Aria Anvar, Jochen Kumm, Raquel Norel, Ernest Fraenkel, Alexander V. Sherman, James D. Berry, Gary L. Pattee, Jun Wang, Jordan R. Green, Vikram Ramanarayanan

We propose a cloud-based multimodal dialog platform for the remote assessment and monitoring of Amyotrophic Lateral Sclerosis (ALS) at scale. This paper presents our vision, technology setup, and an initial investigation of the efficacy of the various acoustic and visual speech metrics automatically extracted by the platform. 82 healthy controls and 54 people with ALS (pALS) were instructed to interact with the platform and completed a battery of speaking tasks designed to probe the acoustic, articulatory, phonatory, and respiratory aspects of their speech. We find that multiple acoustic (rate, duration, voicing) and visual (higher order statistics of the jaw and lip) speech metrics show statistically significant differences between controls, bulbar symptomatic and bulbar pre-symptomatic patients. We report on the sensitivity and specificity of these metrics using five-fold cross-validation. We further conducted a LASSO-LARS regression analysis to uncover the relative contributions of various acoustic and visual features in predicting the severity of patients’ ALS (as measured by their self-reported ALSFRSR scores). Our results provide encouraging evidence of the utility of automatically extracted audiovisual analytics for scalable remote patient assessment and monitoring in ALS.


doi: 10.21437/Interspeech.2021-1801

Cite as: Neumann, M., Roesler, O., Liscombe, J., Kothare, H., Suendermann-Oeft, D., Pautler, D., Navar, I., Anvar, A., Kumm, J., Norel, R., Fraenkel, E., Sherman, A.V., Berry, J.D., Pattee, G.L., Wang, J., Green, J.R., Ramanarayanan, V. (2021) Investigating the Utility of Multimodal Conversational Technology and Audiovisual Analytic Measures for the Assessment and Monitoring of Amyotrophic Lateral Sclerosis at Scale. Proc. Interspeech 2021, 4783-4787, doi: 10.21437/Interspeech.2021-1801

@inproceedings{neumann21b_interspeech,
  author={Michael Neumann and Oliver Roesler and Jackson Liscombe and Hardik Kothare and David Suendermann-Oeft and David Pautler and Indu Navar and Aria Anvar and Jochen Kumm and Raquel Norel and Ernest Fraenkel and Alexander V. Sherman and James D. Berry and Gary L. Pattee and Jun Wang and Jordan R. Green and Vikram Ramanarayanan},
  title={{Investigating the Utility of Multimodal Conversational Technology and Audiovisual Analytic Measures for the Assessment and Monitoring of Amyotrophic Lateral Sclerosis at Scale}},
  year=2021,
  booktitle={Proc. Interspeech 2021},
  pages={4783--4787},
  doi={10.21437/Interspeech.2021-1801}
}