8th European Conference on Speech Communication and Technology

Geneva, Switzerland
September 1-4, 2003


Evaluation of a Speech-Driven Telephone Information Service Using the PARADISE Framework: A Closer Look at Subjective Measures

Paula M.T. Smeele, Juliette A.J.S. Waals

TNO Human Factors, The Netherlands

For the evaluation of a speech-driven telephone flight information service we applied the PARADISE model developed by Walker and colleagues [1] in order to gain insight into the factors affecting the user satisfaction of this service. We conducted an experiment in which participants were asked to call the service and book a flight. During the telephone conversations quantitative measures (e.g. total elapsed time, the number of system errors) were logged. After completion of the telephone calls, the participants judged some quality related aspects such as dialogue presentation and accessability of the system. These subjective measures together represent a value for user satisfaction. Using multivariate linear regression, it was possible to derive a performance function with user satisfaction as the dependent variable and a combination of objective measures as independent variables. The results of the regression analysis also indicated that an extended definition of user satisfaction including a subjective measure `Grade' provides a better prediction than the analysis based on the narrow definition used by Walker et al. Further, we investigated the correlation between the subjective measures by conducting a principal components analysis. The results showed that these measures fell into two groups. Implications are discussed.

Full Paper

Bibliographic reference.  Smeele, Paula M.T. / Waals, Juliette A.J.S. (2003): "Evaluation of a speech-driven telephone information service using the PARADISE framework: a closer look at subjective measures", In EUROSPEECH-2003, 1949-1952.