First ISCA ITRW on Auditory Quality of Systems
April 23-25, 2003
This paper presents the observations and results from the evaluation of the Program Guide Information System (PGIS), which was built as the CATCH-2004 Helsinki demonstrator under the European Union’s 5th Framework Research Programme . The technical implementation is the outcome of the integration of several technologies, including Automatic Speech Recognition and Natural Language Understanding. The paper gives an overview of the architecture and the technical implementation of the system. To achieve a high quality conversational interface implementation, the developed system was tested in several rounds. Both objective and subjective tests were run in an iterative manner: after each test cycle the collected data was analysed and used to improve the overall system and the individual modules’ performance further. Users’ opinions about the usability of the system were taken into account in tuning the user interface, i.e. modifying system prompting, dialogue flow, feedback mechanism, etc. The methodology of the evaluation perspectives is introduced and the main results and observations of the test runs are presented in more details.
Bibliographic reference. Hartikainen, Elviira / Boda, Péter (2003): "Results and observations from the evaluation of a telephony-based conversational information retrieval system", In AQS-2003, 139-144.