7th International Conference on Spoken Language Processing

September 16-20, 2002
Denver, Colorado, USA

Robust Semantic Confidence Scoring

Didier Guillevic, Simona Gandrabur, Yves Normandin

InfoSpace Speech Solutions, Canada

This paper describes an approach for defining robust, applicationindependent confidence measures for dialogue systems. A conceptlevel confidence score is computed using a Multi-Layer Perceptron (MLP) classifier trained to discriminate between correct and incorrect concepts. Three types of concept-level confidence features are considered: features based on the confidence score of the underlying words, parsing specific features, and novel semantic features (weighted semantic purity and time consistency) that are indicators of the coherence among various semantic recognition hypotheses. Confidence scores at the semantic hypothesis and utterance levels are derived from the confidence scores of the corresponding concepts. We report our results on a database of 40,000 utterances from various application contexts. By using features based only on word scores for concept classification we obtained a 46% correct rejection (CR) rate at a 95% correct acceptance (CA) rate. Adding semantic measures to the classifier boosted the CR rate to 71%, which corresponds to a 46.3% relative improvement.

Full Paper

Bibliographic reference.  Guillevic, Didier / Gandrabur, Simona / Normandin, Yves (2002): "Robust semantic confidence scoring", In ICSLP-2002, 853-856.