10th Annual Conference of the International Speech Communication Association

Brighton, United Kingdom
September 6-10, 2009

Classification-Based Strategies for Combining Multiple 5-W Question Answering Systems

Sibel Yaman (1), Dilek Hakkani-Tür (1), Gokhan Tur (2), Ralph Grishman (3), Mary Harper (4), Kathleen R. McKeown (5), Adam Meyers (3), Kartavya Sharma (5)

(2) SRI International, USA
(3) New York University, USA
(4) University of Maryland at College Park, USA
(5) Columbia University, USA

We describe and analyze inference strategies for combining outputs from multiple question answering systems each of which was developed independently. Specifically, we address the DARPA-funded GALE information distillation Year 3 task of finding answers to the 5-Wh questions (who, what, when, where, and why) for each given sentence. The approach we take revolves around determining the best system using discriminative learning. In particular, we train support vector machines with a set of novel features that encode systemsí capabilities of returning as many correct answers as possible. We analyze two combination strategies: one combines multiple systems at the granularity of sentences, and the other at the granularity of individual fields. Our experimental results indicate that the proposed features and combination strategies were able to improve the overall performance by 22% to 36% relative to a random selection, 16% to 35% relative to a majority voting scheme, and 15% to 23% relative to the best individual system.

Full Paper

Bibliographic reference.  Yaman, Sibel / Hakkani-Tür, Dilek / Tur, Gokhan / Grishman, Ralph / Harper, Mary / McKeown, Kathleen R. / Meyers, Adam / Sharma, Kartavya (2009): "Classification-based strategies for combining multiple 5-w question answering systems", In INTERSPEECH-2009, 2703-2706.