The NESPOLE! Speech-to-Speech Translation System

Abstract

This paper describes the first showcase of the NESPOLE! system. NESPOLE! is a speech-to-speech machine translation system designed to provide fully functional speech-to-speech capabilities within real-world settings of common users involved in e-commerce applications. The project is a collaboration between three European research laboratories (IRST in Trento, Italy; ISL at Universitat Karlsruhe (TH) in Germany; and CLIPS at Universite Joseph Fourier in Grenoble, France), one US research group (ISL at Carnegie Mellon University in Pittsburgh, PA) and two industrial partners (APT; Trento, Italy - the Trentino provincial tourism board, and Aethra; Ancona, Italy - a tele-communications company). The project is funded jointly by the European Commission and the US' NSF. The main goal of NESPOLE! is to advance the state-of-the-art of speech-to-speech translation in realistic scenarios and involving naive users. The first showcase presented in this demonstration involves an English-, French-, or German-speaking client enquiring about winter-sports possibilities in the Trentino region of the Italian Alps via a NetMeeting connection. His or her questions are answered by an Italian-speaking agent at APT, while the NESPOLE! system provides speech-to-speech translation and a multimodal Whiteboard, which users can use to point to shared web-sites or draw on shared maps, therefore enhancing the oral communication with extra capabilities. This paper is organized as follows: we will first present a general system description which covers the current hardware setup as well as the design principles of the NESPOLE! system in general. We will then present the user interface, and the results from two system evaluations, one with respect to system performance under different network conditions and an end-to-end evaluation. The references at the end of this paper serve as pointers to further information about the NESPOLE! system.