Vision-based interaction within a multimodal framework

Vítor J. Sá; Cornelius Malerczyk; Michael Schnaider

Vision-based interaction within a multimodal framework

Vítor J. Sá, Cornelius Malerczyk, Michael Schnaider

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Our contribution is to the field of video-based interaction techniques and is integrated in the home environment of the EMBASSI project. This project addresses innovative methods of man-machine interaction achieved through the development of intelligent assistance and anthropomorphic user interfaces. Within this project, multimodal techniques represent a basic requirement, especially considering those related to the integration of modalities. We are using a stereoscopic approach to allow the natural selection of devices via pointing gestures. The pointing hand is segmented from the video images and the 3D position and orientation of the forefinger is calculated. This modality has a subsequent integration with that of speech, in the context of a multimodal interaction infrastructure. In a first phase, we use semantic fusion with amodal input, considering the modalities in a so-called late fusion state.

Original language	English
Title of host publication	Atas do 10º Encontro Português de Computação Gráfica
Publication status	Published - 2001
Externally published	Yes
Event	10º Encontro Português de Computação Gráfica - Lisboa, Portugal Duration: 1 Oct 2001 → 3 Oct 2001 Conference number: 10

Publication series

Name
ISSN (Electronic)	0873-1837

Conference

Conference	10º Encontro Português de Computação Gráfica
Country/Territory	Portugal
City	Lisboa
Period	1/10/01 → 3/10/01

Keywords

EMBASSI project
3D deictic gestures
Multimodal man-machine interaction
Agent-based systems

Cite this

@inproceedings{40499a8c197e4aceb3b4c8b8d15507fc,

title = "Vision-based interaction within a multimodal framework",

abstract = "Our contribution is to the field of video-based interaction techniques and is integrated in the home environment of the EMBASSI project. This project addresses innovative methods of man-machine interaction achieved through the development of intelligent assistance and anthropomorphic user interfaces. Within this project, multimodal techniques represent a basic requirement, especially considering those related to the integration of modalities. We are using a stereoscopic approach to allow the natural selection of devices via pointing gestures. The pointing hand is segmented from the video images and the 3D position and orientation of the forefinger is calculated. This modality has a subsequent integration with that of speech, in the context of a multimodal interaction infrastructure. In a first phase, we use semantic fusion with amodal input, considering the modalities in a so-called late fusion state.",

keywords = "EMBASSI project, 3D deictic gestures, Multimodal man-machine interaction, Agent-based systems",

author = "S{\'a}, {V{\'i}tor J.} and Cornelius Malerczyk and Michael Schnaider",

year = "2001",

language = "English",

booktitle = "Atas do 10º Encontro Portugu{\^e}s de Computa{\c c}{\~a}o Gr{\'a}fica",

note = "10º Encontro Portugu{\^e}s de Computa{\c c}{\~a}o Gr{\'a}fica ; Conference date: 01-10-2001 Through 03-10-2001",

}

TY - GEN

T1 - Vision-based interaction within a multimodal framework

AU - Sá, Vítor J.

AU - Malerczyk, Cornelius

AU - Schnaider, Michael

N1 - Conference code: 10

PY - 2001

Y1 - 2001

N2 - Our contribution is to the field of video-based interaction techniques and is integrated in the home environment of the EMBASSI project. This project addresses innovative methods of man-machine interaction achieved through the development of intelligent assistance and anthropomorphic user interfaces. Within this project, multimodal techniques represent a basic requirement, especially considering those related to the integration of modalities. We are using a stereoscopic approach to allow the natural selection of devices via pointing gestures. The pointing hand is segmented from the video images and the 3D position and orientation of the forefinger is calculated. This modality has a subsequent integration with that of speech, in the context of a multimodal interaction infrastructure. In a first phase, we use semantic fusion with amodal input, considering the modalities in a so-called late fusion state.

AB - Our contribution is to the field of video-based interaction techniques and is integrated in the home environment of the EMBASSI project. This project addresses innovative methods of man-machine interaction achieved through the development of intelligent assistance and anthropomorphic user interfaces. Within this project, multimodal techniques represent a basic requirement, especially considering those related to the integration of modalities. We are using a stereoscopic approach to allow the natural selection of devices via pointing gestures. The pointing hand is segmented from the video images and the 3D position and orientation of the forefinger is calculated. This modality has a subsequent integration with that of speech, in the context of a multimodal interaction infrastructure. In a first phase, we use semantic fusion with amodal input, considering the modalities in a so-called late fusion state.

KW - EMBASSI project

KW - 3D deictic gestures

KW - Multimodal man-machine interaction

KW - Agent-based systems

M3 - Conference contribution

BT - Atas do 10º Encontro Português de Computação Gráfica

T2 - 10º Encontro Português de Computação Gráfica

Y2 - 1 October 2001 through 3 October 2001

ER -

Vision-based interaction within a multimodal framework

Abstract

Publication series

Conference

Keywords

Fingerprint

Cite this