Fourth-Year Project — Embodied conversational agents in augmented reality

October 2000 to May 2001

My fourth-year project was to research and develop an Augmented Reality system for instructional use. My supervisor for this project was Dr Patrick Olivier.

Augmented Reality allows a user to have their view of the world augmented with additional information. For example, someone wishing to repair a photocopier could have their view of the photocopier enhanced with information such as which parts to inspect, or which panel to remove. In particular, I was interested in adding a virtual assistant into the user's world help them with their tasks. As an example, the user should be able to see the virtual assistant as it walks around the augmented world, communicate with speech, and have the assistant understand and reply (with both speech and gesture).

Abstract: Passive methods of instruction, such as a books and videos, are commonplace, but are unable to match the flexibility and expressiveness of human instructors. Spoken natural language is an important part of our communication with each other, yet remains largely unused in our interaction with technology. Embodied conversational agents are animated computer software characters able to converse with users through the understanding and generation of speech, and they often employ gesture to further enhance their expressive powers. Such characters may be used for instructional purposes, combining the benefits of faultless, static reference with the empathic nature of human mentors. Augmented reality, the addition of virtual elements to a view of the real world, represents one of the many challenging and diverse fields of research within Computer Science. A system is sought, which combines the ability of augmented reality to add information to a view of the real world with the versatile nature of embodied conversational agents. The project successfully introduces, reviews, designs, implements, and evaluates an embodied agent-based augmented reality system. The system places particular emphasis on the facial animation and lip-synch of the character by researching and implementing a muscle-based animation system.

scene showing calibration markers
The user's view of the real-world.
scene showing an augmented image scene showing an augmented image from another angle
Initial testing phase: The user's view of the real-world is augmented with additional computer images (two views).
scene showing real coloured blocks scene showing augmented coloured blocks over real ones
Registration testing: (a) View with real-world blocks; (b) Augmented blocks overlaid.
scene showing augmented character incorrectly appearing to be on a real block scene showing augmented character correctly obscured by augmented blocks drawn over the real blocks scene showing augmented character correctly obscured by the real blocks
Obscured Rendering: (a) View with real-world blocks, no obscuring of character; (b) Augmented blocks overlaid, character obscured; (c) Augmented blocks Z-buffer rendered only (hence invisible), character obscured.