ATI | Information | Calendar | Application | ATI 2002 | ATI 2003

Technology

Our state-of-the-art

Our system has evolved over the course of 9 years. It has been ported to three platforms (Sun Microsystems, Silicon Graphics, and Intel/Windows) and multiple graphics libraries. Software capabilities have grown substantially while retaining system accessibility for individuals who are not graphics experts and who have little or no programming experience. This accessibility is made possible by our development of a high-level software interface (Vizard) that removes users from arcane (i.e., low level) programming tasks (e.g., creating scenes with low-level 3D primitives, interfacing to various input devices, configuring stereoscopic projection systems, communicating between networked processors via TCP/IP). Vizard also serves as a layer of abstraction between the creator's virtual environment (VE) and underlying hardware and software components. Thus, it is possible for users to port VEs across various hardware platforms, graphics sub-systems, and even operating systems. Vizard allows creators to share experimental IVEs across laboratories and ameliorates the tendency for software and digital stimuli to become obsolete or incompatible as a result of rapidly changing technology.

Performance and cost have driven our IVET development. Because our technology began as a tool for visual perception research, it was designed with high graphics update rates and low total system lags (e.g., the delay between a users movement and the resulting update of the sensory display). Though this meant keeping VE complexity low at first, we have been able to achieve stereoscopic rendering of environments at graphics update rates of 60Hz or better and total system lags under 50 ms. The cost of the whole system even with added features has decreased substantially over the years because of the availability of high-speed PCs. Indeed, the complete hardware and software costs for our IVE research system is about $25k. Integral eye-tracking and rendering adds about $15k more to our system.

Expanding research interests, funded in large part by RECVEB's current NSF KDI:LIS award, motivated us to develop many new system capabilities. We can now conduct IVET experiments across multiple networked systems allowing two or more users to interact in real time. As this raised the need to represent multiple individuals graphically, we developed our own software and methods for integrating articulated virtual human representations. To increase behavioral realism in virtual social interactions, we have implemented a voice recognition system and have augmented our headmounted displays (HMDs) with microphones that process arbitrary voice commands in a speaker independent fashion. To increase environmental realism, we have developed a working prototype for a video see-through HMD in which dual color video cameras are mounted on the user's head while wearing the HMD. The cameras digitally capture the physical scene as seen by the user. The computer inserts 3D computer graphic images (CGIs) of objects or virtual others into the scene. By continually tracking the user™s head movements, we can stabilize the added CGIs so that they appear stable with respect to the physical scene. Hence, we can effectively mix physical and virtual realities to create powerful hybrid environments and open new areas of research. Finally, looking ahead toward using IVET for tele-collaboration and tele-training, we have integrated live web content into our virtual environments thus making available online data resources and expanding the palette of content development tools.

AvatarTools - This is a toolkit built on top of Vizard and provides its own high-level programming language for creating human-like avatars and agents. The toolkit includes a library of pre-existing bodies and highly realistic faces, and includes methods for constructing new bodies and faces. Industry standard BVH joint parameter files (generated either through motion capture hardware or animation software such as Poser) can be imported directly by this toolkit to create real-time animations. Motions are automatically blended from one pose to another for smooth transitions. Face animation files are also supported, as is real-time lip-syncing of WAV files. The Avatar toolkit is currently being used by myself and collaborators in several behavioral experiments.

Face modeling & tracking – This is a set of hardware and software tools developed at RECVEB that can animate facial expressions on highly realistic human faces in real-time. An important achievement of this project is the ability to animate deformable mesh faces in real-time controlled either through animation scripts or from tracked features on a live human actor's face. This results is a lifelike motion of the animated face.

Future possibilities

We continue to expand the capabilities of high performance affordable IVET technology and believe that the field of virtual humans is on the brink of a major upheaval.  As PC and even consumer gaming consoles continue to break new performance records every few months, we expect that the possibilities for leveraging these advances toward the study of human behavior to be immense.