Progress Report
The Realization of an Avatar-Symbiotic Society where Everyone can Perform Active Roles without Constraint[3] Human level knowledge and concept acquisition
Progress until FY2024
1. Outline of the project
In this project, we work on the acquisition of human knowledge and concepts necessary for advanced cognitive abilities, knowledge sharing among cybernetic avatars (CAs), and understanding of the operator's intentions to realize semi-autonomous functions for CAs that move according to the user's intentions (Figure 1).
This knowledge and concept acquisition requires combining visual information with human-level knowledge and concepts and information from different modalities, including natural language. We also aim for CAs to acquire knowledge and concepts that can be shared with humans through the fusion of different modalities, such as visual information and natural language, and the realization of dialogues for knowledge sharing and acquisition in various real-world environments. Furthermore, observations and dialogues based on the acquired knowledge and concepts are used to understand the intentions of operators and users. Since it is difficult to control the environment for interactions between CA, operators, and users, and since the amount of data obtained is small, we will develop a new method for learning with a small amount of data to acquire new knowledge and concepts.
2. Outcome so far
Effective, smooth communication between the CA and the user is essential for understanding intent and sharing knowledge. A 3D head avatar plays a key role in this process by serving as a dialogue interface and providing a basis for observation and representation. Using a 3D head avatar allows non-verbal cues such as facial expressions, gaze and lip movements to be presented and captured with high fidelity. Integrating these signals with spoken content allows for a richer interpretation, promotes mutual understanding and facilitates the acquisition of human knowledge and concepts through dialogue, ultimately improving intent understanding accuracy.
A 3D head avatar is a digital model that recreates a human head in 3D. It can reproduce facial shapes, expressions, gaze, and mouth movements with high precision. Recently, the use of 3D head avatars has expanded across a wide range of fields, including metaverse, remote communication, medical support, education, and service robots. Interacting through an avatar that resembles the person makes natural and trustworthy dialogue possible, even without face-to-face contact. This contributes to communication that transcends social and physical constraints. This technology has also attracted attention as a means of promoting social participation for diverse groups, including older adults and people with disabilities.
Existing 3D head avatars are generally classified as 2D warping, mesh-based, or neural rendering approaches. These methods have difficulty maintaining multi-view consistency, incorporating non-facial information, and generalizing to unseen identities. In this study, we propose GPAvatar, a framework that can reconstruct 3D head avatars from one or more images with a single forward pass. The framework's key innovation is its point cloud-driven representation field, which enables accurate and expressive facial animation. Additionally, a novel attention module enables the integration of information from multiple input images. GPAvatar achieves faithful identity preservation, precise expression control, and multi-view consistency. It demonstrates promising results in free viewpoint rendering and novel view synthesis.
In the latter half of FY2024, we made further improvements to GPAvatar. Although GPAvatar relies on NeRF (Neural Radiance Fields), it had high rendering costs and slow generation speeds. To address these issues, we developed GAGAvatar: a versatile, animatable, Gaussian-based head avatar that can be reconstructed from a single image in a single forward pass (Figure 2). Our technical innovation lies in the dual-lifting strategy that generates high-fidelity 3D Gaussian distributions that capture fine facial details and identity. Furthermore, we use global image features and 3D morphable models to create Gaussians for controlling expressions. After training, the proposed model can reconstruct unseen identities without personalized optimization and enable real-time rendering. Experimental results demonstrate that our method outperforms existing approaches in terms of reconstruction quality and expression accuracy.
3. Future plans
The challenge will be to advance and integrate each element in the future.