Progress Report
The Realization of an Avatar-Symbiotic Society where Everyone can Perform Active Roles without Constraint[1] Development of a CA which has a humanlike presence and a lifelikeness
Progress until FY2024
1. Outline of the project
The main objective of this R&D item is to develop cybernetic avatars (CAs) with humanlike appearance and functionality, as well as CAs with lifelike characteristics. Additionally, we will create mobile CAs with childlike features that can extend their presence and lifelikeness by being able to move freely in the environment, and huggable CAs supposed to be used in mental health care. Furthermore, an automatic motion generation system will express the teleoperator's intentions, emotions, and personality through CAs. An immersive interface will provide operators with free control over the CAs.
Through these R&D efforts, CAs will serve as substitutes or avatars for teleoperators in social situations, overcoming spatial and temporal constraints. It will enable us to effectively accomplish and excel in tasks that require simultaneous presence at multiple locations.
2. Outcome so far
1. Development of an Autonomous Dialogue System That Generates Speech Reflecting Operator’s Thought Patterns
We developed an autonomous dialogue system that reflects an operator’s personality and thought patterns, and implemented it in the Sonzai-Kan CA “HI-6” (Figure 1). Using Retrieval-Augmented Generation (RAG), the system generates utterances based on digitized data extracted from sources such as the operator’s writings. The CA is also equipped with an automatic gesture generation function that produces expressive motions reflecting the utterance content, further enhancing its ability to convey a presence comparable to that of the operator.
2. Gamification of CA Operation
We developed a gamified system that motivates operators to control a CA by incorporating game-like elements into the interface. Operators earn “CA operation points” by completing tasks, such as attracting passersby at a commercial site. As points accumulate, more control options become available, creating a positive feedback loop between actions taken and new possibilities. Groups of high school or elmentary students were observed en-joying the operation process (Figure 2).
3. Learning Support for Children Using a Huggable CA
We provided reading and English learning support to preschool children using a huggable CA. Placed behind the child engaged in learning activities at childcare facilities, the CA interacted through conversation and gentle touch, helping to enhance his/her motivation to learn (Figure 3). According to a que-stionnaire survey conducted with parents, it was confirmed that the CA provided psycho-logical comfort to the children.
4. Development of the Mobile CA “Yui”
We developed the mobile CA “Yui,” which has a child-like appearance (Figure 4). A head-mounted display is used as the operation interface, allowing the operator to control the CA freely in a way that feels like inhabiting the CA. Yui synchronizes with the operator’s gaze, facial expressions, and whole-body movements, enabling natural behavior that reflects the operator’s personality. Yui will also be used in a demonstration experiment at the signature pavilion “Future of Life” at Expo 2025 Osaka, Kansai.
5. Development of a Bidirectional Avatars System
We developed a bidirectional avatars system that enables pseudo face-to-face interaction by combining systems that directly reflect the operator’s gestures onto CA in both directions (Figure 5). This system allows individuals located remotely to engage in dialogue accompanied by embodied interaction while experiencing each other’s presence.
3. Future plans
The CAs to be developed under this R&D item will free the operator from physical, spatial, and temporal constraints by taking their place. Furthermore, we aim to develop CAs with functions and capabilities that go beyond mere substitution for humans. Through such CAs, we will not only be able to work in different places and at different times, but we will also be able to demonstrate abilities and behaviors that we would not normally be able to do.