The research group led by Kouta Minamizawa, Professor at the Graduate School of Media Design, Keio University is working on the development and social implementation of cybernetic avatar technology,^*1 which will enable the limitations of humans’ physical abilities to be overcome in the society of the future. Through cybernetic avatars, people can transfer and share their physical experiences and skills via a network, and diverse groups of people, including people with disabilities and the elderly, can act freely and take part in social activities.

The cybernetic avatar is, in a sense, the “other self” of an actual person. Cybernetic avatars operated by people sometimes behave as virtual characters in on-screen images, while at other times they act as robots that interact with humans physically in the real world.

To demonstrate part of this research and to offer accessible hands-on experiences of cybernetic avatars to ordinary people, an event entitled the “Cybernetic Avatar Experiment at Avatar Robot Café DAWN ver. β”^*2 was held from May 9 (Tue) to June 16 (Fri), 2023 (Fig. 1). At this event, people who have difficulty going out for various reasons, such as ALS patients, served as pilots (operators) from remote locations by remote-controlling cybernetic avatars at a café in Nihonbashi, Tokyo. This experiment clearly presented a new remote work style that allows people who are unable to go out to serve customers and also demonstrated various other related technologies.

*1: Cybernetic avatar
This demonstration experiment made use of three cybernetic avatar technologies: virtual avatar technology, parallelization technology, and Aiming at a cybernetic avatar society that expands diversity and inclusivity technology. It aimed to construct a creative work environment for pilots as well as to offer as exciting an experience as possible to café visitors.

*2: Avatar Robot Café DAWN ver. β
A permanent experimental café in Nihonbashi, Tokyo operated by OryLab Inc. In this café, employees who have difficulty going out provide service by remotely operating the avatar robots OriHime and OriHime-D. It opened in June 2021.

Cybernetic avatar technologies include not only avatar robots and 3D image avatars but also ICT and robot technologies that augment humans’ physical abilities, cognitive abilities, and perception abilities. The aim is to facilitate free action in the cyber physical space of Society 5.0. The research group led by Prof. Minamizawa is conducting a foundational cybernetic avatar platform and the way how we live in it, with the theme of “Cybernetic Avatar Technology and Social System Design for Harmonious Co-experience and Collective Ability.”

To build the cybernetic avatar platform, after establishing technology that enables a single person to operate multiple avatars to process a task, the number of avatars operated by multiple operators simultaneously must be increased, and multiple tasks must be processed simultaneously. In the end, this platform must enable multiple people to operate many avatars simultaneously to process multiple tasks. As the first step, the target is to develop by 2030 technology and a platform that enable a single person to operate ten or more avatars for performing one task at a speed and with an accuracy equivalent to those of the case of operating a single avatar. As for life lived using such cybernetic avatars, after developing technology that augments the physical, cognitive, and perception abilities in performing specific tasks, it is expected to develop technology that augments the physical, cognitive, and perception abilities to the top level in the given field. Therefore, the 2030 target is to develop technology that augments physical, cognitive, and perception abilities for performing a specific task.

Achieving this target will require various technologies, such as those shown in Fig. 2. Many demonstration experiments, such as this robot café, will be conducted in the process of incorporating these technologies into the cybernetic avatar platform.

In this experiment jointly conducted by the Keio University Graduate School of Media Design, OryLab Inc., and the Nagoya Institute of Technology, a few robots (the avatar robots OriHime and OriHime-D^*3) greeted and guided visitors; waited on customers at each table; and carried and served dishes. From remote locations, the human staff members piloted these robots. They spoke while watching video of the café, and the robots in the café then spoke accordingly.

Since these robots behave as avatars of their pilots, they are called avatar robots. However, each avatar was not represented by a single robot; rather, multiple robots performed the same action at the same time under a single pilot’s operation. For example, the pilot that operated the robot set up at the entrance to greet visitors also operated the robots that waited on those customers at their tables. Thus, different robots performed a series of services, called service by “avatars’ avatar robots,” and a single pilot operated a robot to wait on customers as well as another robot to carry and serve dishes. This demonstration thus implemented operations that cannot usually be performed by a single person (Fig. 3).

In addition, the experiment provided “service by augmented avatars” in which wait staff avatars (CG characters) on-screen changed their appearance according to their pilots’ personalities and the communication between customers and the avatars, allowing the customer moving back and forth between the virtual world and the real world. Moreover, “remote co-creative toppings” was demonstrated, in which multiple pilots operated robots to cooperate in putting toppings on various dishes.

The most important point of this event is that people who were unable to leave their locations for various reasons, such as bedridden ALS patients, served as the pilots operating the avatar robots. A former barista with an intractable disease living in Aichi Prefecture worked as a barista at this café and shared their experience and knowledge with the participants. Human abilities were augmented through avatar robots, and these robots could perform the job of providing service to customers, an extremely human job. The cybernetic avatar technology demonstrated by this experiment that people who have difficulties going out can perform some tasks heretofore impossible by remote work using avatar robots, is a major achievement.

*3: OriHime and OriHime-D
Avatar robots sold by OryLab Inc. Their functions include seeing, talking, and moving; they can be remotely operated from distant locations.

Based on the knowledge acquired through this demonstration experiment, the next mission is to realize a cybernetic avatar society in which everyone can do what they want to do independently and can feel self-growth.

A project of using haptics to transmit tactile sensations that reproduces the sense of touch by digital technology has been launched, and research on transmitting the feel of materials such as textures to distant locations has begun. We expect that cybernetic avatar technology can respond to various social problems that require high expertise to solve. For example, in specialized fields such as disaster relief and infrastructure maintenance, this technology can help experts respond to problems appropriately without visiting the actual sites in person.

However, there are ethical, legal, and social issues with the use of avatars. The research group will clarify these issues regarding daily use of cybernetic avatars and conduct research and consideration on ethics and the design of social systems.