Progress Report

Cybernetic Avatar Technology and Social System Design for Harmonious Co-experience and Collective Ability3. Technology that combines the abilities of multiple bodies into one (Collective Ability Research Group)

Progress until FY2022

1. Outline of the project

A body that can integrate
its own and others' skills
to go beyond
individual capabilities

We will develop technologies to share and integrate the diverse physical skills of multiple people simultaneously connect to a single cybernetic avatar (CA). The system allows users to act while sharing each other's senses and capabilities. By realizing CAs that can support sharing and integration of advanced skills and demonstrate beyond individual capabilities, we will contribute to the achievement of "freedom from physical limitations" as stated in Goal 1.

Yoshihiro TANAKA (Nagoya Institute of Technology)

Development of skill co-creation technology that embraces physical diversity
Yoshihiro TANAKA (Nagoya Institute of Technology)

Hirotaka OSAWA (Keio University)

Development of intention mediation and physical response control technology in body fusion
Hirotaka OSAWA (Keio University)

Hitoshi HIRATA (Nagoya University)

Enabling intrinsic ability activation for frailty treatment by cybernetic avatar technology
Hitoshi HIRATA (Nagoya University)

2. Outcome so far

  • (1) Development of a CA platform for skill fusion
  • (2) Realization of remote co-creation by people with severe disabilities using skills fusion CA
  • (3) Development of a method for remote sharing and learning of skills through the transmission of tactile sensations

In (1), with the aim of enabling physical co-creation across people with different skills and experiences, we developed "Collaborative Avatar," a CA platform for skill fusion in which two individuals simultaneously operate a single robot to perform collaborative tasks. The two operators can share a single physicality by merging their movements according to a specific ratio or by dividing the roles among themselves.

In (2), the skill fusion CA is used to support employment opportunities for people with disabilities. For one month, two people with severe disabilities (who have limited mobility in upper and lower limbs, and experience difficulties going out) used the CA to handcraft customized pancake decorations in real time for customers at an alter-ego robot café. Through this demonstration experiment, it was shown that the operators felt a sense of agency in working remotely and in fusion through the CA, and that they felt an expansion of their individual abilities and creativity through collaboration with others, clarifying that the skill fusion CA can achieve both smooth cooperation and coordination and support the exercise of individual creativity.

In (3), we are developing an infrastructure for remote sharing of skills based on the sharing of tactile sensation with the aim of realizing direct skill sharing from person to person over a network. By measuring the tactile sensation from fingertips and remotely transmitting and sharing the experience alongside video images, we have developed prototypes of a remote palpation system that enables a physical therapist in a remote location to evaluate contractures in a patient, as well as a system that enables users to experience the skills of a craftworker in a remote location. In addition, we have proposed a method to facilitate learning skills by superimposing the tactile experience of a skilled person on oneself across time and space while performing the target task, aiming for higher efficiency in the transmission of skills.

(1) Development of a skill-fusion CA platform (1) Development of a skill-fusion CA platform
(1) Development of a skill-fusion CA platform
(2) Demonstration experiment of pancake toppings by two people with severe physical disabilities utilizing a skill-fusion CA (2) Demonstration experiment of pancake toppings by two people with severe physical disabilities utilizing a skill-fusion CA
(2) Demonstration experiment of pancake toppings by two people with severe physical disabilities utilizing a skill-fusion CA
(3) Trial of skill sharing (palpation/woodworking) and skill learning by transmission of tactile sensation (3) Trial of skill sharing (palpation/woodworking) and skill learning by transmission of tactile sensation (3) Trial of skill sharing (palpation/woodworking) and skill learning by transmission of tactile sensation
(3) Trial of skill sharing (palpation/woodworking) and skill learning by transmission of tactile sensation

3. Future plans

We are promoting fusion between human and AI in CA operation models (AI) and skill fusion between three or more people, aiming at social application and advancement of skill fusion in CA. In fusion with AI, we aim not only to provide functional support but also to co-creation. For direct integration of skills between people, in addition to sensory sharing, we will introduce muscle stimulation, etc., and work on sharing by input/output of motor information. We will verify the effectiveness of sensory and motor sharing for the transmission of skills and prevention of frailty in the elderly.