Progress Report

The Realization of an Avatar-Symbiotic Society where Everyone can Perform Active Roles without Constraint[4] Cooperative control of multiple CAs

«12345678»

Progress until FY2024

1. Outline of the project

This R&D item is responsible for basic research and development of technology for simultaneous remote control and coordinated control of multiple Cybernetic avatars (CAs) (flexible CA control). Achievement of this R&D item will enable a single operator to operate multiple CAs simultaneously, contributing to the realization of an avatar symbiotic society, which is the goal of the project, and to the liberation from space and time constraints, which is the goal of the Moonshot target.
To achieve this, the challenge is to develop technology to use multiple CAs that work while understanding the operator's intentions according to the task and environment. In this R&D item, first, the living and hospital room environments in which CAs work together are actually reproduced, and the semi-autonomous CAs that work in these environments are prototyped based on autonomous CA technology that can be made autonomous from remote operation (Figure.1). Then, a semi-autonomous cooperative control system consisting of multiple CAs will be developed, including BMI and the operation interface.

Fig.1
Figure 1: Multi-CA collaboration in a hospital environment

2. Outcome so far

(1) Integration of Multi-CA Collaboration

A comprehensive demonstration experiment was conducted at Mirai Hive in Osaka Umeda, where two operators (in charge of operation, command, and dialogue) remotely controlled over ten heterogeneous lifestyle-support CAs across five simulated environments: home healthcare, hospital, household, office, and commercial facility. The system successfully verified that multiple CAs could work collaboratively, divide roles, and complement each other autonomously to execute tasks efficiently.

(2) Adaptive CA Manipulation

We developed a deep imitation learning technology that enables operators to flexibly select target objects. This allows for adaptable operations not limited to preset commands, making it possible to respond to dynamically changing environments and objects in real time (Figure.2: Left & Middle).

(3) XR Teleoperation Interface

We realized a predictive teleoperation system using augmented reality (AR). Manipulation and navigation actions are visualized on XR as future predictions, providing operators with an intuitive interface for CA control (Figure.2: Right).

Fig.2
Figure 2: (Left) Intuitive tapping on target object (Middle) Motion generated by robot foundation model (Right) XR-based Predictive Teleoperation
(4) Technology for Environmental Understanding

We improved and conducted a large-scale demonstration of the real-world search engine. The system now supports environments over 100,000 square meters and enables instant language-based search of targets. A representative example includes a large-scale demonstration at the Dubai Mall (Figure.3). A smartphone-compatible real-world crawler was also developed.

Fig.3
Figure 3: (Left) Product shelves in Dubai Mall (Right) Search results
(5) Semi-Autonomous Robotic Hand

A proximity-sensing adaptive robot hand was developed, featuring a “conforming grasp” control mechanism in which the grasping posture is pre-shaped according to the object's shape. This greatly improved the naturalness and stability of CA grasping actions (Figure.4).

Fig.4
Figure 4: Proximity-sensing adaptive robot hand

3. Future plans

We aim to scale up the simultaneous remote operation and autonomous coordination of multiple CAs. Key technologies—such as LLM-based task decomposition, proximity-adaptive hands, and XR teleoperation—will be integrated into a unified system where CAs coordinate through intent arbitration and role sharing. We also seek to enable a single operator to flexibly manage diverse CAs using autonomous, semi-autonomous, shared, and remote control modes. Public demonstrations and validation will support future social deployment.

«12345678»