Research Results

Human beings can be free from limitations of their bodies and space

Three Demonstration Experiments on "Working with Teleoperated Interactive Robots"FY2022

写真:石黒 浩
Ishiguro Hiroshi (Professor, Graduate School of Engineering Science, Osaka University)
Moonshot Research & Development Program
Moonshot Goal 1: Realization of a society in which human beings can be free from limitations of body, brain, space, and time by 2050.
"The Realization of an Avatar-Symbiotic Society where Everyone can Perform Active Roles without Constraint" Project Manager (2020- )

Teleoperated interactive robots contribute to solving problems during the new coronavirus (COVID-19) pandemic.

Robots are now being seen in shopping centers and inside restaurants. Since these robots are pre-programmed with movements and conversations, they mainly interact in a formulaic way. If humans could remotely control and interact with these robots, we would be able to work locally while being far away. This issue was addressed through the "Sota100 Project" with the theme of "working with teleoperated interactive robots."

The project is conducted by the Frontier Intelligent System Collaboration Laboratory (*1) of the Graduate School of Engineering Science, Osaka University (Graduate School of Engineering Science and CyberAgent AI Lab) (*2) as part of a Moonshot Research & Development Program (*3) .

The project focused on fields where the COVID-19 pandemic has greatly restricted face-to-face activities and interfered with business operations. A nursery school, a supermarket, and an amusement park, which have agreed to cooperate in this experiment, were chosen as the demonstration fields to provide customer service through teleoperated interactive robots. The project members investigated whether it would be possible to solve problems and create value for the "users who receive robot customer service," "operation operators," and "demonstration fields."

As a result, it was found that teleoperated interactive robots could contribute to solving the problems faced by each demonstration field. In addition, the benefits and problems of the actual implementation of these robots were clarified.

*1 Frontier Intelligent System (CyberAgent) Collaboration Laboratory
A joint research laboratory between the R&D organization of CyberAgent "AI Lab" and Professor Ishiguro, which was launched on April 1, 2017. The aim is to establish basic technologies for the realization of interactive agents, including robots, that can be harmoniously involved with human and society, and to acquire scientific knowledge about human interactive capabilities.

*2 Research & Development Project "The Realization of an Avatar-Symbiotic Society where Everyone can Perform Active Roles without Constraint"
The project, for which Professor Ishiguro serves as PM, is one of the research teams that aim to achieve "Goal 1: Realization of a society in which human beings can be free from limitations of body, brain, space, and time by 2050" under the moonshot R&D project. Multiple robots capable of hospitable interactive behavior will be freely operated remotely to realize the ability to perform a variety of social activities (work, education, medical care, daily life, etc.) without having to go to the site.

*3 Moonshot Research & Development Program
The Moonshot Research & Development Program is a new program under which the government promotes challenging research and development (moonshots) based on bold ideas that are not extensions of conventional technologies, with the aim of creating disruptive innovations originating in Japan. Of the Moonshot Research & Development Program sets, the Japan Science and Technology Agency (JST) is in charge of "Goal 1, 2, 3, 6, 8, and 9."

Fig. 1

Fig. 1 Remote avatar system that allows "anyone to work anywhere, anytime"

By working with a teleoperated interactive robot, the project aims to create value through solving the problems of operators, users, and the fields.

"Non-contact" and "remote working" are keywords.

Due to the COVID-19 pandemic, face-to-face activities were greatly restricted in a variety of settings. In this context, the presence of robots drew attention. Robots capable of communicating that can be teleoperated by humans are expected to contribute to the solution of issues such as "manpower shortage due to restricted access to outside collaborators caused by admission restrictions" and "difficulty of contactless communication with customers".

Even before the COVID-19 crisis, there was a growing need for human resource matching and the introduction of robots due to the social background of the promotion of work style reforms, the global trend of productivity improvement, and the decrease in the working population caused by the declining birthrate.

As a means of solving these issues, there has been a great deal of interest in the possibility of remote employment, in which remote personnel can provide the same or better experience as local staff via robots.

Benefits and problems of introducing a teleoperated interactive robot

This demonstration project used "a teleoperated robot system that can work anytime, anywhere, and with anyone" developed by the Frontier Intelligent System Collaboration Laboratory. The system is mainly based on video communication technology using WebRTC, and also integrates technologies such as real-time voice conversion and robot motion generation by voice recognition. It realizes a framework where the operating operator can "just talk" and perform basic operations without complicated manual operations.

Fig. 2

Fig. 2 Features of the teleoperated interactive robot system utilized in this study

The design of the robot's exterior and its voice have been carefully devised so that it can be accepted without resistance by a diverse range of people in various situations.

Part 1: "Teleoperated interactive robot x Childcare"
Conducting greeting exercises and activities using a teleoperated interactive robot

Due to the COVID-19 pandemic, nursery schools were restricted from letting in outside personnel, which made it impossible to conduct activities like greeting exercises and activities involving local collaborators and outside instructors. The experiment at the nursery school aimed to solve this situation by using a teleoperated interactive robot.

In this experiment, the operator was able to interact with the local children from a remote location. The children accepted the presence of the robot and were able to create a satisfying time together, which may promote the same "psychological care" effect as when interacting face-to-face.

As a future prospect, it will be important to make it possible for robots alone to take on some of the roles of nursery teachers. In order to achieve this, the technological challenge is to create a sensing environment that can understand what infants want to say from their "non-verbal responses" such as facial expressions, gestures, and eye contact, as little children are not yet developed enough to communicate through words alone.


Fig.3 The teleoperated interactive robot and children at the nursery school

(Left) The greeting exercise for the children on their way to school was conducted by an elderly person via a teleoperated interactive robot.
(Right) Activities such as picture book reading and wordplays are conducted by an external activity instructor from a remote location using the teleoperated interactive robot.

Part 2: "Teleoperated interactive robot x Supermarket"
Conducting sales promotion using a teleoperated interactive robot

Due to the spread of COVID-19, employees at supermarkets were discouraged from actively serving customers. To solve such a communication shortage with customers, a teleoperated interactive robot was used for contactless conversation. In the experiment, recipe flyers were distributed at the shop front, and promotion activities for specific products were conducted within the shop remotely through a robot.

The number of flyers distributed and the percentage of customers stopping by were improved, and the supermarket involved in the experiment gave high marks to the teleoperated interactive robot that was able to "answer questions better than expected." Successful communication between the robot and the customer became the content, and the service (guidance, assistance, caution, recommendation, etc.) was provided during the inspiring experience. However, its contribution to actual sales promotion is weak, and the future challenge will be to balance the need to instantly satisfy or eliminate consumers' attention to the robot and to generate interest in the product to be recommended.


Fig.4 Stopping rate and purchase rate of customers who visited the supermarket

The percentage of people stopping at the store when seeing the teleoperated interactive robot increased by 139% compared to the usual time, but it did not lead to their purchasing behavior.

Part3: "Teleoperated interactive robot x Amusement Park "
Exploring new styles of entertainment provided by a teleoperated interactive robot

Amusement parks are required to guide and assist visitors with consideration for countermeasures against infection so that they can enjoy themselves safely even during the COVID-19 pandemic. The aim was to solve the problems related to this situation with the teleoperated interactive robot. This experiment investigated whether non-contact interaction with a teleoperated interactive robot would be accepted by visitors and how much more exciting the experience could be compared to usual visitor service.

As a result, the teleoperated interactive robot service has achieved a high level of visitor satisfaction and revisit intentions. In addition, if the robot can provide careful guidance and intimate dialogues tailored to each visitor, it can be expected to make a significant contribution to increasing the rate of repeat visits, which amusement facilities are focusing on.

It was also found that a minimum level of skill and knowledge of the robot operation can be acquired simply by the simultaneous operation and customer service with several staff members who share the successful task experience.

Fig. 5

Fig. 5 The teleoperated interactive robot service achieved high customer satisfaction and reuse intentions.

The seven-point Likert scale was used to evaluate the level of satisfaction with the services provided by the robot, and over 70% of the responses were positive.

Teleoperated interactive robots will keep walking into 2030

In the nursery school, supermarket, and amusement park where the demonstration experiments were conducted, it was shown that teleoperated interactive robots can contribute to solving certain problems and creating value in each facility.

However, the limitation of the study is that the results of this experiment could not be compared with the activities of local childcare staff and actual face-to-face customer service. Therefore, please note that this report is based on subjective evaluations by facility staff and users in contrast to the usual procedures, and is thus insufficient as a rigorous conclusion.

In the future, the research on the effects of teleoperated interactive robots will be conducted in industries and business fields different from those in which the demonstration experiment was conducted. At the same time, technologies will continue to be developed to achieve the Moonshot goal: "Development of technologies and infrastructure that allow one person to operate more than 10 avatars for one task at the same speed and accuracy as one avatar by 2030."