Robots are now being seen in shopping centers and inside restaurants. Since these robots are pre-programmed with movements and conversations, they mainly interact in a formulaic way. If humans could remotely control and interact with these robots, we would be able to work locally while being far away. This issue was addressed through the "Sota100 Project" with the theme of "working with teleoperated interactive robots."

The project is conducted by the Frontier Intelligent System Collaboration Laboratory ^(*1) of the Graduate School of Engineering Science, Osaka University (Graduate School of Engineering Science and CyberAgent AI Lab) ^(*2) as part of a Moonshot Research & Development Program ^(*3) .

The project focused on fields where the COVID-19 pandemic has greatly restricted face-to-face activities and interfered with business operations. A nursery school, a supermarket, and an amusement park, which have agreed to cooperate in this experiment, were chosen as the demonstration fields to provide customer service through teleoperated interactive robots. The project members investigated whether it would be possible to solve problems and create value for the "users who receive robot customer service," "operation operators," and "demonstration fields."

As a result, it was found that teleoperated interactive robots could contribute to solving the problems faced by each demonstration field. In addition, the benefits and problems of the actual implementation of these robots were clarified.

*1 Frontier Intelligent System (CyberAgent) Collaboration Laboratory
A joint research laboratory between the R&D organization of CyberAgent "AI Lab" and Professor Ishiguro, which was launched on April 1, 2017. The aim is to establish basic technologies for the realization of interactive agents, including robots, that can be harmoniously involved with human and society, and to acquire scientific knowledge about human interactive capabilities.

*2 Research & Development Project "The Realization of an Avatar-Symbiotic Society where Everyone can Perform Active Roles without Constraint"
The project, for which Professor Ishiguro serves as PM, is one of the research teams that aim to achieve "Goal 1: Realization of a society in which human beings can be free from limitations of body, brain, space, and time by 2050" under the moonshot R&D project. Multiple robots capable of hospitable interactive behavior will be freely operated remotely to realize the ability to perform a variety of social activities (work, education, medical care, daily life, etc.) without having to go to the site.

*3 Moonshot Research & Development Program
The Moonshot Research & Development Program is a new program under which the government promotes challenging research and development (moonshots) based on bold ideas that are not extensions of conventional technologies, with the aim of creating disruptive innovations originating in Japan. Of the Moonshot Research & Development Program sets, the Japan Science and Technology Agency (JST) is in charge of "Goal 1, 2, 3, 6, 8, and 9."

Due to the COVID-19 pandemic, face-to-face activities were greatly restricted in a variety of settings. In this context, the presence of robots drew attention. Robots capable of communicating that can be teleoperated by humans are expected to contribute to the solution of issues such as "manpower shortage due to restricted access to outside collaborators caused by admission restrictions" and "difficulty of contactless communication with customers".

Even before the COVID-19 crisis, there was a growing need for human resource matching and the introduction of robots due to the social background of the promotion of work style reforms, the global trend of productivity improvement, and the decrease in the working population caused by the declining birthrate.

As a means of solving these issues, there has been a great deal of interest in the possibility of remote employment, in which remote personnel can provide the same or better experience as local staff via robots.

This demonstration project used "a teleoperated robot system that can work anytime, anywhere, and with anyone" developed by the Frontier Intelligent System Collaboration Laboratory. The system is mainly based on video communication technology using WebRTC, and also integrates technologies such as real-time voice conversion and robot motion generation by voice recognition. It realizes a framework where the operating operator can "just talk" and perform basic operations without complicated manual operations.

Part 1: "Teleoperated interactive robot x Childcare"
Conducting greeting exercises and activities using a teleoperated interactive robot

Due to the COVID-19 pandemic, nursery schools were restricted from letting in outside personnel, which made it impossible to conduct activities like greeting exercises and activities involving local collaborators and outside instructors. The experiment at the nursery school aimed to solve this situation by using a teleoperated interactive robot.

In this experiment, the operator was able to interact with the local children from a remote location. The children accepted the presence of the robot and were able to create a satisfying time together, which may promote the same "psychological care" effect as when interacting face-to-face.

As a future prospect, it will be important to make it possible for robots alone to take on some of the roles of nursery teachers. In order to achieve this, the technological challenge is to create a sensing environment that can understand what infants want to say from their "non-verbal responses" such as facial expressions, gestures, and eye contact, as little children are not yet developed enough to communicate through words alone.

Part 2: "Teleoperated interactive robot x Supermarket"
Conducting sales promotion using a teleoperated interactive robot

Due to the spread of COVID-19, employees at supermarkets were discouraged from actively serving customers. To solve such a communication shortage with customers, a teleoperated interactive robot was used for contactless conversation. In the experiment, recipe flyers were distributed at the shop front, and promotion activities for specific products were conducted within the shop remotely through a robot.

The number of flyers distributed and the percentage of customers stopping by were improved, and the supermarket involved in the experiment gave high marks to the teleoperated interactive robot that was able to "answer questions better than expected." Successful communication between the robot and the customer became the content, and the service (guidance, assistance, caution, recommendation, etc.) was provided during the inspiring experience. However, its contribution to actual sales promotion is weak, and the future challenge will be to balance the need to instantly satisfy or eliminate consumers' attention to the robot and to generate interest in the product to be recommended.

Part3: "Teleoperated interactive robot x Amusement Park "
Exploring new styles of entertainment provided by a teleoperated interactive robot

Amusement parks are required to guide and assist visitors with consideration for countermeasures against infection so that they can enjoy themselves safely even during the COVID-19 pandemic. The aim was to solve the problems related to this situation with the teleoperated interactive robot. This experiment investigated whether non-contact interaction with a teleoperated interactive robot would be accepted by visitors and how much more exciting the experience could be compared to usual visitor service.

As a result, the teleoperated interactive robot service has achieved a high level of visitor satisfaction and revisit intentions. In addition, if the robot can provide careful guidance and intimate dialogues tailored to each visitor, it can be expected to make a significant contribution to increasing the rate of repeat visits, which amusement facilities are focusing on.

It was also found that a minimum level of skill and knowledge of the robot operation can be acquired simply by the simultaneous operation and customer service with several staff members who share the successful task experience.

In the nursery school, supermarket, and amusement park where the demonstration experiments were conducted, it was shown that teleoperated interactive robots can contribute to solving certain problems and creating value in each facility.

However, the limitation of the study is that the results of this experiment could not be compared with the activities of local childcare staff and actual face-to-face customer service. Therefore, please note that this report is based on subjective evaluations by facility staff and users in contrast to the usual procedures, and is thus insufficient as a rigorous conclusion.

In the future, the research on the effects of teleoperated interactive robots will be conducted in industries and business fields different from those in which the demonstration experiment was conducted. At the same time, technologies will continue to be developed to achieve the Moonshot goal: "Development of technologies and infrastructure that allow one person to operate more than 10 avatars for one task at the same speed and accuracy as one avatar by 2030."