Progress Report

Last updated:

Development of “Jizai Hon-yaku-ki (At-will Translator)” connecting various minds based on brain and body functions[3] Functions of JIZAI Translator

«1234 and 567A1»

Progress until FY2025

1. Outline of the project

In R&D Item 3, we aim to develop JIZAI Translator itself and, specifically, its key functions necessary to support our everyday interactions.

JIZAI Translator consists of two components: an interpreter “reads” the user’s mental state and an expresser “conveys” it to another user.
The primary task of this R&D Item is to develop the two parts with sensitivity to the diversity of contexts and our personalities, so that JIZAI Translator can assist our mundane communication.

Fig.
Illustation of JIZAI Translator (when A speaks to B)

2. Outcome so far

  1. Developed a framework for visualizing users’ mental states by integrating various biosignals
  2. Developed an XR conversation system that enables comfortable communication with others
  3. Developed an AI-agent-based support system that helps users naturally become aware of their own emotions
  4. Successfully estimated speakers’ emotions from speech features
  5. Automatic extraction of nonverbal features and classification of individuals with ASD and TD individuals based on extracted features

Outcome 1: We developed a system that visualizes users’ mental states in real time based on biosignals, including facial expressions, speech, heart rate, and EEG activity.

Fig.
Outcome 1 — illustrative summary
Source: Y. Nagai (U Tokyo)

Outcome 2: We developed an XR conversation system that provides a comfortable communication environment by allowing the appearance and richness of facial expressions to be flexibly adjusted according to each user’s characteristics.
Outcome 3: We developed a support system, called the Mirroring Agent, that estimates users’ emotions from their facial expressions and reflects them in an AI agent. This system enables users to naturally become aware of their own emotions, even when they have difficulty recognizing them.

Fig.
Outcomes 2 & 3 — illustrative summary
Source: M. Inami & H. Saito (U Tokyo)

Outcome 4: We developed a model that estimates gender and emotions from various speech features, achieving high classification accuracies of 98.4% for gender and 77.0% for emotions.

Fig.
Outcome 4 — illustra-tive summary
Source: F. Homae (Tokyo Metropolitan U)

Outcome 5: We automatically extracted nonverbal behavioral features, including posture and gaze (396 and 117 features, respectively), and successfully classified individuals with ASD and typically developing individuals with high accuracy. We also identified ASD-characteristic behavioral indices, such as face-looking ratio and gaze-transition patterns.

Fig.
Outcome 5 — illustrative summary Source: M. Hariyama (Tohoku U)

3. Future plans

We will continue developing an interpreter and an expresser that are sensitive to contexts and personalities. In parallel, we attempt to develop a proof-of-concept product of JIZAI Translator by incorporating the findings from the other five R&D Items.

(U Tokyo: Y. Nagai, M. Inami, H. Saito,
Tokyo Metropolitan U: F. Homae, Tohoku U: M. Hariyama)

«1234 and 567A1»