Progress Report
Development of “Jizai Hon-yaku-ki (At-will Translator)” connecting various minds based on brain and body functions[3] Functions of Jizai Hon-yaku-ki
Progress until FY2023
1. Outline of the project
In R&D Item 3, we aim to develop Jizai Hon-yaku-ki itself and, specifically, its key functions necessary to support our everyday interactions.
Jizai Hon-yaku-ki consists of two components: an interpreter “reads” the user’s mental state and an expresser “conveys” it to another user.
The primary task of this R&D Item is to develop the two parts with sensitivity to the diversity of contexts and our personalities, so that Jizai Hon-yaku-ki can assist our mundane communication.
2. Outcome so far
- Developed tactile feedback devices to create various textures
- Developed a system to share visual perspectives with others
- Developed a computational model to infer emotional states from speech
- Verified the importance of facial information from speakers for understanding unclear speech
- Devised a method to evaluate the naturalness of conversations
Outcome 1: We have developed a new device that allows for the manipulation of various vibration parameters to create a sensation similar to the movement of materials felt at the fingertips. This device can serve as a foundational technology for tactile interaction with others and communication assistance. Currently, we are preparing to conduct user experience evaluations targeting individuals with developmental disabilities as part of R&D Item 4.
Image provided by Prof. Masahiko Inami (U Tokyo)
Outcome 2: We have developed a system using head-mounted displays that enables users to experience viewing things from others’ perspectives. Sharing perspectives can serve as a useful function for communication support. Currently, we are verifying when shared perspectives, such as viewing the same object as others or synchronizing with heartbeats, affect the user's emotional state.
Image provided by Asst. Prof. Hiroto Saito (U Tokyo)
Outcome 3: We have developed a model to estimate emo-tions from voice and found synchronized emotional changes between infants and caregivers using their voice, aiding interpretation device development for diverse users.
Outcome 4: We have found that listening to unclear speech while observing the speaker's face allows for more accurate comprehension. This finding is valuable for de-veloping devices aimed at better understanding others' words.
Outcome 5: We have devised a new method using large-scale language models (GPT) to assess conversation continuity. This is valuable for developing interpretation devices, reflecting diverse personalities and contexts.
3. Future plans
We will continue developing an interpreter and an ex-presser that are sensitive to contexts and personalities.
In parallel, we attempt to develop a proof-of-concept product of Jizai Hon-yaku-ki by incorporating the findings from the other five R&D Items.
(U Tokyo: Y. Nagai, M. Inami, H. Saito,
Tokyo Metropolitan U: F. Homae, Tohoku U: M. Hariyama)