Progress Report
Liberation from Biological Limitations via Physical, Cognitive and Perceptual Augmentation1. Development of IoB Interfaces (IoB: Internet of Brains)
Progress until FY2022
1. Outline of the project
The goal of this theme is to develop technology to extract thoughts and mental states from brain activity using various devices and to implement it in society as an application, thereby promoting the use of brain-machine interface (BMI) technology. Specifically, we are developing algorithms that can extract thoughts and mental states in daily environments for short periods of time by combining gadget-type EEG sensors such as headphones and camera images from cell phones. Using these, we aim to disseminate BMI technology to society by creating applications that enable self-regulation by visualizing daily physical changes of which one is not conscious, and applications that support the communication of intentions of users who are in a state or situation where they are unable to express their intentions externally.
2. Outcome so far
Stylish design, cool to wear. Anyone can use it anywhere, anytime. We have also developed waveform classification AI and inference AI to deal with "sudden noise contamination" and "individual differences in brain wave response," and incorporated them into the system (domestic patent application completed, PCT application in process).
Avatar control
The "BMI Brainpic," in which people with cerebral palsy and neuromuscular intractable diseases and junior and senior high school students used these brainwave sensors to control avatars in the popular game "Fortnite" and compete against each other for time, was held to great success (viewed a total of 33,000 times in Japan and the United States within 3 months after being released on YouTube).
Recently, we have also succeeded in improving the accuracy of BMI control of avatars by using sensors to read "habits" that people do unconsciously, such as leaning or looking in the direction they want to go. We have also succeeded in combining this with technology that allows the AI to recognize obstacles in the avatar's surroundings and avoid collisions fully-automatically. In the future, we plan to apply the IoB interface, which "moves as smoothly as you want," not only to the metaverse but also to real-world environments.
Brain monitoring and training through Avatar
Troubles resulting from brain limitations are widely shared among musicians and music lovers. Performance is unstable from day to day, motor memory is difficult to consolidate, and disease signs are heightened by overtraining.... To solve these problems, we have developed an application to analyze and visualize the acoustics during performance recorded with a consumer microphone and hand posture with a consumer camera. In Japan, we have begun trials at the Music Excellence Project Academy and are planning joint research with the Hanover College of Music and the Munich College of Music. We have also developed and are testing a cognitive-behavioral app that normalizes memory consolidation for parties suffering from traumatic memories.
Furthermore, we are developing a large multidimensional database of functional magnetic resonance imaging, gait data, and diagnostic data for parties with signs of mental illness and have constructed technology to infer mental and physical conditions from the way they walk. We are now able to analyze brain activity components cleanly while walking, which is difficult to do with EEG measurements.
3. Future plans
This fiscal year, we were able to publish world-class academic papers, conduct international joint research activities, and standardize technologies. In the future, we will promote radical innovation while add-on industry-academia collaborative activities, aiming to realize "a society where everyone can pursue their dreams" and "a society where people can enjoy their lives until they are 100 years old without health concerns.