Progress Report
Liberation from Biological Limitations via Physical, Cognitive and Perceptual Augmentation2. IoB Middleware
Progress until FY2022
1. Outline of the project
In order to build an AI-assisted Trusted BMI-CA (Brain-Machine Interface-Controlled Avatar) that can provide the desired support for individuals with physical disabilities, it is necessary to develop technology for transmitting information from the brain and body to the CA.Additionally, the development of CA technology that can provide assistance as desired by the user based on the transmitted information is essential. The 'IoB Middlewaret' is dedicated to the development of 'communication technology between the brain and CA,' using mathematical approaches such as machine learning and information theory.
* AI-assisted BMI-CA: A Cybernetic Avatar (CA) that can accurately decipher the words and actions imagined by the user through the use of machine learning in AI, depending on the combination of different BMI (Brain-Machine Interface) technologies..
In the 'IoB Middleware Development,' we aim to develop BMI technology that supports diverse communication needs, ranging from technologies that interpret speech intentions and serve as alternatives for interpersonal communication to communication of information that is difficult to convey through language, such as recalled images, physical manipulation, cognitive load, and fatigue.
2. Outcome so far
[ChatGPT-supported Brain-Gmail Interface]
Generating AI has made remarkable progress in recent years. Unlike AI systems that select answers from given choices, generating AI can generate answers on its own. By integrating it with BMI, it is expected to enhance user functionality. The group led by Shuntaro Sasai, CRO of Alaya Inc., successfully developed a BMI system that can generate email replies through dialogue with users, leveraging ChatGPT, an AI capable of generating text through conversations. This system decodes colors spoken by users from brain activity and uses that information to manipulate web interfaces of email platforms such as Gmail. Furthermore, ChatGPT automatically generates reply candidates based on received email content and past reply history, allowing for semi-automatic email responses by selecting the desired reply from the candidates.
[Language-based Automatic Control of Robotic Arm]
Shuntaro Sasai, and Alaya Inc’s team leader Kai Arulkumaran have been developing an AI-assisted BMI that can semi-automatically operate a robotic arm based on decoded language from brain activity to perform tasks. The Arulkumaran team has been working on the development of an AI agent that can control a robotic arm automatically based on instructions given in natural language, eliminating the need for precise user control. They successfully conducted an experiment inputting decoded words from the user's brain activity into this AI agent via the Internet, enabling the remote execution of tasks by a robotic arm located in a distant place.
3. Future plans
In the future, we will focus on improving the accuracy of the AI-assisted BMI system currently under development and validating its feasibility and user experience by actual usage among individuals with speech or physical impairments.