Progress Report
Liberation from Biological Limitations via Physical, Cognitive and Perceptual Augmentation3. IoB Core technology
Progress until FY2022
1. Outline of the project
The ability to safely measure brain activity is a core technical element of all BMI development. The IoB Core Technology aims to develop BMI technology that enables users, if they wish, to expand their capabilities in daily life using safe and efficient invasive technology. We are developing the measurement and decoding technologies for human using animals (Figure 1).
We will develop invasive BMI using animals and apply it to humans. In particular, we will develop technology that uses AI to decode imagery contents and intentions and input information into the brain.
We have developed a technology to record and manipulate neural activity with high precision by penetrating electrodes into the brains of marmosets and macaques. By measuring long-term stable neural activities, we have succeeded in decoding animal communication from brain signals. In collaboration with the IoB Middleware Development Group, we have developed brain decoding technology that utilizes AI technology on invasive brain signals to build an AI-assisted Trusted BMI-CA.
2. Outcome so far
[Decoding of multimodal information from the primate cerebral cortex]
A group led by Misako Komatsu, a special associate professor at Tokyo Institute of Technology, has been working on the development of a system for marmosets, primates that are vocal communicators, to enable animals to communicate online like humans. So far, the Komatsu group has successfully set up a wireless measurement system for wide-area ECoG of marmosets to measure speech, behavior, and neural activity during vocal communication with other individuals. They succeeded to decode behavioral categories and types of vocalizations of marmosets, and developed a system that will enable marmosets to communicate online like humans. They found that a higher cortical area called the association cortex commonly contributes to decode both types of information. This research shows the possibility of extending BMI, which has been developed in a single modality, to multimodal applications. These results were reported at an international conference of BMI.
Figure 2 Marmoset wide-area ECoG wireless measurement and decoding
[Displaying images on the screen that people have imagined]
A group led by Professor Takufumi Yanagisawa of Osaka University has been developing a technique to stably measure intracranial EEG in patients with epilepsy. Using this technique, the group has measured highly accurate intracranial EEG from human visual cortex and shown that the imagined image can be presented on the screen when the human imagined the image. This new BMI was reported in a peer-reviewed journal and will be applied to decode imagery contents and intentions for communication.
3. Future plans
- Nishimura Gr. will verify the long-term stability of the brain-muscle electrode implantation technique in macaque monkeys.
- Yanagisawa Gr. will develop a method to decode imagery contents and internal language from ECoG to generate speech and images.
- Komatsu Gr. will connect the multimodal decoder to the XR space for marmosets and realize real-time communication there.