Progress Report
Liberation from Biological Limitations via Physical, Cognitive and Perceptual Augmentation[3] IoB Core technology
Progress until FY2024
1. Outline of the project
The ability to safely measure neural activity is a core technical element of brain-machine interface (BMI) development. The Internet of Brains (IoB) Core Technology aims to develop BMI technology that enables users, if they wish, to expand their capabilities in daily life using safe and efficient invasive technology. We are developing the measurement and decoding technologies for human using animals (Fig.1).
We will develop invasive BMI using animals and apply it to humans. In particular, we will develop technology that uses AI to decode imagery contents and intentions and input information into the brain.
We have developed a technique for recording and manipulating neural activity with high precision and long-term stability by implanting electrodes in the brains of marmosets and monkeys. We have also succeeded in decoding animal communication from neural signals. Furthermore, we have developed a technology to stimulate the mouse brain to input information. Furthermore, based on experiments conducted on humans, we have developed a BMI that outputs images imagined by humans and a BMI that enables avatars to speak words that paralyzed patients attempt to say. In collaboration with the IoB Middleware Development Group, we will develop X-communication, which uses AI technology for invasive brain signals.
2. Outcome so far
[Successful extraction of diverse brain information from the cerebral cortex of primates]
A group led by Associate Professor Misako Komatsu at Institute of Science Tokyo has been working on the development of a system for marmosets, primates that are vocal communicators, to enable animals to communicate online like humans. So far, they have set up a wireless measurement system for marmoset wide-area electrocorticogram (ECoG) to measure speech, behavior and neural activity during vocal communication with other individuals, and have succeeded in reading different information: behavioral categories and vocalization types (Fig.2).
（Komatsu et al, BCI Meeting, 2023）
[Successful neural information transmission at the cellular level]
Using holographic two-photon microscopy, Professor Wake at National Institute for Physiological Sciences has combined this with optogenetics and succeeded in creating artificial sensation (Fig.3). Furthermore, information transmission between individuals is being successfully achieved.
[Developing a BMI that outputs images and words as intended by the user]
Professor Yanagisawa at Osaka University has developed BMI to retrieved images from intracranial EEG of human to present the intended images on the screen (Fig.4). In collaboration with the IoB Middleware Group, this will be applied to technology for extracting and communicating diverse concepts.
(Fukuma et al., Comm.Biol., 2022)
In addition, Professor Edward Chang at University of California, San Francisco has implanted electrodes in the brains of patients with speech difficulties due to illness and measured ECoG. By decoding the ECoG when the patient intended to speak, the AI generated sentences at a rate of 78 words per minute, which were successfully uttered by the avatar. (Fig.5). This achievement shows that BMI-based communication of intent has reached a practical level.
(Metzger et al., Nature, 2023)
3. Future plans
- We have developed technology that enables long-term stable invasive neural recordings and the input of diverse information into the brain. Going forward, we will develop technology that achieves BMI with even lower invasiveness while increasing signal accuracy and the amount of input information.
- We have established a method to estimate information corresponding to memory retrieval and language from human ECoG signals and generate speech or images. We will continue to improve accuracy and output to develop more practical communication technologies.
- By combining AI with inter-individual information transmission technology, we will realize new communication technologies.