In our project, a new humanoid robot has been developed (Figure 1). Our most recent endeavor undertook a collaboration with Duke University, by detecting walking related information from a monkey's brain activity while walking on a treadmill, we were able to relay these data from U.S.A to Japan, we were then able to control our humanoid robot in real time in Japan to walk in a similar manner as the monkey for the first time in the world (Figure 2).
Our results consist of 6 major contributions:
These results were conducted as part of the JST Basic Research Program ICORP Type, Computational Brain Project (Research Director Dr. Mitsuo Kawato of ATR, Japan) and with Duke University (Prof. Miguel A. L. Nicolelis of Neuroscience department). The core of this latest study will be released to the press on 2008/1/15 in U.S.A.
[Background]
Recent studies of how human brain generates behaviors are progressing rapidly, and the development of humanoid robots that act like humans is now a major topic of research. Despite of these progresses, we have yet to sufficiently understand how human daily behaviors are combined, generated robustly and autonomously. In our project, we derived information-processing models of the brain and verified these models on real robots in the view of gaining a better understanding of human brain mechanisms involved in generating behaviors. Furthermore, we aimed to develop humanoid robots that can generate compliant behavior like humans.
As part of our project, commencing in year 2005 (H.17) we undertook the development of a new humanoid robot along with our ICORP project partner, Prof. Christopher G. Atkeson of the Robotics Institute, Carnegie Mellon University. The hardware of robot was developed by the robotic development company. SARCOS (based in the USA, http://www.sarcos.com/). From early last year, we progressed to a collaborative study with the laboratory leaded by Prof. Miguel A. L. Nicolelis, whom is a world leader in the research area of Brain Machine Interface, at Duke University. This collaborative study focused on development of artificial walking device utilizing brain activity.
[Contents and results]
1. Development of new humanoid robot
Feature of the newly developed humanoid robot, CBi (Computational Brain interface) Figure 1
Our new robot extends to higher performance, which can generate full-body compliant movements, which extends from our earlier investigations on the humanoid robot DB (Dynamic Brain), developed during the JST ERATO, Kawato Dynamic Brain Project (1996-2001). Furthermore, information of humans' motor performance studies (eyes movements, motor control) acquired during JST ERATO project was incorporated in the development of CBi. In addition, this new robot (CBi) has increased degrees of freedom (51) with wider rage of motion, and similar performance as that of humans.
2. Bipedal locomotion of robot utilizing real-time network brain interface
In our current study here, we present a world-first proof of concept, in a demonstration of the control of a full-sized humanoid robot via signals taken directly from cortical activity of the brain in real time. An overview of the overall experimental setup is given in Figure 2.
At the Duke University, two monkeys were trained to walk bipedally on a treadmill. In each monkey, activity of several hundreds of neurons was recorded from the leg representation in their sensorimotor cortex, and was converted into predictions of the position of leg joints. These predictions were then adapted for controlling a humanoid robot. In the system that is being developed, feedback is delivered from the robot back to the monkey in the form of visual and somatosensory inputs.
A stream-based network architecture has been made available to allow neural signal recordings, and their derivative control signals to the robot, to be transmitted from the Nicolelis lab in the USA to our robot in ATR, Japan. We show that we can control the humanoid robot CBi through the neural data sent via this network interface. The developed robot control architecture accommodates neural signals at various levels: from fine-grained joint-level control to locomotive control .
[Future extensions]
Our newly developed humanoid robot can widely contribute to area of Brain Machine Interface research at multiple levels. As in the current biped walking study, that was realized through a bi-directional network interface (with visual feedback) between the brain and our robot. Based on this result, we could say that we brought forward a big step to the realization of neural prosthetic device that could one day restore lower limb motor functions to paralyzed patients.
Currently, we are developing a system that could induce feedbacks, visually as well as somatosensory input to the brain via cortical microstimulation. Brain Machine Interface that utilizes proper feedback signal from our robot to the brain forms part of our future scientific investigation.
Figure 1 A newly developed 51 degrees of freedom humanoid robot, CBi. |
Figure 2 Experimental overview |
Technical terms |
Sample Movie (mov format 3,891KB) |
[Publications]
[a] Kawato, M. (2008). From "Understanding the brain by creating the brain" toward manipulative neuroscience. Philosophical Transactions of the Royal Society B, in press.
doi: 10.1098/rstb.2008.2272
[b] Cheng, G., Fitzsimmons, N. A., Morimoto, J., Lebedev, M. A., Kawato, M., & Nicolelis, M. A. L. (2007). Bipedal locomotion with a humanoid robot controlled by cortical ensemble activity. Society for Neuroscience 37th Annual Meeting. San Diego, CA, USA.
doi: 10.13140/2.1.1315.2645
[c] Cheng, G., Hyon, S., Morimoto, J., Ude, A., Hale, J. G., Colvin, G., Scroggin, W., & Jacobsen, S. C. (2007). CB: A humanoid research platform for exploring neuroscience. Journal of Advanced Robotics, 21 (10), 1097-1114.
doi: 10.1163/156855307781389356
[d] Morimoto, J., Endo, G., Nakanishi, J., Hyon, S., Cheng, G., Bentivegna, D., & Atkeson, C. G. (2006). Modulation of simple sinusoidal patterns by a coupled oscillator model for biped walking. IEEE International Conference on Robotics and Automation (pp. 1579-1584). Orlando, FL, USA.
doi: 10.1109/ROBOT.2006.1641932
[e] Kawato, M. (2008). Measurement and control in brain machine interface, Journal of the Society of Instrument and Control Engineers, Vol.46, No.12, pp. 958-963.
PRESS CONTACTS
JST
Computational Brain Project
2-2-2, Hikaridai Seika-cho Soraku-gun, Kyoto 619-0288 JAPAN
Phone:+81-774-95-2404 Fax:+81-774-95-1236
E-mail:
JST
Public Relations Division
5-3, Yonbancho Chiyoda-ku, Tokyo 102-8666 JAPAN
Phone: +81-3-5214-8404 Fax: +81-3-5214-8432
E-mail:
Internet: http://www.jst.go.jp/EN/
Duke University
Melissa Schwarting
Senior Media Relations Strategist
Duke University Medical Center News Office
220 W Main Street, Suite 910-B
Durham, NC 27705
Phone: +1-919-660-1303
E-mail: