Home > Press Releases
JST Press Release

January 15, 2008

5-3, Yonbancho Chiyoda-ku, Tokyo 102-8666
Japan Science and Technology Agency (JST)
URL http://www.jst.go.jp/EN/

World's First Bipedal Locomotion with a Humanoid Robot Controlled by Cortical Ensemble Activity with a Real-time Network Brain Interface

(By relaying monkey's brain activity from USA to Japan, we were able to control our humanoid robot in Japan to walk in similar manner as the monkey in USA)

Japan Science and Technology Agency (JST) has succeeded to make a humanoid robot locomote via data detected from cortical brain activity that transmitted through an internet interface between U.S.A and Japan in real time. Recent studies of how the brain generates behaviors are progressing rapidly. While at the same time as, development of humanoid robots that act like humans is now part of the focus of robotic research. In our project, we developed information-processing models of the brain and verify these models on real robots in order to better understand human brain mechanisms in yielding behaviors. Additionally, we aimed to develop humanoid robots can compliantly behave like humans in facilitating our daily live.

In our project, a new humanoid robot has been developed (Figure 1). Our most recent endeavor undertook a collaboration with Duke University, by detecting walking related information from a monkey's brain activity while walking on a treadmill, we were able to relay these data from U.S.A to Japan, we were then able to control our humanoid robot in real time in Japan to walk in a similar manner as the monkey for the first time in the world (Figure 2).

Our results consist of 6 major contributions:

-1. Humanoid robot: The development of a new full-sized human-like humanoid robot for Neuroscientific research [with Carnegie Mellon University (CMU) et al.];
-2. Real-time Network Brain Interface(1): The development and integration of a real-time networked-based brain machine interface which supports multi-level robot control has been realized;
-3. Cortical ensemble for locomotion: The realization of real-time decoding of locomotive information from the brain [by Duke];
-4. Fine-grained control: we show that brain signals from the brain can control the legs of our humanoid robot over the internet;
-5. Central Pattern Generator for locomotion: The realization of a biological-based controller on a full-sized humanoid robot [with CMU];
-6. A suitable locomotive controller for Brain Machine Interface (BMI): We show that our biological-based walking controller could in fact be interfaced to the brain of a monkey in allowing our robot to walk [with CMU].
We believe, these results form the first stage towards the realization of the restoring of motor functions in severely paralyzed patients, as well as next generation virtual reality communication devices, and towards brain controllable humanoid robots for contributing to our society.

These results were conducted as part of the JST Basic Research Program ICORP Type, Computational Brain Project (Research Director Dr. Mitsuo Kawato of ATR, Japan) and with Duke University (Prof. Miguel A. L. Nicolelis of Neuroscience department). The core of this latest study will be released to the press on 2008/1/15 in U.S.A.

[Background]

Recent studies of how human brain generates behaviors are progressing rapidly, and the development of humanoid robots that act like humans is now a major topic of research. Despite of these progresses, we have yet to sufficiently understand how human daily behaviors are combined, generated robustly and autonomously. In our project, we derived information-processing models of the brain and verified these models on real robots in the view of gaining a better understanding of human brain mechanisms involved in generating behaviors. Furthermore, we aimed to develop humanoid robots that can generate compliant behavior like humans.

As part of our project, commencing in year 2005 (H.17) we undertook the development of a new humanoid robot along with our ICORP project partner, Prof. Christopher G. Atkeson of the Robotics Institute, Carnegie Mellon University. The hardware of robot was developed by the robotic development company. SARCOS (based in the USA, http://www.sarcos.com/). From early last year, we progressed to a collaborative study with the laboratory leaded by Prof. Miguel A. L. Nicolelis, whom is a world leader in the research area of Brain Machine Interface, at Duke University. This collaborative study focused on development of artificial walking device utilizing brain activity.

[Contents and results]

1. Development of new humanoid robot

Feature of the newly developed humanoid robot, CBi (Computational Brain interface) Figure 1

-1. Height: 155cm Weight: 85 kg (without battery), full-size humanoid robot with 51 degrees of freedom(2).
-2. Head has 7 degrees of freedom: vision sensor composed of 4 video cameras; auditory sensor composed of 2 microphones; gyro sensor and accelerometer for detecting posture.
-3. To generate upper-body movement, the torso is able to flex/extend and abduction/adduction similar to that of human. Each arm has 7 degrees of freedom and can carry up to 3 kg objects. 5 fingers can be controlled independently, and can be utilized for pointing, grasping and gripping.
-4. The lower-body is composed of waist and legs. The waist has a yaw rotation degree of freedom and each leg have 7 degrees of freedom.
-5. At each joint, sensor signals for feedback control are made available, such as joint angle, angular velocity, and force. In addition, the control architecture for the robot can utilizes neural signal, and can control each joint at multiple levels, such as joint angle, angular velocity, torque, and impedance.

Our new robot extends to higher performance, which can generate full-body compliant movements, which extends from our earlier investigations on the humanoid robot DB (Dynamic Brain), developed during the JST ERATO, Kawato Dynamic Brain Project (1996-2001). Furthermore, information of humans' motor performance studies (eyes movements, motor control) acquired during JST ERATO project was incorporated in the development of CBi. In addition, this new robot (CBi) has increased degrees of freedom (51) with wider rage of motion, and similar performance as that of humans.

2. Bipedal locomotion of robot utilizing real-time network brain interface

In our current study here, we present a world-first proof of concept, in a demonstration of the control of a full-sized humanoid robot via signals taken directly from cortical activity of the brain in real time. An overview of the overall experimental setup is given in Figure 2.

At the Duke University, two monkeys were trained to walk bipedally on a treadmill. In each monkey, activity of several hundreds of neurons was recorded from the leg representation in their sensorimotor cortex, and was converted into predictions of the position of leg joints. These predictions were then adapted for controlling a humanoid robot. In the system that is being developed, feedback is delivered from the robot back to the monkey in the form of visual and somatosensory inputs.

A stream-based network architecture has been made available to allow neural signal recordings, and their derivative control signals to the robot, to be transmitted from the Nicolelis lab in the USA to our robot in ATR, Japan. We show that we can control the humanoid robot CBi through the neural data sent via this network interface. The developed robot control architecture accommodates neural signals at various levels: from fine-grained joint-level control to locomotive control .

[Future extensions]

Our newly developed humanoid robot can widely contribute to area of Brain Machine Interface research at multiple levels. As in the current biped walking study, that was realized through a bi-directional network interface (with visual feedback) between the brain and our robot. Based on this result, we could say that we brought forward a big step to the realization of neural prosthetic device that could one day restore lower limb motor functions to paralyzed patients.

Currently, we are developing a system that could induce feedbacks, visually as well as somatosensory input to the brain via cortical microstimulation. Brain Machine Interface that utilizes proper feedback signal from our robot to the brain forms part of our future scientific investigation.

Figure 1 A newly developed 51 degrees of freedom humanoid robot, CBi.
Figure 2 Experimental overview
Technical terms
Sample Movie (mov format 3,891KB)

[Publications]

[a] Kawato, M. (2008). From "Understanding the brain by creating the brain" toward manipulative neuroscience. Philosophical Transactions of the Royal Society B, in press.
doi: 10.1098/rstb.2008.2272

[b] Cheng, G., Fitzsimmons, N. A., Morimoto, J., Lebedev, M. A., Kawato, M., & Nicolelis, M. A. L. (2007). Bipedal locomotion with a humanoid robot controlled by cortical ensemble activity. Society for Neuroscience 37th Annual Meeting. San Diego, CA, USA.
doi: 10.13140/2.1.1315.2645

[c] Cheng, G., Hyon, S., Morimoto, J., Ude, A., Hale, J. G., Colvin, G., Scroggin, W., & Jacobsen, S. C. (2007). CB: A humanoid research platform for exploring neuroscience. Journal of Advanced Robotics, 21 (10), 1097-1114.
doi: 10.1163/156855307781389356

[d] Morimoto, J., Endo, G., Nakanishi, J., Hyon, S., Cheng, G., Bentivegna, D., & Atkeson, C. G. (2006). Modulation of simple sinusoidal patterns by a coupled oscillator model for biped walking. IEEE International Conference on Robotics and Automation (pp. 1579-1584). Orlando, FL, USA.
doi: 10.1109/ROBOT.2006.1641932

[e] Kawato, M. (2008). Measurement and control in brain machine interface, Journal of the Society of Instrument and Control Engineers, Vol.46, No.12, pp. 958-963.

PRESS CONTACTS

JST
Computational Brain Project
2-2-2, Hikaridai Seika-cho Soraku-gun, Kyoto 619-0288 JAPAN
Phone:+81-774-95-2404 Fax:+81-774-95-1236
E-mail:

JST
Public Relations Division
5-3, Yonbancho Chiyoda-ku, Tokyo 102-8666 JAPAN
Phone: +81-3-5214-8404 Fax: +81-3-5214-8432
E-mail:
Internet: http://www.jst.go.jp/EN/

Duke University
Melissa Schwarting
Senior Media Relations Strategist
Duke University Medical Center News Office
220 W Main Street, Suite 910-B
Durham, NC 27705
Phone: +1-919-660-1303
E-mail:

Japanese


JST, an integrated organization of science and technology in Japan, establishes an infrastructure for the entire process from the creation of knowledge to the return to the society. For more information, visit http://www.jst.go.jp/EN/