Progress Report
Structuring Spatiotemporal Environmental Information in the Body Using In-body Cybernetic Avatars.[2] Structuring spatiotemporal in-body information and Teleoperation
Progress until FY2024
1. Outline of the project
In this R&D Item, we will research and develop technology to present “spatiotemporal in-body environmental information” in an easy-to-understand manner linked to an individual's biological information and history information (medical treatment, medication, etc.). In addition, we will research and develop an interface technology with excellent operability to freely operate a group of small cybernetic avatars (CAs), and design and test a patient simulation environment necessary for evaluation.
2. Outcome so far
We have developed fundamental technology to link individual biological information (measurement data) and historical information (medical treatment, medication, etc.) to present “spatiotemporal in-body environmental information” within the body. First, we constructed information on the morphology of the digestive tract from CT images obtained during health checkups. We developed a method to classify organs even with limited training data and constructed a large-scale anatomical structure database (CT DB) from over 10,000 CT images. This enables the estimation of internal environmental information, including the digestive tract, from external body images (Figure 1). Additionally, we have achieved the mapping of measurement information obtained from in-body CA to intestinal information. By constructing a 3D mesh of the digestive tract from an individual's internal environment information, we can obtain core line information of the intestines. Furthermore, by estimating the position on the core line from the position of the in-body CA, we can map and visualize the acquired information in intestinal coordinates. This method enables intuitive understanding and observation of the shape and condition of an individual's organs.
We are developing a manipulation interface for intuitive in-body CA manipulation. We have developed a cloth-type robotic suit that immerses the user in the in-body space and intuitively evokes tissue traction and a sense of direction when manipulating the CA with the fingers (Figure 2). Using tactile stimuli capable of intuitive motion induction, we implemented an algorithm for motion induction in a clothed active device. In addition, by presenting a sense of gravity using augmented haptics technology, we have realized a simulated in-body CA (endoscopic type) using a general-purpose robotic arm.
As part of the technology to evaluate the function of CA, we have established a method to fabricate a hydrogel whole colon model (Figure 3). 3D structures can be fabricated with high precision using direct modeling technology with a 3D printer. This enables the reproduction of complex luminal structures such as the colon. In addition, the fabricated biological model integrates a peristaltic mechanism for mimicking the motion of living organisms and a pressure sensing mechanism for quantitative evaluation. Moreover, the physical properties of the model, such as Young's modulus, thermal conductivity, and electrical impedance, can now be adjusted. Compared to conventional elastomeric organ models, this model has physical properties closer to those of living organisms and more precise structural reproducibility than conventional hydrogel models. It enables functional evaluation of in-body CA. It is expected to be utilized as a next-generation evaluation platform with a view to social implementation.
3. Future plans
Interfaces, evaluation techniques, and the presentation of information from the system to the person operating the small CA are needed. We will continue researching and developing elemental technologies to promote system integration to realize in-body CA.