Progress Report

AI & Robots that Harmonize with Humans to Create Knowledge and Cross Its Borders[3] Description & Dialogue AI

Progress until the fiscal year 2023

1. Overview

Researchers typically summarize experimental results, discuss with other researchers, form further hypotheses, and proceed to the next experiment. An AI capable of describing results and updating hypotheses through interactive discussions is necessary.
This project advances R&D on (1) "Paper comprehension and experiment planning AI with Scientist-in-the-loop" for designing experiments as hypotheses, and (2) "Multimodal hypothesis generation using knowledge inference and dialogue" to interactively formulate hypotheses, realizing the Description & Dialogue AI.

2. Progress

Aiming to develop a Multimodal XAI that understands experimental content from figures, tables, and text in papers, we introduced technology reflecting researchers' experiences. Utilizing researcher feedback, we improved LLM performance without large datasets and developed techniques to enhance task allocation and prompt reliability based on expertise.
Furthermore, we worked on constructing paper comprehension and hypothesis generation models using large language models as a foundation for multimodal hypothesis generation.

(1) Paper and Experiment Understanding AI

First, to realize Multimodal XAI, we developed technology to reflect researchers' tacit knowledge in LLMs. Using specific output examples from researchers, we developed a technique for extracting relevant descriptions and outputting their rationale using In-context learning through prompts. This achieved LLM performance improvement without requiring large datasets.
Next, we developed a researcher knowledge understanding AI and techniques to estimate optimal allocation of feedback tasks and prompt reliability based on expertise.
This enabled high-quality output when multiple researchers collaboratively design LLM prompts (Fig. 1).

Fig. 1: Researcher knowledge understanding AI
Fig. 1: Researcher knowledge understanding AI

Furthermore, we worked on understanding experimental content using researcher knowledge acquisition AI and knowledge understanding AI. Targeting papers on object detection technology, we extracted and verified tags, improving accuracy compared to simple LLM usage. We confirmed that incorporating researchers' insights streamlined the understanding of experimental content (Fig. 2).
Lastly, we developed a web application for molecular editing as an interface to embed researchers' insights into molecular synthesis AI. Using RLHF technology, researchers can not only evaluate the foundation model's output but also propose improvements, enabling the reflection of researchers' thought processes in the model.

Fig. 2: Tag extraction and related literature presentation based on experimental content understanding
Fig. 2: Tag extraction and related literature presentation based on experimental content understanding
(2) Multimodal Hypothesis Generation

This research addressed multimodal hypothesis generation using knowledge inference and dialogue. We focused on constructing a paper comprehension framework using large language models, adjusting existing open-calm and T5 models using a paper database obtained from the Patent Office. We used a general masked language model as the objective function. We generated hypotheses by providing prompts to the fine-tuned model. We created 100,000 hypothesis generation data related to causal relationships from patent data.
We also prepared necessary data for hypothesis generation, employing linguistically proficient annotators to construct annotation standards for 10,000 patent data and made over 300 annotations on selected patents.
In constructing the knowledge inference model, we focused on causal relationships and considered methods to select valid hypotheses using generated hypothesis data and external causal relationship dictionaries. Towards building an AI that dialogues with researchers about hypotheses, we explored an AI utterance modification method using reinforcement learning (RL-AIF). We plan to evaluate this method in the future.

3. Future work

We will continue to develop co-evolutionary artificial intelligence technology that improves AI performance through researcher dialogue and feedback, aiming to embed researchers' inspirations for hypothesis generation into AI. The hypothesis inspiration AI will acquire and reason knowledge, collaborating with humans through dialogue to trigger innovative discoveries and innovations.