The 96th Annual Meeting of the Japanese Biochemical Society (JBS) was held from October 31, 2023, to November 2 at the Fukuoka International Congress Center and Marine Messe Fukuoka Building B in Fukuoka City. This report gives an overview of the symposium "Rethinking Assessment and Integrity in Research Activity" held on March 31, organized by Professor Satoshi Tanaka of Kyoto Pharmaceutical University and Professor Takaki Koide of Waseda University.

In this symposium, discussions primarily focused on topics such as numerical evaluation, research competition, and researcher awareness, all in relation to the background of research misconduct. The panel discussion was mainly centered on assessment of research activities in the future. 　

Introduction (Professor Satoshi Tanaka, Kyoto Pharmaceutical University）

As an introduction, Dr. Tanaka raised a question about the frequency of research misconduct in life sciences, which appears to be beyond the researchers' awareness. He inquired about the specific criteria for distinguishing the integrity of researchers within this context.

Dr. Tanaka discussed his current involvement in the JST-RISTEX project titled 'Establishment of Guidelines for Sharing the Common Understanding of Research Integrity in Life Science.' He mentioned the project's objective to develop and propose guidelines for creating a robust research environment. This initiative involves doing various surveys, notably large-scale questionnaires, aiming to facilitate the adoption and dissemination of these guidelines. The project hats a goal of reducing research misconduct by leveraging the genuine motivation of researchers. He also mentioned strengthening collaboration with government, sharing the concept of research integrity, expanding networks involved in research integrity, and focusing on human resource development, and expressed aspirations to develop these initiatives further in the future.

Topic 1 - Let's discuss research integrity
　（Dr. Noriko Osumi, Vice President and Professor, Tohoku University）

Supporting Knowledge Infrastructure

Dr. Osumi explained the current situation regarding the diversification of media in which research results are published given the recent trend toward open access. She also explainedthe background of research misconduct in the fields of medicine and life sciences :1) the abundance of data per paper (necessity of collaborative research, inclusion of excessive data in figures, etc.), 2) issues related to authorship (co-first authors, gift authorship, etc.), 3) problems with peer reviews (dysfunction of peer review, short review periods), and 4) the prevalence of a performance-driven culture.

She gave a summary of the San Francisco Declaration on Research Assessment (DORA) and noted that 16 organizations from Japan, including the Japanese Biochemical Society and the Molecular Biology Society of Japan, have signed the Declaration. She further explained: 1) the use of impact factors as a measure of paper quality which is erroneous and should not be utilized for assessing researchers; 2) the necessity of assessment based on the value of the research itself rather than solely on journal-based metrics; and 3) the importance of fully leveraging the opportunities presented by online publishing. 。
In addition, she explained the Conversion Agreement, which shifts from subscription fees to open access publishing fees and reduces the amount of expenditure on paid journals and expands the number of open access papers (Reference: Publication of papers in the era of open science (Japanese text page)).

Topic 2: Rethinking the ideal form of sound and ethical research assessment　
(Dr. Amane Koizumi, Project Professor, National Institutes of Natural Sciences)

"The research capability at university is like a mountain."

In the lecture, Dr. Koizumi was asked how he measures the "mountain" of research capability.

　
Currently, universities primarily rely on the number of papers(quantity) and the average number of citations (quality) as criteria for evaluating clusters of papers. However, it was noted that they both have issues. It has been observed that, especially in Japan, traditionally prioritizing quantity, such as the number of papers, might have inadvertently created a situation conducive to research misconduct (quantity being the most easily manipulable indicator). There is also the problem of predatory journals. Additionally, metrics like Field-Weighted Citation Impact (FWCI) have improved the assessment of the number of citations by incorporating field-specific adjustments. The average number of citations in a university's cluster of papers is used as an indicator of a university's research impact. However, it was noted that there are several issues with this method of averaging, highlighting increased reliance on papers with notably high citation numbers.
　
To address this, there is a need to explore new assessment methods, such as "substantiality", distinct from the traditional metrics of quantity and quality. Additionally, it's crucial to consider assessments combining these indicators related to quantity, quality, and substantiality. By the way, institutional h5-index (Koizumi, et al. "Research Capability Analysis Index Project" report (2016-2017). 2018)(Japanese text page) was proposed as an indicator to measure the substantiality of research in universities.　
Furthermore, it was emphasized that impact factors serve as assessment metrics for journals and should not be used for individual research or personnel assessment (e.g., DORA). It was also mentioned that in implementing complex composite indicators that combine several metrics, it is crucial to consider the appropriate combinations of purpose, causal relationships, and logic.

Finally, it was concluded that research assessment necessitates a blend of quantitative and qualitative aspects across diverse criteria. For the sake of sound research assessment, the importance lies in determining what to measure and for what purpose, emphasizing the need for meticulous indicator selection, a clear sense of purpose, consideration of causality, and logical reasoning.

Topic 3: Research Integrity and Research Assessment: Implications from the Questionnaire Survey
(Dr. Ryuma Shineha, Associate Professor, Osaka University)　

Dr. Shineha reported on the results of the questionnaire conducted in the JST-RISTEX project 'Establishment of Guidelines for Sharing the Common Understanding of Research Integrity in Life Science'. The survey was publicized through the Union of Japanese Societies for Biological Science, and 947 people, including life science researchers, responded to it

The majority of respondents were in the PI class, with professors accounting for 26.8% and associate professors and lecturers accounting for 28.4%. Based on responses to questions such as 'Pleasure in research activities,' 'Tolerance regarding buying and selling research skills,' and 'Tolerance towards the conduct of fellow researchers,' response trends were classified into four clusters.

Cluster 1 showed a strong interest in their own research and experiments, with less interest in other aspects of their research activities. Cluster 2 was adapted to the modern competitive research environment and also held a stringent stance on research integrity. Cluster 3 didn't exhibit notable characteristics, but tended to have high tolerance for questionable research activities when asked about awareness of research integrity. On the other hand, Cluster 4 tended to prioritize the quality of research over achievements or honors.　　

Dr. Shineha pointed out that this tendency for young researchers to internalize their competitiveness generally aligns with other recent survey findings. Overall, it was inferred that respondents were honest in their answers and suggested the possibility that even straightforward questions might reveal differences in research attitudes.

As a provisional summary of the analysis results, Dr. Shineha highlighted that the attitudes of Groups in Clusters 2 and 4 indicate positive changes in awareness of research integrity. Cluster 3 might appear to be a potentially risky group at first glance, but further investigation is needed. While it is believed that ingenuity is needed in approaching Group 1, which shows little interest in matters beyond their own experimental results, it was mentioned that the necessary measures might differ for each cluster.

Panel Discussion and Summary

　

During the panel discussion, Dr. Koizumi mentioned regarding Topic 3, 'It would be interesting to observe what happens when researchers from different clusters collaborate on a research project, since different clusters have different attitudes toward research integrity. Life sciences are often conducted in large teams, but it was mentioned that if these teams don't interact, it might pose a risk.'
Dr. Osumi mentioned, 'In life sciences, there are various roles such as young researchers and laboratory chiefs, making it challenging to evaluate each one. 'It will be necessary to carefully watch each researcher's contribution to the paper.'
Dr. Shineha commented, 'Even if emphasis is placed on quality rather than quantitative indicators, young researchers are unlikely to take it as it comes.' Additionally, Dr. Tsumoto stated, 'Having discernment in research, encompassing both quantity and quality, becomes essential. Isn't it crucial for everyone to understand that addressing the creation of evaluation criteria?'

In summary, Dr. Koide mentioned the The Leiden Manifesto for research metrics and DORA and research metrology (research measurements). Article 10 of the Leiden Manifesto states, 'Scrutinize indicators regularly and update them.' It was mentioned that if the method of assessment (environment) changes, the behavior of those being assessed will also change.

In sports such as rugby, rules are frequently revised to make the game more interesting. The symposium was concluded by emphasizing the need to adapt and modify appropriate assessment criteria with the times in the academic society, instead of competing using fixed indicators.