Event Reports
Report on the Symposium "Rethinking Assessment and Integrity in Research Activity" at the 96th Annual Meeting of the Japanese Biochemical Society (31 Oct '23)

In this symposium, discussions primarily focused on topics such as numerical evaluation, research competition, and researcher awareness, all in relation to the background of research misconduct. The panel discussion was mainly centered on assessment of research activities in the future.
Introduction (Professor Satoshi Tanaka, Kyoto Pharmaceutical University)

Dr. Tanaka discussed his current involvement in the JST-RISTEX project titled 'Establishment of Guidelines for Sharing the Common Understanding of Research Integrity in Life Science.' He mentioned the project's objective to develop and propose guidelines for creating a robust research environment. This initiative involves doing various surveys, notably large-scale questionnaires, aiming to facilitate the adoption and dissemination of these guidelines. The project hats a goal of reducing research misconduct by leveraging the genuine motivation of researchers. He also mentioned strengthening collaboration with government, sharing the concept of research integrity, expanding networks involved in research integrity, and focusing on human resource development, and expressed aspirations to develop these initiatives further in the future.
Topic 1 - Let's discuss research integrity
(Dr. Noriko Osumi, Vice President and Professor, Tohoku University)
Supporting Knowledge Infrastructure

She gave a summary of the San Francisco Declaration on Research Assessment (DORA) and noted that 16 organizations from Japan, including the Japanese Biochemical Society and the Molecular Biology Society of Japan, have signed the Declaration. She further explained: 1) the use of impact factors as a measure of paper quality which is erroneous and should not be utilized for assessing researchers; 2) the necessity of assessment based on the value of the research itself rather than solely on journal-based metrics; and 3) the importance of fully leveraging the opportunities presented by online publishing. 。
In addition, she explained the Conversion Agreement, which shifts from subscription fees to open access publishing fees and reduces the amount of expenditure on paid journals and expands the number of open access papers (Reference: Publication of papers in the era of open science (Japanese text page)).
Topic 2: Rethinking the ideal form of sound and ethical research assessment
(Dr. Amane Koizumi, Project Professor, National Institutes of Natural Sciences)
"The research capability at university is like a mountain."
In the lecture, Dr. Koizumi was asked how he measures the "mountain" of research capability.
Currently, universities primarily rely on the number of papers(quantity) and the average number of citations (quality) as criteria for evaluating clusters of papers. However, it was noted that they both have issues. It has been observed that, especially in Japan, traditionally prioritizing quantity, such as the number of papers, might have inadvertently created a situation conducive to research misconduct (quantity being the most easily manipulable indicator). There is also the problem of predatory journals. Additionally, metrics like Field-Weighted Citation Impact (FWCI) have improved the assessment of the number of citations by incorporating field-specific adjustments. The average number of citations in a university's cluster of papers is used as an indicator of a university's research impact. However, it was noted that there are several issues with this method of averaging, highlighting increased reliance on papers with notably high citation numbers.
To address this, there is a need to explore new assessment methods, such as "substantiality", distinct from the traditional metrics of quantity and quality. Additionally, it's crucial to consider assessments combining these indicators related to quantity, quality, and substantiality. By the way, institutional h5-index (Koizumi, et al. "Research Capability Analysis Index Project" report (2016-2017). 2018)(Japanese text page) was proposed as an indicator to measure the substantiality of research in universities.
Furthermore, it was emphasized that impact factors serve as assessment metrics for journals and should not be used for individual research or personnel assessment (e.g., DORA). It was also mentioned that in implementing complex composite indicators that combine several metrics, it is crucial to consider the appropriate combinations of purpose, causal relationships, and logic.
Finally, it was concluded that research assessment necessitates a blend of quantitative and qualitative aspects across diverse criteria. For the sake of sound research assessment, the importance lies in determining what to measure and for what purpose, emphasizing the need for meticulous indicator selection, a clear sense of purpose, consideration of causality, and logical reasoning.
Topic 3: Research Integrity and Research Assessment: Implications from the Questionnaire Survey
(Dr. Ryuma Shineha, Associate Professor, Osaka University)

The majority of respondents were in the PI class, with professors accounting for 26.8% and associate professors and lecturers accounting for 28.4%. Based on responses to questions such as 'Pleasure in research activities,' 'Tolerance regarding buying and selling research skills,' and 'Tolerance towards the conduct of fellow researchers,' response trends were classified into four clusters.
Cluster 1 showed a strong interest in their own research and experiments, with less interest in other aspects of their research activities. Cluster 2 was adapted to the modern competitive research environment and also held a stringent stance on research integrity. Cluster 3 didn't exhibit notable characteristics, but tended to have high tolerance for questionable research activities when asked about awareness of research integrity. On the other hand, Cluster 4 tended to prioritize the quality of research over achievements or honors.
Dr. Shineha pointed out that this tendency for young researchers to internalize their competitiveness generally aligns with other recent survey findings. Overall, it was inferred that respondents were honest in their answers and suggested the possibility that even straightforward questions might reveal differences in research attitudes.
As a provisional summary of the analysis results, Dr. Shineha highlighted that the attitudes of Groups in Clusters 2 and 4 indicate positive changes in awareness of research integrity. Cluster 3 might appear to be a potentially risky group at first glance, but further investigation is needed. While it is believed that ingenuity is needed in approaching Group 1, which shows little interest in matters beyond their own experimental results, it was mentioned that the necessary measures might differ for each cluster.
Panel Discussion and Summary

From left: Dr. Koide, Dr. Shineha, Dr. Hiroshi Nishina (Standing Director of the Society), Dr. Osumi, Dr. Kohei Tsumoto, (Director of the Society), Dr. Koizumi, and Dr. Tanaka.
Dr. Osumi mentioned, 'In life sciences, there are various roles such as young researchers and laboratory chiefs, making it challenging to evaluate each one. 'It will be necessary to carefully watch each researcher's contribution to the paper.'
Dr. Shineha commented, 'Even if emphasis is placed on quality rather than quantitative indicators, young researchers are unlikely to take it as it comes.' Additionally, Dr. Tsumoto stated, 'Having discernment in research, encompassing both quantity and quality, becomes essential. Isn't it crucial for everyone to understand that addressing the creation of evaluation criteria?'

In sports such as rugby, rules are frequently revised to make the game more interesting. The symposium was concluded by emphasizing the need to adapt and modify appropriate assessment criteria with the times in the academic society, instead of competing using fixed indicators.