{"id":151496,"date":"2024-12-08T17:31:21","date_gmt":"2024-12-08T17:31:21","guid":{"rendered":"https:\/\/force11.org\/?post_type=group&p=151496"},"modified":"2024-12-19T10:20:34","modified_gmt":"2024-12-19T10:20:34","slug":"ra4ai","status":"publish","type":"group","link":"https:\/\/force11.org\/group\/ra4ai\/","title":{"rendered":"Developing research assessment criteria for the use of AI in scientific and scholarly publications (RA4AI)"},"content":{"rendered":"\n

Group leaders<\/h2>\n\n\n\n
\n
\n\t\t\t\n\t\t\t
\n\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"Profile\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\tGiovanni Cacciamani<\/a><\/strong>\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\n\n

<\/p>\n<\/div><\/div>\n\n\n\n

\n\t\t\t\n\t\t\t
\n\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"Profile\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\tPerihan Elif Ekmekci<\/a><\/strong>\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\n\n

<\/p>\n<\/div><\/div>\n\n\n\n

\n\t\t\t\n\t\t\t
\n\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"Profile\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\tFrancis P. Crawley<\/a><\/strong>\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\n\n

<\/p>\n<\/div><\/div>\n<\/div><\/div><\/div>\n\n\n\n

Membership<\/h2>\n\n\n\n

To join the group please fill out the following form<\/a>.<\/p>\n\n\n\n

Summary<\/h2>\n\n\n\n

The working group \u201cDeveloping research assessment criteria for the use of AI in scientific and scholarly publications (RA4AI)\u201d will develop comprehensive criteria for assessing the use and impact of AI in scholarly communication processes. As AI increasingly influences various aspects of scientific research, from the very conception to the conduct of research to content creation, from authorship and peer review to data curation and dissemination, clear research assessment criteria are needed to promote the responsible use of AI throughout the research and publication process. These criteria are needed in order to maintain trust and credibility in research, as the wide use of AI will come with tremendous tradeoffs between efficiency and (possibly) expert judgement.<\/p>\n\n\n\n

The RA4AI is jointly submitted by CODATA, CANGARU, and the EOSF-Future\/RDA Working Group on Artificial Intelligence & Data Visitation.<\/p>\n\n\n\n

RA4AI will address use the following key questions to help guide its work:<\/p>\n\n\n\n

    \n
  • Are scholarly communication platforms (eg, publishers, data repositories) prepared to assess the use of AI now? If not, what is needed for them to do so in a responsible manner?<\/li>\n\n\n\n
  • What standards have already been developed for AI assessment that we can adapt to the scholarly communication field? <\/li>\n\n\n\n
  • How will the RA4AI outputs actually make scholarly communication more equitable?<\/li>\n<\/ul>\n\n\n\n

    Goals<\/h2>\n\n\n\n

    The group’s key areas of focus will be:<\/p>\n\n\n\n

      \n
    • Ethical considerations: developing criteria on the responsible use of AI in scholarly communications, including assigning attributions, addressing biases, evaluating the transparency of training data and algorithms and standards in reporting, and developing accountability mechanisms for the role played by AI in scientific and scholarly publications. RA4AI will consider how to certify the responsible use of AI in research workflows, ensuring AI tools do not distort the integrity of the research and its presentation\/communication.<\/li>\n\n\n\n
    • The use of AI in the research process and content creation: establishing criteria for evaluating the use of AI in generating scientific and scholarly proposals and outputs, including (but not limited to) outputs for research papers, literature reviews, and summaries. The role of AI-embedded scholarly communication resources as well as AI technologies present in data resources, working systems, project management, and tools for data analysis and presentation will be addressed here. This requires establishing criteria for evaluating authorship, responsibility, accountability, intellectual property, and the accuracy of AI-generated content.<\/li>\n\n\n\n
    • The use of AI in peer review: developing criteria for evaluating the appropriate role for, and use of, AI in automating or assisting peer-review processes as a decision support system for ensuring fairness, transparency, and quality control while maintaining a human-centered approach. It will establish a balance between the role of AI tools used in peer review and the role of human reviewers.<\/li>\n\n\n\n
    • The use of AI in data integrity, data handling, and data curation: based on the FAIR, CARE, and TRUST data principles and their application to the use of AI, the working group will address how AI is (could be) used in managing and curating research data and code during and for the publication process, with a focus on the integrity, reliability, and transparency of AI systems and the data supporting them.<\/li>\n\n\n\n
    • Evaluation of AI-driven research outputs: defining metrics or other criteria to evaluate the influence of AI on research quality and impact, such as the possibly substantial role AI plays in citation patterns, novelty detection, and ranking systems for science communications. These criteria will focus on ensuring the equitable and fair treatment of AI-driven work with reference to traditional research methods.<\/li>\n<\/ul>\n\n\n\n

      Deliverables<\/h2>\n\n\n\n

      The RA4AI working group will work towards the completion of the following outputs:<\/p>\n\n\n\n

        \n
      • Criteria for the ethical use of AI across the publication workflow: develop a set of principles and criteria for using AI across various stages of scientific and scholarly communication and its impact on research assessment.<\/li>\n\n\n\n
      • Toolkits and best practices: practical toolkits for those involved in evaluating researchers, institutions, and publishers to support the adoption of responsible AI use in scientific and scholarly publications.<\/li>\n\n\n\n
      • Impact assessment frameworks: to support the criteria and their implementation, the development of impact assessment frameworks for the use of AI in scientific\/scholarly research and their outputs to measure their real value for the scientific and academic communities as well as for society.<\/li>\n<\/ul>\n