CGIAR recently published guidelines that harmonize how research and science quality is evaluated within CGIAR.  These guidelines draw on work that Science-Metrix completed for CGIAR last year.

In 2022, CGIAR Advisory Services (CAS) contracted Science-Metrix to create recommendations on the use of bibliometrics to evaluate quality of science. CGIAR noted that the technical note Science-Metrix developed, as part of a co-design process with CAS, “made a deep contribution to the associated thinking” in the development of the new guidelines. 

That technical note included four case studies examining Quality of Science evaluation approaches used by CAS to assess research for development projects. These approaches included bibliometric indicators, surveys, and structured interviews.  One of the points made in the technical note was that while bibliometric methods can be an important line of evidence in research assessment, each individual indicator has inherent limitations, and therefore triangulating results by supporting them with additional evidence allows more confidence in the conclusions reached.

The CGIAR guidelines set out for evaluators, primarily at CGIAR but also those working in other agricultural research-for-development contexts, both an approach and “a more common understanding of the Quality of Science (QoS) evaluation criterion and the methods.” The guidelines identify dimensions crucial to evaluating QoS—research design (project proposals for example), inputs (staff, infrastructure, funding modalities), processes (gender equity in experimental practices, collaboration activities), and output (journal publications, data sharing in public repositories, knowledge transfer activities). This is a broad idea that encompasses using multiple data sources and methods. The guidelines also recommend drawing on mixed methods, combining quantitative methods such as altmetric and bibliometric indicators, and qualitative methods, such as desk review and the IDRC rubrics, as appropriate to the evaluation questions.

The guidelines also make an interesting distinction between performance evaluation and process evaluation approaches. While bibliometric strategies are often associated with performance evaluations, Science-Metrix would argue that most of its bibliometric analyses in fact fall within the process approach. This may well be the case for indicators of cross-disciplinarity, gender equity in research, South-North co-publication, or public-private co-publication, to name just a few.

Read about Science-Metrix’s work for CGIAR.

Read the CGIAR Evaluation guidelines, “Applying the CGIAR Quality of Research for Development Framework to Process and Performance Evaluations.”

Image credit: iStock Photo