In January 2020, to address the public health emergency of COVID-19 and accelerate the pace of research, Wellcome, together with the Gates Foundation and UKRI, issued a broad-reaching statement calling for increased open science practices for COVID-19 related publications. Many journals, publishers, and organisations signed on.

Did the open-science practices of the signers and their associated researchers change as a result? 

That is a complicated question to answer. To investigate, Wellcome contracted with Science-Metrix to contribute to an evaluation of the statement’s effects. 

Using innovative and conceptually sophisticated bibliometric analysis, Science-Metrix focused on several key pieces of the statement. The signers had agreed to:

• make peer-reviewed COVID-19-related publications open access (or freely available);
• make these publications available via preprint servers before journal publication; and
• make interim and final research data available (for example, in a data repository).

Science-Metrix analysts developed methods for building data sets related to these three commitments, using methods such as fuzzy matching of preprints to journal publication, regexes to identify signatories and diseases, and Scopus fulltexts to extract data sharing-relevant snippets and excerpts. To test whether the statement had made a difference in open-science practices, Science-Metrix analysts compared the open-access practices of signatories of the statement with non-signatories.

The report lays the groundwork for future assessments of the impact of similar types of statements and calls aimed at reforming science and innovation practices. Monitoring such interventions is important, because, as the report concludes, “it is expected that clearer target setting a priori would lead to more straightforward data collection and monitoring exercises but also to higher signatory alignment with the commitments made. An audit framework, for example, could have led to increased success in areas where the Joint Statement failed to fully deliver, e.g. preprinting and data sharing.”

Read the bibliometric technical report (Annex A) for details on methods. 

Read the Wellcome report “Investigating the effects of open sharing commitments” with the report’s findings and recommendations.