Bibliometrics

Showcase

Below are some examples of projects listed by type of activity or need expressed by clients.

MEASUREMENT AND BENCHMARKING OF SCIENCE AT NATIONAL AND INTERNATIONAL LEVELS

Science and Engineering Indicators 2016 (2016)

Client: SRI International and the National Science Foundation

Description: Science-Metrix was mandated by SRI International and the US National Science Foundation to develop measures and indicators of research and patent activity using bibliometrics and patent data for the Science and Engineering Indicators 2016. The National Science Foundation (NSF) National Center for Science and Engineering Statistics (NCSES) prepares the biennial Science and Engineering Indicators (SEI) report on behalf of the National Science Board, which has the ultimate responsibility to report to the president and Congress about the state of science and engineering in the US. SEI reports high-quality quantitative data bearing on the scope, quality and vitality of the US and international science and engineering enterprise. SEI places particular emphasis on indicators of science and technology across institutional sectors, nations and regions, as well as on time trends in these indicators. A key part of SEI’s coverage is indicators of research and innovation, including scientific publications and patents.

Science-Metrix prepared the bibliometric and technometric statistics for three chapters of the Science and Engineering Indicators 2016 (SEI 2016). These chapters cover Academic Research and Development (Chapter 5), Industry, Technology, and the Global Marketplace (Chapter 6), and State Indicators (Chapter 8). Elsevier’s Scopus database was selected for preparing the bibliometric statistics, which in all previous editions of the publication had been computed using the Science Citation Index (SCI) and the Social Sciences Citation Index (SSCI), both part of Thomson Reuters’ Web of Science at the time of the study. Science-Metrix prepared a comparative report to outline the impact the change of database had on the statistics and to help readers make a more meaningful interpretation of the bibliometric indicators compiled for SEI 2016. The Bibliometrics team also produced a technical documentation report, detailing the various steps for the implementation of the databases used, the cleaning and standardization of the data and the production of statistics. The first section of the report presents the bibliometric methods employed based on scientific publications indexed in Scopus (Elsevier), while the second section presents the methods for the production of technometric data based on patents indexed in LexisNexis (Elsevier).

Following the successful completion of this contract, Science-Metrix was again contracted by SRI and the NSF to collaborate on the preparation of bibliometric and patent statistics for future iterations of the SEI through until 2021.

Access the full Science and Engineering Indicators 2016 via the National Science Foundation website

European Research Area Progress Report 2016 (2016)

Client: European Commission

Description: The European Commission’s Directorate-General for Research & Innovation contracted Science-Metrix to support the Commission’s evidence-based policy development by gathering internationally comparable data and indicators and providing analysis to monitor progress towards the implementation of all European Research Area (ERA) priorities. Deliverables for this contract included a technical report, the ERA Monitoring Handbook and 34 country profile reports.

The Commission’s ERA Monitoring Mechanism (EMM) is an essential component in ERA policymaking as it enables monitoring of the degree to which Member States, research funders and institutions are supporting and implementing the ERA. One of the key components of the EMM is a Progress Report, which contains three parts: policy messages in the form of a Communication, analysis of progress in the ERA Facts and Figures report, and more detailed information in the form of country profiles.

Using a mixed-method approach, our Bibliometrics and Evaluation Services teams collaborated to measure progress at EU, Member State and organizational levels over a short-term and longer-term period. The teams assessed the current state and recent evolution of the ERA, covering both the ERA as a whole and the 41 countries that compose it. Guided by the 2015 ERA Roadmap, they examined progress towards achieving each of the priorities and subpriorities outlined for the ERA, such as more effective national research systems, transnational cooperation, researcher mobility, issues related to gender, knowledge circulation, and open access publications. The primary lens was to focus on the Headline and complementary EMM indicators identified by the European Research Area and Innovation Committee (ERAC), with a secondary focus on additional indicators and composite scores to provide more comprehensive and synthetic overviews. This quantitative approach was bolstered by the collection of substantial qualitative data through document reviews and interviews with key stakeholders across national and sectoral contexts within the ERA.

The team developed the European Research Area Monitoring Handbook of indicators as a resource providing guidance on the theoretical underpinnings, the collection of data and the calculation of all indicators presented in the main report and the individual country profiles. Also included was a discussion of any pertinent considerations for the use of data on progress towards the common ERA at the organizational, Member State/Associated Country and/or European level. Upon future developments and new additions to the EMM, the data collection instruments included in the Handbook will be revised accordingly. It was therefore designed to reflect the state of the art in the mapping and monitoring of the ERA.

Read the reports

Étude bibliométrique sur la recherche tunisienne (2015)

Client: Agence Nationale de Promotion de la Recherche Scientifique (ANPR)

Description: The Agence Nationale de Promotion de la Recherche Scientifique (ANPR) in Tunisia benefitted from the project PASRI (Projet d’Appui au Système de Recherche et de l’Innovation) to conduct a study on the state of research in Tunisia. Science-Metrix was mandated to carry out this study. PASRI was founded by the Ministère tunisien de l’Enseignement Supérieur et de la Recherche Scientifique in collaboration with the Ministère tunisien de l’Industrie. Funded by the European Union, this project aims to improve the contribution of research and innovation for growth and employment development in Tunisia by strengthening the link between research and business.

This report (in French only) presents an analysis of the state of Tunisian research from 2002 to 2013. The study analyses the country's total scientific output, as well as subfields of science, in comparison with other selected countries and regions. Bibliometric data on Tunisian organisations are also provided, as well as data on international cooperation. This study aims to identify the strengths and weaknesses of Tunisian research and to highlight research opportunities that could become priority research fields for the coming years. Some elements of discussion about international cooperation are also presented.

Read the report (French only)

Bibliometric Analysis of the Current State of Science and Technology in the Member States of the African Union (2013)

Client: African Union

Description: Science-Metrix was commissioned by the African Observatory for Science, Technology and Innovation (AOSTI) to conduct a bibliometric assessment of the scientific production and performance of the 54 members in the African Union over 6 thematic domains and 22 scientific fields compared to the world level for the 2005–2010 period.

During the course of this study, Science-Metrix undertook the most exhaustive data cleaning task ever conducted by the firm. Institutions from all country members of the African Union that published more than five publications between the years 2005 and 2012 were standardized. In addition, a list of the 500 most publishing researchers was produced, necessitating the standardization of more than 1,500 names.

The key tasks were the following:

  • Bibliometric data and technometric data
  • Visualization tools
  • Reporting
  • Training, knowledge transfer

Deliverables: Complete set of bibliometrics (databooks), visuals of collaboration patterns, final analytical report.

Read the report

Bibliometric Study to Support the Assessment of Canadian Ocean Science (2013)

Client: Council of Canadian Academies

Description: Science-Metrix was mandated by the Council of Canadian Academies (CCA) to provide bibliometric indicators related to the Canadian performance in ocean science research as well as related sub-topics of high interest.

In its analysis, Science-Metrix provided bibliometric data for the top 25 countries and top 200 institutions publishing the most in the field of ocean science, which provided insights to the CCA regarding the positioning of Canada and Canadian institutions at the international level. Furthermore, the top 100 publishing Canadian institutions were also identified to provide a complete overview of ocean science research in Canada. The results of the study highlighted the high quality of the research lead in this field in Canada and in these sub-topics, with noticeable strengths identified in the sub-topics of arctic, human impact and human health research associated to ocean science.

The results from this study were incorporated in a CCA report prepared by its Expert Panel on Canadian Ocean Science. This report presents a comprehensive portrait of the state of ocean science research in Canada.

Read the report

Cross-cutting Analysis of Scientific Publications versus Other Science, Technology and Innovation Indicators (2013)

Client: European Commission

Description: The last two decades have seen a steady rise in the development of STI indicators. Their use is intended not only to better manage and govern the complex European system but to measure progress towards the achievement of an increasingly wide variety of social and economic objectives. A greater number and variety of actors are now involved in indicator development, contributing new guidelines, new data sources and new areas of inquiry.

A major focus of these efforts has been to find appropriate, quantitative statistical tools that are comparable across systems (i.e., countries, regions, sectors, organisations and industries) and that can strike the best balance between internationally comparable and nationally relevant indicators (Edler & Flanagan, 2011; Lugones & Suarez, 2010). In order to create robust and meaningful measures, ‘positioning’ indicators must also account for the distinct contextual factors that underlie each system, which may be at least as important as formal inputs and outputs to their ultimate performance (Edler & Flanagan, 2011; Lepori, Barré, & Filliatreau, 2008). In the face of often considerable underlying conceptual and methodological difficulties, newer indicators must both consider and attempt to confirm the specific drivers of research output and performance of countries and regions.

This report contributes to this growing body of literature by performing a cross-cutting assessment of performance centering on the European Research Area (ERA) in addition to a number of selected countries. The study uses bibliometric statistics computed by Science-Metrix as part of a project conducted for the European Commission (EC). These data on scientific output are used to examine performance in light of 17 input indicators such as R&D investments. Specifically, this report investigates:

  1. the factors behind publication outputs and productivity (i.e., the efficiency with which entities are converting research inputs into research outputs) of countries/regions, as revealed through an analysis of scientific production; and
  2. the factors behind the production patterns of countries, as revealed through an analysis of scientific concentration (by research area), across thematic domains.

Read the report

MEASUREMENT AND BENCHMARKING OF SCIENCE FOR SPECIFIC RESEARCH PERFORMERS

Provision of bibliometrics in Ocean Science for the Intergovernmental Oceanic Commission – UNESCO (2016)

Client: Intergovernmental Oceanographic Commission (IOC) of UNESCO

Description: Science-Metrix was contracted by the Intergovernmental Oceanographic Commission (IOC) of UNESCO to provide quantitative data on scientific output in the field of Ocean Science. The data were then included in the 2017 Global Ocean Science Report, which provides stakeholders and decision-makers with information on nations’ investments, resources and scientific activity in Ocean Science.

The Science-Metrix study presented an analysis of worldwide scientific literature output in Ocean Science over the period 2010–2014. Using the Web of Science (WoS) by Thomson Reuters as the main data source, bibliometric data were produced on Ocean Science as a whole and for eight subfields (or pillars) of Ocean Science. Datasets were constructed by combining two methods: the first based on scientific journals and the second on specific keywords.

Deliverables consisted of a databook of results and a PowerPoint deck. Nine tables of bibliometrics were produced (one for global Ocean Science and eight for the respective subfields), each presenting the following indicators, by country and for all countries with scientific output in Ocean Science:

  • Number of publications (full counting used)
  • Growth ratio comparing the output of the 2010–2011 period to that of the 2013–2014 period
  • Average of relative citations (ARC)
  • Average of relative impact factors (ARIF)
  • Highly cited publications: top 1% and top 10%
  • Specialization index

Three citation maps were produced showing the most cited countries for Global Ocean Science, one for total number of citations; one based on the ARC, which is a direct normalized impact measure; and another based on the ARIF, which represents an indirect impact measure as it is calculated based on the impact of the journal in which the paper was published. Nine positional analysis graphs (or radar graphs) were also produced, each graph presenting a visual of output (size of bubbles), ARC and specialization index (position of bubbles) for the top 40 most publishing countries. Nine network collaboration graphs were produced for the top 40 countries as well as nine for the top 40 institutions, for a total of 18 network collaboration graphs. Spider graphs presented specialization index scores by pillar, within scientific production as a whole and Ocean Science in particular, for the 10 most cited countries, and, finally, Science-Metrix selected the top 40 countries based on their output over 2010–2014, and showed their ranking over the periods 2000–2004, 2005–2009 and 2010–2014.

Global Ocean Science Report

She Figures 2015 (2015)

Client: European Commission

Description: The She Figures publication is the main source of pan-European, comparable statistics on the state of gender equality in research and innovation. It covers a wide range of themes, including the proportions of women and men amongst top‑level graduates, academic staff and research/advisory boards, the working conditions for women and men researchers, the integration of the gender dimension in the content of peer-reviewed scientific articles, and various indicators measuring gender gaps in scientific and innovation outputs.

Science-Metrix collaborated with KU LeuvenICF International and representatives from the European Commission’s Directorate-General of Research and Innovation on the preparation of She Figures 2015. While the majority of indicators are drawn from Eurostat databases and the Women in Science database, in the 2015 edition of She Figures, the Web of ScienceTM (WoSTM) has been used to produce scientometric indicators by sex (e.g., ratio of women to men authorship of scientific papers, proportion of a country’s scientific production including a sex or gender dimension), and the EPO Worldwide Patent Statistical Database (PATSTAT) has been used to produce a technometric indicator by sex (e.g., ratio of women to men inventorship). The Science-Metrix bibliometrics team were instrumental in developing these indicators.

Released every three years since 2003, the She Figures report provides a key evidence base for policies in the area of gender equality in research and innovation. It is recommended reading for policymakers, researchers and anybody with a general interest in these issues. The co-operation of EU Member States and Associated Countries, the Helsinki Group and their Statistical Correspondents, and Eurostat in preparing She Figures is gratefully acknowledged.

She Figures 2015

She Figures Handbook 2015 

Comprehensive methodology: New research & innovation output indicators

She Figures 2015 leaflet

Scientific Output and Collaboration of Companies Publishing the Most in the ERA (2013)

Client: European Commission

Description: Science-Metrix has been selected as the provider of bibliometric indicators for the European Commission’s Directorate-General for Research & Innovation (DG RTD) that will be integrated into the European Commission’s evidence-based monitoring of progress towards the objectives set forth in the Lisbon framework and the post-Lisbon strategy for the European Research Area (ERA). The bibliometric component of this monitoring system is part of a package of six complementary studies reporting on the dynamics of research activities along the whole route of knowledge, from R&D investments to publications, patents, and licensing.

This report

More specifically, this report aims to measure the scientific production of research-intensive companies within the ERA zone, portray the geographic distribution of their scientific activities, and characterize their collaboration profiles. Data is presented for the top 100 firms that published the most within the ERA zone from 2007 to 2011; this includes multinational companies with activities outside the ERA, although only their production within this zone is considered.

Standardization of author addresses for firms and subsidiaries

The method used to identify and standardize the author addresses for the leading companies is based on two approaches. First, a matching approach was applied based on the list of firms found in the 2012 EU Industrial R&D Scoreboard. The second, and complementary, approach is based on a frequency ranking of the publication output of companies found in Scopus. This involves cleaning the preliminary lists of institutions identified as companies, beginning with those that publish the most. The merger of those two approaches results in a list of identified companies, from which subsidiaries are identified, extracted from the publication database, and associated with their parent companies.  The steps involved in these two approaches are described below.

When the addresses of leading firms and their subsidiaries have been standardized and validated via quality control processes, the unique and standardized organization names are reincorporated into the database, where they can be used to produce robust bibliometric data and statistics at the institutional level.

Bibliometric indicators

The production and collaboration profiles are based on the following set of indicators:

  • number of publications
  • number of exclusively intra-company publications
  • number of exclusively intra-company publications with a single affiliate
  • number of exclusively intra-company publications with multiple affiliates
  • number of co-publications
  • number of co-publications (Acad or RPO)
  • number of co-publications (firm)
  • number of co-publications (unclassified)

The report is primarily descriptive, focusing on the salient points relevant to the report’s two main parts: geographic distribution of the scientific production of the most publishing companies within the ERA and their collaboration patterns.

Read the report

MAPPING OF SCIENTIFIC COLLABORATION

Study on Network Analysis of the 7th Framework Programme Participation (2015)

Client: European Commission

Description: The willingness to increase Europe’s competitiveness by strengthening the scientific and technological bases of industry whilst closing the ‘technology gap’ with the US and later with Japan played an important part in the setting up of a multiannual Framework Programme for Research, Technological Development and Development Activities. From the First Framework Programme (FP1) initiated in 1984, to Horizon 2020, which was launched in 2014, the framework programmes have been among the EU’s main instruments for funding research and innovation in Europe. While the role of the framework programmes has developed over time, two strategic objectives were common to each of them: ‘to strengthen the scientific and technological basis of European industry and to encourage it to become more competitive at international level’ (European Parliament, 2014).

The Seventh Framework Programme (FP7) (2007–2013) was promoted as a key tool to respond to Europe’s needs in terms of jobs and competitiveness, and to maintain leadership in the global knowledge economy. The stated overriding aim of FP7 was ‘to contribute to the Union becoming the world’s leading research area. This requires the Framework Programme to be strongly focused on promoting and investing in world-class state-of-the-art research based primarily upon the principal of excellence in research’ (European Parliament, 2006.). 

This report examined the effectiveness of FP7’s network approach in achieving EU research policy objectives and fostering Europe’s international competitiveness in S&T. In investigating the potentials and limits of this approach, the study examined the effects of multidisciplinary, interdisciplinary, intersectoral and international collaborations on achieving the positive outcomes sought in FP7.

The study’s methods included a network analysis, a survey, case studies including in-depth interviews, a representative stakeholder workshop and regression analyses. The key findings reveal benefits from the continuity of research from FP6 to FP7. However, despite these demonstrated benefits and the increase in participants overall for FP7, the study also found a high attrition rate of organisations from FP6 to FP7.

Read the final report

Bibliometric Study in Support of Norway’s Strategy for International Research Collaboration (2014)

Client: Research Council of Norway

Description: Science-Metrix has been commissioned to provide the Research Council of Norway (RCN) with bibliometric indicators of scientific performance and collaboration for Norway and its international partners. The presentation of these indicators was organized with a view to guiding the RCN’s efforts to establish long-term institutional co-operation between Norwegian institutions and similar institutions in other countries under its International Strategy.

More specifically, the report aims at characterizing the scientific production profiles and collaboration patterns of Norway and 57 selected countries in Europe and elsewhere around the world.

Firstly, the report provides a brief comparative analysis of the current scientific performance of Norway and the 57 selected countries, along with their collaboration patterns in the sciences in general. Collaboration is measured from both an international (co-authorship with any foreign country) and a bilateral (co-authorship specifically with Norway) perspective. The analysis of collaboration with Norway is also performed across the organizational sectors of Higher Education, Government, and Business Enterprise. The key questions addressed include:

  • Which are the leading scientific nations and those that co-publish most with foreign partners and with Norway in particular?
  • What has changed in the past decade—in terms of countries’ relative performance and collaboration patterns—and what is likely to change in the near future?

Secondly, the report analyzes in more detail these countries’ performance and collaboration in 15 scientific themes of significant importance to Norway. This analysis is intended to provide strategic information to the RCN about the scientific partnerships that are most beneficial to the country in each strategic field. It also identifies which of those promising partnerships appear to be under-exploited, with a view to providing recommendations as to potential candidates with which Norway could strengthen partnerships by thematic area.

Thirdly, the report presents further information at the micro-level (i.e., organisational level) to help the RCN target organisations—within the countries identified as potential candidates with which Norway could strengthen partnerships—in the context of setting up specific bilateral agreements. Information on the scientific performance and collaboration patterns of organizations is synthesized in the form of collaboration networks.

Finally, the report presents a detailed analysis of various types of multilateral scientific partnerships to address the following questions: Which type of scientific partnership is most beneficial to a country’s scientific impact and what are the mechanisms underlying such gains in impact? These analyses include:

  • network analysis, which presents international collaboration networks for each of the themes selected for this study on the organisational level;
  • analysis of the effect of multilateral co-authorship on the scientific impact of research output;
  • regional analysis; and
  • cluster analysis.

Constitution of dataset

For this project, data were first produced for the sciences in general. More specifically, only documents published in refereed scientific journals (mostly articles, reviews and conference proceedings) were retained, as these documents were reviewed by peers prior to being accepted for publication.

Data were also produced by field of science using Science-Metrix’s journal-based and mutually exclusive classification scheme. Moreover, 15 themes of strategic interest to Norway were delineated by retrieving relevant publications using keyword-based queries, specialist journals, and entire subfields from Science-Metrix’s classification where appropriate.

Data were also produced for the following regional aggregates: the Nordic countries, EU15 countries, EU28 countries, and the world. Because some of the included countries have large R&D systems characterized by regional differentiations and strongholds, Science-Metrix also performed a regional analysis for these countries. Data for organizations and researchers were also produced. As well, for organizations, data were produced for a number of clusters or networks of highly interconnected organizations as part of optional deliverable.

Bibliometric indicators

Several indicators were computed throughout the course of this project:

  • number of publications
  • international co-publications
  • international co-publications with Norway
  • international collaboration rate
  • collaboration rate with Norway
  • specialization index (SI)
  • growth ratio (GR)
  • relative citations (RC)
  • relative number of authors
  • average of relative citations (ARC)
  • average of relative impact factors (ARIF)
  • transdisciplinarity (TD)
  • collaboration index (CI)
  • affinity index (AI)
  • page rank
  • betweenness centrality
  • share of output available in open access (OA)

Read the report

Intra-European Cooperation Compared to International Collaboration of the ERA Countries (2013)

Client: European Commission

Description: This report is part of the six complementary studies that Science-Metrix has performed for the European Commission’s Directorate-General for Research & Innovation (DG RTD). The work involves the collection, analysis and updating of bibliometric data that will be integrated into the European Commission’s evidence‑based monitoring of progress towards the objectives set forth in the Lisbon framework and the post-Lisbon strategy for the European Research Area (ERA).

The present report

Within the FP7, a strong emphasis has been placed on sponsoring collaborative research within Europe (and beyond) as a way to increase Europe’s competitiveness within the global knowledge-based economy. In particular, the co-operation actions implemented under FP7 are foreseen to foster the achievement of complex research challenges requiring pan-European efforts, to improve research excellence in Europe, as well as to reduce the fragmentation of the European research landscape, thereby contributing to the realisation of the European Research Area (ERA)—an open space for knowledge and growth.

To support the monitoring of collaborative research and its effect under FP7, the current report first provides in-depth analyses of the collaboration patterns of European countries (i.e., those included within the ERA; Israel is also covered), as well as of a selected set of international comparables within and outside of the ERA. It also investigates whether the integration of European countries has progressed since the start of FP7 both within and outside of the ERA, as well as attempts to determine whether the co-operation actions implemented under FP7 appear to have triggered an increased integration of European countries by comparing the magnitude of changes both within and outside of the ERA.

Subsequently, the report characterizes the effect of scientific partnerships on scientific excellence by comparing the scientific impact of various types of publications/co-publications for European countries and a selected set of international comparables. It then provides insights into whether the intensification of cross-border co-operation within Europe promoted under the FP7 can be expected to increase the excellence of European research by comparing the scientific impact of various types of publications/co-publications at the aggregate level for EU27 countries. This analysis also sheds light on which types of partnerships are likely to be most beneficial to scientific excellence.

Read the report

ASSESSMENT OF RESEARCH FUNDING PROGRAMS

Bibliometric analysis of publications on poverty-related and neglected infectious diseases (NIDs) published between 2003 and 2015 (2017)

Client: European & Developing Countries Clinical Trials Partnership (EDCTP)

The European & Developing Countries Clinical Trials Partnership (EDCTP) is a public–public partnership between 14 African and 14 European countries. It supports collaborative research with the aim of accelerating the clinical development of new or improved interventions (drugs, vaccines, microbicides and diagnostics) to prevent or treat HIV/AIDs, malaria, tuberculosis and neglected infectious diseases (NIDs) in sub-Saharan Africa. The first EDCTP program ran from 2003 to 2015, while the second program (EDCTP2) began in 2014 and will operate until 2024. EDCTP2 is co-funded by the European Union (EU) under the Horizon 2020 Framework Programme for research and innovation. The EU will contribute up to a maximum of €683 million to the program, provided this contribution is matched by the contributions of the European Participating States.

Science-Metrix was contracted by EDCTP to update a bibliometric analysis conducted in 2013. The two studies focused on European and sub-Saharan African research output on poverty-related and neglected infectious diseases; the first study covered the period from 2003 to 2011, and Science-Metrix’ update study covered the period 2003 to 2015. The contract involved assessing the geographical and temporal trends in terms of both the quantity and quality of research outputs on HIV/AIDS, TB, malaria and NIDs by European EDCTP member countries and sub-Saharan African countries through a bibliometric analysis. This analysis is being used to inform the interim evaluation of the EDCTP2 program to be conducted in 2017.

Science-Metrix analysts used the Web of Science (WoS) and MEDLINE bibliometric databases to compute a variety of indicators relevant to EDCTP’s remit. The study also presented the open access status of publications using data from oaIndx, compiled by 1science. Findings within each disease area were contextualized by World Health Organization (WHO) data on the burden of disease and disease prevalence data from the Institute for Health Metrics and Evaluation.

For each of the four disease groups, keyword searches using MEDLINE MeSH terms were performed to retrieve all pertinent material, and these were linked to the WoS to enable the preparation of bibliometric analyses. To ensure comparability to the previous study, performed by Thomson Reuters, the queries that were used to identify research related to the four diseases were reused. Slight modifications were made to include relevant additional material that was not covered in the last study, mostly through the inclusion of additional MeSH terms but also by using specialized journals in the WoS related to the diseases to increase the coverage of the datasets.

Bibliometric Technical Report of the Canada Research Chairs Program (2010)

Client: Social Sciences and Humanities Research Council (SSHRC)

Description of project: The Canada Research Chairs Program (CRCP) is a Tri-Agency program that aims to strengthen Canadian research performance in universities, as well as their affiliated institutes and hospitals, in order to position Canada amongst the world leaders in research. Science-Metrix was mandated to perform the tenth-year evaluation of the CRCP; part of this evaluation involves performing a bibliometric assessment of research funded by the program. The present bibliometric assessment is only one of the multiple lines of evidence that are used to address the following questions pertaining to the program’s success:

  • What has been the CRCP’s contribution to the capacity of universities to produce new knowledge?
  • To what extent has the CRCP contributed to universities developing a comparative advantage in strategic areas of research?

The CRCP funds two groups of researchers based on their career status—namely, established researchers who lead in their field (tier 1 Chairs) and emerging researchers who have the potential to lead in their field (tier 2 Chairs). Therefore, the study provides data that will address the following questions for each group of scholars within the research areas of each of the three councils:

  1. Are CRCP Chairholders more productive, on average, than researchers from comparable groups in terms of their number of published journal articles?
  2. Are the research outputs produced by CRCP Chairholders of high impact relative to the average output, published in peer-reviewed journals, of comparable groups?
  3. Are CRCP Chairholders collaborating more often (nationally and internationally), on average, than researchers from comparable groups?
  4. Has CRCP funding contributed to increasing the scientific performance of supported researchers in terms of their number of published journal articles?
  5. Has CRCP funding contributed to increasing the scientific performance of supported researchers in terms of the scientific impact of their journal articles?
  6. Has CRCP funding contributed to increasing the collaboration rate (national and international) of supported researchers?

Ultimately, the findings of the current study will be integrated with those of the other assessment instruments to be used in this evaluation in what is known as the “method of converging partial indicators.” When multiple indicators converge, the conclusions drawn from the evaluation are regarded as being more reliable than those based on a single indicator.

Questions 1, 2 and 3 assess the scientific performance of CRCP Chairholders relative to those of comparable groups of researchers, whereas Questions 4, 5 and 6 assess the effect of the program on the scientific performance of CRCP Chairholders through a comparison of their scientific output produced prior to and after receiving their Chairs (pre–post comparisons). The indicators used to answer these questions are:

  • the number of papers produced by each researcher;
  • the Average of Relative Citations (ARC), an indicator of scientific impact that is normalized by scientific disciplines, ensuring that the indicator can be compared across disciplines;
  • the Average of Relative Impact Factors (ARIF), a proxy for the quality of scientific publications that is also normalized by scientific disciplines; and
  • the national and international collaboration rate.