Below are some examples of projects listed by type of activity or need expressed by clients.
MEASUREMENT AND BENCHMARKING OF SCIENCE AT NATIONAL AND INTERNATIONAL LEVELS
Étude bibliométrique sur la recherche tunisienne (2015)
Client: Agence Nationale de Promotion de la Recherche Scientifique (ANPR)
Description: The Agence Nationale de Promotion de la Recherche Scientifique (ANPR) in Tunisia benefitted from the project PASRI (Projet d’Appui au Système de Recherche et de l’Innovation) to conduct a study on the state of research in Tunisia. Science-Metrix was mandated to carry out this study. PASRI was founded by the Ministère tunisien de l’Enseignement Supérieur et de la Recherche Scientifique in collaboration with the Ministère tunisien de l’Industrie. Funded by the European Union, this project aims to improve the contribution of research and innovation for growth and employment development in Tunisia by strengthening the link between research and business.
This report (in French only) presents an analysis of the state of Tunisian research from 2002 to 2013. The study analyses the country's total scientific output, as well as subfields of science, in comparison with other selected countries and regions. Bibliometric data on Tunisian organisations are also provided, as well as data on international cooperation. This study aims to identify the strengths and weaknesses of Tunisian research and to highlight research opportunities that could become priority research fields for the coming years. Some elements of discussion about international cooperation are also presented.
Bibliometric Analysis of the Current State of Science and Technology in the Member States of the African Union (2013)
Client: African Union
Description: Science-Metrix was commissioned by the African Observatory for Science, Technology and Innovation (AOSTI) to conduct a bibliometric assessment of the scientific production and performance of the 54 members in the African Union over 6 thematic domains and 22 scientific fields compared to the world level for the 2005–2010 period.
During the course of this study, Science-Metrix undertook the most exhaustive data cleaning task ever conducted by the firm. Institutions from all country members of the African Union that published more than five publications between the years 2005 and 2012 were standardized. In addition, a list of the 500 most publishing researchers was produced, necessitating the standardization of more than 1,500 names.
The key tasks were the following:
- Bibliometric data and technometric data
- Visualization tools
- Training, knowledge transfer
Deliverables: Complete set of bibliometrics (databooks), visuals of collaboration patterns, final analytical report.
Bibliometric Study to Support the Assessment of Canadian Ocean Science (2013)
Client: Council of Canadian Academies
Description: Science-Metrix was mandated by the Council of Canadian Academies (CCA) to provide bibliometric indicators related to the Canadian performance in ocean science research as well as related sub-topics of high interest.
In its analysis, Science-Metrix provided bibliometric data for the top 25 countries and top 200 institutions publishing the most in the field of ocean science, which provided insights to the CCA regarding the positioning of Canada and Canadian institutions at the international level. Furthermore, the top 100 publishing Canadian institutions were also identified to provide a complete overview of ocean science research in Canada. The results of the study highlighted the high quality of the research lead in this field in Canada and in these sub-topics, with noticeable strengths identified in the sub-topics of arctic, human impact and human health research associated to ocean science.
The results from this study were incorporated in a CCA report prepared by its Expert Panel on Canadian Ocean Science. This report presents a comprehensive portrait of the state of ocean science research in Canada.
Cross-cutting Analysis of Scientific Publications versus Other Science, Technology and Innovation Indicators (2013)
Client: European Commission
Description: The last two decades have seen a steady rise in the development of STI indicators. Their use is intended not only to better manage and govern the complex European system but to measure progress towards the achievement of an increasingly wide variety of social and economic objectives. A greater number and variety of actors are now involved in indicator development, contributing new guidelines, new data sources and new areas of inquiry.
A major focus of these efforts has been to find appropriate, quantitative statistical tools that are comparable across systems (i.e., countries, regions, sectors, organisations and industries) and that can strike the best balance between internationally comparable and nationally relevant indicators (Edler & Flanagan, 2011; Lugones & Suarez, 2010). In order to create robust and meaningful measures, ‘positioning’ indicators must also account for the distinct contextual factors that underlie each system, which may be at least as important as formal inputs and outputs to their ultimate performance (Edler & Flanagan, 2011; Lepori, Barré, & Filliatreau, 2008). In the face of often considerable underlying conceptual and methodological difficulties, newer indicators must both consider and attempt to confirm the specific drivers of research output and performance of countries and regions.
This report contributes to this growing body of literature by performing a cross-cutting assessment of performance centering on the European Research Area (ERA) in addition to a number of selected countries. The study uses bibliometric statistics computed by Science-Metrix as part of a project conducted for the European Commission (EC). These data on scientific output are used to examine performance in light of 17 input indicators such as R&D investments. Specifically, this report investigates:
- the factors behind publication outputs and productivity (i.e., the efficiency with which entities are converting research inputs into research outputs) of countries/regions, as revealed through an analysis of scientific production; and
- the factors behind the production patterns of countries, as revealed through an analysis of scientific concentration (by research area), across thematic domains.
Bibliometric Analysis for the Expert Panel on the State of Science and Technology in Canada (2011)
Client: Council of Canadian Academies (CCA)
Description: The Council of Canadian Academies (CCA) mandated Science-Metrix to produce bibliometric data that would update and expand the bibliometric analyses from its 2006 report on the state of science and technology (S&T) in Canada. The major theme being addressed by an Expert Panel established by the CCA relates to determining the current state of S&T in Canada. The following questions were developed to explore this theme:
- Considering both basic and applied research fields, what are the scientific disciplines and technological applications in which Canada excels?
- How are these strengths distributed geographically across the country (by provinces)?
- How do these trends compare with what has been taking place in comparable countries?
- In which scientific disciplines and technological applications has Canada shown the greatest improvement/decline in the last five years?
- What major trends have emerged?
- Which scientific disciplines and technological applications have the potential to emerge as areas of prominent strength for Canada?
To answer these questions, Science-Metrix prepared timeline measures of the quantity, specialization, quality and impact of research (based on publications) and innovation (based on patents) performed by countries, with a particular emphasis on Canada and its provinces. The analyses were performed at the field and subfield levels (patent classes were matched as much as possible to scientific disciplines), and additional dimensions of research were investigated, such as scientific collaboration and the cross-country mobility of Canadian researchers. To better rank Canada’s performance by scientific area and to better benchmark its performance relative to other countries, Science-Metrix used its new approach to the multicriteria ranking of alternatives.
During this project a large set of bibliometric data was produced. The performance of Canada relative to other leading countries in the world for all fields of science, the performance of the provinces and territories and the performance of countries and provinces regarding patents were assessed using the following indicators:
- number of papers (full and fractional counts)
- average of relative citations (ARC)
- average of relative impact factor (ARIF)
- number of international and inter-provincial collaborations
- growth index (GI)
- collaboration index (CI)
- collaboration affinity of Canada towards other countries
- collaboration affinity of other countries towards Canada
- number of patents
- specialization index
- flow of IP
A composite indicator combining four aforementioned indicators (number of papers in full counting, ARC, ARIF and CI) was also calculated to assess the relative performance of leading countries, which were then ranked based on the indicator. An identical weight was awarded to each of the four indicators. In addition, a graphical presentation of a collaboration network was produced to examine existing connections between leading countries, along with a positional analysis to outline Canada’s performance in 22 fields of science based on Science-Metrix’s ontology.
MEASUREMENT AND BENCHMARKING OF SCIENCE FOR SPECIFIC RESEARCH PERFORMERS
She Figures 2015 (2015)
Client: European Commission
Description: The She Figures publication is the main source of pan-European, comparable statistics on the state of gender equality in research and innovation. It covers a wide range of themes, including the proportions of women and men amongst top‑level graduates, academic staff and research/advisory boards, the working conditions for women and men researchers, the integration of the gender dimension in the content of peer-reviewed scientific articles, and various indicators measuring gender gaps in scientific and innovation outputs.
Science-Metrix collaborated with KU Leuven, ICF International and representatives from the European Commission’s Directorate-General of Research and Innovation on the preparation of She Figures 2015. While the majority of indicators are drawn from Eurostat databases and the Women in Science database, in the 2015 edition of She Figures, the Web of ScienceTM (WoSTM) has been used to produce scientometric indicators by sex (e.g., ratio of women to men authorship of scientific papers, proportion of a country’s scientific production including a sex or gender dimension), and the EPO Worldwide Patent Statistical Database (PATSTAT) has been used to produce a technometric indicator by sex (e.g., ratio of women to men inventorship). The Science-Metrix bibliometrics team were instrumental in developing these indicators.
Released every three years since 2003, the She Figures report provides a key evidence base for policies in the area of gender equality in research and innovation. It is recommended reading for policymakers, researchers and anybody with a general interest in these issues. The co-operation of EU Member States and Associated Countries, the Helsinki Group and their Statistical Correspondents, and Eurostat in preparing She Figures is gratefully acknowledged.
Scientific Output and Collaboration of Companies Publishing the Most in the ERA (2013)
Client: European Commission
Description: Science-Metrix has been selected as the provider of bibliometric indicators for the European Commission’s Directorate-General for Research & Innovation (DG RTD) that will be integrated into the European Commission’s evidence-based monitoring of progress towards the objectives set forth in the Lisbon framework and the post-Lisbon strategy for the European Research Area (ERA). The bibliometric component of this monitoring system is part of a package of six complementary studies reporting on the dynamics of research activities along the whole route of knowledge, from R&D investments to publications, patents, and licensing.
More specifically, this report aims to measure the scientific production of research-intensive companies within the ERA zone, portray the geographic distribution of their scientific activities, and characterize their collaboration profiles. Data is presented for the top 100 firms that published the most within the ERA zone from 2007 to 2011; this includes multinational companies with activities outside the ERA, although only their production within this zone is considered.
Standardization of author addresses for firms and subsidiaries
The method used to identify and standardize the author addresses for the leading companies is based on two approaches. First, a matching approach was applied based on the list of firms found in the 2012 EU Industrial R&D Scoreboard. The second, and complementary, approach is based on a frequency ranking of the publication output of companies found in Scopus. This involves cleaning the preliminary lists of institutions identified as companies, beginning with those that publish the most. The merger of those two approaches results in a list of identified companies, from which subsidiaries are identified, extracted from the publication database, and associated with their parent companies. The steps involved in these two approaches are described below.
When the addresses of leading firms and their subsidiaries have been standardized and validated via quality control processes, the unique and standardized organization names are reincorporated into the database, where they can be used to produce robust bibliometric data and statistics at the institutional level.
The production and collaboration profiles are based on the following set of indicators:
- number of publications
- number of exclusively intra-company publications
- number of exclusively intra-company publications with a single affiliate
- number of exclusively intra-company publications with multiple affiliates
- number of co-publications
- number of co-publications (Acad or RPO)
- number of co-publications (firm)
- number of co-publications (unclassified)
The report is primarily descriptive, focusing on the salient points relevant to the report’s two main parts: geographic distribution of the scientific production of the most publishing companies within the ERA and their collaboration patterns.
MAPPING OF SCIENTIFIC COLLABORATION
Study on Network Analysis of the 7th Framework Programme Participation (2015)
Client: European Commission
Description: The willingness to increase Europe’s competitiveness by strengthening the scientific and technological bases of industry whilst closing the ‘technology gap’ with the US and later with Japan played an important part in the setting up of a multiannual Framework Programme for Research, Technological Development and Development Activities. From the First Framework Programme (FP1) initiated in 1984, to Horizon 2020, which was launched in 2014, the framework programmes have been among the EU’s main instruments for funding research and innovation in Europe. While the role of the framework programmes has developed over time, two strategic objectives were common to each of them: ‘to strengthen the scientific and technological basis of European industry and to encourage it to become more competitive at international level’ (European Parliament, 2014).
The Seventh Framework Programme (FP7) (2007–2013) was promoted as a key tool to respond to Europe’s needs in terms of jobs and competitiveness, and to maintain leadership in the global knowledge economy. The stated overriding aim of FP7 was ‘to contribute to the Union becoming the world’s leading research area. This requires the Framework Programme to be strongly focused on promoting and investing in world-class state-of-the-art research based primarily upon the principal of excellence in research’ (European Parliament, 2006.).
This report examined the effectiveness of FP7’s network approach in achieving EU research policy objectives and fostering Europe’s international competitiveness in S&T. In investigating the potentials and limits of this approach, the study examined the effects of multidisciplinary, interdisciplinary, intersectoral and international collaborations on achieving the positive outcomes sought in FP7.
The study’s methods included a network analysis, a survey, case studies including in-depth interviews, a representative stakeholder workshop and regression analyses. The key findings reveal benefits from the continuity of research from FP6 to FP7. However, despite these demonstrated benefits and the increase in participants overall for FP7, the study also found a high attrition rate of organisations from FP6 to FP7.
Bibliometric Study in Support of Norway’s Strategy for International Research Collaboration (2014)
Client: Research Council of Norway
Description: Science-Metrix has been commissioned to provide the Research Council of Norway (RCN) with bibliometric indicators of scientific performance and collaboration for Norway and its international partners. The presentation of these indicators was organized with a view to guiding the RCN’s efforts to establish long-term institutional co-operation between Norwegian institutions and similar institutions in other countries under its International Strategy.
More specifically, the report aims at characterizing the scientific production profiles and collaboration patterns of Norway and 57 selected countries in Europe and elsewhere around the world.
Firstly, the report provides a brief comparative analysis of the current scientific performance of Norway and the 57 selected countries, along with their collaboration patterns in the sciences in general. Collaboration is measured from both an international (co-authorship with any foreign country) and a bilateral (co-authorship specifically with Norway) perspective. The analysis of collaboration with Norway is also performed across the organizational sectors of Higher Education, Government, and Business Enterprise. The key questions addressed include:
- Which are the leading scientific nations and those that co-publish most with foreign partners and with Norway in particular?
- What has changed in the past decade—in terms of countries’ relative performance and collaboration patterns—and what is likely to change in the near future?
Secondly, the report analyzes in more detail these countries’ performance and collaboration in 15 scientific themes of significant importance to Norway. This analysis is intended to provide strategic information to the RCN about the scientific partnerships that are most beneficial to the country in each strategic field. It also identifies which of those promising partnerships appear to be under-exploited, with a view to providing recommendations as to potential candidates with which Norway could strengthen partnerships by thematic area.
Thirdly, the report presents further information at the micro-level (i.e., organisational level) to help the RCN target organisations—within the countries identified as potential candidates with which Norway could strengthen partnerships—in the context of setting up specific bilateral agreements. Information on the scientific performance and collaboration patterns of organizations is synthesized in the form of collaboration networks.
Finally, the report presents a detailed analysis of various types of multilateral scientific partnerships to address the following questions: Which type of scientific partnership is most beneficial to a country’s scientific impact and what are the mechanisms underlying such gains in impact? These analyses include:
- network analysis, which presents international collaboration networks for each of the themes selected for this study on the organisational level;
- analysis of the effect of multilateral co-authorship on the scientific impact of research output;
- regional analysis; and
- cluster analysis.
Constitution of dataset
For this project, data were first produced for the sciences in general. More specifically, only documents published in refereed scientific journals (mostly articles, reviews and conference proceedings) were retained, as these documents were reviewed by peers prior to being accepted for publication.
Data were also produced by field of science using Science-Metrix’s journal-based and mutually exclusive classification scheme. Moreover, 15 themes of strategic interest to Norway were delineated by retrieving relevant publications using keyword-based queries, specialist journals, and entire subfields from Science-Metrix’s classification where appropriate.
Data were also produced for the following regional aggregates: the Nordic countries, EU15 countries, EU28 countries, and the world. Because some of the included countries have large R&D systems characterized by regional differentiations and strongholds, Science-Metrix also performed a regional analysis for these countries. Data for organizations and researchers were also produced. As well, for organizations, data were produced for a number of clusters or networks of highly interconnected organizations as part of optional deliverable.
Several indicators were computed throughout the course of this project:
- number of publications
- international co-publications
- international co-publications with Norway
- international collaboration rate
- collaboration rate with Norway
- specialization index (SI)
- growth ratio (GR)
- relative citations (RC)
- relative number of authors
- average of relative citations (ARC)
- average of relative impact factors (ARIF)
- transdisciplinarity (TD)
- collaboration index (CI)
- affinity index (AI)
- page rank
- betweenness centrality
- share of output available in open access (OA)
Intra-European Cooperation Compared to International Collaboration of the ERA Countries (2013)
Client: European Commission
Description: This report is part of the six complementary studies that Science-Metrix has performed for the European Commission’s Directorate-General for Research & Innovation (DG RTD). The work involves the collection, analysis and updating of bibliometric data that will be integrated into the European Commission’s evidence‑based monitoring of progress towards the objectives set forth in the Lisbon framework and the post-Lisbon strategy for the European Research Area (ERA).
The present report
Within the FP7, a strong emphasis has been placed on sponsoring collaborative research within Europe (and beyond) as a way to increase Europe’s competitiveness within the global knowledge-based economy. In particular, the co-operation actions implemented under FP7 are foreseen to foster the achievement of complex research challenges requiring pan-European efforts, to improve research excellence in Europe, as well as to reduce the fragmentation of the European research landscape, thereby contributing to the realisation of the European Research Area (ERA)—an open space for knowledge and growth.
To support the monitoring of collaborative research and its effect under FP7, the current report first provides in-depth analyses of the collaboration patterns of European countries (i.e., those included within the ERA; Israel is also covered), as well as of a selected set of international comparables within and outside of the ERA. It also investigates whether the integration of European countries has progressed since the start of FP7 both within and outside of the ERA, as well as attempts to determine whether the co-operation actions implemented under FP7 appear to have triggered an increased integration of European countries by comparing the magnitude of changes both within and outside of the ERA.
Subsequently, the report characterizes the effect of scientific partnerships on scientific excellence by comparing the scientific impact of various types of publications/co-publications for European countries and a selected set of international comparables. It then provides insights into whether the intensification of cross-border co-operation within Europe promoted under the FP7 can be expected to increase the excellence of European research by comparing the scientific impact of various types of publications/co-publications at the aggregate level for EU27 countries. This analysis also sheds light on which types of partnerships are likely to be most beneficial to scientific excellence.
ASSESSMENT OF RESEARCH FUNDING PROGRAMS
Bibliometric Technical Report of the Canada Research Chairs Program (2010)
Client: Social Sciences and Humanities Research Council (SSHRC)
Description of project: The Canada Research Chairs Program (CRCP) is a Tri-Agency program that aims to strengthen Canadian research performance in universities, as well as their affiliated institutes and hospitals, in order to position Canada amongst the world leaders in research. Science-Metrix was mandated to perform the tenth-year evaluation of the CRCP; part of this evaluation involves performing a bibliometric assessment of research funded by the program. The present bibliometric assessment is only one of the multiple lines of evidence that are used to address the following questions pertaining to the program’s success:
- What has been the CRCP’s contribution to the capacity of universities to produce new knowledge?
- To what extent has the CRCP contributed to universities developing a comparative advantage in strategic areas of research?
The CRCP funds two groups of researchers based on their career status—namely, established researchers who lead in their field (tier 1 Chairs) and emerging researchers who have the potential to lead in their field (tier 2 Chairs). Therefore, the study provides data that will address the following questions for each group of scholars within the research areas of each of the three councils:
- Are CRCP Chairholders more productive, on average, than researchers from comparable groups in terms of their number of published journal articles?
- Are the research outputs produced by CRCP Chairholders of high impact relative to the average output, published in peer-reviewed journals, of comparable groups?
- Are CRCP Chairholders collaborating more often (nationally and internationally), on average, than researchers from comparable groups?
- Has CRCP funding contributed to increasing the scientific performance of supported researchers in terms of their number of published journal articles?
- Has CRCP funding contributed to increasing the scientific performance of supported researchers in terms of the scientific impact of their journal articles?
- Has CRCP funding contributed to increasing the collaboration rate (national and international) of supported researchers?
Ultimately, the findings of the current study will be integrated with those of the other assessment instruments to be used in this evaluation in what is known as the “method of converging partial indicators.” When multiple indicators converge, the conclusions drawn from the evaluation are regarded as being more reliable than those based on a single indicator.
Questions 1, 2 and 3 assess the scientific performance of CRCP Chairholders relative to those of comparable groups of researchers, whereas Questions 4, 5 and 6 assess the effect of the program on the scientific performance of CRCP Chairholders through a comparison of their scientific output produced prior to and after receiving their Chairs (pre–post comparisons). The indicators used to answer these questions are:
- the number of papers produced by each researcher;
- the Average of Relative Citations (ARC), an indicator of scientific impact that is normalized by scientific disciplines, ensuring that the indicator can be compared across disciplines;
- the Average of Relative Impact Factors (ARIF), a proxy for the quality of scientific publications that is also normalized by scientific disciplines; and
- the national and international collaboration rate.
Bibliometric Analysis of Individuals Supported by Alberta Ingenuity Fund (2008)
Client: Alberta Ingenuity Fund (AIF)
Description: This study examined the scientific production of researchers and graduate students who received financial support from the Alberta Ingenuity Fund (AIF). The scientific output of about 120 researchers and 300 graduate students was analyzed based on three bibliometric indicators of scientific performance: (1) number of papers, (2) average of relative citations, and (3) average of relative impact factors.
The selection of high-calibre researchers and academically superior graduate students is a crucial component of AIF funding programs. As such, by comparing the scientific output of successful researchers and students to comparable groups, the analysis provided insight as to whether AIF has effectively selected outstanding researchers and highly promising students.
In addition, the effect of AIF funding on the volume and impact of the scientific production of funded applicants was investigated. In the case of graduate students, the analysis also looked at the effect of AIF funding on mobility.