How is Brazilian research performing?
Public universities are complex institutions that fulfil a huge variety of functions and roles within society. They have differing missions, levels of output and attending a variety of different sectors of society. The universities who feature in international rankings are research-intensive – they are dedicated to the production of new knowledge and publish heavily in international journals.
In both the Shanghai Jiao Tong and CWTS Leiden Ranking, there are 23 Brazilian universities included. In the NTU ranking, 11 universities are listed. This is far more than the rest of Latin America combined (see table 1). These rankings were chosen for their focus on bibliometric indicators, in other words, in the publication of articles in indexed reputable journals. They do not consider international reputation, teaching, or financial resources available. This means that they mostly measure the amount of attention that the science produced by a university is recognised by others through a mention (citation) or an international award.
Table 1 -Number of universities included in research rankings
|Leiden||Shanghai Jiao Tong ARWU||NTU Ranking|
|Rest of Latin America||11||12||7|
Brazil is, by a long distance, the best represented in global rankings in South and Central America, with more universities listed than the rest of the continent combined. It is also better represented than both Russia and India, whose governments have invested billions into their higher education systems over the past few years.
Brazil has a very large, and very active higher education sector. Brazil has a relatively much larger number of ranking research intensive universities than other emerging economies. Given that Webometrics recognises over 28,000 universities, appearing in the top 500 places a university within the top 1.7% of institutions worldwide on citation impact performance.
Furthermore, there is a select group of elite universities; USP, Unicamp, Unesp, UFRJ, UFMG, Unifesp and UFRGS who regularly position in the world’s top 500 across different rankings. These seven institutions form the backbone of Brazil’s university research system.
Table 2- Bibliometric performance of leading research universities.
|University||P||P top 1%||P top 10%||P top 50%||PP top 1%||PP top 10%||PP top 50%|
|Universidade Estadual Paulista||12350||89||801||5402||0.7%||6.5%||43.7%|
|University of São Paulo||33295||351||2858||15804||1.1%||8.6%||47.5%|
|University of Campinas||11325||103||998||5564||0.9%||8.8%||49.1%|
|Universidade Federal do Rio de Janeiro||10460||96||897||4919||0.9%||8.6%||47.0%|
|Universidade Federal de São Paulo||6556||75||483||2847||1.1%||7.4%||43.4%|
|Universidade Federal de Minas Gerais||8013||87||657||3729||1.1%||8.2%||46.5%|
|Federal University of Rio Grande do Sul||9686||95||771||4530||1.0%||8.0%||46.8%|
Legend: P – Number of papers on Web of Science; P top X% – Number of papers among the x% most cited for area of knowledge and year; PP top X% – Proportion of total papers among X% most cited for area of knowledge and year.
However, a number of countries have improved very rapidly, driven by large government spending initiatives in China, Germany, South Korea, Israel, among others. Brazilian universities have lacked the ability to keep up with the financial resources available to carry out cutting edge science in these countries.
Brazilian universities did not accompany this growth of investment necessary to produce cutting edge science. The lack of rigid and reliable public funding has meant that the universities have had less ability to launch ambitious projects than their international peers.
We will develop the theme responding to two questions, in order to clarify some of the care and restrictions that should be placed on citation based analyses.
What is the impact of Brazilian research?
This is a complex question that cannot be answered with a single number. A holistic view informed by a variety of different indicators is needed to come close to expressing the broad nature of the impact that universities have on intellectual, social, cultural and economic life. Citation rates are an important indicator of the intellectual impact of research, when used responsibly.
There are two types of citation based indicator – size dependent and size independent. Size dependent indicators measure how much impact the university has exerted on the world – the number of highly cited papers, for example. Size independent indicators are useful to describing internal characteristics of a research community – the percentage of highly cited papers relative to all papers, or an average number of citations.
In a league table of Latin American universities by average citations per paper weighted for area of knowledge, for example, Brazil has a lower average than Colombia, Argentina and Chile, and yet it produces more articles and more highly cited articles than the three of them combined. Therefore, it is possible to draw misleading conclusions without a grasp of both indicators. It does, on the other hand, publish less than India, but publishes a higher proportion of its research as highly cited with a higher average citation rate.
South Korea, because of its heavy public investment and commitment to education, science and technology has been able to publish much more than the other countries here, at a much higher citation rate.
Table 2: Citation data of Latin American countries and other peers.
|Name||Rank||Web of Science Documents||Category Normalized Citation Impact||Times Cited||Citation Impact||Highly Cited Papers||% Highly Cited Papers|
Source: InCites Database, 2010-2020, accessed on 17.04.2020.
Citation impact is a useful measure of the influence of ideas on science for some areas of knowledge. It is predominantly used to assess natural and biological sciences, which are properly covered in sources like Web of Science. This is not the case for social sciences and humanities, which use other indexing resources and other publication media, is cited and recognised differently. It should not be used alone to judge the influence of social sciences or humanities, and especially not those published outside of the US or Europe, where research fields are oriented to local needs and not international norms. This means that while citation analyses can identify areas of outstanding performance, they cannot be used alone to identify the presence of underperforming ones. For example, while citation rates for arts and humanities and social sciences appear low for USP, the most recent QS ranking placed it among the 50 best on earth. For these areas, citations at best tell only part of the story.
Citations are not a measure of how research generates returns for society, either social or economic. While indicators of social and economic impact are sometimes correlated with citation impact, they are not always, and so they cannot be used as a proxy for them.
Is the citation impact of science improving or declining?
There are two distinct movements that can be observed in Brazilian bibliometrics over the past 15 years. The first is the huge expansion in the quantity of publication, and in the number of locations where research is happening. This means expressive increases in the northeast, north and central western regions, as well as in the southeast and south, which hold the country’s traditional research base.
As these new networks take time to build, looking at an overall mean citation score gives the impression that impact is not growing very fast. However, this is not a like-for-like comparison, as the number of actors grows. If we take just the seven elite research institutions who have been large knowledge producers over time, we can see that not only has their research improved in overall citation rate, but the number and proportion of highly cited research has also improved.
In that respect, we can conclusively say that the impact of Brazilian science has increased vastly compared to the year 2000 among the elites, but has also expanded into new territories and engaged new actors among emerging institutions. There is no evidence to suggest that Brazilian science is in decline.
Graph 1 shows the evolution of research impact of the seven largest research intensive universities from the year 2000 to 2016), normalised by area of knowledge and publication year. CNCI becomes unreliable for smaller data sets for more recent years for reasons of varying citation behaviour. This shows that all have increased significantly over the past twenty years.
Graph 1: Category normalised citation impact from Web of Science for the group of 7 (2000-2016)
Similarly, graph 2 shows that the number of highly cited papers (in the top 1% for its area of knowledge and publication year) have increased significantly since measurement started in 2008. USP has many more of these papers, but this is to be expected given that it is a much larger university. Proportionally the other universities perform similarly.
Graph 2 – Number of highly cited papers from Web of Science, 2000-2018
Finally, we can see in graph 3 that the category normalised citation impact for the whole country has increased significantly over the past decade – Brazil’s research has higher impact relative to the rest of the world. The only country from this benchmark to improve more rapidly is Russia, who started from a much lower baseline value. It should be noted that all of the other countries in this baseline have had the benefit of clear planning and massive investment in their higher education systems over this period. Bearing in mind that this is the total (including new research actors), the overall impact of the elite institutions has grown much higher, and is around the same value as South Korea – the baseline in graph 2 is 1.1, the same as the CNCI of South Korea and Mainland China.
Graph 3 – Category normalised citation impact of comparable countries 2007-2017
University communications must become more honest. We cannot complain about being judged on the basis of simplistic ranking results if we present the same results when they are positive. Communications must avoid presenting simplistic readings of rankings that focus on position without contextualisation. Too often reporting of results lacks explanation of the difference between areas of knowledge, and understanding of the difference between absolute and relative indicators. The perspective that Brazilian science is getting worse, or delivering nothing is based on a poor use of bibliometrics and lack of understanding of how rankings work. These narratives are not difficult to combat with improved communications and understanding of the underlying logic of them.
Presenting the rankings as objective measures without reflection has allowed what should be a broad normative debate become constrained into a very narrow technical one. The question should be – “what is the best the universities can be for Brazilian society?” instead it becomes “which is the best university?”. Rankings and bibliometric analyses have a role to play in answering the first question, they make up the whole of the second.
Ranking results and citation analyses should be used to supplement more sensitive perspectives, not supplant them.
In the medium term, universities must emphasise the good that they do and the relevance that they have, and learn to express this better. Making it clearer that citation impact, quality and relevance are separate, but sometimes interrelated concepts is important.
In the long term, universities must open more channels, and more permanent channels of communication and decision making with society and stakeholders. Shared governance of the institution both reduces its vulnerability to outside attacks. It becomes less possible to accuse the university as existing for itself when a greater diversity of actors are involved in its decision making. This also has the benefit of increasing understanding of the local impact and relevance of the university, as well as informing strategies on how to further it.
Finally, it has been a long standing problem that while much of the university’s production is internationalised, and on a personal level Brazilian scientists are heavily engaged with the global production of knowledge, the way in which it is evaluated and represented is uniquely Brazilian. This leads to a wide gap in understanding between how universities in other countries view research performance, and how Brazilian institutions view it. In part, this is an obstacle to raising the impact of Brazilian science further, as the incentives towards publishing high impact and high relevance research are much lower than in other countries with more sophisticated research governance.
What the universities should do
The universities should pay attention to the way in which their communications teams handle these data. This should include ensuring that they understand what is being measured, what is not being measured and how it is normalised and presented.
An understanding of these aspects should be reinforced through specific training in responsible use of indicators and rankings, and the production of guidelines orienting teams on how to comment on them.
The universities should seek to build better and more rigorous tools to measure, compare and express what they do well. Representing local social and economic impacts in all areas, as well as representing the cultural benefits of a comprehensive university should be a priority to turn the debate from purely defensive into a positive representation of what is produced. This means expanding the number of types of production measured, as well as the range and sophistication of the measurement of the impacts that they have.
Higher education institutions and funding bodies should look to include international specialists into the process of evaluation in order to ensure that research governance accompanies the evolution of research behavior both in Brazil and elsewhere.
The universities must consider building durable and significant channels for society to contribute to university governance. Whether this is through a board of trustees, the staging of regular public fora or some other method of shared governance.