How good are Brazilian universities?

How is Brazilian research performing?

Public universities are complex institutions that fulfil a huge variety of functions and roles within society. They have differing missions, levels of output and attending a variety of different sectors of society. The universities who feature in international rankings are research-intensive – they are dedicated to the production of new knowledge and publish heavily in international journals. 

In both the Shanghai Jiao Tong and CWTS Leiden Ranking, there are 23 Brazilian universities included. In the NTU ranking, 11 universities are listed. This is far more than the rest of Latin America combined (see table 1). These rankings were chosen for their focus on bibliometric indicators, in other words, in the publication of articles in indexed reputable journals. They do not consider international reputation, teaching, or financial resources available. This means that they mostly measure the amount of attention that the science produced by a university is recognised by others through a mention (citation) or an international award.

Table 1 -Number of universities included in research rankings

	Leiden	Shanghai Jiao Tong ARWU	NTU Ranking
Brazil	23	23	11
Rest of Latin America	11	12	7
Russia	3	11	5
India	25	16	11
China	165	132	99

Brazil is, by a long distance, the best represented in global rankings in South and Central America, with more universities listed than the rest of the continent combined. It is also better represented than both Russia and India, whose governments have invested billions into their higher education systems over the past few years. 

Brazil has a very large, and very active higher education sector. Brazil has a relatively much larger number of ranking research intensive universities than other emerging economies. Given that Webometrics recognises over 28,000 universities, appearing in the top 500 places a university within the top 1.7% of institutions worldwide on citation impact performance.

Furthermore, there is a select group of elite universities; USP, Unicamp, Unesp, UFRJ, UFMG, Unifesp and UFRGS who regularly position in the world’s top 500 across different rankings. These seven institutions form the backbone of Brazil’s university research system.

Table 2- Bibliometric performance of leading research universities.

University	P	P top 1%	P top 10%	P top 50%	PP top 1%	PP top 10%	PP top 50%
Universidade Estadual Paulista	12350	89	801	5402	0.7%	6.5%	43.7%
University of São Paulo	33295	351	2858	15804	1.1%	8.6%	47.5%
University of Campinas	11325	103	998	5564	0.9%	8.8%	49.1%
Universidade Federal do Rio de Janeiro	10460	96	897	4919	0.9%	8.6%	47.0%
Universidade Federal de São Paulo	6556	75	483	2847	1.1%	7.4%	43.4%
Universidade Federal de Minas Gerais	8013	87	657	3729	1.1%	8.2%	46.5%
Federal University of Rio Grande do Sul	9686	95	771	4530	1.0%	8.0%	46.8%

However, a number of countries have improved very rapidly, driven by large government spending initiatives in China, Germany, South Korea, Israel, among others. Brazilian universities have lacked the ability to keep up with the  financial resources available to carry out cutting edge science in these countries. 

Brazilian universities did not accompany this growth of investment necessary to produce cutting edge science. The lack of rigid and reliable public funding has meant that the universities have had less ability to launch ambitious projects than their international peers.

We will develop the theme responding to two questions, in order to clarify some of the care and restrictions that should be placed on citation based analyses.

What is the impact of Brazilian research?

This is a complex question that cannot be answered with a single number. A holistic view informed by a variety of different indicators is needed to come close to expressing the broad nature of the impact that universities have on intellectual, social, cultural and economic life. Citation rates are an important indicator of the intellectual impact of research, when used responsibly.

There are two types of citation based indicator – size dependent and size independent. Size dependent indicators measure how much impact  the university has exerted on the world – the number of highly cited papers, for example. Size independent indicators are useful to describing internal characteristics of a research community – the percentage of highly cited papers relative to all papers, or an average number of citations.

In a league table of Latin American universities by average citations per paper weighted for area of knowledge, for example, Brazil has a lower average than Colombia, Argentina and Chile, and yet it produces more articles and more highly cited articles than the three of them combined. Therefore, it is possible to draw misleading conclusions without a grasp of both indicators. It does, on the other hand, publish less than India, but publishes a higher proportion of its research as highly cited with a higher average citation rate. 

South Korea, because of its heavy public investment and commitment to education, science and technology has been able to publish much more than the other countries here, at a much higher citation rate.

Table 2: Citation data of Latin American countries and other peers.

Name	Rank	Web of Science Documents	Category Normalized Citation Impact	Times Cited	Citation Impact	Highly Cited Papers	% Highly Cited Papers
SOUTH KOREA	14	721,740	0.95	7,132,071	9.88	4,987	0.69
INDIA	15	909,495	0.81	6,764,465	7.44	4,206	0.46
BRAZIL	18	577,380	0.85	4,492,292	7.78	3,207	0.56
RUSSIA	26	490,622	0.75	2,801,272	5.71	2,078	0.42
MEXICO	39	188,425	0.83	1,527,758	8.11	1,305	0.69
ARGENTINA	41	121,590	0.96	1,186,841	9.76	999	0.82
CHILE	44	102,993	1.13	1,079,828	10.48	1,075	1.04
COLOMBIA	49	65,227	1.00	577,733	8.86	749	1.15

Source: InCites Database, 2010-2020, accessed on 17.04.2020.

Citation impact is a useful measure of the influence of ideas on science for some areas of knowledge. It is predominantly used to assess natural and biological sciences, which are properly covered in sources like Web of Science. This is not the case for social sciences and humanities, which use other indexing resources and other publication media, is cited and recognised differently. It should not be used alone to judge the influence of social sciences or humanities, and especially not those published outside of the US or Europe, where research fields are oriented to local needs and not international norms. This means that while citation analyses can identify areas of outstanding performance, they cannot be used alone to identify the presence of underperforming ones. For example, while citation rates for arts and humanities and social sciences appear low for USP, the most recent QS ranking placed it among the 50 best on earth. For these areas, citations at best tell only part of the story.

Citations are not a measure of how research generates returns for society, either social or economic. While indicators of social and economic impact are sometimes correlated with citation impact, they are not always, and so they cannot be used as a proxy for them.

Is the citation impact of science improving or declining?

There are two distinct movements that can be observed in Brazilian bibliometrics over the past 15 years. The first is the huge expansion in the quantity of publication, and in the number of locations where research is happening. This means expressive increases in the northeast, north and central western regions, as well as in the southeast and south, which hold the country’s traditional research base.

 As these new networks take time to build, looking at an overall mean citation score gives the impression that impact is not growing very fast. However, this is not a like-for-like comparison, as the number of actors grows. If we take just the seven elite research institutions who have been large knowledge producers over time, we can see that not only has their research improved in overall citation rate, but the number and proportion of highly cited research has also improved. 

In that respect, we can conclusively say that the impact of Brazilian science has increased vastly compared to the year 2000 among the elites, but has also expanded into new territories and engaged new actors among emerging institutions. There is no evidence to suggest that Brazilian science is in decline.

Graph 1 shows the evolution of research impact of the seven largest research intensive universities from the year 2000 to 2016), normalised by area of knowledge and publication year. CNCI becomes unreliable for smaller data sets for more recent years for reasons of varying citation behaviour. This shows that all have increased significantly over the past twenty years.

Graph 1: Category normalised citation impact from Web of Science for the group of 7 (2000-2016)

Similarly, graph 2 shows that the number of highly cited papers (in the top 1% for its area of knowledge and publication year) have increased significantly since measurement started in 2008. USP has many more of these papers, but this is to be expected given that it is a much larger university. Proportionally the other universities perform similarly.

Graph 2 – Number of highly cited papers from Web of Science, 2000-2018

Finally, we can see in graph 3 that the category normalised citation impact for the whole country has increased significantly over the past decade – Brazil’s research has higher impact relative to the rest of the world. The only country from this benchmark to improve more rapidly is Russia, who started from a much lower baseline value. It should be noted that all of the other countries in this baseline have had the benefit of clear planning and massive investment in their higher education systems over this period. Bearing in mind that this is the total (including new research actors), the overall impact of the elite institutions has grown much higher, and is around the same value as South Korea – the baseline in graph 2 is 1.1, the same as the CNCI of South Korea and Mainland China.

Graph 3 – Category normalised citation impact of comparable countries 2007-2017

Conclusion

University communications must become more honest. We cannot complain about being judged on the basis of simplistic ranking results if we present the same results when they are positive. Communications must avoid presenting simplistic readings of rankings that focus on position without contextualisation. Too often reporting of results lacks explanation of the difference between areas of knowledge, and understanding of the difference between absolute and relative indicators. The perspective that Brazilian science is getting worse, or delivering nothing is based on a poor use of bibliometrics and lack of understanding of how rankings work. These narratives are not difficult to combat with improved communications and understanding of the underlying logic of them.

Presenting the rankings as objective measures without reflection has allowed what should be a broad normative debate become constrained into a very narrow technical one. The question should be – “what is the best the universities can be for Brazilian society?” instead it becomes “which is the best university?”. Rankings and bibliometric analyses have a role to play in answering the first question, they make up the whole of the second.

Ranking results and citation analyses should be used to supplement more sensitive perspectives, not supplant them. 

In the medium term, universities must emphasise the good that they do and the relevance that they have, and learn to express this better. Making it clearer that citation impact, quality and relevance are separate, but sometimes interrelated concepts is important.

In the long term, universities must open more channels, and more permanent channels of communication and decision making with society and stakeholders. Shared governance of the institution both reduces its vulnerability to outside attacks. It becomes less possible to accuse the university as existing for itself when a greater diversity of actors are involved in its decision making. This also has the benefit of increasing understanding of the local impact and relevance of the university, as well as informing strategies on how to further it.

Finally, it has been a long standing problem that while much of the university’s production is internationalised, and on a personal level Brazilian scientists are heavily engaged with the global production of knowledge, the way in which it is evaluated and represented is uniquely Brazilian. This leads to a wide gap in understanding between how universities in other countries view research performance, and how Brazilian institutions view it. In part, this is an obstacle to raising the impact of Brazilian science further, as the incentives towards publishing high impact and high relevance research are much lower than in other countries with more sophisticated research governance.