Impact Profile™

Visualization of an organization or country’s Impact Profile™ was first proposed and demonstrated in an academic paper published in 20071. More recently, the ISI report on Profiles, not Metrics published in 2019, described in detail how the methodology could be used to reveal additional information hidden beneath a simple metric of average citation impact. You can download the full report here.

For example, to show how citations can be more effectively understood when seen in an Impact Profile™, we compare the research output of three organizations: Harvard University, the University of California system (all the institutions and campuses that belong to the University) and the Centre National de la Recherche Scientifique (the CNRS, which is the French national research institute system).

In the following table, we show the number and citation impact of those documents that were categorized as Journal Articles published in the Web of Science Cell Biology journal category for the 10 years between 2010-2019.

Table image of number in citation impact

There are two versions of citation impact. The middle column shows the average number of citations per paper, which is only a rough measure of citation impact. It’s not a good comparator because citations accumulate over time at a rate that is discipline dependent. For this reason, most analysts prefer Category Normalized Citation Impact (CNCI) when comparing the research output of institutions. This uses the ratio between the actual citation count for a document and the world average for all other documents of the same type published in the same year and in the same journal category. This is called normalization and the CNCI for a group of articles is then the average of their individual ratios.

CNCI is a much better indicator of citation impact than the average citation count. It also has the advantage shared by all indicators in that it’s a simple summary reference number. However, because it is a single point metric that is being used to tell us about a distribution of real values (more than 12,000 values in the case of California), it still has inherent limitations.

Citation counts are always very skewed, with many low and a few high values in any sample. Many of the documents in a sample are not cited. At the far end of the skew will be a small number of very highly cited documents. These high values can have a huge effect on the average index value, but it’s not immediately apparent when we only look at CNCI.

A much more informed analysis that helps us to interpret our data more effectively comes from visualizing this spread across the skew, so we can see how many papers are at the high end and what the grouping around the center looks like. To do this, we categorize the individual CNCI values relative to the world average.

Our first step is to take all the uncited documents and put them in a separate category or bin. We then split the cited documents into those with CNCI below world average and those with CNCI above world average (where CNCI=1.0). For each side of the world average we then allocate documents to one of four bins, doubling or halving the CNCI threshold or boundary each time. Above world average we have bins that cover from one to two times the world average CNCI, then 2-4x, 4-8x and finally a bin for those high impact documents with CNCI more than 8x world average.

At the same time, below CNCI =1.0, we take the counts from 0.5-1.0, then 0.25 to 0.5 (i.e., ¼ to ½), then 0.125-0.25 (⅛ to ¼) and then those cited but less often than one-eighth world average CNCI.

The last step in the procedure is to display the data, not as document counts, but as percentages of the total count for each organization in our picture. We do this because of the variations in sample size. It allows us to make informative comparisons between such different entities as individual universities and their home country.

The overall Impact Profile of each entity we want to view is revealed by this categorization across the uncited plus eight cited bins, presenting a picture of the real spread of more and less well-cited papers. It shows us the balance between cited and uncited, the balance either side of world average CNCI, and the way in which the distribution spreads up into the most impactful categories. It ensures that our understanding of the data and our interpretation of relative performance is not affected by an average index that can be skewed by a few exceptional documents.

 Impact Profile Category Normalization Citation Impact

This procedure delivers a more informative picture than the summary CNCI values in our first table. In the visualization, the spread of the papers across the first seven bins (including uncited) appear relatively similar. In the bins below world average = 1.0, however, there is a consistently smaller share of articles for Harvard University compared to the other two institutions. Consequently, in the two bins at the highest CNCI thresholds, we see Harvard University has a relatively greater share of articles performing above the world average. The net outcome, which we saw in the table, is Harvard having a higher average CNCI = 2.89 in Cell Biology than the others.

By contrast, the CNRS, which had the lowest CNCI in the table, can now be seen to have relatively more article bins below world average and relatively fewer in bins above world average. As such, it’s possible to understand the overall values clearer than when using the summary metric alone.

Impact Profile™ visualizations are available at the levels of Organization, Location, Journal, and Funding Agency entities. This allows you to effectively visualize and compare research performance across multiple entities. Comparison is always a sound path to improved understanding and interpretation.

 

1Adams, J., Gurney, K. and Marshall, S. (2007). Profiling citation impact: a new methodology. Scientometrics, 72, 325-344. DOI: 10.1007/s11192-007-1696-x