International scientific journals as seen through the Journal Citation Reports : analysis of the ISI evaluation system. Application to INRA output.
M.-H. MAGRI 1, A. SOLARI 2, K. RERAT 1
1 INRA, Centre de recherche
de Jouy-en-Josas, Unité centrale de documentation,
78352, Jouy-en-Josas cedex, France.
e-mail : firstname.lastname@example.org
2 INRA, Centre de recherche
de Jouy-en-Josas, Station de physiologie animale, Unité
78352, Jouy-en-Josas cedex, France.
e-mail : email@example.com
version en français
As scientific journals are the main means of disseminating research findings they are subject to expertise in particular in the case of evaluation of research systems and scientists. This expertise is based on several methods, the most famous being citation analysis. This method has been widely adopted over the past twenty years, largely due to the influence of the Institute for Scientific Information (ISI), which produces a large amount of data extracted from about 4500 scientific journals which is then published annually in the Science Citation Index Journal Citation Reports (SCI JCR). In the social sciences, the Social Science Citation Index, Journal Citation Reports covers 1500 journals. The SCI JCR has become more or less an authority for evaluating scientific journals using only one of its bibliometric indicators, the impact factor. Only a more comprehensive analysis of the overall quantitative data will render the use of the SCI JCR more exhaustive. Without this more extensive analysis, the use of the SCI JCR leads to an over-simplified vision of the complex reality of the world of scientific journals. In this communication, we will present the SCI JCR briefly and explain the descriptive statistical analysis of the SCI JCR that we have undertaken. We used this method to divide the journals into four groups according to their indicator values and created a reference system which generates a synthetic image of the SCI JCR quantitative data. We studied various applications using this reference system, in particular INRA's scientific output for 1994.
In the majority of scientific fields, journals are still the main means of disseminating research findings. Yet during the process of evaluation, research is often assimilated to the journal in which it is published. This would explain why scientific and publishing communities who work with periodicals have imparted qualitative notions to these journals such as being "famous", "prestigious" and so on. This caricatured and over-simplified vision of journals often leads to their arbitrary division into "good" or "bad".
Of the various different methods used to evaluate journals (Reed-Scott, 1988), only one, based on citation analysis, has really imposed itself under the influence of an indicator, the impact factor. The impact factor was created twenty years ago by the Institute for Scientific Information (ISI) 1 (Garfield, 1972,1979) and is published annually along with a series of other indicators in what is commonly known as the JCR - the Journal Citation Reports. Two editions of the JCR exist : one called Social Sciences and the other Science (see annexe). The results presented in this communication concern the Science edition.
The Science edition of the JCR (SCI JCR) is a collection of figures about approximately 4500 of the 140000 journals published throughout the world 2. Let us take the example of journal A belonging to the 4500 selected journals. The SCI JCR provides the following information on this and the other 4499 journals :
- the number of articles published each year by journal A (source items),
- the number of citations received by the articles of journal A (total citations, citations to the two previous years),
- a series of other indicators calculated from the two above indicators: the impact factor, immediacy index, cited half-life and citing half-life 3.
The ISI constructs its indicators from the bibliographic references in articles. Every year the ISI extracts the title of the periodical cited and the year of publication of the article cited 4. Only the bibliographic references of articles from journals selected by the ISI 5 (so-called source or citing journals) are analysed. The 4500 journals listed in the SCI JCR are made up of these source journals and journals cited by the latter.
According to the ISI, the SCI JCR which is unique of its kind, makes it possible to rank, evaluate, categorise and compare journals (Garfield, 1996, p.5). However the reality is not so simple. When consulting the SCI JCR we are confronted with columns of figures representing indicator values (table 1).
Table 1 : Extract of the Journal rankings sorted by impact factor of the Journal Citation Reports on CD-ROM (Science ed.) (Copyright, ISI, 1996).
|2||*ANNU REV IMMUNOL||0732-0582||9088||49.509||3.333||24||4.4|
|3||*ANNU REV BIOCHEM||0066-4154||17825||44.414||3.167||30||7.4|
|5||*ABSTR PAP AM CHEM S||0065-7727||66||31.000||0.000||7|
|6||*ANNU REV CELL BIOL||0743-4634||5223||30.548||5.1|
|8||*ANNU REV NEUROSCI||0147-006X||5552||29.083||3.600||20||5.3|
|12||*NEW ENGL J MED||0028-4793||103033||22.412||4.913||413||6.4|
Table 2 : Extract of the subject category listing section of the Science Citation Index. Journal Citation Reports. Printed guide to the microfiche edition of the SCI JCR 1994 (Copyright, ISI, 1995). View larger version
We have calculated that the number of figures provided each year by the ISI in the SCI JCR amounts to some 6 million - the non-initiated are easily overwhelmed. Handling this data can be a laborious, boring and confusing task 6. Moreover, the information undergoes very little analysis. A biref analysis is presented, since ISI has arranged the indicators in ascending or descending order (table 1) and separating the journals into various scientific subject categories (table 2). Problems abound when trying to interpret the indicator values of the SCI JCR. For example, what meaning should we give to an isolated impact factor of 0.3 or 60 ? Our aim here is to synthesise this huge quantity of data by applying descriptive statistical methods.
An overview of the 4500 journals can be obtained by distributing journals into the variation interval of each indicator or frequency distribution. To portray this distribution, we have used a traditional descriptive statistical method, the box-plot. The construction of a box-plot is based on calculating quartiles (Q25, Q50 and Q75), the lower and upper adjacent values and the maxima and minima observed (Chambers et al., 1983). Its graphic representation outlines zones which we have defined as groups of journals (for an example of a box-plot, see figure 1). We labelled these groups LOW, CENTRAL, HIGH and EXTREME 7 :
- the LOW GROUP contains the journals whose indicators values were situated between the lower adjacent value (always equivalent to the minimum value of each indicator) and Q25. It is represented by an orange segment;
- the CENTRAL GROUP contains the journals whose indicators values were situated between Q25 and Q75. It is shown as a blue rectangle of a varying length. The median Q50 is represented by a horizontal line across this blue rectangle;
- the HIGH GROUP contains the journals whose indicators values were situated between Q75 and the upper adjacent value. It is represented by a green segment;
- the EXTREME GROUP contains the journals whose indicators values are equal or superior to the upper adjacent value. It is represented by red dotted line, the maximum observed is symbolised by an asterisk.
We determined these groups for all the journals in the SCI JCR (all subject categories combined) and by subject category. In this communication we will only present the results of the impact factor for 1994. Several different examples of applications were studied using these reference systems.
1) Constitution of groups of journals according to the impact factor
a) All subject categories combined (fig. 1)
In 1994, an impact factor was allocated to 4445 of the 4514 journals figuring in the SCI JCR. This impact factor varied from 0 to 57.778. The 4445 journals were distributed as follows (fig. 1) :
- the LOW GROUP contained 25% of the journals (n = 1111). In this group, the impact factor was equal to or lower than 0.294;
- the CENTRAL GROUP contained by definition, 50% of the journals (n = 2225). The impact factor of these journals ranged from 0.294 to 1.380. The median was 0.655;
- the HIGH GROUP contained 17% of the journals (n = 774), their impact factor ranged from 1.380 to 3.008;
- the EXTREME GROUP contained nearly 8% of the journals (n = 335). It combined all the journals whose impact factor ranged from 3.008 to 57.778. The 335 journals in this group occupied more than 90% of the impact factor variation interval. This group contained the fewest journals, yet had the widest range of indicator values and included journals that are qualified as "prestigious" and "renowned". In statistical terms, this group has a particular significance. Take, for example, the spread of different weights in a sample of animals, if some of these animals had an exceptionally high weight, they would almost certainly be placed in the extreme group. However, these outside values are far from being representative of the population under examination. In the same way, the 335 journals in the extreme group do not represent the journal population of the SCI JCR. We might ask ourselves if it is really judicious in all circumstances to use these journals as a reference system in the evaluation processes involving journals ?
|Figure 1. Journal groups determined from the box-plot of the impact factor of the 4445 journals of the SCI JCR 94 and the number of journals in these groups (%). View larger version|
The reference system we have set up in the form of an evaluation scale is more synthetic than the "4445" ranks given by ISI. It enabled us to section off the 8% of journals whose indicators were exceptionally high and thereby bring the other 92% of journals, whose existence was overshadowed, back into prominence. These reference systems were calculated for the twenty years since the SCI JCR was set up and for six of the SCI JCR indicators (total citations, citations to the two previous years, source items, immediacy index, impact factor, cited half-life) (Magri et al., 1995, Magri and Solari, 1996).
b) According to the scientific subject category of the journals (fig. 2)
In the subject category listing section of the SCI JCR, the ISI has separated the journals into subject categories since 1979 (176 subject categories in 1994). A journal can belong to 1, 2, 3 or 4 different subject categories 8. In each subject category, the ISI classes the journals in decreasing order of their impact factor and gives them a rank (table 2). Important differences exist between categories. For example in 1994, 29 journals could be found in the "Behavioural Science" category, whilst there were 193 in the "Biochemistry & Molecular Biology" category. Another example of the differences between subject categories is that the highest impact factor value for the behavioural science category was 3.79, whilst it was 42.169 for the "Biochemistry & Molecular Biology" category (table 2).
The box-plot method was used to construct reference systems by subject category in order to study these differences and make better use of the subject category section, all the subject categories listed in the SCI JCR since 1979 are being examined, (Solari and Magri, 1997). This study is still being carried out, but at least three families of subject categories can be outlined. We chose to illustrate these three families in fields of interest to INRA (fig. 2). This figure recalls the box-plot of the frequency distribution for all journals (all subject categories combined). The three other box-plots display the frequency distribution of the impact factor of the subject categories "Biochemistry & Molecular Biology", "Plant Sciences" and "Food Science & Technology". These box-plots cannot be superposed. In the "Biochemistry & Molecular Biology" category, the four journal groups described above can be found, but the group limits are higher than those of the total population of the SCI JCR journals. The second family of subject categories represented by "Plant Sciences", has group limits which are more or less equivalent to those of the total journal population. Finally, the third family represented by "Food Science & Technology", has group limits which are lower than those of the total population. Moreover, in this subject category, no extreme group exists.
|Figure 2. Construction of journal groups from the impact factor of all the journals in the SCI JCR 94 and journals from 3 scientific subject categories. Group in which the two journals A and B belong according to their impact factor. View larger version|
By introducing the notion of belonging to a journal group, these reference systems shed light on the differences between subject categories and make intra- and inter-journal subject category evaluation possible.
2) Examples of applications
a) Evaluation of journals listed in different subject categories (fig. 2)
Take as an example two studies published in two different journals. The first study is published in journal A whose impact factor is 3 and listed in the "Biochemistry & Molecular Biology" category. The second is published in journal B whose impact factor is 1.5 and belongs to the "Food Science & Technology" category. If we were to apply the reasoning that is currently practised with the impact factor, the first study published in journal A in the "Biochemistry & Molecular Biology" category would certainly be the better considered of the two as its impact factor is twice as high. Yet, in its subject category, journal A comes into the central group. On the other hand, journal B in which the second study has been published is one of best in the "Food Science & Technology" category since it belongs to the high group. It therefore becomes possible to compare the journals of the high group of "Food Science & Technology" whose impact factors are between 1 and 2, with the "Biochemistry & Molecular Biology" journals in the high group whose higher impact factors range from 3.5 to 7. In other words, dividing journals into groups make comparison between subject categories possible, thereby avoiding misleading interpretations when just the impact factor is consulted.
b) Evaluation of a journal listed in several subject categories (fig. 3)
Take for example the Japanese journal Bioscience, Biotechnology and Biochemistry. Published since 1924, this journal has an impact factor of 0.863 in 1994. The ISI classes this journal in four subject categories including "Biochemistry & Molecular Biology" and "Food Science & Technology". In the overall population, i.e. the 4445 journals, this journal belongs to the central group. It belongs to the low group in the "Biochemistry & Molecular Biology" category, and is at the boundary of the central and high groups in the "Food Science & Technology" category, where no extreme group exists. If the journal is only examined in its "Biochemistry & Molecular Biology" context, judgement is harsh.
|Figure 3. Position of the journal Bioscience, Biotechnology and Biochemistry within the population of 4445 journals and within the subject categories population from its impact factor. View larger version|
c) Study of INRA production in 1994 (fig. 4)
These reference systems were used to study publication activity at INRA in 1994 9 (Magri et al., 1997). This communication will only present analysis on the impact factor. INRA's written production for 1994 in the Institute's database of researcher's publications, comes to 6303 10. We estimated that 28% of INRA's production figures in the journals of the SCI JCR. This figure of 28% represents 1740 articles published in 486 different journals.
Figure 4. Determination
of the group to which INRA articles belong by the impact factor
of the journal in which they are published :
a - distribution of the 1740 INRA 1994 articles in all the SCI JCR journals (in %);
b - distribution of the 212 INRA 1994 articles in the "Biochemistry & Molecular Biology" subject category (in %);
c - distribution of the 237 INRA 1994 articles in the "Plant Sciences" subject category (in %). View larger version
We started by examining how the 1740 articles were spread over the reference system for the whole journal population, all subject categories combined (fig. 4a). We observed that little is published in low group journals (4%). Half of INRA's production is published in central group journals and the other half in the high and extreme groups, with a substantial percentage, (17%) in the extreme group.
This same production will now be analysed within the SCI JCR scientific subject categories (figs 4b and 4c). INRA is present in various ways in 95 of the 176 subject categories defined by the ISI. We will only present the results of two of those subject categories, "Biochemistry & Molecular Biology" and "Plant Sciences" in which the highest number of articles are published.
This brief look at INRA production in the 1994 SCI JCR, shows that our reference systems make it possible to see "at a glance" the structure needing evaluation within the ISI indicator system.
The aim of our study was not to touch on the methodology adopted by the ISI and how citations are counted, as they have been dealt with widely in the literature (e.g. Courtial, 1990; Callon et al., 1993). However, we would like to point out that the SCI JCR characterises the container (the journal) rather than the contents (the article and its intrinsic qualities). Nevertheless, the SCI JCR is implicitly recognised as a system of evaluation by many scientific communities. It is true that it is the only tool of its kind and for the last 20 years has continuously and methodically gathered information on approximately 4500 scientific journals supposedly covering all subject categories.
Today, very high impact factor journals are taken as references. This use which is essentially based on isolated impact factor values has no doubt led to the over- or under-estimation of journals and in consequence of research findings and laboratories. Our rereading of the data based on the notion of belonging to a group of journals enables fuller more nuanced judgements to be made on journals. Our reference systems could be used in processes of evaluation that are based on scientific journals.
This study has been carried out with the technical collaboration of Mme M. Désiré. We also wish to thank Mr D. Marie for his constructive comments. This article was translated by Kathi Archbold of the Translation Department at INRA.
CHAMBERS J.M., CLEVELAND W.S., KLEINER B., TUKEY P.A., 1983. Graphical methods for data analysis, Wadsworth & Brooks, California, 386 p.
COURTIAL J-P., 1990. Introduction à la scientométrie : de la bibliométrie à la veille technologique. Anthropos- Economica, Paris, 126 p.
CALLON M., COURTIAL J-P., PENAN H., 1993. La scientométrie. Que sais-je? n°2727. Presses Universitaires de France, Paris, 126 p.
GARFIELD E., 1972. Citation analysis as a tool in journal evaluation. Science, 78 (4060) 471-479.
GARFIELD E., 1979. Citation indexing - its theory and application in science, technology, and humanities. John Wiley & Sons, New York, 274 p.
GARFIELD E., 1990. How ISI selects journals for coverage : quantitative and qualitative considerations. Current Contents. Life Sciences, 33 (22) 5-13.
GARFIELD E., 1995. A bibliometric analysis of science journals in the ISI database. Science Citation Index. Journal Citation Reports. Printed guide to the microfiche edition of the SCI JCR 1994, 142p.
GARFIELD E., 1996. A bibliometric analysis of science journals in the ISI database. Science Citation Index. Journal Citation Reports. Printed guide to the microfiche edition of the SCI JCR 1995, 147p.
GARFIELD E., 1996. A bibliometric analysis of science journals in the ISI database. Journal Citation Reports on CD-ROM (Science ed).
MAGRI M.-H., SOLARI A., RÉRAT K., 1995. Le SCI Journal Citation Reports ou comment positionner une revue dans l'environnement ISI. Journées d'étude sur les systèmes d'information élaborée, 30 Mai - 2 Juin, Ile Rousse, session poster.
MAGRI M.-H., SOLARI A., 1996. The SCI Journal Citation Reports : a potential tool for studying journals? I Description of the JCR journal population based on the number of citations received, number of source items, impact factor, immediacy index and cited half-life. Scientometrics, 35 (1) 93-117.
MAGRI M.-H., SOLARI A., MIQUEL F., OBUKO Y., 1997. La production de l'INRA dans le système d'évaluation de l'ISI (en préparation)
REED-SCOTT J., 1988. Guide de l'inventaire des collections des bibliothèques de recherches en Amérique du Nord. Association of research libraries, office of management services, S.l., 68p.
SOLARI A., MAGRI M.-H., 1997. The SCI Journal Citation Reports : a potential tool for studying journals? II Description by subject category of the JCR journal population based on the impact factor (en préparation).
1) Editions and formats of the Journal Citation Reports (JCR)
The publisher of this tool is the Institute for Scientific Information 11. Today, two editions of the JCR exist, a Science edition and a Social Sciences edition. Both these editions are available on CD-ROM, microfiche and partially on paper.
- From 1975 to 1988, a paper version of the SCI Journal Citation Reports was published with cumulative annual volumes from the Science Citation Index database. The same holds for the SSCI Journal Citation Reports published with the Social Science Citation Index from 1977 to 1988.
- From 1989, the Science Citation Index, Journal Citation Reports (microfiche edition) and the Social Science Citation Index were published on microfiches. A partial printed version accompanied the following microfiche editions: The Science Citation Index, Journal Citation Reports. Printed guide to the microfiche edition of the SCI JCR and the Social Science Citation Index, Journal Citation Reports. Printed guide to the microfiche edition of the SSCI JCR.
- As of 1994 and on CD-ROM, the Journal Citation Reports on CD-ROM (Science ed.) and the Journal Citation Reports on CD-ROM (Social Sciences ed.).
These serial publications are published annually in the last quarter of the current year. The indicators provide information on the previous year : for example, the JCR of December 1996 is based on the 1995 indicators.
2) Definitions of the indicators used in the Journal Citation Reports"
These definitions 12 are a quick reminder of the terms and bibliometric indicators created by the ISI (Garfield, 1996, p.10, 11, 14) :
When one document (A) mentions or refers to another document (B), the latter has been cited by the former as a source of information, as support for a point of view, as authority for a statement of fact, etc. The term "citation" is used to indicate not only the fact that document B has been cited in a reference of document A, but also the description of document B contained in the reference. In this sense, citation and reference are frequently used interchangeably.
citations to the two previous years
...the portion of the total citations that were received by articles that the journal published in 1994 and the portion of the total citations that were received by articles that the journal published in 1993 ...
The number of journal publication years going back from the current year which accounts for 50% of the total citations received by the cited journal in the current year.
The number of journal publication years going back from the current year which accounts for 50% of the total citations given by the citing journal in the current year.
...The immediacy index of journal X would be calculated by dividing the number of all current citations of current source items published in journal X by the total number of articles journal X published that year...
... The JCR impact factor is basically a ratio between citations and recent citable items published. Thus, the impact factor of Journal X would be calculated by dividing the number of all current citations of source items published in Journal X during the two previous years by the number of articles Journal X published in those two years...
Source item generally refers to an item published in any journals processed for coverage in a citation index. In the JCR, however, only original research articles, review articles are counted for all listed journals.
This is a journal that is covered in the SCI, SSCI or A&HCI, so called because it is the source of published items processed for compilation of the four sections of the citation indexes. In the JCR a source journal is a citing journal.
...the total number of times the journal was cited by ISI source items during 1995 ...
1 The ISI is a private company specialising in the supply of information and is well-known in research laboratories due to one of its products : the Current Contents.
2 We obtained this estimation by adding together the number of current periodicals indexed in the ISSN register using classes 5 et/ou 6 of the Universal Decimal or Dewey Classification (ISSN Compact, August 96).
3 A definition of these indicators can be found in the annexe.
4 Which represents about 12 million bibliographic references that are sorted each year by the ISI (Garfield, 1996 p.7).
5 In our opinion, the information published on the selection criteria for the source journal (Garfield, 1990) lack precision.
6 This job which was particularly tricky using microfiches, has improved since the publication of the CD-ROM version.
7 For a more detailed description of the box-plot methods as applied to the SCI JCR journals see Magri et Solari, 1996.
8 To our knowledge, precise criteria for the division of journals into subject category have not been published. This classification, is more often than not, the only section of the SCI JCR that scientists and a large number of those working with the information sciences are aware of. This is the list that is most frequently seen in laboratories and evaluation committees.
9 This study is the result of collaboration with the laboratoire Stratégie & technologie de l'Ecole centrale de Paris.
10 This number corresponds to the different types of documents produced by the Institute's researchers : periodicals articles, monographs, chapters, theses, posters, patents, maps, films, etc. (consultation of the PUBINRA database in October 1996).
11 Institute for Scientific Information, Inc. 3501 Market Street Philadelphia, Pa. 19104 USA. In Europe the ISI can be found at : ISI Europe, Brunel Science Park. Uxbridge UB8 3PQ, Great-Britain.
12 When dates are given in the definitions, the examples given correspond to indicator calculations for 1995 published in 1996.