Should the impact factor of the year of publication or the last available one be used when evaluating scientists?
Abstract
Aim of study: A common procedure when evaluating scientists is considering the journal’s quartile of impact factors (within a category), many times considering the quartile in the year of publication instead of the last available ranking. We tested whether the extra work involved in considering the quartiles of each particular year is justified
Area of study: Europe
Material and methods: we retrieved information from all papers published in 2008-2012 by researchers of AGROTECNIO, a centre focused in a range of agri-food subjects. Then, we validated the results observed for AGROTECNIO against five other European independent research centres: Technical University of Madrid (UPM) and the Universities of Nottingham (UK), Copenhagen (Denmark), Helsinki (Finland), and Bologna (Italy).
Main results: The relationship between the actual impact of the papers and the impact factor quartile of a journal within its category was not clear, although for evaluations based on recently published papers there might not be much better indicators. We found unnecessary to determine the rank of the journal for the year of publication as the outcome of the evaluation using the last available rank was virtually the same.
Research highlights: We confirmed that the journal quality reflects only vaguely the quality of the papers, and reported for the first time evidences that using the journal rank from the particular year that papers were published represents an unnecessary effort and therefore evaluation should be done simply considering the last available rank.Downloads
References
Abramo G, D'Angelo CA, Di Costa F, 2010. Citations versus journal impact factor as proxy of quality: Could the latter ever be preferable? Scientometrics 84: 821-833. https://doi.org/10.1007/s11192-010-0200-1
Abramo G, D'Angelo CA, Felici G, 2019. Predicting publication long-term impact through a combination of early citations and journal impact factor. J Informetrics 1: 32-49. https://doi.org/10.1016/j.joi.2018.11.003
Albarrán P, Crespo J, Ortuño I, Ruiz-Castillo J, 2011. The skewness of science in 219 sub-fields and a number of aggregates. Scientometrics 88: 385-397. https://doi.org/10.1007/s11192-011-0407-9
Ancaiani A, Anfossi AF, Barbara A, Benedetto S, Blasi B, Carletti V, Cicero T, Ciolfi A, Costa F, Colizza G, et al. 2015. Evaluating scientific research in Italy: The 2004-10 research evaluation exercise. Res Eval 24: 242-255. https://doi.org/10.1093/reseval/rvv008
Bartneck C, Kokkelmans S, 2011. Detecting h-index manipulation through self-citation analysis. Scientometrics 87: 85-98. https://doi.org/10.1007/s11192-010-0306-5
Baveye PC, Trevors JT, 2011. How can we encourage peer-reviewing? Water Air Soil Pollut 214: 1-3. https://doi.org/10.1007/s11270-010-0355-7
Bornmann L, Leydesdorff L, 2017. Skewness of citation impact data and covariates of citation distributions: A large-scale empirical analysis based on Web of Science data. J Informetrics 11: 164-175. https://doi.org/10.1016/j.joi.2016.12.001
Bornmann L, Leydesdorff L, Wang J, 2013. Which percentile-based approach should be preferred for calculating normalized citation impact values? An empirical comparison of five approaches including a newly developed citation-rank approach (p100). J Informetrics 7: 933-944. https://doi.org/10.1016/j.joi.2013.09.003
Bradford SC, 1934. Sources of information on specific subjects. Engineering 137: 85-86.
Butler L, 2002. A list of published papers is no measure of value. Nature 419: 877. https://doi.org/10.1038/419877a
Chavarro D, Ràfols I, Tang P, 2018. To what extent is inclusion in the Web of Science an indicator of journal 'quality'? Res Eval 27: 106-118. https://doi.org/10.1093/reseval/rvy001
Cooke S, Donaldson M, 2014. Self-citation by researchers: Narcissism or an inevitable outcome of a cohesive and sustained research program? Ideas Ecol Evol 7: 1-2. https://doi.org/10.4033/iee2014.7.1.e
Didegah F, Thelwall M, 2013. Determinants of research citation impact in nanoscience and nanotechnology. J Am Soc Inform Sci Technol 64: 1055-1064. https://doi.org/10.1002/asi.22806
Egghe L, 2011. A disadvantage of h-type indices for comparing the citation impact of two researchers. Res Eval 20: 341-346. https://doi.org/10.3152/095820211X13164389670356
Finardi U, 2013. Correlation between journal impact factor and citation performance: an experimental study. J Informetrics 7: 357-370. https://doi.org/10.1016/j.joi.2012.12.004
Fox CW, 2017. Difficulty of recruiting reviewers predicts review scores and editorial decisions at six journals of ecology and evolution. Scientometrics 113: 465-477. https://doi.org/10.1007/s11192-017-2489-5
Frank M, 2003. IFs: arbiter of excellence? J Medical Library Assoc 91: 4-6.
Hirsch JE, 2005. An index to quantify an individual's scientific research output. P Nat Acad Sci USA 102: 16569-16572. https://doi.org/10.1073/pnas.0507655102
Huang DW, 2016. Positive correlation between quality and quantity in academic journals, J Informetrics 10: 329-335. https://doi.org/10.1016/j.joi.2016.02.002
Ioannidis JPA, 2015. A generalized view of self-citation: Direct, co-author, collaborative, and coercive induced self-citation. J Psychosomatic Res 78: 7-11. https://doi.org/10.1016/j.jpsychores.2014.11.008
Kacem A, Flatt JW, Mayr P, 2020. Traking self-citations in academic publishing. Scientometrics 123: 1157-1165. https://doi.org/10.1007/s11192-020-03413-9
Kreiman G, Maunsell JH, 2011. Nine criteria for a measure of scientific output. Front Comput Neurosci 5: 48. https://doi.org/10.3389/fncom.2011.00048
Langfeldt L, Bloch C, Sivertsen G, 2015. Options and limitations in measuring the impact of research grants - evidence from Denmark and Norway. Res Eval 24: 256-270. https://doi.org/10.1093/reseval/rvv012
Levitt JM, Thelwall M, 2008. Patterns of annual citation of highly cited articles and the prediction of their citation ranking: A comparison across subjects. Scientometrics 77: 41-60. https://doi.org/10.1007/s11192-007-1946-y
Leydesdorff L, 2008. Caveats for the use of citation indicators in research and journal evaluations. J Am Soc Inform Sci Technol 59: 278-287. https://doi.org/10.1002/asi.20743
Liu XL, Gai SS, Zhang SL, Wang P, 2015. An analysis of peer-reviewed scores and impact factors with different citation time windows: A case study of 28 ophthalmologic journals. PLoS ONE 10 (8): e0135583. https://doi.org/10.1371/journal.pone.0135583
Logan JM, Bean SB, Myers AE, 2017. Author contributions to ecological publications: What does it mean to be an author in modern ecological research? PLoS ONE 12 (6): e0179956. https://doi.org/10.1371/journal.pone.0179956
Moed HF, 2008. UK research assessment exercises: Informed judgments on research quality or quantity? Scientometrics 74: 153-161. https://doi.org/10.1007/s11192-008-0108-1
Mutz R, Daniel HD, 2012. Skewed citation distribution and bias factor: Solutions to two core problems with the journal impact factor. J Informetrics 6: 169-176. https://doi.org/10.1016/j.joi.2011.12.006
Owlia P, Vasei M, Goliaei B, Nassiri I, 2011. Normalized impact factor (NIF): An adjusted method for calculating the citation rate of biomedical journals. J Biomed Informatics 44: 216-220. https://doi.org/10.1016/j.jbi.2010.11.002
Pajić D, 2015. On the stability of citation-based journal rankings. J Informetrics 9: 990-1006. https://doi.org/10.1016/j.joi.2015.08.005
Rajasekaran S, Shan RLP, Finnoff JT, 2014. Honorary authorship: Frequency and associated factors in physical medicine and rehabilitation research articles. Archiv Phys Med Rehabil 95: 418-428. https://doi.org/10.1016/j.apmr.2013.09.024
Sahel JA, 2011. Quality versus quantity: Assessing individual research performance. Sci Transl Med 3: 84cm13. https://doi.org/10.1126/scitranslmed.3002249
Seglen PO, 1992. The skewness of science. J Am Soc Inform Sci 43: 628-638. https://doi.org/10.1002/(SICI)1097-4571(199210)43:9<628::AID-ASI5>3.0.CO;2-0
Seglen PO, 1997. Why the IF of journals should not be used for evaluating research. Brit Med J 314: 497-502. https://doi.org/10.1136/bmj.314.7079.497
Slafer GA, 2005. Multiple authorship of crop science papers: are there too many co-authors? Field Crops Res 94: 272-276. https://doi.org/10.1016/j.fcr.2004.11.011
Slafer GA, 2008. Should crop scientists consider a journal's impact factor in deciding where to publish? Eur J Agron 29: 208-212. https://doi.org/10.1016/j.eja.2008.07.001
Stegehuis C, Litvak N, Waltman L, 2015. Predicting the long-term citation impact of recent publications. J Informetrics 9: 642-657. https://doi.org/10.1016/j.joi.2015.06.005
van Raan AFJ, 2013. Universities scale like cities. PLoS ONE 8: e59384. https://doi.org/10.1371/journal.pone.0059384
Vanclay JK, 2013. Factors affecting citation rates in environmental science. J Informetrics 7: 265-271. https://doi.org/10.1016/j.joi.2012.11.009
Waltman L, 2016. A review of the literature on citation impact indicators. J Informetrics 10: 365-391. https://doi.org/10.1016/j.joi.2016.02.007
© CSIC. Manuscripts published in both the printed and online versions of this Journal are the property of Consejo Superior de Investigaciones Científicas, and quoting this source is a requirement for any partial or full reproduction.
All contents of this electronic edition, except where otherwise noted, are distributed under a “Creative Commons Attribution 4.0 International” (CC BY 4.0) License. You may read here the basic information and the legal text of the license. The indication of the CC BY 4.0 License must be expressly stated in this way when necessary.
Self-archiving in repositories, personal webpages or similar, of any version other than the published by the Editor, is not allowed.
