University of Sussex
Browse
EWPB13.pdf (485.26 kB)

The assessment of science: the relative merits of post- publication review, the impact factor, and the number of citations

Download (485.26 kB)
journal contribution
posted on 2023-06-08, 16:02 authored by Adam Eyre-WalkerAdam Eyre-Walker, Nina Stoletzki
The assessment of scientific publications is an integral part of the scientific process. Here we investigate three methods of assessing the merit of a scientific paper: subjective post-publication peer review, the number of citations gained by a paper, and the impact factor of the journal in which the article was published. We investigate these methods using two datasets in which subjective post-publication assessments of scientific publications have been made by experts. We find that there are moderate, but statistically significant, correlations between assessor scores, when two assessors have rated the same paper, and between assessor score and the number of citations a paper accrues. However, we show that assessor score depends strongly on the journal in which the paper is published, and that assessors tend to over-rate papers published in journals with high impact factors. If we control for this bias, we find that the correlation between assessor scores and between assessor score and the number of citations is weak, suggesting that scientists have little ability to judge either the intrinsic merit of a paper or its likely impact. We also show that the number of citations a paper receives is an extremely error-prone measure of scientific merit. Finally, we argue that the impact factor is likely to be a poor measure of merit, since it depends on subjective assessment. We conclude that the three measures of scientific merit considered here are poor; in particular subjective assessments are an error-prone, biased, and expensive method by which to assess merit. We argue that the impact factor may be the most satisfactory of the methods we have considered, since it is a form of pre-publication review. However, we emphasise that it is likely to be a very error-prone measure of merit that is qualitative, not quantitative.

History

Publication status

  • Published

File Version

  • Published version

Journal

PLoS Biology

ISSN

1544-9173

Publisher

Public Library of Science

Issue

10

Volume

11

Article number

e1001675

Department affiliated with

  • Biology and Environmental Science Publications

Full text available

  • Yes

Peer reviewed?

  • Yes

Legacy Posted Date

2014-01-09

First Open Access (FOA) Date

2014-01-09

First Compliant Deposit (FCD) Date

2013-10-10

Usage metrics

    University of Sussex (Publications)

    Categories

    No categories selected

    Licence

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC