Does changing examiner stations during UK postgraduate surgery objective structured clinical examinations influence examination reliability and candidates' scores?

Brennan, Peter A, Croke, David T, Reed, Malcolm, Smith, Lee, Munro, Euan, Foulkes, John and Arnett, Richard (2016) Does changing examiner stations during UK postgraduate surgery objective structured clinical examinations influence examination reliability and candidates' scores? Journal of Surgical Education, 73 (4). pp. 616-623. ISSN 1931-7204

[img] PDF - Accepted Version
Download (863kB)

Abstract

Objective

Objective structured clinical examinations (OSCE) are widely used for summative assessment in surgery. Despite standardizing these as much as possible, variation, including examiner scoring, can occur which may affect reliability. In study of a high-stakes UK postgraduate surgical OSCE, we investigated whether examiners changing stations once during a long examining day affected marking, reliability, and overall candidates’ scores compared with examiners who examined the same scenario all day.

Design, Setting, and Participants

An observational study of 18,262 examiner-candidate interactions from the UK Membership of the Royal College of Surgeons examination was carried at 3 Surgical Colleges across the United Kingdom. Scores between examiners were compared using analysis of variance. Examination reliability was assessed with Cronbach’s alpha, and the comparative distribution of total candidates’ scores for each day was evaluated using t-tests of unit-weighted z scores.

Results

A significant difference was found in absolute scores differences awarded in the morning and afternoon sessions between examiners who changed stations at lunchtime and those who did not (p < 0.001). No significant differences were found for the main effects of either broad content area (p = 0.290) or station content area (p = 0.450). The reliability of each day was not affected by examiner switching (p = 0.280). Overall, no difference was found in z-score distribution of total candidate scores and categories of examiner switching.

Conclusions

This large study has found that although the range of marks awarded varied when examiners change OSCE stations, examination reliability and the likely candidate outcome were not affected. These results may have implications for examination design and examiner experience in surgical OSCEs and beyond.

Item Type: Article
Schools and Departments: Brighton and Sussex Medical School > Brighton and Sussex Medical School
Subjects: R Medicine > R Medicine (General)
R Medicine > R Medicine (General) > R735 Medical education
R Medicine > RD Surgery
Depositing User: Esme Acton-Stewart
Date Deposited: 22 Apr 2016 15:06
Last Modified: 08 Jul 2017 06:47
URI: http://sro.sussex.ac.uk/id/eprint/60522

View download statistics for this item

📧 Request an update