A competency-based mixed-methods approach to evaluating a leadership development intervention in the NHS

Evaluating the success of interventions in applied organisational settings poses a number of problems for work and organisational psychologists. It is increasingly recognised that random control trials are practically difficult to implement in real-world intervention designs. Our aim is to demonstrate that intervention evaluation research can still be rigorous and informative, even when applied organisational interventions are limited in their design. In this study, an evaluation of a 9-month leadership development programme, with 41 senior NHS commissioners, was conducted. However, with non-random allocation of participants, and no control comparison group available, we developed a competency-based, mixed-methods approach to evaluate the programme’s success. This served to overcome issues traditionally associated with within-person intervention evaluations, whereby changes occurring from the pre to the post-intervention period cannot confidently be attributed to the intervention itself (Shadish, Cook & Campbell, 2002). Our study offers a thorough and novel approach by: (1) applying a repeated-measures design to capture information on intervention content and style of delivery at each of the 7 sessions that formed the leadership development programme; (2) operating a mixed-method design, incorporating structured observations, self-report competency assessments and evaluations, other-report competency assessments, and objective data about line-manager support; and, (3) taking a longitudinal approach (months, rather than days or weeks) to measuring intervention impact, rarely applied when using momentary-based repeated measures designs. Data will be analysed using multi-level random coefficient modelling, (by ML-Win), and will be available by EAWOP 2017. Implications for evaluation research, specifically from a socio-technical systems perspective are discussed.