External validity of multi-criteria preference data obtained from non-random sampling: measuring cohesiveness within and between groups

An important component of multi-criteria decision analysis (MCDA) in the public sector is the elicitation and aggregation of preferences data collected via surveys into the relative importance of the criteria for the decision at hand. These aggregated preferences data, usually in the form of mean weights on the criteria, are intended to represent the preferences of the relevant population overall. However, random sampling is often not feasible for public-sector MCDA for logistical reasons, including the expense involved in identifying and recruiting participants. Instead, non-random sampling methods such as convenience, purposive or snowball sampling are widely used. Nonetheless, provided the preferences data collected are sufficiently ‘cohesive’ in terms of the extent to which the weights of the individuals belonging to the various exogenously defined groups in the sample are similar, non-random sampling can still produce externally valid aggregate preferences data. We explain a method for measuring cohesiveness using the Kemeny and Hellinger distance measures, which involve measuring the ‘distance’ of participants’ weights (and the corresponding rankings of the criteria) from each other, within and between the groups respectively. As an illustration, these distance measures are applied to data from a MCDA to rank non-communicable diseases according to their overall burden to society. We conclude that the method is useful for evaluating the external validity of preferences data obtained from non-random sampling.