Ambiguity helps: classification with disagreements in crowdsourced annotations

Sharmanska, Viktoriia, Hernández-Lobato, Daniel, Hernández-Lobato, Jose Miguel and Quadrianto, Novi (2016) Ambiguity helps: classification with disagreements in crowdsourced annotations. Published in: Proceedings 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016); Las Vegas, Nevada; 26 June - 1 July 2016. 2194-2202. Institute of Electrical and Electronics Engineers ISSN 1063-6919 ISBN 9781467388504

[img] PDF - Accepted Version
Download (1MB)


Imagine we show an image to a person and ask her/him to decide whether the scene in the image is warm or not warm, and whether it is easy or not to spot a squirrel in the image. For exactly the same image, the answers to those questions are likely to differ from person to person. This is because the task is inherently ambiguous. Such an ambiguous, therefore challenging, task is pushing the boundary of computer vision in showing what can and can not be learned from visual data. Crowdsourcing has been invaluable for collecting annotations. This is particularly so for a task that goes beyond a clear-cut dichotomy as multiple human judgments per image are needed to reach a consensus. This paper makes conceptual and technical contributions. On the conceptual side, we define disagreements among annotators as privileged information about the data instance. On the technical side, we propose a framework to incorporate annotation disagreements into the classifiers. The proposed framework is simple, relatively fast, and outperforms classifiers that do not take into account the disagreements, especially if tested on high confidence annotations.

Item Type: Conference Proceedings
Schools and Departments: School of Engineering and Informatics > Informatics
Subjects: Q Science > QA Mathematics > QA0276 Mathematical statistics
Related URLs:
Depositing User: Novi Quadrianto
Date Deposited: 18 Apr 2016 09:28
Last Modified: 16 Jun 2017 10:20

View download statistics for this item

📧 Request an update