Sharmanska_Ambiguity_Helps_Classification_CVPR_2016_paper.pdf (1.59 MB)
Ambiguity helps: classification with disagreements in crowdsourced annotations
conference contribution
posted on 2023-06-09, 00:55 authored by Viktoriia SharmanskaViktoriia Sharmanska, Daniel Hernández-Lobato, Jose Miguel Hernández-Lobato, Novi QuadriantoNovi QuadriantoImagine we show an image to a person and ask her/him to decide whether the scene in the image is warm or not warm, and whether it is easy or not to spot a squirrel in the image. For exactly the same image, the answers to those questions are likely to differ from person to person. This is because the task is inherently ambiguous. Such an ambiguous, therefore challenging, task is pushing the boundary of computer vision in showing what can and can not be learned from visual data. Crowdsourcing has been invaluable for collecting annotations. This is particularly so for a task that goes beyond a clear-cut dichotomy as multiple human judgments per image are needed to reach a consensus. This paper makes conceptual and technical contributions. On the conceptual side, we define disagreements among annotators as privileged information about the data instance. On the technical side, we propose a framework to incorporate annotation disagreements into the classifiers. The proposed framework is simple, relatively fast, and outperforms classifiers that do not take into account the disagreements, especially if tested on high confidence annotations.
History
Publication status
- Published
File Version
- Accepted version
Journal
Proceedings 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016); Las Vegas, Nevada; 26 June - 1 July 2016ISSN
1063-6919Publisher
Institute of Electrical and Electronics EngineersExternal DOI
Page range
2194-2202ISBN
9781467388504Department affiliated with
- Informatics Publications
Full text available
- Yes
Peer reviewed?
- Yes
Legacy Posted Date
2016-04-18First Open Access (FOA) Date
2016-09-23First Compliant Deposit (FCD) Date
2016-04-18Usage metrics
Categories
No categories selectedKeywords
Licence
Exports
RefWorks
BibTeX
Ref. manager
Endnote
DataCite
NLM
DC