University of Sussex
Browse
Sharmanska_Ambiguity_Helps_Classification_CVPR_2016_paper.pdf (1.59 MB)

Ambiguity helps: classification with disagreements in crowdsourced annotations

Download (1.59 MB)
conference contribution
posted on 2023-06-09, 00:55 authored by Viktoriia SharmanskaViktoriia Sharmanska, Daniel Hernández-Lobato, Jose Miguel Hernández-Lobato, Novi QuadriantoNovi Quadrianto
Imagine we show an image to a person and ask her/him to decide whether the scene in the image is warm or not warm, and whether it is easy or not to spot a squirrel in the image. For exactly the same image, the answers to those questions are likely to differ from person to person. This is because the task is inherently ambiguous. Such an ambiguous, therefore challenging, task is pushing the boundary of computer vision in showing what can and can not be learned from visual data. Crowdsourcing has been invaluable for collecting annotations. This is particularly so for a task that goes beyond a clear-cut dichotomy as multiple human judgments per image are needed to reach a consensus. This paper makes conceptual and technical contributions. On the conceptual side, we define disagreements among annotators as privileged information about the data instance. On the technical side, we propose a framework to incorporate annotation disagreements into the classifiers. The proposed framework is simple, relatively fast, and outperforms classifiers that do not take into account the disagreements, especially if tested on high confidence annotations.

History

Publication status

  • Published

File Version

  • Accepted version

Journal

Proceedings 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016); Las Vegas, Nevada; 26 June - 1 July 2016

ISSN

1063-6919

Publisher

Institute of Electrical and Electronics Engineers

Page range

2194-2202

ISBN

9781467388504

Department affiliated with

  • Informatics Publications

Full text available

  • Yes

Peer reviewed?

  • Yes

Legacy Posted Date

2016-04-18

First Open Access (FOA) Date

2016-09-23

First Compliant Deposit (FCD) Date

2016-04-18

Usage metrics

    University of Sussex (Publications)

    Categories

    No categories selected

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC