University of Sussex
Browse
acl2016.pdf (197.99 kB)

A critique of word similarity as a method for evaluating distributional semantic models

Download (197.99 kB)
conference contribution
posted on 2023-06-09, 02:11 authored by Miroslav Batchkarov, Thomas Kober, Jeremy ReffinJeremy Reffin, Julie WeedsJulie Weeds, David WeirDavid Weir
This paper aims to re-think the role of the word similarity task in distributional semantics research. We argue while it is a valuable tool, it should be used with care because it provides only an approximate measure of the quality of a distributional model. Word similarity evaluations assume there exists a single notion of similarity that is independent of a particular application. Further, the small size and low inter-annotator agreement of existing data sets makes it challenging to find significant differences between models.

Funding

A Unified Model of Compositional and Distributional Semantics: Theory and Applications; G0853; EPSRC-ENGINEERING & PHYSICAL SCIENCES RESEARCH COUNCIL; EP/IO37458/1

History

Publication status

  • Published

File Version

  • Accepted version

Journal

Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP

Publisher

Association for Computational Linguistics

Page range

7-12

Event name

The 1st Workshop on Evaluating Vector Space Representations for NLP

Event location

Berlin

Event type

conference

Event date

12th August 2016

Place of publication

Berlin, Germany

Department affiliated with

  • Informatics Publications

Full text available

  • Yes

Peer reviewed?

  • Yes

Legacy Posted Date

2016-07-19

First Open Access (FOA) Date

2016-07-19

First Compliant Deposit (FCD) Date

2016-07-19

Usage metrics

    University of Sussex (Publications)

    Categories

    No categories selected

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC