2022.coling-1.359.pdf (3.06 MB)
Testing large language models on compositionality and inference with phrase-level adjective-noun entailment
conference contribution
posted on 2023-06-10, 04:54 authored by Lorenzo Scott Bertolini, Julie WeedsJulie Weeds, David WeirDavid WeirPrevious work has demonstrated that pre-trained large language models (LLM) acquire knowledge during pre-training which enables reasoning over relationships between words (e.g, hyponymy) and more complex inferences over larger units of meaning such as sentences. Here, we investigate whether lexical entailment (LE, i.e. hyponymy or the is a relation between words) can be generalised in a compositional manner. Accordingly, we introduce PLANE (Phrase-Level Adjective-Noun Entailment), a new benchmark to test models on fine-grained compositional entailment using adjective-noun phrases. Our experiments show that knowledge extracted via In–Context and transfer learning is not enough to solve PLANE. However, a LLM trained on PLANE can generalise well to out–of–distribution sets, since the required knowledge can be stored in the representations of subwords (SW) tokens.
History
Publication status
- Published
File Version
- Published version
Journal
Proceedings of the 29th International Conference on Computational LinguisticsPublisher
International Committee on Computational LinguisticsPublisher URL
Page range
4084-4100Event name
The 29th International Conference on Computational Linguistics (COLING)Event location
Gyeongju, Republic of KoreaEvent type
conferenceEvent date
October 12-17, 2022Place of publication
Gyeongju, Republic of KoreaSeries
COLING'2022Department affiliated with
- Informatics Publications
Full text available
- Yes
Peer reviewed?
- Yes
Legacy Posted Date
2022-09-29First Open Access (FOA) Date
2022-10-19First Compliant Deposit (FCD) Date
2022-09-29Usage metrics
Categories
No categories selectedKeywords
Licence
Exports
RefWorks
BibTeX
Ref. manager
Endnote
DataCite
NLM
DC