2021.repl4nlp-1.7.pdf (288.64 kB)
Structure-aware sentence encoder in Bert-based siamese network
conference contribution
posted on 2023-06-10, 01:22 authored by Qiwei Peng, David WeirDavid Weir, Julie WeedsJulie WeedsRecently, impressive performance on various natural language understanding tasks has been achieved by explicitly incorporating syntax and semantic information into pre-trained models, such as BERT and RoBERTa. However, this approach depends on problem-specific fine-tuning, and as widely noted, BERT-like models exhibit weak performance, and are inefficient, when applied to unsupervised similarity comparison tasks. Sentence-BERT (SBERT) has been proposed as a general-purpose sentence embedding method, suited to both similarity comparison and downstream tasks. In this work, we show that by incorporating structural information into SBERT, the resulting model outperforms SBERT and previous general sentence encoders on unsupervised semantic textual similarity (STS) datasets and transfer classification tasks.
History
Publication status
- Published
File Version
- Published version
Journal
Proceedings of the 6th Workshop on Representation Learning for NLPPublisher
Association for Computational LinguisticsExternal DOI
Page range
57-63Event name
Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)Event location
OnlineEvent type
conferenceEvent date
August 6, 2021Place of publication
Bangkok, ThailandISBN
9781954085725Department affiliated with
- Informatics Publications
Full text available
- Yes
Peer reviewed?
- Yes
Legacy Posted Date
2021-10-12First Open Access (FOA) Date
2021-10-12First Compliant Deposit (FCD) Date
2021-10-12Usage metrics
Categories
No categories selectedKeywords
Licence
Exports
RefWorks
BibTeX
Ref. manager
Endnote
DataCite
NLM
DC