Papers with SciNCL
Contrastive Learning Using Graph Embeddings for Domain Adaptation of Language Models in the Process Industry (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Recent trends in NLP utilize knowledge graphs to enhance pretrained language models by incorporating additional knowledge from the graph structures to learn domain-specific terminology or relationships between documents that might otherwise be overlooked. |
| Approach: | They propose to use graph-aware neighborhood contrastive learning methodology SciNCL to enhance pretrained language models by incorporating additional knowledge from graph structures. |
| Outcome: | The proposed graph-aware neighborhood contrastive learning methodology outperforms a state-of-the-art mE5-large text encoder on the process industry text embedding benchmark while having 3 times fewer parameters. |
SciRepEval: A Multi-Format Benchmark for Scientific Document Representations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for evaluating scientific document representations fail to capture the diversity of relevant tasks. |
| Approach: | They propose a benchmark for training and evaluating scientific document representations that includes 24 challenging and realistic tasks across four formats: classification, regression, ranking and search. |
| Outcome: | The proposed model outperforms existing models by over 2 points absolute. |
Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prior work relies on discrete citation relations to generate contrast samples, but discrete ones enforce a hard cut-off to similarity. |
| Approach: | They propose to use nearest neighbor sampling to learn continuous similarity and to sample hard-to-learn negatives and positives by controlling the sampling margin between them. |
| Outcome: | The proposed method outperforms the state-of-the-art on the SciDocs benchmark and can train (or tune) language models sample-efficiently. |