| Challenge: | Conventional text embedding methods suffer from information loss if directly adapted to hyper-documents. |
| Approach: | They propose an embedding approach for hyper-documents that incorporates four criteria to preserve necessary information for embeddable models. |
| Outcome: | The proposed model outperforms several existing models on two tasks in the academic domain. |
Similar Papers
Domain-Specific Word Embeddings with Structure Prediction (2023.tacl-1)
Copied to clipboard
| Challenge: | Current word embedding methods do not provide a way to use or predict information on structure between sub-corpora, time or domain. |
| Approach: | They propose a word embedding method that provides general word representations for the whole corpus, domain-specific representations and embeddable alignment simultaneously. |
| Outcome: | The proposed method provides better performance than baselines on a dataset of science and philosophy articles. |
Self-Discriminative Learning for Unsupervised Document Embedding (N19-1)
Copied to clipboard
| Challenge: | Existing methods for document embedding learning do not consider inter-document relationships. |
| Approach: | They propose to exploit the inter-document information and directly model the relations of documents in embedding space with a discriminative network and a novel objective. |
| Outcome: | The proposed method has errors that are 5 to 13% lower than state-of-the-art models and is even more pronounced in scarce label setting. |
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |
S2ORC: The Semantic Scholar Open Research Corpus (2020.acl-main)
Copied to clipboard
| Challenge: | Academic papers are an increasingly important textual domain for natural language processing (NLP) research. |
| Approach: | They propose to aggregate 81.1M English-language academic papers into a unified source . they hope this resource will facilitate research and development of tools for text mining over academic text. |
| Outcome: | The proposed corpus includes metadata, abstracts, bibliographic references, and structured full text for 8.1M open access papers. |
Improving Embedding-based Large-scale Retrieval via Label Enhancement (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for large-scale retrieval are trained with 0-1 hard labels that indicate whether a query is relevant to a document, ignoring rich information of the relevance degree. |
| Approach: | They propose to introduce label enhancement for the first time to characterize query-document relevance degree by embedding label distribution into contextual embeddables. |
| Outcome: | The proposed method significantly outperforms existing retrieval models and its counterparts equipped with two alternative methods on English and Chinese large-scale retrieval tasks. |
Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? (2023.findings-eacl)
Copied to clipboard
| Challenge: | obtaining document embeddings at document level is challenging due to computational requirements and lack of appropriate data. |
| Approach: | They compare methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. |
| Outcome: | The proposed methods produce document-level representations from sentences in 8 languages . the results show that a clever combination of sentence embeddings is usually better than encoding the full document as a single unit. |
XDoc: Unified Pre-training for Cross-Format Document Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pre-trained models target one document format at a time, making it difficult to combine knowledge from multiple document formats. |
| Approach: | They propose a unified pre-trained model which deals with different document formats in a single model. |
| Outcome: | The proposed model achieves comparable or even better performance on a variety of downstream tasks compared with the individual pre-trained models. |
Understanding the Influence of Synthetic Data for Text Embedders (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in general purpose text embedders have been driven by training on synthetic training data. |
| Approach: | They propose to use GPT-4 to produce high quality synthetic data that expands existing training datasets for embeddings to new tasks. |
| Outcome: | The proposed dataset is high quality and leads to consistent improvements in performance. |
Word Mover’s Embedding: From Word2Vec to Document Embedding (D18-1)
Copied to clipboard
Lingfei Wu, Ian En-Hsu Yen, Kun Xu, Fangli Xu, Avinash Balakrishnan, Pin-Yu Chen, Pradeep Ravikumar, Michael J. Witbrock
| Challenge: | Recent work has demonstrated that Word Mover’s Distance (WMD) that aligns semantically similar words yields unprecedented KNN classification accuracy. |
| Approach: | They propose a Word Mover’s Distance (WMD) method that aligns semantically similar words to generate unsupervised sentences or documents embeddings. |
| Outcome: | The proposed method consistently outperforms state-of-the-art techniques on 9 benchmark text classification datasets and 22 textual similarity tasks. |
HG2Vec: Improved Word Embeddings from Dictionary and Thesaurus Based Heterogeneous Graph (2022.coling-1)
Copied to clipboard
| Challenge: | Existing models that learn word embeddings rely on a large corpus of data . however, these models require massive time and space for data pre-processing and training . |
| Approach: | They propose a model that learns word embeddings utilizing only dictionaries and thesauri . they exploit a new context-focused loss model that models transitive relationships between word pairs . |
| Outcome: | The proposed model reaches the state-of-art on multiple word similarity and relatedness benchmarks. |