| Challenge: | Detection problems involving positive instances are often deficient in information extraction tasks . a number of researches have employed neural network models to solve detection problems . |
| Approach: | They propose an algorithm which can handle positive sparsity problem and directly optimize over F-measure . they borrow the idea of marginal utility from economics and propose a theoretical framework for instance importance measuring . |
| Outcome: | The proposed algorithm improves on positive sparsity problem and over F-measure . it leads to more effective and stable training of neural network based detection models. |
Similar Papers
Scalable Evaluation and Improvement of Document Set Expansion via Neural Positive-Unlabeled Learning (2021.eacl-main)
Copied to clipboard
| Challenge: | Current methods for document set expansion for large collections are based on word-frequency or bag-of-words document similarity metrics. |
| Approach: | They propose to extend the IR approach by treating the problem as an instance of positive-unlabeled (PU) learning . they propose solutions for each challenge and empirically validate them with ablation tests . |
| Outcome: | The proposed method improves on a PubMed abstract retrieval task . it is compared with existing methods and empirically validated with ablation tests . |
Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Attention pruning techniques have been developed to identify and exploit sparseness . previous work has taken pioneering steps to discover and explain the sparsity in attention patterns . |
| Approach: | They propose a framework that observes attention patterns in a fixed dataset and generates a global sparseness mask. |
| Outcome: | The proposed approach saves 90% of computations and maintains quality of results. |
Threshold-driven Pruning with Segmented Maximum Term Weights for Approximate Cluster-based Sparse Retrieval (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using rank score thresholding, sparse retrieval skips the index at cluster and document levels. |
| Approach: | They propose a pruning control scheme with a probabilistic guarantee on rank-safeness competitiveness. |
| Outcome: | The proposed pruning control scheme improves accuracy and safeness while delivering low latency on single-threaded CPU. |
Adaptive Document Retrieval for Deep Question Answering (D18-1)
Copied to clipboard
| Challenge: | Existing methods for deep question answering do not understand the exact interplay between document retrieval and machine comprehension. |
| Approach: | They propose an adaptive document retrieval model that learns the optimal document number, conditional on the size of the corpus and the query. |
| Outcome: | The proposed model outperforms state-of-the-art methods on multiple benchmark datasets and in the context of corpora with variable sizes. |
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Sparse attention is a promising strategy to extend long-context capabilities in LLMs . but its efficiency–accuracy trade-offs remain unclear due to the lack of comprehensive evaluation . |
| Approach: | They evaluate sparse attention methods across multiple model families and sizes . they find larger sparser models outperform smaller dense ones at equivalent cost . |
| Outcome: | The proposed methods outperform smaller sparse models at equivalent cost and improve the Pareto frontier. |
Improve Interpretability of Neural Networks via Sparse Contrastive Coding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | XAI has achieved remarkable advances, but few efforts have been devoted to solving the problem. |
| Approach: | They propose a model-agnostic explanation method termed Sparse Contrastive Coding . they use model-based explanations to explain the black-box in a more model-oriented way . |
| Outcome: | The proposed method outperforms five state-of-the-art methods in interpretability and classification metrics. |
Efficient Sparse Attention needs Adaptive Token Release (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide array of text-centric tasks, however, their ‘large’ scale introduces significant computational and storage challenges, particularly in managing the key-value states of the transformer, which limits their wider applicability. |
| Approach: | They propose to release resources from caches and rebuild key-value states by a lightweight controller module to approximate an ideal top-K sparse attention. |
| Outcome: | The proposed method achieves a significant throughput improvement of 221.8% over full attention and a model with 7 billion tokens. |
Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training (2023.findings-acl)
Copied to clipboard
| Challenge: | Dense retrievers have impressive performance, but their demand for abundant training data limits their application scenarios. |
| Approach: | They propose a method which uses unlabeled data to construct pseudo-positive examples from unlabelled data and then contrastively weighs the contrastive loss of different pairs according to the estimated relevance. |
| Outcome: | The proposed method beats the SOTA unsupervised Contriever model on BEIR and open-domain QA retrieval benchmarks and is a good few-shot learner. |
Learning with Noise-Contrastive Estimation: Easing training by learning to scale (C18-1)
Copied to clipboard
| Challenge: | Neural language models have recently shown great improvement, but they share a common issue: large output vocabulary, computational time, and high dimensional space. |
| Approach: | They propose to make scaling factor a trainable parameter and use noise distribution to initialize output bias. |
| Outcome: | The proposed training strategies yield stable and competitive performances in small and large scale language modelling tasks. |
AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for augmented large language models suffer from irrelevant retrieved content . existing methods struggle to adapt compression rates for different context, maintain low latency . |
| Approach: | We propose an adaptive, efficient and context-aware compression framework to reduce retrieved content . AttnComp uses a top-p compression algorithm to retain the minimal set of documents whose attention weights exceed a threshold. |
| Outcome: | Experiments show that AttnComp outperforms existing compression methods and uncompressed baselines in achieving higher accuracy with substantial compression rates and lower latency. |