PROBES : Performance and Relevance Observation for BEtter Search (2026.eacl-industry)
Copied to clipboard
| Challenge: | Qualitative search is essential for the success of online platforms, authors say . large-scale evaluation of search systems is essential to ensure high-quality user experiences . |
| Approach: | They propose a multi-task system powered by Large Language Models for end-to-end evaluation of semantic search systems. |
| Outcome: | The proposed system provides more precise and consistent relevance assessments across query categories. |
Similar Papers
Revisiting LLM Value Probing Strategies: Are They Robust and Expressive? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing value probing methods that capture in-context information and predict models’ real-world actions are limited and lack systematic comparisons. |
| Approach: | They compare three widely used value probing methods: token likelihood, sequence perplexity, and text generation. |
| Outcome: | The proposed methods exhibit large variances under non-semantic perturbations in prompts and option formats, with sequence perplexity being the most robust overall. |
Redefining Retrieval Evaluation in the Era of LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Traditional IR metrics assume that humans examine documents sequentially with diminishing attention to lower ranks. |
| Approach: | They propose a utility-based annotation schema that quantifies positive contribution of relevant passages and negative impact of distracting ones. |
| Outcome: | The proposed metric improves correlation with the end-to-end answer accuracy by up to 36% compared to traditional metrics. |
Does My Representation Capture X? Probe-Ably (2021.acl-demo)
Copied to clipboard
| Challenge: | Probing (or diagnostic classification) has become a popular strategy for investigating whether a given set of intermediate features is present in the representations of neural models. |
| Approach: | They propose to use an extendable probing framework to automate the application of probing methods to the user’s inputs. |
| Outcome: | The proposed framework automates the application of probing methods to the user’s inputs. |
Over-Searching in Search-Augmented Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Search-augmented large language models (LLMs) excel at knowledge-intensive tasks by integrating external retrieval. |
| Approach: | They conduct a systematic evaluation of over-searching across multiple dimensions including query types, model categories, retrieval conditions, and multi-turn conversations. |
| Outcome: | The proposed model improves answer accuracy on answerable queries but harms abstention on unanswerable ones . |
False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work has leveraged probing-based approaches to study the separability of malicious and benign inputs in Large Language Models’ internal representations. |
| Approach: | They propose to use probing-based methods to study separability of malicious and benign inputs in LLMs' internal representations to detect harmful and benign content. |
| Outcome: | The proposed methods show that they learn superficial patterns rather than semantic harmfulness. |
Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probes (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are often used as automated judges to evaluate text, but their effectiveness can be hindered by unintentional biases. |
| Approach: | They propose to use linear classifying probes to access LLMs’ latent knowledge and extract more accurate preferences by leveraging differences between contrasting pairs of prompts. |
| Outcome: | The proposed method outperforms generation-based judgements while maintaining similar computational costs. |
INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning (2024.acl-long)
Copied to clipboard
Yutao Zhu, Peitian Zhang, Chenghao Zhang, Yifei Chen, Binyu Xie, Zheng Liu, Ji-Rong Wen, Zhicheng Dou
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks, but their application to information retrieval tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language. |
| Approach: | They propose to use instruction tuning to enhance LLMs' proficiency in IR tasks by combining a dataset with manually written templates to analyze the effects of instruction design, template diversity, few-shot demonstrations, and the volume of instructions. |
| Outcome: | The proposed model can be used to perform query understanding, document understanding, and query-document relationship understanding tasks. |
Beyond Superficial Tests: Adversarial Refinement for Reliable Property-Based Testing (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable proficiency in code generation, yet their application to Property-Based Testing (PBT) remains fraught with a superficiality gap. |
| Approach: | They propose an agentic framework that hardens software properties through Adversarial Refinement. |
| Outcome: | a new framework hardens software properties through Adversarial Refinement that detects and fixes bugs in top-tier libraries. |
Assessing LLMs for Zero-shot Abstractive Summarization Through the Lens of Relevance Paraphrasing (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have achieved state-of-the-art performance at zero-shot summarization of abstractive summaries for given articles, but little is known about their robustness at this task. |
| Approach: | They propose a strategy that uses the most relevant sentences to generate an ideal summary and then paraphrases them to obtain a minimally perturbed dataset. |
| Outcome: | The proposed approach can be used to measure the robustness of LLMs as summarizers on a minimally perturbed dataset. |
How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have focused on enhancing the factualness of large language models using context knowledge. |
| Approach: | They propose to use ChatGPT to construct probing datasets that provide diverse and coherent evidence corresponding to various facts. |
| Outcome: | The proposed model can encode knowledge across different layers, and it is compared with existing models. |