Papers with Llama-3.1
Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approaches (2025.naacl-short)
Copied to clipboard
| Challenge: | Summarizing large text collections is a valuable tool for document research . a multi-stage pipeline and lack of global context are challenges for large-scale summarization systems. |
| Approach: | They compare compression and full-text systems for large-scale multi-document summarization . they find that compression-based methods outperform full-context methods . |
| Outcome: | The proposed methods outperform compression-based methods on three datasets . however, they suffer information loss due to their multi-stage pipeline and lack of global context. |
The Dangers of Indirect Prompt Injection Attacks on LLM-based Autonomous Web Navigation Agents: A Demonstration (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Large Language Model (LLM)-integrated applications are becoming more popular to support, augment, and automate tasks. |
| Approach: | They propose to embed universal adversarial triggers in webpage HTML to hijack agents . they also use a browser-gym agent powered by Llama-3.1 to test their system . |
| Outcome: | The proposed system software is released under the MIT License . |
A Case Study of Cross-Lingual Zero-Shot Generalization for Classical Languages in LLMs (2025.findings-acl)
Copied to clipboard
V.S.D.S.Mahesh Akavarapu, Hrishikesh Terdalkar, Pramit Bhattacharyya, Shubhangi Agarwal, Dr. Vishakha Deulgaonkar, Chaitali Dangarikar, Pralay Manna, Arnab Bhattacharya
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable generalization capabilities across diverse tasks and languages. |
| Approach: | They focus on named entity recognition and machine translation into English to examine factors affecting cross-lingual zero-shot generalization. |
| Outcome: | The proposed models perform better than fine-tuned baselines on out-of-domain data, but smaller models struggle with niche or abstract entity types. |
MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) generate plausible but factually incorrect outputs, posing serious risks to patient safety and clinical decision-making. |
| Approach: | They propose a benchmark for medical hallucination detection using 10,000 question-answer pairs derived from PubMedQA. |
| Outcome: | The proposed model achieves an F1 score as low as 0.625 for detecting 'hard' category hallucinations. |
Semi-supervised Fine-tuning for Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing LLMs require labeled data, which can be costly in real-world applications. |
| Approach: | They propose a framework that can fully exploit labeled and unlabeled data for LLM fine-tuning . they conducted experiments using GPT-4o-mini and Llama-3.1 on seven general or domain-specific datasets . |
| Outcome: | The proposed framework can fully exploit labeled and unlabeled data for LLM alignment from a propagate-and-select manner. |
POP: Prefill-Only Pruning for Efficient Large Model Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing structured pruning methods suffer from significant accuracy degradation . Existing pruning methods are expensive and require specialized hardware and kernels to perform . |
| Approach: | They propose a stage-agnostic pruning approach that overlooks asymmetric roles between prefill and decode stages. |
| Outcome: | The proposed pruning approach achieves 1.37 speedup in prefill latency with minimal performance loss. |
SelfPrompt: Autonomously Evaluating LLM Robustness via Domain-Constrained Knowledge Guidelines and Refined Adversarial Prompts (2025.coling-main)
Copied to clipboard
| Challenge: | Existing frameworks for evaluating robustness of large language models rely on standardized benchmarks that can escalate costs and limit evaluations across domains. |
| Approach: | They propose a framework to evaluate the robustness of large language models using adversarial prompts and domain-constrained knowledge guidelines. |
| Outcome: | The proposed framework reduces dependency on conventional benchmarks and provides efficient evaluations in constrained domains. |
EC-RAFT: Automated Generation of Clinical Trial Eligibility Criteria through Retrieval-Augmented Fine-Tuning (2025.findings-acl)
Copied to clipboard
| Challenge: | Eligibility criteria (EC) are critical components of clinical trial design, specifying parameters for participant inclusion and exclusion. |
| Approach: | They propose a method that utilizes Retrieval-Augmented Fine-Tuning to generate structured and cohesive EC directly from clinical trial titles and descriptions. |
| Outcome: | The proposed method outperforms Llama-3.1-8B-Instruct and Llm-as-a-Judge models in BERTScore and EC score. |
Exploring Safety-Utility Trade-Offs in Personalized Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Prior studies have shown that large language models can exhibit bias against specific demographic groups and engage in the generation of stereotypical responses. |
| Approach: | They propose a framework to evaluate LLM performance along two axes: safety and utility. |
| Outcome: | The proposed framework evaluates the performance of LLMs along two axes: safety and utility. |
AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research (2025.acl-long)
Copied to clipboard
Yilun Zhao, Weiyuan Chen, Zhijian Xu, Manasi Patwardhan, Chengye Wang, Yixin Liu, Lovekesh Vig, Arman Cohan
| Challenge: | a benchmark designed to evaluate the capabilities of LLMs in designing ablation studies for scientific research is available online. |
| Approach: | They propose to use a benchmark to evaluate LLMs' ability to design ablation studies . they investigate whether current automated evaluation methods are not reliable . |
| Outcome: | The benchmark compared leading LLMs with human experts on generating detailed ablation study designs . the results show that current evaluation methods are not reliable for the task . |
Layer-aware Dual-directional Modulation for Low-resource Machine Translation (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated excellent performance in Machine Translation (MT) however, a performance gap persists between high-resource and low-resourced languages due to imbalanced pre-training data. |
| Approach: | They propose a layer-wise metric to quantify the activation divergence between high- and low-resource languages. |
| Outcome: | The proposed model outperforms standard LoRA fine-tuning on Chinese-to-seven low-resource language translations. |
Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been successful in medical text summarization . however, they do not perform fine-grained evaluations under difficult settings . |
| Approach: | They show that large language models show a significant performance drop for data points with high concentration of out-of-vocabulary words or with high novelty. |
| Outcome: | The proposed model shows a significant performance drop for data points with high concentration of out-of-vocabulary words or with high novelty. |
Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RL (2026.findings-acl)
Copied to clipboard
Zhaofeng Wu, Shiqi Wang, Boya Peng, Anuj Kumar Goyal, Melanie Kambadur, Sebastian Ruder, Yoon Kim, Chloe Bi
| Challenge: | Modern language models demonstrate impressive coding capabilities in common programming languages (PLs) but their performance in lower-resource PLs is often limited by training data availability. |
| Approach: | They propose a zero-shot cross-programming-language transfer task for code RL . they propose RL training in a source PL fails to improve performance on other target PLs . |
| Outcome: | The proposed approach improves transferability in Llama-3.1 code generation on parallel-stack model . it also improves performance on other target PLs, compared to single-PL SFT . |
From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies on mechanistic interpretability revealed that long-context factuality is closely related to a set of attention heads, retrieval heads. |
| Approach: | They propose a method that generates training signals by contrasting normal model outputs with those from an ablated variant. |
| Outcome: | The proposed method achieves significant improvements on LLMs with a sparse retrieval score distribution. |