Papers with reproducibility
Quantified Reproducibility Assessment of NLP Results (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for reproducibility assessment are based on concepts and definitions from metrology. |
| Approach: | They propose a method for quantified reproducibility assessment that is based on metrology. |
| Outcome: | The proposed method produces comparable scores across multiple studies . authors find that it facilitates insights into causes of variation between studies - and conclusions can be drawn about improvements. |
Inducing Stereotypical Character Roles from Plot Structure (2021.emnlp-main)
Copied to clipboard
| Challenge: | Stereotypical character roles are important aids to narrative understanding and are often referred to as archetypes or dramatis personae. |
| Approach: | They propose an unsupervised method for learning stereotypical roles given only structural plot information using Vladimir Propp’s structural theory of Russian folktales. |
| Outcome: | The proposed method induces six out of seven of Vladimir Propp’s dramatis personae with F1 measures of up to 0.70 (0.58 average), with an additional category for minor characters. |
Towards LLM-driven Dialogue State Tracking (2023.emnlp-main)
Copied to clipboard
| Challenge: | emergence of large language models (LLMs) such as GPT3 and ChatGPT has sparked considerable interest in assessing their efficacy across diverse applications. |
| Approach: | They present a framework for a domain-slot instruction tuning method that allows LDST to achieve performance on par with ChatGPT. |
| Outcome: | The proposed framework performs better in zero-shot and few-shot settings than previous SOTA methods. |
TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidation (2024.acl-long)
Copied to clipboard
| Challenge: | Current methods for Continual Dialogue State Tracking (DST) struggle with catastrophic forgetting and knowledge transfer between tasks. |
| Approach: | They propose a framework for task skill localization and consolidation that enables effective knowledge transfer without relying on memory replay. |
| Outcome: | The proposed framework shows a 7.6% increase in Avg. JGA and 11% rise in BWT metrics over existing state-of-the-art methods. |
Auto-Weighted Group Relative Preference Optimization for Multi-Objective Text Generation Tasks (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Failing to balance the objectives in advance can lead to overfitting or insufficient learning of each reward function. |
| Approach: | They propose a method that adjusts reward weights according to learning progress . they evaluate AW-GRPO on advertising text generation problem . |
| Outcome: | The proposed method outperforms GRPO on advertising text generation tasks . it overfits BLEURT and jReadability at the expense of BLeurT performance . |
DP-GTR: Differentially Private Prompt Protection via Group Text Rewriting (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for prompt privacy focus on document-level rewriting, neglecting rich, multi-granular representations of text. |
| Approach: | a framework that leverages local differential privacy and composition theorem via group text rewriting is proposed . the framework is compatible with existing rewrite techniques and is publicly available at anonymous.4open.science for reproducibility. |
| Outcome: | DP-GTR is the first framework to integrate document-level and word-level information while exploiting in-context learning to improve privacy and utility. |
Non-Autoregressive Neural Machine Translation: A Call for Clarity (2022.emnlp-main)
Copied to clipboard
| Challenge: | Non-autoregressive translation models require a single forward pass to generate the output sequence instead of iteratively producing each predicted token. |
| Approach: | They propose to use a single forward pass to generate the output sequence instead of iteratively producing each predicted token. |
| Outcome: | The proposed models improve translation quality and speed under third-party testing environments. |
Experimental Standards for Deep Learning in Natural Language Processing Research (2022.findings-emnlp)
Copied to clipboard
Dennis Ulmer, Elisa Bassignana, Max Müller-Eberstein, Daniel Varab, Mike Zhang, Rob van der Goot, Christian Hardmeier, Barbara Plank
| Challenge: | a lack of common experimental standards remains an open challenge to the field at large . |
| Approach: | They propose to distill discussions on experimental standards into a single, widely-applicable methodology. |
| Outcome: | Using best practices, we can strengthen experimental evidence, improve reproducibility and enable scientific progress. |
What do Large Language Models Need for Machine Translation Evaluation? (2024.emnlp-main)
Copied to clipboard
Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orasan, Tharindu Ranasinghe, Fred Blain
| Challenge: | Existing research shows that large language models can perform better in machine translation tasks. |
| Approach: | They propose to use large language models for machine translation evaluations . authors explore what translation information is needed for LLMs to evaluate MT quality . |
| Outcome: | The proposed model performs comparable to fine-tuned multilingual pre-trained models. |
Neural Unification for Logic Reasoning over Natural Language (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Automated Theorem Proving (ATP) is a computer program that can show that conjectures are logical consequences of a set of axioms. |
| Approach: | They propose a transformer-based architecture for deriving conjectures given axioms . they propose 'neural unifier' and relative training procedure to train the model . |
| Outcome: | The proposed architectures are able to answer queries with deep queries with a relatively low training time. |
LLMs can Perform Multi-Dimensional Analytic Writing Assessments: A Case Study of L2 Graduate-Level Academic English Writing (2025.acl-long)
Copied to clipboard
| Challenge: | a growing number of studies have indicated the general usefulness of LLMs for automated writing assessments. |
| Approach: | They propose a framework that evaluates LLMs' ability to provide scores and comments based on multiple assessment criteria. |
| Outcome: | The proposed framework is interpretable, cost-efficient, scalable, and reproducible . it is compared to existing methods that rely on manual judgments . |
Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that contextual language models display outlier dimensions . this is true for monolingual and multilingual models, but little work has been done on multilingual contexts . |
| Approach: | They investigate outlier dimensions and their relationship to anisotropy in multilingual contexts . they focus on cross-lingual semantic similarity tasks . |
| Outcome: | The proposed model improves on cross-lingual semantic similarity tasks. |
Towards large language model-based personal agents in the enterprise: Current trends and open problems (2023.findings-emnlp)
Copied to clipboard
Vinod Muthusamy, Yara Rizk, Kiran Kate, Praveen Venkateswaran, Vatche Isahagian, Ashu Gulati, Parijat Dube
| Challenge: | Existing large language models (LLMs) are brittle to input changes and can produce inconsistent results for the same inputs. |
| Approach: | They propose to use large language models to reason about complex goals and orchestrate a set of pluggable tools or APIs to accomplish a goal. |
| Outcome: | The proposed use cases have many open problems in an exciting area of NLP research, such as trust and explainability, consistency and reproducibility, and the need for new metrics and benchmarks. |
Zero-shot Cross-domain Dialogue State Tracking via Context-aware Auto-prompting and Instruction-following Contrastive Decoding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Previous studies have implemented slot-based input improvements, such as schema-driven descriptions and question-answering formats, but still suffer from negative transfer for seen slots and inefficient transfer for unseen slots due to the significant source-target domain gap. |
| Approach: | They propose a framework that generates dynamic, context-aware slot queries to improve model transferability by penalizing deviations from the provided instructions. |
| Outcome: | Experiments on two datasets show that the proposed model performs better than existing models on the restaurant domain. |
Catch Me If You Can? Not Yet: LLMs Still Struggle to Imitate the Implicit Writing Styles of Everyday Authors (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Personal style is often subtle and implicit, making it difficult to specify through prompts yet essential for user-aligned generation. |
| Approach: | They evaluate LLMs' ability to imitate personal writing styles via in-context learning from user-authored samples. |
| Outcome: | The proposed model can imitate personal writing styles from a small number of user-authored samples. |
Dual Hierarchical Dialogue Policy Learning for Legal Inquisitive Conversational Agents (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing systems for conversational AI are user-driven, but in many real-world situations, they do not extract information to achieve its own objectives. |
| Approach: | They propose an inquisitive conversational agent that learns when and how to ask probing questions . they also propose a framework for a conversational ICA specifically tailored to the court . |
| Outcome: | The proposed method outperforms single-agent RL baselines on a U.S. Supreme Court dataset. |
EmbedTextNet: Dimension Reduction with Weighted Reconstruction and Correlation Losses for Efficient Text Embedding (2023.findings-acl)
Copied to clipboard
| Challenge: | EmbedTextNet is a light add-on network that can be appended to an arbitrary language model to generate a compact embedding without requiring any changes in its architecture or training procedure. |
| Approach: | They propose an add-on network that can be appended to an arbitrary language model to generate a compact embedding without requiring any changes in its architecture or training procedure. |
| Outcome: | The proposed network can be appended to an arbitrary language model to generate a compact embedding without any changes in its architecture or training procedure. |
MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Metadata extraction relies heavily on manual annotation of documents. |
| Approach: | They propose a framework that leverages Large Language Models to automatically extract metadata attributes from scientific papers covering datasets of languages other than Arabic. |
| Outcome: | The proposed framework automates the extraction of metadata attributes from Arabic scientific papers using large language models. |
AIMMerging: Adaptive Iterative Model Merging Using Training Trajectories for Language Model Continual Learning (2025.emnlp-main)
Copied to clipboard
Yujie Feng, Jian Li, Xiaoyu Dong, Pengfei Xu, Xiaohui Zhou, Yujia Zhang, Zexin Lu, Yasha Wang, Alan Zhao, Xu Chu, Xiao-Ming Wu
| Challenge: | Recent model merging-based methods struggle to effectively manage the trade-off between learning new knowledge and preventing catastrophic forgetting. |
| Approach: | They propose a model merging framework that utilizes learning and forgetting signals from the training trajectory to dynamically monitor the model’s training status. |
| Outcome: | The proposed framework achieves significant performance improvements over existing state-of-the-art methods on three CL benchmarks with various model sizes (from 770M to 13B). |
LVLMs are Bad at Overhearing Human Referential Communication (2025.emnlp-main)
Copied to clipboard
| Challenge: | a crucial skill for embodied AI agents working with humans is grounding in referential communication. |
| Approach: | They use large vision language models to overhear spontaneous conversations between humans . they find that current LVLMs fail to show consistent performance improvement . |
| Outcome: | The proposed models fail to show consistent performance improvement over previous models . the authors release the results to facilitate future research . |
Link, Synthesize, Retrieve: Universal Document Linking for Zero-Shot Information Retrieval (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for zero-shot learning are sparse, but have been used for dense retrieval (DR) . |
| Approach: | They propose a novel Universal Document Linking algorithm which links similar documents to enhance synthetic query generation across multiple datasets with different characteristics. |
| Outcome: | The proposed algorithm surpasses state-of-the-art methods in zero-shot cases. |
Revealing the impact of synthetic native samples and multi-tasking strategies in Hindi-English code-mixed humour and sarcasm detection (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Specifically, we tried native sample mixing, multi-task learning, and prompting and instruction finetuning very large multilingual language models (VMLMs). |
| Approach: | They used native sample mixing, multi-task learning and prompting and instruction finetuning to improve code-mixed humour and sarcasm detection. |
| Outcome: | The proposed methods improve humour and sarcasm detection by adding native samples to training sets and multitask learning and prompting and instruction finetuning VMLMs. |
LLM-Based Multi-Task Bangla Hate Speech Detection: Type, Severity, and Target (2026.acl-long)
Copied to clipboard
| Challenge: | Existing work on social media platforms is limited in its ability to detect hate speech . a lack of reliable and scalable automated hate speech detection systems is a challenge for low-resource languages like Bangla. |
| Approach: | They propose to use a single-task, single-targeted, single language dataset to identify hate speech in Bangla. |
| Outcome: | The proposed dataset is the largest manually annotated Bangla hate-speech dataset to date. |