Papers by Shafiq Joty
Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Cross-lingual transfer of language models trained on high-resource languages such as English has been limited due to the high cost of obtaining non-English conversational data. |
| Approach: | They introduce a parallel and large-scale multilingual conversation dataset that is used for cross-lingual alignment pretraining by translating the English-only Schema-Guided Dialogue dataset into 105 other languages. |
| Outcome: | The proposed model performs well on slot-filling and intent classification tasks, and is able to perform well in other languages. |
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023.findings-acl)
Copied to clipboard
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, Jimmy Huang
| Challenge: | Currently, the evaluation of large language models (LLMs) such as ChatGPT in academic datasets is difficult due to the difficulty of evaluating the generative outputs produced by this model against the ground truth. |
| Approach: | They evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in academic datasets. |
| Outcome: | The proposed model performs well on 140 tasks and generates 255K responses in these datasets. |
Chart-to-Text: A Large-Scale Benchmark for Chart Summarization (2022.acl-long)
Copied to clipboard
Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, Shafiq Joty
| Challenge: | Inferring key insights from charts can be challenging and time-consuming. |
| Approach: | They propose a task where the goal is to explain a chart and summarize key takeaways from it in natural language. |
| Outcome: | The proposed model produces fluent summaries but suffers from hallucinations and factual errors . the proposed model is compared with other models and can be used to generate BLEU scores . |
Towards Low-Resource Automatic Program Repair with Meta-Learning and Pretrained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in deep learning (DL) based APR models have demonstrated promising results by learning from large-scale bug-fix examples in a data-driven manner. |
| Approach: | They propose a meta-learning framework integrated with code pretrained language models to generate fixes for low-resource bugs with limited training samples. |
| Outcome: | The proposed framework learns better error-specific knowledge from high-resource bugs through efficient first-order meta-learning optimization, which allows for a faster adaptation to the target low-resourced bugs. |
GeDi: Generative Discriminator Guided Sequence Generation (2021.findings-emnlp)
Copied to clipboard
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, Nazneen Fatema Rajani
| Challenge: | Existing methods for controlling LMs have limitations. |
| Approach: | They propose a class-conditional LM that uses a control code to control text generation. |
| Outcome: | The proposed algorithm is much faster than the existing methods for generating from the LM directly. |
On Positional Bias of Faithfulness for Long-form Summarization (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models exhibit positional bias in long-context settings, under-attending to information in the middle. |
| Approach: | They compile eight human-annotated long-form summarization datasets to evaluate faithfulness . they find that LLMs faithfully summarize beginning and end of documents but neglect middle content . |
| Outcome: | The proposed methods show that LLMs under-attend to information in the middle of inputs. |
Direct Judgement Preference Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing judge models are largely trained with supervised finetuning on small data scales to perform limited types of evaluation tasks, limiting generalization. |
| Approach: | They propose to train judge models at large data scales with direct preference optimization . they use four training tasks to form three types of preference pairs targeting different aspects of evaluation . |
| Outcome: | The proposed model outperforms GPT-4o and other similar models on 13 benchmarks. |
J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly being used for reasoning intensive tasks. |
| Approach: | They propose an algorithm that trains judges to be robust to positional biases . they also propose a benchmark that evaluates judges in diverse reasoning settings . |
| Outcome: | The proposed algorithm outperforms GPT-4o and the next best small judge by 6.7% and 9% on ReasoningJudgeBench and JudgeBench. |
It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on societal bias in NLP focuses on race and gender . linguistic background is a unique attribute that has been largely ignored in the field . |
| Approach: | They examine linguistic background to craft plausible adversarial examples that expose biases in popular NLP models. |
| Outcome: | The proposed model improves robustness without sacrificing performance on clean data. |
FOLIO: Natural Language Reasoning with First-Order Logic (2024.emnlp-main)
Copied to clipboard
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, Dragomir Radev
| Challenge: | Existing benchmarks for logical reasoning in large language models lack language naturalness or limited complexity. |
| Approach: | They propose to use first-order logic annotations to evaluate logical reasoning capabilities of large language models. |
| Outcome: | The proposed dataset evaluates the FOL reasoning ability of supervised fine-tuning on medium-sized language models. |
Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision Language Models struggle with visual arithmetic, seemingly simple tasks like object counting or length comparison, which are essential for relevant complex tasks like chart understanding and geometric reasoning. |
| Approach: | They propose a novel post-training strategy inspired by Piaget’s theory of cognitive development that trains VLMs to recognize invariant properties under visual transformations. |
| Outcome: | The proposed approach outperforms supervised fine-tuning methods while requiring 60% less training data. |
SWiPE: A Dataset for Document-Level Simplification of Wikipedia Pages (2023.acl-long)
Copied to clipboard
| Challenge: | Prior work on document-level simplification has focused on sentence-level edits, while many desirable edits require document- level context. |
| Approach: | They propose a dataset that reconstructs the document-level editing process from English Wikipedia to paired Simple Wikipedia articles. |
| Outcome: | The proposed dataset reconstructs the document-level editing process from English Wikipedia (EW) articles to paired Simple Wikipedia (SEW) pages. |
On Context Utilization in Summarization with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models excel in abstractive summarization tasks, delivering fluent and pertinent summaries. |
| Approach: | They conduct the first comprehensive study on context utilization and position bias in summarization. |
| Outcome: | The proposed benchmark compares two methods to alleviate position bias in summarization tasks. |
What Makes a Good Natural Language Prompt? (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies on prompt quality show imbalanced support across models and tasks, and research gaps. |
| Approach: | They propose a property- and human-centric framework for evaluating prompt quality . they propose comparing prompt quality to other factors such as adverbs and apverbs . |
| Outcome: | The proposed framework reveals imbalanced support across models and tasks and substantial research gaps. |
Effective Fine-Tuning Methods for Cross-lingual Adaptation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large scale multilingual pre-trained language models have shown promising results in zero- and few-shot cross-lingual tasks. |
| Approach: | They propose a co-tuning method that aims to learn more generalized semantic equivalences when the languages are structurally dissimilar. |
| Outcome: | The proposed method improves on cross-lingual inference and review tasks by capturing the semantic relationship in the parallel data when a few translation pairs are available. |
Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning? (2025.acl-industry)
Copied to clipboard
Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub, Ahmed Masry, Mizanur Rahman, Amran Bhuiyan, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque, Jimmy Huang
| Challenge: | Large Vision-Language Models (LVLMs) are expensive and time-consuming to evaluate . however, they are limited in their use in industrial settings due to their limited availability and limited resources. |
| Approach: | They evaluate 13 open-source LVLMs as judges for diverse chart comprehension and reasoning tasks. |
| Outcome: | The proposed models can be used to assess chart comprehension and reasoning tasks, but they are expensive and time-consuming. |
UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for chart-based data analysis neglect explicit modeling of chart structures. |
| Approach: | They propose a pretrained model for chart comprehension and reasoning that encodes relevant text, data, and visual elements of charts and uses a chart-grounded text decoder for text generation. |
| Outcome: | The proposed model outperforms existing methods that lack explicit modeling of chart structures and lacks explicit modeling. |
Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have a number of shortcomings, including lack of factual correctness. |
| Approach: | They propose a framework to increase prediction factuality by post-editing reasoning chains . they propose to use large language models to generate interpretable reasoning chains. |
| Outcome: | The proposed framework leads to accuracy improvements in open-domain question-answering tasks. |
Alleviating Sparsity of Open Knowledge Graphs with Ternary Contrastive Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to learning KG triplets ignore ternary propagation patterns and ignore zero-shot, few-shot and synonymity problems. |
| Approach: | They propose a framework for contrastive learning based on ternary propagation patterns among head, relation and tail. |
| Outcome: | Experiments on benchmarks show that TernaryCL is superior to state-of-the-art models. |
Retrieving Multimodal Information for Augmented Generation: A Survey (2023.findings-emnlp)
Copied to clipboard
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, Shafiq Joty
| Challenge: | Large Language Models (LLMs) are increasingly using multimodality to augment their generation ability, but there is no unified perception of at which stage and how to incorporate different modalities. |
| Approach: | They propose to use multimodality to augment Large Language Models (LLMs) this will provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs. |
| Outcome: | The proposed methods improve factuality, reasoning, interpretability, and robustness of the generated content. |
VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Prior work focused on attention mechanisms to model complex interactions in visual dialog . a new framework for visual dialog is based on pretrained BERT language models . |
| Approach: | They propose a framework for a vision-dialog Transformer that leverages pretrained BERT language models for Visual Dialog tasks. |
| Outcome: | The proposed framework achieves the top position on the visual dialog leaderboard without pretraining on external vision-language data. |
Online Conversation Disentanglement with Pointer Networks (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for disentangling textual conversations rely on dataset specific features that hinder generalization and adaptability. |
| Approach: | They propose an end-to-end online framework for conversation disentanglement that embeds the whole utterance that comprises timestamp, speaker, and message text. |
| Outcome: | The proposed method performs state-of-the-art on the Ubuntu IRC dataset and on other social and organizational platforms. |
UXLA: A Robust Unsupervised Data Augmentation Framework for Zero-Resource Cross-Lingual NLP (2021.acl-long)
Copied to clipboard
| Challenge: | Transfer learning has yielded state-of-the-art (SoTA) results, but annotated data for every target task in every target language is rare, especially for low-resource languages. |
| Approach: | They propose a framework for unsupervised data augmentation for zero-resource transfer learning scenarios that performs simultaneous self-training with data hausse and unsupervised sample selection. |
| Outcome: | The proposed framework outperforms baselines on three zero-resource transfer tasks and achieves SoTA results in all the tasks. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
Coherence Modeling of Asynchronous Conversations: A Neural Entity Grid Approach (P18-1)
Copied to clipboard
| Challenge: | Existing coherence models are not able to distinguish coherent discourses from incoherent ones. |
| Approach: | They propose a novel coherence model for written asynchronous conversations . they propose to lexicalize the model's entity transitions and extend it to asynchron conversations based on conversational structure . |
| Outcome: | The proposed model outperforms existing models on coherence assessment and thread reconstruction tasks. |
DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | Retrievalaugmented LLMs have been used to ground LLM in external knowledge . a gap exists in the current landscape regarding the effectiveness of grounding LLM on heterogeneous knowledge sources. |
| Approach: | They propose a model that uses symbolic language to generate symbolic queries . they use a dataset that is generated using predefined reasoning chains and human annotation . |
| Outcome: | The proposed model outperforms previous approaches by a significant margin in QA tasks over text. |
Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encoding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Inflectional variation is a common feature of World Englishes such as Colloquial Singapore English and African American Vernacular English. |
| Approach: | They propose a method to tokenize English text by reducing inflected words to their base forms before reinjecting the grammatical information as special symbols. |
| Outcome: | The proposed method defends against inflectional adversaries while maintaining performance on clean data. |
P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains (2024.findings-emnlp)
Copied to clipboard
Simeng Han, Aaron Yu, Rui Shen, Zhenting Qi, Martin Riddell, Wenfei Zhou, Yujie Qiao, Yilun Zhao, Semih Yavuz, Ye Liu, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Dragomir Radev, Rex Ying, Arman Cohan
| Challenge: | Existing methods on understanding the capabilities of LLMs in logical reasoning rely on binary entailment classification or synthetically derived rationales. |
| Approach: | They propose to annotate a human-annotated dataset consisting of diverse and complex reasoning chains for a set of realistic logical reasoning stories also written by humans. |
| Outcome: | The proposed model outperforms existing methods on understanding the capabilities of LLMs in logical reasoning by 10% or more. |
Learning Auxiliary Tasks Improves Reference-Free Hallucination Detection in Open-Domain Long-Form Generation (2025.acl-short)
Copied to clipboard
Chengwei Qin, Wenxuan Zhou, Karthik Abinav Sankararaman, Nanshu Wang, Tengyu Xu, Alexander Radovic, Eryk Helenowski, Arya Talebzadeh, Aditya Tayade, Sinong Wang, Shafiq Joty, Han Fang, Hao Ma
| Challenge: | Existing methods for detecting hallucination in long-form tasks focus on limited domains or rely heavily on external fact-checking tools, which may not always be available. |
| Approach: | They propose a new paradigm that augments fine-tuning with an auxiliary task for the model to jointly learn with the main task of hallucination detection. |
| Outcome: | The proposed method outperforms existing methods for detecting hallucination in open-domain long-form generation and is more accurate than random guessing. |
Contrastive Learning with Generated Representations for Inductive Knowledge Graph Embedding (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for inductive knowledge Graphs are limited by sparsity and implicit transfer. |
| Approach: | They propose a Contrastive Learning framework with graph guided Variational autoencoder on Meta-KGs to capture and transfer entities. |
| Outcome: | The proposed framework outperforms state-of-the-art methods with extensive experiments. |
MulDA: A Multilingual Data Augmentation Framework for Low-Resource Cross-Lingual NER (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual NER are labeled sequence translation and instance-based transfer via machine translation (MT) Existing methods to cross NER include label projection and labeling, but they are expensive and time-consuming. |
| Approach: | They propose a simple but effective labeled sequence translation method to translate source-language training data to target languages and avoids word order change and entity span determination. |
| Outcome: | The proposed method avoids word order change and entity span determination and can be generalized with the language-specific features from the target-language synthetic data and the language independent features from multilingual synthetic data. |
ParaICL: Towards Parallel In-Context Learning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to improve ICL performance are limited by the length of the input context. |
| Approach: | They propose a method that utilizes all demonstration examples without exceeding the manageable context length. |
| Outcome: | The proposed method can be scaled up to integrate with existing methods. |
Is GPT-3 a Good Data Annotator? (2023.acl-long)
Copied to clipboard
| Challenge: | Data annotation is the process of labeling data that could be used to train machine learning models. |
| Approach: | They evaluate the performance of a large-scale language model developed by OpenAI . they compare it with traditional methods and analyze its output on a range of tasks . |
| Outcome: | The proposed model has shown impressive performance on a range of NLP tasks. |
A Unified Speaker Adaptation Approach for ASR (2021.emnlp-main)
Copied to clipboard
| Challenge: | Adapting a model to target speakers requires a lot of compute and may cause catastrophic forgetting to the existing speakers. |
| Approach: | They propose a unified speaker adaptation approach consisting of feature adaptation and model adaptation. |
| Outcome: | The proposed model outperforms baseline models with 20.58% relative WER reduction and surpasses finetuning method by 2.54% on target speaker adaptation. |
Evaluating Psychological Safety of Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | a recent study evaluated the psychological safety of large language models. |
| Approach: | They designed unbiased prompts to evaluate the psychological safety of large language models. |
| Outcome: | The proposed prompts showed that they were fine-tuned with behavioral metrics to reduce toxicity. |
DataNarrative: Automated Data-Driven Storytelling with Visualizations and Texts (2024.emnlp-main)
Copied to clipboard
| Challenge: | Data-driven storytelling uses visual aids and visualizations to convey insights. |
| Approach: | They propose a task for data story generation using large language models and a benchmark containing 1,449 stories from diverse sources. |
| Outcome: | The proposed framework outperforms non-agentic counterparts in both model-based and human evaluations, but also reveals unique challenges in data story generation. |
Exploring Self-supervised Logic-enhanced Training for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Traditional attempts to enhance the logical reasoning abilities of language models often rely on supervised fine-tuning, limiting their generalization to new tasks or domains. |
| Approach: | They propose a framework for integrating logical reasoning capabilities into LLMs and activating them via in-context learning. |
| Outcome: | The proposed framework achieves comparable results to existing models on three language understanding benchmarks. |
Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are known to perform tasks by simply observing few exemplars, but performance among under-represented languages falls behind due to pre-training data imbalance. |
| Approach: | They propose to assemble synthetic exemplars from high-resource languages to prompt LLMs to translate from any language into English and use them to create intra-lingual exemplar models to perform tasks in target languages. |
| Outcome: | The proposed method outperforms supervised few-shot learning in LLMs of different sizes for translations between English and 13 Indic and 21 African low-resource languages. |
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets that focus on complex reasoning questions do not address such questions as they are template-based and answers come from a fixed-vocabulary. |
| Approach: | They propose a large-scale benchmark that uses visual and logical reasoning to answer questions using a transformer-based model. |
| Outcome: | The proposed models achieve state-of-the-art on the previous datasets and on the current one, but also show that they have several challenges in answering complex reasoning questions. |
XL-HeadTags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tags (2024.findings-acl)
Copied to clipboard
| Challenge: | XL-HeadTags is a dataset that includes 20 languages across 6 diverse language families. |
| Approach: | They propose to leverage auxiliary information such as images and captions embedded in news articles to retrieve relevant sentences and utilize instruction tuning with variations to generate both headlines and tags for news articles in a multilingual context. |
| Outcome: | The proposed approach generates headlines and tags in a multilingual context using images and captions embedded in the articles and instruction tuning with variations. |
Pronoun-Targeted Fine-tuning for NMT with Hybrid Losses (2020.emnlp-main)
Copied to clipboard
| Challenge: | Popular machine translation model training uses backtranslation to improve BLEU scores . we use generative-discriminative hybrid losses to fine-tune a trained model . |
| Approach: | They propose a class of conditional generative-discriminative hybrid losses to fine-tune a machine translation model. |
| Outcome: | The proposed model improves on a sentence-level and contextual model without additional data. |
Sentence-Level Evidence Embedding for Claim Verification with Hierarchical Attention Networks (P19-1)
Copied to clipboard
| Challenge: | Claim verification is cumbersome and inefficient for human fact-checkers to find consistent pieces of evidence. |
| Approach: | They propose an end-to-end hierarchical attention network that learns to represent coherent evidence and their semantic relatedness with the claim. |
| Outcome: | The proposed model outperforms state-of-the-art models on three datasets . it is based on a coherence-based attention layer and entailment-based one . |
LNMap: Departures from Isomorphic Assumption in Bilingual Lexicon Induction Through Non-Linear Mapping in Latent Space (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for bilingual lexicon induction are mapping-based, but they do not hold for closely related languages. |
| Approach: | They propose a semi-supervised method to learn cross-lingual word embeddings for BLI using a linear mapping function and a latent space of two independently trained autoencoders. |
| Outcome: | The proposed method outperforms existing models on 15 different language pairs on both directions. |
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation (2021.naacl-main)
Copied to clipboard
Alexander Fabbri, Simeng Han, Haoyuan Li, Haoran Li, Marjan Ghazvininejad, Shafiq Joty, Dragomir Radev, Yashar Mehdad
| Challenge: | Recent advances in summarization are driven by the availability of large datasets such as the CNN-DailyMail corpus and the New York Times corpus. |
| Approach: | They propose a method for fine-tuning pretrained models for summarization in unsupervised manner . they use Wikipedia data to produce pseudo-summaries which contain characteristics of target dataset . |
| Outcome: | The proposed method achieves state-of-the-art, zero-shot abstractive summarization performance on CNN-DailyMail dataset and compares with other methods on other datasets. |
Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long context inputs, but this comes at the cost of increased computational resources and latency. |
| Approach: | They propose an algorithm that uses early LLM layers as filters to select and compress input tokens, reducing the context length for subsequent processing. |
| Outcome: | The proposed method outperforms existing techniques on the Needle in a Haystack task while demonstrating comparable performance on the LongBench challenge. |
Hierarchical Pointer Net Parsing (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to parsing are greedy transition-based and globally optimized . however, the decision-making process is based on local information, causing error propagation to subsequent steps. |
| Approach: | They propose hierarchical pointer network parsers and apply them to dependency and sentence-level discourse parsing tasks. |
| Outcome: | The proposed method outperforms existing methods and sets new state-of-the-art methods on benchmark datasets. |
Explaining Language Model Predictions with High-Impact Concepts (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to explain large language models (LLMs) are mostly correlational and lack causal features due to compositional nature of languages. |
| Approach: | They propose a framework to provide impact-aware explanations for large language models that are robust to feature changes and influential to the model’s predictions. |
| Outcome: | The proposed explanations improve on real and synthetic tasks and are robust to feature changes and influential to the model’s predictions. |
NLP+Vis: NLP Meets Visualization (2023.emnlp-tutorial)
Copied to clipboard
| Challenge: | This tutorial will introduce NLP+Vis with a focus on two main threads of work: NLP for Vis and Vis for NLP. |
| Approach: | tutorial will introduce NLP+Vis with a focus on two main threads of work . overview of research topics on combining NLP and Vis techniques will be covered . |
| Outcome: | The tutorial will introduce NLP+Vis with a focus on two main threads of work . it will provide an overview of research topics on combining NLP and Vis techniques . |
Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation (2023.emnlp-main)
Copied to clipboard
Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev
| Challenge: | Compared to neural systems, automatic metrics should be interpretable and provide intuitive insights into system performance and output quality. |
| Approach: | They propose to use a two-stage evaluation pipeline to extract basic information units from one text sequence and check the extracted units in another sequence. |
| Outcome: | The proposed metrics can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability. |
Text2Vis: A Challenging and Diverse Benchmark for Generating Multimodal Visualizations from Text (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown promise in generating visualizations from natural language, but lack of comprehensive benchmarks limits their capabilities. |
| Approach: | They propose a framework that jointly refines the textual answer and visualization code to improve GPT-4o's pass rate from 26% to 42% over direct approach. |
| Outcome: | The proposed framework increases GPT-4o’s pass rate from 26% to 42% over the direct approach and improves chart quality. |
Explicit Memory Tracker with Coarse-to-Fine Reasoning for Conversational Machine Reading (2020.acl-main)
Copied to clipboard
Yifan Gao, Chien-Sheng Wu, Shafiq Joty, Caiming Xiong, Richard Socher, Irwin King, Michael Lyu, Steven C.H. Hoi
| Challenge: | Existing approaches to answer user questions are limited in their decision making due to struggles in extracting question-related rules and reasoning about them. |
| Approach: | They propose a conversational machine reading framework that uses a Explicit Memory Tracker to track whether conditions in the rule text have already been satisfied to make a decision. |
| Outcome: | The proposed framework achieves state-of-the-art on the ShARC benchmark and is more interpretable by visualizing the entailment-oriented reasoning process as the conversation flows. |
Open-RAG: Enhanced Retrieval Augmented Reasoning with Open-Source Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to integrate Large Language Models with external knowledge suffer from limited reasoning capabilities, especially when using open-source LLMs. |
| Approach: | They propose a framework that transforms an arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE) model capable of handling complex reasoning tasks. |
| Outcome: | The proposed framework transforms an arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE) model capable of handling complex reasoning tasks, including both single- and multi-hop queries. |
Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations (2023.acl-long)
Copied to clipboard
| Challenge: | Using hidden representations, pretrained language models are prone to overfitting due to the huge amount of parameters. |
| Approach: | They propose a method that inserts random autoencoders between hidden layers of a PLM to transform activations from the previous layers into multi-view compressed representations before feeding them into the upper layers. |
| Outcome: | The proposed method improves performance across sequence- and token-level lowresource tasks. |
Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings (2022.coling-1)
Copied to clipboard
| Challenge: | Existing approaches to learn cross-lingual word embeddings are sense agnostic . a novel framework to align contextual embeddables at the sense level is proposed . |
| Approach: | They propose a framework to align contextual embeddings at the sense level by leveraging cross-lingual signal from bilingual dictionaries only. |
| Outcome: | The proposed framework improves word sense disambiguation tasks by leveraging bilingual dictionaries . compared with baseline results, the proposed models achieve 0.52%, 2.09% and 1.29% performance improvements . |
ReIFE: Re-evaluating Instruction-Following Evaluation (2025.naacl-long)
Copied to clipboard
Yixin Liu, Kejian Shi, Alexander Fabbri, Yilun Zhao, PeiFeng Wang, Chien-Sheng Wu, Shafiq Joty, Arman Cohan
| Challenge: | Existing evaluations of large language models (LLMs) for instruction following are incomplete. |
| Approach: | They propose to use 25 base LLMs and 15 recently proposed evaluation protocols to evaluate instruction following on 4 human-annotated datasets. |
| Outcome: | The proposed evaluations identify the best-performing base LLMs and evaluation protocols with a high degree of robustness. |
Adaptation of Hierarchical Structured Models for Speech Act Recognition in Asynchronous Conversation (N19-1)
Copied to clipboard
| Challenge: | asynchronous domains lack large labeled datasets to train an effective speech act recognition model. |
| Approach: | They propose methods to leverage abundant unlabeled conversational data and available labeled data from synchronous domains to train an effective SAR model. |
| Outcome: | The proposed method outperforms existing methods when trained on in-domain data only. |
Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling (2022.acl-long)
Copied to clipboard
| Challenge: | Prior work on text generation models focused on new architectures for permuted document tasks. |
| Approach: | They propose to use a basic model architecture to improve coherence evaluation of machine generated text. |
| Outcome: | The proposed model improves on a task-independent test set and shows significant improvements in coherence evaluations of downstream tasks. |
From Charts to Fair Narratives: Uncovering and Mitigating Geo-Economic Biases in Chart-to-Text (2025.emnlp-main)
Copied to clipboard
Ridwan Mahbub, Mohammed Saidul Islam, Mir Tafseer Nayeem, Md Tahmid Rahman Laskar, Mizanur Rahman, Shafiq Joty, Enamul Hoque
| Challenge: | Existing VLMs produce more positive descriptions for high-income countries compared to middle- or low-income nations, even when country attribution is the only variable changed. |
| Approach: | They propose to automate the process by generating textual summaries of charts using vision-language models to understand how a country’s economic status influences the sentiment of generated summary. |
| Outcome: | The proposed model amplifys geo-economic biases in 6,000 chart-country pairs from six widely used vision-language models to understand how a country’s economic status influences the sentiment of generated summaries. |
Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges (2024.findings-acl)
Copied to clipboard
Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu, Shafiq Joty
| Challenge: | Data augmentation (DA) is a key technique for enhancing model performance by diversifying training examples without the need for additional data collection. |
| Approach: | They examine various strategies that utilize LLMs for data augmentation, including a novel exploration of learning paradigms where LLM-generated data is used for diverse forms of further training. |
| Outcome: | The proposed approach addresses the primary open challenges faced by LLMs in the field of large language models and aims to serve as a comprehensive guide for researchers and practitioners. |
Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks (2021.eacl-main)
Copied to clipboard
| Challenge: | Coherence models are typically evaluated only on synthetic tasks, which may not be representative of their performance in downstream applications. |
| Approach: | They compare models' performance on synthetic sentences with those on retrieval-based dialog. |
| Outcome: | The proposed models perform poorly on synthetic sentences and retrieval-based dialog tasks. |
Data Selection Curriculum for Neural Machine Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Neural Machine Translation models are typically trained on heterogeneous data that are concatenated and randomly shuffled. |
| Approach: | They propose a two-stage curriculum training framework where a NMT model is fine-tuned on subsets of data, selected by deterministic scoring and online scoring. |
| Outcome: | The proposed framework improves on six language pairs comprising low- and high-resource languages and shows up to +2.2 BLEU improvement and faster convergence. |
DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks (2020.emnlp-main)
Copied to clipboard
Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kruengkrai, Thien Hai Nguyen, Shafiq Joty, Luo Si, Chunyan Miao
| Challenge: | Data augmentation techniques are widely used to improve machine learning performance . however, due to the complexity of language, it is difficult to generalize such rules for languages. |
| Approach: | They propose a method to generate high quality synthetic data for low-resource tagging tasks . they use unlabeled data only and unlabelled data plus a knowledge base . |
| Outcome: | The proposed method outperforms baselines on NER, part of speech and target based sentiment analysis tasks. |
DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are scalable and economical evaluators, but how reliable they are is still under-explored. |
| Approach: | They propose a framework which breaks down the evaluation process into decomposition and aggregation stages based on pedagogical practices and provides an interpretable window for how well LLMs evaluate . |
| Outcome: | The proposed framework improves performance on a variety of meta-evaluation benchmarks by providing an interpretable window for how well LLMs evaluate . |
Discern: Discourse-Aware Entailment Reasoning Network for Conversational Machine Reading (2020.emnlp-main)
Copied to clipboard
Yifan Gao, Chien-Sheng Wu, Jingjing Li, Shafiq Joty, Steven C.H. Hoi, Caiming Xiong, Irwin King, Michael Lyu
| Challenge: | Document interpretation and dialog understanding are the two major challenges for conversational machine reading. |
| Approach: | They propose a discourse-aware entailment reasoning network to strengthen the connection and enhance the understanding of document and dialog. |
| Outcome: | The proposed model improves document interpretation and dialog understanding on the ShARC benchmark. |
Deploying Tiny LVLM Judges for Real-World Evaluation of Chart Models: Lessons Learned and Best Practices (2025.emnlp-industry)
Copied to clipboard
Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub, Mizanur Rahman, Amran Bhuiyan, Israt Jahan, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque, Jimmy Huang
| Challenge: | Large Vision-Language Models (LVLMs) with only 7B parameters perform poorly as judges in resource-constrained settings. |
| Approach: | They propose two approaches to ensure costefficient evaluation by combining multiple criteria into a single query and domainadaptive transfer learning to create a 2Bparameter VLM on a chart dataset. |
| Outcome: | The proposed model can effectively transfer knowledge from one dataset to another to make it a more specialized model. |
Personalized Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that close-sourced LLMs lack the ability to integrate into real-world applications due to their high associated costs and ethical concerns. |
| Approach: | They propose to use student model to refine its own solution by querying ChatGPT to generate task instruction and solution pairs and querying data to refine model. |
| Outcome: | The proposed model outperforms standard distillation with only one third of the data. |
A Unified Neural Coherence Model (D19-1)
Copied to clipboard
| Challenge: | Existing models for coherence modeling fail on harder tasks with more realistic application scenarios. |
| Approach: | They propose a unified coherence model that incorporates sentence grammar, inter-sentence coherent relations, and global coherency patterns into a common neural framework. |
| Outcome: | The proposed model outperforms existing models on local and global discrimination tasks and outperformed existing models by a good margin. |
Adaptation of Large Language Models (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | a tutorial on adaptation of large language models addresses the growing demand for models that go beyond static capabilities. |
| Approach: | This tutorial will provide an overview of dynamic, domain-specific, and task-adaptive LLM adaptation techniques. |
| Outcome: | This tutorial will outline dynamic, domain-specific, and task-adaptive LLM adaptation techniques. |
Discourse Analysis and Its Applications (P19-4)
Copied to clipboard
| Challenge: | Discourse processing is a suite of NLP tasks to uncover linguistic structures from texts at several levels, which can support many downstream applications. |
| Approach: | They present a set of tasks to uncover linguistic structures from texts at several levels, which can support many downstream applications. |
| Outcome: | The tutorial covers the basic concepts of discourse analysis and linguistic structures in monologue vs. conversation, synchronous v. asynchronous conversation, and key linguistic structure in discourse analysis. |
Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learning (2025.acl-long)
Copied to clipboard
Chengwei Qin, Wenhan Xia, Fangkai Jiao, Chen Chen, Yuchen Hu, Bosheng Ding, Ruirui Chen, Shafiq Joty
| Challenge: | Existing methods to train student models on the generated outputs of teacher models are not efficient for ICL. |
| Approach: | They propose to align the output of smaller (student) models with that of larger (teacher) models by incorporating a ranking loss and aligning the token-level output distribution. |
| Outcome: | The proposed model outperforms baseline models on a variety of tasks involving language understanding, reasoning, and coding. |
Differentiable Window for Dynamic Local Attention (2020.acl-main)
Copied to clipboard
| Challenge: | Existing general purpose components for learning differentiable windows are hard to optimize. |
| Approach: | They propose a new neural module and general purpose component for dynamic window selection that can enable more focused attentions over the input regions. |
| Outcome: | The proposed approach improves on a myriad of NLP tasks including machine translation, sentiment analysis, subject-verb agreement and language modeling. |
A Unified Linear-Time Framework for Sentence-Level Discourse Parsing (P19-1)
Copied to clipboard
| Challenge: | a new neural framework for sentence-level discourse analysis is proposed . a discourse segmenter and a parser are based on pointer networks and operate in linear time . |
| Approach: | They propose a neural framework for sentence-level discourse analysis in accordance with Rhetorical Structure Theory . they use a discourse segmenter and a parser to construct a discursive tree in a top-down fashion . |
| Outcome: | The proposed framework surpasses previous approaches on both tasks and human agreement on both. |
DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboards (2026.findings-eacl)
Copied to clipboard
Aaryaman Kartha, Ahmed Masry, Mohammed Saidul Islam, Thinh Lang, Shadikur Rahman, Ridwan Mahbub, Mizanur Rahman, Mahir Ahmed, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty
| Challenge: | Existing question-answering benchmarks for data visualizations focus on static charts instead of interactive dashboards. |
| Approach: | They propose a benchmark to assess how vision-language GUI agents comprehend and interact with real-world dashboards. |
| Outcome: | The first benchmark explicitly designed to assess how vision-language GUI agents comprehend and interact with real-world dashboards. |
Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing continual relation learning methods rely on labeled training data for learning new tasks, which can be expensive and time-consuming. |
| Approach: | They propose a method that embeds space regularization and data augmentation to learn relational patterns with very few labeled data while avoiding catastrophic forgetting of previous task knowledge. |
| Outcome: | The proposed method outperforms existing state-of-the-art methods in CFRL task settings. |
Unsupervised Summarization Re-ranking (2023.findings-acl)
Copied to clipboard
| Challenge: | Abstractive summarization models have been gaining popularity, but performance of unsupervised models still lags behind supervised models. |
| Approach: | They propose to re-rank summary candidates in an unsupervised manner to close the performance gap between unsupervised and supervised models. |
| Outcome: | The proposed model improves unsupervised models by up to 7.27% and ChatGPT by up 6.86% relative mean ROUGE across four widely-adopted summarization benchmarks. |
NLP for Conversations: Sentiment, Summarization, and Group Dynamics (C18-3)
Copied to clipboard
| Challenge: | a tutorial focuses on computational models for conversational structure, summarization and sentiment detection, and group dynamics. |
| Approach: | a tutorial will provide examples of specific NLP tasks for conversational structure, summarization and sentiment detection, and group dynamics. |
| Outcome: | The tutorial focuses on the three areas of conversational structure, summarization and sentiment detection, and group dynamics. |
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization (2022.acl-long)
Copied to clipboard
| Challenge: | Sequence-to-sequence neural networks have enabled great progress in abstractive summarization. |
| Approach: | They propose to train a second-stage model performing re-ranking on a set of summary candidates by using a mixture of experts. |
| Outcome: | The proposed model outperforms the base model on CNN- DailyMail, XSum and Reddit TIFU with a base PEGASUS. |
SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization (2023.emnlp-main)
Copied to clipboard
Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, Chien-Sheng Wu
| Challenge: | Existing factual consistency benchmarks are inadequate to detect factual inconsistencies in LLMs. |
| Approach: | They propose a protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. |
| Outcome: | The proposed method is 20 times more cost-effective per sample and highly reproducible, as it estimates inter-annotator agreement at about 0.9. |
Prompt Leakage effect and mitigation strategies for multi-turn LLM Applications (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Prompt leakage poses a compelling security and privacy threat in LLM applications. |
| Approach: | They propose a model which leverages the LLM sycophancy effect and a threat model which fine tunes an open-source model to defend against prompt leakage attempts. |
| Outcome: | The proposed model elevates the average attack success rate (ASR) from 17.7% to 86.2% in a multi-turn setting. |
Joint Multitask Learning for Community Question Answering Using Task-Specific Embeddings (D18-1)
Copied to clipboard
| Challenge: | Stack-Overflow, Quora, and Yahoo! Answers forums are not moderated, which results in noisy and redundant content. |
| Approach: | They use deep neural networks to learn meaningful task-specific embeddings . they incorporate the embeddables into a conditional random field model . |
| Outcome: | The proposed task improves significantly across evaluation metrics. |
Randomized Smoothing with Masked Inference for Adversarially Robust Text Classifications (2023.acl-long)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are brittle against specifically crafted adversarial examples, leading to increasing interest in probing the adversariality of NLP systems. |
| Approach: | They propose a two-stage framework that combines randomized smoothing and masked inference to improve the adversarial robustness of NLP systems. |
| Outcome: | The proposed framework improves adversarial robustness by 2 to 3 times over existing state-of-the-art methods on benchmark datasets. |
XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have shown impressive abilities in generating codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving relevant code segments. |
| Approach: | They propose to use a multilingual multitask benchmark to evaluate large language models that can generate codes from natural language descriptions, repair buggy codes, and translate between languages. |
| Outcome: | The proposed model performs 7 tasks covering up to 11 languages with execution-level parallelism and 25 M document-level coding examples (16.5 B tokens) |
Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models have shown impressive performance in following natural language instructions to solve unseen tasks. |
| Approach: | They propose two strategies to help large language models better leverage task instructions . they propose to remove 60% of tokens from the task definitions while maintaining model performance . |
| Outcome: | The proposed approach achieves 4.2 Rouge-L improvement over 119 unseen test tasks. |
RST Parsing from Scratch (2021.naacl-main)
Copied to clipboard
| Challenge: | Fig. 1 shows a document level discourse parser that performs top-down end-to-end parsing without requiring segmentation . |
| Approach: | They propose a top-down end-to-end formulation of document level discourse parsing in the Rhetorical Structure Theory framework. |
| Outcome: | The proposed model outperforms existing methods in end-to-end parsing and parse with gold segmentation without handcrafted features. |
Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite (D19-1)
Copied to clipboard
| Challenge: | Currently, machine translation is performed at the level of individual sentences, in isolation from the rest of the document. |
| Approach: | They propose a dataset that can be used as a test suite for pronoun translation . they propose an evaluation measure to differentiate good and bad pronounce translations . |
| Outcome: | The proposed dataset can be used as a test suite for pronoun translation in English . it covers multiple source languages and different pronouner errors drawn from real system translations . |
Domain Adaptation with Adversarial Training and Graph Embeddings (P18-1)
Copied to clipboard
| Challenge: | Existing models for deep neural networks can handle data distributions between source and target domains, but they must deal with data distribution drifts. |
| Approach: | They propose a model that leverages unlabeled and labeled data from a related domain to deal with distribution drifts. |
| Outcome: | The proposed model improves over baselines on two real-world disaster datasets. |
Lost in Translation: Do LVLM Judges Generalize Across Languages? (2026.findings-acl)
Copied to clipboard
Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem, Amran Bhuiyan, Mizanur Rahman, Shafiq Joty, Enamul Hoque, Jimmy Huang
| Challenge: | MM-JudgeBench is the first large-scale benchmark for multilingual and multimodal judge model evaluation. |
| Approach: | They propose a multilingual benchmark for multilingual and multimodal judge model evaluation that includes over 60K pairwise preference instances spanning 25 typologically diverse languages. |
| Outcome: | The proposed benchmark includes over 60K pairwise preference instances spanning 25 languages. |
Demystifying Domain-adaptive Post-training for Financial LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Domain-adaptive post-training of large language models (LLMs) has emerged as promising approach for specialized domains such as medicine and finance. |
| Approach: | They propose a system to identify optimal adaptation criteria and training strategies for LLMs for the finance domain. |
| Outcome: | The proposed model achieves state-of-the-art performance across a wide range of financial tasks. |
Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles (2024.naacl-long)
Copied to clipboard
Kung-Hsiang Huang, Philippe Laban, Alexander Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, Chien-Sheng Wu
| Challenge: | Existing studies on multi-document summarization focus on collating information that all sources agree upon, but the task of summarizing diverse information remains underexplored. |
| Approach: | They propose a task of summarizing diverse information encountered in multiple news articles encompassing the same event using a dataset curated by a large language model. |
| Outcome: | The proposed task aims to summarize diverse information in multiple news articles encompassing the same event . the proposed task is difficult due to its limited coverage and verbosity biases . |
AdaPT: A Set of Guidelines for Hyperbolic Multimodal Multilingual NLP (2024.findings-naacl)
Copied to clipboard
| Challenge: | Euclidean space is used for training neural models and performing arithmetic operations, but many data types have complex geometries and cannot be captured in the Euclidesan space. |
| Approach: | They propose a set of guidelines for initialization, parametrization, and training of neural networks that can be generalized over existing neural network training methodologies. |
| Outcome: | The proposed framework outperforms Euclidean methods on three tasks over 12 languages and modalities on a variety of domains. |
Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization (2024.findings-naacl)
Copied to clipboard
Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, Arman Cohan
| Challenge: | Recent studies have found that large language models (LLMs) can achieve state-of-the-art performance on generic summarization benchmarks, but their performance on more complex summarizing task settings is less studied. |
| Approach: | They benchmark large language models on instruction controllable text summarization . they use 4 evaluation protocols and 11 LLMs to evaluate their performance . |
| Outcome: | The proposed model performs well on instruction controllable text summarization tasks with 4 evaluation protocols and 11 LLMs. |
Modeling What-to-ask and How-to-ask for Answer-unaware Conversational Question Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to generate conversational question are naive and do not account for the answer span. |
| Approach: | They propose a framework for generating a conversational question from a context. |
| Outcome: | The proposed framework achieves state-of-the-art in two different settings compared to existing models . it uses a sentence as the rationale and extracts the answer span from it . |
OpenCQA: Open-ended Question Answering with Charts (2022.emnlp-main)
Copied to clipboard
| Challenge: | OpenCQA is a task to answer open-ended questions about charts with descriptive texts. |
| Approach: | They propose a task to answer open-ended questions about charts with descriptive texts. |
| Outcome: | The proposed task is to answer an open-ended question about a chart with descriptive texts. |
Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning? (2023.acl-long)
Copied to clipboard
| Challenge: | Prompt tuning (PT) based on frozen pre-trained language models has shown remarkable performance in few-shot learning . however, it relies heavily on good initialization of the prompt embeddings. |
| Approach: | They propose to use meta prompt tuning to improve cross-task generalization by learning to initialize prompt embeddings from other relevant tasks. |
| Outcome: | The proposed method outperforms PT on classification tasks, but not multi-task learning. |
Aligning Text, Code, and Vision: A Multi-Objective Reinforcement Learning Framework for Text-to-Visualization (2026.eacl-long)
Copied to clipboard
| Challenge: | Text2Vis systems generate functional code but resulting charts lack semantic alignment and clarity. |
| Approach: | They propose a framework that integrates post-execution feedback with textual accuracy, code validity, and visualization quality. |
| Outcome: | The proposed framework outperforms strong zero-shot and supervised baselines and shows robust generalization to out-of-domain datasets. |
Towards Summary Candidates Fusion (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for abstractive summarization are limited by the quality of the first-stage candidates. |
| Approach: | They propose a method that fuses several summary candidates to produce a novel abstractive second-stage summary. |
| Outcome: | The proposed method improves ROUGE scores and qualitative properties of fused summaries on several summarization datasets. |
ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Charts provide visual representations of data and are used for analyzing information, addressing queries, and conveying insights to others. |
| Approach: | They propose a chart-specific vision-language Instruction-following dataset with 191K instructions and a pipeline model that extracts chart data tables and inputs them into a LLM. |
| Outcome: | The proposed model can solve a wide range of chart-related tasks, achieving state-of-the-art results on four tasks. |
Reliability Testing for Natural Language Processing Systems (2021.acl-long)
Copied to clipboard
| Challenge: | a lack of rigorous testing and ML implicit assumption of identical training and testing distributions may result in systems that discriminate against minorities. |
| Approach: | They argue that reliability testing is needed to address the issue of demographics . they argue that adversarial attacks can be reframed for this goal . |
| Outcome: | The proposed framework will enable rigorous and targeted testing and aid in the enactment and enforcement of industry standards. |
CoHS-CQG: Context and History Selection for Conversational Question Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies focus on single-turn question generation, but few studies have studied the challenges of multiturn QG. |
| Approach: | They propose a two-stage conversational question generation framework that shortens the context and history of the input and calculates relevance scores. |
| Outcome: | The proposed framework achieves state-of-the-art on CoQA in answer-aware and answer-unaware settings. |
Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have raised concerns regarding the hallucination and flaws in their reasoning process. |
| Approach: | They propose a framework to learn planning-based reasoning through Direct Preference Optimization on collected trajectories, which are ranked according to synthesized process rewards. |
| Outcome: | The proposed model surpasses GPT-3.5-Turbo on logical reasoning benchmarks on a set of logically-based reasoning tasks. |
BotSIM: An End-to-End Bot Simulation Framework for Commercial Task-Oriented Dialog Systems (2022.emnlp-demos)
Copied to clipboard
| Challenge: | evaluating and troubleshooting production TOD systems is still a largely manual process requiring large amount of human conversations with the systems. |
| Approach: | They propose a data-efficient end-to-end Bot SIMulation framework for commercial task-oriented dialog systems that can generate user queries and generate semantic-level dialog acts and entities from bot definitions. |
| Outcome: | The proposed framework is able to infer semantic-level dialog acts and entities from bot definitions and generate user queries via model-based paraphrasing. |
Revisiting Adversarial Autoencoder for Unsupervised Word Translation with Cycle Consistency and Improved Training (N19-1)
Copied to clipboard
| Challenge: | Recent work has shown superior performance for non-adversarial methods in more challenging language pairs. |
| Approach: | They propose to use adversarial autoencoder to map monolingual embeddings to a shared space and to put the target encoders as an adversary against the corresponding discriminator. |
| Outcome: | The proposed method is more robust and achieves better performance than previously proposed adversarial and non-adversarial methods. |
Using Clinical Notes with Time Series Data for ICU Management (D19-1)
Copied to clipboard
| Challenge: | Existing work on monitoring patients in ICU has focused on using time series signals from medical instruments. |
| Approach: | They propose to add clinical notes to the time-series data to improve model performance for three benchmark tasks: in-hospital mortality prediction, modeling decompensation, and length of stay forecasting. |
| Outcome: | The proposed model improves on three benchmark tasks: in-hospital mortality prediction, modeling decompensation, and length of stay forecasting. |
ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering (2025.findings-acl)
Copied to clipboard
Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aaryaman Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadikur Rahman, Mehrad Shahmohammadi, Megh Thakkar, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty
| Challenge: | Chart Question Answering systems are limited in their ability to interpret data visually and reason with visual representations. |
| Approach: | They propose a chart-based chart question-answering system that includes 1,341 charts from 99 diverse sources and 1,948 questions in various types. |
| Outcome: | The new benchmark includes 1,341 charts from 99 diverse sources and 1,948 questions in various types. |
Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown impressive results in single-document summarization, but their performance on MDS still leaves room for improvement. |
| Approach: | They propose a topic-guided reinforcement learning approach to improve content selection in MDS . explicit prompting models with topic labels enhances the informativeness, they show . |
| Outcome: | The proposed method outperforms baselines on multi-News and multi-XScience datasets. |
A Conditional Splitting Framework for Efficient Constituency Parsing (2021.acl-long)
Copied to clipboard
| Challenge: | Developing efficient and effective parsing solutions has always been a key focus in NLP. |
| Approach: | They propose a generic seq2seq parsing framework that casts constituency parsers into a series of conditional splitting decisions. |
| Outcome: | The proposed framework outperforms state-of-the-art (SoTA) methods in discourse parsing . it is based on a syntactic and discourse parsed model and is linear in number of nodes . |
AugVic: Exploiting BiText Vicinity for Low-Resource NMT (2021.findings-acl)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) systems often exhibit poor performance due to the lack of large bitext training corpora in low-resource languages. |
| Approach: | They propose a data augmentation framework which exploits the vicinal samples of the given bitext without using extra monolingual data explicitly. |
| Outcome: | The proposed framework can diversify in-domain bitext data with finer level control on four low-resource language pairs. |
LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs (2025.naacl-long)
Copied to clipboard
Do Xuan Long, Ngoc-Hai Nguyen, Tiviatis Sim, Hieu Dao, Shafiq Joty, Kenji Kawaguchi, Nancy F. Chen, Min-Yen Kan
| Challenge: | Using format-following capabilities, state-of-the-art large language models (LLMs) can be leveraged to tailor outputs to specific task formats. |
| Approach: | They propose to define a format bias evaluation metric and establish effective strategies to reduce it. |
| Outcome: | The proposed evaluation reduces the variance in ChatGPT’s performance among wrapping formats from 235.33 to 0.71 (%2) |
Relevant or Random: Can LLMs Truly Perform Analogical Reasoning? (2025.findings-acl)
Copied to clipboard
| Challenge: | Analogical reasoning is a unique ability of humans to address unfamiliar challenges by transferring strategies from relevant past experiences. |
| Approach: | They propose to use self-generated random examples to improve performance on a variety of reasoning tasks by incorporating relevant examples from relevant past experiences. |
| Outcome: | The proposed methods achieve comparable or even better performance on GSM8K with random biological examples. |
Lifelong Event Detection with Embedding Space Separation and Compaction (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing methods for event detection are prone to forgetting due to overlap between memory data and the previously learned embedding space. |
| Approach: | They propose a method that embeds feature distributions away from the previous embedding space and mitigates overfitting by a memory calibration mechanism. |
| Outcome: | The proposed method outperforms existing state-of-the-art methods with extensive experiments. |
Learning Label Modular Prompts for Text Classification in the Wild (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in parameter efficient tuning of pretrained language models have limited performance. |
| Approach: | They propose a label-modular prompt tuning framework for text classification tasks that emulates the transient nature of real-world. |
| Outcome: | The proposed framework outperforms baselines in two formidable settings and shows strong generalisation ability. |
Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to problem-solving for large language models fail to provide accurate reasoning and factual accuracy. |
| Approach: | They propose a framework that leverages fine-tuned critic models to guide reasoning and retrieval processes. |
| Outcome: | The proposed framework outperforms baselines on domain-knowledge-intensive tasks . it can be used to iterate retrieval and reasoning, and improve retrieval relevance . |
Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines (2025.findings-acl)
Copied to clipboard
Do Xuan Long, Duong Ngoc Yen, Do Xuan Trong, Anh Tuan Luu, Kenji Kawaguchi, Shafiq Joty, Min-Yen Kan, Nancy F. Chen
| Challenge: | In-context learning is an important but not fully understood ability of pre-trained large language models. |
| Approach: | They propose a tool that generates two streams of guidelines capturing task language and format distributions and prompts them to define them by prompting. |
| Outcome: | The proposed model improves both strong open- and closed-source LLMs by over 5% in both zero- and few-shot settings. |
Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Math (2026.acl-long)
Copied to clipboard
| Challenge: | Large language model (LLM)-based reasoning systems have recently achieved gold medal-level performance in the IMO 2025 competition . |
| Approach: | They propose a human-annotated step-level verification benchmark that measures step- level verifiers at the frontier. |
| Outcome: | The proposed benchmark outperforms closed-source models in step-level verification and the impact of scaling verifier compute. |
ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild (2025.coling-industry)
Copied to clipboard
| Challenge: | Existing methods for chart understanding and reasoning are weakly aligned and rely on underlying data tables. |
| Approach: | They propose a chart-based understanding and reasoning model that is trained on instruction-tuning data generated directly from chart images. |
| Outcome: | The proposed model achieves state-of-the-art results across 5 benchmarks spanning chart summarization, question answering, and fact-checking. |
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models for Natural Languages (NL) like BERT and GPT have been shown to transfer well to Programming Languages. |
| Approach: | They propose a unified pre-trained encoder-decoder Transformer model that leverages the code semantics conveyed from the developer-assigned identifiers. |
| Outcome: | The proposed model outperforms existing models on understanding and generation tasks and can capture semantic information from code. |
HPE: Answering Complex Questions over Text by Hybrid Question Parsing and Execution (2023.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end neural networks excel at answering natural language questions but fail on complex ones . a proposed framework for question parsing and execution on textual QA is designed to combine the strengths of neural and symbolic methods. |
| Approach: | They propose a framework for question parsing and execution on textual QA . they parse questions into an intermediate representation and use deterministic rules to translate them . |
| Outcome: | The proposed framework outperforms existing methods in supervised, few-shot, and zero-shot settings while preserving its underlying reasoning process. |
Addressing the Vulnerability of NMT in Input Perturbations (2021.naacl-industry)
Copied to clipboard
| Challenge: | Recent advances in NMT have improved translation quality but are vulnerable to input perturbations. |
| Approach: | They propose a method to reduce the effect of noisy inputs by using a Context-Enhanced Reconstruction approach. |
| Outcome: | The proposed approach improves robustness on Chinese-English and French-English translation tasks. |
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2023.acl-long)
Copied to clipboard
Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev
| Challenge: | Existing studies for summarization evaluation exhibit low inter-annotator agreement or lack scale. |
| Approach: | They propose a modified summarization salience protocol based on fine-grained semantic units and a robust summarizing evaluation benchmark. |
| Outcome: | The proposed protocol is based on fine-grained semantic units and allows for high inter-annotator agreement. |
Lifelong Sequence Generation with Dynamic Module Expansion and Adaptation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods focus on maintaining old knowledge while paying little attention to knowledge transfer across tasks. |
| Approach: | They propose to train a model on a sequence of generation tasks to learn new generation patterns while avoiding the forgetting of previous knowledge. |
| Outcome: | The proposed model outperforms existing methods in different settings. |
Efficient Constituency Parsing by Pointing (2020.acl-main)
Copied to clipboard
| Challenge: | Constituency parsing is a core task in natural language processing (NLP) Existing methods for constituency paring are greedy transition-based or globally optimized. |
| Approach: | They propose a constituency parsing model that casts the problem into a series of pointing tasks. |
| Outcome: | The proposed model achieves 92.78 F1 without pre-trained models, which is faster than existing models. |
Response Selection for Multi-Party Conversations with Dynamic Topic Tracking (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing response selection methods focus on a two-party single-conversation scenario. |
| Approach: | They propose a multi-task learning framework that frames response selection as a dynamic topic tracking task to match the topic between the response and relevant conversation context. |
| Outcome: | The proposed framework outperforms existing methods on an Ubuntu IRC dataset in response selection and topic disentanglement tasks. |
Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for language model pretraining use limited knowledge graph data for knowledge-intensive tasks. |
| Approach: | They propose to make better use of multilingual annotations and language agnostic properties of KG triples for pretraining LMs. |
| Outcome: | The proposed models show significant performance improvements on a wide range of knowledge-intensive cross-lingual tasks. |
Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings (2025.acl-long)
Copied to clipboard
| Challenge: | Contextual evaluation is challenging for state-of-the-art judge models . evaluation criteria are often conditional and dependent on practitioner priorities . |
| Approach: | They propose a judge benchmark that evaluates large language models as judges in contexts . they use human annotations and model-based perturbations to build the benchmark . |
| Outcome: | The proposed benchmark aims to evaluate large language models in contexts with 2,000 challenging response pairs. |
CEMTM: Contextual Embedding-based Multimodal Topic Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | CEMTM is a context-enhanced multimodal topic model that can infer coherent topic structures from documents . traditional multimodal topics failed to capture deeper cross-modal interactions . large vision language models (LLMs) and LVLMs have shown remarkable capacity to encode rich semantic knowledge from vast corpora. |
| Approach: | They propose a context-enhanced multimodal topic model that uses tokens to weight contributions to topic inference. |
| Outcome: | The proposed model outperforms unimodal and multimodal benchmarks on six multimodal domains and captures semantics in scientific articles. |
Multimodal Large Language Models for Human-AI Interaction: Foundations, Agents, and Inclusive Applications (2026.eacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents foundations, agentic capabilities, and inclusive applications of multimodal large language models. |
| Approach: | This tutorial presents foundations, agentic capabilities, and inclusive applications of multimodal large language models. |
| Outcome: | This tutorial covers foundations, agentic capabilities, and inclusive applications of multimodal large language models. |
GlobalWoZ: Globalizing MultiWoZ to Develop Multilingual Task-Oriented Dialogue Systems (2022.acl-long)
Copied to clipboard
| Challenge: | Existing multilingual task-oriented dialogue datasets lack high-quality data curation due to the high expense and challenges of human annotation. |
| Approach: | They propose a method that generates a multilingual ToD dataset globalized from an English ToD data set for three unexplored use cases of multilingual toD systems. |
| Outcome: | The proposed method generates a large-scale multilingual ToD dataset globalized from an English ToD data set for three unexplored use cases of multilingual toD systems. |
Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots (2021.naacl-main)
Copied to clipboard
| Challenge: | Multilingual models have demonstrated impressive cross-lingual transfer abilities. |
| Approach: | They propose two strong adversarial attacks that target multilingual models that can handle code-mixed sentences using bilingual dictionaries. |
| Outcome: | The proposed model has a success rate of 89.75% against XLM-R-large, bringing its average accuracy of 79.85 down to 8.18 on XNLI. |