GPT-3

Small Models are Valuable Plug-ins for Large Language Models (2024.findings-acl)

Challenge:	This tutorial aims to bring awareness of the important and emerging research area of open-domain creative generation.
Approach:	They will review recent studies on creative language generation at sentence level as well as longer forms of text.
Outcome:	This paper reviews recent studies on creative language generation at sentence level as well as longer forms of text.

Copied to clipboard

Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chenguang Zhu, Julian McAuley

Evaluating Transformer Language Models on Arithmetic Operations Using Number Decomposition (2022.lrec-1)

Challenge:	Large-scale pre-trained language models are difficult to fine-tune due to their huge weights and limited context length.
Approach:	They propose an approach which allows black-box LLMs to work with locally fine-tuned smaller models, resulting in superior performance on supervised tasks.
Outcome:	The proposed approach overcomes the challenges of poor performance and instability of In-Context Learning (ICL) while reducing the complexity of in-context learning.

Copied to clipboard

Matteo Muffo, Aldo Cocco, Enrico Bertino

“John is 50 years old, can his son be 65?” Evaluating NLP Models’ Understanding of Feasibility (2023.eacl-main)

Challenge:	Large Language Models such as GPT-3 have demonstrated on-the-fly reasoning capabilities in NLP tasks, but they struggle with arithmetic operations.
Approach:	They propose a Transformer Language Model that decomposes numbers in units, tens, and so on . they introduce a pipeline that allows them to perform arithmetic operations between decomposed numbers .
Outcome:	The proposed model improves accuracy in addition, subtractions and multiplication tasks by 63% . the model is fine-tuned to perform arithmetic operations between decomposed numbers .

Copied to clipboard

Himanshu Gupta, Neeraj Varshney, Swaroop Mishra, Kuntal Kumar Pal, Saurabh Arjun Sawant, Kevin Scaria, Siddharth Goyal, Chitta Baral

Causal Reasoning of Entities and Events in Procedural Texts (2023.findings-eacl)

Challenge:	Recent work has found that large-scale language models lack commonsense reasoning ability . a dataset evaluating large-level language models is needed to evaluate their understanding of feasibility .
Approach:	They propose a question-answering dataset that tests understanding of feasibility . they propose to use commonsense reasoning to reason about when an action is feasible .
Outcome:	The proposed dataset shows that state-of-the-art models struggle to answer feasibility questions correctly.

Copied to clipboard

Li Zhang, Hainiu Xu, Yue Yang, Shuyan Zhou, Weiqiu You, Manni Arora, Chris Callison-Burch

Do large language models and humans have similar behaviours in causal inference with script knowledge? (2024.starsem-1)

Challenge:	Existing work on entity state tracking or event reasoning is limited to procedural texts.
Approach:	They propose a benchmark for causal reasoning of event plausibility and entity states . they represent entities as programming languages while prompting language models .
Outcome:	The proposed model outperforms existing models on human reasoning and event reasoning.

Copied to clipboard

Xudong Hong, Margarita Ryzhova, Daniel Biondi, Vera Demberg

Query Generation Using GPT-3 for CLIP-Based Word Sense Disambiguation for Image Retrieval (2023.starsem-1)

Challenge:	Recent studies show pre-trained language models have superior language understanding abilities, including zero-shot causal reasoning.
Approach:	They used a script-based story to manipulate event B in a story which causally depends on a previous event A.
Outcome:	The results show that only recent LLMs, like GPT-3 or Vicuna, correlate with human behavior in the A B condition.

Copied to clipboard

Xiaomeng Pan, Zhousi Chen, Mamoru Komachi

Recursion of Thought: A Divide-and-Conquer Approach to Multi-Context Reasoning with Language Models (2023.findings-acl)

Challenge:	Existing studies show that human-like prompts with quotes benefit both CLIP and GPT-3 as implicit word sense disambiguation components.
Approach:	They propose using the GPT-3 as a query generator for the backend of CLIP as an implicit word sense disambiguation component for the SemEval 2023 shared task Visual Word Sense Disambiguation.
Outcome:	The proposed query generator for CLIP is an implicit word sense disambiguation component for the SemEval 2023 shared task Visual Word Sense Disambiguation (VWSD). human-like prompts adapted for WSD with quotes benefit both CLIP and GPT-3, whereas plain phrases or poorly templated prompts give the worst results.

Copied to clipboard

Soochan Lee, Gunhee Kim

MulZDG: Multilingual Code-Switching Framework for Zero-shot Dialogue Generation (2022.coling-1)

Challenge:	Existing methods to generate intermediate steps (CoT) are limited by the maximum context size due to various reasons.
Approach:	They propose a new inference framework that introduces several special tokens that the models can output to trigger context-related operations.
Outcome:	Extensive experiments with multiple architectures including GPT-3 show that the proposed framework significantly improves LMs’ inference capability.

Copied to clipboard

Yongkang Liu, Shi Feng, Daling Wang, Yifei Zhang

Dialogue Systems & Conversational Agents

Challenge:	Existing zero-shot dialogue generation systems rely on large-scale pre-trained language models.
Approach:	They propose a multilingual learning framework for zero-shot dialogue generation that can transfer knowledge from an English corpus to a non-English corpus with zero samples.
Outcome:	The proposed framework can transfer knowledge from an English corpus to a non-English corpus with zero samples.

Does GPT-3 Generate Empathetic Dialogues? A Novel In-Context Example Selection Method and Automatic Evaluation Metric for Empathetic Dialogue Generation (2022.coling-1)

Copied to clipboard

Young-Jun Lee, Chae-Gyun Lim, Ho-Jin Choi

Dialogue Systems & Conversational Agents

Challenge:	Empathy is a multi-dimensional concept consisting of cognitive and affective aspects.
Approach:	They propose two new in-context example selection methods that utilize emotion and situational information.
Outcome:	The proposed method is effective in measuring the degree of human empathy.

Does GPT-3 Grasp Metaphors? Identifying Metaphor Mappings with Generative Language Models (2023.acl-long)

Copied to clipboard

Lennart Wachowiak, Dagmar Gromann

LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning (2023.acl-short)

Challenge:	Existing approaches to detect whether natural language sequences are metaphoric or literal focus on detecting the transfer of knowledge structures to pre-trained language models.
Approach:	They propose to probe the ability of GPT-3 to detect metaphoric language and predict the metaphor’s source domain without any pre-set domains.
Outcome:	The proposed model generates the correct source domain for a new sample with an accuracy of 65.15% in English and 34.65% in Spanish.

Copied to clipboard

Amirhossein Abaskohi, Sascha Rothe, Yadollah Yaghoobzadeh

Dissecting In-Context Learning of Translations in GPT-3 (2023.findings-emnlp)

Challenge:	Recent advances in pre-trained language models have been limited when fine-tuned on small datasets.
Approach:	They propose to add contrastive learning to prompt-based fine-tuning to improve model performance.
Outcome:	The proposed approach outperforms other methods on multiple text classification benchmarks.

Copied to clipboard

Vikas Raunak, Arul Menezes, Hany Awadalla

How Large Language Models are Transforming Machine-Paraphrase Plagiarism (2022.emnlp-main)

Challenge:	Recent work on fewshot prompting in Large Language Models has focused on selecting the few-shot samples for prompting.
Approach:	They propose a method to add demonstration attributes to prompting in machine translations by perturbations of high-quality in-domain demonstrations.
Outcome:	The proposed method improves upon the zero-shot translation performance of GPT-3, even making it competitive with few-shot prompted translations.

Copied to clipboard

Jan Philip Wahle, Terry Ruas, Frederic Kirstein, Bela Gipp

Text Generation

Challenge:	Autoregressive paraphrasing tools can be used to generate convincing plagiarized texts with minimal effort.
Approach:	They evaluate the detection performance of large autoregressive models for machine-paraphrase generation on scientific articles from arXiv, student theses, and Wikipedia.
Outcome:	The proposed models generate paraphrases indistinguishable from original work and human experts rate the quality of generated examples as high as originals.

RL with KL penalties is better viewed as Bayesian inference (2022.findings-emnlp)

Copied to clipboard

Tomasz Korbak, Ethan Perez, Christopher Buckley

A Guide To Effectively Leveraging LLMs for Low-Resource Text Summarization: Data Augmentation and Semi-supervised Approaches (2025.findings-naacl)

Challenge:	Reinforcement learning (RL) is used in fine-tuning large language models to penalize them for undesirable features of generated sequences.
Approach:	They analyze challenges associated with treating a language model as an RL policy . they find that RL is equivalent to variational inference: approximating a Bayesian posterior .
Outcome:	The proposed approach is flawed because it turns the LM into a degenerate distribution, the authors show . they show that the proposed approach avoids the distribution collapse problem and offers a first-principles derivation for its objective.

Copied to clipboard

Gaurav Sahu, Olga Vechtomova, Issam H. Laradji

Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It (2026.findings-eacl)

Challenge:	Existing approaches for low-resource text summarization use large language models (LLMs) but such models suffer from inconsistent outputs and are difficult to adapt to domain-specific data.
Approach:	They propose two methods to effectively utilize large language models for low-resource text summarization.
Outcome:	The proposed methods synthesize high-quality documents using LLaMA-3-70b-Instruct model . they achieve competitive ROUGE scores as a fully supervised method with 5% of the labeled data.

Copied to clipboard

Seyed Mahed Mousavi, Edoardo Cecchinato, Lucia Horníková, Giuseppe Riccardi

Do GPTs Produce Less Literal Translations? (2023.acl-short)

Challenge:	Using social reasoning benchmarks, we uncover pervasive flaws in both benchmark items and evaluation methodology.
Approach:	They audit three widely used social reasoning benchmarks and identify flaws in their design and evaluation methodology.
Outcome:	The results challenge the validity of current benchmark-based claims about social reasoning in large language models.

Copied to clipboard

Vikas Raunak, Arul Menezes, Matt Post, Hany Hassan

Multilinguality

Challenge:	Large Language Models (LLMs) are general-purpose language models capable of many natural language generation or understanding tasks.
Approach:	They investigate how LLMs differ qualitatively from standard Neural Machine Translation models by measuring literalness and monotonicity.
Outcome:	The proposed models achieve close to state-of-the-art translation performance under few-shot prompting . the results are backed up by human evaluations and a newer MT quality metrics .

Transformer Language Models without Positional Encodings Still Learn Positional Information (2022.findings-emnlp)

Copied to clipboard

Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, Omer Levy

Do language models have coherent mental models of everyday things? (2023.acl-long)

Challenge:	Using positional embeddings, Causal transformer language models learn an implicit notion of absolute positions.
Approach:	They propose to use positional embeddings to encode positional information in transformer language models.
Outcome:	The proposed model learns an implicit notion of absolute positions across datasets, model sizes, and sequence lengths.

Copied to clipboard

Yuling Gu, Bhavana Dalvi Mishra, Peter Clark

Instruction Induction: From Few Examples to Natural Language Task Descriptions (2023.acl-long)

Challenge:	Psychologists and cognitive scientists hypothesize that humans develop mental models of the world, namely internal, conceptual representations of the environment which we base our decisions and actions on.
Approach:	They propose to add a constraint satisfaction layer to the LM's raw predictions to apply commonsense constraints to reduce incoherence.
Outcome:	The proposed extension removes inconsistencies and improves accuracy by 16-20%.

Copied to clipboard

Or Honovich, Uri Shaham, Samuel R. Bowman, Omer Levy

Large language models are few-shot clinical information extractors (2022.emnlp-main)

Challenge:	Large language models can perform unseen tasks by conditioning on a few input-output demonstrations, but task inference is implicit and the ability of models to explicitly reason about it remains unexplored.
Approach:	They propose an instruction induction challenge in which a model is asked to generate a natural language instruction that fits a set of labeled examples.
Outcome:	The proposed model achieves 65.7% of human performance while the original model only reaches 9.8% of human performances.

Copied to clipboard

Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, David Sontag

Information Extraction & Text Mining

Challenge:	a long-running goal of clinical NLP is the extraction of important variables trapped in clinical notes.
Approach:	They propose to use large language models to tackle diverse clinical extraction tasks . they propose to reannote existing CASI datasets to compare their models with clinical text.
Outcome:	The proposed models outperform existing models on few-shot clinical information extraction tasks.

Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts (2022.emnlp-main)

Copied to clipboard

Ben Zhou, Kyle Richardson, Xiaodong Yu, Dan Roth

Generate then Select: Open-ended Visual Question Answering Guided by World Knowledge (2023.findings-acl)

Challenge:	Existing approaches to end-to-end questionanswering assume that pre-trained language can decompose complex tasks into more straightforward sub-tasks.
Approach:	They propose to use distant supervision to train decomposition-based transformers for large-scale parallel news.
Outcome:	The proposed model improves on semantic parsing and on hotpotQA and strategyQA datasets by 20% to 30%.

Copied to clipboard

Xingyu Fu, Sheng Zhang, Gukyeong Kwon, Pramuditha Perera, Henghui Zhu, Yuhao Zhang, Alexander Hanbo Li, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, Dan Roth, Bing Xiang

Multimodality

Challenge:	Open-ended Visual Question Answering (VQA) requires models to reason over visual and natural language inputs using world knowledge.
Approach:	They propose a new VQA pipeline that deploys a generate-then-select strategy guided by world knowledge for the first time.
Outcome:	The proposed pipeline expands the knowledge coverage from in-domain training data by 4.1% on OK-VQA, without additional computation cost.

Hence, Socrates is mortal: A Benchmark for Natural Language Syllogistic Reasoning (2023.findings-acl)

Copied to clipboard

Yongkang Wu, Meng Han, Yutao Zhu, Lei Li, Xinyu Zhang, Ruofei Lai, Xiaoguang Li, Yuanhang Ren, Zhicheng Dou, Zhao Cao

BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology (2023.emnlp-main)

Challenge:	SylloBase is a benchmark for syllogistic reasoning, a critical capability widely required in natural language understanding tasks, such as text entailment and question answering.
Approach:	They propose to use a benchmark to learn syllogistic reasoning on a set of templates and to use them to generate and understand slogisms.
Outcome:	The proposed benchmark covers a complete taxonomy of syllogism reasoning patterns, and contains both automatically and manually constructed samples.

Copied to clipboard

Odhran O’Donoghue, Aleksandar Shtedritski, John Ginger, Ralph Abboud, Ali Ghareeb, Samuel Rodriques

BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks (2021.acl-long)

Challenge:	Large Language Models (LLMs) have impressive capabilities on a wide range of tasks, such as question answering and the generation of coherent text and code.
Approach:	They propose a framework for automatic evaluation of large language models on open-ended planning problems and a dataset of biology protocols with corresponding pseudocode representations.
Outcome:	The proposed framework evaluates an LLM on a dataset of biology protocols with corresponding pseudocode representations.

Copied to clipboard

Jong-Hoon Oh, Ryu Iida, Julien Kloetzer, Kentaro Torisawa

Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection (2022.emnlp-main)

Challenge:	Existing models of NLP are fading away, but new ones are needed to maintain their dominance.
Approach:	They propose a method to pretrain a CNN using Wikipedia data and integrate it with standard TLMs.
Outcome:	The proposed method outperforms the original ALBERT on GLUE tasks and achieves similar performance to SOTA on open-domain QA tasks.

Copied to clipboard

Suchin Gururangan, Dallas Card, Sarah Dreier, Emily Gade, Leroy Wang, Zeyu Wang, Luke Zettlemoyer, Noah A. Smith

Performance and Risk Trade-offs for Multi-word Text Prediction at Scale (2023.findings-eacl)

Challenge:	Language models rely on massive web crawls for diverse text data, but are rife with undesirable content.
Approach:	They analyze newspaper articles written by students from across the country to determine whose language is preferred by a quality filter.
Outcome:	The results show that newspapers from wealthier, educated, and urban zones are more likely to be classified as high quality.

Copied to clipboard

Aniket Vashishtha, S Sai Prasad, Payal Bajaj, Vishrav Chaudhary, Kate Cook, Sandipan Dandapat, Sunayana Sitaram, Monojit Choudhury

Challenge:	Large Language Models (LLMs) generate ethically inappropriate texts even for seemingly innocuous contexts.
Approach:	They propose to use large language models to detect and filter toxic content in text prediction tasks by evaluating their toxicity detection approaches against a manually crafted CheckList of harms.
Outcome:	The proposed methods are compared against a checklist of harms targeted at different groups and different levels of severity in English.

From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds? (2023.findings-acl)

Copied to clipboard

Albert Coil, Vered Shwartz

Memory-assisted prompt editing to improve GPT-3 after deployment (2022.emnlp-main)

Challenge:	Noun compound interpretation is the task of expressing a noun compound in a free-text paraphrase that makes the relationship between the constituent nouns explicit.
Approach:	They propose modifications to the standard task and propose a new task that solves it.
Outcome:	The proposed task solves the standard task of paraphrasing a noun compound in a free-text paraphrase that makes the relationship between the constituent nouns explicit.

Copied to clipboard

Aman Madaan, Niket Tandon, Peter Clark, Yiming Yang

It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2021.naacl-main)

Challenge:	Large LMs such as GPT-3 can commit mistakes that are obvious to humans, such as interpreting “What word is similar to good?” to mean a homophone, while the user intended a synonym.
Approach:	They pair GPT-3 with a growing memory of cases where the model misunderstood the user’s intents, along with user feedback for clarification.
Outcome:	The proposed model can correct misunderstandings on four lexical tasks and two advanced ethical reasoning tasks without retraining.

Copied to clipboard

Timo Schick, Hinrich Schütze

GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation (2021.findings-emnlp)

Challenge:	Pretraining ever-larger language models on massive corpora requires enormous amounts of compute.
Approach:	They propose to convert textual inputs into cloze questions that contain a task description . they also exploit unlabeled data to improve their performance .
Outcome:	The proposed model outperforms GPT-3 with PET/iPET with cloze questions and unlabeled data.

Copied to clipboard

Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, Woomyoung Park

Benchmarking Hallucination in Large Language Models Based on Unanswerable Math Word Problem (2024.lrec-main)

Challenge:	Recent studies report that prompt-based direct classification eliminates the need for fine-tuning but lacks data and inference scalability.
Approach:	They propose a data augmentation technique that leverages large-scale language models to generate real text samples from a mixture of real samples.
Outcome:	The proposed method outperforms existing methods on diverse classification tasks.

Copied to clipboard

YuHong Sun, Zhangyue Yin, Qipeng Guo, Jiawen Wu, Xipeng Qiu, Hui Zhao

Explanation-based Finetuning Makes Models More Robust to Spurious Cues (2023.acl-long)

Challenge:	Large language models (LLMs) are highly effective in various natural language processing tasks, but can produce unreliable conjectures in ambiguous contexts, which is known as hallucination.
Approach:	They propose a method to evaluate LLM hallucination in Question Answering based on the unanswerable math word problem (UMWP) . they combine text similarity and mathematical expression detection to determine whether LLM considers the question unanswered.
Outcome:	The proposed method combines text similarity and mathematical expression detection to determine whether the LLM considers the question unanswerable.

Copied to clipboard

Josh Magnus Ludan, Yixuan Meng, Tai Nguyen, Saurabh Shah, Qing Lyu, Marianna Apidianaki, Chris Callison-Burch

Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021.findings-emnlp)

Challenge:	Large Language Models (LLMs) learn correlations between labels and features that are irrelevant to the task, leading to poor generalization on out-of-distribution data.
Approach:	They propose an explanation-based approach to fine tune large language models to generate a free-text explanation supporting their answer.
Outcome:	The proposed model is more robust against spurious cues in terms of accuracy drop across four classification tasks: ComVE (+1.2), CREAK (+9.1), e-SNLI (+5.4), and SBIC (+6.5).

Copied to clipboard

Ruiqi Zhong, Kristy Lee, Zheng Zhang, Dan Klein

CANarEx: Contextually Aware Narrative Extraction for Semantically Rich Text-as-data Applications (2022.findings-emnlp)

Challenge:	Large pre-trained language models (LMs) have a surprising ability to perform zero-shot learning.
Approach:	They propose to fine-tune pre-trained language models to optimize the zero-shot learning objective by aggregating 43 existing datasets and annotating 441 label descriptions in a question-answering format.
Outcome:	The proposed model outperforms a same-sized QA model and the previous SOTA zero-shot learning system on unseen tasks.

Copied to clipboard

Nandini Anantharama, Simon Angus, Lachlan O’Neill

Mixed-effects transformers for hierarchical adaptation (2022.emnlp-main)

Challenge:	Narrative modelling is a field of active research that conceptualizes narratives as connected entity chains.
Approach:	They propose an alternative narrative extraction approach using semantic role labeling to extract tuples from text, then dimensionality reduction to reduce the space of entities and connections separately.
Outcome:	The proposed approach improves on a text-as-data task and improves accuracy and recall.

Copied to clipboard

Julia White, Noah Goodman, Robert Hawkins

Definitions Matter: Guiding GPT for Multi-label Classification (2023.findings-emnlp)

Challenge:	Language differs dramatically from context to context, but prompting can be ineffective when contexts are sparse, out-of-sample, or extra-textual.
Approach:	They propose a mixed-effects transformer approach for learning hierarchically-structured prefixes to account for structured variation in language use.
Outcome:	The proposed approach can be extended to transformer-based architectures while generalizing well to unseen contexts.

Copied to clipboard

Youri Peskine, Damir Korenčić, Ivan Grubisic, Paolo Papotti, Raphael Troncy, Paolo Rosso

Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness (2024.acl-long)

Challenge:	Recent success of Large Language Models (LLMs) is due to their superior performance on various tasks such as text generation, summarization, question answering, and inductive reasoning.
Approach:	They propose to generate definitions from examples and use them for zero-shot classification and to investigate how an LLM makes use of the definitions.
Outcome:	The proposed method improves the definitions of class labels and improves their understanding of the definition.

Copied to clipboard

Jiuhai Chen, Jonas Mueller

Making Language Models Better Reasoners with Step-Aware Verifier (2023.acl-long)

Challenge:	BSDetector is a method for detecting bad and speculative answers from a pretrained Large Language Model.
Approach:	They propose a method for detecting bad and speculative answers from a pretrained Large Language Model by estimating a confidence score for any output it generated.
Outcome:	Experiments on open-form Question-Answer benchmarks show that BSDetector more accurately identifies incorrect LLM responses than alternative uncertainty estimation procedures.

Copied to clipboard

Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, Weizhu Chen

TestAug: A Framework for Augmenting Capability-based NLP Tests (2022.coling-1)

Challenge:	Large language models have made impressive progress in few-shot learning but still face difficulties in reasoning tasks such as GSM8K.
Approach:	They propose a new approach that uses a verifier to filter out incorrect answers based on a weighted voting scheme to improve reasoning ability of language models.
Outcome:	The proposed approach improves GSM8K reasoning rate by 17.9% to 58.1%.

Copied to clipboard

Guanqun Yang, Mirazul Haque, Qiaochu Song, Wei Yang, Xueqing Liu

Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Text (2023.findings-acl)

Challenge:	Existing work on capability-based testing requires the developer to compose each individual test template from scratch.
Approach:	They propose a capability-based NLP testing framework that requires the developer to only annotate a few test templates while leveraging the GPT-3 engine to generate the majority of test cases.
Outcome:	The proposed framework saves the developer's manual efforts and guarantees the correctness of the generated suites with a validity checker.

Copied to clipboard

Zhun Yang, Adam Ishay, Joohyung Lee

CONAN-MT-SP: A Spanish Corpus for Counternarrative Using GPT Models (2024.lrec-main)

Challenge:	Large language models (LLMs) are robust and general, but their reasoning ability is not at a level to compete with the best models trained for specific natural language reasoning problems.
Approach:	They propose to use large language models as a few-shot semantic parser to convert natural language sentences into a logical form that serves as input for answer set programs.
Outcome:	The proposed model can handle multiple question-answering tasks without requiring retraining for each new task.

Copied to clipboard

María Estrella Vallecillo Rodríguez, Maria Victoria Cantero Romero, Isabel Cabrera De Castro, Arturo Montejo Ráez, María Teresa Martín Valdivia

Multilinguality

Challenge:	a new study evaluates the performance of GPT-based models to generate CNs for hate speech in Spanish . a growing number of social interactions through digital platforms have led to inappropriate behavior .
Approach:	They propose to use GPT-based models to generate CNs for Hate Speech in Spanish . they use the DeepL API to automatically translate the HS segment into Spanish based on the original CN pairs translated into spanish .
Outcome:	The proposed models outperform human models in most instances, the authors say . the results will be made available to the research community .

Harnessing Black-Box Control to Boost Commonsense in LM’s Generation (2023.emnlp-main)

Copied to clipboard

Yufei Tian, Felix Zhang, Nanyun Peng

Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again (2022.findings-emnlp)

Challenge:	Recent years have seen remarkable progress in massively Pre-Trained Language Models such as GPT-3 . however, their generated outputs lack commonsense at times .
Approach:	They propose a framework that steers a frozen Pre-Trained Language Model towards more commonsense generation by training an auxiliary model.
Outcome:	The proposed framework produces plausible outputs that incorporate concepts in a meaningful way.

Copied to clipboard

Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, Yu Su

Fine-tuned LLMs Know More, Hallucinate Less with Few-Shot Sequence-to-Sequence Semantic Parsing over Wikidata (2023.emnlp-main)

Challenge:	Large pre-trained language models (PLMs) such as GPT-3 have shown strong in-context learning capabilities, which are appealing for domains such as biomedicine that feature high and diverse demands of language technologies but also high data annotation costs.
Approach:	They propose to compare the few-shot performance of GPT-3 in-context learning with fine-tuning smaller (i.e., BERT-sized) PLMs on two representative biomedical information extraction tasks: named entity recognition and relation extraction.
Outcome:	The proposed model underperforms on two representative biomedical information extraction tasks.

Copied to clipboard

Silei Xu, Shicheng Liu, Theo Culhane, Elizaveta Pertseva, Meng-Hsi Wu, Sina Semnani, Monica Lam

SQuARe: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created through Human-Machine Collaboration (2023.acl-long)

Challenge:	Large language models can answer many questions correctly, but can also hallucinate and give wrong answers.
Approach:	They propose a question-answering benchmark for Wikidata that uses SPARQL to ground large language models.
Outcome:	The proposed method outperforms the state-of-the-art for QALD-7 by 3.6% in F1 score.

Copied to clipboard

Hwaran Lee, Seokhee Hong, Joonsuk Park, Takyoung Kim, Meeyoung Cha, Yejin Choi, Byoungpil Kim, Gunhee Kim, Eun-Ju Lee, Yong Lim, Alice Oh, Sangchul Park, Jung-Woo Ha

Challenge:	Existing studies focus on coping with social harms that large language models pose . however, discussions on sensitive issues can become toxic even if the users are well-intentioned.
Approach:	They propose to use Korean dataset to test whether LLMs can generate offensive content and propagate prejudices.
Outcome:	The proposed dataset shows that acceptable response generation improves for HyperCLOVA and GPT-3.

FLamE: Few-shot Learning from Natural Language Explanations (2023.acl-long)

Copied to clipboard

Yangqiaoyu Zhou, Yiming Zhang, Chenhao Tan

MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems (2023.findings-emnlp)

Challenge:	Recent work has shown limited utility of natural language explanations in improving classification.
Approach:	They propose a two-stage few-shot learning framework that generates explanations and fine-tunes a smaller model with generated explanations.
Outcome:	The proposed framework increases inference accuracy over strong baselines, but human evaluation reveals that the majority of generated explanations does not adequately justify classification decisions.

Copied to clipboard

Jakub Macina, Nico Daheim, Sankalan Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, Mrinmaya Sachan

When and how to paraphrase for named entity recognition? (2023.acl-long)

Challenge:	Existing models for automatic dialogue tutoring fail to provide accurate feedback or reveal solutions to students too early.
Approach:	They propose a framework to generate one-to-one teacher-student tutoring dialogues by pairing human teachers with a Large Language Model (LLM) they use scaffolding questions and annotations to fine-tune models to be more effective tutors .
Outcome:	The proposed framework can generate 3k one-to-one teacher-student tutoring dialogues grounded in multi-step math reasoning problems.

Copied to clipboard

Saket Sharma, Aviral Joshi, Yiyun Zhao, Namrata Mukhija, Hanoz Bhathena, Prateek Singh, Sashank Santhanam

Information Extraction & Text Mining

Challenge:	Named entity recognition (NER) is a key component underpinning many industrial pipelines for a variety of downstream applications.
Approach:	They propose to use back translation to annotate entity spans in generations and propose a paraphraser with a larger dataset.
Outcome:	The proposed method improves NER performance across different datasets with gold annotations and paraphrasing strength.

Evaluation of Question Generation Needs More References (2023.findings-acl)

Copied to clipboard

Shinhyeok Oh, Hyojun Go, Hyeongdon Moon, Yunsung Lee, Myeongho Jeong, Hyun Seung Lee, Seungtaek Choi

Text Generation

Challenge:	Existing evaluations of QG methods rely on single reference-based similarity metrics . multiple (pseudo) references are more effective for QG evaluation .
Approach:	They propose to paraphrase the reference question for a more robust QG evaluation.
Outcome:	The proposed frameworks show higher correlation with human evaluations than evaluation with a single reference.

Few-Shot Semantic Parsing with Language Models Trained on Code (2022.naacl-main)

Copied to clipboard

Richard Shin, Benjamin Van Durme

Deconstructing In-Context Learning: Understanding Prompts via Corruption (2024.lrec-main)

Challenge:	Large language models can perform semantic parsing with little training data, when prompted with in-context examples.
Approach:	They propose to map natural language to a controlled natural language-like representation . they find that OpenAI Codex performs better on such tasks than equivalent GPT-3 models .
Outcome:	The proposed model performs better on large parsing tasks than GPT-3 models on Overnight and SMCalFlow.

Copied to clipboard

Namrata Shivagunde, Vladislav Lialin, Sherin Muckatira, Anna Rumshisky

Contrastive Learning for Prompt-based Few-shot Language Learners (2022.naacl-main)

Challenge:	Prior work examined how modifying different elements of the prompt can affect model performance, but this limited number of elements made replication challenging.
Approach:	They decompose the entire prompt into four components: task description, demonstration inputs, labels, and inline instructions provided for each demonstration.
Outcome:	The proposed model is robust to minor prompt modifications, but its underlying pre-trained backbone is brittle . previous studies focused on models with fewer than 15 billion parameters or exclusively examined black-box models like GPT-3 or PaLM, making replication challenging.

Copied to clipboard

Yiren Jian, Chongyang Gao, Soroush Vosoughi

Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning (2023.emnlp-main)

Challenge:	a recent study has shown that GPT-3 fine-tuning models with limited examples is effective . a contrastive learning framework clusters inputs from the same class under different augmented “views” and repels those from different classes.
Approach:	They propose a supervised contrastive framework that clusters inputs from the same class under different augmented "views" they combine a contrastive loss with the standard masked language modeling loss in prompt-based few-shot learners .
Outcome:	The proposed framework improves on the state-of-the-art methods in a diverse set of 15 language tasks.

Copied to clipboard

Ximing Lu, Faeze Brahman, Peter West, Jaehun Jung, Khyathi Chandu, Abhilasha Ravichander, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouha Dziri, Jillian Fisher, Bill Lin, Skyler Hallinan, Lianhui Qin, Xiang Ren, Sean Welleck, Yejin Choi

Evaluating the Knowledge Base Completion Potential of GPT (2023.findings-emnlp)

Challenge:	Extreme-scale language models have shown exceptional performance on a variety of language tasks, but the degree of control offered by these models through pure prompting is limited.
Approach:	They propose an inference-time policy adapter which tailors a large base model without fine-tuning it.
Outcome:	The proposed model outperforms baseline methods on five challenging text generation tasks and even over GPT-4.

Copied to clipboard

Blerta Veseli, Simon Razniewski, Jan-Christoph Kalo, Gerhard Weikum

RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs (2023.acl-long)

Challenge:	Language models (LMs) have been proposed for unsupervised knowledge base completion (KBC) however, their ability to do this at scale and with high accuracy remains an open question.
Approach:	They propose to use language models to complete a large public KB, Wikidata, with 90% precision.
Outcome:	The proposed models can extend Wikidata by 27M facts at 90% precision.

Copied to clipboard

Afra Feyza Akyurek, Ekin Akyurek, Ashwin Kalyan, Peter Clark, Derry Tanti Wijaya, Niket Tandon

Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters (2022.emnlp-main)

Challenge:	Despite their success, even the largest language models make mistakes.
Approach:	They propose a framework where one language model can generate critiques to improve its peer's performance.
Outcome:	The proposed framework improves the performance of a fixed model 200 times its size by 10% over other models.

Copied to clipboard

Hongyu Zhao, Hao Tan, Hongyuan Mei

Green & Sustainable NLP

Challenge:	Adapter-tuning is a paradigm that transfers a pretrained language model to downstream tasks . Previously proposed adapters are all feed-forward neural networks .
Approach:	They propose to use tiny-attention attention with extremely small per-head dimensionality as adapters to modify hidden states at each position . they propose to average multiple attention heads' weights during deployment to reduce its inference computation cost.
Outcome:	The proposed adapter outperforms other adapter-tuning methods on the GLUE benchmark . it uses attention with extremely small per-head dimensionality to modify hidden states .

Conditional Natural Language Inference (2023.findings-emnlp)

Copied to clipboard

Youngwoo Kim, Razieh Rahimi, James Allan

Textual Inference

Challenge:	False positive: widely-used feature-attribution explanation models are not suitable for finding conditions when sentences are long and are written independently.
Approach:	They propose a task of conditional natural language inference to explain sentence pairs that provide contradictory (different) information for different conditions.
Outcome:	The proposed model outperforms the full cross-encoder and other baselines in extracting conditions while maintaining comparable accuracy.

FLUTE: Figurative Language Understanding through Textual Explanations (2022.emnlp-main)

Copied to clipboard

Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, Smaranda Muresan

Sharpness-Aware Minimization Improves Language Model Generalization (2022.acl-long)

Challenge:	Figurative language understanding is a recognizing textual entailment task, but lacks data for figurative language.
Approach:	They propose to use a dataset to analyze figurative NLI instances with explanations to improve models' performance.
Outcome:	The proposed dataset can scale up models even for figurative language using human annotations.

Copied to clipboard

Dara Bahri, Hossein Mobahi, Yi Tay

Benchmarking Large Language Model Capabilities for Conditional Generation (2023.acl-long)

Challenge:	Comparatively little work has been done to improve the generalization of language models . recent work shows that Sharpness-Aware Minimization (SAM) can improve generalization without much computational overhead.
Approach:	They propose a Sharpness-Aware Minimization procedure that encourages convergence to flatter minima to improve generalization of language models without much computational overhead.
Outcome:	The proposed Sharpness-Aware Minimization procedure can improve language models without much computational overhead.

Copied to clipboard

Joshua Maynez, Priyanka Agrawal, Sebastian Gehrmann

Multi-Dimensional Evaluation of Text Summarization with In-Context Learning (2023.findings-acl)

Challenge:	Autoregressive and pre-trained large language models have shifted the field from application-specific to generation-based approaches.
Approach:	They propose to adapt existing application-specific generation benchmarks to pre-trained large language models to better suit different tasks.
Outcome:	The proposed models differ in their applicability to different data regimes and their generalization to multiple languages.

Copied to clipboard

Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes, Pengfei Liu, Graham Neubig, Chunting Zhou

Do Large Language Models Know What They Don’t Know? (2023.findings-acl)

Challenge:	In-context learning-based evaluators are competitive with learned evaluation frameworks for text summarization tasks.
Approach:	They propose to use large language models as multi-dimensional evaluators using in-context learning to evaluate text summarization tasks.
Outcome:	The proposed frameworks are competitive with existing frameworks on relevance and factual consistency, the authors show .

Copied to clipboard

Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, Xuanjing Huang

Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity (2022.acl-long)

Challenge:	Large language models (LLMs) have vast knowledge that allows them to excel in various NLP tasks.
Approach:	They propose an automated method to detect uncertainty in the responses of large language models and a dataset to measure their self-knowledge.
Outcome:	The proposed method detects uncertainty in the responses of large language models and provides a novel measure of their self-knowledge.

Copied to clipboard

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, Pontus Stenetorp

SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023.emnlp-main)

Challenge:	Large pretrained language models can generate text classification results that match fully supervised models.
Approach:	They propose to use a few sample training to determine which permutations are performant . they use generative language models to construct an artificial development set .
Outcome:	The proposed model outperforms fully-supervised models in eleven text classification tasks.

Copied to clipboard

Potsawee Manakul, Adian Liusie, Mark Gales

Text Classification via Large Language Models (2023.findings-emnlp)

Challenge:	Existing fact-checking approaches require access to external databases or external databases . a lack of external databases can undermine trust in large language models.
Approach:	They propose a sampling-based approach to fact-check black-box models without external databases.
Outcome:	The proposed approach can be used to fact-check black-box models without external databases . it can detect non-factual and factual sentences and rank passages in terms of factuality .

Copied to clipboard

Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, Guoyin Wang

Few-shot Learning with Multilingual Generative Language Models (2022.emnlp-main)

Challenge:	Large-scale Language Models (LLMs) have shown the ability for in-context learning.
Approach:	They propose a progressive reasoning strategy tailored to addressing complex linguistic phenomena such as intensification, contrast, irony and limited number of tokens allowed in in-context learning.
Outcome:	The proposed model performs better on 4 out of 5 widely-used text-classification benchmarks, while demonstrating comparable performance to SOTA on MR.

Copied to clipboard

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li

Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations (2023.acl-long)

Challenge:	Large-scale generative language models such as GPT-3 are competitive few-shot learners.
Approach:	They train multilingual generative language models on a corpus covering a diverse set of languages and study their few- and zero-shot learning capabilities.
Outcome:	The proposed model outperforms GPT-3 on 171 out of 182 directions with 32 training examples and surpasses the official supervised baseline in 45 directions.

Copied to clipboard

Chenglei Si, Dan Friedman, Nitish Joshi, Shi Feng, Danqi Chen, He He

Power doesn’t reside in size: A Low Parameter Hybrid Language Model (HLM) for Sentiment Analysis in Code-mixed data (2025.emnlp-main)

Challenge:	In-context learning is an important paradigm for adapting large language models to new tasks . but the generalization behavior of ICL remains poorly understood .
Approach:	They characterize the feature biases of large language models by constructing underspecified demonstrations . they find that LLMs exhibit clear feature bias, and they evaluate interventions .
Outcome:	The proposed model prefers the "default" task features over distractor features more often than the base model.

Copied to clipboard

Pavan Sai Balaga, Nagasamudram Karthik, Challa Vishwanath, Raksha Sharma, Rudra Murthy, Ashish Mittal

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022.emnlp-main)

Challenge:	Code-mixed text presents significant challenges for machine learning due to interplay of distinct grammatical structures, effectively forming a hybrid language.
Approach:	They propose a Hybrid Language Model that combines a multilingual encoder and a lightweight decoder to achieve sentiment classification performance comparable to those of fine-tuned Large Language Models.
Outcome:	The proposed model outperforms models trained individually in sentiment detection tasks.

Copied to clipboard

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer

From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models (2023.acl-long)

Challenge:	Large language models can in-context learn by conditioning on a few input-label pairs and making predictions for new inputs.
Approach:	They propose to use ground truth demonstrations to replace labels in demonstrations . they also show that other aspects of the demonstrations are key drivers of endtask performance .
Outcome:	The proposed model outperforms zeroshot inference on a wide range of tasks using ground truth demonstrations.

Copied to clipboard

Julia Mendelsohn, Ronan Le Bras, Yejin Choi, Maarten Sap

CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors (2023.acl-long)

Challenge:	This work sheds light on the theoretical and applied importance of dogwhistles in both NLP and computational social science.
Approach:	They propose a typology of dogwhistles, curate a glossary of over 300 dogwhitles and analyze their usage in historical U.S. politicians’ speeches.
Outcome:	The proposed model identifies dogwhistles and their meanings and shows that harmful content containing dogwhitles avoids toxicity detection.

Copied to clipboard

Peng Li, Tianxiang Sun, Qiong Tang, Hang Yan, Yuanbin Wu, Xuanjing Huang, Xipeng Qiu

Revisiting Relation Extraction in the era of Large Language Models (2023.acl-long)

Challenge:	Large language models pre-trained on massive corpora have shown impressive few-shot learning ability on many NLP tasks.
Approach:	They propose to recast structured output in the form of code instead of natural language and use generative LLMs of code to perform IE tasks.
Outcome:	The proposed method outperforms fine-tuning moderate-size pre-trained models and prompting NL-LLMs under few-shot settings.

Copied to clipboard

Somin Wadhwa, Silvio Amir, Byron Wallace