Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

Speculative Contrastive Decoding (2024.acl-short)

Challenge:	Several approaches to active learning are available, including confidence-based, diversity-based and committee-based.
Approach:	They propose to use a baseline and a skyline to measure the accuracy of the unannotated sample pool.
Outcome:	The proposed model outperforms a random selection baseline and a skyline approach.

Copied to clipboard

Hongyi Yuan, Keming Lu, Fei Huang, Zheng Yuan, Chang Zhou

RDRec: Rationale Distillation for LLM-based Recommendation (2024.acl-short)

Challenge:	Large language models (LLMs) exhibit exceptional performance in language tasks, yet their auto-regressive inference is limited due to high computational requirements and is sub-optimal due to the exposure bias.
Approach:	They propose a decoding approach that leverages predictions from smaller language models to achieve both decoding acceleration and quality improvement.
Outcome:	The proposed method achieves both decoding acceleration and quality improvement on four diverse language tasks.

Copied to clipboard

Xinfeng Wang, Jin Cui, Yoshimi Suzuki, Fumiyo Fukumoto

Isotropy, Clusters, and Classifiers (2024.acl-short)

Challenge:	Existing models that bridge users and items through textual prompts for effective semantic reasoning do not consider the underlying rationales behind interactions, such as user preferences and item attributes.
Approach:	They propose a rationale distillation recommender model that learns rationales generated by a larger language model (LM) by leveraging reviews related to users and items.
Outcome:	The proposed model achieves state-of-the-art (SOTA) performance in top-N and sequential recommendations.

Copied to clipboard

Timothee Mickus, Stig-Arne Grönroos, Joseph Attieh

Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic Tasks (2024.acl-short)

Challenge:	Existing evidence supports and challenges the use of isotropy in embedding spaces.
Approach:	They propose to formalize this connection mathematically and empirically and prove it's true . they argue that isotropy imposes requirements on embedding space that are not compatible with clusters .
Outcome:	The proposed method sheds light on previous studies focusing on anisotropy in embedding spaces.

Copied to clipboard

Andrew Gambardella, Yusuke Iwasawa, Yutaka Matsuo

Simpson’s Paradox and the Accuracy-Fluency Tradeoff in Translation (2024.acl-short)

Challenge:	Despite the generality and far-reaching consequences of large language models, there are still significant limitations making it difficult to apply them to certain tasks.
Approach:	They show that large language models can perform arithmetic tasks more robustly when conditioned on all of the correct higher-order digits.
Outcome:	The proposed model can predict the first digit of n-digit by m-digit multiplication without chain of thought reasoning, but in practice it fails to correctly predict the last digit on n digit by 1-digit multiplikation .

Copied to clipboard

Zheng Wei Lim, Ekaterina Vylomova, Trevor Cohn, Charles Kemp

UltraSparseBERT: 99% Conditionally Sparse Language Modelling (2024.acl-short)

Challenge:	Existing studies suggest that accuracy and fluency should trade off against each other, and that capturing every detail of the source is difficult for human raters to distinguish.
Approach:	They propose to evaluate the relationship between accuracy and fluency at the segment level and to use probabilities to estimate probabilities.
Outcome:	The proposed model relies on human judgments of accuracy and fluency collected in prior work on translation quality estimation.

Copied to clipboard

Peter Belcak, Roger Wattenhofer

SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark (2024.acl-short)

Challenge:	UltraSparseBERT uses 0.3% of its neurons during inference, compared to similar BERT models.
Approach:	They propose a BERT variant that selectively engages just 12 out of 4095 neurons for each layer inference.
Outcome:	The proposed model employs 0.3% of its neurons during inference while performing on par with similar models.

Copied to clipboard

Zhenwen Liang, Kehan Guo, Gang Liu, Taicheng Guo, Yujun Zhou, Tianyu Yang, Jiajun Jiao, Renjie Pi, Jipeng Zhang, Xiangliang Zhang

Natural Language Interfaces

Challenge:	SceMQA focuses on core science subjects including Mathematics, Physics, Chemistry, and Biology.
Approach:	They propose to use SceMQA to evaluate multimodal question answering at college entrance level.
Outcome:	The proposed model provides specific knowledge points for each problem and detailed explanations for each answer.

On the Role of Long-tail Knowledge in Retrieval Augmented Large Language Models (2024.acl-short)

Copied to clipboard

Dongyang Li, Junbing Yan, Taolin Zhang, Chengyu Wang, Xiaofeng He, Longtao Huang, Hui Xue’, Jun Huang

IEPile: Unearthing Large Scale Schema-Conditioned Information Extraction Corpus (2024.acl-short)

Challenge:	Existing RAG methods focus on improving the task performance, without fine-grained process of knowledge.
Approach:	They propose a method that detects long-tail knowledge in large language models by analyzing retrieved documents and enhancing queries indiscriminately with retrieved information.
Outcome:	The proposed method achieves over 4x speedup in average inference time and consistent performance improvement in downstream tasks compared to existing pipelines.

Copied to clipboard

Honghao Gui, Lin Yuan, Hongbin Ye, Ningyu Zhang, Mengshu Sun, Lei Liang, Huajun Chen

Bi-Directional Multi-Granularity Generation Framework for Knowledge Graph-to-Text with Large Language Model (2024.acl-short)

Challenge:	Large Language Models exhibit a significant performance gap in Information Extraction (IE) high-quality instruction data is the vital key for enhancing LLMs' specific capabilities .
Approach:	They propose a bilingual (English and Chinese) IE instruction corpus that contains 0.32B tokens.
Outcome:	The proposed model improves the performance of LLMs for IE with zero-shot generalization.

Copied to clipboard

Haowei Du, Chen Li, Dinghao Zhang, Dongyan Zhao

Code-Switching Can be Better Aligners: Advancing Cross-Lingual SLU through Representation-Level and Prediction-Level Alignment (2024.acl-short)

Challenge:	Existing methods generate whole text based on all KG triples at once and may incorporate incorrect KG Triples for each sentence.
Approach:	They propose a bi-directional multi-granularity generation framework that generates graph-level sentences based on KG triples instead of the whole text at a time.
Outcome:	The proposed framework achieves state-of-the-art in benchmark dataset WebNLG and further analysis shows the efficiency of different modules.

Copied to clipboard

Zhihong Zhu, Xuxin Cheng, Zhanpeng Chen, Xianwei Zhuang, Zhiqi Huang, Yuexian Zou

AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models (2024.acl-short)

Challenge:	Existing code-switching-based cross-lingual spoken language understanding frameworks are limited to low-resource languages.
Approach:	They propose a cross-lingual spoken language understanding framework that leverages both code-switched and original sentences to achieve multi-level alignment.
Outcome:	The proposed framework can achieve multi-level alignment on two benchmarks across ten languages.

Copied to clipboard

Zeyu Liu, Souvik Kundu, Anni Li, Junrui Wan, Lianghao Jiang, Peter Beerel

DDPrompt: Differential Diversity Prompting in Large Language Models (2024.acl-short)

Challenge:	Pre-trained language models have demonstrated commendable performance on various NLP tasks.
Approach:	They propose a Parameter-Efficient Fine-Tuning (PEFT) method that incrementally freezes low-rank matrices during fine-tuning to reduce computation and alleviate over-fitting.
Outcome:	The proposed method achieves state-of-the-art performance with an average improvement of 0.85% on the GLUE benchmark while yielding up to 1.86 improvement as opposed to similar PEFT alternatives.

Copied to clipboard

Lin Mu, Wenhao Zhang, Yiwen Zhang, Peiquan Jin

Monotonic Representation of Numeric Attributes in Language Models (2024.acl-short)

Challenge:	Large Language Models (LLMs) have shown that their reasoning ability can be enhanced through approaches like Chain-of-Thought (CoT) prompting.
Approach:	They propose a method that generates differentially diverse reasoning paths for different types of questions by voting on the optimal prompts.
Outcome:	The proposed method improves LLMs' reasoning ability on complex reasoning tasks by learning from demonstrations while keeping their parameters frozen.

Copied to clipboard

Benjamin Heinzerling, Kentaro Inui

Two Issues with Chinese Spelling Correction and A Refinement Solution (2024.acl-short)

Challenge:	Language models (LMs) can express factual knowledge involving numeric properties such as Karl Popper was born in 1902, but how this information is encoded in the model’s internal representations is not understood well.
Approach:	They propose a method for finding and editing representations of numeric properties such as Karl Popper’s birth year.
Outcome:	The proposed method can express an increasingly late birthyear by patching activations along a “birthyear” direction.

Copied to clipboard

Changxuan Sun, Linlin She, Xuesong Lu

DynaSemble: Dynamic Ensembling of Textual and Structure-Based Models for Knowledge Graph Completion (2024.acl-short)

Challenge:	Existing models contain many spelling errors, resulting in performance bottlenecks and performance issues.
Approach:	They propose to fix the SIGHAN datasets and re-evaluate four representative Chinese Spelling Correction models using the fixed datasets.
Outcome:	The proposed model improves the models in all metrics by notable margins.

Copied to clipboard

Ananjan Nandi, Navdeep Kaur, Parag Singla, Mausam .

Fine-Tuning Pre-Trained Language Models with Gaze Supervision (2024.acl-short)

Challenge:	Existing approaches to Knowledge Graph Completion use textual descriptions of the KG entities and relations to perform the task.
Approach:	They propose a method to combine two popular approaches to Knowledge Graph Completion . structure-based models perform better when gold answer is easily reachable . textual models exploit textual descriptions to give good performance .
Outcome:	The proposed method achieves 6.8 pt MRR and 8.3 pTits@1 gains over the best baseline model for WN18RR dataset.

Copied to clipboard

Shuwen Deng, Paul Prasse, David Reich, Tobias Scheffer, Lena Jäger

Growing Trees on Sounds: Assessing Strategies for End-to-End Dependency Parsing of Speech (2024.acl-short)

Challenge:	Existing pre-trained language models lack a gaze module to exploit cognitive signals.
Approach:	They propose to integrate a gaze module into pre-trained language models at the fine-tuning stage to exploit cognitive signals.
Outcome:	The proposed model improves performance on the GLUE benchmark and standard fine-tuning and text augmentation baselines.

Copied to clipboard

Adrien Pupier, Maximin Coavoux, Jérôme Goulian, Benjamin Lecouteux

Sketch-Guided Constrained Decoding for Boosting Blackbox Large Language Models without Logit Access (2024.acl-short)

Challenge:	Direct dependency parsing of the speech signal is proposed as a way of incorporating prosodic information into the parser and bypassing the limitations of a pipeline approach.
Approach:	They propose to use graph-based parsing and sequence labeling based parses to integrate prosodic information into the parser and bypass limitations of pipeline approaches.
Outcome:	The proposed graph based approach outperforms a pipeline approach on a large treebank of spoken french, despite having 30% fewer parameters.

Copied to clipboard

Saibo Geng, Berkay Döner, Chris Wendler, Martin Josifoski, Robert West

On the Semantic Latent Space of Diffusion-Based Text-To-Speech Models (2024.acl-short)

Challenge:	Constrained decoding is a technique for enforcing constraints on language model outputs.
Approach:	They propose a technique for enforcing constraints on language model outputs . they propose auxiliary model that refines the initial output without constraints .
Outcome:	The proposed approach is able to refine the output of an unconstrained blackbox LLM without access to logits.

Copied to clipboard

Miri Varshavsky-Hassid, Roy Hirsch, Regev Cohen, Tomer Golany, Daniel Freedman, Ehud Rivlin

Learnable Privacy Neurons Localization in Language Models (2024.acl-short)

Challenge:	Denoising Diffusion Models (DDMs) are a powerful generative tool for text-to-speech (TTS) but their semantic capabilities are unknown and control of synthesized speech’s vocal properties remains a challenge.
Approach:	They explore the latent space of frozen TTS models composed of latent bottleneck activations of the DDM’s denoiser and propose methods for finding semantic directions within it.
Outcome:	The proposed methods enable off-the-shelf audio editing without any training, architectural changes or data requirements.

Copied to clipboard

Ruizhe Chen, Tianxiang Hu, Yang Feng, Zuozhu Liu

Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation of Non-Literal Intent Resolution in LLMs (2024.acl-short)

Challenge:	Large Language Models (LLMs) memorize and disclose private information, especially Personally Identifiable Information (PII) concerns regarding privacy and security within human society remain poorly understood.
Approach:	They propose to use learnable binary weight masks to localize PII-sensitive neurons within LLMs by deactivating localized privacy neurons.
Outcome:	The proposed method localizes PII-sensitive neurons across all layers and shows the property of PI I specificity.

Copied to clipboard

Akhila Yerukola, Saujas Vaduguru, Daniel Fried, Maarten Sap

Challenge:	Existing work on discriminative evaluations of large language models has focused on discrimination, but this paper examines their intention understanding by examining their responses to non-literal utterances.
Approach:	They propose a framework to evaluate large language models’ intention understanding by examining their responses to non-literal utterances.
Outcome:	The proposed framework compares large language models' responses to human-like expectations and provides nuanced evaluations of their intention understanding.

Generating Harder Cross-document Event Coreference Resolution Datasets using Metaphoric Paraphrasing (2024.acl-short)

Copied to clipboard

Shafiuddin Rehan Ahmed, Zhiyong Wang, George Baker, Kevin Stowe, James Martin

Soft Self-Consistency Improves Language Models Agents (2024.acl-short)

Challenge:	Existing methods for Cross-Document Event Coreference Resolution (CDEC) are biased towards lexical similarities, limiting a crucial avenue of research in event comprehension.
Approach:	They propose a lexically rich variant of Event Coref Bank Plus (ECB+) for CDEC on symbolic and metaphoric language.
Outcome:	The proposed method avoids the reannotation of expensive coreference links on symbolic and metaphoric language.

Copied to clipboard

Han Wang, Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal

RecGPT: Generative Pre-training for Text-based Recommendation (2024.acl-short)

Challenge:	Current “sample and select” methods rely on majority voting to score answers . however, when tasks have many distinct and valid answers, selection by voting requires a large number of samples.
Approach:	They introduce a method that replaces SC's discontinuous scoring with a continuous score computed from model likelihoods to increase selection even when actions are sparsely distributed.
Outcome:	The proposed method improves performance and efficiency on long-horizon interactive tasks by replacing SC’s discontinuous scoring with a continuous score computed from model likelihoods.

Copied to clipboard

Hoang Ngo, Dat Quoc Nguyen

MTP: A Dataset for Multi-Modal Turning Points in Casual Conversations (2024.acl-short)

Challenge:	Existing models for text-based recommendation lack data sparsity and flexibility to capture fluctuations in user preferences over time.
Approach:	They present the first domain-adapted and fully-trained large language model for text-based recommendation.
Outcome:	The proposed model outperforms baseline models on rating prediction and sequential recommendation tasks.

Copied to clipboard

Gia-Bao Ho, Chang Tan, Zahra Darban, Mahsa Salehi, Reza Haf, Wray Buntine

Dialogue Systems & Conversational Agents

Challenge:	a new problem setting is designed to detect critical moments in conversations . a human-annotated multi-modal dataset is used to classify and detect turning points .
Approach:	They propose a problem setting focusing on turning points in conversations as TPs . they propose MTPC, MTPD, & MTPR tasks to classify and detect turning points .
Outcome:	The proposed model achieves an F1-score of 0.88 in classification and 0.61 in detection . it uses state-of-the-art vision-language models to construct a narrative from the videos .

What Does Parameter-free Probing Really Uncover? (2024.acl-short)

Copied to clipboard

Tommi Buder-Gröndahl

ATLAS: Improving Lay Summarisation with Attribute-based Control (2024.acl-short)

Challenge:	Probing large language models (LLMs) has been criticized for using pre-defined label-laden target labels.
Approach:	They extend a parameter-free probing technique called perturbed masking applied to BERT to examine the relationship between UD and BERT.
Outcome:	The proposed method is compared to the UD formalism for English and shows that it lacks correlations with linguistic theory.

Copied to clipboard

Zhihao Zhang, Tomas Goldsack, Carolina Scarton, Chenghua Lin

EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models (2024.acl-short)

Challenge:	Lay summarisation aims to produce scientific summaries that are comprehensible to non-experts.
Approach:	They propose an abstractive summarisation approach that can control properties contributing to overall "layness" they evaluate ATLAS on a combination of biomedical lay summarization datasets.
Outcome:	The proposed approach outperforms state-of-the-art summarisation metrics on biomedical datasets and shows that it can be discriminatory and emergently influenced.

Copied to clipboard

Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, Zhongyu Wei

Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark (2024.acl-short)

Challenge:	Recent studies have revealed significant deficiencies of LVLMs in understanding visual contents, leaving the gap between current embodied intelligence and large vision-language models (LVLM) .
Approach:	They propose to use a benchmark to evaluate LVLMs' spatial understanding of embodied environments to evaluate their ability to understand visual contents.
Outcome:	The proposed benchmark is derived from embodied scenes and covers 6 spatial relationships from an egocentric perspective.

Copied to clipboard

Niklas Wretblad, Fredrik Riseby, Rahul Biswas, Amin Ahmadi, Oskar Holmström

Programming Languages in NLP

Challenge:	BIRD-Bench was created to model dirty and noisy database values, but it was not created to contain noise and errors in the questions and gold SQL queries.
Approach:	They analysed the distribution and types of noise in the widely used BIRD-Bench benchmark and the impact of noise on models.
Outcome:	The BIRD-Bench benchmark contains ambiguous questions and syntactical errors, but it was not designed to contain noise and errors in the questions and gold SQL queries.

Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval (2024.acl-short)

Copied to clipboard

João Coelho, Bruno Martins, Joao Magalhaes, Jamie Callan, Chenyan Xiong

That’s Optional: A Contemporary Exploration of “that” Omission in English Subordinate Clauses (2024.acl-short)

Challenge:	Existing studies have shown that Transformer-based language models lose information in the middle of input sequences, especially in the context of web document retrieval.
Approach:	They examine position biases at multiple stages of the training pipeline for an encoder-decoder neural retrieval model, namely language model pre-training, contrastive pre- training, and contrastive fine-tuning.
Outcome:	The proposed model generates embeddings that better capture the beginning of the input content, with fine-tuning further aggravating this effect.

Copied to clipboard

Ella Rabinovich

Syntactic Text Processing

Challenge:	Uniform information density (UID) hypothesis posits speakers optimize the communicative properties of their utterances by avoiding spikes in information.
Approach:	They propose to extend the information-uniformity principles by the notion of entropy to estimate the UID manifestations in the usecase of syntactic reduction choices.
Outcome:	The proposed hypothesis is based on the optional omission of the connector "that" in English subordinate clauses.

Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender? (2024.acl-short)

Copied to clipboard

Haozhe An, Christabel Acquaye, Colin Wang, Zongxia Li, Rachel Rudinger

Explainability and Hate Speech: Structured Explanations Make Social Media Moderators Faster (2024.acl-short)

Challenge:	We study whether large language models exhibit race- and gender-based name discrimination in hiring decisions .
Approach:	They propose templatic prompts to LLMs to write an email to a named job applicant informing them of a hiring decision.
Outcome:	The proposed model generates an acceptance or rejection email based on the applicant's first name .

Copied to clipboard

Agostina Calabrese, Leonardo Neves, Neil Shah, Maarten Bos, Björn Ross, Mirella Lapata, Francesco Barbieri

Born Differently Makes a Difference: Counterfactual Study of Bias in Biography Generation from a Data-to-Text Perspective (2024.acl-short)

Challenge:	Existing studies have shown that explanations can support content moderators to make faster decisions, but the benefits of such models have not been studied.
Approach:	They propose to use structured explanations to support content moderators to make faster decisions by 7.4%.
Outcome:	The proposed models lower the speed of real-world moderators by 7.4% compared to generic explanations and are often ignored . previous studies have shown that explanations can support moderator's decision making by detecting violations of policies but the benefits have not been studied .

Copied to clipboard

Biaoyan Fang, Ritvik Dinesh, Xiang Dai, Sarvnaz Karimi

Sign Language Translation with Sentence Embedding Supervision (2024.acl-short)

Challenge:	Current research shows that biographies reflect bias from society such as gender and religions.
Approach:	They propose a method that manipulates the personal attributes of interest while keeping the co-occurring attributes unchanged.
Outcome:	The proposed method expands the analysis of gender-centered bias in text generation.

Copied to clipboard

Hamidullah Yasser, Josef Genabith, Cristina España-Bonet

STREAM: Simplified Topic Retrieval, Exploration, and Analysis Module (2024.acl-short)

Challenge:	State-of-the-art sign language translation systems facilitate learning through gloss annotations when available at scale.
Approach:	They propose to use sentence embeddings of the target sentences at training time that take the role of glosses to supervise the learning process.
Outcome:	The proposed method significantly outperforms gloss-free approaches on German and American sign languages and with mono- and multilingual sentence embeddings and translation systems.

Copied to clipboard

Anton Thielmann, Arik Reuter, Christoph Weisser, Gillian Kant, Manish Kumar, Benjamin Säfken

DocFinQA: A Long-Context Financial Reasoning Dataset (2024.acl-short)

Challenge:	Topic modeling is a widely used technique to analyze large document corpora.
Approach:	They propose a module for topic retrieval, exploration, and analysis that implements multiple intruder-word based topic evaluation metrics.
Outcome:	The proposed module implements multiple intruder-word based topic evaluation metrics and extends existing datasets.

Copied to clipboard

Varshini Reddy, Rik Koncel-Kedziorski, Viet Lai, Michael Krumdick, Charles Lovering, Chris Tanner

Question Answering

Challenge:	Existing work on automating financial numerical reasoning focuses on unrealistically specific document snippets, failing to reflect the broader and more realistic scenarios faced by analysts.
Approach:	They propose a long-document financial QA task that augments 7,437 questions from existing FinQA dataset with full-document context, extending the average context length from under 700 words in FinQA to 123k words in DocFinQA.
Outcome:	The proposed task extends the average context length from under 700 words in FinQA to 123k words in DocFinQA.

MaskLID: Code-Switching Language Identification through Iterative Masking (2024.acl-short)

Copied to clipboard

Amir Hossein Kargaran, François Yvon, Hinrich Schuetze

An Empirical Analysis on Large Language Models in Debate Evaluation (2024.acl-short)

Challenge:	Sentence-level LIDs are classifiers trained on monolingual texts to provide single labels, typically using a softmax layer to turn scores into probabilities.
Approach:	They propose a simple yet effective code-switching language identification method that uses the LID itself to mask features associated with L1 and L2 in the next round.
Outcome:	The proposed method is based on two open-source LIDs based in the FastText architecture and does not require any external resources.

Copied to clipboard

Xinyi Liu, Pinxin Liu, Hangfeng He

Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains (2024.acl-short)

Challenge:	Prior research in automatic debate evaluation relied on pre-trained encoders and the modeling of argument relations and structures.
Approach:	They investigate the capabilities and inherent biases of advanced large language models (LLMs) such as GPT-3.5 and GPT-4 in the context of debate evaluation.
Outcome:	The proposed models outperform state-of-the-art methods on extensive datasets and show that they are more accurate than previous models.

Copied to clipboard

Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan Wang, Brian Thompson

IndicIRSuite: Multilingual Dataset and Neural Information Models for Indian Languages (2024.acl-short)

Challenge:	a new dataset examines whether fine-tuned metrics are robust to domain shifts between training and inference.
Approach:	They use an annotated multidimensional quality metrics dataset to examine whether they are robust to domain shifts between training and inference.
Outcome:	The proposed metrics exhibit a substantial performance drop in the unseen domain scenario compared to metrics that rely on the surface form and pre-trained metrics that are not fine-tuned on MT quality judgments.

Copied to clipboard

Saiful Haq, Ashutosh Sharma, Omar Khattab, Niyati Chhaya, Pushpak Bhattacharyya

AGR: Reinforced Causal Agent-Guided Self-explaining Rationalization (2024.acl-short)

Challenge:	IndicIRSuite is the first attempt at building large-scale Neural Information Retrieval resources for a large number of Indian languages.
Approach:	They introduce Neural Information Retrieval resources for 11 widely spoken Indian Languages from two major Indian language families.
Outcome:	Experiments show that Indic-ColBERT improves on INDIC-MARCO datasets for 11 languages, and that it can be used to improve IR for Indian languages.

Copied to clipboard

Yunxiao Zhao, Zhiqiang Wang, Xiaoli Li, Jiye Liang, Ru Li

Shoulders of Giants: A Look at the Degree and Utility of Openness in NLP Research (2024.acl-short)

Challenge:	Existing rationalization approaches are susceptible to degeneration due to lack of effective control over the learning direction of the model during training.
Approach:	They propose an agent-guided rationalization approach that guides the next step of the model based on its current training state.
Outcome:	The proposed approach outperforms state-of-the-art methods on BeerAdvocate and HotelReview datasets.

Copied to clipboard

Surangika Ranathunga, Nisansa De Silva, Dilith Jayakody, Aloka Fernando

The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Models (2024.acl-short)

Challenge:	We analysed a sample of NLP research papers published in ACL Anthology . we observe a wide language-wise disparity in publicly available NLP-related artefacts .
Approach:	They analysed NLP research papers archived in ACL Anthology to quantify degree of openness and benefit of such an open culture in the NLP community.
Outcome:	The results show that more than 30% of the papers published in ACL Anthology do not release their artefacts publicly.

Copied to clipboard

Noah Siegel, Oana-Maria Camburu, Nicolas Heess, Maria Perez-Ortiz

Naming, Describing, and Quantifying Visual Objects in Humans and LLMs (2024.acl-short)

Challenge:	In many applications of ML systems it is important to understand why the system came to a particular answer.
Approach:	They introduce a faithfulness metric based on counterfactual input edits that takes into account not just the binary label change, but the total shift in the model’s predicted label distribution.
Outcome:	The proposed explanations are more likely to mention factors when they are impactful to the model’s prediction, with the degree of association increasing with model size but varying significantly by task.

Copied to clipboard

Alberto Testoni, Juell Sprott, Sandro Pezzelle

Visual Data in NLP

Challenge:	Recent work has highlighted that speakers display a wide range of variability when asked to utter sentences, resulting in inter-speaker variability but also variability over time for the same speaker.
Approach:	They evaluate Vision & Language Large Language Models (VLLMs) on three categories where humans show great subjective variability concerning the distribution over plausible labels.
Outcome:	The proposed models can mimic human distributions over plausible labels, but fail to assign quantifiers, a task that requires more accurate, high-level reasoning.

Are LLMs classical or nonmonotonic reasoners? Lessons from generics (2024.acl-short)

Copied to clipboard

Alina Leidinger, Robert Van Rooij, Ekaterina Shutova

Reasoning

Challenge:	Recent research on nonmonotonic reasoning has provided evidence of impressive performance and flexible adaptation to machine generated or human critique.
Approach:	They propose to use generics to explain why birds fly and exceptions such as penguins don't fly to maintain stable beliefs on truth conditions of generics.
Outcome:	The proposed task features generics, such as ‘Birds fly’, and exceptions, ‘Penguins don’t fly’.

ConstitutionalExperts: Training a Mixture of Principle-based Prompts (2024.acl-short)

Copied to clipboard

Savvas Petridis, Ben Wedin, Ann Yuan, James Wexler, Nithum Thain

Time Sensitive Knowledge Editing through Efficient Finetuning (2024.acl-short)

Challenge:	Large language models (LLMs) are capable at a variety of tasks given the right prompt, but writing one remains a difficult and tedious process.
Approach:	They propose a method for learning a prompt consisting of constitutional principles, given a training dataset.
Outcome:	The proposed method outperforms other prompt optimization techniques by 10.9% and improves all techniques, suggesting its broad applicability.

Copied to clipboard

Xiou Ge, Ali Mousavi, Edouard Grave, Armand Joulin, Kun Qian, Benjamin Han, Mostafa Arefiyan, Yunyao Li

PRewrite: Prompt Rewriting with Reinforcement Learning (2024.acl-short)

Challenge:	Existing locate-and-edit knowledge editing methods suffer from two limitations: they are infeasible for large scale KE in practice and require long run-time.
Approach:	They propose to use parametric fine-tuning techniques to update obsolete knowledge and induce new knowledge into LLMs.
Outcome:	The proposed methods improve the performance of KE and knowledge update in a temporal dataset with knowledge update and knowledge injection examples.

Copied to clipboard

Weize Kong, Spurthi Hombaiah, Mingyang Zhang, Qiaozhu Mei, Michael Bendersky

Paraphrasing in Affirmative Terms Improves Negation Understanding (2024.acl-short)

Challenge:	Prompt engineering is done manually in a trial-and-error ad-hoc fashion, authors say .
Approach:	They propose a method to rewrite an under-optimized prompt to a more effective prompt.
Outcome:	The proposed method rewrites an under-optimized prompt to a more effective prompt.

Copied to clipboard

MohammadHossein Rezaei, Eduardo Blanco

Exploring Conditional Variational Mechanism to Pinyin Input Method for Addressing One-to-Many Mappings in Low-Resource Scenarios (2024.acl-short)

Challenge:	Negation is a common linguistic phenomenon in human languages . however, language models face challenges with negation in many tasks .
Approach:	They propose to incorporate affirmative interpretations into models to make them more robust against negation.
Outcome:	The proposed models are more robust against negation when negation is present in input . the proposed models can be used to analyze large corpus and natural language understanding tasks .

Copied to clipboard

Bin Sun, Jianfeng Li, Hao Zhou, Fandong Meng, Kan Li, Jie Zhou

Consistency Training by Synthetic Question Generation for Conversational Question Answering (2024.acl-short)

Challenge:	Experimental results demonstrate the superior performance of our method.
Approach:	They propose to leverage conditional variational mechanism to simplify pinyin IME . they employ a strategy that facilitates interaction between pinyan and Chinese character information .
Outcome:	The proposed method improves the performance of pinyin input method engine (IME) under low-resource conditions.

Copied to clipboard

Hamed Hemati, Hamid Beigy

Natural Language Interfaces

Challenge:	Existing methods that use historical information to address user queries in conversational question-answering (CQA) contexts use the gold answers of history instead of the predicted ones.
Approach:	They propose a model-agnostic approach that augments historical information with synthetic questions and employs consistency training to implicitly make the reasoning robust to irrelevant history.
Outcome:	The proposed model improves in later turns of the conversation when dealing with questions with a large historical context.

How Good is Zero-Shot MT Evaluation for Low Resource Indian Languages? (2024.acl-short)

Copied to clipboard

Anushka Singh, Ananya Sai, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh Khapra

Low-Resource NLP

Challenge:	a recent study focused on machine translation evaluation for low-resource languages . linguistic aspects that vary across languages are factors that will exacerbate the problem in low-source languages due to the reliance on extensive data resources.
Approach:	They propose to use multi-dimensional quality metrics and DA annotations to meta-evaluate MT evaluation metrics for low-resource languages.
Outcome:	The proposed evaluation metrics are based on human scores on the candidate translations of assamese, maithili, and Punjabi.

Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languages (2024.acl-short)

Copied to clipboard

Mofetoluwa Adeyemi, Akintunde Oladipo, Ronak Pradeep, Jimmy Lin

Cross-Modal Projection in Multimodal LLMs Doesn’t Really Project Visual Attributes to Textual Space (2024.acl-short)

Challenge:	Large language models (LLMs) have shown impressive zero-shot capabilities in various passage ranking tasks.
Approach:	They analyze and compare the effectiveness of monolingual reranking using query or document translations and evaluate the effectiveness when leveraging their own generated translations.
Outcome:	The proposed models perform better when using their own translations than when using query or document translations.

Copied to clipboard

Gaurav Verma, Minje Choi, Kartik Sharma, Jamelle Watson-Daniels, Sejoon Oh, Srijan Kumar

Guidance-Based Prompt Data Augmentation in Specialized Domains for Named Entity Recognition (2024.acl-short)

Challenge:	Existing multimodal large language models are limited to general-purpose multimodal tasks like question-answering on natural images.
Approach:	They propose to use cross-modal projection networks and a large language model to model domain-specific visual attributes of MLLMs.
Outcome:	The proposed models gain domain-specific visual capabilities when the projection is fine-tuned, but the updates do not extract relevant domain-specific visual attributes.

Copied to clipboard

Hyeonseok Kang, Hyein Seo, Jeesu Jung, Sangkeun Jung, Du-Seong Chang, Riwoo Chung

Low-Resource NLP

Challenge:	specialized fields such as science and biology face significant challenges due to the scarcity of quality data.
Approach:	They propose a guidance data augmentation technique that abstracts context and sentence structure and maintains context-entity relationships for DA.
Outcome:	The proposed method enhances the training performance of named entity recognition tasks while maintaining context-entity relationships.

Aligning Large Language Models via Fine-grained Supervision (2024.acl-short)

Copied to clipboard

Dehong Xu, Liang Qiu, Minseok Kim, Faisal Ladhak, Jaeyoung Do

Annotating FrameNet via Structure-Conditioned Language Generation (2024.acl-short)

Challenge:	Pre-trained large-scale language models often generate biased or toxic text, misaligning with human intentions.
Approach:	They propose to use human feedback to improve LLM alignment by fine-grained token supervision . they ask annotators to edit less preferred responses to make them more favorable .
Outcome:	The proposed method improves LLM alignment by up to 5.1% in terms of win rate compared with the traditional model.

Copied to clipboard

Xinyue Cui, Swabha Swayamdipta

DUAL-REFLECT: Enhancing Large Language Models for Reflective Translation through Dual Learning Feedback Mechanisms (2024.acl-short)

Challenge:	despite the remarkable generative capabilities of language models, their effectiveness on explicit manipulation and generation of linguistic structures remains understudied.
Approach:	They propose a framework to generate frame-semantically annotated sentences following FrameNet . they use explicit semantic information to generate frames with high human acceptance .
Outcome:	The proposed framework produces frame-semantic annotations with high human acceptance . generating high-quality, semantically rich data is effective in low-resource settings, but not under higher resource settings.

Copied to clipboard

Andong Chen, Lianzhang Lou, Kehai Chen, Xuefeng Bai, Yang Xiang, Muyun Yang, Tiejun Zhao, Min Zhang