Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Student Research Workshop)

53 papers
Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation (2025.naacl-srw)

Copied to clipboard

Challenge: Large language models excel in machine translation, but most studies focus on sentence-level translation.
Approach: They propose to use LLMs as a judge paradigm to evaluate document-level translations by directly prompting them to translate entire documents in a single pass.
Outcome: The proposed method improves translation quality even without document-level fine-tuning compared to translating sentences separately .
INSIGHTBUDDY-AI: Medication Extraction and Entity Linking using Pre-Trained Language Models and Ensemble Learning (2025.naacl-srw)

Copied to clipboard

Challenge: InsightBuddy-AI is a system for extracting medication mentions and their associated attributes.
Approach: They propose a system for extracting medication mentions and their associated attributes . they use stacked and voting ensembles built upon pre-trained language models .
Outcome: The proposed system outperforms fine-tuned models in the extraction of medication mentions and associated attributes.
Linguistic Features in German BERT: The Role of Morphology, Syntax, and Semantics in Multi-Class Text Classification (2025.naacl-srw)

Copied to clipboard

Challenge: a monolingual German BERT model is used for semantic classification of newspaper articles . linguistic features identified in English affect classification in German, but suggest important language- and task-specific features as well.
Approach: They examine a monolingual German BERT model using a semantic classification task on newspaper articles.
Outcome: The proposed model uses the TüBa-D/Z corpus, a resource with gold-standard annotations for linguistic features.
Thesis Proposal: Uncertainty in Knowledge Graph Embeddings (2025.naacl-srw)

Copied to clipboard

Challenge: Knowledge graph embedding (KGE) methods map entities and relations from knowledge graphs into numerical vector spaces.
Approach: They propose to investigate various types of uncertainty in knowledge graph embedding methods and explore strategies to quantify, mitigate, and reason under uncertainty effectively.
Outcome: The proposed methods have shown to be reliable in high-stakes domains and provide greater confidence in their use beyond benchmark datasets.
Detecting Sexism in Tweets: A Sentiment Analysis and Graph Neural Network Approach (2025.naacl-srw)

Copied to clipboard

Challenge: a new tool to detect sexism on social media platforms is being developed to identify such behavior . sexist ideologies such as sextism and gender-based violence can be spread through social media .
Approach: They propose to use BERT and GraphSAGE to analyze tweets for sexism detection . they also use sentiment analysis and natural language processing techniques to classify tweets .
Outcome: The proposed tool analyzes tweets for sexism detection and classifies them into five categories.
Towards Codec-LM Co-design for Neural Codec Language Models (2025.naacl-srw)

Copied to clipboard

Challenge: Neural codec language models (or codec LMs) are emerging as a powerful framework for text-to-speech (TTS) despite the close interdependence of codecs and LM, research on codec and lms has largely remained siloed.
Approach: They propose a frame-wise codec encoder that improves both LM log-likelihood and TTS metrics . they also propose LM codebook level dropout to efficiently navigate a portion of codec-LM design space .
Outcome: The proposed codec-LM co-design improves intelligibility, audio quality and speaker control compared to a siloed baseline.
Low-resource Machine Translation for Code-switched Kazakh-Russian Language Pair (2025.naacl-srw)

Copied to clipboard

Challenge: Existing methods to train machine translation models for low-resource languages are not available.
Approach: They propose to build a machine translation model for code-switched Kazakh-Russian language pair with no labeled data.
Outcome: The proposed method beats an existing commercial system by human evaluation on a Kazakh-Russian language pair with no labeled data.
Generative Product Recommendations for Implicit Superlative Queries (2025.naacl-srw)

Copied to clipboard

Challenge: Existing retrieval and ranking systems struggle with implicit superlative queries . lack of explicit attribute mentions and complexity of the query complicates ranking .
Approach: They propose a four-point schema for annotating the best product candidates for superlative queries . they propose pointwise, deliberated pointwise and pairwise methods to analyze the results .
Outcome: The proposed schema can be used to rank products with implicit attributes and reason over them.
ConQuer: A Framework for Concept-Based Quiz Generation (2025.naacl-srw)

Copied to clipboard

Challenge: Existing concepts-based quiz generation frameworks that leverage external knowledge sources are challenging and labor intensive.
Approach: They propose a concept-based quiz generation framework that leverages external knowledge sources to assess the quality of the generated quizzes, using LLMs as judges.
Outcome: The proposed framework shows a 4.8% improvement in evaluation scores and a 77.52% win rate in pairwise comparisons against baseline quiz sets.
What is it? Towards a Generalizable Native American Language Identification System (2025.naacl-srw)

Copied to clipboard

Challenge: Despite their cultural and historical significance, Native American languages remain unsupported by major commercial language identification systems.
Approach: They propose to curate linguistic resources across all Native American languages for robust training and tailor data augmentation to generate synthetic yet linguistically coherent training samples.
Outcome: The proposed system would be generalizable across all Native American languages . it would also generate coherent training samples for low-resource languages based on Plains Apache .
Med-CoDE: Medical Critique based Disagreement Evaluation Framework (2025.naacl-srw)

Copied to clipboard

Challenge: Existing evaluation methods for large language models lack robustness and accuracy in medical contexts.
Approach: They propose an evaluation framework for medical LLMs that measures disagreement between model-generated responses and established medical ground truths.
Outcome: The proposed evaluation framework captures accuracy and reliability in medical settings.
Sentimatic: Sentiment-guided Automatic Generation of Preference Datasets for Customer Support Dialogue System (2025.naacl-srw)

Copied to clipboard

Challenge: Existing methods for supervised fine-tuning and preference optimization (PO) rely on human-authored evaluation criteria for practical application.
Approach: They propose a method to generate customer preference datasets without human intervention using a publicly available dataset constructed for SFT.
Outcome: The proposed method classifies responses by sentiment, fine-tunes models on them, and applies advanced sampling and evaluation techniques to ensure diversity and quality.
Privacy-Preserving Federated Learning for Hate Speech Detection (2025.naacl-srw)

Copied to clipboard

Challenge: a federated learning system with differential privacy is tailored to low-resource languages . data with fewer than 20 sentences per client struggled due to excessive noise .
Approach: They propose a federated learning system with differential privacy for hate speech detection . they fine-tuned pre-trained language models to find it to be the most effective .
Outcome: The proposed learning system outperforms other models in low-resource languages . balanced datasets and augmenting hateful data with non-hateful examples proved critical .
From Annotation to Adaptation: Metrics, Synthetic Data, and Aspect Extraction for Aspect-Based Sentiment Analysis with Large Language Models (2025.naacl-srw)

Copied to clipboard

Challenge: Using a synthetic sports feedback dataset, we evaluate open-weight LLMs’ ability to extract aspect-polarity pairs.
Approach: They propose a metric to facilitate the evaluation of aspect extraction with generative models.
Outcome: The proposed metric improves the performance of open-weight LLMs in the Aspect-Based Sentiment Analysis task.
Developing Japanese CLIP Models Leveraging an Open-weight LLM for Large-scale Dataset Translation (2025.naacl-srw)

Copied to clipboard

Challenge: lack of large-scale open Japanese image-text pairs poses a significant barrier to the development of vision-language models.
Approach: They construct large-scale Japanese image-text pairs using machine translation and pre-trained CLIP models on a Japanese dataset.
Outcome: The results show that pre-trained models achieve competitive average scores on Japanese culture tasks compared to models of similar size.
Self-Vocabularizing Training for Neural Machine Translation (2025.naacl-srw)

Copied to clipboard

Challenge: Past vocabulary learning techniques identify relevant vocabulary before training, relying on corpus statistics or frequency counts without considering contextual information or the model's ability to represent it.
Approach: They propose a method that self-vocabularizes a smaller, more optimal vocabulary by pairing source sentences with the model's predictions to define a new vocabulary.
Outcome: The proposed method produces a 1.49 BLEU improvement in the simulated model and an increase in unique token usage and a 6–8% reduction in vocabulary size.
CCT-Code: Cross-Consistency Training for Multilingual Clone Detection and Code Search (2025.naacl-srw)

Copied to clipboard

Challenge: clone detection is crucial in software development for identifying semantically similar code . clones can be found in the same language code snippets, but there is little research on multilingual clonage detection.
Approach: They propose a novel training procedure leveraging cross-lingual similarity to train language models on source code in various programming languages.
Outcome: The proposed method achieves state-of-the-art on C++ and Python clone detection benchmarks with comparable performance on decoder-based models.
Text Compression for Efficient Language Generation (2025.naacl-srw)

Copied to clipboard

Challenge: Existing models rely on sub-word tokens for text generation, but there is no evidence for a more efficient way to generate text.
Approach: They propose a hierarchical transformer language model capable of text generation by compressing text into sentence embeddings and employing a sentence attention mechanism.
Outcome: The proposed model achieves an up to an order of magnitude improvement in FLOPs efficiency and a threefold increase in runtime speed compared to equally-sized models in the low-size regime.
Multilingual Native Language Identification with Large Language Models (2025.naacl-srw)

Copied to clipboard

Challenge: Native Language Identification (NLI) is the task of automatically identifying the native language (L1) of individuals based on their second language production.
Approach: They evaluated the performance of several LLMs on non-English NLI corpora compared to traditional statistical machine learning models and language-specific BERT-based models.
Outcome: The proposed models outperform statistical models and language-specific BERT-based models on English, Italian, Norwegian, and Portuguese.
Generating Synthetic Free-text Medical Records with Low Re-identification Risk using Masked Language Modeling (2025.naacl-srw)

Copied to clipboard

Challenge: Existing methods to generate medical records using Causal Language Modelling are limited due to privacy concerns.
Approach: They propose a method for generating medical records using Masked Language Modelling using Causal language models.
Outcome: The proposed method produces high-quality synthetic data with a re-identification risk of only 3.5% and a patient recall of 96%.
How many words does it take to understand a low-resource language? (2025.naacl-srw)

Copied to clipboard

Challenge: We evaluated the documentation needed to create a sentence embedding space using widely spoken languages.
Approach: They propose to use widely spoken languages as a proxy for low-resource languages to evaluate the documentation needed to create a sentence embedding space.
Outcome: The proposed language model can be used to improve the performance of sentences embedded in low-resource languages.
Linear Relational Decoding of Morphology in Language Models (2025.naacl-srw)

Copied to clipboard

Challenge: Recent work has shown that affine transformations on subject representations can faithfully approximate model outputs for certain subject-object relations.
Approach: They propose to use affine transformations to adapt the Bigger Analogy Test Set to test faithfulness of morphological relations.
Outcome: The proposed method achieves 90% faithfulness on morphological relations, with similar findings across languages and models.
SPY: Enhancing Privacy with Synthetic PII Detection Dataset (2025.naacl-srw)

Copied to clipboard

Challenge: Historically, Named Entity Recognition (NER) has been employed for PII detection, but PI I entities constitute a subset of NER entities.
Approach: They propose to use Large Language Models to generate a synthetic dataset that emulates real-world PII scenarios and validate its quality.
Outcome: The proposed dataset is validated and provides a benchmark for PII detection.
Tighter Clusters, Safer Code? Improving Vulnerability Detection with Enhanced Contrastive Loss (2025.naacl-srw)

Copied to clipboard

Challenge: a recent study shows that supervised contrastive learning (SCL) improves embedding separation but struggles with intra-class clustering . code vulnerability detection is a cornerstone of software security, especially as the world undergoes rapid digitization.
Approach: They propose an extension of SCL with a distance-based regularization term that tightens intra-class clustering while maintaining inter-class separation.
Outcome: The proposed method improves F1 score on CodeBERT and GraphCodeBERT with BCE, BCE + SCL, and BCE+ CESCL.
Text Extraction and Script Completion in Images of Arabic Script-Based Calligraphy: A Thesis Proposal (2025.naacl-srw)

Copied to clipboard

Challenge: despite its artistic elements, Arabic calligraphy is difficult to read, even for those fluent in Arabic.
Approach: They analyze the variability in calligraphic styles and the influence of artistic distortions to improve text extraction and script completion.
Outcome: The proposed methods improve text extraction and script completion in Arabic calligraphy . the authors show that the proposed techniques are more efficient than traditional methods .
Subasa - Adapting Language Models for Low-resourced Offensive Language Detection in Sinhala (2025.naacl-srw)

Copied to clipboard

Challenge: A major challenge in the field of NLP are the disparities between high- and low-resource languages.
Approach: They propose fine-tuning strategies that have not been previously explored for Sinhala in the downstream task of offensive language detection.
Outcome: The proposed models outperform baseline models on the Sinhala offensive language detection task.
Integrating Symbolic Execution into the Fine-Tuning of Code-Generating LLMs (2025.naacl-srw)

Copied to clipboard

Challenge: Code-generating Large Language Models (LLMs) have become essential tools in modern software development, enhancing productivity and accelerating development.
Approach: They propose to use Reinforcement Learning and Direct Preference Optimization to fine-tune code-generating Large Language Models (LLMs) by enhancing the training data with symbolic execution techniques.
Outcome: The proposed model improves on the CodeRL benchmark and shows that it is more accurate and objective than the baseline model.
Through the Looking Glass: Common Sense Consistency Evaluation of Weird Images (2025.naacl-srw)

Copied to clipboard

Challenge: Existing methods to measure image common sense inconsistentness are difficult to implement because of their complexity.
Approach: They propose a visual commonsense model that leverages large vision-language models to extract atomic facts from images and a compact attention-pooling classifier to fine-tune it over encoded atomic fact.
Outcome: The proposed method outperforms existing methods on the WHOOPS! and WEIRD datasets while maintaining a compact attention-pooling classifier over encoded atomic facts.
ColorFoil: Investigating Color Blindness in Large Vision and Language Models (2025.naacl-srw)

Copied to clipboard

Challenge: Several studies indicate a lack of robustness of the models when dealing with complex linguistics and visual attributes.
Approach: They propose a new V&L benchmark by creating color-related foils to assess the models’ perception ability to detect colors like red, white, green, etc.
Outcome: The proposed benchmark evaluates seven state-of-the-art V&L models including CLIP, ViLT, GroupViT, and BridgeTower in a zero-shot setting and demonstrates that they have better color perception capabilities than CLIP and its variants and GroupVit.
Towards Practical and Knowledgeable LLMs for a Multilingual World: A Thesis Proposal (2025.naacl-srw)

Copied to clipboard

Challenge: a proposed thesis examines the role that multilinguality occupies in the development of practical and knowledgeable LLMs.
Approach: They propose to use multilingual knowledge to improve LLM performance on NLP tasks . they extend the territorial disputes benchmark to retrieval-augmented generation setting .
Outcome: The proposed methods improve LLMs' performance on standard natural language processing tasks by leveraging their existing multilingual knowledge.
MDC3: A Novel Multimodal Dataset for Commercial Content Classification in Bengali (2025.naacl-srw)

Copied to clipboard

Challenge: Identifying commercial posts in resource-constrained languages remains a challenge for automatic text classification tasks.
Approach: They propose a dataset for Bengali social media posts classified as commercial and noncommercial . they include an annotation guideline to aid future dataset creation in resource-constrained languages .
Outcome: The proposed dataset is based on an annotation guideline for future dataset creation in resource-constrained languages.
DateLogicQA: Benchmarking Temporal Biases in Large Language Models (2025.naacl-srw)

Copied to clipboard

Challenge: DateLogicQA examines temporal biases in Large Language Models (LLMs) 190 questions are curated by humans to examine temporal reasoning across date formats and contexts .
Approach: They propose a human-curated benchmark of 190 questions specifically designed to understand temporal bias in Large Language Models.
Outcome: The proposed dataset covers seven date formats across past, present, and future contexts . it examines four reasoning types: commonsense, factual, conceptual, and numerical .
AMR-RE: Abstract Meaning Representations for Retrieval-Based In-Context Learning in Relation Extraction (2025.naacl-srw)

Copied to clipboard

Challenge: Existing in-context learning methods for relation extraction often overlook entity relationships . Existing methods for RE prioritize language similarity over structural similarity .
Approach: They propose an AMR-enhanced retrieval-based ICL method for relation extraction . their method retrieves in-context examples based on semantic structure similarity .
Outcome: The proposed method outperforms baselines on four English RE datasets and in the more demanding unsupervised setting.
Linguistic Analysis of Veteran Job Interviews to Assess Effectiveness in Translating Military Expertise to the Civilian Workforce (2025.naacl-srw)

Copied to clipboard

Challenge: Using natural language processing, veterans are more likely to be hired than general population.
Approach: They conduct NLP experiments to evaluate the degree of explanation in veteran job interview responses as a proxy for perceived hireability.
Outcome: The results show that the proposed method is robust to linguistic features and features.
MetaMeme: A Dataset for Meme Template and Meta-Category Classification (2025.naacl-srw)

Copied to clipboard

Challenge: a new dataset for classifying memes by their template and communicative intent is presented.
Approach: They propose a new dataset for classifying memes by their template and communicative intent.
Outcome: The proposed method outperforms existing methods in classifying memes by their template and communicative intent.
Representing and Clustering Errors in Offensive Language Detection (2025.naacl-srw)

Copied to clipboard

Challenge: Sentence-BERT embeddings of Large Language Model (LLM)-generated linguistic features give the most interpretable clustering for Arabic errors.
Approach: They evaluate the K-Means clustering of four text representations for the task of offensive language detection in English and Levantine Arabic.
Outcome: The proposed clustering of four text representations for offensive language detection in English and Levantine Arabic gives the most human-interpretable clustering for English errors and the grouping is mainly based on the targeted group in the text.
ELIOT: Zero-Shot Video-Text Retrieval through Relevance-Boosted Captioning and Structural Information Extraction (2025.naacl-srw)

Copied to clipboard

Challenge: Recent advances in video-text retrieval (VTR) have relied on supervised learning and fine-tuning.
Approach: They propose a zero-shot video-text retrieval framework that leverages off-the-shelf captioners, large language models, and text retrieval methods without additional training or annotated data.
Outcome: The proposed framework outperforms existing methods on video-text retrieval benchmarks without data.
Can Large Language Models Advance Crosswalks? The Case of Danish Occupation Codes (2025.naacl-srw)

Copied to clipboard

Challenge: Large Language Models (LLMs) are used to map classification systems to each other . however, their use is labor-intensive and requires domain expertise .
Approach: They propose a prompt-based framework where LLMs perform similarity assessments between classification codes and identify final mappings through a guided decision process.
Outcome: The proposed framework shows that LLMs perform better than the embedding-based framework in creating crosswalks.
Paraphrase-based Contrastive Learning for Sentence Pair Modeling (2025.naacl-srw)

Copied to clipboard

Challenge: Existing methods to improve performance of sentence pair modeling are not available on a large-scale for non-English languages.
Approach: They propose a method to apply contrastive learning to pre-trained masked language models . they use sentence embeddings of paraphrase pairs to make similar sentences .
Outcome: The proposed method can be used on four sentence pair modeling tasks in English and Japanese.
Do Video Language Models really understand the video contexts? (2025.naacl-srw)

Copied to clipboard

Challenge: Recent advances in VideoQA performance have shown that visual language models are effective but the processes of understanding and reasoning in VLMs remain under-explored.
Approach: They propose a framework that incorporates a fine-grained question generation and answering process to measure how well VLMs understand video question answering tasks.
Outcome: The proposed framework incorporates a fine-grained question generation and answering process to measure how well the responses generated by VLMs align with what the model understands.
Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics? (2025.naacl-srw)

Copied to clipboard

Challenge: Text style transfer (TST) is a multidimensional task requiring the assessment of style transfer accuracy, content preservation, and naturalness.
Approach: They propose to use text style transfer metrics to evaluate outputs of text editors . they also investigate the potential of large language models as tools for TST evaluation .
Outcome: The proposed methods provide better insights than existing metrics, the authors show . their meta-evaluation through correlation with hu-man judgments shows they are effective .
(CPER) From Guessing to Asking: An Approach to Resolving Persona Knowledge Gap in LLMs during Multi-Turn Conversations (2025.naacl-srw)

Copied to clipboard

Challenge: Existing methods for identifying and resolving persona knowledge gaps are underexplored.
Approach: They propose a framework that dynamically detects and resolves persona knowledge gaps using intrinsic uncertainty quantification and feedback-driven refinement.
Outcome: The proposed framework detects and resolves persona knowledge gaps using intrinsic uncertainty quantification and feedback-driven refinement on two real-world datasets: CCPE-M for preferential movie recommendations and ESConv for mental health support.
Streamlining LLMs: Adaptive Knowledge Distillation for Tailored Language Models (2025.naacl-srw)

Copied to clipboard

Challenge: Large language models (LLMs) have transformative potential across industries, e.g., enhancing customer service, revolutionizing medical diagnostics, or identifying crises in news articles.
Approach: They propose to distill compact, parameter-efficient tailored language models from LLMs for domain-specific tasks with comparable performance.
Outcome: The proposed framework outperforms knowledge distillation frameworks in the crisis domain, where labeled data is scarce.
LLM DEBATE OPPONENT : Counter-argument Generation focusing on Implicit and Critical Premises (2025.naacl-srw)

Copied to clipboard

Challenge: Recent advances in Large Language Models (LLMs) show promise in automating counter-argument generation.
Approach: They compare multi-step and one-step generation methods for counter-arguments across 100 debate topics.
Outcome: The proposed model outperforms multi-step and one-step pipelines for counter-arguments across 100 debate topics.
AutoML Meets Hugging Face: Domain-Aware Pretrained Model Selection for Text Classification (2025.naacl-srw)

Copied to clipboard

Challenge: Existing methods for fine-tuning text classification models are resource-intensive and require substantial computational power and time.
Approach: They propose a corpus-driven domain mapping pipeline that integrates pre-fine-tuned models from Hugging Face Model Hub into AutoML systems to improve model selection.
Outcome: The proposed pipeline improves model selection and streamlines workflows and reduces computational costs.
Paraphrasing Attack Resilience of Various Machine-Generated Text Detection Methods (2025.naacl-srw)

Copied to clipboard

Challenge: Recent large-scale emergence of LLMs has left an open space for dealing with consequences, such as plagiarism or the spread of false information on the Internet.
Approach: They evaluate the parsing attack resilience of three machine-generated text detection methods and their ensembles using Random Forest classifiers.
Outcome: The proposed methods yield the strongest results, but suffer significant losses during attacks.
Detecting, Generating, and Evaluating in the Writing Style of Different Authors (2025.naacl-srw)

Copied to clipboard

Challenge: In recent years, stylometry has been investigated in many different fields.
Approach: They propose to use sentences from different books to generate and evaluate stylistic texts according to the authors' writing styles.
Outcome: The proposed model can detect, generate, and evaluate documents according to the authors' writing styles with unpaired samples.
Collaborative Data Exploration through Visualization: A Thesis Proposal Analyzing Impact of Conversational Assistants (2025.naacl-srw)

Copied to clipboard

Challenge: Data visualization is integral to any Exploratory Data Analysis task, but the steep learning curve and cognitive load of traditional tools excludes non-technical users.
Approach: They propose to develop a conversational assistant capable of understanding user-initiated clarification requests and generating accurate responses to user requests.
Outcome: The proposed conversational assistant can understand user-initiated clarification requests and generate accurate responses.
MENDER: Multi-hop Commonsense and Domain-specific CoT Reasoning for Knowledge-grounded Empathetic Counseling of Crime Victims (2025.naacl-srw)

Copied to clipboard

Challenge: Experimental evaluations on counseling dialogue dataset, POEM validate MENDER’s efficacy in generating coherent, knowledge-grounded responses.
Approach: They propose a multi-hop commonsensE and domaiN-specific Chain-of-Thought reasoning framework that integrates commonsense and domain knowledge via multi-hopping reasoning over the dialogue context.
Outcome: Experimental evaluations on counseling dialogue dataset validate MENDER’s efficacy in generating coherent, empathetic, knowledge-grounded responses.
SkipCLM: Enhancing Crosslingual Alignment of Decoder Transformer Models via Contrastive Learning and Skip Connection (2025.naacl-srw)

Copied to clipboard

Challenge: Existing contrastive learning methods for cross-lingual alignment are not effective for multilingual machine translation tasks.
Approach: They propose a method that augments contrastive learning for cross-lingual alignment with a trainable skip connection to preserve information crucial for accurate target language generation.
Outcome: Experiments with XGLM-564M on the Flores-101 benchmark show that the proposed method preserves crucial information crucial for accurate target language generation.
Towards LLMs Robustness to Changes in Prompt Format Styles (2025.naacl-srw)

Copied to clipboard

Challenge: Existing prompt engineering techniques for adapting pre-trained LLMs to specific tasks are zero-shot prompting and few-shot supervised fine-tuning.
Approach: They propose a technique for addressing prompt brittleness by diversifying the styles used in the prompt few-shot examples by using computer vision techniques.
Outcome: Empirical results show that the proposed technique reduces style-induced prompt brittleness while improving overall performance across prompt variations and different datasets.
Reliability of Distribution Predictions by LLMs: Insights from Counterintuitive Pseudo-Distributions (2025.naacl-srw)

Copied to clipboard

Challenge: Recent studies highlight the use of Large Language Models (LLMs) for predicting response distributions as a cost-effective survey method.
Approach: They examine whether LLMs can rationally estimate distributions when presented with explanations that are against commonsense.
Outcome: The proposed models can rationally estimate distributions when presented with explanations that are against commonsense, but smaller or less human-optimized models follow explanations uncritically, compared to larger models that resist counterintuitive explanations by leveraging their pretraining-acquired knowledge.
Rosetta-PL: Propositional Logic as a Benchmark for Large Language Model Reasoning (2025.naacl-srw)

Copied to clipboard

Challenge: Large Language Models (LLMs) are primarily trained on high-resource natural languages, limiting their effectiveness in low-resourced settings and in tasks requiring deep logical reasoning.
Approach: They propose to use a dataset of logical propositions from Lean into a custom logical language to evaluate LLMs' logical reasoning and generalization capabilities in a controlled environment.
Outcome: The proposed model improves accuracy and accuracy beyond 20,000 training samples.

What is GenGO?

GenGO is an NLP powered publication search system. It currenctly indexes 30k+ papers from ACL Anthology, and implements multi-aspect summarization, semantic search, and more!

Information

About
Limitations