Papers with macro
Copied to clipboard
| Challenge: | Recommender systems are a key component of our day-to-day lives, but evaluation remains a challenge due to the gap between offline metrics and online behaviors. |
| Approach: | They propose a framework that enables users to build believable human proxies from historical data. |
| Outcome: | The proposed framework exhibits closer alignment with real humans than previous work, both at micro and macro levels. |
Copied to clipboard
| Challenge: | Existing research indicates that disfluencies can constitute up to 5.9% of words in spontaneous speech, with repetitions accounting for over half of these disfluency. |
| Approach: | They propose to use a dataset to analyze reduplication and repetition in speech using computational linguistics to evaluate transformer-based models. |
| Outcome: | The proposed models achieve macro F1 scores of up to 85.62% in Hindi, 83.95% in Telugu, and 84.82% in Marathi for reduplication-repetition classification. |
Copied to clipboard
| Challenge: | Existing models rely on rigid, hand-crafted rules to model nuanced behavior in urban environments. |
| Approach: | They propose an urban simulator that generates realistic daily schedules using a recursive value-driven approach that balances mandatory activities, personal habits, and situational factors. |
| Outcome: | The proposed urban simulator exhibits closer alignment with real humans than previous work. |
Copied to clipboard
| Challenge: | Existing efforts in misinformation detection focus on written text, leaving a significant gap in addressing the complexity of spoken text in video transcripts. |
| Approach: | They propose to annotate video transcripts in three languages and six topics using a custom annotation tool. |
| Outcome: | The proposed tool shows strong cross-validation performance but challenges for generalization to unseen topics. |
Copied to clipboard
| Challenge: | Discourse parsing is a challenging task and plays a critical role in discourse analysis. |
| Approach: | They propose a macro discourse structure presentation schema to present the macro level discourse structure analysis. |
| Outcome: | The proposed corpus is based on two tasks of macro discourse structure analysis, including structure identification and nuclearity recognition. |
Copied to clipboard
| Challenge: | Large language models have shown increasing in-context learning capabilities with scaling up the model and data sizes. |
| Approach: | They propose a benchmark and suite of analyses to evaluate reasoning skills of large language models. |
| Outcome: | The proposed model compares pre-trained and fine-tuned models on tasks that require reasoning skills to solve. |
Copied to clipboard
| Challenge: | Unsupervised speech recognition (ASR) is the problem of learning automatic speech recognition systems from unpaired speech-only and text-only corpora. |
| Approach: | They propose a general theoretical framework to study the properties of pasted macro ‘ASRU’/ systems based on random matrix theory and the theory of neural tangent kernels. |
| Outcome: | The proposed framework proves various learnability conditions and sample complexity bounds on synthetic languages with three classes of transition graphs. |
Copied to clipboard
| Challenge: | Recent studies have shown that most abstractive summarization models are unfaithful and suffer from a wide range of hallucination. |
| Approach: | They propose a candidate summary generation and ranking technique to improve summary factuality without sacrificing quality. |
| Outcome: | The proposed method shows that the model trained using the proposed method improves on factuality and similarity-based metrics without conflicting with the model. |
Copied to clipboard
| Challenge: | EmoGist is a training-free, in-context learning method for visual emotion classification . context-dependent definitions of emotion labels could allow more accurate predictions of emotions . |
| Approach: | They introduce EmoGist, a training-free, in-context learning method for performing visual emotion classification with LVLMs. |
| Outcome: | The proposed method improves micro F1 scores and macro F1 with LVLMs. |
Copied to clipboard
| Challenge: | Traditionally, clinical coding is manual and laborintensive task prone to human error. |
| Approach: | They analyze 25 years of electronic health records from the Landspitali University Hospital in Icelandic to explore the potential of using NLP for clinical coding. |
| Outcome: | The best-performing model achieves competitive results in micro and macro F1 scores, with label attention contributing significantly to its success. |
Copied to clipboard
| Challenge: | Chinese discourse parsing has not yet a consistent evaluation metric . micro vs. macro F1 scores, binary v. multiway ground truth, and left-heavy v . right-heaviness binarization are important for Chinese discourses . |
| Approach: | They propose a neural network model that unifies a pre-trained transformer and a CKY-like algorithm and compare it with previous models with different evaluation scenarios. |
| Outcome: | The proposed model outperforms the previous models with different evaluation scenarios. |
Copied to clipboard
| Challenge: | 108K drug overdose deaths in 2022, according to NIDA . |
| Approach: | They propose a large-scale study of OUD-related myths on YouTube with clinical experts to validate 8 pervasive myths and release an expert-labeled video dataset. |
| Outcome: | The proposed model reduces annotation time and cost by over 76% compared to experts and full LLM labeling. |
Copied to clipboard
| Challenge: | Open-ended Visual Question Answering (VQA) requires models to reason over visual and natural language inputs using world knowledge. |
| Approach: | They propose a new VQA pipeline that deploys a generate-then-select strategy guided by world knowledge for the first time. |
| Outcome: | The proposed pipeline expands the knowledge coverage from in-domain training data by 4.1% on OK-VQA, without additional computation cost. |
Copied to clipboard
| Challenge: | Existing work on identifying complaints in social media has focused on feature-based and task-specific neural network models. |
| Approach: | They evaluate a battery of neural models underpinned by transformer networks and combine them with linguistic information to predict complaints. |
| Outcome: | The proposed models outperform state-of-the-art methods on a publicly available dataset achieving a macro F1 up to 87. |
Copied to clipboard
| Challenge: | Understanding and generating collaborative stories remains an underexplored area due to the lack of open-domain corpora. |
| Approach: | They propose to use a dataset of 40,000 collaborative stories written by 9,400 different authors from an online platform to generate a multi-task benchmark. |
| Outcome: | The proposed model achieves the best performance on fully-supervised, few-shot, and zero-shot scenarios while achieving the best results on the fully-supervised tasks. |
Copied to clipboard
| Challenge: | Complaining is a speech act used by humans to communicate a negative mismatch between reality and expectations . recent work on modeling complaints in natural language processing (NLP) has focused on distinguishing complaints from non-complaints in social media. |
| Approach: | They propose to classify complaints into various severity levels based on the face-threat that the complainer is willing to undertake and their purpose. |
| Outcome: | The proposed model achieves 55.7 macro F1 on binary complaint classification and 88.2 macro F1. |
Copied to clipboard
| Challenge: | Using publicly available materials science text data, we construct a benchmark for evaluating the performance of natural language processing (NLP) models on materials science texts. |
| Approach: | They propose a natural language benchmark for evaluating the performance of natural language processing (NLP) models on materials science text. |
| Outcome: | The proposed model outperforms BERT-based models on scientific text and a model pretrained on materials science journals. |
Copied to clipboard
| Challenge: | Auxiliary information from multiple sources has been demonstrated to be effective in zero-shot fine-grained entity typing (ZFET) however, there is no comprehensive understanding of how to make better use of the existing information sources and how they affect the performance of ZFET. |
| Approach: | They propose a multi-source fusion model targeting auxiliary information from multiple sources to improve zero-shot fine-grained entity typing (ZFET) |
| Outcome: | The proposed model achieves 11.42% and 22.84% gains over state-of-the-art baselines on BBN and Wiki respectively with regard to macro F1 scores. |
Copied to clipboard
| Challenge: | Opioid related aberrant behaviors (ORABs) present novel risk factors for opioid overdose. |
| Approach: | They propose to use a biomedical natural language processing benchmark dataset to classify ORABs from patients’ EHR notes into nine categories: confirmed aberrant behavior, suggested aberrant behaviors, Opioids, indication, diagnosed opioid dependency, Benzodiazepines, medication changes, and Central Nervous System-related. |
| Outcome: | The proposed dataset outperforms two state-of-the-art models in most categories and the gains are especially higher among uncommon classes. |
Copied to clipboard
| Challenge: | Existing methods for text-based person anomaly search fail to address the pose-semantic gap . asymmetric cross-modal information poses a challenge to accurately establishing retrieval relationships . |
| Approach: | They propose a video retrieval framework that partitions visual features into two categories based on relevance to the text query and performs effective interaction. |
| Outcome: | The proposed framework achieves leading retrieval performance on five benchmark datasets. |
Copied to clipboard
| Challenge: | Existing studies have examined the quality of labeled data in non-English languages. |
| Approach: | They annotate how datasets are created, input text and label sources, tools used to build them and what they study. |
| Outcome: | The results show that language-proficient NLP researchers' estimated availability correlates with dataset availability. |
Copied to clipboard
| Challenge: | Bragging is a speech act employed to build a favorable self-image through positive statements about oneself. |
| Approach: | They propose to use tweets annotated for bragging to build a model that can predict bragging with macro F1 up to 72.42 and 35.95 for binary and multi-class bragging classification tasks respectively. |
| Outcome: | The proposed models predict bragging with macro F1 up to 72.42 and 35.95 in binary and multi-class classification tasks respectively. |
Copied to clipboard
| Challenge: | Discourse analysis is becoming increasingly important in the field of natural language processing. |
| Approach: | They propose to annotate macro discourse information and additional discourse information to make annotation more objective and accurate. |
| Outcome: | The results show that the annotations are more objective and accurate than the previous ones. |
Copied to clipboard
| Challenge: | Discourse structure analysis is an important research topic in natural language processing. |
| Approach: | They propose to construct a macro discourse structure framework and annotate 147 Newswire articles. |
| Outcome: | The proposed framework can lay the foundation for further analysis of macro discourse structure. |
Copied to clipboard
| Challenge: | Existing methods for fake news detection "zoom in" to verify content with knowledge sources or check readers’ replies to posts but neglect information in the external news environment where a fake news post is created and disseminated. |
| Approach: | They propose a framework to capture news environment signals and a module to perceive useful signals and assist final prediction. |
| Outcome: | The proposed framework can improve the performance of basic fake news detectors by capturing the environmental signals of news posts and analyzing the results. |
Copied to clipboard
| Challenge: | Existing LLMs generally assign a higher score to factually consistent summaries than to factualally inconsistent summary. |
| Approach: | They propose a benchmark to measure whether large language models prefer factually consistent continuations of inputs. |
| Outcome: | The proposed benchmark compares the scores an LLM assigns to a factually consistent versus a inconsistent summary for an input news article. |
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated exceptional abilities in both text understanding and generation. |
| Approach: | They propose an Embedding Watermark method that implants backdoors on embeddings to protect copyright of large language models. |
| Outcome: | The proposed method protects the copyright of large language models without compromising service quality while minimizing the adverse impact on the original embeddings’ utility. |
Copied to clipboard
| Challenge: | This study introduces a dataset that focuses on the validity of statements in legal wills. |
| Approach: | They propose a dataset that focuses on the validity of statements in legal wills. |
| Outcome: | The proposed model achieves 80% macro F1 and accuracy, but group accuracy is in mid 80s at best, suggesting that the models’ understanding of the task remains superficial. |
Copied to clipboard
| Challenge: | Audio deepfake detection systems do not generalize well to realistic in-the-wild deepfakkes. |
| Approach: | They propose a novel In-Context Learning paradigm with comparison-guidance for Audio Deepfake detection framework that uses audio language models for training-free generalization to unseen deepfakes. |
| Outcome: | The proposed framework improves macro F1 over specialized detectors on in-the-wild datasets with up to 2 relative improvement over existing models. |
Copied to clipboard
| Challenge: | Concept and Named Entity Recognition (CNER) is a new unified task that handles concepts and entities mentioned in unstructured texts seamlessly. |
| Approach: | They propose a new unified task that handles concepts and entities mentioned in unstructured texts seamlessly. |
| Outcome: | The proposed task gains +5.4 and +8 macro F1 points when performed as a unified task compared to specialized named entity and concept recognition systems. |
Copied to clipboard
| Challenge: | Vulgar words are employed in language use for several different functions, including expressing aggression, signaling group identity or the informality of the communication. |
| Approach: | They present a dataset of 7,800 tweets with six categories of vulgarity in which all instances of vulgar words are annotated with one of the six categories. |
| Outcome: | The proposed model can predict the category of a vulgar word based on the immediate context it appears in with 67.4 macro F1 across six classes. |
Copied to clipboard
| Challenge: | Existing aspects target sentiment classification models are not trainable if annotated data are not available. |
| Approach: | They propose an approach that solves ATSC with natural language prompts by 24.13 accuracy points and 33.14 macro F1 points. |
| Outcome: | The proposed model outperforms supervised SOTA approaches under few-shot scenarios and under supervised settings, especially for few-shot cases. |
Copied to clipboard
| Challenge: | Quantization is a viable solution for pre-trained language models, but most existing methods are task-specific and require customized training and quantization with a large number of trainable parameters. |
| Approach: | They propose a "quantize before fine-tuning" framework that allows for quantization with a large number of trainable parameters on each individual task. |
| Outcome: | The proposed framework is compatible with quantization-aware training and post-training quantization and corrects quantization errors. |
Copied to clipboard
| Challenge: | Existing methods for ultra-low bit quantization cause severe accuracy drops . a novel Dual-Binarization method is proposed for efficient Large Language Models . |
| Approach: | They propose a Dual-Binarization method that takes 2-bit-width and binarization into account . they propose DB-LLM, which uses a 2-bit binarized weighted model to represent weights efficiently . |
| Outcome: | The proposed method surpasses the current State-of-the-Art in ultra-low bit quantization and achieves 20% reduction in computational consumption compared to the SOTA method under the same bit-width. |
Copied to clipboard
| Challenge: | Figures permeate human communication, but are understudied in NLP. |
| Approach: | They create a figurative language inference dataset for seven languages associated with a variety of cultures, using cultural and regional concepts for figurativ expressions. |
| Outcome: | The results show that the most common figurative expressions are found in Hindi, Indonesian, Javanese, Kannada, Sundanese, Swahili and Yoruba. |
Copied to clipboard
| Challenge: | a dataset of cooking recipes in French is highly imbalanced due to collaborative nature of the dataset . authors propose a neural-based model to address the first task of the DEFT 2013 shared task . |
| Approach: | They propose a neural-based model to address the first task of the DEFT 2013 shared task . they use state-of-the-art embedding approaches and deep architectures to address imbalanced dataset . |
| Outcome: | The proposed model outperforms models that use only pretrained embeddings in micro and macro F1 scores. |
Copied to clipboard
| Challenge: | State-of-the-art (SOTA) methods use the cross-encoder architecture to concatenate a mention (and its context) with each type and feed it into a pretrained language model (PLM) to score their relevance. |
| Approach: | They propose to perform entity typing in a recall-expand-filter manner and use a novel model to encode and score all these K candidates in one forward pass. |
| Outcome: | The proposed method is thousands of times faster than the CE-based architecture and is very efficient in fine-grained (130 types) and coarse-grain (9 types) entity typing. |
Copied to clipboard
| Challenge: | Extensive experiments on seven semantic textual similarity tasks show our method achieves consistent improvement over the contrastive learning baseline and sets new states of the art. |
| Approach: | They propose a whitening-based contrastive learning method for sentence embedding learning which combines contrastive and shuffled group whitening. |
| Outcome: | The proposed method achieves better alignment and uniformity on seven semantic textual similarity tasks. |
Copied to clipboard
| Challenge: | Existing guardrails rely on rule-based filtering or single-pass classification, limiting their ability to handle nuanced safety violations. |
| Approach: | They propose a critique-augmented guardrail model that distills knowledge from high-capacity LLMs by generating structured critiques alongside safety labels. |
| Outcome: | The proposed model outperforms existing guardrail models on multiple safety benchmarks and achieves the highest average F1 and AUPRC. |
Copied to clipboard
| Challenge: | Existing evaluation practices for recommender systems rely on few-shot prompting and offline metrics are often misaligned with online behavior. |
| Approach: | They propose a framework that learns world-model-driven agents from human interactions. |
| Outcome: | The proposed framework enables agents to express rich preferences and feedback in natural language and interact with recommender systems in a simulation. |
Copied to clipboard
| Challenge: | Existing tools for sexism detection fail to capture subtle distinctions within sexist content, limiting their practical applicability. |
| Approach: | They propose two techniques to address class imbalance and nuanced nature of sexist language . definition-based data augmentation leverages category-specific definitions to generate semantically-aligned examples . |
| Outcome: | The proposed techniques improve accuracy across all tasks and improve reliability. |
Copied to clipboard
| Challenge: | Existing alignment methods share a common topology of information flow, but their alternatives have not been thoroughly explored. |
| Approach: | They propose a theory of reward generalization in reinforcement learning from human feedback . they propose induced Bayesian networks to model the impact of dataset topologies on reward generalisation . |
| Outcome: | The proposed method achieves an average win rate of 65% on three NLP tasks. |
Copied to clipboard
| Challenge: | Pretrained language models have transformed text classification, but their computational demands often render them impractical for resource-constrained settings. |
| Approach: | They propose a linguistically-grounded framework for context minimization that leverages theme-rheme structure to preserve critical classification signals while reducing input complexity. |
| Outcome: | The proposed framework preserves critical classification signals while reducing input complexity. |
Copied to clipboard
| Challenge: | Existing methods to search for information are limited by data scarcity and lack contextual semantic understanding. |
| Approach: | They propose a new post-training paradigm with three self-supervised tasks to initialize the conversational search model to enhance dialogue structure and contextual semantic understanding. |
| Outcome: | The proposed method can initialize the conversational search model to enhance dialogue structure and contextual semantic understanding. |
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have made impressive results in a wide range of NLP tasks. |
| Approach: | They propose a pre-training model with editable and scalable key-value memory and leverage knowledge in an explainable manner by knowledge retrieval in the pasted macro ‘MEMORY’. |
| Outcome: | The proposed model decouples the knowledge storage from model parameters with an editable and scalable key-value memory and leverages knowledge in an explainable manner by knowledge retrieval in the pasted macro ‘MEMORY’. |
Copied to clipboard
| Challenge: | Existing studies focus on enhancing token-level interactions, but lack sufficient modeling of discourse structure information. |
| Approach: | They propose to use a discourse structure called "thread" to enhance token interaction among different utterances. |
| Outcome: | The proposed model achieves state-of-the-art on two datasets. |
Copied to clipboard
| Challenge: | LLM-empowered agent simulations generate rich, adaptive, and often nonlinear interaction patterns. |
| Approach: | They propose an automated Causal discovery framework for LLM agent simulations that converts mechanistic hypotheses into computable factors and learns a compact causal representation centered on an emergent target. |
| Outcome: | Experiments across four emergent settings demonstrate the promise of CAMO. |
Copied to clipboard
| Challenge: | Argument mining aims to identify the structural elements of arguments, denoted as information and inference, in online discourses. |
| Approach: | They propose to use Twitter Arguments to identify structural elements of arguments, denoted as information and inference, in a dataset that uses 1,814 tweets and an annotation framework that incorporates definitions from the Cambridge Dictionary to define and identify argument components. |
| Outcome: | The proposed dataset identifies arguments on Twitter and achieves an 85.06% macro F1 score in detecting arguments. |
Copied to clipboard
| Challenge: | Large language models (LLMs) have raised hopes for automated end-to-end fact-checking, but prior studies report mixed results. |
| Approach: | They evaluate 15 large language models on 6,000 claims fact-checked by PolitiFact . standard models perform poorly, reasoning offers minimal benefits, and web search provides only moderate gains . |
| Outcome: | The models predict claim veracity and a curated RAG system improved macro F1 by 233% on average across model variants. |
Copied to clipboard
| Challenge: | Recent studies have shown that LLMs can be significantly improved by integrating external tools. |
| Approach: | They propose a framework that integrates external tools into large language models to evaluate their ability to generate action plans. |
| Outcome: | The proposed framework evaluates the ability of large language models to generate action plans and generate action plan templates. |
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) provide potential for uncovering structured, human-interpretable representations in Large Language Models (LLMs). |
| Approach: | They analyze SAEs for interpretable feature extraction from Large Language Models in safety-critical classification tasks. |
| Outcome: | The proposed framework outperforms hidden-state and BoW models while demonstrating cross-lingual toxicity detection and visual classification tasks. |
Copied to clipboard
| Challenge: | Fei Xiaotong’s Differential Order Pattern characterizes rural society as egocentric and relationally graded, with cooperation attenuating over social distance. |
| Approach: | They propose a multi-agent framework grounded in Affect Control Theory, Social Identity Theory, and Durkheimian collective affect. |
| Outcome: | Extensive simulations support interpreting Differential Order as a structure-sensitive emergent outcome of general social mechanisms. |