Papers with predictors
Eyes Don’t Lie: Subjective Hate Annotation and Detection with Gaze (2024.emnlp-main)
Copied to clipboard
| Challenge: | Hate speech is a complex and subjective phenomenon. |
| Approach: | They propose a dataset that provides gaze data collected in a hate speech annotation experiment and introduce a first gaze-integrated HSD model. |
| Outcome: | The proposed dataset provides gaze data from hate speech annotation experiments. |
Beyond Literal Token Overlap: Token Alignability for Multilinguality (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing studies have shown that token overlap is a strong predictor of multilinguality and cross-lingual knowledge transfer between languages with different scripts. |
| Approach: | They propose a subword token alignability metric to understand the impact and quality of multilingual tokenisation. |
| Outcome: | The proposed metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low. |
Evaluating a Century of Progress on the Cognitive Science of Adjective Ordering (2023.tacl-1)
Copied to clipboard
| Challenge: | a new study examines the performance of cognitive hypotheses for adjective ordering in 32 languages . linguists and cognitive scientists have proposed an array of hypothese predicting adjective ordering . |
| Approach: | They compare the combined performance of existing adjective ordering proposals across 32 languages . they propose to use a baseline that reflects random chance accuracy and a higher baseline that measures idealized order . |
| Outcome: | The proposed hypotheses are compared with baselines in 32 languages and with random and idealized baselines. |
Don’t be my Doctor! Recognizing Healthcare Advice in Large Language Models (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) are becoming increasingly popular in everyday use, especially in highly regulated domains such as healthcare, where misleading advice may influence users to commit malpractice. |
| Approach: | They present a large-scale health-advice benchmark dataset that evaluates large language models' ability to recognize health-related advice in industrial settings. |
| Outcome: | The proposed model can be misinterpreted as direct advice in highly regulated domains such as healthcare, but the results are not enough to protect them from misinterpreting them as medical advice. |
Deconvolutional Time Series Regression: A Technique for Modeling Temporally Diffuse Effects (D18-1)
Copied to clipboard
| Challenge: | a confound exists in time series data that violates assumptions of linear models . time series may violate assumptions through temporal diffusion . |
| Approach: | They propose a statistical model that borrows from digital signal processing to fit latent impulse response functions of arbitrary shape. |
| Outcome: | The proposed model recovers true latent IRFs and improves prediction quality . it is based on a new technique that borrows from digital signal processing . |
CDRNN: Discovering Complex Dynamics in Human Language Processing (2021.acl-long)
Copied to clipboard
| Challenge: | Behavioral and fMRI experiments reveal detailed and plausible estimates of human language processing dynamics . central questions in psycholinguistics concern the mental processes involved in incremental human language understanding . |
| Approach: | They propose a continuous-time deconvolutional regressive neural network that captures time-varying, non-linear, and delayed influences of predictors on the response. |
| Outcome: | The proposed neural network captures time-varying, non-linear, and delayed influences on the response . Behavioral and fMRI experiments show it generalizes better than baselines . |
Towards More Fine-grained and Reliable NLP Performance Prediction (2021.eacl-main)
Copied to clipboard
| Challenge: | Performance prediction is a task of estimating a system’s performance without performing experiments. |
| Approach: | They propose to understand reliability of performance prediction models from two angles: confidence intervals and calibration. |
| Outcome: | The proposed methods demonstrate the feasibility of fine-grained performance prediction and the necessity to perform reliability analysis for performance prediction methods in the future. |
Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models (2022.acl-long)
Copied to clipboard
| Challenge: | Massively Multilingual Transformer based Language Models have been shown to be effective on zero-shot transfer across languages, though performance varies from language to language depending on pivot language(s) used for fine-tuning. |
| Approach: | They propose to combine multi-task learning problems with multi-lingual Transformers to model zero-shot transfer across languages. |
| Outcome: | The proposed model can predict zero-shot transfer across languages with a multi-task learning problem with pretraining data in very few languages. |
Predicting Long-Term Citations from Short-Term Linguistic Influence (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to quantify linguistic influence in timestamped documents are not informative about extent to which a paper affected subsequent publications. |
| Approach: | They propose to quantify linguistic influence in timestamped document collections by estimating a Hawkes process with a low-rank parameter matrix and identify lexical and semantic changes using contextual embeddings and word frequencies. |
| Outcome: | The proposed method is based on an online evaluation with incremental temporal training/test splits, in comparison with a strong baseline that includes predictors for initial citation counts, topics, and lexical features. |
Topics to Avoid: Demoting Latent Confounds in Text Classification (D19-1)
Copied to clipboard
| Challenge: | Despite impressive performance on many text classification tasks, deep neural networks tend to learn frequent superficial patterns that are specific to the training data and do not always generalize well. |
| Approach: | They propose a method that represents latent topical confounds and a model which “unlearns” confounding features by predicting both the label of the input text and the confound. |
| Outcome: | The proposed model generalizes better and learns features indicative of the writing style rather than the content. |
Multi-Source Domain Adaptation with Mixture of Experts (D18-1)
Copied to clipboard
| Challenge: | Existing methods for domain adaptation from multiple sources are designed to transfer supervision from a single source domain. |
| Approach: | They propose to capture the relationship between a target example and different source domains by a point-to-set metric. |
| Outcome: | The proposed method outperforms baselines and can handle negative transfer. |
Is the Brain Mechanism for Hierarchical Structure Building Universal Across Languages? An fMRI Study of Chinese and English (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that the brain builds hierarchical syntactic structures, but it is unknown whether they are universal across languages. |
| Approach: | They analyze the working memory requirements when applying parsing strategies to two languages: Chinese and English. |
| Outcome: | The proposed method shows that the brain adopts parsing strategies with less memory load according to different language structures. |
Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for review rating prediction ignore hierarchies among data . paper review rating predictions are important for improving paper review process . |
| Approach: | They propose a Hierarchical bi-directional self-attention Network framework for paper review rating prediction and recommendation . they leverage hierarchical structure of paper reviews with three levels of encoders . |
| Outcome: | The proposed approach can be used to make an effective decision-making tool for the academic paper review process. |
MS-LaTTE: A Dataset of Where and When To-do Tasks are Completed (2022.lrec-1)
Copied to clipboard
| Challenge: | Tasks are a fundamental unit of work in the daily lives of people, who are increasingly using digital means to keep track of, organize, triage, and act on them. |
| Approach: | They compile and release a large-scale dataset that captures location and time for tasks and a BERT-fine-tuned model that predicts task co-occurrence. |
| Outcome: | The proposed framework captures location and time, and predicts task co-occurrence with a BERT fine-tuned model outperforming baselines. |
Disentangling Uncertainty in Machine Translation Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Trainable evaluation metrics for machine translation (MT) exhibit strong correlation with human judgements, but they are often hard to interpret and might produce unreliable scores under noisy or out-of-domain data. |
| Approach: | They propose to use Monte Carlo dropout and deep ensembles to quantify uncertainty in machine translation and assess their ability to target different sources of aleatoric and epistemic uncertainty. |
| Outcome: | The proposed measures can target different sources of aleatoric and epistemic uncertainty, with a reduction in computational costs. |
Argument Mining for Review Helpfulness Prediction (2022.emnlp-main)
Copied to clipboard
| Challenge: | Argumentational features have been shown to be promising indicators of product review helpfulness, but their utility has been limited due to the lack of resources and large-scale experiments investigating their utility. |
| Approach: | They present an argumentational argumentation model that annotates 878 Amazon reviews on headphones and uses it to evaluate argument quality. |
| Outcome: | The proposed model improves the state-of-the-art model under text-only and text-and-image settings. |
Entropy- and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisal (2022.emnlp-main)
Copied to clipboard
| Challenge: | Transformer-based large language models are trained to make predictions about the next word by aggregating representations of previous tokens through their self-attention mechanism. |
| Approach: | They propose an entropy-based predictor that quantifies the diffuseness of self-attention and a distance-based one that captures the incremental change in attention patterns across timesteps. |
| Outcome: | The proposed models perform better over a rigorous baseline including GPT-2 surprisal than previous models that used entropy-based predictors and distance-based ones. |
Cross-Lingual Generalization and Compression: From Language-Specific to Shared Neurons (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evidence suggests that multilingual language models can transfer knowledge across languages without explicit cross-lingual supervision. |
| Approach: | They analyze the parameter spaces of three multilingual language models to examine their representations . they find that models evolve from language-specific representations to more specialized layer functions . |
| Outcome: | The proposed model can generate coherent English text, rather than spanish text, and it can generate generalized representations, the authors show. |
Predicting Performance for Natural Language Processing Tasks (2020.acl-main)
Copied to clipboard
| Challenge: | Natural language processing (NLP) is a vast field, with a wide variety of tasks, languages, and domains. |
| Approach: | They build regression models to predict evaluation score of an NLP experiment . they find that their models can produce meaningful predictions over unseen languages . |
| Outcome: | The proposed model outperforms baseline models and human experts on 9 different tasks. |
Modeling the Impact of Syntactic Distance and Surprisal on Cross-Slavic Text Comprehension (2022.lrec-1)
Copied to clipboard
| Challenge: | Using symmetric measures of insertion, deletion and movement of syntactic units, we investigate phonetic and orthographic asymmetries between selected languages. |
| Approach: | They focus on the syntactic variation and measure syntaktic distances between nine Slavic languages using symmetric measures of insertion, deletion and movement of syntak units in parallel sentences of the fable “The North Wind and the Sun”. |
| Outcome: | The proposed measures are validated on spoken and written cloze tests for Slavic native speakers to determine whether variations in syntax lead to slower or impeded intercomprehension of Slav texts. |
A linguistically-motivated evaluation methodology for unraveling model’s abilities in reading comprehension tasks (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models fail for linguistic characteristics of input examples, despite the impressive quantity of scientific studies dedicated to them, the capabilities, limitations, and risks of these models remain largely unknown. |
| Approach: | They propose to use semantic frame annotation to characterize examples by a small number of complexity factors to account for model’s difficulty. |
| Outcome: | The proposed evaluation methodology is based on the intuition that certain examples consistently yield lower scores regardless of model size or architecture. |
Surprise! Uniform Information Density Isn’t the Whole Story: Predicting Surprisal Contours in Long-form Discourse (2024.emnlp-main)
Copied to clipboard
| Challenge: | Uniform Information Density (UID) hypothesis posits that speakers tend to distribute information evenly across linguistic units to achieve efficient communication. |
| Approach: | They propose a functional pressure that speakers modulate information rate based on location within a hierarchically-structured model of discourse. |
| Outcome: | The proposed hypothesis posits that speakers tend to distribute information evenly across linguistic units to achieve efficient communication. |
Two Counterexamples to Tokenization and the Noiseless Channel (2024.lrec-main)
Copied to clipboard
| Challenge: | Nevertheless, Rényi efficiency is not perfect and the metric is difficult to evaluate because training multiple tokenizers can be prohibitively expensive and takes days or weeks. |
| Approach: | They propose to use Rényi efficiency as an intrinsic mechanism to evaluate a tokenizer for NLP tasks without the expensive step of training multiple models with different tokenizers. |
| Outcome: | The proposed metric is better correlated to downstream model performance than a percentile frequency metric. |
Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal (2026.acl-long)
Copied to clipboard
| Challenge: | cloze task is being replaced with LM probabilities for how predictable a word is in its context . clozen task is used to measure how predictable word is compared to unexpected words . |
| Approach: | They propose to use LM probabilities to quantify how predictable a word is . they propose to test whether LMs are better at distinguishing semantically similar words . |
| Outcome: | The results show that LM probabilities outperform cloze probabilities for the right reasons . they also show that human-like prediction is more sensitive to LM probability distinctions . |