| Challenge: | In some pathological situations, such a stochastic process may "leak" probability mass onto the set of infinite strings. |
| Approach: | They propose to view a language model as a discrete stochastic process X t : t = = t + . |
| Outcome: | The proposed conditions of tightness are generalized to language models and the literature. |
Similar Papers
A Measure-Theoretic Characterization of Tight Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Language modeling is a core task in natural language processing. |
| Approach: | They propose to characterize leakage onto the set of infinite sequences by a measure-theoretic approach. |
| Outcome: | The proposed language model families are tight, meaning they will not leak . the proposed language models are based on the 'sequence leakage' hypothesis . |
Limitations of Autoregressive Models and Their Alternatives (2021.naacl-main)
Copied to clipboard
| Challenge: | Standard autoregressive language models only perform polynomial-time computation to compute probability of next symbol. |
| Approach: | authors propose alternative to standard autoregressive language models that use polynomial-time computation to compute probability of next symbol. |
| Outcome: | a large model size can grow superpolynomially in length, allowing it to store precomputed results and verify solutions. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe modelsâ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
What Kind of Language Is Hard to Language-Model? (P19-1)
Copied to clipboard
| Challenge: | a recent study suggests that language models perform poorly across languages. |
| Approach: | They propose a model that fits a paired-sample multiplicative mixed-effects model to obtain language difficulty coefficients from at least-pairwise parallel corpora. |
| Outcome: | The proposed model is able to handle missing data and is aware of inter-sentence variation. |
Are All Languages Equally Hard to Language-Model? (N18-2)
Copied to clipboard
| Challenge: | a fair comparison of language models is tricky because of the size of the corpora and the variability of orthographic systems. |
| Approach: | They propose a framework for fair cross-linguistic comparison of language models . they show that in some languages, textual expression is harder to predict with n-gram models compared to LSTM models based on translated text . |
| Outcome: | The proposed framework is based on translated text and language models on 21 languages. |
What Can String Probability Tell Us About Grammaticality? (2026.tacl-1)
Copied to clipboard
| Challenge: | linguistic theories have argued that language models have largely achieved grammatical competence, but they will assign non-zero probability to all strings. |
| Approach: | They propose a theoretical framework for analyzing string probabilities in linguistics based on simple assumptions about the generative process of corpus data. |
| Outcome: | The proposed framework makes three predictions using 280K sentence pairs in English and Chinese. |
Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work has shown that language models can select the most likely or plausible of a set of possible events, but they are far from robust. |
| Approach: | They focus on whether language models can select the most likely or plausible of a set of possibilities and compare them to a broader behavior that humans exhibit largely unconsciously. |
| Outcome: | The proposed models perform worse than expected under certain conditions, compared with Llama 3, Gemma 2, and Mistral NeMo, and they are significantly more sensible than leaves. |
Language Model Evaluation Beyond Perplexity (2021.acl-long)
Copied to clipboard
| Challenge: | a nascent literature on probing language models has focused on studying linguistic phenomena. |
| Approach: | They propose a framework for evaluating the fit of language models to natural language tendencies. |
| Outcome: | The proposed framework evaluates language models to the tendencies of natural language . it shows that the models learn only a subset of the tendancies considered . |
Understanding the Inner-workings of Language Models Through Representation Dissimilarity (2023.emnlp-main)
Copied to clipboard
| Challenge: | Dissimilarity measures measure the extent to which two modelâs internal representations differ . they can identify and locate generalization properties of models that are invisible via in-distribution test set performance. |
| Approach: | They propose to use representation dissimilarity measures to measure the extent to which two modelâs internal representations differ. |
| Outcome: | The proposed dissimilarity measures can identify and locate generalization properties of models that are invisible via in-distribution test set performance and new evaluations of how language model features vary as width and depth are increased. |