Consistent Bidirectional Language Modelling: Expressive Power and Representational Conciseness (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing bidirectional language models lack the ability to utilise future contexts and the pre-determined left-to-right generation order. |
| Approach: | They propose a class of bidirectional language models that are consistent by definition and can be efficiently used both for generation and scoring of sequences. |
| Outcome: | The proposed models are consistent by definition and can be efficiently used both for generation and scoring of sequences. |
Similar Papers
Credible without Credit: Domain Experts Assess Generative Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | ChatGPT has been criticized for its lack of accuracy and coherence . authors argue that language models could replace search engines and make college essays obsolete . |
| Approach: | a team of 10 domain experts conducts an initial assessment of language models using 100 expert-written questions. |
| Outcome: | The results show that language models are mixed in their accuracy. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current Large Language Models (LLMs) are unparalleled in their ability to generate grammatically correct, fluent text. |
| Approach: | They argue that LLMs only parrot statistical patterns in training data and that language learning in LLM cannot inform human language learning. |
| Outcome: | The proposed model can generate grammatically correct, fluent text without requiring human intervention. |
In What Languages are Generative Language Models the Most Formal? Analyzing Formality Distribution across Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual generative language models (LMs) are fluent in a large variety of languages. |
| Approach: | They analyze formality distributions of XGLM and BLOOM’s predictions in 5 languages and classify 1,200 generations per language as formal, informal, or incohesive. |
| Outcome: | The proposed models generate a significant amount of informal predictions even when prompted with formal text. |
Pragmatically Informative Text Generation (N19-1)
Copied to clipboard
| Challenge: | Existing approaches to pragmatics have been used to improve the informativeness of generated text in grounded language learning problems. |
| Approach: | They propose to use pragmatics to improve the informativeness of conditional text models . they propose to apply pragmatic reasoning to more traditional language generation tasks . |
| Outcome: | The proposed methods improve the performance of strong existing systems for abstractive summarization and generation from structured meaning representations. |
Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive capabilities, but still suffer from inconsistency issues. |
| Approach: | They develop a ConsisEval benchmark to evaluate LLMs' inconsistency . they find that LLM models can paradoxically fail at easier problems . |
| Outcome: | The proposed model achieves highest consistency score but inconsistent to specific questions due to distraction by redundant information, misinterpretation of questions, etc. |
How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly studied as repositories of linguistic knowledge. |
| Approach: | They compare LLMs’ performance as pragmatic listeners and as pragmatic speakers . they find a robust asymmetry between pragmatic evaluation and pragmatic generation . |
| Outcome: | The proposed models perform better as listeners than speakers, and produce more appropriate language than speakers. |
Unlocking Anticipatory Text Generation: A Constrained Approach for Large Language Models Decoding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have shown a powerful ability for text generation, but undesired behaviors such as toxicity and hallucinations can manifest. |
| Approach: | They propose to formalize text generation as a future-constrained generation problem to minimize undesirable behaviors and enforce faithfulness to instructions. |
| Outcome: | The proposed approach is effective across three tasks, including keyword-constrained generation, toxicity reduction, and factual correctness in question-answering. |
Factuality of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Das, Preslav Nakov
| Challenge: | Large language models (LLMs) are factually incorrect, which limits their applicability in real-world scenarios. |
| Approach: | They analyze existing work to identify major challenges and their associated causes . they propose to evaluate LLMs using a variety of measures to mitigate factual errors . |
| Outcome: | The proposed methods are based on a variety of datasets and proposed strategies to mitigate factual errors. |
Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges (2025.acl-long)
Copied to clipboard
Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, Barbara Plank
| Challenge: | linguistics studies how context influences meaning of language and how people use it to convey implied meanings, emotions, and intentions. |
| Approach: | They analyze task designs, data collection methods, evaluation approaches and their relevance to real-world applications. |
| Outcome: | The findings highlight emerging trends, challenges, and gaps in existing benchmarks . the findings will contribute to more nuanced and context-aware NLP models . |