How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) have exhibited impressive abilities in generating code from natural language requirements. |
| Approach: | They propose to introduce various metrics with inter-code similarity to evaluate the diversity of generated code by comparing model-generated solutions with human-written ones. |
| Outcome: | The proposed method leverages LMs’ capabilities in code understanding and reasoning, resulting in a set of metrics that represent the number of algorithms in model-generated solutions. |
Similar Papers
The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new study examines the effects of training language models on synthetic data generated by their predecessors. |
| Approach: | They propose to use recursive finetuning techniques to assess linguistic diversity of models. |
| Outcome: | The proposed metrics show a decrease in diversity of model outputs through successive iterations, especially for tasks demanding high levels of creativity. |
NLP+Code: Code Intelligence in Language Models (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | Language models have shown impressive abilities in a range of natural language processing tasks. |
| Approach: | This tutorial will provide an overview of the latest advances in natural language processing . it will provide preliminaries of training foundation models on code and their common practices . |
| Outcome: | This tutorial aims to provide an overview of recent advances in code modeling . it provides preliminaries of training foundation models on code and their common practices . |
Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Voting (2023.emnlp-main)
Copied to clipboard
Preethi Lahoti, Nicholas Blumm, Xiao Ma, Raghavendra Kotikalapudi, Sahitya Potluri, Qijun Tan, Hansa Srinivasan, Ben Packer, Ahmad Beirami, Alex Beutel, Jilin Chen
| Challenge: | Existing studies on diversity in large language models focus on the understudied class of fairness and inclusion concern in LLMs. |
| Approach: | They propose a technique to measure diversity in generated responses along people and culture axes by collective-critique and self-voting. |
| Outcome: | The proposed approach outperforms baseline methods and human evaluations with human and automated evaluations. |
Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models are contaminated with data from pretraining and finetuning tasks. |
| Approach: | They perform extensive analysis on the factors that affect model memorization and generalization, such as model size, problem difficulty, and question length. |
| Outcome: | The results show that models perform better on the subset of the benchmarks where similar solutions are seen during training. |
Improving Diversity of Commonsense Generation by Large Language Models via In-Context Learning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown proficiency in enhancing the generation quality across various tasks without the need for any fine-tuning. |
| Approach: | They propose a method that diversifies the LLM generations while preserving their quality. |
| Outcome: | The proposed method can be used as training data to improve diversity in existing commonsense generators. |
The Program Testing Ability of Large Language Models for Code (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent development of large language models (LLMs) for code shows promise in achieving code intelligence. |
| Approach: | They explore the ability of large language models to generate automated test cases . they show +11.77% and +4.22% higher code pass rates on HumanEval+ . |
| Outcome: | The proposed models show higher pass rates on humanEval+ compared with the current state-of-the-art models. |
Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be used to generate text data for training and evaluating other models. |
| Approach: | They propose to use logit suppression and temperature sampling to diversify text generation but at the cost of data accuracy. |
| Outcome: | The proposed approach can increase diversity but at the cost of data accuracy. |
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. |
| Approach: | They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them. |
| Outcome: | The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods. |
Evaluating the Evaluation of Diversity in Natural Language Generation (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for controlling diversity by tuning a “decoding parameter” affect form but not meaning. |
| Approach: | They propose a framework that measures correlation between a diversity metric and a parameter that controls some aspect of diversity in generated text. |
| Outcome: | The proposed framework outperforms existing methods in estimating diversity . it shows that humans outperformed existing methods but affect form but not meaning . |
Thesis proposal: Are We Losing Textual Diversity to Natural Language Processing? (2026.eacl-srw)
Copied to clipboard
| Challenge: | Using Neural Machine Translation, we examine whether the algorithms used in NMT have inherent inductive biases that are beneficial for most types of inputs but might harm the processing of untypical texts. |
| Approach: | They propose to use a set of measures to quantify text diversity based on its statistical properties to determine whether NMT systems struggle with maintaining the diversity of such texts. |
| Outcome: | The proposed approaches maintain the diversity and complexity of language and allow for better global planning of the output generation. |