Multi-level Diagnosis and Evaluation for Robust Tabular Feature Engineering with Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models have shown promise in feature engineering for tabular data, but reliability concerns persist due to variability in generated outputs. |
| Approach: | They propose a multi-level diagnosis and evaluation framework to assess the robustness of large language models in feature engineering across diverse domains. |
| Outcome: | The proposed framework assesses the robustness of large language models across domains. |
Similar Papers
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models (2025.naacl-industry)
Copied to clipboard
| Challenge: | Typical evaluations of Large Language Models (LLMs) report a single accuracy metric per dataset, often derived from an optimized setup. |
| Approach: | They propose a framework for non-adversarial evaluation of large language models that evaluates models by repeatedly testing them on the same benchmarks in various setups. |
| Outcome: | The proposed framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. |
Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists (2026.findings-eacl)
Copied to clipboard
| Challenge: | Current benchmarks assess LLMs on more isolated capabilities, such as language understanding and question-answering. |
| Approach: | They propose a benchmark to evaluate the ability of large language models (LLMs) to perform feature engineering. |
| Outcome: | The proposed benchmark evaluates the ability of large language models to perform feature engineering, a critical and knowledge-intensive task in data science. |
Predicting Language Models’ Success at Zero-Shot Probabilistic Prediction (2025.findings-emnlp)
Copied to clipboard
Kevin Ren, Santiago Cortes-Gomez, Carlos Miguel Patiño, Ananya Joshi, Ruiqi Lyu, Jingjing Tang, Alistair Turcan, Khurram Yamin, Steven Wu, Bryan Wilder
| Challenge: | Recent work has investigated the capabilities of large language models (LLMs) as zero-shot models for generating individual-level characteristics. |
| Approach: | They conduct a large-scale empirical study of large language models’ zero-shot predictive capabilities across a wide range of tabular prediction tasks. |
| Outcome: | The results show that LLMs perform well on the base prediction task, and when they perform well, they are more likely to provide high-quality predictions. |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |
Rethinking Tabular Data Understanding with Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are capable of various tasks, yet their capability in interpreting and reasoning over tabular data remains an underexplored area. |
| Approach: | They propose a method for table structure normalization to improve model performance . they propose aggregation of multiple reasoning pathways to improve performance based on textual and symbolic reasoning. |
| Outcome: | The proposed method improves performance on symbolic reasoning tasks with textual reasoning slightly outperforming symbolic reasoning on tables. |
Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit impressive abilities in various domains such as text generation, instruction following, and reasoning. |
| Approach: | They propose a method to decompose the activations of Large Language Models into a sparse linear combination of SAE features. |
| Outcome: | The proposed method shows that some features are strongly related to specific languages, while others are unaffected by ablating them. |
Benchmarking and Improving LLM Robustness for Personalized Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluations focus on whether a model’s responses align with a user’s preferences, but factuality is an important yet overlooked dimension. |
| Approach: | They propose a scalable framework for evaluating robustness of large language models in personalization and a new dataset, PERGData. |
| Outcome: | The proposed framework improves robustness by 25% across models. |
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. |
| Approach: | They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them. |
| Outcome: | The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods. |
A Survey of Confidence Estimation and Calibration in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks in various domains, but they can be unreliable due to factual errors in their generations. |
| Approach: | They summarize recent advances in LLM confidence estimation and calibration and outline their main lessons learned. |
| Outcome: | The proposed methods can be used to assess the reliability of models and to calibrate them across tasks. |