Uncovering Limitations of Large Language Models in Information Seeking from Tables (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for Table Information Seeking (TabIS) are lacking in reliable evaluation. |
| Approach: | They propose a benchmark to evaluate the table information seeking abilities of large language models . they use a single-choice question format instead of a text-based evaluation . |
| Outcome: | The proposed benchmark is more reliable than existing models and is available online. |
Similar Papers
Investigating Table-to-Text Generation Capabilities of Large Language Models in Real-World Information Seeking Scenarios (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Existing table-to-text generation techniques that transform complex tabular data into comprehensible narratives are lacking in real-world applications. |
| Approach: | They investigate the table-to-text capabilities of different LLMs using four datasets within two real-world information seeking scenarios. |
| Outcome: | The proposed models can generate table-to-text data in two real-world information seeking scenarios and perform better than existing models. |
Rethinking Tabular Data Understanding with Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are capable of various tasks, yet their capability in interpreting and reasoning over tabular data remains an underexplored area. |
| Approach: | They propose a method for table structure normalization to improve model performance . they propose aggregation of multiple reasoning pathways to improve performance based on textual and symbolic reasoning. |
| Outcome: | The proposed method improves performance on symbolic reasoning tasks with textual reasoning slightly outperforming symbolic reasoning on tables. |
Exploring the Best Practices of Query Expansion with Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are foundational in language technologies, particularly in information retrieval (IR). |
| Approach: | They propose a framework that leverages large language models for query expansion . they use LLMs to generate multiple pseudo-references and integrate them with original queries . |
| Outcome: | The proposed framework enhances sparse and dense retrieval methods without pre-indexing. |
DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | Retrievalaugmented LLMs have been used to ground LLM in external knowledge . a gap exists in the current landscape regarding the effectiveness of grounding LLM on heterogeneous knowledge sources. |
| Approach: | They propose a model that uses symbolic language to generate symbolic queries . they use a dataset that is generated using predefined reasoning chains and human annotation . |
| Outcome: | The proposed model outperforms previous approaches by a significant margin in QA tasks over text. |
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2023.emnlp-main)
Copied to clipboard
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, Zhaochun Ren
| Challenge: | Existing work utilizes generative LLMs for Information Retrieval (IR) rather than direct passage ranking. |
| Approach: | They investigate generative LLMs such as ChatGPT and GPT-4 for relevance ranking in IR and use a test set to verify the model’s ability to rank unknown knowledge. |
| Outcome: | The proposed model outperforms a 3B supervised model on the BEIR benchmark. |
TabularMath: Understanding Math Reasoning over Tables with Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Mathematical reasoning has long been a key benchmark for evaluating large language models. |
| Approach: | They propose a framework that transforms math word problems into scalable tabular reasoning tasks. |
| Outcome: | The proposed framework transforms math word problems into scalable and verified tabular reasoning tasks. |
CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models (2024.acl-long)
Copied to clipboard
Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, Tat-Seng Chua
| Challenge: | Large language models are used to meet user information needs, but their effectiveness in dealing with user queries that contain various types of ambiguity remains unknown. |
| Approach: | They propose a benchmark for evaluating large language models using a well-organized taxonomy. |
| Outcome: | The proposed model is based on a well-organized taxonomy and compares it with other models. |
TabSQLify: Enhancing Reasoning Capabilities of LLMs Through Table Decomposition (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models struggle with large tables due to their limited input length . a novel method that decomposes tables into smaller and relevant sub-tables reduces the computational load on LLMs . |
| Approach: | They propose a method that leverages text-to-SQL generation to decompose tables into smaller and relevant sub-tables . the method can reduce the input context length significantly, making it more scalable and efficient . |
| Outcome: | The proposed method performs remarkably well on the WikiTQ benchmark and on the TabFact benchmark. |
Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Table Question Answering (TQA) aims to answer natural language questions using tabular data. |
| Approach: | They propose a systematic overview of TQA research using large language models and summarize available benchmarks based on task features. |
| Outcome: | The proposed framework provides a comprehensive overview of the current state of the art in the field of Table Question Answering. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |