| Challenge: | Existing work on tabular representation-learning jointly models tables and associated text using self-supervised objective functions derived from pretrained language models such as BERT. |
| Approach: | They propose a tabular representation-learning model that integrates tabular data with a pretraining objective function that detects corrupted cells. |
| Outcome: | The proposed model understands complex table semantics and numerical trends. |
Similar Papers
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data (2020.acl-main)
Copied to clipboard
| Challenge: | Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language understanding tasks. |
| Approach: | They propose a pretrained language model that jointly learns representations for NL sentences and (semi-)structured tables. |
| Outcome: | The proposed model performs best on the weakly-supervised semantic parsing benchmark WikiTableQuestions while performing competitively on the text-to-SQL dataset Spider. |
Structural Encoding and Pre-training Matter: Adapting BERT for Table-Based Fact Verification (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing research on fact verification focuses on unstructured textual evidence, but it is still underexplored. |
| Approach: | They propose to use a table-based language model to verify textual statements . they use cell embeddings and numerical information to improve accuracy . |
| Outcome: | The proposed method outperforms the state-of-the-art model on question answering tasks even without modeling numerical information. |
Generative Table Pre-training Empowers Models for Tabular Prediction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to use table pre-training to boost tabular prediction performance remain open . a bachelor's degree earns less than 50K, and a generative LM can be used to unify tasks via one LM. |
| Approach: | They propose a method that leverages table pre-training to empower tabular prediction models. |
| Outcome: | The proposed method outperforms baseline models on 12 datasets and can be easily combined with various backbone models. |
TabEmb: Joint Semantic-Structure Embedding for Table Annotation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing tables learn by linearizing the 2D table into a 1D token sequence and encoding it with pretrained language models (PLMs) such as BERT, but this leads to limited semantic quality and weaker generalization to unseen or rare values compared to modern LLMs. |
| Approach: | They propose a table annotation module called TabEmb which decouples semantic encoding from structural modeling by creating semantically rich embeddings for each column. |
| Outcome: | Experiments show that TabEmb outperforms baselines on different table annotation tasks. |
Transformers for Tabular Data Representation: A Survey of Models and Applications (2023.tacl-1)
Copied to clipboard
| Challenge: | Recent research efforts extend LMs by developing neural representations for structured data. |
| Approach: | They propose to extend transformer-based language models to tabular data by analyzing inputs, model training, and supported downstream tasks. |
| Outcome: | The proposed models are compared against existing models and are based on a traditional pipeline. |
How Much Pretraining Does Structured Data Need? (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models are increasingly adopted for handling structured data, despite pretraining on unstructured text. |
| Approach: | They propose to re-initialize subsets of layers with random weights before fine-tuning on structured datasets. |
| Outcome: | The proposed models are compared to unstructured datasets and show that they perform well over structured data. |
Tag&Tab: Pretraining Data Detection in Large Language Models Using Keyword-Based Membership Inference Attack (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies on detecting pretraining data in large language models have focused on sentence-level membership inference attacks (MIAs) but these methods often exhibit poor accuracy, failing to account for the semantic importance of textual content and word significance. |
| Approach: | They propose a method that leverages established natural language processing techniques to tag keywords in input text and then uses them to obtain probabilities and calculate their average log-likelihood to determine input text membership. |
| Outcome: | The proposed method exploits established natural language processing techniques to tag keywords in input text and calculate their average log-likelihood to determine input text membership. |
Understanding tables with intermediate pre-training (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Textual entailment is well studied, but is less well studied for table enlargement . a new dataset of millions of examples is used to train the model . |
| Approach: | They adapt a table-based BERT model to recognize entailment from a dataset . they evaluate table pruning techniques as a pre-processing step to improve model efficiency . |
| Outcome: | The proposed model improves training and prediction efficiency at a moderate drop in accuracy. |
Enhancing Tabular Reasoning with Pattern Exploiting Training (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing methods based on pre-trained language models have shown superior performance over tabular tasks despite showing inherent problems such as not using the right evidence and inconsistent predictions across inputs. |
| Approach: | They utilize Pattern-Exploiting Training (PET) on pre-trained language models to strengthen tabular reasoning models’ pre-existing knowledge and reasoning abilities. |
| Outcome: | The proposed model exhibits superior understanding of knowledge facts and tabular reasoning compared to baseline models. |
Towards Table-to-Text Generation with Pretrained Language Model: A Table Structure Understanding and Text Deliberating Approach (2022.emnlp-main)
Copied to clipboard
| Challenge: | Currently, the generalization issues hinder the applicability of neural table-to-text models due to the limited source tables. |
| Approach: | They propose a table-structureaware text generation model with pretrained language model and propose TASD to bridge the gap between the structured table and text input. |
| Outcome: | The proposed model bridges the gap between the structured table and text input and generates accurate and fluent descriptive texts on two public datasets. |