Papers with Longformer
‘Am I the Bad One’? Predicting the Moral Judgement of the Crowd Using Pre–trained Language Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on NLP touch upon moral contexts in text. |
| Approach: | They construct a dataset that can be used for moral judgement tasks on a popular reddit subreddit. |
| Outcome: | The proposed model passes moral judgements on posts from a popular reddit subreddit . it shows that the model can be fine tuned and improves across the datasets . |
What Helps Transformers Recognize Conversational Structure? Importance of Context, Punctuation, and Labels in Dialog Act Recognition (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing punctuation in the transcripts has a massive effect on the models’ performance, and specific label set specificity does not affect dialog act segmentation performance. |
| Approach: | They apply two pre-trained transformer models to a conversation transcript as a sequence of dialog acts and achieve strong results on Switchboard Dialog Act and Meeting Recorder Dialog Act corpora. |
| Outcome: | The proposed models achieve 8.4% and 14.2% error rates on the Switchboard Dialog Act and Meeting Recorder Dialog Act corpora. |
An LCF-IDF Document Representation Model Applied to Long Document Classification (2024.lrec-main)
Copied to clipboard
| Challenge: | Document representation models have been used for years in NLP and Text Mining tasks but are limited when it comes to capturing the deeper semantics and context of textual data. |
| Approach: | They propose to use a Latent Concept Frequency-Inverse Document Frequence model to exploit the advantages of TF-IDF while incorporating semantic context into the model. |
| Outcome: | The proposed model outperforms existing models on the Long Document Classification task and shows that it performs better than TF-IDF and BERT-like representation models. |
NLQuAD: A Non-Factoid Long Question Answering Data Set (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing data sets for document-level question answering are limited in their ability to detect short text and require multiple-sentence descriptive answers and opinions. |
| Approach: | They introduce a new data set with baseline methods for non-factoid long question answering . they compare BERT, RoBERTa, and Longformer models to establish baseline performances . |
| Outcome: | Experimental results show that Longformer outperforms the other architectures but human evaluations show that it is far behind the human upper bound. |
Accurate and Well-Calibrated ICD Code Assignment Through Attention Over Diverse Label Embeddings (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to assigning ICD codes to clinical text are time-consuming, labor intensive, and error-prone. |
| Approach: | They propose to adapt a Transformer-based model to a longformer model and use it to encode clinical narratives. |
| Outcome: | The proposed approach outperforms current state-of-the-art models in ICD coding with the label embeddings contributing to the good performance. |
Simple Local Attentions Remain Competitive for Long-Context Tasks (2022.naacl-main)
Copied to clipboard
Wenhan Xiong, Barlas Oguz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Scott Yih, Yashar Mehdad
| Challenge: | Existing models for NLP tasks require long text sequences beyond the length limit of pretrained models. |
| Approach: | They propose to pretrain large-size NLP models using the same long-doc corpus and fine tune them for real-world long-context tasks. |
| Outcome: | The proposed models can perform better under standard pretraining paradigms than longformer and Longformer. |
CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding (2021.findings-acl)
Copied to clipboard
| Challenge: | Scientific document understanding is challenging due to the highly domain specific nature of scientific language. |
| Approach: | They propose a large, contextualized, rigorously cleaned labelled dataset for cite-worthiness detection built from extracted scientific documents. |
| Outcome: | The proposed model improves on a paragraphlevel contextualized sentence labelling model based on Longformer . the model shows a 5 F1 point improvement over SciBERT which considers only individual sentences . |
The Accuracy Paradox in RLHF: When Better Reward Models Don’t Yield Better Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) significantly enhances Natural Language Processing by aligning language models with human expectations. |
| Approach: | They propose to integrate feedback from humans into RLHF to improve language models by capturing human-like preferences. |
| Outcome: | The proposed model outperforms models trained with moderately accurate reward models on relevance, factuality, and completeness tasks. |
The NLP Task Effectiveness of Long-Range Transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing benchmarks on long-range attention models have not been sufficient to develop efficient Transformers and their practical application on complex NLP tasks. |
| Approach: | They propose to benchmark 7 Transformer variants on 5 difficult NLP tasks and 7 datasets to examine their capacity for long-range attention. |
| Outcome: | The proposed models have advantages on content selection and query-guided decoding, but they come with previously unrecognized drawbacks such as insufficient attention to distant tokens and accumulated approximation error. |
Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent supervised neural models have greatly promoted the development of topic segmentation, but the deeper relationship between coherence and topic segmenting is underexplored. |
| Approach: | They propose to use topic-aware Sentence Structure Prediction and Contrastive Semantic Similarity Learning to capture coherence from logical structure and semantic similarity perspectives to further improve topic segmentation performance. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on WIKI-727K and achieves an average relative reduction of 4.3% on Pk on WikiSection. |
LittleBird: Efficient Faster & Longer Transformer for Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for BERT have a limitation dealing with long inputs due to its attention mechanism. |
| Approach: | They propose a model based on BigBird with improved speed and memory footprint . they propose 'pack and unpack attention' to replace global attention . |
| Outcome: | The proposed model can work on long inputs even after being pre-trained on short inputs. |
Multi-level Contrastive Learning for Script-based Character Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Scripts are written text for plays, movies, or broadcasts. |
| Approach: | They propose a multi-level contrastive learning framework to capture characters’ global information in a fine-grained manner. |
| Outcome: | The proposed framework improves on three character understanding sub-tasks by a considerable margin. |
An Exploratory Study on Long Dialogue Summarization: What Works and What’s Next (2021.findings-emnlp)
Copied to clipboard
Yusen Zhang, Ansong Ni, Tao Yu, Rui Zhang, Chenguang Zhu, Budhaditya Deb, Asli Celikyilmaz, Ahmed Hassan Awadallah, Dragomir Radev
| Challenge: | Existing models for dialogue summarization focus on extracting the main events of short conversations, but real-world dialogues are difficult to train. |
| Approach: | They propose three strategies to deal with the lengthy input problem and locate relevant information using long dialogue datasets. |
| Outcome: | The retrieve-then-summarize pipeline models yield the best performance on three long dialogue datasets. |