Papers by Maarten Rijke
Improving Multi-label Malevolence Detection in Dialogues through Multi-faceted Label Correlation Enhancement (2022.acl-long)
Copied to clipboard
| Challenge: | Current methods for detecting dialogue malevolence neglect label correlation. |
| Approach: | They propose to crowdsource a multi-label dataset for detecting malevolent dialogue responses and a model with label correlation enhanced CRF to measure the correlation between malevolence and negative emotions. |
| Outcome: | The proposed model outperforms the best performing baseline method on precision, recall, F1, and Jaccard score by 16.1%, 11.9%, 12.0%, and 6.1% on malevolence. |
Real World Conversational Entity Linking Requires More Than Zero-Shots (2024.findings-acl)
Copied to clipboard
| Challenge: | Entity linking (EL) in conversations is a key component of many downstream tasks such as semantic search. |
| Approach: | They propose to use Fandom and Wikipedia to evaluate EL models' ability to generalize to a new unfamiliar KB without prior training. |
| Outcome: | The proposed evaluation framework and dataset are tailored to facilitate the study. |
MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning (2024.acl-long)
Copied to clipboard
Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten Rijke, Zhumin Chen, Jiahuan Pei
| Challenge: | Large language models (LLMs) are the default paradigm for natural language processing (NLP) as the models’ scale and the diversity of tasks increase, fine-tuning becomes infeasible. |
| Approach: | They propose to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters and reduce their rank by 8 times . |
| Outcome: | The proposed model uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential. |
CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems (2024.findings-acl)
Copied to clipboard
| Challenge: | a number of studies have evaluated user satisfaction estimation in TOD systems . current benchmarks for user satisfaction estimates are highly skewed towards dialogues for which the user is satisfied. |
| Approach: | They leverage large language models to generate satisfaction-aware counterfactual dialogues to augment original dialogues of a test collection. |
| Outcome: | The proposed models show higher robustness to increase in dissatisfaction labels than fine-tuned models. |
Bootstrapped Pre-training with Dynamic Identifier Prediction for Generative Retrieval (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for document retrieval rely on static document identifiers . experimental results show that generative retrieval is outperforms dense retrieval in document retrievals. |
| Approach: | They propose a bootstrapped pre-training method that dynamically adjusts document identifiers during pre-train to accommodate the continuing memorization of the corpus. |
| Outcome: | The proposed method significantly outperforms existing pre-training generative retrieval baselines and performs well even in zero-shot settings. |
Table Question Answering for Low-resourced Indic Languages (2024.emnlp-main)
Copied to clipboard
| Challenge: | TableQA is the task of answering questions over tables of structured information, returning individual cells or tables as output. |
| Approach: | They propose a fully automatic large-scale tableQA data generation process for low-resource languages with limited budget. |
| Outcome: | The proposed method outperforms state-of-the-art LLMs on two Indic languages with no tableQA datasets and models on different aspects including mathematical reasoning capabilities and zero-shot cross-lingual transfer. |
What Makes a Good and Useful Summary? Incorporating Users in Automatic Summarization Research (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing research on automatic text summarization does not fully align with students’ needs. |
| Approach: | They propose a survey methodology that can be used to investigate the needs of users of automatically generated summaries. |
| Outcome: | The proposed method can be easily adjusted to investigate different user groups. |
Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to detect text in training corpus are limited due to their low token probabilities. |
| Approach: | They propose a method to calibrate token probabilities for pretraining data detection by using a divergence-based calibration method. |
| Outcome: | The proposed method significantly outperforms existing methods on Chinese text on English-language benchmarks and patents. |
KnowTuning: Knowledge-aware Fine-tuning for Large Language Models (2024.emnlp-main)
Copied to clipboard
Yougang Lyu, Lingyong Yan, Shuaiqiang Wang, Haibo Shi, Dawei Yin, Pengjie Ren, Zhumin Chen, Maarten Rijke, Zhaochun Ren
| Challenge: | Large language models (LLMs) are a default solution for many natural language processing tasks. |
| Approach: | They propose a knowledge-aware fine-tuning method to improve LLMs' knowledge awareness . they propose augmentation and comparison stages to improve accuracy and reliability . |
| Outcome: | The proposed method generates more facts with less factual error rate under fine-grained facts evaluation. |
Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies suggest using only a portion of the dialogue context in the annotation process, but the impact of this limitation on label quality remains unexplored. |
| Approach: | They propose to use large language models to summarize the dialogue context to provide a rich and short description of the dialogue and to examine the impact of doing so on the annotator’s performance. |
| Outcome: | The proposed model reduces the context and produces higher quality ratings but introduces ambiguity in usefulness ratings. |
The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models (2024.findings-emnlp)
Copied to clipboard
Xinyi Chen, Baohao Liao, Jirui Qi, Panagiotis Eustratiadis, Christof Monz, Arianna Bisazza, Maarten Rijke
| Challenge: | Current evaluation resources for instruction following focus on single task instructions, but the instruction sequences in these benchmarks often lack coherence. |
| Approach: | They propose to evaluate models’ abilities to follow multiple instructions through sequential instruction following tasks using four tasks to assess different aspects of sequential instruction followed. |
| Outcome: | The proposed benchmark outperforms open-source and closed-source models on four tasks assessing different aspects of sequential instruction following. |