Papers by Chang-Tien Lu
Mitigating Uncertainty in Document Classification (N19-1)
Copied to clipboard
| Challenge: | Existing models for uncertainty measurement are time-consuming and unable to handle large-scale data sets. |
| Approach: | They propose a new dropout-entropy method for uncertainty measurement and a metric learning method on feature representations to boost the performance of dropout based uncertainty methods. |
| Outcome: | The proposed method improves accuracy from 0.78 to 0.92 when 30% of the most uncertain predictions were handed over to human experts in “20NewsGroup” data. |
Uncertainty-Aware Cross-Lingual Transfer with Pseudo Partial Labels (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to train pre-trained language models for zero-shot cross-lingual tasks are noisy and lack confidence. |
| Approach: | They propose an uncertainty-aware cross-lingual transfer framework with pseudo-partial-label to maximize the utilization of unlabeled data by reducing noise. |
| Outcome: | The proposed framework outperforms baselines on named entity recognition and natural language inference tasks on 40 languages. |
TART: Improved Few-shot Text Classification Using Task-Adaptive Reference Transformation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fewshot text classification depend on inter-class variance . Existing approaches suffer from MLADA, which performs poorly on tasks with high inter- class variance whereas it fails to distinguish samples from tasks with low inter-group variance. |
| Approach: | They propose a task-adaptive reference transformation network to transform class prototypes to per-class fixed reference points in task-adapted metric spaces. |
| Outcome: | The proposed method surpasses state-of-the-art methods in 1-shot and 5-shot classifications on the 20 Newsgroups dataset. |
Can We Trust the Performance Evaluation of Uncertainty Estimation Methods in Text Summarization? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Text summarization is a key natural language generation task, but the high cost of inaccurate summaries raises concerns about the reliability of uncertainty estimation on text summarisation (UE-TS) evaluation methods. |
| Approach: | They propose a UE-TS benchmark that evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets. |
| Outcome: | The proposed benchmark evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets, with human-annotation analysis incorporated where applicable. |
InternalInspector I2: Robust Confidence Estimation in LLMs through Internal States (2024.findings-emnlp)
Copied to clipboard
Mohammad Beigi, Ying Shen, Runing Yang, Zihao Lin, Qifan Wang, Ankith Mohan, Jianfeng He, Ming Jin, Chang-Tien Lu, Lifu Huang
| Challenge: | Large Language Models (LLMs) often struggle with generating reliable outputs, often producing high-confidence inaccuracies known as hallucinations. |
| Approach: | They propose a framework that leverages contrastive learning on internal states including attention states, feed-forward states, and activation states of all layers to enhance confidence estimation in LLMs. |
| Outcome: | The framework outperforms existing methods in the hallucination detection benchmark HaluEval and the previous methods at the same time. |
Don’t Go To Extremes: Revealing the Excessive Sensitivity and Calibration Limitations of LLMs in Implicit Hate Speech Detection (2024.acl-long)
Copied to clipboard
| Challenge: | Several studies have examined whether large language models exhibit bias or discrimination against individuals or groups in terms of protected attributes like race, gender, or religion. |
| Approach: | They evaluate LLMs' ability to detect implicit hate speech and express confidence in their responses by considering prompt patterns and mainstream uncertainty estimation methods. |
| Outcome: | The proposed models exhibit two extremes: (1) excessive sensitivity towards groups or topics that may cause fairness issues, resulting in misclassifying benign statements as hate speech; (2) confidence scores for each method excessively concentrate on a fixed range, remaining unchanged regardless of the dataset’s complexity. |
Uncertainty Estimation on Sequential Labeling via Uncertainty Transmission (2024.findings-naacl)
Copied to clipboard
| Challenge: | Named entity recognition tasks are often suboptimal for NER . previous work focused on UE-NER, which estimates uncertainty scores for ner . |
| Approach: | They propose to use a Sequential Labeling Posterior Network to estimate uncertainty for NER . they propose to consider wrong-span cases and to evaluate the specificity of wrong-pan cases. |
| Outcome: | The proposed system improves on three datasets and AUPR on MIT-Restaurant datasets. |