Papers by Jackie Cheung
Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation (2022.findings-acl)
Copied to clipboard
| Challenge: | Current language generation models suffer from issues such as repetition, incoherence, and hallucinations . |
| Approach: | They propose to analyze exposure bias from an imitation learning perspective and prove it is a problem . they show that exposure bias leads to an accumulation of errors during generation . |
| Outcome: | The proposed model fails to capture errors during generation and poor generation quality. |
GLIMPSE: Pragmatically Informative Multi-Document Summarization for Scholarly Reviews (2024.acl-long)
Copied to clipboard
| Challenge: | Scientific peer review is essential for the quality of academic publications. |
| Approach: | They propose a method that summarises scholarly reviews using a Rational Speech Act framework and novel uniqueness scores. |
| Outcome: | The proposed method generates more discriminative summaries than baseline methods in terms of human evaluation while achieving comparable performance with these methods in term of automatic metrics. |
Evaluating Dependencies in Fact Editing for Language Models: Specificity and Implication Awareness (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on editing LLMs neglects the dependency between a fact and its logical implications. |
| Approach: | They propose an evaluation protocol that assesses the editing process using StandUp . they aim to ensure that the editing respects internal logical constraints . |
| Outcome: | The proposed evaluation protocol assesses the editing process using a standup question-answering dataset. |
Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations (2024.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art language models (LMs) sometimes generate that misalign with world knowledge. |
| Approach: | They propose a method to mitigate hallucinations by restoring the LM's internal fact recall pipeline by a targeted restoration of its internal fact-recall pipeline. |
| Outcome: | The proposed method shows superior performance compared to baselines. |
Balaur: Language Model Pretraining with Lexical Semantic Relations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Lexical semantic relations (LSRs) characterize meaning relationships between words and play an important role in systematic generalization on lexical inference tasks. |
| Approach: | They propose a model that models LSRs directly in the LM’s hidden states throughout pretraining by allowing them to be controlled by the model. |
| Outcome: | The proposed model improves performance on hypernymy-informed tasks and on the original LM objective. |
CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have found that information relevant to the next token prediction task accumulates in the hidden representations of just a few tokens. |
| Approach: | They propose a method that integrates attention preferences useful for a downstream task into the eviction process of hidden states. |
| Outcome: | The proposed method performs better on comprehension and retrieval tasks while preserving language modeling perplexity. |
Qualitative Code Suggestion: A Human-Centric Approach to Qualitative Coding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Qualitative coding is a content analysis method that assigns descriptive labels or qualitative codes to passages. |
| Approach: | They propose a qualitative code suggestion task where a ranked list of previously assigned qualitative codes is suggested from an identified passage. |
| Outcome: | The proposed method integrates previously ignored properties such as the sequence in which passages are annotated, the importance of rare codes and the differences in annotation styles between coders. |
Characterizing Idioms: Conventionality and Contingency (2022.acl-long)
Copied to clipboard
| Challenge: | idioms have non-canonical meanings, but non-conventional meanings are contingent on other words . a recent study shows that idiomatic expressions are not homogeneous among idiomas . |
| Approach: | They propose to use a contingency relationship between words in an idiom and non-canonical meanings of words in the idiome. |
| Outcome: | a new study shows that idioms fall at the expected intersection of the two dimensions, but that the dimensions themselves are not correlated. |
Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective (2024.findings-acl)
Copied to clipboard
| Challenge: | a recent study shows that evaluations of CR models on multiple datasets conflate different factors concerning what is being measured. |
| Approach: | They propose to view evaluations through the lens of measurement modeling . they show that evaluations risk conflating different factors concerning what is being measured . |
| Outcome: | The evaluations on seven datasets show that models that reflect coreference generalization are often correlated with differences in how coreference is defined and operationalized. |
Separately Parameterizing Singleton Detection Improves End-to-end Neural Coreference Resolution (2024.naacl-short)
Copied to clipboard
| Challenge: | Current end-to-end coreference resolution models combine detection of singleton mentions and antecedent linking into a single step. |
| Approach: | They add a singleton detector to a coarse-to-fine coreference model and design an anaphoricity-aware span embedding and singletont detection loss. |
| Outcome: | The proposed method significantly improves model performance on OntoNotes and four additional datasets. |
Investigating the Effect of Pre-finetuning BERT Models on NLI Involving Presuppositions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a study of presupposition, discourse and sarcasm suggests that pre-finetuning can improve models' performance on presimplified cases. |
| Approach: | They propose to leverage the connection between presupposition, discourse and sarcasm to improve models' performance. |
| Outcome: | The proposed model improves on cases involving presupposition by pre-finetuning on additional tasks and datasets. |
Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing work on question answering focuses on the pre-deployment stage; building an accurate model for deployment. |
| Approach: | They collect feedback from users and train a neural model with the feedback data. |
| Outcome: | The proposed model can explain the correctness or incorrectness of an answer. |
COSMIC: Mutual Information for Task-Agnostic Summarization Evaluation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for summarizing text are not well aligned with human judgments. |
| Approach: | They propose a task-oriented evaluation approach that assesses the quality of summarizers based on their capacity to produce summaries while preserving task outcomes. |
| Outcome: | The proposed method is able to predict task performance in a variety of contexts and tasks. |
Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing evidence suggests that pre-trained Transformers encode commonsense knowledge . however, the extent to which this knowledge is acquired is unclear . |
| Approach: | They inject verbalized knowledge into pre-training minibatches and evaluate generalization . they find generalization does not improve over the course of pre- training from scratch . |
| Outcome: | The proposed model generalizes to supported inferences after pre-training on the injected knowledge. |
From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards (2024.findings-acl)
Copied to clipboard
Khaoula Chehbouni, Megha Roshan, Emmanuel Ma, Futian Wei, Afaf Taik, Jackie Cheung, Golnoosh Farnadi
| Challenge: | Recent advances in large language models have also introduced additional safety risks and raised concerns regarding their detrimental impact on already marginalized populations. |
| Approach: | They propose to use LLMs to evaluate their safety responses on already mitigated biases by evaluating models on already encoded assumptions. |
| Outcome: | The proposed model can encode harmful assumptions, but it can also be harmful for certain demographic groups. |
Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization (2022.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art abstractive summarization systems often generate hallucinations, i.e., content that is not directly inferable from the source document. |
| Approach: | They propose a detection approach that separates factual from non-factual hallucinations of entities by masked language models. |
| Outcome: | The proposed method outperforms baselines in accuracy and F1 scores and has a strong correlation with human judgments on factuality classification tasks. |