Papers by Samson Tan
ReCode: Robustness Evaluation of Code Generation Models (2023.acl-long)
Copied to clipboard
Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, Bing Xiang
| Challenge: | Existing work on robustness in text or code tasks has focused on classification, while robustness for code generation tasks is an uncharted area. |
| Approach: | They propose a robustness evaluation benchmark for code generation models that customizes over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format. |
| Outcome: | The proposed model performs better on human annotators and on SOTA models with human annnotators. |
Fine-tuning Language Models for Joint Rewriting and Completion of Code with Potential Bugs (2024.findings-acl)
Copied to clipboard
| Challenge: | Previous work has demonstrated shortcomings of large language models of code (CodeLLMs) in completing drafty partial code with potential bugs. |
| Approach: | They propose to use large language models of code to fine-tune their models to rewrite and complete drafty partial code into functional full programs. |
| Outcome: | The proposed approach achieves superior pass rates over baselines and preserves the integrity of the original partial implementations. |
It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on societal bias in NLP focuses on race and gender . linguistic background is a unique attribute that has been largely ignored in the field . |
| Approach: | They examine linguistic background to craft plausible adversarial examples that expose biases in popular NLP models. |
| Outcome: | The proposed model improves robustness without sacrificing performance on clean data. |
Learning to Generate Answers with Citations via Factual Consistency Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) frequently hallucinate, impeding their reliability in mission-critical situations. |
| Approach: | They propose a weakly-supervised fine-tuning method leveraging factual consistency models (FCMs) they propose to generate citations to retrieved text passages alongside LLM-produced statements . |
| Outcome: | The proposed method improves on the ALCE few-shot citation benchmark with various instruction-tuned LLMs with an average improvement of 34.1, 15.5, and 10.5 citation F1 points. |
Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encoding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Inflectional variation is a common feature of World Englishes such as Colloquial Singapore English and African American Vernacular English. |
| Approach: | They propose a method to tokenize English text by reducing inflected words to their base forms before reinjecting the grammatical information as special symbols. |
| Outcome: | The proposed method defends against inflectional adversaries while maintaining performance on clean data. |
Extreme Miscalibration and the Illusion of Adversarial Robustness (2024.acl-long)
Copied to clipboard
| Challenge: | emergence of the Adversarial Training paradigm has shown some success in training models to be more robust to these small adversarial perturbations. |
| Approach: | They propose to use adversarial examples to detect adversarials by miscalibrating models to mask gradients in a way that interferes with adversarial attack search methods. |
| Outcome: | The proposed model gains are an illusion of robustness (IOR) and urge the NLP community to incorporate test-time temperature scaling into their robustness evaluations to ensure that any observed gains are genuine. |
Interpreting the Robustness of Neural NLP Models to Textual Perturbations (2022.findings-acl)
Copied to clipboard
| Challenge: | Modern Natural Language Processing models are sensitive to input perturbations and their performance can decrease when applied to noisy data. |
| Approach: | They propose to explain the extent to which a model is affected by an unseen textual perturbation by the learnability of the perturbation. |
| Outcome: | The proposed model is better at identifying a perturbation (higher learnability) but worse at ignoring it (lower robustness). |
Whodunit? Learning to Contrast for Authorship Attribution (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing approaches to authorship attribution are dataset-dependent and yield inconsistent performance across corpora. |
| Approach: | They propose to fine-tune pre-trained generic language representations with a contrastive objective to learn author-specific representations by identifying clusters of authors. |
| Outcome: | The proposed approach improves on multiple human and machine authorship attribution benchmarks, but at the cost of sacrificing performance for some authors. |
Reliability Testing for Natural Language Processing Systems (2021.acl-long)
Copied to clipboard
| Challenge: | a lack of rigorous testing and ML implicit assumption of identical training and testing distributions may result in systems that discriminate against minorities. |
| Approach: | They argue that reliability testing is needed to address the issue of demographics . they argue that adversarial attacks can be reframed for this goal . |
| Outcome: | The proposed framework will enable rigorous and targeted testing and aid in the enactment and enforcement of industry standards. |
BotSIM: An End-to-End Bot Simulation Framework for Commercial Task-Oriented Dialog Systems (2022.emnlp-demos)
Copied to clipboard
| Challenge: | evaluating and troubleshooting production TOD systems is still a largely manual process requiring large amount of human conversations with the systems. |
| Approach: | They propose a data-efficient end-to-end Bot SIMulation framework for commercial task-oriented dialog systems that can generate user queries and generate semantic-level dialog acts and entities from bot definitions. |
| Outcome: | The proposed framework is able to infer semantic-level dialog acts and entities from bot definitions and generate user queries via model-based paraphrasing. |
TraVLR: Now You See It, Now You Don’t! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoning (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing visio-linguistic (V+L) models do not represent visual and linguistic concepts in a unified space. |
| Approach: | They propose to use cross-modal transfer to evaluate the extent to which visio-linguistic (V+L) representations are represented in a unified space. |
| Outcome: | The proposed evaluation settings include cross-modal transfer and a global accuracy score on the entire dataset making the specific sources of success and failure difficult to diagnose. |
Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots (2021.naacl-main)
Copied to clipboard
| Challenge: | Multilingual models have demonstrated impressive cross-lingual transfer abilities. |
| Approach: | They propose two strong adversarial attacks that target multilingual models that can handle code-mixed sentences using bilingual dictionaries. |
| Outcome: | The proposed model has a success rate of 89.75% against XLM-R-large, bringing its average accuracy of 79.85 down to 8.18 on XNLI. |