Papers by Ta-Chung Chi
Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models of working memory are limited by self-attention, but we propose a new variant of regular programming that can model regular languages. |
| Approach: | They propose a Transformer variant that constructs working memory along the depth dimension . they find it rediscovers the local windowed attention effect needed for length extrapolation . |
| Outcome: | The proposed model rediscovers the local windowed attention effect needed for length extrapolation. |
CLUSE: Cross-Lingual Unsupervised Sense Embeddings (D18-1)
Copied to clipboard
| Challenge: | Existing models for learning bilingual sense embeddings that encode semantics are weak for recognizing multi-sense word representations . elucidation of word embeddables is difficult because they do not allow a word to have different meanings in different contexts. |
| Approach: | They propose a sense induction and representation learning model that learns bilingual sense embeddings that align well in the vector space. |
| Outcome: | The proposed model shows that the learned embeddings are aligned well in the vector space. |
PESCO: Prompt-enhanced Self Contrastive Learning for Zero-shot Text Classification (2023.acl-long)
Copied to clipboard
| Challenge: | Existing text classification frameworks require large amounts of human-labeled documents to train . |
| Approach: | They propose a contrastive learning framework that improves zero-shot text classification . they add prompts to enhance label retrieval and use retrieved labels to enrich training . |
| Outcome: | The proposed framework achieves state-of-the-art on four benchmark text classification datasets. |
Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for length extrapolation are tailored for natural language modeling, a task known to have strong recency bias. |
| Approach: | They propose two attention alignment strategies to improve T5's long-context utilization capability without fine-tuning. |
| Outcome: | The proposed methods improve the long-context utilization capability of T5 on language modeling, retrieval, multi-document question answering, and code completion tasks without any fine-tuning. |
Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings (2023.acl-short)
Copied to clipboard
| Challenge: | Recent research has called into question the necessity of positional embeddings in transformer language models. |
| Approach: | They propose to discard positional embeddings in transformer language models to facilitate more efficient pretraining. |
| Outcome: | The proposed model encodes strong positional information through shrinkage of self-attention variance. |
Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis (2023.acl-long)
Copied to clipboard
| Challenge: | Length extrapolation allows training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. |
| Approach: | They propose a relative positional embedding design that uses longer than the training sequence to create sandwich. |
| Outcome: | The proposed model can extrapolate to L ex L tr much better than other models. |
Advancing Regular Language Reasoning in Linear Recurrent Neural Networks (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing linear recurrent neural networks have been used for natural language and long-range modeling for decades. |
| Approach: | They propose a linear recurrent neural network with a block-diagonal transition matrix and a transition matrix for LRNNs. |
| Outcome: | The proposed model is the only one capable of performing length extrapolation on regular language tasks such as Sum, Even Pair, and Modular Arithmetic. |
Zero-Shot Dialogue Disentanglement by Self-Supervised Entangled Response Selection (2021.emnlp-main)
Copied to clipboard
| Challenge: | a zero-shot dialogue disentanglement solution is difficult due to the need for manual annotation. |
| Approach: | They propose a zero-shot dialogue disentanglement solution using a web dataset . they train a model on the data and fine-tune the model using labeled data . |
| Outcome: | The proposed model achieves a cluster F1 score of 25 without labeling data . it can be used to analyze discourses and to perform response selection . |