Papers by Mariko Kato
Token-based Decision Criteria Are Suboptimal in In-context Learning (2025.naacl-long)
Copied to clipboard
| Challenge: | In-Context Learning (ICL) typically utilizes output probabilities of manually selected label tokens, but such calibrations lead to suboptimal decision boundaries. |
| Approach: | They propose a method which renounces token probabilities and uses the nearest centroid classifier on the Language Model’s last hidden states to predict the label of the nearest ctroid. |
| Outcome: | The proposed method outperforms current token-based baselines by about 20%50% and provides a strong state-of-the-art in ICL. |
Understanding Token Probability Encoding in Output Embeddings (2025.coling-main)
Copied to clipboard
| Challenge: | a common log-linear encoding of output token probabilities is used in language models, but it is sparse and inaccurate. |
| Approach: | They propose an approximate log-linear encoding of output token probabilities within the output embedding vectors and show that it is accurate and sparse. |
| Outcome: | The proposed output embeddings capture the corpus token frequency information in early steps, even before an obvious convergence of parameters starts. |