Papers by Freda Shi
On the Effect of Hyperparameters in Language Modeling for Computational Linguistics (2026.acl-long)
Copied to clipboard
| Challenge: | Training language models and examining their linguistic behaviors is a common protocol in computational linguistics for studying linguistic phenomena and modeling human language processing. |
| Approach: | They replicate three prior studies with hyperparameters varied within a practical range and show that modest hyperparametric changes can alter qualitative conclusions about models’ linguistic abilities. |
| Outcome: | The results show that hyperparameter changes can alter qualitative conclusions and reverse the ranking of models. |
SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data (2025.acl-long)
Copied to clipboard
| Challenge: | Vision-language models struggle with spatial reasoning, a skill that humans excel at. |
| Approach: | They propose to use a spatial-reasoning Enhanced (SpaRE) VLM to improve spatial reasoning in visual question answering and robotics. |
| Outcome: | The proposed model achieves a 49% performance gain on the What's Up benchmark while maintaining strong results on general tasks. |
Translation or Recitation? Calibrating Evaluation Scores for Machine Translation of Extremely Low-Resource Languages (2026.acl-short)
Copied to clipboard
| Challenge: | Existing studies show that performance across low-resource settings is variable, resulting in a significant barrier for the MT community. |
| Approach: | They propose to use FRED Difficulty Metrics to contextualize reported performance across different language pairs to determine whether breakthroughs reported in other contexts are artifacts of benchmark collection. |
| Outcome: | The proposed metrics explain a significant portion of result variability rather than model capability. |
Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing (2024.acl-long)
Copied to clipboard
| Challenge: | Recent work has proposed a new task of textless speech constituency parsing that uses textless parsers to parse spoken word boundaries over automatically recognized spoken word borders. |
| Approach: | They propose a metric that compares a constituency parse tree over spoken word boundaries with a ground-truth parser tree over written words. |
| Outcome: | The proposed metric shows higher tolerance to syntactically plausible parses than PARSEVAL. |
Natural Language to Code Translation with Execution (2022.emnlp-main)
Copied to clipboard
| Challenge: | Generative code models do not explicitly incorporate program semantics during training, but they are able to generate correct solutions for many problems. |
| Approach: | They introduce execution result-based minimum Bayes risk decoding for program selection . they select output programs from a generated candidate set by marginalizing over implementations that share the same semantics . |
| Outcome: | The proposed model outperforms all other methods on natural language-to-code translation. |
LingGym: How Far Are LLMs from Thinking Like Field Linguists? (2025.emnlp-main)
Copied to clipboard
| Challenge: | LINGGYM is a benchmark that evaluates LLMs’ capacity for meta-linguistic reasoning using Interlinear Glossed Text and grammatical descriptions extracted from 18 typologically diverse reference grammars. |
| Approach: | They propose a benchmark that evaluates LLMs’ capacity for meta-linguistic reasoning using Interlinear Glossed Text and grammatical descriptions extracted from 18 typologically diverse reference grammars. |
| Outcome: | The proposed model can generalize linguistic inference across low-resource languages and structures not seen during training. |
FORG3D: Flexible Object Rendering for Generating Vision-Language Spatial Reasoning Data from 3D Scenes (2025.acl-demo)
Copied to clipboard
| Challenge: | FORG3D synthesizes vision-language data for use in cognitive experiments . it provides precise control over object placement, orientation, and camera configurations . |
| Approach: | They propose a 3D rendering toolkit that synthesizes vision-language data with Blender and Python. |
| Outcome: | The toolkit synthesizes vision-language data for two primary purposes: supporting cognitive experiments and improving visual reasoning capabilities of large vision-linguistic models. |
From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens (2025.emnlp-demos)
Copied to clipboard
Hala Sheta, Eric Haoran Huang, Shuyu Wu, Ilia Alenabi, Jiajun Hong, Ryker Lin, Ruoxi Ning, Daniel Wei, Jialin Yang, Jiawei Zhou, Ziqiao Ma, Freda Shi
| Challenge: | Existing vision-language models are based on exactmatch based accuracy and its derivations to evaluate performance. |
| Approach: | They propose a toolkit that supports systematic benchmarking, analysis, and interpretation of vision-language models by extracting intermediate outputs from any layer during the forward pass of open-source VLMs. |
| Outcome: | The proposed toolkit supports 16 state-of-the-art base VLMs and their over 30 variants and is extensible to accommodate new models without changing the core logic. |
How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them (2026.acl-long)
Copied to clipboard
| Challenge: | Tokenization is the first step in every language model (LM), yet it never takes the sounds of words into account. |
| Approach: | They propose a lightweight IPA-based fine-tuning method that infuses phonological awareness into LMs. |
| Outcome: | The proposed method improves phonological awareness across three phonology-related tasks while preserving math and general reasoning ability. |
Distribution Prompting: Understanding the Expressivity of Language Models Through the Next-Token Distributions They Can Produce (2025.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive neural language models (LMs) generate a probability distribution over tokens at each time step given a prompt. |
| Approach: | They propose to find a prompt that induces LMs to output a distribution as close as possible to the target, using either soft or hard gradient-based prompt tuning. |
| Outcome: | The proposed model is able to generate a distribution as close as possible to a target given a prompt, and it can be used to approximate distributions with low or high entropy. |
Blessing of Multilinguality: A Systematic Analysis of Multilingual In-Context Learning (2025.findings-acl)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a widely adopted technique for learning large language models . however, there is little systematic understanding of when and why it works well . |
| Approach: | They analyze multilingual in-context learning using demonstrations in HRLs to enhance cross-lingual transfer. |
| Outcome: | The proposed method outperforms English-only models on high-resource languages . the study shows that the presence of irrelevant non-English sentences in the prompt yields measurable gains . |
Logical forms complement probability in understanding language model (and human) performance (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies on LLMs have shown that they perform well on logical reasoning problems, but there is still a lack of fine-grained understanding of the logical forms. |
| Approach: | They propose a dataset of hypothetical and disjunctive syllogisms in propositional and modal logic and use it as the testbed for understanding LLM performance. |
| Outcome: | The proposed model performs well on proposi-tional and modal logics, but does it exhibit preferences for certain argument forms? |
Substructure Distribution Projection for Zero-Shot Cross-Lingual Dependency Parsing (2022.acl-long)
Copied to clipboard
| Challenge: | Substructure distribution projection (SubDP) projects a distribution over structures in one domain to another, using substructure distributions as silver labels. |
| Approach: | They evaluate substructure distribution projection technique on zero shot cross-lingual dependency parsing using dependency arcs as substructures. |
| Outcome: | The proposed technique achieves better unlabeled attachment score than previous work on eight target languages and best labeled attachment scores on six languages. |
LogogramNLP: Comparing Visual and Textual Representations of Ancient Logographic Writing Systems for NLP (2024.acl-long)
Copied to clipboard
| Challenge: | Existing pipelines for natural language processing only process symbolic representations of language, which are labor-intensive and noisy . a large portion of logographic data persists in a purely visual form due to the absence of transcription . this issue poses a bottleneck for researchers seeking to apply NLP to ancient logographic languages . |
| Approach: | They propose a benchmark for NLP analysis of ancient logographic languages using visual representations of writing. |
| Outcome: | The proposed pipeline outperforms existing pipelines for some tasks . the results could unlock large amounts of cultural heritage data of ancient logographic languages . |
Knowledge Distillation for Language Models (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) aims to transfer knowledge from a teacher to a student . this tutorial will cover topics ranging from LLM sequence compression to LLM self-distillation . |
| Approach: | They propose to introduce intermediate-layer matching and prediction matching . they will then present advanced techniques such as reinforcement learning-based KD and multi-teacher distillation . |
| Outcome: | This tutorial aims to provide participants with a comprehensive understanding of the techniques and applications of knowledge distillation for language models. |
Learning Language through Grounding (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | This tutorial provides a historical overview of grounding and discusses its use in computational linguistics and in computational language processing. |
| Approach: | They introduce the concept of grounding and discuss future directions and open challenges . they will delve into recent progress in learning lexical semantics, syntax, and complex meanings through various forms of ground. |
| Outcome: | This course will provide an overview of the field of grounding and discuss future directions and challenges related to large language models and scaling. |