Papers by Najoung Kim
Are they lovers or friends? Evaluating LLMs’ Social Reasoning in English and Korean Dialogues (2026.acl-long)
Copied to clipboard
Eunsu Kim, Junyeong Park, Juhyun Oh, Kiwoong Park, Seyoung Song, A. Seza Doğruöz, Alice Oh, Najoung Kim
| Challenge: | Existing studies on LLMs' ability to infer social relationships have limited results for Korean and English. |
| Approach: | They propose a social reasoning task based on a 1.1k-dialogue dataset in English and Korean sourced from movie scripts to evaluate LLMs' ability to infer the social relationships between speakers. |
| Outcome: | The proposed task evaluates the ability of LLMs to infer the social relationships between speakers in 1.1k-dialogue datasets in English and Korean. |
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2024.naacl-long)
Copied to clipboard
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, Yoon Kim
| Challenge: | Recent language models possess impressive performance across a wide range of tasks . however, they often rely on narrow, non-transferable procedures for task-solving . |
| Approach: | They propose to evaluate language models using "counterfactual" task variants that deviate from standard tasks. |
| Outcome: | The proposed framework shows that language models perform better on a wide range of tasks compared to the default conditions. |
CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation protocols for text generation suffer from rating inconsistencies . lexical overlap-based metrics align poorly with human judgments . |
| Approach: | They propose a checklist-based evaluation framework that improves rating reliability via decomposed binary questions. |
| Outcome: | The proposed framework improves rating reliability by decomposing binary questions . it improves agreement across evaluator models by 0.45 and reduces score variance . human evaluation remains the gold standard, but it #, Equal contribution. |
COGS: A Compositional Generalization Challenge Based on Semantic Interpretation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Natural language is characterized by compositionality: meaning of complex expressions is constructed from the meanings of its constituent parts. |
| Approach: | They propose a semantic parsing dataset based on a fragment of English to assess compositional generalization abilities. |
| Outcome: | The proposed model can generalize meanings in a given sentence in 96–99% of the tests, but generalization accuracy is lower and the generalization sensitivity is higher. |
Inverse Scaling Can Become U-Shaped (2023.emnlp-main)
Copied to clipboard
| Challenge: | Scaling up language models has been shown to improve performance on a wide range of downstream tasks, but are there any tasks for which performance gets worse as models scale? |
| Approach: | They evaluate models trained on five times more compute and evaluated them on 280B parameters and 500 zettaFLOPs of training compute. |
| Outcome: | The proposed tasks show that performance decreases as models scale and increases again as models get larger. |
Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering (2021.acl-long)
Copied to clipboard
| Challenge: | Existing Question-Answering (QA) datasets contain unanswerable questions . however, their treatment in QA systems remains primitive . |
| Approach: | They propose a framework that provides answers based on presupposition failure over oracle behavior of existing QA systems. |
| Outcome: | The proposed system provides responses based on presupposition failure over oracle behavior of existing QA systems. |
Entity Tracking in Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies on the ability of large language models to track discourse entities have not been conducted. |
| Approach: | They propose to investigate whether large language models can track entities . they first investigate whether Flan-T5, GPT-3 and GPT-3.5 can track the state of entities based on an English description of the initial state and a series of state-changing operations. |
| Outcome: | The proposed task investigates whether language models can track entities based on language descriptions and state-changing operations. |
RExBench: Can coding agents autonomously implement AI research extensions? (2026.acl-long)
Copied to clipboard
| Challenge: | Existing large language model (LLM) agents are not capable of performing research extension tasks autonomously. |
| Approach: | They propose a benchmark to evaluate LLM agents' ability to extend existing AI research . they use extensions of 12 recently published research papers accompanied by domain expert-written instructions . |
| Outcome: | The proposed benchmark evaluates 12 LLM agents implemented using aider and OpenHands. |
SLOG: A Structural Generalization Benchmark for Semantic Parsing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing compositional generalization benchmarks focus on lexical generalisation, the interpretation of novel lexicals in syntactic structures familiar from training. |
| Approach: | They propose a semantic parsing dataset that extends COGS with 17 structural generalization cases to evaluate how well models generalize to new complex linguistic expressions. |
| Outcome: | The proposed model generalization accuracy is far below the near-perfect accuracy of existing models on COGS, demonstrating the role of SLOG in foregrounding the large discrepancy between models’ lexical and structural generalization capacities. |
(QA)2: Question Answering with Questionable Assumptions (2023.acl-long)
Copied to clipboard
| Challenge: | a number of questions contain questionable assumptions, such as when did Marie Curie discover Uranium, that cannot be answered as a true when question. |
| Approach: | They propose an open-domain evaluation dataset that can detect questionable assumptions . they propose a method that can be used to produce adequate responses for questions with questionable assumption. |
| Outcome: | The proposed model detects questionable assumptions and produces adequate responses for both types of questions. |
LAMBADA: Backward Chaining for Automated Reasoning in Natural Language (2023.acl-long)
Copied to clipboard
| Challenge: | Recent advances in automated reasoning with natural text suffer from a combinatorial explosion of the search space and high failure rates for problems requiring longer chains of reasoning. |
| Approach: | They propose a Backward Chaining algorithm that decomposes reasoning into four sub-modules and implements it by few-shot prompted LLM inference. |
| Outcome: | The proposed algorithm achieves sizable accuracy boosts over state-of-the-art forward reasoning methods on two challenging logical reasoning datasets. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
Implicit Discourse Relation Classification: We Need to Talk about Evaluation (2020.acl-main)
Copied to clipboard
| Challenge: | Lack of consistency in preprocessing and evaluation poses challenges to fair comparison of results in literature. |
| Approach: | They propose an improved evaluation protocol for implicit relation classification on PDTB 2.0 . they report strong baseline results from pretrained sentence encoders . |
| Outcome: | The proposed evaluation protocol improves the existing framework and provides strong baseline results. |
Semantic Training Signals Promote Hierarchical Syntactic Generalization in Transformers (2024.emnlp-main)
Copied to clipboard
| Challenge: | Neural networks without hierarchical biases struggle to learn linguistic rules that come naturally to humans . et al., 2018: Transformers trained on form and meaning favor hierarchically generalization more than those trained on forms alone. |
| Approach: | They examine whether neural networks without hierarchical biases can generalize more like humans . they find that Transformers trained on form and meaning favor hierarchic generalization . |
| Outcome: | The proposed neural networks perform better on syntactic evaluations when trained on form and meaning compared to those trained on forms alone. |
Personas as a Way to Model Truthfulness in Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models are trained on vast amounts of text from the internet, which contains factual and misleading information. |
| Approach: | They hypothesize that the pretraining data is generated by groups of (un)truthful agents whose outputs share common features and form a (un-truthfully persona) this allows the model to separate truth from falsehoods and controls the truthfulness of its generation. |
| Outcome: | The proposed model can infer truth from falsehoods by finetuning its model on a set of facts and finetuned it on unseen topics. |
Reconstruction Probing (2023.findings-acl)
Copied to clipboard
| Challenge: | a new analysis method for contextualized representations is proposed . contextualization boosts reconstructability of tokens close to the token being reconstructed . |
| Approach: | They propose a method for contextualized representations based on reconstruction probabilities in masked language models. |
| Outcome: | The proposed method compares reconstruction probabilities of tokens in masked language models . it finds that contextualization boosts reconstructability of token that are close to the token being reconstructed . |