Papers by Nikolay Malkin
Studying word order through iterative shuffling (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on large language models has made this hypothesis popular . but, word order is not important enough to make sentence structure relevant . |
| Approach: | They propose an efficient procedure that finds word order having highest likelihood under a fixed language model. |
| Outcome: | The proposed procedure can be used to find the ordering of a bag of words having the highest likelihood under a fixed language model. |
Mixtures of In-Context Learners (2025.acl-long)
Copied to clipboard
| Challenge: | In-context learning is sensitive to the choice of in-con context demonstrations and processing many demonstrations can be computationally demanding. |
| Approach: | They propose a method that uses subsets of demonstrations to train experts via ICL and learns a weighting function to merge their output distributions via gradient-based optimisation. |
| Outcome: | The proposed approach improves on 5 out of 7 classification datasets compared to strong baselines and reduces the inference time needed to achieve the same performance with fewer demonstrations. |
ThinkSum: Probabilistic reasoning over sets using large language models (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have a substantial capacity for high-level analogical reasoning, but they fail in scenarios that require reasoning over multiple objects or facts and making sequences of logical deductions. |
| Approach: | They propose a two-stage probabilistic inference paradigm, ThinkSum, which reasons over sets of objects or facts in a structured manner. |
| Outcome: | The proposed paradigm improves on the BIG-bench suite of evaluation tasks. |
GPT Perdetry Test: Generating new meanings for new words (2021.naacl-main)
Copied to clipboard
| Challenge: | We create a set of nonce words and prompt GPT-3 to generate their dictionary definitions. |
| Approach: | They create a set of nonce words and prompt GPT-3 to generate their dictionary definitions. |
| Outcome: | The proposed model can process new words and make them 'neologisms' . it can also adapt to and extend a changing vocabulary, the authors found . |
Coherence boosting: When your pretrained language model is not paying enough attention (2022.acl-long)
Copied to clipboard
| Challenge: | Long-range semantic coherence remains a challenge in automatic language generation and understanding. |
| Approach: | They propose a procedure that increases a model’s focus on a long context by distributional analyses of generated ordinary text and dialog responses. |
| Outcome: | The proposed procedure increases the model's focus on a long context. |