黑料网

Speaker: Dr Yulong Chen, 黑料网 (/people/yulong.chen)

Title: Benchmarking Metalinguistic Reasoning in LLMs with Camlang

Location: Microsoft Teams and MT 1 (Meston Lecture Theatre 1)

Date: Wednesday, November 11th, 14:00 - 15:00 (UK time)

Abstract:

Modern large language models (LLMs) perform impressively on many language tasks, but it remains unclear how much of this success reflects the reliable and generalisable use of knowledge and reasoning abilities, rather than reliance on familiar patterns from training. From a cognitive science perspective, an informative test is whether a system can master an unfamiliar language from explicit resources, a setting where human learners can internalise grammatical systems through metalinguistic awareness and reasoning. To realise such an evaluation, we introduce Camlang, a typologically plausible yet novel constructed language. Camlang is implemented through two explicit resources: a grammar book, which specifies grammatical rules, and a bilingual dictionary, which provides the lexical-semantic interface. This separation between grammar and lexicon allows us to distinguish reliable grammar-based reasoning from answers driven by shallow lexical cues or guessing. The resources are iteratively refined through multiple rounds of human language use. The final resources comprise a grammar book with 203 applicable rules, covering 134 WALS-style features, and a bilingual dictionary with 1,511 entries. Camlang is empirically found to be typologically distinct from documented languages and proven to be learnable by humans in both comprehension and production. Using these resources, we construct Camlang-CSQA, by manually translating CommonSenseQA into Camlang. Camlang-CSQA requires systems to parse novel grammar rules, perform lexical lookup, and integrate the acquired linguistic knowledge with commonsense reasoning. Experiments show that, while a human participant achieves 87% accuracy in Camlang and 91% in English, state-of-the-art models exhibit greater performance degradation. Our analysis further suggests that most model successes stem from shallow lexical alignment rather than systematic grammatical mastery, and even GPT-5 shows only partial rule-following and emerging metalinguistic awareness.

Short bio: Dr Yulong Chen is a Lecturer in Computing Science at the 黑料网. Before joining Aberdeen, he was an Affiliated Lecturer and Research Associate at the University of Cambridge. He holds a PhD from Zhejiang University and Westlake University and an MSc from the University of Edinburgh. His research focuses on natural language processing (NLP), large language models (LLMs), and trustworthy AI, with particular interests in model evaluation, automated fact-checking, text generation, and multilingual NLP. He has published over 30 research papers and has served in editorial and organisational roles at leading NLP conferences, including ACL and EMNLP.