BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//ºÚÁÏÍø//NONSGML Events//EN
METHOD:PUBLISH
BEGIN:VTIMEZONE
TZID:Europe/London
BEGIN:DAYLIGHT
TZOFFSETFROM:+0000
TZOFFSETTO:+0100
DTSTART:20210328T010000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU
TZNAME:BST
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0100
TZOFFSETTO:+0000
DTSTART:20211031T020000
RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU
TZNAME:GMT
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
UID:25417
DTSTAMP:20261103T024920Z
DTSTART;TZID=Europe/London:20261111T140000
DTEND;TZID=Europe/London:20261111T150000
LOCATION:Meston Lecture Theatre 1
SUMMARY:CS Seminar by Dr Yulong Chen
DESCRIPTION:Speaker: Dr Yulong Chen, ºÚÁÏÍø (/people/yulong.chen)

Title: Benchmarking Metalinguistic Reasoning in LLMs with Camlang

Location: Microsoft Teams and MT 1 (Meston Lecture Theatre 1)

Date: Wednesday, November 11th, 14:00 - 15:00 (UK time)

Abstract:

Modern large language models (LLMs) perform impressively on many language tasks, but it remains unclear how much of this success reflects the reliable and generalisable use of knowledge and reasoning abilities, rather than reliance on familiar patterns from training. From a cognitive science perspective, an informative test is whether a system can master an unfamiliar language from explicit resources, a setting where human learners can internalise grammatical systems through metalinguistic awareness and reasoning. To realise such an evaluation, we introduce Camlang, a typologically plausible yet novel constructed language. Camlang is implemented through two explicit resources: a grammar book, which specifies grammatical rules, and a bilingual dictionary, which provides the lexical-semantic interface. This separation between grammar and lexicon allows us to distinguish reliable grammar-based reasoning from answers driven by shallow lexical cues or guessing. The resources are iteratively refined through multiple rounds of human language use. The final resources comprise a grammar book with 203 applicable rules, covering 134 WALS-style features, and a bilingual dictionary with 1,511 entries. Camlang is empirically found to be typologically distinct from documented languages and proven to be learnable by humans in both comprehension and production. Using these resources, we construct Camlang-CSQA, by manually translating CommonSenseQA into Camlang. Camlang-CSQA requires systems to parse novel grammar rules, perform lexical lookup, and integrate the acquired linguistic knowledge with commonsense reasoning. Experiments show that, while a human participant achieves 87% accuracy in Camlang and 91% in English, state-of-the-art models exhibit greater performance degradation. Our analysis further suggests that most model successes stem from shallow lexical alignment rather than systematic grammatical mastery, and even GPT-5 shows only partial rule-following and emerging metalinguistic awareness.

Short bio: Dr Yulong Chen is a Lecturer in Computing Science at the ºÚÁÏÍø. Before joining Aberdeen, he was an Affiliated Lecturer and Research Associate at the University of Cambridge. He holds a PhD from Zhejiang University and Westlake University and an MSc from the University of Edinburgh. His research focuses on natural language processing (NLP), large language models (LLMs), and trustworthy AI, with particular interests in model evaluation, automated fact-checking, text generation, and multilingual NLP. He has published over 30 research papers and has served in editorial and organisational roles at leading NLP conferences, including ACL and EMNLP. Visit /events/25417 for further details about this event.
FBTYPE:BUSY-TENTATIVE
X-MICROSOFT-CDO-BUSYSTATUS:TENTATIVE
END:VEVENT
END:VCALENDAR