Evaluating the Quality of a Corpus Annotation Scheme Using Pretrained Language Models

By Furkan Akkurt, O. Gungor, Büsra Marsan, Tunga Gungor, Balkiz Ozturk Basaran, Arzucan Özgür, S. Uskudarli. International Conference on Language Resources and Evaluation, 2024

5 citations recorded by the source corpus.

Pretrained language models and large language models are increasingly used to assist in a great variety of natural language tasks. In this work, we explore their use in evaluating the quality of alternative corpus annotation schemes. For this purpose, we analyze two alternative annotations of the Turkish BOUN treebank, versions 2.8 and 2.11, in the Universal Dependencies framework using large language models. Using a suitable prompt generated using treebank annotations, large language models are used to recover the surface forms of sentences. Based on the idea that the large language models capture the characteristics of the languages, we expect that the better annotation scheme would yield the sentences with higher success. The experiments conducted on a subset of the treebank show that the new annotation scheme (2.11) results in a successful recovery percentage of about 2 points higher. All the code developed for this work is available at https://github.com/boun-tabi/eval-ud .

Subfields: Large Language Models, Evaluation & Benchmarks, Computational Linguistics & Semantics

View Paper: https://doi.org/10.63317/2zy52xp7m2v5

Citations in the corpus

• Extending Retag to Conversion Error Detection: A Case Study on SynTagRus Morphology (2026, unknown): “A common proxy for annotation quality is to measure performance on tasks that depend on the annotation, such as parsing (Johannsen et al., 2015; Türk et al., 2019), morphological tagging (Pyysalo et al., 2015; Mc-Carthy et al., 2018), or language modeling (Akkurt et al., 2024).”
• Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies (2026, unknown): “UD data have also been used to evaluate LLMs performance on tasks such as morphological analysis and syntactic annotation (Akkurt et al., 2024; Acs et al., 2024).”
• Analysis of Three Complement-Taking Predicates in Spoken Turkish: bil, san, and zannet (2025, unknown): “VOS orders were not observed by Slobin and Bever (1982) and recent research shows that VOS orders are rare in written Turkish (0.15%; Akkurt et al., 2024).”
• Large corpora and large language models: a replicable method for automating grammatical annotation (2024, unknown): “…can be an e ff ective and reliable tool for a range of di ff erent topics, such as sentiment analysis (Belal et al. 2023), the evaluation of corpus annotation schemes in treebanks (Akkurt et al. 2024), and the annotation of pragmatic and discourse related phenomena in corpora (Yu et al. 2024).”