Comparison of automatic vs. manual language identification in multilingual social media texts

Publication type
B2
Publication status
Published
Authors
Frey, J., Stemle, E., & Doğruöz, A.S.
Editor
Ciara R. Wigham and Egon W. Stemle
Series
Building computer mediated corpora for sociolinguistic analysis
Volume
8
Pagination
28-44
Publisher
Presses universitaires Blaise Pascal
Download
(.pdf)
View in Biblio
(externe link)

Abstract

Multilingual speakers communicate in more than one language in daily life and on social media. In order to process or investigate multilingual communication, there is a need for language identification. This study compares the performance of human annotators with automatic ways of language identification on a multilingual (mainly German-Italian-English) social media corpus collected in South Tyrol, Italy. Our results indicate that humans and Natural Language Processing (NLP) systems follow their individual techniques to make a decision about multilingual text messages. This results in low agreement when different annotators or NLP systems execute the same task. In general, annotators agree with each other more than NLP systems. However, there is also variation in human agreement depending on the prior establishment of guidelines for the annotation task or not.