Linguistic difference of human-human and human-chatbot dialogues about COVID-19 in the russian language

Abstract

This work describes the quantitative analysis of the linguistic difference in human-human and human-chatbot dialogues. The research is based on conducting a set of experiments where respondents communicate with a human or a chatbot in the domain of COVID-19 questions. In the case of the human-human dialogues, the approach of the inverted “Wizard of Oz” experimental setting is used. During the experiments, 35 human-human and 68 human-chatbot dialogues in Russian language were performed. The dialogues were collected during 4 months and thereafter analyzed with a set of quantitative text measures such as descriptive statistics of a text, syntactic complexity, lexical density, and readability. As a result, a set of measures demonstrated a statistically significant linguistic difference between the language structure of questions that were asked to the human and to the chatbot. Specifically, respondents were using shorter sentences and words, simpler syntax while communicating with the chatbot. Moreover, lexical richness of the human-chatbot dialogue data is lower while the readability is higher – these markers indicate that humans use simpler language constructions while speaking with a chatbot.