Are chatbots being optimized?

: unveiling the efficiency-effectiveness dilemma in rule-based and large language models chatbots

Abstract

As digital and technological evolution advances, the entrepreneurial world is adapting and capitalizing on these transformations for a diverse range of applications. This progress has consequently led to a dynamic change in consumers’ expectations, as they seek more personalized, efficient, and innovative experiences. Therefore, this paper focuses on one of these emerging technologies: automated messaging bots, or chatbots, applied to Customer Service. In the past few years, chatbots have been widely adopted by companies across all industries for customer service, however, as of 2023, a more advanced model of automated conversational systems with generative artificial intelligence was introduced in the equation – Large Langue Models chatbots. To the best of our knowledge and supported by the study by Kshetri et al. (2023), it remains to be clarified how this newest version differs in terms of performance from the long-standing one, hence this gap is the focus of the present study. In an effort to make a comparison between the two aforementioned chatbots, a mixed-methods approach was conducted. Initially, a quantitative analysis was performed to assess key performance metrics of the more traditional type. Subsequently, a qualitative analysis was undertaken to delve deeper into the nuances underlying the disparities observed in these performance metrics between the two versions. In terms of chatbot performance, in contrast to prior assumptions, our findings indicate that even though chatbots can be highly effective in resolving customer tickets, this ability does not influence customers to return to the same conversational interface. Nonetheless, it was found that the Efficiency rate, more than the Effectiveness rate, may determine whether customers will use the same automation again or opt for other channels of communication with the company, underlining the importance of streamlined chatbots. When comparing the two types across these parameters, it became evident that, for the time being, the more traditional chatbots still surpass the more technologically advanced ones. In addition to addressing the gap identified in the literature, this study has identified inherent characteristics of these generative artificial intelligence leveraged chatbots that hinder their performance. However, resolving these issues has the potential to elevate customer service to a level where the distinction between robot and human support can become indistinguishable.

Date of Award	10 Jul 2024
Original language	English
Awarding Institution	Universidade Católica Portuguesa
Supervisor	Susana Silva (Supervisor) & Roberta De Cicco (Co-Supervisor)