how to get literature review for chatbot

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Artif Intell

Are We There Yet? - A Systematic Literature Review on Chatbots in Education

Sebastian wollny.

1 Information Center for Education, DIPF | Leibniz Institute for Research and Information in Education, Frankfurt am Main, Germany

Jan Schneider

Daniele di mitri, joshua weidlich, marc rittberger, hendrik drachsler.

2 Educational Science Faculty, Open University of the Netherlands, Heerlen, Netherlands

3 Computer Science Faculty, Goethe University, Frankfurt am Main, Germany

Diana Rosario Perez Marin , Rey Juan Carlos University, Spain

Associated Data

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding authors.

Chatbots are a promising technology with the potential to enhance workplaces and everyday life. In terms of scalability and accessibility, they also offer unique possibilities as communication and information tools for digital learning. In this paper, we present a systematic literature review investigating the areas of education where chatbots have already been applied, explore the pedagogical roles of chatbots, the use of chatbots for mentoring purposes, and their potential to personalize education. We conducted a preliminary analysis of 2,678 publications to perform this literature review, which allowed us to identify 74 relevant publications for chatbots’ application in education. Through this, we address five research questions that, together, allow us to explore the current state-of-the-art of this educational technology. We conclude our systematic review by pointing to three main research challenges: 1) Aligning chatbot evaluations with implementation objectives, 2) Exploring the potential of chatbots for mentoring students, and 3) Exploring and leveraging adaptation capabilities of chatbots. For all three challenges, we discuss opportunities for future research.

Introduction

Educational Technologies enable distance learning models and provide students with the opportunity to learn at their own pace. They have found their way into schools and higher education institutions through Learning Management Systems and Massive Open Online Courses, enabling teachers to scale up good teaching practices ( Ferguson and Sharples, 2014 ) and allowing students to access learning material ubiquitously ( Virtanen et al., 2018 ).

Despite the innovative power of educational technologies, most commonly used technologies do not substantially change teachers’ role. Typical teaching activities like providing students with feedback, motivating them, or adapting course content to specific student groups are still entrusted exclusively to teachers, even in digital learning environments. This can lead to the teacher-bandwidth problem ( Wiley and Edwards, 2002 ), the result of a shortage of teaching staff to provide highly informative and competence-oriented feedback at large scale. Nowadays, however, computers and other digital devices open up far-reaching possibilities that have not yet been fully exploited. For example, incorporating process data can provide students with insights into their learning progress and bring new possibilities for formative feedback, self-reflection, and competence development ( Quincey et al., 2019 ). According to ( Hattie, 2009 ), feedback in terms of learning success has a mean effect size of d = 0.75, while ( Wisniewski et al., 2019 ) even report a mean effect of d = 0.99 for highly informative feedback. Such feedback provides suitable conditions for self-directed learning ( Winne and Hadwin, 2008 ) and effective metacognitive control of the learning process ( Nelson and Narens, 1994 ).

One of the educational technologies designed to provide actionable feedback in this regard is Learning Analytics. Learning Analytics is defined as the research area that focuses on collecting traces that learners leave behind and using those traces to improve learning ( Duval and Verbert, 2012 ; Greller and Drachsler, 2012 ). Learning Analytics can be used both by students to reflect on their own learning progress and by teachers to continuously assess the students’ efforts and provide actionable feedback. Another relevant educational technology is Intelligent Tutoring Systems. Intelligent Tutoring Systems are defined as computerized learning environments that incorporate computational models ( Graesser et al., 2001 ) and provide feedback based on learning progress. Educational technologies specifically focused on feedback for help-seekers, comparable to raising hands in the classroom, are Dialogue Systems and Pedagogical Conversational Agents ( Lester et al., 1997 ). These technologies can simulate conversational partners and provide feedback through natural language ( McLoughlin and Oliver, 1998 ).

Research in this area has recently focused on chatbot technology, a subtype of dialog systems, as several technological platforms have matured and led to applications in various domains. Chatbots incorporate generic language models extracted from large parts of the Internet and enable feedback by limiting themselves to text or voice interfaces. For this reason, they have also been proposed and researched for a variety of applications in education ( Winkler and Soellner, 2018 ). Recent literature reviews on chatbots in education ( Winkler and Soellner, 2018 ; Hobert, 2019a ; Hobert and Meyer von Wolff, 2019 ; Jung et al., 2020 ; Pérez et al., 2020 ; Smutny and Schreiberova, 2020 ; Pérez-Marín, 2021 ) have reported on such applications as well as design guidelines, evaluation possibilities, and effects of chatbots in education.

In this paper, we contribute to the state-of-the-art of chatbots in education by presenting a systematic literature review, where we examine so-far unexplored areas such as implementation objectives, pedagogical roles, mentoring scenarios, the adaptations of chatbots to learners, and application domains. This paper is structured as follows: First, we review related work ( section 2 ), derive research questions from it, then explain the applied method for searching related studies ( section 3 ), followed by the results ( section 4 ), and finally, we discuss the findings ( section 5 ) and point to future research directions in the field ( section 5 ).

Related Work

In order to accurately cover the field of research and deal with the plethora of terms for chatbots in the literature (e.g. chatbot, dialogue system or pedagogical conversational agent) we propose the following definition:

Chatbots are digital systems that can be interacted with entirely through natural language via text or voice interfaces. They are intended to automate conversations by simulating a human conversation partner and can be integrated into software, such as online platforms, digital assistants, or be interfaced through messaging services.

Outside of education, typical applications of chatbots are in customer service ( Xu et al., 2017 ), counseling of hospital patients ( Vaidyam et al., 2019 ), or information services in smart speakers ( Ram et al., 2018 ). One central element of chatbots is the intent classification, also named the Natural Language Understanding (NLU) component, which is responsible for the sense-making of human input data. Looking at the current advances in chatbot software development, it seems that this technology’s goal is to pass the Turing Test ( Saygin et al., 2000 ) one day, which could make chatbots effective educational tools. Therefore, we ask ourselves “ Are we there yet? - Will we soon have an autonomous chatbot for every learner?”

To understand and underline the current need for research in the use of chatbots in education, we first examined the existing literature, focusing on comprehensive literature reviews. By looking at research questions in these literature reviews, we identified 21 different research topics and extracted findings accordingly. To structure research topics and findings in a comprehensible way, a three-stage clustering process was applied. While the first stage consisted of coding research topics by keywords, the second stage was applied to form overarching research categories ( Table 1 ). In the final stage, the findings within each research category were clustered to identify and structure commonalities within the literature reviews. The result is a concept map, which consists of four major categories. Those categories are CAT1. Applications of Chatbots, CAT2. Chatbot Designs, CAT3. Evaluation of Chatbots and CAT4. Educational Effects of Chatbots. To standardize the terminology and concepts applied, we present the findings of each category in a separate sub-section, respectively ( see Figure 1 , Figure 2 , Figure 3 , and Figure 4 ) and extended it with the outcomes of our own literature study that will be reported in the remaining parts of this article. Due to the size of the concept map a full version can be found in Appendix A .

Assignment of coded research topics identified in related literature reviews to research categories.

Applications of chatbots in related literature reviews (CAT1).

Chatbot designs in related literature reviews (CAT2).

Evaluation of chatbots in related literature reviews (CAT3).

Educational Effects of chatbots in related literature reviews (CAT4).

Regarding the applications of chatbots (CAT1), application clusters (AC) and application statistics (AS) have been described in the literature, which we visualized in Figure 1 . The study of ( Pérez et al., 2020 ) identifies two application clusters, defined through chatbot activities: “service-oriented chatbots” and “teaching-oriented chatbots.” ( Winkler and Soellner, 2018 ) identify applications clusters by naming the domains “health and well-being interventions,” “language learning,” “feedback and metacognitive thinking” as well as “motivation and self-efficacy.” Concerning application statistics (AS), ( Smutny and Schreiberova, 2020 ) found that nearly 47% of the analyzed chatbots incorporate informing actions, and 18% support language learning by elaborating on chatbots integrated into the social media platform Facebook. Besides, the chatbots studied had a strong tendency to use English, at 89%. This high number aligns with results from ( Pérez-Marín, 2021 ), where 75% of observed agents, as a related technology, were designed to interact in the English language. ( Pérez-Marín, 2021 ) also shows that 42% of the analyzed chatbots had mixed interaction modalities. Finally, ( Hobert and Meyer von Wolff, 2019 ) observed that only 25% of examined chatbots were incorporated in formal learning settings, the majority of published material focuses on student-chatbot interaction only and does not enable student-student communication, as well as nearly two-thirds of the analyzed chatbots center only on a single domain. Overall, we can summarize that so far there are six application clusters for chatbots for education categorized by chatbot activities or domains. The provided statistics allow for a clearer understanding regarding the prevalence of chatbots applications in education ( see Figure 1 ).

Regarding chatbot designs (CAT2), most of the research questions concerned with chatbots in education can be assigned to this category. We found three aspects in this category visualized in Figure 2 : Personality (PS), Process Pipeline (PP), and Design Classifications (DC). Within these, most research questions can be assigned to Design Classifications (DC), which are separated into Classification Aspects (DC2) and Classification Frameworks (DC1). One classification framework is defined through “flow chatbots,” “artificially intelligent chatbots,” “chatbots with integrated speech recognition,” as well as “chatbots with integrated context-data” by ( Winkler and Soellner, 2018 ). A second classification framework by ( Pérez-Marín, 2021 ) covers pedagogy, social, and HCI features of chatbots and agents, which themselves can be further subdivided into more detailed aspects. Other Classification Aspects (DC2) derived from several publications, provide another classification schema, which distinguishes between “retrieval vs. generative” based technology, the “ability to incorporate context data,” and “speech or text interface” ( Winkler and Soellner, 2018 ; Smutny and Schreiberova, 2020 ). By specifying text interfaces as “Button-Based” or “Keyword Recognition-Based” ( Smutny and Schreiberova, 2020 ), text interfaces can be subdivided. Furthermore, a comparison of speech and text interfaces ( Jung et al., 2020 ) shows that text interfaces have advantages for conveying information, and speech interfaces have advantages for affective support. The second aspect of CAT2 concerns the chatbot processing pipeline (PP), highlighting user interface and back-end importance ( Pérez et al., 2020 ). Finally, ( Jung et al., 2020 ) focuses on the third aspect, the personality of chatbots (PS). Here, the study derives four guidelines helpful in education: positive or neutral emotional expressions, a limited amount of animated or visual graphics, a well-considered gender of the chatbot, and human-like interactions. In summary, we have found in CAT2 three main design aspects for the development of chatbots. CAT2 is much more diverse than CAT1 with various sub-categories for the design of chatbots. This indicates the huge flexibility to design chatbots in various ways to support education.

Regarding the evaluation of chatbots (CAT3), we found three aspects assigned to this category, visualized in Figure 3 : Evaluation Criteria (EC), Evaluation Methods (EM), and Evaluation Instruments (EI). Concerning Evaluation Criteria, seven criteria can be identified in the literature. The first and most important in the educational field, according to ( Smutny and Schreiberova, 2020 ) is the evaluation of learning success ( Hobert, 2019a ), which can have subcategories such as how chatbots are embedded in learning scenarios ( Winkler and Soellner, 2018 ; Smutny and Schreiberova, 2020 ) and teaching efficiency ( Pérez et al., 2020 ). The second is acceptance, which ( Hobert, 2019a ) names as “acceptance and adoption” and ( Pérez et al., 2020 ) as “students’ perception.” Further evaluation criteria are motivation, usability, technical correctness, psychological, and further beneficial factors ( Hobert, 2019a ). These Evaluation Criteria show broad possibilities for the evaluation of chatbots in education. However, ( Hobert, 2019a ) found that most evaluations are limited to single evaluation criteria or narrower aspects of them. Moreover, ( Hobert, 2019a ) introduces a classification matrix for chatbot evaluations, which consists of the following Evaluation Methods (EM): Wizard-of-Oz approach, laboratory studies, field studies, and technical validations. In addition to this, ( Winkler and Soellner, 2018 ) recommends evaluating chatbots by their embeddedness into a learning scenario, a comparison of human-human and human-chatbot interactions, and comparing spoken and written communication. Instruments to measure these evaluation criteria were identified by ( Hobert, 2019a ) by naming quantitative surveys, qualitative interviews, transcripts of dialogues, and technical log files. Regarding CAT3, we found three main aspects for the evaluation of chatbots. We can conclude that this is a more balanced and structured distribution in comparison to CAT2, providing researchers with guidance for evaluating chatbots in education.

Regarding educational effects of chatbots (CAT4), we found two aspects visualized in Figure 4 : Effect Size (ES) and Beneficial Chatbot Features for Learning Success (BF). Concerning the effect size, ( Pérez et al., 2020 ) identified a strong dependency between learning and the related curriculum, while ( Winkler and Soellner, 2018 ) elaborate on general student characteristics that influence how students interact with chatbots. They state that students’ attitudes towards technology, learning characteristics, educational background, self-efficacy, and self-regulation skills affect these interactions. Moreover, the study emphasizes chatbot features, which can be regarded as beneficial in terms of learning outcomes (BF): “Context-Awareness,” “Proactive guidance by students,” “Integration in existing learning and instant messaging tools,” “Accessibility,” and “Response Time.” Overall, for CAT4, we found two main distinguishing aspects for chatbots, however, the reported studies vary widely in their research design, making high-level results hardly comparable.

Looking at the related work, many research questions for the application of chatbots in education remain. Therefore, we selected five goals to be further investigated in our literature review. Firstly, we were interested in the objectives for implementing chatbots in education (Goal 1), as the relevance of chatbots for applications within education seems to be not clearly delineated. Secondly, we aim to explore the pedagogical roles of chatbots in the existing literature (Goal 2) to understand how chatbots can take over tasks from teachers. ( Winkler and Soellner, 2018 ) and ( Pérez-Marín, 2021 ), identified research gaps for supporting meta-cognitive skills with chatbots such as self-regulation. This requires a chatbot application that takes a mentoring role, as the development of these meta-cognitive skills can not be achieved solely by information delivery. Within our review we incorporate this by reviewing the mentoring role of chatbots as (Goal 3). Another key element for a mentoring chatbot is adaptation to the learners needs. Therefore, (Goal 4) of our review lies in the investigation of the adaptation approaches used by chatbots in education. For (Goal 5), we want to extend the work of ( Winkler and Soellner, 2018 ) and ( Pérez et al., 2020 ) regarding Application Clusters (AC) and map applications by further investigating specific learning domains in which chatbots have been studied.

To delineate and map the field of chatbots in education, initial findings were collected by a preliminary literature search. One of the takeaways is that the emerging field around educational chatbots has seen much activity in the last two years. Based on the experience of this preliminary search, search terms, queries, and filters were constructed for the actual structured literature review. This structured literature review follows the PRISMA framework ( Liberati et al., 2009 ), a guideline for reporting systematic reviews and meta-analyses. The framework consists of an elaborated structure for systematic literature reviews and sets requirements for reporting information about the review process ( see section 3.2 to 3.4).

Research Questions

Contributing to the state-of-the-art, we investigate five aspects of chatbot applications published in the literature. We therefore guided our research with the following research questions:

RQ1: Which objectives for implementing chatbots in education can be identified in the existing literature?

RQ2: Which pedagogical roles of chatbots can be identified in the existing literature?

RQ3: Which application scenarios have been used to mentor students?

RQ4: To what extent are chatbots adaptable to personal students’ needs?

RQ5: What are the domains in which chatbots have been applied so far?

Sources of Information

As data sources, Scopus, Web of Science, Google Scholar, Microsoft Academics, and the educational research database “Fachportal Pädagogik” (including ERIC) were selected, all of which incorporate all major publishers and journals. In ( Martín-Martín et al., 2018 ) it was shown that for the social sciences only 29.8% and for engineering and computer science, 46.8% of relevant literature is included in all of the first three databases. For the topic of chatbots in education, a value between these two numbers can be assumed, which is why an approach of integrating several publisher-independent databases was employed here.

Search Criteria

Based on the findings from the initial related work search, we derived the following search query:

( Education OR Educational OR Learning OR Learner OR Student OR Teaching OR School OR University OR Pedagogical ) AND Chatbot.

It combines education-related keywords with the “chatbot” keyword. Since chatbots are related to other technologies, the initial literature search also considered keywords such as “pedagogical agents,” “dialogue systems,” or “bots” when composing the search query. However, these increased the number of irrelevant results significantly and were therefore excluded from the query in later searches.

Inclusion and Exclusion Criteria

The queries were executed on 23.12.2020 and applied twice to each database, first as a title search query and secondly as a keyword-based search. This resulted in a total of 3.619 hits, which were checked for duplicates resulting in 2.678 candidate publications. The overall search and filtering process is shown in Figure 5 .

PRISMA flow chart.

In the case of Google Scholar, the number of results sorted by relevance per query was limited to 300, as this database also delivers many less relevant works. The value was determined by looking at the search results in detail using several queries to exclude as few relevant works as possible. This approach showed promising results and, at the same time, did not burden the literature list with irrelevant items.

The further screening consisted of a four-stage filtering process. First, eliminating duplicates in the results of title and keyword queries of all databases independently and second, excluding publications based on the title and abstract that:

• • were not available in English
• • did not describe a chatbot application
• • were not mainly focused on learner-centered chatbots applications in schools or higher education institutions, which is according to the preliminary literature search the main application area within education.

Third, we applied another duplicate filter, this time for the merged set of publications. Finally, a filter based on the full text, excluding publications that were:

• • limited to improve chatbots technically (e.g., publications that compare or develop new algorithms), as research questions presented in these publications were not seeking for additional insights on applications in education
• • exclusively theoretical in nature (e.g., publications that discuss new research projects, implementation concepts, or potential use cases of chatbots in education), as they either do not contain research questions or hypotheses or do not provide conclusions from studies with learners.

After the first, second, and third filters, we identified 505 candidate publications. We continued our filtering process by reading the candidate publications’ full texts resulting in 74 publications that were used for our review. Compared to 3.619 initial database results, the proportion of relevant publications is therefore about 2.0%.

The final publication list can be accessed under https://bit.ly/2RRArFT .

To analyze the identified publications and derive results according to the research questions, full texts were coded, considering for each publication the objectives for implementing chatbots (RQ1), pedagogical roles of chatbots (RQ2), their mentoring roles (RQ3), adaptation of chatbots (RQ4), as well as their implementation domains in education (RQ5) as separated sets of codes. To this end, initial codes were identified by open coding and iteratively improved through comparison, group discussion among the authors, and subsequent code expansion. Further, codes were supplemented with detailed descriptions until a saturation point was reached, where all included studies could be successfully mapped to codes, suggesting no need for further refinement. As an example, codes for RQ2 (Pedagogical Roles) were adapted and refined in terms of their level of abstraction from an initial set of only two codes, 1 ) a code for chatbots in the learning role and 2 ) a code for chatbots in a service-oriented role. After coding a larger set of publications, it became clear that the code for service-oriented chatbots needed to be further distinguished. This was because it summarized e.g. automation activities with activities related to self-regulated learning and thus could not be distinguished sharply enough from the learning role. After refining the code set in the next iteration into a learning role, an assistance role, and a mentoring role, it was then possible to ensure the separation of the individual codes. In order to avoid defining new codes for singular or a very small number of publications, studies were coded as “other” (RQ1) or “not defined” (RQ2), if their occurrence was less than eight publications, representing less than 10% of the publications in the final paper list.

By grouping the resulting relevant publications according to their date of publication, it is apparent that chatbots in education are currently in a phase of increased attention. The release distribution shows slightly lower publication numbers in the current than in the previous year ( Figure 6 ), which could be attributed to a time lag between the actual publication of manuscripts and their dissemination in databases.

Identified chatbot publications in education per year.

Applying the curve presented in Figure 6 to Gartner’s Hype Cycle ( Linden and Fenn, 2003 ) suggests that technology around chatbots in education may currently be in the “Innovation Trigger” phase. This phase is where many expectations are placed on the technology, but the practical in-depth experience is still largely lacking.

Objectives for Implementing Chatbots in Education

Regarding RQ1, we extracted implementation objectives for chatbots in education. By analyzing the selected publications we identified that most of the objectives for chatbots in education can be described by one of the following categories: Skill improvement, Efficiency of Education, and Students’ Motivation ( see Figure 7 ). First, the “improvement of a student’s skill” (or Skill Improvement ) objective that the chatbot is supposed to help with or achieve. Here, chatbots are mostly seen as a learning aid that supports students. It is the most commonly cited objective for chatbots. The second objective is to increase the Efficiency of Education in general. It can occur, for example, through the automation of recurring tasks or time-saving services for students and is the second most cited objective for chatbots. The third objective is to increase Students’ Motivation . Finally, the last objective is to increase the Availability of Education . This objective is intended to provide learning or counseling with temporal flexibility or without the limitation of physical presence. In addition, there are other, more diverse objectives for chatbots in education that are less easy to categorize. In cases of a publication indicating more than one objective, the publication was distributed evenly across the respective categories.

Objectives for implementing chatbots identified in chatbot publications.

Given these results, we can summarize four major implementing objectives for chatbots. Of these, Skill Improvement is the most popular objective, constituting around one-third of publications (32%). Making up a quarter of all publications, Efficiency of Education is the second most popular objective (25%), while addressing Students’ Motivation and Availability of Education are third (13%) and fourth (11%), respectively. Other objectives also make up a substantial amount of these publications (19%), although they were too diverse to categorize in a uniform way. Examples of these are inclusivity ( Heo and Lee, 2019 ) or the promotion of student teacher interactions ( Mendoza et al., 2020 ).

Pedagogical Roles

Regarding RQ2, it is crucial to consider the use of chatbots in terms of their intended pedagogical role. After analyzing the selected articles, we were able to identify four different pedagogical roles: a supporting learning role, an assisting role, and a mentoring role.

In the supporting learning role ( Learning ), chatbots are used as an educational tool to teach content or skills. This can be achieved through a fixed integration into the curriculum, such as conversation tasks (L. K. Fryer et al., 2020 ). Alternatively, learning can be supported through additional offerings alongside classroom teaching, for example, voice assistants for leisure activities at home ( Bao, 2019 ). Examples of these are chatbots simulating a virtual pen pal abroad ( Na-Young, 2019 ). Conversations with this kind of chatbot aim to motivate the students to look up vocabulary, check their grammar, and gain confidence in the foreign language.

In the assisting role ( Assisting ), chatbot actions can be summarized as simplifying the student's everyday life, i.e., taking tasks off the student’s hands in whole or in part. This can be achieved by making information more easily available ( Sugondo and Bahana, 2019 ) or by simplifying processes through the chatbot’s automation ( Suwannatee and Suwanyangyuen, 2019 ). An example of this is the chatbot in ( Sandoval, 2018 ) that answers general questions about a course, such as an exam date or office hours.

In the mentoring role ( Mentoring ), chatbot actions deal with the student’s personal development. In this type of support, the student himself is the focus of the conversation and should be encouraged to plan, reflect or assess his progress on a meta-cognitive level. One example is the chatbot in ( Cabales, 2019 ), which helps students develop lifelong learning skills by prompting in-action reflections.

The distribution of each pedagogical role is shown in Figure 8 . From this, it can be seen that Learning is the most frequently used role of the examined publications (49%), followed by Assisting (20%) and Mentoring (15%). It should be noted that pedagogical roles were not identified for all the publications examined. The absence of a clearly defined pedagogical role (16%) can be attributed to the more general nature of these publications, e.g. focused on students’ small talk behaviors ( Hobert, 2019b ) or teachers’ attitudes towards chatbot applications in classroom teaching (P. K. Bii et al., 2018 ).

Pedagogical roles identified in chatbot publications.

Looking at pedagogical roles in the context of objectives for implementing chatbots, relations among publications can be inspected in a relations graph ( Figure 9 ). According to our results, the strongest relation in the examined publications can be considered between Skill Improvement objective and the Learning role. This strong relation is partly because both, the Skill Improvement objective and the Learning role, are the largest in their respective categories. In addition, two other strong relations can be observed: Between the Students’ Motivation objective and the Learning role, as well as between Efficiency of Education objective and Assisting role.

Relations graph of pedagogical roles and objectives for implementing chatbots.

By looking at other relations in more detail, there is surprisingly no relation between Skill Improvement as the most common implementation objective and Assisting , as the 2nd most common pedagogical role. Furthermore, it can be observed that the Mentoring role has nearly equal relations to all of the objectives for implementing chatbots.

The relations graph ( Figure 9 ) can interactively be explored through bit.ly/32FSKQM.

Mentoring Role

Regarding RQ3, we identified eleven publications that deal with chatbots in this regard. The Mentoring role in these publications can be categorized in two dimensions. Starting with the first dimension, the mentoring method, three methods can be observed:

• • Scaffolding ( n = 7)
• • Recommending ( n = 3)
• • Informing ( n = 1)

An example of Scaffolding can be seen in ( Gabrielli et al., 2020 ), where the chatbot coaches students in life skills, while an example of Recommending can be seen in ( Xiao et al., 2019 ), where the chatbot recommends new teammates. Finally, Informing can be seen in ( Kerly et al., 2008 ), where the chatbot informs students about their personal Open Learner Model.

The second dimension is the addressed mentoring topic, where the following topics can be observed:

• • Self-Regulated Learning ( n = 5)
• • Life Skills ( n = 4)
• • Learning Skills ( n = 2)

While Mentoring chatbots to support Self-Regulated Learning are intended to encourage students to reflect on and plan their learning progress, Mentoring chatbots to support Life Skills address general student’s abilities such as self-confidence or managing emotions. Finally, Mentoring chatbots to support Learning Skills , in contrast to Self-Regulated Learning , address only particular aspects of the learning process, such as new learning strategies or helpful learning partners. An example for Mentoring chatbots supporting Life Skill is the Logo counseling chatbot, which promotes healthy self-esteem ( Engel et al., 2020 ). CALMsystem is an example of a Self-Regulated Learning chatbot, which informs students about their data in an open learner model ( Kerly et al., 2008 ). Finally, there is the Learning Skills topic. Here, the MCQ Bot is an example that is designed to introduce students to transformative learning (W. Huang et al., 2019 ).

Regarding RQ4, we identified six publications in the final publication list that address the topic of adaptation. Within these publications, five adaptation approaches are described:

The first approach (A1) is proposed by ( Kerly and Bull, 2006 ) and ( Kerly et al., 2008 ), dealing with student discussions based on success and confidence during a quiz. The improvement of self-assessment is the primary focus of this approach. The second approach (A2) is presented in ( Jia, 2008 ), where the personality of the chatbot is adapted to motivate students to talk to the chatbot and, in this case, learn a foreign language. The third approach (A3), as shown in the work of ( Vijayakumar et al., 2019 ), is characterized by a chatbot that provides personalized formative feedback to learners based on their self-assessment, again in a quiz situation. Here, the focus is on Hattie and Timperley’s three guiding questions: “Where am I going?,” “How am I going?” and “Where to go next?” ( Hattie and Timperley, 2007 ). In the fourth approach (A4), exemplified in ( Ruan et al., 2019 ), the chatbot selects questions within a quiz. Here, the chatbot estimates the student’s ability and knowledge level based on the quiz progress and sets the next question accordingly. Finally, a similar approach (A5) is shown in ( Davies et al., 2020 ). In contrast to ( Ruan et al., 2019 ), this chatbot adapts the amount of question variation and takes psychological features into account which were measured by psychological tests before.

We examined these five approaches by organizing them according to their information sources and extracted learner information. The results can be seen in Table 2 .

Adaptation approaches of chatbots in education.

Four out of five adaptation approaches (A1, A3, A4, and A5) are observed in the context of quizzes. These adaptations within quizzes can be divided into two mainstreams: One is concerned about students’ feedback (A1 and A3), while the other is concerned about learning material selection (A4 and A5). The only different adaptation approach is shown in A2, which focuses on the adaptation of the chatbot personality within a language learning application.

Domains for Chatbots in Education

Regarding RQ5, we identified 20 domains of chatbots in education. These can broadly be divided by their pedagogical role into three domain categories (DC): Learning Chatbots , Assisting Chatbots , and Mentoring Chatbots . The remaining publications are grouped in the Other Research domain category. The complete list of identified domains can be seen in Table 3 .

Domains of chatbots in education.

The domain category Learning Chatbots , which deals with chatbots incorporating the pedagogical role Learning , can be subdivided into seven domains: 1 ) Language Learning , 2 ) Learn to Program , 3 ) Learn Communication Skills , 4 ) Learn about Educational Technologies , 5 ) Learn about Cultural Heritage , 6 ) Learn about Laws , and 7 ) Mathematics Learning . With more than half of publications (53%), chatbots for Language Learning play a prominent role in this domain category. They are often used as chat partners to train conversations or to test vocabulary. An example of this can be seen in the work of ( Bao, 2019 ), which tries to mitigate foreign language anxiety by chatbot interactions in foreign languages.

The domain category Assisting Chatbots , which deals with chatbots incorporating the pedagogical role Assisting , can be subdivided into four domains: 1 ) Administrative Assistance , 2 ) Campus Assistance , 3 ) Course Assistance , and 4 ) Library Assistance . With one-third of publications (33%), chatbots in the Administrative Assistance domain that help to overcome bureaucratic hurdles at the institution, while providing round-the-clock services, are the largest group in this domain category. An example of this can be seen in ( Galko et al., 2018 ), where the student enrollment process is completely shifted to a conversation with a chatbot.

The domain category Mentoring Chatbots , which deals with chatbots incorporating the pedagogical role Mentoring , can be subdivided into three domains: 1 ) Scaffolding Chatbots , 2 ) Recommending Chatbots , and 3 ) Informing Chatbots . An example of a Scaffolding Chatbots is the CRI(S) chatbot ( Gabrielli et al., 2020 ), which supports life skills such as self-awareness or conflict resolution in discussion with the student by promoting helpful ideas and tricks.

The domain category Other Research , which deals with chatbots not incorporating any of these pedagogical roles, can be subdivided into three domains: 1 ) General Chatbot Research in Education , 2 ) Indian Educational System , and 3 ) Chatbot Interfaces . The most prominent domain, General Chatbot Research , cannot be classified in one of the other categories but aims to explore cross-cutting issues. An example for this can be seen in the publication of ( Hobert, 2020 ), which researches the importance of small talk abilities of chatbots in educational settings.

Discussions

In this paper, we investigated the state-of-the-art of chatbots in education according to five research questions. By combining our results with previously identified findings from related literature reviews, we proposed a concept map of chatbots in education. The map, reported in Appendix A , displays the current state of research regarding chatbots in education with the aim of supporting future research in the field.

Answer to Research Questions

Concerning RQ1 (implementation objectives), we identified four major objectives: 1 ) Skill Improvement , 2 ) Efficiency of Education , 3 ) Students’ Motivation, and 4 ) Availability of Education . These four objectives cover over 80% of the analyzed publications ( see Figure 7 ). Based on the findings on CAT3 in section 2 , we see a mismatch between the objectives for implementing chatbots compared to their evaluation. Most researchers only focus on narrow aspects for the evaluation of their chatbots such as learning success, usability, and technology acceptance. This mismatch of implementation objectives and suitable evaluation approaches is also well known by other educational technologies such as Learning Analytics dashboards ( Jivet et al., 2017 ). A more structured approach of aligning implementation objectives and evaluation procedures is crucial to be able to properly assess the effectiveness of chatbots. ( Hobert, 2019a ), suggested a structured four-stage evaluation procedure beginning with a Wizard-of-Oz experiment, followed by technical validation, a laboratory study, and a field study. This evaluation procedure systematically links hypotheses with outcomes of chatbots helping to assess chatbots for their implementation objectives. “Aligning chatbot evaluations with implementation objectives” is, therefore, an important challenge to be addressed in the future research agenda.

Concerning RQ2 (pedagogical roles), our results show that chatbots’ pedagogical roles can be summarized as Learning , Assisting , and Mentoring . The Learning role is the support in learning or teaching activities such as gaining knowledge. The Assisting role is the support in terms of simplifying learners’ everyday life, e.g. by providing opening times of the library. The Mentoring role is the support in terms of students’ personal development, e.g. by supporting Self-Regulated Learning. From a pedagogical standpoint, all three roles are essential for learners and should therefore be incorporated in chatbots. These pedagogical roles are well aligned with the four implementation objectives reported in RQ1. While Skill Improvement and Students’ Motivation is strongly related to Learning , Efficiency of Education is strongly related to Assisting . The Mentoring role instead, is evenly related to all of the identified objectives for implementing chatbots. In the reviewed publications, chatbots are therefore primarily intended to 1 ) improve skills and motivate students by supporting learning and teaching activities, 2 ) make education more efficient by providing relevant administrative and logistical information to learners, and 3 ) support multiple effects by mentoring students.

Concerning RQ3 (mentoring role), we identified three main mentoring method categories for chatbots: 1 ) Scaffolding , 2 ) Recommending , and 3 ) Informing . However, comparing the current mentoring of chatbots reported in the literature with the daily mentoring role of teachers, we can summarize that the chatbots are not at the same level. In order to take over mentoring roles of teachers ( Wildman et al., 1992 ), a chatbot would need to fulfill some of the following activities in their mentoring role. With respect to 1 ) Scaffolding , chatbots should provide direct assistance while learning new skills and especially direct beginners in their activities. Regarding 2 ) Recommending , chatbots should provide supportive information, tools or other materials for specific learning tasks to life situations. With respect to 3 ) Informing, chatbots should encourage students according to their goals and achievements, and support them to develop meta-cognitive skills like self-regulation. Due to the mismatch of teacher vs. chatbot mentoring we see here another research challenge, which we call “Exploring the potential of chatbots for mentoring students.”

Regarding RQ4 (adaptation), only six publications were identified that discuss an adaptation of chatbots, while four out of five adaptation approaches (A1, A3, A4, and A5) show similarities by being applied within quizzes. In the context of educational technologies, providing reasonable adaptations for learners requires a high level of experience. Based on our results, the research on chatbots does not seem to be at this point yet. Looking at adaptation literature like ( Brusilovsky, 2001 ) or ( Benyon and Murray, 1993 ), it becomes clear that a chatbot needs to consider the learners’ personal information to fulfill the requirement of the adaptation definition. Personal information must be retrieved and stored at least temporarily, in some sort of learner model. For learner information like knowledge and interest, adaptations seem to be barely explored in the reviewed publications, while the model of ( Brusilovsky and Millán, 2007 ) points out further learner information, which can be used to make chatbots more adaptive: personal goals, personal tasks, personal background, individual traits, and the learner’s context. We identify research in this area as a third future challenge and call it the “Exploring and leveraging adaptation capabilities of chatbots” challenge.

In terms of RQ5 (domains), we identified a detailed map of domains applying chatbots in education and their distribution ( see Table 3 ). By systematically analyzing 74 publications, we identified 20 domains and structured them according to the identified pedagogical role into four domain categories: Learning Chatbots , Assisting Chatbots , Mentoring Chatbots , and Other Research . These results extend the taxonomy of Application Clusters (AC) for chatbots in education, which previously comprised the work from ( Pérez et al., 2020 ), who took the chatbot activity as characteristic, and ( Winkler and Soellner, 2018 ), who characterized the chatbots by domains. It draws relationships between these two types of Application Clusters (AC) and structures them accordingly. Our structure incorporates Mentoring Chatbots and Other Research in addition to the “service-oriented chatbots” (cf. Assisting Chatbots ) and “teaching-oriented chatbots” (cf. Learning Chatbots ) identified by (Perez). Furthermore, the strong tendencies of informing students already mentioned by ( Smutny and Schreiberova, 2020 ) can also be recognized in our results, especially in Assisting Chatbots . Compared to ( Winkler and Soellner, 2018 ), we can confirm the prominent domains of “language learning” within Learning Chatbots and “metacognitive thinking” within Mentoring Chatbots . Moreover, through Table 3 , a more detailed picture of chatbot applications in education is reflected, which could help researchers to find similar works or unexplored application areas.

Limitations

One important limitation to be mentioned here is the exclusion of alternative keywords for our search queries, as we exclusively used chatbot as keyword in order to avoid search results that do not fit our research questions. Though we acknowledge that chatbots share properties with pedagogical agents, dialog systems, and bots, we carefully considered this trade-off between missing potentially relevant work and inflating our search procedure by including related but not necessarily pertinent work. A second limitation may lie in the formation of categories and coding processes applied, which, due to the novelty of the findings, could not be built upon theoretical frameworks or already existing code books. Although we have focused on ensuring that codes used contribute to a strong understanding, the determination of the abstraction level might have affected the level of detail of the resulting data representation.

In this systematic literature review, we explored the current landscape of chatbots in education. We analyzed 74 publications, identified 20 domains of chatbots and grouped them based on their pedagogical roles into four domain categories. These pedagogical roles are the supporting learning role ( Learning ), the assisting role ( Assisting ), and the mentoring role ( Mentoring ). By focusing on objectives for implementing chatbots, we identified four main objectives: 1 ) Skill Improvement , 2 ) Efficiency of Education , 3 ) Students’ Motivation, and 4 ) Availability of Education . As discussed in section 5 , these objectives do not fully align with the chosen evaluation procedures. We focused on the relations between pedagogical roles and objectives for implementing chatbots and identified three main relations: 1 ) chatbots to improve skills and motivate students by supporting learning and teaching activities, 2 ) chatbots to make education more efficient by providing relevant administrative and logistical information to learners, and 3 ) chatbots to support multiple effects by mentoring students. We focused on chatbots incorporating the Mentoring role and found that these chatbots are mostly concerned with three mentoring topics 1 ) Self-Regulated Learning , 2 ) Life Skills , and 3 ) Learning Skills and three mentoring methods 1 ) Scaffolding , 2 ) Recommending , and 3 ) Informing . Regarding chatbot adaptations, only six publications with adaptations were identified. Furthermore, the adaptation approaches found were mostly limited to applications within quizzes and thus represent a research gap.

Based on these outcomes we consider three challenges for chatbots in education that offer future research opportunities:

Challenge 1: Aligning chatbot evaluations with implementation objectives . Most chatbot evaluations focus on narrow aspects to measure the tool’s usability, acceptance or technical correctness. If chatbots should be considered as learning aids, student mentors, or facilitators, the effects on the cognitive, and emotional levels should also be taken into account for the evaluation of chatbots. This finding strengthens our conclusion that chatbot development in education is still driven by technology, rather than having a clear pedagogical focus of improving and supporting learning.

Challenge 2: Exploring the potential of chatbots for mentoring students . In order to better understand the potentials of chatbots to mentor students, more empirical studies on the information needs of learners are required. It is obvious that these needs differ from schools to higher education. However, so far there are hardly any studies investigating the information needs with respect to chatbots nor if chatbots address these needs sufficiently.

Challenge 3: Exploring and leveraging adaptation capabilities of chatbots . There is a large literature on adaptation capabilities of educational technologies. However, we have seen very few studies on the effect of adaptation of chatbots for education purposes. As chatbots are foreseen as systems that should personally support learners, the area of adaptable interactions of chatbots is an important research aspect that should receive more attention in the near future.

By addressing these challenges, we believe that chatbots can become effective educational tools capable of supporting learners with informative feedback. Therefore, looking at our results and the challenges presented, we conclude, “No, we are not there yet!” - There is still much to be done in terms of research on chatbots in education. Still, development in this area seems to have just begun to gain momentum and we expect to see new insights in the coming years.

APPENDIX A AConcept map of chatbots in education

Data Availability Statement

Author contributions.

SW, JS†, DM†, JW†, MR, and HD.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

• Abbasi S., Kazi H., Hussaini N. N. (2019). Effect of Chatbot Systems on Student’s Learning Outcomes . Sylwan 163 ( 10 ). [ Google Scholar ]
• Abbasi S., Kazi H. (2014). Measuring Effectiveness of Learning Chatbot Systems on Student's Learning Outcome and Memory Retention . Asian J. Appl. Sci. Eng. 3 , 57. 10.15590/AJASE/2014/V3I7/53576 [ CrossRef ] [ Google Scholar ]
• Almahri F. A. J., Bell D., Merhi M. (2020). “ Understanding Student Acceptance and Use of Chatbots in the United Kingdom Universities: A Structural Equation Modelling Approach ,” in 2020 6th IEEE International Conference on Information Management, ICIM 2020, London, United Kingdom, March 27–29, 2020, (IEEE), 284–288. 10.1109/ICIM49319.2020.244712 [ CrossRef ] [ Google Scholar ]
• Bao M. (2019). Can Home Use of Speech-Enabled Artificial Intelligence Mitigate Foreign Language Anxiety - Investigation of a Concept . Awej 5 , 28–40. 10.24093/awej/call5.3 [ CrossRef ] [ Google Scholar ]
• Benyon D., Murray D. (1993). Applying User Modeling to Human-Computer Interaction Design . Artif. Intell. Rev. 7 ( 3-4 ), 199–225. 10.1007/BF00849555 [ CrossRef ] [ Google Scholar ]
• Bii P. K., Too J. K., Mukwa C. W. (2018). Teacher Attitude towards Use of Chatbots in Routine Teaching . Univers. J. Educ. Res. . 6 ( 7 ), 1586–1597. 10.13189/ujer.2018.060719 [ CrossRef ] [ Google Scholar ]
• Bii P., Too J., Langat R. (2013). An Investigation of Student’s Attitude Towards the Use of Chatbot Technology in Instruction: The Case of Knowie in a Selected High School . Education Research 4 , 710–716. 10.14303/er.2013.231 [ CrossRef ] [ Google Scholar ]
• Bos A. S., Pizzato M. C., Vettori M., Donato L. G., Soares P. P., Fagundes J. G., et al. (2020). Empirical Evidence During the Implementation of an Educational Chatbot with the Electroencephalogram Metric . Creative Education 11 , 2337–2345. 10.4236/CE.2020.1111171 [ CrossRef ] [ Google Scholar ]
• Brusilovsky P. (2001). Adaptive Hypermedia . User Model. User-Adapted Interaction 11 ( 1 ), 87–110. 10.1023/a:1011143116306 [ CrossRef ] [ Google Scholar ]
• Brusilovsky P., Millán E. (2007). “ User Models for Adaptive Hypermedia and Adaptive Educational Systems ,” in The Adaptive Web: Methods and Strategies of Web Personalization . Editors Brusilovsky P., Kobsa A., Nejdl W.. Berlin: Springer, 3–53. 10.1007/978-3-540-72079-9_1 [ CrossRef ] [ Google Scholar ]
• Cabales V. (2019). “ Muse: Scaffolding metacognitive reflection in design-based research ,” in CHI EA’19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, Scotland, United Kingdom, May 4–9, 2019, (ACM), 1–6. 10.1145/3290607.3308450 [ CrossRef ] [ Google Scholar ]
• Carayannopoulos S. (2018). Using Chatbots to Aid Transition. Int. J. Info. Learn. Tech. 35 , 118–129. 10.1108/IJILT-10-2017-0097 [ CrossRef ] [ Google Scholar ]
• Chan C. H., Lee H. L., Lo W. K., Lui A. K.-F. (2018). Developing a Chatbot for College Student Programme Advisement . in 2018 International Symposium on Educational Technology, ISET 2018, Osaka, Japan, July 31–August 2, 2018. Editors Wang F. L., Iwasaki C., Konno T., Au O., Li C., (IEEE), 52–56. 10.1109/ISET.2018.00021 [ CrossRef ] [ Google Scholar ]
• Chang M.-Y., Hwang J.-P. (2019). “ Developing Chatbot with Deep Learning Techniques for Negotiation Course ,” in 2019 8th International Congress on Advanced Applied Informatics, IIAI-AAI 2019, Toyama, Japan, July 7–11, 2019, (IEEE), 1047–1048. 10.1109/IIAI-AAI.2019.00220 [ CrossRef ] [ Google Scholar ]
• Chen C.-A., Yang Y.-T., Wu S.-M., Chen H.-C., Chiu K.-C., Wu J.-W., et al. (2018). “ A Study of Implementing AI Chatbot in Campus Consulting Service ”, in TANET 2018-Taiwan Internet Seminar , 1714–1719. 10.6861/TANET.201810.0317 [ CrossRef ] [ Google Scholar ]
• Chen H.-L., Widarso G. V., Sutrisno H. (2020). A ChatBot for Learning Chinese: Learning Achievement and Technology Acceptance . J. Educ. Comput. Res. 58 ( 6 ), 1161–1189. 10.1177/0735633120929622 [ CrossRef ] [ Google Scholar ]
• Daud S. H. M., Teo N. H. I., Zain N. H. M. (2020). E-java Chatbot for Learning Programming Language: A post-pandemic Alternative Virtual Tutor . Int. J. Emerging Trends Eng. Res. 8 (7) . 3290–3298. 10.30534/ijeter/2020/67872020 [ CrossRef ] [ Google Scholar ]
• Davies J. N., Verovko M., Verovko O., Solomakha I. (2020). “ Personalization of E-Learning Process Using Ai-Powered Chatbot Integration ,” in Selected Papers of 15th International Scientific-practical Conference, MODS, 2020: Advances in Intelligent Systems and Computing, Chernihiv, Ukraine, June 29–July 01, 2020. Editors Shkarlet S., Morozov A., Palagin A., (Springer; ) Vol. 1265 , 209–216. 10.1007/978-3-030-58124-4_20 [ CrossRef ] [ Google Scholar ]
• Diachenko A. V., Morgunov B. P., Melnyk T. P., Kravchenko O. I., Zubchenko L. V. (2019). The Use of Innovative Pedagogical Technologies for Automation of the Specialists' Professional Training . Int. J. Hydrogen. Energy. 8 , 288–295. 10.5430/ijhe.v8n6p288 [ CrossRef ] [ Google Scholar ]
• Dibitonto M., Leszczynska K., Tazzi F., Medaglia C. M. (2018). “ Chatbot in a Campus Environment: Design of Lisa, a Virtual Assistant to Help Students in Their university Life ,” in 20th International Conference, HCI International 2018, Las Vegas, NV, USA, July 15–20, 2018, Lecture Notes in Computer Science. Editors Kurosu M., (Springer), 103–116. 10.1007/978-3-319-91250-9 [ CrossRef ] [ Google Scholar ]
• Durall E., Kapros E. (2020). “ Co-design for a Competency Self-Assessment Chatbot and Survey in Science Education ,” in 7th International Conference, LCT 2020, Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Lecture Notes in Computer Science. Editors Zaphiris P., Ioannou A., Berlin: Springer; Vol. 12206 , 13–23. 10.1007/978-3-030-50506-6_2 [ CrossRef ] [ Google Scholar ]
• Duval E., Verbert K. (2012). Learning Analytics . Eleed 8 ( 1 ). [ Google Scholar ]
• Engel J. D., Engel V. J. L., Mailoa E. (2020). Interaction Monitoring Model of Logo Counseling Website for College Students' Healthy Self-Esteem , I. J. Eval. Res. Educ. 9 , 607–613. 10.11591/ijere.v9i3.20525 [ CrossRef ] [ Google Scholar ]
• Febriani G. A., Agustia R. D. (2019). Development of Line Chatbot as a Learning Media for Mathematics National Exam Preparation . Elibrary.Unikom.Ac.Id . https://elibrary.unikom.ac.id/1130/14/UNIKOM_GISTY%20AMELIA%20FEBRIANI_JURNAL%20DALAM%20BAHASA%20INGGRIS.pdf .
• Ferguson R., Sharples M. (2014). “ Innovative Pedagogy at Massive Scale: Teaching and Learning in MOOCs ,” in 9th European Conference on Technology Enhanced Learning, EC-TEL 2014, Graz, Austria, September 16–19, 2014, Lecture Notes in Computer Science. Editors Rensing C., de Freitas S., Ley T., Muñoz-Merino P. J., (Berlin: Springer) Vol. 8719 , 98–111. 10.1007/978-3-319-11200-8_8 [ CrossRef ] [ Google Scholar ]
• Fryer L. K., Ainley M., Thompson A., Gibson A., Sherlock Z. (2017). Stimulating and Sustaining Interest in a Language Course: An Experimental Comparison of Chatbot and Human Task Partners . Comput. Hum. Behav. 75 , 461–468. 10.1016/j.chb.2017.05.045 [ CrossRef ] [ Google Scholar ]
• Fryer L. K., Nakao K., Thompson A. (2019). Chatbot Learning Partners: Connecting Learning Experiences, Interest and Competence . Comput. Hum. Behav. 93 , 279–289. 10.1016/j.chb.2018.12.023 [ CrossRef ] [ Google Scholar ]
• Fryer L. K., Thompson A., Nakao K., Howarth M., Gallacher A. (2020). Supporting Self-Efficacy Beliefs and Interest as Educational Inputs and Outcomes: Framing AI and Human Partnered Task Experiences . Learn. Individual Differences , 80. 10.1016/j.lindif.2020.101850 [ CrossRef ] [ Google Scholar ]
• Gabrielli S., Rizzi S., Carbone S., Donisi V. (2020). A Chatbot-Based Coaching Intervention for Adolescents to Promote Life Skills: Pilot Study . JMIR Hum. Factors 7 ( 1 ). 10.2196/16762 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Galko L., Porubän J., Senko J. (2018). “ Improving the User Experience of Electronic University Enrollment ,” in 16th IEEE International Conference on Emerging eLearning Technologies and Applications, ICETA 2018, Stary Smokovec, Slovakia, Nov 15–16, 2018. Editors Jakab F., (Piscataway, NJ: IEEE; ), 179–184. 10.1109/ICETA.2018.8572054 [ CrossRef ] [ Google Scholar ]
• Goda Y., Yamada M., Matsukawa H., Hata K., Yasunami S. (2014). Conversation with a Chatbot before an Online EFL Group Discussion and the Effects on Critical Thinking . J. Inf. Syst. Edu. 13 , 1–7. 10.12937/EJSISE.13.1 [ CrossRef ] [ Google Scholar ]
• Graesser A. C., VanLehn K., Rose C. P., Jordan P. W., Harter D. (2001). Intelligent Tutoring Systems with Conversational Dialogue . AI Mag. 22 ( 4 ), 39–51. 10.1609/aimag.v22i4.1591 [ CrossRef ] [ Google Scholar ]
• Greller W., Drachsler H. (2012). Translating Learning into Numbers: A Generic Framework for Learning Analytics . J. Educ. Tech. Soc. 15 ( 3 ), 42–57. 10.2307/jeductechsoci.15.3.42 [ CrossRef ] [ Google Scholar ]
• Haristiani N., Rifa’i M. M. Combining Chatbot and Social Media: Enhancing Personal Learning Environment (PLE) in Language Learning . Indonesian J Sci Tech. 5 ( 3 ), 487–506. 10.17509/ijost.v5i3.28687 [ CrossRef ] [ Google Scholar ]
• Hattie J., Timperley H. (2007). The Power of Feedback . Rev. Educ. Res. 77 ( 1 ), 81–112. 10.3102/003465430298487 [ CrossRef ] [ Google Scholar ]
• Hattie J. (2009). Visible Learning: A Synthesis of over 800 Meta-Analyses Relating to Achievement . Abingdon, UK: Routledge. [ Google Scholar ]
• Heller B., Proctor M., Mah D., Jewell L., Cheung B. (2005). “ Freudbot: An Investigation of Chatbot Technology in Distance Education ,” in Proceedings of ED-MEDIA 2005–World Conference on Educational Multimedia, Hypermedia and Telecommunications, Montréal, Canada, June 27–July 2, 2005. Editors Kommers P., Richards G., (AACE; ), 3913–3918. [ Google Scholar ]
• Heo J., Lee J. (2019). “ CiSA: An Inclusive Chatbot Service for International Students and Academics ,” in 21st International Conference on Human-Computer Interaction, HCII 2019: Communications in Computer and Information Science, Orlando, FL, USA, July 26–31, 2019. Editors Stephanidis C., (Springer; ) 11786 , 153–167. 10.1007/978-3-030-30033-3 [ CrossRef ] [ Google Scholar ]
• Hobert S. (2019a). “ How Are You, Chatbot? Evaluating Chatbots in Educational Settings - Results of a Literature Review ,” in 17. Fachtagung Bildungstechnologien, DELFI 2019 - 17th Conference on Education Technologies, DELFI 2019, Berlin, Germany, Sept 16–19, 2019. Editors Pinkwart N., Konert J., 259–270. 10.18420/delfi2019_289 [ CrossRef ] [ Google Scholar ]
• Hobert S., Meyer von Wolff R. (2019). “ Say Hello to Your New Automated Tutor - A Structured Literature Review on Pedagogical Conversational Agents ,” in 14th International Conference on Wirtschaftsinformatik, Siegen, Germany, Feb 23–27, 2019. Editors Pipek V., Ludwig T., (AIS; ). [ Google Scholar ]
• Hobert S. (2019b). Say Hello to ‘Coding Tutor’! Design and Evaluation of a Chatbot-Based Learning System Supporting Students to Learn to Program in International Conference on Information Systems (ICIS) 2019 Conference, Munich, Germany, Dec 15–18, 2019, AIS; 2661 , 1–17. [ Google Scholar ]
• Hobert S. (2020). Small Talk Conversations and the Long-Term Use of Chatbots in Educational Settings ‐ Experiences from a Field Study in 3rd International Workshop on Chatbot Research and Design, CONVERSATIONS 2019, Amsterdam, Netherlands, November 19–20: Lecture Notes in Computer Science. Editors Folstad A., Araujo T., Papadopoulos S., Law E., Granmo O., Luger E., Brandtzaeg P., (Springer; ) 11970 , 260–272. 10.1007/978-3-030-39540-7_18 [ CrossRef ] [ Google Scholar ]
• Hsieh S.-W. (2011). Effects of Cognitive Styles on an MSN Virtual Learning Companion System as an Adjunct to Classroom Instructions . Edu. Tech. Society 2 , 161–174. [ Google Scholar ]
• Huang J.-X., Kwon O.-W., Lee K.-S., Kim Y.-K. (2018). Improve the Chatbot Performance for the DB-CALL System Using a Hybrid Method and a Domain Corpus in Future-proof CALL: language learning as exploration and encounters–short papers from EUROCALL 2018, Jyväskylä, Finland, Aug 22–25, 2018. Editors Taalas P., Jalkanen J., Bradley L., Thouësny S., (Research-publishing.net; ). 10.14705/rpnet.2018.26.820 [ CrossRef ] [ Google Scholar ]
• Huang W., Hew K. F., Gonda D. E. (2019). Designing and Evaluating Three Chatbot-Enhanced Activities for a Flipped Graduate Course . Int. J. Mech. Engineer. Robotics. Research. 813–818. 10.18178/ijmerr.8.5.813-818 [ CrossRef ] [ Google Scholar ]
• Ismail M., Ade-Ibijola A. (2019). “ Lecturer's Apprentice: A Chatbot for Assisting Novice Programmers ,”in Proceedings - 2019 International Multidisciplinary Information Technology and Engineering Conference (IMITEC), Vanderbijlpark, South Africa, (IEEE), 1–8. 10.1109/IMITEC45504.2019.9015857 [ CrossRef ] [ Google Scholar ]
• Jia J. (2008). “ Motivate the Learners to Practice English through Playing with Chatbot CSIEC ,” in 3rd International Conference on Technologies for E-Learning and Digital Entertainment, Edutainment 2008, Nanjing, China, June 25–27, 2008, Lecture Notes in Computer Science, (Springer) 5093 , 180–191. 10.1007/978-3-540-69736-7_20 [ CrossRef ] [ Google Scholar ]
• Jia J. (2004). “ The Study of the Application of a Keywords-Based Chatbot System on the Teaching of Foreign Languages ,” in Proceedings of SITE 2004--Society for Information Technology and Teacher Education International Conference, Atlanta, Georgia, USA. Editors Ferdig R., Crawford C., Carlsen R., Davis N., Price J., Weber R., Willis D., (AACE), 1201–1207. [ Google Scholar ]
• Jivet I., Scheffel M., Drachsler H., Specht M. (2017). “ Awareness is not enough: Pitfalls of learning analytics dashboards in the educational practice ,” in 12th European Conference on Technology Enhanced Learning, EC-TEL 2017, Tallinn, Estonia, September 12–15, 2017, Lecture Notes in ComputerScience. Editors Lavoué E., Drachsler H., Verbert K., Broisin J., Pérez-Sanagustín M., (Springer), 82–96. 10.1007/978-3-319-66610-5_7 [ CrossRef ] [ Google Scholar ]
• Jung H., Lee J., Park C. (2020). Deriving Design Principles for Educational Chatbots from Empirical Studies on Human-Chatbot Interaction . J. Digit. Contents Society , 21 , 487–493. 10.9728/dcs.2020.21.3.487 [ CrossRef ] [ Google Scholar ]
• Kerly A., Bull S. (2006). “ The Potential for Chatbots in Negotiated Learner Modelling: A Wizard-Of-Oz Study ,” in 8th International Conference on Intelligent Tutoring Systems, ITS 2006, Jhongli, Taiwan, June 26–30, 2006, Lecture Notes in Computer Science. Editors Ikeda M., Ashley K. D., Chan T. W., (Springer; ) 4053 , 443–452. 10.1007/11774303 [ CrossRef ] [ Google Scholar ]
• Kerly A., Ellis R., Bull S. (2008). CALMsystem: A Conversational Agent for Learner Modelling . Knowledge-Based Syst. 21 , 238–246. 10.1016/j.knosys.2007.11.015 [ CrossRef ] [ Google Scholar ]
• Kerly A., Hall P., Bull S. (2007). Bringing Chatbots into Education: Towards Natural Language Negotiation of Open Learner Models . Knowledge-Based Syst. , 20 , 177–185. 10.1016/j.knosys.2006.11.014 [ CrossRef ] [ Google Scholar ]
• Kumar M. N., Chandar P. C. L., Prasad A. V., Sumangali K. (2016). “ Android Based Educational Chatbot for Visually Impaired People ,” in 2016 IEEE International Conference on Computational Intelligence and Computing Research, Chennai, India, December 15–17, 2016, 1–4. 10.1109/ICCIC.2016.7919664 [ CrossRef ] [ Google Scholar ]
• Lee K., Jo J., Kim J., Kang Y. (2019). Can Chatbots Help Reduce the Workload of Administrative Officers? - Implementing and Deploying FAQ Chatbot Service in a University in 21st International Conference on Human-Computer Interaction, HCII 2019: Communications in Computer and Information Science, Orlando, FL, USA, July 26–31, 2019. Editors Stephanidis C., (Springer; ) 1032 , 348–354. 10.1007/978-3-030-23522-2 [ CrossRef ] [ Google Scholar ]
• Lester J. C., Converse S. A., Kahler S. E., Barlow S. T., Stone B. A., Bhogal R. S. (1997). “ The Persona Effect: Affective Impact of Animated Pedagogical Agents ,” in Proceedings of the ACM SIGCHI Conference on Human factors in computing systems, Atlanta, Georgia, USA, March 22–27, 1997, (ACM), 359–366. [ Google Scholar ]
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A., et al. (2009). The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies that Evaluate Health Care Interventions: Explanation and Elaboration . J. Clin. Epidemiol. 62 ( 10 ), e1–e34. 10.1016/j.jclinepi.2009.06.006 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Lin M. P.-C., Chang D. (2020). Enhancing Post-secondary Writers’ Writing Skills with a Chatbot . J. Educ. Tech. Soc. 23 , 78–92. 10.2307/26915408 [ CrossRef ] [ Google Scholar ]
• Lin Y.-H., Tsai T. (2019). “ A Conversational Assistant on Mobile Devices for Primitive Learners of Computer Programming ,” in TALE 2019 - 2019 IEEE International Conference on Engineering, Technology and Education, Yogyakarta, Indonesia, December 10–13, 2019, (IEEE), 1–4. 10.1109/TALE48000.2019.9226015 [ CrossRef ] [ Google Scholar ]
• Linden A., Fenn J. (2003). Understanding Gartner’s Hype Cycles . Strategic Analysis Report No. R-20-1971 8 . Stamford, CT: Gartner, Inc. [ Google Scholar ]
• Liu Q., Huang J., Wu L., Zhu K., Ba S. (2020). CBET: Design and Evaluation of a Domain-specific Chatbot for mobile Learning . Univ. Access Inf. Soc. , 19 , 655–673. 10.1007/s10209-019-00666-x [ CrossRef ] [ Google Scholar ]
• Mamani J. R. C., Álamo Y. J. R., Aguirre J. A. A., Toledo E. E. G. (2019). “ Cognitive Services to Improve User Experience in Searching for Academic Information Based on Chatbot ,” in Proceedings of the 2019 IEEE 26th International Conference on Electronics, Electrical Engineering and Computing (INTERCON), Lima, Peru, August 12–14, 2019, (IEEE), 1–4. 10.1109/INTERCON.2019.8853572 [ CrossRef ] [ Google Scholar ]
• Martín-Martín A., Orduna-Malea E., Thelwall M., Delgado López-Cózar E. (2018). Google Scholar, Web of Science, and Scopus: A Systematic Comparison of Citations in 252 Subject Categories . J. Informetrics 12 ( 4 ), 1160–1177. 10.1016/j.joi.2018.09.002 [ CrossRef ] [ Google Scholar ]
• Matsuura S., Ishimura R. (2017). Chatbot and Dialogue Demonstration with a Humanoid Robot in the Lecture Class , in 11th International Conference on Universal Access in Human-Computer Interaction, UAHCI 2017, held as part of the 19th International Conference on Human-Computer Interaction, HCI 2017, Vancouver, Canada, July 9–14, 2017, Lecture Notes in Computer Science. Editors Antona M., Stephanidis C., (Springer) Vol. 10279 , 233–246. 10.1007/978-3-319-58700-4 [ CrossRef ] [ Google Scholar ]
• Matsuura S., Omokawa R. (2020). Being Aware of One’s Self in the Auto-Generated Chat with a Communication Robot in UAHCI 2020 , 477–488. 10.1007/978-3-030-49282-3 [ CrossRef ] [ Google Scholar ]
• McLoughlin C., Oliver R. (1998). Maximising the Language and Learning Link in Computer Learning Environments . Br. J. Educ. Tech. 29 ( 2 ), 125–136. 10.1111/1467-8535.00054 [ CrossRef ] [ Google Scholar ]
• Mendoza S., Hernández-León M., Sánchez-Adame L. M., Rodríguez J., Decouchant D., Meneses-Viveros A. (2020). “ Supporting Student-Teacher Interaction through a Chatbot ,” in 7th International Conference, LCT 2020, Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Lecture Notes in Computer Science. Editors Zaphiris P., Ioannou A., (Springer; ) 12206 , 93–107. 10.1007/978-3-030-50506-6 [ CrossRef ] [ Google Scholar ]
• Meyer V., Wolff R., Nörtemann J., Hobert S., Schumann M. (2020). “ Chatbots for the Information Acquisition at Universities ‐ A Student’s View on the Application Area ,“in 3rd International Workshop on Chatbot Research and Design, CONVERSATIONS 2019, Amsterdam, Netherlands, November 19–20, Lecture Notes in Computer Science. Editors Folstad A., Araujo T., Papadopoulos S., Law E., Granmo O., Luger E., Brandtzaeg P., (Springer) 11970 , 231–244. 10.1007/978-3-030-39540-7 [ CrossRef ] [ Google Scholar ]
• Na-Young K. (2018c). A Study on Chatbots for Developing Korean College Students’ English Listening and Reading Skills . J. Digital Convergence 16 . 19–26. 10.14400/JDC.2018.16.8.019 [ CrossRef ] [ Google Scholar ]
• Na-Young K. (2019). A Study on the Use of Artificial Intelligence Chatbots for Improving English Grammar Skills . J. Digital Convergence 17 , 37–46. 10.14400/JDC.2019.17.8.037 [ CrossRef ] [ Google Scholar ]
• Na-Young K. (2018a). Chatbots and Korean EFL Students’ English Vocabulary Learning . J. Digital Convergence 16 . 1–7. 10.14400/JDC.2018.16.2.001 [ CrossRef ] [ Google Scholar ]
• Na-Young K. (2018b). Different Chat Modes of a Chatbot and EFL Students’ Writing Skills Development . 1225–4975. 10.16933/sfle.2017.32.1.263 [ CrossRef ] [ Google Scholar ]
• Na-Young K. (2017). Effects of Different Types of Chatbots on EFL Learners’ Speaking Competence and Learner Perception . Cross-Cultural Studies 48 , 223–252. 10.21049/ccs.2017.48.223 [ CrossRef ] [ Google Scholar ]
• Nagata R., Hashiguchi T., Sadoun D. (2020). Is the Simplest Chatbot Effective in English Writing Learning Assistance? , in 16th International Conference of the Pacific Association for Computational Linguistics, PACLING, Hanoi, Vietnam, October 11–13, 2019, Communications in Computer and Information Science. Editors Nguyen L.-M., Tojo S., Phan X.-H., Hasida K., (Springer; ) Vol. 1215 , 245–246. 10.1007/978-981-15-6168-9 [ CrossRef ] [ Google Scholar ]
• Nelson T. O., Narens L. (1994). Why Investigate Metacognition . in Metakognition: Knowing About Knowing . Editors Metcalfe J., Shimamura P., (MIT Press) 13 , 1–25. [ Google Scholar ]
• Nghi T. T., Phuc T. H., Thang N. T. (2019). Applying Ai Chatbot for Teaching a Foreign Language: An Empirical Research . Int. J. Sci. Res. 8 . [ Google Scholar ]
• Ondas S., Pleva M., Hládek D. (2019). How Chatbots Can Be Involved in the Education Process . in ICETA 2019 - 17th IEEE International Conference on Emerging eLearning Technologies and Applications, Proceedings, Stary Smokovec, Slovakia, November 21–22, 2019. Editors Jakab F., (IEEE), 575–580. 10.1109/ICETA48886.2019.9040095 [ CrossRef ] [ Google Scholar ]
• Pereira J., Fernández-Raga M., Osuna-Acedo S., Roura-Redondo M., Almazán-López O., Buldón-Olalla A. (2019). Promoting Learners' Voice Productions Using Chatbots as a Tool for Improving the Learning Process in a MOOC . Tech. Know Learn. 24 , 545–565. 10.1007/s10758-019-09414-9 [ CrossRef ] [ Google Scholar ]
• Pérez J. Q., Daradoumis T., Puig J. M. M. (2020). Rediscovering the Use of Chatbots in Education: A Systematic Literature Review . Comput. Appl. Eng. Educ. 28 , 1549–1565. 10.1002/cae.22326 [ CrossRef ] [ Google Scholar ]
• Pérez-Marín D. (2021). A Review of the Practical Applications of Pedagogic Conversational Agents to Be Used in School and University Classrooms . Digital 1 ( 1 ), 18–33. 10.3390/digital1010002 [ CrossRef ] [ Google Scholar ]
• Pham X. L., Pham T., Nguyen Q. M., Nguyen T. H., Cao T. T. H. (2018). “ Chatbot as an Intelligent Personal Assistant for mobile Language Learning ,” in ACM International Conference Proceeding Series 10.1145/3291078.3291115 [ CrossRef ] [ Google Scholar ]
• Quincey E. de., Briggs C., Kyriacou T., Waller R. (2019). “ Student Centred Design of a Learning Analytics System ,” in Proceedings of the 9th International Conference on Learning Analytics & Knowledge, Tempe Arizona, USA, March 4–8, 2019, (ACM), 353–362. 10.1145/3303772.3303793 [ CrossRef ] [ Google Scholar ]
• Ram A., Prasad R., Khatri C., Venkatesh A., Gabriel R., Liu Q, et al. (2018). Conversational Ai: The Science behind the Alexa Prize , in 1st Proceedings of Alexa Prize (Alexa Prize 2017) . ArXiv [Preprint]. Available at: https://arxiv.org/abs/1801.03604 .
• Rebaque-Rivas P., Gil-Rodríguez E. (2019). Adopting an Omnichannel Approach to Improve User Experience in Online Enrolment at an E-Learning University , in 21st International Conference on Human-Computer Interaction, HCII 2019: Communications in Computer and Information Science, Orlando, FL, USA, July 26–31, 2019. Editors Stephanidis C., (Springer; ), 115–122. 10.1007/978-3-030-23525-3 [ CrossRef ] [ Google Scholar ]
• Robinson C. (2019). Impressions of Viability: How Current Enrollment Management Personnel And Former Students Perceive The Implementation of A Chatbot Focused On Student Financial Communication. Higher Education Doctoral Projects.2 . https://aquila.usm.edu/highereddoctoralprojects/2 . [ Google Scholar ]
• Ruan S., Jiang L., Xu J., Tham B. J.-K., Qiu Z., Zhu Y., Murnane E. L., Brunskill E., Landay J. A. (2019). “ QuizBot: A Dialogue-based Adaptive Learning System for Factual Knowledge ,” in 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019, Glasgow, Scotland, United Kingdom, May 4–9, 2019, (ACM), 1–13. 10.1145/3290605.3300587 [ CrossRef ] [ Google Scholar ]
• Sandoval Z. V. (2018). Design and Implementation of a Chatbot in Online Higher Education Settings . Issues Inf. Syst. 19 , 44–52. 10.48009/4.iis.2018.44-52 [ CrossRef ] [ Google Scholar ]
• Sandu N., Gide E. (2019). “ Adoption of AI-Chatbots to Enhance Student Learning Experience in Higher Education in india ,” in 18th International Conference on Information Technology Based Higher Education and Training, Magdeburg, Germany, September 26–27, 2019, (IEEE), 1–5. 10.1109/ITHET46829.2019.8937382 [ CrossRef ] [ Google Scholar ]
• Saygin A. P., Cicekli I., Akman V. (2000). Turing Test: 50 Years Later . Minds and Machines 10 ( 4 ), 463–518. 10.1023/A:1011288000451 [ CrossRef ] [ Google Scholar ]
• Sinclair A., McCurdy K., Lucas C. G., Lopez A., Gaševic D. (2019). “ Tutorbot Corpus: Evidence of Human-Agent Verbal Alignment in Second Language Learner Dialogues ,” in EDM 2019 - Proceedings of the 12th International Conference on Educational Data Mining. [ Google Scholar ]
• Smutny P., Schreiberova P. (2020). Chatbots for Learning: A Review of Educational Chatbots for the Facebook Messenger . Comput. Edu. 151 , 103862. 10.1016/j.compedu.2020.103862 [ CrossRef ] [ Google Scholar ]
• Song D., Rice M., Oh E. Y. (2019). Participation in Online Courses and Interaction with a Virtual Agent . Int. Rev. Res. Open. Dis. 20 , 44–62. 10.19173/irrodl.v20i1.3998 [ CrossRef ] [ Google Scholar ]
• Stapić Z., Horvat A., Vukovac D. P. (2020). Designing a Faculty Chatbot through User-Centered Design Approach , in 22nd International Conference on Human-Computer Interaction,HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Lecture Notes in Computer Science. Editors Stephanidis C., Harris D., Li W. C., Schmorrow D. D., Fidopiastis C. M., Zaphiris P., (Springer; ), 472–484. 10.1007/978-3-030-60128-7 [ CrossRef ] [ Google Scholar ]
• Subramaniam N. K. (2019). Teaching and Learning via Chatbots with Immersive and Machine Learning Capabilities . In International Conference on Education (ICE 2019) Proceedings, Kuala Lumpur, Malaysia, April 10–11, 2019. Editors Ali S. A. H., Subramaniam T. T., Yusof S. M., 145–156. [ Google Scholar ]
• Sugondo A. F., Bahana R. (2019). “ Chatbot as an Alternative Means to Access Online Information Systems ,” in 3rd International Conference on Eco Engineering Development, ICEED 2019, Surakarta, Indonesia, November 13–14, 2019, IOP Conference Series: Earth and Environmental Science, (IOP Publishing) 426 . 10.1088/1755-1315/426/1/012168 [ CrossRef ] [ Google Scholar ]
• Suwannatee S., Suwanyangyuen A. (2019). “ Reading Chatbot” Mahidol University Library and Knowledge Center Smart Assistant ,” in Proceedings for the 2019 International Conference on Library and Information Science (ICLIS), Taipei, Taiwan, July 11–13, 2019. [ Google Scholar ]
• Vaidyam A. N., Wisniewski H., Halamka J. D., Kashavan M. S., Torous J. B. (2019). Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape . Can. J. Psychiatry 64 ( 7 ), 456–464. 10.1177/0706743719828977 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Vijayakumar B., Höhn S., Schommer C. (2019). “ Quizbot: Exploring Formative Feedback with Conversational Interfaces ,” in 21st International Conference on Technology Enhanced Assessment, TEA 2018, Amsterdam, Netherlands, Dec 10-11, 2018. Editors Draaijer S., Joosten-ten B. D., Ras E., (Springer; ), 102–120. 10.1007/978-3-030-25264-9 [ CrossRef ] [ Google Scholar ]
• Virtanen M. A., Haavisto E., Liikanen E., Kääriäinen M. (2018). Ubiquitous Learning Environments in Higher Education: A Scoping Literature Review . Educ. Inf. Technol. 23 ( 2 ), 985–998. 10.1007/s10639-017-9646-6 [ CrossRef ] [ Google Scholar ]
• Wildman T. M., Magliaro S. G., Niles R. A., Niles J. A. (1992). Teacher Mentoring: An Analysis of Roles, Activities, and Conditions . J. Teach. Edu. 43 ( 3 ), 205–213. 10.1177/0022487192043003007 [ CrossRef ] [ Google Scholar ]
• Wiley D., Edwards E. K. (2002). Online Self-Organizing Social Systems: The Decentralized Future of Online Learning . Q. Rev. Distance Edu. 3 ( 1 ), 33–46. [ Google Scholar ]
• Winkler R., Soellner M. (2018). Unleashing the Potential of Chatbots in Education: A State-Of-The-Art Analysis . in Academy of Management Annual Meeting Proceedings 2018 2018 ( 1 ), 15903. 10.5465/AMBPP.2018.15903abstract [ CrossRef ] [ Google Scholar ]
• Winne P. H., Hadwin A. F. (2008). “ The Weave of Motivation and Self-Regulated Learning ,” in Motivation and Self-Regulated Learning: Theory, Research, and Applications . Editors Schunk D. H., Zimmerman B. J., (Mahwah, NJ: Lawrence Erlbaum Associates Publishers; ), 297–314. [ Google Scholar ]
• Wisniewski B., Zierer K., Hattie J. (2019). The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research . Front. Psychol. 10 , 1664–1078. 10.3389/fpsyg.2019.03087 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Wolfbauer I., Pammer-Schindler V., Rose C. P. (2020). “ Rebo Junior: Analysis of Dialogue Structure Quality for a Reflection Guidance Chatbot ,” in Proceedings of the Impact Papers at EC-TEL 2020, co-located with the 15th European Conference on Technology-Enhanced Learning “Addressing global challenges and quality education” (EC-TEL 2020), Virtual, Sept 14–18, 2020. Editors Broos T., Farrell T., 1–14. [ Google Scholar ]
• Xiao Z., Zhou M. X., Fu W.-T. (2019). “ Who should be my teammates: Using a conversational agent to understand individuals and help teaming ,” in IUI’19: Proceedings of the 24th International Conference on Intelligent User Interfaces, Marina del Ray, California, USA, March 17–20, 2019, (ACM), 437–447. 10.1145/3301275.3302264 [ CrossRef ] [ Google Scholar ]
• Xu A., Liu Z., Guo Y., Sinha V., Akkiraju R. (2017). “ A New Chatbot for Customer Service on Social media ,” in Proceedings of the 2017 CHI conference on human factors in computing systems, Denver, Colorado, USA, May 6–11, 2017, ACM, 3506–3510. 10.1145/3025453.3025496 [ CrossRef ] [ Google Scholar ]
• Yin J., Goh T.-T., Yang B., Xiaobin Y. (2020). Conversation Technology with Micro-learning: The Impact of Chatbot-Based Learning on Students' Learning Motivation and Performance . J. Educ. Comput. Res. 59 , 154–177. 10.1177/0735633120952067 [ CrossRef ] [ Google Scholar ]

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

A literature survey of recent advances in chatbots.

1. Introduction

2. chatbots background, 3. methodology, 3.1. stage one: information gathering, search terms and databases identification, 3.2. stage two: article filtering and reviewing, 3.2.1. filtering articles, 3.2.2. reviewing articles.

• Chatbots’ History and Evolution: this aspect encompasses all papers that presented a detailed description of chatbots’ evolution over time. This category is fundamental since it helped us understand the trends and technologies that ascended or discarded over time, indicating the evolution of the chatbot. It also helped us discover how and why chatbots emerged and how their applications and purposes changed over time. Section 2 offers overview of our finding on chatbots history and evolution.
• Chatbots’ Implementation: this aspect includes papers that present examples of chatbots architectural design and implementation. This category allowed us to identify the commonly used algorithms for chatbots and the specific algorithms that are used for diverse types of chatbots based on the purpose of chatbot application. This also allowed to identify the industry standards in terms of chatbots’ models and algorithms, as well as their shortcomings and limitations. Detailed implementation approaches to chatbots are given in Section 4.1 .
• Chatbots’ Evaluation: For this aspect, some articles focused on the evaluation methods and metrics used for measuring chatbots performance. It was important to identify these papers in order to understand the way chatbots are evaluated and the evaluation metrics and methods used. We outline the various evaluation metrics in Section 4.3 .
• Chatbots’ Applications: this aspect encompasses all examples of chatbots applied to a specific domain, such as education, finance, customer support and psychology. Papers pertaining to this category helped us tie information from previous categories and get a better understanding of what models and what features are used for which applications in order to serve different purposes. We identify and offer overview on the application of chatbots in Section 4.4 .
• Dataset: this category was used to classify chatbots depending on the dataset used to train machine learning algorithms for the development of language model. Section 4.2 highlights the main datasets that have been used in previous studies.

4. Literature Review Analysis

4.1. implementation approaches to chatbots, 4.1.1. rule-based chatbots, 4.1.2. artificial intelligence chatbots, 4.2. datasets used, 4.3. evaluation, 4.4. applications of chatbots.

• Machine Learning in general and Deep Learning in particular, require a large amount of training data; although training data is becoming increasingly available but finding a suitable dataset might still represent a challenge. Furthermore, data needs to be preprocessed in order to be used and might often contain unwanted noise.
• Training is costly in terms of infrastructure and human resources, and time consuming.
• Chatbots, when they are not used for social or companion chatbots, are usually applied to a specific domain, which means that they require domain-specific training data (e.g., products information and details, financial information, educational material, healthcare information). This type of data is often confidential due to its nature; they are not readily available in open access to train a Deep Learning engine. Furthermore, given the nature of the data needed and of the tasks the chatbot is required to carry out (e.g., access a customer’s purchase history, or give more information about a product feature), Information Retrieval might be the best solution for most use-case applications.

5. Related Works

6. discussion, 7. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

• Jia, J. The Study of the Application of a Keywords-based Chatbot System on the Teaching of Foreign Languages. arXiv 2003 , arXiv:cs/0310018. [ Google Scholar ]
• Sojasingarayar, A. Seq2Seq AI Chatbot with Attention Mechanism. Master’s Thesis, Department of Artificial Intelligence, IA School/University-GEMA Group, Boulogne-Billancourt, France, 2020. [ Google Scholar ]
• Bala, K.; Kumar, M.; Hulawale, S.; Pandita, S. Chat-Bot For College Management System Using A.I. Int. Res. J. Eng. Technol. (IRJET) 2017 , 4 , 4. [ Google Scholar ]
• Ayanouz, S.; Abdelhakim, B.A.; Benhmed, M. A Smart Chatbot Architecture based NLP and Machine Learning for Health Care Assistance. In Proceedings of the 3rd International Conference on Networking, Information Systems & Security, Marrakech, Morocco, 31 March–2 April 2020; pp. 1–6. [ Google Scholar ] [ CrossRef ]
• Kumar, R.; Ali, M.M. A Review on Chatbot Design and Implementation Techniques. Int. J. Eng. Technol. 2020 , 7 , 11. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Cahn, J. CHATBOT: Architecture, Design, & Development. Ph.D. Thesis, University of Pennsylvania, School of Engineering and Applied Science, Philadelphia, PA, USA, 2017. [ Google Scholar ]
• Okuda, T.; Shoda, S. AI-based Chatbot Service for Financial Industry. FUJITSU Sci. Tech. J. 2018 , 54 , 5. [ Google Scholar ]
• Brandtzaeg, P.B.; Følstad, A. Why People Use Chatbots. In Internet Science ; Kompatsiaris, I., Cave, J., Satsiou, A., Carle, G., Passani, A., Kontopoulos, E., Diplaris, S., McMillan, D., Eds.; Lecture Notes in Computer Science; Springer International Publishing: Berlin/Heidelberg, Germany, 2017; Volume 10673, pp. 377–392. [ Google Scholar ]
• Costa, P. Conversing with personal digital assistants: On gender and artificial intelligence. J. Sci. Technol. Arts 2018 , 10 , 59–72. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Go, E.; Sundar, S.S. Humanizing chatbots: The effects of visual, identity and conversational cues on humanness perceptions. Comput. Hum. Behav. 2019 , 97 , 304–316. [ Google Scholar ] [ CrossRef ]
• Luo, X.; Tong, S.; Fang, Z.; Qu, Z. Frontiers: Machines vs. Humans: The Impact of Artificial Intelligence Chatbot Disclosure on Customer Purchases. Mark. Sci. 2019 , 38 , 913–1084. [ Google Scholar ] [ CrossRef ]
• Christensen, S.; Johnsrud, S.; Ruocco, M.; Ramampiaro, H. Context-Aware Sequence-to-Sequence Models for Conversational Systems. arXiv 2018 , arXiv:1805.08455. [ Google Scholar ]
• Fernandes, A. NLP, NLU, NLG and How Chatbots Work. 2018. Available online: https://chatbotslife.com/nlp-nlu-nlg-and-how-chatbots-work-dd7861dfc9df (accessed on 19 May 2021).
• Turing, A.M. Computing Machinery and Intelligence. Mind 1950 , LIX , 433–460. [ Google Scholar ] [ CrossRef ]
• Weizenbaum, J. ELIZA–A Computer Program for the Study of Natural Language Communication between Man and Machine. Commun. ACM 1966 , 9 , 36–45. [ Google Scholar ] [ CrossRef ]
• Shum, H.y.; He, X.d.; Li, D. From Eliza to XiaoIce: Challenges and opportunities with social chatbots. Front. Inf. Technol. Electron. Eng. 2018 , 19 , 10–26. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Zemčík, T. A Brief History of Chatbots. DEStech Trans. Comput. Sci. Eng. 2019 , 2019 , 14–18. [ Google Scholar ] [ CrossRef ]
• Bradeško, L.; Mladenić, D. A Survey of Chabot Systems through a Loebner Prize Competition. In Proceedings of the Slovenian Language Technologies Society Eighth Conference of Language Technologies, Ljubljana, Slovenia, 8–9 October 2012; Institut Jožef Stefan: Ljubljana, Slovenia, 2012; pp. 34–37. [ Google Scholar ]
• Wilcox, B. Winning the Loebner’s. 2014. Available online: https://www.gamasutra.com/blogs/BruceWilcox/20141020/228091/Winning_the_Loebners.php (accessed on 26 November 2020).
• AbuShawar, B.; Atwell, E. ALICE Chatbot: Trials and Outputs. Comput. Y Sist. 2015 , 19 , 4. [ Google Scholar ] [ CrossRef ]
• Dormehl, L. Microsoft’s Friendly Xiaoice A.I Can Figure out What You Want—Before You Ask. 2018. Available online: https://www.digitaltrends.com/cool-tech/xiaoice-microsoft-future-of-ai-assistants/ (accessed on 5 May 2021).
• Spencer, G. Much More Than a Chatbot: China’s Xiaoice Mixes AI with Emotions and Wins over Millions of Fans. Microsoft Asia News Cent . 2018. Available online: https://news.microsoft.com/apac/features/much-more-than-a-chatbot-chinas-xiaoice-mixes-ai-with-emotions-and-wins-over-millions-of-fans/ (accessed on 29 June 2021).
• Zhou, L.; Gao, J.; Li, D.; Shum, H.Y. The Design and Implementation of XiaoIce, an Empathetic Social Chatbot. arXiv 2019 , arXiv:1812.08989. [ Google Scholar ] [ CrossRef ]
• Radziwill, N.; Benton, M. Evaluating Quality of Chatbots and Intelligent Conversational Agents. arXiv 2017 , arXiv:1704.04579. [ Google Scholar ]
• Sheehan, B.; Jin, H.S.; Gottlieb, U. Customer service chatbots: Anthropomorphism and adoption. J. Bus. Res. 2020 , 115 , 14–24. [ Google Scholar ] [ CrossRef ]
• Adamopoulou, E.; Moussiades, L. An Overview of Chatbot Technology. Artif. Intell. Appl. Innov. 2020 , 584 , 373–383. [ Google Scholar ] [ CrossRef ]
• Nuruzzaman, M.; Hussain, O.K. A Survey on Chatbot Implementation in Customer Service Industry through Deep Neural Networks. In Proceedings of the 2018 IEEE 15th International Conference on e-Business Engineering (ICEBE), Xi’an, China, 12–14 October 2018; pp. 54–61. [ Google Scholar ] [ CrossRef ]
• Ketakee, N.; Champaneria, T. Chatbots: An Overview Types, Architecture, Tools and Future Possibilities. IJSRD-Int. J. Sci. Res. Dev. 2017 , 5 , 6. [ Google Scholar ]
• Yan, R.; Song, Y.; Wu, H. Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval-SIGIR ’16, Pisa, Italy, 17–21July 2016; ACM Press: Pisa, Italy, 2016; pp. 55–64. [ Google Scholar ] [ CrossRef ]
• Abdul-Kader, S.A.; Woods, D.J. Survey on Chatbot Design Techniques in Speech Conversation Systems. Int. J. Adv. Comput. Sci. Appl. 2015 , 6 , 9. [ Google Scholar ]
• Lu, Z.; Li, H. A Deep Architecture for Matching Short Texts. Adv. Neural Inf. Process. Syst. 2013 , 26 , 1367–1375. [ Google Scholar ]
• Shang, L.; Lu, Z.; Li, H. Neural Responding Machine for Short-Text Conversation. arXiv 2015 , arXiv:1503.02364. [ Google Scholar ]
• Sordoni, A.; Galley, M.; Auli, M.; Brockett, C.; Ji, Y.; Mitchell, M.; Nie, J.Y.; Gao, J.; Dolan, B. A Neural Network Approach to Context-Sensitive Generation of Conversational Responses. arXiv 2015 , arXiv:1506.06714. [ Google Scholar ]
• Vinyals, O.; Le, Q. A Neural Conversational Model. arXiv 2015 , arXiv:1506.05869. [ Google Scholar ]
• Sutskever, I.; Vinyals, O.; Le, Q.V. Sequence to Sequence Learning with Neural Networks. Adv. Neural Inf. Process. Syst. 2014 , 2 , 3104–3112. [ Google Scholar ]
• Jurafsky, D.; Martin, J. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition ; Dorling Kindersley Pvt, Limited: London, UK, 2020; Volume 2. [ Google Scholar ]
• Strigér, A. End-to-End Trainable Chatbot for Restaurant Recommendations. Master’s Thesis, KTH Royal Institute of Technology, Stockholm, Sweden, 2017. [ Google Scholar ]
• Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I. Attention Is All You Need. arXiv 2017 , arXiv:1706.03762. [ Google Scholar ]
• Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv 2019 , arXiv:1810.04805. [ Google Scholar ]
• Kitaev, N.; Kaiser, L.; Levskaya, A. Reformer: The Efficient Transformer. arXiv 2020 , arXiv:2001.04451. [ Google Scholar ]
• Dai, Z.; Yang, Z.; Yang, Y.; Carbonell, J.; Le, Q.V.; Salakhutdinov, R. Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. arXiv 2019 , arXiv:1901.02860. [ Google Scholar ]
• Adiwardana, D.; Luong, M.T.; So, D.R.; Hall, J.; Fiedel, N.; Thoppilan, R.; Yang, Z.; Kulshreshtha, A.; Nemade, G.; Lu, Y.; et al. Towards a Human-like Open-Domain Chatbot. arXiv 2020 , arXiv:2001.09977. [ Google Scholar ]
• Bahdanau, D.; Cho, K.; Bengio, Y. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv 2016 , arXiv:1409.0473. [ Google Scholar ]
• So, D.R.; Liang, C.; Le, Q.V. The Evolved Transformer. arXiv 2019 , arXiv:1901.11117. [ Google Scholar ]
• Dauphin, Y.N.; Fan, A.; Auli, M.; Grangier, D. Language Modeling with Gated Convolutional Networks. arXiv 2017 , arXiv:cs.CL/1612.08083. [ Google Scholar ]
• Ramachandran, P.; Zoph, B.; Le, Q.V. Searching for Activation Functions. arXiv 2017 , arXiv:cs.NE/1710.05941. [ Google Scholar ]
• Lison, P.; Tiedemann, J. OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Portorož, Slovenia, 23–28 May 2016; European Language Resources Association (ELRA): Portorož, Slovenia, 2016; pp. 923–929. [ Google Scholar ]
• Zhong, P.; Wang, D.; Miao, C. An Affect-Rich Neural Conversational Model with Biased Attention and Weighted Cross-Entropy Loss. arXiv 2018 , arXiv:1811.07078. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Klein, G.; Kim, Y.; Deng, Y.; Senellart, J.; Rush, A. OpenNMT: Open-Source Toolkit for Neural Machine Translation. In Proceedings of the ACL 2017, System Demonstrations, Vancouver, QC, Canada, 30 July–4 August 2017; Association for Computational Linguistics: Vancouver, QC, Canada, 2017; pp. 67–72. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Li, J.; Monroe, W.; Ritter, A.; Galley, M.; Gao, J.; Jurafsky, D. Deep Reinforcement Learning for Dialogue Generation. arXiv 2016 , arXiv:1606.01541. [ Google Scholar ]
• Li, Y.; Su, H.; Shen, X.; Li, W.; Cao, Z.; Niu, S. DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Taipei, Taiwan, 27 November–1 December 2017; Asian Federation of Natural Language Processing: Taipei, Taiwan, 2017; pp. 986–995. [ Google Scholar ]
• Li, J.; Monroe, W.; Shi, T.; Jean, S.; Ritter, A.; Jurafsky, D. Adversarial Learning for Neural Dialogue Generation. arXiv 2017 , arXiv:1701.06547. [ Google Scholar ]
• Danescu-Niculescu-Mizil, C.; Lee, L. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, Portland, OR, USA, 23 June 2011. [ Google Scholar ]
• He, T.; Liu, J.; Cho, K.; Ott, M.; Liu, B.; Glass, J.; Peng, F. Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models. arXiv 2021 , arXiv:1910.07117. [ Google Scholar ]
• Roller, S.; Dinan, E.; Goyal, N.; Ju, D.; Williamson, M.; Liu, Y.; Xu, J.; Ott, M.; Shuster, K.; Smith, E.M.; et al. Recipes for building an open-domain chatbot. arXiv 2020 , arXiv:2004.13637. [ Google Scholar ]
• Ghandeharioun, A.; Shen, J.H.; Jaques, N.; Ferguson, C.; Jones, N.; Lapedriza, A.; Picard, R. Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems. arXiv 2019 , arXiv:1906.09308. [ Google Scholar ]
• Kim, J.; Oh, S.; Kwon, O.W.; Kim, H. Multi-Turn Chatbot Based on Query-Context Attentions and Dual Wasserstein Generative Adversarial Networks. Appl. Sci. 2019 , 9 , 3908. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Walker, M.A.; Litman, D.J.; Kamm, C.A.; Abella, A. PARADISE: A framework for evaluating spoken dialogue agents. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics, Madrid, Spain, 7–12 July 1997; pp. 271–280. [ Google Scholar ] [ CrossRef ]
• Hung, V.; Elvir, M.; Gonzalez, A.; DeMara, R. Towards a method for evaluating naturalness in conversational dialog systems. In Proceedings of the 2009 IEEE International Conference on Systems, Man and Cybernetics, San Antonio, TX, USA, 11–14 October 2009. [ Google Scholar ]
• Kempson, R.; Gargett, A.; Gregoromichelaki, E. Clarification Requests: An Incremental Account. Decalog 2007 , 2007 , 65. [ Google Scholar ]
• Li, M.; Weston, J.; Roller, S. ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons. arXiv 2019 , arXiv:1909.03087. [ Google Scholar ]
• Lin, Z.; Liu, Z.; Winata, G.I.; Cahyawijaya, S.; Madotto, A.; Bang, Y.; Ishii, E.; Fung, P. XPersona: Evaluating Multilingual Personalized Chatbot. arXiv 2020 , arXiv:2003.07568. [ Google Scholar ]
• Sedoc, J.; Ippolito, D.; Kirubarajan, A.; Thirani, J.; Ungar, L.; Callison-Burch, C. ChatEval: A Tool for Chatbot Evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), Minneapolis, MN, USA, 2–7 June 2019; Association for Computational Linguistics: Minneapolis, MN, USA, 2019; pp. 60–65. [ Google Scholar ] [ CrossRef ]
• Przegalinska, A.; Ciechanowski, L.; Stroz, A.; Gloor, P.; Mazurek, G. In bot we trust: A new methodology of chatbot performance measures. Bus. Horizons 2019 , 62 , 785–797. [ Google Scholar ] [ CrossRef ]
• Saikh, T.; Naskar, S.K.; Ekbal, A.; Bandyopadhyay, S. Textual Entailment Using Machine Translation Evaluation Metrics. In Computational Linguistics and Intelligent Text Processing ; Gelbukh, A., Ed.; Series Title: Lecture Notes in Computer Science; Springer International Publishing: Berlin/Heidelberg, Germany, 2018; Volume 10761, pp. 317–328. [ Google Scholar ] [ CrossRef ]
• Wood, T. F-Score. Available online: https://deepai.org/machine-learning-glossary-and-terms/f-score (accessed on 1 November 2021).
• Xu, J.; Ju, D.; Li, M.; Boureau, Y.L.; Weston, J.; Dinan, E. Recipes for Safety in Open-domain Chatbots. arXiv 2020 , arXiv:2010.07079. [ Google Scholar ]
• Cuayáhuitl, H.; Lee, D.; Ryu, S.; Cho, Y.; Choi, S.; Indurthi, S.; Yu, S.; Choi, H.; Hwang, I.; Kim, J. Ensemble-based deep reinforcement learning for chatbots. Neurocomputing 2019 , 366 , 118–130. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Chen, S.; Beeferman, D.; Rosenfeld, R. Evaluation Metrics for Language Models. 1998, p. 6. Available online: https://kilthub.cmu.edu/articles/journalcontribution/EvaluationMetricsForLanguageModels/6605324 (accessed on 8 June 2021).
• Dhyani, M.; Kumar, R. An intelligent Chatbot using deep learning with Bidirectional RNN and attention model. Mater. Today Proc. 2020 , 34 , 817–824. [ Google Scholar ] [ CrossRef ]
• John, A.; Di Caro, L.; Robaldo, L.; Boella, G. Legalbot: A Deep Learning-Based Conversational Agent in the Legal Domain ; Springer: Berlin/Heidelberg, Germany, 2017; p. 273. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Higashinaka, R.; Imamura, K.; Meguro, T.; Miyazaki, C.; Kobayashi, N.; Sugiyama, H.; Hirano, T.; Makino, T.; Matsuo, Y. Towards an open-domain conversational system fully based on natural language processing. In Proceedings of the COLING 2014, the 25th International Conference on Computational Linguistics, Dublin, Ireland, 23–29 August 2014; p. 12. [ Google Scholar ]
• Agarwal, A.; Lavie, A. Meteor, M-BLEU and M-TER: Evaluation Metrics for High-Correlation with Human Rankings of Machine Translation Output. In Proceedings of the Third Workshop on Statistical Machine Translation, Columbus, OH, USA, 19 June 2008; Association for Computational Linguistics: Columbus, OH, USA, 2008; pp. 115–118. [ Google Scholar ]
• Banerjee, S.; Lavie, A. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the Acl Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, MI, USA, 29 June 2005; p. 8. [ Google Scholar ]
• Xu, K.; Lei, J.; Kiros, R.; Cho, K.; Courville, A.; Salakhutdinov, R.; Zemel, R.S.; Bengio, Y. Show, Attend and Tell: Neural Image CaptionGeneration with Visual Attention. Proc. Int. Conf. Mach. Learn. 2015 , 37 , 10. [ Google Scholar ]
• Snover, M.; Dorr, B.; Schwartz, R.; Micciulla, L.; Makhoul, J. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of the Association for Machine Translation in the Americas, Cambridge, MA, USA, 8–12 August 2006; p. 9. [ Google Scholar ]
• Kannan, A.; Vinyals, O. Adversarial Evaluation of Dialogue Models. arXiv 2017 , arXiv:1701.08198. [ Google Scholar ]
• Kuksenok, K.; Martyniv, A. Evaluation and Improvement of Chatbot Text Classification Data Quality Using Plausible Negative Examples. In Proceedings of the First Workshop on NLP for Conversational AI, Florence, Italy, 1 August 2019; Association for Computational Linguistics: Florence, Italy, 2019; pp. 87–95. [ Google Scholar ] [ CrossRef ]
• Ebner, M.; Ebner, M. Potential of Bots for Encyclopedia. IPSI BgD Trans. 2020 , 16 , 54–60. [ Google Scholar ]
• Arifi, J.; Ebner, M.; Ebner, M. Potentials of Chatbots for Spell Check among Youngsters. Int. J. Learn. Anal. Artif. Intell. Educ. 2019 , 1 , 77. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Palasundram, K.; Sharef, N.M.; Nasharuddin, N.A.; Kasmiran, K.A.; Azman, A. Sequence to Sequence Model Performance for Education Chatbot. Int. J. Emerg. Technol. Learn. 2019 , 14 , 56–68. [ Google Scholar ] [ CrossRef ]
• Nwankwo, W. Interactive Advising with Bots: Improving Academic Excellence in Educational Establishments. Am. J. Oper. Manag. Inf. Syst. 2018 , 3 , 6. [ Google Scholar ] [ CrossRef ]
• Fei, Y.; Petrina, S. Using Learning Analytics to Understand the Design of an Intelligent Language Tutor—Chatbot Lucy. Int. J. Adv. Comput. Sci. Appl. 2013 , 4 , 124–131. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Augello, A.; Pilato, G.; Machi, A.; Gaglio, S. An Approach to Enhance Chatbot Semantic Power and Maintainability: Experiences within the FRASI Project. In Proceedings of the 2012 IEEE Sixth International Conference on Semantic Computing, Palermo, Italy, 19–21 September 2012. [ Google Scholar ]
• Berger, R.; Ebner, M.; Ebner, M. Conception of a Conversational Interface to Provide a Guided Search of Study Related Data. Int. J. Emerg. Technol. Learn. (IJET) 2019 , 14 , 37. [ Google Scholar ] [ CrossRef ]
• Athota, L.; Shukla, V.K.; Pandey, N.; Rana, A. Chatbot for Healthcare System Using Artificial Intelligence. In Proceedings of the 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, India, 4–5 June 2020; IEEE: Noida, India, 2020; pp. 619–622. [ Google Scholar ] [ CrossRef ]
• Cui, L.; Huang, S.; Wei, F.; Tan, C.; Duan, C.; Zhou, M. SuperAgent: A Customer Service Chatbot for E-commerce Websites. In Proceedings of the ACL 2017, System Demonstrations, Vancouver, QC, Canada, 30 July–4 August 2017; Association for Computational Linguistics: Vancouver, QC, Canada, 2017; pp. 97–102. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Ikumoro, A.O.; Jawad, M.S. Intention to Use Intelligent Conversational Agents in e-Commerce among Malaysian SMEs: An Integrated Conceptual Framework Based on Tri-theories including Unified Theory of Acceptance, Use of Technology (UTAUT), and T-O-E. Int. J. Acad. Res. Bus. Soc. Sci. 2019 , 9 , 205–235. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Singh, R.; Paste, M.; Shinde, N.; Patel, H.; Mishra, N. Chatbot using TensorFlow for small Businesses. In Proceedings of the 2018 Second International Conference on Inventive Communication and Computational Technologies (ICICCT), Coimbatore, India, 20–21 April 2018; pp. 1614–1619. [ Google Scholar ] [ CrossRef ]
• Casillo, M.; Colace, F.; Fabbri, L.; Lombardi, M.; Romano, A.; Santaniello, D. Chatbot in Industry 4.0: An Approach for Training New Employees. In Proceedings of the 2020 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), Takamatsu, Japan, 8–11 December 2020; pp. 371–376. [ Google Scholar ] [ CrossRef ]
• Melo, G.; Law, E.; Alencar, P.; Cowan, D. Exploring Context-Aware Conversational Agents in Software Development. arXiv 2020 , arXiv:2006.02370. [ Google Scholar ]
• Sheikh, S.; Tiwari, V.; Bansal, S. Generative model chatbot for Human Resource using Deep Learning. In Proceedings of the 2019 International Conference on Data Science and Engineering, Patna, India, 26–28 September 2019; p. 132. [ Google Scholar ] [ CrossRef ]
• Rahman, A.M.; Mamun, A.A.; Islam, A. Programming challenges of chatbot: Current and future prospective. In Proceedings of the 2017 IEEE Region 10 Humanitarian Technology Conference (R10-HTC), Dhaka, Bangladesh, 21–23 December 2017; pp. 75–78. [ Google Scholar ] [ CrossRef ]
• Bernardini, A.A.; Sônego, A.A.; Pozzebon, E. Chatbots: An Analysis of the State of Art of Literature. In Proceedings of the 1st Workshop on Advanced Virtual Environments and Education (WAVE2 2018), Florianópolis, Brazil, 4–5 October 2018; Brazilian Computer Society (Sociedade Brasileira de Computação-SBC): Florianópolis, Santa Catarina, Brazil, 2018; p. 1. [ Google Scholar ] [ CrossRef ] [ Green Version ]

Click here to enlarge figure

Share and Cite

Caldarini, G.; Jaf, S.; McGarry, K. A Literature Survey of Recent Advances in Chatbots. Information 2022 , 13 , 41. https://doi.org/10.3390/info13010041

Caldarini G, Jaf S, McGarry K. A Literature Survey of Recent Advances in Chatbots. Information . 2022; 13(1):41. https://doi.org/10.3390/info13010041

Caldarini, Guendalina, Sardar Jaf, and Kenneth McGarry. 2022. "A Literature Survey of Recent Advances in Chatbots" Information 13, no. 1: 41. https://doi.org/10.3390/info13010041

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Literature Reviews

• Getting Started
• Choosing a Type of Review
• Developing a Research Question
• Searching the Literature
• Searching Tips
• ChatGPT [beta]
• ChatGPT Resources
• Documenting your Search
• Using Citation Managers
• Concept Mapping
• Writing the Review
• Further Resources

UM's Report on GenerativeAI

The initial report from the U-M Generative Artificial Intelligence Advisory (GAIA) Committee.

—User-friendly Interface that allows faculty, staff and students to engage in chat-based queries and benefit from the expertise of GENAI technology

— U-M GPT, enhances, teaching learning research in collaboration, providing a valuable and equitable resource for the entire university community

— designed to be accessible, including for use with screen readers

— provides access to GPT-3.5 and GPT-4 with no charge

https://chat.openai.com

• The knowledge cutoff for the ChatGPT 3.5 is September 2021 and it has no access to the Internet. Academic users may consider alternatives such as  Semantic Scholar,  Elicit,  Consensus  or other AI-powered tools for doing  real-time  literature reviews.
• Input/ Output length  - ChatGPT-3.5 allows a maximum token limit of 4096 tokens. According to ChatGPT " On average, a token in English is roughly equivalent to 4 bytes or characters. English words are typically around 5 characters long. This means that, very roughly, you could fit around 800 to 1000 English words within 4096 tokens." 
• Notes : (1) According to ChatGPT, the generated responses are non-deterministic by default. So if you run the searches again and get slightly or very different results than the ones listed on this guide, it's likely due to this factor. (2) While ChatGPT or Bing Chat searches offer conversational interaction, they might not be a substitute for the rigor and reliability of traditional academic database searches.  

KEYWORD GENERATION

Write a literature review abstract based on the following keywords: Keywords: Artificial Intelligence, Employment Trends, Skill Development, Future Workforce

Infant death Neonatal mortality  (referring specifically to deaths within the first 28 days of life) Perinatal mortality  (referring to deaths occurring during the period from 22 completed weeks of gestation to 7 completed days after birth) Early childhood mortality  (referring to deaths occurring within the first five years of life) Child mortality  (referring to deaths occurring before the age of 18) [Press the  Regenerate  button to get more.]

• Example -   RELATED WORDS What are the related words of infant mortality?

Neonatal mortality, Post-neonatal mortality, Stillbirths, Low birth weight, Malnutrition, Infectious diseases, Vaccination, Maternal health,  Access to healthcare, Poverty, Social inequality, Sanitation, Hygiene, Water quality, Childbirth complications, Congenital abnormalities, Birth defects, Maternal age, Under-five mortality, Child mortality, Perinatal mortality, Preterm birth, Low birth weight, Sudden infant death syndrome (SIDS), Maternal mortality, Postnatal care, Malnutrition, Immunization, Access to healthcare, Clean water and sanitation, Congenital anomalies, Infant health disparities, Infant mortality rate (IMR), Infant survival.  

(("Infant Death"[Mesh] OR "Sudden Infant Death Syndrome"[Mesh] OR "Infant Mortality"[Mesh] OR "Perinatal Mortality"[Mesh] OR "Neonatal Mortality"[Mesh] OR "Fetal Mortality"[Mesh] OR "Infant Death" OR "Sudden Infant Death Syndrome" OR "Infant Mortality" OR "Perinatal Mortality" OR "Neonatal Mortality" OR "Fetal Mortality") AND ("Risk Factors"[Mesh] OR "Etiology"[Mesh] OR "Prevention and Control"[Mesh] OR "Interventions"[Mesh] OR "Public Health"[Mesh] OR "Health Disparities"[Mesh] OR "Healthcare Disparities"[Mesh] OR "Poverty"[Mesh] OR "Malnutrition"[Mesh] OR "Infection"[Mesh] OR "Congenital Abnormalities"[Mesh] OR "Prematurity"[Mesh] OR "Low Birth Weight"[Mesh] OR "Maternal Education"[Mesh] OR "Maternal Health"[Mesh] OR "Risk Factors" OR "Etiology" OR "Prevention and Control" OR "Interventions" OR "Public Health" OR "Health Disparities" OR "Healthcare Disparities" OR "Poverty" OR "Malnutrition" OR "Infection" OR "Congenital Abnormalities" OR "Prematurity" OR "Low Birth Weight" OR "Maternal Education" OR "Maternal Health"))

Comments and Suggestions: #1 -This search string may not work on all database platforms. For example,  Medline (OVID)  may not be able to run it,  Medline  &  CINAHL (EBSCO)  can run it without any problems while   PubMed  will run it but it will indicate which MeSH terms are not indexed. See PubMed's search  History and Details  under  Advanced . #2 -This is just one possible search string and you may need to adjust it based on your specific research questions and the databases you are using. #3 - It might be easier if you break the task into 3 parts: ask the chatbot give you the synonyms > ask it to give you the thesauri terms > and ask it to combine them.

RESEARCH QUESTION

Increased risk of sudden infant death syndrome (SIDS) Increased risk of premature birth Increased risk of low birth weight Increased risk of respiratory problems in newborns Increased risk of respiratory problems in infants exposed to secondhand smoke Increased risk of developing asthma and other respiratory illnesses later in life for infants exposed to secondhand smoke Please generate a literature review that addresses the research question: 'What are the effects of social media on the mental health of adolescents? [Author's note : If you ask for the specific studies, ChatGPT may generate   non-existent references or false knowledge. To find out   why Large Language Models   hallucinate, check out this WiKi article:  Hallucination (artificial intelligence)   and this blog post -   A Gentle Introduction to Hallucinations in Large Language Models  by Adrian Tam ]

DATABASE RECOMMENDATIONS  

Heath Science  -  Which databases to use to find articles on the EFFECTS OF MATERNAL SMOKING ON INFANT DEATH? PubMed:  This is a database of biomedical literature from the National Library of Medicine. It includes articles from medical journals and other sources on a wide range of health-related topics. Embase:  This is a database of biomedical literature that covers more international and pharmaceutical content than PubMed. It includes articles from medical journals and other sources on a wide range of health-related topics. Cochrane Library : This is a collection of databases that provides systematic reviews and meta-analyses of healthcare interventions, including interventions related to maternal smoking and infant death. Scopus:  This is a multidisciplinary database that covers research from a wide range of fields, including health sciences. It includes articles from scientific journals, books, and conference proceedings. Web of Science:  This is a database that covers research from a wide range of fields, including health sciences. It includes articles from scientific journals, books, and conference proceedings. CINAHL:  This is a database of nursing and allied health literature, including articles on maternal smoking and infant death. [Note: The response above was copied and pasted directly from ChatGPT. No editing was added.] Education  --  I am doing a  meta-analysis  on k-12 STEM education. What databases should I use? ERIC : This is a database of education research and information, including articles from journals, reports, and conference papers. It is a commonly used database for educational research. Education Source : This is a comprehensive database of education research and information, including articles from journals, books, and conference papers. Web of Science:  This is a multidisciplinary database that covers research from a wide range of fields, including education. It includes articles from scientific journals, books, and conference proceedings. Scopus:  This is another multidisciplinary database that covers research from a wide range of fields, including education. It includes articles from scientific journals, books, and conference proceedings. PsycINFO:  This is a database of psychology research, including articles from journals, books, and dissertations. It can be useful for research on STEM education that includes psychological factors. Cochrane Library : This is a collection of databases that provides systematic reviews and meta-analyses of healthcare interventions including interventions related to education.

Example:   ACT-AS-AN-EXPERT PROMPT Here's an alternative way to consolidate all the questions into a single consultation by using this popular Act-As-An-Expert prompt, aka, system message prompt:  I am going to conduct a systematic review on games and mathematics related to K-12 education  I want you to act as a systematic review expert. I will ask you some questions related to my research topic and you advise me on how to do it. 

GREY LITERATURE  

Please show me all the resources that offer grey literature for conducting research in the field of education     ERIC (Education Resources Information Center): https://eric.ed.gov/     OpenGrey: http://www.opengrey.eu/     British Education Index: https://www.ebscohost.com/academic/british-education-index     WorldCat: https://www.worldcat.org/     ProQuest Dissertations & Theses Global: https://www.proquest.com/products-services/pqdtglobal.html     EThOS (Electronic Theses Online Service): https://ethos.bl.uk/     Institutional Repositories of universities and academic institutions (search for specific institution repositories)     American Educational Research Association (AERA): https://www.aera.net/     National Education Association (NEA): https://www.nea.org/     Association for Supervision and Curriculum Development (ASCD): http://www.ascd.org/     RAND Corporation: https://www.rand.org/     National Center for Education Research (NCER): https://ies.ed.gov/ncer/     National Center for Education Statistics (NCES): https://nces.ed.gov/     European Educational Research Association (EERA): https://www.eera-ecer.de/     UNESCO International Institute for Educational Planning (IIEP): http://www.iiep.unesco.org/     Organization for Economic Cooperation and Development (OECD): https://www.oecd.org/education/     Education Development Center (EDC): https://www.edc.org/     National Institute for Educational Policy Research (NIER) - Japan: https://www.nier.go.jp/     Australian Council for Educational Research (ACER): https://www.acer.org/     Canadian Education Association (CEA): https://www.cea-ace.ca/ My topic is about stem education. Which grey literature databases should I use?    STEM Education Research Centre (SERC) Grey Literature Database    National STEM Education Research Repository (NSERR)    National Science Digital Library (NSDL)    STEM Education Resource Center (SERC) GrayLit Network    STEM Education Grey Literature Library

MORE RESOURCES

• << Previous: AI Lit Searching [beta]
• Next: Organizing the Literature >>
• Last Updated: May 9, 2024 11:44 AM
• URL: https://guides.lib.umich.edu/litreview

How To Use ChatGPT To Write A Literature Review: Prompts & References

In the rapidly evolving world of academic research, the integration of AI tools like ChatGPT has transformed the traditional approach to literature reviews. As a researcher, you should also leverage on this tool to make your research work easier.

In this post, explore how ChatGPT can enhance the literature review process. We’ll explore how specific prompts can effectively guide this advanced AI model to generate insightful content, while ensuring accuracy, relevance, and academic integrity in their scholarly work.

How to Use ChatGPT for Writing a Literature Review

Understanding ChatGPT’s Limitations

While it can efficiently generate content, streamline the research process, and provide a comprehensive understanding of relevant literature, its capabilities are not without constraints. Here are some for you to consider:

Dependence On Pre-Existing Datasets

Since ChatGPT is a language model trained on available data, it may not include the most recent research papers or cutting-edge findings in a specific field. This gap can lead to a lack of current state-of-research insights, particularly crucial in fields like technology and science where advancements happen rapidly.

May Lack Depth And Context

ChatGPT, while able to produce summaries and synthesize information, might not fully grasp the nuanced arguments or complex theories specific to a research topic. This limitation necessitates that researchers critically evaluate and supplement AI-generated text with thorough analysis and insights from recent systematic reviews and primary sources.

Risk Of Plagiarism

Although ChatGPT can generate human-like text, it’s vital to ensure that the content for your literature review is original and properly cited. Relying solely on ChatGPT to write a literature review defeats the purpose of engaging deeply with the material and developing a personal understanding of the literature.

Not A Total Replacement of A Researcher

While ChatGPT can assist non-native English speakers in crafting clear and concise academic writing, it’s not a replacement for the human ability to contextualize and interpret research findings. Researchers must guide the AI model with specific prompts and leverage it as a tool rather than a substitute for comprehensive analysis.

By keeping these limitations in mind, ChatGPT can be a valuable aid in the literature review process, but it should be used judiciously and in conjunction with traditional research methods.

Defining Research Objective

When starting on writing a literature review, the initial step involves using ChatGPT to define your research question or hypothesis.

The AI model’s ability to respond with a summary of the current state of research in your field can provide a comprehensive understanding, especially for systematic reviews or research papers.

For example, by inputting a prompt related to your research topic, ChatGPT can generate human-like text, summarizing prior research and highlighting relevant literature.

One insider tip for effectively using ChatGPT in the literature review process is to leverage its natural language processing capabilities to identify relevant keywords.

These keywords are crucial for non-native English speakers or those new to a research field, as they streamline the search for pertinent academic writing. Additionally, ChatGPT can guide you in understanding the context of your research topic, offering insights that are often challenging to find.

Using AI language models like ChatGPT for generating content for your literature review is efficient and effective, saving valuable time. However, it’s vital to critically evaluate the generated text to ensure it aligns with your research objectives and to avoid plagiarism.

ChatGPT’s ability to synthesize large amounts of information can aid in developing a clear and concise outline, but remember, it’s a guide, not a replacement for human analysis.

Despite these limitations, ChatGPT provides a unique advantage in conducting literature reviews. It can automate mundane tasks, allowing researchers to focus on analysis and critical thinking.

Identifying Keywords and Search Terms

Using ChatGPT to identify relevant keywords related to your research topic can significantly streamline your workflow.

For instance, when you input a summary of your research question into ChatGPT, the AI model can generate a list of pertinent keywords.

These keywords are not just randomly selected; they are based on the vast amounts of information in ChatGPT’s dataset, making them highly relevant and often inclusive of terms that are current in your research field.

An insider tip for leveraging ChatGPT effectively is to guide the AI with clear and concise prompts.

For example, asking ChatGPT to: “summarize key themes in [specific field] research papers from the last five years” can yield a list of keywords and phrases that are not only relevant but also reflective of the current state of research.

This approach is particularly beneficial for conducting systematic reviews or for non-native English speakers who might be unfamiliar with specific academic jargon.

While ChatGPT can provide a comprehensive understanding of relevant literature and help automate the identification of keywords, it’s important to critically evaluate the generated content.

Researchers should use ChatGPT as a tool to augment their research process, not as a replacement for human insight.

It’s crucial to mind the limitations of the AI model and ensure that the keywords identified align with the research topic and objectives.

Creating an Initial Literature Review Outline

The key to using ChatGPT effectively in crafting an initial outline lies in its ability to generate content based on specific prompts.

For instance, a researcher working on organic photovoltaic devices can input a prompt into ChatGPT, such as “Help me create a structure for a literature review on organic photovoltaic devices.”

The AI model, using its comprehensive understanding of the research topic, can then produce a preliminary structure, including sections like:

• Introduction
• Advances in materials and technology, performance, and efficiency.

This generated outline serves as a valuable starting point. It helps in organizing thoughts and determining the key areas that the literature review should cover. I

mportantly, researchers can refine and expand this initial outline as they delve deeper into their topic, ensuring it aligns with their specific research question and the current state of research.

However, while ChatGPT can streamline the review process and save valuable time in creating an initial outline, researchers should not solely rely on it.

The content generated by ChatGPT must be critically evaluated and supplemented with in-depth research. This involves:

• Reading systematic reviews
• Reading research papers, and
• Summarizing relevant literature to ensure the review is comprehensive and up-to-date.

Get ChatGPT To Help You During Research, Using The Right Prompts

The key to effectively using ChatGPT in this process lies in crafting the right prompts, guiding the AI to generate relevant and useful content. 

When initiating a literature review, the prompt should aim for a broad understanding of the research topic. For instance, asking ChatGPT to:

• “Give a brief overview of research done on [topic]”
• “What are some of the recent findings on the [topic] in research?” or 
• “Summarize the historical development of [topic] in academia”

Helps in capturing the general landscape of the field. These prompts assist in identifying key theories, methodologies, and authors within the research area. As the review progresses, more specific prompts are necessary to delve deeper into individual studies. Queries like:

• “Summarize the main arguments and findings of [specific paper]” or
• “What are the strengths and weaknesses of [specific paper]?”

enable ChatGPT to provide detailed insights into particular research papers, aiding in understanding their contribution to the broader field. Comparative prompts are also crucial in synthesizing information across multiple works. Asking ChatGPT to:

• “Compare and contrast the methodologies of [paper 1] and [paper 2]” or
• “How do the findings of [paper 1] and [paper 2] agree or disagree?”

helps in discerning the nuances and disparities in the literature. In the final stages of the literature review, prompts should focus on summarizing findings and identifying emerging trends or gaps. For example:

• “What trends or patterns have emerged from the literature on [topic]?” or
• “What future research directions are suggested by the literature on [topic]?”

We will share more on these ChatGPT prompts in the later part of this post, read on.

Reviewing ChatGPT’s Responses

When using ChatGPT to write a literature review, it’s crucial to critically evaluate its responses.

Firstly, researchers should cross-reference the information provided by ChatGPT with actual research papers.

This step ensures the accuracy of the data and helps in identifying any discrepancies or outdated information, given that ChatGPT’s dataset may not include the most recent studies.

Another essential aspect is assessing the coherence and depth of the AI-generated text. ChatGPT can summarize and synthesize information efficiently, but it might not capture the nuances of complex theories or research arguments.

Researchers should ensure that the content aligns with their research question and systematically reviews the topic comprehensively. This is where a researcher’s value comes in.

Additionally, verifying the originality of the content is vital to avoid plagiarism. While ChatGPT can generate human-like text, researchers must ensure that the AI-generated content is used as a guide rather than a verbatim source. 

Proper citations and references are essential to maintain the integrity of the literature review. Avoid torpedoing your own research by committing plagiarism.

Ensuring Coherence and Flow

One of the challenges when using such advanced AI language models is ensuring the coherence and flow of the final document. This aspect is crucial as it determines the readability and academic rigor of the literature review.

ChatGPT can generate vast amounts of content on a wide range of topics, responding efficiently to prompts and synthesizing information from its extensive dataset.

However, the content generated by ChatGPT, while informative, might not always align seamlessly with the specific research question or maintain a consistent narrative flow.

To tackle this, researchers need to take an active role in guiding ChatGPT and subsequently refining its output.

A practical approach is to use ChatGPT as a starting point, leveraging its ability to quickly provide summaries, synthesize relevant literature, and identify key references and keywords related to the research topic. For example, prompts like:

• “Summarize the current research on [topic]” or
• “Identify key debates in [topic]”

Can yield valuable initial insights.

Once this foundational information is obtained, the crucial task is to carefully review and edit the AI-generated content.

This involves connecting the dots between different sections, ensuring that each part contributes meaningfully to addressing the research question, and refining the language to maintain academic standards.

It’s also essential to check for and avoid plagiarism, ensuring that all sources are correctly cited.

In addition, considering the vast amounts of information ChatGPT can access, it’s vital to verify the accuracy and relevance of the content.

Researchers should cross-reference AI-generated summaries with actual research papers, especially the most recent ones, as ChatGPT’s dataset may not include the latest studies.

Editing and Proofreading

Now that your literature review is mostly written out, now focus on the editing and proofreading. The content generated by ChatGPT needs to be meticulously reviewed and edited. Here are the steps:

• Verifying the accuracy of the information. Researchers must cross-check the AI-generated content against actual research papers and systematic reviews. This ensures that the latest studies are accurately represented.
• Improve coherence and flow. Researchers should restructure sentences, ensure logical progression of ideas, and maintain a consistent academic tone throughout the document.
• Checking for plagiarism. Despite ChatGPT’s ability to generate human-like text, researchers must ensure that all sources are correctly cited and that the review does not inadvertently replicate existing material.
• Check Grammar and Spelling: Editing should encompass grammar checks, vocabulary refinement, and ensuring that the language used is appropriate for an academic audience.
• Update Citation: Review citation, or reference list to ensure everything is cited correctly, and the citation list is written out to your required standard, be it MLA, Chicago, or APA.

What ChatGPT Prompts To Use When Writing A Literature Review?

There are many ways to use ChatGPT to write literature review, usually by using the right prompts. Here’s how specific types of prompts can be effectively employed, with multiple examples for each category:

• “Provide a comprehensive overview of the latest research on [topic].”
• “Summarize the current understanding and key findings in the field of [topic].”
• “Detail the dominant theoretical frameworks currently used in [topic].”
• “Describe the evolution of theoretical approaches in [topic] over the past decade.”
• “Identify and discuss the major debates or controversies in [topic].”
• “What are the conflicting viewpoints or schools of thought in [topic]?”
• “List the leading researchers in [topic] and summarize their key contributions.”
• “Who are the emerging authors in [topic], and what unique perspectives do they offer?”
• “Explain the most common research methodologies used in studies about [topic].”
• “How have the methodologies in [topic] research evolved recently?”
• “Trace the historical development and major milestones in [topic].”
• “Provide a timeline of the key discoveries and shifts in understanding in [topic].”
• “What significant paradigm shifts have occurred in [topic] in the last twenty years?”
• “How has the focus of research in [topic] changed over time?”
• “Analyze the methodology and conclusions of [specific paper].”
• “Discuss the impact and reception of [specific paper] in the field of [topic].”
• “Compare the results and methodologies of [paper 1] and [paper 2] in [topic].”
• “How do [paper 1] and [paper 2] differ in their approach to [topic]?”
• “Based on current literature, what are the suggested future research directions in [topic]?”
• “Identify gaps in the literature of [topic] that could be explored in future studies.”

By using these types of prompts, researchers can guide ChatGPT to produce content that is not only relevant to their literature review but also rich in detail and scope.

Wrapping Up: Use Other AI Tools Too, Not Just ChatGPT

In conclusion, while ChatGPT serves as a powerful ally in the literature review process, it’s important to recognize it as one of many AI tools available to researchers as well. Diversifying your AI toolkit can enhance the depth and breadth of your review, offering varied perspectives and methodologies.

As AI continues to evolve, embracing a range of these tools can lead to more comprehensive, nuanced, and innovative academic writing, expanding the horizons of research and scholarly exploration beyond what we currently envision.

Dr Andrew Stapleton has a Masters and PhD in Chemistry from the UK and Australia. He has many years of research experience and has worked as a Postdoctoral Fellow and Associate at a number of Universities. Although having secured funding for his own research, he left academia to help others with his YouTube channel all about the inner workings of academia and how to make it work for you.

Thank you for visiting Academia Insider.

We are here to help you navigate Academia as painlessly as possible. We are supported by our readers and by visiting you are helping us earn a small amount through ads and affiliate revenue - Thank you!

2024 © Academia Insider

Advertisement

Interacting with educational chatbots: A systematic review

• Open access
• Published: 09 July 2022
• Volume 28 , pages 973–1018, ( 2023 )

Cite this article

You have full access to this open access article

• Mohammad Amin Kuhail 1 ,
• Nazik Alturki 2 ,
• Salwa Alramlawi 3 &
• Kholood Alhejori 4  

55k Accesses

20 Altmetric

Explore all metrics

Chatbots hold the promise of revolutionizing education by engaging learners, personalizing learning activities, supporting educators, and developing deep insight into learners’ behavior. However, there is a lack of studies that analyze the recent evidence-based chatbot-learner interaction design techniques applied in education. This study presents a systematic review of 36 papers to understand, compare, and reflect on recent attempts to utilize chatbots in education using seven dimensions: educational field, platform, design principles, the role of chatbots, interaction styles, evidence, and limitations. The results show that the chatbots were mainly designed on a web platform to teach computer science, language, general education, and a few other fields such as engineering and mathematics. Further, more than half of the chatbots were used as teaching agents, while more than a third were peer agents. Most of the chatbots used a predetermined conversational path, and more than a quarter utilized a personalized learning approach that catered to students’ learning needs, while other chatbots used experiential and collaborative learning besides other design principles. Moreover, more than a third of the chatbots were evaluated with experiments, and the results primarily point to improved learning and subjective satisfaction. Challenges and limitations include inadequate or insufficient dataset training and a lack of reliance on usability heuristics. Future studies should explore the effect of chatbot personality and localization on subjective satisfaction and learning effectiveness.

Similar content being viewed by others

The effectiveness of using chatbot-based environment on learning process, students’ performances and perceptions: A mixed exploratory study

Conversational agent-based guidance: examining the effect of chatbot usage frequency and satisfaction on visual design self-efficacy, engagement, satisfaction, and learner autonomy

Integrating chatbots in education: insights from the Chatbot-Human Interaction Satisfaction Model (CHISM)

Avoid common mistakes on your manuscript.

1 Introduction

Chatbots, also known as conversational agents, enable the interaction of humans with computers through natural language, by applying the technology of natural language processing (NLP) (Bradeško & Mladenić, 2012 ). Due to their ability to emulate human conversations and thus automate services and reduce effort, chatbots are increasingly becoming popular in several domains, including healthcare (Oh et al., 2017 ), consumer services (Xu et al., 2017 ), education (Anghelescu & Nicolaescu, 2018 ), and academic advising (Alkhoori et al., 2020 ). In fact, the size of the chatbot market worldwide is expected to be 1.23 billion dollars in 2025 (Kaczorowska-Spychalska, 2019 ). In the US alone, the chatbot industry was valued at 113 million US dollars and is expected to reach 994.5 million US dollars in 2024 Footnote 1 .

The adoption of educational chatbots is on the rise due to their ability to provide a cost-effective method to engage students and provide a personalized learning experience (Benotti et al., 2018 ). Chatbot adoption is especially crucial in online classes that include many students where individual support from educators to students is challenging (Winkler & Söllner, 2018 ). Chatbots can facilitate learning within the educational context, for instance by instantaneously providing students with course content (Cunningham-Nelson et al., 2019 ), assignments (Ismail & Ade-Ibijola, 2019 ), rehearsal questions (Sinha et al., 2020 ), and study resources (Mabunda, 2020 ). Moreover, chatbots may interact with students individually (Hobert & Meyer von Wolff, 2019 ) or support collaborative learning activities (Chaudhuri et al., 2009 ; Tegos et al., 2014 ; Kumar & Rose, 2010 ; Stahl, 2006 ; Walker et al., 2011 ). Chatbot interaction is achieved by applying text, speech, graphics, haptics, gestures, and other modes of communication to assist learners in performing educational tasks.

Existing literature review studies attempted to summarize current efforts to apply chatbot technology in education. For example, Winkler and Söllner ( 2018 ) focused on chatbots used for improving learning outcomes. On the other hand, Cunningham-Nelson et al. ( 2019 ) discussed how chatbots could be applied to enhance the student’s learning experience. The study by Pérez et al. ( 2020 ) reviewed the existing types of educational chatbots and the learning results expected from them. Smutny and Schreiberova ( 2020 ) examined chatbots as a learning aid for Facebook Messenger. Thomas ( 2020 ) discussed the benefits of educational chatbots for learners and educators, showing that the chatbots are successful educational tools, and their benefits outweigh the shortcomings and offer a more effective educational experience. Okonkwo and Ade-Ibijola ( 2021 ) analyzed the main benefits and challenges of implementing chatbots in an educational setting.

The existing review studies contributed to the literature, albeit their main emphasis was using chatbots for improving the learning experience and outcomes (Winkler & Söllner, 2018 ; Cunningham-Nelson et al., 2019 ; Smutny & Schreiberova, 2020 ; Thomas, 2020 ), identifying the types of educational chatbots (Pérez et al., 2020 ), and determining the benefits, and challenges of implementing educational chatbots (Okonkwo & Ade-Ibijola, 2021 ). Nonetheless, the existing review studies have not concentrated on the chatbot interaction type and style, the principles used to design the chatbots, and the evidence for using chatbots in an educational setting.

Given the magnitude of research on educational chatbots, there is a need for a systematic literature review that sheds light on several vital dimensions: field of application, platform, role in education, interaction style, design principles, empirical evidence, and limitations.

By systematically analyzing 36 articles presenting educational chatbots representing various interaction styles and design approaches, this study contributes: (1) an in-depth analysis of the learner-chatbot interaction approaches and styles currently used to improve the learning process, (2) a characterization of the design principles used for the development of educational chatbots, (3) an in-depth explanation of the empirical evidence used to back up the validity of the chatbots, and (4) the discussion of current challenges and future research directions specific to educational chatbots. This study will help the education and human-computer interaction community aiming at designing and evaluating educational chatbots. Potential future chatbots might adopt some ideas from the chatbots surveyed in this study while addressing the discussed challenges and considering the suggested future research directions. This study is structured as follows: In Section  2 , we present background information about chatbots, while Section  3 discusses the related work. Section  4 explains the applied methodology, while Section  5 presents the study’s findings. Section  6 presents the discussion and future research directions. Finally, we present the conclusion and the study’s limitations in Section  7 .

2 Background

Chatbots have existed for more than half a century. Prominent examples include ELIZA, ALICE, and SmarterChild. ELIZA, the first chatbot, was developed by Weizenbaum ( 1966 ). The chatbot used pattern matching to emulate a psychotherapist conversing with a human patient. ALICE was a chatbot developed in the mid-1990s. It used Artificial Intelligence Markup Language (AIML) to identify an accurate response to user input using knowledge records (AbuShawar and Atwell, 2015 ). Another example is Smart Child (Chukhno et al., 2019 ), which preceded today’s modern virtual chatbot-based assistants such as Alexa Footnote 2 and Siri Footnote 3 , which are available on messaging applications with the ability to emulate conversations with quick data access to services.

Chatbots have been utilized in education as conversational pedagogical agents since the early 1970s (Laurillard, 2013 ). Pedagogical agents, also known as intelligent tutoring systems, are virtual characters that guide users in learning environments (Seel, 2011 ). Conversational Pedagogical Agents (CPA) are a subgroup of pedagogical agents. They are characterized by engaging learners in a dialog-based conversation using AI (Gulz et al., 2011 ). The design of CPAs must consider social, emotional, cognitive, and pedagogical aspects (Gulz et al., 2011 ; King, 2002 ).

A conversational agent can hold a discussion with students in a variety of ways, ranging from spoken (Wik & Hjalmarsson, 2009 ) to text-based (Chaudhuri et al., 2009 ) to nonverbal (Wik & Hjalmarsson, 2009 ; Ruttkay & Pelachaud, 2006 ). Similarly, the agent’s visual appearance can be human-like or cartoonish, static or animated, two-dimensional or three-dimensional (Dehn & Van Mulken, 2000 ). Conversational agents have been developed over the last decade to serve a variety of pedagogical roles, such as tutors, coaches, and learning companions (Haake & Gulz, 2009 ). Furthermore, conversational agents have been used to meet a variety of educational needs such as question-answering (Feng et al., 2006 ), tutoring (Heffernan & Croteau, 2004 ; VanLehn et al., 2007 ), and language learning (Heffernan & Croteau, 2004 ; VanLehn et al., 2007 ).

When interacting with students, chatbots have taken various roles such as teaching agents, peer agents, teachable agents, and motivational agents (Chhibber & Law, 2019 ; Baylor, 2011 ; Kerry et al., 2008 ). Teaching agents play the role of human teachers and can present instructions, illustrate examples, ask questions (Wambsganss et al., 2020 ), and provide immediate feedback (Kulik & Fletcher, 2016 ). On the other hand, peer agents serve as learning mates for students to encourage peer-to-peer interactions. The agent of this approach is less knowledgeable than the teaching agent. Nevertheless, peer agents can still guide the students along a learning path. Students typically initiate the conversation with peer agents to look up certain definitions or ask for an explanation of a specific topic. Peer agents can also scaffold an educational conversation with other human peers.

Students can teach teachable agents to facilitate gradual learning. In this approach, the agent acts as a novice and asks students to guide them along a learning route. Rather than directly contributing to the learning process, motivational agents serve as companions to students and encourage positive behavior and learning (Baylor, 2011 ). An agent could serve as a teaching or peer agent and a motivational one.

Concerning their interaction style, the conversation with chatbots can be chatbot or user-driven (Følstad et al., 2018 ). Chatbot-driven conversations are scripted and best represented as linear flows with a limited number of branches that rely upon acceptable user answers (Budiu, 2018 ). Such chatbots are typically programmed with if-else rules. When the user provides answers compatible with the flow, the interaction feels smooth. However, problems occur when users deviate from the scripted flow.

User-driven conversations are powered by AI and thus allow for a flexible dialogue as the user chooses the types of questions they ask and thus can deviate from the chatbot’s script. There are one-way and two-way user-driven chatbots. One-way user-driven chatbots use machine learning to understand what the user is saying (Dutta, 2017 ), and the responses are selected from a set of premade answers. In contrast, two-way user-driven chatbots build accurate answers word by word to users (Winkler & Söllner, 2018 ). Such chatbots can learn from previous user input in similar contexts (De Angeli & Brahnam, 2008 ).

In terms of the medium of interaction, chatbots can be text-based, voice-based, and embodied. Text-based agents allow users to interact by simply typing via a keyboard, whereas voice-based agents allow talking via a mic. Voice-based chatbots are more accessible to older adults and some special-need people (Brewer et al., 2018 ). An embodied chatbot has a physical body, usually in the form of a human, or a cartoon animal (Serenko et al., 2007 ), allowing them to exhibit facial expressions and emotions.

Concerning the platform, chatbots can be deployed via messaging apps such as Telegram, Facebook Messenger, and Slack (Car et al., 2020 ), standalone web or phone applications, or integrated into smart devices such as television sets.

3 Related work

Recently several studies reviewed chatbots in education. The studies examined various areas of interest concerning educational chatbots, such as the field of application (Smutny & Schreiberova, 2020 ; Wollny et al., 2021 ; Hwang & Chang, 2021 ), objectives and learning experience (Winkler & Söllner, 2018 ; Cunningham-Nelson et al., 2019 ; Pérez et al., 2020 ; Wollny et al., 2021 ; Hwang & Chang, 2021 ), how chatbots are applied (Winkler & Söllner, 2018 ; Cunningham-Nelson et al., 2019 ; Wollny et al., 2021 ), design approaches (Winkler & Söllner, 2018 ; Martha & Santoso, 2019 ; Hwang & Chang, 2021 ), the technology used (Pérez et al., 2020 ), evaluation methods used (Pérez et al., 2020 ; Hwang & Chang, 2021 ; Hobert & Meyer von Wolff, 2019 ), and challenges in using educational chatbots (Okonkwo & Ade-Ibijola, 2021 ). Table  1 summarizes the areas that the studies explored.

Winkler and Söllner ( 2018 ) reviewed 80 articles to analyze recent trends in educational chatbots. The authors found that chatbots are used for health and well-being advocacy, language learning, and self-advocacy. Chatbots are either flow-based or powered by AI, concerning approaches to their designs.

Several studies have found that educational chatbots improve students’ learning experience. For instance, Okonkwo and Ade-Ibijola ( 2021 ) found out that chatbots motivate students, keep them engaged, and grant them immediate assistance, particularly online. Additionally, Wollny et al. ( 2021 ) argued that educational chatbots make education more available and easily accessible.

Concerning how they are applied, Cunningham-Nelson et al. ( 2019 ) identified two main applications: answering frequently-asked questions (FAQ) and performing short quizzes, while Wollny et al. ( 2021 ) listed three other applications, including scaffolding, activity recommendations, and informing them about activities.

In terms of the design of educational chatbots, Martha and Santoso ( 2019 ) found out that the role and appearance of the chatbot are crucial elements in designing the educational chatbots, while Winkler and Söllner ( 2018 ) identified various types of approaches to designing educational chatbots such as flow and AI-based, in addition to chatbots with speech recognition capabilities.

Pérez et al. ( 2020 ) identified various technologies used to implement chatbots such as Dialogflow Footnote 4 , FreeLing (Padró and Stanilovsky, 2012 ), and ChatFuel Footnote 5 . The study investigated the effect of the technologies used on performance and quality of chatbots.

Hobert and Meyer von Wolff ( 2019 ), Pérez et al. ( 2020 ), and Hwang and Chang ( 2021 ) examined the evaluation methods used to assess the effectiveness of educational chatbots. The authors identified that several evaluation methods such as surveys, experiments, and evaluation studies measure acceptance, motivation, and usability.

Okonkwo and Ade-Ibijola ( 2021 ) discussed challenges and limitations of chatbots including ethical, programming, and maintenance issues.

Although these review studies have contributed to the literature, they primarily focused on chatbots as a learning aid and thus how they can be used to improve educational objectives. Table  2 compares this study and the related studies in terms of the seven dimensions that this study focuses on: field of application, platform, educational role, interaction style, design principles, evaluation, and limitations.

Only four studies (Hwang & Chang, 2021 ; Wollny et al., 2021 ; Smutny & Schreiberova, 2020 ; Winkler & Söllner, 2018 ) examined the field of application. None of the studies discussed the platforms on which the chatbots run, while only one study (Wollny et al., 2021 ) analyzed the educational roles the chatbots are playing. The study used “teaching,” “assisting,” and “mentoring” as categories for educational roles. This study, however, uses different classifications (e.g., “teaching agent”, “peer agent”, “motivational agent”) supported by the literature in Chhibber and Law ( 2019 ), Baylor ( 2011 ), and Kerlyl et al. ( 2006 ). Other studies such as (Okonkwo and Ade-Ibijola, 2021 ; Pérez et al., 2020 ) partially covered this dimension by mentioning that chatbots can be teaching or service-oriented.

Only two articles partially addressed the interaction styles of chatbots. For instance, Winkler and Söllner ( 2018 ) classified the chatbots as flow or AI-based, while Cunningham-Nelson et al. ( 2019 ) categorized the chatbots as machine-learning-based or dataset-based. In this study, we carefully look at the interaction style in terms of who is in control of the conversation, i.e., the chatbot or the user. As such, we classify the interactions as either chatbot or user-driven.

Only a few studies partially tackled the principles guiding the design of the chatbots. For instance, Martha and Santoso ( 2019 ) discussed one aspect of the design (the chatbot’s visual appearance). This study focuses on the conceptual principles that led to the chatbot’s design.

In terms of the evaluation methods used to establish the validity of the articles, two related studies (Pérez et al., 2020 ; Smutny & Schreiberova, 2020 ) discussed the evaluation methods in some detail. However, this study contributes more comprehensive evaluation details such as the number of participants, statistical values, findings, etc.

Regarding limitations, Pérez et al. ( 2020 ) examined the technological limitations that have an effect on the quality of the educational chatbots, while Okonkwo and Ade-Ibijola ( 2021 ) presented some challenges and limitations facing educational chatbots such as ethical, technical, and maintenance matters. While the identified limitations are relevant, this study identifies limitations from other perspectives such as the design of the chatbots and the student experience with the educational chatbots. To sum up, Table  2 shows some gaps that this study aims at bridging to reflect on educational chatbots in the literature.

4 Methodology

The literature related to chatbots in education was analyzed, providing a background for new approaches and methods, and identifying directions for further research. This study follows the guidelines described by Keele et al. ( 2007 ). The process includes these main steps: (1) defining the review protocol, including the research questions, how to answer them, search strategy, and inclusion and exclusion criteria. (2) running the study by selecting the articles, assessing their quality, and synthesizing the results. (3) reporting the findings.

4.1 Research questions

Based on the shortcomings of the existing related literature review studies, we formulated seven main research questions:

- In what fields are the educational chatbots used?

- What platforms do the educational chatbots operate on?

- What role do the educational chatbots play when interacting with students?

- What are the interaction styles supported by the educational chatbots?

- What are the principles used to guide the design of the educational chatbots?

- What empirical evidence exists to support the validity of the educational chatbots?

- What are the challenges of applying and using the chatbots in the classroom?

The first question identifies the fields of the proposed educational chatbots, while the second question presents the platforms the chatbots operate on, such as web or phone-based platforms. The third question discusses the roles chatbots play when interacting with students. For instance, chatbots could be used as teaching or peer agents. The fourth question sheds light on the interaction styles used in the chatbots, such as flow-based or AI-powered. The fifth question addresses the principles used to design the proposed chatbots. Examples of such principles could be collaborative and personalized learning. The sixth question focuses on the evaluation methods used to prove the effectiveness of the proposed chatbots. Finally, the seventh question discusses the challenges and limitations of the works behind the proposed chatbots and potential solutions to such challenges.

4.2 Search process

The search process was conducted during the period (2011 - 2021) in the following databases: ACM Digital Library, Scopus, IEEE Xplore, and SpringerLink. We analyzed our research questions, objectives, and related existing literature review studies to identify keywords for the search string of this study. Subsequently, we executed and refined the keywords and the search string iteratively until we arrived at promising results. We used these search keywords: “Chatbot” and “Education.” Correlated keywords for “Chatbot” are “Conversational Agent” and “Pedagogical Agent.” Further, correlated keywords for “Education” are ”Learning,” “Learner,” “Teaching,” “Teacher,” and “Student.”

The search string was defined using the Boolean operators as follows:

(‘Chatbot’ OR ‘Conversational Agent’ OR ‘Pedagogical Agent’) AND (‘Education’ OR ‘Learning’ OR ‘Learner’ OR ‘Teaching’ OR ‘Teacher’ OR ‘Student’)

According to their relevance to our research questions, we evaluated the found articles using the inclusion and exclusion criteria provided in Table  3 . The inclusion and exclusion criteria allowed us to reduce the number of articles unrelated to our research questions. Further, we excluded tutorials, technical reports, posters, and Ph.D. thesis since they are not peer-reviewed.

After defining the criteria, our search query was performed in the selected databases to begin the inclusion and exclusion process. Initially, the total of studies resulting from the databases was 1208 studies. The metadata of the studies containing; title, abstract, type of article (conference, journal, short paper), language, and keywords were extracted in a file format (e.g., bib file format). Subsequently, it was imported into the Rayyan tool Footnote 6 , which allowed for reviewing, including, excluding, and filtering the articles collaboratively by the authors.

The four authors were involved in the process of selecting the articles. To maintain consistency amongst our decisions and inter-rater reliability, the authors worked in two pairs allowing each author to cross-check the selection and elimination of the author they were paired with. The process of selecting the articles was carried out in these stages:

Reading the articles’ metadata and applying the inclusion criteria of IC-1 and the exclusion criteria of EC-1. As a result, the number of studies was reduced to 1101.

As a first-round, we applied the inclusion criterion IC-2 by reading the studies’ title, abstract, and keywords. Additionally, the EC-2 exclusion criterion was applied in the same stage. As a result, only 197 studies remained.

In this stage, we eliminated the articles that were not relevant to any of our research questions and applied the EC-3 criteria. As a result, the articles were reduced to 71 papers.

Finally, we carefully read the entire content of the articles having in mind IC-3. Additionally, we excluded studies that had no or little empirical evidence for their effectiveness of the educational chatbot (EC-4 criterion). As a result, the articles were reduced to 36 papers.

Figure  1 . shows the flowchart of the selecting processes, in which the final stage of the selection resulted in 36 papers.

Flowchart of the process of the selection of the studies

Figure  2 shows the number and types of articles plotted against time. 63.88% (23) of the selected articles are conference papers, while 36.11% (13) were published in journals. Most conference papers were published after 2017. Interestingly, 38.46% (5) of the journal articles were published recently in 2020. Concerning the publication venues, two journal articles were published in the Journal of IEEE Transactions on Learning Technologies (TLT), which covers various topics such as innovative online learning systems, intelligent tutors, educational software applications and games, and simulation systems for education. Intriguingly, one article was published in Computers in Human Behavior journal. The remaining journal articles were published in several venues such as IEEE Transactions on Affective Computing, Journal of Educational Psychology, International Journal of Human-Computer Studies, ACM Transactions on Interactive Intelligent System. Most of these journals are ranked Q1 or Q2 according to Scimago Journal and Country Rank Footnote 7 .

A timeline of the selected studies

Figure  3 . shows the geographical mapping of the selected articles. The total sum of the articles per country in Fig.  3 is more than 36 (the number of selected articles) as the authors of a single article could work in institutions located in different countries. The vast majority of selected articles were written or co-written by researchers from American universities. However, the research that emerged from all European universities combined was the highest in the number of articles (19 articles). Asian universities have contributed 10 articles, while American universities contributed 9 articles. Further, South American universities have published 5 articles. Finally, universities from Africa and Australia contributed 4 articles (2 articles each).

A geographical mapping of the selected articles

5.1 RQ1: What fields are the educational chatbots used in?

Recently, chatbots have been utilized in various fields (Ramesh et al., 2017 ). Most importantly, chatbots played a critical role in the education field, in which most researchers (12 articles; 33.33%) developed chatbots used to teach computer science topics (Fig.  4 ). For example, some chatbots were used as tutors for teaching programming languages such as Java (Coronado et al., 2018 ; Daud et al., 2020 ) and Python (Winkler et al., 2020 ), while other researchers proposed educational chatbots for computer networks (Clarizia et al., 2018 ; Lee et al., 2020 ), databases (Latham et al., 2011 ; Ondáš et al., 2019 ), and compilers (Griol et al., 2011 ).

The fields of the chatbots in the selected articles

Table  4 . shows that ten (27.77%) articles presented general-purpose educational chatbots that were used in various educational contexts such as online courses (Song et al., 2017 ; Benedetto & Cremonesi, 2019 ; Tegos et al., 2020 ). The approach authors use often relies on a general knowledge base not tied to a specific field.

In comparison, chatbots used to teach languages received less attention from the community (6 articles; 16.66%;). Interestingly, researchers used a variety of interactive media such as voice (Ayedoun et al., 2017 ; Ruan et al., 2021 ), video (Griol et al., 2014 ), and speech recognition (Ayedoun et al., 2017 ; Ruan et al., 2019 ).

A few other subjects were targeted by the educational chatbots, such as engineering (Mendez et al., 2020 ), religious education (Alobaidi et al., 2013 ), psychology (Hayashi, 2013 ), and mathematics (Rodrigo et al., 2012 ).

5.2 RQ2: What platforms do the proposed chatbots operate on?

Table  5 . shows an overview of the platforms the educational chatbots operate on. Most researchers (25 articles; 69.44%) developed chatbots that operate on the web (Fig.  5 ). The web-based chatbots were created for various educational purposes. For example, KEMTbot (Ondáš et al., 2019 ) is a chatbot system that provides information about the department, its staff, and their offices. Other chatbots acted as intelligent tutoring systems, such as Oscar (Latham et al., 2011 ), used for teaching computer science topics. Moreover, other web-based chatbots such as EnglishBot (Ruan et al., 2021 ) help students learn a foreign language.

The platforms of the chatbots in the selected articles

Six (16.66%) articles presented educational chatbots that exclusively operate on a mobile platform (e.g., phone, tablet). The articles were published recently in 2019 and 2020. The mobile-based chatbots were used for various purposes. Examples include Rexy (Benedetto & Cremonesi, 2019 ), which helps students enroll in courses, shows exam results, and gives feedback. Another example is the E-Java Chatbot (Daud et al., 2020 ), a virtual tutor that teaches the Java programming language.

Five articles (13.88%) presented desktop-based chatbots, which were utilized for various purposes. For example, one chatbot focused on the students’ learning styles and personality features (Redondo-Hernández & Pérez-Marín, 2011 ). As another example, the SimStudent chatbot is a teachable agent that students can teach (Matsuda et al., 2013 ).

In general, most desktop-based chatbots were built in or before 2013, probably because desktop-based systems are cumbersome to modern users as they must be downloaded and installed, need frequent updates, and are dependent on operating systems. Unsurprisingly, most chatbots were web-based, probably because the web-based applications are operating system independent, do not require downloading, installing, or updating. Mobile-based chatbots are on the rise. This can be explained by users increasingly desiring mobile applications. According to an App Annie report, users spent 120 billion dollars on application stores Footnote 8 .

5.3 RQ3 - What role do the educational chatbots play when interacting with students?

Chatbots have been found to play various roles in educational contexts, which can be divided into four roles (teaching agents, peer agents, teachable agents, and peer agents), with varying degrees of success (Table  6 , Fig.  6 ). Exceptionally, a chatbot found in (D’mello & Graesser, 2013 ) is both a teaching and motivational agent.

By far, the majority (20; 55.55%) of the presented chatbots play the role of a teaching agent, while 13 studies (36.11%) discussed chatbots that are peer agents. Only two studies used chatbots as teachable agents, and two studies used them as motivational agents.

Teaching agents

The teaching agents presented in the different studies used various approaches. For instance, some teaching agents recommended tutorials to students based upon learning styles (Redondo-Hernández & Pérez-Marín, 2011 ), students’ historical learning (Coronado et al., 2018 ), and pattern matching (Ondáš et al., 2019 ). In some cases, the teaching agent started the conversation by asking the students to watch educational videos (Qin et al., 2020 ) followed by a discussion about the videos. In other cases, the teaching agent started the conversation by asking students to reflect on past learning (Song et al., 2017 ). Other studies discussed a scenario-based approach to teaching with teaching agents (Latham et al., 2011 ; D’mello & Graesser, 2013 ). The teaching agent simply mimics a tutor by presenting scenarios to be discussed with students. In other studies, the teaching agent emulates a teacher conducting a formative assessment by evaluating students’ knowledge with multiple-choice questions (Rodrigo et al., 2012 ; Griol et al., 2014 ; Mellado-Silva et al., 2020 ; Wambsganss et al., 2020 ).

Moreover, it has been found that teaching agents use various techniques to engage students. For instance, some teaching agents engage students with a discussion in a storytelling style (Alobaidi et al., 2013 ; Ruan et al., 2019 ), whereas other chatbots engage students with effective channeling, with empathetic phrases as “uha” to show interest (Ayedoun et al., 2017 ). Other teaching agents provide adaptive feedback (Wambsganss et al., 2021 ).

Peer agents

Most peer agent chatbots allowed students to ask for specific help on demand. For instance, the chatbots discussed in (Clarizia et al., 2018 ; Lee et al., 2020 ) allowed students to look up specific terms or concepts, while the peer agents in (Verleger & Pembridge, 2018 ; da Silva Oliveira et al., 2019 ; Mendez et al., 2020 ) were based on a Question and Answer (Q&A) database, and as such answered specific questions. Other peer agents provide more advanced assistance. For example, students may ask the peer agent in (Janati et al., 2020 ) how to use a particular technology (e.g., using maps in Oracle Analytics), while the peer agent described in (Tegos et al., 2015 ; Tegos et al., 2020 ; Hayashi, 2013 ) scaffolded a group discussion. Interestingly, the only peer agent that allowed for a free-style conversation was the one described in (Fryer et al., 2017 ), which could be helpful in the context of learning a language.

Teachable agents

Only two articles discussed teachable agent chatbots. In general, the followed approach with these chatbots is asking the students questions to teach students certain content. For example, the chatbot discussed in (Matsuda et al., 2013 ) presents a mathematical equation and then asks the student of each required step to gradually solve the equation, while in the work presented in (Law et al., 2020 ), students individually or in a group teach a classification task to chatbots in several topics.

Motivational agents

Two studies presented chatbots as motivational agent-based chatbots. One of them presented in (D’mello & Graesser, 2013 ) asks the students a question, then waits for the student to write an answer. Then the motivational agent reacts to the answer with varying emotions, including empathy and approval, to motivate students. Similarly, the chatbot in (Schouten et al., 2017 ) shows various reactionary emotions and motivates students with encouraging phrases such as “you have already achieved a lot today” .

The roles of the chatbots in the selected articles

5.4 RQ4 – What are the interaction styles supported by the educational chatbots?

As shown in Table  7 and Fig.  7 , most of the articles (88.88%) used the chatbot-driven interaction style where the chatbot controls the conversation. 52.77% of the articles used flow-based chatbots where the user had to follow a specific learning path predetermined by the chatbot. Notable examples are explained in (Rodrigo et al., 2012 ; Griol et al., 2014 ), where the authors presented a chatbot that asks students questions and provides them with options to choose from. Other authors, such as (Daud et al., 2020 ), used a slightly different approach where the chatbot guides the learners to select the topic they would like to learn. Subsequently, the assessment of specific topics is presented where the user is expected to fill out values, and the chatbot responds with feedback. The level of the assessment becomes more challenging as the student makes progress. A slightly different interaction is explained in (Winkler et al., 2020 ), where the chatbot challenges the students with a question. If they answer incorrectly, they are explained why the answer is incorrect and then get asked a scaffolding question.

The remaining articles (13 articles; 36.11%) present chatbot-driven chatbots that used an intent-based approach. The idea is the chatbot matches what the user says with a premade response. The matching could be done using pattern matching as discussed in (Benotti et al., 2017 ; Clarizia et al., 2018 ) or simply by relying on a specific conversational tool such as Dialogflow Footnote 9 as in (Mendez et al., 2020 ; Lee et al., 2020 ; Ondáš et al., 2019 ).

Only four (11.11%) articles used chatbots that engage in user-driven conversations where the user controls the conversation and the chatbot does not have a premade response. For example, the authors in (Fryer et al., 2017 ) used Cleverbot, a chatbot designed to learn from its past conversations with humans. The authors used Cleverbot for foreign language education. User-driven chatbots fit language learning as students may benefit from an unguided conversation. The authors in (Ruan et al., 2021 ) used a similar approach where students freely speak a foreign language. The chatbot assesses the quality of the transcribed text and provides constructive feedback. In comparison, the authors in (Tegos et al., 2020 ) rely on a slightly different approach where the students chat together about a specific programming concept. The chatbot intervenes to evoke curiosity or draw students’ attention to an interesting, related idea.

The interaction styles of the chatbots in the selected articles

5.5 RQ5 – What are the principles used to guide the design of the educational chatbots?

Various design principles, including pedagogical ones, have been used in the selected studies (Table  8 , Fig.  8 ). We discuss examples of how each of the principles was applied.

The principles used to design the chatbots

Personalized Learning The ability to tailor chatbots to the individual user may help meet students’ needs (Clarizia et al., 2018 ). Many studies claim that students learn better when the chatbot is represented by a personalized method rather than a non-personalized one (Kester et al., 2005 ). From our selected studies, ten (27.77%) studies have applied personalized learning principles. For instance, the study in (Coronado et al., 2018 ) designed a chatbot to teach Java. The students’ learning process is monitored by collecting information on all interactions between the students and the chatbot. Thus, direct and customized instruction and feedback are provided to students. Another notable example can be found in (Latham et al., 2011 ), where students were given a learning path designed to their learning styles. With this approach, the students received 12% more accurate answers than those given chatbots without personalized learning materials. Moreover, other articles, such as the one mentioned in (Villegas-Ch et al., 2020 ), used AI for activity recommendation, depending on each student’s needs and learning paths. The chatbot evaluates and identifies students’ weaknesses and allows the AI model to be used in personalized learning.

Experiential Learning Experiential learning utilizes reflection on experience and encourages individuals to gain and construct knowledge by interacting with their environment through a set of perceived experiences (Felicia, 2011 ). Reflection on experience is the most important educational activity for developing comprehension skills and constructing knowledge. It is primarily based on the individual’s experience. Song et al. ( 2017 ) describe reflection as an intellectual activity that supports the course’s weekly reflection for online learners. The chatbot asks questions to help students reflect and construct their knowledge. D’mello and Graesser ( 2013 ) have presented a constructivist view of experiential learning. The embodied chatbot mimics the conversation movements of human tutors who advise students in gradually developing explanations to problems.

Social Dialog Social dialog, also called small talk, is a chit-chat that manages social situations rather than content exchange (Klüwer, 2011 ). The advantage of incorporating social dialog in the development of conversational agents is to establish relationships with users to engage users and gain their trust. For example, the chatbot presented in (Wambsganss et al., 2021 ) uses a casual chat mode allowing students to ask the chatbot to tell jokes, fun facts, or talk about unrelated content such as the weather to take a break from the main learning activity. As another example, Qin et al. ( 2020 ) suggested the usage of various social phrases that show interest, agreement, and social presence.

Collaborative learning Collaborative learning is an approach that involves groups of learners working together to complete a task or solve a problem. Collaborative learning has been demonstrated to be beneficial in improving students’ knowledge and improving the students’ critical thinking and argumentation (Tegos et al., 2015 ). One of the techniques used to support collaborative learning is using an Animated Conversational Agent (ACA) (Zedadra et al., 2014 ). This cognitive agent considers all the pedagogical activities related to Computer-Supported Collaborative Learning (CSCL), such as learning, collaboration, and tutoring. On the other hand, the collaborative learning approach that Tegos et al. ( 2020 ) used provides an open-ended discussion, encouraging the students to work collaboratively as a pair to provide answers for a question. Before beginning the synchronous collaborative activity, the students were advised to work on a particular unit material that contained videos, quizzes, and assignments. Additionally, Tegos et al. ( 2015 ) proposed a conversational agent named MentorChat, a cloud-based CSCL, to help teachers build dialog-based collaborative activities.

Affective learning Affective learning is a form of empathetic feedback given to the learner to keep the interest, attention, or desire to learn (Ayedoun et al., 2017 ). Two articles used this form of learning. For instance, Ayedoun et al. ( 2017 ) provided various types of affective feedback depending on the situation: congratulatory, encouraging, sympathetic, and reassuring. The idea is to support learners, mainly when a problematic situation arises, to increase their learning motivation. On the other hand, to assess the learning for low-literate people, Schouten et al. ( 2017 ) built their conversation agent to categorize four basic emotions: anger, fear, sadness, and happiness. Depending on the situation, the chatbot shows students an empathetic reaction. The researchers showed that this is helpful for learners and agents to express themselves, especially in the event of difficulty.

Learning by teaching Learning by teaching is a well-known pedagogical approach that allows students to learn through the generation of explanations to others (Chase et al., 2009 ). Two studies used this pedagogical technique. The first study (Matsuda et al., 2013 ) described a chatbot that learns from students’ answers and activities. Students are supposed to act as “tutors” and provide the chatbot with examples and feedback. The second study (Law et al., 2020 ) describes a teachable agent which starts by asking students low or high-level questions about a specific topic to evoke their curiosity. The student answers the questions, and the chatbot simulates learning. The chatbot provides a variety of questions by filling a pre-defined sentence template. To confirm its learning and make the conversation interesting, the chatbot seeks feedback from students by asking questions such as, “ Am I smart? ”, “ Am I learning? ” and “ Do you think I know more now than before? ”.

Scaffolding In the educational field, scaffolding is a term describing several teaching approaches used to gradually bring students toward better comprehension and, eventually, more independence in the learning process (West et al., 2017 ). Teachers provide successive degrees of temporary support that aid students in reaching excellent comprehension and skill development levels that they would not be able to attain without assistance (Maybin et al., 1992 ). In the same way, scaffolding was used as a learning strategy in a chatbot named Sara to improve students’ learning (Winkler et al., 2020 ). The chatbot provided voice and text-based scaffolds when needed. The approach had a significant improvement during learning in programming tasks.

5.6 RQ6 – What empirical evidence is there to substantiate the effectiveness of the proposed chatbots in education?

The surveyed articles used different types of empirical evaluation to assess the effectiveness of chatbots in educational settings. In some instances, researchers combined multiple evaluation methods, possibly to strengthen the findings.

We classified the empirical evaluation methods as follows: experiment, evaluation study, questionnaire, and focus group. An experiment is a scientific test performed under controlled conditions (Cook et al., 2002 ); one factor is changed at a time, while other factors are kept constant. It is the most familiar type of evaluation. It includes a hypothesis, a variable that the researcher can manipulate, and variables that can be measured, calculated, and compared. An evaluation study is a test to provide insights into specific parameters (Payne and Payne, 2004 ). There is typically no hypothesis to prove, and the results are often not statistically significant. A questionnaire is a data collection method for evaluation that focuses on a specific set of questions (Mellenbergh & Adèr, 2008 ). These questions aim to extract information from participants’ answers. It can be carried on by mail, telephone, face-to-face interview, and online using the web or email. A focus group allows researchers to evaluate a small group or sample that represents the community (Morgan, 1996 ). The idea behind the focus group is to examine some characteristics or behaviors of a sample when it’s difficult to examine all groups.

Table  9 and Fig.  9 show the various evaluation methods used by the articles. Most articles (13; 36.11%) used an experiment to establish the validity of the used approach, while 10 articles (27.77%) used an evaluation study to validate the usefulness and usability of their approach. The remaining articles used a questionnaire (10; 27.7%) and a focus group (3; 8.22%) as their evaluation methods.

Empirical evaluation methods applied in the selected studies

Experiments

Table  10 shows the details of the experiments the surveyed studies had used. Eight articles produced statistically significant results pointing to improved learning when using educational chatbots compared to a traditional learning setting, while a few other articles pointed to improved engagement, interest in learning, as well as subjective satisfaction.

A notable example of a conducted experiment includes the one discussed in (Wambsganss et al., 2021 ). The experiment evaluated whether adaptive tutoring implemented via the chatbot helps students write more convincing texts. The author designed two groups: a treatment group and a control group. The result showed that students using the chatbot (treatment group) to conduct a writing exercise wrote more convincing texts with a better formal argumentation quality than the traditional approach (control group). Another example is the experiment conducted by the authors in (Benotti et al., 2017 ), where the students worked on programming tasks. The experiment assessed the students’ learning by a post-test. Comparing the treatment group (students who interacted with the chatbot) with a control group (students in a traditional setting), the students in the control group have improved their learning and gained more interest in learning. Another study (Hayashi, 2013 ) evaluated the effect of text and audio-based suggestions of a chatbot used for formative assessment. The result shows that students receiving text and audio-based suggestions have improved learning.

Despite most studies showing overwhelming evidence for improved learning and engagement, one study (Fryer et al., 2017 ) found that students’ interest in communicating with the chatbot significantly declined in an 8-week longitudinal study where a chatbot was used to teach English.

Evaluation studies

In general, the studies conducting evaluation studies involved asking participants to take a test after being involved in an activity with the chatbot. The results of the evaluation studies (Table  12 ) point to various findings such as increased motivation, learning, task completeness, and high subjective satisfaction and engagement.

As an example of an evaluation study, the researchers in (Ruan et al., 2019 ) assessed students’ reactions and behavior while using ‘BookBuddy,’ a chatbot that helps students read books. The participants were five 6-year-old children. The researchers recorded the facial expressions of the participants using webcams. It turned out that the students were engaged more than half of the time while using BookBuddy.

Another interesting study was the one presented in (Law et al., 2020 ), where the authors explored how fourth and fifth-grade students interacted with a chatbot to teach it about several topics such as science and history. The students appreciated that the robot was attentive, curious, and eager to learn.

Questionnaires

Studies that used questionnaires as a form of evaluation assessed subjective satisfaction, perceived usefulness, and perceived usability, apart from one study that assessed perceived learning (Table  11 ). Assessing students’ perception of learning and usability is expected as questionnaires ultimately assess participants’ subjective opinions, and thus, they don’t objectively measure metrics such as students’ learning.

While using questionnaires as an evaluation method, the studies identified high subjective satisfaction, usefulness, and perceived usability. The questionnaires used mostly Likert scale closed-ended questions, but a few questionnaires also used open-ended questions.

A notable example of a study using questionnaires is ‘Rexy,’ a configurable educational chatbot discussed in (Benedetto & Cremonesi, 2019 ). The authors designed a questionnaire to assess Rexy. The questionnaires elicited feedback from participants and mainly evaluated the effectiveness and usefulness of learning with Rexy. The results largely point to high perceived usefulness. However, a few participants pointed out that it was sufficient for them to learn with a human partner. One student indicated a lack of trust in a chatbot.

Another example is the study presented in (Ondáš et al., 2019 ), where the authors evaluated various aspects of a chatbot used in the education process, including helpfulness, whether users wanted more features in the chatbot, and subjective satisfaction. The students found the tool helpful and efficient, albeit they wanted more features such as more information about courses and departments. About 62.5% of the students said they would use the chatbot again. In comparison, 88% of the students in (Daud et al., 2020 ) found the tool highly useful.

Focus group

Only three articles were evaluated by the focus group method. Only one study pointed to high usefulness and subjective satisfaction (Lee et al., 2020 ), while the others reported low to moderate subjective satisfaction (Table  13 ). For instance, the chatbot presented in (Lee et al., 2020 ) aims to increase learning effectiveness by allowing students to ask questions related to the course materials. The authors invited 10 undergraduate students to evaluate the chatbot. It turned out that most of the participants agreed that the chatbot is a valuable educational tool that facilitates real-time problem solving and provides a quick recap on course material. The study mentioned in (Mendez et al., 2020 ) conducted two focus groups to evaluate the efficacy of chatbot used for academic advising. While students were largely satisfied with the answers given by the chatbot, they thought it lacked personalization and the human touch of real academic advisors. Finally, the chatbot discussed by (Verleger & Pembridge, 2018 ) was built upon a Q&A database related to a programming course. Nevertheless, because the tool did not produce answers to some questions, some students decided to abandon it and instead use standard search engines to find answers.

5.7 RQ7: What are the challenges and limitations of using proposed chatbots?

Several challenges and limitations that hinder the use of chatbots were identified in the selected studies, which are summarized in Table  14 and listed as follow:

Insufficient or Inadequate Dataset Training The most recurring limitation in several studies is that the chatbots are either trained with a limited dataset or, even worse, incorrectly trained. Learners using chatbots with a limited dataset experienced difficulties learning as the chatbot could not answer their questions. As a result, they became frustrated (Winkler et al., 2020 ) and could not wholly engage in the learning process (Verleger & Pembridge, 2018 ; Qin et al., 2020 ). Another example that caused learner frustration is reported in (Qin et al., 2020 ), where the chatbot gave incorrect responses.

To combat the issues arising from inadequate training datasets, authors such as (Ruan et al., 2021 ) trained their chatbot using standard English language examination materials (e.g., IELTS and TOEFL). The evaluation suggests an improved engagement. Further, Song et al. ( 2017 ) argue that the use of Natural Language Processing (NLP) supports a more natural conversation instead of one that relies on a limited dataset and a rule-based mechanism.

User-centered design User-centered design (UCD) refers to the active involvement of users in several stages of the software cycle, including requirements gathering, iterative design, and evaluation (Dwivedi et al., 2012 ). The ultimate goal of UCD is to ensure software usability. One of the challenges mentioned in a couple of studies is the lack of student involvement in the design process (Verleger and Pembridge, 2018 ) which may have resulted in decreased engagement and motivation over time. As another example, Law et al. ( 2020 ) noted that personality traits might affect how learning with a chatbot is perceived. Thus, educators wishing to develop an educational chatbot may have to factor students’ personality traits into their design.

Losing Interest Over Time Interestingly, apart from one study, all of the reviewed articles report educational chatbots were used for a relatively short time. Fryer et al. ( 2017 ) found that students’ interest in communicating with the chatbot significantly dropped in a longitudinal study. The decline happened between the first and the second tasks suggesting a novelty effect while interacting with the chatbot. Such a decline did not happen when students were interacting with a human partner.

Lack of Feedback Feedback is a crucial element that affects learning in various environments (Hattie and Timperley, 2007 ). It draws learners’ attention to understanding gaps and supports them gain knowledge and competencies (Narciss et al., 2014 ). Moreover, feedback helps learners regulate their learning (Chou & Zou, 2020 ). Villegas-Ch et al. ( 2020 ) noted that the lack of assessments and exercises coupled with the absence of the feedback mechanism negatively affected the chatbot’s success.

Distractions Usability heuristics call for a user interface that focuses on the essential elements and does not distract users from necessary information (Inostroza et al., 2012 ). In the context of educational chatbots, this would mean that the design must focus on the essential interactions between the chatbot and the student. Qin et al. ( 2020 ) identified that external links and popups suggested by the chatbot could be distracting to students, and thus, must be used judiciously.

6 Discussion and future research directions

The purpose of this work was to conduct a systematic review of the educational chatbots to understand their fields of applications, platforms, interaction styles, design principles, empirical evidence, and limitations.

Seven general research questions were formulated in reference to the objectives.

RQ1 examined the fields the educational chatbots are used in. The results show that the surveyed chatbots were used to teach several fields. More than a third of the chatbots were developed to teach computer science topics, including programming languages and networks. Fewer chatbots targeted foreign language education, while slightly less than a third of the studies used general-purpose educational chatbots. Our findings are somewhat similar to (Wollny et al., 2021 ), and (Hwang and Chang, 2021 ), although both of the review studies reported that language learning was the most targeted educational topic, followed by computer programming. Other review studies such as (Winkler & Söllner, 2018 ) highlighted that chatbots were used to educate students on health, well-being, and self-advocacy.

RQ2 identified the platforms the educational chatbots operate on. Most surveyed chatbots are executed within web-based platforms, followed by a few chatbots running on mobile and desktop platforms. The web offers a versatile platform as multiple devices can access it, and it does not require installation. Other review studies such as (Cunningham-Nelson et al., 2019 ) and (Pérez et al., 2020 ) did not discuss the platform but mentioned the tools used to develop the chatbots. Popular tools include Dialogflow Footnote 10 , QnA Maker Footnote 11 , ChatFuel Footnote 12 . Generally, these tools allow for chatbot deployment on web and mobile platforms. Interestingly, Winkler and Söllner ( 2018 ) highlighted that mobile platforms are popular for chatbots used for medical education.

RQ3 explored the roles of the chatbots when interacting with students. More than half of the surveyed chatbots were used as teaching agents that recommended educational content to students or engaged students in a discussion on relevant topics. Our results are similar to those reported in (Smutny & Schreiberova, 2020 ) which classified most chatbots as teaching agents that recommend content, conducted formative assessments, and set learning goals.

Slightly more than a third of the surveyed chatbots acted as peer agents which helped students ask for help when needed. Such help includes term definition, FAQ (Frequently Asked Questions), and discussion scaffolding. No studies reported the use of peer agents. However, a review study (Wollny et al., 2021 ) reported that some chatbots were used for scaffolding which correlates with our findings.

Two chatbots were used as motivational agents showing empathetic and encouraging feedback as students learn. A few review studies such as (Okonkwo & Ade-Ibijola, 2021 ) and (Winkler & Söllner, 2018 ) identified that chatbots are used for motivation and engagement, but no details were given.

Finally, only two surveyed chatbots acted as teachable agents where students gradually taught the chatbots.

RQ4 investigated the interaction styles supported by the educational chatbots. Most surveyed chatbots used a chatbot-driven conversation where the chatbot was in control of the conversation. Some of these chatbots used a predetermined path, whereas others used intents that were triggered depending on the conversation. In general, related review studies did not distinguish between intent-based or flow-based chatbots. However, a review study surveyed chatbot-driven agents that were used for FAQ (Cunningham-Nelson et al., 2019 ). Other review studies, such as (Winkler & Söllner, 2018 ) highlighted that some chatbots are flow-based. However, no sufficient details were mentioned.

Only a few surveyed chatbots allowed for a user-driven conversation where the user can initiate and lead the conversation. Other review studies reported that such chatbots rely on AI algorithms (Winkler & Söllner, 2018 ).

RQ5 examined the principles used to guide the design of the educational chatbots. Personalized learning is a common approach where the learning content is recommended, and instruction and feedback are tailored based on students’ performance and learning styles. Most related review studies did not refer to personalized learning as a design principle, but some review studies such as (Cunningham-Nelson et al., 2019 ) indicated that some educational chatbots provided individualized responses to students.

Scaffolding has also been used in some chatbots where students are provided gradual guidance to help them become independent learners. Scaffolding chatbots can help when needed, for instance, when students are working on a challenging task. Other review studies such as (Wollny et al., 2021 ) also revealed that some chatbots scaffolded students’ discussions to help their learning.

Other surveyed chatbots supported collaborative learning by advising the students to work together on tasks or by engaging a group of students in a conversation. A related review study (Winkler & Söllner, 2018 ) highlighted that chatbots could be used to support collaborative learning.

The remaining surveyed chatbots engaged students in various methods such as social dialog, affective learning, learning by teaching, and experiential learning. However, none of the related review studies indicated such design principles behind educational chatbots.

A few surveyed chatbots have used social dialog to engage students. For instance, some chatbots engaged students with small talk and showed interest and social presence. Other chatbots used affective learning in the form of sympathetic and reassuring feedback to support learners in problematic situations. Additionally, learning by teaching was also used by two chatbots where the chatbot acted as a student and asked the chatbot for answers and examples. Further, a surveyed chatbot used experiential learning by asking students to develop explanations to problems gradually.

RQ6 studied the empirical evidence used to back the validity of the chatbots. Most surveyed chatbots were evaluated with experiments that largely proved with statistical significance that chatbots could improve learning and student satisfaction. A related review study (Hwang & Chang, 2021 ) indicated that many studies used experiments to substantiate the validity of chatbots. However, no discussion of findings was reported.

Some of the surveyed chatbots used evaluation studies to assess the effect of chatbots on perceived usefulness and subjective satisfaction. The results are in favor of the chatbots. A related review study (Hobert & Meyer von Wolff, 2019 ) mentioned that qualitative studies using pre/post surveys were used. However, no discussion of findings was reported.

Questionnaires were also used by some surveyed chatbots indicating perceived subjective satisfaction, ease of learning, and usefulness. Intriguingly, a review study (Pérez et al., 2020 ) suggested that questionnaires were the most common method of evaluation of chatbots. Such questionnaires pointed to high user satisfaction and no failure on the chatbot’s part.

Finally, only this study reported using focus groups as an evaluation method. Only three chatbots were evaluated with this method with a low number of participants, and the results showed usefulness, reasonable subjective satisfaction, and lack of training.

RQ7 examined the challenges and limitations of using educational chatbots. A frequently reported challenge was a lack of dataset training which caused frustration and learning difficulties. A review study (Pérez et al., 2020 ) hinted at a similar issue by shedding light on the complex task of collecting data to train the chatbots.

Two surveyed studies also noticed the novelty effect. Students seem to lose interest in talking to chatbots over time. A similar concern was reported by a related review study (Pérez et al., 2020 ).

Other limitations not highlighted by related review studies include the lack of user-centered design, the lack of feedback, and distractions. In general, the surveyed chatbots were not designed with the involvement of students in the process. Further, one surveyed chatbot did not assess the students’ knowledge, which may have negatively impacted the chatbot’s success. Finally, a surveyed study found that a chatbot’s external links and popup messages distracted the students from the essential tasks.

The main limitation not identified in our study is chatbot ethics. A review study (Okonkwo & Ade-Ibijola, 2021 ) discussed that ethical issues such as privacy and trust must be considered when designing educational chatbots.

To set the ground for future research and practical implementation of chatbots, we shed some light on several areas that should be considered when designing and implementing chatbots

Usability Principles Usability is a quality attribute that evaluates how easy a user interface is to use (Nathoo et al., 2019 ). Various usability principles can serve as guidance for designing user interfaces. For instance, Nielson presented ten heuristics considered rules of thumb Footnote 13 . Moreover, Shneiderman mentioned eight golden user interface design rules (Shneiderman et al., 2016 ). Further, based on the general usability principles and heuristics, some researchers devised usability heuristics for designing and evaluating chatbots (conversational user interfaces). The heuristics are based on traditional usability heuristics in conjunction with principles specific to conversation and language studies. In terms of the design phase, it is recommended to design user interfaces iteratively by involving users during the design phase (Lauesen, 2005 ).

The chatbots discussed in the reviewed articles aimed at helping students with the learning process. Since they interact with students, the design of the chatbots must pay attention to usability principles. However, none of the chatbots explicitly discussed the reliance on usability principles in the design phase. However, it could be argued that some of the authors designed the chatbots with usability in mind based on some design choices. For instance, Alobaidi et al. ( 2013 ) used contrast to capture user attention, while Ayedoun et al. ( 2017 ) designed their chatbot with subjective satisfaction in mind. Further, Song et al. ( 2017 ) involved the users in their design employing participatory design, while Clarizia et al. ( 2018 ) ensured that the chatbot design is consistent with existing popular chatbots. Similarly, Villegas-Ch et al. ( 2020 ) developed the user interface of their chatbot to be similar to that of Facebook messenger.

Nevertheless, we argue that it is crucial to design educational chatbots with usability principles in mind explicitly. Further, we recommend that future educators test for the chatbot’s impact on learning or student engagement and assess the usability of the chatbots.

Chatbot Personality Personality describes consistent and characteristic patterns of behavior, emotions, and cognition (Smestad and Volden, 2018 ). Research suggests that users treat chatbots as if they were humans (Chaves & Gerosa, 2021 ), and thus chatbots are increasingly built to have a personality. In fact, researchers have also used the Big Five model to explain the personalities a chatbot can have when interacting with users (Völkel & Kaya, 2021 ; McCrae & Costa, 2008 ). Existing studies experimented with various chatbot personalities such as agreeable, neutral, and disagreeable (Völkel & Kaya, 2021 ). An agreeable chatbot uses family-oriented words such as “family” or “together” (Hirsh et al., 2009 ), words that are regarded as positive emotionally such as “like” or “nice” (Hirsh et al., 2009 ), words indicating assurance such as “sure” (Nass et al., 1994 ), as well as certain emojis (Völkel et al., 2019 ), as suggested by the literature. On the other hand, a disagreeable chatbot does not show interest in the user and might be critical and uncooperative (Andrist et al., 2015 ).

Other personalities have also been attributed to chatbots, such as casual and formal personalities, where a formal chatbot uses a standardized language with proper grammar and punctuation, whereas a casual chatbot includes everyday, informal language (Andrist et al., 2015 ; Cafaro et al., 2016 ).

Despite the interest in chatbot personalities as a topic, most of the reviewed studies shied away from considering chatbot personality in their design. A few studies, such as (Coronado et al., 2018 ; Janati et al., 2020 ; Qin et al., 2020 ; Wambsganss et al., 2021 ), integrated social dialog into the design of the chatbot. However, the intention of the chatbots primarily focused on the learning process rather than the chatbot personality. We argue that future studies should shed light on how chatbot personality could affect learning and subjective satisfaction.

Chatbot Localization and Acceptance Human societies’ social behavior and conventions, as well as the individuals’ views, knowledge, laws, rituals, practices, and values, are all influenced by culture. It is described as the underlying values, beliefs, philosophy, and methods of interacting that contribute to a person’s unique psychological and social environment. Shin et al. ( 2022 ) defines culture as the common models of behaviors and interactions, cognitive frameworks, and perceptual awareness gained via socialization in a cross-cultural environment. The acceptance of chatbots involves a cultural dimension. The cultural and social circumstances in which the chatbot is used influence how students interpret the chatbot and how they consume and engage with it. For example, the study by (Rodrigo et al., 2012 ) shows evidence that the chatbot ‘Scooter’ was regarded and interacted with differently in the Philippines than in the United States. According to student gaming behavior in the Philippines, Scooter’s interface design did not properly exploit Philippine society’s demand for outwardly seamless interpersonal relationships.

Nevertheless, all other studies didn’t focus on localization as a design element crucial to the chatbot’s effectiveness and acceptance. We encourage future researchers and educators to assess how the localization of chatbots affects students’ acceptance of the chatbots and, consequently, the chatbot’s success as a learning mate.

Development Framework As it currently stands, the literature offers little guidance on designing effective usable chatbots. None of the studies used a certain framework or guiding principles in designing the chatbots. Future works could contribute to the Human-Computer Interaction (HCI) and education community by formulating guiding principles that assist educators and instructional designers in developing effective usable chatbots. Such guiding principles must assist educators and researchers across multiple dimensions, including learning outcomes and usability principles. A software engineering approach can be adopted, which guides educators in four phases: requirements, design, deployment, and assessment. A conceptual framework could be devised as a result of analyzing quantitative and qualitative data from empirical evaluations of educational chatbots. The framework could guide designing a learning activity using chatbots by considering learning outcomes, interaction styles, usability guidelines, and more.

End-user development of chatbots End-User Development (EUD) is a field that is concerned with tools and activities enabling end-users who are not professional software developers to write software programs (Lieberman et al., 2006 ). EUD uses various approaches such as visual programming (Kuhail et al., 2021 ) and declarative formulas (Kuhail and Lauesen, 2012 ). Since end-users outnumber software engineers by a factor of 30-to-1, EUD empowers a much larger pool of people to participate in software development (Kuhail et al., 2021 ). Only a few studies (e.g., (Ondáš et al., 2019 ; Benedetto & Cremonesi, 2019 ) have discussed how the educational chatbots were developed using technologies such as Google Dialogflow and IBM Watson Footnote 14 . Nevertheless, such technologies are only accessible to developers. Recently, commercial tools such as Google Dialogflow CX Footnote 15 emerged to allow non-programmers to develop chatbots with visual programming, allowing end-users to create a program by putting together graphical visual elements rather than specifying them textually.

Future studies could experiment with existing EUD tools that allow educational chatbots’ development. In particular, researchers could assess the usability and expressiveness of such tools and their suitability in the educational context.

7 Conclusion

This study described how several educational chatbot approaches empower learners across various domains. The study analyzed 36 educational chatbots proposed in the literature. To analyze the tools, the study assessed each chatbot within seven dimensions: educational field, platform, educational role, interaction style, design principles, empirical principles, and challenges as well as limitations.

The results show that the chatbots were proposed in various areas, including mainly computer science, language, general education, and a few other fields such as engineering and mathematics. Most chatbots are accessible via a web platform, and a fewer chatbots were available on mobile and desktop platforms. This choice can be explained by the flexibility the web platform offers as it potentially supports multiple devices, including laptops, mobile phones, etc.

In terms of the educational role, slightly more than half of the studies used teaching agents, while 13 studies (36.11%) used peer agents. Only two studies presented a teachable agent, and another two studies presented a motivational agent. Teaching agents gave students tutorials or asked them to watch videos with follow-up discussions. Peer agents allowed students to ask for help on demand, for instance, by looking terms up, while teachable agents initiated the conversation with a simple topic, then asked the students questions to learn. Motivational agents reacted to the students’ learning with various emotions, including empathy and approval.

In terms of the interaction style, the vast majority of the chatbots used a chatbot-driven style, with about half of the chatbots using a flow-based with a predetermined specific learning path, and 36.11% of the chatbots using an intent-based approach. Only four chatbots (11.11%) used a user-driven style where the user was in control of the conversation. A user-driven interaction was mainly utilized for chatbots teaching a foreign language.

Concerning the design principles behind the chatbots, slightly less than a third of the chatbots used personalized learning, which tailored the educational content based on learning weaknesses, style, and needs. Other chatbots used experiential learning (13.88%), social dialog (11.11%), collaborative learning (11.11%), affective learning (5.55%), learning by teaching (5.55%), and scaffolding (2.77%).

Concerning the evaluation methods used to establish the validity of the approach, slightly more than a third of the chatbots used experiment with mostly significant results. The remaining chatbots were evaluated with evaluation studies (27.77%), questionnaires (27.77%), and focus groups (8.33%). The findings point to improved learning, high usefulness, and subjective satisfaction.

Some studies mentioned limitations such as inadequate or insufficient dataset training, lack of user-centered design, students losing interest in the chatbot over time, and some distractions.

There are several challenges to be addressed by future research. None of the articles explicitly relied on usability heuristics and guidelines in designing the chatbots, though some authors stressed a few usability principles such as consistency and subjective satisfaction. Further, none of the articles discussed or assessed a distinct personality of the chatbots though research shows that chatbot personality affects users’ subjective satisfaction.

Future studies should explore chatbot localization, where a chatbot is customized based on the culture and context it is used in. Moreover, researchers should explore devising frameworks for designing and developing educational chatbots to guide educators to build usable and effective chatbots. Finally, researchers should explore EUD tools that allow non-programmer educators to design and develop educational chatbots to facilitate the development of educational chatbots. Adopting EUD tools to build chatbots would accelerate the adoption of the technology in various fields.

Study Limitations

We established some limitations that may affect this study. We restricted our research to the period January 2011 to April 2021. This limitation was necessary to allow us to practically begin the analysis of articles, which took several months. We potentially missed other interesting articles that could be valuable for this study at the date of submission.

We conducted our search using four digital libraries: ACM, Scopus, IEEE Xplore, and SpringerLink. We may have missed other relevant articles found in other libraries such as Web of Science.

Our initial search resulted in a total of 1208 articles. We applied exclusion criteria to find relevant articles that were possible to assess. As such, our decision might have caused a bias: for example, we could have excluded short papers presenting original ideas or papers without sufficient evidence.

Since different researchers with diverse research experience participated in this study, article classification may have been somewhat inaccurate. As such, we mitigated this risk by cross-checking the work done by each reviewer to ensure that no relevant article was erroneously excluded. We also discussed and clarified all doubts and gray areas after analyzing each selected article.

There is also a bias towards empirically evaluated articles as we only selected articles that have an empirical evaluation, such as experiments, evaluation studies, etc. Further, we only analyzed the most recent articles when many articles discussed the same concept by the same researchers.

At last, we could have missed articles that report an educational chatbot that could not be found in the selected search databases. To deal with this risk, we searched manually to identify significant work beyond the articles we found in the search databases. Nevertheless, the manual search did not result in any articles that are not already found in the searched databases.

Data Availability Statement

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

https://www.statista.com/statistics/656596/worldwidechatbot-market/

https://developer.amazon.com/en-US/alexa

https://www.apple.com/sa-ar/siri/

https://dialogflow.cloud.google.com

https://chatfuel.com/

https://www.rayyan.ai/

https://www.scimagojr.com/

https://www.appannie.com/en/insights/market-data/state-of-mobile-2020-infographic/

https://dialogflow.cloud.google.com/

https://www.qnamaker.ai/

https://www.nngroup.com/articles/ten-usability-heuristics/

https://www.ibm.com/sa-ar/watson

https://cloud.google.com/dialogflow/cx/docs/basics

AbuShawar, B., & Atwell, E. (2015). Alice chatbot: Trials and outputs. Computación y Sistemas , 19 (4), 625–632.

Article   Google Scholar  

Baylor, A.L (2011). The design of motivational agents and avatars. Educational Technology Research and Development , 59 (2), 291–300.

Benotti, L., Martnez, M.C., & Schapachnik, F. (2017). Atool for introducing computer science with automatic formative assessment. IEEE Transactions on Learning Technologies , 11 (2), 179–192.

Benotti, L., Martnez, M.C., & Schapachnik, F. (2018). Atool for introducing computer science with automatic formative assessment. IEEE Transactions on Learning Technologies , 11 (2), 179–192. https://doi.org/10.1109/TLT.2017.2682084 .

Cafaro, A., Vilhjálmsson, H.H., & Bickmore, T. (2016). First impressions in human–agent virtual encounters. ACM Transactions on Computer-Human Interaction (TOCHI) , 23 (4), 1–40.

Car, L.T., Dhinagaran, D.A., Kyaw, B.M., Kowatsch, T., Joty, S., Theng, Y.-L., & Atun, R. (2020). Conversational agents in health care: scoping review and conceptual analysis. Journal of medical Internet research , 22 (8), e17158.

Chase, C.C, Chin, D.B, Oppezzo, M.A, & Schwartz, D.L (2009). Teachable agents and the protégé effect: Increasing the effort towards learning. Journal of Science Education and Technology , 18 (4), 334–352.

Chaves, A.P., & Gerosa, M.A. (2021). How should my chatbot interact? a survey on social characteristics in human–chatbot interaction design. International Journal of Human–Computer Interaction , 37 (8), 729–758.

Chou, C.-Y., & Zou, N.-B. (2020). An analysis of internal and external feedback in self-regulated learning activities mediated by self-regulated learning tools and open learner models. International Journal of Educational Technology in Higher Education , 17 (1), 1–27.

Cook, T.D, Campbell, D.T., & Shadish, W. (2002). Experimental and quasi-experimental designs for generalized causal inference . MA: Houghton Mifflin Boston.

Google Scholar  

Coronado, M., Iglesias, C.A., Carrera, Á., & Mardomingo, A. (2018). A cognitive assistant for learning java featuring social dialogue. International Journal of Human-Computer Studies , 117 , 55–67.

Cunningham-Nelson, S., Baktashmotlagh, M., & Boles, W. (2019). Visualizing student opinionthroughtext analysis. IEEE Transactions on Education , 62 (4), 305–311.

Daud, S.H.M., Teo, N.H.I., & Zain, N.H.M. (2020). Ejava chatbot for learning programming language: Apost-pandemic alternative virtual tutor. International Journal , 8 (7), 3290–3298.

De Angeli, A., & Brahnam, S. (2008). I hate you! disinhibition with virtualpartners. Interacting with computers , 20 (3), 302–310.

Dehn, D.M, & Van Mulken, S. (2000). The impact of animated interface agents: a review of empirical research. International journal of human-computer studies , 52 (1), 1–22.

D’mello, S., & Graesser, A. (2013). Autotutor and affective autotutor: Learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Transactions on Interactive Intelligent Systems (TiiS) , 2 (4), 1–39.

Dwivedi, M., Upadhyay, M.S., & Tripathi, A. (2012). A working framework for the user-centered design approach and a survey of the available methods. International Journal of Scientific and Research Publications , 2 (4), 12–19.

Fryer, L.K, Ainley, M., Thompson, A., Gibson, A., & Sherlock, Z. (2017). Stimulating and sustaining interest in a language course: An experimental comparison of chatbot and human task partners. Computers in Human Behavior , 75 , 461–468.

Haake, M., & Gulz, A. (2009). A look atthe roles of look & roles in embodied pedagogical agents–a user preference perspective. International Journal of Artificial Intelligence in Education , 19 (1), 39–71.

Hattie, J., & Timperley, H. (2007). Thepower of feedback. Review of Educational Research , 77 (1), 81–112.

Hirsh, J.B, DeYoung, C.G, & Peterson, J.B (2009). Metatraits of the big five differentially predict engagement and restraint of behavior. Journal of Personality , 77 (4), 1085–1102.

Kester, L., Kirschner, P.A, & Van Merriënboer, J.J. (2005). The management of cognitive load during complex cognitive skill acquisition by means of computer-simulated problem solving. British Journal of Educational Psychology , 75 (1), 71–85.

King, F.B (2002). A virtual student: Not an ordinary joe. The Internet and Higher Education , 5 (2), 157–166.

Kulik, J.A, & Fletcher, J.D. (2016). Effectiveness of intelligent tutoring systems: a meta-analytic review. Review of Educational Research , 86 (1), 42–78.

Kumar, R., & Rose, C.P (2010). Architecture for building conversational agents that support collaborative learning. IEEE Transactions on Learning Technologies , 4 (1), 21–34.

Martha, A.S.D., & Santoso, H.B (2019). The design and impact of the pedagogical agent: A systematic literature review. Journal of Educators Online , 16 (1), n1.

Matsuda, N., Yarzebinski, E., Keiser, V., Raizada, R., Cohen, W.W, Stylianides, G.J, & Koedinger, K.R (2013). Cognitive anatomy of tutor learning: Lessons learned with simstudent. Journal of Educational Psychology , 105 (4), 1152.

Morgan, D.L (1996). Focus groups. Annual Review of Sociology , 22 (1), 129–152.

Narciss, S., Sosnovsky, S., Schnaubert, L., Andrès, E., Eichelmann, A., Goguadze, G., & Melis, E. (2014). Exploring feedback and student characteristics relevant for personalizing feedback strategies. Computers & Education , 71 , 56–76.

Okonkwo, C.W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence , 2 , 100033.

Pérez, J.Q., Daradoumis, T., & Puig, J.M.M. (2020). Rediscovering the use of chatbots in education: A systematic literature review. Computer Applications in Engineering Education , 28 (6), 1549–1565.

Rodrigo, M.M.T, Baker, R.S., Agapito, J., Nabos, J., Repalam, M.C., Reyes, S.S, & San Pedro, M.O.C. (2012). The effects of an interactive software agent on student affective dynamics while using; an intelligent tutoring system. IEEE Transactions on Affective Computing , 3 (2), 224–236.

Ruttkay, Z., & Pelachaud, C. (2006). From browstotrust: Evaluating embodied conversational agents Vol. 7. Berlin: Springer Science & Business Media.

MATH   Google Scholar  

Schouten, D.G., Venneker, F., Bosse, T., Neerincx, M.A, & Cremers, A.H. (2017). A digital coach that provides affective and social learning support to low-literate learners. IEEE Transactions on Learning Technologies , 11 (1), 67–80.

Seel, N.M. (2011). Encyclopedia of the sciences of learning . Berlin: Springer Science & Business Media.

Serenko, A., Bontis, N., & Detlor, B. (2007). End-user adoption of animated interface agent sin everyday work applications. Behaviour & Information Technology , 26 (2), 119–132.

Shin, D., Chotiyaputta, V., & Zaid, B. (2022). The effects of cultural dimensions on algorithmic news: How do cultural value orientations affect how people perceive algorithms? Computers in Human Behavior , 126 , 107007.

Shneiderman, B., Plaisant, C., Cohen, M.S, Jacobs, S., Elmqvist, N., & Diakopoulos, N. (2016). Designingthe user interface: strategies for effective human-computer interaction . London: Pearson.

Smutny, P., & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the facebook messenger. Computers & Education , 151 , 103862.

Stahl, G. (2006). Group cognition: Computer support for building collaborative knowledge (acting with technology) . Penguin: The MIT Press.

Book   Google Scholar  

Tegos, S., Demetriadis, S., & Karakostas, A. (2015). Promoting academically productive talk with conversational agent interventions in collaborative learning settings. Computers & Education , 87 , 309–325.

Tegos, S., Demetriadis, S., & Tsiatsos, T. (2014). A configurable conversational agent to trigger students’ productive dialogue: a pilot study in the call domain. International Journal of Artificial Intelligence in Education , 24 (1), 62–91.

VanLehn, K., Graesser, A., Jackson, G.T., Jordan, P., Olney, A., & Rosé, C.P. (2007). Natural language tutoring: A comparison of human tutors, computer tutors, and text. Cognitive Science , 31 (1), 3–52.

Villegas-Ch, W., Arias-Navarrete, A., & Palacios-Pacheco, X. (2020). Proposal of an architecture for the integration of a chatbot with artificial intelligence in a smart campus for the improvement of learning. Sustainability , 12 (4), 1500.

Walker, E., Rummel, N., & Koedinger, K.R (2011). Designing automated adaptive support to improve student helping behaviors in a peer tutoring activity. International Journal of Computer-Supported Collaborative Learning , 6 (2), 279–306.

Weizenbaum, J. (1966). Eliza–a computer program for the study of natural language communication between man and machine. Communications of the ACM , 9 (1), 36–45.

Wik, P., & Hjalmarsson, A. (2009). Embodied conversational agents in computer assisted language learning. Speech Communication , 51 (10), 1024–1037.

Alkhoori, A., Kuhail, M.A., & Alkhoori, A. (2020). Unibud: A virtual academic adviser. In 2020 12th annual undergraduate research conference on applied computing (URC) (pp. 1–4).

Alobaidi, O.G, Crockett, K.A, O’Shea, J.D, & Jarad, T.M (2013). Abdullah: An intelligent arabic conversational tutoring system for modern islamic education. In Proceedings ofthe world congress on engineering , Vol. 2.

Andrist, S., Mutlu, B., & Tapus, A. (2015). Look like me: matching robot personality via gazeto increase motivation. In Proceedings ofthe 33rd annual ACM conference on human factors in computing systems (pp. 3603–3612).

Anghelescu, P., & Nicolaescu, S.V. (2018). Chatbot application using search engines and teaching methods. In 2018 10th international conference on electronics, computers and artificial intelligence (ECAI) (pp. 1–6).

Ayedoun, E., Hayashi, Y., & Seta, K. (2017). Communication strategies and affective back channels for conversational agents to enhance learners’ willingnessto communicate in a second language. In International conference on artificial intelligence in education (pp. 459–462).

Benedetto, L., & Cremonesi, P. (2019). Rexy, a configurable application for building virtual teaching assistants. In IFIP conference on human-computer interaction (pp. 233–241).

Bradeško, L., & Mladenić, D. (2012). A survey of chatbot systemsthrough a loebner prize competition. In Proceedings of Slovenian language technologies society eighth conference of language technologies (pp. 34–37).

Brewer, R.N, Findlater, L., Kaye, J., Lasecki, W., Munteanu, C., & Weber, A. (2018). Accessible voice interfaces. In Companion ofthe 2018 ACM conference on computer supported cooperative work and social computing (pp. 441–446).

Budiu, R. (2018). The user experience of chatbots. Nielsen Norman Group 25 .

Chaudhuri, S., Kumar, R., Howley, I.K., & Rosé, C.P. (2009). Engaging collaborative learners with helping agents. In AIED (pp. 365–372).

Chhibber, N., & Law, E. (2019). Using conversational agents to support learning by teaching. arXiv: 1909.13443 .

Chukhno, O., Chukhno, N., Samouylov, K.E, & Shorgin, S. (2019). A chatbot as an environment for carrying out the group decision making process. In ITTMM (Selected Papers) (pp. 15–25).

Clarizia, F., Colace, F., Lombardi, M., Pascale, F., & Santaniello, D. (2018). Chatbot: An education support system for student. In International symposium on cyberspace safety and security (pp. 291–302).

Cunningham-Nelson, S., Boles, W., Trouton, L., & Margerison, E. (2019). A review of chatbots in education:practical steps forward. In 30th annual conference for the australasian association for engineering education (AAEE 2019): educators becoming agents of change: innovate, integrate, motivate (pp. 299–306).

da Silva Oliveira, J., Espíndola, D.B., Barwaldt, R., Ribeiro, L.M., & Pias, M. (2019). Ibm watson application as faq assistant about moodle. In 2019 ieee frontiers in education conference (FIE) (pp. 1–8).

Dutta, D. (2017). Developing an intelligent chat-bottool to assist high school students for learning general knowledge subjects (Tech. rep.) Georgia Institute of Technology.

Felicia, P. (2011). Handbook of research on improving learning and motivation through educational games: Multidisciplinary approaches: Multidisciplinary approaches. iGi Global.

Feng, D., Shaw, E., Kim, J., & Hovy, E. (2006). An intelligent discussion-bot for answering student queries in threaded discussions. In Proceedings ofthe 11th international conference on intelligent user interfaces (pp. 171–177).

Følstad, A., Skjuve, M., & Brandtzaeg, P.B. (2018). Different chatbots for different purposes:towards a typology of chatbots to understand interaction design. In International conference on internet science (pp. 145–156).

Griol, D., Baena, I., Molina, J.M., & de Miguel, A.S. (2014). A multimodal conversational agent for personalized language learning. In Ambient intelligence-software and applications (pp. 13–21). Springer.

Griol, D., García-Herrero, J., & Molina, J. (2011). The educagent platform: Intelligent conversational agents for e-learning applications. In Ambient intelligence-software and applications (pp. 117–124). Springer.

Gulz, A., Haake, M., Silvervarg, A., Sjödén, B., & Veletsianos, G. (2011). Building a social conversational pedagogical agent: Design challenges and methodological approaches. In Conversational agents and natural language interaction: Techniques and effective practices (pp. 128–155). IGI Global.

Hayashi, Y. (2013). Learner-support agents for collaborative interaction: A study on affect and communication channels.

Heffernan, N.T, & Croteau, E.A (2004). Web-based evaluations showing differential learning for tutorial strategies employed by the ms. lindquisttutor. In International conference on intelligent tutoring systems (pp. 491–500).

Hobert, S., & Meyer von Wolff, R. (2019). Say helloto your new automatedtutor–a structured literature review on pedagogical conversational agents.

Hwang, G.-J., & Chang, C.-Y. (2021). A review of opportunities and challenges of chatbots in education. Interactive Learning Environments, 1–14.

Inostroza, R., Rusu, C., Roncagliolo, S., Jimenez, C., & Rusu, V. (2012). Usability heuristics for touchscreen-based mobile devices. In 2012 ninth international conference on information technology-new generations (pp. 662–667).

Ismail, M., & Ade-Ibijola, A. (2019). Lecturer’s apprentice: A chatbot for assisting novice programmers. In 2019 international multidisciplinary information technology and engineering conference (IMITEC) (pp. 1–8).

Janati, S.E., Maach, A., & Ghanami, D.E. (2020). Adaptive e-learning ai-powered chatbot based on multimedia indexing. International Journal of Advanced Computer Science and Applications 11 (12). Retrieved from https://doi.org/10.14569/IJACSA.2020.0111238 .

Kaczorowska-Spychalska, D. (2019). Chatbots in marketing. Management 23 (1).

Keele, S., et al. (2007). Guidelines for performing systematic literature reviews in software engineering (Tech. Rep.) Citeseer.

Kerlyl, A., Hall, P., & Bull, S. (2006). Bringing chatbots into education: Towards natural language negotiation of open learner models. In International conference on innovative techniques and applications of artificial intelligence (pp. 179–192).

Kerry, A., Ellis, R., & Bull, S. (2008). Conversational agents in e-learning. In International conference on innovative techniques and applications of artificial intelligence (pp. 169–182).

Klüwer, T. (2011). “i like your shirt”-dialogue acts for enabling socialtalk in conversational agents. In International workshop on intelligent virtual agents (pp. 14–27).

Kuhail, M.A., & Lauesen, S. (2012). Customizable visualizations with formula-linked building blocks. In GRAPP/IVAPP (pp. 768–771).

Kuhail, M.A., Farooq, S., Hammad, R., & Bahja, M. (2021). Characterizing visual programming approaches for end-user developers: A systematic review. IEEE Access.

Latham, A., Crockett, K., McLean, D., & Edmonds, B. (2011). Oscar: an intelligent adaptive conversational agent tutoring system. In KES international symposium on agent and multi-agent systems: technologies and applications (pp. 563–572).

Lauesen, S. (2005). User interface design: a software engineering perspective. Pearson Education.

Laurillard, D. (2013). Rethinking university teaching: A conversational framework for the effective use of learning technologies. Routledge.

Law, E., Baghaei Ravari, P., Chhibber, N., Kulic, D., Lin, S., Pantasdo, K.D, Ceha, J., Suh, S., & Dillen, N. (2020). Curiosity notebook: A platform for learning by teaching conversational agents. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–9).

Lee, L.-K., Fung, Y.-C., Pun, Y.-W., Wong, K.-K., Yu, M.T.-Y., & Wu, N.-I. (2020). Using a multiplatform chatbot as an onlinetutor in a university course. In 2020 international symposium on educational technology (ISET) (pp. 53–56). IEEE.

Lieberman, H., Paternò, F., Klann, M., & Wulf, V. (2006). End-user development: An emerging paradigm. In End user development (pp. 1–8). Springer.

Mabunda, K. (2020). An intelligent chatbot for guiding visitors and locating venues.

Maybin, J., Mercer, N., & Stierer, B. (1992). Scaffolding learning in the classroom. Thinking voices: The work ofthe national oracy project 186–195.

McCrae, R.R., & Costa, P.T.Jr. (2008). The five-factortheory of personality.

Mellado-Silva, R., Faúndez-Ugalde, A., & Blanco-Lobos, M. (2020). Effective learning of tax regulations using different chatbot techniques.

Mellenbergh, G.J, & Adèr, H.J. (2008). Tests and questionnaires: Construction and administration. Advising on Research Methods: A Consultant’s Companion 211–236.

Mendez, S., Johanson, K., Martin Conley, V., Gosha, K., A Mack, N., Haynes, C., & A Gerhardt, R. (2020). Chatbots: Atoolto supplementthe future faculty mentoring of doctoral engineering students. International Journal of Doctoral Studies 15 .

Nass, C., Steuer, J., & Tauber, E.R (1994). Computers are social actors. In Proceedings ofthe SIGCHI conference on Human factors in computing systems (pp. 72–78).

Nathoo, A., Gangabissoon, T., & Bekaroo, G. (2019). Exploringthe use of tangible user interfaces for teaching basic java programming concepts: A usability study. In 2019 conference on next generation computing applications (NextComp) (pp. 1–5).

Oh, K.-J., Lee, D., Ko, B., & Choi, H.-J. (2017). A chatbot for psychiatric counseling in mental healthcare service based on emotional dialogue analysis and sentence generation. In 2017 18th IEEE international conference on mobile data management (MDM) (pp. 371–375).

Ondáš, S., Pleva, M., & Hládek, D. (2019). How chatbots can be involved in the education process. In 2019 17th international conference on emerging elearning technologies and applications (ICETA) (pp. 575–580).

Padró, L., & Stanilovsky, E. (2012). Freeling 3.0: Towards wider multilinguality. In LREC2012 .

Payne, G., & Payne, J. (2004). Key concepts in social research. Sage.

Qin, C., Huang, W., & Hew, K.F. (2020). Usingthe community of inquiry framework to develop an educational chatbot: lesson learned from a mobile instant messaging learning environment. In Proceedings ofthe 28th international conference on computers in education .

Ramesh, K., Ravishankaran, S., Joshi, A., & Chandrasekaran, K. (2017). A survey of design techniques for conversational agents. In International conference on information, communication and computing technology (pp. 336–350).

Redondo-Hernández, A., & Pérez-Marín, D. (2011). Aprocedureto automatically adapt questions in student–pedagogic conversational agent dialogues. In International conference on user modeling, adaptation, and personalization (pp. 122–134).

Ruan, S., Jiang, L., Xu, Q., Liu, Z., Davis, G.M, Brunskill, E., & Landay, J.A. (2021). Englishbot: An ai-powered conversational system for second language learning. In 26th international conference on intelligent user interfaces (pp. 434–444).

Ruan, S., Willis, A., Xu, Q., Davis, G.M., Jiang, L., Brunskill, E., & Landay, J.A (2019). Bookbuddy: Turning digital materials into interactive foreign language lessonsthrough a voice chatbot. In Proceedings ofthe Sixth (2019) ACM conference on learning@ scale (pp. 1–4).

Sinha, S., Basak, S., Dey, Y., & Mondal, A. (2020). An educational chatbot for answering queries, Springer.

Smestad, T.L., & Volden, F. (2018). Chatbot personalities matters. In International conference on internet science (pp. 170–181).

Song, D., Oh, E.Y., & Rice, M. (2017). Interacting with a conversational agent system for educational purposes in online courses. In 2017 10th international conference on human system interactions (HSI) (pp. 78–82). IEEE.

Tegos, S., Psathas, G., Tsiatsos, T., Katsanos, C., Karakostas, A., Tsibanis, C., & Demetriadis, S. (2020). Enriching synchronous collaboration in online courses with configurable conversational agents. In International Conference on Intelligent Tutoring Systems (pp. 284–294).

Thomas, H. (2020). Critical literature review on chatbots in education.

Verleger, M., & Pembridge, J. (2018). Apilot study integrating an ai-driven chatbot in an introductory programming course. In 2018 ieee frontiers in education conference (FIE) (pp. 1–4).

Völkel, S.T., Buschek, D., Pranjic, J., & Hussmann, H. (2019). Understanding emoji interpretation through user personality and message context. In Proceedings of the 21st international conference on human-computer interaction with mobile devices and services (pp. 1–12).

Völkel, S.T., & Kaya, L. (2021). Examining user preference for agreeableness in chatbots. In CUI 2021-3rd conference on conversational user interfaces (pp. 1–6).

Wambsganss, T., Kueng, T., Soellner, M., & Leimeister, J.M. (2021). Arguetutor: an adaptive dialog-based learning system for argumentation skills. In Proceedings of the 2021 CHI conference on human factors in computing systems (pp. 1–13).

Wambsganss, T., Winkler, R., Schmid, P., & Söllner, M. (2020). Designing a conversational agent as a formative course evaluation tool.

Wambsganss, T., Winkler, R., Söllner, M., & Leimeister, J. M. (2020). A conversational agent to improve response quality in course evaluations. In Extended Abstracts of the 2020 CHI conference on human factors in computing systems (pp. 1–9).

West, A., Swanson, J., & Lipscomb, L. (2017). Ch. 11 scaffolding. Instructional methods, strategies and technologies to meet the needs of all learners.

Winkler, R., Hobert, S., Salovaara, A., Söllner, M., & Leimeister, Jan Marco (2020). Sara, the lecturer: Improving learning in online education with a scaffolding-based conversational agent. In Proceedings of the 2020 CHI conference on human factors in computing systems (pp. 1–14).

Winkler, R., & Söllner, M. (2018). Unleashing the potential of chatbots in education: A state-of-the-art analysis. In Academy of management annual meeting (AOM) .

Wollny, S., Schneider, J., Di Mitri, D., Weidlich, J., Rittberger, M., & Drachsler, H. (2021). Are we there yet?-a systematic literature review on chatbots in education. Frontiers in artificial intelligence 4 .

Xu, A., Liu, Z., Guo, Y., Sinha, V., & Akkiraju, R. (2017). A new chatbot for customer service on social media. In Proceedings of the 2017 CHI conference on human factors in computing systems (pp. 3506–3510).

Zedadra, A., Lafifi, Y., & Zedadra, O. (2014). Interpreting learners’traces in collaborative learning environments. In 2014 4th international symposium isko-maghreb: concepts and tools for knowledge management (isko-maghreb) (pp. 1–8).

Download references

No funding was received to assist with the preparation of this manuscript.

The authors have no relevant financial or non-financial interests to disclose

Author information

Authors and affiliations.

College of Technological Innovation, Zayed University, Abu Dhabi, 144534, United Arab Emirates

Mohammad Amin Kuhail

Information System department, College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University, Riyadh, 11671, Saudi Arabia

Nazik Alturki

College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University, Riyadh, 11671, Saudi Arabia

Salwa Alramlawi

Independent Scholar, Yanbu, 46423, Saudi Arabia

Kholood Alhejori

You can also search for this author in PubMed   Google Scholar

Contributions

Mohammad Amin Kuhail: Conceptualization, Methodology, Validation, Investigation, Data Curation, Writing, Original Draft, Writing - Review & Editing, Supervision, Project administration

Nazik Alturki: Conceptualization, Methodology, Investigation, Data Curation, Writing, Writing - Review & Editing, Supervision.

Salwa Alramlawi: Investigation, Data Curation, Writing, Formal Analysis, Writing, Writing - Review & Editing.

Kholood Alhejori: Investigation, Data Curation, Writing, Writing - Review & Editing.

Corresponding author

Correspondence to Nazik Alturki .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Kuhail, M.A., Alturki, N., Alramlawi, S. et al. Interacting with educational chatbots: A systematic review. Educ Inf Technol 28 , 973–1018 (2023). https://doi.org/10.1007/s10639-022-11177-3

Download citation

Received : 28 February 2022

Accepted : 14 June 2022

Published : 09 July 2022

Issue Date : January 2023

DOI : https://doi.org/10.1007/s10639-022-11177-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Conversational Agent
• Educational Bot
• Literature Review
• Interaction Styles
• Human-Computer Interaction
• Find a journal
• Publish with us
• Track your research

A literature review on users' behavioral intention toward chatbots' adoption

Applied Computing and Informatics

ISSN : 2634-1964

Article publication date: 5 July 2022

Despite the fact that chatbots have been largely adopted for the last few years, a comprehensive literature review research focusing on the intention of individuals to adopt chatbots is rather scarce. In this respect, the present paper attempts a literature review investigation of empirical studies focused on the specific issue in nine scientific databases during 2017-2021. Specifically, it aims to classify extant empirical studies which focus on the context of individuals' adoption intention toward chatbots.

Design/methodology/approach

The research is based on PRISMA methodology, which revealed a total of 39 empirical studies examining users' intention to adopt and utilize chatbots.

After a thorough investigation, distinct categorization criteria emerged, such as research field, applied theoretical models, research types, methods and statistical measures, factors affecting intention to adopt and further use chatbots, the countries/continents where these surveys took place as well as relevant research citations and year of publication. In addition, the paper highlights research gaps in the examined issue and proposes future research directions in such a promising information technology solution.

Originality/value

As far as the authors are concerned, there has not been any other comprehensive literature review research to focus on examining previous empirical studies of users' intentions to adopt and use chatbots on the aforementioned period. According to the authors' knowledge, the present paper is the first attempt in the field which demonstrates broad literature review data of relevant empirical studies.

• Literature review
• Adoption intention
• Artificial intelligence

Gatzioufa, P. and Saprikis, V. (2022), "A literature review on users' behavioral intention toward chatbots' adoption", Applied Computing and Informatics , Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/ACI-01-2022-0021

Emerald Publishing Limited

Copyright © 2022, Paraskevi Gatzioufa and Vaggelis Saprikis

Published in Applied Computing and Informatics . Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode

1. Introduction

The widespread use of the Internet and the development of modern technologies have brought about significant changes, including artificial intelligence (AI) agents or chatbots. Chatbots are programs which, using AI, can answer users' questions usually during a text-based conversation [ 1–4 ]. Thus, in many cases they replace employees in customer service transactions, who, in the context of interaction with customers, answer their questions, propose solutions and redefine suggestions according to preferences and choices [ 5–7 ].

Chatbots have been variously defined in the international literature. They are frequently described as “software agents that facilitate automated conversation through natural language processing” [ 8 ], or as “an artificial construct that is designed to converse with human beings using natural language as input and output” [ 9 ], and “Artificial Conversational Entities or computer programs, based on AI, which are very interactive and conduct a conversation via auditory or textual method” [ 10 ].

Text chatbots have transformed communication and interaction between businesses and customers, by providing immediate response to requests, without time or space constraints and without human intervention.

In recent years, the use of chatbots has been widely adopted as part of the companies' marketing strategy [ 11 ]. Based on their utilization, customer service has been improved by reducing response time to requests and increasing loyalty. Typically, chatbots have been used for providing customers with entertainment and useful information, easily and fast, 24 hours a day with personalized help, saving both costs and manpower [ 3 ].

On the other hand, conventional customer service practices have not been abandoned. It is worth noting that a potential significant deterrent to the adoption of chatbots by users is the fact that a large number of customers tend to use traditional communication channels (i.e. mail, website and telephone) when communicating with companies, mainly because of security and privacy of personal data, which are critical issues requiring special attention in terms of their management [ 11 ]. Thus, as research has shown, trust and privacy concerns affect customers [ 12 ].

In the extant literature, various researchers have focused on a number of aspects related to chatbots. Remarkably, a significant number of researchers have focused on the intention to adopt and use chatbots by investigating factors which affect users in specific research areas, such as health [ 13 ], financial services [ 14–16 ], tourism [ 5, 17–20 ], customer service (e.g. Refs [ 1 ,  21–34 ]), mobile commerce [ 35–37 ], business [ 38, 39 ], insurance [ 12, 40 ] and education [ 41, 42 ].

The purpose of the present research is to provide a comprehensive literature review of the existing empirical studies in the field regarding individuals' intentions to adopt and use chatbots. More specifically, the research intends to categorize these studies in terms of a number of criteria, such as applied research methods, areas of chatbots' utilization, theoretical models, influential factors, the countries/continents where most studies have been carried out in the specific field and relevant research citations and year of publication. These classifications are expected to provide a cumulative and better view of the examined topic. As there has not been any other comprehensive literature review research to focus on examining previous empirical studies of users' intentions to adopt and use chatbots, the present paper is the first attempt in the field which demonstrates literature review data of relevant empirical studies.

Which behavioral theories have been most frequently used in the research of individuals' intention to adopt and use chatbots?

Which are the most commonly observed factors that influence users toward the adoption and use of chatbots?

In which sectors are the use of chatbots more frequently observed?

In which countries/continents has extensive and focused research been carried out?

Apart from the specific issues in relation to chatbot adoption and use, the paper aims to identify research gaps in the context of individuals' adoption intention toward chatbots, as well as reveal future research prospects in such a promising information technology solution for contemporary e- and m-business models.

The rest of the paper is divided into four sections. Section 2 discusses the applied research methodology, whereas Section 3 provides a literature review classification. Finally, Section 4 includes the conclusions drawn from the relevant literature review as well as the potential limitations before recommending a series of suggestions for future research and practice.

2. Research methodology

As already mentioned, the present paper discusses the researched literature review using PRISMA methodology (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), a most suitable methodology tool for the objectives of this study, by encompassing all the empirical studies concerning individuals' intention to adopt and use chatbots during 2017-2021 (i.e. from January 2017 to September 2021). The researched papers were selected in terms of specific inclusion and exclusion criteria, such as language–all texts should be written in English – and specific search keywords: chatbot, intention, adoption, usage, text-bot and AI , either in the paper title, abstract or keywords. Specifically, the search statement was as follows: (“chatbot” or “text-bot” OR “AI”) AND (“intention” OR “adoption” OR “usage”). Moreover, it was decided to focus on the most contemporary period when chatbots have been adopted and utilized. Hence, this is the reason why the research has included empirical studies from the last five years (2017–2021).Finally, it should me mentioned that the survey took place between February 2020 and September 2021.

Collection of data was based on the following procedure: Initially, a search of scientific databases was carried out, during which 90 empirical research papers emerged, 20 of which were readily excluded due to duplicate registrations. Next, 10 papers were rejected; eight of them because they did not meet the search criteria and three because they were not written in English. Full access to text was possible for 59. Of these 59 papers and after an in-depth investigation, 20 were not taken into consideration since they were not related to the scope of the paper. Of the remaining 39 sources, eight were conference papers, 27 were published in journals, 1 was an MSc Dissertation, 1 a workshop paper, 1 a symposium paper and 1 a book chapter. Our research relied on the following online academic databases: Science Direct, Emerald, Taylor and Francis, Elsevier, Research Gate, Wiley, IEEE Explore, ACM Digital Libraries and Google Scholar. The quality of the papers that are included to these databases guarantees the trustworthiness of the results of this literature review study.

All researched papers were carefully reviewed, and through their examination, a classification of the prevalent categories emerged. Notably, most research studies include all categories. The classification was based on eight different criteria: Types of Data Analysis, Research Methods, Statistical Methods of Analysis, Field of Study, Behavioral Theories Used, Factors which Affect Adoption Intention, Citations and Year of Publication, Country/Continent, all of which correspond to the specific research questions. The results of the classification were then organized into tables, followed by related comments, aiming to answer the research questions.

The use of certain criteria reflects the studies which were conducted, and describes the issues examined, distinguishing several categories, in accordance with those discussed by Misirlis and Vlachopoulou [ 43 ]. The review raises the research questions which reveal the trends in the specific field of individuals' behavioral intention to adopt and use chatbots. The applied methodology is shown in Figure 1 below.

3. Literature classification

The literature review classification was conducted in terms of the following eight criteria: types of data analysis, research methods, statistical methods of analysis, field of study, behavioral theories used by previous researchers, factors affecting the adoption and use of AI agents, number of citations and year of publication of the empirical research and countries/continents where the specific studies were conducted. Following previous literature review studies (e.g. Ref. [ 43 ]), the selection of these categorizations is expected to better present the extant and most contemporary empirical studies in the examined issue as well as reveal potential research gaps in the context of individuals' adoption intention toward chatbots.

3.1 Types of data analysis

Overall, the most common research method regarding users' intention to adopt and use text chatbots is quantitative research ( N  = 20, 51.2%), whereas qualitative research was used by authors to a much lesser extent in the literature ( N  = 4, 10.2%). Overall, most researchers have applied quantitative analysis methods to investigate the intention to adopt and use chatbots, such as Van Den Broeck et al. work [ 21 ]. In the specific empirical study 245 Facebook users were asked to rate their experience of using a chatbot (Cinebot). Qualitative methods were used by authors such as Mogaji et al. [ 15 ], who investigated the interaction of 36 Nigerians with chatbots in the banking sector.

However, there have also been mixed methods that include both qualitative and quantitative research ( N  = 15, 38.4%). For example, Cardona et al. [ 40 ] examined the factors which affect the adoption of chatbots in the insurance sector in Germany, using a sample of 300 respondents via email and social networks, as well as seven interviews with experts.

Summarizing the information presented, it can be deduced that quantitative studies seem to be more suitable for investigations in the context of individuals' behavioral intention toward chatbot adoption. The outcome of the type of analyses classified in quantitative, qualitative and mixed methods is presented in Table 1 .

3.2 Research methods

Regarding the research methods applied, almost all empirical studies utilized e-questionnaires ( N  = 19, 48.7%), which was the main data collection method. For example, Soni and Pooja [ 26 ] developed an e-questionnaire to determine the factors affecting the adoption of chatbots by generation Z.

With regard to interviewing, only three (7.7%) researchers utilized it as a data collection method. Interviews were used by researchers, such as Folstad et al. [ 44 ], and Mogaji et al. [ 15 ] who, by conducting semi-structured interviews, expected to receive answers to specific research questions.

However, several research studies utilized mixed research methods, such as experimental investigations and questionnaires (e.g. Refs [ 27, 28 ]) or a combination of interviews and experimental methods [ 29 ].

By combining these methods, the researchers tried to investigate the interaction with chatbots and the effect they had on the subsequent intention to use chatbots (e.g. Refs [ 45, 46 ]), as shown in Table 2 .

The above discussion demonstrated that e-questionnaires have various key advantages compared to other research methods, such as easy, fast and inexpensive access to a broad base of potential respondents, as well as anonymity, which makes it ideal for relevant empirical studies. Thus, it was applied by the majority of previous researchers, whereas a considerable number of studies preferred a combination of e-questionnaires with experimental investigations. In effect, the interaction of individuals with a chatbot, combined with a set of relevant questions, is another attractive way to get and analyze data about a number of users who interact for the first time with a new chatbot agent. Such studies can significantly help companies to further improve their emerging or utilized AI solution.

3.3 Statistical methods of analysis

Concerning data analyses, the researched quantitative surveys used statistical processing methods to draw valuable conclusions. The principal method was PLS-SEM ( N  = 13, 33.3%), followed by descriptive statistics measures ( N  = 10, 25.6%). Methods with lower frequency of utilization were regression analysis ( N  = 8, 20.5 %), ANOVA ( N  = 9, 23%), T -test and chi-square test ( N  = 7, 18%). Factor Analysis ( N  = 4, 10.2%) and correlation analysis ( N  = 5, 12.8%) were used to a lesser extent, as shown in Table 3 . Finally, four research papers did not use statistical methods of analysis ( N  = 4, 10.2%); they used content analysis after interviews.

It is worth noting that in several research papers a combination of statistical methods and analyses were applied. For example, Malik et al. [ 30 ] applied ANOVA and factor analysis. However, the predominance of PLM-SEM as the most suitable applied method can be attributed to the fact that it can analyze a large number of factors via various simultaneous regressions, which can have any direct and indirect impact of the examined factors on a single structural model.

3.4 Field of study

As discussed in the introduction, although chatbots have been applied in various areas, the largest percentage of previous empirical studies were focused on e-commerce and customer service ( N  = 17, 43.6%). Companies, especially those with strong customer interaction, have realized the importance of incorporating modern technologies, such as chatbots, in their operations. Hence, they compete with each other to offer the highest quality of customer experience, and their marketing practices have been restructured and oriented toward new tactics, including chatbots, which are an alternative to traditional customer service, by providing an additional level of support of services anytime, anywhere (e.g. Refs [ 1, 21 ]).

A smaller percentage surveyed the intention to use in the tourism industry ( N  = 5, 12.8%), where this technology is widely applied (e.g. Refs [ 5, 17 ]) and has been greatly benefited by the utilization of technologies, such as chatbots.

The research also demonstrated adoption of agents by students in 5.1% ( N  = 2) of cases; similarly, factors that lead to chatbot adoption using mobile phones was also observed in 7.7% ( N  = 3) of the cases [ 35–37 ]. Overall, mobile commerce is an easy and convenient process, with no space or time constraints. As all consumers who have a mobile phone can make purchases, chat-based marketing is one of the most popular digital tools.

In addition, 5.1% ( N  = 2) of the papers examined the acceptance of chatbots among employees in the banking sector [ 14–16 ]. An increasing percentage of financial institutions have already adopted the technology of chatbots aiming to facilitate the support to their clients' financial decisions and transactions. With regards to insurance companies, two studies (5.1%) were identified. Finally, one survey in the public transport [ 47 ], one in the health industry [ 13 ] (2.5%) and one in veterinary medicine [ 48 ] were also identified ( Table 4 ). The specific discussion of the sectors where chatbots have been examined is related to the 3rd research question of the paper.

3.5 Behavioral theories

The empirical studies examined in the present study were based on well-known behavioral theories, schemes and models to investigate specific factors. More specifically, the most frequently used approach in previous studies is UTAUT (Unified theory of acceptance and use of technology) and UTAUT2 ( N  = 5, 12.8%), proposed by Venkatesh et al. [ 49 ] to explain user intentions and subsequent use behavior in relation to information systems. According to this theory, the factors which affect the intention to use technology are performance expectancy, effort expectancy, social influence and facilitating conditions. Venkatesh et al. [ 50 ] expanded it to UTAUT2 theory, by adding three intrinsic factors affecting usage intention: hedonic motivation, price value and habit. According to the above characteristics, this theory fitted well and was applied in two research papers of this investigation [ 19, 42 ].

Another theoretical model observed in research papers is TAM (technology acceptance model), formulated by Davis et al. [ 51 ], to predict users' behavior toward adopting a new technology. In specific TAM, on its original form, was utilized by two studies [ 12, 18 ]. However, TAM was widely applied by being combined with other theories. Particularly, in five studies combined theories have been used (TAM and DOI; TAM and ECM and ISS; TAM & SST; TAM, DOI and TOE) [ 1, 29, 35, 37, 40 ], whereas in seven (18%) research papers information system (IS) continuance models such as SOR, U&G, TPB, CAT, TRA, SERVQUAL, extended post acceptance model were applied. Cheng et al. [ 52 ] used SOR theory to explain the behavior of consumers toward chatbots in the context of e-commerce.

Finally, it is worth noting that 51.3% of the research papers did not employ a common theoretical model ( N  = 20). In specific, these empirical studies base their proposed framework by combining various factors used in a number of past papers and have been already proved for their validity and reliability. Despite the fact that UTAUT(2) and TAM are the most widely applied behavioral theories in previous empirical studies, it should be emphasized that a large number of other theories have been applied in the examined field as well. The results are presented in Table 5 and are associated with the 1st research question of the paper.

3.6 Factors affecting adoption intention

Regarding the second research question, the results ( Table 6 ) demonstrate that the main determinants of TAM and UTAUT (both versions), which are the main behavioral theories, and on which previous researchers based their empirical studies, have been more commonly applied. More specifically, perceived usefulness, which is a key factor of TAM, is top on the list, followed by performance expectancy, trust and attitude. Apart from these factors, effort expectancy, habit, perceived enjoyment, perceived ease of use and social influence were also examined and confirmed in various relevant empirical studies. It should be emphasized, however, that these factors have a direct effect on individuals to adopt and use chatbots.

On the other hand, there are factors which indirectly affect users toward their behavioral intention to adopt and use chatbots (i.e. perceived completeness and communication style) [ 53 ]. There is also a significant number of factors with both direct and indirect impact on individuals in the examined topic, such as perceived usefulness, perceived enjoyment, perceived ease of use, trust and attitude.

3.7 Citations and year of publication

The research demonstrated the top five most popular citations until 7/10/2021 ( Table 7 ). Obviously, the research article by Brandtzaeg and Folstad [ 8 ] ranks first with 410 cross-references, followed by Ciechanowski et al. [ 45 ] and Go and Sundar's [ 54 ] empirical work.

As shown in Figure 2 , research in the examined topic has significantly increased since 2019, peaking in 2020. Two papers were carried out in 2017 and 2018, nine in 2019, 15 in 2020 and 11 until September of 2021. Therefore, the results show an increase in the published papers during the last two years.

3.8 Researched countries/continents

With regard to the countries where these empirical studies were undertaken, the countries where there is the vast majority of investigations on users' behavioral intention to adopt and use chatbots were the United States ( N  = 7, 18%) and India ( N  = 6, 15.4%), followed by the United Kingdom ( N  = 5, 12.8%). These three countries are followed by Germany, Italy, the Netherlands, Norway and China ( N  = 2, 5.1%), whereas in Poland, Japan, South Korea, Indonesia, Nigeria, Spain, Taiwan and the Philippines only one research paper was identified in each (2.6%). In addition, one empirical investigation was conducted in two countries – the United States and the Netherlands, whereas in four (12.8%) studies the countries where the investigations took place were not mentioned.

As regards the continents where the greatest number of surveys has been carried out, Europe ranks first with 15 surveys, followed by Asia where 13 empirical studies were identified, whereas in America, except for the USA, no other countries have carried out any investigations. Similarly, one empirical study was observed in Africa (Nigeria), whereas there are no such studies in Oceania, which is quite surprising. Taking into consideration the broad and continuous adopt of chatbots from numerous e- and m-business models worldwide, it is very surprising that there have not been analogous studies in a large number of countries so far. Table 8 summarizes the relevant information and demonstrates the relation to the 4th research question of the paper.

4. Conclusions

The present paper is a literature review study concerning the empirical investigation of users' behavioral intention to adopt and use chatbots during the last five years. By analyzing key characteristic points of these empirical research studies, a number of significant findings were drawn.

According to the research results in terms of the theoretical models applied, the most commonly used approaches are UTAUT(2) and TAM. Regarding the factors which affect the intention to use and adopt chatbots, performance expectancy, effort expectancy, social influence, trust and attitude are the most significant considerations associated with the behavioral theories on which they were based, whereas in relation to the areas on which most of the research work was focused, customer service ranks first by far. It can, thus, be concluded that an increasing number of companies are focusing their marketing strategies on adopting such technologies to provide rapid and effective services through websites or mobile apps, as a great number of consumers spend most of their time online, either for fun or for informational and/or professional issues.

The present research discussed surveys based mostly on quantitative data collection methods through e-questionnaires, whereas most researchers employed the PLS-SEM statistical method. As regards the countries and continents where empirical research has been carried out, most surveys have taken place in Europe and Asia. Notably, however, there have been no such studies in Oceania and in a large number of developed countries, such as France, Canada, Sweden, Switzerland, etc., where chatbots have been widely applied and greatly welcomed by users in various e- and m-business models. Therefore, this is a significant research gap, on which researchers are expected to focus attention in the upcoming period.

Furthermore, taking into consideration that there is a great interest in using chatbots mostly in customer service, companies should understand their role, especially in the context of the pandemic and the ensuing social distancing, so that they would be able to meet their customers' needs promptly, safely and beyond geographical or time constraints. However, despite the fact that most empirical studies have focused on customer service, there is a significant research gap in specific fields, such as the telecommunication industry, where chatbots have been already applied. Thus, a number of studies on such fields might provide useful insights for chatbots' adoption and further use.

The present literature review highlights the need for the academia to employ knowledge and further investigation of additional factors and dimensions requiring the application and use of chatbots in e- and m-commerce, which have not been identified to date. The specific factors may provide the ground for empirical studies in digital contexts. The categorization ensued should stimulate further efforts for development and evaluation for improving and advancing the relevant research.

Moreover, the research aims at emphasizing the developments in the relevant field by highlighting the factors that cause users to adopt AI agents and provides the appropriate theoretical background for both the academia and the industry. It also underscores additional components which have not been researched and may play a role in e-commerce and m-commerce, such as time response, efficiency and effectiveness of users' experience, mobile apps convenience, internal barriers, pre- and post-use behavior and how it relates to intention and satisfaction resulting from the quality of chatbot service.

The implications of the survey for practitioners are associated with the policies they must follow to enhance the factors which affect the intention to use chatbots, as well as create the conditions that will make individuals integrate and further utilize chatbots in their transactions with companies. In addition, as already discussed, a significant number of such investigations should take place in countries and sectors where chatbots have been applied or intended to be used. Hence, companies could definitely have a more comprehensive view about their chatbot investments and how they can derive better outcomes from this promising IT solution.

With regard to further future research recommendations, particular attention should be paid to the generalization of the research results, as this is related to various factors, which seem contradictory and cover multi-dimensional perspectives in a way. In detail, the most common determinants are the socio-economic and cultural environment of each country and its status in terms of development, as well as digitization level and access-use of the Internet in each country. An additional key factor of differentiation is individuals' personality and innovation characteristics. Thus, a person who is reluctant to accept anything new and innovative will not adopt chatbots as readily as a person who is eager to try new and innovative e-solutions. On the other hand, a fourth factor is to build trust and ensure the quality of service between individuals and companies as far as chatbot is concerned. Companies, especially those involved in customer service, make efforts to respond as quick and efficient as possible to their customers' requests, aiming to satisfy and subsequently gain their loyalty. Therefore, e-readiness to integrate such marketing practices in their business strategy is interesting to be further investigated. Chatbots have already been widely acceptable in some countries, and an important issue is how to get most benefits from their capabilities. In contrast, as aforementioned, there are a number of countries that lag behind because of socio-economic factors and technology-related determinants. To sum up, a multi-dimensional future examination of the aforementioned factors is expected to reveal useful insights and possibly reveal novel chatbots' perspectives to both academia and the industry.

PRISMA methodology procedure

Number of references per year

Types of data analysis

Research methods

Statistical methods of analysis

Field of study

Behavioral theories

Factors influencing intention

List of the top-5 most cited papers, as of October 7th, 2021

Chatbot adoption-intention papers per continent and country

1 Ashfaq M , Yun J , Yu S , Loureiro SMC. , Chatbot I . Modeling the determinants of users' satisfaction and continuance intention of AI-power service agents . Telematics Inform . 2020 ; 54 : 17 .

2 Przegalinska A , Ciechanowski L , Stroz A , Gloor P , Mazurek G . In bot we trust: a new methodology of chatbot performance measures . Bus Horiz . 2019 ; 62 ( 6 ): 785 - 97 .

3 Radziwill NM , Benton MC . Evaluating quality of chatbots and intelligent conversational agents . Comput Sci (Internet) . 2017 . Available from: http://arxiv.org/abs/1704.04579.pdf .

4 Sivaramakrishnan S , Wan F , Tang Z . Giving an “e-human touch” to e-tailing: the moderating roles of static information quantity and consumption motive in the effectiveness of an anthropomorphic information agent . J Interact Mark . 2007 ; 21 ( 1 ): 60 - 75 .

5 Dash M , Bakshi S . An exploratory study of customer perceptions of usage of chatbots in the hospitality industry . Int J Cust Relat . 2019 ; 7 ( 2 ): 27 - 33 .

6 Rowley J . Product searching with shopping bots . Internet Res . 2000 ; 10 ( 3 ): 203 - 14 .

7 Smith MD . The impact of shopbots on electronic markets . J Acad Mark Sci . 2002 ; 30 ( 4 ): 446 - 54 .

8 Brandtzaeg PB , Folstad A . Why people use chatbots . In: International Conference on Internet Science ; 2017 Nov 22-24 ; Thessaloniki . 377 - 92 . doi: 10.1007/978-3-319-70284-1_30 .

9 Brennan K . The managed teacher: emotional labour, education, and technology . Educ Insights . 2006 ; 10 ( 2 ): 55 - 65 .

10 Shawar BA , Atwell E . Different measurements metrics to evaluate a chatbot system . In: Proceedings of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies ; 2007 . 89 - 96 .

11 Zumstein D , Hundertmark S . Chatbots - an interactive technology for personalized communication, transactions and services . IADIS Int J WWW/Internet . 2017 ; 15 ( 1 ): 96 - 109 .

12 Cardona DR , Janssen A , Guhr N , Breitnet MH , Milde J . A matter of trust? Examination of chatbot usage in insurance business . In: Proceedings of the 54th Hawaii International Conference on System Sciences 2021 Maui ; Hawaii .

13 Nadarzynski T , Miles O , Cowie A , Ridge D . Acceptability of artificial intelligence (AI)-led chatbot services in healthcare: a mixed-methods study . Digital Health . 2019 ; 5 : 1 - 12 .

14 Quah JT , Chua YW . Chatbot assisted marketing in financial service industry . International conference on services computing . Cham : Springer ; 2019 . 107 - 14 . doi: 10.1007/978-3-030-23554-3_8 .

15 Mogaji E , Balakrishnan J , Nwoba Christian A , Nguyen P . Emerging-market consumers' interactions with banking chatbots . Telemat Inform . 2021 ; 65 : 101711 . Elsevier .

16 Coopamootoo NM , Toreini E , Aitken M , Elliot K , Van Moorse A . Simulating the effects of social presence on trust, privacy concerns & usage intentions in automated bots for finance . In: 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) . np .

17 Li L , Lee KY , Emokpae E , Yang S-B . What makes you continuously use chatbot services? Evidence from Chinese online travel agencies . Electron Mark . 2021 . doi: 10.1007/s12525-020-00454-z .

18 Pillai R , Sivathanu B . Adoption of AI-based chatbots for hospitality and tourism . Int J Contemp Hosp Manag . 2020 ; 32 ( 10 ): 3199 - 226 .

19 Gonzalez Melian S , Tano Gutierrez D , Gidumal Bulchand J . Predicting the intentions to use chatbots for travel and tourism . Curr Issues Tour . 2021 ; 24 ( 2 ): 192 - 210 .

20 Roy R , Naidoo V . Enhancing chatbot effectiveness: the role of anthropomorphic conversational styles and time orientation . J Bus Res . 2021 ; 126 : 23 - 34 .

21 Zarouali B , Van den Broeck E , Walrave M , Poels K . Predicting consumer responses to a chatbot on Facebook . Cyberpsychol Behav Soc Netw . 2018 ; 21 ( 8 ): 491 - 497 .

22 Chung M , Ko E , Joung H , Kim SJ . Chatbot e-service and customer satisfaction regarding luxury brands . J Bus Res . 2020 ; 117 : 587 - 95 .

23 Luo X , Tong S , Fang Z , Qu Z . Frontiers: machines vs humans: the impact of artificial intelligence chatbot disclosure on customer purchases . Mark Sci . 2019 ; 38 ( 6 ): 937 - 47 .

24 Kvale K , Freddi E , Hodnebrog S , Sell OA , Folstad A . Understanding the user experience of customer service chatbots: what can we learn from customer satisfaction surveys? Chatbot Research and Design . 4th International Workshop, CONVERSATIONS ; 2020 Nov 23-24 ; 2021 . 205 - 18 .

25 Van den Broeck E , Zarouali B , Poels K . Chatbot advertising effectiveness: when does the message get through? Comput Hum Behav . 2019 ; 98 : 150 - 7 .

26 Soni R , Pooja B . Trust in chatbots: investigating key factors influencing the adoption of chatbots by Generation Z . MuktShabd J . 2020 ; 9 ( 5 ): 5528 - 43 .

27 Sands S. , Ferraro C. , Campell C. , Tsao HY . Managing the human-chatbot divide:how service scripts influence service experience . J Serv Manag . 2021 . ISSN 1757-5818 Elsevier .

28 De Cicco R , Da Costa e Silva SCL , Alparone FR . It's on its way: chatbots applied for online food delivery services, social or task-oriented interaction style? J Food Serv Bus Res . 2020 ; 24 ( 2 ): 140 - 64 .

29 Van der Goot MJ , Pilgrim T . Exploring age differences in motivations for and acceptance of chatbot communication in a customer service context . International Workshop on Chatbot Research and Design, Book Series (LNCS, Volume 11970) . Springer Link ; 2020 . 173 - 86 .

30 Malik P , Gautam S , Srivastava S . A study on behavior intention for using chatbots . In: 8th International Conference on Reliability ; 2020 Jun 4-5 ; Noida, India : Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) ,Institute of Electrical and Electronics Engineers (IEEE) . np .

31 Huang Yu S. , Kao WK . Chatbot service usage during a pandemic: fear and social distancing . Serv Ind J . 2021 ; 41 ( 13-14 ): 964 - 84 .

32 Sheehan B. , Jin HS. , Gottlieb U . Customer service chatbots: anthropomorphism and adoption . J Bus Res . 2020 ; 115 : 14 - 24 .

33 Soni R , Tyagi V . Acceptance of chat bots by millennial consumers . Int J Res Eng Manag . 2019 ; 4 ( 10 ): 429 - 32 . ISSN 2454-9150 .

34 Trapero H , Ilao J , Lacaza R . An integrated theory for chatbot use in air travel: questionnaire development and validation . In: 2020 IEEE Region 10 Conference (TENCON) ; 2020 Nov 16-19 ; Osaka, Japan . 652 - 7 . doi: 10.1109/TENCON50793.2020.9293710 .

35 Van Eeuwen M . Mobile conversational commerce: messenger chatbots as the next interface between businesses and consumers . Master's thesis . University of Twente ; 2017 .

36 De Cosmo LM , Piper L , Di Vittorio A . The role of attitude toward chatbots and privacy concern on the relationship between attitude toward mobile advertising and behavioral intent to use chatbots . Ital J Mark . 2021 ; ( 1-2 ): 83 - 102 .

37 Kasilingam DL . Understanding the attitude and intention to use smartphone chatbots for shopping . Technol Soc . 2020 ; 62 : 15 .

38 Brachten F , Kissmer T , Stieglitz S . The acceptance of chatbots in an enterprise context – a survey study . Int J Inf Manag . 2021 ; 60 ( C ). doi: 10.1016/j.ijinfomgt.2021.102375 .

39 Selamat MA , Windasari NA . Chatbot for SMEs: integrating customer and business owner perspectives . Technol Soc . 2021 ; 66 ( C ). Elsevier , 101685 .

40 Cardona DR , Werth O , Schonborn S , Breitner MH . A mixed methods analysis of the adoption and diffusion of chatbot technology in the German insurance sector . In: Twenty-fifth Americas Conference on Information System ; Mexico : Cancun ; 2019 .

41 Fryer L , Nakao K , Thompson A . Chatbot learning partners: connecting learning experiences, interest and competence . Comput Hum Behav . 2019 ; 93 : 279 - 89 .

42 Almahri Amer Jid F , Bell D , Merhi M . Understanding student acceptance and use of chatbots in the United Kingdom universities: a structural equation modeling approach . In: 6th IEEE International Conference on Information Management ; UK : IEEE Xplore ; 2020 . 284 - 8 .

43 Misirlis N , Vlachopoulou M . Social media metrics and analytics in marketing-S3M: a mapping literature review . Int J Inf Manag . 2018 ; 38 ( 1 ): 270 - 6 .

44 Folstad A , Nordheum CB , Bjorkli CA . What makes users trust a chatbot for customer service. An exploratory interview study . In: 5 th International Conference on Internet Science- INSCI ; 2018 Oct 24-26 ; St. Petersburg, Russia : Internet Science ; 2018 . 194 - 208 .

45 Ciechanowski L , Przegalinska A , Magnuski M , Gloor P . In the shades of the uncanny valley: an experimental study of human-chatbot interaction . Future Gener Comput Syst . 2019 ; 92 : 539 - 48 .

46 Lee S. , Lee N. , Sah YJ . Perceiving a mind in a chatbot: effect of mind perception and social cues on co-presence, closeness, and intention to use . Int J Hum-Comput Int . 2020 ; 36 ( 10 ): 930 - 40 .

47 Kuberkar S , Singhal TK . Factors influencing adoption intention of AI powered chatbot for public transport services within a smart city . Int J Emerg Tech . 2020 ; 11 ( 3 ): 948 - 58 .

48 Huang D.H , Chueh HE . Chatbot usage intention analysis: veterinary consultation . J Innov Knowl . 2021 ; 6 ( 3 ): 135 - 44 .

49 Venkatesh V , Morris MG , Davis GB , Davis FD . User acceptance of information technology: toward a unified view . MIS Q Manag Inf Syst . 2003 ; 27 ( 3 ): 425 - 78 .

50 Venkatesh V , Thong JYL , Xu X . Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology . MIS Q Manag Inf Syst . 2012 ; 36 ( 1 ): 157 - 78 .

51 Davis FD , Bagozzi RP , Warshaw PR . User acceptance of computer technology: a comparison of two theoretical models . Manag Sci . 1989 ; 35 ( 8 ): 982 - 1003 .

52 Cheng X , Bao Y , Zarifis A , Gong W. , Mou J . Exploring consumers' response to text-based chatbots in e-commerce: the moderating role of task complexity and chatbot disclosure . Internet Res . 2021 ; 32 ( 2 ): 1066 - 2243 . Emerald Publishing . doi: 10.1108/INTR-08-2020-0460 .

53 Silva S , De Cicco R , Alparone F . What kind of chatbot do millenials prefer to interact with? In: Proceedings of the 49th European Annual Marketing Academy Conference ; 2020 May 26-29 ; Budapest .

54 Go E , Sundar SS . Humanizing chatbots: the effects of visual, identity and conversational cues on humanness perceptions . Comput Hum Behav . 2019 ; 97 : 304 - 16 .

Corresponding author

Related articles, all feedback is valuable.

Please share your general feedback

Report an issue or find answers to frequently asked questions

Contact Customer Support

• Interesting
• Scholarships
• UGC-CARE Journals

Step-by-Step Guide: How to Use ChatGPT to Write a Literature Review With Prompts

Step-by-Step Guide: How to Use ChatGPT for Writing a Literature Review

Writing a literature review can be a challenging task for researchers and students alike. It requires a comprehensive understanding of the existing body of research on a particular topic. However, with the advent of advanced language models like ChatGPT, the process has become more accessible and efficient.

In this step-by-step guide, ilovephd will explore how you can leverage ChatGPT to write a compelling literature review that is both informative.

A Step-by-Step Guide: How to Use ChatGPT for Writing a Literature Review

Step 1: Defining Your Research Objective Before diving into the literature review process, it is crucial to define your research objective.

Clearly articulate the topic, research question, or hypothesis you aim to address through your literature review. This step will help you maintain focus and guide your search for relevant sources.

Step 2: Identifying Keywords and Search Terms To effectively use ChatGPT to assist in your literature review, you need to identify relevant keywords and search terms related to your research topic.

These keywords will help you narrow down your search and gather pertinent information. Consider using tools like Google Keyword Planner or other keyword research tools to discover commonly used terms in your field.

Step 3: Familiarizing Yourself with ChatGPT Before engaging with ChatGPT, it is essential to understand its capabilities and limitations. Familiarize yourself with the prompts and commands that work best with the model.

Keep in mind that ChatGPT is an AI language model trained on a vast amount of data, so it can provide valuable insights and suggestions, but it’s important to critically evaluate and validate the information it generates.

Step 4: Generating an Initial Literature Review Outline Start by creating an outline for your literature review. Outline the main sections, such as the introduction, methodology, results, discussion, and conclusion.

Within each section, jot down the key points or subtopics you want to cover. This will help you organize your thoughts and structure your review effectively.

Step 5: Engaging with ChatGPT for Research Assistance Once you have your outline ready, engage with ChatGPT for research assistance.

Begin by providing a clear and concise prompt that specifies the topic, context, and any specific questions you have. For example, “What are the current trends in [your research topic]?” or “Can you provide an overview of the main theories on [your research question]?”

Step 6: Reviewing and Selecting Generated Content ChatGPT will generate a response based on your prompt. Carefully review the content generated, considering its relevance, accuracy, and coherence.

Extract key points, relevant references, and insightful arguments from the response and incorporate them into your literature review. Be sure to cite and attribute the sources appropriately.

Step 7: Ensuring Coherence and Flow While ChatGPT can provide valuable content, it’s important to ensure the coherence and flow of your literature review.

Use your critical thinking skills to connect the generated content with your research objective and existing knowledge. Rearrange, rephrase, and expand upon the generated text to ensure it aligns with the structure and purpose of your review.

Step 8: Editing and Proofreading Once you have incorporated the generated content into your literature review, thoroughly edit and proofread the document.

Check for grammatical errors, consistency in referencing, and overall clarity. This step is crucial to ensure your literature review is polished and professional.

ChatGPT prompts to Write a Literature Review

Prompts you can use when engaging with ChatGPT for research assistance in writing a literature review:

• “Can you provide an overview of the main theories and concepts related to [your research topic]?”
• “What are the current trends and developments in [your research field]?”
• “Can you suggest some key studies or research papers on [specific aspect of your research topic]?”
• “What are the main methodologies used in conducting research on [your research topic]?”
• “Can you provide a critical analysis of the existing literature on [your research question]?”
• “Are there any gaps or areas of controversy in the literature on [your research topic] that need further exploration?”
• “What are the key findings and conclusions from the most recent studies on [your research topic]?”
• “Can you suggest some reputable journals or publications explore for relevant literature in [your research field]?”
• “What are the different perspectives or schools of thought in the literature on [your research topic]?”
• “Can you provide a summary of the historical background and evolution of research on [your research topic]?”

Remember to provide clear and specific instructions in your prompts to guide ChatGPT in generating relevant and accurate content for your literature review.

Using ChatGPT to write a literature review can greatly facilitate the research process. By following a step-by-step approach, researchers can effectively leverage ChatGPT’s capabilities to gather insights, generate content, and enhance the quality of their literature review. However, it is important to approach the generated content critically, validate it with reliable sources, and ensure coherence within the review.

• Academic Writing
• AI writing tool
• literature review guide
• literature review tips
• research assistance
• research process
• research writing
• write literature review

Advantages and disadvantages of getting a patent

Working sci-hub proxy links – 2024, 24 best online plagiarism checker free – 2024, most popular, icmr call for research proposal 2024, call for applications: dst inspire faculty fellowship (2024), should you quit your phd explore reasons & alternatives, india – sri lanka joint research funding opportunity, how to check scopus indexed journals 2024, apply for the dst-jsps indo-japan call 2024, india-eu partner up for explainable and robust ai research, list of open access sci journals in computer science, best for you, what is phd, popular posts, scopus indexed journals list 2024, how to write a research paper in a month, popular category.

• POSTDOC 317
• Interesting 258
• Journals 234
• Fellowship 131
• Research Methodology 102
• All Scopus Indexed Journals 92

Mail Subscription

iLovePhD is a research education website to know updated research-related information. It helps researchers to find top journals for publishing research articles and get an easy manual for research tools. The main aim of this website is to help Ph.D. scholars who are working in various domains to get more valuable ideas to carry out their research. Learn the current groundbreaking research activities around the world, love the process of getting a Ph.D.

Contact us: [email protected]

Google News

Copyright © 2024 iLovePhD. All rights reserved

• Artificial intelligence

SYSTEMATIC REVIEW article

Are we there yet - a systematic literature review on chatbots in education.

• 1 Information Center for Education, DIPF | Leibniz Institute for Research and Information in Education, Frankfurt am Main, Germany
• 2 Educational Science Faculty, Open University of the Netherlands, Heerlen, Netherlands
• 3 Computer Science Faculty, Goethe University, Frankfurt am Main, Germany

Chatbots are a promising technology with the potential to enhance workplaces and everyday life. In terms of scalability and accessibility, they also offer unique possibilities as communication and information tools for digital learning. In this paper, we present a systematic literature review investigating the areas of education where chatbots have already been applied, explore the pedagogical roles of chatbots, the use of chatbots for mentoring purposes, and their potential to personalize education. We conducted a preliminary analysis of 2,678 publications to perform this literature review, which allowed us to identify 74 relevant publications for chatbots’ application in education. Through this, we address five research questions that, together, allow us to explore the current state-of-the-art of this educational technology. We conclude our systematic review by pointing to three main research challenges: 1) Aligning chatbot evaluations with implementation objectives, 2) Exploring the potential of chatbots for mentoring students, and 3) Exploring and leveraging adaptation capabilities of chatbots. For all three challenges, we discuss opportunities for future research.

Introduction

Educational Technologies enable distance learning models and provide students with the opportunity to learn at their own pace. They have found their way into schools and higher education institutions through Learning Management Systems and Massive Open Online Courses, enabling teachers to scale up good teaching practices ( Ferguson and Sharples, 2014 ) and allowing students to access learning material ubiquitously ( Virtanen et al., 2018 ).

Despite the innovative power of educational technologies, most commonly used technologies do not substantially change teachers’ role. Typical teaching activities like providing students with feedback, motivating them, or adapting course content to specific student groups are still entrusted exclusively to teachers, even in digital learning environments. This can lead to the teacher-bandwidth problem ( Wiley and Edwards, 2002 ), the result of a shortage of teaching staff to provide highly informative and competence-oriented feedback at large scale. Nowadays, however, computers and other digital devices open up far-reaching possibilities that have not yet been fully exploited. For example, incorporating process data can provide students with insights into their learning progress and bring new possibilities for formative feedback, self-reflection, and competence development ( Quincey et al., 2019 ). According to ( Hattie, 2009 ), feedback in terms of learning success has a mean effect size of d = 0.75, while ( Wisniewski et al., 2019 ) even report a mean effect of d = 0.99 for highly informative feedback. Such feedback provides suitable conditions for self-directed learning ( Winne and Hadwin, 2008 ) and effective metacognitive control of the learning process ( Nelson and Narens, 1994 ).

One of the educational technologies designed to provide actionable feedback in this regard is Learning Analytics. Learning Analytics is defined as the research area that focuses on collecting traces that learners leave behind and using those traces to improve learning ( Duval and Verbert, 2012 ; Greller and Drachsler, 2012 ). Learning Analytics can be used both by students to reflect on their own learning progress and by teachers to continuously assess the students’ efforts and provide actionable feedback. Another relevant educational technology is Intelligent Tutoring Systems. Intelligent Tutoring Systems are defined as computerized learning environments that incorporate computational models ( Graesser et al., 2001 ) and provide feedback based on learning progress. Educational technologies specifically focused on feedback for help-seekers, comparable to raising hands in the classroom, are Dialogue Systems and Pedagogical Conversational Agents ( Lester et al., 1997 ). These technologies can simulate conversational partners and provide feedback through natural language ( McLoughlin and Oliver, 1998 ).

Research in this area has recently focused on chatbot technology, a subtype of dialog systems, as several technological platforms have matured and led to applications in various domains. Chatbots incorporate generic language models extracted from large parts of the Internet and enable feedback by limiting themselves to text or voice interfaces. For this reason, they have also been proposed and researched for a variety of applications in education ( Winkler and Soellner, 2018 ). Recent literature reviews on chatbots in education ( Winkler and Soellner, 2018 ; Hobert, 2019a ; Hobert and Meyer von Wolff, 2019 ; Jung et al., 2020 ; Pérez et al., 2020 ; Smutny and Schreiberova, 2020 ; Pérez-Marín, 2021 ) have reported on such applications as well as design guidelines, evaluation possibilities, and effects of chatbots in education.

In this paper, we contribute to the state-of-the-art of chatbots in education by presenting a systematic literature review, where we examine so-far unexplored areas such as implementation objectives, pedagogical roles, mentoring scenarios, the adaptations of chatbots to learners, and application domains. This paper is structured as follows: First, we review related work (section 2), derive research questions from it, then explain the applied method for searching related studies (section 3), followed by the results (section 4), and finally, we discuss the findings (section 5) and point to future research directions in the field (section 5).

Related Work

In order to accurately cover the field of research and deal with the plethora of terms for chatbots in the literature (e.g. chatbot, dialogue system or pedagogical conversational agent) we propose the following definition:

Chatbots are digital systems that can be interacted with entirely through natural language via text or voice interfaces. They are intended to automate conversations by simulating a human conversation partner and can be integrated into software, such as online platforms, digital assistants, or be interfaced through messaging services.

Outside of education, typical applications of chatbots are in customer service ( Xu et al., 2017 ), counseling of hospital patients ( Vaidyam et al., 2019 ), or information services in smart speakers ( Ram et al., 2018 ). One central element of chatbots is the intent classification, also named the Natural Language Understanding (NLU) component, which is responsible for the sense-making of human input data. Looking at the current advances in chatbot software development, it seems that this technology’s goal is to pass the Turing Test ( Saygin et al., 2000 ) one day, which could make chatbots effective educational tools. Therefore, we ask ourselves “ Are we there yet? - Will we soon have an autonomous chatbot for every learner?”

To understand and underline the current need for research in the use of chatbots in education, we first examined the existing literature, focusing on comprehensive literature reviews. By looking at research questions in these literature reviews, we identified 21 different research topics and extracted findings accordingly. To structure research topics and findings in a comprehensible way, a three-stage clustering process was applied. While the first stage consisted of coding research topics by keywords, the second stage was applied to form overarching research categories ( Table 1 ). In the final stage, the findings within each research category were clustered to identify and structure commonalities within the literature reviews. The result is a concept map, which consists of four major categories. Those categories are CAT1. Applications of Chatbots, CAT2. Chatbot Designs, CAT3. Evaluation of Chatbots and CAT4. Educational Effects of Chatbots. To standardize the terminology and concepts applied, we present the findings of each category in a separate sub-section, respectively ( see Figure 1 , Figure 2 , Figure 3 , and Figure 4 ) and extended it with the outcomes of our own literature study that will be reported in the remaining parts of this article. Due to the size of the concept map a full version can be found in Appendix A .

TABLE 1 . Assignment of coded research topics identified in related literature reviews to research categories.

FIGURE 1 . Applications of chatbots in related literature reviews (CAT1).

FIGURE 2 . Chatbot designs in related literature reviews (CAT2).

FIGURE 3 . Evaluation of chatbots in related literature reviews (CAT3).

FIGURE 4 . Educational Effects of chatbots in related literature reviews (CAT4).

Regarding the applications of chatbots (CAT1), application clusters (AC) and application statistics (AS) have been described in the literature, which we visualized in Figure 1 . The study of ( Pérez et al., 2020 ) identifies two application clusters, defined through chatbot activities: “service-oriented chatbots” and “teaching-oriented chatbots.” ( Winkler and Soellner, 2018 ) identify applications clusters by naming the domains “health and well-being interventions,” “language learning,” “feedback and metacognitive thinking” as well as “motivation and self-efficacy.” Concerning application statistics (AS), ( Smutny and Schreiberova, 2020 ) found that nearly 47% of the analyzed chatbots incorporate informing actions, and 18% support language learning by elaborating on chatbots integrated into the social media platform Facebook. Besides, the chatbots studied had a strong tendency to use English, at 89%. This high number aligns with results from ( Pérez-Marín, 2021 ), where 75% of observed agents, as a related technology, were designed to interact in the English language. ( Pérez-Marín, 2021 ) also shows that 42% of the analyzed chatbots had mixed interaction modalities. Finally, ( Hobert and Meyer von Wolff, 2019 ) observed that only 25% of examined chatbots were incorporated in formal learning settings, the majority of published material focuses on student-chatbot interaction only and does not enable student-student communication, as well as nearly two-thirds of the analyzed chatbots center only on a single domain. Overall, we can summarize that so far there are six application clusters for chatbots for education categorized by chatbot activities or domains. The provided statistics allow for a clearer understanding regarding the prevalence of chatbots applications in education ( see Figure 1 ).

Regarding chatbot designs (CAT2), most of the research questions concerned with chatbots in education can be assigned to this category. We found three aspects in this category visualized in Figure 2 : Personality (PS), Process Pipeline (PP), and Design Classifications (DC). Within these, most research questions can be assigned to Design Classifications (DC), which are separated into Classification Aspects (DC2) and Classification Frameworks (DC1). One classification framework is defined through “flow chatbots,” “artificially intelligent chatbots,” “chatbots with integrated speech recognition,” as well as “chatbots with integrated context-data” by ( Winkler and Soellner, 2018 ). A second classification framework by ( Pérez-Marín, 2021 ) covers pedagogy, social, and HCI features of chatbots and agents, which themselves can be further subdivided into more detailed aspects. Other Classification Aspects (DC2) derived from several publications, provide another classification schema, which distinguishes between “retrieval vs. generative” based technology, the “ability to incorporate context data,” and “speech or text interface” ( Winkler and Soellner, 2018 ; Smutny and Schreiberova, 2020 ). By specifying text interfaces as “Button-Based” or “Keyword Recognition-Based” ( Smutny and Schreiberova, 2020 ), text interfaces can be subdivided. Furthermore, a comparison of speech and text interfaces ( Jung et al., 2020 ) shows that text interfaces have advantages for conveying information, and speech interfaces have advantages for affective support. The second aspect of CAT2 concerns the chatbot processing pipeline (PP), highlighting user interface and back-end importance ( Pérez et al., 2020 ). Finally, ( Jung et al., 2020 ) focuses on the third aspect, the personality of chatbots (PS). Here, the study derives four guidelines helpful in education: positive or neutral emotional expressions, a limited amount of animated or visual graphics, a well-considered gender of the chatbot, and human-like interactions. In summary, we have found in CAT2 three main design aspects for the development of chatbots. CAT2 is much more diverse than CAT1 with various sub-categories for the design of chatbots. This indicates the huge flexibility to design chatbots in various ways to support education.

Regarding the evaluation of chatbots (CAT3), we found three aspects assigned to this category, visualized in Figure 3 : Evaluation Criteria (EC), Evaluation Methods (EM), and Evaluation Instruments (EI). Concerning Evaluation Criteria, seven criteria can be identified in the literature. The first and most important in the educational field, according to ( Smutny and Schreiberova, 2020 ) is the evaluation of learning success ( Hobert, 2019a ), which can have subcategories such as how chatbots are embedded in learning scenarios ( Winkler and Soellner, 2018 ; Smutny and Schreiberova, 2020 ) and teaching efficiency ( Pérez et al., 2020 ). The second is acceptance, which ( Hobert, 2019a ) names as “acceptance and adoption” and ( Pérez et al., 2020 ) as “students’ perception.” Further evaluation criteria are motivation, usability, technical correctness, psychological, and further beneficial factors ( Hobert, 2019a ). These Evaluation Criteria show broad possibilities for the evaluation of chatbots in education. However, ( Hobert, 2019a ) found that most evaluations are limited to single evaluation criteria or narrower aspects of them. Moreover, ( Hobert, 2019a ) introduces a classification matrix for chatbot evaluations, which consists of the following Evaluation Methods (EM): Wizard-of-Oz approach, laboratory studies, field studies, and technical validations. In addition to this, ( Winkler and Soellner, 2018 ) recommends evaluating chatbots by their embeddedness into a learning scenario, a comparison of human-human and human-chatbot interactions, and comparing spoken and written communication. Instruments to measure these evaluation criteria were identified by ( Hobert, 2019a ) by naming quantitative surveys, qualitative interviews, transcripts of dialogues, and technical log files. Regarding CAT3, we found three main aspects for the evaluation of chatbots. We can conclude that this is a more balanced and structured distribution in comparison to CAT2, providing researchers with guidance for evaluating chatbots in education.

Regarding educational effects of chatbots (CAT4), we found two aspects visualized in Figure 4 : Effect Size (ES) and Beneficial Chatbot Features for Learning Success (BF). Concerning the effect size, ( Pérez et al., 2020 ) identified a strong dependency between learning and the related curriculum, while ( Winkler and Soellner, 2018 ) elaborate on general student characteristics that influence how students interact with chatbots. They state that students’ attitudes towards technology, learning characteristics, educational background, self-efficacy, and self-regulation skills affect these interactions. Moreover, the study emphasizes chatbot features, which can be regarded as beneficial in terms of learning outcomes (BF): “Context-Awareness,” “Proactive guidance by students,” “Integration in existing learning and instant messaging tools,” “Accessibility,” and “Response Time.” Overall, for CAT4, we found two main distinguishing aspects for chatbots, however, the reported studies vary widely in their research design, making high-level results hardly comparable.

Looking at the related work, many research questions for the application of chatbots in education remain. Therefore, we selected five goals to be further investigated in our literature review. Firstly, we were interested in the objectives for implementing chatbots in education (Goal 1), as the relevance of chatbots for applications within education seems to be not clearly delineated. Secondly, we aim to explore the pedagogical roles of chatbots in the existing literature (Goal 2) to understand how chatbots can take over tasks from teachers. ( Winkler and Soellner, 2018 ) and ( Pérez-Marín, 2021 ), identified research gaps for supporting meta-cognitive skills with chatbots such as self-regulation. This requires a chatbot application that takes a mentoring role, as the development of these meta-cognitive skills can not be achieved solely by information delivery. Within our review we incorporate this by reviewing the mentoring role of chatbots as (Goal 3). Another key element for a mentoring chatbot is adaptation to the learners needs. Therefore, (Goal 4) of our review lies in the investigation of the adaptation approaches used by chatbots in education. For (Goal 5), we want to extend the work of ( Winkler and Soellner, 2018 ) and ( Pérez et al., 2020 ) regarding Application Clusters (AC) and map applications by further investigating specific learning domains in which chatbots have been studied.

To delineate and map the field of chatbots in education, initial findings were collected by a preliminary literature search. One of the takeaways is that the emerging field around educational chatbots has seen much activity in the last two years. Based on the experience of this preliminary search, search terms, queries, and filters were constructed for the actual structured literature review. This structured literature review follows the PRISMA framework ( Liberati et al., 2009 ), a guideline for reporting systematic reviews and meta-analyses. The framework consists of an elaborated structure for systematic literature reviews and sets requirements for reporting information about the review process ( see section 3.2 to 3.4).

Research Questions

Contributing to the state-of-the-art, we investigate five aspects of chatbot applications published in the literature. We therefore guided our research with the following research questions:

RQ1: Which objectives for implementing chatbots in education can be identified in the existing literature?

RQ2: Which pedagogical roles of chatbots can be identified in the existing literature?

RQ3: Which application scenarios have been used to mentor students?

RQ4: To what extent are chatbots adaptable to personal students’ needs?

RQ5: What are the domains in which chatbots have been applied so far?

Sources of Information

As data sources, Scopus, Web of Science, Google Scholar, Microsoft Academics, and the educational research database “Fachportal Pädagogik” (including ERIC) were selected, all of which incorporate all major publishers and journals. In ( Martín-Martín et al., 2018 ) it was shown that for the social sciences only 29.8% and for engineering and computer science, 46.8% of relevant literature is included in all of the first three databases. For the topic of chatbots in education, a value between these two numbers can be assumed, which is why an approach of integrating several publisher-independent databases was employed here.

Search Criteria

Based on the findings from the initial related work search, we derived the following search query:

( Education OR Educational OR Learning OR Learner OR Student OR Teaching OR School OR University OR Pedagogical ) AND Chatbot.

It combines education-related keywords with the “chatbot” keyword. Since chatbots are related to other technologies, the initial literature search also considered keywords such as “pedagogical agents,” “dialogue systems,” or “bots” when composing the search query. However, these increased the number of irrelevant results significantly and were therefore excluded from the query in later searches.

Inclusion and Exclusion Criteria

The queries were executed on 23.12.2020 and applied twice to each database, first as a title search query and secondly as a keyword-based search. This resulted in a total of 3.619 hits, which were checked for duplicates resulting in 2.678 candidate publications. The overall search and filtering process is shown in Figure 5 .

FIGURE 5 . PRISMA flow chart.

In the case of Google Scholar, the number of results sorted by relevance per query was limited to 300, as this database also delivers many less relevant works. The value was determined by looking at the search results in detail using several queries to exclude as few relevant works as possible. This approach showed promising results and, at the same time, did not burden the literature list with irrelevant items.

The further screening consisted of a four-stage filtering process. First, eliminating duplicates in the results of title and keyword queries of all databases independently and second, excluding publications based on the title and abstract that:

• were not available in English

• did not describe a chatbot application

• were not mainly focused on learner-centered chatbots applications in schools or higher education institutions, which is according to the preliminary literature search the main application area within education.

Third, we applied another duplicate filter, this time for the merged set of publications. Finally, a filter based on the full text, excluding publications that were:

• limited to improve chatbots technically (e.g., publications that compare or develop new algorithms), as research questions presented in these publications were not seeking for additional insights on applications in education

• exclusively theoretical in nature (e.g., publications that discuss new research projects, implementation concepts, or potential use cases of chatbots in education), as they either do not contain research questions or hypotheses or do not provide conclusions from studies with learners.

After the first, second, and third filters, we identified 505 candidate publications. We continued our filtering process by reading the candidate publications’ full texts resulting in 74 publications that were used for our review. Compared to 3.619 initial database results, the proportion of relevant publications is therefore about 2.0%.

The final publication list can be accessed under https://bit.ly/2RRArFT .

To analyze the identified publications and derive results according to the research questions, full texts were coded, considering for each publication the objectives for implementing chatbots (RQ1), pedagogical roles of chatbots (RQ2), their mentoring roles (RQ3), adaptation of chatbots (RQ4), as well as their implementation domains in education (RQ5) as separated sets of codes. To this end, initial codes were identified by open coding and iteratively improved through comparison, group discussion among the authors, and subsequent code expansion. Further, codes were supplemented with detailed descriptions until a saturation point was reached, where all included studies could be successfully mapped to codes, suggesting no need for further refinement. As an example, codes for RQ2 (Pedagogical Roles) were adapted and refined in terms of their level of abstraction from an initial set of only two codes, 1 ) a code for chatbots in the learning role and 2 ) a code for chatbots in a service-oriented role. After coding a larger set of publications, it became clear that the code for service-oriented chatbots needed to be further distinguished. This was because it summarized e.g. automation activities with activities related to self-regulated learning and thus could not be distinguished sharply enough from the learning role. After refining the code set in the next iteration into a learning role, an assistance role, and a mentoring role, it was then possible to ensure the separation of the individual codes. In order to avoid defining new codes for singular or a very small number of publications, studies were coded as “other” (RQ1) or “not defined” (RQ2), if their occurrence was less than eight publications, representing less than 10% of the publications in the final paper list.

By grouping the resulting relevant publications according to their date of publication, it is apparent that chatbots in education are currently in a phase of increased attention. The release distribution shows slightly lower publication numbers in the current than in the previous year ( Figure 6 ), which could be attributed to a time lag between the actual publication of manuscripts and their dissemination in databases.

FIGURE 6 . Identified chatbot publications in education per year.

Applying the curve presented in Figure 6 to Gartner’s Hype Cycle ( Linden and Fenn, 2003 ) suggests that technology around chatbots in education may currently be in the “Innovation Trigger” phase. This phase is where many expectations are placed on the technology, but the practical in-depth experience is still largely lacking.

Objectives for Implementing Chatbots in Education

Regarding RQ1, we extracted implementation objectives for chatbots in education. By analyzing the selected publications we identified that most of the objectives for chatbots in education can be described by one of the following categories: Skill improvement, Efficiency of Education, and Students’ Motivation ( see Figure 7 ). First, the “improvement of a student’s skill” (or Skill Improvement ) objective that the chatbot is supposed to help with or achieve. Here, chatbots are mostly seen as a learning aid that supports students. It is the most commonly cited objective for chatbots. The second objective is to increase the Efficiency of Education in general. It can occur, for example, through the automation of recurring tasks or time-saving services for students and is the second most cited objective for chatbots. The third objective is to increase Students’ Motivation . Finally, the last objective is to increase the Availability of Education . This objective is intended to provide learning or counseling with temporal flexibility or without the limitation of physical presence. In addition, there are other, more diverse objectives for chatbots in education that are less easy to categorize. In cases of a publication indicating more than one objective, the publication was distributed evenly across the respective categories.

FIGURE 7 . Objectives for implementing chatbots identified in chatbot publications.

Given these results, we can summarize four major implementing objectives for chatbots. Of these, Skill Improvement is the most popular objective, constituting around one-third of publications (32%). Making up a quarter of all publications, Efficiency of Education is the second most popular objective (25%), while addressing Students’ Motivation and Availability of Education are third (13%) and fourth (11%), respectively. Other objectives also make up a substantial amount of these publications (19%), although they were too diverse to categorize in a uniform way. Examples of these are inclusivity ( Heo and Lee, 2019 ) or the promotion of student teacher interactions ( Mendoza et al., 2020 ).

Pedagogical Roles

Regarding RQ2, it is crucial to consider the use of chatbots in terms of their intended pedagogical role. After analyzing the selected articles, we were able to identify four different pedagogical roles: a supporting learning role, an assisting role, and a mentoring role.

In the supporting learning role ( Learning ), chatbots are used as an educational tool to teach content or skills. This can be achieved through a fixed integration into the curriculum, such as conversation tasks (L. K. Fryer et al., 2020 ). Alternatively, learning can be supported through additional offerings alongside classroom teaching, for example, voice assistants for leisure activities at home ( Bao, 2019 ). Examples of these are chatbots simulating a virtual pen pal abroad ( Na-Young, 2019 ). Conversations with this kind of chatbot aim to motivate the students to look up vocabulary, check their grammar, and gain confidence in the foreign language.

In the assisting role ( Assisting ), chatbot actions can be summarized as simplifying the student's everyday life, i.e., taking tasks off the student’s hands in whole or in part. This can be achieved by making information more easily available ( Sugondo and Bahana, 2019 ) or by simplifying processes through the chatbot’s automation ( Suwannatee and Suwanyangyuen, 2019 ). An example of this is the chatbot in ( Sandoval, 2018 ) that answers general questions about a course, such as an exam date or office hours.

In the mentoring role ( Mentoring ), chatbot actions deal with the student’s personal development. In this type of support, the student himself is the focus of the conversation and should be encouraged to plan, reflect or assess his progress on a meta-cognitive level. One example is the chatbot in ( Cabales, 2019 ), which helps students develop lifelong learning skills by prompting in-action reflections.

The distribution of each pedagogical role is shown in Figure 8 . From this, it can be seen that Learning is the most frequently used role of the examined publications (49%), followed by Assisting (20%) and Mentoring (15%). It should be noted that pedagogical roles were not identified for all the publications examined. The absence of a clearly defined pedagogical role (16%) can be attributed to the more general nature of these publications, e.g. focused on students’ small talk behaviors ( Hobert, 2019b ) or teachers’ attitudes towards chatbot applications in classroom teaching (P. K. Bii et al., 2018 ).

FIGURE 8 . Pedagogical roles identified in chatbot publications.

Looking at pedagogical roles in the context of objectives for implementing chatbots, relations among publications can be inspected in a relations graph ( Figure 9 ). According to our results, the strongest relation in the examined publications can be considered between Skill Improvement objective and the Learning role. This strong relation is partly because both, the Skill Improvement objective and the Learning role, are the largest in their respective categories. In addition, two other strong relations can be observed: Between the Students’ Motivation objective and the Learning role, as well as between Efficiency of Education objective and Assisting role.

FIGURE 9 . Relations graph of pedagogical roles and objectives for implementing chatbots.

By looking at other relations in more detail, there is surprisingly no relation between Skill Improvement as the most common implementation objective and Assisting , as the 2nd most common pedagogical role. Furthermore, it can be observed that the Mentoring role has nearly equal relations to all of the objectives for implementing chatbots.

The relations graph ( Figure 9 ) can interactively be explored through bit.ly/32FSKQM.

Mentoring Role

Regarding RQ3, we identified eleven publications that deal with chatbots in this regard. The Mentoring role in these publications can be categorized in two dimensions. Starting with the first dimension, the mentoring method, three methods can be observed:

• Scaffolding ( n = 7)

• Recommending ( n = 3)

• Informing ( n = 1)

An example of Scaffolding can be seen in ( Gabrielli et al., 2020 ), where the chatbot coaches students in life skills, while an example of Recommending can be seen in ( Xiao et al., 2019 ), where the chatbot recommends new teammates. Finally, Informing can be seen in ( Kerly et al., 2008 ), where the chatbot informs students about their personal Open Learner Model.

The second dimension is the addressed mentoring topic, where the following topics can be observed:

• Self-Regulated Learning ( n = 5)

• Life Skills ( n = 4)

• Learning Skills ( n = 2)

While Mentoring chatbots to support Self-Regulated Learning are intended to encourage students to reflect on and plan their learning progress, Mentoring chatbots to support Life Skills address general student’s abilities such as self-confidence or managing emotions. Finally, Mentoring chatbots to support Learning Skills , in contrast to Self-Regulated Learning , address only particular aspects of the learning process, such as new learning strategies or helpful learning partners. An example for Mentoring chatbots supporting Life Skill is the Logo counseling chatbot, which promotes healthy self-esteem ( Engel et al., 2020 ). CALMsystem is an example of a Self-Regulated Learning chatbot, which informs students about their data in an open learner model ( Kerly et al., 2008 ). Finally, there is the Learning Skills topic. Here, the MCQ Bot is an example that is designed to introduce students to transformative learning (W. Huang et al., 2019 ).

Regarding RQ4, we identified six publications in the final publication list that address the topic of adaptation. Within these publications, five adaptation approaches are described:

The first approach (A1) is proposed by ( Kerly and Bull, 2006 ) and ( Kerly et al., 2008 ), dealing with student discussions based on success and confidence during a quiz. The improvement of self-assessment is the primary focus of this approach. The second approach (A2) is presented in ( Jia, 2008 ), where the personality of the chatbot is adapted to motivate students to talk to the chatbot and, in this case, learn a foreign language. The third approach (A3), as shown in the work of ( Vijayakumar et al., 2019 ), is characterized by a chatbot that provides personalized formative feedback to learners based on their self-assessment, again in a quiz situation. Here, the focus is on Hattie and Timperley’s three guiding questions: “Where am I going?,” “How am I going?” and “Where to go next?” ( Hattie and Timperley, 2007 ). In the fourth approach (A4), exemplified in ( Ruan et al., 2019 ), the chatbot selects questions within a quiz. Here, the chatbot estimates the student’s ability and knowledge level based on the quiz progress and sets the next question accordingly. Finally, a similar approach (A5) is shown in ( Davies et al., 2020 ). In contrast to ( Ruan et al., 2019 ), this chatbot adapts the amount of question variation and takes psychological features into account which were measured by psychological tests before.

We examined these five approaches by organizing them according to their information sources and extracted learner information. The results can be seen in Table 2 .

TABLE 2 . Adaptation approaches of chatbots in education.

Four out of five adaptation approaches (A1, A3, A4, and A5) are observed in the context of quizzes. These adaptations within quizzes can be divided into two mainstreams: One is concerned about students’ feedback (A1 and A3), while the other is concerned about learning material selection (A4 and A5). The only different adaptation approach is shown in A2, which focuses on the adaptation of the chatbot personality within a language learning application.

Domains for Chatbots in Education

Regarding RQ5, we identified 20 domains of chatbots in education. These can broadly be divided by their pedagogical role into three domain categories (DC): Learning Chatbots , Assisting Chatbots , and Mentoring Chatbots . The remaining publications are grouped in the Other Research domain category. The complete list of identified domains can be seen in Table 3 .

TABLE 3 . Domains of chatbots in education.

The domain category Learning Chatbots , which deals with chatbots incorporating the pedagogical role Learning , can be subdivided into seven domains: 1 ) Language Learning , 2 ) Learn to Program , 3 ) Learn Communication Skills , 4 ) Learn about Educational Technologies , 5 ) Learn about Cultural Heritage , 6 ) Learn about Laws , and 7 ) Mathematics Learning . With more than half of publications (53%), chatbots for Language Learning play a prominent role in this domain category. They are often used as chat partners to train conversations or to test vocabulary. An example of this can be seen in the work of ( Bao, 2019 ), which tries to mitigate foreign language anxiety by chatbot interactions in foreign languages.

The domain category Assisting Chatbots , which deals with chatbots incorporating the pedagogical role Assisting , can be subdivided into four domains: 1 ) Administrative Assistance , 2 ) Campus Assistance , 3 ) Course Assistance , and 4 ) Library Assistance . With one-third of publications (33%), chatbots in the Administrative Assistance domain that help to overcome bureaucratic hurdles at the institution, while providing round-the-clock services, are the largest group in this domain category. An example of this can be seen in ( Galko et al., 2018 ), where the student enrollment process is completely shifted to a conversation with a chatbot.

The domain category Mentoring Chatbots , which deals with chatbots incorporating the pedagogical role Mentoring , can be subdivided into three domains: 1 ) Scaffolding Chatbots , 2 ) Recommending Chatbots , and 3 ) Informing Chatbots . An example of a Scaffolding Chatbots is the CRI(S) chatbot ( Gabrielli et al., 2020 ), which supports life skills such as self-awareness or conflict resolution in discussion with the student by promoting helpful ideas and tricks.

The domain category Other Research , which deals with chatbots not incorporating any of these pedagogical roles, can be subdivided into three domains: 1 ) General Chatbot Research in Education , 2 ) Indian Educational System , and 3 ) Chatbot Interfaces . The most prominent domain, General Chatbot Research , cannot be classified in one of the other categories but aims to explore cross-cutting issues. An example for this can be seen in the publication of ( Hobert, 2020 ), which researches the importance of small talk abilities of chatbots in educational settings.

Discussions

In this paper, we investigated the state-of-the-art of chatbots in education according to five research questions. By combining our results with previously identified findings from related literature reviews, we proposed a concept map of chatbots in education. The map, reported in Appendix A , displays the current state of research regarding chatbots in education with the aim of supporting future research in the field.

Answer to Research Questions

Concerning RQ1 (implementation objectives), we identified four major objectives: 1 ) Skill Improvement , 2 ) Efficiency of Education , 3 ) Students’ Motivation, and 4 ) Availability of Education . These four objectives cover over 80% of the analyzed publications ( see Figure 7 ). Based on the findings on CAT3 in section 2, we see a mismatch between the objectives for implementing chatbots compared to their evaluation. Most researchers only focus on narrow aspects for the evaluation of their chatbots such as learning success, usability, and technology acceptance. This mismatch of implementation objectives and suitable evaluation approaches is also well known by other educational technologies such as Learning Analytics dashboards ( Jivet et al., 2017 ). A more structured approach of aligning implementation objectives and evaluation procedures is crucial to be able to properly assess the effectiveness of chatbots. ( Hobert, 2019a ), suggested a structured four-stage evaluation procedure beginning with a Wizard-of-Oz experiment, followed by technical validation, a laboratory study, and a field study. This evaluation procedure systematically links hypotheses with outcomes of chatbots helping to assess chatbots for their implementation objectives. “Aligning chatbot evaluations with implementation objectives” is, therefore, an important challenge to be addressed in the future research agenda.

Concerning RQ2 (pedagogical roles), our results show that chatbots’ pedagogical roles can be summarized as Learning , Assisting , and Mentoring . The Learning role is the support in learning or teaching activities such as gaining knowledge. The Assisting role is the support in terms of simplifying learners’ everyday life, e.g. by providing opening times of the library. The Mentoring role is the support in terms of students’ personal development, e.g. by supporting Self-Regulated Learning. From a pedagogical standpoint, all three roles are essential for learners and should therefore be incorporated in chatbots. These pedagogical roles are well aligned with the four implementation objectives reported in RQ1. While Skill Improvement and Students’ Motivation is strongly related to Learning , Efficiency of Education is strongly related to Assisting . The Mentoring role instead, is evenly related to all of the identified objectives for implementing chatbots. In the reviewed publications, chatbots are therefore primarily intended to 1 ) improve skills and motivate students by supporting learning and teaching activities, 2 ) make education more efficient by providing relevant administrative and logistical information to learners, and 3 ) support multiple effects by mentoring students.

Concerning RQ3 (mentoring role), we identified three main mentoring method categories for chatbots: 1 ) Scaffolding , 2 ) Recommending , and 3 ) Informing . However, comparing the current mentoring of chatbots reported in the literature with the daily mentoring role of teachers, we can summarize that the chatbots are not at the same level. In order to take over mentoring roles of teachers ( Wildman et al., 1992 ), a chatbot would need to fulfill some of the following activities in their mentoring role. With respect to 1 ) Scaffolding , chatbots should provide direct assistance while learning new skills and especially direct beginners in their activities. Regarding 2 ) Recommending , chatbots should provide supportive information, tools or other materials for specific learning tasks to life situations. With respect to 3 ) Informing, chatbots should encourage students according to their goals and achievements, and support them to develop meta-cognitive skills like self-regulation. Due to the mismatch of teacher vs. chatbot mentoring we see here another research challenge, which we call “Exploring the potential of chatbots for mentoring students.”

Regarding RQ4 (adaptation), only six publications were identified that discuss an adaptation of chatbots, while four out of five adaptation approaches (A1, A3, A4, and A5) show similarities by being applied within quizzes. In the context of educational technologies, providing reasonable adaptations for learners requires a high level of experience. Based on our results, the research on chatbots does not seem to be at this point yet. Looking at adaptation literature like ( Brusilovsky, 2001 ) or ( Benyon and Murray, 1993 ), it becomes clear that a chatbot needs to consider the learners’ personal information to fulfill the requirement of the adaptation definition. Personal information must be retrieved and stored at least temporarily, in some sort of learner model. For learner information like knowledge and interest, adaptations seem to be barely explored in the reviewed publications, while the model of ( Brusilovsky and Millán, 2007 ) points out further learner information, which can be used to make chatbots more adaptive: personal goals, personal tasks, personal background, individual traits, and the learner’s context. We identify research in this area as a third future challenge and call it the “Exploring and leveraging adaptation capabilities of chatbots” challenge.

In terms of RQ5 (domains), we identified a detailed map of domains applying chatbots in education and their distribution ( see Table 3 ). By systematically analyzing 74 publications, we identified 20 domains and structured them according to the identified pedagogical role into four domain categories: Learning Chatbots , Assisting Chatbots , Mentoring Chatbots , and Other Research . These results extend the taxonomy of Application Clusters (AC) for chatbots in education, which previously comprised the work from ( Pérez et al., 2020 ), who took the chatbot activity as characteristic, and ( Winkler and Soellner, 2018 ), who characterized the chatbots by domains. It draws relationships between these two types of Application Clusters (AC) and structures them accordingly. Our structure incorporates Mentoring Chatbots and Other Research in addition to the “service-oriented chatbots” (cf. Assisting Chatbots ) and “teaching-oriented chatbots” (cf. Learning Chatbots ) identified by (Perez). Furthermore, the strong tendencies of informing students already mentioned by ( Smutny and Schreiberova, 2020 ) can also be recognized in our results, especially in Assisting Chatbots . Compared to ( Winkler and Soellner, 2018 ), we can confirm the prominent domains of “language learning” within Learning Chatbots and “metacognitive thinking” within Mentoring Chatbots . Moreover, through Table 3 , a more detailed picture of chatbot applications in education is reflected, which could help researchers to find similar works or unexplored application areas.

Limitations

One important limitation to be mentioned here is the exclusion of alternative keywords for our search queries, as we exclusively used chatbot as keyword in order to avoid search results that do not fit our research questions. Though we acknowledge that chatbots share properties with pedagogical agents, dialog systems, and bots, we carefully considered this trade-off between missing potentially relevant work and inflating our search procedure by including related but not necessarily pertinent work. A second limitation may lie in the formation of categories and coding processes applied, which, due to the novelty of the findings, could not be built upon theoretical frameworks or already existing code books. Although we have focused on ensuring that codes used contribute to a strong understanding, the determination of the abstraction level might have affected the level of detail of the resulting data representation.

In this systematic literature review, we explored the current landscape of chatbots in education. We analyzed 74 publications, identified 20 domains of chatbots and grouped them based on their pedagogical roles into four domain categories. These pedagogical roles are the supporting learning role ( Learning ), the assisting role ( Assisting ), and the mentoring role ( Mentoring ). By focusing on objectives for implementing chatbots, we identified four main objectives: 1 ) Skill Improvement , 2 ) Efficiency of Education , 3 ) Students’ Motivation, and 4 ) Availability of Education . As discussed in section 5, these objectives do not fully align with the chosen evaluation procedures. We focused on the relations between pedagogical roles and objectives for implementing chatbots and identified three main relations: 1 ) chatbots to improve skills and motivate students by supporting learning and teaching activities, 2 ) chatbots to make education more efficient by providing relevant administrative and logistical information to learners, and 3 ) chatbots to support multiple effects by mentoring students. We focused on chatbots incorporating the Mentoring role and found that these chatbots are mostly concerned with three mentoring topics 1 ) Self-Regulated Learning , 2 ) Life Skills , and 3 ) Learning Skills and three mentoring methods 1 ) Scaffolding , 2 ) Recommending , and 3 ) Informing . Regarding chatbot adaptations, only six publications with adaptations were identified. Furthermore, the adaptation approaches found were mostly limited to applications within quizzes and thus represent a research gap.

Based on these outcomes we consider three challenges for chatbots in education that offer future research opportunities:

Challenge 1: Aligning chatbot evaluations with implementation objectives . Most chatbot evaluations focus on narrow aspects to measure the tool’s usability, acceptance or technical correctness. If chatbots should be considered as learning aids, student mentors, or facilitators, the effects on the cognitive, and emotional levels should also be taken into account for the evaluation of chatbots. This finding strengthens our conclusion that chatbot development in education is still driven by technology, rather than having a clear pedagogical focus of improving and supporting learning.

Challenge 2: Exploring the potential of chatbots for mentoring students . In order to better understand the potentials of chatbots to mentor students, more empirical studies on the information needs of learners are required. It is obvious that these needs differ from schools to higher education. However, so far there are hardly any studies investigating the information needs with respect to chatbots nor if chatbots address these needs sufficiently.

Challenge 3: Exploring and leveraging adaptation capabilities of chatbots . There is a large literature on adaptation capabilities of educational technologies. However, we have seen very few studies on the effect of adaptation of chatbots for education purposes. As chatbots are foreseen as systems that should personally support learners, the area of adaptable interactions of chatbots is an important research aspect that should receive more attention in the near future.

By addressing these challenges, we believe that chatbots can become effective educational tools capable of supporting learners with informative feedback. Therefore, looking at our results and the challenges presented, we conclude, “No, we are not there yet!” - There is still much to be done in terms of research on chatbots in education. Still, development in this area seems to have just begun to gain momentum and we expect to see new insights in the coming years.

Data Availability Statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding authors.

Author Contributions

SW, JS†, DM†, JW†, MR, and HD.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Abbasi, S., Kazi, H., and Hussaini, N. N. (2019). Effect of Chatbot Systems on Student’s Learning Outcomes. Sylwan 163(10).

CrossRef Full Text

Abbasi, S., and Kazi, H. (2014). Measuring Effectiveness of Learning Chatbot Systems on Student's Learning Outcome and Memory Retention. Asian J. Appl. Sci. Eng. 3, 57. doi:10.15590/AJASE/2014/V3I7/53576

CrossRef Full Text | Google Scholar

Almahri, F. A. J., Bell, D., and Merhi, M. (2020). “Understanding Student Acceptance and Use of Chatbots in the United Kingdom Universities: A Structural Equation Modelling Approach,” in 2020 6th IEEE International Conference on Information Management, ICIM 2020 , London, United Kingdom , March 27–29, 2020 , (IEEE), 284–288. doi:10.1109/ICIM49319.2020.244712

Bao, M. (2019). Can Home Use of Speech-Enabled Artificial Intelligence Mitigate Foreign Language Anxiety - Investigation of a Concept. Awej 5, 28–40. doi:10.24093/awej/call5.3

Benyon, D., and Murray, D. (1993). Applying User Modeling to Human-Computer Interaction Design. Artif. Intell. Rev. 7 (3-4), 199–225. doi:10.1007/BF00849555

Bii, P. K., Too, J. K., and Mukwa, C. W. (2018). Teacher Attitude towards Use of Chatbots in Routine Teaching. Univers. J. Educ. Res. . 6 (7), 1586–1597. doi:10.13189/ujer.2018.060719

Bii, P., Too, J., and Langat, R. (2013). An Investigation of Student’s Attitude Towards the Use of Chatbot Technology in Instruction: The Case of Knowie in a Selected High School. Education Research 4, 710–716. doi:10.14303/er.2013.231

Google Scholar

Bos, A. S., Pizzato, M. C., Vettori, M., Donato, L. G., Soares, P. P., Fagundes, J. G., et al. (2020). Empirical Evidence During the Implementation of an Educational Chatbot with the Electroencephalogram Metric. Creative Education 11, 2337–2345. doi:10.4236/CE.2020.1111171

Brusilovsky, P. (2001). Adaptive Hypermedia. User Model. User-Adapted Interaction 11 (1), 87–110. doi:10.1023/a:1011143116306

Brusilovsky, P., and Millán, E. (2007). “User Models for Adaptive Hypermedia and Adaptive Educational Systems,” in The Adaptive Web: Methods and Strategies of Web Personalization . Editors P. Brusilovsky, A. Kobsa, and W. Nejdl. Berlin: Springer , 3–53. doi:10.1007/978-3-540-72079-9_1

Cabales, V. (2019). “Muse: Scaffolding metacognitive reflection in design-based research,” in CHI EA’19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems , Glasgow, Scotland, United Kingdom , May 4–9, 2019 , (ACM), 1–6. doi:10.1145/3290607.3308450

Carayannopoulos, S. (2018). Using Chatbots to Aid Transition. Int. J. Info. Learn. Tech. 35, 118–129. doi:10.1108/IJILT-10-2017-0097

Chan, C. H., Lee, H. L., Lo, W. K., and Lui, A. K.-F. (2018). Developing a Chatbot for College Student Programme Advisement. in 2018 International Symposium on Educational Technology, ISET 2018 , Osaka, Japan , July 31–August 2, 2018 . Editors F. L. Wang, C. Iwasaki, T. Konno, O. Au, and C. Li, (IEEE), 52–56. doi:10.1109/ISET.2018.00021

Chang, M.-Y., and Hwang, J.-P. (2019). “Developing Chatbot with Deep Learning Techniques for Negotiation Course,” in 2019 8th International Congress on Advanced Applied Informatics, IIAI-AAI 2019 , Toyama, Japan , July 7–11, 2019 , (IEEE), 1047–1048. doi:10.1109/IIAI-AAI.2019.00220

Chen, C.-A., Yang, Y.-T., Wu, S.-M., Chen, H.-C., Chiu, K.-C., Wu, J.-W., et al. (2018). “A Study of Implementing AI Chatbot in Campus Consulting Service”, in TANET 2018-Taiwan Internet Seminar , 1714–1719. doi:10.6861/TANET.201810.0317

Chen, H.-L., Widarso, G. V., and Sutrisno, H. (2020). A ChatBot for Learning Chinese: Learning Achievement and Technology Acceptance. J. Educ. Comput. Res. 58 (6), 1161–1189. doi:10.1177/0735633120929622

Daud, S. H. M., Teo, N. H. I., and Zain, N. H. M. (2020). E-java Chatbot for Learning Programming Language: A post-pandemic Alternative Virtual Tutor. Int. J. Emerging Trends Eng. Res. 8(7). 3290–3298. doi:10.30534/ijeter/2020/67872020

Davies, J. N., Verovko, M., Verovko, O., and Solomakha, I. (2020). “Personalization of E-Learning Process Using Ai-Powered Chatbot Integration,” in Selected Papers of 15th International Scientific-practical Conference, MODS, 2020: Advances in Intelligent Systems and Computing , Chernihiv, Ukraine , June 29–July 01, 2020 . Editors S. Shkarlet, A. Morozov, and A. Palagin, ( Springer ) Vol. 1265, 209–216. doi:10.1007/978-3-030-58124-4_20

Diachenko, A. V., Morgunov, B. P., Melnyk, T. P., Kravchenko, O. I., and Zubchenko, L. V. (2019). The Use of Innovative Pedagogical Technologies for Automation of the Specialists' Professional Training. Int. J. Hydrogen. Energy. 8, 288–295. doi:10.5430/ijhe.v8n6p288

Dibitonto, M., Leszczynska, K., Tazzi, F., and Medaglia, C. M. (2018). “Chatbot in a Campus Environment: Design of Lisa, a Virtual Assistant to Help Students in Their university Life,” in 20th International Conference, HCI International 2018 , Las Vegas, NV, USA , July 15–20, 2018 , Lecture Notes in Computer Science. Editors M. Kurosu, (Springer), 103–116. doi:10.1007/978-3-319-91250-9

Durall, E., and Kapros, E. (2020). “Co-design for a Competency Self-Assessment Chatbot and Survey in Science Education,” in 7th International Conference, LCT 2020, Held as Part of the 22nd HCI International Conference, HCII 2020 , Copenhagen, Denmark , July 19–24, 2020 , Lecture Notes in Computer Science. Editors P. Zaphiris, and A. Ioannou, Berlin: Springer Vol. 12206, 13–23. doi:10.1007/978-3-030-50506-6_2

Duval, E., and Verbert, K. (2012). Learning Analytics. Eleed 8 (1).

Engel, J. D., Engel, V. J. L., and Mailoa, E. (2020). Interaction Monitoring Model of Logo Counseling Website for College Students' Healthy Self-Esteem, I. J. Eval. Res. Educ. 9, 607–613. doi:10.11591/ijere.v9i3.20525

Febriani, G. A., and Agustia, R. D. (2019). Development of Line Chatbot as a Learning Media for Mathematics National Exam Preparation. Elibrary.Unikom.Ac.Id . https://elibrary.unikom.ac.id/1130/14/UNIKOM_GISTY%20AMELIA%20FEBRIANI_JURNAL%20DALAM%20BAHASA%20INGGRIS.pdf .

Ferguson, R., and Sharples, M. (2014). “Innovative Pedagogy at Massive Scale: Teaching and Learning in MOOCs,” in 9th European Conference on Technology Enhanced Learning, EC-TEL 2014 , Graz, Austria , September 16–19, 2014 , Lecture Notes in Computer Science. Editors C. Rensing, S. de Freitas, T. Ley, and P. J. Muñoz-Merino, ( Berlin : Springer) Vol. 8719, 98–111. doi:10.1007/978-3-319-11200-8_8

Fryer, L. K., Ainley, M., Thompson, A., Gibson, A., and Sherlock, Z. (2017). Stimulating and Sustaining Interest in a Language Course: An Experimental Comparison of Chatbot and Human Task Partners. Comput. Hum. Behav. 75, 461–468. doi:10.1016/j.chb.2017.05.045

Fryer, L. K., Nakao, K., and Thompson, A. (2019). Chatbot Learning Partners: Connecting Learning Experiences, Interest and Competence. Comput. Hum. Behav. 93, 279–289. doi:10.1016/j.chb.2018.12.023

Fryer, L. K., Thompson, A., Nakao, K., Howarth, M., and Gallacher, A. (2020). Supporting Self-Efficacy Beliefs and Interest as Educational Inputs and Outcomes: Framing AI and Human Partnered Task Experiences. Learn. Individual Differences , 80. doi:10.1016/j.lindif.2020.101850

Gabrielli, S., Rizzi, S., Carbone, S., and Donisi, V. (2020). A Chatbot-Based Coaching Intervention for Adolescents to Promote Life Skills: Pilot Study. JMIR Hum. Factors 7 (1). doi:10.2196/16762

PubMed Abstract | CrossRef Full Text | Google Scholar

Galko, L., Porubän, J., and Senko, J. (2018). “Improving the User Experience of Electronic University Enrollment,” in 16th IEEE International Conference on Emerging eLearning Technologies and Applications, ICETA 2018 , Stary Smokovec, Slovakia , Nov 15–16, 2018 . Editors F. Jakab, (Piscataway, NJ: IEEE ), 179–184. doi:10.1109/ICETA.2018.8572054

Goda, Y., Yamada, M., Matsukawa, H., Hata, K., and Yasunami, S. (2014). Conversation with a Chatbot before an Online EFL Group Discussion and the Effects on Critical Thinking. J. Inf. Syst. Edu. 13, 1–7. doi:10.12937/EJSISE.13.1

Graesser, A. C., VanLehn, K., Rose, C. P., Jordan, P. W., and Harter, D. (2001). Intelligent Tutoring Systems with Conversational Dialogue. AI Mag. 22 (4), 39–51. doi:10.1609/aimag.v22i4.1591

Greller, W., and Drachsler, H. (2012). Translating Learning into Numbers: A Generic Framework for Learning Analytics. J. Educ. Tech. Soc. 15 (3), 42–57. doi:10.2307/jeductechsoci.15.3.42

Haristiani, N., and Rifa’i, M. M. Combining Chatbot and Social Media: Enhancing Personal Learning Environment (PLE) in Language Learning. Indonesian J Sci Tech. 5 (3), 487–506. doi:10.17509/ijost.v5i3.28687

Hattie, J., and Timperley, H. (2007). The Power of Feedback. Rev. Educ. Res. 77 (1), 81–112. doi:10.3102/003465430298487

Hattie, J. (2009). Visible Learning: A Synthesis of over 800 Meta-Analyses Relating to Achievement . Abingdon, UK: Routledge .

Heller, B., Proctor, M., Mah, D., Jewell, L., and Cheung, B. (2005). “Freudbot: An Investigation of Chatbot Technology in Distance Education,” in Proceedings of ED-MEDIA 2005–World Conference on Educational Multimedia, Hypermedia and Telecommunications , Montréal, Canada , June 27–July 2, 2005 . Editors P. Kommers, and G. Richards, ( AACE ), 3913–3918.

Heo, J., and Lee, J. (2019). “CiSA: An Inclusive Chatbot Service for International Students and Academics,” in 21st International Conference on Human-Computer Interaction, HCII 2019: Communications in Computer and Information Science , Orlando, FL, USA , July 26–31, 2019 . Editors C. Stephanidis, ( Springer ) 11786, 153–167. doi:10.1007/978-3-030-30033-3

Hobert, S. (2019a). “How Are You, Chatbot? Evaluating Chatbots in Educational Settings - Results of a Literature Review,” in 17. Fachtagung Bildungstechnologien, DELFI 2019 - 17th Conference on Education Technologies, DELFI 2019 , Berlin, Germany , Sept 16–19, 2019 . Editors N. Pinkwart, and J. Konert, 259–270. doi:10.18420/delfi2019_289

Hobert, S., and Meyer von Wolff, R. (2019). “Say Hello to Your New Automated Tutor - A Structured Literature Review on Pedagogical Conversational Agents,” in 14th International Conference on Wirtschaftsinformatik , Siegen, Germany , Feb 23–27, 2019 . Editors V. Pipek, and T. Ludwig, ( AIS ).

Hobert, S. (2019b). Say Hello to ‘Coding Tutor’! Design and Evaluation of a Chatbot-Based Learning System Supporting Students to Learn to Program in International Conference on Information Systems (ICIS) 2019 Conference , Munich, Germany , Dec 15–18, 2019 , AIS 2661, 1–17.

Hobert, S. (2020). Small Talk Conversations and the Long-Term Use of Chatbots in Educational Settings ‐ Experiences from a Field Study in 3rd International Workshop on Chatbot Research and Design, CONVERSATIONS 2019 , Amsterdam, Netherlands , November 19–20 : Lecture Notes in Computer Science. Editors A. Folstad, T. Araujo, S. Papadopoulos, E. Law, O. Granmo, E. Luger, and P. Brandtzaeg, ( Springer ) 11970, 260–272. doi:10.1007/978-3-030-39540-7_18

Hsieh, S.-W. (2011). Effects of Cognitive Styles on an MSN Virtual Learning Companion System as an Adjunct to Classroom Instructions. Edu. Tech. Society 2, 161–174.

Huang, J.-X., Kwon, O.-W., Lee, K.-S., and Kim, Y.-K. (2018). Improve the Chatbot Performance for the DB-CALL System Using a Hybrid Method and a Domain Corpus in Future-proof CALL: language learning as exploration and encounters–short papers from EUROCALL 2018 , Jyväskylä, Finland , Aug 22–25, 2018 . Editors P. Taalas, J. Jalkanen, L. Bradley, and S. Thouësny, ( Research-publishing.net ). doi:10.14705/rpnet.2018.26.820

Huang, W., Hew, K. F., and Gonda, D. E. (2019). Designing and Evaluating Three Chatbot-Enhanced Activities for a Flipped Graduate Course. Int. J. Mech. Engineer. Robotics. Research. 813–818. doi:10.18178/ijmerr.8.5.813-818

Ismail, M., and Ade-Ibijola, A. (2019). “Lecturer's Apprentice: A Chatbot for Assisting Novice Programmers,”in Proceedings - 2019 International Multidisciplinary Information Technology and Engineering Conference (IMITEC) , Vanderbijlpark, South Africa , (IEEE), 1–8. doi:10.1109/IMITEC45504.2019.9015857

Jia, J. (2008). “Motivate the Learners to Practice English through Playing with Chatbot CSIEC,” in 3rd International Conference on Technologies for E-Learning and Digital Entertainment, Edutainment 2008 , Nanjing, China , June 25–27, 2008 , Lecture Notes in Computer Science, (Springer) 5093, 180–191. doi:10.1007/978-3-540-69736-7_20

Jia, J. (2004). “The Study of the Application of a Keywords-Based Chatbot System on the Teaching of Foreign Languages,” in Proceedings of SITE 2004--Society for Information Technology and Teacher Education International Conference , Atlanta, Georgia, USA . Editors R. Ferdig, C. Crawford, R. Carlsen, N. Davis, J. Price, R. Weber, and D. Willis, (AACE), 1201–1207.

Jivet, I., Scheffel, M., Drachsler, H., and Specht, M. (2017). “Awareness is not enough: Pitfalls of learning analytics dashboards in the educational practice,” in 12th European Conference on Technology Enhanced Learning, EC-TEL 2017 , Tallinn, Estonia , September 12–15, 2017 , Lecture Notes in ComputerScience. Editors E. Lavoué, H. Drachsler, K. Verbert, J. Broisin, and M. Pérez-Sanagustín, (Springer), 82–96. doi:10.1007/978-3-319-66610-5_7

Jung, H., Lee, J., and Park, C. (2020). Deriving Design Principles for Educational Chatbots from Empirical Studies on Human-Chatbot Interaction. J. Digit. Contents Society , 21, 487–493. doi:10.9728/dcs.2020.21.3.487

Kerly, A., and Bull, S. (2006). “The Potential for Chatbots in Negotiated Learner Modelling: A Wizard-Of-Oz Study,” in 8th International Conference on Intelligent Tutoring Systems, ITS 2006 , Jhongli, Taiwan , June 26–30, 2006 , Lecture Notes in Computer Science. Editors M. Ikeda, K. D. Ashley, and T. W. Chan, ( Springer ) 4053, 443–452. doi:10.1007/11774303

Kerly, A., Ellis, R., and Bull, S. (2008). CALMsystem: A Conversational Agent for Learner Modelling. Knowledge-Based Syst. 21, 238–246. doi:10.1016/j.knosys.2007.11.015

Kerly, A., Hall, P., and Bull, S. (2007). Bringing Chatbots into Education: Towards Natural Language Negotiation of Open Learner Models. Knowledge-Based Syst. , 20, 177–185. doi:10.1016/j.knosys.2006.11.014

Kumar, M. N., Chandar, P. C. L., Prasad, A. V., and Sumangali, K. (2016). “Android Based Educational Chatbot for Visually Impaired People,” in 2016 IEEE International Conference on Computational Intelligence and Computing Research , Chennai, India , December 15–17, 2016 , 1–4. doi:10.1109/ICCIC.2016.7919664

Lee, K., Jo, J., Kim, J., and Kang, Y. (2019). Can Chatbots Help Reduce the Workload of Administrative Officers? - Implementing and Deploying FAQ Chatbot Service in a University in 21st International Conference on Human-Computer Interaction, HCII 2019: Communications in Computer and Information Science , Orlando, FL, USA , July 26–31, 2019 . Editors C. Stephanidis, ( Springer ) 1032, 348–354. doi:10.1007/978-3-030-23522-2

Lester, J. C., Converse, S. A., Kahler, S. E., Barlow, S. T., Stone, B. A., and Bhogal, R. S. (1997). “The Persona Effect: Affective Impact of Animated Pedagogical Agents,” in Proceedings of the ACM SIGCHI Conference on Human factors in computing systems , Atlanta, Georgia, USA , March 22–27, 1997 , (ACM), 359–366.

Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., et al. (2009). The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies that Evaluate Health Care Interventions: Explanation and Elaboration. J. Clin. Epidemiol. 62 (10), e1–e34. doi:10.1016/j.jclinepi.2009.06.006

Lin, M. P.-C., and Chang, D. (2020). Enhancing Post-secondary Writers’ Writing Skills with a Chatbot. J. Educ. Tech. Soc. 23, 78–92. doi:10.2307/26915408

Lin, Y.-H., and Tsai, T. (2019). “A Conversational Assistant on Mobile Devices for Primitive Learners of Computer Programming,” in TALE 2019 - 2019 IEEE International Conference on Engineering, Technology and Education , Yogyakarta, Indonesia , December 10–13, 2019 , (IEEE), 1–4. doi:10.1109/TALE48000.2019.9226015

Linden, A., and Fenn, J. (2003). Understanding Gartner’s Hype Cycles. Strategic Analysis Report No. R-20-1971 8. Stamford, CT: Gartner, Inc .

Liu, Q., Huang, J., Wu, L., Zhu, K., and Ba, S. (2020). CBET: Design and Evaluation of a Domain-specific Chatbot for mobile Learning. Univ. Access Inf. Soc. , 19, 655–673. doi:10.1007/s10209-019-00666-x

Mamani, J. R. C., Álamo, Y. J. R., Aguirre, J. A. A., and Toledo, E. E. G. (2019). “Cognitive Services to Improve User Experience in Searching for Academic Information Based on Chatbot,” in Proceedings of the 2019 IEEE 26th International Conference on Electronics, Electrical Engineering and Computing (INTERCON) , Lima, Peru , August 12–14, 2019 , (IEEE), 1–4. doi:10.1109/INTERCON.2019.8853572

Martín-Martín, A., Orduna-Malea, E., Thelwall, M., and Delgado López-Cózar, E. (2018). Google Scholar, Web of Science, and Scopus: A Systematic Comparison of Citations in 252 Subject Categories. J. Informetrics 12 (4), 1160–1177. doi:10.1016/j.joi.2018.09.002

Matsuura, S., and Ishimura, R. (2017). Chatbot and Dialogue Demonstration with a Humanoid Robot in the Lecture Class, in 11th International Conference on Universal Access in Human-Computer Interaction, UAHCI 2017, held as part of the 19th International Conference on Human-Computer Interaction, HCI 2017 , Vancouver, Canada , July 9–14, 2017 , Lecture Notes in Computer Science. Editors M. Antona, and C. Stephanidis, (Springer) Vol. 10279, 233–246. doi:10.1007/978-3-319-58700-4

Matsuura, S., and Omokawa, R. (2020). Being Aware of One’s Self in the Auto-Generated Chat with a Communication Robot in UAHCI 2020 , 477–488. doi:10.1007/978-3-030-49282-3

McLoughlin, C., and Oliver, R. (1998). Maximising the Language and Learning Link in Computer Learning Environments. Br. J. Educ. Tech. 29 (2), 125–136. doi:10.1111/1467-8535.00054

Mendoza, S., Hernández-León, M., Sánchez-Adame, L. M., Rodríguez, J., Decouchant, D., and Meneses-Viveros, A. (2020). “Supporting Student-Teacher Interaction through a Chatbot,” in 7th International Conference, LCT 2020, Held as Part of the 22nd HCI International Conference, HCII 2020 , Copenhagen, Denmark , July 19–24, 2020 , Lecture Notes in Computer Science. Editors P. Zaphiris, and A. Ioannou, ( Springer ) 12206, 93–107. doi:10.1007/978-3-030-50506-6

Meyer, V., Wolff, R., Nörtemann, J., Hobert, S., and Schumann, M. (2020). “Chatbots for the Information Acquisition at Universities ‐ A Student’s View on the Application Area,“in 3rd International Workshop on Chatbot Research and Design, CONVERSATIONS 2019 , Amsterdam, Netherlands , November 19–20 , Lecture Notes in Computer Science. Editors A. Folstad, T. Araujo, S. Papadopoulos, E. Law, O. Granmo, E. Luger, and P. Brandtzaeg, (Springer) 11970, 231–244. doi:10.1007/978-3-030-39540-7

Na-Young, K. (2018c). A Study on Chatbots for Developing Korean College Students’ English Listening and Reading Skills. J. Digital Convergence 16. 19–26. doi:10.14400/JDC.2018.16.8.019

Na-Young, K. (2019). A Study on the Use of Artificial Intelligence Chatbots for Improving English Grammar Skills. J. Digital Convergence 17, 37–46. doi:10.14400/JDC.2019.17.8.037

Na-Young, K. (2018a). Chatbots and Korean EFL Students’ English Vocabulary Learning. J. Digital Convergence 16. 1–7. doi:10.14400/JDC.2018.16.2.001

Na-Young, K. (2018b). Different Chat Modes of a Chatbot and EFL Students’ Writing Skills Development . 1225–4975. doi:10.16933/sfle.2017.32.1.263

Na-Young, K. (2017). Effects of Different Types of Chatbots on EFL Learners’ Speaking Competence and Learner Perception. Cross-Cultural Studies 48, 223–252. doi:10.21049/ccs.2017.48.223

Nagata, R., Hashiguchi, T., and Sadoun, D. (2020). Is the Simplest Chatbot Effective in English Writing Learning Assistance?, in 16th International Conference of the Pacific Association for Computational Linguistics , PACLING, Hanoi, Vietnam , October 11–13, 2019 , Communications in Computer and Information Science. Editors L.-M. Nguyen, S. Tojo, X.-H. Phan, and K. Hasida, ( Springer ) Vol. 1215, 245–246. doi:10.1007/978-981-15-6168-9

Nelson, T. O., and Narens, L. (1994). Why Investigate Metacognition. in Metakognition: Knowing About Knowing . Editors J. Metcalfe, and P. Shimamura, (MIT Press) 13, 1–25.

Nghi, T. T., Phuc, T. H., and Thang, N. T. (2019). Applying Ai Chatbot for Teaching a Foreign Language: An Empirical Research. Int. J. Sci. Res. 8.

Ondas, S., Pleva, M., and Hládek, D. (2019). How Chatbots Can Be Involved in the Education Process. in ICETA 2019 - 17th IEEE International Conference on Emerging eLearning Technologies and Applications, Proceedings, Stary Smokovec , Slovakia , November 21–22, 2019 . Editors F. Jakab, (IEEE), 575–580. doi:10.1109/ICETA48886.2019.9040095

Pereira, J., Fernández-Raga, M., Osuna-Acedo, S., Roura-Redondo, M., Almazán-López, O., and Buldón-Olalla, A. (2019). Promoting Learners' Voice Productions Using Chatbots as a Tool for Improving the Learning Process in a MOOC. Tech. Know Learn. 24, 545–565. doi:10.1007/s10758-019-09414-9

Pérez, J. Q., Daradoumis, T., and Puig, J. M. M. (2020). Rediscovering the Use of Chatbots in Education: A Systematic Literature Review. Comput. Appl. Eng. Educ. 28, 1549–1565. doi:10.1002/cae.22326

Pérez-Marín, D. (2021). A Review of the Practical Applications of Pedagogic Conversational Agents to Be Used in School and University Classrooms. Digital 1 (1), 18–33. doi:10.3390/digital1010002

Pham, X. L., Pham, T., Nguyen, Q. M., Nguyen, T. H., and Cao, T. T. H. (2018). “Chatbot as an Intelligent Personal Assistant for mobile Language Learning,” in ACM International Conference Proceeding Series doi:10.1145/3291078.3291115

Quincey, E. de., Briggs, C., Kyriacou, T., and Waller, R. (2019). “Student Centred Design of a Learning Analytics System,” in Proceedings of the 9th International Conference on Learning Analytics & Knowledge , Tempe Arizona, USA , March 4–8, 2019 , (ACM), 353–362. doi:10.1145/3303772.3303793

Ram, A., Prasad, R., Khatri, C., Venkatesh, A., Gabriel, R., Liu, Q, et al. (2018). Conversational Ai: The Science behind the Alexa Prize, in 1st Proceedings of Alexa Prize (Alexa Prize 2017) . ArXiv [Preprint]. Available at: https://arxiv.org/abs/1801.03604 .

Rebaque-Rivas, P., and Gil-Rodríguez, E. (2019). Adopting an Omnichannel Approach to Improve User Experience in Online Enrolment at an E-Learning University, in 21st International Conference on Human-Computer Interaction, HCII 2019: Communications in Computer and Information Science , Orlando, FL, USA , July 26–31, 2019 . Editors C. Stephanidis, ( Springer ), 115–122. doi:10.1007/978-3-030-23525-3

Robinson, C. (2019). Impressions of Viability: How Current Enrollment Management Personnel And Former Students Perceive The Implementation of A Chatbot Focused On Student Financial Communication. Higher Education Doctoral Projects.2 . https://aquila.usm.edu/highereddoctoralprojects/2 .

Ruan, S., Jiang, L., Xu, J., Tham, B. J.-K., Qiu, Z., Zhu, Y., Murnane, E. L., Brunskill, E., and Landay, J. A. (2019). “QuizBot: A Dialogue-based Adaptive Learning System for Factual Knowledge,” in 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019 , Glasgow, Scotland, United Kingdom , May 4–9, 2019 , (ACM), 1–13. doi:10.1145/3290605.3300587

Sandoval, Z. V. (2018). Design and Implementation of a Chatbot in Online Higher Education Settings. Issues Inf. Syst. 19, 44–52. doi:10.48009/4.iis.2018.44-52

Sandu, N., and Gide, E. (2019). “Adoption of AI-Chatbots to Enhance Student Learning Experience in Higher Education in india,” in 18th International Conference on Information Technology Based Higher Education and Training , Magdeburg, Germany , September 26–27, 2019 , (IEEE), 1–5. doi:10.1109/ITHET46829.2019.8937382

Saygin, A. P., Cicekli, I., and Akman, V. (2000). Turing Test: 50 Years Later. Minds and Machines 10 (4), 463–518. doi:10.1023/A:1011288000451

Sinclair, A., McCurdy, K., Lucas, C. G., Lopez, A., and Gaševic, D. (2019). “Tutorbot Corpus: Evidence of Human-Agent Verbal Alignment in Second Language Learner Dialogues,” in EDM 2019 - Proceedings of the 12th International Conference on Educational Data Mining .

Smutny, P., and Schreiberova, P. (2020). Chatbots for Learning: A Review of Educational Chatbots for the Facebook Messenger. Comput. Edu. 151, 103862. doi:10.1016/j.compedu.2020.103862

Song, D., Rice, M., and Oh, E. Y. (2019). Participation in Online Courses and Interaction with a Virtual Agent. Int. Rev. Res. Open. Dis. 20, 44–62. doi:10.19173/irrodl.v20i1.3998

Stapić, Z., Horvat, A., and Vukovac, D. P. (2020). Designing a Faculty Chatbot through User-Centered Design Approach, in 22nd International Conference on Human-Computer Interaction,HCII 2020 , Copenhagen, Denmark , July 19–24, 2020 , Lecture Notes in Computer Science. Editors C. Stephanidis, D. Harris, W. C. Li, D. D. Schmorrow, C. M. Fidopiastis, and P. Zaphiris, ( Springer ), 472–484. doi:10.1007/978-3-030-60128-7

Subramaniam, N. K. (2019). Teaching and Learning via Chatbots with Immersive and Machine Learning Capabilities. In International Conference on Education (ICE 2019) Proceedings , Kuala Lumpur, Malaysia , April 10–11, 2019 . Editors S. A. H. Ali, T. T. Subramaniam, and S. M. Yusof, 145–156.

Sugondo, A. F., and Bahana, R. (2019). “Chatbot as an Alternative Means to Access Online Information Systems,” in 3rd International Conference on Eco Engineering Development, ICEED 2019 , Surakarta, Indonesia , November 13–14, 2019 , IOP Conference Series: Earth and Environmental Science, (IOP Publishing) 426. doi:10.1088/1755-1315/426/1/012168

Suwannatee, S., and Suwanyangyuen, A. (2019). “Reading Chatbot” Mahidol University Library and Knowledge Center Smart Assistant,” in Proceedings for the 2019 International Conference on Library and Information Science (ICLIS) , Taipei, Taiwan , July 11–13, 2019 .

Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S., and Torous, J. B. (2019). Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape. Can. J. Psychiatry 64 (7), 456–464. doi:10.1177/0706743719828977

Vijayakumar, B., Höhn, S., and Schommer, C. (2019). “Quizbot: Exploring Formative Feedback with Conversational Interfaces,” in 21st International Conference on Technology Enhanced Assessment, TEA 2018 , Amsterdam, Netherlands , Dec 10-11, 2018 . Editors S. Draaijer, B. D. Joosten-ten, and E. Ras, ( Springer ), 102–120. doi:10.1007/978-3-030-25264-9

Virtanen, M. A., Haavisto, E., Liikanen, E., and Kääriäinen, M. (2018). Ubiquitous Learning Environments in Higher Education: A Scoping Literature Review. Educ. Inf. Technol. 23 (2), 985–998. doi:10.1007/s10639-017-9646-6

Wildman, T. M., Magliaro, S. G., Niles, R. A., and Niles, J. A. (1992). Teacher Mentoring: An Analysis of Roles, Activities, and Conditions. J. Teach. Edu. 43 (3), 205–213. doi:10.1177/0022487192043003007

Wiley, D., and Edwards, E. K. (2002). Online Self-Organizing Social Systems: The Decentralized Future of Online Learning. Q. Rev. Distance Edu. 3 (1), 33–46.

Winkler, R., and Soellner, M. (2018). Unleashing the Potential of Chatbots in Education: A State-Of-The-Art Analysis. in Academy of Management Annual Meeting Proceedings 2018 2018 (1), 15903. doi:10.5465/AMBPP.2018.15903abstract

Winne, P. H., and Hadwin, A. F. (2008). “The Weave of Motivation and Self-Regulated Learning,” in Motivation and Self-Regulated Learning: Theory, Research, and Applications . Editors D. H. Schunk, and B. J. Zimmerman, (Mahwah, NJ: Lawrence Erlbaum Associates Publishers ), 297–314.

Wisniewski, B., Zierer, K., and Hattie, J. (2019). The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research. Front. Psychol. 10, 1664–1078. doi:10.3389/fpsyg.2019.03087

Wolfbauer, I., Pammer-Schindler, V., and Rose, C. P. (2020). “Rebo Junior: Analysis of Dialogue Structure Quality for a Reflection Guidance Chatbot,” in Proceedings of the Impact Papers at EC-TEL 2020, co-located with the 15th European Conference on Technology-Enhanced Learning “Addressing global challenges and quality education” (EC-TEL 2020) , Virtual , Sept 14–18, 2020 . Editors T. Broos, and T. Farrell, 1–14.

Xiao, Z., Zhou, M. X., and Fu, W.-T. (2019). “Who should be my teammates: Using a conversational agent to understand individuals and help teaming,” in IUI’19: Proceedings of the 24th International Conference on Intelligent User Interfaces, Marina del Ray , California, USA , March 17–20, 2019 , (ACM), 437–447. doi:10.1145/3301275.3302264

Xu, A., Liu, Z., Guo, Y., Sinha, V., and Akkiraju, R. (2017). “A New Chatbot for Customer Service on Social media,” in Proceedings of the 2017 CHI conference on human factors in computing systems , Denver, Colorado, USA , May 6–11, 2017 , ACM, 3506–3510. doi:10.1145/3025453.3025496

Yin, J., Goh, T.-T., Yang, B., and Xiaobin, Y. (2020). Conversation Technology with Micro-learning: The Impact of Chatbot-Based Learning on Students' Learning Motivation and Performance. J. Educ. Comput. Res. 59, 154–177. doi:10.1177/0735633120952067

Appendix a aconcept map of chatbots in education

Keywords: chatbots, education, literature review, pedagogical roles, domains

Citation: Wollny S, Schneider J, Di Mitri D, Weidlich J, Rittberger M and Drachsler H (2021) Are We There Yet? - A Systematic Literature Review on Chatbots in Education. Front. Artif. Intell. 4:654924. doi: 10.3389/frai.2021.654924

Received: 17 January 2021; Accepted: 10 June 2021; Published: 15 July 2021.

Reviewed by:

Copyright © 2021 Wollny, Schneider, Di Mitri, Weidlich, Rittberger and Drachsler. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Sebastian Wollny, [email protected] ; Jan Schneider, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

How to get started with Claude AI

Claude 3.5: from songwriter to dungeon master

Teach me a new skill

Create a song, pretend you’re a choose-your-own-adventure game, tell me a very funny joke, practise learning a language, develop a workout routine, create ascii art.

Among the various AI chatbots and assistants available today , I've found Claude to be particularly impressive in its capabilities and user interaction. While Open AI 's ChatGPT has been a prominent player in the AI assistant space, Claude is quickly making a name for itself as a powerful alternative. What I appreciate about Claude is its more natural conversational tone. In essence, it's human-ness . 

In my experience, Claude avoids the often impersonal customer-service-chatbot tone that other AI assistants like ChatGPT are prone to, which is a breath of fresh air. Since playing around with Claude, I've been really impressed by its language comprehension, as well as the intuitive interface. It often seems to anticipate what I need, which I've found incredibly helpful — whether I'm working on more complex tasks or broadly exploring its capabilities.

For those seeking an AI chatbot which offers its users a personable interaction blended with optimal efficiency, Claude is definitely worth it. Let's take a closer look at what it can do.

1. Log into Claude

If you already have a Claude account, go onto the Claude website and click Continue with Google or email to begin chatting.

If you're using Claude for the first time, go onto the Claude website and provide your email address. Once provided, you'll be taken to the chat interface. It's important to note that you'll need to verify your phone number for security reasons when signing up. 

2. Start chatting

Type your Claude prompt in the message bar and press Enter to generate a response.

You can also upload up to 5 documents or images, with a size limit of 10 MB each, and ask questions related to them.

3. Explore your options

After Claude generates a response, you have several editing options available. If you're not a fan of Claude's initial response, simply click the Retry button. The chatbot will generate an entirely new response to your original question, often with a different approach or insight. 

One of Claude's standout features is its ability to remember the ongoing conversation, allowing you to adjust your query and refine Claude's response without starting over or repeating yourself.

If you find Claude's answer helpful and want to use it elsewhere, click Copy. To provide feedback, click the thumbs up or thumbs down icon.

Now we've covered the fundamentals, let's dive into Claude's additional capabilities! 

When I prompted Claude to teach me a new skill , it asked me to choose out of the following options: a practical skill, a creative pursuit, a technical skill, a language or communication skill, or a physical activity. I chose technical skill . 

Claude decided to teach me the basics of Python. The instructions were really clear and concise, and also included accompanying pictures. There was also an option to move onto a simple project to practice my newfound skill. 

Type in the idea for your song. I inputted the following: Develop a song about self-discovery and embracing one's true identity, with lyrics that explore the journey of overcoming doubt and celebrating individuality.

While the lyrics are pretty saccharine, Claude's use of rhyme and metaphor was actually quite impressive! Significantly less cringe than I anticipated. 

With this prompt, you can be as general or specific as you want. You could type something simple like: Pretend you're a choose-your-own adventure game. Set the scene and give me four options to proceed to the next stage. 

For my prompt, I provided specific instructions for the game: Create a choose-your-own-adventure story where the reader can explore an ancient castle, a hidden cave guarded by a dragon, or an enchanted forest filled with magical creatures. Provide multiple choices at each decision point, leading to different outcomes based on the reader's selections. 

I had a lot of fun playing with this and found the range of choices very compelling. Claude did an excellent job of immersing me as a player, making the game feel truly original. 

If you're a pro subscriber (as you'll need more than five generated responses to really get into it) you won't want to miss this.

I asked Claude to tell me a very funny joke. The results were interesting. A few of the jokes were actually quite clever and did make me chuckle. As a more humanised AI chatbot, Claude's responses felt like your typical dad jokes. 

Claude also suggested I could try a joke in a different style of humour. I chose observational humour with an edge . I was really surprised how well Claude roasted modern society, and it got a genuine laugh out of me, too. 

If you're looking to learn and practise speaking a new language, Claude's got you covered. In the prompt box, I typed I'm a beginner, trying to learn Spanish. Offer language learning tips and practice conversations. 

Claude provided some basic tips to start with, before asking if I'd like to practise a conversation. 

From a practical standpoint, Claude was an excellent teacher. It conversed with me in Spanish, affirming when I got it right and clearly suggesting corrections when I got it wrong. Additionally, it provided follow-up phrases to help expand the conversation. If you use this in tandem with Duolingo, you'll be fluent in no time.

I wanted to test how well Claude could tailor a workout to my needs, so I asked it to develop a routine for someone with a repetitive strain injury in the wrist.

Claude not only provided a detailed breakdown of specific exercises and set times but also included important considerations, such as avoiding exercises that put direct stress on the wrist, like traditional push-ups or planks on hands.

ASCII art is making pictures using just the letters, numbers, and symbols you can type on a keyboard. Imagine trying to draw a smiley face, but you can only use things like colons, dashes, and parentheses. So you might end up with something like this: :-) 

I asked Claude to create a unique ASCII artwork. At first it created a really a simple tree with a triangular canopy. As I encouraged it to be more creative, the results seriously improved. 

Claude seemed to generate the best responses when they related to landscapes or objects. It created a ship, a mountain landscape and an underwater scene. The best by far, was the train. I thought the use of '@' and '0' to indicate the dissipating steam was a really nice touch.

Some other prompts to try...

• Create a cipher or code system for secret messages
• Explain complex topics in simple terms
• Provide step-by-step cooking recipes
• Create a mock syllabus for a college course
• Help troubleshoot basic tech issues
• Design a basic tarot spread and provide interpretations
• Help create schedules and to-do lists for better time management
• Create a set of rules for a new sport
• Design a themed escape room puzzle
• Generate creative writing prompts or story ideas

Anthropic has ambitiously claimed that Claude 3.5 Sonnet outperforms OpenAI 's GPT-4, and many users of Claude aren't surprised. The AI chatbot is impressively human-like, funny and at times a little sassy, thanks to character training added during the fine tuning process. And with three model tiers, there's a Claude for everyone. 

Sign up to get the BEST of Tom’s Guide direct to your inbox.

Upgrade your life with a daily dose of the biggest tech news, lifestyle hacks and our curated analysis. Be the first to know about cutting-edge gadgets and the hottest deals.

Kaycee is an Editor at Tom’s Guide and has been writing for as long as she can remember. Her journey into the tech world began as Cazoo's Knowledge Content Specialist, igniting her enthusiasm for technology. When she’s not exploring the latest gadgets and innovations, Kaycee can be found immersed in her favorite video games, or penning her second poetry collection. 

Moshi Chat's GPT-4o advanced voice competitor tried to argue with me — OpenAI doesn't need to worry just yet

You can now talk to 1000s of AI models on Mac or Windows, thanks to this huge GPT4All 3.0 update

My favorite iPhone app is the one that irritates me the most — here's why

Most Popular

• 2 Marvel claims its new Disney Plus show is ‘really scary’ — but I don't believe it
• 3 Here are 3 Prime Day beating deals on Shark vacuum cleaners that you can buy right now
• 4 Spice up your kitchen with these 9 early Prime Day deals
• 5 Dell's Black Friday in July sale is live — 5 deals I recommend starting at $79

ChatGPT: Here's What You Get With the Gen AI Tool That Started It All

ChatGPT quickly swept us away with its mind-blowing skills. Its latest model, GPT-4o, is faster, cheaper and can generate more text than its predecessors.

• Shankland covered the tech industry for more than 25 years and was a science writer for five years before that. He has deep expertise in microprocessors, digital photography, computer hardware and software, internet standards, web technology, and more.

In late 2022, OpenAI wowed the world when it introduced ChatGPT , a chatbot with an entirely new level of power, breadth and usefulness, thanks to the generative AI technology behind it. Since then, ChatGPT has continued to evolve, including its most recent development: the launch of its GPT-4o model .

ChatGPT and generative AI aren't a novelty anymore, but keeping track of what they can do poses a challenge as new abilities arrive. Most notably, OpenAI now provides easier access to anyone who wants to use it. It also lets anyone write custom AI apps called GPTs and share them on its own app store, while on a smaller scale ChatGPT can now speak its responses to you. OpenAI has been leading the generative AI charge , but it's hotly pursued by Microsoft, Google and startups far and wide.

Generative AI still hasn't shaken a core problem -- it makes up information that sounds plausible but isn't necessarily correct. But there's no denying AI has fired the imaginations of computer scientists, loosened the purse strings of venture capitalists and caught the attention of everyone from teachers to doctors to artists and more, all wondering how AI will change their work and their lives. 

If you're trying to get a handle on ChatGPT, this FAQ is for you. Here's a look at what's up.

Read more :  ChatGPT 3.5 Review: First Doesn't Mean Best

What is ChatGPT?

ChatGPT is an online chatbot that responds to "prompts" -- text requests that you type. ChatGPT has countless uses . You can request relationship advice, a summarized history of punk rock or an explanation of the ocean's tides. It's particularly good at writing software, and it can also handle some other technical tasks, like creating 3D models .

ChatGPT is called a generative AI because it generates these responses on its own. But it can also display more overtly creative output like screenplays, poetry, jokes and student essays. That's one of the abilities that really caught people's attention.

Much of AI has been focused on specific tasks, but ChatGPT is a general-purpose tool. This puts it more into a category like a search engine.

That breadth makes it powerful but also hard to fully control. OpenAI has many mechanisms in place to try to screen out abuse and other problems, but there's an active cat-and-mouse game afoot by researchers and others who try to get ChatGPT to do things like offer bomb-making recipes.

ChatGPT really blew people's minds when it began passing tests. For example, AnsibleHealth researchers reported in 2023 that " ChatGPT performed at or near the passing threshold " for the United States Medical Licensing Exam, suggesting that AI chatbots "may have the potential to assist with medical education, and potentially, clinical decision-making."

We're a long way from fully fledged doctor-bots you can trust, but the computing industry is investing billions of dollars to solve the problems and expand AI into new domains like visual data, too. OpenAI is among those at the vanguard. So strap in, because the AI journey is going to be a sometimes terrifying, sometimes exciting thrill.

What's ChatGPT's origin?

Artificial intelligence algorithms had been ticking away for years before ChatGPT arrived. These systems were a big departure from traditional programming, which follows a rigid if-this-then-that approach. AI, in contrast, is trained to spot patterns in complex real-world data. AI has been busy for more than a decade screening out spam, identifying our friends in photos, recommending videos and translating our Alexa voice commands into computerese.

A Google technology called transformers helped propel AI to a new level, leading to a type of AI called a large language model, or LLM . These AIs are trained on enormous quantities of text, including material like books, blog posts, forum comments and news articles. The training process internalizes the relationships between words, letting chatbots process input text and then generate what it believes to be appropriate output text. 

A second phase of building an LLM is called reinforcement learning through human feedback, or RLHF. That's when people review the chatbot's responses and steer it toward good answers or away from bad ones. That significantly alters the tool's behavior and is one important mechanism for trying to stop abuse.

OpenAI's LLM is called GPT, which stands for "generative pretrained transformer." Training a new model is expensive and time consuming, typically taking weeks and requiring a data center packed with thousands of expensive AI acceleration processors. OpenAI's latest LLM is called GPT-4o. Other LLMs include Google's Gemini (formerly called Bard), Anthropic's Claude and Meta's Llama .

ChatGPT is an interface that lets you easily prompt GPT for responses. When it arrived as a free tool in November 2022, its use exploded far beyond what OpenAI expected.

When OpenAI launched ChatGPT, the company didn't even see it as a product. It was supposed to be a mere "research preview," a test that could draw some feedback from a broader audience, said ChatGPT product leader Nick Turley. Instead, it went viral, and OpenAI scrambled to just keep the service up and running under the demand.

"It was surreal," Turley said. "There was something about that release that just struck a nerve with folks in a way that we certainly did not expect. I remember distinctly coming back the day after we launched and looking at dashboards and thinking, something's broken, this couldn't be real, because we really didn't make a very big deal out of this launch."

ChatGPT, a name only engineers could love, was launched as a research project in November 2022, but quickly caught on as a consumer product.

How do I use ChatGPT?

The ChatGPT website is the most obvious method. Open it up, select the LLM version you want from the drop-down menu in the upper left corner, and type in a query.

As of April 1, OpenAI is allowing consumers to use ChatGPT without first signing up for an account. According to a blog post , the move was meant to make the tool more accessible. OpenAI also said in the post that as part of the move, it's introducing added content safeguards, blocking prompts in a wider range of categories.

However, users with accounts will be able to do more with the tool, such as save and review their history, share conversations and tap into features like voice conversations and custom instructions.

In 2023, OpenAI released a ChatGPT app for iPhones and for Android phones . In February 2024, ChatGPT for Apple Vision Pro arrived , too, adding the chatbot's abilities to the "spatial computing" headset. Be careful to look for the genuine article, because other developers can create their own chatbot apps that link to OpenAI's GPT.

In January 2024, OpenAI opened its GPT Store , a collection of custom AI apps that focus ChatGPT's all-purpose design to specific jobs. A lot more on that later, but in addition to finding them through the store you can invoke them with the @ symbol in a prompt, the way you might tag a friend on Instagram.

Microsoft uses GPT for its Bing search engine, which means you can also try out ChatGPT there.

ChatGPT has sprouted up in various hardware devices, including Volkswagen EVs , Humane's voice-controlled AI pin and the squarish Rabbit R1 device .

How much does ChatGPT cost?

It's free, though you have to set up an account to take advantage of all of its features.

For more capability, there's also a subscription called ChatGPT Plus that costs $20 per month that offers a variety of advantages: It responds faster, particularly during busy times when the free version is slow or sometimes tells you to try again later. It also offers access to newer AI models, including GPT-4 Turbo , which arrived in late 2023 with more up-to-date responses and an ability to ingest and output larger blocks of text.

The free ChatGPT uses GPT-4o, which launched in May of this year.

ChatGPT is growing beyond its language roots. With ChatGPT Plus, you can upload images, for example, to ask what type of mushroom is in a photo.

Perhaps most importantly, ChatGPT Plus lets you use GPTs.

What are these GPTs?

GPTs are custom versions of ChatGPT from OpenAI, its business partners and thousands of third-party developers who created their own GPTs.

Sometimes when people encounter ChatGPT, they don't know where to start. OpenAI calls it the "empty box problem." Discovering that led the company to find a way to narrow down the choices, Turley said.

"People really benefit from the packaging of a use case -- here's a very specific thing that I can do with ChatGPT," like travel planning, cooking help or an interactive, step-by-step tool to build a website, Turley said.

OpenAI CEO Sam Altman announces custom AI apps called GPTs at a developer event in November 2023.

Think of GPTs as OpenAI trying to make the general-purpose power of ChatGPT more refined the same way smartphones have a wealth of specific tools. (And think of GPTs as OpenAI's attempt to take control over how we find, use and pay for these apps, much like Apple has a commanding role over iPhones through its App Store.)

What GPTs are available now?

OpenAI's GPT store now offers millions of GPTs , though as with smartphone apps, you'll probably not be interested in most of them. A range of GPT custom apps are available, including AllTrails personal trail recommendations , a Khan Academy programming tutor , a Canva design tool , a book recommender , a fitness trainer , the laundry buddy clothes washing label decoder, a music theory instructor , a haiku writer and the Pearl for Pets for vet advice bot .

One person excited by GPTs is Daniel Kivatinos, co-founder of financial services company JustPaid . His team is building a GPT designed to take a spreadsheet of financial data as input and then let executives ask questions. How fast is a startup going through the money investors gave it? Why did that employee just file a $6,000 travel expense?

JustPaid hopes that GPTs will eventually be powerful enough to accept connections to bank accounts and financial software. For now, the developers are focusing on guardrails to avoid problems like hallucinations -- those answers that sound plausible but are actually wrong -- or making sure the GPT is answering based on the users' data, not on some general information in its AI model, Kivatinos said.

Anyone can create a GPT, at least in principle. OpenAI's GPT editor walks you through the process with a series of prompts. Just like with the regular ChatGPT, your ability to craft the right prompt will generate better results.

Another notable difference from regular ChatGPT: GPTs let you upload extra data that's relevant to your particular GPT, like a collection of essays or a writing style guide.

Some of the GPTs draw on OpenAI's Dall-E tool for turning text into images, which can be useful and entertaining. For example, there is a coloring book picture creator , a logo generator and a tool that turns text prompts into diagrams like company org charts. OpenAI calls Dall-E a GPT.

How up to date is ChatGPT?

Not very, and that can be a problem. For example, a Bing search using ChatGPT to process results said OpenAI hadn't yet released its ChatGPT Android app. Search results from traditional search engines can help to "ground" AI results, and indeed that's part of the Microsoft-OpenAI partnership that can tweak ChatGPT Plus results.

GPT-4 Turbo is trained on data up through April 2023. But it's nothing like a search engine whose bots crawl news sites many times a day for the latest information.

Can you trust ChatGPT responses?

No. Well, sometimes, but you need to be wary.

Large language models work by stringing words together, one after another, based on what's probable each step of the way. But it turns out that the generative AI fueled by LLMs works better and sounds more natural with a little spice of randomness added to the word selection recipe. That's the basic statistical nature that underlies the criticism that LLMs are mere "stochastic parrots" rather than sophisticated systems that in some way understand the world's complexity.

The result of this system, combined with the steering influence of the human training, is an AI that produces results that sound plausible but that aren't necessarily true. ChatGPT does better with information that's well represented in training data and undisputed -- for instance, red traffic signals mean stop, Plato was a philosopher who wrote the Allegory of the Cave , an Alaskan earthquake in 1964 was the largest in US history at magnitude 9.2.

We humans interact with AI chatbots by writing prompts -- questions or statements that seek an answer from the information stored in the chatbot's underlying large language model. 

When facts are more sparsely documented, controversial or off the beaten track of human knowledge, LLMs don't work as well. Unfortunately, they sometimes produce incorrect answers with a convincing, authoritative voice. That's what tripped up a lawyer who used ChatGPT to bolster his legal case only to be reprimanded when it emerged ChatGPT fabricated some cases that appeared to support his arguments. "I did not comprehend that ChatGPT could fabricate cases ," he said, according to The New York Times.

Such fabrications are called hallucinations in the AI business.

That means when you're using ChatGPT, it's best to double check facts elsewhere.

But there are plenty of creative uses for ChatGPT that don't require strictly factual results.

Want to use ChatGPT to draft a cover letter for a job hunt or give you ideas for a themed birthday party? No problem. Looking for hotel suggestions in Bangladesh? ChatGPT can give useful travel itineraries , but confirm the results before booking anything.

Is the hallucination problem getting better?

Yes, but we haven't seen a breakthrough.

"Hallucinations are a fundamental limitation of the way that these models work today," Turley said. LLMs just predict the next word in a response, over and over, "which means that they return things that are likely to be true, which is not always the same as things that are true," Turley said.

But OpenAI has been making gradual progress. "With nearly every model update, we've gotten a little bit better on making the model both more factual and more self aware about what it does and doesn't know," Turley said. "If you compare ChatGPT now to the original ChatGPT, it's much better at saying, 'I don't know that' or 'I can't help you with that' versus making something up."

Hallucinations are so much a part of the zeitgeist that Dictionary.com touted it as a new word it added to its dictionary in 2023.

Can you use ChatGPT for wicked purposes?

You can try, but lots of it will violate OpenAI's terms of use , and the company tries to block it too. The company prohibits use that involves sexual or violent material, racist caricatures, and personal information like Social Security numbers or addresses.

OpenAI works hard to prevent harmful uses. Indeed, its basic sales pitch is trying to bring the benefits of AI to the world without the drawbacks. But it acknowledges the difficulties, for example in its GPT-4 "system card" that documents its safety work.

"GPT-4 can generate potentially harmful content, such as advice on planning attacks or hate speech. It can represent various societal biases and worldviews that may not be representative of the user's intent, or of widely shared values. It can also generate code that is compromised or vulnerable," the system card says. It also can be used to try to identify individuals and could help lower the cost of cyberattacks.

Through a process called red teaming, in which experts try to find unsafe uses of its AI and bypass protections, OpenAI identified lots of problems and tried to nip them in the bud before GPT-4 launched. For example, a prompt to generate jokes mocking a Muslim boyfriend in a wheelchair was diverted so its response said, "I cannot provide jokes that may offend someone based on their religion, disability or any other personal factors. However, I'd be happy to help you come up with some light-hearted and friendly jokes that can bring laughter to the event without hurting anyone's feelings."

Researchers are still probing LLM limits. For example, Italian researchers discovered they could use ChatGPT to fabricate fake but convincing medical research data . And Google DeepMind researchers found that telling ChatGPT to repeat the same word forever eventually caused a glitch that made the chatbot blurt out training data verbatim. That's a big no-no, and OpenAI barred the approach .

LLMs are still new. Expect more problems and more patches.

And there are plenty of uses for ChatGPT that might be allowed but ill-advised. The website of Philadelphia's sheriff published more than 30 bogus news stories generated with ChatGPT .

What about ChatGPT and cheating in school?

ChatGPT is well suited to short essays on just about anything you might encounter in high school or college, to the chagrin of many educators who fear students will type in prompts instead of thinking for themselves.

Microsoft CEO Satya Nadella touted his company's partnership with OpenAI at a November 2023 event for OpenAI developers. Microsoft uses OpenAI's GPT large language model for its Bing search engine, Office productivity tools and GitHub Copilot programming assistant.

ChatGPT also can solve some math problems, explain physics phenomena, write chemistry lab reports and handle all kinds of other work students are supposed to handle on their own. Companies that sell anti-plagiarism software have pivoted to flagging text they believe an AI generated.

But not everyone is opposed, seeing it more like a tool akin to Google search and Wikipedia articles that can help students.

"There was a time when using calculators on exams was a huge no-no," said Alexis Abramson, dean of Dartmouth's Thayer School of Engineering. "It's really important that our students learn how to use these tools, because 90% of them are going into jobs where they're going to be expected to use these tools. They're going to walk in the office and people will expect them, being age 22 and technologically savvy, to be able to use these tools."

ChatGPT also can help kids get past writer's block and can help kids who aren't as good at writing, perhaps because English isn't their first language, she said.

So for Abramson, using ChatGPT to write a first draft or polish their grammar is fine. But she asks her students to disclose that fact.

"Anytime you use it, I would like you to include what you did when you turn in your assignment," she said. "It's unavoidable that students will use ChatGPT, so why don't we figure out a way to help them use it responsibly?"

Is ChatGPT coming for my job?

The threat to employment is real as managers seek to replace expensive humans with cheaper automated processes. We've seen this movie before: elevator operators were replaced by buttons, bookkeepers were replaced by accounting software, welders were replaced by robots. 

ChatGPT has all sorts of potential to blitz white-collar jobs: paralegals summarizing documents, marketers writing promotional materials, tax advisers interpreting IRS rules, even therapists offering relationship advice.

But so far, in part because of problems with things like hallucinations, AI companies present their bots as assistants and "copilots," not replacements.

And so far, sentiment is more positive than negative about chatbots, according to a survey by consulting firm PwC. Of 53,912 people surveyed around the world, 52% expressed at least one good expectation about the arrival of AI, for example that AI would increase their productivity. That compares with 35% who had at least one negative thing to say, for example that AI will replace them or require skills they're not confident they can learn.

How will ChatGPT affect programmers?

Software development is a particular area where people have found ChatGPT and its rivals useful. Trained on millions of lines of code, it internalized enough information to build websites and mobile apps. It can help programmers frame up bigger projects or fill in details.

One of the biggest fans is Microsoft's GitHub , a site where developers can host projects and invite collaboration. Nearly a third of people maintaining GitHub projects use its GPT-based assistant, called Copilot, and 92% of US developers say they're using AI tools .

"We call it the industrial revolution of software development," said Github Chief Product Officer Inbal Shani. "We see it lowering the barrier for entry. People who are not developers today can write software and develop applications using Copilot."

It's the next step in making programming more accessible, she said. Programmers used to have to understand bits and bytes, then higher-level languages gradually eased the difficulties. "Now you can write coding the way you talk to people," she said.

And AI programming aids still have a lot to prove. Researchers from Stanford and the University of California-San Diego found in a  study of 47 programmers  that those with access to an OpenAI programming help " wrote significantly less secure code  than those without access."

And they raise a variation of the cheating problem that some teachers are worried about: copying software that shouldn't be copied, which can lead to copyright problems. That's why Copyleaks, a maker of plagiarism detection software, offers a tool called the  Codeleaks Source Code AI Detector  designed to spot AI-generated code from ChatGPT, Google Gemini and GitHub Copilot. AIs could inadvertently copy code from other sources, and the latest version is designed to spot copied code based on its semantic structures, not just verbatim software.

At least in the next five years, Shani doesn't see AI tools like Copilot as taking humans out of programming.

"I don't think that it will replace the human in the loop. There's some capabilities that we as humanity have -- the creative thinking, the innovation, the ability to think beyond how a machine thinks in terms of putting things together in a creative way. That's something that the machine can still not do."

CNET's Lisa Lacy contributed to this report.

Computing Guides

• Best Laptop
• Best Chromebook
• Best Budget Laptop
• Best Cheap Gaming Laptop
• Best 2-in-1 Laptop
• Best Windows Laptop
• Best Macbook
• Best Gaming Laptop
• Best Macbook Deals
• Best Desktop PC
• Best Gaming PC
• Best Monitor Under 200
• Best Desktop Deals
• Best Monitors
• M2 Mac Mini Review
• Best PC Speakers
• Best Printer
• Best External Hard Drive SSD
• Best USB C Hub Docking Station
• Best Keyboard
• Best Webcams
• Best Laptop Backpack
• Best Camera to Buy
• Best Vlogging Camera
• Best Tripod
• Best Waterproof Camera
• Best Action Camera
• Best Camera Bag and Backpack
• Best E-Ink Tablets
• Best iPad Deals
• Best E-Reader
• Best Tablet
• Best Android Tablet
• Best 3D Printer
• Best Budget 3D Printer
• Best 3D Printing Filament
• Best 3D Printer Deals

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 02 July 2024

Western scientists more likely to get rejected papers published — and do it faster

You can also search for this author in PubMed   Google Scholar

Academics hope to publish their research in journals (shown), but their initial submissions are often rejected. Credit: Getty

Like actors and writers, researchers experience their fair share of rejection. Scientists submit their work to journals, hoping that it will be accepted, but many manuscripts are rejected from their authors’ top-choice publication and eventually get accepted by another. A considerable number of submissions don’t ever find a home.

A study 1 sheds light on this process of rejection and resubmission, which it argues can be skewed by the differing attitudes and behaviours of researchers around the world.

After following the fate of some 126,000 rejected manuscripts, the research team found that authors in Western countries are almost 6% more likely than are those based in other parts of the world to successfully publish a paper after it has been rejected. This could be, the authors suggest, because of regional differences in access to ‘procedural knowledge’ of how to deal with rejection — how to interpret negative reviews, revise accordingly and resubmit to a journal that is likely to accept the work. (Many academic journals are based in Western countries.)

“Maybe it’s something about being in the right networks and being able to get the right kind of advice at the right time,” says co-author Misha Teplitskiy, a sociologist studying innovation in science and technology at the University of Michigan in Ann Arbor.

Rejection review

Teplitskiy and his colleagues worked with data provided by IOP Publishing (IOPP), a company based in Bristol, UK, that publishes more than 90 English-language journals and is owned by the Institute of Physics.

They examined around 203,000 manuscripts that were submitted to 62 of IOPP’s physical-sciences journals between 2018 and 2022. Some 62% were rejected. The researchers scoured a bibliometric database to see whether the same (or similar) work was subsequently published elsewhere. They then sorted these publications by the geographical region of the corresponding author — the researcher who is usually in charge of a study’s publication process — and compared the outcomes for authors from the West (which they define as North America, Europe and Oceania) with those from the rest of the world.

Source: Ref. 1

To compare the fate of rejected papers as fairly as possible, the authors categorized them by quality, using the ratings and comments of the original peer reviewers recorded in the IOPP data. In this way, they could compare ‘like for like’: for example, looking at whether low-quality papers from Western authors had different outcomes from those rated as similar quality but written by authors from other parts of the world.

The analysis — published ahead of peer review as a preprint on the SSRN server 1 — showed that corresponding authors from Western countries are 5.7% more likely to publish a manuscript after rejection than those from other regions. In a process that often takes up to 300 days, they did so 23 days faster, on average. These authors also revised the abstract of their manuscript — a proxy for the overall paper — 5.9% less often, as defined by a computational ‘edit distance’ metric. And, ultimately, they published in journals with 0.8% higher impact factors. This metric reflects how often papers in a journal are cited, but is equated by some with the journal’s reach and prestige.

In a breakdown by country, the team’s analysis showed that around 70% of papers from Asian nations such as China and India were published eventually, compared with 85% from the United States, and close to 90% for many European countries (see ‘Publishing outcomes by country’).

What’s responsible for these differences? It’s hard to be sure, Teplitskiy says, but the results are consistent — at least in part — with the idea that the tacit norms and rules of the publishing process circulate more widely in the West, which leads to a higher likelihood of successful responses by Western scientists to rejections. His team tried to ask the authors of rejected papers about this hypothesis in a follow-up survey, but got few responses.

“People hate surveys in general, but they really don’t like surveys about their rejected papers,” he says.

Navigating the system

The way the authors rated and compared papers of similar quality is a good approach, says Honglin Bao, a data scientist at Harvard Business School in Boston, Massachusetts, who worked previously in China: “I think this works.”

The true cost of science’s language barrier for non-native English speakers

Differing procedural knowledge could contribute to the well-known bias in the peer-review system against researchers who are not based in Western countries, Bao says. Another possibility is that cultural factors work against researchers and add to the system’s bias. For example, many journals are written in English, which puts researchers whose first language is not English at a disadvantage , and could contribute to their poorer performance after rejection.

Teplitskiy will now face the possible rejection–resubmission cycle himself. He has submitted the study to the journal Proceedings of the National Academy of Sciences for peer review, but is realistic about the probable outcome. “I think this paper’s great, but I know the process is noisy,” he says. “We expect that it will bounce around early on and then land somewhere.”

doi: https://doi.org/10.1038/d41586-024-02142-w

Chen, H., Rider, C. I., Jurgens, D. & Teplitskiy, M. Preprint at https://ssrn.com/abstract=4872023 (2024).

Download references

Reprints and permissions

Related Articles

• Scientific community

Spy on millions of sleeping butterflies and more — June’s best science images

News 02 JUL 24

‘It can feel like there’s no way out’ — political scientists face pushback on their work

News Feature 19 JUN 24

How climate change is hitting Europe: three graphics reveal health impacts

News 18 JUN 24

‘All things that wander in the heavens’: how I swapped my ivory tower for the world of science fiction

Career Q&A 04 JUL 24

Give UK science the overhaul it urgently needs

Comment 04 JUL 24

Japan’s scientists demand more money for basic science

News 04 JUL 24

Staff Scientist in Computational Metabolomics

A position as a Staff scientist in Computational Metabolomics is available at the SciLifeLab Metabolomics Platform.

Umeå (Kommun), Västerbotten (SE)

Umeå University (KBC)

Group Leader in Functional Genomics

APPLICATION CLOSING DATE: August 15th, 2024 Human Technopole (HT) is an interdisciplinary life science research institute, created and supported by...

Human Technopole

Faculty Positions & Postdocs at Institute of Physics (IOP), Chinese Academy of Sciences

IOP is the leading research institute in China in condensed matter physics and related fields. Through the steadfast efforts of generations of scie...

Beijing, China

Institute of Physics (IOP), Chinese Academy of Sciences (CAS)

Faculty and Research Positions, Postdoctoral Recruitment

Jointly sponsored by the Hangzhou Municipal People's Government and the University of Chinese Academy of Sciences.

Hangzhou, Zhejiang, China

Hangzhou Institute of Advanced Study, UCAS

Postdoctoral Research Scientist: DNA Replication and Repair in Haematopoietic Stem Cells

An exciting opportunity has arisen for a highly motivated Postdoctoral Research Scientist to join Professor Chapman’s Group, to investigate how DNA...

Oxford, Oxfordshire

University of Oxford, Radcliffe Department of Medicine

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies