literature review of academic journal

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Try for free

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved April 9, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, unlimited academic ai-proofreading.

✔ Document error-free in 5minutes ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• CAREER FEATURE
• 04 December 2020
• Correction 09 December 2020

How to write a superb literature review

Andy Tay is a freelance writer based in Singapore.

You can also search for this author in PubMed   Google Scholar

Literature reviews are important resources for scientists. They provide historical context for a field while offering opinions on its future trajectory. Creating them can provide inspiration for one’s own research, as well as some practice in writing. But few scientists are trained in how to write a review — or in what constitutes an excellent one. Even picking the appropriate software to use can be an involved decision (see ‘Tools and techniques’). So Nature asked editors and working scientists with well-cited reviews for their tips.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

doi: https://doi.org/10.1038/d41586-020-03422-x

Interviews have been edited for length and clarity.

Updates & Corrections

Correction 09 December 2020 : An earlier version of the tables in this article included some incorrect details about the programs Zotero, Endnote and Manubot. These have now been corrected.

Hsing, I.-M., Xu, Y. & Zhao, W. Electroanalysis 19 , 755–768 (2007).

Article   Google Scholar  

Ledesma, H. A. et al. Nature Nanotechnol. 14 , 645–657 (2019).

Article   PubMed   Google Scholar  

Brahlek, M., Koirala, N., Bansal, N. & Oh, S. Solid State Commun. 215–216 , 54–62 (2015).

Choi, Y. & Lee, S. Y. Nature Rev. Chem . https://doi.org/10.1038/s41570-020-00221-w (2020).

Download references

Related Articles

• Research management

How I harnessed media engagement to supercharge my research career

Career Column 09 APR 24

How we landed job interviews for professorships straight out of our PhD programmes

Career Column 08 APR 24

Three ways ChatGPT helps me in my academic writing

How two PhD students overcame the odds to snag tenure-track jobs

Adopt universal standards for study adaptation to boost health, education and social-science research

Correspondence 02 APR 24

Is ChatGPT corrupting peer review? Telltale words hint at AI use

News 10 APR 24

Rwanda 30 years on: understanding the horror of genocide

Editorial 09 APR 24

Junior Group Leader Position at IMBA - Institute of Molecular Biotechnology

The Institute of Molecular Biotechnology (IMBA) is one of Europe’s leading institutes for basic research in the life sciences. IMBA is located on t...

Austria (AT)

IMBA - Institute of Molecular Biotechnology

Open Rank Faculty, Center for Public Health Genomics

Center for Public Health Genomics & UVA Comprehensive Cancer Center seek 2 tenure-track faculty members in Cancer Precision Medicine/Precision Health.

Charlottesville, Virginia

Center for Public Health Genomics at the University of Virginia

Husbandry Technician I

Memphis, Tennessee

St. Jude Children's Research Hospital (St. Jude)

Lead Researcher – Department of Bone Marrow Transplantation & Cellular Therapy

Researcher in the center for in vivo imaging and therapy.

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Academic Writing

• Strategies for Writing
• Punctuation
• Plagiarism & Self-Plagiarism

How to Build a Literature Review

• PRISMA - Systematic reviews & meta-analyses
• Other Resources
• Using Zotero for Bibliographies
• Abstract Writing Tips
• Writing Assistance
• Locating a Journal
• Assessing Potential Journals
• Finding a Publisher
• Types of Peer Review
• Author Rights & Responsibilities
• Copyright Considerations
• What is a Lit Review?
• Why Write a Lit Review?

Structure of a Literature Review

Preliminary steps for literature review.

• Basic Example
• More Examples

What is a Literature Review?

A literature review is a comprehensive summary and analysis of previously published research on a particular topic. Literature reviews should give the reader an overview of the important theories and themes that have previously been discussed on the topic, as well as any important researchers who have contributed to the discourse. This review should connect the established conclusions to the hypothesis being presented in the rest of the paper.

What a Literature Review Is Not:

• Annotated Bibliography: An annotated bibliography summarizes and assesses each resource individually and separately. A literature review explores the connections between different articles to illustrate important themes/theories/research trends within a larger research area. 
• Timeline: While a literature review can be organized chronologically, they are not simple timelines of previous events. They should not be a list of any kind. Individual examples or events should be combined to illustrate larger ideas or concepts.
• Argumentative Paper: Literature reviews are not meant to be making an argument. They are explorations of a concept to give the audience an understanding of what has already been written and researched about an idea. As many perspectives as possible should be included in a literature review in order to give the reader as comprehensive understanding of a topic as possible.

Why Write a Literature Review?

After reading the literature review, the reader should have a basic understanding of the topic. A reader should be able to come into your paper without really knowing anything about an idea, and after reading the literature, feel more confident about the important points.

A literature review should also help the reader understand the focus the rest of the paper will take within the larger topic. If the reader knows what has already been studied, they will be better prepared for the novel argument that is about to be made.

A literature review should help the reader understand the important history, themes, events, and ideas about a particular topic. Connections between ideas/themes should also explored. Part of the importance of a literature review is to prove to experts who do read your paper that you are knowledgeable enough to contribute to the academic discussion. You have to have done your homework.

A literature review should also identify the gaps in research to show the reader what hasn't yet been explored. Your thesis should ideally address one of the gaps identified in the research. Scholarly articles are meant to push academic conversations forward with new ideas and arguments. Before knowing where the gaps are in a topic, you need to have read what others have written.

As mentioned in other tabs, literature reviews should discuss the big ideas that make up a topic. Each literature review should be broken up into different subtopics. Each subtopic should use groups of articles as evidence to support the ideas. There are several different ways of organizing a literature review. It will depend on the patterns one sees in the groups of articles as to which strategy should be used. Here are a few examples of how to organize your review:

Chronological

If there are clear trends that change over time, a chronological approach could be used to organize a literature review. For example, one might argue that in the 1970s, the predominant theories and themes argued something. However, in the 1980s, the theories evolved to something else. Then, in the 1990s, theories evolved further. Each decade is a subtopic, and articles should be used as examples. 

Themes/Theories

There may also be clear distinctions between schools of thought within a topic, a theoretical breakdown may be most appropriate. Each theory could be a subtopic, and articles supporting the theme should be included as evidence for each one. 

If researchers mainly differ in the way they went about conducting research, literature reviews can be organized by methodology. Each type of method could be a subtopic,  and articles using the method should be included as evidence for each one.

• Define your research question
• Compile a list of initial keywords to use for searching based on question
• Search for literature that discusses the topics surrounding your research question
• Assess and organize your literature into logical groups
• Identify gaps in research and conduct secondary searches (if necessary)
• Reassess and reorganize literature again (if necessary)
• Write review

Here is an example of a literature review, taken from the beginning of a research article. You can find other examples within most scholarly research articles. The majority of published scholarship includes a literature review section, and you can use those to become more familiar with these reviews.

Source:  Perceptions of the Police by LGBT Communities

There are many books and internet resources about literature reviews though most are long on how to search and gather the literature. How to literally organize the information is another matter.

Some pro tips:

• Be thoughtful in naming the folders, sub-folders, and sub, sub-folders.  Doing so really helps your thinking and concepts within your research topic.
• Be disciplined to add keywords under the tabs as this will help you search for ALL the items on your concepts/topics.
• Use the notes tab to add reminders, write bibliography/annotated bibliography
• Your literature review easily flows from your statement of purpose (SoP).  Therefore, does your SoP say clearly and exactly the intent of your research?  Your research assumption and argument is obvious?
• Begin with a topic outline that traces your argument. pg99: "First establish the line of argumentation you will follow (the thesis), whether it is an assertion, a contention, or a proposition.
• This means that you should have formed judgments about the topic based on the analysis and synthesis of the literature you are reviewing."
• Keep filling it in; flushing it out more deeply with your references

Other Resources/Examples

• ISU Writing Assistance The Julia N. Visor Academic Center provides one-on-one writing assistance for any course or need. By focusing on the writing process instead of merely on grammar and editing, we are committed to making you a better writer.
• University of Toronto: The Literature Review Written by Dena Taylor, Health Sciences Writing Centre
• Purdue OWL - Writing a Lit Review Goes over the basic steps
• UW Madison Writing Center - Review of Literature A description of what each piece of a literature review should entail.
• USC Libraries - Literature Reviews Offers detailed guidance on how to develop, organize, and write a college-level research paper in the social and behavioral sciences.
• Creating the literature review: integrating research questions and arguments Blog post with very helpful overview for how to organize and build/integrate arguments in a literature review
• Understanding, Selecting, and Integrating a Theoretical Framework in Dissertation Research: Creating the Blueprint for Your “House” Article focusing on constructing a literature review for a dissertation. Still very relevant for literature reviews in other types of content.

A note that many of these examples will be far longer and in-depth than what's required for your assignment. However, they will give you an idea of the general structure and components of a literature review. Additionally, most scholarly articles will include a literature review section. Looking over the articles you have been assigned in classes will also help you.

• Understanding, Selecting, and Integrating a Theoretical Framework in Dissertation Research: Creating the Blueprint for Your “House” Excellent article detailing how to construct your literature review.
• Sample Literature Review (Univ. of Florida) This guide will provide research and writing tips to help students complete a literature review assignment.
• Sociology Literature Review (Univ. of Hawaii) Written in ASA citation style - don't follow this format.
• Sample Lit Review - Univ. of Vermont Includes an example with tips in the footnotes.

Attribution

Content on this page was provided by Grace Allbaugh

• << Previous: Writing a Literature Review
• Next: PRISMA - Systematic reviews & meta-analyses >>
• Last Updated: Dec 6, 2023 3:23 PM
• URL: https://guides.library.illinoisstate.edu/academicwriting

Additional Links

• Directions and Parking
• Accessibility Services
• Library Spaces
• Staff Directory

Libraries | Research Guides

Literature reviews, what is a literature review, learning more about how to do a literature review.

• Planning the Review
• The Research Question
• Choosing Where to Search
• Organizing the Review
• Writing the Review

A literature review is a review and synthesis of existing research on a topic or research question. A literature review is meant to analyze the scholarly literature, make connections across writings and identify strengths, weaknesses, trends, and missing conversations. A literature review should address different aspects of a topic as it relates to your research question. A literature review goes beyond a description or summary of the literature you have read. 

• Sage Research Methods Core Collection This link opens in a new window SAGE Research Methods supports research at all levels by providing material to guide users through every step of the research process. SAGE Research Methods is the ultimate methods library with more than 1000 books, reference works, journal articles, and instructional videos by world-leading academics from across the social sciences, including the largest collection of qualitative methods books available online from any scholarly publisher. – Publisher

• Next: Planning the Review >>
• Last Updated: Jan 17, 2024 10:05 AM
• URL: https://libguides.northwestern.edu/literaturereviews

• University of Texas Libraries

Literature Reviews

• What is a literature review?
• Steps in the Literature Review Process
• Define your research question
• Determine inclusion and exclusion criteria
• Choose databases and search
• Review Results
• Synthesize Results
• Analyze Results
• Librarian Support

What is a Literature Review?

A literature or narrative review is a comprehensive review and analysis of the published literature on a specific topic or research question. The literature that is reviewed contains: books, articles, academic articles, conference proceedings, association papers, and dissertations. It contains the most pertinent studies and points to important past and current research and practices. It provides background and context, and shows how your research will contribute to the field. 

A literature review should: 

• Provide a comprehensive and updated review of the literature;
• Explain why this review has taken place;
• Articulate a position or hypothesis;
• Acknowledge and account for conflicting and corroborating points of view

From  S age Research Methods

Purpose of a Literature Review

A literature review can be written as an introduction to a study to:

• Demonstrate how a study fills a gap in research
• Compare a study with other research that's been done

Or it can be a separate work (a research article on its own) which:

• Organizes or describes a topic
• Describes variables within a particular issue/problem

Limitations of a Literature Review

Some of the limitations of a literature review are:

• It's a snapshot in time. Unlike other reviews, this one has beginning, a middle and an end. There may be future developments that could make your work less relevant.
• It may be too focused. Some niche studies may miss the bigger picture.
• It can be difficult to be comprehensive. There is no way to make sure all the literature on a topic was considered.
• It is easy to be biased if you stick to top tier journals. There may be other places where people are publishing exemplary research. Look to open access publications and conferences to reflect a more inclusive collection. Also, make sure to include opposing views (and not just supporting evidence).

Source: Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal, vol. 26, no. 2, June 2009, pp. 91–108. Wiley Online Library, doi:10.1111/j.1471-1842.2009.00848.x.

Meryl Brodsky : Communication and Information Studies

Hannah Chapman Tripp : Biology, Neuroscience

Carolyn Cunningham : Human Development & Family Sciences, Psychology, Sociology

Larayne Dallas : Engineering

Janelle Hedstrom : Special Education, Curriculum & Instruction, Ed Leadership & Policy ​

Susan Macicak : Linguistics

Imelda Vetter : Dell Medical School

For help in other subject areas, please see the guide to library specialists by subject .

Periodically, UT Libraries runs a workshop covering the basics and library support for literature reviews. While we try to offer these once per academic year, we find providing the recording to be helpful to community members who have missed the session. Following is the most recent recording of the workshop, Conducting a Literature Review. To view the recording, a UT login is required.

• October 26, 2022 recording
• Last Updated: Oct 26, 2022 2:49 PM
• URL: https://guides.lib.utexas.edu/literaturereviews

How to Write a Literature Review

• What Is a Literature Review

What Is the Literature

• Writing the Review

The "literature" that is reviewed is the collection of publications (academic journal articles, books, conference proceedings, association papers, dissertations, etc) written by scholars and researchers for scholars and researchers. The professional literature is one (very significant) source of information for researchers, typically referred to as the secondary literature, or secondary sources. To use it, it is useful to know how it is created and how to access it.

The "Information Cycle"

The diagram below is a brief general picture of how scholarly literature is produced and used. Research does not have a beginning or an end; researchers build on work that has already been done in order to add to it, thus providing more resources for other researchers to build on. They read the professional literature of their field to see what issues, questions, and problems are current, then formulate a plan to address one or a few of those issues. Then they make a more focused review of the literature, which they use to refine their research plan. After carrying out the research, they present their results (presentations at conferences, published articles, etc) to other scholars in the field, i.e. they add to the general subject reading ("the literature").

  Research may not have a beginning or an end, but researchers have to begin somewhere. As noted above, the professional literature is typically referred to as secondary sources. Primary and tertiary sources also play important roles in research. Note, though, that these labels are not rigid distinctions; the same resource can overlap categories.

• Lab reports (yours or someone else's) - Records of the results of experiments.
• Field notes, measurements, etc (yours or someone else's) - Records of observations of the natural world (electrons, elephants, earthquakes, etc).
• Journal articles, conference proceedings , and similar publications reporting results of original research.
• Historical documents - Official papers, maps, treaties, etc.
• Government publications - Census statistics, economic data, court reports, etc.
• Statistical data - Measurements (counts, surveys, etc.) compiled by researchers.
• First-person accounts - Diaries, memoirs, letters, interviews, speeches
• Newspapers - Some types of articles, e.g. stories on a breaking issue, or journalists reporting the results of their investigations.
• Published writings - Novels, stories, poems, essays, philosophical treatises, etc
• Works of art - Paintings, sculptures, etc.
• Recordings - audio, video, photographic
• Conference proceedings - Scholars and researchers getting together and presenting their latest ideas and findings
• Internet - Web sites that publish the author's findings or research; e.g. your professor's home page listing research results. Note: use extreme caution when using the Internet as a primary source … remember, anyone with internet access can post whatever they want.
• Archives - Records (minutes of meetings, purchase invoices, financial statements, etc.) of an organization (e.g. The Nature Conservancy), institution (e.g. Wesleyan University), business, or other group entity (even the Grateful Dead has an archivist on staff).
• Artifacts - manufactured items such as clothing, furniture, tools, buildings
• Manuscript collections - Collected writings, notes, letters, diaries, and other unpublished works.
• Books or articles - Depending on the purpose and perspective of your project, works intended as secondary sources -- analyzing or critiquing primary sources -- can serve as primary sources for your research.
• Secondary - Books, articles, and other writings by scholars and researchers reporting their analysis of their primary sources to others. They may be reporting the results of their own primary research or critiquing the work of others. As such, these sources are usually a major focus of a literature review: this is where you go to find out in detail what has been and is being done in a field, and thus to see how your work can contribute to the field.   
• Summaries / Introductions - Encyclopedias, dictionaries, textbooks, yearbooks, and other sources which provide an introductory or summary state of the art of the research in the subject areas covered. They are an efficient means to quickly build a general framework for understanding a field.
• Indexes to publications - Provide lists of primary and secondary sources of more extensive information. They are an efficient means of finding books, articles, conference proceedings, and other publications in which scholars report the results of their research.

Work backwards . Usually, your research should begin with tertiary sources:

• Tertiary - Start by finding background information on your topic by consulting reference sources for introductions and summaries, and to find bibliographies or citations of secondary and primary sources.
• Secondary - Find books, articles, and other sources providing more extensive and thorough analyses of a topic. Check to see what other scholars have to say about your topic, find out what has been done and where there is a need for further research, and discover appropriate methodologies for carrying out that research. 
• Primary - Now that you have a solid background knowledge of your topic and a plan for your own research, you are better able to understand, interpret, and analyze the primary source information. See if you can find primary source evidence to support or refute what other scholars have said about your topic, or posit an interpretation of your own and look for more primary sources or create more original data to confirm or refute your thesis. When you present your conclusions, you will have produced another secondary source to aid others in their research.

Publishing the Literature

There are a variety of avenues for scholars to report the results of their research, and each has a role to play in scholarly communication. Not all of these avenues result in official or easily findable publications, or even any publication at all. The categories of scholarly communication listed here are a general outline; keep in mind that they can vary in type and importance between disciplines.

Peer Review - An important part of academic publishing is the peer review, or refereeing,  process. When a scholar submits an article to an academic journal or a book manuscript to a university publisher, the editors or publishers will send copies to other scholars and experts in that field who will review it. The reviewers will check to make sure the author has used methodologies appropriate to the topic, used those methodologies properly, taken other relevant work into account, and adequately supported the conclusions, as well as consider the relevance and importance to the field. A submission may be rejected, or sent back for revisions before being accepted for publication.

Peer review does not guarantee that an article or book is 100% correct. Rather, it provides a "stamp of approval" saying that experts in the field have judged this to be a worthy contribution to the professional discussion of an academic field.

Peer reviewed journals typically note that they are peer reviewed, usually somewhere in the first few pages of each issue. Books published by university presses typically go through a similar review process. Other book publishers may also have a peer review process. But the quality of the reviewing can vary among different book or journal publishers. Use academic book reviews or check how often and in what sources articles in a journal are cited, or ask a professor or two in the field, to get an idea of the reliability and importance of different authors, journals, and publishers.

Informal Sharing - In person or online, researchers discuss their ongoing projects to let others know what they are up to or to give or receive assistance in their work. Conferences, listservs, and online discussion boards are common avenues for these discussions. Increasingly, scholars are using personal web sites to present their work.

Conference Presentations - Many academic organizations sponsor conferences at which scholars read papers, display at poster sessions, or otherwise present the results of their work. To give a presentation, scholars must submit a proposal which is reviewed by those sponsoring the conference. Unless a presentation is published in another venue, it will likely be difficult to find a copy, or even to know what was presented. Some subject specific indexes and other sources list conference proceedings along with the author and contact information.

Conference Papers / Association Papers / Working Papers - Papers presented at a conference, submitted but not yet accepted for publication, works in progress, or not otherwise published are sometimes made available by academic associations. These are often not easy to find, but many are indexed in subject specific indexes or available in subject databases. Sometimes a collection of papers presented at a conference will be published in a book.

Journals - Articles in journals contain specific analyses of particular aspects of a topic. Journal articles can be written and published more quickly than books, academic libraries subscribe to many journals, and the contents of these journals are indexed in a variety of sources so others can easily find them. So, researchers commonly use articles to report their findings to a wide audience, and journals are a good readily available source for anyone researching current information on a topic.

• Research journals - Articles reporting in detail the results of research.
• Review journals - Articles reviewing the literature and work done on particular topics.
• News/Letters journals - News reports, brief research reports, short discussions of current issues.
• Proceedings/Transactions journals - A common venue for publishing conference papers or other proceedings of academic conferences.
• General interest magazines - News and other magazines that report scholarly findings for a general, nonacademic audience. These are usually written by journalists (who are usually not academically trained in the field), but sometimes are written by researchers (or at least by journalists with training in the field). Magazines are not peer reviewed, and are usually not academically useful sources of information for research purposes, but they can alert you to work being done in your field and give you a quick summary.
• Trade journals and magazines - These are written for people working in a particular industry or profession, such as advertising, banking, construction, dentistry, education. Articles are generally written by and for people working in that trade, and focus on current topics and developments in the trade. They do not present academic analyses of their topics, but they can provide useful background or context for academic work if the articles are relevant to your research.

Books - Books take a longer time than articles or conference presentations to get from research to publication, but they can cover a broader range of topics, or cover a topic much more thoroughly. University press books typically go through some sort of a peer review process. There is a wide range of review processes (from rigorous to none at all) among other book publishers.

Dissertations/Theses - Graduate students working on advanced degrees typically must perform a substantial piece of original work, and then present the results in the form of a thesis or dissertation. A master's thesis is typically somewhere between an article and a book in length, and a doctoral dissertation is typically about the length of a book. Both should include extensive bibliographies of their topics. 

Web sites - In addition to researchers informally presenting and discussing their work on personal web pages, there are an increasing number of peer reviewed web sites publishing academic work. The rigor, and even existence, of peer reviewing can vary widely on the web, and it can be difficult to determine the reliability of information presented on the web, so always be careful in relying on a web-based information source. Do your own checking and reviewing to make sure the web site and the information it presents are reliable.

Reference Sources - Subject encyclopedias, dictionaries, and other reference sources present brief introductions to or summaries of the current work in a field or on a topic. These are typically produced by a scholar and/or publisher serving as an editor who invites submissions for articles from experts on the topics covered.

How to Find the Literature

Just as there are many avenues for the literature to be published and disseminated, there are many avenues for searching for and finding the literature. There are, for example, a variety of general and subject specific indexes which list citations to publications (books, articles, conference proceedings, dissertations, etc). The Wesleyan Library web site has links to the library catalog and many indexes and databases in which to search for resources, along with subject guides to list resources appropriate for specific academic disciplines. When you find some appropriate books, articles, etc, look in their bibliographies for other publications and also for other authors writing about the same topics. For research assistance tailored to your topic, you can sign up for a Personal Research Session with a librarian.

• << Previous: What Is a Literature Review
• Next: Writing the Review >>
• Last Updated: Jan 11, 2024 9:48 AM
• URL: https://libguides.wesleyan.edu/litreview

• Writing & Citing
• Paraphrasing
• Literature Reviews
• Primary & Secondary Sources
• Database Citations
• BibTheo Style Guide

By the time you're reading this, you've probably written at least a few term papers during your time in school, whether at the high school or college level, but now you're in your major courses and professors are saying that it's time to write a "Literature Review." If this is the first time you're hearing of a paper like this, you're not alone! Literature reviews can seem overwhelming, but they are doable. This guide will help you determine what a literature review is, how to structure your literature review, how to summarize a journal article, and where to find your peer-reviewed resources.

What is a Literature Review?

There is one major difference between the term papers you've written before and the literature review you're writing now: Goal of the final product.

Term papers are written to research a specific topic that you have an opinion on and, in a way, provide informaiton to prove that your opinion is the most accurate according to the supporting research. You'll often find that term papers include things like counterarguments and emotion-weighted words. These aren't bad things! They're very important to term papers! However, literature reviews have a different end-goal.

Literature reviews are written to do one thing and one thing only: Review the literature. If you have an opinion on your topic (which, hopefully, you do!), the literature review  is not  the time to talk about it. For a literature review, you're going over the research that has already been done to establish a baseline of knowledge between you and your reader. You're looking to figure out what the experts already know and what they haven't figured out yet (also referred to as "gaps in the literature"). You're summarizing articles and drawing connections between them; not much more and not much less.

How to Structure a Literature View

If your professor has already given you an outline, ignore this box completely, and follow your professor's provided outline!

If you're writing your literature review from a blank slate, you can choose what kind of structure you want your literature review to have:

• Chronological = If you're writing about the history of a topic for your literature review, you can structure your review in such a way that indicates theory origins, early research, and more recent research and applications. For example, if you're researching applications for a theory like "Color Theory," then you may benefit most from a historical look from past to present.
• Topical  = If your topic doesn't necessarily have a clear historical timeline or you're looking at a specific application of a theory, you'll likely need an outline that allows you to review topics of interest to your research. For example, if your topic is "Experiences of Hope in Religious Students," then you don't need to give a history; you only need to review current research that covers the topic you're looking for. Your topical outline may include headings like "Christianity and Experiences of Hope," "Buddhism and Experiences of Hope," etc.  or  "Elementary Students' Experiences of Hope," "Middle School Students' Experiences of Hope," etc.
• Synthetic  = Honestly, don't pick this one unless you have to; this is the hardest kind of literature review to write, but is best if only a little bit of research has been done on your topic of interest. For example, if your topic is "Information seeking habits of squirrels on a college campus," not much research exists on this topic. Instead, you'll need to search for something like "how squirrels learn," "habits of squirrels on a college campus," and then combine (AKA synthesize) the information you found to draw conclusions where necessary. This is the most difficult structure because it requires advanced writing skills; this outline might be the right one if you're having trouble finding relevant resources to your topic, but talk to a librarian first, and see if we aren't able to help you find sources!

The structure you choose will determine how your outline is best set up; however, every outline should include both an introduction and a conclusion. Everything that's mentioned above is to help you figure out all the stuff in the middle.

Writing Your Conclusion & Introduction

Every term paper you've written up until now should have included an introduction and a conclusion, and your literature review is no different in that regard! Your conclusion will be the same as it has always been: A paragraph-ish summary of your paper, tying up all of your loose ends, and drawing any final conclusions for your reader. In literature reviews, your conclusion can (and should!) also include information on gaps in the literature; these are those areas or facets that very few people (or no one at all!) have researched yet. This is a great place to talk about where future research can go, including your current research that you're doing.

While your conclusion is still just a conclusion but with an extra flair, your introduction is likely to look a bit different than the ones you've written for past term papers. It's going to be much longer than 4-5 sentences, and it will include a great deal more in it. Here's something of an outline that you can consider for your introduction:

• Hook (an attention-catching statement that gets your reader to actually read your paper)
• Statistics (include any relevant statistics to your topic; consider things like how many people are affected by your topic)
• Definitions (define any terms/phrases/keywords you'll be using throughout your paper; even if you already think your reader knows the definition of an academic term, define it anyway to make sure you and your reader are on the same page) [Note: Figuring out what terms to define might be a bit tricky at first. Start with any term that could have multiple interpretations. For example, if your paper is about "Experiences of Hope in Religious Students," you may find it best to define the terms "Hope" and "Religious/Religion" so your readers don't misunderstand.]
• History (this is especially useful if you're doing a topical or synthetic outline, but it also applies to chronological; provide short biographical information, timelines, or key historical events to your topic.)
• Theories/Approached (this may not apply to every literature review or every topic, but, in general, think about whether or not there is already a baseline idea that your topic is based off of and discuss the main tenants of that theory/approach [e.g. "Cognitive Behavioral Theory" or "Catholocism"].)
• Thesis statement (avoid "I" and "You" statements like "In this paper, I am going to teach you about..." These statements aren't academic; if you need help formatting a thesis statement, you can visit the learning center on campus or reach out to writing center if you're online.)

Keep in mind that this is a generic/general outline. Your paper's introduction may include more (or even a little bit less!) than what's been listed here. It may be in a different order (maybe you define your terms and then give statistics). Not every introduction is going to look exactly the same, nor should they! As long as they give your reader the most basic understanding of what your paper discusses, you're well on your way to a passing literature review!

How to Summarize an Article

So we know now that a literature review, however it's structured, doesn't involve your own opinion. That leaves one major question: What  does  go in a literature review? What does "review the literature" actually mean? At the foundational level, what goes into a literature review are summaries of the peer-reviewed/scholarly/academic journal articles you find while researching. These summaries will help your readers understand what research already exists and how it applies to your theory or research topic. All you need to do after writing a summary is make the information connect by drawing bridges between articles, using transition statements (you can visit the Learning Center on campus or reach out to the Writing Center online if you need help with your writing and transitions), and pointing out agreements (or disagreements where appropriate!) in the research you're summarizing.

(Pro tip! Ever heard of an article abstract being referred to as that article's summary? The summaries you'll be writing and the article's abstract are pretty different. This means you can't just copy and paste an article's abstract into your paper. Not only is this not the right kind of summary your professor is looking for, this is also considered plagiarism. You can  read  an article abstract to help you figure out what might be important to your paper, but do not  copy  the abstract and paste it into your paper.)

Sounds simple enough, right? But if you've never summarized an article, how do you know what information to include? Our best recommendation is to use the following resource that was created and refined through a collaborative process between librarians and professors. This Journal Article Review Worksheet gives you step-by-step guidance on summarizing an article effectively and includes websites to help you determine key pieces of information like what kind of research you're looking at and how it can be used. After you've gotten all of your information into the worksheet, you can combine the aspects most important to your research/topic/theory into a 1-paragraph (sometimes 2-paragraph) summary that you'll be able to copy and paste into your literature review paper (don't forget to make your summary flow!).

• Journal Article Review Worksheet Note: While this document focuses primarily on research in the behavioral sciences (e.g. psychology research articles), the information should be relatively generalizable to most research articles. If this document isn't helpful, you don't have to use it at all, but it may be a good idea to look it over to understand the broad concepts of article summary.

Where to Find Peer-Reviewed/Academic/Scholarly Journal Articles

Finding your resources should be a breeze with Nelson Memorial Library's variety of databases and FAQs on how to search them! We recommend starting at the guide that's been created and curated with your subject in mind. Remember to choose the subject guide that matches the topic you're researching most closely (i.e. don't use the psychology subject guide for theological research or vis versa).

• Subject Guides

If you're having trouble figuring out how to use the databases you've found, check out our Library FAQs to see if we don't already have an answer for you!

​​​​​​THIS IS A TEST

If all else fails and you're still having trouble doing your research, never fear: That's exactly why your librarians are here to help you! You can schedule an appointment with any librarian by using our online scheduling service.

• << Previous: Paraphrasing
• Next: Primary & Secondary Sources >>
• Last Updated: Apr 12, 2024 12:51 PM
• URL: https://sagu.libguides.com/writingciting

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health ( m-health ) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)
• Disable Glossary Links

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

• Research Guides
• Vanderbilt University Libraries

Peabody Library

Literature reviews.

• Publishing in Academic Journals
• Steps to Success: The Literature Review Process
• Literature Reviews Webinar Recording
• Literature Review Table
• Writing Like an Academic
• Managing Citations This link opens in a new window

Calls for Papers

• H-Net CFPs in the social sciences and humanities.
• CFPS from AERA Compiled by the American Educational Research Association.

Why Publish?

As a grad/professional student, you will write (or already have written!) a LOT of papers. Rather than tossing that paper aside after you've received your grade, consider turning it into a manuscript submission. There are many benefits for publishing as a grad student:

• You've already done a lot of the work! You may just need to update your research a bit, and re-format your paper for the journal you're submitting to (more on that below).
• It's a great CV booster. Whether you're looking for a job, applying to another graduate program, or just looking for an internship, having a published or accepted publication on your resume looks great.
• Publishing is also a great way to grow your professional network. While the peer review process means you most likely won't know the identify of your reviewers, journal editors are often available to discuss why your article might be a good fit for their journal. Not only are they providing valuable feedback on your writing, you now have a new professional contact!

Things To Think About

• Where Should You Publish?
• What is Impact Factor?
• Crafting Your Submission
• Copyright for Authors

Things to consider:

• Reputation - The reputation of the publisher, journal, editor and editorial board can give an indication of the quality of the journal.
• Scope and focus of the journal - It is important that your article reaches the readers who can most benefit from it and who can most benefit you. The scope and aim of the journal will give an indication of who the journal’s readers are, e.g. national or international, limited to a select area of research or with a multidisciplinary focus.
• Turnaround time - What is the length of the review process? Average length of time from submission to acceptance or rejection; from acceptance to publication? Frequency of publication?
• Included in prominent indexes - Are articles from the journal indexed in journal databases relevant to your field, or in citation databases such as Scopus or Web of Science?
• Editorial standards / Journal information - The competence of a journal’s editorial office can hugely influence the success or failure of an article. Make sure that the “Instructions to Authors” are easily accessible and that they set out clearly what is expected from authors. Does the journal come out on time or is it often two or three years behind? Is the journal carefully produced with a professional appearance, or does it have many typing errors, poor paper quality and other signs of neglect? Does the journal accept electronic submissions? This simplifies the submission process, allows swift management of manuscripts and makes it possible for authors to track the position of their manuscripts in the review process.
• Acceptance rate - The acceptance rate gives an indication of how competitive a journal is. Journals with a low acceptance rate are considered to be amongst the most prestigious in their field, the assumption being that only the very best articles are selected.
• Cost - Be aware that some journals charge either a submission fee, an acceptance fee, page fees or fees for use of colour images or other special media formats.
• Rights for authors - Check the journal website or their copyright form for information on author rights. Are you allowed to re-use the article after publication or to submit the post-print to the University’s research repository?
• Type of publication - Some journals only accept certain types of articles for publication.

More resources:

• Ulrichsweb This link opens in a new window ISSN, publisher, language, subject, abstracting & indexing coverage, full-text database coverage, tables of contents, and reviews written by librarians on periodicals of all types: academic and scholarly journals, e-journals, peer-reviewed titles, popular magazines, newspapers, newsletters, etc. more... less... User Limit: 5.

• Cabell's directory of publishing opportunities in educational psychology and administration Call Number: Peabody Reference Z286 .E3 C323
• Vanderbilt University Institutional Repository Vanderbilt's Institutional Repository. This is a great way to gain a wider audience for your work.

A  journal's  Impact Factor  ind icates the average number of times articles from the publication have been cited over a two-year period.  The higher the journal's impact factor, the more prominent the journal is in the canon of literature.

• Web of Knowledge Journal Citation Reports The official entity providing a journal's impact factor.
• CiteFactor A repository of open access journals, including publication information about journals, such as impact factor.
• Altmetrics Altmetrics considers how scholarship-based activities such as blog posts, tweets, downloads, shares, and views may provide important information about the significance of a research article or other research "product." Find more information on this research guide.

Author guidelines of most journals can be found on the journal website, which can be found via Google or by using Ulrich's.

Make sure to follow the guidelines closely and prepare your submission in accordance with these guidelines. Things like word count, format of the manuscript (text, illustrations, etc), and citation style are all very important.

Make sure to also check if the journal accepts submissions on a rolling basis, or if you need to inquire with the editor first.

• Journal of scholarly publishing Provides practical advice for authors, editors and publishers for a wide range of topics.
• Tips from Editors on Getting Published

• More information on Copyright Videos include: What are Copyright and Using Others' Work
• Authors' Rights Find out more about your rights as an artist, author, or creator of any sort
• Open Access Publishing Guide from Vanderbilt Library
• << Previous: Writing Like an Academic
• Next: Managing Citations >>
• Last Updated: Feb 29, 2024 4:20 PM
• URL: https://researchguides.library.vanderbilt.edu/peabody/litreviews

Reviews of Peer-Reviewed Journals in the Humanities and Social Sciences

We give you the scuttlebutt on academic journals—aiding you in selecting the right journal for publication—in reviews that are sometimes snarky, sometimes lengthy, always helpful. Written by Princeton University graduate students and Wendy Laura Belcher.

Category Archives: Literary Studies Journals

• Journal Article Prize Competitions
• Journal Disciplines
• Journals Reviewed
• Journals Ranking

MLN (Modern Language Notes)

For those interested in publishing articles that explore French, German, Hispanic, or Italian literature, film, theory and pedagogy. MLN may not be quite as well-known … Continue reading →

PMLA (Publications of the Modern Language Association)

For those interested publishing articles that discuss timely and culturally relevant literature and archival materials. For those looking to make broad connections (not close reading) … Continue reading →

Journal of Modern Literature

For those interested in publishing articles that deal with literature from any language or tradition, 1900-present. Ranked 19 out of all literature journals by Google … Continue reading →

French Forum

For those interested in publishing articles on French and Francophone Literature and films of all periods but preferably post-1900. Founded by Virginia and Raymond La … Continue reading →

Essays in Romanticism

For those interested in publishing articles that engage some aspect of Romantic literature (mainly British or, in a few instances, German)—often complicating or developing a … Continue reading →

Comparative Literature Studies

For those interested in publishing articles of a comparative nature on literary history, the history of ideas, critical theory, studies between authors, and literary relations … Continue reading →

Modernism/modernity

For those interested in publishing articles that, as the journal’s name suggest, deal with either Modernism or the more broadly defined concept of modernity. As … Continue reading →

The Eighteenth Century: Theory and Interpretation

For those interested in publishing articles on literature of the eighteenth century, mainly British (but articles can have New World or colonial connections), often with … Continue reading →

Nineteenth-Century French Studies

For those who would like to publish scholarly articles on nineteenth-century French literature, culture, and social/political issues. NCFS is independent journal that purports to publish … Continue reading →

Philosophy and Literature

For those who wish to publish articles in which they address an aesthetic object (usually literature) with an attitude explicitly attuned to formal concerns and … Continue reading →

French Studies

For those interested in publishing articles that concern a broad range of topics in French and Francophone literature, film, culture, and theory from the medieval … Continue reading →

Modern Philology

For those interested in publishing articles that speak to an aesthetic problem in a canonical text written before 1900 by a Western author, that deeply … Continue reading →

For those interested in publishing very formal articles that directly take up narrative theory and narratology and who are ready to subordinate their politics and … Continue reading →

MELUS (Multi-Ethnic Literature of the United States)

For those with articles that provincialize or problematize the idea of the U.S. and its literary output. MELUS is a quarterly journal, and one of … Continue reading →

African American Review

For those with articles that have something substantial and innovative to say about canonical and non-canonical African American Literary texts; this journal is an especially … Continue reading →

Callaloo: A Journal of African Diaspora Arts and Letters

For those interested in publishing articles that creatively and/or critically engage with the work of African Americans and peoples of African descent throughout the African Diaspora. … Continue reading →

Victorian Literature and Culture (stub)

Submit here if you have a really interesting, interdisciplinary article about Science, Race/Imperialism, Gender (mostly women), Death, Religion, Sympathy, Victorian institutions (museums, pawnshops, libraries, etc), … Continue reading →

Victorians Institute Journal (stub)

For those interested in publishing close readings of a single work of a popular Victorian author, especially Hardy or Dickens.

Victorian Review (stub)

For those interested in publishing an article on a lesser known Victorian author or text, or one on an obscure topic in relation to a … Continue reading →

Nineteenth Century Literature (stub)

For those interested in publishing articles with very broad claims about an author, discourse, or text.

Dickens Studies Annual (stub)

For those interested in publishing articles with close readings of any of Dickens’s novels or short stories or of his life and legacy.

Eighteenth-Century Life

Submit here if you’re writing about a canonical British author, have a historical (not theoretical) perspective, and want to talk about the author’s journals and … Continue reading →

Eighteenth-Century Studies

Submit here if you have a substantial amount of historical or contemporary (aka 18th century) theoretical background or if you’re writing about something global (not … Continue reading →

Journal of Postcolonial Writing

For those interested in publishing on texts written in English after 1950, particularly on an individual postcolonial author or text; a critique of contemporary postcolonial … Continue reading →

Comparative Literature

For those interested in publishing theoretically rigorous articles with large claims on different national or linguistic literatures and with a focus on transmission or circulation. … Continue reading →

SEL: Studies in English Literature, 1500-1900

Submit here if you don’t mind waiting for the annual issue on your time period; if you’re working in the 19th century, submit if you have … Continue reading →

Tulsa Studies in Women’s Literature

Submit here if you’re writing about women (especially multiple works by a single author) and talk really explicitly about the larger implications of your argument. … Continue reading →

ELH (English Literary History)

For those interested in publishing a traditionally structured article (historical/ theoretical background followed by close reading); if you’re working in the 19th century, write about … Continue reading →

A Systematic Literature Review of Substance-Use Prevention Programs Amongst Refugee Youth

• Original Paper
• Open access
• Published: 09 April 2024

Cite this article

You have full access to this open access article

• Elijah Aleer 1 ,
• Khorshed Alam   ORCID: orcid.org/0000-0003-2232-0745 2 &
• Afzalur Rashid   ORCID: orcid.org/0000-0003-3413-1757 1  

38 Accesses

Explore all metrics

This paper aims at exploring existing literature on substance use prevention programs, focusing on refugee youth. A comprehensive search for relevant articles was conducted on Scopus, PubMed, and EBSCOhost Megafile databases including Academic Search Ultimate, APA PsycArticles, APA PsycInfo, CINAHL with Full Text, E-Journals, Humanities Source Ultimate, Psychology and Behavioural Sciences Collection, and Sociology Source Ultimate. Initially, a total of 485 studies were retrieved; nine papers were retained for quality assessment after removing duplicates. Of the nine studies that met the inclusion criteria, only three are found to partially addressed substance use prevention programs. The two substance use prevention programs that emerge from the study are Adelante Social and Marketing Campaign (ASMC), and Screening and Brief Intervention (SBI). Six others explored protective factors and strategies for preventing substance use. The study findings show that refugee youth held negative attitudes toward institutions that provide substance use prevention programs. This review concluded that refugee youth often experience persistent substance use as they are not aware of prevention programs that may reduce the prevalence and/or severity of such misuse.

Similar content being viewed by others

Substance Abuse Amongst Refugee Youth: A Scoping Review

Elijah Aleer, Khorshed Alam, … Douglas Eacersall

Preventing Substance Use Among Indigenous Adolescents in the USA, Canada, Australia and New Zealand: a Systematic Review of the Literature

Mieke Snijder, Lexine Stapinski, … Nicola Newton

Substance Use Among Refugee and Conflict-Affected Children and Adolescents

Avoid common mistakes on your manuscript.

Introduction

Increasingly, literature suggests that refugee youth face a heightened vulnerability to substance use, coupled with a limited awareness of substance use prevention programs. Refugees’ susceptibility to substance use is linked to adverse living conditions and maladaptive coping mechanisms (Posselt et al., 2015 ; Ramachandran et al., 2019 ; Roberts et al., 2011 ). As a result, research suggests that the prevalence of substance use amongst refugees ranges from 17 to 37% in camps and 4% to7% in the community setting (Horyniak et al., 2016a ). Another study revealed that 14.9% of men and 0.7% of women from refugee background exhibited substance use (Ramachandran et al., 2019 ). The concerning aspect of this situation lies in the fact that substance use and its associated risks are well-documented within refugee setting (Gire et al., 2019 ; Luitel et al., 2013 ), with a growing call to integrate substance use prevention programs into refugee services due to the prevalence of the phenomenon (Horyniak et al., 2016a ). Such recommendation emphasises the importance of addressing the knowledge gap on substance use prevention programs amongst the refugee youth. Research indicates that if the substance use prevention programs are not made known to those at risk individuals, it could have detrimental effects on such individuals (Bauman and Phongsavan,  1999 ). Failure to address the knowledge gap of substance use prevention programs could place, refugee youth at an increasing risk of various negative outcomes such as disorder, higher mortality, accidental injury, liver diseases, violence, dysfunctional work, and school dropout due to substance use (Ji et al., 2021 ; Kuntsche et al., 2017 ; Li et al., 2017 ; Momeñe et al., 2021 ). Hence, it is important to document the knowledge of substance use prevention programs amongst refugee youth in the literature to ensure that the groups are informed about the negative consequences.

As per this study, substance use prevention programs refer to a myriad of substance-free and medication treatments administered to assist individuals to reduce substance use (Alayan et al., 2021 ). While substance use refers to as a prolonged harmful use of any substance, which can result in problems such as non-fulfilling social roles, withdrawal and tolerance symptoms, substance use disorders and attributable to burden of disease and mortality (American Psychiatric Association, 2013 ; Rehm et al., 2013 ). In this case, substances can include alcohol, cannabis, methamphetamine and other stimulants drugs, non-medical use of pharmaceutical drugs, illicit opioids including heroin, tobacco and other emerging psychoactive substances (AIHW, 2020 ). In Australia, youth refers to a person aged between 12 and 24 years (AIHW, 2021 ). Accordingly, refugee youth in this study are those between the ages of 12 and 24.

Substance Use Prevention Programs

There are several substance use prevention programs in the literature, the aim of which are to reduce harms of substance use. The last two decades have witnessed a surge in studies conducted on substance use prevention programs for different socio-demographic groups that produced information about the initiation, prevalence and associated behavioural, social, and educational outcomes (Fishbein et al., 2006 ; Gau et al., 2012 ; Gruenewald et al., 2009 ; Springer et al., 2004 ). The surge in research reaffirms that substance use prevention programs play an important role in reducing the consequences of substance use. Notably, there are several factors which permit individuals to engage in use substance. These include peer pressure, poor neighbourhood, inability to cope with difficulties, cultural norms, family history of drug use and lower level of education. Family structure and mental disorder play a vital role in initiation and maintenance of substance use (Gattamorta et al., 2017 ; Peloso et al., 2021 ). The knowledge of various factors, that induce individuals to use substances is vital as they play a significant role when designing substance use prevention programs.

Some of the known substance use prevention programs include individual and group counselling, alternative programs, and family and community interventions (Barrett et al., 1988 ; Foss-Kelly et al., 2021 ; Radoi, 2014 ). These programs are designed to influence social and psychological factors associated with the initiation and maintenance of substance use (Barrett et al., 1988 ). The social factors include peer pressure, a deviation from conventional values. Including those of one’s family, school, and religion, while the psychological characteristics include low self-esteem and an attitude of tolerance towards deviancy (Barrett et al., 1988 ; Hater et al., 1984 ; Radoi, 2014 ). Substance use prevention programs aim to approach social and psychological factors in a unique way depending on their goal and outcome. Each of the factors requires a different approach when designing a substance use prevention program. For example, the primary objective of providing counselling to young individuals who engage in substance use is to assist them in overcoming their low self-esteem and embracing the positive societal norms that are linked to such behaviour (Barrett et al., 1988 ). The effectiveness of an individual program depends on the participants’ attitude toward intervention and their outcomes (Espada et al., 2015 ). For instance, participants sometimes refuse to join the prevention program due to fear of being reported to authorities (Kvillemo et al., 2021 ).

Peer pressure is widely acknowledged as a significant source of the initiation and maintenance of substance use amongst youth. According to social learning theory, youth substance use is a consequence of peer pressures originating from their reference groups (Watkins, 2016 ). To address the substance use where such pressure is deemed to be the initiation and maintenance factor, group counselling is believed to be a key prevention program (Barrett et al., 1988 ). This is because peer relations play a powerful influence, and therefore, researchers often use group counselling rather than individual counselling to promote healthy and acceptable relationships, foster social skills, and thus to develop healthy forms of recreational activities amongst peers.

Apart from counselling, adopting alternative programs such as substance-free strategies reduce the initiation and maintenance factors of substance use. Behavioural economic theory suggests that an increase in rewarding substance-free activities can lead to a reduction in substance use (Murphy et al., 2019 ). The structured substance-free activities approach is based on the relationship between the reinforcement derived from substance-related activities to the reinforcement derived from substance-free activities (Correia et al., 2005 ). Research shows that substance use programs that are supplemented with either relaxation training or a behavioural economic session focused on increasing substance-free activities are associated with reductions in substance use (Murphy et al., 2019 ). Notably, increasing substance-free activities is suggested to be useful in substance use prevention in vulnerable youth (Andrabi et al., 2017 ).

Community, family, academic engagements, work, and religious activities play a significant role in reducing the initiation and maintenance of substance use and its related consequences. Similarly, individual and group counselling, alternative programs, and family and community interventions have also led to a reduction in the initiation and maintenance of substance use amongst youth. Research demonstrated a negative relationship between commitment to conventional values such as family, religion, and education, and substance use amongst the youth (Sussman et al., 2006 ). This evidence is supported by social bond theory, which postulates that commitment to conventional values of one’s family, religion, and school act to prevent deviant responses (Nijdam-Jones et al., 2015 ). Similarly, the Family Interaction Theory suggests that social learning, parent attachment, and intrapersonal characteristics equally discourage youth risk-taking behaviours (Ismayilova et al., 2019 ). The evidence appeared in several substance use prevention programs (Huang et al., 2014 ; Ishaak et al., 2015 ; Liddle et al., 2006 ). For instance, the Adolescent Day Treatment Program (ADTP) in Canada implements a social learning approach stressing positive support for appropriate substance, anti-social coping behaviour, and social skills (Liddle et al., 2006 ).

Some substance use prevention programs are designed to assist individuals with the development of skills and attitudes through a community approach. The approach has seen youth cessation of substance use and helped them make changes leading to substance-free lifestyles (Wade-Mdivanian et al., 2016 ). One of the substance use prevention programs, which adopts a community approach is Multidimensional Family Treatment (DFT). DFT targets the initiation and maintenance of youth substance use by addressing coping strategies, parenting practices, other family members, and interactional patterns that contribute to the continuation of substance use and related consequences (Liddle et al., 2006 ). DFT also addresses the functioning of youth and family using the social systems influencing the youth’s life such as school, work, peer networks, and the juvenile justice system (Liddle et al., 2006 ; Valente et al., 2007 ). In support of the community approach, researchers argue for the inclusion of the perspectives of community members in substance use prevention programs because they understand the unique needs of the people with whom they share a bond (Bermea et al., 2019 ). Researchers also focus the interconnected nature of their socio-environmental relationships that can facilitate advocacy for change at the community level (Bermea et al., 2019 ).

Research Gap

Despite the vast knowledge of substance use prevention programs in the literature, research on the refugee youth remains scarce. The lack of research on substance use prevention programs for refugee youth may be due to many factors. First, scholars might have ignored the severity of the issues amongst the groups. Secondly, the socio-economic benefits of the prevention programs might have been underestimated in the literature. Thirdly, the political aspect of substance use prevention programs for refugee youth might have not been thoroughly evaluated in the policy frameworks. The socio-economic benefit of substance use prevention programs underscores a pressing need to begin synthesizing evidence given the deleterious nature of substance use if it is left unmitigated. The knowledge of substance use prevention programs is significant to vulnerable groups like refugee youth because they seek assistance whenever they succumb to substance use. As a result, they will avoid the negative consequences of substance use and subsequently exploit the social benefit. Furthermore, the knowledge of substance use prevention programs can assist organisations and advocacy groups assisting refugee youth to provide them with better services.

This study aims at contributing to substance use prevention programs literature by conducting a systematic literature review to synthesize evidence on such programs, their attitudes towards the program, and amongst refugee youth to fill the gaps in knowledge and provide directions for future research.

Research Questions

The following questions are designed to achieve the aims and objectives of the systematic literature review:

What different substance use prevention programs are used to assist refugee youth with substance use?

What is the refugee youth’s attitude toward substance use prevention programs?

What are the outcomes of a substance use prevention program?

To ensure the validity and reliability of this study, systematic review guidelines are followed (Toews, 2017 ). This is because the systematic review is useful in mapping out areas of uncertainty, identifying the lack of research on a particular topic, and pointing out an area where research is needed (Rethlefsen et al., 2021 ). The systematic review method provides complete and accurate reporting, which facilitates assessment of how well reviews have been conducted (Toews, 2017 ).

Unlike a traditional review, a systematic review uses a transparent, replicable, and scientific steps purposely to mitigate the risk of bias by conducting a comprehensive literature search and providing an audit trail of procedures, decisions, and conclusions (Caldwell and Bennett,  2020 ). The systematic review reports a reproducible search strategy that increases the reliability and validity of the study.

By following systematic review guidelines, this study will mitigate bias and increase its validity and reliability. The following steps are adopted to conduct the systematic review:

Step 1: Identifying Keywords

To synthesize the evidence of substance use prevention programs available in the literature amongst refugee youth, a database search began with a simple string of “substance use AND Prevention AND Refugee AND youth” in the library. Then other search terms were obtained using a permutation of the keywords in EBSCOhost Megafile Ultimate (Table  1 ).

Step 2: Search Strategy

In the next step, a comprehensive search for relevant articles was conducted on 12th of October 2021 on three major databases: Scopus, PubMed, and EBSCOhost Megafile databases including Academic Search Ultimate, APA PsycArticles, APA PsycInfo, CINAHL with Full Text, E-Journals, Humanities Source Ultimate, Psychology and Behavioural Sciences Collection, and Sociology Source Ultimate. A total of 485 studies were retrieved following the comprehensive search of the databases (Table  2 ).

Study Selection

All the retrieved studies were exported to Endnote X9, and 199 duplicates were removed. The titles and abstracts of the remaining 286 studies were reviewed and 253 studies were excluded for not focusing on substance use prevention programs. A total of 33 studies were further screened using inclusion and exclusion criteria. As a result of the exercise, 24 studies were excluded and nine were included for quality assessment. The PRISMA workflow diagram below shows the process of identifying and selecting eligible studies for this systematic review (Fig.  1 ). The data visualisation displays identified, included, and excluded papers and their explanations.

PRISMA of workflow

Exclusion and Inclusion Criteria

This systematic literature review on substance use prevention programs amongst refugee youth was conducted after adopting exclusion and inclusion criteria. To assist in the process of selecting relevant studies in this systematic literature review, studies were limited to peer-reviewed articles published in the English language. Unpublished articles were excluded, and no restriction was placed on the date of publication of the studies.

Furthermore, the selection of articles was restricted to the following eligibility criteria:

Inclusion Criteria

Studies that explored substance use and prevention/reduction/treatment/intervention programs amongst refugee youth.

Studies that explored substance use amongst refugee youth included another perspective of substance use prevention programs.

Studies that investigated and reported motivation for substance use refugee youth.

Exclusion Criteria

Studies that addressed substance use but did not include any intervention.

Studies that addressed substance use prevention and never mentioned refugee youth.

Studies that addressed substance use prevention programs amongst refugees in general.

Studies that addressed immigrant youth but did not mention refugees.

Quality Assessment

The quality of studies included in the systematic literature review was evaluated using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement checklist (von Elm et al., 2007 ). This quality assessment tool is chosen for this study because to its usefulness and applicability to all studies (Vandenbroucke et al., 2014 ; von Elm et al., 2007 ). The explanation and elaboration of the different components of the STROBE provide readers with a clear understanding of the study (Vandenbroucke et al., 2014 ).

A total of twenty STROBE items from the checklist were used to assess the quality of the studies. These include 1 A. title, 1B. abstract, 2. background/rationale, 3. objective, 4. design, 5. setting, 6. eligibility of the participants, 7. variables, 8. data source/measurement, 10. study size, 13a. participant number, 14a. descriptive data, 15. outcome data, 16a. main result, 16b. Category of Continuous variable, 19. limitation, 20. interpretation, 21. generalisation, and 22. funding (items 1 A, 1B, 2, 3, 4, 5, 6, 7, 8, 10, 13a, 14a, 15, 16a, 18, 19, 20, 21, 22). Each item was coded as: Y = present, N = not present, P = partially present, N/A = not applicable, and finally, the percentage of the positive judgement’s total calculation (Table  3 ). If an article’s total percentage of positive judgement is less than 50%, then it is deemed poor quality and excluded from the study.

Table  3 : Quality Assessment .

Data Extraction

Systematic reviews conduct data extraction to minimise human error and bias (Tranfield et al., 2003 ). The purpose of the data extraction is to directly link to the formulated review question and the planned assessment of the incorporated studies, providing as a visual representation and historical record of decisions made during the process, and as the data-repository for the analysis (Tranfield et al., 2003 ). Below is the data extraction table developed for this systematic literature review (Table  4 ). Data extractions contain valuable information such as title, author, findings, concepts, journal, study design, setting, population, and emerging themes.

Study Characteristics

Study objectives and designs.

The study designs include four qualitative, one ethnographic, two mixed methods, one random controlled trial, and one two-cluster sample. The studies were published in nine different journals (Table  5 ).

Study Setting and Participants

Nine peer-reviewed articles met the inclusion criteria for this systematic literature review. They were published from 2009 to 2020. Four studies were conducted in the USA, two in Australia, two in the Middle East, and one in Kenya. Participants in these studies are refugees youth from these host countries.

The findings revealed a gap in the literature about substance use prevention programs amongst refugee youth. In the nine articles that met the inclusion criteria for this study, only two substance use prevention programs emerged. The substance use prevention programs identified in the study included Adelante Social and Marketing Campaign (ASMC) and Screening and Brief Intervention (SBI).

ASMC is a community-based intervention program offered by the Advance Centre for the Advancement of Immigrant/Refugee Health in Washington, DC, USA. This is a well-known primary prevention program, which addresses risk factors for substance use and other co-occurrences amongst Latino adolescents aged 12 to 19 years in a suburb of Washington, DC (Andrade et al., 2018 ; Edberg et al., 2015 ). The study employed the 4-year Adelante primary prevention program to address risk factors for substance use and other issues amongst Latino adolescents, aged 12 to 19 years (Andrade et al., 2018 ). In the two studies, ASMC was used to investigate two distinct scenarios. Firstly, it was used to identify post contents and features that resulted in greater user engagement (Andrade et al., 2018 ). Secondly, Edberg et al. ( 2015 ) used ASMC to provide a brief description of the background for community-level health disparities intervention that aims to help close the gap. The intervention is organised in a group of one to five short psychotherapeutic sessions for substance users (Karno et al., 2021 ; Widmann et al., 2017 ). Participants engage in a standardized screening for substance use problems, receive systematic feedback on substance-related risks, and participate in a motivational intervention to reduce substance use (Saitz, 2014 ).

On the other hand, SBI is used by non-psychiatric healthcare providers for substance use prevention. The approach relies on motivational interviewing focusing on empowering patients during the intervention (Karno et al., 2021 ; Widmann et al., 2017 ). SBI was successfully used to assist refugee youth in addressing substance use issues.

Six studies explore the strategies and protective factors for substance use prevention. Giuliani et al. ( 2010 ) and McCann et al. ( 2016 ) identified protective factors that influence the cessation of substance use amongst refugee youth, including strong community support systems, family, and friends. Protective factors such as trustworthiness, confidentiality of help sources, perceived expertise of formal help sources, and increasing young people’s and parents’ substance use literacy play a vital role in reducing the initiation and maintenance of substance use. Research has shown that providing refugee youth woth counselling, ongoing case management coordination, residential detoxification programmes, and individual strategies such as self-imposed physical isolation can mitigate substance use amongst them (Horyniak et al., 2016a ; McCleary et al., 2016 ). Moreover, researchers identified protective factors including academic success, and participation in voluntary activities can assist in reducing substance use (Massad et al., 2016 ).

The findings highlight protective factors that shield refugee youth from substance-use. These protective factors included religion, positive peer pressure, health, relief, and social services (Giuliani et al., 2010 ; Khader et al., 2009 ; McCann et al., 2016 ). More importantly, connecting with substance use treatment is suggested to be one way refugee youth can reduce substance use (McCann et al., 2016 ; McCleary et al., 2016 ).

Participants’ Attitudes toward Substance use Prevention Programs

The studies that attempt to investigate the attitude of refugee youth towards substance use prevention programs have revealed mixed results. First and foremost, refugee youth demonstrated a lack of confidence in the institution that provides substance use prevention programs (Massad et al., 2016 ; McCann et al., 2016 ). For instance, refugee youth in substance use treatment expressed a sense of scepticism towards the institution that provides counselling and rehabilitation (McCann et al., 2016 ; McCleary et al., 2016 ). Other researchers found out that refugee youth’s participation in substance use treatments is not motivated and therefore they are too reluctant to seek treatment (McCann et al., 2016 ; McCleary et al., 2016 ). While other research shows that refugee youth are unaware of any local institutions to support youth with substance use problems (Massad et al., 2016 ). The refugee youth who participated in the Adelante intervention and utilise social media demonstrated a positive propensity towards engaging in more passive forms of social media usage (Andrade et al., 2018 ).

Outcomes of Substance Use Prevention Programs

ASMC showed that prevention topics were significantly associated with post-engagement behaviour, such as substance use (Andrade et al., 2018 ). ASMC also identified the inequalities that promote substance use amongst the refugee youth such as a lack of community attachment, social support and social space, isolation rather than connection, and a racialized identity (Andrade et al., 2018 ; Edberg et al., 2015 ). The study indicated lack of social space leading to refugee youth finding sanctuary in gang activities (Edberg et al., 2015 ). ASMC also indicated that the most engaging topic discussed in social media posts was substance use prevention, which accounted for 8.4% of the posts with the p-value < 0.001 (Andrade et al., 2018 ).

The outcome for SBI was significant. The findings indicate that there was a decline in the amount of time that refugee youth spent using substances as their functional time increased among refugee youth (Widmann et al., 2017 ). As a result, SBIs appear to reduce substance use to some extent.

Overview of the Findings

The study aimed to explore different substance use prevention programs, summarise refugee youth’s attitudes towards these programs and outline the outcomes of the prevention programs. This systematic literature review appeared to be the first of its kind to systematically synthesis substance use prevention programs amongst refugee youth. The findings from this study supported the hypothesis that research on substance use prevention programs amongst refugee youth is scarce. Only two substance use prevention programs were identified in the study: SBI and ASMC. Although ASMC was included in only one study on substance use prevention programs, its main objectives were to identify the activities in which refugee youth participate and to outline potential areas for intervention. ASMC did not employ strategies to reduce substance use. Moreover, most studies included in this context outlined strategies and protective factors that assist in reducing substance use and related consequences amongst refugee youth. If refugee youth adhere to protective factors such as family attachment, religion, and commitment to social norms, then there is a likelihood that they can avoid the initiation and maintenance factors of substance use. Another important strategy that emerges from this study is the need to increase refugee youth and parents’ substance use literacy. Increasing literacy can help refugee youth to understand the risk substance use can have on their health, social interactions, and economic wellbeing.

Previous studies asserted that the efficacy of substance use prevention program depends on the participants’ attitude towards intervention and its outcomes (Espada et al., 2015 ). However, what is alarming is refugee youth have a negative attitude about institutions providing substance reduction services. Although the ASMC and SBI demonstrated positive outcome, such an approach can be associated with high dropout rates and subsequently, poor outcomes in substance use prevention programs. Individuals who have confidence in professional services are more likely to seek assistance and therefore, reduce substance use.

Implication

The dearth of research on substance use prevention programs programmes may have significant ramifications, considering the substantial body of literature indicating the widespread occurrence of substance use amongst refugee youth. There exists convincing evidence that the refugee youth cohort could be at risk of substance use disorders but are not seeking help. Substance use has a debilitating impact on an individual’s health, social and economic well-being. For refugee youth not seeking assistance to reduce substance use may indicate they are suffering significant consequences on top of their challenges before and after migration.

Previous studies conducted on youth in general has identified many substance use prevention programs in the literature that can mitigate the prevalence of substance use and related consequences (Barrett et al., 1988 ). However, little is known in the literature about the extent and effectiveness of substance use prevention programs including individual and group counselling, alternative programs, and family and community interventions, applicable for refugee youth (Barrett et al., 1988 ; Foss-Kelly et al., 2021 ; Radoi, 2014 ). Researchers only indicated that refugees are aware of some substance use treatment services. There are substantial differences between being aware of a service and actively interacting and engaging with it. Therefore, it is significant for refugee youth to be aware of substance use prevention programs and seek assistance to reduce substance dependence.

Refugee youth’s lack understanding of substance use prevention programs might be compounded by their inability to seek professional help. Scholarly literature suggest that refugee youth do not seek professional help because of barriers including lack of understanding of the new health system, poor mental literacy, language problem, limited transportation and cultural differences (Posselt et al., 2014 ; Shaw et al., 2019 ). Additionally, young refugees, particularly those who are forced to flee their countries due to persecution or violence, frequently encounter substantial trauma and stress without adequate access to mental health services. The pressures encompass a dearth of livelihood opportunities, familial separation, risky journeys, and vulnerability to assault and abuse. Despite managing to escape life-threatening situations in their native countries, these youth individuals often face further prejudice and become targets of in their host countries. They frequently encounter challenges accessing appropriate services, especially when it comes to disparities in mental healthcare services caused by socio-cultural factors. While additional resources and support are necessary, it is crucial to provide culturally sensitive and customised interventions to refugee youth.

Conclusion and Future Research

In conclusion, prevention programs for substance use remain obscure despite the prevalence of substance use amongst refugee youth. The prominent finding of this review is that the majority of the investigations failed to address substance use prevention programs, as their focus was primarily on protective factors and strategies to reduce substance-use. While the study does make an attempt to address substance use prevention programs, it also incorporates other risk behaviours as well. In such investigations, it is difficult to deduce the outcome and attitudes of the participants. Future research is warranted regarding the implementation of substance use prevention programs amongst refugee youth. The findings are an indication of the need to conduct a robust substance use prevention program such as individual and group counselling, alternative programs, and family and community interventions tailored specifically to refugee youth. Furthermore, research should demonstrate the efficacy of each substance use prevention program by exploring participants’ attitudes towards intervention and measuring the outcome of the study. This can fill the gap in the literature with empirical evidence on how refugee youth participate in substance use prevention programs and maximise the benefits by reducing substance use.

It is essential to acknowledge the limitations of this study. The primary constraint lies in the study’s narrow focus on refugee youth, restricting the search to this specific keyword. Consequently, fewer articles satisfied the inclusion criteria. The study may have overlooked relevant articles that employ alternative terms such as ‘immigrant’, ‘migrant’, or ‘cultural linguistic diverse individuals’. Using broader and more inclusive terms can improve the quality of future research by redesigning the search strategy. .

AIHW (2020). Australian Institute Of Health And Welfare Retrieved from  https://www.aihw.gov.au/reports/alcohol/alcohol-tobacco-other-drugs-australia/contents/summary .

AIHW (2021). Australian’s Youth Retrieved from June 2022.  https://www.aihw.gov.au/reports/cws/81/australias-youth .

Alayan, N., Naal, H., Makhoul, M., Avedissian, T., Assaf, G., Talih, F., et al. (2021). Primary care screening, brief intervention, and referral to treatment for adolescent substance use in lebanon: A National Cross-Sectional Study. Substance Abuse: Research & Treatment, 1. https://doi.org/10.1177/1178221821994608

Article   Google Scholar  

American Psychiatric Association (2013). American psychiatric association, arlington, Va: diagnostic and statistical manual of mental disorders (Dsm–5) Retrieved from https://www.psychiatry.org/psychiatrists/practice/dsm .

Andrade, L., Evans, D., Barrett, N., Edberg, C., & Cleary, D. (2018). Strategies to increase latino immigrant youth engagement in health promotion using social media: Mixed-methods study. Jmir Public Health Surveill, 4 (4), E71. https://doi.org/10.2196/Publichealth.9332

Article   PubMed   PubMed Central   Google Scholar  

Andrabi, N., Khoddam, R., & Leventhal, A. M. (2017). Socioeconomic disparities in adolescent substance use: Role of enjoyable alternative substance-free activities. Social Science & Medicine (1982) , 176 , 175–182. https://doi.org/10.1016/J.Socscimed.2016.12.032 .

Barrett, E., Simpson, D., & Lehman, K. (1988). Behavioral changes of adolescents in drug abuse intervention programs. Journal of Clinical Psychology , 44 (3), 461–473.

Article   CAS   PubMed   Google Scholar  

Bauman, A., & Phongsavan, P. (1999). Epidemiology of substance use in adolescence: Prevalence, trends and policy implications. Drug and Alcohol Dependence, 55 (3), 187–207. https://doi.org/10.1016/S0376-8716(99)00016-2

Bermea, M., Lardier, T., Forenza, B., Garcia-Reid, P., & Reid, J. (2019). Communitarianism and youth empowerment: Motivation for participation in A Community‐based substance abuse Prevention Coalition. Journal of Community Psychology , 47 (1), 49–62. https://doi.org/10.1002/Jcop.22098 .

Article   PubMed   Google Scholar  

Caldwell, Y., & Bennett, T. (2020). Easy Guide to conducting a systematic review. Journal of Paediatrics and Child Health , 56 (6), 853–856. https://doi.org/10.1111/Jpc.14853 .

Correia, J., Benson, T. A., & Carey, B. (2005). Decreased substance use following increases in alternative behaviors: A preliminary investigation. Addictive Behaviors, 30 (1), 19–27. https://doi.org/10.1016/J.Addbeh.2004.04.006

Edberg, M., Cleary, S., Simmons, B., Cubilla-Batista, I., Andrade, L., & Gudger, G. (2015). Defining the ‘Community’: Applying ethnographic methods for a latino immigrant health intervention. Human Organization, 74 (1), 27–41. https://doi.org/10.17730/Humo.74.1.6561p4u727582850

Espada, P., Gonzálvez, T., Lloret, D., Guillén-Riquelme, A., & Orgilés, M. (2015). Meta-analysis of the effectiveness of school substance abuse prevention programs in Spain. Meta-Análisis De La Eficacia De Los Programas De Prevención Escolares Del Abuso De Sustancias En España, 27 (1), 5–12. https://doi.org/10.7334/Psicothema2014.106

Fishbein, H., Hyde, C., Eldreth, D., Paschall, J., Hubal, R., & Das, A. (2006). Neurocognitive skills moderate urban male adolescents’ responses to preventive intervention materials. Drug & Alcohol Dependence , 82 (1), 47–60. https://doi.org/10.1016/J.Drugalcdep.2005.08.008 .

Foss-Kelly, L., Generali, M., & Crowley, M. J. (2021). Making choices and reducing risk (Mcarr): School counseling primary prevention of substance use. Professional Counselor, 11 (3), 352–369. https://doi.org/10.15241/Llfk.11.3.352

Gattamorta, A., Varela, A., Mccabe, E., Mena, P., & Santisteban, D. A. (2017). Psychiatric symptoms, parental attachment, and reasons for Use as correlates of heavy substance use among treatment-seeking hispanic adolescents. Substance Use & Misuse, 52 (3), 392–400. https://doi.org/10.1080/10826084.2016.1229338

Gau, J., Stice, E., Rohde, P., & Seeley, J. (2012). Negative life events and substance use moderate cognitive behavioral adolescent depression prevention intervention. Cognitive Behaviour Therapy, 41 (3), 241–250. https://doi.org/10.1080/16506073.2011.649781

Gire, T., & Ibaishwa, R. L. (2019). Patterns of substance Use among internally displaced persons in Borno State, Nigeria. African Journal of Drug and Alcohol Studies, 18 (1), 17–30.

Google Scholar  

Giuliani, W., Mire, O., Ehrlich, C., Stigler, H., & Dubois, K. (2010). Characteristics and prevalence of Tobacco use among Somali Youth in Minnesota. American Journal of Preventive Medicine, 39 (6, Suppl 1), S48–S55. https://doi.org/10.1016/J.Amepre.2010.09.009

Gruenewald, J., Johnson, K., Shamblen, R., Ogilvie, A., & Collins, D. (2009). Reducing adolescent use of harmful legal products: Intermediate effects of a community prevention intervention. Substance Use & Misuse, 44 (14), 2080–2098. https://doi.org/10.3109/10826080902855223

Hater, J., Singh, K., & Simpson, D. (1984). Influence of family and religion on long-term outcomes among opioid addicts. Advances in Alcohol & Substance Abuse, 4 (1), 29–40. https://doi.org/10.1300/J251v04n01_04

Article   CAS   Google Scholar  

Horyniak, D., Melo, S., Farrell, M., Ojeda, D., & Strathdee, A. (2016a). Epidemiology of substance use among forced migrants: A global systematic review. Plos One, 11 (7), E0159134–E0159134. https://doi.org/10.1371/Journal.Pone.0159134

Horyniak, D., Higgs, P., Cogger, S., Dietze, P., & Bofu, T. (2016b). Heavy alcohol consumption among marginalised African Refugee Young people in Melbourne, Australia: Motivations for drinking, experiences of alcohol-related problems and strategies for managing drinking. Ethnicity & Health, 21 (3), 284–299. https://doi.org/10.1080/13557858.2015.1061105

Huang, S., Cordova, D., Estrada, Y., Brincks, M., Asfour, S., & Prado, G. (2014). An application of the complier average causal effect analysis to examine the effects of a family intervention in reducing illicit drug use among high-risk hispanic adolescents. Family Process, 53 (2), 336–347. https://doi.org/10.1111/Famp.12068

Ishaak, F., De Vries, K., & Van Der Wolf, K. (2015). Design of study without drugs–A surinamese school-based drug-prevention program for adolescents. Bmc Public Health, 15 (1), 1046–1057. https://doi.org/10.1186/S12889-015-2374-1

Ismayilova, L., Terlikbayeva, A., & Rozental, Y. (2019). Computerized intervention to prevent drug use among at-risk adolescents in Central Asia: Preliminary family-level findings from a pilot mixed methods trial. International Journal of Drug Policy, 68 , 75–85. https://doi.org/10.1016/J.Drugpo.2019.03.022

Ji, X., Cummings, R., Mertens, C., Wen, H., & Effinger, E. (2021). Substance use, substance use disorders, and treatment in adolescent and young adult cancer survivors—results from a national survey. Cancer (0008543x), 127 (17), 3223–3231. https://doi.org/10.1002/Cncr.33634

Karno, P., Rawson, R., Rogers, B., Spear, S., Grella, C., & Mooney, J. (2021). Effect of screening, brief intervention and referral to treatment for unhealthy alcohol and other drug use in mental health treatment settings: A randomized controlled trial. Addiction (Abingdon England), 116 (1), 159–169. https://doi.org/10.1111/Add.15114

Khader, A., Shaheen, Y., Turki, Y., El Awa, F., Fouad, H., & Warren, W. (2009). Tobacco use among palestine refugee students (Unrwa) aged 13–15. Preventive Medicine, 49 (1), 224–228. https://doi.org/10.1016/J.Ypmed.2009.06.001

Kuntsche, E., Kuntsche, S., Thrul, J., & Gmel, G. (2017). Binge drinking: Health impact, prevalence, correlates and interventions. Psychology & Health, 32 (8), 976–1017. https://doi.org/10.1080/08870446.2017.1325889

Kvillemo, P., Hiltunen, L., Demetry, Y., Carlander, K., Hansson, T., & Gripenberg, J. (2021). How to prevent alcohol and illicit drug use among students in affluent areas: A qualitative study on motivation and attitudes towards prevention. Substance Abuse Treatment Prevention & Policy, 16 (1), 1–13. https://doi.org/10.1186/S13011-021-00420-8

Li, J., Cho, B., Salvatore, E., Edenberg, J., Agrawal, A., & Chorlian, B. (2017). The impact of peer substance use and polygenic risk on trajectories of Heavy Episodic drinking across adolescence and emerging adulthood. Alcoholism Clinical and Experimental Research, 41 (1), 65–75. https://doi.org/10.1111/Acer.13282

Liddle, A., Rowe, L., Gonzalez, A., Henderson, E., Dakof, A., & Greenbaum, E. (2006). ChangiNg provider practices, program environment, and improving outcomes by transporting multidimensional family therapy to an adolescent drug treatment setting. American Journal on Addictions, 15 , 102-S112. https://doi.org/10.1080/10550490601003698

Luitel, P., Jordans, M., Murphy, A., Roberts, B., & Mccambridge, J. (2013). Prevalence and patterns of hazardous and harmful alcohol consumption assessed using the audit among Bhutanese refugees in Nepal. Alcohol and Alcoholism , 48 (3), 349–355. https://doi.org/10.1093/Alcalc/Agt009 .

Massad, S. G., Shaheen, M., Karam, R., Brown, R., Glick, P., & Linnemay, S. (2016). Substance use among palestinian youth in the West Bank, Palestine: A qualitative investigation. Bmc Public Health, 16 (1), 1–9. https://doi.org/10.1186/S12889-016-3472-4

Mccann, V., Mugavin, J., Renzaho, A., & Lubman, I. (2016). Sub-Saharan African Migrant Youths’ help-seeking barriers and facilitators for mental health and substance use problems: A qualitative study. Bmc Psychiatry . https://doi.org/10.1186/S12888-016-0984-5

Mccleary, S., Shannon, J., & Cook, L. (2016). Connecting refugees to substance use treatment: A qualitative study. Social Work in Public Health, 31 (1), 1–8. https://doi.org/10.1080/19371918.2015.1087906

Momeñe, J., Estévez, A., Pérez-García, A. M., Jiménez, J., Dolores Chávez-Vera, M., Olave, L., & Iruarrizaga, I. (2021). Substance abuse and its relationship to emotional dependence, attachment and emotional regulation in adolescents. El Consumo De Sustancias Y Su Relación Con La Dependencia Emocional El Apego Y La Regulación Emocional En Adolescentes, 37 (1), 121–132. https://doi.org/10.6018/Analesps.404671

Murphy, G., Dennhardt, A., Martens, P., Borsari, B., Witkiewitz, K., & Meshesha, Z. (2019). A randomized clinical trial evaluating the efficacy of a brief alcohol intervention supplemented with a substance-free activity session or relaxation training. Journal of Consulting and Clinical Psychology, 87 (7), 657–669. https://doi.org/10.1037/Ccp0000412

Nijdam-Jones, A., Livingston, D., Verdun-Jones, S., & Brink, J. (2015). Using social bonding theory to examine ‘recovery’ in a forensic mental health hospital: A qualitative study. Criminal Behaviour and Mental Health, 25 (3), 157–168. https://doi.org/10.1002/Cbm.1918

Peloso, M., De Freitas, C., Henriques, D., De Souza Magalhães, B., Marques, S., & Lima, M. (2021). Factors associated with drug use among adolescents. Revista Ciência E Natura, 43 , 1–15. https://doi.org/10.5902/2179460x43556

Posselt, M., Galletly, C., De Crespigny, C., & Procter, N. (2014). Mental health and drug and alcohol comorbidity in young people of refugee background: A review of the literature. Mental Health & Substance Use: Dual Diagnosis, 7 (1), 19–30. https://doi.org/10.1080/17523281.2013.772914

Posselt, M., Procter, N., Galletly, C., & De Crespigny, C. (2015). Aetiology of coexisting mental health and alcohol and other drug disorders: Perspectives of refugee youth and service providers. Australian Psychologist, 50 (2), 130–140. https://doi.org/10.1111/Ap.12096

Radoi, M. (2014). Preventing drug abuse - from explanatory theories to intervention models. Scientific Annals of the ‘Al I Cuza’ University Iasi Sociology & Social Work / Analele Stiintifice Ale Universitatii ‘Al I Cuza’ Iasi Sociologie Si Asistenta Sociala , 7 (1), 116–128.

Ramachandran, A., Makhashvili, N., Javakhishvili, J., Karachevskyy, A., Kharchenko, N., Shpiker, M., Ezard, M., Fuhr, N., & Roberts, B. (2019). Alcohol use among conflict-affected persons in Ukraine: Risk factors, coping and access to mental health services. European Journal of Public Health, 29 (6), 1141–1146. https://doi.org/10.1093/Eurpub/Ckz117

Rehm, J., Marmet, S., Anderson, P., Gual, A., Kraus, L., & Nutt, J. (2013). Defining substance use disorders: Do we really need more than heavy use? Alcohol and Alcoholism, 48 (6), 633–640. https://doi.org/10.1093/Alcalc/Agt127

Rethlefsen, L., Kirtley, S., Waffenschmidt, S., Ayala, P., Moher, D., & Page, J. (2021). Prisma-S: An extension to the prisma statement for reporting literature searches in systematic reviews. Systematic Reviews, 10 (1), 39–39. https://doi.org/10.1186/S13643-020-01542-Z

Roberts, B., Felix Ocaka, K., Browne, J., Oyok, T., & Sondorp, E. (2011). Alcohol disorder amongst forcibly displaced persons in Northern Uganda. Addictive Behaviors , 36 (8), 870–873. https://doi.org/10.1016/J.Addbeh.2011.03.006 .

Saitz, R. (2014). Screening and brief intervention for unhealthy drug use: Little or no efficacy. Frontiers in Psychiatry, 5 , 121–121. https://doi.org/10.3389/Fpsyt.2014.00121

Shaw, A., Ward, P., & Pillai, V. (2019). Assessing mental health and service needs among refugees in Malaysia. International Journal of Social Welfare, 28 (1), 44–52. https://doi.org/10.1111/Ijsw.12313

Springer, F., Sale, E., Hermann, J., Sambrano, S., Kasim, R., & Nistler, M. (2004). Characteristics of effective substance abuse prevention programs for high-risk youth. The Journal of Primary Prevention, 25 (2), 171–194. https://doi.org/10.1023/B:Jopp.0000042388.63695.3f

Sussman, S., Skara, S., Rodriguez, Y., & Pokhrel, P. (2006). Non drug use- and drug use-specific spirituality as one-year predictors of drug use among high-risk youth. Substance Use & Misuse, 41 (13), 1801–1816. https://doi.org/10.1080/10826080601006508

Toews, C. (2017). Compliance of systematic reviews in Veterinary journals with preferred reporting items for systematic reviews and meta-analysis (prisma) literature search reporting guidelines. Journal of the Medical Library Association, 105 (3), 233–239. https://doi.org/10.5195/Jmla.2017.246

Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management , 14 (3), 207–222. https://doi.org/10.1111/1467-8551.00375 .

Valente, W., Ritt-Olson, A., Stacy, A., Unger, B., Okamoto, J., & Sussman, S. (2007). Peer acceleration: Effects of a social network tailored substance abuse prevention program among high-risk adolescents. Addiction (Abingdon England), 102 (11), 1804–1815. https://doi.org/10.1111/J.1360-0443.2007.01992.X

Vandenbroucke, P., Von Elm, E., Altman, G., Gøtzsche, C., Mulrow, D., & Pocock, S. J. (2014). Strengthening the reporting of observational studies in epidemiology (Strobe): Explanation and elaboration. International Journal of Surgery (London England), 12 (12), 1500–1524. https://doi.org/10.1016/J.Ijsu.2014.07.014

Von Elm, D., Altman, P., Egger, P., Pocock, P., Gøtzsche, D., & Vandenbroucke, P. (2007). The strengthening the reporting of observational studies in epidemiology (Strobe) statement: Guidelines for reporting observational studies. The Lancet (British Edition), 370 (9596), 1453–1457. https://doi.org/10.1016/S0140-6736(07)61602-X

Wade-Mdivanian, R., Anderson-Butcher, D., Newman, J., Ruderman, E., Smock, J., & Christie, S. (2016). Exploring the long-term impact of a positive youth development-based alcohol, tobacco and other drug prevention program. Journal of Alcohol & Drug Education, 60 (3), 67–90.

Watkins, C. (2016). A social learning approach to prescription drug misuse among college students. Deviant Behavior, 37 (6), 601–614. https://doi.org/10.1080/01639625.2015.1060799

Widmann, M., Apondi, B., Musau, A., Warsame, A. H., Isse, M., Mutiso, V., Veltrup, V., Ndetei, D., & Odenwald, M. (2017). Comorbid psychopathology and everyday functioning in a brief intervention study to reduce Khat use among somalis living in Kenya: Description of baseline multimorbidity, its effects of intervention and its moderation effects on substance use. Social Psychiatry and Psychiatric Epidemiology: The International Journal for Research in Social and Genetic Epidemiology and Mental Health Services, 52 (11), 1425–1434. https://doi.org/10.1007/S00127-017-1368-Y

Download references

Open Access funding enabled and organized by CAUL and its Member Institutions. Open Access funding enabled and organized by CAUL and its Member Institutions.

Author information

Authors and affiliations.

Faculty of Business, Education, Law and Arts, School of Business, University of Southern Queensland, Toowoomba, QLD, 4350, Australia

Elijah Aleer & Afzalur Rashid

Faculty of Business, Education, Law and Arts, School of Business, Centre for Health Research, University of Southern Queensland, Toowoomba, QLD, 4350, Australia

Khorshed Alam

You can also search for this author in PubMed   Google Scholar

Contributions

Elijah Aleer: study concept, developed review protocol and conceptual framework for study classification, data acquisition, extraction, analysis and interpretation of data, initial draft and critical revision of manuscript, and characteristics of studies tables. Khorshed Alam: review supervision, study concept, review protocol and conceptual framework for study classification, data acquisition, extraction, analysis and interpretation of data, draft and critical revision of manuscript. Afzalur Rashid: review supervision, peer reviewed of search strategies, data acquisition, extraction and interpretation, critical revision of protocol and manuscript.

Corresponding author

Correspondence to Afzalur Rashid .

Ethics declarations

Conflict of interest.

The authors report no conflict of interest.

Ethical Approval

Ethical standards were followed with this research and a formal human ethics approval were not required as the data were collected from the secondary available sources.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Aleer, E., Alam, K. & Rashid, A. A Systematic Literature Review of Substance-Use Prevention Programs Amongst Refugee Youth. Community Ment Health J (2024). https://doi.org/10.1007/s10597-024-01267-6

Download citation

Received : 18 September 2023

Accepted : 09 March 2024

Published : 09 April 2024

DOI : https://doi.org/10.1007/s10597-024-01267-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Refugee youth
• Systematic literature review
• Substance use prevention programs
• Find a journal
• Publish with us
• Track your research

• Search Menu
• Volume 2024, Issue 4, April 2024 (In Progress)
• Volume 2024, Issue 3, March 2024
• Bariatric Surgery
• Breast Surgery
• Cardiothoracic Surgery
• Colorectal Surgery
• Colorectal Surgery, Upper GI Surgery
• Gynaecology
• Hepatobiliary Surgery
• Interventional Radiology
• Neurosurgery
• Ophthalmology
• Oral and Maxillofacial Surgery
• Otorhinolaryngology - Head & Neck Surgery
• Paediatric Surgery
• Plastic Surgery
• Transplant Surgery
• Trauma & Orthopaedic Surgery
• Upper GI Surgery
• Vascular Surgery
• Author Guidelines
• Submission Site
• Open Access
• Reasons to Submit
• About Journal of Surgical Case Reports
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Introduction, case series, author contributions, conflict of interest statement, data availability, consent to publish, surgical and radiological perspectives for the spinal accessory nerve passing through a fenestrated internal jugular vein: case series and literature review.

• Article contents
• Figures & tables
• Supplementary Data

Pierre Guarino, Paolo Tesauro, Leone Giordano, Claudio Donadio Caporale, Livio Presutti, Francesco Mattioli, Surgical and radiological perspectives for the spinal accessory nerve passing through a fenestrated internal jugular vein: case series and literature review, Journal of Surgical Case Reports , Volume 2024, Issue 4, April 2024, rjae099, https://doi.org/10.1093/jscr/rjae099

• Permissions Icon Permissions

The preservation of the spinal accessory nerve represents a key goal in head and neck oncologic surgery during selective neck dissection. This study aims to illustrate the anatomical variants of the XI cranial nerve, delving into the relationship between the spinal nerve and the internal jugular vein, as well as the surgical implications. Two cases of patients who underwent oncologic surgery with neck dissection are described. Both cases found the spinal accessory nerve passing through the fenestration of the internal jugular vein. Alongside this case series, an independent literature review was conducted using the Medline and PubMed databases. In the majority of cases (67% – 96%), the spinal accessory nerve traces a lateral course to the internal jugular vein. Less frequently, the XI cranial nerve courses medial to the internal jugular vein. More rarely, as described in this case series, the nerve crosses through the fenestration of the vein (0.48% – 3.3%). 

During neck dissection, the spinal accessory nerve (SAN) is encountered on Level II at the level of the posterior belly of the digastric muscle where it crosses laterally and anterior the internal jugular vein (IJV). A medial course of the nerve is reported in literature with an incidence that varies from work to work. Rarely, the SAN can pass through a fenestration of the IJV or can split around the IJV [ 1 , 2 ]. Nowadays, preservation of the SAN is the standard of care even though iatrogenic injury to the SAN and to the IJV still occurs. In order to reduce the risk of iatrogenic injury a detailed study of the radiological anatomy together with an extensive knowledge of the anatomical variants of the SAN and of the IJV represent the surgeon’s most important tools [ 3–6 ].

A 78-year-old man with a follow-up PET for a Type B mantle cell lymphoma finding of uptake in in the epiglottis and left laterocervical region is reported here. The patient was diagnosed with squamous cell carcinoma (SCC) after fine needle aspiration of an adenopathy at Level II. His workup included an MRI of the neck with contrast enhancement, which revealed laterocervical lymphadenopathy (bilateral level IIa and left Level III with the biggest adenopathy with maximum diameter of 12 × 12 mm) and thickening of the epiglottis extended at the floor of the left vallecula. No mention was made of an IJV abnormality in the report even though it was noticed during the preoperative evaluation by the surgeon ( Fig. 1 ). The fibroendoscopic evaluation of the neck showed an asymmetry of the epiglottis with the presence of vegetating tissue at the level of the left lingual border of the epiglottis, preserved motility of the vocal cords, and normal breathing space. The patient’s clinical stadiation was cT2 N2c M0. The multidisciplinary meeting indicated an endoscopic partial laryngectomy and bilateral laterocervical dissection. The cervical lymph node dissection on the left side discovered a duplication of the IJV which resulted dilated in comparison to the IJV on the right side. The SAN passed medially to the anterior vein and laterally to the posterior vein. Both the IJV and the SAN were preserved during the procedure ( Fig. 2 ).

MRI showing the fenestration of the IJV; left image: sagittal section showing the fenestration (arrowhead) of the left IJV; right image: axial section showing the anterior (dotted arrowhead) and the posterior (arrowhead) part of the fenestrated IJV.

SAN passes through medially to the anterior part (filled triangle) and laterally to the posterior part (star) of the fenestrated IJV (arrowhead); ICA: internal carotid artery; SCM: sternocleidomastoid muscle.

A 72-year-old woman underwent total laryngectomy + thyroid isthmectomy + bilateral (II–IV) laterocervical dissection for a G3 SCC of the right hemilarynx (involving the right true and false vocal cord and extending to the anterior commissure and left vocal cord). The preoperative radiology report did not mention any IJV variations. Nonetheless, a left IJV fenestration was observed in the preoperative study, raising the suspect of a possible SAN variant ( Fig. 3 ). Pathological stadiation of the patient was pT4a (infiltration of thyroid cartilage and prethyroid soft tissue) pN2c, cM0. Intraoperatively, the fenestration of the left IJV, that was noticed during the CT, was confirmed. As for the first patient presented, the SAN nerve passed medially to the anterior vein and laterally to the posterior vein ( Fig. 4 ).

CT scan showing the fenestration of the IJV; left image: sagittal section showing the fenestration (arrowhead) of the left IJV; central image: coronal section showing the anterior (dotted arrowhead) and the posterior (arrowhead) part of the fenestrated IJV; right image: axial section showing the anterior (dotted arrowhead) and the posterior (arrowhead) part of the fenestrated IJV.

SAN passes through medially to the anterior part (filled triangle) and laterally to the posterior part (star) of the fenestrated IJV (arrowhead); CCA: common carotid artery.

An in-depth understanding of the anatomy of the lateral neck is essential to avoid injury to the IJV during SAN dissection. A literature review of the anatomic relationship of the SAN to the IJV is presented in Table 1 .

Literature review of the incidence of the anatomical variants of the SAN.

As shown, Krause, Soo, and Kierner conducted cadaveric studies [ 7–9 ]. Soo and Kierner reported an equal frequency of the lateral (56%) and medial (44%) position of the nerve compared to the IJV, while Krause reported an incidence of 72.5% of the nerve crossing lateral to the IJV. Krause also observed one case of SAN “fenestrating” through the IJV. Saman’s cadaveric study supported Krause’s data, reporting a 79.8% of SAN coursing lateral to the IJV, 19% of SAN medial to the IJV, and 1.2% of SAN “fenestrating” through the IJV [ 10 ].

In vivo studies reported a higher frequencies of the nerve crossing lateral to the IJV. Levy et al. [ 11 ], Hinsley et al. [ 12 ] and Taylor et al. [ 13 ] showed the SAN lateral to the IJV in 99.25%, 96.6%, 95.7% and the SAN medial to the IJV in 0.75%, 2.6% and 2.8%. Taylor and Hinsley also reported <1% of cases of SAN “fenestrating” to the IJV; moreover, Taylor described a new anatomical variant of the SAN in which the nerve divides and travels both medial and lateral to the IJV.

The higher incidence of the nerve crossing lateral to the IJV in in vivo studies compared to cadaveric is explained by Hinsley and Taylor as follows. The IJV of cadavers can collapse, determining a higher incidence of the medial position of the nerve in relation to the IJV. Moreover, in vivo studies which are conducted on oncological patients focus on the relation between the nerve and IJV at the level of the posterior belly of the digastric muscle, while cadaveric studies do not always keep the same reference points.

In 2009, Lee published an in vivo study in which he found a surprising 57.4% incidence of SAN medial to the IJV. However, these results have not been confirmed by other studies [ 14 ].

In his in vivo study, Lee pointed out that the variation of the course of the SAN correlates with a variation in the number of lymph node of the Levels IIa and IIb. Therefore, a lateral course of the SAN increases the Level IIb area, and subsequently, the number of lymph nodes.

Regarding the rarer anatomic variants of the SAN, Table 1 shows an incidence of the SAN “fenestrating” the IJV ranging from 0.8% to 2.2% [ 15–17 ]. Taylor also describes a new anatomical variant of the spinal nerve splitting around the IJV. However, there are no other reports of this variant of the SAN and no iconographic material results available.

So what tools can a surgeon use to uncover a rare anatomical variant of the SAN? In 2012, Hashimoto published a case report where they were able to identify the fenestration of the IJV with a contrast-enhanced CT. However, Ozturk warns us that even though the fenestration of the IJV is strongly associated with the SAN traveling through the fenestration, this is not to be taken for granted as it can also travel medial or lateral to the fenestration [ 18 ].

In both cases presented in this article, the surgeon noticed the fenestration of the IJV during the preoperative imaging which was not indicated in the radiologist report. As the radiologist priority concerns the extension of the tumor and lymph nodal spread, it falls to the surgeon to enquire about potential anatomical variants of the IJV during the scan.

The literature review conducted in this study confirms that the SAN crosses lateral to the IJV in the majority of cases. However, incidental finding of variants of the SAN, as presented in the case reports, can occur intraoperatively, resulting in iatrogenic lesion of the SAN or of the IJV. Therefore, an upfront radiological study together with a deep knowledge of the possible variants of the SAN are the surgeon’s keystones to reduce the cases of damage to the SAN or to the IJV. The authors also hope that this study could be a starting point for further investigations about the relationship between radiological and surgical anatomy.

All authors conceptualized the study. P.T. wrote the manuscript and prepared the original draft. PG reviewed and edited the manuscript. F.M., L.G., C.D.C., and L.P. supervised the study. P.T., P.G., and L.G. confirm the authenticity of all the raw data. All authors have read and approved the final manuscript.

None declared.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

All data generated or analyzed during this study are included in this published article.

The patients have provided written informed consent for the publication of any associated data and accompanying images.

Caliot P , Bousquet V , Midy D , et al. . A contribution to the study of the accessory nerve: surgical implications . Surg Radiol Anat   1989 ; 11 : 11 – 5 . https://doi.org/10.1007/BF02102238 .

Google Scholar

Crile G . Excision of cancer of the head and neck . JAMA   1906 ; 47 : 1780 – 6 .

Dargent M , Papillon J . Motor complications of neck dissection: how to avoid them [in French] . Lyon Chir   1945 ; 40 : 718 .

Nahum AM , Mullally W , Marmor L . A syndrome resulting from radical neck dissection . Arch Otolaryngol   1961 ; 74 : 424 – 8 .

Suarez O . The problem of distant lymphatic metastasis in cancer of the larynx and hypopharynx [in Spanish] . Rev Otorinolaryngol Santiago   1963 ; 23 : 83 – 99 .

Bocca E , Pignataro O , Oldini C , et al. . Functional neck dissection: an evaluation and review of 843 cases . Laryngoscope   1984 ; 94 : 942 – 5 .

Krause HR , Bremerich A , Herrmann M . The innervation of the trapezius muscle in connection with radical neck-dissection. An anatomical study . J Craniomaxillofac Surg   1991 ; 19 : 87 – 9 . https://doi.org/10.1016/s1010-5182(05)80613-4 . Erratum in: J Craniomaxillofac Surg 1991;19:231 .

Kierner AC , Zelenka I , Heller S , et al. . Surgical anatomy of the spinal accessory nerve and the trapezius branches of the cervical plexus . Arch Surg   2000 ; 135 : 1428 – 31 . https://doi.org/10.1001/archsurg.135.12.1428 .

Soo KC , Hamlyn PJ , Pegington J , et al. . Anatomy of the accessory nerve and its cervical contributions in the neck . Head Neck Surg   1986 ; 9 : 111 – 5 . https://doi.org/10.1002/hed.2890090207 .

Saman M , Etebari P , Pakdaman MN , et al. . Anatomic relationship between the spinal accessory nerve and the jugular vein: a cadaveric study . Surg Radiol Anat   2011 ; 33 : 175 – 9 . https://doi.org/10.1007/s00276-010-0737-y .

Lévy D , Vacher C , Lézy JP . Relations of the accessory nerve with the internal jugular vein: surgical implications in cervical lymph node clearances . Surg Radiol Anat   2001 ; 23 : 155 – 7 . https://doi.org/10.1007/s00276-001-0155-2 .

Hinsley ML , Hartig GK . Anatomic relationship between the spinal accessory nerve and internal jugular vein in the upper neck . Otolaryngol Head Neck Surg   2010 ; 143 : 239 – 41 . https://doi.org/10.1016/j.otohns.2010.03.033 .

Taylor CB , Boone JL , Schmalbach CE , et al. . Intraoperative relationship of the spinal accessory nerve to the internal jugular vein: variation from cadaver studies . Am J Otolaryngol   2013 ; 34 : 527 – 9 . https://doi.org/10.1016/j.amjoto.2013.05.008 ..

Lee SH , Lee JK , Jin SM , et al. . Anatomical variations of the spinal accessory nerve and its relevance to level IIb lymph nodes . Otolaryngol Head Neck Surg   2009 ; 141 : 639 – 44 . https://doi.org/10.1016/j.otohns.2009.07.005 .

Hashimoto Y , Otsuki N , Morimoto K , et al. . Four cases of spinal accessory nerve passing through the fenestrated internal jugular vein . Surg Radiol Anat   2012 ; 34 : 373 – 5 . https://doi.org/10.1007/s00276-011-0875-x .

Contrera KJ , Aygun N , Ward BK , et al. . Internal jugular vein duplication and fenestration: case series and literature review . Laryngoscope   2016 ; 126 : 1585 – 8 . https://doi.org/10.1002/lary.25743 .

Prades JM , Timoshenko A , Dumollard JM , et al. . High duplication of the internal jugular vein: clinical incidence in the adult and surgical consequences, a report of three clinical cases . Surg Radiol Anat   2002 ; 24 : 129 – 32 . https://doi.org/10.1007/s00276-002-0020-y .

Ozturk NC , Talas DU . Fenestration of internal jugular vein and relation to spinal accessory nerve: case report and review of the literature . Clin Anat . 2010 ; 23 : 883 – 4 . https://doi.org/10.1002/ca.20897 .

Email alerts

Citing articles via, affiliations.

• Online ISSN 2042-8812
• Copyright © 2024 Oxford University Press and JSCR Publishing Ltd
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• Open access
• Published: 13 April 2024

Enhancing academic performance prediction with temporal graph networks for massive open online courses

• Qionghao Huang 1 , 2 &
• Jili Chen 1 , 2  

Journal of Big Data volume  11 , Article number:  52 ( 2024 ) Cite this article

Metrics details

Educational big data significantly impacts education, and Massive Open Online Courses (MOOCs), a crucial learning approach, have evolved to be more intelligent with these technologies. Deep neural networks have significantly advanced the crucial task within MOOCs, predicting student academic performance. However, most deep learning-based methods usually ignore the temporal information and interaction behaviors during the learning activities, which can effectively enhance the model’s predictive accuracy. To tackle this, we formulate the learning processes of e-learning students as dynamic temporal graphs to encode the temporal information and interaction behaviors during their studying. We propose a novel academic performance prediction model (APP-TGN) based on temporal graph neural networks. Specifically, in APP-TGN, a dynamic graph is constructed from online learning activity logs. A temporal graph network with low-high filters learns potential academic performance variations encoded in dynamic graphs. Furthermore, a global sampling module is developed to mitigate the problem of false correlations in deep learning-based models. Finally, multi-head attention is utilized for predicting academic outcomes. Extensive experiments are conducted on a well-known public dataset. The experimental results indicate that APP-TGN significantly surpasses existing methods and demonstrates excellent potential in automated feedback and personalized learning.

Introduction

With advancements in deep learning technologies, big data, and artificial intelligence (AI) technologies are now widespread in various domains, such as smart grids [ 1 ], telecommunications [ 2 ], and so on. Educational big data also profoundly influences and reshapes academic research and industrial applications in education [ 3 , 4 ]. Massive Open Online Courses (MOOCs) [ 5 ], generating a large amount of diversified learning behavior data, has been a research hotspot in educational big data and AI + Education  [ 6 , 7 ]. Student academic performance prediction, as a fundamental technique in realizing intelligent educational applications [ 8 ], has received more and more attention in MOOCs [ 9 ]. Predicting student performance is a process that involves estimating how students will fare in future evaluations or exams. This process is crucial in identifying students at risk of failing or dropping out, enabling timely intervention and support. Such a process holds significant importance in the context of massive open online courses [ 10 ].

Many research efforts have been devoted to predicting students’ academic performance with machine learning techniques. For instance, traditional machine learning techniques successfully applied to academic performance prediction, e.g., logistic regression, random forest, artificial neural network, support vector machine [ 11 , 12 ]. Deep neural network-based methods also make significant progress, e.g., recurrent neural networks [ 13 , 14 ], convolutional neural networks [ 15 ], attention networks [ 8 , 16 ]. However, most existing academic performance prediction methods exploit learning behavior data with simple feature engineering, e.g., using a statistic figure (number of occurrences) to denote the feature of a specific learning activity. These settings may result in severe value information loss due to inappropriate data structures. To encode the learning behavior data with graph structures [ 17 ], the most expressive data structure, which can retain valuable clues for performance predictions [ 18 ]. Furthermore, many kinds of research show that sequential patterns of learning behaviors or interaction activities can exhibit the academic states of students [ 14 , 19 , 20 ]. Thus, encoding the online learning behavior data in graph structures with a temporal property may better retain the value learning cues for their academic performance predictions. This work will demonstrate that this temporal graph structure is vital for academic performance prediction. Nevertheless, finding a suitable graph structure to encode students’ learning cues and corresponding processing techniques remains challenging in the domain.

To bridge this gap, a novel model, APP-TGN, utilizing temporal graph neural networks, is introduced to predict academic performance for problem-solving. Specifically, within APP-TGN, a dynamic graph is constructed from the online learning activity logs. The generated graph is forwarded to a temporal graph network with low-high filters to learn potential academic performance variations encoded in dynamic graphs. Furthermore, a global sampling module is developed to mitigate the problem of false correlations in deep learning-based models. Finally, the learned representations from global sampling and local processing (with TGN) are passed through a multi-head attention module, predicting academic performances. The proposed approach’s utility is assessed through comprehensive experimentation using the widely recognized public dataset, OULA [ 21 ], derived from a practical educational application. Specifically, the empirical study seeks to address three research questions: ( i ) How does the proposed APP-TGN perform when predicting student academic performance in terms of accuracy, F1-score, and recall? ( ii ) What is the improvement in early predicting at-risk students when using APP-TGN against other state-of-the-art methods? ( iii ) What contribution does each proposed component of APP-TCN make to the final prediction performance in terms of accuracy? The experimental results indicate that the proposed APP-TGN significantly surpasses existing methods and holds great potential for automated feedback and personalized learning in practical educational applications. Ablation studies also highlight the superiority and value of the proposed techniques within APP-TGN.

The main technical contributions of the paper are summarized as follows:

A novel framework for predicting academic performance is introduced, utilizing temporal graph networks and local and global sampling techniques. This framework leverages temporal information and interaction behaviors to achieve high prediction accuracy in the model.

An efficient temporal graph neural network with low-high filters is designed to deal with temporal-evolving dynamic graphs formed by complex learning interaction activities.

To the best of our knowledge, this paper is the first work to formulate the academic performance prediction tasks as the problem of classifying temporal dynamic graphs. Furthermore, a data bias deduction module is also developed in APP-TGN to mitigate the issue of false correlations in deep learning-based models.

Literature review

Academic performance predictions.

As an important research task in intelligent education, academic performance prediction has attracted the attention of many researchers. The initial discussion is dedicated to exploring research works that utilize traditional machine learning techniques.

Methods with traditional machine learning Academic performance prediction with traditional machine learning has been investigated for decades [ 22 ]. Three logistic regression models were developed by Marbouti et al. to pinpoint students who were at risk in the first grade engineering curriculum. These models were applied at three crucial junctures throughout the semester, and the findings underscored the significance of devising a prediction model tailored to a specific curriculum [ 11 ]. Ren et al. formulated a linear multiple regression approach tailored to individual students to forecast their academic performance in the curriculum. This was achieved by monitoring student participation in MOOCs. The approach effectively highlighted critical aspects of the student’s learning behaviors and studied habits [ 23 ]. Chui and his team introduced a model known as the Reduced Training Vector-based Support Vector Machine (RTV-SVM) for identifying students who are marginal or at risk. By minimizing the number of training vectors, this model effectively cuts down the duration of training while maintaining accuracy [ 24 ]. To find at-risk students at an early stage and promote the realization of pedagogical and economic goal outcomes, Coussement et al. proposed a logit leaf model (LLM). They visualized it to balance predictive performance and comprehensibility, effectively improving the prediction of student dropout [ 25 ]. Riestra et al. utilized five algorithms, decision tree, naive Bayes, logical regression, multi-layer perceptron, and support vector machine, to anticipate student performance in the early stages of a course, based on an analysis of LMS log information available at the time of prediction. In addition, they employed a clustering algorithm to examine various patterns of cluster interaction [ 26 ]. Turabieh et al. introduced a method that enhances the Harris Hawkes optimization (HHO) approach. This method addresses the issue of premature convergence by managing population diversity. They also employed the k-nearest neighbor (kNN) method as a strategy for clustering, which allowed them to monitor the performance of HHO in adjusting population diversity [ 27 ]. In their work, Mubarak et al. put forward Sequential Logistic Regression along with Input Output Hidden Markov Model (IOHMM) for scrutinizing student learning behavior. This approach proves effective in pinpointing students who are at risk of discontinuing their studies [ 28 ]. A model based on genetic programming was developed by Jiao et al. for forecasting student academic performance. This model demonstrated robust performance compared to conventional AI methods such as ANN and SVM [ 29 ]. In summary, traditional machine learning algorithms have a limited capacity for feature learning [ 30 ], which can hinder their ability to model students’ complex learning processes accurately.

Methods with deep neural networks Research on applying deep neural networks has become increasingly popular in recent years [ 31 , 32 ]. Yang et al. proposed the 1-channel & 3-channel learning image recognition based on convolutional neural networks for transforming students’ curriculum participation into images for predictive analysis [ 15 ]. Giannakas et al. introduced a Deep Neural Network framework with two hidden layers in software engineering. This framework was designed to predict teams’ performance early and demonstrated superior performance compared to traditional methods [ 33 ]. It was specifically tailored to handle two-category classification tasks. Wang and colleagues presented AS-SAN, Adaptive Sparse Self-Attention Network, which predicts the fine-grained performance of students in online courses [ 8 ]. Karimi et al. constructed a knowledge map using the DOPE, the Deep Online Performance Evaluation method. They employed recurrent neural networks for encoding sequence learning, which aids in predicting student performance in curriculum [ 34 ]. Waheed et al. utilized deep artificial neural networks in virtual learning environments for early intervention with at-risk students. This approach, which extracted features from clickstream data, outperformed baseline models such as logistic regression or support vector machines [ 35 ]. Du et al. introduced a comprehensive model that leverages Latent Variation Auto Encoder (LVAE) and Deep Neural Network (DNN) to address imbalances in education datasets. This approach enhances the model’s capacity for early identification of students at risk [ 36 ]. Leveraging the growing popularity of graph neural networks [ 31 ], a novel pipeline, MTGNN [ 18 ], has been developed for predicting student performance. This innovative approach utilizes multi-topology graph neural networks, capitalizing on graph structures to mirror student relationships. Sun et al. [ 37 ] propose an adversarial reinforcement learning method for time-relevant scoring systems. They aim to optimize student scores within a limited time while minimizing detection risk. The attacking problem is formulated as a Markov decision process, and a deep Q-network is used for policy learning. Li et al. introduced a unique method, MVHGNN, for predicting students’ academic performance [ 38 ]. This approach utilizes hypergraphs, meta-paths, and a CAT module to establish high-order relations between students and determine the weight of various behaviors. Despite their effectiveness, these models do not incorporate temporal learning process information in simulating learning performance, indicating potential areas for enhancement.

Graph neural networks in educational applications

Graph Neural Networks (GNNs) have garnered significant interest recently due to their exceptional ability to extract information from non-Euclidean spaces [ 39 ]. As a versatile tool compatible with various learning paradigms, such as graph prompt learning [ 40 , 41 ], GNNs have been widely applied in a range of domains, including natural language processing, recommendation systems, and materials science [ 42 , 43 , 44 ]. In line with the advancements in intelligent education, GNNs have also made their mark in the educational sector.

Cognitive diagnosis For instance, cognitive diagnosis, a fundamental aspect of intelligent education, assesses a student’s grasp of specific knowledge areas [ 45 ]. Gao and colleagues introduced a unique framework for Cognitive Diagnosis driven by Relation maps (RCD), based on the interplay among students, exercises, and concepts. This framework successfully integrates both structural and interactive relationships [ 46 ]. Zhang et al. introduced a graph-based approach to knowledge tracing for cognitive diagnosis, known as GKT-CD [ 47 ]. They utilized Gated-GNN within GKT-CD to monitor students’ knowledge records and dynamically ascertain their knowledge mastery abilities. Mao et al. proposed an approach for cognitive diagnosis that is aware of learning behavior (LCD). This method employs GCN to distill features from exercises and videos, thereby enhancing the depiction of students’ knowledge proficiency [ 48 ]. The graph-based Cognitive Diagnosis model (GCDM), proposed by Su et al. facilitates the extraction of interactions between students, skills, and questions from heterogeneous cognitive graphs [ 49 ]. It also uncovers potential higher-order relations between these entities. The ICD, a cognitive diagnostic model proposed by Qi et al. uses three layers of neural networks to model the influence of exercises on concepts, the interaction between concepts, and the influence of concepts on exercises, aiming to address the interaction among knowledge concepts and the quantitative relation between exercises and concepts [ 50 ]. These models have shown the comparable capacity of graph neural networks in modeling the complex learning interaction among students.

Knowledge tracing Knowledge tracing is another important task in intelligent education, which aims to judge students’ knowledge states by tracing their historical learning [ 51 , 52 ]. In the work of Nakagawa et al., a Graph Neural Network was utilized for the first time to transform knowledge structures and apply graph networks for interactive feature extraction, leading to the creation of a unique approach to knowledge tracing known as GKT [ 53 ]. In the study by Yang et al., a unique approach was introduced, known as Graph-based Interaction Knowledge Tracing (GIKT). This approach leveraged a graph convolution network, allowing it to discern the correlation between questions and skills [ 54 ]. Tong et al. introduced a hierarchical graph knowledge tracing approach, HGKT, was introduced. This approach involved the construction of a hierarchical exercise graph, effectively capturing the dependencies in exercise learning [ 55 ]. Song et al. introduced a Joint graph convolutional network-based deep Knowledge Tracing (JKT) system that connects exercises across different concepts, grasps high-level semantic details, and enhances the model’s interpretability [ 56 ]. Wu et al. introduced a session graph-based knowledge tracing (SGKT) that captures dynamic graphs through student interactions during a session and mimics the student response process. Additionally, they utilized a gated graph neural network to discern the knowledge states of students [ 57 ]. A Bi-Graph Contrastive Learning-based Knowledge Tracing (Bi-CLKT) model was proposed to obtain better concept representation through contrastive learning [ 58 ]. Some models with self-supervised methods and graph neural networks are also investigated [ 59 , 60 ]. These studies highlight the importance of simulating complex interactions during learning to improve model prediction performance.

Other educational applications Graph neural networks are also widely used in other intelligent education fields [ 61 ]. Ying et al. introduced an efficient Graph Convolutional Network that produces node embeddings using random walks and graph convolution techniques[ 62 ], and this approach has demonstrated outstanding performance in large-scale network recommendation systems. To counter cold start plus data sparsity issues in recommender systems based on collaborative filtering, Wang et al. introduced a Knowledge Graph Convolutional Network [ 63 ]. This network adeptly identifies item correlations by exploring attributes linked in knowledge graphs. In addressing costliness plus rigidity in conventional Automatic Short Answer Grading (ASAG) tasks, Tan et al. employed a two-layer graph convolutional network, transforming a heterogeneous graph representing student responses, effectively resolving these issues [ 64 ]. Agarwal et al. proposed a Multi-Relational Graph Transformer (MitiGaTe) to mine the structural context of the sentence and achieved remarkable performance on the ASAG task [ 65 ]. Li et al. used interactive information to model the relationship between students and questions. They proposed a GNN model named R2GCN, which can be applied to heterogeneous networks to predict students’ performance in interactive online question banks [ 66 ]. Li et al. leveraged interactive data for mapping relationships between students and questions, proposing an R2GCN GNN variant. This variant, applicable on heterogeneous networks, forecasts student performance for interactive online question banks [ 66 ]. A GNN model named R2GCN was proposed to model the relationship between students and questions using interactive information, this model can be applied to heterogeneous networks to predict student performance in interactive online question banks  [ 67 ]. Asadi et al. suggest using graph neural networks to model irregular multivariate time series, which can achieve accuracy comparable or superior to hand-crafted features when applied to raw time series click streams [ 20 ]. These models demonstrate the promising performance of graph neural networks in these applications.

Methodology

An academic performance prediction model (APP-TGN) based on a revised low-high filtering temporal graph network is proposed in this section. The proposed APP-TGN considers temporal information and interaction behaviors to enhance the performance of model predictions. Furthermore, a data bias deduction module with global sampling techniques is developed to mitigate the problem of false correlations in deep learning-based models. The section introduces the details of the proposed APP-TGN. Firstly, a brief introduction of the framework of APP-TGN is presented, followed by an explanation of the different components of APP-TGN.

The framework of APP-TGN

Figure  1 illustrates the architecture of our solution with APP-TGN. It mainly consists of five main components: Data Collection & Pre-processing, Dynamic Graph Construction, Global Sampling Module, Low-High Filtering Temporal Graph Networks(LHFTGN), Academic Performance Representation & Prediction .

The framework of the proposed APP-TGN for academic performance prediction

Procedures of APP-TGN Data Collection & Pre-processing includes attribution selection, data cleaning, and data transformation. With the pre-processed data from online learning systems, a dynamic graph construction method is presented to provide temporal graphs as the input for LHFTGN in Dynamic Graph Construction . After that, a revised temporal graph neural network with low-high filtering operators is applied to the generated dynamic graphs, from which a local representation of the academic performance is learned for the candid student. Meanwhile, a global representation of the group cognition is also obtained from Global Sampling Module . The local and global representations are concatenated and forwarded to a multi-head attention module to learn an unbiased academic performance representation. With an MLP-based classifier, the academic performances are predicted from the learned representations of these candidate students.

Data cleaning and pre-processing

To perform a training or prediction task for APP-TGN, we need to prepare well-format data from the interaction logs of learning management systems (LMS) to fulfill the requirement of APP-TGN through data cleaning and pre-processing. Usually, the data collection and pre-processing have several essential steps to obtain the desirable formatted data, such as attribute selection, data cleaning, data transformation, etc. Attribute selection refers to choosing a suitable subset of data to achieve better performance on a specific task, as there are many attribute features from the logs of LMS, not all of them can contribute to the model’s performance. Data cleaning is to fix or remove incomplete or unreasonable data to produce a qualified dataset for model training or testing. More importantly, the data format or type may not fulfill the requirement of the model inputs. Some data transformation techniques are often employed to get the exact data types or structures for specific tasks. From Fig.  1 , we can see that the input for APP-TGN can be divided into parts: Ones are used for generating dynamic graphs, and the others are forwarded to the global sampling module.

Data preparation for dynamic graph construction This paper mainly uses temporal dynamic graphs to encode the temporal information and learning behaviors to facilitate academic performance prediction. Thus, we need to prepare the candidate data to generate dynamic graphs. To generate a graph from the raw log data, the key is to determine the types of nodes and edges. As the target graph has a temporal property, we choose online activities as the nodes \(V=\{v_1, v_2,..., v_{N_v}\}\) , where \(v_i\) denotes the i th type of learning activities. The type of edges \(E=\{e_1, e_2,...\}\) are usually the possible interactions between these nodes. We use the notation ac ( i , 1) to denote the required data to generate a dynamic graph. ac ( i , 1) represents a data unit from the sequence of learning activity logs of learner \(l_i\) . A sequence of learning activities for the learner \(l_i\) can be formulated by Eq. ( 1 ).

where \(L=\{l_1, l_2,..., l_m\}\) , Fg ( L ) represents a collection activity log data for a collection of learners L (with M learners), \(N_{aci}\) denotes the length of an activity log for the learner \(l_i\) . The following subsection will detail how these interaction activity logs are converted into dynamic graphs.

Data preparation for global sampling module A global sampling technique is applied in APP-TGN to mitigate the problem of false correlations in deep learning-based models. To achieve this goal, we must select the proper attributes to participate in the global sampling process. We use the notation at ( i , 1) to denote the i th chosen attribute (e.g., Gender, Region, Disability, Highest_education, etc.) from learning management systems. at ( i , 1) can be real-valued scalar or integer numbers obtained by a one-hot or multi-hot encoding method. Thus a record for the learner \(l_i\) can be formulated as follows:

where Fa ( L ) represents selected attribute feature records for a collection of learners L (with M learners), \(N_{at}\) denotes that we choose \(N_{at}\) attribute features for the global sampling. Specifically, Fa ( L ) is generated only from the training dataset, not all the raw data from LMS, through which the problem of predicting the current states with possible future information can be avoided.

Dynamic graph construction

The subsection details how to use the data from Eq. ( 2 ) to construct the dynamic temporal graphs as the input for low-high filtering temporal graph networks in APP-TGN. Temporal graphs are a kind of dynamic graphs that are temporally changing with node or edge events. In our setting, as mentioned in Eq. ( 1 ) and ( 2 ), we use a sequence of online learning activities to generate a temporal graph \({\mathcal {G}}\) , the temporal graph can be formulated as follows:

where \(x(t_i)\) denotes a node-wise or interaction event in the sequence of online learning activities \(\Upsilon\) . A node-wise event \(v_i(t)\) is an online learning activity from a collection of candidate online learning activities V . An interaction event is a directed temporal edge \(e_{i,j}(t)\) between node \(v_i\) (source) and node \(v_j\) (target), usually denoting the transition from the learning activity \(v_i\) to the learning activity \(v_j\) . \({\mathcal {N}}_i(T) = \{j: (i,j) \in \Omega (T)\}\) refers to the neighborhood of node \(v_i(t)\) in time interval T .

To be specific, the number of the node types in a temporal graph \({\mathcal {G}}(T)\) is determined by the types of online learning activities, i.e., \(N_v\) , which means that we can see a temporal graph \({\mathcal {G}}\) as a static graph with \(N_v\) nodes at a specific duration, denoted as \({\mathcal {G}}(t) = ({\mathcal {V}}[0,t], {\mathcal {E}}[0,t])\) . Therefore, we can apply spectral-based or spatial-based techniques to obtain the temporal embedding \({\textbf{v}}_i(t)\) of \(v_i(t)\) in temporal graph convolutional operators. The features of node \(v_i\) are denoted as a tuple \((v_{i,1},..., v_{i,j},...)\) , where \(v_{i,j}\) denotes the j th feature of \(v_i\) , e.g., the type of learning activity, or the duration of the learning activity, and so on. The features of temporal edge \(e_{i,j}(t)\) are denoted as a tuple \((e_{i,j,1},..., e_{i,j, k},...)\) , where \(e_{i,j, k}\) denotes the k th feature of \(e_{i,j}(t)\) , e.g. the timestamp of transition. Furthermore, we can define the node or edge features with different time intervals for efficient computation with dynamic graphs. Together with temporal graphs and their node or edge features, an effective temporal graph network is proposed to obtain the representation of a sequence of online learning activities in the following subsection.

Low-high filtering temporal graph networks

From Fig.  1 , we can see that there are two crucial temporary representations of academic performance to reach the final representation, one is generated from temporal graph networks (locally), which is detailed in this subsection, the other is from the global sampling module (globally), detailed in the following subsection.

Following the conventions, we also adapt an encoder-decoder architecture to realize the temporal graph networks for a local representation of online learning activities. There may exist an over-smooth problem [ 67 ] in temporal graph learning after several propagation operations with different online learning transitions. Thus, we propose an adaptive low-high filtering temporal graph neural network for problem-solving.

Propagation function From the process of dynamic graph construction, we know that dynamic graphs are temporal event-driven in online learning activities. Therefore, the transition between online learning activities is simulated as propagation functions in TGN and can be expressed as:

where \(\varvec{v}^s_i(t^-)\) denotes the memory representation of node \(v_i\) before time t , \(v_i(t)\) is the raw feature, as a source node in the transition between online learning activities, \(\varvec{v}_j^d(t^-)\) for the destination one, \(\sigma\) is a learnable gate function. If the transition of activities is self-loop, the propagation is expressed as:

where pgf is the similar learnable propagation function as Eqn. ( 9 ) and ( 10 ).

Low-high filtering aggregator We will perform information aggregation several times after information propagation as Eqn ( 9 ), ( 10 ) and ( 11 ). Inspired by the work [ 68 ], we propose an adaptive low-high filtering aggregator for temporal graph networks for online learning interaction activities, which can be formulated as follows:

where \({\mathcal {N}}\) denotes the neighboring operator, \(\alpha _{i,j}^L\) and \(\alpha _{i,j}^H\) are coefficient to feature representation node \(v_i\) with the relation \(\alpha _{i,j}^L + \alpha _{i,j}^H=1\) , \({\mathcal {F}}^L_l\) and \({\mathcal {F}}^H_l\) are low-high filters similar in [ 68 ], \({\mathcal {F}}^L_r\) and \({\mathcal {F}}^H_r\) are operators of element-wise attention mechanisms between \(\varvec{p}_i\) and \(\varvec{p}_j\) .

Memory updater and local representation As previously mentioned, one part of the final representation of student academic performances is generated locally from a temporal graph network. Thus, we first need to obtain the node-wise features of the online activities, which can be formulated as follows:

where upd can be implemented by a learnable neural network, e.g., GRU or LSTM, and \(\varvec{s}_i(t)\) is the temporal state of node \(v_i\) at time step t . The local representation of student academic performance can be learned with Eqn. ( 13 ). It can be defined as:

where CPooling denotes a column-wise average or max pooling technique to obtain the local representation of student academic performance, i.e., \(\hat{\varvec{z}}^L(T)\) .

Global sampling module

With the collection of interaction features \(\textit{FI}(S)\) , we can apply a K-means clustering algorithm to construct the target global interaction feature dictionary.

Global sampling Specifically, some datasets’ whole interaction features may be too large to perform a clustering algorithm. We may choose a subset of them for constructing the dictionary, and we will note this in the experimental settings. The process to obtain the global interaction feature dictionary Gdict for \(\textit{FI}(S)\) can be formulated as follows:

where Gdict ( FI (S)) is a matrix with the size of \(N\times d_{k}\) , and \(d_{k}\) is the dimension of interaction features. The optimization object to get N cluster-shaped dictionary is formulated by

where \(f^{(j)}\) denotes \(in(i,j)\in \textit{In}(s_i)\) , \(\mu ^{(n)}\) denotes the n th candidate vector of the global interaction feature dictionary, \(||*||^\delta\) represents a distance function. A cosine similarity or Euclid distance function is often employed in the algorithm. This setting ensures that global and local sampling estimates are based on the same distribution.

Linear transformation layer The feature vectors \(\varvec{z}^G\) from Global Sampling may not be in a well-aligned space to the features from TGN. Thus, we introduce a simple linear transformation layer to obtain a feature representation from a global perspective. The process can be formulated as follows:

where \(D_k\) , head are parameters for the attention mechanism, \(\varvec{L}_i\) is a feature vector for student \(l_i\) , \(\tilde{\otimes }\) is multiplication with broadcasting property.

Academic performance representation and prediction

As Fig.  1 shown, the final representation of academic performance is generated from a local branch of TGN and a global branch of the global sampling module. We apply a simplified multi-head attention mechanism to fuse these local–global features to obtain the academic performance representation. It can be defined as:

where \(\varvec{z}\) is the output of CPooling as in Eqn. ( 14 ). With the final representation \(\varvec{z}\) , an MLP-based classifier is applied to \(\varvec{z}\) to obtain the academic performance prediction of online candidate learners, i.e., \(y= MLP(\varvec{V})\) , y is the predicted result on a given representation \(\varvec{V}\) . Following the convention of classification tasks with neural networks, a cross-entropy loss is utilized to train our APP-TGN model.

Research questions

A case study on the widely recognized OULA dataset [ 21 ] validates the superior performance of APP-TGN in forecasting student academic outcomes. The study aims to answer the following research questions:

Question One ( Q1 ): How does the proposed APP-TGN perform when predicting student academic performance in terms of classification accuracy, F1-score, and recall?

Question Two ( Q2 ): What is the improvement in early prediction of at-risk students when using APP-TGN against other state-of-the-art methods?

Question Three ( Q3 ): What contribution does each proposed component of APP-TCN make to the final prediction performance in terms of classification accuracy?

Dataset and baselines

Dataset A subset of the Open University Learning Analytics dataset (OULA) [ 21 ], specifically code-Module FFF (2013B, 2013J), is chosen for evaluation. The refined data encompasses academic records of 3897 students, encapsulating student details, online learning interaction logs, and academic performance. Figure  2 visually represents the spread of students’ grades. For the sake of simplicity in our study, students were categorized into three groups: Pass (encompassing Pass and Distinction ), Withdrawn , and Fail , as depicted in (b). Besides the basic information ( e.g. gender, region, highest_education) of students, Table  1 summarizes online learning activities to construct dynamic graphs.

Statistics of code-Module FFF in OULA

Baselines The case study employs a variety of machine learning models as baselines to evaluate our proposed APP-TGN. They are - optimized multiple layer perception (OMLP) [ 69 ], ProbSAP [ 70 ], CNN-LSTM [ 71 ], graph neural networks MTGNN [ 18 ] and a modified multi-view graph transformer from [ 31 ] (noted as AP-GT), hybrid recurrent networks (HRNs) [ 72 ] and a variant of our model, denoted APP-TGN1, where APP-TGN substitutes the TGN module for TGN as per  [ 73 ]. This variant serves the role of baseline models, and we contrast it against our newly introduced APP-TGN. Both the reference models and our APP-TGN are built using PyTorch and Python.

Experimental settings

Training and testing setup We partition the dataset, allocating 80% of the samples for training purposes and reserving the remaining 20% for testing. The training set undergoes further partitioning. Here, 90% of the samples form the training set, while the remaining portion aids in the process of identifying optimal hyper-parameters and model configurations. As for the sequential models like GRU, APP-TGN1, and APP-TGN, we will tune the hyper-parameter of the window size to achieve their best performance. To be specific, as we detail in Sect. " Data Cleaning and Pre-processing " and " Dynamic graph construction ", the dynamic graph construction involves feature selection for the process, we choose the learning materials (denoted as id_site in the dataset) as the nodes. Not all the learning materials or learning activities are employed in the graph construction, the ones used in the process are summarized in Table  1 . We cannot build a directed edge between nodes because each learning activity has no fine-grained timestamps. We suppose the materials or nodes used within a day have a non-directional edge between them. The raw features for a node are a tuple ( site_id , sum_click , date ).

Hyperparameter tuning and optimization In the APP-TGN framework, a thorough process of hyperparameter tuning and optimization was carried out. Different propagation and gate functions were experimented with for the low-high filtering temporal graph network module. The challenge lay in striking a balance between complexity and performance. The Identity function for propagation and a three-layer MLP for the gate function yielded the best results. Various configurations were tested for the low-high filter aggregator for the low and high filters. The primary challenge was to ensure the filters effectively captured the relationships among the neighboring vectors. The best performance was achieved when the low filter was set as the addition of neighboring vectors and the high filter as the subtraction of neighboring vectors. The linear transformation layer and the FNN function in the global sampling module were optimized. A three-layer MLP for the FNN function, with three heads and a \(D_k\) of 100, yielded the best results. For the academic performance representation and prediction module, a three-layer MLP was also used for the FNN function. The challenge was to ensure that the output vector had the right dimensionality. An output vector with a dimensionality of 100 proved to be the best. ReLU activation functions were used throughout the entire process, and the APP-TGN was initialized with random parameters following a normal distribution with a standard deviation of 0.1. The main challenge was to prevent overfitting while achieving high performance. This setup provided a good balance between model complexity and performance. Overall, hyperparameter tuning and optimization was a complex task requiring careful experimentation and considering trade-offs between different factors. However, the effort was worthwhile as it significantly improved the performance of the APP-TGN framework.

Evaluation metrics The task of predicting student performance is approached as a binary classification problem. The metrics listed below serve as the basis for comparing performance:

Classification Accuracy(ACC):

where TP, FP, FN, and TN denote the count of True Positive, False Positive, False Negative, and True Negative instances in the confusion matrix.

Recall(REL):

where REL is the proportion that the model is accurately classifying the true positives;

F1-score(F1):

where F1 is the harmonic mean of REL (REL = TP / (TP + FN)) and PRE (Precision, defined as the proportion of true positives among predicted positives).

Results and discussions

This subsection details the empirical study results of APP-TGN and other baselines from two perspectives. The first experimental study is to answer the research question one, i.e., How does the proposed APP-TGN perform when predicting student academic performance in terms of classification accuracy, F1-score, and recall? The task in the experiments for the evaluated models is to exploit students’ learning logs of the whole semester to predict their academic performances in the course, e.g., Pass/Fail , or Pass/Withdrawn . The second experimental study aims to answer the second research question, i.e., What is the improvement in early prediction of at-risk students when using APP-TGN against other state-of-the-art methods? The merits of APP-TGN in comparison to other baselines for the early identification of students at risk of not excelling in the initial weeks of the term are examined.

Academic performance prediction with whole online learning logs ( Q1 )

The experiment involves two distinct tasks: identifying students who might fail and those who might withdraw. To identify students who might fail, students are classified as either Pass or Fail . Similarly, to identify students who might withdraw, students are classified as either Pass or Withdrawn .

ACC(%) of APP-TGN against other baselines for predicting at-risk students

Table  2 reports the experimental results of the tasks, and we use bold font to denote the best performance. We find several observations in the following. Firstly, superior performance of APP-TGN : Our APP-TGN model outperforms the baseline models in both sub-tasks, achieving an accuracy of 83.22% in the Pass/Fail task and 77.06% in the Pass/Withdrawn task. Secondly, advantage of graph-Based models : Graph-based models (MTGNN, AP-GT, APP-TGN1, and APP-TGN) consistently surpass non-graph-based models (ProbSAP, CNN-LSTM, OMLP, HRNs) in all metrics, demonstrating their effectiveness in predicting academic performance. Thirdly, comparison of AP-GT and MTGNN : AP-GT and MTGNN, utilizing multiple graphs, show similar prediction performance. However, AP-GT performs slightly better due to its deep feature transformation after GNN representation, a technique also used in our model. Fourthly, benefit of temporal graph structure : Models incorporating the TGN module (APP-TGN and APP-TGN1) outperform static graph neural networks (AP-GT, MTGNN), indicating that a temporal graph structure can more effectively encode learning behavior data for academic performance prediction. In particular, effectiveness of low-high filtering mechanism : Our APP-TGN model, which includes a low-high filtering mechanism, surpasses the APP-TGN with a standard TGN module in three metrics, demonstrating the practical effectiveness of this mechanism. Our APP-TGN introduces a suitable graph structure with temporal property to encode the learning behavior data, which can capture academic states in their complex learning processes, so its predictive performance improves. Furthermore, consistent performance across various training sizes : As depicted in Fig.  3 , our APP-TGN model maintains superior performance across various sizes of training sets, demonstrating its robust ability to discern students’ academic states from learning behavior data. In summary, our APP-TGN model introduces a suitable graph structure with temporal property to encode the learning behavior data, which can capture academic states in their complex learning processes, thereby improving its predictive performance. Further experimental studies will scrutinize the effectiveness of the components of our APP-TGN model.

Early prediction for at-risk students with partial online learning logs( Q2 )

The task of this experiment is to answer the second research question, i.e., What is the improvement in early prediction of at-risk students when using APP-TGN against other state-of-the-art methods? Early prediction of students’ performance is an important application in online learning management systems, as we can identify students at risk of failing or dropping out early. Some active invention policies or actions can be applied promptly, giving them enough time to improve their abilities and understanding. We have split the task into two sub-tasks: predicting early on whether students are at risk of failure, categorized as Pass or Fail , and identifying students who may drop out prematurely, categorized as Pass or Withdrawn . Following a similar experimental setting except for the duration (weeks 5, 10, 15, and 20) of learning logs for training and testing.

APP-TGN against MTGNN, APP-TGN1 for early predicting at-risk students in terms of ACC(%)

The comparison between the baseline models and APP-TGN in predicting at-risk students early is presented in Table  3 . It is evident that APP-TGN consistently surpasses the other baseline models in accuracy across all learning periods. Among the baseline models, graph-based models, including AP-GT and MTGNN, exhibit competitive performance compared to non-graph-based models. This suggests that the graph-based approach, which captures complex interactions among learning activities, is beneficial for this prediction task. Interestingly, APP-TGN and its variant, APP-TGN1, outperform AP-GT and MTGNN and perform better in different periods. This indicates that the techniques proposed in APP-TGN, such as temporal graph networks, are effective for early prediction tasks. Moreover, it is worth noting that the performance of all models improves over time, as more academic information becomes available. However, APP-TGN shows the most significant improvement, further highlighting its effectiveness in utilizing temporal information for prediction. Specifically, Fig.  4 a illustrates how APP-TGN achieves an accuracy rate of 81.65% in predicting students who might fail, and Fig.  4 b shows an accuracy rate of 71.13% in predicting students who might withdraw. These figures highlight the potential for early identification of students who are at risk. Moreover, Fig.  4 illustrates that APP-TGN surpasses other compared methods in early prediction, showcasing its high capacity for early intervention. This is important for addressing student issues promptly and encouraging their learning journey.

Effectiveness of APP-TGN ( Q3 )

This part aims to answer the second research question, i.e., What contribution does each proposed component of APP-TCN make to the final prediction performance regarding classification accuracy? As our APP-TGN consists of several significant components and hyper-parameters, we investigate their contribution to the performance of model predictions with ablation study and parameter sensitivities.

Effectiveness of different components of APP-TGN To evaluate the impact of different components of APP-TGN on the prediction performance, we introduce some notations to denote different ablation settings of APP-TGN: APP-GS denotes the APP-TGN without global sampling module, and takes \(\varvec{L_i}\) as \(\varvec{z}^G\) directly; APP-LTL denotes the global sampling module without a linear transformation layer; APP-GRU denotes the APP-TGN with a GRU network [ 13 ] as TGN module; APP-TGN1 denotes the APP-TGN with a normal temporal graph network [ 73 ] as the TGN module. Table  4 shows the accuracy of different components of APP-TGN, and the numbers in the parentheses are deviations from the best prediction performance. We can make the following observations from the table. First, we can see that all main components of APP-TGN are important for the prediction performance for both Pass/Fail and Pass/Withdrawn classification, indicating that the proposed techniques can effectively capture the temporal and relational features of online learning behavior data. It shows that the APP-TGN model can provide a comprehensive and dynamic representation of students’ academic performance, which can help educators and students monitor and improve their learning outcomes. Second, we can see that the GS module can help reduce data bias due to the training dataset. Without the GS module, there is a 1.07% and 1.32% decrease in Pass/Fail and Pass/Withdrawn , respectively. This suggests that the GS module can enhance the APP-TGN model’s generalization ability, making it more robust to different learning scenarios and student groups. Third, the APP-GRU model does not include a TGN module and, therefore, ignores interaction information between learning behavior data. This can result in a significant decrease in prediction performance. APP-GRU has the lowest prediction performance for both Pass/Fail and Pass/Withdrawn sub-tasks, at 81.11% and 74.21%, respectively. That is, the interaction information between learning behavior data is crucial for understanding students’ academic performance, and the TGN module can effectively model such information. Fourth, APP-TGN1 and APP-TGN both have a TGN module in their models, but we can see that APP-TGN shows a better prediction performance over APP-TGN1 for two sub-tasks. The difference is that the TGN module in our APP-TGN adapts a low-high filtering information aggregation design. In contrast, the TGN module in APP-TGN1 adapts a conventional implementation [ 73 ], implying that the low-high filtering design is a better solution to capture more academic information during their learning processes. It demonstrates that the low-high filtering design can help the APP-TGN model distinguish between different learning behavior data levels, focusing on the most relevant and informative ones for academic performance prediction.

Parameter sensitivity in APP-TGN A parameter sensitivity analysis is performed on the main hyper-parameters in APP-TGN. Dynamic graph construction is crucial in APP-TGN, with the window size for updating a temporal graph being a critical hyperparameter that impacts prediction performance. Experimental results from various window size settings are presented in Table  5 . The prediction performances of these two sub-tasks are pretty sensitive to these hyper-parameter settings. APP-TGN achieves the best performance at a window size of 6 days. For the two sub-tasks, the performance of APP-TGN decreases when the window size exceeds 6 days. This suggests that using a large window size to update a dynamic graph may result in information loss and poor graph construction. It implies that the online learning logs of students are more informative and relevant when they are closer in time, and that older logs may not reflect students’ current state and behavior. As the window size increases beyond 6 days, the performance worsens. Furthermore, from Table  4 , we can see that the global sampling module plays a vital role in APP-TGN, which is an effective technique for reducing data bias. This means that the model can learn from a more representative and diverse set of students rather than focusing on a few dominant or frequent ones. The experimental results for APP-TGN and APP-LTL, concerning different hyper-parameter settings for the feature vectors N , are visualized in Fig.  5 . As shown in Fig.  5 a, APP-TGN delivers optimal performance with N as 300, while APP-LTL requires a larger amount of feature vectors, precisely 500, for optimal performance. Figure  5 b also shows a similar result, demonstrating the effectiveness of the linear transformation layer in the global sampling module. The layer can help reduce the dimensionality and complexity of the feature vectors, making them more suitable for temporal graph networks.

ACC(%) of APP-TGN and APP-LTL with different settings of the number of feature vector N in a global sampling module

Feature Importance and Contribution In the experiment conducted by us, the goal was to comprehend how different types of interactions influence student outcomes. Seven interaction features were utilized (as listed in Table  1 ), and an ablation study was carried out. This study involved the omission of one feature at a time from our APP-TGN model. The changes in prediction accuracy (%) for each performance category, resulting from this process, were documented and are displayed in Table  6 . The analysis brought to light that the Quiz and Forumng features have a significant bearing on the performance prediction of the model. The accuracy experienced a considerable drop when these features were removed, suggesting their critical role in capturing students’ learning behaviors and progress. It implies that future strategies for data collection could prioritize obtaining more detailed data concerning quizzes and forum interactions. Conversely, features such as Homepage, Subpage, Resource , among others, had a less noticeable impact on the prediction accuracy. This could be attributed to the redundancy or lower relevance of these features for the task at hand. Hence, future enhancements to the model could consider exploring techniques for feature selection or transformation to minimize redundancy and boost the predictive power of the input features. Interestingly, it was also observed that the influence of each feature differs across various performance categories, indicating that different features might be capturing distinct aspects of student performance. For example, a feature that is highly predictive for one category (e.g., Pass ) might not be as informative for another category (e.g., Fail ). This insight could steer the development of models specific to each category or the application of multi-task learning techniques to harness the differential predictive power of the features. In conclusion, the comprehensive analysis of the importance and contribution of features offers valuable insights that can enhance the model’s performance and guide future strategies for data collection.

Model Complexity and Computation Cost of APP-TGN The APP-TGN model is designed with computational efficiency in mind, making it suitable for handling large-scale MOOC data. The computational complexity of APP-TGN can be estimated by considering its components. A 1-layer GCN has a complexity of \({\mathcal {O}}(|E|d_id_o)\) where | E | is the number of edges, \(d_i\) is the input feature dimension, and \(d_o\) is the output feature dimension. A GAT-like layer [ 74 ] has a complexity of \({\mathcal {O}}(N_vd_id_o + |E|d_o)\) , where \(N_v\) is the number of activity types. The linear transformation attention in APP-TGN has a linear complexity of \({\mathcal {O}}(N_v)\) , similar to Linformer [ 75 ]. The k-Means feature clustering in the global sampling module is pre-processed and remains constant during training and testing. Therefore, the overall complexity of APP-TGN can be estimated as \({\mathcal {O}}(S|E|d_id_o+Sd_md_o)\) , where S the step size for prediction, \(d_m\) denotes the number of neurons in MLP for realizing learnable functions. Since the graph in each step is usually sparse, the computational cost of APP-TGN is similar when S is small. We report the FLOPs of several baselines and our APP-TGN (with a window size of 6 days). The FLOPs are as follows: OMLP - 0.151M, HRNs - 0.263M, CNN-LSTM - 0.924M, MTGNN - 1.705M, and APP-TGN - 0.6621M. Our computational cost is less than that of MTGNN. Compared to computer vision models like ResNet (1.8G FLOPs), the computational cost of these models is relatively small for this task and is not yet a significant concern. This further underscores the efficiency and scalability of APP-TGN for large-scale MOOC data.

Visualization of academic performance representations We visualize the academic representation of the category of Pass/Withdrawn in Fig.  6 . Figure  6 a shows the representations from the original feature spaces, where the features of Withdrawn and Pass overlap together in a feature space, making it difficult to classify a specific feature. Figure  6 b displays the representations learned from our APP-TGN of the category of Withdrawn and Pass . It can be seen that most features learned by APP-TGN are separable in the feature space. Compared to those not learned by APP-TGN, Feature representations learned by it have a more structured form and clear category boundaries. Thus, our APP-TGN can effectively cluster students’ academic performances within the same category, which can help educators identify students’ learning patterns, strengths, and weaknesses and provide personalized feedback and intervention.

Visualization of academic performance representations

Model Interpretability in Educational Context In this section, we discuss how the predictions of APP-TGN can be interpreted in an educational context based on the analysis of the model components and the experimental results. First, the dynamic graph construction module captures students’ temporal information and interaction behaviors during their online learning activities, which reflect their learning processes and states. The temporal graphs can be visualized to show the patterns and transitions of different learning activities, such as watching videos, reading texts, or taking quizzes. Second, the low-high filtering temporal graph network module learns the potential academic performance variations encoded in the dynamic graphs, representing student knowledge and skills changes over time. The low-high filters can identify the nodes’ and edges’ important and relevant features in the temporal graphs, such as the frequency, duration, order, or correlation of the learning activities. Third, the global sampling module mitigates the problem of false correlations in deep learning-based models by incorporating students’ demographic and contextual features, such as gender, region, disability, or highest education. The global sampling module can also provide a way to compare and contrast the performance of different groups of students based on these features. Finally, the academic performance representation and prediction module combines students’ local and global representations and uses a multi-head attention mechanism to generate the final predictions of academic outcomes. The attention weights can be interpreted as the importance or relevance of different features or components for the prediction task. For example, the attention weights can indicate which types of learning activities or which demographic or contextual factors are more influential in predicting a specific student’s performance or group of students. By providing these interpretations, APP-TGN can help educators and learners understand the factors and processes that affect students’ academic performance in online courses and provide feedback and guidance for improving their learning outcomes.

Implications

This paper introduces APP-TGN, a new method that uses online learning logs to predict academic performance. APP-TGN does not rely on any existing framework but instead constructs a dynamic graph from the raw data and applies temporal graph networks to learn the academic performance representation and prediction. Our framework leverages temporal graph networks to capture the dynamic and complex relationships between learning behaviors and academic outcomes. We also introduced a global sampling module to improve the representation learning for temporal graphs and a low-high filtering technique that eliminates the noise in online learning data. Our APP-TGN model achieved high accuracy rates in two prediction tasks, outperforming several baseline models by a significant margin. Specifically, in the experimental study of the first research question, our APP-TGN model achieved accuracy rates of 83.22% and 77.06% for two different tasks. These results represent statistically significant improvements over other models, with increases ranging from 1.23% to 8.29%. In the experimental study of the second research question, our APP-TGN model showed better statistically significant improvements over other models in early predicting at-risk students, with increases ranging from 2.99% to 12.97%. Our APP-TGN model is particularly effective in mining the dynamic relationship between learning behavior data and accurately predicting at-risk students. The third research question also demonstrates the effectiveness and superiority of our proposed techniques in APP-TGN. Overall, our model has great potential for use in automated feedback and personalized learning in real-world educational applications.

Limitations The APP-TGN prediction model has some limitations regarding data, algorithm, ethics, and generalizability. Firstly, there are few course interactions that form the model’s basis and could benefit from more data. Secondly, the APP-TGN algorithm cannot learn incrementally, or interactively like other supervised AI methods. However, an APP-TGN with a more extensive database could be used for quasi-real-time analysis. Thirdly, ethical considerations such as the potential influence of AI-enabled models on student learning outcomes should be considered. Future work could deliver real-time predictions, timely alerts, and suggestions to ensure positive outcomes from AI prediction methods. Lastly, the prediction method must enhance its generalizability through empirical research in various educational contexts and by considering external factors like offline classroom activities or social interactions.

Conclusions

Student academic performance prediction is fundamental in implementing intelligent services for massive open online courses. The paper explores exploiting temporal information and interaction behaviors during learning activities to promote the performance of model predictions. We represent the learning processes of e-learning students as dynamic temporal graphs that capture the temporal information and interaction behaviors during their studying. We also introduce APP-TGN, a new method for academic performance prediction that utilizes temporal graph neural networks. Specifically, in APP-TGN, a dynamic graph is constructed from the online learning activity logs. Generated graphs are forwarded to a revised temporal graph network with low-high filters to learn potential academic performance variations encoded in dynamic graphs. Furthermore, a global sampling module is developed to mitigate the problem of false correlations in deep learning-based models. Finally, the learned representations from the global sampling and local processing (with TGN) are forwarded to a multi-head attention module to get the predicted academic performances. We perform a case study with a popular dataset from a real-world educational application that is publicly available. Empirical study results indicate that APP-TGN, which we introduce, surpasses other methods by a large margin. The ablation study also reveals the effectiveness and superiority of our APP-TGN techniques.

Future work and extensions We intend to explore the following directions: (i)(i) Heterogeneous Data Sources: The primary focus of our existing model is structured data derived from learning management systems. However, the nature of educational data is often heterogeneous, incorporating text from student essays, audio from spoken responses, and video from recorded presentations. Our goal is to broaden the scope of our model to accommodate these varied data types. For example, we could employ natural language processing techniques for text data analysis, while audio and video data might be processed using deep learning models tailored for these specific data types. (ii) Incorporation of Additional Educational Data: Beyond the data currently in use, there are other forms of educational data that could offer valuable insights. These include demographic information, data on student learning styles, and affective states. The integration of these supplementary data sources could enhance the precision of our predictions and provide a more comprehensive understanding of student performance. (iii) Forecasting of Additional Educational Outcomes: Although our present focus is on predicting academic performance, the model has the potential to be modified to forecast other vital educational outcomes. These might encompass student retention rates, degrees of student engagement, or even student satisfaction. Each of these outcomes holds significant importance in the educational context, and their accurate prediction could have substantial implications for educational institutions. (iv) Pretraining-fine-tuning Schema: We are also keen on investigating a pretraining-fine-tuning schema in APP-TGN for a range of educational analytical tasks. This would involve retraining the model on a large dataset to discern general patterns, followed by fine-tuning it on a specific task with a smaller dataset. This method has proven effective in various domains and could enhance the performance of our model.

Availability of data and materials

The authors have no permission to share the dataset.

Huang C, Huang Q, Wang D. Stochastic configuration networks based adaptive storage replica management for power big data processing. IEEE Trans Ind Inf. 2019;16(1):373–83.

Article   Google Scholar  

Alsaroah AH, Al-Turjman F. Combining Cloud Computing with Artificial intelligence and Its Impact on Telecom Sector. NEU J Artif Intell Internet Things. 2023;2(3).

Baig MI, Shuib L, Yadegaridehkordi E. Big data in education: a state of the art, limitations, and future research directions. Int J Educ Technol High Educ. 2020;17(1):1–23.

Wang J. Comprehensive test and evaluation path of college teachers’ professional development based on a cloud education big data platform. Int J Emerg Technol Learn (Online). 2023;18(5):79.

Morris W, Crossley S, Holmes L, Trumbore A. Using transformer language models to validate peer-assigned essay scores in massive open online courses (MOOCs). In: LAK23: 13th international learning analytics and knowledge conference; 2023; p. 315–323.

Zheng Y, Yin B. Big data analytics in MOOCs. In: 2015 IEEE international conference on computer and information technology; ubiquitous computing and communications; dependable, autonomic and secure computing; pervasive intelligence and computing. IEEE; 2015; p. 681–686.;

Crompton H, Burke D. Artificial intelligence in higher education: the state of the field. Int J Educ Technol High Educ. 2023;20(1):1–22.

Wang X, Mei X, Huang Q, Han Z, Huang C. Fine-grained learning performance prediction via adaptive sparse self-attention networks. Inf Sci. 2021;545:223–40.

Article   MathSciNet   Google Scholar  

Liang J, Yang J, Wu Y, Li C, Zheng L, Big data application in education: dropout prediction in edx MOOCs. In: IEEE second international conference on multimedia big data (BigMM). IEEE. 2016;2016:440–3.

Chen Y, Zhai L. A comparative study on student performance prediction using machine learning. Educ Inf Technol. 2023;28:1–19.

Marbouti F, Diefes-Dux HA, Strobel J. Building course-specific regression-based models to identify at-risk students. In: 2015 ASEE Annual Conference and Exposition; 2015; p. 26–304.

Xu X, Wang J, Peng H, Wu R. Prediction of academic performance associated with internet usage behaviors using machine learning algorithms. Comput Human Behav. 2019;98:166–73.

Wang X, Wu P, Liu G, Huang Q, Hu X, Xu H. Learning performance prediction via convolutional GRU and explainable neural networks in e-learning environments. Computing. 2019;101(6):587–604.

Kukkar A, Mohana R, Sharma A, Nayyar A. Prediction of student academic performance based on their emotional wellbeing and interaction on various e-learning platforms. Educ Inf Technol. 2023;28:1–30.

Yang Z, Yang J, Rice K, Hung JL, Du X. Using convolutional neural network to recognize learning images for early warning of at-risk students. IEEE Trans Learn Technol. 2020;13(3):617–30.

Waheed H, Hassan SU, Nawaz R, Aljohani NR, Chen G, Gasevic D. Early prediction of learners at risk in self-paced education: a neural network approach. Expert Syst Appl. 2023;213: 118868.

Sun X, Cheng H, Liu B, Li J, Chen H, Xu G, et al. Self-supervised hypergraph representation learning for sociological analysis. IEEE Trans Knowl Data Eng. 2023. https://doi.org/10.1109/TKDE.2023.3235312 .

Li M, Wang X, Wang Y, Chen Y, Chen Y. Study-GNN: a novel pipeline for student performance prediction based on multi-topology graph neural networks. Sustainability. 2022;14(13):7965.

Wang C, Fang T, Gu Y. Learning performance and behavioral patterns of online collaborative learning: impact of cognitive load and affordances of different multimedia. Comput Educ. 2020;143: 103683.

Asadi M, Swamy V, Frej J, Vignoud J, Marras M, Käser T. Ripple: Concept-based interpretation for raw time series models in education. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2023; 13. p. 15903–15911.

Kuzilek J, Hlosta M, Zdrahal Z. Open university learning analytics dataset. Sci Data. 2017;4(1):1–8.

Ouyang F, Wu M, Zheng L, Zhang L, Jiao P. Integration of artificial intelligence performance prediction and learning analytics to improve student learning in online engineering course. Int J Educ Technol High Educ. 2023;20(1):1–23.

Ren Z, Rangwala H, Johri A. Predicting performance on MOOC assessments using multi-regression models. arXiv preprint arXiv:1605.02269 . 2016.

Chui KT, Fung DCL, Lytras MD, Lam TM. Predicting at-risk university students in a virtual learning environment via a machine learning algorithm. Comput Human Behav. 2020;107: 105584.

Coussement K, Phan M, De Caigny A, Benoit DF, Raes A. Predicting student dropout in subscription-based online learning environments: The beneficial impact of the logit leaf model. Decis Support Syst. 2020;135: 113325.

Riestra-González M, del Puerto Paule-Ruíz M, Ortin F. Massive LMS log data analysis for the early prediction of course-agnostic student performance. Comput Educ. 2021;163: 104108.

Turabieh H, Azwari SA, Rokaya M, Alosaimi W, Alharbi A, Alhakami W, et al. Enhanced Harris Hawks optimization as a feature selection for the prediction of student performance. Computing. 2021;103(7):1417–38.

Mubarak AA, Cao H, Zhang W. Prediction of students’ early dropout based on their interaction logs in online learning environment. Interact Learn Environ. 2022;30(8):1414–33.

Jiao P, Ouyang F, Zhang Q, Alavi AH. Artificial intelligence-enabled prediction model of student academic performance in online engineering education. Artif Intell Rev. 2022;55(8):6321–44.

Batool S, Rashid J, Nisar MW, Kim J, Kwon HY, Hussain A. Educational data mining to predict students’ academic performance: a survey study. Educ Inf Technol. 2023;28(1):905–71.

Peng T, Liang Y, Wu W, Ren J, Pengrui Z, Pu Y. CLGT: A graph transformer for student performance prediction in collaborative learning. In: Proceedings of the AAAI conference on artificial intelligence; 2023; p. 15947–15954.

Huang Q, Zeng Y. Improving academic performance predictions with dual graph neural networks. Complex and Intelligent Systems. 2024;p. 1–19.

Giannakas F, Troussas C, Voyiatzis I, Sgouropoulou C. A deep learning classification framework for early prediction of team-based academic performance. Appl Soft Comput. 2021;106: 107355.

Karimi H, Derr T, Huang J, Tang J. Online academic course performance prediction using relational graph convolutional neural network. In: Proceedings of the 13th international conference on educational data mining (EDM 2020); 2020. p. 444–450.

Waheed H, Hassan SU, Aljohani NR, Hardman J, Alelyani S, Nawaz R. Predicting academic performance of students from VLE big data using deep learning models. Comput Human Behav. 2020;104: 106189.

Du X, Yang J, Hung JL. An integrated framework based on latent variational autoencoder for providing early warning of at-risk students. IEEE Access. 2020;8:10110–22.

Sun X, Cheng H, Dong H, Qiao B, Qin S, Lin Q. Counter-empirical attacking based on adversarial reinforcement learning for time-relevant scoring system. IEEE Trans Knowl Data Eng. 2023. https://doi.org/10.1109/TKDE.2023.3341430 .

Li M, Zhang Y, Li X, Cai L, Yin B. Multi-view hypergraph neural networks for student academic performance prediction. Eng Appl Artif Intell. 2022;114:105174.

Sun X, Yin H, Liu B, Chen H, Cao J, Shao Y, et al. Heterogeneous hypergraph embedding for graph classification. In: Proceedings of the 14th ACM international conference on web search and data mining; 2021; p. 725–733.

Sun X, Zhang J, Wu X, Cheng H, Xiong Y, Li J. Graph Prompt Learning: A Comprehensive Survey and Beyond. arXiv preprint arXiv:2311.16534 . 2023.

Sun X, Cheng H, Li J, Liu B, Guan J. All in One: Multi-Task Prompting for Graph Neural Networks. In: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. KDD ’23. New York, NY, USA: Association for Computing Machinery; 2023; p. 2120–2131. https://doi.org/10.1145/3580305.3599256 .

Wu Z, Pan S, Chen F, Long G, Zhang C, Philip SY. A comprehensive survey on graph neural networks. IEEE Trans Neural Netw Learn Syst. 2020;32(1):4–24.

Zhou J, Cui G, Hu S, Zhang Z, Yang C, Liu Z, et al. Graph neural networks: a review of methods and applications. AI Open. 2020;1:57–81.

Zhang Z, Cui P, Zhu W. Deep learning on graphs: a survey. IEEE Trans Knowl Data Eng. 2020;34:249.

Wang S, Zeng Z, Yang X, Zhang X. Self-supervised Graph Learning for Long-tailed Cognitive Diagnosis. In: Proceedings of the AAAI conference on artificial intelligence; 2023; p. 110–118.

Gao W, Liu Q, Huang Z, Yin Y, Bi H, Wang MC, et al. Rcd: Relation map driven cognitive diagnosis for intelligent education systems. In: Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval; 2021; p. 501–510.

Zhang J, Mo Y, Chen C, He X. GKT-CD: Make Cognitive Diagnosis Model Enhanced by Graph-based Knowledge Tracing. In: 2021 International joint conference on neural networks (IJCNN). IEEE; 2021; p. 1–8.

Mao Y, Xu B, Yu J, Fang Y, Yuan J, Li J, et al. Learning behavior-aware cognitive diagnosis for online education systems. In: International conference of pioneering computer scientists, engineers and educators. Springer; 2021; p. 385–398.

Su Y, Cheng Z, Wu J, Dong Y, Huang Z, Wu L, et al. Graph-based cognitive diagnosis for intelligent tutoring systems. Knowl-Based Syst. 2022;253:109547.

Qi T, Ren M, Guo L, Li X, Li J, Zhang L. ICD: a new interpretable cognitive diagnosis model for intelligent tutor systems. Expert Syst Appl. 2023;215:119309.

Chen Y, Wang S, Jiang F, Tu Y, Huang Q. DCKT: a novel dual-centric learning model for knowledge tracing. Sustainability. 2022;14(23):16307.

Abdelrahman G, Wang Q, Nunes B. Knowledge tracing: a survey. ACM Comput Surv. 2023;55(11):1–37.

Nakagawa H, Iwasawa Y, Matsuo Y, Graph-based knowledge tracing: modeling student proficiency using graph neural network. In: IEEE/WIC/ACM international conference on web intelligence (WI). IEEE. 2019;2019:156–63.

Yang Y, Shen J, Qu Y, Liu Y, Wang K, Zhu Y, et al. GIKT: a graph-based interaction model for knowledge tracing. In: Joint European conference on machine learning and knowledge discovery in databases. Springer; 2020; p. 299–315.

Tong H, Wang Z, Liu Q, Zhou Y, Han W. HGKT: Introducing Hierarchical Exercise Graph for Knowledge Tracing. arXiv preprint arXiv:2006.16915 . 2020.

Song X, Li J, Tang Y, Zhao T, Chen Y, Guan Z. Jkt: a joint graph convolutional network based deep knowledge tracing. Inf Sci. 2021;580:510–23.

Wu Z, Huang L, Huang Q, Huang C, Tang Y. SGKT: session graph-based knowledge tracing for student performance prediction. Expert Syst Appl. 2022;206:117681.

Song X, Li J, Lei Q, Zhao W, Chen Y, Mian A. Bi-CLKT: Bi-graph contrastive learning based knowledge tracing. Knowl-Based Syst. 2022;241: 108274.

Wu T, Ling Q. Self-supervised heterogeneous hypergraph network for knowledge tracing. Inf Sci. 2023;624:200–16.

Wu T, Ling Q. Fusing hybrid attentive network with self-supervised dual-channel heterogeneous graph for knowledge tracing. Expert Syst Appl. 2023;225:120212.

Tan H, Wang C, Duan Q, Lu Y, Zhang H, Li R. Automatic short answer grading by encoding student responses via a graph convolutional network. Interact Learn Environ. 2023;31(3):1636–50.

Ying R, He R, Chen K, Eksombatchai P, Hamilton WL, Leskovec J. Graph convolutional neural networks for web-scale recommender systems. In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining; 2018; p. 974–983.

Wang H, Zhao M, Xie X, Li W, Guo M. Knowledge graph convolutional networks for recommender systems. In: The world wide web conference; 2019; p. 3307–3313.

Tan H, Wang C, Duan Q, Lu Y, Zhang H, Li R. Automatic short answer grading by encoding student responses via a graph convolutional network. Interact Learn Environ. 2020;31:1–15.

Google Scholar  

Agarwal R, Khurana V, Grover K, Mohania M, Goyal V. Multi-relational graph transformer for automatic short answer grading. In: Proceedings of the 2022 Conference of the North American chapter of the association for computational linguistics: human language technologies; 2022; p. 2001–2012.

Li H, Wei H, Wang Y, Song Y, Qu H. Peer-inspired student performance prediction in interactive online question pools with graph neural network. In: Proceedings of the 29th ACM international conference on information & knowledge management; 2020; p. 2589–2596.

Zhou Y, Zheng H, Huang X, Hao S, Li D, Zhao J. Graph neural networks: taxonomy, advances, and trends. ACM Trans Intell Syst Technol (TIST). 2022;13(1):1–54.

Bo D, Wang X, Shi C, Shen H. Beyond low-frequency information in graph convolutional networks. In: Proceedings of the AAAI conference on artificial intelligence. 2021; vol. 35. p. 3950–3957.

Michira MK, Rimiru RM, Mwangi WR. Improved multilayer perceptron neural networks weights and biases based on the grasshopper optimization algorithm to predict student performance on ambient learning. In: Proceedings of the 2023 7th international conference on machine learning and soft computing; 2023; p. 61–68.

Wang X, Zhao Y, Li C, Ren P. ProbSAP: a comprehensive and high-performance system for student academic performance prediction. Pattern Recognit. 2023;137: 109309.

Talebi K, Torabi Z, Daneshpour N. Ensemble models based on CNN and LSTM for dropout prediction in MOOC. Expert Syst Appl. 2024;235:121187.

Kukkar A, Mohana R, Sharma A, Nayyar A. A novel methodology using RNN + LSTM + ML for predicting student’s academic performance. Education and Information Technologies. 2024;p. 1–37.

Rossi E, Chamberlain B, Frasca F, Eynard D, Monti F, Bronstein M. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637 . 2020.

Veličković P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y. Graph Attention Networks. In: International conference on learning representations; 2018; p. 1–12.

Wang S, Li BZ, Khabsa M, Fang H, Ma H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 . 2020.

Download references

Acknowledgements

The research project is supported by National Natural Science Foundation of China (No. 62207028), and partially by Zhejiang Provincial Natural Science Foundation (No. LY23F020009), Key R &D Program of Zhejiang Province (No. 2022C03106), and National Natural Science Foundation of China (No. 62007031, 62177016), and Zhejiang Province Education Science Planning Annual General Planning Project (Universities) (No. 2023SCG367), and Open Research Fund of College of Teacher Education, Zhejiang Normal University (No. jykf22006).

Author information

Authors and affiliations.

Zhejiang Key Laboratory of Intelligent Education Technology and Application, Zhejiang Normal University, Yinbin Ave., Jinhua, 321004, Zhejiang, China

Qionghao Huang & Jili Chen

College of Education, Zhejiang Normal University, Yinbin Ave., Jinhua, 321004, Zhejiang , China

You can also search for this author in PubMed   Google Scholar

Contributions

The first author: Writing—original draft, Conceptualization, Software, Investigation, Writing—review and editing. The second author: Data curation, Visualization, Writing—review and editing.

Corresponding author

Correspondence to Qionghao Huang .

Ethics declarations

Ethics approval and consent to participate.

The study has no ethical issues, and no potentially identifiable personal information is presented in this study.

Competing interests

The authors declare that they have no Competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Huang, Q., Chen, J. Enhancing academic performance prediction with temporal graph networks for massive open online courses. J Big Data 11 , 52 (2024). https://doi.org/10.1186/s40537-024-00918-5

Download citation

Received : 25 December 2023

Accepted : 07 April 2024

Published : 13 April 2024

DOI : https://doi.org/10.1186/s40537-024-00918-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial intelligence
• Temporal graph networks
• Academic performance prediction