literature review of systematic literature reviews

• Locations and Hours
• UCLA Library
• Research Guides
• Biomedical Library Guides

Systematic Reviews

• Types of Literature Reviews

What Makes a Systematic Review Different from Other Types of Reviews?

• Planning Your Systematic Review
• Database Searching
• Creating the Search
• Search Filters and Hedges
• Grey Literature
• Managing and Appraising Results
• Further Resources

Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x

• << Previous: Home
• Next: Planning Your Systematic Review >>
• Last Updated: Jul 23, 2024 3:40 PM
• URL: https://guides.library.ucla.edu/systematicreviews

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Prevent plagiarism. Run a free check.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved October 15, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, what is your plagiarism score.

Reference management. Clean and simple.

How to write a systematic literature review [9 steps]

What is a systematic literature review?

Where are systematic literature reviews used, what types of systematic literature reviews are there, how to write a systematic literature review, 1. decide on your team, 2. formulate your question, 3. plan your research protocol, 4. search for the literature, 5. screen the literature, 6. assess the quality of the studies, 7. extract the data, 8. analyze the results, 9. interpret and present the results, registering your systematic literature review, frequently asked questions about writing a systematic literature review, related articles.

A systematic literature review is a summary, analysis, and evaluation of all the existing research on a well-formulated and specific question.

Put simply, a systematic review is a study of studies that is popular in medical and healthcare research. In this guide, we will cover:

• the definition of a systematic literature review
• the purpose of a systematic literature review
• the different types of systematic reviews
• how to write a systematic literature review

➡️ Visit our guide to the best research databases for medicine and health to find resources for your systematic review.

Systematic literature reviews can be utilized in various contexts, but they’re often relied on in clinical or healthcare settings.

Medical professionals read systematic literature reviews to stay up-to-date in their field, and granting agencies sometimes need them to make sure there’s justification for further research in an area. They can even be used as the starting point for developing clinical practice guidelines.

A classic systematic literature review can take different approaches:

• Effectiveness reviews assess the extent to which a medical intervention or therapy achieves its intended effect. They’re the most common type of systematic literature review.
• Diagnostic test accuracy reviews produce a summary of diagnostic test performance so that their accuracy can be determined before use by healthcare professionals.
• Experiential (qualitative) reviews analyze human experiences in a cultural or social context. They can be used to assess the effectiveness of an intervention from a person-centric perspective.
• Costs/economics evaluation reviews look at the cost implications of an intervention or procedure, to assess the resources needed to implement it.
• Etiology/risk reviews usually try to determine to what degree a relationship exists between an exposure and a health outcome. This can be used to better inform healthcare planning and resource allocation.
• Psychometric reviews assess the quality of health measurement tools so that the best instrument can be selected for use.
• Prevalence/incidence reviews measure both the proportion of a population who have a disease, and how often the disease occurs.
• Prognostic reviews examine the course of a disease and its potential outcomes.
• Expert opinion/policy reviews are based around expert narrative or policy. They’re often used to complement, or in the absence of, quantitative data.
• Methodology systematic reviews can be carried out to analyze any methodological issues in the design, conduct, or review of research studies.

Writing a systematic literature review can feel like an overwhelming undertaking. After all, they can often take 6 to 18 months to complete. Below we’ve prepared a step-by-step guide on how to write a systematic literature review.

• Decide on your team.
• Formulate your question.
• Plan your research protocol.
• Search for the literature.
• Screen the literature.
• Assess the quality of the studies.
• Extract the data.
• Analyze the results.
• Interpret and present the results.

When carrying out a systematic literature review, you should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

You may also need to team up with a librarian to help with the search, literature screeners, a statistician to analyze the data, and the relevant subject experts.

Define your answerable question. Then ask yourself, “has someone written a systematic literature review on my question already?” If so, yours may not be needed. A librarian can help you answer this.

You should formulate a “well-built clinical question.” This is the process of generating a good search question. To do this, run through PICO:

• Patient or Population or Problem/Disease : who or what is the question about? Are there factors about them (e.g. age, race) that could be relevant to the question you’re trying to answer?
• Intervention : which main intervention or treatment are you considering for assessment?
• Comparison(s) or Control : is there an alternative intervention or treatment you’re considering? Your systematic literature review doesn’t have to contain a comparison, but you’ll want to stipulate at this stage, either way.
• Outcome(s) : what are you trying to measure or achieve? What’s the wider goal for the work you’ll be doing?

Now you need a detailed strategy for how you’re going to search for and evaluate the studies relating to your question.

The protocol for your systematic literature review should include:

• the objectives of your project
• the specific methods and processes that you’ll use
• the eligibility criteria of the individual studies
• how you plan to extract data from individual studies
• which analyses you’re going to carry out

For a full guide on how to systematically develop your protocol, take a look at the PRISMA checklist . PRISMA has been designed primarily to improve the reporting of systematic literature reviews and meta-analyses.

When writing a systematic literature review, your goal is to find all of the relevant studies relating to your question, so you need to search thoroughly .

This is where your librarian will come in handy again. They should be able to help you formulate a detailed search strategy, and point you to all of the best databases for your topic.

➡️ Read more on on how to efficiently search research databases .

The places to consider in your search are electronic scientific databases (the most popular are PubMed , MEDLINE , and Embase ), controlled clinical trial registers, non-English literature, raw data from published trials, references listed in primary sources, and unpublished sources known to experts in the field.

➡️ Take a look at our list of the top academic research databases .

Tip: Don’t miss out on “gray literature.” You’ll improve the reliability of your findings by including it.

Don’t miss out on “gray literature” sources: those sources outside of the usual academic publishing environment. They include:

• non-peer-reviewed journals
• pharmaceutical industry files
• conference proceedings
• pharmaceutical company websites
• internal reports

Gray literature sources are more likely to contain negative conclusions, so you’ll improve the reliability of your findings by including it. You should document details such as:

• The databases you search and which years they cover
• The dates you first run the searches, and when they’re updated
• Which strategies you use, including search terms
• The numbers of results obtained

➡️ Read more about gray literature .

This should be performed by your two reviewers, using the criteria documented in your research protocol. The screening is done in two phases:

• Pre-screening of all titles and abstracts, and selecting those appropriate
• Screening of the full-text articles of the selected studies

Make sure reviewers keep a log of which studies they exclude, with reasons why.

➡️ Visit our guide on what is an abstract?

Your reviewers should evaluate the methodological quality of your chosen full-text articles. Make an assessment checklist that closely aligns with your research protocol, including a consistent scoring system, calculations of the quality of each study, and sensitivity analysis.

The kinds of questions you'll come up with are:

• Were the participants really randomly allocated to their groups?
• Were the groups similar in terms of prognostic factors?
• Could the conclusions of the study have been influenced by bias?

Every step of the data extraction must be documented for transparency and replicability. Create a data extraction form and set your reviewers to work extracting data from the qualified studies.

Here’s a free detailed template for recording data extraction, from Dalhousie University. It should be adapted to your specific question.

Establish a standard measure of outcome which can be applied to each study on the basis of its effect size.

Measures of outcome for studies with:

• Binary outcomes (e.g. cured/not cured) are odds ratio and risk ratio
• Continuous outcomes (e.g. blood pressure) are means, difference in means, and standardized difference in means
• Survival or time-to-event data are hazard ratios

Design a table and populate it with your data results. Draw this out into a forest plot , which provides a simple visual representation of variation between the studies.

Then analyze the data for issues. These can include heterogeneity, which is when studies’ lines within the forest plot don’t overlap with any other studies. Again, record any excluded studies here for reference.

Consider different factors when interpreting your results. These include limitations, strength of evidence, biases, applicability, economic effects, and implications for future practice or research.

Apply appropriate grading of your evidence and consider the strength of your recommendations.

It’s best to formulate a detailed plan for how you’ll present your systematic review results. Take a look at these guidelines for interpreting results from the Cochrane Institute.

Before writing your systematic literature review, you can register it with OSF for additional guidance along the way. You could also register your completed work with PROSPERO .

Systematic literature reviews are often found in clinical or healthcare settings. Medical professionals read systematic literature reviews to stay up-to-date in their field and granting agencies sometimes need them to make sure there’s justification for further research in an area.

The first stage in carrying out a systematic literature review is to put together your team. You should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

Your systematic review should include the following details:

A literature review simply provides a summary of the literature available on a topic. A systematic review, on the other hand, is more than just a summary. It also includes an analysis and evaluation of existing research. Put simply, it's a study of studies.

The final stage of conducting a systematic literature review is interpreting and presenting the results. It’s best to formulate a detailed plan for how you’ll present your systematic review results, guidelines can be found for example from the Cochrane institute .

Literature Reviews

• Overview of Literature Reviews and Systematic Reviews
• How to Get Started and Developing a Research Question
• Finding and Evaluating Sources
• Citations This link opens in a new window
• Synthesizing Sources
• Writing the Literature Review
• Systematic Reviews This link opens in a new window
• Suggested Readings

Literature Reviews and Systematic Reviews

• What is a Literature Review
• What is a Systematic Review
• Literature Review vs. Systematic Review

A literature review summarizes and analyzes the relevant publications on a topic. It demonstrates to your readers that you are knowledgeable of the ongoing scholarly conversation and how your research fits within the broader field of study. An effective literature review will lay the foundation for the importance of your stated problem and research question.

Literature Reviews: An Overview for Graduate Students from NC State University Libraries on Vimeo .

A systematic review attempts to identify, appraise and synthesize all available relevant evidence to answer a specific, focused research question. Researchers conducting systematic reviews use standardized, systematic methods and pre-selected eligibility criteria to reduce the risk of bias in identifying, selecting, and analyzing relevant studies.

Prepared by the Cochrane Consumers and Communication Group, La Trobe University and generously support by Cochrane Australia. Written by Jack Nunn and Sophie Hill.

Robinson, P. and Lowe, J. (2015), Literature reviews vs systematic reviews. Australian and New Zealand Journal of Public Health, 39: 103-103.  https://doi.org/10.1111/1753-6405.12393

Why Do a Literature Review

Why do a literature review?

• To increase your knowledge of this topic
• To identify other researchers and seminal works in this field of study
• To provide context for your work
• To locate gaps in the literature
• To demonstrate the credibility of your research

What is 'The Literature'

What is 'the literature' that is reviewed in a literature review? 

‘The literature’ consists of the published works that document a scholarly conversation in a field of study, including:

• scholarly articles
• conference proceedings
• dissertations

The literature can also include newspapers, encyclopedias, textbooks, as well as websites and reports written by government agencies and professional organizations ("grey literature").

Attribution

The content of this page was developed from Chapter 1, "Introduction", and Chapter 2, "What is a Literature Review?", in:

Frederiksen, L., & Phelps, S. F. (2017). Literature reviews for education and nursing graduate students. Rebus Community.  https://open.umn.edu/opentextbooks/textbooks/literature-reviews-for-education-and-nursing-graduate-students

• Next: How to Get Started and Developing a Research Question >>
• Last Updated: Nov 30, 2023 1:07 PM
• URL: https://libguides.chapman.edu/literature_reviews

Library Services

UCL LIBRARY SERVICES

• Guides and databases
• Library skills
• Systematic reviews

What are systematic reviews?

• Types of systematic reviews
• Formulating a research question
• Identifying studies
• Searching databases
• Describing and appraising studies
• Synthesis and systematic maps
• Software for systematic reviews
• Online training and support
• Live and face to face training
• Individual support
• Further help

Systematic reviews are a type of literature review of research that require equivalent standards of rigour to primary research. They have a clear, logical rationale that is reported to the reader of the review. They are used in research and policymaking to inform evidence-based decisions and practice. They differ from traditional literature reviews in the following elements of conduct and reporting.

Systematic reviews: 

• use explicit and transparent methods
• are a piece of research following a standard set of stages
• are accountable, replicable and updateable
• involve users to ensure a review is relevant and useful.

For example, systematic reviews (like all research) should have a clear research question, and the perspective of the authors in their approach to addressing the question is described. There are clearly described methods on how each study in a review was identified, how that study was appraised for quality and relevance and how it is combined with other studies in order to address the review question. A systematic review usually involves more than one person in order to increase the objectivity and trustworthiness of the reviews methods and findings.

Research protocols for systematic reviews may be peer-reviewed and published or registered in a suitable repository to help avoid duplication of reviews and for comparisons to be made with the final review and the planned review.

• History of systematic reviews to inform policy (EPPI-Centre)
• Six reasons why it is important to be systematic (EPPI-Centre)
• Evidence Synthesis International (ESI): Position Statement Describes the issues, principles and goals in synthesising research evidence to inform policy, practice and decisions

On this page

Should all literature reviews be 'systematic reviews', different methods for systematic reviews, reporting standards for systematic reviews.

Literature reviews provide a more complete picture of research knowledge than is possible from individual pieces of research. This can be used to: clarify what is known from research, provide new perspectives, build theory, test theory, identify research gaps or inform research agendas.

A systematic review requires a considerable amount of time and resources, and is one type of literature review.

If the purpose of a review is to make justifiable evidence claims, then it should be systematic, as a systematic review uses rigorous explicit methods. The methods used can depend on the purpose of the review, and the time and resources available.

A 'non-systematic review' might use some of the same methods as systematic reviews, such as systematic approaches to identify studies or quality appraise the literature. There may be times when this approach can be useful. In a student dissertation, for example, there may not be the time to be fully systematic in a review of the literature if this is only one small part of the thesis. In other types of research, there may also be a need to obtain a quick and not necessarily thorough overview of a literature to inform some other work (including a systematic review). Another example, is where policymakers, or other people using research findings, want to make quick decisions and there is no systematic review available to help them. They have a choice of gaining a rapid overview of the research literature or not having any research evidence to help their decision-making. 

Just like any other piece of research, the methods used to undertake any literature review should be carefully planned to justify the conclusions made. 

Finding out about different types of systematic reviews and the methods used for systematic reviews, and reading both systematic and other types of review will help to understand some of the differences. 

Typically, a systematic review addresses a focussed, structured research question in order to inform understanding and decisions on an area. (see the  Formulating a research question  section for examples). 

Sometimes systematic reviews ask a broad research question, and one strategy to achieve this is the use of several focussed sub-questions each addressed by sub-components of the review.  

Another strategy is to develop a map to describe the type of research that has been undertaken in relation to a research question. Some maps even describe over 2,000 papers, while others are much smaller. One purpose of a map is to help choose a sub-set of studies to explore more fully in a synthesis. There are also other purposes of maps: see the box on  systematic evidence maps  for further information. 

Reporting standards specify minimum elements that need to go into the reporting of a review. The reporting standards refer mainly to methodological issues but they are not as detailed or specific as critical appraisal for the methodological standards of conduct of a review.

A number of organisations have developed specific guidelines and standards for both the conducting and reporting on systematic reviews in different topic areas.  

• PRISMA PRISMA is a reporting standard and is an acronym for Preferred Reporting Items for Systematic Reviews and Meta-Analyses. The Key Documents section of the PRISMA website links to a checklist, flow diagram and explanatory notes. PRISMA is less useful for certain types of reviews, including those that are iterative.
• eMERGe eMERGe is a reporting standard that has been developed for meta-ethnographies, a qualitative synthesis method.
• ROSES: RepOrting standards for Systematic Evidence Syntheses Reporting standards, including forms and flow diagram, designed specifically for systematic reviews and maps in the field of conservation and environmental management.

Useful books about systematic reviews

Systematic approaches to a successful literature review

An introduction to systematic reviews

Cochrane handbook for systematic reviews of interventions

Systematic reviews: crd's guidance for undertaking reviews in health care.

Finding what works in health care: Standards for systematic reviews

Systematic Reviews in the Social Sciences

Meta-analysis and research synthesis.

Research Synthesis and Meta-Analysis

Doing a Systematic Review

Literature reviews.

• What is a literature review?
• Why are literature reviews important?
• << Previous: Systematic reviews
• Next: Types of systematic reviews >>
• Last Updated: Aug 2, 2024 9:22 AM
• URL: https://library-guides.ucl.ac.uk/systematic-reviews

Literature Review: Systematic literature reviews

• Traditional or narrative literature reviews
• Scoping Reviews
• Systematic literature reviews
• Annotated bibliography
• Keeping up to date with literature
• Finding a thesis
• Evaluating sources and critical appraisal of literature
• Managing and analysing your literature
• Further reading and resources

Systematic reviews

Systematic and systematic-like reviews

Charles Sturt University library has produced a comprehensive guide for Systematic and systematic-like literature reviews. A comprehensive systematic literature review can often take a team of people up to a year to complete. This guide provides an overview of the steps required for systematic reviews:

• Identify your research question
• Develop your protocol
• Conduct systematic searches (including the search strategy, text mining, choosing databases, documenting and reviewing
• Critical appraisal
• Data extraction and synthesis
• Writing and publishing .
• Systematic and systematic-like reviews Library Resource Guide

Systematic literature review

A systematic literature review (SLR) identifies, selects and critically appraises research in order to answer a clearly formulated question (Dewey, A. & Drahota, A. 2016). The systematic review should follow a clearly defined protocol or plan where the criteria is clearly stated before the review is conducted. It is a comprehensive, transparent search conducted over multiple databases and grey literature that can be replicated and reproduced by other researchers. It involves planning a well thought out search strategy which has a specific focus or answers a defined question. The review identifies the type of information searched, critiqued and reported within known timeframes. The search terms, search strategies (including database names, platforms, dates of search) and limits all need to be included in the review.

Pittway (2008) outlines seven key principles behind systematic literature reviews

• Transparency
• Integration
• Accessibility

Systematic literature reviews originated in medicine and are linked to evidence based practice. According to Grant & Booth (p 91, 2009) "the expansion in evidence-based practice has lead to an increasing variety of review types". They compare and contrast 14 review types, listing the strengths and weaknesses of each review. 

Tranfield et al (2003) discusses the origins of the evidence-based approach to undertaking a literature review and its application to other disciplines including management and science.

References and additional resources

Dewey, A. & Drahota, A. (2016) Introduction to systematic reviews: online learning module Cochrane Training   https://training.cochrane.org/interactivelearning/module-1-introduction-conducting-systematic-reviews

Gough, David A., David Gough, Sandy Oliver, and James Thomas. An Introduction to Systematic Reviews. Systematic Reviews. London: SAGE, 2012.

Grant, M. J. & Booth, A. (2009) A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal 26(2), 91-108

Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol, 18(1), 143. https://doi.org/10.1186/s12874-018-0611-x 

Pittway, L. (2008) Systematic literature reviews. In Thorpe, R. & Holt, R. The SAGE dictionary of qualitative management research. SAGE Publications Ltd doi:10.4135/9780857020109

Tranfield, D., Denyer, D & Smart, P. (2003) Towards a methodology for developing evidence-informed management knowledge by means of systematic review . British Journal of Management 14 (3), 207-222

Evidence based practice - an introduction : Literature reviews/systematic reviews

Evidence based practice - an introduction is a library guide produced at CSU Library for undergraduates. The information contained in the guide is also relevant for post graduate study and will help you to understand the types of research and levels of evidence required to conduct evidence based research.

• Evidence based practice an introduction
• << Previous: Scoping Reviews
• Next: Annotated bibliography >>
• Last Updated: Aug 11, 2024 4:07 PM
• URL: https://libguides.csu.edu.au/review

Charles Sturt University is an Australian University, TEQSA Provider Identification: PRV12018. CRICOS Provider: 00005F.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Systematic reviews of the literature: an introduction to current methods

Affiliation.

• 1 Department of Health Research Methods, Evidence, and Impact; McMaster University.
• PMID: 39038802
• DOI: 10.1093/aje/kwae232

Systematic reviews are a type of evidence synthesis in which authors develop explicit eligibility criteria, collect all the available studies that meet these criteria, and summarize results using reproducible methods that minimize biases and errors. Systematic reviews serve different purposes and use a different methodology than other types of evidence synthesis that include narrative reviews, scoping reviews, and overviews of reviews. Systematic reviews can address questions regarding effects of interventions or exposures, diagnostic properties of tests, and prevalence or prognosis of diseases. All rigorous systematic reviews have common processes that include: 1) determining the question and eligibility criteria, including a priori specification of subgroup hypotheses 2) searching for evidence and selecting studies, 3) abstracting data and assessing risk of bias of the included studies, 4) summarizing the data for each outcome of interest, whenever possible using meta-analyses, and 5) assessing the certainty of the evidence and drawing conclusions. There are several tools that can guide and facilitate the systematic review process, but methodological and content expertise are always necessary.

Keywords: Evidence synthesis; Systematic Reviews.

© The Author(s) 2024. Published by Oxford University Press on behalf of the Johns Hopkins Bloomberg School of Public Health.

PubMed Disclaimer

Similar articles

• Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas. Crider K, Williams J, Qi YP, Gutman J, Yeung L, Mai C, Finkelstain J, Mehta S, Pons-Duran C, Menéndez C, Moraleda C, Rogers L, Daniels K, Green P. Crider K, et al. Cochrane Database Syst Rev. 2022 Feb 1;2(2022):CD014217. doi: 10.1002/14651858.CD014217. Cochrane Database Syst Rev. 2022. PMID: 36321557 Free PMC article.
• The future of Cochrane Neonatal. Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
• The Effectiveness of Integrated Care Pathways for Adults and Children in Health Care Settings: A Systematic Review. Allen D, Gillen E, Rixson L. Allen D, et al. JBI Libr Syst Rev. 2009;7(3):80-129. doi: 10.11124/01938924-200907030-00001. JBI Libr Syst Rev. 2009. PMID: 27820426
• Impact of social protection on gender equality in low- and middle-income countries: A systematic review of reviews. Perera C, Bakrania S, Ipince A, Nesbitt-Ahmed Z, Obasola O, Richardson D, Van de Scheur J, Yu R. Perera C, et al. Campbell Syst Rev. 2022 May 25;18(2):e1240. doi: 10.1002/cl2.1240. eCollection 2022 Jun. Campbell Syst Rev. 2022. PMID: 36913187 Free PMC article. Review.
• What guidance is available for researchers conducting overviews of reviews of healthcare interventions? A scoping review and qualitative metasummary. Pollock M, Fernandes RM, Becker LA, Featherstone R, Hartling L. Pollock M, et al. Syst Rev. 2016 Nov 14;5(1):190. doi: 10.1186/s13643-016-0367-5. Syst Rev. 2016. PMID: 27842604 Free PMC article. Review.
• Development of the International Federation for Surgery of Obesity and Metabolic Disorders-European Chapter (IFSO-EC) Grade-Based Guidelines on the Surgical Treatment of Obesity Using Multimodal Strategies: Design and Methodological Aspects. De Luca M, Belluzzi A, Salminen P, Bueter M, Pujol-Rafols J, Sakran N, Stier C, Taskin HE, Chiappetta S, Carrano FM, Di Lorenzo N, Nienhuijs S, Puy RV, Stenberg E, Emous M, Prager G, Himpens J, Felsenreich DM, Iannelli A, Parmar C, Copaescu C, Fried M, Ruiz-Úcar E, Cohen RV, Olmi S, Angrisani L, Ribeiro R, Bandini G, Scoccimarro D, Ragghianti B, Monami M; Panel for the IFSO-EC on the Surgical Treatment of Obesity Using Multimodal Strategies. De Luca M, et al. J Clin Med. 2024 Aug 28;13(17):5106. doi: 10.3390/jcm13175106. J Clin Med. 2024. PMID: 39274320 Free PMC article.

LinkOut - more resources

Full text sources.

• Silverchair Information Systems

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Systematic literature reviews

Systematic approaches to literature review searching.

This guide is primarily for those undertaking a literature review. It outlines how to approach the searching phase systematically in order to identify relevant literature on a research question.

Why be systematic? This approach can:

• Provide a robust overview of the available literature on your topic
• Ensure relevant literature is identified and key publications are not overlooked
• Reduce irrelevant search results through search planning
• Help you to create a reproducible search strategy.

In addition, applying a systematic approach will allow you to work more efficiently.

A literature review may form an essential part of the research process, for example as a major component of a thesis or dissertation. Alternatively, a review may constitute a research project in itself - as a peer-reviewed publication in a journal, or as a report from a research funded project.

While the searching phase of any literature review should be approached in a systematic manner, you do not need to follow all of the techniques outlined in this guide. The methods you choose are dependent on the time and resources you have available, and the purpose of your literature review.

Systematic reviews vs. systematic approaches

A full systematic review aims to comprehensively identify, evaluate and integrate the findings of all relevant studies on a particular research question. A systematic approach involves a rigorous and structured search strategy, without necessarily attempting to include all available research on a particular topic.

What is a Systematic Literature Review?

A systematic literature review (SLR) is an independent academic method that aims to identify and evaluate all relevant literature on a topic in order to derive conclusions about the question under consideration. "Systematic reviews are undertaken to clarify the state of existing research and the implications that should be drawn from this." (Feak & Swales, 2009, p. 3) An SLR can demonstrate the current state of research on a topic, while identifying gaps and areas requiring further research with regard to a given research question. A formal methodological approach is pursued in order to reduce distortions caused by an overly restrictive selection of the available literature and to increase the reliability of the literature selected (Tranfield, Denyer & Smart, 2003). A special aspect in this regard is the fact that a research objective is defined for the search itself and the criteria for determining what is to be included and excluded are defined prior to conducting the search. The search is mainly performed in electronic literature databases (such as Business Source Complete or Web of Science), but also includes manual searches (reviews of reference lists in relevant sources) and the identification of literature not yet published in order to obtain a comprehensive overview of a research topic.

An SLR protocol documents all the information gathered and the steps taken as part of an SLR in order to make the selection process transparent and reproducible. The PRISMA flow-diagram support you in making the selection process visible.

In an ideal scenario, experts from the respective research discipline, as well as experts working in the relevant field and in libraries, should be involved in setting the search terms . As a rule, the literature is selected by two or more reviewers working independently of one another. Both measures serve the purpose of increasing the objectivity of the literature selection. An SLR must, then, be more than merely a summary of a topic (Briner & Denyer, 2012). As such, it also distinguishes itself from “ordinary” surveys of the available literature. The following table shows the differences between an SLR and an “ordinary” literature review.

• Charts of BSWL workshop (pdf, 2.88 MB)
• Listen to the interview (mp4, 12.35 MB)

Differences to "common" literature reviews

What are the objectives of SLRs?

• Avoidance of research redundancies despite a growing amount of publications
• Identification of research areas, gaps and methods
• Input for evidence-based management, which allows to base management decisions on scientific methods and findings
• Identification of links between different areas of researc

Process steps of an SLR

A SLR has several process steps which are defined differently in the literature (Fink 2014, p. 4; Guba 2008, Transfield et al. 2003). We distinguish the following steps which are adapted to the economics and management research area:

1. Defining research questions

Briner & Denyer (2009, p. 347ff.) have developed the CIMO scheme to establish clearly formulated and answerable research questions in the field of economic sciences:

C – CONTEXT:  Which individuals, relationships, institutional frameworks and systems are being investigated?

I – Intervention:  The effects of which event, action or activity are being investigated?

M – Mechanisms:  Which mechanisms can explain the relationship between interventions and results? Under what conditions do these mechanisms take effect?

O – Outcomes:  What are the effects of the intervention? How are the results measured? What are intended and unintended effects?

The objective of the systematic literature review is used to formulate research questions such as “How can a project team be led effectively?”. Since there are numerous interpretations and constructs for “effective”, “leadership” and “project team”, these terms must be particularized.

With the aid of the scheme, the following concrete research questions can be derived with regard to this example:

Under what conditions (C) does leadership style (I) influence the performance of project teams (O)?

Which constructs have an effect upon the influence of leadership style (I) on a project team’s performance (O)?          

Research questions do not necessarily need to follow the CIMO scheme, but they should:

• ... be formulated in a clear, focused and comprehensible manner and be answerable;
• ... have been determined prior to carrying out the SLR;
• ... consist of general and specific questions.

As early as this stage, the criteria for inclusion and exclusion are also defined. The selection of the criteria must be well-grounded. This may include conceptual factors such as a geographical or temporal restrictions, congruent definitions of constructs, as well as quality criteria (journal impact factor > x).

2. Selecting databases and other research sources

The selection of sources must be described and explained in detail. The aim is to find a balance between the relevance of the sources (content-related fit) and the scope of the sources.

In the field of economic sciences, there are a number of literature databases that can be searched as part of an SLR. Some examples in this regard are:

• Business Source Complete
• ProQuest One Business
• EconBiz        

Our video " Selecting the right databases " explains how to find relevant databases for your topic.

Literature databases are an important source of research for SLRs, as they can minimize distortions caused by an individual literature selection (selection bias), while offering advantages for a systematic search due to their data structure. The aim is to find all database entries on a topic and thus keep the retrieval bias low (tutorial on retrieval bias ).  Besides articles from scientific journals, it is important to inlcude working papers, conference proceedings, etc to reduce the publication bias ( tutorial on publication bias ).

Our online self-study course " Searching economic databases " explains step 2 und 3.

3. Defining search terms

Once the literature databases and other research sources have been selected, search terms are defined. For this purpose, the research topic/questions is/are divided into blocks of terms of equal ranking. This approach is called the block-building method (Guba 2008, p. 63). The so-called document-term matrix, which lists topic blocks and search terms according to a scheme, is helpful in this regard. The aim is to identify as many different synonyms as possible for the partial terms. A precisely formulated research question facilitates the identification of relevant search terms. In addition, keywords from particularly relevant articles support the formulation of search terms.

A document-term matrix for the topic “The influence of management style on the performance of project teams” is shown in this example .

Identification of headwords and keywords

When setting search terms, a distinction must be made between subject headings and keywords, both of which are described below:

• appear in the title, abstract and/or text
• sometimes specified by the author, but in most cases automatically generated
• non-standardized
• different spellings and forms (singular/plural) must be searched separately

Subject headings

• describe the content
• are generated by an editorial team
• are listed in a standardized list (thesaurus)
• may comprise various keywords
• include different spellings
• database-specific

Subject headings are a standardized list of words that are generated by the specialists in charge of some databases. This so-called index of subject headings (thesaurus) helps searchers find relevant articles, since the headwords indicate the content of a publication. By contrast, an ordinary keyword search does not necessarily result in a content-related fit, since the database also displays articles in which, for example, a word appears once in the abstract, even though the article’s content does not cover the topic.

Nevertheless, searches using both headwords and keywords should be conducted, since some articles may not yet have been assigned headwords, or errors may have occurred during the assignment of headwords. 

To add headwords to your search in the Business Source Complete database, please select the Thesaurus tab at the top. Here you can find headwords in a new search field and integrate them into your search query. In the search history, headwords are marked with the addition DE (descriptor).

The EconBiz database of the German National Library of Economics (ZBW – Leibniz Information Centre for Economics), which also contains German-language literature, has created its own index of subject headings with the STW Thesaurus for Economics . Headwords are integrated into the search by being used in the search query.

Since the indexes of subject headings divide terms into synonyms, generic terms and sub-aspects, they facilitate the creation of a document-term matrix. For this purpose it is advisable to specify in the document-term matrix the origin of the search terms (STW Thesaurus for Economics, Business Source Complete, etc.).

Searching in literature databases

Once the document-term matrix has been defined, the search in literature databases begins. It is recommended to enter each word of the document-term matrix individually into the database in order to obtain a good overview of the number of hits per word. Finally, all the words contained in a block of terms are linked with the Boolean operator OR and thereby a union of all the words is formed. The latter are then linked with each other using the Boolean operator AND. In doing so, each block should be added individually in order to see to what degree the number of hits decreases.

Since the search query must be set up separately for each database, tools such as  LitSonar  have been developed to enable a systematic search across different databases. LitSonar was created by  Professor Dr. Ali Sunyaev (Institute of Applied Informatics and Formal Description Methods – AIFB) at the Karlsruhe Institute of Technology.

Advanced search

Certain database-specific commands can be used to refine a search, for example, by taking variable word endings into account (*) or specifying the distance between two words, etc. Our overview shows the most important search commands for our top databases.

Additional searches in sources other than literature databases

In addition to literature databases, other sources should also be searched. Fink (2014, p. 27) lists the following reasons for this:

• the topic is new and not yet included in indexes of subject headings;
• search terms are not used congruently in articles because uniform definitions do not exist;
• some studies are still in the process of being published, or have been completed, but not published.

Therefore, further search strategies are manual search, bibliographic analysis, personal contacts and academic networks (Briner & Denyer, p. 349). Manual search means that you go through the source information of relevant articles and supplement your hit list accordingly. In addition, you should conduct a targeted search for so-called gray literature, that is, literature not distributed via the book trade, such as working papers from specialist areas and conference reports. By including different types of publications, the so-called publication bias (DBWM video “Understanding publication bias” ) – that is, distortions due to exclusive use of articles from peer-reviewed journals – should be kept to a minimum.

The PRESS-Checklist can support you to check the correctness of your search terms.

4. Merging hits from different databases

In principle, large amounts of data can be easily collected, structured and sorted with data processing programs such as Excel. Another option is to use reference management programs such as EndNote, Citavi or Zotero. The Saxon State and University Library Dresden (SLUB Dresden) provides an  overview of current reference management programs  . Software for qualitative data analysis such as NVivo is equally suited for data processing. A comprehensive overview of the features of different tools that support the SLR process can be found in Bandara et al. (2015).

Our online-self study course "Managing literature with Citavi" shows you how to use the reference management software Citavi.

When conducting an SLR, you should specify for each hit the database from which it originates and the date on which the query was made. In addition, you should always indicate how many hits you have identified in the various databases or, for example, by manual search.

Exporting data from literature databases

Exporting from literature databases is very easy. In  Business Source Complete  , you must first click on the “Share” button in the hit list, then “Email a link to download exported results” at the very bottom and then select the appropriate format for the respective literature program.

Exporting data from the literature database  EconBiz  is somewhat more complex. Here you must first create a marked list and then select each hit individually and add it to the marked list. Afterwards, articles on the list can be exported.

After merging all hits from the various databases, duplicate entries (duplicates) are deleted.

5. Applying inclusion and exclusion criteria

All publications are evaluated in the literature management program applying the previously defined criteria for inclusion and exclusion. Only those sources that survive this selection process will subsequently be analyzed. The review process and inclusion criteria should be tested with a small sample and adjustments made if necessary before applying it to all articles. In the ideal case, even this selection would be carried out by more than one person, with each working independently of one another. It needs to be made clear how discrepancies between reviewers are dealt with. 

The review of the criteria for inclusion and exclusion is primarily based on the title, abstract and subject headings in the databases, as well as on the keywords provided by the authors of a publication in the first step. In a second step the whole article / source will be read.

You can create tag words for the inclusion and exclusion in your literature management tool to keep an overview.

In addition to the common literature management tools, you can also use software tools that have been developed to support SLRs. The central library of the university in Zurich has published an overview and evaluation of different tools based on a survey among researchers. --> View SLR tools

The selection process needs to be made transparent. The PRISMA flow diagram supports the visualization of the number of included / excluded studies.

Forward and backward search

Should it become apparent that the number of sources found is relatively small, or if you wish to proceed with particular thoroughness, a forward-and-backward search based on the sources found is recommendable (Webster & Watson 2002, p. xvi). A backward search means going through the bibliographies of the sources found. A forward search, by contrast, identifies articles that have cited the relevant publications. The Web of Science and Scopus databases can be used to perform citation analyses.

6. Perform the review

As the next step, the remaining titles are analyzed as to their content by reading them several times in full. Information is extracted according to defined criteria and the quality of the publications is evaluated. If the data extraction is carried out by more than one person, a training ensures that there will be no differences between the reviewers.

Depending on the research questions there exist diffent methods for data abstraction (content analysis, concept matrix etc.). A so-called concept matrix can be used to structure the content of information (Webster & Watson 2002, p. xvii). The image to the right gives an example of a concept matrix according to Becker (2014).

Particularly in the field of economic sciences, the evaluation of a study’s quality cannot be performed according to a generally valid scheme, such as those existing in the field of medicine, for instance. Quality assessment therefore depends largely on the research questions.

Based on the findings of individual studies, a meta-level is then applied to try to understand what similarities and differences exist between the publications, what research gaps exist, etc. This may also result in the development of a theoretical model or reference framework.

Example concept matrix (Becker 2013) on the topic Business Process Management

7. Synthesizing results

Once the review has been conducted, the results must be compiled and, on the basis of these, conclusions derived with regard to the research question (Fink 2014, p. 199ff.). This includes, for example, the following aspects:

• historical development of topics (histogram, time series: when, and how frequently, did publications on the research topic appear?);
• overview of journals, authors or specialist disciplines dealing with the topic;
• comparison of applied statistical methods;
• topics covered by research;
• identifying research gaps;
• developing a reference framework;
• developing constructs;
• performing a meta-analysis: comparison of the correlations of the results of different empirical studies (see for example Fink 2014, p. 203 on conducting meta-analyses)

Publications about the method

Bandara, W., Furtmueller, E., Miskon, S., Gorbacheva, E., & Beekhuyzen, J. (2015). Achieving Rigor in Literature Reviews: Insights from Qualitative Data Analysis and Tool-Support.  Communications of the Association for Information Systems . 34(8), 154-204.

Booth, A., Papaioannou, D., and Sutton, A. (2012)  Systematic approaches to a successful literature review.  London: Sage.

Briner, R. B., & Denyer, D. (2012). Systematic Review and Evidence Synthesis as a Practice and Scholarship Tool. In Rousseau, D. M. (Hrsg.),  The Oxford Handbook of Evidenence Based Management . (S. 112-129). Oxford: Oxford University Press.

Durach, C. F., Wieland, A., & Machuca, Jose A. D. (2015). Antecedents and dimensions of supply chain robustness: a systematic literature review . International Journal of Physical Distribution & Logistic Management , 46 (1/2), 118-137. doi:  https://doi.org/10.1108/IJPDLM-05-2013-0133

Feak, C. B., & Swales, J. M. (2009). Telling a Research Story: Writing a Literature Review.  English in Today's Research World 2.  Ann Arbor: University of Michigan Press. doi:  10.3998/mpub.309338

Fink, A. (2014).  Conducting Research Literature Reviews: From the Internet to Paper  (4. Aufl.). Los Angeles, London, New Delhi, Singapore, Washington DC: Sage Publication.

Fisch, C., & Block, J. (2018). Six tips for your (systematic) literature review in business and management research.  Management Review Quarterly,  68, 103–106 (2018).  doi.org/10.1007/s11301-018-0142-x

Guba, B. (2008). Systematische Literaturrecherche.  Wiener Medizinische Wochenschrift , 158 (1-2), S. 62-69. doi:  doi.org/10.1007/s10354-007-0500-0  Hart, C.  Doing a literature review: releasing the social science research imagination.  London: Sage.

Jesson, J. K., Metheson, L. & Lacey, F. (2011).  Doing your Literature Review - traditional and Systematic Techniques . Los Angeles, London, New Delhi, Singapore, Washington DC: Sage Publication.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi: 10.1136/bmj.n71.

Petticrew, M. and Roberts, H. (2006).  Systematic Reviews in the Social Sciences: A Practical Guide . Oxford:Blackwell. Ridley, D. (2012).  The literature review: A step-by-step guide . 2nd edn. London: Sage. 

Chang, W. and Taylor, S.A. (2016), The Effectiveness of Customer Participation in New Product Development: A Meta-Analysis,  Journal of Marketing , American Marketing Association, Los Angeles, CA, Vol. 80 No. 1, pp. 47–64.

Tranfield, D., Denyer, D. & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review.  British Journal of Management , 14 (3), S. 207-222. doi:  https://doi.org/10.1111/1467-8551.00375

Webster, J., & Watson, R. T. (2002). Analyzing the Past to Prepare for the Future: Writing a Literature Review.  Management Information Systems Quarterly , 26(2), xiii-xxiii.  http://www.jstor.org/stable/4132319

Durach, C. F., Wieland, A. & Machuca, Jose. A. D. (2015). Antecedents and dimensions of supply chain robustness: a systematic literature review. International Journal of Physical Distribution & Logistics Management, 45(1/2), 118 – 137.

What is particularly good about this example is that search terms were defined by a number of experts and the review was conducted by three researchers working independently of one another. Furthermore, the search terms used have been very well extracted and the procedure of the literature selection very well described.

On the downside, the restriction to English-language literature brings the language bias into play, even though the authors consider it to be insignificant for the subject area.

Bos-Nehles, A., Renkema, M. & Janssen, M. (2017). HRM and innovative work behaviour: a systematic literature review. Personnel Review, 46(7), pp. 1228-1253

• Only very specific keywords used
• No precise information on how the review process was carried out (who reviewed articles?)
• Only journals with impact factor (publication bias)

Jia, F., Orzes, G., Sartor, M. & Nassimbeni, G. (2017). Global sourcing strategy and structure: towards a conceptual framework. International Journal of Operations & Production Management, 37(7), 840-864

• Research questions are explicitly presented
• Search string very detailed
• Exact description of the review process
• 2 persons conducted the review independently of each other

Franziska Klatt

[email protected]

+49 30 314-29778

Privacy notice: The TU Berlin offers a chat information service. If you enable it, your IP address and chat messages will be transmitted to external EU servers. more information

The chat is currently unavailable.

Please use our alternative contact options.

About Systematic Reviews

Understanding the Differences Between a Systematic Review vs Literature Review

Automate every stage of your literature review to produce evidence-based research faster and more accurately.

Let’s look at these differences in further detail.

Goal of the Review

The objective of a literature review is to provide context or background information about a topic of interest. Hence the methodology is less comprehensive and not exhaustive. The aim is to provide an overview of a subject as an introduction to a paper or report. This overview is obtained firstly through evaluation of existing research, theories, and evidence, and secondly through individual critical evaluation and discussion of this content.

A systematic review attempts to answer specific clinical questions (for example, the effectiveness of a drug in treating an illness). Answering such questions comes with a responsibility to be comprehensive and accurate. Failure to do so could have life-threatening consequences. The need to be precise then calls for a systematic approach. The aim of a systematic review is to establish authoritative findings from an account of existing evidence using objective, thorough, reliable, and reproducible research approaches, and frameworks.

Level of Planning Required

The methodology involved in a literature review is less complicated and requires a lower degree of planning. For a systematic review, the planning is extensive and requires defining robust pre-specified protocols. It first starts with formulating the research question and scope of the research. The PICO’s approach (population, intervention, comparison, and outcomes) is used in designing the research question. Planning also involves establishing strict eligibility criteria for inclusion and exclusion of the primary resources to be included in the study. Every stage of the systematic review methodology is pre-specified to the last detail, even before starting the review process. It is recommended to register the protocol of your systematic review to avoid duplication. Journal publishers now look for registration in order to ensure the reviews meet predefined criteria for conducting a systematic review [1].

Search Strategy for Sourcing Primary Resources

Learn more about distillersr.

(Article continues below)

Quality Assessment of the Collected Resources

A rigorous appraisal of collected resources for the quality and relevance of the data they provide is a crucial part of the systematic review methodology. A systematic review usually employs a dual independent review process, which involves two reviewers evaluating the collected resources based on pre-defined inclusion and exclusion criteria. The idea is to limit bias in selecting the primary studies. Such a strict review system is generally not a part of a literature review.

Presentation of Results

Most literature reviews present their findings in narrative or discussion form. These are textual summaries of the results used to critique or analyze a body of literature about a topic serving as an introduction. Due to this reason, literature reviews are sometimes also called narrative reviews. To know more about the differences between narrative reviews and systematic reviews , click here.

A systematic review requires a higher level of rigor, transparency, and often peer-review. The results of a systematic review can be interpreted as numeric effect estimates using statistical methods or as a textual summary of all the evidence collected. Meta-analysis is employed to provide the necessary statistical support to evidence outcomes. They are usually conducted to examine the evidence present on a condition and treatment. The aims of a meta-analysis are to determine whether an effect exists, whether the effect is positive or negative, and establish a conclusive estimate of the effect [2].

Using statistical methods in generating the review results increases confidence in the review. Results of a systematic review are then used by clinicians to prescribe treatment or for pharmacovigilance purposes. The results of the review can also be presented as a qualitative assessment when the end goal is issuing recommendations or guidelines.

Risk of Bias

Literature reviews are mostly used by authors to provide background information with the intended purpose of introducing their own research later. Since the search for included primary resources is also less exhaustive, it is more prone to bias.

One of the main objectives for conducting a systematic review is to reduce bias in the evidence outcome. Extensive planning, strict eligibility criteria for inclusion and exclusion, and a statistical approach for computing the result reduce the risk of bias.

Intervention studies consider risk of bias as the “likelihood of inaccuracy in the estimate of causal effect in that study.” In systematic reviews, assessing the risk of bias is critical in providing accurate assessments of overall intervention effect [3].

With numerous review methods available for analyzing, synthesizing, and presenting existing scientific evidence, it is important for researchers to understand the differences between the review methods. Choosing the right method for a review is crucial in achieving the objectives of the research.

[1] “Systematic Review Protocols and Protocol Registries | NIH Library,” www.nihlibrary.nih.gov . https://www.nihlibrary.nih.gov/services/systematic-review-service/systematic-review-protocols-and-protocol-registries

[2] A. B. Haidich, “Meta-analysis in medical research,” Hippokratia , vol. 14, no. Suppl 1, pp. 29–37, Dec. 2010, [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3049418/#:~:text=Meta%2Danalyses%20are%20conducted%20to

3 Reasons to Connect

Systematic literature reviews

• Introduction

Systematic literature reviews defined

How these articles can be useful in getting an overview.

• Types of reviews
• Finding literature review articles
• Conducting a systematic literature review

Contact the library

If you're a student and need help, you can use the chat service, stop by the info desk, send us an e-mail or book a guidance session with a librarian. See more information on the student portal.

PhD candidates and researchers: check out information on this service on the intranet

We suggest two alternative definitions of  review articles:

"There is no commonly agreed definition of review articles and different disciplines might value them differently. However, in general parlance review articles are defined as articles that do not contain original data and simply collect, review and synthesize earlier research, without including substantial theoretical or conceptual development." (Harzing, 2013, 24) .

"a systematic, explicit, and reproducible method for identifying, evaluating, and synthesizing the existing body of completed and recorded work produced by researchers, scholars, and practitioners. (Fink, 2020, 254)

The main focus of a systematic review is to synthesize previous research. These articles do  not present any new, original data . In other words, to perform a (systematic) literature review, is a research method in its own right. In general, these types of study typically:

• require a research question that can be answered by studying a group of studies that all are trying to answer the same question
• collect, sort and code a group of studies that are very similar (Wang, 2019, pp. 4-5)

Typically, the methodology section of a review article may explain how and where the author has identified relevant studies and which criteria have been used for inclusion and exclusion. Such information can include the search string used, list of databases or a time frame. So, a review article will usually have a specific scope and not necessarily cover the complete body of research within an area.

Most research articles contain a literature review section, where the author links the study to relevant literature and theory. However, this does not make them review articles. "Book review" is another example of a document type that is not relevant here.

For different types of reviews, see this page . 

Searching for already published review articles on your research question, may be a good starting point for your topic and may give you a kick start in getting an overview, especially if you're able to find a fairly recent review article.  These particular type of articles:

• Provide a summary of research on a topic
• Identify researchers and specific works within the area (great way to find relevant authors and studies related to a research topic)
• Often identify gaps in the research and indicate where more research is needed (may help you to find and formulate a research question)

One should not expect to find literature reviews within all research areas and certainly not in extensive numbers. Also note that new research areas are not likely to have been covered (yet).

• << Previous: Introduction
• Next: Types of reviews >>
• Last Updated: Jun 20, 2024 1:16 PM
• URL: https://libguides.bi.no/systematic_literature_reviews

Penn State University Libraries

• Home-Articles and Databases
• Asking the clinical question
• PICO & Finding Evidence
• Evaluating the Evidence
• Systematic Review vs. Literature Review
• Fall 2024 Workshops
• Nursing Library Instruction Course
• Ethical & Legal Issues for Nurses
• Useful Nursing Resources
• Writing Resources
• LionSearch and Finding Articles
• The Catalog and Finding Books

Know the Difference! Systematic Review vs. Literature Review

It is common to confuse systematic and literature reviews as both are used to provide a summary of the existent literature or research on a specific topic.  Even with this common ground, both types vary significantly.  Please review the following chart (and its corresponding poster linked below) for the detailed explanation of each as well as the differences between each type of review.

• What's in a name? The difference between a Systematic Review and a Literature Review, and why it matters by Lynn Kysh, MLIS, University of Southern California - Norris Medical Library
• << Previous: Evaluating the Evidence
• Next: Fall 2024 Workshops >>
• Last Updated: Aug 27, 2024 1:21 PM
• URL: https://guides.libraries.psu.edu/nursing

Psychology: Systematic Reviews and Literature Reviews

• Find Articles
• Find Journals
• Dissertations
• Videos & More
• Systematic Reviews and Literature Reviews

Guidelines for Systematic Reviews

• PRISMA Checklist The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) has developed a 27 checklist items pertaining to the content of a systematic review and meta-analysis, which include the title, abstract, methods, results, discussion and funding.
• NIH Guidelines This website provides guidelines developed by the Institute of Medicine (IOM). more... less... In 2011, the Institute of Medicine (IOM) defined a systematic evidence review as "a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies. It may include a quantitative synthesis (meta-analysis), depending on the available data."

Register your Systematic Review

Prospero Registry

"PROSPERO is an international database of prospectively registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome." (Website- About)

Conducting a Literature Review

• The Literature Review (25 minutes Video - opens in a new window)) This in-depth video lecture explains how to write a Literature Review, and examines which elements are required in one. This video was created the Student Learning Centre at Massey University, Auckland.

Systematic Reviews - What to Consider

Before you Begin: 

• Did you know, it takes an average of 18 months to complete a systematic review?
• Did you know that it's recommended to have at least 3 people on a systematic review team? 
• Did you know that a systematic review answers a very specific type of research question? Is your question a good fit for a systematic review? 

Contact your librarian, Emily Hart , to learn more about systematic reviews. 

Types of Reviews

Comparison of Review Types from Cornell University Libraries

" A systematic review is a review of the literature that is conducted in a methodical manner based on a pre-specified protocol and with the aim of synthesizing the retrieved information often by means of a meta-analysis."

" A literature review is a descriptive and/or analytic summary of the existing material relating to some topic or area of study."

(Sage Research Methods Online)

Systematic Review vs. Literature Review - What's the Difference?

• << Previous: Videos & More
• Last Updated: Aug 5, 2024 12:38 PM
• URL: https://researchguides.library.syr.edu/psychology
• Print This Page

Human Movement

• Key resources
• Biomechanics and Anatomy
• Nutrition and Dietetics
• Socio-cultural studies of sport
• Exercise and sport science
• Sports Coaching
• Statistics and organisations

What is a literature review?

What is a systematic review, helpful guides, books and articles.

• Referencing

A literature review is a critical assessment of the literature pertaining to a particular topic or subject.  It is a 'systematic, explicit, and reproducible method for identifying, evaluating and synthesising the existing body of completed and recorded work produced by researchers, scholars, and practitioners'  Fink, (2005).

What's its purpose?

• justify your research
• provide context for your research
• identify new ways, to interpret and highlight gaps in previous research
• ensure that the research has not been done before
• signpost a way forward for further research
• show where the research fits into the existing literature
• highlight flaws in previous research

"Systematic reviews aim to identify, evaluate and summarise the findings of all relevant individual studies, thereby making the available evidence more accessible to decisionmakers. When appropriate, combining the results of several studies gives a more reliable and precise estimate of an intervention’s effectiveness than one study alone." 

Centre for Reviews and Dissemination 2009, Systematic Reviews: CRD's guidance for undertaking reviews in health care , Centre for Reviews and Dissemination, York.

  The key characteristics of a systematic review are:

• a clearly stated set of objectives with pre-defined eligibility criteria for studies;
• an explicit, reproducible methodology;
• a systematic search that attempts to identify all studies that would meet the eligibility criteria; learn more about how to conduct a thorough systematic search ;
• an assessment of the validity of the findings of the included studies, for example through the assessment of risk of bias; and
• a systematic presentation, and synthesis, of the characteristics and findings of the included studies.

Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from  http://handbook-5-1.cochrane.org/ 

• Literature reviews library guide
• Systematic reviews library guide
• Finding grey literature library guide
• Using NVivo software in systematic reviews
• UQ Student Support - writing a literature review A guide to the process of planning and writing a literature review.
• Doing a literature review in nursing, health and social care
• The essential guide to doing a health and social care literature review
• Health sciences literature review made easy: The matrix method
• The literature review: A step-by-step guide for students
• The literature review: Six steps to success
• Systematic approaches to a successful literature review
• Writing a literature review Includes checklists
• Writing an effective literature review
• << Previous: Statistics and organisations
• Next: Referencing >>
• Last Updated: Oct 11, 2024 6:33 PM
• URL: https://guides.library.uq.edu.au/human-movement

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

The PMC website is updating on October 15, 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• PMC11463120

Medical student perceptions of gender and pain: a systematic review of the literature

Maia patrick-smith.

Imperial College London, Exhibition Rd, South Kensington, London, SW7 2AZ UK

Stephanie Bull

Associated data.

Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.

No datasets were generated or analysed during the current study.

Gender bias exists in healthcare and affects how pain is assessed and managed. This bias affects patient outcomes and their trust in healthcare professionals. We also know that future clinicians develop their attitudes early in training. Medical school is therefore an opportunity to shape the values of future doctors and to combat systemic gender bias in healthcare. This systematic review aims to explore medical student perceptions of the relationship between patients’ gender and their pain, so that recommendations can be made for developing medical education.

Embase, MEDLINE, PsychINFO, LILACS, Global Index Medicus, PakMediNet and ERIC were searched for articles relating to medical student perceptions of gender and pain, with no geographical or language limitations. Quality was assessed using the Medical Education Research Study Quality Index and the Critical Appraisal Skills Programme checklist.

Nine publications were identified, two qualitative and seven quantitative. All studies had methodological limitations. Many different study designs were used, although most involved simulated patients. All studies referred to gender as binary. Multiple studies found that women’s pain is more likely to be underestimated by medical students and that the patient’s gender drives different approaches during clinical history taking, examination and management in these simulated situations. Only one study found no effect of patients’ gender on students’ perception of their pain.

Conclusions

Whilst there is a paucity of high-quality studies in this area, patients’ gender was found to affect how their pain is perceived by medical students. No studies explored where students’ attitudes towards gender and pain arise from, and few involved ‘real life situations’. We propose that further work into medical student perceptions in ‘real situations’ is needed. This will help to inform how undergraduate medical education can be developed to tackle gender bias, and ultimately improve outcomes for patients.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12916-024-03660-0.

It is well established that there is a gender bias in the assessment and management of pain by healthcare professionals [ 1 – 4 ]. Women’s pain is systematically underestimated by clinicians and undertreated in comparison to pain experienced by men [ 1 ]. Women have been found to receive less analgesia after surgery [ 5 ] and are more likely to be prescribed psychological treatment rather than traditional analgesia for their pain [ 3 ]. This is on a background of significant pain-related morbidity in women, with one in five women in the USA living with chronic pain [ 6 ]. The difference in the way that pain is assessed and managed based on a patient’s gender results in women receiving inadequate pain management [ 5 , 7 ] and it taking women longer to receive a diagnosis than men for the same types of pain [ 8 ]. This impacts on their immediate and long-term health, as well as their trust in healthcare professionals [ 4 ].

Two main types of gender bias are seen in medicine: gender stereotyping , where patients are treated differently because of their gender in a clinically unjustified way; and gender blindness , where clinically important differences between the genders are overlooked [ 9 , 10 ]. Though we know that both forms of bias play a role in how doctors respond to and manage pain, what is less clear are the origins of these biases [ 11 ].

Medical students are the doctors of the future and medical school, as well as medical school faculty and the wider clinical team that students encounter, shapes many of their professional values and perspectives [ 12 ]. This may be through formal learning from the official medical school curriculum, or through informal learning which can be termed the ‘hidden curriculum’ [ 13 ]. There have been multiple calls for greater discussion of gender bias in medical school curricula [ 9 , 10 ], and some medical schools have already developed tools to help students improve their gender awareness [ 14 , 15 ]. However, without a greater understanding of the pre-existing attitudes and perceptions amongst medical students, and of whether they hold similar attitudes to clinicians in relation to patients’ gender and their pain, it is difficult to determine where efforts should be targeted, and which interventions will have the best chance of success.

The aim of this study is to conduct a systematic review of the literature [ 16 ] to explore medical student perceptions of the relationship between patients’ gender and their pain. We are seeking to understand what is known, what remains unknown and where there is uncertainty around findings. The ambition is that by understanding the current state of knowledge, we can make recommendations for what should come next, so that we can develop medical education that better supports the reduction of gender bias in medical students, our future doctors, and ultimately improve patient care.

The systematic review approach followed PRISMA guidelines ( see Additional file 1: Supplementary file 1 and Additional file 2: Supplementary file 2) and is described below.

MP-S and SB created the initial search to identify articles relating to medical students’ perception of the relationship between a patient’s gender and their pain, considering the wide range of keywords and Medical Subject Headings (MeSH) that could be used to describe the concepts of gender, pain and medical students. Search testing was performed by MP-S in Embase. This involved reviewing the dominant MeSH terms used to tag 3 known relevant articles and adapting the search so that these were included. Further refinement to the MeSH terms and keywords were made to reflect this.

Keywords, and their Boolean combinations, were used across databases Embase, MEDLINE, PsychINFO, LILACS, Global Index Medicus, PakMediNet and ERIC. MeSH terms were available in MEDLINE, Embase, PsychINFO, LILACS, Global Index Medicus and ERIC and were varied to reflect the vocabulary of each database. PakMediNet does not use MeSH terms, and so keywords were used only.

Searches were conducted on 18 May 2023. The search string for Embase is displayed below with MESH terms indicated in bold:

( Medical School OR Medical Student OR Medical Education OR BMBS OR MBBS OR BMBCh OR medic* adj2 student*) AND ( Gender OR Gender Equity OR Gender Inequality OR gender* OR Gender Bias OR Sexism ) AND ( Pain OR Pain Assessment OR Pain Intensity OR Pain Severity OR Chronic Pain OR Pain Threshold OR Psychogenic Pain OR pain* OR Analgesia ).

The search string was translated into the syntax and vocabulary of each additional database (Additional file 3: Supplementary file 3).

The results from each database search were imported into Endnote 21 (Clarivate PLC, Jersey, UK) for ease of de-duplication. After de-duplication, studies were uploaded to a screening platform Covidence (Veritas Health Innovation, Melbourne, Australia) to facilitate team collaboration whilst screening. Titles and abstracts of studies were screened against the listed inclusion and exclusion criteria by one reviewer (MP-S). At this stage, a cautious approach was adopted, only excluding articles that obviously bore no relation to the research question.

Study selection

The following inclusion and exclusion criteria were used:

• Studies involving medical students but not qualified healthcare professionals or students studying any other healthcare discipline.
• Studies where it was not possible to separate outcomes or analysis of medical students’ perceptions from other groups of healthcare professionals were excluded.
• Studies looking at medical students’ perception of (attitudes towards, opinions on, experience of, or evaluations of) the relationship between a patient’s gender and their pain (assessment and/or management).
• Studies that did not focus on the medical students’ perception of gender and pain (e.g. studies solely looking at the prevalence of pain between genders) were excluded.
• Studies published in any language—translation to the English language was conducted using Google Translate.
• Studies from any geographical location.
• Qualitative or quantitative empirical studies, opinion or comment pieces and conference proceedings were included.
• Literature indexed within databases: Embase, PsychINFO, ERIC, MEDLINE, Global Index Medicus, PakMediNet and LILACS.
• The reference lists of all included published studies were also searched.

Three people (MP-S, SB and MB — see the ‘Acknowledgements’ section) reviewed the full text of the articles against the inclusion criteria. MP-S and SB met to discuss any uncertainties over inclusion, adding clarifying details to the inclusion and exclusion criteria to enhance transparency. Uncertainties over inclusion were discussed by two reviewers (MP-S and SB) until a consensus was formed. The reference lists of included articles were screened by MP-S to search for any relevant articles that had not been identified in the preliminary search. One study required translation from French which was performed using Google Translate (Fig. ​ (Fig.1 1 ).

Flow diagram (based on PRISMA template [ 19 ]) showing the data collection process for the included studies

Data extraction

Data was extracted from each included paper by MP-S and SB. Data extracted included paper identifiers (authorship, date), study type (empirical qualitative, quantitative), study setting (geographical location and type of setting), the aims of the study that were relevant to the research question and details of the study including methods, results and conclusions (in relation to medical students perceptions of gender and pain) that can be drawn from each study.

Quality assessment

The quality of included empirical studies involving quantitative and qualitative research was assessed by one reviewer (MP-S), using the Medical Education Research Study Quality Index (MERSQI) [ 17 ] and the Critical Appraisal Skills Programme (CASP) checklist [ 18 ]. This checklist is not designed to assign a numerical score to assess quality and has been used to identify where there are perceived methodological limitations. Uncertainties were resolved through discussion with SB.

A descriptive synthesis of studies was conducted taking care to address the main research question: how do medical students perceive the relationship between patients’ gender and their pain?

Reflexivity

Whilst the authorship team have conducted a systematic search and synthesis of the evidence, they would also like to share their positionality with regard to the research topic. Both identify as women, are of white ethnicity and have roles as an educator and an early career clinician who both teach medical students. Both SB and MP-S have experienced pain requiring assessment and treatment by healthcare professionals. Each was satisfied with the pain management that they received and felt that it was not affected by their gender. The researchers took time to reflect on whether their experiences may be influencing their interpretation of the findings during the research process.

Nine empirical studies met the inclusion criteria (Additional file 4: Table 1), made up of two qualitative studies [ 15 , 20 ] and seven quantitative studies [ 2 , 21 – 26 ]. No review articles, opinion or comment pieces were found during the search. The quantitative studies ranged in size, with two studies having less than 30 participants [ 2 , 25 ], one with over 200 participants [ 22 ] and the remainder between the two. Methodological limitations were identified in all publications during the quality appraisal process, although there was a variation in quality across the studies (MERSQI score: 9.5–12.5, CASP: 1–5 limitations). A cut-off MERSQI score of 14 out of a possible 18 has been used to indicate high quality in previous literature [ 27 ]. Notably, in many of the studies, the role of gender in relation to pain was not the main focus of the research, rather it was a variable that was being taken into consideration in a study with other goals. All studies were from the Global North [ 28 ], with three from the USA, two from the UK and four from mainland Europe. Seven studies were based at a single site and two were multi-site studies [ 21 , 22 ].

Study setting

The two qualitative studies [ 15 , 20 ] looked at student reflections of real clinical cases that they had observed, whereas all other studies used simulations—either actors, mannikins or written vignettes—where the student acted as the clinician. Two studies [ 21 , 22 ] considered the perception of gender and pain in a paediatric patient population and the rest considered the relationship between gender and pain in adults. Six studies looked at acute pain [ 15 , 22 – 26 ], two used examples of chronic pain [ 2 , 21 ] and one had a mixed sample [ 20 ]. All studies referred to gender as binary and none clarified whether they referred to gender identity or biological sex when using terms such as ‘man/men’, ‘woman/women’, ‘boy’, ‘girl’, ‘male’ or ‘female’ within their manuscripts.

Perceptions and biases

Eight of the nine studies concluded that gender influenced student’s perceptions of patient’s pain, measured through assigned pain ratings or through student reflections. In the only study that did not make this association, the patient population were children undergoing venepuncture [ 22 ]. Notably, the study used a video of a child that had been ‘validated’ to have a gender-neutral appearance, and the authors assigned the child a typically male or typically female name within each vignette to denote the child’s gender. Two studies found that medical students assigned women lower pain ratings than men despite the identical nature of the other aspects of the case [ 2 , 26 ]. Another showed that medical students felt that, in children, girls were more distressed by their pain than boys. The same study also tested medical students’ attitudes towards gender and pain using an implicit association test (designed for measuring attitudes towards adults in pain) and found that students perceive women to be more pain-sensitive than men [ 21 ]. One qualitative study described a student reflection of a clinical encounter where the attitude held by medical professionals was that men express pain less than women [ 15 ]. The student felt this had affected the quality of care that the man was given as non-pharmaceutical options to alleviate pain were not explored.

Clinical history, examination and investigations

A gender bias in the clinical history-taking and examination of patients with pain was found in three studies. One study found that the characteristics of a patient’s pain were explored more thoroughly by medical students when the patient was a woman [ 23 ] and in another qualitative study, a student reflected that women in pain may be examined differently by clinicians due to concerns about modesty or dignity [ 20 ]. More subtle gender differences were revealed in a study that showed that the students’ own gender influences the ability to interpret pain expressions in patients of the opposite gender when there is a disconnect between the student’s belief in the pain level and the patient’s (in this case a mannikin) response to pain. This altered their examination technique with students palpating the abdomen more forcefully when the patients’ pain was disbelieved [ 25 ].

The only study that looked at investigations for patients with pain found no difference in the frequency with which computed tomography (CT) scans were proposed for patients who were men and women with symptoms of acute aortic dissection during an Observed Structured Clinical Examination [ 23 ].

Diagnosis and management

The relationship between pain and psychological distress was explored in two publications [ 23 , 24 ]. One study found that, in the context of stress and anxiety, women’s pain was more likely to be perceived as psychogenic than men’s pain [ 24 ]. Another study identified that for patients with identical presentations, anxiety was more frequently diagnosed as the causative complaint in women than men [ 23 ].

Students reflected that in circumstances where women’s pain is believed to be psychological, then less analgesia is likely to be given and there can be a potential delay in diagnosis [ 20 ]. Another student reflected that women may be more likely to be offered non-pharmacological interventions for their pain than men [ 15 ]. In a quantitative study, however, despite perceiving a gender difference in the level of distress that a child displayed whilst in pain, there was no difference in how the medical student treated the patient [ 21 ].

We know that healthcare professionals’ perception of patients’ pain is influenced by patient gender; however, it was not previously clear whether medical students hold similar perceptions with regard to patients’ gender and their pain, and if so, how, why and when these perceptions develop.

This review has shown that medical students do hold the perception that women are more pain sensitive than men and that they regard women to feel more distressed and to be inclined to exaggerate the level of pain that they feel. However, we also identified that there is a paucity of literature specifically on medical student perceptions of the relationship between patients’ gender and their pain and that the studies that have been conducted are of relatively low methodological quality. Therefore, a cautious approach should be adopted when interpreting study findings, and more studies addressing this research area would be beneficial. All but two studies occurred in simulated situations, meaning that it is unclear how these perceptions may play out in real-life clinical encounters. In addition, whilst it is thought that gender can affect all aspects of a patient’s pain-related care, from the assessment (history-taking, examination and investigations) to management, there was only one study that had looked at the impact of gender on medical students approach to ordering clinical investigations [ 23 ]. This is another gap in the literature where further work would be useful.

There are many examples of how gender influences patient care. Women are underrepresented in research trials, which results in gaps in our knowledge about women’s health and means that it is often assumed that conditions present in the same manner in both men and women. Men are more likely to experience left-sided chest pain when having a ‘heart attack’ or myocardial infarction (MI), and therefore the symptoms that women are more likely to experience such as nausea or abdominal discomfort are labelled as ‘atypical’. This is likely to explain why, in the case of MI, there is often a delay in women presenting to the hospital and receiving treatment [ 29 ]. When we consider some of the outcomes in clinical practice that may be specifically associated with gender bias and pain, we see that women are more likely to be prescribed antidepressants or receive a mental health referral for pain symptoms than men [ 3 , 4 , 11 ] and that women presenting with chest pain are less likely to receive diagnostic and interventional cardiac procedures [ 30 ]. We also know that women report feeling disbelieved by physicians [ 31 ] and that this disbelief can lead to a breakdown of the therapeutic relationship and can cause worsening of the patient’s pain [ 32 ]. It is therefore of particular interest that we also found that when features of stress and anxiety were added to vignettes of patients with chest pain, medical students deemed women more likely to have a psychogenic cause of their pain whereas no effect was seen with men [ 24 ]. It was also noted that medical students asked women more in-depth questions about their pain than men [ 23 ] and whilst the reason for this was not explored further, might it represent a lack of belief in the woman’s pain and a feeling that women’s pain is more likely to be driven by poor mental health rather than having a physical basis? It has been seen that medical students assess patients’ pain levels as higher when there is clear medical evidence to support why a patient is in pain [ 33 ].

We know that perceptions of gender are complex and that gender roles within our society exist. Such roles may mean that a stereotypical man is seen as stoic and less emotional [ 34 ], and a stereotypical woman is seen as dramatic and irrational [ 35 ] and as having a greater tendency to overtly express their pain [ 36 ]. This can then affect the way that their pain is perceived by others [ 1 , 37 ]. It may also be that the gender bias experienced by a patient in relation to their pain is affected by the extent to which they display more typically ‘masculine’ or ‘feminine’ traits within their gender identity. Factors other than gender are known to affect the perception of pain by clinicians, the most studied demographic in this context being ethnicity [ 38 ]; however, this review has focused solely on gender. In children, the influence of gender in relation to pain perception by clinicians appears to be less strong and strengthens during adolescence [ 39 ]. It is interesting that the only study in this review where students did not express gender bias [ 22 ] involved a child patient with a gender-neutral appearance. As the aim of this review was to document what is known about the perception of pain by medical students according to patient gender and where the gaps are, it was important to include patients from any demographic.

Whilst existing studies show that gender does affect the way pain is perceived by medical students, there is limited explanation of why this is the case and how, why and when these views have developed. As educators, we are particularly interested to understand the influence of medical training on these perceptions. Further research is required in this field, and in particular explorations of how students’ values and perspectives are shaped by both the formal and informal medical curriculum. Further exploration of this topic is important as medical students are our future clinicians and we know that medical school has an impact on the values that they hold [ 12 ]. Medical school is a timely opportunity to shape these values and although we now know that medical students do perceive pain differently depending on the patient’s gender, at present there is not enough research looking into where these attitudes come from. With this piece of the puzzle missing, it becomes difficult to know how best to integrate learning on this topic into teaching.

This is the first published literature review on this subject to our knowledge. Multiple databases were searched, and care was taken to broaden the search beyond the Global North by including databases indexing literature with a wider geographical reach. The search strategy and inclusion criteria were developed in an iterative fashion, with continuous testing, and regular discussion between the authors. The authors also spent time reflecting on how their own experiences and perceptions may affect their interpretation of the research findings.

Limitations

Despite the care taken to conduct an exhaustive search, the included studies remained geographically limited and as such their findings are bounded in the values and cultures of those conducting the research. The methodological quality of the studies identified was generally low, and so our interpretations of the findings need to be viewed within this context. This review has not taken an intersectional approach and does not consider other variables that may affect medical students’ perceptions of patients’ pain beyond gender, such as ethnicity and social class. At present, there is not enough literature in the field to be able to take an intersectional approach; however, this would be an important future work. The study also did not use explicit search terms to include patients with transgender or non-binary gender identities, although the MeSH term ‘Gender’ was included across all databases. Research into how biological sex affects pain has been researched by many teams over the last 40 years [ 40 , 41 ]. It is therefore possible that some of the medical students within the studies we have included have been exposed to this research and that their perception of gender and pain has been affected by this exposure. This was not explored within this review but would be an interesting topic for future research. Despite the careful reflection carried out by the authorship team throughout the research process, the interpretations may be limited by the experiences, ethnicity and gender of the authors.

Existing studies have shown that medical students’ perception of pain is influenced by the gender of the patient. The biases seen in medical students align with those that are seen in clinical practice and are seen across all aspects of the patient journey, including both the assessment and management of a patient’s pain. The evidence base is, however, weak and gaps remain in our knowledge, particularly with relation to how, why and when medical students develop these biases. Further work addressing these questions and ultimately considering pain perception in a more intersectional context would be valuable. Understanding more about medical students’ perception of pain and gender is key to the planning and implementation of effective learning within the undergraduate medical school curriculum.

Acknowledgements

We would like to thank Dr. Megan Brown for her contribution during the full-text review stage of our systematic review and Dr. Jo Horsburgh for her critical review of the manuscript.

Abbreviations

Authors’ contributions

MP-S and SB contributed equally to the study conception and design. The literature search, title/abstract screening and quality assessment was carried out by MP-S. Full text review, data extraction and synthesis were performed by both MP-S and SB. The first draft of the manuscript was written by MP-S with extensive input from SB. All authors read and approved the final manuscript. Both authors are accountable for all the work and the manuscript submitted.

This article presents independent research commissioned by the National Institute for Health Research (NIHR) under the Applied Health Research (ARC) programme for North West London. The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Availability of data and materials

Data availability, declarations.

Not applicable.

The authors declare no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.