critical appraisal of literature reviews

• University of Texas Libraries
• UT Libraries

Systematic Reviews & Evidence Synthesis Methods

Critical appraisal.

• Types of Reviews
• Formulate Question
• Find Existing Reviews & Protocols
• Register a Protocol
• Searching Systematically
• Supplementary Searching
• Managing Results
• Deduplication
• Glossary of terms
• Librarian Support
• Video tutorials This link opens in a new window
• Systematic Review & Evidence Synthesis Boot Camp

Some reviews require a critical appraisal for each study that makes it through the screening process. This involves a risk of bias assessment and/or a quality assessment. The goal of these reviews is not just to find all of the studies, but to determine their methodological rigor, and therefore, their credibility.

"Critical appraisal is the balanced assessment of a piece of research, looking for its strengths and weaknesses and them coming to a balanced judgement about its trustworthiness and its suitability for use in a particular context." 1

It's important to consider the impact that poorly designed studies could have on your findings and to rule out inaccurate or biased work.

Selection of a valid critical appraisal tool, testing the tool with several of the selected studies, and involving two or more reviewers in the appraisal are good practices to follow.

1. Purssell E, McCrae N. How to Perform a Systematic Literature Review: A Guide for Healthcare Researchers, Practitioners and Students. 1st ed. Springer ;  2020.

Evaluation Tools

• The Appraisal of Guidelines for Research & Evaluation Instrument (AGREE II) The Appraisal of Guidelines for Research & Evaluation Instrument (AGREE II) was developed to address the issue of variability in the quality of practice guidelines.
• Centre for Evidence-Based Medicine (CEBM). Critical Appraisal Tools "contains useful tools and downloads for the critical appraisal of different types of medical evidence. Example appraisal sheets are provided together with several helpful examples."
• Critical Appraisal Skills Programme (CASP) Checklists Critical Appraisal checklists for many different study types
• Critical Review Form for Qualitative Studies Version 2, developed out of McMaster University
• Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS) Downes MJ, Brennan ML, Williams HC, et al. Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS). BMJ Open 2016;6:e011458. doi:10.1136/bmjopen-2016-011458
• Downs & Black Checklist for Assessing Studies Downs, S. H., & Black, N. (1998). The Feasibility of Creating a Checklist for the Assessment of the Methodological Quality Both of Randomised and Non-Randomised Studies of Health Care Interventions. Journal of Epidemiology and Community Health (1979-), 52(6), 377–384.
• GRADE The Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group "has developed a common, sensible and transparent approach to grading quality (or certainty) of evidence and strength of recommendations."
• Grade Handbook Full handbook on the GRADE method for grading quality of evidence.
• MAGIC (Making GRADE the Irresistible choice) Clear succinct guidance in how to use GRADE
• Joanna Briggs Institute. Critical Appraisal Tools "JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers." Includes checklists for 13 types of articles.
• Latitudes Network This is a searchable library of validity assessment tools for use in evidence syntheses. This website also provides access to training on the process of validity assessment.
• Mixed Methods Appraisal Tool A tool that can be used to appraise a mix of studies that are included in a systematic review - qualitative research, RCTs, non-randomized studies, quantitative studies, mixed methods studies.
• RoB 2 Tool Higgins JPT, Sterne JAC, Savović J, Page MJ, Hróbjartsson A, Boutron I, Reeves B, Eldridge S. A revised tool for assessing risk of bias in randomized trials In: Chandler J, McKenzie J, Boutron I, Welch V (editors). Cochrane Methods. Cochrane Database of Systematic Reviews 2016, Issue 10 (Suppl 1). dx.doi.org/10.1002/14651858.CD201601.
• ROBINS-I Risk of Bias for non-randomized (observational) studies or cohorts of interventions Sterne J A, Hernán M A, Reeves B C, Savović J, Berkman N D, Viswanathan M et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions BMJ 2016; 355 :i4919 doi:10.1136/bmj.i4919
• Scottish Intercollegiate Guidelines Network. Critical Appraisal Notes and Checklists "Methodological assessment of studies selected as potential sources of evidence is based on a number of criteria that focus on those aspects of the study design that research has shown to have a significant effect on the risk of bias in the results reported and conclusions drawn. These criteria differ between study types, and a range of checklists is used to bring a degree of consistency to the assessment process."
• The TREND Statement (CDC) Des Jarlais DC, Lyles C, Crepaz N, and the TREND Group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. Am J Public Health. 2004;94:361-366.
• Assembling the Pieces of a Systematic Reviews, Chapter 8: Evaluating: Study Selection and Critical Appraisal.
• How to Perform a Systematic Literature Review, Chapter: Critical Appraisal: Assessing the Quality of Studies.

Other library guides

• Duke University Medical Center Library. Systematic Reviews: Assess for Quality and Bias
• UNC Health Sciences Library. Systematic Reviews: Assess Quality of Included Studies
• Last Updated: Aug 12, 2024 8:26 AM
• URL: https://guides.lib.utexas.edu/systematicreviews

Best Practice for Literature Searching

• Literature Search Best Practice
• What is literature searching?
• What are literature reviews?
• Hierarchies of evidence
• 1. Managing references
• 2. Defining your research question
• 3. Where to search
• 4. Search strategy
• 5. Screening results
• 6. Paper acquisition
• 7. Critical appraisal
• Further resources
• Training opportunities and videos
• Join FSTA student advisory board This link opens in a new window
• Chinese This link opens in a new window
• Italian This link opens in a new window
• Persian This link opens in a new window
• Portuguese This link opens in a new window
• Spanish This link opens in a new window

Deciding what to include in your review through critical appraisal

Once you have narrowed down your pool of results, it's time to begin critically appraising your articles.  Using a checklist helps you scrutinise articles in a consistent, structured way.  

Questions to consider include: 

• Are the aims of the study clearly stated?
• Is the study design suitable for the aims?
• Are the measurements and methods used clearly described?
• Are the correct measurement tools used?
• Are the statistical methods described?
• Was the sample size adequate? 
• Are the methods overall described in enough detail that you could replicate the study?
• Does the discussion overall reflect the results?
• Who funded this study?
• What are the specific limitations of what can be concluded from the study?

Working through the questions will help you identify the strengths and weakness of each article, and also identify points to draw on when you write about the literature. 

• DOWNLOAD THE CRITICAL APPRAISAL CHECKLIST

Additional critical appraisal checklists

REFLECT provides a  checklist for evaluating randomized control trials in livestock and food safety. 

CASP provides  checklists  for critical appraisal of studies related to health.

JBI provides checklists for critical appraisal of studies related to health.

Documenting critical appraisal decisions

As you closely examine full articles, you will be making judgements about why to include  or exclude  each study from your review.  Documenting your reasoning will help you reassure yourself and demonstrate to others that you have been systematic and unbiased in your appr aisal decisions.

Keeping track of what you have excluded, and why, will be very helpful if you must defend your work—for instance, if your literature review is part of a dissertation or thesis. 

Pulling all the literature you will include in your review into a single chart is a good way to begin to synthesise the literature. 

• DOWNLOAD THE FULL TEXT SCREENING CHART

Best practice!

BEST PRACTICE RECOMMENDATION :  If you include any direct quotes in your chart (or in any notes) be sure to use quotation marks so that you don’t later mistake the words for your own.

BEST PRACTICE RECOMMENDATION: The more carefully you record each of the steps of your process, the more easily reproducible it will be. This is especially important for research abstracts and articles found in conference proceedings.

• << Previous: 6. Paper acquisition
• Next: Further resources >>
• Last Updated: May 17, 2024 5:48 PM
• URL: https://ifis.libguides.com/literature_search_best_practice

Conducting a Literature Review

• Literature Review
• Developing a Topic
• Planning Your Literature Review
• Developing a Search Strategy
• Managing Citations
• Critical Appraisal Tools
• Writing a Literature Review

Appraise Your Research Articles

The structure of a literature review should include the following :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance  -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology  -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity  -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness  -- which of the author's theses are most convincing or least convincing?
• Value  -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

Reviewing the Literature

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what the articles are saying, but how are they saying it.

Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?
• When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Tools for Critical Appraisal

Now, that you have found articles based on your research question you can appraise the quality of those articles. These are resources you can use to appraise different study designs.

Centre for Evidence Based Medicine (Oxford)

University of Glasgow

"AFP uses the Strength-of-Recommendation Taxonomy (SORT), to label key recommendations in clinical review articles."

• SORT: Rating the Strength of Evidence    American Family Physician and other family medicine journals use the Strength of Recommendation Taxonomy (SORT) system for rating bodies of evidence for key clinical recommendations.
• The Interprofessional Health Sciences Library
• 123 Metro Boulevard
• Nutley, NJ 07110
• [email protected]
• Student Services
• Parents and Families
• Career Center
• Web Accessibility
• Visiting Campus
• Public Safety
• Disability Support Services
• Campus Security Report
• Report a Problem
• Login to LibApps

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Published: 08 April 2022

How to appraise the literature: basic principles for the busy clinician - part 1: randomised controlled trials

• Aslam Alkadhimi 1 ,
• Samuel Reeves 2 &
• Andrew T. DiBiase 3  

British Dental Journal volume  232 ,  pages 475–481 ( 2022 ) Cite this article

878 Accesses

1 Citations

2 Altmetric

Metrics details

Critical appraisal is the process of carefully, judiciously and systematically examining research to adjudicate its trustworthiness and its value and relevance in clinical practice. The first part of this two-part series will discuss the principles of critically appraising randomised controlled trials. The second part will discuss the principles of critically appraising systematic reviews and meta-analyses.

Evidence-based dentistry (EBD) is the integration of the dentist's clinical expertise, the patient's needs and preferences and the most current, clinically relevant evidence. Critical appraisal of the literature is an invaluable and indispensable skill that dentists should possess to help them deliver EBD.

This article seeks to act as a refresher and guide for generalists, specialists and the wider readership, so that they can efficiently and confidently appraise research - specifically, randomised controlled trials - that may be pertinent to their daily clinical practice.

Evidence-based dentistry is discussed.

Efficient techniques for critically appraising randomised controlled trials are described.

Important methodological and statistical considerations are explicated.

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

Receive 24 print issues and online access

251,40 € per year

only 10,48 € per issue

Buy this article

• Purchase on SpringerLink
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

How to appraise the literature: basic principles for the busy clinician - part 2: systematic reviews and meta-analyses

Making sense of the literature: an introduction to critical appraisal for the primary care practitioner

Critical appraisal of systematic reviews of intervention in dentistry published between 2019-2020 using the AMSTAR 2 tool

Burls A. What is critical appraisal? 2014. Available at http://www.whatisseries.co.uk/whatiscritical-appraisal/ (accessed April 2021).

Hong B, Plugge E. Critical appraisal skills teaching in UK dental schools. Br Dent J 2017; 222: 209-213.

Isham A, Bettiol S, Hoang H, Crocombe L. A Systematic Literature Review of the Information-Seeking Behaviour of Dentists in Developed Countries. J Dent Educ 2016; 80: 569-577.

Critical Appraisal Skills Programme. CASP Checklist. Available at https://casp-uk.net/wp-content/uploads/2018/01/CASP-Randomised-Controlled-Trial-Checklist-2018.pdf (accessed April 2021).

Schulz K F, Altman D G, Moher D, CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Int Med 2010; 152 : 726-732.

Sterne J A C, Savović J, Page M J et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019; DOI: 10.1136/bmj.l4898.

Petrou S, Grey A. Economic evaluation alongside randomised controlled trials: design, conduct, analysis, and reporting. BMJ 2011; DOI: 10.1136/bmj.d1548.

Black W C. The CE plane: a graphic representation of cost-effectiveness. Med Decis Making 1990; 10: 212-214.

Download references

Author information

Authors and affiliations.

Senior Registrar in Orthodontics, The Royal London Hospital Barts Health NHS Trust and East Kent Hospitals University NHS Foundation Trust, London, UK

Aslam Alkadhimi

Dental Core Trainee, East Kent Hospitals University NHS Foundation Trust, UK

Samuel Reeves

Consultant Orthodontist, East Kent Hospitals University NHS Foundation Trust, UK

Andrew T. DiBiase

You can also search for this author in PubMed   Google Scholar

Contributions

Aslam Alkadhimi contributed to conceptualisation, literature search, original draft preparation and drafting and critically revising the manuscript; Samuel Reeves contributed to original draft preparation and editing; and Andrew DiBiase contributed to supervision, draft editing and critically revising the manuscript.

Corresponding author

Correspondence to Aslam Alkadhimi .

Ethics declarations

The authors declare no competing interests.

Ethical approval and consent to participate did not apply to this study.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Alkadhimi, A., Reeves, S. & DiBiase, A. How to appraise the literature: basic principles for the busy clinician - part 1: randomised controlled trials. Br Dent J 232 , 475–481 (2022). https://doi.org/10.1038/s41415-022-4096-y

Download citation

Received : 31 January 2021

Accepted : 25 April 2021

Published : 08 April 2022

Issue Date : 08 April 2022

DOI : https://doi.org/10.1038/s41415-022-4096-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Student Supports
• About U of M
• Current students
• Faculty and staff
• News and events
• Elizabeth Dafoe, Sciences & Technology, and Neil John Maclean Health Sciences Libraries are open for students, faculty, and staff. Archives is closed.
• Physical materials can be requested for contactless pickup at the pickup lockers at Elizabeth Dafoe Library (Fort Garry Campus) and Neil John Maclean Health Sciences Library (only accessible for Bannatyne campus students, staff, and faculty who have swipe card access).
• Our online information resources are available 24x7. Begin your search here .
• Need help? Use our Ask Us chat service and stay tuned to our update page for the most current libraries' services information.

How to Write a Literature Review

• Step #1: Build a Search Question This link opens in a new window
• How to Search This link opens in a new window
• How to Search PubMed This link opens in a new window
• How to Search in Ovid This link opens in a new window
• How to Search Scopus This link opens in a new window
• How to Search Cinahl This link opens in a new window
• Step #3: Reading & Analyzing
• Step #4: Writing
• Step #5: Citation Management This link opens in a new window
• Tools to support the Lit Review
• Last Updated: Jul 24, 2024 11:51 AM
• URL: https://libguides.lib.umanitoba.ca/literaturereview

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Clin Diagn Res
• v.11(5); 2017 May

Critical Appraisal of Clinical Research

Azzam al-jundi.

1 Professor, Department of Orthodontics, King Saud bin Abdul Aziz University for Health Sciences-College of Dentistry, Riyadh, Kingdom of Saudi Arabia.

Salah Sakka

2 Associate Professor, Department of Oral and Maxillofacial Surgery, Al Farabi Dental College, Riyadh, KSA.

Evidence-based practice is the integration of individual clinical expertise with the best available external clinical evidence from systematic research and patient’s values and expectations into the decision making process for patient care. It is a fundamental skill to be able to identify and appraise the best available evidence in order to integrate it with your own clinical experience and patients values. The aim of this article is to provide a robust and simple process for assessing the credibility of articles and their value to your clinical practice.

Introduction

Decisions related to patient value and care is carefully made following an essential process of integration of the best existing evidence, clinical experience and patient preference. Critical appraisal is the course of action for watchfully and systematically examining research to assess its reliability, value and relevance in order to direct professionals in their vital clinical decision making [ 1 ].

Critical appraisal is essential to:

• Combat information overload;
• Identify papers that are clinically relevant;
• Continuing Professional Development (CPD).

Carrying out Critical Appraisal:

Assessing the research methods used in the study is a prime step in its critical appraisal. This is done using checklists which are specific to the study design.

Standard Common Questions:

• What is the research question?
• What is the study type (design)?
• Selection issues.
• What are the outcome factors and how are they measured?
• What are the study factors and how are they measured?
• What important potential confounders are considered?
• What is the statistical method used in the study?
• Statistical results.
• What conclusions did the authors reach about the research question?
• Are ethical issues considered?

The Critical Appraisal starts by double checking the following main sections:

I. Overview of the paper:

• The publishing journal and the year
• The article title: Does it state key trial objectives?
• The author (s) and their institution (s)

The presence of a peer review process in journal acceptance protocols also adds robustness to the assessment criteria for research papers and hence would indicate a reduced likelihood of publication of poor quality research. Other areas to consider may include authors’ declarations of interest and potential market bias. Attention should be paid to any declared funding or the issue of a research grant, in order to check for a conflict of interest [ 2 ].

II. ABSTRACT: Reading the abstract is a quick way of getting to know the article and its purpose, major procedures and methods, main findings, and conclusions.

• Aim of the study: It should be well and clearly written.
• Materials and Methods: The study design and type of groups, type of randomization process, sample size, gender, age, and procedure rendered to each group and measuring tool(s) should be evidently mentioned.
• Results: The measured variables with their statistical analysis and significance.
• Conclusion: It must clearly answer the question of interest.

III. Introduction/Background section:

An excellent introduction will thoroughly include references to earlier work related to the area under discussion and express the importance and limitations of what is previously acknowledged [ 2 ].

-Why this study is considered necessary? What is the purpose of this study? Was the purpose identified before the study or a chance result revealed as part of ‘data searching?’

-What has been already achieved and how does this study be at variance?

-Does the scientific approach outline the advantages along with possible drawbacks associated with the intervention or observations?

IV. Methods and Materials section : Full details on how the study was actually carried out should be mentioned. Precise information is given on the study design, the population, the sample size and the interventions presented. All measurements approaches should be clearly stated [ 3 ].

V. Results section : This section should clearly reveal what actually occur to the subjects. The results might contain raw data and explain the statistical analysis. These can be shown in related tables, diagrams and graphs.

VI. Discussion section : This section should include an absolute comparison of what is already identified in the topic of interest and the clinical relevance of what has been newly established. A discussion on a possible related limitations and necessitation for further studies should also be indicated.

Does it summarize the main findings of the study and relate them to any deficiencies in the study design or problems in the conduct of the study? (This is called intention to treat analysis).

• Does it address any source of potential bias?
• Are interpretations consistent with the results?
• How are null findings interpreted?
• Does it mention how do the findings of this study relate to previous work in the area?
• Can they be generalized (external validity)?
• Does it mention their clinical implications/applicability?
• What are the results/outcomes/findings applicable to and will they affect a clinical practice?
• Does the conclusion answer the study question?
• -Is the conclusion convincing?
• -Does the paper indicate ethics approval?
• -Can you identify potential ethical issues?
• -Do the results apply to the population in which you are interested?
• -Will you use the results of the study?

Once you have answered the preliminary and key questions and identified the research method used, you can incorporate specific questions related to each method into your appraisal process or checklist.

1-What is the research question?

For a study to gain value, it should address a significant problem within the healthcare and provide new or meaningful results. Useful structure for assessing the problem addressed in the article is the Problem Intervention Comparison Outcome (PICO) method [ 3 ].

P = Patient or problem: Patient/Problem/Population:

It involves identifying if the research has a focused question. What is the chief complaint?

E.g.,: Disease status, previous ailments, current medications etc.,

I = Intervention: Appropriately and clearly stated management strategy e.g.,: new diagnostic test, treatment, adjunctive therapy etc.,

C= Comparison: A suitable control or alternative

E.g.,: specific and limited to one alternative choice.

O= Outcomes: The desired results or patient related consequences have to be identified. e.g.,: eliminating symptoms, improving function, esthetics etc.,

The clinical question determines which study designs are appropriate. There are five broad categories of clinical questions, as shown in [ Table/Fig-1 ].

[Table/Fig-1]:

Categories of clinical questions and the related study designs.

2- What is the study type (design)?

The study design of the research is fundamental to the usefulness of the study.

In a clinical paper the methodology employed to generate the results is fully explained. In general, all questions about the related clinical query, the study design, the subjects and the correlated measures to reduce bias and confounding should be adequately and thoroughly explored and answered.

Participants/Sample Population:

Researchers identify the target population they are interested in. A sample population is therefore taken and results from this sample are then generalized to the target population.

The sample should be representative of the target population from which it came. Knowing the baseline characteristics of the sample population is important because this allows researchers to see how closely the subjects match their own patients [ 4 ].

Sample size calculation (Power calculation): A trial should be large enough to have a high chance of detecting a worthwhile effect if it exists. Statisticians can work out before the trial begins how large the sample size should be in order to have a good chance of detecting a true difference between the intervention and control groups [ 5 ].

• Is the sample defined? Human, Animals (type); what population does it represent?
• Does it mention eligibility criteria with reasons?
• Does it mention where and how the sample were recruited, selected and assessed?
• Does it mention where was the study carried out?
• Is the sample size justified? Rightly calculated? Is it adequate to detect statistical and clinical significant results?
• Does it mention a suitable study design/type?
• Is the study type appropriate to the research question?
• Is the study adequately controlled? Does it mention type of randomization process? Does it mention the presence of control group or explain lack of it?
• Are the samples similar at baseline? Is sample attrition mentioned?
• All studies report the number of participants/specimens at the start of a study, together with details of how many of them completed the study and reasons for incomplete follow up if there is any.
• Does it mention who was blinded? Are the assessors and participants blind to the interventions received?
• Is it mentioned how was the data analysed?
• Are any measurements taken likely to be valid?

Researchers use measuring techniques and instruments that have been shown to be valid and reliable.

Validity refers to the extent to which a test measures what it is supposed to measure.

(the extent to which the value obtained represents the object of interest.)

• -Soundness, effectiveness of the measuring instrument;
• -What does the test measure?
• -Does it measure, what it is supposed to be measured?
• -How well, how accurately does it measure?

Reliability: In research, the term reliability means “repeatability” or “consistency”

Reliability refers to how consistent a test is on repeated measurements. It is important especially if assessments are made on different occasions and or by different examiners. Studies should state the method for assessing the reliability of any measurements taken and what the intra –examiner reliability was [ 6 ].

3-Selection issues:

The following questions should be raised:

• - How were subjects chosen or recruited? If not random, are they representative of the population?
• - Types of Blinding (Masking) Single, Double, Triple?
• - Is there a control group? How was it chosen?
• - How are patients followed up? Who are the dropouts? Why and how many are there?
• - Are the independent (predictor) and dependent (outcome) variables in the study clearly identified, defined, and measured?
• - Is there a statement about sample size issues or statistical power (especially important in negative studies)?
• - If a multicenter study, what quality assurance measures were employed to obtain consistency across sites?
• - Are there selection biases?
• • In a case-control study, if exercise habits to be compared:
• - Are the controls appropriate?
• - Were records of cases and controls reviewed blindly?
• - How were possible selection biases controlled (Prevalence bias, Admission Rate bias, Volunteer bias, Recall bias, Lead Time bias, Detection bias, etc.,)?
• • Cross Sectional Studies:
• - Was the sample selected in an appropriate manner (random, convenience, etc.,)?
• - Were efforts made to ensure a good response rate or to minimize the occurrence of missing data?
• - Were reliability (reproducibility) and validity reported?
• • In an intervention study, how were subjects recruited and assigned to groups?
• • In a cohort study, how many reached final follow-up?
• - Are the subject’s representatives of the population to which the findings are applied?
• - Is there evidence of volunteer bias? Was there adequate follow-up time?
• - What was the drop-out rate?
• - Any shortcoming in the methodology can lead to results that do not reflect the truth. If clinical practice is changed on the basis of these results, patients could be harmed.

Researchers employ a variety of techniques to make the methodology more robust, such as matching, restriction, randomization, and blinding [ 7 ].

Bias is the term used to describe an error at any stage of the study that was not due to chance. Bias leads to results in which there are a systematic deviation from the truth. As bias cannot be measured, researchers need to rely on good research design to minimize bias [ 8 ]. To minimize any bias within a study the sample population should be representative of the population. It is also imperative to consider the sample size in the study and identify if the study is adequately powered to produce statistically significant results, i.e., p-values quoted are <0.05 [ 9 ].

4-What are the outcome factors and how are they measured?

• -Are all relevant outcomes assessed?
• -Is measurement error an important source of bias?

5-What are the study factors and how are they measured?

• -Are all the relevant study factors included in the study?
• -Have the factors been measured using appropriate tools?

Data Analysis and Results:

- Were the tests appropriate for the data?

- Are confidence intervals or p-values given?

• How strong is the association between intervention and outcome?
• How precise is the estimate of the risk?
• Does it clearly mention the main finding(s) and does the data support them?
• Does it mention the clinical significance of the result?
• Is adverse event or lack of it mentioned?
• Are all relevant outcomes assessed?
• Was the sample size adequate to detect a clinically/socially significant result?
• Are the results presented in a way to help in health policy decisions?
• Is there measurement error?
• Is measurement error an important source of bias?

Confounding Factors:

A confounder has a triangular relationship with both the exposure and the outcome. However, it is not on the causal pathway. It makes it appear as if there is a direct relationship between the exposure and the outcome or it might even mask an association that would otherwise have been present [ 9 ].

6- What important potential confounders are considered?

• -Are potential confounders examined and controlled for?
• -Is confounding an important source of bias?

7- What is the statistical method in the study?

• -Are the statistical methods described appropriate to compare participants for primary and secondary outcomes?
• -Are statistical methods specified insufficient detail (If I had access to the raw data, could I reproduce the analysis)?
• -Were the tests appropriate for the data?
• -Are confidence intervals or p-values given?
• -Are results presented as absolute risk reduction as well as relative risk reduction?

Interpretation of p-value:

The p-value refers to the probability that any particular outcome would have arisen by chance. A p-value of less than 1 in 20 (p<0.05) is statistically significant.

• When p-value is less than significance level, which is usually 0.05, we often reject the null hypothesis and the result is considered to be statistically significant. Conversely, when p-value is greater than 0.05, we conclude that the result is not statistically significant and the null hypothesis is accepted.

Confidence interval:

Multiple repetition of the same trial would not yield the exact same results every time. However, on average the results would be within a certain range. A 95% confidence interval means that there is a 95% chance that the true size of effect will lie within this range.

8- Statistical results:

• -Do statistical tests answer the research question?

Are statistical tests performed and comparisons made (data searching)?

Correct statistical analysis of results is crucial to the reliability of the conclusions drawn from the research paper. Depending on the study design and sample selection method employed, observational or inferential statistical analysis may be carried out on the results of the study.

It is important to identify if this is appropriate for the study [ 9 ].

• -Was the sample size adequate to detect a clinically/socially significant result?
• -Are the results presented in a way to help in health policy decisions?

Clinical significance:

Statistical significance as shown by p-value is not the same as clinical significance. Statistical significance judges whether treatment effects are explicable as chance findings, whereas clinical significance assesses whether treatment effects are worthwhile in real life. Small improvements that are statistically significant might not result in any meaningful improvement clinically. The following questions should always be on mind:

• -If the results are statistically significant, do they also have clinical significance?
• -If the results are not statistically significant, was the sample size sufficiently large to detect a meaningful difference or effect?

9- What conclusions did the authors reach about the study question?

Conclusions should ensure that recommendations stated are suitable for the results attained within the capacity of the study. The authors should also concentrate on the limitations in the study and their effects on the outcomes and the proposed suggestions for future studies [ 10 ].

• -Are the questions posed in the study adequately addressed?
• -Are the conclusions justified by the data?
• -Do the authors extrapolate beyond the data?
• -Are shortcomings of the study addressed and constructive suggestions given for future research?
• -Bibliography/References:

Do the citations follow one of the Council of Biological Editors’ (CBE) standard formats?

10- Are ethical issues considered?

If a study involves human subjects, human tissues, or animals, was approval from appropriate institutional or governmental entities obtained? [ 10 , 11 ].

Critical appraisal of RCTs: Factors to look for:

• Allocation (randomization, stratification, confounders).
• Follow up of participants (intention to treat).
• Data collection (bias).
• Sample size (power calculation).
• Presentation of results (clear, precise).
• Applicability to local population.

[ Table/Fig-2 ] summarizes the guidelines for Consolidated Standards of Reporting Trials CONSORT [ 12 ].

[Table/Fig-2]:

Summary of the CONSORT guidelines.

Critical appraisal of systematic reviews: provide an overview of all primary studies on a topic and try to obtain an overall picture of the results.

In a systematic review, all the primary studies identified are critically appraised and only the best ones are selected. A meta-analysis (i.e., a statistical analysis) of the results from selected studies may be included. Factors to look for:

• Literature search (did it include published and unpublished materials as well as non-English language studies? Was personal contact with experts sought?).
• Quality-control of studies included (type of study; scoring system used to rate studies; analysis performed by at least two experts).
• Homogeneity of studies.

[ Table/Fig-3 ] summarizes the guidelines for Preferred Reporting Items for Systematic reviews and Meta-Analyses PRISMA [ 13 ].

[Table/Fig-3]:

Summary of PRISMA guidelines.

Critical appraisal is a fundamental skill in modern practice for assessing the value of clinical researches and providing an indication of their relevance to the profession. It is a skills-set developed throughout a professional career that facilitates this and, through integration with clinical experience and patient preference, permits the practice of evidence based medicine and dentistry. By following a systematic approach, such evidence can be considered and applied to clinical practice.

Financial or other Competing Interests

Please enter both an email address and a password.

Account login

• Show/Hide Password Show password Hide password
• Reset Password

Need to reset your password?  Enter the email address which you used to register on this site (or your membership/contact number) and we'll email you a link to reset it. You must complete the process within 2hrs of receiving the link.

We've sent you an email.

An email has been sent to Simply follow the link provided in the email to reset your password. If you can't find the email please check your junk or spam folder and add [email protected] to your address book.

• About RCS England

• Dissecting the literature: the importance of critical appraisal

08 Dec 2017

Kirsty Morrison

This post was updated  in 2023.

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context.

Amanda Burls, What is Critical Appraisal?

Why is critical appraisal needed?

Literature searches using databases like Medline or EMBASE often result in an overwhelming volume of results which can vary in quality. Similarly, those who browse medical literature for the purposes of CPD or in response to a clinical query will know that there are vast amounts of content available. Critical appraisal helps to reduce the burden and allow you to focus on articles that are relevant to the research question, and that can reliably support or refute its claims with high-quality evidence, or identify high-level research relevant to your practice.

Critical appraisal allows us to:

• reduce information overload by eliminating irrelevant or weak studies
• identify the most relevant papers
• distinguish evidence from opinion, assumptions, misreporting, and belief
• assess the validity of the study
• assess the usefulness and clinical applicability of the study
• recognise any potential for bias.

Critical appraisal helps to separate what is significant from what is not. One way we use critical appraisal in the Library is to prioritise the most clinically relevant content for our Current Awareness Updates .

How to critically appraise a paper

There are some general rules to help you, including a range of checklists highlighted at the end of this blog. Some key questions to consider when critically appraising a paper:

• Is the study question relevant to my field?
• Does the study add anything new to the evidence in my field?
• What type of research question is being asked? A well-developed research question usually identifies three components: the group or population of patients, the studied parameter (e.g. a therapy or clinical intervention) and outcomes of interest.
• Was the study design appropriate for the research question? You can learn more about different study types and the hierarchy of evidence here .
• Did the methodology address important potential sources of bias? Bias can be attributed to chance (e.g. random error) or to the study methods (systematic bias).
• Was the study performed according to the original protocol? Deviations from the planned protocol can affect the validity or relevance of a study, e.g. a decrease in the studied population over the course of a randomised controlled trial .
• Does the study test a stated hypothesis? Is there a clear statement of what the investigators expect the study to find which can be tested, and confirmed or refuted.
• Were the statistical analyses performed correctly? The approach to dealing with missing data, and the statistical techniques that have been applied should be specified. Original data should be presented clearly so that readers can check the statistical accuracy of the paper.
• Do the data justify the conclusions? Watch out for definite conclusions based on statistically insignificant results, generalised findings from a small sample size, and statistically significant associations being misinterpreted to imply a cause and effect.
• Are there any conflicts of interest? Who has funded the study and can we trust their objectivity? Do the authors have any potential conflicts of interest, and have these been declared?

And an important consideration for surgeons:

• Will the results help me manage my patients?

At the end of the appraisal process you should have a better appreciation of how strong the evidence is, and ultimately whether or not you should apply it to your patients.

Further resources:

• How to Read a Paper by Trisha Greenhalgh
• The Doctor’s Guide to Critical Appraisal by Narinder Kaur Gosall
• CASP checklists
• CEBM Critical Appraisal Tools
• Critical Appraisal: a checklist
• Critical Appraisal of a Journal Article (PDF)
• Introduction to...Critical appraisal of literature
• Reporting guidelines for the main study types

Kirsty Morrison, Information Specialist

Share this page:

• Library Blog

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Critical appraisal of published literature

Affiliations.

• 1 Department of Anaesthesia, Dharwad Institute of Mental Health and Neuro Sciences, Dharwad, Karnataka, India.
• 2 Department of Anaesthesiology, Universal Hospital, Abu Dhabi, UAE.
• 3 Department of Pulmonary Medicine, Kasturba Medical College, Manipal University, Manipal, Karnataka, India.
• PMID: 27729695
• PMCID: PMC5037949
• DOI: 10.4103/0019-5049.190624

With a large output of medical literature coming out every year, it is impossible for readers to read every article. Critical appraisal of scientific literature is an important skill to be mastered not only by academic medical professionals but also by those involved in clinical practice. Before incorporating changes into the management of their patients, a thorough evaluation of the current or published literature is an important step in clinical practice. It is necessary for assessing the published literature for its scientific validity and generalizability to the specific patient community and reader's work environment. Simple steps have been provided by Consolidated Standard for Reporting Trial statements, Scottish Intercollegiate Guidelines Network and several other resources which if implemented may help the reader to avoid reading flawed literature and prevent the incorporation of biased or untrustworthy information into our practice.

Keywords: Allocation concealment; bias; conflict of interest; critical appraisal; randomisation; study design.

PubMed Disclaimer

Similar articles

• Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas. Crider K, Williams J, Qi YP, Gutman J, Yeung L, Mai C, Finkelstain J, Mehta S, Pons-Duran C, Menéndez C, Moraleda C, Rogers L, Daniels K, Green P. Crider K, et al. Cochrane Database Syst Rev. 2022 Feb 1;2(2022):CD014217. doi: 10.1002/14651858.CD014217. Cochrane Database Syst Rev. 2022. PMID: 36321557 Free PMC article.
• The future of Cochrane Neonatal. Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
• A critical appraisal of epidemiological studies comes from basic knowledge: a reader's guide to assess potential for biases. Boccia S, La Torre G, Persiani R, D'Ugo D, van Duijn CM, Ricciardi G. Boccia S, et al. World J Emerg Surg. 2007 Mar 15;2:7. doi: 10.1186/1749-7922-2-7. World J Emerg Surg. 2007. PMID: 17359550 Free PMC article.
• Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, Dias S, Schulz KF, Plint AC, Moher D. Turner L, et al. Cochrane Database Syst Rev. 2012 Nov 14;11(11):MR000030. doi: 10.1002/14651858.MR000030.pub2. Cochrane Database Syst Rev. 2012. PMID: 23152285 Free PMC article. Review.
• Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Philips Z, Ginnelly L, Sculpher M, Claxton K, Golder S, Riemsma R, Woolacoot N, Glanville J. Philips Z, et al. Health Technol Assess. 2004 Sep;8(36):iii-iv, ix-xi, 1-158. doi: 10.3310/hta8360. Health Technol Assess. 2004. PMID: 15361314 Review.
• Levels of Evidence, Quality Assessment, and Risk of Bias: Evaluating the Internal Validity of Primary Research. Sargeant JM, Brennan ML, O'Connor AM. Sargeant JM, et al. Front Vet Sci. 2022 Jul 12;9:960957. doi: 10.3389/fvets.2022.960957. eCollection 2022. Front Vet Sci. 2022. PMID: 35903128 Free PMC article. Review.
• Critical appraisal of published research papers - A reinforcing tool for research methodology: Questionnaire-based study. Gajbhiye S, Tripathi R, Parmar U, Khatri N, Potey A. Gajbhiye S, et al. Perspect Clin Res. 2021 Apr-Jun;12(2):100-105. doi: 10.4103/picr.PICR_107_18. Epub 2019 May 14. Perspect Clin Res. 2021. PMID: 34012907 Free PMC article.
• Burls A. (What is...? Series) 2nd ed. Oxford, UK: University of Oxford; 2009. [Last accessed on 2016 Jun 19]. What is Critical Appraisal? Available from: http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/what_is_c... .
• Glasziou PP. Information overload: What's behind it, what's beyond it? Med J Aust. 2008;189:84–5. - PubMed
• Strite SA, Stuart ME. Getting started with critical appraisal of medical literature: It is easier than you think. Calif Pharm. 2009;LVI:52–5.
• Forder PM, Gebski VJ, Keech AC. Allocation concealment and blinding: When ignorance is bliss. Med J Aust. 2005;182:87–9. - PubMed
• Bialocerkowski A, Klupp N, Bragge P. How to read and critically appraise a reliability article. Int J Ther Rehabil. 2010;17:114–20.

Publication types

• Search in MeSH

LinkOut - more resources

Full text sources.

• Europe PubMed Central
• Medknow Publications and Media Pvt Ltd
• PubMed Central

Other Literature Sources

• scite Smart Citations
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Pardon Our Interruption

As you were browsing something about your browser made us think you were a bot. There are a few reasons this might happen:

• You've disabled JavaScript in your web browser.
• You're a power user moving through this website with super-human speed.
• You've disabled cookies in your web browser.
• A third-party browser plugin, such as Ghostery or NoScript, is preventing JavaScript from running. Additional information is available in this support article .

To regain access, please make sure that cookies and JavaScript are enabled before reloading the page.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Integrating circular economy principles into agri-food supply chain management: a systematic literature review.

1. Introduction

2. theoretical background, 3. materials and methods, 3.1. systematic literature review process, 3.2. literature search, 3.3. screening and filtering.

• Title and Abstract Screening: ChatGPT-4 reviewed the titles and abstracts of the initial 1200 articles to exclude studies that were clearly irrelevant to the topic. This step reduced the pool to 900 articles.
• Full-Text Review: The remaining 900 articles underwent a full-text review, assisted by ChatGPT-4, which highlighted key sections relevant to our thematic issues. This automated highlighting helped streamline the identification of pertinent studies, reducing the pool to 400 articles.
• Quality Assessment: Using the CASP (Critical Appraisal Skills Programme) checklists, ChatGPT-4 assisted in evaluating the methodological quality of the 400 articles. This step ensured that only high and medium-quality studies were included, resulting in a final selection of 100 articles.

3.4. Data Extraction and Synthesis

3.5. ethical considerations in using chatgpt-4, 3.6. systematic review process, 3.7. selection for core literatures.

• Citation Impact:

4.1. Temporal Distribution of Reviewed Papers

4.2. classification by paper stance, 4.3. thematic issues in ce research, 4.4. geographical distribution of research, 4.5. classification by supply chain stage, 4.6. core literatures of circular economy in supply chain management, 4.6.1. the foundational framework of circular economy, 4.6.2. defining and refining circular economy concepts, 4.6.3. integrating environmental and economic systems, 4.6.4. addressing the limitations and challenges of circular economy, 4.6.5. evolving towards resource-efficient supply chains, 5. discussion, 6. conclusions, author contributions, data availability statement, conflicts of interest.

• Geissdoerfer, M.; Savaget, P.; Bocken, N.M.P.; Hultink, E.J. The Circular Economy—A new sustainability paradigm? J. Clean. Prod. 2017 , 143 , 757–768. [ Google Scholar ] [ CrossRef ]
• Completing the Picture: How the Circular Economy Tackles Climate Change. 2019. Available online: https://www.ellenmacarthurfoundation.org/completing-the-picture (accessed on 14 June 2024).
• Korhonen, J.; Honkasalo, A.; Seppälä, J. Circular Economy: The Concept and its Limitations. Ecol. Econ. 2018 , 143 , 37–46. [ Google Scholar ] [ CrossRef ]
• Moving Forward on Food Loss and Waste Reduction. State of Food and Agriculture. 2019. Available online: https://www.fao.org/3/ca6030en/ca6030en.pdf (accessed on 14 June 2024).
• Springmann, M.; Clark, M.; Mason-D’Croz, D.; Wiebe, K.; Bodirsky, B.L.; Lassaletta, L.; de Vries, W.; Vermeulen, S.J.; Herrero, M.; Carlson, K.M.; et al. Options for keeping the food system within environmental limits. Nature 2018 , 562 , 519–525. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Gustavsson, J.; Cederberg, C.; Sonesson, U.; van Otterdijck, R.; Meybeck, A. Extent of food losses and waste. In Global Food Losses and Food Waste ; FAO: Rome, Italy, 2011; pp. 4–9. [ Google Scholar ]
• Kirchherr, J.; Reike, D.; Hekkert, M. Conceptualizing the circular economy: An analysis of 114 definitions. Resour. Conserv. Recycl. 2017 , 127 , 221–232. [ Google Scholar ] [ CrossRef ]
• Esposito, M.; Tse, T.; Soufani, K. Introducing a Circular Economy: New Thinking with New Managerial and Policy Implications. Calif. Manag. Rev. 2018 , 60 , 5–19. [ Google Scholar ] [ CrossRef ]
• de Jesus, A.; Mendonça, S. Lost in Transition? Drivers and Barriers in the Eco-innovation Road to the Circular Economy. Ecol. Econ. 2018 , 145 , 75–89. [ Google Scholar ] [ CrossRef ]
• Prieto-Sandoval, V.; Jaca, C.; Ormazabal, M. Towards a consensus on the circular economy. J. Clean. Prod. 2018 , 179 , 605–615. [ Google Scholar ] [ CrossRef ]
• Venkata Mohan, S.; Nikhil, G.N.; Chiranjeevi, P.; Nagendranatha Reddy, C.; Rohit, M.V.; Kumar, A.N.; Sarkar, O. Waste biorefinery models towards sustainable circular bioeconomy: Critical review and future perspectives. Bioresour. Technol. 2016 , 215 , 2–12. [ Google Scholar ] [ CrossRef ]
• Ghisellini, P.; Cialani, C.; Ulgiati, S. A review on circular economy: The expected transition to a balanced interplay of environmental and economic systems. J. Clean. Prod. 2016 , 114 , 11–32. [ Google Scholar ] [ CrossRef ]
• Onyango, C.M.; Nyaga, J.M.; Wetterlind, J.; Söderström, M.; Piikki, K. Precision Agriculture for Resource Use Efficiency in Smallholder Farming Systems in Sub-Saharan Africa: A Systematic Review. Sustainability 2021 , 13 , 1158. [ Google Scholar ] [ CrossRef ]
• Matopoulos, A.; Barros, A.C.; van der Vorst, J.G.A.J. Resource-efficient supply chains: A research framework, literature review and research agenda. Supply Chain Manag. 2015 , 20 , 218–236. [ Google Scholar ] [ CrossRef ]
• Kirchherr, J.; Piscicelli, L.; Bour, R.; Kostense-Smit, E.; Muller, J.; Huibrechtse-Truijens, A.; Hekkert, M. Barriers to the Circular Economy: Evidence From the European Union (EU). Ecol. Econ. 2018 , 150 , 264–272. [ Google Scholar ] [ CrossRef ]
• Reike, D.; Vermeulen, W.J.V.; Witjes, S. The circular economy: New or Refurbished as CE 3.0?—Exploring Controversies in the Conceptualization of the Circular Economy through a Focus on History and Resource Value Retention Options. Resour. Conserv. Recycl. 2018 , 135 , 246–264. [ Google Scholar ] [ CrossRef ]
• Hamam, M.; Chinnici, G.; Di Vita, G.; Pappalardo, G.; Pecorino, B.; Maesano, G.; D’Amico, M. Circular economy models in agro-food systems: A review. Sustainability 2021 , 13 , 3453. [ Google Scholar ] [ CrossRef ]
• Moher, D.; Liberati, A.; Tetzlaff, J.; Altman, D.G.; Antes, G.; Atkins, D.; Barbour, V.; Barrowman, N.; Berlin, J.A.; Clark, J.; et al. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med. 2009 , 6 , e1000097. [ Google Scholar ] [ CrossRef ]
• Gorokhova, T.; Shpatakova, O.; Toponar, O.; Zolotarova, O.; Pavliuk, S. Circular Economy As an Alternative to the Traditional Linear Economy: Case Study of the EU. Rev. Gest. Soc. Ambient. 2023 , 17 , e03385. [ Google Scholar ] [ CrossRef ]
• Lambert, D.M.; Cooper, M.C. Issues in Supply Chain Management. Ind. Mark. Manag. 2000 , 29 , 65–83. [ Google Scholar ] [ CrossRef ]
• Christopher, M. Innovating the Global Supply Chain. In Logistics & Supply Chain Management ; Prentice Hall: London, UK, 2011; Available online: https://books.google.com/books/about/Logistics_Supply_Chain_Management.html?hl=es&id=2dsYQwAACAAJ (accessed on 15 June 2024).
• Nasir, M.H.A.; Genovese, A.; Acquaye, A.A.; Koh, S.C.L.; Yamoah, F. Comparing linear and circular supply chains: A case study from the construction industry. Int. J. Prod. Econ. 2017 , 183 , 443–457. [ Google Scholar ] [ CrossRef ]
• Hobbs, J.E. Food supply chains during the COVID-19 pandemic. Can. J. Agric. Econ. Can. D’agroecon. 2020 , 68 , 171–176. [ Google Scholar ] [ CrossRef ]
• Haryanti, N.; Zueni, A. Identifikasi mutu fisik, kimia dan organoleptik es krim daging kulit manggis ( Garcinia mangostana L.) dengan variasi susu krim. AGRITEPA J. Ilmu Dan Teknol. Pertan. 2015 , 2 , 143–156. [ Google Scholar ] [ CrossRef ]
• Pomponi, F.; Moncaster, A. Circular economy for the built environment: A research framework. J. Clean. Prod. 2017 , 143 , 710–718. [ Google Scholar ] [ CrossRef ]
• de la Torre-López, J.; Ramírez, A.; Romero, J.R. Artificial intelligence to automate the systematic review of scientific literature. Computing 2023 , 105 , 2171–2194. Available online: https://link.springer.com/article/10.1007/s00607-023-01181-x (accessed on 15 June 2024). [ CrossRef ]
• Zeb, A.; Ullah, R.; Karim, R. Exploring the role of ChatGPT in higher education: Opportunities, challenges and ethical considerations. Int. J. Inf. Learn. Technol. 2024 , 41 , 99–111. [ Google Scholar ] [ CrossRef ]
• O’Dea, R.E.; Lagisz, M.; Jennions, M.D.; Koricheva, J.; Noble, D.W.A.; Parker, T.H.; Gurevitch, J.; Page, M.J.; Stewart, G.; Moher, D.; et al. Preferred reporting items for systematic reviews and meta-analyses in ecology and evolutionary biology: A PRISMA extension. Biol. Rev. 2021 , 96 , 1695–1722. [ Google Scholar ] [ CrossRef ]
• Luhnen, M.; Prediger, B.; Neugebauer, E.A.M.; Mathes, T. Systematic reviews of health economic evaluations: A structured analysis of characteristics and methods applied. Res. Synth. Methods 2019 , 10 , 195–206. [ Google Scholar ] [ CrossRef ]
• Oláh, J.; Krisán, E.; Kiss, A.; Lakner, Z.; Popp, J. PRISMA Statement for Reporting Literature Searches in Systematic Reviews of the Bioethanol Sector. Energies 2020 , 13 , 2323. [ Google Scholar ] [ CrossRef ]
• Zhu, Q.; Sarkis, J.; Lai, K.-h. Initiatives and outcomes of green supply chain management implementation by Chinese manufacturers. J. Environ. Manag. 2007 , 85 , 179–189. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Blomsma, F.; Brennan, G. The Emergence of Circular Economy: A New Framing around Prolonging Resource Productivity. J. Ind. Ecol. 2017 , 21 , 603–614. [ Google Scholar ] [ CrossRef ]
• Bocken, N.M.P.; de Pauw, I.; Bakker, C.; van der Grinten, B. Product design and business model strategies for a circular economy. J. Ind. Prod. Eng. 2016 , 33 , 308–320. [ Google Scholar ] [ CrossRef ]
• Phiri, R.; Mavinkere Rangappa, S.; Siengchin, S. Agro-waste for renewable and sustainable green production: A review. J. Clean. Prod. 2024 , 434 , 139989. [ Google Scholar ] [ CrossRef ]
• Ahmad, S.; Wong, K.Y.; Tseng, M.L.; Wong, W.P. Sustainable product design and development: A review of tools, applications and research prospects. Resour. Conserv. Recycl. 2018 , 132 , 49–61. [ Google Scholar ] [ CrossRef ]
• Garcia-Garcia, G.; Stone, J.; Rahimifard, S. Opportunities for waste valorisation in the food industry—A case study with four UK food manufacturers. J. Clean. Prod. 2019 , 211 , 1339–1356. [ Google Scholar ] [ CrossRef ]
• Aibar-Guzmán, B.; García-Sánchez, I.M.; Aibar-Guzmán, C.; Hussain, N. Sustainable product innovation in agri-food industry: Do ownership structure and capital structure matter? J. Innov. Knowl. 2022 , 7 , 100160. [ Google Scholar ] [ CrossRef ]
• Khan, F.; Ali, Y. Moving towards a sustainable circular bio-economy in the agriculture sector of a developing country. Ecol. Econ. 2022 , 196 , 107402. [ Google Scholar ] [ CrossRef ]
• Abbasi, I.A.; Shamim, A.; Shad, M.K.; Ashari, H.; Yusuf, I. Circular economy-based integrated farming system for indigenous chicken: Fostering food security and sustainability. J. Clean. Prod. 2024 , 436 , 140368. [ Google Scholar ] [ CrossRef ]
• Qian, J.; Yu, Q.; Jiang, L.; Yang, H.; Wu, W. Food cold chain management improvement: A conjoint analysis on COVID-19 and food cold chain systems. Food Control 2022 , 137 , 108940. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• OECD. Green Growth Studies Improving Energy Efficiency in the Agro-Food Chain ; OECD Publishing: Paris, France, 2017. [ Google Scholar ]
• Ur Rahman, S.; Chwialkowska, A.; Hussain, N.; Akbar Bhatti, W.; Luomala, H. Cross-cultural perspective on sustainable consumption: Implications for consumer motivations and promotion. Environ. Dev. Sustain. 2021 , 25 , 997–1016. Available online: https://link.springer.com/article/10.1007/s10668-021-02059-8 (accessed on 23 July 2024). [ CrossRef ]
• Nunes, C.; Rocha, D.; Riesenegger, L.; Hübner, A. Reducing Food Waste at Retail Stores—An Explorative Study. Sustainability 2022 , 14 , 2494. [ Google Scholar ] [ CrossRef ]
• Kresnanto, N.C.; Putri, W.H.; Lantarsih, R.; Harjiyatni, F.R. Efficient agri food supply chain in a sustainable transportation perspective. IOP Conf. Ser. Earth Environ. Sci. 2021 , 892 , 012105. [ Google Scholar ] [ CrossRef ]
• Babagolzadeh, M.; Shrestha, A.; Abbasi, B.; Zhang, Y.; Woodhead, A.; Zhang, A. Sustainable cold supply chain management under demand uncertainty and carbon tax regulation. Transp. Res. Part D Transp. Environ. 2020 , 80 , 102245. [ Google Scholar ] [ CrossRef ]
• Durkin, A.; Guo, M.; Wuertz, S.; Stuckey, D.C. Resource recovery from food-processing wastewaters in a circular economy: A methodology for the future. Curr. Opin. Biotechnol. 2022 , 76 , 102735. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ojha, S.; Bußler, S.; Schlüter, O.K. Food waste valorisation and circular economy concepts in insect production and processing. Waste Manag. 2020 , 118 , 600–609. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Sharma, P.; Gaur, V.K.; Sirohi, R.; Varjani, S.; Kim, S.H.; Wong, J.W.C. Sustainable processing of food waste for production of bio-based products for circular bioeconomy. Bioresour. Technol. 2021 , 325 , 124684. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Jinil Persis, D.; Venkatesh, V.G.; Raja Sreedharan, V.; Shi, Y.; Sankaranarayanan, B. Modelling and analysing the impact of Circular Economy; Internet of Things and ethical business practices in the VUCA world: Evidence from the food processing industry. J. Clean. Prod. 2021 , 301 , 126871. [ Google Scholar ] [ CrossRef ]
• Yang, Y.; Barnes, H.; Yang, B.; Onofrei, G.; Nguyen, H. Food waste management for the UK grocery retail sector—A supply chain collaboration perspective. Prod. Plan. Control 2023 , 1–14. [ Google Scholar ] [ CrossRef ]
• Gallardo, A.; Bovea, M.D.; Colomer, F.J.; Prades, M. Analysis of collection systems for sorted household waste in Spain. Waste Manag. 2012 , 32 , 1623–1633. [ Google Scholar ] [ CrossRef ]
• Jimenez-Fernandez, A.; Aramendia-Muneta, M.E.; Alzate, M. Consumers’ awareness and attitudes in circular fashion. Clean. Responsible Consum. 2023 , 11 , 100144. [ Google Scholar ] [ CrossRef ]
• Ormazabal, M.; Prieto-Sandoval, V.; Puga-Leal, R.; Jaca, C. Circular Economy in Spanish SMEs: Challenges and opportunities. J. Clean. Prod. 2018 , 185 , 157–167. [ Google Scholar ] [ CrossRef ]
• Correani, L.; Morganti, P.; Silvestri, C.; Ruggieri, A. Food waste, circular economy, and policy with oligopolistic retailers. J. Clean. Prod. 2023 , 407 , 137092. [ Google Scholar ] [ CrossRef ]
• Beulque, R.; Micheaux, H.; Ntsondé, J.; Aggeri, F.; Steux, C. Sufficiency-based circular business models: An established retailers’ perspective. J. Clean. Prod. 2023 , 429 , 139431. [ Google Scholar ] [ CrossRef ]
• Liu, W.; Oosterveer, P.; Spaargaren, G. Promoting sustainable consumption in China: A conceptual framework and research review. J. Clean. Prod. 2016 , 134 , 13–21. [ Google Scholar ] [ CrossRef ]
• Tiboldo, G.; Arata, L.; Coderoni, S. Back to the future: Are consumers ready to eat insect-fed poultry food products from a circular farming system? An assessment for Italy. Future Foods 2024 , 9 , 100290. [ Google Scholar ] [ CrossRef ]
• Cela, N.; Giorgione, V.; Fassio, F.; Torri, L. Impact of circular economy information on sensory acceptability, purchase intention and perceived value of upcycled foods by young consumers. Food Res. Int. 2024 , 175 , 113765. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Paparella, A.; Vecchio, R.; Cembalo, L.; Lombardi, A. Measuring consumer effort in circular economy initiatives in the food domain: An exploratory analysis. Heliyon 2023 , 9 , e13373. [ Google Scholar ] [ CrossRef ] [ PubMed ]

Click here to enlarge figure

Share and Cite

Cahyadi, E.R.; Hidayati, N.; Zahra, N.; Arif, C. Integrating Circular Economy Principles into Agri-Food Supply Chain Management: A Systematic Literature Review. Sustainability 2024 , 16 , 7165. https://doi.org/10.3390/su16167165

Cahyadi ER, Hidayati N, Zahra N, Arif C. Integrating Circular Economy Principles into Agri-Food Supply Chain Management: A Systematic Literature Review. Sustainability . 2024; 16(16):7165. https://doi.org/10.3390/su16167165

Cahyadi, Eko Ruddy, Nurul Hidayati, Nisa Zahra, and Chusnul Arif. 2024. "Integrating Circular Economy Principles into Agri-Food Supply Chain Management: A Systematic Literature Review" Sustainability 16, no. 16: 7165. https://doi.org/10.3390/su16167165

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals