critical appraisal research

• CASP Checklists
• How to use our CASP Checklists
• Referencing and Creative Commons
• Online Training Courses
• CASP Workshops
• What is Critical Appraisal
• Study Designs
• Useful Links
• Bibliography
• View all Tools and Resources
• Testimonials

What is Critical Appraisal?

Critical Appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context. It is an essential skill for evidence-based medicine because it allows people to find and use research evidence reliably and efficiently. All of us would like to enjoy the best possible health we can. To achieve this, we need reliable information about what might harm or help us when we make healthcare decisions.

Why is Critical Appraisal important?

Critical appraisal skills are important as they enable you to assess systematically the trustworthiness, relevance and results of published papers. Where an article is published, or who wrote it should not be an indication of its trustworthiness and relevance.

Randomised Controlled Trials (RCTs): An experiment that randomises participants into two groups: one that receives the treatment and another that serves as the control. RCTs are often used in healthcare to test the efficacy of different treatments.

Learn more about how to critically appraise an RCT.

Systematic Reviews : A thorough and structured analysis of all relevant studies on a particular research question. These are often used in evidence-based practice to evaluate the effects of health and social interventions.

Discover what systematic reviews are, and why they are important .

Cohort Studies : This is an observational study where two or more groups (cohorts) of individuals are followed over time and their outcomes are compared. It's used often in medical research to investigate the potential causes of disease.

Learn more about cohort studies .

Case-Control Studies : This is an observational study where two groups differing in outcome are identified and compared on the basis of some supposed causal attribute. These are often used in epidemiological research.

Check out this article to better understand what a case-control study is in research .

Cross-Sectional Studies : An observational study that examines the relationship between health outcomes and other variables of interest in a defined population at a single point in time. They're useful for determining prevalence and risk factors.

Discover what a cross-sectonal study is and when to use one .

Qualitative Research : An in-depth analysis of a phenomenon based on unstructured data, such as interviews, observations, or written material. It's often used to gain insights into behaviours, value systems, attitudes, motivations, or culture.

This guide will help you increase your knowledge of qualitative research .

Economic Evaluation : A comparison of two or more alternatives in terms of their costs and consequences. Often used in healthcare decision making to maximise efficiency and equity.

Diagnostic Studies : Evaluates the performance of a diagnostic test in predicting the presence or absence of a disease. It is commonly used to validate the accuracy and utility of a new diagnostic procedure.

Case Series : Describes characteristics of a group of patients with a particular disease or who have undergone a specific procedure. Used in clinical medicine to present preliminary observations.

Case Studies : Detailed examination of a single individual or group. Common in psychology and social sciences, this can provide in-depth understanding of complex phenomena in their real-life context.

Aren’t we already doing it?

To some extent, the answer to this question is “yes”. Evidence-based journals can give us reliable, relevant summaries of recent research; guidelines, protocols, and pathways can synthesise the best evidence and present it in the context of a clinical problem. However, we still need to be able to assess research quality to be able to adapt what we read to what we do.

There are still significant gaps in access to evidence.

The main issues we need to address are:

Health and Social Care provision must be based on sound decisions.

In order to make well-informed and sensible choices, we need evidence that is rigorous in methodology and robust in findings.

What types of questions does a critical appraisal encourage you to ask?

• What is the main objective of the research?
• Who conducted the research and are they reputable?
• How was the research funded? Are there any potential conflicts of interest?
• How was the study designed?
• Was the sample size large enough to provide accurate results?
• Were the participants or subjects selected appropriately?
• What data collection methods were used and were they reliable and valid?
• Was the data analysed accurately and rigorously?
• Were the results and conclusions drawn directly from the data or were there assumptions made?
• Can the findings be generalised to the broader population?
• How does this research contribute to existing knowledge in this field?
• Were ethical standards maintained throughout the study?
• Were any potential biases accounted for in the design, data collection or data analysis?
• Have the researchers made suggestions for future research based on their findings?
• Are the findings of the research replicable?
• Are there any implications for policy or practice based on the research findings?
• Were all aspects of the research clearly explained and detailed?

How do you critically appraise a paper?

Critically appraising a paper involves examining the quality, validity, and relevance of a published work to identify its strengths and weaknesses.

This allows the reader to judge its trustworthiness and applicability to their area of work or research. Below are general steps for critically appraising a paper:

Decide how trustworthy a piece of research is (Validity)

• Determine what the research is telling us (Results)
• Weigh up how useful the research will be in your context (Relevance)

You need to understand the research question, do a methodology evaluation, analyse the results, check the conclusion and review the implications and limitations.

That's just a quick summary but we provide a range of in-depth  training courses  and  workshops  to help you improve your knowledge around how to successfully perform critical appraisals so book onto one today or contact us for more information.

Is Critical Appraisal In Research Different To Front-Line Usage In Nursing, Etc?

Critical appraisal in research is different from front-line usage in nursing.

Critical appraisal in research involves a careful analysis of a study's methodology, results, and conclusions to assess the quality and validity of the study. This helps researchers to determine if the study's findings are robust, reliable and applicable in their own research context. It requires a specific set of skills including understanding of research methodology, statistics, and evidence-based practices.

Front-line usage in nursing refers to the direct application of evidence-based practice and research findings in patient care settings. Nurses need to appraise the evidence critically too but their focus is on the direct implications of the research on patient care and health outcomes. The skills required here would be the ability to understand the clinical implications of research findings, communicate these effectively to patients, and incorporate these into their practice.

Both require critical appraisal but the purpose, context, and skills involved are different. Critical appraisal in research is more about evaluating research for validity and reliability whereas front-line usage in nursing is about effectively applying valid and reliable research findings to improve patient care.

How do you know if you're performing critical appraisals correctly?

Thorough Understanding : You've thoroughly read and understood the research, its aims, methodology, and conclusions. You should also be aware of the limitations or potential bias in the research.

Using a Framework or Checklist : Various frameworks exist for critically appraising research (including CASP’s own!). Using these can provide structure and make sure all key points are considered. By keeping a record of your appraisal you will be able to show your reasoning behind whether you’ve implemented a decision based on research.

Identifying Research Methods : Recognising the research design, methods used, sample size, and how data was collected and analysed are crucial in assessing the research's validity and reliability.

Checking Results and Conclusions : Check if the conclusions drawn from the research are justified by the results and data provided, and if any biases could have influenced these conclusions.

Relevance and applicability : Determine if the research's results and conclusions can be applied to other situations, particularly those relevant to your context or question.

Updating Skills : Continually updating your skills in research methods and statistical analysis will improve your confidence and ability in critically appraising research.

Finally, getting feedback from colleagues or mentors on your critical appraisals can also provide a good check on how well you're doing. They can provide an additional perspective and catch anything you might have missed. If possible, we would always recommend doing appraisals in small groups or pairs, working together is always helpful for another perspective, or if you can – join and take part in a journal club.

Ready to Learn more?

Critical Appraisal Training Courses

Critical Appraisal Workshops

• CASP Checklist

Need more information?

• Online Learning
• Privacy Policy

Critical Appraisal Skills Programme

Critical Appraisal Skills Programme (CASP) will use the information you provide on this form to be in touch with you and to provide updates and marketing. Please let us know all the ways you would like to hear from us:

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.

Copyright 2024 CASP UK - OAP Ltd. All rights reserved Website by Beyond Your Brand

• Mayo Clinic Libraries
• Systematic Reviews
• Critical Appraisal by Study Design

Systematic Reviews: Critical Appraisal by Study Design

• Knowledge Synthesis Comparison
• Knowledge Synthesis Decision Tree
• Standards & Reporting Results
• Materials in the Mayo Clinic Libraries
• Training Resources
• Review Teams
• Develop & Refine Your Research Question
• Develop a Timeline
• Project Management
• Communication
• PRISMA-P Checklist
• Eligibility Criteria
• Register your Protocol
• Other Resources
• Other Screening Tools
• Grey Literature Searching
• Citation Searching
• Data Extraction Tools
• Minimize Bias
• Synthesis & Meta-Analysis
• Publishing your Systematic Review

Tools for Critical Appraisal of Studies

“The purpose of critical appraisal is to determine the scientific merit of a research report and its applicability to clinical decision making.” 1 Conducting a critical appraisal of a study is imperative to any well executed evidence review, but the process can be time consuming and difficult. 2 The critical appraisal process requires “a methodological approach coupled with the right tools and skills to match these methods is essential for finding meaningful results.” 3 In short, it is a method of differentiating good research from bad research.

Critical Appraisal by Study Design (featured tools)

• Non-RCTs or Observational Studies
• Diagnostic Accuracy
• Animal Studies
• Qualitative Research
• Tool Repository
• AMSTAR 2 The original AMSTAR was developed to assess the risk of bias in systematic reviews that included only randomized controlled trials. AMSTAR 2 was published in 2017 and allows researchers to “identify high quality systematic reviews, including those based on non-randomised studies of healthcare interventions.” 4 more... less... AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews)
• ROBIS ROBIS is a tool designed specifically to assess the risk of bias in systematic reviews. “The tool is completed in three phases: (1) assess relevance(optional), (2) identify concerns with the review process, and (3) judge risk of bias in the review. Signaling questions are included to help assess specific concerns about potential biases with the review.” 5 more... less... ROBIS (Risk of Bias in Systematic Reviews)
• BMJ Framework for Assessing Systematic Reviews This framework provides a checklist that is used to evaluate the quality of a systematic review.
• CASP Checklist for Systematic Reviews This CASP checklist is not a scoring system, but rather a method of appraising systematic reviews by considering: 1. Are the results of the study valid? 2. What are the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Systematic Reviews Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• JBI Critical Appraisal Tools, Checklist for Systematic Reviews JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• NHLBI Study Quality Assessment of Systematic Reviews and Meta-Analyses The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• RoB 2 RoB 2 “provides a framework for assessing the risk of bias in a single estimate of an intervention effect reported from a randomized trial,” rather than the entire trial. 6 more... less... RoB 2 (revised tool to assess Risk of Bias in randomized trials)
• CASP Randomised Controlled Trials Checklist This CASP checklist considers various aspects of an RCT that require critical appraisal: 1. Is the basic study design valid for a randomized controlled trial? 2. Was the study methodologically sound? 3. What are the results? 4. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CONSORT Statement The CONSORT checklist includes 25 items to determine the quality of randomized controlled trials. “Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in the report.” 7 more... less... CONSORT (Consolidated Standards of Reporting Trials)
• NHLBI Study Quality Assessment of Controlled Intervention Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• JBI Critical Appraisal Tools Checklist for Randomized Controlled Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• ROBINS-I ROBINS-I is a “tool for evaluating risk of bias in estimates of the comparative effectiveness… of interventions from studies that did not use randomization to allocate units… to comparison groups.” 8 more... less... ROBINS-I (Risk Of Bias in Non-randomized Studies – of Interventions)
• NOS This tool is used primarily to evaluate and appraise case-control or cohort studies. more... less... NOS (Newcastle-Ottawa Scale)
• AXIS Cross-sectional studies are frequently used as an evidence base for diagnostic testing, risk factors for disease, and prevalence studies. “The AXIS tool focuses mainly on the presented [study] methods and results.” 9 more... less... AXIS (Appraisal tool for Cross-Sectional Studies)
• NHLBI Study Quality Assessment Tools for Non-Randomized Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. • Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies • Quality Assessment of Case-Control Studies • Quality Assessment Tool for Before-After (Pre-Post) Studies With No Control Group • Quality Assessment Tool for Case Series Studies more... less... NHLBI (National Heart, Lung, and Blood Institute)
• Case Series Studies Quality Appraisal Checklist Developed by the Institute of Health Economics (Canada), the checklist is comprised of 20 questions to assess “the robustness of the evidence of uncontrolled, [case series] studies.” 10
• Methodological Quality and Synthesis of Case Series and Case Reports In this paper, Dr. Murad and colleagues “present a framework for appraisal, synthesis and application of evidence derived from case reports and case series.” 11
• MINORS The MINORS instrument contains 12 items and was developed for evaluating the quality of observational or non-randomized studies. 12 This tool may be of particular interest to researchers who would like to critically appraise surgical studies. more... less... MINORS (Methodological Index for Non-Randomized Studies)
• JBI Critical Appraisal Tools for Non-Randomized Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis. • Checklist for Analytical Cross Sectional Studies • Checklist for Case Control Studies • Checklist for Case Reports • Checklist for Case Series • Checklist for Cohort Studies
• QUADAS-2 The QUADAS-2 tool “is designed to assess the quality of primary diagnostic accuracy studies… [it] consists of 4 key domains that discuss patient selection, index test, reference standard, and flow of patients through the study and timing of the index tests and reference standard.” 13 more... less... QUADAS-2 (a revised tool for the Quality Assessment of Diagnostic Accuracy Studies)
• JBI Critical Appraisal Tools Checklist for Diagnostic Test Accuracy Studies JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• STARD 2015 The authors of the standards note that “[e]ssential elements of [diagnostic accuracy] study methods are often poorly described and sometimes completely omitted, making both critical appraisal and replication difficult, if not impossible.”10 The Standards for the Reporting of Diagnostic Accuracy Studies was developed “to help… improve completeness and transparency in reporting of diagnostic accuracy studies.” 14 more... less... STARD 2015 (Standards for the Reporting of Diagnostic Accuracy Studies)
• CASP Diagnostic Study Checklist This CASP checklist considers various aspects of diagnostic test studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Diagnostic Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• SYRCLE’s RoB “[I]mplementation of [SYRCLE’s RoB tool] will facilitate and improve critical appraisal of evidence from animal studies. This may… enhance the efficiency of translating animal research into clinical practice and increase awareness of the necessity of improving the methodological quality of animal studies.” 15 more... less... SYRCLE’s RoB (SYstematic Review Center for Laboratory animal Experimentation’s Risk of Bias)
• ARRIVE 2.0 “The [ARRIVE 2.0] guidelines are a checklist of information to include in a manuscript to ensure that publications [on in vivo animal studies] contain enough information to add to the knowledge base.” 16 more... less... ARRIVE 2.0 (Animal Research: Reporting of In Vivo Experiments)
• Critical Appraisal of Studies Using Laboratory Animal Models This article provides “an approach to critically appraising papers based on the results of laboratory animal experiments,” and discusses various “bias domains” in the literature that critical appraisal can identify. 17
• CEBM Critical Appraisal of Qualitative Studies Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• CASP Qualitative Studies Checklist This CASP checklist considers various aspects of qualitative research studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• Quality Assessment and Risk of Bias Tool Repository Created by librarians at Duke University, this extensive listing contains over 100 commonly used risk of bias tools that may be sorted by study type.
• Latitudes Network A library of risk of bias tools for use in evidence syntheses that provides selection help and training videos.

References & Recommended Reading

1.     Kolaski, K., Logan, L. R., & Ioannidis, J. P. (2024). Guidance to best tools and practices for systematic reviews .  British Journal of Pharmacology ,  181 (1), 180-210

2.    Portney LG.  Foundations of clinical research : applications to evidence-based practice.  Fourth edition. ed. Philadelphia: F A Davis; 2020.

3.     Fowkes FG, Fulton PM.  Critical appraisal of published research: introductory guidelines.   BMJ (Clinical research ed).  1991;302(6785):1136-1140.

4.     Singh S.  Critical appraisal skills programme.   Journal of Pharmacology and Pharmacotherapeutics.  2013;4(1):76-77.

5.     Shea BJ, Reeves BC, Wells G, et al.  AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.   BMJ (Clinical research ed).  2017;358:j4008.

6.     Whiting P, Savovic J, Higgins JPT, et al.  ROBIS: A new tool to assess risk of bias in systematic reviews was developed.   Journal of clinical epidemiology.  2016;69:225-234.

7.     Sterne JAC, Savovic J, Page MJ, et al.  RoB 2: a revised tool for assessing risk of bias in randomised trials.  BMJ (Clinical research ed).  2019;366:l4898.

8.     Moher D, Hopewell S, Schulz KF, et al.  CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials.  Journal of clinical epidemiology.  2010;63(8):e1-37.

9.     Sterne JA, Hernan MA, Reeves BC, et al.  ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.  BMJ (Clinical research ed).  2016;355:i4919.

10.     Downes MJ, Brennan ML, Williams HC, Dean RS.  Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS).   BMJ open.  2016;6(12):e011458.

11.   Guo B, Moga C, Harstall C, Schopflocher D.  A principal component analysis is conducted for a case series quality appraisal checklist.   Journal of clinical epidemiology.  2016;69:199-207.e192.

12.   Murad MH, Sultan S, Haffar S, Bazerbachi F.  Methodological quality and synthesis of case series and case reports.  BMJ evidence-based medicine.  2018;23(2):60-63.

13.   Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J.  Methodological index for non-randomized studies (MINORS): development and validation of a new instrument.   ANZ journal of surgery.  2003;73(9):712-716.

14.   Whiting PF, Rutjes AWS, Westwood ME, et al.  QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.   Annals of internal medicine.  2011;155(8):529-536.

15.   Bossuyt PM, Reitsma JB, Bruns DE, et al.  STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies.   BMJ (Clinical research ed).  2015;351:h5527.

16.   Hooijmans CR, Rovers MM, de Vries RBM, Leenaars M, Ritskes-Hoitinga M, Langendam MW.  SYRCLE's risk of bias tool for animal studies.   BMC medical research methodology.  2014;14:43.

17.   Percie du Sert N, Ahluwalia A, Alam S, et al.  Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0.  PLoS biology.  2020;18(7):e3000411.

18.   O'Connor AM, Sargeant JM.  Critical appraisal of studies using laboratory animal models.   ILAR journal.  2014;55(3):405-417.

• << Previous: Minimize Bias
• Next: GRADE >>
• Last Updated: Apr 12, 2024 9:46 AM
• URL: https://libraryguides.mayo.edu/systematicreviewprocess

Systematic Reviews and Meta-Analyses: Critical Appraisal

• Get Started
• Exploratory Search
• Where to Search
• How to Search
• Grey Literature
• What about errata and retractions?
• Eligibility Screening

Critical Appraisal

• Data Extraction
• Synthesis & Discussion
• Assess Certainty
• Share & Archive

All relevant studies must undergo a critical appraisal to evaluate the risk of bias , or internal and external validity, of all relevant references.

This step often occurs simultaneously with the Data Extraction  phase. It is a vital stage of the systematic review process to uphold the cornerstone of reducing bias .

Risk of Bias Tools

• Presenting Results

Critical Appraisal 

Critical appraisal is also referred to as quality assessment , risk of bias assessment , and similar variations. Sometimes the critical appraisal phase is confused with the assessment of certainty of evidence  - although related, these are independent  stages of the systematic review process.

According to the Center for Evidence-Based Medicine (CEBM): 

"Critical appraisal is the process of carefully and systematically assessing the outcome of scientific research (evidence) to judge its trustworthiness, value and relevance in a particular context. Critical appraisal looks at the way a study is conducted and examines factors such as internal validity , generalizability and relevance."

Systematic reviews require a formal, systematic, uniform appraisal of the quality - or  risk of bias  - of all   relevant  studies. In a critical appraisal, you are examining the methods   not  the results .

Process Details

Use risk of bias tools  f or this stage - these tools are often formatted as checklists. You can find more about risk of bias tools in the next tab! If a refresher of some common biases, definitions, and examples is helpful, check out the Catalogue of Bias  from the University of Oxford and CEBM.

Just like the other stages of a systematic review,  2 reviewers  should assess risk of bias in each reference . As such, your team should calculate and report interrater reliability , deciding ahead of time how to resolve conflicts. Oftentimes the critical appraisal occurs at the same time as data extraction .

In addition to the formal risk of bias assessment, your team should also consider meta-biases like publication bias, selective reporting, etc. Search for errata and retractions related to included research, and consider other limitations of and concerns about the included studies and how this may impact the reliability of your review.

Note: Subjectivity of Critical Appraisal 

The critical appraisal is inherently subjective , from the selection of the RoB tool(s) to the final assessment of each study. Therefore, it is important to consider how tools compare, and how this process may impact the results of your review. Check out these studies evaluating Risk of Bias Tools:

Page MJ, McKenzie JE, Higgins JPT  Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review   BMJ Open 2018;8:e019703. doi:  10.1136/bmjopen-2017-019703

Losilla, J.-M., Oliveras, I., Marin-Garcia, J. A., & Vives, J. (2018).  Three risk of bias tools lead to opposite conclusions in observational research synthesis.   Journal of Clinical Epidemiology ,  101 , 61–72.  https://doi.org/10.1016/j.jclinepi.2018.05.021

Margulis, A. V., Pladevall, M., Riera-Guardia, N., Varas-Lorenzo, C., Hazell, L., Berkman, N., Viswanathan, M., & Perez-Gutthann, S. (2014).  Quality assessment of observational studies in a drug-safety systematic review, comparison of two tools: The Newcastle-Ottawa Scale and the RTI item bank .  Clinical Epidemiology , 359.  https://doi.org/10.2147/CLEP.S66677

Select Risk of Bias Tool(s)

When you think of a critical appraisal in a systematic review and/or meta-analysis, think of assessing the  risk of bias  of included studies. The potential biases to consider will vary by study design. Therefore, risk of bias tool(s) should be selected based on the designs of included studies.  If you include more than one study design , you'll include more than one risk of bias tool.  Whenever possible, select tools developed for a discipline relevant to your topic.

Risk of bias tools  are simply checklists used to consider bias specific to a study design, and sometimes discipline. 

• Cochrane Risk of Bias Tool  | randomized trials, health
• Collaboration for Environmental Evidence (CEE) Critical Appraisal Tool, Prototype  | environmental management focused
• Crowe Critical Appraisal Tool  | mixed methods
• Meta-QAT  | public health focused
• Meta-QAT Grey Literature Companion | grey literature
• Mixed Method Appraisal Tool (MMAT) | mixed method ( more detail )
• Newcastle-Ottawa Scale | non-randomized studies
• RTI Item Bank  | observational studies
• SYRCLE's Risk of Bias Tool | animal studies
• Quality Checklist for Blogs | blogs
• Quality Checklist for Podcasts | podcasts

Risk of Bias Toolsets

Risk of bias tool sets  are a series of tools developed by the same group or organization, where each tool addresses a specific study design. The organization is usually discipline specific. Note that many also include a systematic review and/or meta-analysis quality assessment tool, but that these tools will not be useful during this stage as existing reviews will not be folded into your synthesis.

Critical Appraisal Skills Programme (CASP) Checklists include tools for:

• Randomized Controlled Trials 
• Qualitative Studies
• Cohort Study
• Diagnostic Study
• Case Control Study
• Economic Evaluation
• Clinical Prediction Rule 

National Institutes of Health (NIH) Study Quality Assessment Tools include tools for:

• Controlled intervention studies
• Observational cohort and cross-sectional studies
• Case-control studies
• Before-after (pre-post) studies without control
• Case series studies

Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) includes tools for:

• Case-control
• Cross-sectional
• Conference abstracts

Joanna Briggs Institute (JBI) Manual for Evidence Synthesis includes the following tools found in respective relevant chapters:

• Qualitative research (appendix 2.1)
• Randomized controlled trials (appendix 3.1)
• Quasi-experimental studies (non-randomized experimental studies; appendix 3.3)
• Text and opinion (appendix 4.1)  with explanation (Appendix 4.2)
• Prevalence studies (appendix 5.1)
• Cohort studies (appendix 7.1)
• Case-control studies (appendix 7.2)
• Case series (appendix 7.3)
• Case reports (appendix 7.4)
• Cross sectional studies (appendix 7.5)
• Diagnostic test accuracy (appendix 9.1)

Latitudes Network 

• Systematic Reviews ( ROBIS )
• Randomized Controlled Trials ( RoB 2 ) 
• Cohort studies - interventions ( ROBINS-I )
• Cohort studies - exposure ( ROBINS-E ) 
• Diagnostic accuracy studies ( QUADAS-2 ; QUADAS-C ) 
• Prognostic accuracy studies ( QUAPAS ) 
• Prediction models ( PROBAST ) 
• Reliability studies ( COSMIN )

Risk of Bias Tool Repositories

Risk of bias tool repositories  are curated lists of existing tools - kind of like what we've presented above. Although we update this guide with new tools as we find them, these repositories may contain additional resources:

• Quality Assessment and Risk of Bias Tool Repository , from Duke University's Medical Center Library & Archives
• Interactive Tableau Dataset of 68 Risk of Bias Tools , from the National Toxicology Program

Presenting Critical Appraisal Results

Risk of bias within each reference should be presented in a table like the one seen below. Studies are presented along the y-axis and biases considered (what is addressed by the tool) along the x-axis, such that each row belongs to a study , and each column belongs to a bias (or domain/category of biases).

It is also best practice to present the bias across the included set of literature  (seen below). Each bias or bias category  is represented as a row and each row is associated with a bar showing the  percentage of the total included literature  that was rated as low risk, some risk, high risk, or unable to determine the risk. 

The images above can be created using the ROBVIS package of metaverse for evidence synthesis in R. You can create your own graphics without using this software.

Methodological Guidance

• Health Sciences
• Animal, Food Sciences
• Social Sciences
• Environmental Sciences

Cochrane Handbook  -  Part 2: Core Methods

Chapter 7 : Considering bias and conflicts of interest among the included studies

• 7.2 Empirical evidence of bias
• 7.3 General procedures for risk-of-bias assessment
• 7.4 Presentation of assessment of risk of bias
• 7.5 Summary assessments of risk of bias 
• 7.6 Incorporating assessment of risk of bias into analyses 
• 7.7 Considering risk of bias due to missing results
• 7.8 Considering source of funding and conflict of interest of authors of included studies 

Chapter 8: Assessing risk of bias in randomized trial

• 8.2 Overview of RoB 2
• 8.3 Bias arising from the randomization process
• 8.4 Bias due to deviations from intended interventions
• 8.5 Bias due to missing outcome data 
• 8.6 Bias in measurement of the outcome
• 8.7 Bias in selection of the reported result
• 8.8 Differences from the previous version of the tool

Chapter 25:  Risk of bias in non-randomized studies

SYREAF Resources

Step 3: identifying eligible papers.

Conducting systematic reviews of intervention questions II: Relevance screening, data extraction, assessing risk of bias , presenting the results and interpreting the findings.  Sargeant JM, O’Connor AM. Zoonoses Public Health. 2014 Jun;61 Suppl 1:39-51. doi: 10.1111/zph.12124. PMID: 24905995

Campbell -  MECCIR

C51. Assessing risk of bias / study quality ( protocol & review / final manuscript )

C52. Assessing risk of bias / study quality in duplicate  ( protocol & review / final manuscript )

C53. Supporting judgements of risk of bias / study quality ( review / final manuscript )

C54. Providing sources of information for risk of bias / study quality assessments ( review / final manuscript )

C55. Differentiating between performance bias and detection bias  ( protocol & review / final manuscript )

C56. If applicable, assessing risk of bias due to lack of blinding for different outcomes ( review / final manuscript )

C57. If applicable, assessing completeness of data for different outcomes ( review / final manuscript )

C58. If applicable, summarizing risk of bias when using the Cochrane Risk of Bias tool ( review / final manuscript )

C59. Addressing risk of bias / study quality in the synthesis  ( review / final manuscript )

C60. Incorporating assessments of risk of bias  ( review / final manuscript )

CEE  -  Guidelines and Standards for Evidence synthesis in Environmental Management

Section 7.  critical appraisal of study validity.

CEE Standards for conduct and reporting

7.1.2   Internal validity

7.1.3  External validity 

Reporting in Protocol and Final Manuscript

• Final Manuscript

In the Protocol |  PRISMA-P

Risk of bias individual studies (item 14).

...planned approach to assessing risk of bias should include the constructs being assessed and a definition for each, reviewer judgment options (high, low, unclear), the number of assessors ...training, piloting, previous risk of bias assessment experience...method(s) of assessment (independent or in duplicate)...

Protocol for reporting results

" ...summarise risk of bias assessments across studies or outcomes ..."

Protocol for reporting  impact on synthesis

"...describe how risk of bias assessments will be incorporated into data synthesis (that is, subgroup or sensitivity analyses) and their potential influence on findings of the review (Item 15c) in the protocol..."

In the Final Manuscript |  PRISMA

For the critical appraisal stage, PRISMA requires specific items to be addressed in both the methods and results section.

Study Risk of Bias Assessment (Item 11; report in methods )

Essential items.

• Specify the tool(s) (and version) used to assess risk of bias in the included studies.
• Specify the methodological domains/components/items of the risk of bias tool(s) used.
• Report whether an overall risk of bias judgment that summarised across domains/components/items was made, and if so, what rules were used to reach an overall judgment.
• If any adaptations to an existing tool to assess risk of bias in studies were made (such as omitting or modifying items), specify the adaptations.
• If a new risk of bias tool was developed for use in the review, describe the content of the tool and make it publicly accessible.
• Report how many reviewers assessed risk of bias in each study, whether multiple reviewers worked independently (such as assessments performed by one reviewer and checked by another), and any processes used to resolve disagreements between assessors.
• Report any processes used to obtain or confirm relevant information from study investigators.
• If an automation tool was used to assess risk of bias in studies, report how the automation tool was used (such as machine learning models to extract sentences from articles relevant to risk of bias88), how the tool was trained , and details on the tool’s performance and internal validation

Risk of Bias in Studies (Item 18; report in results )

• Present tables or figures indicating for each study the risk of bias in each domain /component/item assessed and overall study-level risk of bias.
• Present justification for each risk of bias judgment—for example, in the form of relevant quotations from reports of included studies.

Additional Items

If assessments of risk of bias were done for specific outcomes or results in each study , consider displaying risk of bias judgments on a forest plot, next to the study results, so that the limitations of studies contributing to a particular meta-analysis are evident (see Sterne et al86 for an example forest plot).

We host a workshop each fall on critical appraisal, check out our latest recording !

• << Previous: Eligibility Screening
• Next: Data Extraction >>
• Last Updated: Apr 12, 2024 12:41 PM
• URL: https://guides.lib.vt.edu/SRMA

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Published: 20 January 2009

How to critically appraise an article

• Jane M Young 1 &
• Michael J Solomon 2  

Nature Clinical Practice Gastroenterology & Hepatology volume  6 ,  pages 82–91 ( 2009 ) Cite this article

52k Accesses

100 Citations

435 Altmetric

Metrics details

Critical appraisal is a systematic process used to identify the strengths and weaknesses of a research article in order to assess the usefulness and validity of research findings. The most important components of a critical appraisal are an evaluation of the appropriateness of the study design for the research question and a careful assessment of the key methodological features of this design. Other factors that also should be considered include the suitability of the statistical methods used and their subsequent interpretation, potential conflicts of interest and the relevance of the research to one's own practice. This Review presents a 10-step guide to critical appraisal that aims to assist clinicians to identify the most relevant high-quality studies available to guide their clinical practice.

Critical appraisal is a systematic process used to identify the strengths and weaknesses of a research article

Critical appraisal provides a basis for decisions on whether to use the results of a study in clinical practice

Different study designs are prone to various sources of systematic bias

Design-specific, critical-appraisal checklists are useful tools to help assess study quality

Assessments of other factors, including the importance of the research question, the appropriateness of statistical analysis, the legitimacy of conclusions and potential conflicts of interest are an important part of the critical appraisal process

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

Receive 12 print issues and online access

195,33 € per year

only 16,28 € per issue

Buy this article

• Purchase on Springer Link
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

Making sense of the literature: an introduction to critical appraisal for the primary care practitioner

How to appraise the literature: basic principles for the busy clinician - part 2: systematic reviews and meta-analyses

How to appraise the literature: basic principles for the busy clinician - part 1: randomised controlled trials

Druss BG and Marcus SC (2005) Growth and decentralisation of the medical literature: implications for evidence-based medicine. J Med Libr Assoc 93 : 499–501

PubMed   PubMed Central   Google Scholar  

Glasziou PP (2008) Information overload: what's behind it, what's beyond it? Med J Aust 189 : 84–85

PubMed   Google Scholar  

Last JE (Ed.; 2001) A Dictionary of Epidemiology (4th Edn). New York: Oxford University Press

Google Scholar  

Sackett DL et al . (2000). Evidence-based Medicine. How to Practice and Teach EBM . London: Churchill Livingstone

Guyatt G and Rennie D (Eds; 2002). Users' Guides to the Medical Literature: a Manual for Evidence-based Clinical Practice . Chicago: American Medical Association

Greenhalgh T (2000) How to Read a Paper: the Basics of Evidence-based Medicine . London: Blackwell Medicine Books

MacAuley D (1994) READER: an acronym to aid critical reading by general practitioners. Br J Gen Pract 44 : 83–85

CAS   PubMed   PubMed Central   Google Scholar  

Hill A and Spittlehouse C (2001) What is critical appraisal. Evidence-based Medicine 3 : 1–8 [ http://www.evidence-based-medicine.co.uk ] (accessed 25 November 2008)

Public Health Resource Unit (2008) Critical Appraisal Skills Programme (CASP) . [ http://www.phru.nhs.uk/Pages/PHD/CASP.htm ] (accessed 8 August 2008)

National Health and Medical Research Council (2000) How to Review the Evidence: Systematic Identification and Review of the Scientific Literature . Canberra: NHMRC

Elwood JM (1998) Critical Appraisal of Epidemiological Studies and Clinical Trials (2nd Edn). Oxford: Oxford University Press

Agency for Healthcare Research and Quality (2002) Systems to rate the strength of scientific evidence? Evidence Report/Technology Assessment No 47, Publication No 02-E019 Rockville: Agency for Healthcare Research and Quality

Crombie IK (1996) The Pocket Guide to Critical Appraisal: a Handbook for Health Care Professionals . London: Blackwell Medicine Publishing Group

Heller RF et al . (2008) Critical appraisal for public health: a new checklist. Public Health 122 : 92–98

Article   Google Scholar  

MacAuley D et al . (1998) Randomised controlled trial of the READER method of critical appraisal in general practice. BMJ 316 : 1134–37

Article   CAS   Google Scholar  

Parkes J et al . Teaching critical appraisal skills in health care settings (Review). Cochrane Database of Systematic Reviews 2005, Issue 3. Art. No.: cd001270. 10.1002/14651858.cd001270

Mays N and Pope C (2000) Assessing quality in qualitative research. BMJ 320 : 50–52

Hawking SW (2003) On the Shoulders of Giants: the Great Works of Physics and Astronomy . Philadelphia, PN: Penguin

National Health and Medical Research Council (1999) A Guide to the Development, Implementation and Evaluation of Clinical Practice Guidelines . Canberra: National Health and Medical Research Council

US Preventive Services Taskforce (1996) Guide to clinical preventive services (2nd Edn). Baltimore, MD: Williams & Wilkins

Solomon MJ and McLeod RS (1995) Should we be performing more randomized controlled trials evaluating surgical operations? Surgery 118 : 456–467

Rothman KJ (2002) Epidemiology: an Introduction . Oxford: Oxford University Press

Young JM and Solomon MJ (2003) Improving the evidence-base in surgery: sources of bias in surgical studies. ANZ J Surg 73 : 504–506

Margitic SE et al . (1995) Lessons learned from a prospective meta-analysis. J Am Geriatr Soc 43 : 435–439

Shea B et al . (2001) Assessing the quality of reports of systematic reviews: the QUORUM statement compared to other tools. In Systematic Reviews in Health Care: Meta-analysis in Context 2nd Edition, 122–139 (Eds Egger M. et al .) London: BMJ Books

Chapter   Google Scholar  

Easterbrook PH et al . (1991) Publication bias in clinical research. Lancet 337 : 867–872

Begg CB and Berlin JA (1989) Publication bias and dissemination of clinical research. J Natl Cancer Inst 81 : 107–115

Moher D et al . (2000) Improving the quality of reports of meta-analyses of randomised controlled trials: the QUORUM statement. Br J Surg 87 : 1448–1454

Shea BJ et al . (2007) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology 7 : 10 [10.1186/1471-2288-7-10]

Stroup DF et al . (2000) Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 283 : 2008–2012

Young JM and Solomon MJ (2003) Improving the evidence-base in surgery: evaluating surgical effectiveness. ANZ J Surg 73 : 507–510

Schulz KF (1995) Subverting randomization in controlled trials. JAMA 274 : 1456–1458

Schulz KF et al . (1995) Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273 : 408–412

Moher D et al . (2001) The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Medical Research Methodology 1 : 2 [ http://www.biomedcentral.com/ 1471-2288/1/2 ] (accessed 25 November 2008)

Rochon PA et al . (2005) Reader's guide to critical appraisal of cohort studies: 1. Role and design. BMJ 330 : 895–897

Mamdani M et al . (2005) Reader's guide to critical appraisal of cohort studies: 2. Assessing potential for confounding. BMJ 330 : 960–962

Normand S et al . (2005) Reader's guide to critical appraisal of cohort studies: 3. Analytical strategies to reduce confounding. BMJ 330 : 1021–1023

von Elm E et al . (2007) Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ 335 : 806–808

Sutton-Tyrrell K (1991) Assessing bias in case-control studies: proper selection of cases and controls. Stroke 22 : 938–942

Knottnerus J (2003) Assessment of the accuracy of diagnostic tests: the cross-sectional study. J Clin Epidemiol 56 : 1118–1128

Furukawa TA and Guyatt GH (2006) Sources of bias in diagnostic accuracy studies and the diagnostic process. CMAJ 174 : 481–482

Bossyut PM et al . (2003)The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med 138 : W1–W12

STARD statement (Standards for the Reporting of Diagnostic Accuracy Studies). [ http://www.stard-statement.org/ ] (accessed 10 September 2008)

Raftery J (1998) Economic evaluation: an introduction. BMJ 316 : 1013–1014

Palmer S et al . (1999) Economics notes: types of economic evaluation. BMJ 318 : 1349

Russ S et al . (1999) Barriers to participation in randomized controlled trials: a systematic review. J Clin Epidemiol 52 : 1143–1156

Tinmouth JM et al . (2004) Are claims of equivalency in digestive diseases trials supported by the evidence? Gastroentrology 126 : 1700–1710

Kaul S and Diamond GA (2006) Good enough: a primer on the analysis and interpretation of noninferiority trials. Ann Intern Med 145 : 62–69

Piaggio G et al . (2006) Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA 295 : 1152–1160

Heritier SR et al . (2007) Inclusion of patients in clinical trial analysis: the intention to treat principle. In Interpreting and Reporting Clinical Trials: a Guide to the CONSORT Statement and the Principles of Randomized Controlled Trials , 92–98 (Eds Keech A. et al .) Strawberry Hills, NSW: Australian Medical Publishing Company

National Health and Medical Research Council (2007) National Statement on Ethical Conduct in Human Research 89–90 Canberra: NHMRC

Lo B et al . (2000) Conflict-of-interest policies for investigators in clinical trials. N Engl J Med 343 : 1616–1620

Kim SYH et al . (2004) Potential research participants' views regarding researcher and institutional financial conflicts of interests. J Med Ethics 30 : 73–79

Komesaroff PA and Kerridge IH (2002) Ethical issues concerning the relationships between medical practitioners and the pharmaceutical industry. Med J Aust 176 : 118–121

Little M (1999) Research, ethics and conflicts of interest. J Med Ethics 25 : 259–262

Lemmens T and Singer PA (1998) Bioethics for clinicians: 17. Conflict of interest in research, education and patient care. CMAJ 159 : 960–965

Download references

Author information

Authors and affiliations.

JM Young is an Associate Professor of Public Health and the Executive Director of the Surgical Outcomes Research Centre at the University of Sydney and Sydney South-West Area Health Service, Sydney,

Jane M Young

MJ Solomon is Head of the Surgical Outcomes Research Centre and Director of Colorectal Research at the University of Sydney and Sydney South-West Area Health Service, Sydney, Australia.,

Michael J Solomon

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Jane M Young .

Ethics declarations

Competing interests.

The authors declare no competing financial interests.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Young, J., Solomon, M. How to critically appraise an article. Nat Rev Gastroenterol Hepatol 6 , 82–91 (2009). https://doi.org/10.1038/ncpgasthep1331

Download citation

Received : 10 August 2008

Accepted : 03 November 2008

Published : 20 January 2009

Issue Date : February 2009

DOI : https://doi.org/10.1038/ncpgasthep1331

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Emergency physicians’ perceptions of critical appraisal skills: a qualitative study.

• Sumintra Wood
• Jacqueline Paulis
• Angela Chen

BMC Medical Education (2022)

An integrative review on individual determinants of enrolment in National Health Insurance Scheme among older adults in Ghana

• Anthony Kwame Morgan
• Anthony Acquah Mensah

BMC Primary Care (2022)

Autopsy findings of COVID-19 in children: a systematic review and meta-analysis

• Anju Khairwa
• Kana Ram Jat

Forensic Science, Medicine and Pathology (2022)

The use of a modified Delphi technique to develop a critical appraisal tool for clinical pharmacokinetic studies

• Alaa Bahaa Eldeen Soliman
• Shane Ashley Pawluk
• Ousama Rachid

International Journal of Clinical Pharmacy (2022)

Critical Appraisal: Analysis of a Prospective Comparative Study Published in IJS

• Ramakrishna Ramakrishna HK
• Swarnalatha MC

Indian Journal of Surgery (2021)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

How to Perform a Systematic Literature Review pp 51–68 Cite as

Critical Appraisal: Assessing the Quality of Studies

• Edward Purssell   ORCID: orcid.org/0000-0003-3748-0864 3 &
• Niall McCrae   ORCID: orcid.org/0000-0001-9776-7694 4  
• First Online: 05 August 2020

7351 Accesses

There is great variation in the type and quality of research evidence. Having completed your search and assembled your studies, the next step is to critically appraise the studies to ascertain their quality. Ultimately you will be making a judgement about the overall evidence, but that comes later. You will see throughout this chapter that we make a clear differentiation between the individual studies and what we call the body of evidence , which is all of the studies and anything else that we use to answer the question or to make a recommendation. This chapter deals with only the first of these—the individual studies. Critical appraisal, like everything else in systematic literature reviewing, is a scientific exercise that requires individual judgement, and we describe some tools to help you.

This is a preview of subscription content, log in via an institution .

Buying options

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Oxford Centre for Evidence-Based Medicine (OCEBM) (2016) OCEBM levels of evidence. In: CEBM. https://www.cebm.net/2016/05/ocebm-levels-of-evidence/ . Accessed 17 Apr 2020

Aromataris E, Munn Z (eds) (2017) Joanna Briggs Institute reviewer’s manual. The Joanna Briggs Institute, Adelaide

Google Scholar  

Daly J, Willis K, Small R et al (2007) A hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol 60:43–49. https://doi.org/10.1016/j.jclinepi.2006.03.014

Article   PubMed   Google Scholar  

EQUATOR Network (2020) What is a reporting guideline?—The EQUATOR Network. https://www.equator-network.org/about-us/what-is-a-reporting-guideline/ . Accessed 7 Mar 2020

Tong A, Sainsbury P, Craig J (2007) Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 19:349–357. https://doi.org/10.1093/intqhc/mzm042

von Elm E, Altman DG, Egger M et al (2007) The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med 4:e296. https://doi.org/10.1371/journal.pmed.0040296

Article   Google Scholar  

Brouwers MC, Kerkvliet K, Spithoff K, AGREE Next Steps Consortium (2016) The AGREE reporting checklist: a tool to improve reporting of clinical practice guidelines. BMJ 352:i1152. https://doi.org/10.1136/bmj.i1152

Article   PubMed   PubMed Central   Google Scholar  

Moher D, Liberati A, Tetzlaff J et al (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6:e1000097. https://doi.org/10.1371/journal.pmed.1000097

Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A (2019) Chapter 7: Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019), Cochrane. https://www.training.cochrane.org/handbook

Critical Appraisal Skills Programme (2018) CASP checklists. In: CASP—critical appraisal skills programme. https://casp-uk.net/casp-tools-checklists/ . Accessed 7 Mar 2020

Higgins JPT, Savović J, Page MJ et al (2019) Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JPT, Thomas J, Chandler J et al (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, London

Chapter   Google Scholar  

Guyatt GH, Oxman AD, Kunz R et al (2011) GRADE guidelines 6. Rating the quality of evidence—imprecision. J Clin Epidemiol 64:1283–1293. https://doi.org/10.1016/j.jclinepi.2011.01.012

Sterne JAC, Savović J, Page MJ et al (2019) RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 366:l4898. https://doi.org/10.1136/bmj.l4898

Sterne JA, Hernán MA, Reeves BC et al (2016) ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355:i4919. https://doi.org/10.1136/bmj.i4919

Wells GA, Shea B, O’Connell D et al (2019) The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa Hospital Research Institute, Ottawa. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp . Accessed 7 Mar 2020

Cochrane Community (2020) Glossary—Cochrane community. https://community.cochrane.org/glossary#letter-R . Accessed 8 Mar 2020

Messick S (1989) Meaning and values in test validation: the science and ethics of assessment. Educ Res 18:5–11. https://doi.org/10.3102/0013189X018002005

Sparkes AC (2001) Myth 94: qualitative health researchers will agree about validity. Qual Health Res 11:538–552. https://doi.org/10.1177/104973230101100409

Article   CAS   PubMed   Google Scholar  

Aguinis H, Solarino AM (2019) Transparency and replicability in qualitative research: the case of interviews with elite informants. Strat Manag J 40:1291–1315. https://doi.org/10.1002/smj.3015

Lincoln YS, Guba EG (1985) Naturalistic inquiry. Sage Publications, Beverly Hills, CA

Book   Google Scholar  

Hannes K (2011) Chapter 4: Critical appraisal of qualitative research. In: Noyes J, Booth A, Hannes K et al (eds) Supplementary guidance for inclusion of qualitative research in Cochrane systematic reviews of interventions. Cochrane Collaboration Qualitative Methods Group, London

Munn Z, Porritt K, Lockwood C et al (2014) Establishing confidence in the output of qualitative research synthesis: the ConQual approach. BMC Med Res Methodol 14:108. https://doi.org/10.1186/1471-2288-14-108

Toye F, Seers K, Allcock N et al (2013) ‘Trying to pin down jelly’—exploring intuitive processes in quality assessment for meta-ethnography. BMC Med Res Methodol 13:46. https://doi.org/10.1186/1471-2288-13-46

Katikireddi SV, Egan M, Petticrew M (2015) How do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? A methodological study. J Epidemiol Community Health 69:189–195. https://doi.org/10.1136/jech-2014-204711

McKenzie JE, Brennan SE, Ryan RE et al (2019) Chapter 9: Summarizing study characteristics and preparing for synthesis. In: Higgins JPT, Thomas J, Chandler J et al (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, London

Deeks JJ, Higgins JPT, Altman DG (2019) Chapter 10: Analysing data and undertaking meta-analyses. In: Higgins JPT, Thomas J, Chandler J et al (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, London

Download references

Author information

Authors and affiliations.

School of Health Sciences, City, University of London, London, UK

Edward Purssell

Florence Nightingale Faculty of Nursing, Midwifery & Palliative Care, King’s College London, London, UK

Niall McCrae

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Edward Purssell .

Rights and permissions

Reprints and permissions

Copyright information

© 2020 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Cite this chapter.

Purssell, E., McCrae, N. (2020). Critical Appraisal: Assessing the Quality of Studies. In: How to Perform a Systematic Literature Review. Springer, Cham. https://doi.org/10.1007/978-3-030-49672-2_6

Download citation

DOI : https://doi.org/10.1007/978-3-030-49672-2_6

Published : 05 August 2020

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-49671-5

Online ISBN : 978-3-030-49672-2

eBook Packages : Medicine Medicine (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Our services

Critical Appraisal Questionnaires

Critical appraisal is the process of carefully and systematically assessing the outcome of scientific research (evidence) to judge its trustworthiness, value and relevance in a particular context. Critical appraisal looks at the way a study is conducted and examines factors such as internal validity, generalizability and relevance.

Some initial appraisal questions you could ask are:

• Is the evidence from a known, reputable source?
• Has the evidence been evaluated in any way? If so, how and by whom?
• How up-to-date is the evidence?

Second, you could look at the study itself and ask the following general appraisal questions:

• How was the outcome measured?
• Is that a reliable way to measure?
• How large was the effect size?
• What implications does the study have for your practice? Is it relevant?
• Can the results be applied to your organization?

Questionnaires

If you would like to critically appraise a study, we strongly recommend using the app we have developed for iOS and Android:  CAT Manager App

You could also consider using the following appraisal questionnaires (checklists) for specific study designs, but we do not recommend this. 

Appraisal of a meta-analysis or systematic review

Appraisal of a controlled study, appraisal of a cohort or panel study, appraisal of a case control study, appraisal of a cross-sectional study (survey), appraisal of a qualitative study, appraisal of a case study.

• Navigate To
• Members area
• Bargelaan 200
• 2333 CW Leiden
• The Netherlands
• Want to stay up to date?

• {{region.title}}
• Others (a full list)

{{region_detail.title}} back

{{region_detail.text}}

• {{person.title}}
• {{organization.title}}

{{person_detail.title}} back

{{person_detail.text}}

{{organization_detail.title}} back

{{organization_detail.text}}

coming soon

{{series.title}}.

{{series.description | limitTo:140}}

March 27, 2024

Russian offensive campaign assessment, march 27, 2024.

March 27, 2024, 5:10pm ET

Click here to see ISW’s interactive map of the Russian invasion of Ukraine. This map is updated daily alongside the static maps present in this report.

Click here to see ISW’s 3D control of terrain topographic map of Ukraine. Use of a computer (not a mobile device) is strongly recommended for using this data-heavy tool.

Click here to access ISW’s archive of interactive time-lapse maps of the Russian invasion of Ukraine. These maps complement the static control-of-terrain map that ISW produces daily by showing a dynamic frontline. ISW will update this time-lapse map archive monthly.

Note: The data cut-off for this product was 2:15pm ET on March 27. ISW will cover subsequent reports in the March 28 Russian Offensive Campaign Assessment.

The UN Human Rights Monitoring Mission in Ukraine (HRMMU) released its 38th report on the human rights situation in Ukraine on March 26, confirming several of ISW’s longstanding assessments about Russia’s systematic violations of international human rights and humanitarian law in occupied territories and towards Ukrainian prisoners of war (POWs). [1] The HRMMU report details activities between December 1, 2023 and February 29 2024, and includes new findings about Russia’s abuse of Ukrainian POWs during this timeframe, based on interviews with 60 recently released male POWs. [2] Nearly all of the POWs that HRMMU interviewed detailed how they were tortured by Russian forces with beatings and electric shocks and threatened with execution, and over half of the interviewees experienced sexual violence. HRMMU also reported that it has evidence of Russian forces executing at least 32 POWs in 12 different incidents during the reporting period and independently verified three of the executions. ISW observed open-source evidence of several POW executions during this reporting period: the execution of three Ukrainian POWs near Robotyne, Zaporizhia Oblast on December 27, 2023; the execution of one Ukrainian POW near Klishchiivka, Donetsk Oblast on February 9, 2024; the executions of three Ukrainian POWs near Robotyne, the execution of six Ukrainian POWs near Avdiivka, Donetsk Oblast, and the executions of two Ukrainian POWs near Vesele, Donetsk Oblast on or around February 18, 2024; and the execution of nine Ukrainian POWs near Ivanivske, Donetsk Oblast, on February 25. [3] The summary execution and mistreatment of POWs is a violation of Article 3 of the Geneva Convention relative to the Treatment of Prisoners of War. [4] The HRMMU report also details the forced Russification of Ukrainian populations in occupied areas, including the imposition of Russian political, legal, and administrative systems onto occupied Ukraine in violation of Russia’s international legal obligations as an occupying power. [5] ISW has reported at length on the specifics of Russia’s illegal occupation of Ukraine, consistent with the findings of the UN HRMMU report. [6]

Russian officials are tying the US and the West to a broader set of “terrorist” attacks against Russia following the Crocus City Hall attack, likely to intensify rhetoric about alleged Western and Ukrainian threats to generate greater domestic support for the war in Ukraine. The Russian Investigative Committee and Prosecutor General’s Office stated on March 27 that they will consider an appeal from the Russian State Duma to investigate American and Western financing and organization of terrorist attacks against Russia. [7] The Russian Investigative Committee, Prosecutor General’s Office, and the Duma Deputies that made the appeal did not explicitly reference the Crocus City Hall attack. [8] Kremlin officials have previously tied Ukraine and the West to the Crocus City Hall attack but have yet to make a formal accusation, and the Kremlin may refrain from issuing an official accusation as all available evidence continues to show that the Islamic State (IS) is very likely responsible for the attack. [9] Russian officials routinely describe Ukrainian military strikes against legitimate military targets in occupied Ukraine and Russia as terrorism and consistently claim that Western actors help organize these strikes. [10] The Kremlin likely aims to seize on wider Russian social fears and anger following the Crocus City Hall attack by portraying Ukraine, the US, and the West as immediate terrorist threats. The Kremlin likely hopes that perceptions of Ukrainian and Western involvement in the Crocus City Hall attack will increase domestic support for the war in Ukraine, and Russian officials will likely invoke a broader view of what they consider terrorism to further cast Ukrainians as terrorists and the West as a sponsor of terrorism. [11] The Kremlin may still formally accuse Ukraine of conducting the Crocus City Hall attack if it believes that these other informational efforts are insufficient to generate the domestic response it likely desires. [12]

Russian authorities are increasing legal pressure against migrants in Russia following recent Russian officials’ proposals for harsher, measures against migrant communities in response to the March 22 Crocus City Hall attack. BBC News Russian Service stated that there has been a significant increase in the number of cases related to violations of the rules of entry for foreign citizens into Russia following the Crocus City Hall attack. [13] BBC News Russian Service reported on March 27 that 784 such cases have been registered since the morning of March 25, as compared with 1,106 during the entire previous week. A Russian lawyer who often works with Tajik citizens reportedly told BBC News Russian Service that over 100 people waited for a Moscow district court to hear their cases on March 25 alone and that Russian authorities are especially targeting migrants from Tajikistan during searches. BBC News Russian Service reported that representatives of the Tajik diaspora in Russia are expecting Russian authorities to conduct a large wave of deportations following the Crocus City Hall attack. A Russian insider source claimed on March 27 that unspecified actors gave the Moscow Ministry of Internal Affairs (MVD) an “unspoken” order to “not spare” migrants and for MVD employees to use their own judgement in the field. [14] The insider source claimed that a source suggested that Russian authorities are not preparing to conduct raids on migrant communities but will apply the “strictest measures” to migrants in “controversial situations.” Kremlin newswire TASS stated on March 27 that Russian police and Rosgvardia conducted a raid at the Wildberries warehouse in Elektrostal, Moscow Oblast to check the documents of migrant workers, and Russian opposition outlet Baza reported that Russian authorities detained 21 people during the raid. [15] Several Russian ultranationalist milbloggers complained that the way Russian-language schools in Tajikistan are teaching about Russia’s historical imperial occupation of Tajikistan is discouraging Tajik migrants from integrating into Russian society, essentially blaming migrants for the alienation that Russian society subjects them to. [16] Select Russian officials recently called for the introduction of several anti-migrant policies, which Russian authorities are unlikely to enact given Russia’s reliance on migrants for its force generation and labor needs. [17] Russian authorities may continue the practice of raiding migrant workplaces and increase crackdowns at border crossings to temporarily placate emotional cries for retribution following the March 22 attack as the Kremlin continues to develop a cogent and practical response.

Key Takeaways:

• The UN Human Rights Monitoring Mission in Ukraine (HRMMU) released its 38th report on the human rights situation in Ukraine on March 26, confirming several of ISW’s longstanding assessments about Russia’s systematic violations of international human rights and humanitarian law in occupied territories and towards Ukrainian prisoners of war (POWs).
• Russian officials are tying the US and the West to a broader set of “terrorist” attacks against Russia following the Crocus City Hall attack, likely to intensify rhetoric about alleged Western and Ukrainian threats to generate greater domestic support for the war in Ukraine.
• Russian authorities are increasing legal pressure against migrants in Russia following recent Russian officials’ proposals for harsher, measures against migrant communities in response to the March 22 Crocus City Hall attack.
• Russian forces recently made confirmed advances near Avdiivka and southwest of Donetsk City on March 27.
• Russian Storm-Z personnel continue to complain about their poor treatment by the Russian Ministry of Defense (MoD) as the MoD tries to posture efficacy in its force generation and social benefit allocation system.

We do not report in detail on Russian war crimes because these activities are well-covered in Western media and do not directly affect the military operations we are assessing and forecasting. We will continue to evaluate and report on the effects of these criminal activities on the Ukrainian military and the Ukrainian population and specifically on combat in Ukrainian urban areas. We utterly condemn Russian violations of the laws of armed conflict and the Geneva Conventions and crimes against humanity even though we do not describe them in these reports.  

• Russian Main Effort – Eastern Ukraine (comprised of two subordinate main efforts)
• Russian Subordinate Main Effort #1 – Capture the remainder of Luhansk Oblast and push westward into eastern Kharkiv Oblast and encircle northern Donetsk Oblast
• Russian Subordinate Main Effort #2 – Capture the entirety of Donetsk Oblast
• Russian Supporting Effort – Southern Axis
• Russian Air, Missile, and Drone Campaign
• Russian Mobilization and Force Generation Efforts
• Russian Technological Adaptations
• Activities in Russian-occupied areas
• Ukrainian Defense Industrial Base Efforts

Russian Information Operations and Narratives

• Significant Activity in Belarus

Russian Main Effort – Eastern Ukraine

Russian Subordinate Main Effort #1 – Luhansk Oblast (Russian objective: Capture the remainder of Luhansk Oblast and push westward into eastern Kharkiv Oblast and northern Donetsk Oblast)

Positional engagements continued along the Kupyansk-Svatove-Kreminna line on March 27, but there were no confirmed changes to the frontline in this area. Ukrainian and Russian sources stated that positional engagements continued northeast of Kupyansk near Synkivka and Lake Lyman; southeast of Kupyansk near Ivanivka; west of Kreminna near Terny and Yampolivka; and south of Kreminna near Bilohorivka. [18] Russian milbloggers claimed that Russian forces advanced near Terny, but ISW has not observed visual confirmation of this claim. [19] Chechen Republic Head Ramzan Kadyrov stated that elements of the Chechen Akhmat Spetsnaz “Aida” detachment are operating near Bilohorivka. [20]

Ukrainian officials reported that Russian forces struck Kharkiv City with a D-30 universal joint glide munition (UMPB), a guided glide bomb, on March 27. [21] Ukrainian officials noted that the strike was the first Russian glide bomb strike against Kharkiv City since the beginning of the full-scale invasion in 2022. [22] Ukrainian Kharkiv Oblast Military Administration Head Oleh Synehubov stated that the UMPB D-30 has a range of up to 90 kilometers and that Russian forces can launch the bomb from aircraft or ground-based Smerch multiple rocket launch systems (MLRS). [23] Russian forces struck Myrnohrad, Donetsk Oblast with three UMPB D-30SN guided glide bombs on March 10. [24]

Russian Subordinate Main Effort #2 – Donetsk Oblast (Russian objective: Capture the entirety of Donetsk Oblast, the claimed territory of Russia’s proxies in Donbas)

Russian forces reportedly advanced west of Bakhmut, although there were no confirmed changes to the frontline in the area on March 27. Russian milbloggers claimed that Russian forces advanced west of Bakhmut along a railway line and a section of the O0506 (Khromove-Chasiv Yar) highway by 1.15 kilometers in depth and 1.85 kilometers in width. [25] A Russian milblogger claimed that elements of the 98th Airborne (VDV) Division are advancing near Ivanivske and are within 500 meters of the city limits of Chasiv Yar (west of Bakhmut). [26] Russian Defense Minister Sergei Shoigu credited elements of the Russian 102nd Motorized Rifle Regiment (150th Motorized Rifle Division, 8th Combined Arms Army [CAA], Southern Military District [SMD]) with seizing Ivanivske on March 24, although ISW has yet to observe visual evidence confirming that Russian forces have seized Ivanivske. [27] Positional fighting continued northeast of Bakhmut near Vesele; northwest of Bakhmut near Bohdanivka; west of Bakhmut near Ivanivske; southwest of Bakhmut near Klishchiivka and Andriivka; and south of Bakhmut near Shumy and Pivdenne. [28] A Ukrainian military observer reported that Russian forces have intensified transfers of equipment and personnel along ground lines of communication (GLOCs) through Kadiivka, Pervomaisk, and Popasna (all east of Bakhmut), but did not specify the destination of these transfers. [29] Kadiivka, Pervomaisk, and Popasna all lie along the T0504 Luhansk City-Bakhmut highway that runs directly from the Russian rear in occupied Luhansk Oblast into Bakhmut, however.

Russian forces recently advanced west of Avdiivka amid continued positional fighting in the area on March 27. Geolocated footage published on March 27 indicates that Russian forces recently advanced within Berdychi (northwest of Avdiivka) and in Orlivka (west of Avdiivka). [30] Russian milbloggers claimed that Russian forces entered Semenivka (northwest of Avdiivka) and are attacking Ukrainian positions within the settlement but that Ukrainian forces are actively counterattacking in the area. [31] A Russian milblogger claimed that Russian forces advanced 200 meters west of Orlivka on the western bank of the Durna River, 200 meters west of Tonenke (west of Avdiivka), 200 meters in the direction of Umanske (west of Avdiivka), 300 meters south of Tonenke towards Pervomaiske (southwest of Avdiivka), and 100 meters south of Nevelske (southwest of Avdiivka). [32] ISW has not observed visual confirmation of these claims. Positional fighting continued northwest of Avdiivka near Berdychi and Semenivka; west of Avdiivka near Orlivka, Tonenke, and Umanske; and southwest of Avdiivka near Vodyane, Nevelske, and Pervomaiske. [33]

Russian forces recently advanced southwest of Donetsk City amid continued positional fighting west and southwest of Donetsk City on March 27. Geolocated footage published on March 27 indicates that Russian forces recently advanced within central Novomykhailivka (southwest of Donetsk City). [34] Positional fighting continued west of Donetsk City near Heorhiivka and Krasnohorivka and southwest of Donetsk City near Novomykhailivka and Pobieda. [35] Elements of the Russian 5th Motorized Rifle Brigade (1st Donetsk People’s Republic [DNR] Army Corps [AC]) are reportedly operating near Krasnohorivka. [36]

Positional engagements continued south of Velyka Novosilka near Staromayorske and Urozhaine in the Donetsk-Zaporizhia Oblast border area on March 27. [37]

Russian Supporting Effort – Southern Axis (Russian objective: Maintain frontline positions and secure rear areas against Ukrainian strikes)

Positional engagements continued in western Zaporizhia Oblast on March 27, but there were no confirmed changes to the frontline. Positional engagements continued near Robotyne, near Mala Tokmachka (northeast of Robotyne), northeast of Novoprokopivka (south of Robotyne), and northwest of Verbove (east of Robotyne). [38] Elements of the Russian 71st Motorized Rifle Regiment (42nd Motorized Rifle Division, 58th Combined Arms Army [CAA], Southern Military District [SMD]) reportedly continue operating within Robotyne. [39]

Positional engagements continued in east (left) bank Kherson Oblast, including near Krynky, on March 27. [40]

Russian Air, Missile, and Drone Campaign (Russian Objective: Target Ukrainian military and civilian infrastructure in the rear and on the frontline)

Russian forces conducted a series of drone and missile strikes against Ukraine on the night of March 26 to 27 and on March 27. The Ukrainian Air Force reported that Russian forces launched 13 Shahed-136/131 drones from Kursk Oblast and that Ukrainian forces shot down 10 drones over Kharkiv, Sumy, and Kyiv oblasts on the night of March 26 to 27. [41] Ukrainian officials reported that Russian drones struck civilian infrastructure in Izyum, Kharkiv Oblast. [42] Ukrainian Kharkiv Oblast Head Oleh Synehubov stated that a Russian Kh-35U subsonic anti-ship cruise missile struck Kharkiv City on the morning of March 27. [43] Ukraine’s Eastern Air Command reported that Ukrainian forces shot down an unspecified Russian cruise missile over Dnipropetrovsk Oblast on March 27. [44] Ukrainian officials stated that Russian forces struck an industrial enterprise in Mykolaiv City with an Iskander-M ballistic missile on the afternoon of March 27. [45]

Ukraine’s Southern Operational Command Spokesperson Colonel Nataliya Humenyuk stated that Russian forces have stored “several dozen” Zircon missiles in military facilities in occupied Crimea. [46] Ukrainian Air Force Spokesperson Major Ilya Yevlash stated that Ukrainian air defense systems, such as Patriot and SAMP/T systems, can intercept Zircon missiles when they slow down to about 3,700 kilometers per hour on approach to a target. [47]

Russian Mobilization and Force Generation Efforts (Russian objective: Expand combat power without conducting general mobilization)

Russian Storm-Z personnel continue to complain about their poor treatment by the Russian Ministry of Defense (MoD) as the MoD tries to present the efficacy of its force generation and social benefit allocation system. Russian opposition outlet Mobilization News posted a video appeal from Storm-Z fighters from Kaluga Oblast on March 27 wherein one fighter claimed that after signing contracts with the Russian MoD, Russian command sent a Storm-Z unit of 230 people to the frontline, of whom only 38 survived combat. [48] The Storm-Z fighter complained that he has been unable to receive combat veteran status or promised payments from the Russian authorities for his service. [49] Mobilization News released another video on March 27 wherein relatives of killed and wounded Storm-Z fighters complain to Russian President Vladimir Putin that Russian authorities have not issued the Storm-Z fighters combat status or granted payments in the event of their death or injury in Ukraine. [50] The relatives of the Storm-Z fighters blamed the Russian MoD and Defense Minister Sergei Shoigu for the poor treatment and lack of benefits for Storm-Z fighters. The Russian MoD relies heavily on Storm-Z recruits from penal colonies to carry out costly infantry-led frontal assaults against Ukrainian positions and is very unlikely to address complaints concerning their poor treatment. The Russian MoD claimed on March 27 that it is issuing electronic combat veteran certificates and streamlining and digitizing the process for veterans to obtain payments and social benefits — but these privileges evidently do not apply evenly to all personnel who have signed contracts with the Russian MoD. [51]

Russian news outlet Vedemosti reported that US-sanctioned Russian company Baikal Electronics is struggling to domestically package semiconductor chips to produce processors and that over half of its domestically produced processors are defective. [52] Vedemosti reported that Baikal Electronics began to experiment with domestically packaging chips in Russia at the end of 2021 and that outdated equipment and a lack of experienced employees caused the large amount of processor defects.

Russian Technological Adaptations (Russian objective: Introduce technological innovations to optimize systems for use in Ukraine)

Russian drone developer Albatross LLC told Kremlin newswire TASS that Russian forces used the Albatross M5 long-range reconnaissance drones to guide aviation and artillery strikes while repelling recent pro-Ukrainian Russian raids into Belgorod Oblast. [53] Albatross LLC noted that the modernized Albatross M5 drone has a maximum range of 60-80 kilometers.

Russian state news outlet RIA Novosti reported that Russian T-72B3, T-72B3M, T-80BVM, and T-90M tanks operating in Ukraine use Reflex-M guided weapon systems with the Invar-M/M1 anti-tank guided missiles to strike Ukrainian and Western-made vehicles. [54]

Ukrainian Defense Industrial Efforts (Ukrainian objective: Develop its defense industrial base to become more self-sufficient in cooperation with US, European, and international partners)

ISW is not publishing coverage of Ukrainian defense industrial efforts today.

Activities in Russian-occupied areas (Russian objective: Consolidate administrative control of annexed areas; forcibly integrate Ukrainian citizens into Russian sociocultural, economic, military, and governance systems)

ISW is not publishing coverage of activities in Russian-occupied areas of Ukraine today.

Russian officials are weaponizing international responses to the Crocus City Hall attack to accuse the West of espousing Russophobic policies and to baselessly blame Ukraine of involvement in the attack. Russian Ambassador to Austria Dmitry Lyubinsky claimed on March 27 that while the Austrian government reacted to the Crocus City Hall attack, it did not use the words “terrorist attack” or condemn the attack. [55] Lyubinsky accused Austria of having “taken a very special position in its hypocrisy” and a “daze of permissiveness” towards Ukraine and reiterated the Kremlin narrative baselessly connecting Ukraine to the attack. Russian Foreign Ministry Spokesperson Maria Zakharova reported that Russia has received 24-hour non-stop words of support from around the globe following the attack, but immediately pivoted to accuse Ukraine of involvement in the attack and blame NATO members of monopolizing the global fight against terror. [56]

Significant activity in Belarus (Russian efforts to increase its military presence in Belarus and further integrate Belarus into Russian-favorable frameworks and Wagner Group activity in Belarus)

Nothing significant to report.

Note: ISW does not receive any classified material from any source, uses only publicly available information, and draws extensively on Russian, Ukrainian, and Western reporting and social media as well as commercially available satellite imagery and other geospatial data as the basis for these reports. References to all sources used are provided in the endnotes of each update.

[1] https://ukraine.un.org/sites/default/files/2024-03/2024-03-26%20OHCHR%2038th%20Periodic%20Report.pdf

[2] https://ukraine.un.org/en/264368-un-says-russia-continues-torture-execute-ukrainian-pows

[3] https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-20-2024 ; https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-18-2024 ; https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-10-2024 ; https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-january-3-2024 ; https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-20-2024 ; https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-december-27-2023

[4] https://www.ohchr.org/en/instruments-mechanisms/instruments/geneva-convention-relative-treatment-prisoners-war

[5] https://ukraine.un.org/sites/default/files/2024-03/2024-03-26%20OHCHR%2038th%20Periodic%20Report.pdf

[6] https://www.understandingwar.org/sites/default/files/24-210-01%20ISW%20Occupation%20playbook.pdf

[7] https://t.me/tass_agency/240300 ; https://t.me/astrapress/52521 ; https://t.me/tass_agency/240322

[8] https://ria dot ru/20240327/rassledovanie-1936142056.html ; https://meduza dot io/news/2024/03/27/deputaty-gosdumy-potrebovali-ot-sk-rassledovat-akty-terrorizma-kotorye-ssha-sovmestno-so-stranami-nato-i-spetssluzhbami-ukrainy-osuschestvlyayut-v-rossii

[9] https://isw.pub/UkrWar032324 ; https://isw.pub/UkrWar032424 ; https://isw.pub/UkrWar032524 ; https://isw.pub/UkrWar032624

[10] https://t.me/tass_agency/239253%C2%A0;%C2%A0https://isw.pub/UkrWar020624%C2%A0;%C2%A0https://isw.pub/UkrWar031824%C2%A0 ; https://www.reuters.com/world/europe/putin-calls-ukrainian-attack-belgorod-terrorism-promises-more-strikes-2024-01-01/ ; https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-march-23-2024 ; https://isw.pub/RusCampaignOct10

[11] https://isw.pub/UkrWar032324

[12] https://isw.pub/UkrWar032324

[13] https://t.me/bbcrussian/62850

[14] https://t.me/vchkogpu/47045

[15] https://t.me/bazabazon/26432 ; https://t.me/bazabazon/26440 ; https://meduza dot io/news/2024/03/27/politsiya-i-rosgvardiya-priehali-s-reydom-na-sklad-wildberries-v-podmoskovnoy-elektrostali-u-rabotnikov-proveryayut-dokumenty-nekotoryh-uvozyat-v-voenkomat ; https://t.me/tass_agency/240303 ; https://t.me/tass_agency/240290

[16] https://t.me/rybar/58588 ; https://t.me/notes_veterans/16295 ; https://t.me/historiographe/12011 ; https://t.me/voenacher/63252

[17] https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-march-26-2024 ; https://understandingwar.org/backgrounder/russian-offensive-campaign-assessment-march-24-2024

[18] https://www.facebook.com/GeneralStaff.ua/posts/pfbid02rxTJAPqhSGh5mqY7C4XDTQiRjiVX25K4Tmx6tT6GCypPhjw8tmKBZAmRa5jaETbGl ; https://www.facebook.com/GeneralStaff.ua/posts/pfbid02ReTBwNLG8czu42xB89ixKbv1WzZE2LqsgMcXwngSeHHpRjAXoaR3esPk1eCxZiZ8l ; https://t.me/mod_russia/37036 ; https://t.me/wargonzo/19025 ; https://t.me/luhanskaVTSA/17835 ; https://t.me/wargonzo/19025

[19] https://t.me/dva_majors/38313 ; https://t.me/DnevnikDesantnika/8702

[20] https://t.me/RKadyrov_95/4620

[21] https://suspilne dot media/714544-zelenskij-zminiv-sekretara-rnbo-zvit-oon-sodo-stracenih-ukrainskih-polonenih-763-den-vijni-onlajn/?anchor=live_1711553688&utm_source=copylink&utm_medium=ps ; https://armyinform dot com.ua/2024/03/27/boyeprypas-yakym-rosiyany-vdaryly-po-harkovu-mozhe-letity-na-vidstan-do-90-km-oleg-synyegubov/

[22] https://suspilne dot media/714544-zelenskij-zminiv-sekretara-rnbo-zvit-oon-sodo-stracenih-ukrainskih-polonenih-763-den-vijni-onlajn/?anchor=live_1711553688&utm_source=copylink&utm_medium=ps; https://armyinform dot com.ua/2024/03/27/boyeprypas-yakym-rosiyany-vdaryly-po-harkovu-mozhe-letity-na-vidstan-do-90-km-oleg-synyegubov/

[23] https://armyinform dot com.ua/2024/03/27/boyeprypas-yakym-rosiyany-vdaryly-po-harkovu-mozhe-letity-na-vidstan-do-90-km-oleg-synyegubov/

[24] https://isw.pub/UkrWar031024

[25] https://t.me/RVvoenkor/64758; https://t.me/basurin_e/10068 ; https://t.me/rusich_army/13845

[26] https://t.me/rusich_army/13845

[27] https://t.me/mod_russia/37029 ; https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-march-23-2024

[28] https://t.me/mod_russia/37044 ; https://t.me/mod_russia/37051 ; https://www.facebook.com/GeneralStaff.ua/posts/pfbid02Lh7wn9dDbMDZcCSUP4kHDoHuABYPPUB5vnfakuyQw21x2MKXQ1fcsLqAgYeuSQVWl ; https://www.facebook.com/GeneralStaff.ua/posts/pfbid02rxTJAPqhSGh5mqY7C4XDTQiRjiVX25K4Tmx6tT6GCypPhjw8tmKBZAmRa5jaETbGl; https://www.facebook.com/GeneralStaff.ua/posts/pfbid02ReTBwNLG8czu42xB89ixKbv1WzZE2LqsgMcXwngSeHHpRjAXoaR3esPk1eCxZiZ8l ; https://t.me/DnevnikDesantnika/8702 ; https://t.me/negumanitarnaya_pomosch_Z/16170 ; https://t.me/wargonzo/19025 ; https://t.me/rusich_army/13845 ;

[29] https://t.me/samotniyskhid/4868

[30] https://t.me/creamy_caprice/4888; https://t.me/kultshturmovika_ukraine/1773 ; https://t.me/creamy_caprice/4889; https://t.me/c/1595839251/3625; https://x.com/GeoConfirmed/status/1772981767139430744?s=20

[31] https://t.me/DnevnikDesantnika/8702 ; https://t.me/dva_majors/38373 ; https://t.me/negumanitarnaya_pomosch_Z/16183 ; https://t.me/DnevnikDesantnika/8724 ; https://t.me/rybar/58575

[32] https://t.me/DnevnikDesantnika/8720

[33] https://www.facebook.com/GeneralStaff.ua/posts/pfbid02rxTJAPqhSGh5mqY7C4XDTQiRjiVX25K4Tmx6tT6GCypPhjw8tmKBZAmRa5jaETbGl; https://www.facebook.com/GeneralStaff.ua/posts/pfbid02ReTBwNLG8czu42xB89ixKbv1WzZE2LqsgMcXwngSeHHpRjAXoaR3esPk1eCxZiZ8l ; https://www.facebook.com/GeneralStaff.ua/posts/pfbid02Lh7wn9dDbMDZcCSUP4kHDoHuABYPPUB5vnfakuyQw21x2MKXQ1fcsLqAgYeuSQVWl ; https://t.me/mod_russia/37044 ; https://t.me/mod_russia/37051 ; https://t.me/dva_majors/38313 ; https://t.me/DnevnikDesantnika/8720 ; https://t.me/DnevnikDesantnika/8702 ; https://t.me/wargonzo/19025 ; https://t.me/voenkorKotenok/55225

[34] https://t.me/tivaz_artillery/3650; https://t.me/creamy_caprice/4893

[35] https://www.facebook.com/GeneralStaff.ua/posts/pfbid02Lh7wn9dDbMDZcCSUP4kHDoHuABYPPUB5vnfakuyQw21x2MKXQ1fcsLqAgYeuSQVWl ; https://www.facebook.com/GeneralStaff.ua/posts/pfbid02rxTJAPqhSGh5mqY7C4XDTQiRjiVX25K4Tmx6tT6GCypPhjw8tmKBZAmRa5jaETbGl; https://www.facebook.com/GeneralStaff.ua/posts/pfbid02ReTBwNLG8czu42xB89ixKbv1WzZE2LqsgMcXwngSeHHpRjAXoaR3esPk1eCxZiZ8l ; https://t.me/dva_majors/38313 ; https://t.me/wargonzo/19025 ; https://t.me/boris_rozhin/118101 ; https://t.me/voenkorKotenok/55225

[36] https://t.me/boris_rozhin/118105

[37] https://www.facebook.com/GeneralStaff.ua/posts/pfbid02rxTJAPqhSGh5mqY7C4XDTQiRjiVX25K4Tmx6tT6GCypPhjw8tmKBZAmRa5jaETbGl; https://www.facebook.com/GeneralStaff.ua/posts/pfbid02ReTBwNLG8czu42xB89ixKbv1WzZE2LqsgMcXwngSeHHpRjAXoaR3esPk1eCxZiZ8l ; https://t.me/mod_russia/37044 ; https://t.me/mod_russia/37052 ; https://www.facebook.com/GeneralStaff.ua/posts/pfbid02Lh7wn9dDbMDZcCSUP4kHDoHuABYPPUB5vnfakuyQw21x2MKXQ1fcsLqAgYeuSQVWl

[38] https://www.facebook.com/GeneralStaff.ua/posts/pfbid02Lh7wn9dDbMDZcCSUP4kHDoHuABYPPUB5vnfakuyQw21x2MKXQ1fcsLqAgYeuSQVWl ; https://www.facebook.com/GeneralStaff.ua/posts/pfbid02rxTJAPqhSGh5mqY7C4XDTQiRjiVX25K4Tmx6tT6GCypPhjw8tmKBZAmRa5jaETbGl; https://www.facebook.com/GeneralStaff.ua/posts/pfbid02ReTBwNLG8czu42xB89ixKbv1WzZE2LqsgMcXwngSeHHpRjAXoaR3esPk1eCxZiZ8l ; https://t.me/SJTF_Odes/7591 ; https://t.me/rybar/58575 ; https://t.me/dva_majors/38313 ; https://t.me/DnevnikDesantnika/8715 ; https://t.me/DnevnikDesantnika/8692 ; https://t.me/wargonzo/19025

[39] https://t.me/batalyon15/4045

[40] https://www.facebook.com/GeneralStaff.ua/posts/pfbid02rxTJAPqhSGh5mqY7C4XDTQiRjiVX25K4Tmx6tT6GCypPhjw8tmKBZAmRa5jaETbGl; https://www.facebook.com/GeneralStaff.ua/posts/pfbid02ReTBwNLG8czu42xB89ixKbv1WzZE2LqsgMcXwngSeHHpRjAXoaR3esPk1eCxZiZ8l ; https://t.me/dva_majors/38313

[41] https://t.me/kpszsu/12330

[42] https://t.me/pgo_gov_ua/22717 ; https://armyinform.com dot ua/2024/03/27/vijska-rf-atakuvaly-izyum-shahedamy-poshkodzheno-gimnaziyu-poraneno-ohoronczya/ ; https://t.me/synegubov/8827?single

[43] https://t.me/synegubov/8827

[44] https://www.facebook.com/pvkshid/posts/pfbid0LGmUtBDdzmxud8zZ23FDoN8eKarYJkLS6YrsSUzB62HVo7uSrXWhxPxnnzAhuSUyl

[45] https://t.me/mykolaivskaODA/8840 ; https://t.me/dsns_mykolaiv/4948 ; https://t.me/SJTF_Odes/7600

[46] https://armyinform.com dot ua/2024/03/27/u-sylah-oborony-povidomyly-pro-kilkist-rosijskyh-czyrkoniv-u-krymu/

[47] https://armyinform.com dot ua/2024/03/27/u-povitryanyh-sylah-povidomyly-pro-sposoby-zbyttya-rosijskyh-czyrkoniv/

[48] https://t.me/mobilizationnews/18111

[49] https://t.me/mobilizationnews/18111

[50] https://t.me/mobilizationnews/18114

[51] https://t.me/mod_russia/37031

[52] https://www.severreal.org/a/bolshe-poloviny-rossiyskih-protsessorov-baykal-okazalis-brakovannymi/32879476.html ; https://www.vedomosti dot ru/technology/articles/2024/03/26/1027924-razrabotchik-protsessorov-baikal-lokalizuet-odin-iz-etapov-proizvodstva

[53] https://t.me/tass_agency/240240 ; https://t.me/tass_agency/240241 ; https://t.me/tass_agency/240268

[54] https://ria dot ru/20240327/rakety-1936068479.html

[55] https://t.me/RusBotWien_RU/4869

[56] https://t.me/MID_Russia/38112

Reading List

{{currentView.title}}

{{nextView.title}}

Get Notified

Currently receiving 0 of 2 possible notifications for this sort of content.

• LOCATION Ukraine
• LOCATION Russia

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Bras Pneumol
• v.44(6); Nov-Dec 2018

Critical appraisal of the literature. Why do we care?

Avaliação crítica da literatura. por que nos importamos, juliana carvalho ferreira.

1 . Methods in Epidemiologic, Clinical, and Operations Research-MECOR-program, American Thoracic Society/Asociación Latinoamericana del Tórax, Montevideo, Uruguay.

2 . Divisão de Pneumologia, Instituto do Coração, Hospital das Clínicas, Faculdade de Medicina, Universidade de São Paulo, São Paulo (SP) Brasil.

Cecilia Maria Patino

3 . Department of Preventive Medicine, Keck School of Medicine, University of Southern California, Los Angeles, CA, USA.

PRACTICAL SCENARIO

Investigators conducted a noninferiority, double-blind clinical trial involving 4,215 patients with mild asthma, randomly assigned to receive twice-daily placebo plus budesonide-formoterol used as needed vs. maintenance therapy with twice-daily budesonide plus terbutaline as needed. They found that budesonide-formoterol used as needed was noninferior to twice-daily budesonide concerning the rate of severe asthma exacerbations but was inferior in controlling symptoms. 1

HOW TO CRITICALLY APPRAISE THE MEDICAL LITERATURE

As clinicians, when we read a paper reporting the benefit of a given intervention, we make a judgment regarding whether we should use those results to inform how we care for our patients. In our example, after reading the paper, we ask ourselves: should a clinician working in a public hospital in Brazil start prescribing budesonide-formoterol as needed rather than maintenance budesonide for her patients with mild asthma? What criteria should guide her decision to adopt a new intervention? One may think that if a study is published in a high-impact, peer-reviewed journal, it is of high quality and should therefore be used to guide clinical decision making. However, if the population included in the study or the context is different from her population, that may not be the case. Therefore, examining the external validity of a study is critical to informing local practice.

Other commonly used criteria are related to evaluating the quality of the evidence by evaluating the type of study design used. The pyramid of evidence puts meta-analyses at the top (as providing the highest quality of evidence), followed by systematic reviews and randomized controlled trials; then come observational studies (cohort, case-control, and cross-sectional studies); whereas case reports and case series are categorized as offering the lowest quality of evidence. Although those criteria may be helpful, making a detailed appraisal of a paper, taking into account aspects other than the study design, is a skill that researchers and clinicians can learn and apply when reading the literature.

Critical appraisal is the systematic evaluation of clinical research papers that helps us establish if the results are valid and if they could be used to inform medical decision in a given local population and context. There are several published guidelines for critically appraising the scientific literature, most of which are structured as checklists and address specific study designs. 2 Although different appraisal tools may vary, the general structure is shown in Table 1 .

The items in Table 1 are a guide to appraising the content of a research article. There are also guidelines for appraising the quality of reporting of health research which focus on the reporting accuracy and completeness of research studies. 3 These two types of appraisal (content and reporting) are complementary and should both be used, because it is possible that a research paper has high reporting quality but is not relevant to the context in question.

KEY MESSAGE

Critical appraisal of the literature is an essential skill for researchers and clinicians, and there are easy-to-use guidelines. Clinicians have the responsibility to help patients make health-related decisions, which should be based on high-quality, valid research that is applicable in their context.