critical appraisal of nursing research example

Your web browser is outdated and may be insecure

The RCN recommends using an updated browser such as Microsoft Edge or Google Chrome

Critical Appraisal

Use this guide to find information resources about critical appraisal including checklists, books and journal articles.

Key Resources

• This online resource explains the sections commonly used in research articles. Understanding how research articles are organised can make reading and evaluating them easier View page
• Critical appraisal checklists
• Worksheets for appraising systematic reviews, diagnostics, prognostics and RCTs. View page
• A free online resource for both healthcare staff and patients; four modules of 30–45 minutes provide an introduction to evidence based medicine, clinical trials and Cochrane Evidence. View page
• This tool will guide you through a series of questions to help you to review and interpret a published health research paper. View page
• The PRISMA flow diagram depicts the flow of information through the different phases of a literature review. It maps out the number of records identified, included and excluded, and the reasons for exclusions. View page
• A useful resource for methods and evidence in applied social science. View page
• A comprehensive database of reporting guidelines. Covers all the main study types. View page
• A tool to assess the methodological quality of systematic reviews. View page

• Borrow from RCN Library services

• Chapter 5 covers critical appraisal of the literature. View this eBook

• Chapter 6 covers assessing the evidence base. Borrow from RCN Library services

• Section 1 covers an introduction to critical appraisal. Section 3 covers appraising difference types of papers including qualitative papers and observational studies. View this eBook

• Chapter 6 covers critically appraising the literature. Borrow from RCN Library services

• View this eBook

• Chapter 8 covers critical appraisal of the evidence. View this eBook

• Chapter 18 covers critical appraisal of nursing studies. View this eBook

• Borrow from RCN Library Services

Book subject search

• Critical appraisal

Journal articles

• View article

Shea BJ and others (2017) AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions or both, British Medical Journal, 358.

• An outline of AMSTAR 2 and its use for as a critical appraisal tool for systematic reviews. View article (open access)
• View articles

Editor of this guide

RCN Library and Archive Service

Upcoming events relating to this subject guide

Know How to Search CINAHL

Learn about using the CINAHL database for literature searches at this event for RCN members.

Library Search ... in 30 minutes

Learn how RCN members can quickly and easily search for articles, books and more using our fantastic and easy to use Library Search tool.

Know How to Reference Accurately and Avoid Plagiarism

Learn how to use the Harvard refencing style and why referencing is important at this event for RCN members.

Easy referencing ... in 30 minutes

Learn how to generate quick references and citations using free, easy to use, online tools.

Page last updated - 08/02/2024

Your Spaces

• RCNi Profile
• RCN Starting Out
• Steward Portal
• Careers at the RCN
• RCN Foundation
• RCN Library

Work & Venue

• RCNi Nursing Jobs
• Work for the RCN
• RCN Working with us

Further Info

• Manage Cookie Preferences
• Modern slavery statement
• Accessibility
• Press office

Connect with us:

© 2024 Royal College of Nursing

Fastest Nurse Insight Engine

• MEDICAL ASSISSTANT
• Abdominal Key
• Anesthesia Key
• Basicmedical Key
• Otolaryngology & Ophthalmology
• Musculoskeletal Key
• Obstetric, Gynecology and Pediatric
• Oncology & Hematology
• Plastic Surgery & Dermatology
• Clinical Dentistry
• Radiology Key
• Thoracic Key
• Veterinary Medicine
• Gold Membership

Critical Appraisal of Quantitative and Qualitative Research for Nursing Practice

Chapter 12 Critical Appraisal of Quantitative and Qualitative Research for Nursing Practice Chapter Overview When Are Critical Appraisals of Studies Implemented in Nursing? Students’ Critical Appraisal of Studies Critical Appraisal of Studies by Practicing Nurses, Nurse Educators, and Researchers Critical Appraisal of Research Following Presentation and Publication Critical Appraisal of Research for Presentation and Publication Critical Appraisal of Research Proposals What Are the Key Principles for Conducting Intellectual Critical Appraisals of Quantitative and Qualitative Studies? Understanding the Quantitative Research Critical Appraisal Process Step 1: Identifying the Steps of the Research Process in Studies Step 2: Determining the Strengths and Weaknesses in Studies Step 3: Evaluating the Credibility and Meaning of Study Findings Example of a Critical Appraisal of a Quantitative Study Understanding the Qualitative Research Critical Appraisal Process Step 1: Identifying the Components of the Qualitative Research Process in Studies Step 2: Determining the Strengths and Weaknesses in Studies Step 3: Evaluating the Trustworthiness and Meaning of Study Finding Example of a Critical Appraisal of a Qualitative Study Key Concepts References Learning Outcomes After completing this chapter, you should be able to: 1.  Describe when intellectual critical appraisals of studies are conducted in nursing. 2.  Implement key principles in critically appraising quantitative and qualitative studies. 3.  Describe the three steps for critically appraising a study: (1) identifying the steps of the research process in the study; (2) determining study strengths and weaknesses; and (3) evaluating the credibility and meaning of the study findings. 4.  Conduct a critical appraisal of a quantitative research report. 5.  Conduct a critical appraisal of a qualitative research report. Key Terms Confirmability, p. 392 Credibility, p. 392 Critical appraisal, p. 362 Critical appraisal of qualitative studies, p. 389 Critical appraisal of quantitative studies, p. 362 Dependability, p. 392 Determining strengths and weaknesses in the studies, p. 370 Evaluating the credibility and meaning of study findings, p. 374 Identifying the steps of the research process in studies, p. 366 Intellectual critical appraisal of a study, p. 365 Qualitative research critical appraisal process, p. 389 Quantitative research critical appraisal process, p. 366 Referred journals, p. 364 Transferable, p. 392 Trustworthiness, p. 392 The nursing profession continually strives for evidence-based practice (EBP), which includes critically appraising studies, synthesizing the findings, applying the scientific evidence in practice, and determining the practice outcomes ( Brown, 2014 ; Doran, 2011 ; Melnyk & Fineout-Overholt, 2011 ). Critically appraising studies is an essential step toward basing your practice on current research findings. The term critical appraisal or critique is an examination of the quality of a study to determine the credibility and meaning of the findings for nursing. Critique is often associated with criticize, a word that is frequently viewed as negative. In the arts and sciences, however, critique is associated with critical thinking and evaluation—tasks requiring carefully developed intellectual skills. This type of critique is referred to as an intellectual critical appraisal. An intellectual critical appraisal is directed at the element that is created, such as a study, rather than at the creator, and involves the evaluation of the quality of that element. For example, it is possible to conduct an intellectual critical appraisal of a work of art, an essay, and a study. The idea of the intellectual critical appraisal of research was introduced earlier in this text and has been woven throughout the chapters. As each step of the research process was introduced, guidelines were provided to direct the critical appraisal of that aspect of a research report. This chapter summarizes and builds on previous critical appraisal content and provides direction for conducting critical appraisals of quantitative and qualitative studies. The background provided by this chapter serves as a foundation for the critical appraisal of research syntheses (systematic reviews, meta-analyses, meta-syntheses, and mixed-methods systematic reviews) presented in Chapter 13. This chapter discusses the implementation of critical appraisals in nursing by students, practicing nurses, nurse educators, and researchers. The key principles for implementing intellectual critical appraisals of quantitative and qualitative studies are described to provide an overview of the critical appraisal process. The steps for critical appraisal of quantitative studies , focused on rigor, design validity, quality, and meaning of findings, are detailed, and an example of a critical appraisal of a published quantitative study is provided. The chapter concludes with the critical appraisal process for qualitative studies and an example of a critical appraisal of a qualitative study. When are Critical Appraisals of Studies Implemented in Nursing? In general, studies are critically appraised to broaden understanding, summarize knowledge for practice, and provide a knowledge base for future research. Studies are critically appraised for class projects and to determine the research evidence ready for use in practice. In addition, critical appraisals are often conducted after verbal presentations of studies, after a published research report, for selection of abstracts when studies are presented at conferences, for article selection for publication, and for evaluation of research proposals for implementation or funding. Therefore nursing students, practicing nurses, nurse educators, and nurse researchers are all involved in the critical appraisal of studies. Students’ Critical Appraisal of Studies One aspect of learning the research process is being able to read and comprehend published research reports. However, conducting a critical appraisal of a study is not a basic skill, and the content presented in previous chapters is essential for implementing this process. Students usually acquire basic knowledge of the research process and critical appraisal process in their baccalaureate program. More advanced analysis skills are often taught at the master’s and doctoral levels. Performing a critical appraisal of a study involves the following three steps, which are detailed in this chapter: (1) identifying the steps or elements of the study; (2) determining the study strengths and limitations; and (3) evaluating the credibility and meaning of the study findings. By critically appraising studies, you will expand your analysis skills, strengthen your knowledge base, and increase your use of research evidence in practice. Striving for EBP is one of the competencies identified for associate degree and baccalaureate degree (prelicensure) students by the Quality and Safety Education for Nurses ( QSEN, 2013 ) project, and EBP requires critical appraisal and synthesis of study findings for practice ( Sherwood & Barnsteiner, 2012 ). Therefore critical appraisal of studies is an important part of your education and your practice as a nurse. Critical Appraisal of Studies by Practicing Nurses, Nurse Educators, and Researchers Practicing nurses need to appraise studies critically so that their practice is based on current research evidence and not on tradition or trial and error ( Brown, 2014 ; Craig & Smyth, 2012 ). Nursing actions need to be updated in response to current evidence that is generated through research and theory development. It is important for practicing nurses to design methods for remaining current in their practice areas. Reading research journals and posting or e-mailing current studies at work can increase nurses’ awareness of study findings but are not sufficient for critical appraisal to occur. Nurses need to question the quality of the studies, credibility of the findings, and meaning of the findings for practice. For example, nurses might form a research journal club in which studies are presented and critically appraised by members of the group ( Gloeckner & Robinson, 2010 ). Skills in critical appraisal of research enable practicing nurses to synthesize the most credible, significant, and appropriate evidence for use in their practice. EBP is essential in agencies that are seeking or maintaining Magnet status. The Magnet Recognition Program was developed by the American Nurses Credentialing Center ( ANCC, 2013 ) to “recognize healthcare organizations for quality patient care, nursing excellence, and innovations in professional nursing,” which requires implementing the most current research evidence in practice (see http://www.nursecredentialing.org/Magnet/ProgramOverview.aspx ). Your faculty members critically appraise research to expand their clinical knowledge base and to develop and refine the nursing educational process. The careful analysis of current nursing studies provides a basis for updating curriculum content for use in clinical and classroom settings. Faculty serve as role models for their students by examining new studies, evaluating the information obtained from research, and indicating which research evidence to use in practice. For example, nursing instructors might critically appraise and present the most current evidence about caring for people with hypertension in class and role-model the management of patients with hypertension in practice. Nurse researchers critically appraise previous research to plan and implement their next study. Many researchers have a program of research in a selected area, and they update their knowledge base by critically appraising new studies in this area. For example, selected nurse researchers have a program of research to identify effective interventions for assisting patients in managing their hypertension and reducing their cardiovascular risk factors. Critical Appraisal of Research Following Presentation and Publication When nurses attend research conferences, they note that critical appraisals and questions often follow presentations of studies. These critical appraisals assist researchers in identifying the strengths and weaknesses of their studies and generating ideas for further research. Participants listening to study critiques might gain insight into the conduct of research. In addition, experiencing the critical appraisal process can increase the conference participants’ ability to evaluate studies and judge the usefulness of the research evidence for practice. Critical appraisals have been published following some studies in research journals. For example, the research journals Scholarly Inquiry for Nursing Practice: An International Journal and Western Journal of Nursing Research include commentaries after the research articles. In these commentaries, other researchers critically appraise the authors’ studies, and the authors have a chance to respond to these comments. Published research critical appraisals often increase the reader’s understanding of the study and the quality of the study findings ( American Psychological Association [APA], 2010 ). A more informal critical appraisal of a published study might appear in a letter to the editor. Readers have the opportunity to comment on the strengths and weaknesses of published studies by writing to the journal editor. Critical Appraisal of Research for Presentation and Publication Planners of professional conferences often invite researchers to submit an abstract of a study they are conducting or have completed for potential presentation at the conference. The amount of information available is usually limited, because many abstracts are restricted to 100 to 250 words. Nevertheless, reviewers must select the best-designed studies with the most significant outcomes for presentation at nursing conferences. This process requires an experienced researcher who needs few cues to determine the quality of a study. Critical appraisal of an abstract usually addresses the following criteria: (1) appropriateness of the study for the conference program; (2) completeness of the research project; (3) overall quality of the study problem, purpose, methodology, and results; (4) contribution of the study to nursing’s knowledge base; (5) contribution of the study to nursing theory; (6) originality of the work (not previously published); (7) implication of the study findings for practice; and (8) clarity, conciseness, and completeness of the abstract ( APA, 2010 ; Grove, Burns, & Gray, 2013 ). Some nurse researchers serve as peer reviewers for professional journals to evaluate the quality of research papers submitted for publication. The role of these scientists is to ensure that the studies accepted for publication are well designed and contribute to the body of knowledge. Journals that have their articles critically appraised by expert peer reviews are called peer-reviewed journals or referred journals ( Pyrczak, 2008 ). The reviewers’ comments or summaries of their comments are sent to the researchers to direct their revision of the manuscripts for publication. Referred journals usually have studies and articles of higher quality and provide excellent studies for your review for practice. Critical Appraisal of Research Proposals Critical appraisals of research proposals are conducted to approve student research projects, permit data collection in an institution, and select the best studies for funding by local, state, national, and international organizations and agencies. You might be involved in a proposal review if you are participating in collecting data as part of a class project or studies done in your clinical agency. More details on proposal development and approval can be found in Grove et al. (2013, Chapter 28) . Research proposals are reviewed for funding from selected government agencies and corporations. Private corporations develop their own format for reviewing and funding research projects ( Grove et al., 2013 ). The peer review process in federal funding agencies involves an extremely complex critical appraisal. Nurses are involved in this level of research review through national funding agencies, such as the National Institute of Nursing Research ( NINR, 2013 ) and the Agency for Healthcare Research and Quality ( AHRQ, 2013 ). What are the Key Principles for Conducting Intellectual Critical Appraisals of Quantitative and Qualitative Studies? An intellectual critical appraisal of a study involves a careful and complete examination of a study to judge its strengths, weaknesses, credibility, meaning, and significance for practice. A high-quality study focuses on a significant problem, demonstrates sound methodology, produces credible findings, indicates implications for practice, and provides a basis for additional studies ( Grove et al., 2013 ; Hoare & Hoe, 2013 ; Hoe & Hoare, 2012 ). Ultimately, the findings from several quality studies can be synthesized to provide empirical evidence for use in practice ( O’Mathuna, Fineout-Overholt, & Johnston, 2011 ). The major focus of this chapter is conducting critical appraisals of quantitative and qualitative studies. These critical appraisals involve implementing some key principles or guidelines, outlined in Box 12-1 . These guidelines stress the importance of examining the expertise of the authors, reviewing the entire study, addressing the study’s strengths and weaknesses, and evaluating the credibility of the study findings ( Fawcett & Garity, 2009 ; Hoare & Hoe, 2013 ; Hoe & Hoare, 2012 ; Munhall, 2012 ). All studies have weaknesses or flaws; if every flawed study were discarded, no scientific evidence would be available for use in practice. In fact, science itself is flawed. Science does not completely or perfectly describe, explain, predict, or control reality. However, improved understanding and increased ability to predict and control phenomena depend on recognizing the flaws in studies and science. Additional studies can then be planned to minimize the weaknesses of earlier studies. You also need to recognize a study’s strengths to determine the quality of a study and credibility of its findings. When identifying a study’s strengths and weaknesses, you need to provide examples and rationale for your judgments that are documented with current literature. Box 12-1    Key Principles for Critically Appraising Quantitative and Qualitative Studies 1.  Read and critically appraise the entire study. A research critical appraisal involves examining the quality of all aspects of the research report. 2.  Examine the organization and presentation of the research report. A well-prepared report is complete, concise, clearly presented, and logically organized. It does not include excessive jargon that is difficult for you to read. The references need to be current, complete, and presented in a consistent format. 3.  Examine the significance of the problem studied for nursing practice. The focus of nursing studies needs to be on significant practice problems if a sound knowledge base is to be developed for evidence-based nursing practice. 4.  Indicate the type of study conducted and identify the steps or elements of the study. This might be done as an initial critical appraisal of a study; it indicates your knowledge of the different types of quantitative and qualitative studies and the steps or elements included in these studies. 5.  Identify the strengths and weaknesses of a study. All studies have strengths and weaknesses, so attention must be given to all aspects of the study. 6.  Be objective and realistic in identifying the study’s strengths and weaknesses. Be balanced in your critical appraisal of a study. Try not to be overly critical in identifying a study’s weaknesses or overly flattering in identifying the strengths. 7.  Provide specific examples of the strengths and weaknesses of a study. Examples provide evidence for your critical appraisal of the strengths and weaknesses of a study. 8.  Provide a rationale for your critical appraisal comments. Include justifications for your critical appraisal, and document your ideas with sources from the current literature. This strengthens the quality of your critical appraisal and documents the use of critical thinking skills. 9.  Evaluate the quality of the study. Describe the credibility of the findings, consistency of the findings with those from other studies, and quality of the study conclusions. 10.  Discuss the usefulness of the findings for practice. The findings from the study need to be linked to the findings of previous studies and examined for use in clinical practice. Critical appraisal of quantitative and qualitative studies involves a final evaluation to determine the credibility of the study findings and any implications for practice and further research (see Box 12-1 ). Adding together the strong points from multiple studies slowly builds a solid base of evidence for practice. These guidelines provide a basis for the critical appraisal process for quantitative research discussed in the next section and the critical appraisal process for qualitative research (see later). Understanding the Quantitative Research Critical Appraisal Process The quantitative research critical appraisal process includes three steps: (1) identifying the steps of the research process in studies; (2) determining study strengths and weaknesses; and (3) evaluating the credibility and meaning of study findings. These steps occur in sequence, vary in depth, and presume accomplishment of the preceding steps. However, an individual with critical appraisal experience frequently performs two or three steps of this process simultaneously. This section includes the three steps of the quantitative research critical appraisal process and provides relevant questions for each step. These questions have been selected as a means for stimulating the logical reasoning and analysis necessary for conducting a critical appraisal of a study. Those experienced in the critical appraisal process often formulate additional questions as part of their reasoning processes. We will identify the steps of the research process separately because those new to critical appraisal start with this step. The questions for determining the study strengths and weaknesses are covered together because this process occurs simultaneously in the mind of the person conducting the critical appraisal. Evaluation is covered separately because of the increased expertise needed to perform this step. Step 1: Identifying the Steps of the Research Process in Studies Initial attempts to comprehend research articles are often frustrating because the terminology and stylized manner of the report are unfamiliar. Identifying the steps of the research process in a quantitative study is the first step in critical appraisal. It involves understanding the terms and concepts in the report, as well as identifying study elements and grasping the nature, significance, and meaning of these elements. The following guidelines will direct you in identifying a study’s elements or steps. Guidelines for Identifying the Steps of the Research Process in Studies The first step involves reviewing the abstract and reading the study from beginning to end. As you read, think about the following questions regarding the presentation of the study: •  Was the study title clear? •  Was the abstract clearly presented? •  Was the writing style of the report clear and concise? •  Were relevant terms defined? You might underline the terms you do not understand and determine their meaning from the glossary at the end of this text. •  Were the following parts of the research report plainly identified ( APA, 2010 )? •   Introduction section, with the problem, purpose, literature review, framework, study variables, and objectives, questions, or hypotheses •   Methods section, with the design, sample, intervention (if applicable), measurement methods, and data collection or procedures •   Results section, with the specific results presented in tables, figures, and narrative •   Discussion section, with the findings, conclusions, limitations, generalizations, implications for practice, and suggestions for future research ( Fawcett & Garity, 2009 ; Grove et al., 2013 ) We recommend reading the research article a second time and highlighting or underlining the steps of the quantitative research process that were identified previously. An overview of these steps is presented in Chapter 2 . After reading and comprehending the content of the study, you are ready to write your initial critical appraisal of the study. To write a critical appraisal identifying the study steps, you need to identify each step of the research process concisely and respond briefly to the following guidelines and questions. 1.  Introduction a.  Describe the qualifications of the authors to conduct the study (e.g., research expertise conducting previous studies, clinical experience indicated by job, national certification, and years in practice, and educational preparation that includes conducting research [PhD]). b.  Discuss the clarity of the article title. Is the title clearly focused and does it include key study variables and population? Does the title indicate the type of study conducted—descriptive, correlational, quasi-experimental, or experimental—and the variables ( Fawcett & Garity, 2009 ; Hoe & Hoare, 2012 ; Shadish, Cook, & Campbell, 2002 )? c.  Discuss the quality of the abstract (includes purpose, highlights design, sample, and intervention [if applicable], and presents key results; APA, 2010 ). 2.  State the problem. a.  Significance of the problem b.  Background of the problem c.  Problem statement 3.  State the purpose. 4.  Examine the literature review. a.  Are relevant previous studies and theories described? b.  Are the references current (number and percentage of sources in the last 5 and 10 years)? c.  Are the studies described, critically appraised, and synthesized ( Brown, 2014 ; Fawcett & Garity, 2009 )? Are the studies from referred journals? d.  Is a summary provided of the current knowledge (what is known and not known) about the research problem? 5.  Examine the study framework or theoretical perspective. a.  Is the framework explicitly expressed, or must you extract the framework from statements in the introduction or literature review of the study? b.  Is the framework based on tentative, substantive, or scientific theory? Provide a rationale for your answer. c.  Does the framework identify, define, and describe the relationships among the concepts of interest? Provide examples of this. d.  Is a map of the framework provided for clarity? If a map is not presented, develop a map that represents the study’s framework and describe the map. e.  Link the study variables to the relevant concepts in the map. f.  How is the framework related to nursing’s body of knowledge ( Alligood, 2010 ; Fawcett & Garity, 2009 ; Smith & Liehr, 2008 )? 6.  List any research objectives, questions, or hypotheses. 7.  Identify and define (conceptually and operationally) the study variables or concepts that were identified in the objectives, questions, or hypotheses. If objectives, questions, or hypotheses are not stated, identify and define the variables in the study purpose and results section of the study. If conceptual definitions are not found, identify possible definitions for each major study variable. Indicate which of the following types of variables were included in the study. A study usually includes independent and dependent variables or research variables, but not all three types of variables. a.  Independent variables: Identify and define conceptually and operationally. b.  Dependent variables: Identify and define conceptually and operationally. c.  Research variables or concepts: Identify and define conceptually and operationally. 8.  Identify attribute or demographic variables and other relevant terms. 9.  Identify the research design. a.  Identify the specific design of the study (see Chapter 8 ). b.  Does the study include a treatment or intervention? If so, is the treatment clearly described with a protocol and consistently implemented? c.  If the study has more than one group, how were subjects assigned to groups? d.  Are extraneous variables identified and controlled? Extraneous variables are usually discussed as a part of quasi-experimental and experimental studies. e.  Were pilot study findings used to design this study? If yes, briefly discuss the pilot and the changes made in this study based on the pilot ( Grove et al., 2013 ; Shadish et al., 2002 ). 10.  Describe the sample and setting. a.  Identify inclusion and exclusion sample or eligibility criteria. b.  Identify the specific type of probability or nonprobability sampling method that was used to obtain the sample. Did the researchers identify the sampling frame for the study? c.  Identify the sample size. Discuss the refusal number and percentage, and include the rationale for refusal if presented in the article. Discuss the power analysis if this process was used to determine sample size ( Aberson, 2010 ). d.  Identify the sample attrition (number and percentage) for the study. e.  Identify the characteristics of the sample. f.  Discuss the institutional review board (IRB) approval. Describe the informed consent process used in the study. g.  Identify the study setting and indicate whether it is appropriate for the study purpose. 11.  Identify and describe each measurement strategy used in the study. The following table includes the critical information about two measurement methods, the Beck Likert scale and a physiological instrument to measure blood pressure. Completing this table will allow you to cover essential measurement content for a study ( Waltz, Strickland, & Lenz, 2010 ). a.  Identify each study variable that was measured. b.  Identify the name and author of each measurement strategy. c.  Identify the type of each measurement strategy (e.g., Likert scale, visual analog scale, physiological measure, or existing database). d.  Identify the level of measurement (nominal, ordinal, interval, or ratio) achieved by each measurement method used in the study ( Grove, 2007 ). e.  Describe the reliability of each scale for previous studies and this study. Identify the precision of each physiological measure ( Bialocerkowski, Klupp, & Bragge, 2010 ; DeVon et al., 2007 ). f.  Identify the validity of each scale and the accuracy of physiological measures ( DeVon et al., 2007 ; Ryan-Wenger, 2010 ). Variable Measured Name of Measurement Method (Author) Type of Measurement Method Level of Measurement Reliability or Precision Validity or Accuracy Depression Beck Depression Inventory (Beck) Likert scale Interval Cronbach alphas of 0.82-0.92 from previous studies and 0.84 for this study; reading level at 6th grade. Construct validity—content validity from concept analysis, literature review, and reviews of experts; convergent validity of 0.04 with Zung Depression Scale; predictive validity of patients’ future depression episodes; successive use validity with the conduct of previous studies and this study. Blood pressure Omron blood pressure (BP) equipment (equipment manufacturer) Physiological measurement method Ratio Test-retest values of BPs in previous studies; BP equipment new and recalibrated every 50 BP readings in this study; average three BP readings to determine BP. Documented accuracy of systolic and diastolic BPs to 1 mm Hg by company developing Omron BP cuff; designated protocol for taking BP average three BP readings to determine BP. 12.  Describe the procedures for data collection. 13.  Describe the statistical analyses used. a.  List the statistical procedures used to describe the sample ( Grove, 2007 ). b.  Was the level of significance or alpha identified? If so, indicate what it was (0.05, 0.01, or 0.001). c.  Complete the following table with the analysis techniques conducted in the study: (1) identify the focus (description, relationships, or differences) for each analysis technique; (2) list the statistical analysis technique performed; (3) list the statistic; (4) provide the specific results; and (5) identify the probability ( p ) of the statistical significance achieved by the result ( Grove, 2007 ; Grove et al., 2013 ; Hoare & Hoe, 2013 ; Plichta & Kelvin, 2013 ). Purpose of Analysis Analysis Technique Statistic Results Probability (p) Description of subjects’ pulse rate Mean Standard deviation Range M SD range 71.52 5.62 58-97   Difference between adult males and females on blood pressure t -Test t 3.75 p  = 0.001 Differences of diet group, exercise group, and comparison group for pounds lost in adolescents Analysis of variance F 4.27 p  = 0.04 Relationship of depression and anxiety in older adults Pearson correlation r 0.46 p  = 0.03 14.  Describe the researcher’s interpretation of findings. a.  Are the findings related back to the study framework? If so, do the findings support the study framework? b.  Which findings are consistent with those expected? c.  Which findings were not expected? d.  Are the findings consistent with previous research findings? ( Fawcett & Garity, 2009 ; Grove et al., 2013 ; Hoare & Hoe, 2013 ) 15.  What study limitations did the researcher identify? 16.  What conclusions did the researchers identify based on their interpretation of the study findings? 17.  How did the researcher generalize the findings? 18.  What were the implications of the findings for nursing practice? 19.  What suggestions for further study were identified? 20.  Is the description of the study sufficiently clear for replication? Step 2: Determining the Strengths and Weaknesses in Studies The second step in critically appraising studies requires determining strengths and weaknesses in the studies . To do this, you must have knowledge of what each step of the research process should be like from expert sources such as this text and other research sources ( Aberson, 2010 ; Bialocerkowski et al., 2010 ; Brown, 2014 ; Creswell, 2014 ; DeVon et al., 2007 ; Doran, 2011 ; Fawcett & Garity, 2009 ; Grove, 2007 ; Grove et al., 2013 ; Hoare & Hoe, 2013 ; Hoe & Hoare, 2012 ; Morrison, Hoppe, Gillmore, Kluver, Higa, & Wells, 2009 ; O’Mathuna et al., 2011 ; Ryan-Wenger, 2010 ; Santacroce, Maccarelli, & Grey, 2004 ; Shadish et al., 2002 ; Waltz et al., 2010 ). The ideal ways to conduct the steps of the research process are then compared with the actual study steps. During this comparison, you examine the extent to which the researcher followed the rules for an ideal study, and the study elements are examined for strengths and weaknesses. You also need to examine the logical links or flow of the steps in the study being appraised. For example, the problem needs to provide background and direction for the statement of the purpose. The variables identified in the study purpose need to be consistent with the variables identified in the research objectives, questions, or hypotheses. The variables identified in the research objectives, questions, or hypotheses need to be conceptually defined in light of the study framework. The conceptual definitions should provide the basis for the development of operational definitions. The study design and analyses need to be appropriate for the investigation of the study purpose, as well as for the specific objectives, questions, or hypotheses. Examining the quality and logical links among the study steps will enable you to determine which steps are strengths and which steps are weaknesses. Guidelines for Determining the Strengths and Weaknesses in Studies The following questions were developed to help you examine the different steps of a study and determine its strengths and weaknesses. The intent is not for you to answer each of these questions but to read the questions and then make judgments about the steps in the study. You need to provide a rationale for your decisions and document from relevant research sources, such as those listed previously in this section and in the references at the end of this chapter. For example, you might decide that the study purpose is a strength because it addresses the study problem, clarifies the focus of the study, and is feasible to investigate ( Brown, 2014 ; Fawcett & Garity, 2009 ; Hoe & Hoare, 2012 ). 1.  Research problem and purpose a.  Is the problem significant to nursing and clinical practice ( Brown, 2014 )? b.  Does the purpose narrow and clarify the focus of the study ( Creswell, 2014 ; Fawcett & Garity, 2009 )? c.  Was this study feasible to conduct in terms of money commitment, the researchers’ expertise; availability of subjects, facilities, and equipment; and ethical considerations? 2.  Review of literature a.  Is the literature review organized to demonstrate the progressive development of evidence from previous research ( Brown, 2014 ; Creswell, 2014 ; Hoe & Hoare, 2012 )? b.  Is a clear and concise summary presented of the current empirical and theoretical knowledge in the area of the study ( O’Mathuna et al., 2011 )? c.  Does the literature review summary identify what is known and not known about the research problem and provide direction for the formation of the research purpose? 3.  Study framework a.  Is the framework presented with clarity? If a model or conceptual map of the framework is present, is it adequate to explain the phenomenon of concern ( Grove et al., 2013 )? b.  Is the framework related to the body of knowledge in nursing and clinical practice? c.  If a proposition from a theory is to be tested, is the proposition clearly identified and linked to the study hypotheses ( Alligood, 2010 ; Fawcett & Garity, 2009 ; Smith & Liehr, 2008 )? 4.  Research objectives, questions, or hypotheses a.  Are the objectives, questions, or hypotheses expressed clearly? b.  Are the objectives, questions, or hypotheses logically linked to the research purpose? c.  Are hypotheses stated to direct the conduct of quasi-experimental and experimental research ( Creswell, 2014 ; Shadish et al., 2002 )? d.  Are the objectives, questions, or hypotheses logically linked to the concepts and relationships (propositions) in the framework ( Chinn & Kramer, 2011 ; Fawcett & Garity, 2009 ; Smith & Liehr, 2008 )? 5.  Variables a.  Are the variables reflective of the concepts identified in the framework? b.  Are the variables clearly defined (conceptually and operationally) and based on previous research or theories ( Chinn & Kramer, 2011 ; Grove et al., 2013 ; Smith & Liehr, 2008 )? c.  Is the conceptual definition of a variable consistent with the operational definition? 6.  Design a.  Is the design used in the study the most appropriate design to obtain the needed data ( Creswell, 2014 ; Grove et al., 2013 ; Hoe & Hoare, 2012 )? b.  Does the design provide a means to examine all the objectives, questions, or hypotheses? c.  Is the treatment clearly described ( Brown, 2002 )? Is the treatment appropriate for examining the study purpose and hypotheses? Does the study framework explain the links between the treatment (independent variable) and the proposed outcomes (dependent variables)? Was a protocol developed to promote consistent implementation of the treatment to ensure intervention fidelity ( Morrison et al., 2009 )? Did the researcher monitor implementation of the treatment to ensure consistency ( Santacroce et al., 2004 )? If the treatment was not consistently implemented, what might be the impact on the findings? d.  Did the researcher identify the threats to design validity (statistical conclusion validity, internal validity, construct validity, and external validity [see Chapter 8 ]) and minimize them as much as possible ( Grove et al., 2013 ; Shadish et al., 2002 )? e.  If more than one group was used, did the groups appear equivalent? f.  If a treatment was implemented, were the subjects randomly assigned to the treatment group or were the treatment and comparison groups matched? Were the treatment and comparison group assignments appropriate for the purpose of the study? 7.  Sample, population, and setting a.  Is the sampling method adequate to produce a representative sample? Are any subjects excluded from the study because of age, socioeconomic status, or ethnicity, without a sound rationale? b.  Did the sample include an understudied population, such as young people, older adults, or minority group? c.  Were the sampling criteria (inclusion and exclusion) appropriate for the type of study conducted ( O’Mathuna et al., 2011 )? d.  Was a power analysis conducted to determine sample size? If a power analysis was conducted, were the results of the analysis clearly described and used to determine the final sample size? Was the attrition rate projected in determining the final sample size ( Aberson, 2010 )? e.  Are the rights of human subjects protected ( Creswell, 2014 ; Grove et al., 2013 )? f.  Is the setting used in the study typical of clinical settings? g.  Was the rate of potential subjects’ refusal to participate in the study a problem? If so, how might this weakness influence the findings? h.  Was sample attrition a problem? If so, how might this weakness influence the final sample and the study results and findings ( Aberson, 2010 ; Fawcett & Garity, 2009 ; Hoe & Hoare, 2012 )? 8.  Measurements a.  Do the measurement methods selected for the study adequately measure the study variables? Should additional measurement methods have been used to improve the quality of the study outcomes ( Waltz et al., 2010 )? b.  Do the measurement methods used in the study have adequate validity and reliability? What additional reliability or validity testing is needed to improve the quality of the measurement methods ( Bialocerkowski et al., 2010 ; DeVon et al., 2007 ; Roberts & Stone, 2003 )? c.  Respond to the following questions, which are relevant to the measurement approaches used in the study: 1)  Scales and questionnaires (a)  Are the instruments clearly described? (b)  Are techniques to complete and score the instruments provided? (c)  Are the validity and reliability of the instruments described ( DeVon et al., 2007 )? (d)  Did the researcher reexamine the validity and reliability of the instruments for the present sample? (e)  If the instrument was developed for the study, is the instrument development process described ( Grove et al., 2013 ; Waltz et al., 2010 )? 2)  Observation (a)  Is what is to be observed clearly identified and defined? (b)  Is interrater reliability described? (c)  Are the techniques for recording observations described ( Waltz et al., 2010 )? 3)  Interviews (a)  Do the interview questions address concerns expressed in the research problem? (b)  Are the interview questions relevant for the research purpose and objectives, questions, or hypotheses ( Grove et al., 2013 ; Waltz et al., 2010 )? 4)  Physiological measures (a)  Are the physiological measures or instruments clearly described ( Ryan-Wenger, 2010 )? If appropriate, are the brand names of the instruments identified, such as Space Labs or Hewlett-Packard? (b)  Are the accuracy, precision, and error of the physiological instruments discussed ( Ryan-Wenger, 2010 )? (c)  Are the physiological measures appropriate for the research purpose and objectives, questions, or hypotheses? (d)  Are the methods for recording data from the physiological measures clearly described? Is the recording of data consistent? 9.  Data collection a.  Is the data collection process clearly described ( Fawcett & Garity, 2009 ; Grove et al., 2013 )? b.  Are the forms used to collect data organized to facilitate computerizing the data? c.  Is the training of data collectors clearly described and adequate? d.  Is the data collection process conducted in a consistent manner? e.  Are the data collection methods ethical? f.  Do the data collected address the research objectives, questions, or hypotheses? g.  Did any adverse events occur during data collection, and were these appropriately managed? 10.  Data analysis a.  Are data analysis procedures appropriate for the type of data collected ( Grove, 2007 ; Hoare & Hoe, 2013 ; Plichta & Kelvin, 2013 )? b.  Are data analysis procedures clearly described? Did the researcher address any problem with missing data, and explain how this problem was managed? c.  Do the data analysis techniques address the study purpose and the research objectives, questions, or hypotheses ( Fawcett & Garity, 2009 ; Grove et al., 2013 ; Hoare & Hoe, 2013 )? d.  Are the results presented in an understandable way by narrative, tables, or figures or a combination of methods ( APA, 2010 )? e.  Is the sample size sufficient to detect significant differences, if they are present? f.  Was a power analysis conducted for nonsignificant results (Aberson, 2010) ? g.  Are the results interpreted appropriately?

Share this:

• Click to share on Twitter (Opens in new window)
• Click to share on Facebook (Opens in new window)

Related posts:

• Outcomes Research
• Understanding Statistics in Research
• Research Problems, Purposes, and Hypotheses
• Clarifying Quantitative Research Designs

Stay updated, free articles. Join our Telegram channel

Comments are closed for this page.

Full access? Get Clinical Tree

Postgraduate Nursing: Critical appraisal and Evaluation of research

• EndNote & Referencing
• Academic Writing
• Research & Research Methods
• Introduction to Evidence-Informed Practice
• Asking a Clinical Question
• Types of Research & Levels of Evidence
• Searching for the Evidence
• Critical appraisal and Evaluation of research
• Searching and appraising evidence demonstration
• Journals & Journal Articles
• Grey Literature & Statistics
• Search Tips
• CINAHL Plus with Full Text
• Medline & PsycINFO
• Cochrane Library
• Clinical Governance
• Anatomy & Physiology Resources
• Drug Resources
• Clinical Education
• Contact and feedback

Introduction to critical appraisal and evaluation

The information you use in your research and study must all be credible, reliable and relevant. Part of the Evidence-Based Practice process is to critically appraise scientific papers, but in general, all the resources you refer to should be evaluated carefully to ensure their credibility.

How can you tell whether the resources you've found are credible and suitable for you to reference? To evaluate Information you have found on websites, see the video below and the box on using Internet sites. Journal articles and academic texts should at least have gone through a process of peer review (see the video about peer review on the Journals page of this guide).

Critical appraisal of scientific papers takes the evaluation to another level. Once you have asked the clinical question and searched for evidence, it's often not enough that you've checked for peer review if you want to find the very best evidence - it will ensure that studies with scientific flaws are disregarded, and the ones you include are relevant to your question.

In the Evidence-Based Practice process, and especially in the process of evaluating primary research (which hasn't be pre-appraised or filtered by others), we need to go beyond the usual general information evaluation and make sure the evidence we are using is scientifically rigorous. The main questions to address are:

• Is the study relevant to your clinical question?
• How well (scientifically) was the study done, especially taking care to eliminate bias?
• What do the results mean and are they statistically valid (and not just due to chance)?

For a more detailed look at Critical Appraisal, head to the Systematic Review Guide - Critical Appraisal and the Evidence-Based Practic Guide - Appraise.

Critical appraisal tools

Fortunately, there have been some great checklist tools developed for different types of studies. Here are some examples:

• The Joanna Briggs Institute (JBI) provides access to critical appraisal tools, a collection of checklists that you can use to help you appraise or evaluate research.
• Critical Appraisal Skills Programme (CASP) is part of Better Value Healthcare based in Oxford, UK. It includes a series of checklists , suitable for different types of studies and designed to be used when reading research.
• The Equator Network is devoted to Enhancing the QUAlity and Transparency Of health Research. Among other functions, they include a  Toolkit for Peer Reviewing Health Research   which is very useful as a guide for critically appraising studies.
• Critical Appraisal Tools (CEBM)  - This site from the Centre of Evidence Based Medicine includes tools and worksheets for the critical appraisal of different types of medical evidence.
• Critical Appraisal Tools (iCAHE) - This site from the International Centre of Allied Health Evidence (at the University of South Australia) has a range of tools for various types of studies.
• Understanding Health Research - is from the Medical Research Council in the UK. It's a very handy all-purpose tool which takes you through a series of questions about a particular article, highlighting the good points and possible problem areas. You can print off a summary at the end of your checklist

Critical appraisal tools from the NHS in Scotland links interactively to all sorts of resources on how to identify the study type and build your critical appraisal skills, as well as to tools themselves.

Critical reading and understanding research

A useful series of articles for nurses about critiquing and understanding types of research has been published in the Australian Journal of Advanced Nursing by Rebecca Ingham-Broomfield, from the University of New South Wales:

Ingham-Broomfield, R. (2014). A nurses' guide to the critical reading of research . Australian Journal of Advanced Nursing , 32 (1), 37-44. [Updated from 2008.]

Ingham-Broomfield, R. (2014). A nurses' guide to quantitative research . Australian Journal of Advanced Nursing, 32 (2), 32-38. 

Ingham-Broomfield, R. (2015). A nurses' guide to qualitative research . Australian Journal of Advanced Nursing, 32 (3), 34-40. 

Ingham-Broomfield, R. (2016). A nurses' guide to mixed methods research . Australian Journal of Advanced Nursing, 33 (4), 46-52. 

Ingham-Broomfield, R. (2016). A nurses' guide to the hierarchy of research designs and evidence . The Australian Journal of Advanced Nursing, 33 (3), 38-43. 

Evaluate internet resources

The website domain gives you an idea of the reliability of a website:

Critical appraisal resources

Introduction to Critical Appraisal -  This short video from the library at the University of Sheffield in the UK looks at the background to critical appraisal, what it is, and why we do it. A very useful introduction to the topic.

Evaluating information

• << Previous: Searching for the Evidence
• Next: Searching and appraising evidence demonstration >>
• Last Updated: Mar 7, 2024 11:12 AM
• URL: https://libguides.csu.edu.au/MN

Charles Sturt University is an Australian University, TEQSA Provider Identification: PRV12018. CRICOS Provider: 00005F.

Critical Appraisal Resources for Evidence-Based Nursing Practice

What is critical appraisal, critical appraisal tools, video: learn more about the joanna briggs institute.

• Levels of Evidence
• Systematic Reviews
• Randomized Controlled Trials
• Quasi-Experimental Studies
• Case-Control Studies
• Cohort Studies
• Analytical Cross-Sectional Studies
• Qualitative Research

Recommended Books on Critical Appraisal

Critical appraisal is an essential and important step in the evidence-based practice (EBP) process.  It involves analyzing and critiquing the methodology and data of published research studies (both quantitative and qualitative designs) to determine the value, reliability, trustworthiness, and relevance of those studies in answering a clinical question.  

Looking for critical appraisal tools? Click here to access. 

RECOMMENDED READING:

Buccheri, R. K., & Sharifi, C. (2017). Critical appraisal tools and reporting guidelines for evidence-based practice .  Worldviews on Evidence-Based Nursing ,  14 (6), 463–472. https://doi.org/10.1111/wvn.12258

Definitions of critical appraisal are provided below:

“Judging the quality of information in terms of its validity and degree of bias (quantitative research) and credibility and dependability (qualitative research). This is a critical step in the evidence-based practice process” (Hopp & Rittenmeyer, 2021, p. 360).

“During appraisal, the study design, how the research was conducted, and the data analysis are all scrutinized to ensure that the study was sound” (Schmidt & Brown, 2019, p. 405).

“Critical appraisal is an assessment of the benefits and strengths of research against its flaws and weaknesses” (Holly, Salmond, & Saimbert, 2012, p. 147).

Holly, C., Salmond, S.W., & Saimbert, M. (2012). Comprehensive systematic review for advanced nursing practice. New York: Springer.

Hopp, L., & Rittenmeyer, L. (2021). Introduction to evidence-based practice: A practical guide for nursing. Philadelphia: F.A. Davis.

Schmidt, N.A., & Brown, J.M. (2019). Evidence-based practice for nurses: Appraisal and application of research. Burlington, MA: Jones & Bartlett. 

A variety of critical appraisal tools are available from different organizations to help guide you through the appraisal process.

The following links will connect you to these tools. 

• Joanna Briggs Institute (JBI) - Critical Appraisal Tools
• Centre for Evidence-Based Medicine (CEBM) - Critical Appraisal Tools
• Critical Appraisal Skills Programme (CASP) - Critical Appraisal Tools
• Critical Appraisal Tools Collection of links to various checklists by study type
• AMSTAR Checklist Tool for appraising systematic reviews
• AGREE Tools Tools for appraising practice guidelines

The Joanna Briggs Institute is a non-profit, international research and development organization for the promotion and implementation of evidence-based practice in healthcare.  The JBI Critical Appraisal Checklists are utilized the world over by healthcare practitioners and researchers who conduct EBP.  Learn more about JBI by visiting their website or watch the following video:  

• Next: Levels of Evidence >>
• Last Updated: Feb 22, 2024 11:26 AM
• URL: https://libguides.utoledo.edu/nursingappraisal

• NCSBN Member Login Submit

Access provided by

Login to your account

If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password

If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password

Download started.

• Academic & Personal: 24 hour online access
• Corporate R&D Professionals: 24 hour online access
• Add To Online Library Powered By Mendeley
• Add To My Reading List
• Export Citation
• Create Citation Alert

Nursing Research: Methods and Critical Appraisal for Evidence-Based Practice

• Geri LoBiondo-Wood Geri LoBiondo-Wood Search for articles by this author
• Judith Haber Judith Haber Search for articles by this author

Purchase one-time access:

• For academic or personal research use, select 'Academic and Personal'
• For corporate R&D use, select 'Corporate R&D Professionals'

Article info

© 2014 Elsevier Mosby. ISBN: 978-0-323-10086-1

Identification

DOI: https://doi.org/10.1016/S2155-8256(15)30102-2

ScienceDirect

Related articles.

• Download Hi-res image
• Download .PPT
• Access for Developing Countries
• Articles & Issues
• Current Issue
• List of Issues
• Supplements
• For Authors
• Guide for Authors
• Author Services
• Permissions
• Researcher Academy
• Submit a Manuscript
• Journal Info
• About the Journal
• Contact Information
• Editorial Board
• New Content Alerts
• Call for Papers
• January & April 2022 Issues

The content on this site is intended for healthcare professionals.

• Privacy Policy   
• Terms and Conditions   
• Accessibility   
• Help & Contact

• Cancer Nursing Practice
• Emergency Nurse
• Evidence-Based Nursing
• Learning Disability Practice
• Mental Health Practice
• Nurse Researcher
• Nursing Children and Young People
• Nursing Management
• Nursing Older People
• Nursing Standard
• Primary Health Care
• RCN Nursing Awards
• Nursing Live
• Nursing Careers and Job Fairs
• CPD webinars on-demand
• --> Advanced -->

• Clinical articles
• CPD articles
• CPD Quizzes
• Expert advice
• Clinical placements
• Study skills
• Clinical skills
• University life
• Person-centred care
• Career advice
• Revalidation

Research Previous     Next

Guidelines on conducting a critical research evaluation, gill hek , senior lecturer, university of the west of england, bristol.

This article outlines the reasons why nurses need to be able to read and evaluate research reports critically, and provides a step-by-step approach to conducting a critical appraisal of a research article

The ability to evaluate or appraise research in a critical manner is a skill that all nurses must develop. By acquiring and using these skills nurses will be able to understand and appreciate research. For those nurses who have qualified in the last four or five years, critical appraisal or critical evaluation skills may have been learnt during their pre-registration course. However, the vast majority of qualified nurses who undertook their training prior to Project 2000, are unlikely to have had the opportunity to develop such skills. Furthermore, many nurses undertake post-registration courses, and most of these courses will require them to gain competency in critical evaluation skills, often assessed through assignments and project work.

Nursing Standard . 11, 6, 40-43. doi: 10.7748/ns.11.6.40.s48

■ Nursing research - ■ Nursing literature - ■ Quality of nursing practice

User not found

Want to read more?

Already have access log in, 3-month trial offer for £5.25/month.

• Unlimited access to all 10 RCNi Journals
• RCNi Learning featuring over 175 modules to easily earn CPD time
• NMC-compliant RCNi Revalidation Portfolio to stay on track with your progress
• Personalised newsletters tailored to your interests
• A customisable dashboard with over 200 topics

Alternatively, you can purchase access to this article for the next seven days. Buy now

Are you a student? Our student subscription has content especially for you. Find out more

30 October 1996 / Vol 11 issue 6

TABLE OF CONTENTS

DIGITAL EDITION

• LATEST ISSUE
• SIGN UP FOR E-ALERT
• WRITE FOR US
• PERMISSIONS

Share article: Guidelines on conducting a critical research evaluation

We use cookies on this site to enhance your user experience.

By clicking any link on this page you are giving your consent for us to set cookies.

Evidence Based Practice Guide for Nursing Students: Appraisal

• Getting Started
• Levels of Evidence
• APA Style Guides

What is Critical Appraisal?

Critical appraisal.

Critical appraisal is an essential part of the Evidence Based Practice process. Critical appraisal identifies possible flaws or problems with the study methodology, the transparency of the study design as written in the article, the quality of the research, and  level of evidence.

Appraisal Concepts - Validity & Reliability

What is validity.

Internal validity is the extent to which the experiment demonstrated a cause-effect relationship between the independent and dependent variables.

External validity is the extent to which one may safely generalize from the sample studied to the defined target population and to other populations.

What is reliability?

Reliability is the extent to which the results of the experiment are replicable.  The research methodology should be described in detail so that the experiment could be repeated with similar results.

More Useful Scientific Terminology

Critically Appraised Topics (CATS)

CATs are critical summaries of a research article.  They are concise, standardized, and provide an appraisal of the research.

If a CAT already exists for an article, it can be read quickly and the clinical bottom line can be put to use as the clinician sees fit.  If a CAT does not exist, the CAT format provides a template to appraise the article of interest.

• CEBM's CATMaker tool helps you create your own CATs

Evaluating a Study

Start by asking simple questions about the article:

• Have the study aims been clearly stated?
• Does the sample accurately reflect the population?
• Has the sampling method and size been described and justified?
• Have exclusions been stated?
• Is the control group easily identified?
• Is the loss to follow-up detailed?
• Are enough details included so that the results could be replicated?
• Are there confounding factors?
• Are the conclusions logical?
• Do the findings match the study aims?
• Can the results be extrapolated to other populations?

Attribution Statement:  University of Illinois Chicago. Library of the Health Sciences. Evidence Based Medicine.  https://researchguides.uic.edu/ebm . Used under a CC BY-NC license.

Critical Appraisal Tools

• CASP UK Critical Appraisal Skills Programme
• Centre for Evidence Based Medicine
• Cochrane Handbook for Systematic Reviews
• Critical Appraisal Skills Programme Checklists
• GRADE Working Group

Videos from NCCMT

• Number Needed to Treat (10:42 min.)
• Relative Risk (10:40 min.)
• Types of Reviews: What Type of Review Do You Need? (9:22 min.)
• Importance of Clinical Significance (3:41 min.)
• How to Calculate an Odds Ratio (5:51)
• Understanding a Confidence Interval (5:29 min.)

NCCMT, National Collaborating Centre for Methods and Tools, is a Canadian agency that provides leadership, education and expertise to promote informed decision-making in public health.

• << Previous: Search
• Next: APA Style Guides >>
• Last Updated: Feb 19, 2024 12:10 PM
• URL: https://uwyo.libguides.com/EBP-BSN

How to critically appraise a qualitative health research study

Affiliations.

• 1 RN, MScN, PhD, School of Nursing, Dept. of Medicine and Surgery, University of Milano - Bicocca, Milan, Italy. Email: [email protected].
• 2 RN MScN PhD student, School of Nursing, McMaster University,Hamilton, Ontario, Canada.
• 3 RN, PhD student, School of Nursing, McMaster University, Hamilton, Ontario, Canada.
• 4 RN, MScN PhD, School of Nursing, Dept. of Medicine and Surgery, University of Milano - Bicocca, Milan, Italy.
• 5 RN, PhD, School of Nursing, McMaster University, Hamilton, Ontario, Canada.
• PMID: 32243743

Abstract in English, Italian

Evidence-based nursing is a process that requires nurses to have the knowledge, skills, and confidence to critically reflect on their practice, articulate structured questions, and then reliably search for research evidence to address the questions posed. Many types of research evidence are used to inform decisions in health care and findings from qualita- tive health research studies are useful to provide new insights about individuals' experi- ences, values, beliefs, needs, or perceptions. Before qualitative evidence can be utilized in a decision, it must be critically appraised to determine if the findings are trustworthy and if they have relevance to the identified issue or decision. In this article, we provide practical guidance on how to select a checklist or tool to guide the critical appraisal of qualitative studies and then provide an example demonstrating how to apply the critical appraisal process to a clinical scenario.

L’Evidence-Based Nursing Ë un processo che richiede agli infermieri di avere le conoscenze, le competenze e la fiducia necessarie per riflettere criticamente sulla loro pratica, articolare domande strutturate e poi cercare in modo affidabile la letteratura per rispondere alle domande poste. Ci sono molti tipi di evidence che vengono utilizzate per informare le decisioni nell'as- sistenza sanitaria e i risultati di studi di ricerca qualitativa sanitaria sono utili per fornire nuove intuizioni sulle esperienze, i valori, le convinzioni, i bisogni o le percezioni degli individui. Prima che l'evidence qualitativa possa essere utilizzata in una decisione, deve essere valutata criticamente per determinare se i risultati sono affidabili e se hanno rilevanza per la questione o la decisione identificata. In questo articolo forniamo una guida pratica su come selezionare una checklist o uno strumento per guidare la valutazione critica degli studi qualitativi e, poi, forniamo un esempio che dimostra come applicare il processo di valutazione critica a uno scenario clinico.

• Clinical Competence
• Delivery of Health Care / organization & administration
• Delivery of Health Care / standards
• Evidence-Based Nursing / organization & administration*
• Health Knowledge, Attitudes, Practice
• Health Services Research / organization & administration
• Health Services Research / standards
• Nurses / organization & administration*
• Nurses / standards
• Qualitative Research*

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 21, Issue 4
• How to appraise quantitative research
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

This article has a correction. Please see:

• Correction: How to appraise quantitative research - April 01, 2019

• Xabi Cathala 1 ,
• Calvin Moorley 2
• 1 Institute of Vocational Learning , School of Health and Social Care, London South Bank University , London , UK
• 2 Nursing Research and Diversity in Care , School of Health and Social Care, London South Bank University , London , UK
• Correspondence to Mr Xabi Cathala, Institute of Vocational Learning, School of Health and Social Care, London South Bank University London UK ; cathalax{at}lsbu.ac.uk and Dr Calvin Moorley, Nursing Research and Diversity in Care, School of Health and Social Care, London South Bank University, London SE1 0AA, UK; Moorleyc{at}lsbu.ac.uk

https://doi.org/10.1136/eb-2018-102996

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

Some nurses feel that they lack the necessary skills to read a research paper and to then decide if they should implement the findings into their practice. This is particularly the case when considering the results of quantitative research, which often contains the results of statistical testing. However, nurses have a professional responsibility to critique research to improve their practice, care and patient safety. 1  This article provides a step by step guide on how to critically appraise a quantitative paper.

Title, keywords and the authors

The authors’ names may not mean much, but knowing the following will be helpful:

Their position, for example, academic, researcher or healthcare practitioner.

Their qualification, both professional, for example, a nurse or physiotherapist and academic (eg, degree, masters, doctorate).

This can indicate how the research has been conducted and the authors’ competence on the subject. Basically, do you want to read a paper on quantum physics written by a plumber?

The abstract is a resume of the article and should contain:

Introduction.

Research question/hypothesis.

Methods including sample design, tests used and the statistical analysis (of course! Remember we love numbers).

Main findings.

Conclusion.

The subheadings in the abstract will vary depending on the journal. An abstract should not usually be more than 300 words but this varies depending on specific journal requirements. If the above information is contained in the abstract, it can give you an idea about whether the study is relevant to your area of practice. However, before deciding if the results of a research paper are relevant to your practice, it is important to review the overall quality of the article. This can only be done by reading and critically appraising the entire article.

The introduction

Example: the effect of paracetamol on levels of pain.

My hypothesis is that A has an effect on B, for example, paracetamol has an effect on levels of pain.

My null hypothesis is that A has no effect on B, for example, paracetamol has no effect on pain.

My study will test the null hypothesis and if the null hypothesis is validated then the hypothesis is false (A has no effect on B). This means paracetamol has no effect on the level of pain. If the null hypothesis is rejected then the hypothesis is true (A has an effect on B). This means that paracetamol has an effect on the level of pain.

Background/literature review

The literature review should include reference to recent and relevant research in the area. It should summarise what is already known about the topic and why the research study is needed and state what the study will contribute to new knowledge. 5 The literature review should be up to date, usually 5–8 years, but it will depend on the topic and sometimes it is acceptable to include older (seminal) studies.

Methodology

In quantitative studies, the data analysis varies between studies depending on the type of design used. For example, descriptive, correlative or experimental studies all vary. A descriptive study will describe the pattern of a topic related to one or more variable. 6 A correlational study examines the link (correlation) between two variables 7  and focuses on how a variable will react to a change of another variable. In experimental studies, the researchers manipulate variables looking at outcomes 8  and the sample is commonly assigned into different groups (known as randomisation) to determine the effect (causal) of a condition (independent variable) on a certain outcome. This is a common method used in clinical trials.

There should be sufficient detail provided in the methods section for you to replicate the study (should you want to). To enable you to do this, the following sections are normally included:

Overview and rationale for the methodology.

Participants or sample.

Data collection tools.

Methods of data analysis.

Ethical issues.

Data collection should be clearly explained and the article should discuss how this process was undertaken. Data collection should be systematic, objective, precise, repeatable, valid and reliable. Any tool (eg, a questionnaire) used for data collection should have been piloted (or pretested and/or adjusted) to ensure the quality, validity and reliability of the tool. 9 The participants (the sample) and any randomisation technique used should be identified. The sample size is central in quantitative research, as the findings should be able to be generalised for the wider population. 10 The data analysis can be done manually or more complex analyses performed using computer software sometimes with advice of a statistician. From this analysis, results like mode, mean, median, p value, CI and so on are always presented in a numerical format.

The author(s) should present the results clearly. These may be presented in graphs, charts or tables alongside some text. You should perform your own critique of the data analysis process; just because a paper has been published, it does not mean it is perfect. Your findings may be different from the author’s. Through critical analysis the reader may find an error in the study process that authors have not seen or highlighted. These errors can change the study result or change a study you thought was strong to weak. To help you critique a quantitative research paper, some guidance on understanding statistical terminology is provided in  table 1 .

• View inline

Some basic guidance for understanding statistics

Quantitative studies examine the relationship between variables, and the p value illustrates this objectively.  11  If the p value is less than 0.05, the null hypothesis is rejected and the hypothesis is accepted and the study will say there is a significant difference. If the p value is more than 0.05, the null hypothesis is accepted then the hypothesis is rejected. The study will say there is no significant difference. As a general rule, a p value of less than 0.05 means, the hypothesis is accepted and if it is more than 0.05 the hypothesis is rejected.

The CI is a number between 0 and 1 or is written as a per cent, demonstrating the level of confidence the reader can have in the result. 12  The CI is calculated by subtracting the p value to 1 (1–p). If there is a p value of 0.05, the CI will be 1–0.05=0.95=95%. A CI over 95% means, we can be confident the result is statistically significant. A CI below 95% means, the result is not statistically significant. The p values and CI highlight the confidence and robustness of a result.

Discussion, recommendations and conclusion

The final section of the paper is where the authors discuss their results and link them to other literature in the area (some of which may have been included in the literature review at the start of the paper). This reminds the reader of what is already known, what the study has found and what new information it adds. The discussion should demonstrate how the authors interpreted their results and how they contribute to new knowledge in the area. Implications for practice and future research should also be highlighted in this section of the paper.

A few other areas you may find helpful are:

Limitations of the study.

Conflicts of interest.

Table 2 provides a useful tool to help you apply the learning in this paper to the critiquing of quantitative research papers.

Quantitative paper appraisal checklist

• 1. ↵ Nursing and Midwifery Council , 2015 . The code: standard of conduct, performance and ethics for nurses and midwives https://www.nmc.org.uk/globalassets/sitedocuments/nmc-publications/nmc-code.pdf ( accessed 21.8.18 ).
• Gerrish K ,
• Moorley C ,
• Tunariu A , et al
• Shorten A ,

Competing interests None declared.

Patient consent Not required.

Provenance and peer review Commissioned; internally peer reviewed.

Correction notice This article has been updated since its original publication to update p values from 0.5 to 0.05 throughout.

Linked Articles

• Miscellaneous Correction: How to appraise quantitative research BMJ Publishing Group Ltd and RCN Publishing Company Ltd Evidence-Based Nursing 2019; 22 62-62 Published Online First: 31 Jan 2019. doi: 10.1136/eb-2018-102996corr1

Read the full text or download the PDF:

• Open access
• Published: 19 March 2024

Interventions, methods and outcome measures used in teaching evidence-based practice to healthcare students: an overview of systematic reviews

• Lea D. Nielsen 1 ,
• Mette M. Løwe 2 ,
• Francisco Mansilla 3 ,
• Rene B. Jørgensen 4 ,
• Asviny Ramachandran 5 ,
• Bodil B. Noe 6 &
• Heidi K. Egebæk 7  

BMC Medical Education volume  24 , Article number:  306 ( 2024 ) Cite this article

304 Accesses

Metrics details

To fully implement the internationally acknowledged requirements for teaching in evidence-based practice, and support the student’s development of core competencies in evidence-based practice, educators at professional bachelor degree programs in healthcare need a systematic overview of evidence-based teaching and learning interventions. The purpose of this overview of systematic reviews was to summarize and synthesize the current evidence from systematic reviews on educational interventions being used by educators to teach evidence-based practice to professional bachelor-degree healthcare students and to identify the evidence-based practice-related learning outcomes used.

An overview of systematic reviews. Four databases (PubMed/Medline, CINAHL, ERIC and the Cochrane library) were searched from May 2013 to January 25th, 2024. Additional sources were checked for unpublished or ongoing systematic reviews. Eligibility criteria included systematic reviews of studies among undergraduate nursing, physiotherapist, occupational therapist, midwife, nutrition and health, and biomedical laboratory science students, evaluating educational interventions aimed at teaching evidence-based practice in classroom or clinical practice setting, or a combination. Two authors independently performed initial eligibility screening of title/abstracts. Four authors independently performed full-text screening and assessed the quality of selected systematic reviews using standardized instruments. Data was extracted and synthesized using a narrative approach.

A total of 524 references were retrieved, and 6 systematic reviews (with a total of 39 primary studies) were included. Overlap between the systematic reviews was minimal. All the systematic reviews were of low methodological quality. Synthesis and analysis revealed a variety of teaching modalities and approaches. The outcomes were to some extent assessed in accordance with the Sicily group`s categories; “skills”, “attitude” and “knowledge”. Whereas “behaviors”, “reaction to educational experience”, “self-efficacy” and “benefits for the patient” were rarely used.

Conclusions

Teaching evidence-based practice is widely used in undergraduate healthcare students and a variety of interventions are used and recognized. Not all categories of outcomes suggested by the Sicily group are used to evaluate outcomes of evidence-based practice teaching. There is a need for studies measuring the effect on outcomes in all the Sicily group categories, to enhance sustainability and transition of evidence-based practice competencies to the context of healthcare practice.

Peer Review reports

Evidence-based practice (EBP) enhances the quality of healthcare, reduces the cost, improves patient outcomes, empowers clinicians, and is recognized as a problem-solving approach [ 1 ] that integrates the best available evidence with clinical expertise and patient preferences and values [ 2 ]. A recent scoping review of EBP and patient outcomes indicates that EBPs improve patient outcomes and yield a positive return of investment for hospitals and healthcare systems. The top outcomes measured were length of stay, mortality, patient compliance/adherence, readmissions, pneumonia and other infections, falls, morbidity, patient satisfaction, patient anxiety/ depression, patient complications and pain. The authors conclude that healthcare professionals have a professional and ethical responsibility to provide expert care which requires an evidence-based approach. Furthermore, educators must become competent in EBP methodology [ 3 ].

According to the Sicily statement group, teaching and practicing EBP requires a 5-step approach: 1) pose an answerable clinical question (Ask), 2) search and retrieve relevant evidence (Search), 3) critically appraise the evidence for validity and clinical importance (Appraise), 4) applicate the results in practice by integrating the evidence with clinical expertise, patient preferences and values to make a clinical decision (Integrate), and 5) evaluate the change or outcome (Evaluate /Assess) [ 4 , 5 ]. Thus, according to the World Health Organization, educators, e.g., within undergraduate healthcare education, play a vital role by “integrating evidence-based teaching and learning processes, and helping learners interpret and apply evidence in their clinical learning experiences” [ 6 ].

A scoping review by Larsen et al. of 81 studies on interventions for teaching EBP within Professional bachelor-degree healthcare programs (PBHP) (in English undergraduate/ bachelor) shows that the majority of EBP teaching interventions include the first four steps, but the fifth step “evaluate/assess” is less often applied [ 5 ]. PBHP include bachelor-degree programs characterized by combined theoretical education and clinical training within nursing, physiotherapy, occupational therapy, radiography, and biomedical laboratory students., Furthermore, an overview of systematic reviews focusing on practicing healthcare professionals EBP competencies testifies that although graduates may have moderate to high level of self-reported EBP knowledge, skills, attitudes, and beliefs, this does not translate into their subsequent EBP implementation [ 7 ]. Although this cannot be seen as direct evidence of inadequate EBP teaching during undergraduate education, it is irrefutable that insufficient EBP competencies among clinicians across healthcare disciplines impedes their efforts to attain highest care quality and improved patient outcomes in clinical practice after graduation.

Research shows that teaching about EBP includes different types of modalities. An overview of systematic reviews, published by Young et al. in 2014 [ 8 ] and updated by Bala et al. in 2021 [ 9 ], synthesizes the effects of EBP teaching interventions including under- and post graduate health care professionals, the majority being medical students. They find that multifaceted interventions with a combination of lectures, computer lab sessions, small group discussion, journal clubs, use of current clinical issues, portfolios and assignments lead to improvement in students’ EBP knowledge, skills, attitudes, and behaviors compared to single interventions or no interventions [ 8 , 9 ]. Larsen et al. find that within PBHP, collaboration with clinical practice is the second most frequently used intervention for teaching EBP and most often involves four or all five steps of the EBP teaching approach [ 5 ]. The use of clinically integrated teaching in EBP is only sparsely identified in the overviews by Young et al. and Bala et al. [ 8 , 9 ]. Therefore, the evidence obtained within Bachelor of Medicine which is a theoretical education [ 10 ], may not be directly transferable for use in PBHP which combines theoretical and mandatory clinical education [ 11 ].

Since the overview by Young et al. [ 8 ], several reviews of interventions for teaching EBP used within PBHP have been published [ 5 , 12 , 13 , 14 ].

We therefore wanted to explore the newest evidence for teaching EBP focusing on PBHP as these programs are characterized by a large proportion of clinical teaching. These healthcare professions are certified through a PBHP at a level corresponding to a University Bachelor Degree, but with strong focus on professional practice by combining theoretical studies with mandatory clinical teaching. In Denmark, almost half of PBHP take place in clinical practice. These applied science programs qualify “the students to independently analyze, evaluate and reflect on problems in order to carry out practice-based, complex, and development-oriented job functions" [ 11 ]. Thus, both the purpose of these PBHP and the amount of clinical practice included in the educations contrast with for example medicine.

Thus, this overview, identifies the newest evidence for teaching EBP specifically within PBHP and by including reviews using quantitative and/or qualitative methods.

We believe that such an overview is important knowledge for educators to be able to take the EBP teaching for healthcare professions to a higher level. Also reviewing and describing EBP-related learning outcomes, categorizing them according to the seven assessment categories developed by the Sicily group [ 2 ], will be useful knowledge to educators in healthcare professions. These seven assessment categories for EBP learning including: Reaction to the educational experience, attitudes, self-efficacy, knowledge, skills, behaviors and benefits to patients, can be linked to the five-step EBP approach. E.g., reactions to the educational experience: did the educators teaching style enhance learners’ enthusiasm for asking questions? (Ask), self-efficacy: how well do learners think they critically appraise evidence? (Appraise), skills: can learners come to a reasonable interpretation of how to apply the evidence? (Integrate) [ 2 ]. Thus, this set of categories can be seen as a basic set of EBP-related learning outcomes to classify the impact from EBP educational interventions.

Purpose and review questions

A systematic overview of which evidence-based teaching interventions and which EBP-related learning outcomes that are used will give teachers access to important knowledge on what to implement and how to evaluate EBP teaching.

Thus, the purpose of this overview is to synthesize the latest evidence from systematic reviews about EBP teaching interventions in PBHP. This overview adds to the existing evidence by focusing on systematic reviews that a) include qualitative and/ or quantitative studies regardless of design, b) are conducted among PBHP within nursing, physiotherapy, occupational therapy, midwifery, nutrition and health and biomedical laboratory science, and c) incorporate the Sicily group's 5-step approach and seven assessment categories when analyzing the EBP teaching interventions and EBP-related learning outcomes.

The questions of this overview of systematic reviews are:

Which educational interventions are described and used by educators to teach EBP to Professional Bachelor-degree healthcare students?

What EBP-related learning outcomes have been used to evaluate teaching interventions?

The study protocol was guided by the Cochrane Handbook on Overviews of Reviews [ 15 ] and the review process was reported in accordance with The Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement [ 16 ] when this was consistent with the Cochrane Handbook.

Inclusion criteria

Eligible reviews fulfilled the inclusion criteria for publication type, population, intervention, and context (see Table  1 ). Failing a single inclusion criterion implied exclusion.

Search strategy

On January 25th 2024 a systematic search was conducted in; PubMed/Medline, CINAHL (EBSCOhost), ERIC (EBSCOhost) and the Cochrane library from May 2013 to January 25th, 2024 to identify systematic reviews published after the overview by Young et al. [ 8 ]. In collaboration with a research librarian, a search strategy of controlled vocabulary and free text terms related to systematic reviews, the student population, teaching interventions, teaching context, and evidence-based practice was developed (see Additional file 1 ). For each database, the search strategy was peer reviewed, revised, modified and subsequently pilot tested. No language restrictions were imposed.

To identify further eligible reviews, the following methods were used: Setting email alerts from the databases to provide weekly updates on new publications; backward and forward citation searching based on the included reviews by screening of reference lists and using the “cited by” and “similar results” function in PubMed and CINAHL; broad searching in Google Scholar (Advanced search), Prospero, JBI Evidence Synthesis and the OPEN Grey database; contacting experts in the field via email to first authors of included reviews, and by making queries via Twitter and Research Gate on any information on unpublished or ongoing reviews of relevance.

Selection and quality appraisal process

Database search results were merged, duplicate records were removed, and title/abstract were initially screened via Covidence [ 17 ]. The assessment process was pilot tested by four authors independently assessing eligibility and methodological quality of one potential review followed by joint discussion to reach a common understanding of the criteria used. Two authors independently screened each title/abstract for compliance with the predefined eligibility criteria. Disagreements were resolved by a third author. Four authors were paired for full text screening, and each pair assessed independently 50% of the potentially relevant reviews for eligibility and methodological quality.

For quality appraisal, two independent authors used the AMSTAR-2 (A MeaSurement Tool to Assess systematic Reviews) for reviews including intervention studies [ 18 ] and the Joanna Briggs Institute Checklist for systematic reviews and research Synthesis (JBI checklist) [ 19 ] for reviews including both quantitative and qualitative or only qualitative studies. Uncertainties in assessments were resolved by requesting clarifying information from first authors of reviews and/or discussion with co-author to the present overview.

Overall methodological quality for included reviews was assessed using the overall confidence criteria of AMSTAR 2 based on scorings in seven critical domains [ 18 ] appraised as high (none or one non-critical flaw), moderate (more than one non-critical flaw), low (one critical weakness) or critically low (more than one critical weakness) [ 18 ]. For systematic reviews of qualitative studies [ 13 , 20 , 21 ] the critical domains of the AMSTAR 2, not specified in the JBI checklist, were added.

Data extraction and synthesis process

Data were initially extracted by the first author, confirmed or rejected by the last author and finally discussed with the whole author group until consensus was reached.

Data extraction included 1) Information about the search and selection process according to the PRISMA statement [ 16 , 22 ], 2) Characteristics of the systematic reviews inspired by a standard in the Cochrane Handbook (15), 3) A citation index inspired by Young et al. [ 8 ] used to illustrate overlap of primary studies in the included systematic reviews, and to ensure that data from each primary study were extracted only once [ 15 ], 4) Data on EBP teaching interventions and EBP-related outcomes. These data were extracted, reformatted (categorized inductively into two categories: “Collaboration interventions” and “  Educational interventions ”) and presented as narrative summaries [ 15 ]. Data on outcome were categorized according to the seven assessment categories, defined by the Sicily group, to classify the impact from EBP educational interventions: Reaction to the educational experience, attitudes, self-efficacy, knowledge, skills, behaviors and benefits to patients [ 2 ]. When information under points 3 and 4 was missing, data from the abstracts of the primary study articles were reviewed.

Results of the search

The database search yielded 691 references after duplicates were removed. Title and abstract screening deemed 525 references irrelevant. Searching via other methods yielded two additional references. Out of 28 study reports assessed for eligibility 22 were excluded, leaving a total of six systematic reviews. Screening resulted in 100% agreement among the authors. Figure  1 details the search and selection process. Reviews that might seem relevant but did not meet the eligibility criteria [ 15 ], are listed in Additional file 2 . One protocol for a potentially relevant review was identified as ongoing [ 23 ].

PRISMA flow diagram on search and selection of systematic reviews

Characteristics of included systematic reviews and overlap between them

The six systematic reviews originated from the Middle East, Asia, North America, Europe, Scandinavia, and Australia. Two out of six reviews did not identify themselves as systematic reviews but did fulfill this eligibility criteria [ 12 , 20 ]. All six represented a total of 64 primary studies and a total population of 6649 students (see Table  2 ). However, five of the six systematic reviews contained a total of 17 primary studies not eligible to our overview focus (e.g., postgraduate students) (see Additional file 3 ). Results from these primary studies were not extracted. Of the remaining primary studies, six were included in two, and one was included in three systematic reviews. Data from these studies were extracted only once to avoid double-counting. Thus, the six systematic reviews represented a total of 39 primary studies and a total population of 3394 students. Nursing students represented 3280 of these. One sample of 58 nutrition and health students and one sample of 56 mixed nursing and midwife students were included but none from physiotherapy, occupational therapy, or biomedical laboratory scientists. The majority ( n  = 28) of the 39 primary studies had a quantitative design whereof 18 were quasi-experimental (see Additional file 4 ).

Quality of systematic review

All the included systematic reviews were assessed as having critically low quality with 100% concordance between the two designed authors (see Fig.  2 ) [ 18 ]. The main reasons for the low quality of the reviews were a) not demonstrating a registered protocol prior to the review [ 13 , 20 , 24 , 25 ], b) not providing a list of excluded studies with justification for exclusion [ 12 , 13 , 21 , 24 , 25 ] and c) not accounting for the quality of the individual studies when interpreting the result of the review [ 12 , 20 , 21 , 25 ].

Overall methodological quality assessment for systematic reviews. Quantitative studies [ 12 , 24 , 25 ] were assessed following the AMSTAR 2 critical domain guidelines. Qualitative studies [ 13 , 20 , 21 ] were assessed following the JBI checklist. For overall classification, qualitative studies were also assessed with the following critical AMSTAR 2 domains not specified in the JBI checklist (item 2. is the protocol registered before commencement of the review, item 7. justification for excluding individual studies and item 13. consideration of risk of bias when interpreting the results of the review)

Missing reporting of sources of funding for primary studies and not describing the included studies in adequate detail were, most often, the two non-critical items of the AMSTAR 2 and the JBI checklist, not met.

Most of the included reviews did report research questions including components of PICO, performed study selection and data extraction in duplicate, used appropriate methods for combining studies and used satisfactory techniques for assessing risk of bias (see Fig.  2 ).

Main findings from the systematic reviews

As illustrated in Table  2 , this overview synthesizes evidence on a variety of approaches to promote EBP teaching in both classroom and clinical settings. The systematic reviews describe various interventions used for teaching in EBP, which can be summarized into two themes: Collaboration Interventions and Educational Interventions.

Collaboration interventions to teach EBP

In general, the reviews point that interdisciplinary collaboration among health professionals and/or others e.g., librarian and professionals within information technologies is relevant when planning and teaching in EBP [ 13 , 20 ].

Interdisciplinary collaboration was described as relevant when planning teaching in EBP [ 13 , 20 ]. Specifically, regarding literature search Wakibi et al. found that collaboration between librarians, computer laboratory technicians and nurse educators enhanced students’ skills [ 13 ]. Also, in terms of creating transfer between EBP teaching and clinical practice, collaboration between faculty, library, clinical institutions, and teaching institutions was used [ 13 , 20 ].

Regarding collaboration with clinical practice, Ghaffari et al. found that teaching EBP integrated in clinical education could promote students’ knowledge and skills [ 25 ]. Horntvedt et al. found that during a six-week course in clinical practice, students obtained better skills in reading research articles and orally presenting the findings to staff and fellow students [ 20 ]. Participation in clinical research projects combined with instructions in analyzing and discussing research findings also “led to a positive approach and EBP knowledge” [ 20 ]. Moreover, reading research articles during the clinical practice period enhances the students critical thinking skills. Furthermore, Horntvedt et al. mention, that students found it meaningful to conduct a “mini” – research project in clinical settings, as the identified evidence became relevant [ 20 ].

Educational interventions

Educational interventions can be described as “Framing Interventions” understood as different ways to set up a framework for teaching EBP, and “  Teaching methods ” understood as specific methods used when teaching EBP.

Various educational interventions were described in most reviews [ 12 , 13 , 20 , 21 ]. According to Patelarou et al., no specific educational intervention regardless of framing and methods was in favor to “ increase knowledge, skills and competency as well as improve the beliefs, attitudes and behaviors of nursing students”  [ 12 ].

Framing interventions

The approaches used to set up a framework for teaching EBP were labelled in different ways: programs, interactive teaching strategies, educational programs, courses etc. Approaches of various durations from hours to months were described as well as stepwise interventions [ 12 , 13 , 20 , 21 , 24 , 25 ].

Some frameworks [ 13 , 20 , 21 , 24 ] were based on the assessments categories described by the Sicily group [ 2 ] or based on theory [ 21 ] or as mentioned above clinically integrated [ 20 ]. Wakibi et al. identified interventions used to foster a spirit of inquiry and EBP culture reflecting the “5-step approach” of the Sicily group [ 4 ], asking PICOT questions, searching for best evidence, critical appraisal, integrating evidence with clinical expertise and patient preferences to make clinical decisions, evaluating outcomes of EBP practice, and disseminating outcomes useful [ 13 ]. Ramis et al. found that teaching interventions based on theory like Banduras self-efficacy or Roger’s theory of diffusion led to positive effects on students EBP knowledge and attitudes [ 21 ].

Teaching methods

A variety of teaching methods were used such as, lectures [ 12 , 13 , 20 ], problem-based learning [ 12 , 20 , 25 ], group work, discussions [ 12 , 13 ], and presentations [ 20 ] (see Table  2 ). The most effective method to achieve the skills required to practice EBP as described in the “5-step approach” by the Sicely group is a combination of different teaching methods like lectures, assignments, discussions, group works, and exams/tests.

Four systematic reviews identified such combinations or multifaceted approaches [ 12 , 13 , 20 , 21 ]. Patelarou et al. states that “EBP education approaches should be blended” [ 12 ]. Thus, combining the use of video, voice-over, PowerPoint, problem-based learning, lectures, team-based learning, projects, and small groups were found in different studies. This combination had shown “to be effective” [ 12 ]. Similarly, Horntvedt et al. found that nursing students reported that various teaching methods improved their EBP knowledge and skills [ 20 ].

According to Ghaffari et al., including problem-based learning in teaching plans “improved the clinical care and performance of the students”, while the problem-solving approach “promoted student knowledge” [ 25 ]. Other teaching methods identified, e.g., flipped classroom [ 20 ] and virtual simulation [ 12 , 20 ] were also characterized as useful interactive teaching interventions. Furthermore, face-to-face approaches seem “more effective” than online teaching interventions to enhance students’ research and appraisal skills and journal clubs enhance the students critically appraisal-skills [ 12 ].

As the reviews included in this overview primarily are based on qualitative, mixed methods as well as quasi-experimental studies and to a minor extent on randomized controlled trials (see Table  2 ) it is not possible to conclude of the most effective methods. However, a combination of methods and an innovative collaboration between librarians, information technology professionals and healthcare professionals seem the most effective approach to achieve EBP required skills.

EBP-related outcomes

Most of the systematic reviews presented a wide array of outcome assessments applied in EBP research (See Table  3 ). Analyzing the outcomes according to the Sicily group’s assessment categories revealed that assessing “knowledge” (used in 19 out of 39 primary studies), “skills” (used in 18 out of 39 primary studies) and “attitude” (used in 17 out of 39) were by far the most frequently used assessment categories, whereas outcomes within the category of “behaviors” (used in eight studies) “reaction to educational experience” (in five studies), “self-efficacy” (in two studies), and “benefits for the patient” (in one study), were used to a far lesser extent. Additionally, outcomes, that we were not able to categorize within the seven assessment categories, were “future use” and “Global EBP competence”.

The purpose of this overview of systematic reviews was to collect and summarize evidence of the diversity of EBP teaching interventions and outcomes measured among professional bachelor- degree healthcare students.

Our results give an overview of “the state of the art” of using and measuring EBP in PBHP education. However, the quality of included systematic reviews was rated critically low. Thus, the result cannot support guidelines of best practice.

The analysis of the interventions and outcomes described in the 39 primary studies included in this overview, reveals a wide variety of teaching methods and interventions being used and described in the scientific literature on EBP teaching of PBHP students. The results show some evidence of the five step EBP approach in accordance with the inclusion criteria “interventions aimed at teaching one or more of the five EBP steps; Ask, Search, Appraise, Integrate, Assess/evaluate”. Most authors state, that the students´ EBP skills, attitudes and knowledge improved by almost any of the described methods and interventions. However, descriptions of how the improvements were measured were less frequent.

We evaluated the described outcome measures and assessments according to the seven categories proposed by the Sicily group and found that most assessments were on “attitudes”, “skills” and “knowledge”, sometimes on “behaviors” and very seldom on” reaction to educational experience”, “self-efficacy” and “benefits to the patients”. To our knowledge no systematic review or overview has made this evaluation on outcome categories before, but Bala et al. [ 9 ] also stated that knowledge, skills, and attitudes are the most common evaluated effects.

Comparing the outcomes measured between mainly medical [ 9 ] and nursing students, the most prevalent outcomes in both groups are knowledge, skills and attitudes around EBP. In contrast, measuring on the students´ patient care or on the impact of the EBP teaching on benefits for the patients is less prevalent. In contrast Wu et al.’s systematic review shows that among clinical nurses, educational interventions supporting implementation of EBP projects can change patient outcomes positively. However, they also conclude that direct causal evidence of the educational interventions is difficult to measure because of the diversity of EBP projects implemented [ 26 ]. Regarding EBP behavior the Sicily group recommend this category to be assessed by monitoring the frequency of the five step EBP approach, e.g., ASK questions about patients, APPRAISE evidence related to patient care, EVALUATE their EBP behavior and identified areas for improvement [ 2 ]. The results also showed evidence of student-clinician transition. “Future use” was identified in two systematic reviews [ 12 , 13 ] and categorized as “others”. This outcome is not included in the seven Sicily categories. However, a systematic review of predictive modelling studies shows, that future use or the intention to use EBP after graduation are influenced by the students EBP familiarity, EBP capability beliefs, EBP attitudes and academic and clinical support [ 27 ].

Teaching and evaluating EBP needs to move beyond aiming at changes in knowledge, skills, and attitudes, but also start focusing on changing and assessing behavior, self-efficacy and benefit to the patients. We recommend doing this using validated tools for the assessment of outcomes and in prospective studies with longer follow-up periods, preferably evaluating the adoption of EBP in clinical settings bearing in mind, that best teaching practice happens across sectors and settings supported and supervised by multiple professions.

Based on a systematic review and international Delphi survey, a set of interprofessional EBP core competencies that details the competence content of each of the five steps has been published to inform curriculum development and benchmark EBP standards [ 28 ]. This consensus statement may be used by educators as a reference for both learning objectives and EBP content descriptions in future intervention research. The collaboration with clinical institutions and integration of EBP teaching components such as EBP assignments or participating in clinical research projects are important results. Specifically, in the light of the dialectic between theoretical and clinical education as a core characteristic of Professional bachelor-degree healthcare educations.

Our study has some limitations that need consideration when interpreting the results. A search in the EMBASE and Scopus databases was not added in the search strategy, although it might have been able to bring additional sources. Most of the 22 excluded reviews included primary studies among other levels/ healthcare groups of students or had not critically appraised their primary studies. This constitutes insufficient adherence to methodological guidelines for systematic reviews and limits the completeness of the reviews identified. Often, the result sections of the included reviews were poorly reported and made it necessary to extract some, but not always sufficient, information from the primary study abstracts. As the present study is an overview and not a new systematic review, we did not extract information from the result section in the primary studies. Thus, the comprehensiveness and applicability of the results of this overview are limited by the methodological limitations in the six included systematic reviews.

The existing evidence is based on different types of study designs. This heterogeneity is seen in all the included reviews. Thus, the present overview only conveys trends around the comparative effectiveness of the different ways to frame, or the methods used for teaching EBP. This can be seen as a weakness for the clarity and applicability of the overview results. Also, our protocol is unpublished, which may weaken the transparency of the overview approach, however our search strategies are available as additional material (see Additional file 1 ). In addition, the validity of data extraction can be discussed. We extracted data consecutively by the first and last author and if needed consensus was reached by discussion with the entire research group. This method might have been strengthened by using two blinded reviewers to extract data and present data with supporting kappa values.

The generalizability of the results of this overview is limited to undergraduate nursing students. Although, we consider it a strength that the results represent a broad international perspective on framing EBP teaching, as well as teaching methods and outcomes used among educators in EBP. Primary studies exist among occupational therapy and physiotherapy students [ 5 , 29 ] but have not been systematically synthesized. However, the evidence is almost non-existent among midwife, nutrition and health and biomedical laboratory science students. This has implications for further research efforts because evidence from within these student populations is paramount for future proofing the quality assurance of clinical evidence-based healthcare practice.

Another implication is the need to compare how to frame the EBP teaching, and the methods used both inter-and mono professionally among these professional bachelor-degree students. Lastly, we support the recommendations of Bala et al. of using validated tools to increase the focus on measuring behavior change in clinical practice and patient outcomes, and to report in accordance with the GREET guidelines for educational intervention studies [ 9 ].

This overview demonstrates a variety of approaches to promote EBP teaching among professional bachelor-degree healthcare students. Teaching EBP is based on collaboration with clinical practice and the use of different approaches to frame the teaching as well as different teaching methods. Furthermore, this overview has elucidated, that interventions often are evaluated according to changes in the student’s skills, knowledge and attitudes towards EBP, but very rarely on self-efficacy, behaviors, benefits to the patients or reaction to the educational experience as suggested by the Sicily group. This might indicate that educators need to move on to measure the effect of EBP on outcomes comprising all categories, which are important to enhance sustainable behavior and transition of knowledge into the context of practices where better healthcare education should have an impact. In our perspective these gaps in the EBP teaching are best met by focusing on more collaboration with clinical practice which is the context where the final endpoint of teaching EBP should be anchored and evaluated.

Availability of data and materials

The datasets used an/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Evidence-Based Practice

Professional bachelor-degree healthcare programs

Mazurek Melnyk B, Fineout-Overholt E. Making the Case for Evidence-Based Practice and Cultivalting a Spirit of Inquiry. I: Mazurek Melnyk B, Fineout-Overholt E, redaktører. Evidence-Based Practice in Nursing and Healthcare A Guide to Best Practice. 4. ed. Wolters Kluwer; 2019. p. 7–32.

Tilson JK, Kaplan SL, Harris JL, Hutchinson A, Ilic D, Niederman R, et al. Sicily statement on classification and development of evidence-based practice learning assessment tools. BMC Med Educ. 2011;11(78):1–10.

Google Scholar  

Connor L, Dean J, McNett M, Tydings DM, Shrout A, Gorsuch PF, et al. Evidence-based practice improves patient outcomes and healthcare system return on investment: Findings from a scoping review. Worldviews Evid Based Nurs. 2023;20(1):6–15.

Article   PubMed   Google Scholar  

Dawes M, Summerskill W, Glasziou P, Cartabellotta N, Martin J, Hopayian K, et al. Sicily statement on evidence-based practice. BMC Med Educ. 2005;5(1):1–7.

Article   PubMed   PubMed Central   Google Scholar  

Larsen CM, Terkelsen AS, Carlsen AF, Kristensen HK. Methods for teaching evidence-based practice: a scoping review. BMC Med Educ. 2019;19(1):1–33.

Article   CAS   Google Scholar  

World Health Organization. Nurse educator core competencies. 2016 https://apps.who.int/iris/handle/10665/258713 Accessed 21 Mar 2023.

Saunders H, Gallagher-Ford L, Kvist T, Vehviläinen-Julkunen K. Practicing healthcare professionals’ evidence-based practice competencies: an overview of systematic reviews. Worldviews Evid Based Nurs. 2019;16(3):176–85.

Young T, Rohwer A, Volmink J, Clarke M. What Are the Effects of Teaching Evidence-Based Health Care (EBHC)? Overview of Systematic Reviews PLoS ONE. 2014;9(1):1–13.

Bala MM, Poklepović Peričić T, Zajac J, Rohwer A, Klugarova J, Välimäki M, et al. What are the effects of teaching Evidence-Based Health Care (EBHC) at different levels of health professions education? An updated overview of systematic reviews. PLoS ONE. 2021;16(7):1–28.

Article   Google Scholar  

Copenhagen University. Bachelor in medicine. 2024 https://studier.ku.dk/bachelor/medicin/undervisning-og-opbygning/ Accessed 31 Jan 2024.

Ministery of Higher Education and Science. Professional bachelor programmes. 2022 https://ufm.dk/en/education/higher-education/university-colleges/university-college-educations Accessed 31 Jan 2024.

Patelarou AE, Mechili EA, Ruzafa-Martinez M, Dolezel J, Gotlib J, Skela-Savič B, et al. Educational Interventions for Teaching Evidence-Based Practice to Undergraduate Nursing Students: A Scoping Review. Int J Env Res Public Health. 2020;17(17):1–24.

Wakibi S, Ferguson L, Berry L, Leidl D, Belton S. Teaching evidence-based nursing practice: a systematic review and convergent qualitative synthesis. J Prof Nurs. 2021;37(1):135–48.

Fiset VJ, Graham ID, Davies BL. Evidence-Based Practice in Clinical Nursing Education: A Scoping Review. J Nurs Educ. 2017;56(9):534–41.

Pollock M, Fernandes R, Becker L, Pieper D, Hartling L. Chapter V: Overviews of Reviews. I: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 62. 2021 https://training.cochrane.org/handbook Accessed 31 Jan 2024.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, m.fl. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:1-9

Covidence. Covidence - Better systematic review management. https://www.covidence.org/ Accessed 31 Jan 2024.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;21(358):1–9.

Joanna Briggs Institute. Critical Appraisal Tools. https://jbi.global/critical-appraisal-tools Accessed 31 Jan 2024.

Horntvedt MT, Nordsteien A, Fermann T, Severinsson E. Strategies for teaching evidence-based practice in nursing education: a thematic literature review. BMC Med Educ. 2018;18(1):1–11.

Ramis M-A, Chang A, Conway A, Lim D, Munday J, Nissen L. Theory-based strategies for teaching evidence-based practice to undergraduate health students: a systematic review. BMC Med Educ. 2019;19(1):1–13.

Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, et al. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021;10(1):1–19.

Song CE, Jang A. Simulation design for improvement of undergraduate nursing students’ experience of evidence-based practice: a scoping-review protocol. PLoS ONE. 2021;16(11):1–6.

Cui C, Li Y, Geng D, Zhang H, Jin C. The effectiveness of evidence-based nursing on development of nursing students’ critical thinking: A meta-analysis. Nurse Educ Today. 2018;65:46–53.

Ghaffari R, Shapoori S, Binazir MB, Heidari F, Behshid M. Effectiveness of teaching evidence-based nursing to undergraduate nursing students in Iran: a systematic review. Res Dev Med Educ. 2018;7(1):8–13.

Wu Y, Brettle A, Zhou C, Ou J, Wang Y, Wang S. Do educational interventions aimed at nurses to support the implementation of evidence-based practice improve patient outcomes? A systematic review. Nurse Educ Today. 2018;70:109–14.

Ramis MA, Chang A, Nissen L. Undergraduate health students’ intention to use evidence-based practice after graduation: a systematic review of predictive modeling studies. Worldviews Evid Based Nurs. 2018;15(2):140–8.

Albarqouni L, Hoffmann T, Straus S, Olsen NR, Young T, Ilic D, et al. Core competencies in evidence-based practice for health professionals: consensus statement based on a systematic review and Delphi survey. JAMA Netw Open. 2018;1(2):1–12.

Hitch D, Nicola-Richmond K. Instructional practices for evidence-based practice with pre-registration allied health students: a review of recent research and developments. Adv Health Sci Educ Theory Pr. 2017;22(4):1031–45.

Download references

Acknowledgements

The authors would like to acknowledge research librarian Rasmus Sand for competent support in the development of literature search strategies.

This work was supported by the University College of South Denmark, which was not involved in the conduct of this study.

Author information

Authors and affiliations.

Nursing Education & Department for Applied Health Science, University College South Denmark, Degnevej 17, 6705, Esbjerg Ø, Denmark

Lea D. Nielsen

Department of Oncology, Hospital of Lillebaelt, Beriderbakken 4, 7100, Vejle, Denmark

Mette M. Løwe

Biomedical Laboratory Science & Department for Applied Health Science, University College South Denmark, Degnevej 17, 6705, Esbjerg Ø, Denmark

Francisco Mansilla

Physiotherapy Education & Department for Applied Health Science, University College South Denmark, Degnevej 17, 6705, Esbjerg Ø, Denmark

Rene B. Jørgensen

Occupational Therapy Education & Department for Applied Health Science, University College South Denmark, Degnevej 17, 6705, Esbjerg Ø, Denmark

Asviny Ramachandran

Department for Applied Health Science, University College South Denmark, Degnevej 17, 6705, Esbjerg Ø, Denmark

Bodil B. Noe

Centre for Clinical Research and Prevention, Section for Health Promotion and Prevention, Bispebjerg and Frederiksberg Hospital, Nordre Fasanvej 57, 2000, Frederiksberg, Denmark

Heidi K. Egebæk

You can also search for this author in PubMed   Google Scholar

Contributions

All authors have made substantial contributions to the conception and design of the study, acquisition of data, analysis, and interpretation of data, writing the main manuscript, preparing figures and tables and revising the manuscripts.

Corresponding author

Correspondence to Lea D. Nielsen .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., supplementary material 4., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Nielsen, L.D., Løwe, M.M., Mansilla, F. et al. Interventions, methods and outcome measures used in teaching evidence-based practice to healthcare students: an overview of systematic reviews. BMC Med Educ 24 , 306 (2024). https://doi.org/10.1186/s12909-024-05259-8

Download citation

Received : 29 May 2023

Accepted : 04 March 2024

Published : 19 March 2024

DOI : https://doi.org/10.1186/s12909-024-05259-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• MH "Students, Health occupations+"
• MH "Students, occupational therapy"
• MH "Students, physical therapy"
• MH "Students, Midwifery"
• “Students, Nursing"[Mesh]
• “Teaching"[Mesh]
• MH "Teaching methods+"
• "Evidence-based practice"[Mesh]

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.11(2); Spring 2007

Critical Appraisal of Clinical Studies: An Example from Computed Tomography Screening for Lung Cancer

Introduction.

Every physician is familiar with the impact that findings from studies published in scientific journals can have on medical practice, especially when the findings are amplified by popular press coverage and direct-to-consumer advertising. New studies are continually published in prominent journals, often proposing significant and costly changes in clinical practice. This situation has the potential to adversely affect the quality, delivery, and cost of care, especially if the proposed changes are not supported by the study's data. Reports about the results of a single study do not portray the many considerations inherent in a decision to recommend or not recommend an intervention in the context of a large health care organization like Kaiser Permanente (KP).

… in many cases, published articles do not discuss or acknowledge the weaknesses of the research …

Moreover, in many cases, published articles do not discuss or acknowledge the weaknesses of the research, and the reader must devote a considerable amount of time to identifying them. This creates a problem for the busy physician, who often lacks the time for systematic evaluation of the methodologic rigor and reliability of a study's findings. The Southern California Permanente Medical Group's Technology Assessment and Guidelines (TAG) Unit critically appraises studies published in peer-reviewed medical journals and provides evidence summaries to assist senior leaders and physicians in applying study findings to clinical practice. In the following sections, we provide a recent example of the TAG Unit's critical appraisal of a highly publicized study, highlighting key steps involved in the critical appraisal process.

Critical Appraisal: The I-ELCAP Study

In its October 26, 2006, issue, the New England Journal of Medicine published the results of the International Early Lung Cancer Action Program (I-ELCAP) study, a large clinical research study examining annual computed tomography (CT) screening for lung cancer in asymptomatic persons. Though the authors concluded that the screening program could save lives, and suggested that this justified screening asymptomatic populations, they offered no discussion of the shortcomings of the study. This report was accompanied by a favorable commentary containing no critique of the study's limitations, 1 and it garnered positive popular media coverage in outlets including the New York Times , CNN, and the CBS Evening News . Nevertheless, closer examination shows that the I-ELCAP study had significant limitations. Important harms of the study intervention were ignored. A careful review did not support the contention that screening for lung cancer with helical CT is clinically beneficial or that the benefits outweigh its potential harms and costs.

Critical appraisals of published studies address three questions:

• Are the study's results valid?
• What are the results?
• Will the results help in caring for my patient?

We discuss here the steps of critical appraisal in more detail and use the I-ELCAP study as an example of the way in which this process can identify important flaws in a given report.

Are the Study's Results Valid?

Assessing the validity of a study's results involves addressing three issues. First, does the study ask a clearly focused clinical question ? That is, does the paper clearly define the population of interest, the nature of the intervention, the standard of care to which the intervention is being compared, and the clinical outcomes of interest? If these are not obvious, it can be difficult to determine which patients the results apply to, the nature of the change in practice that the article proposes, and whether the intervention produces effects that both physician and patient consider important.

The clinical question researched in the I-ELCAP study 2 of CT screening for lung cancer is only partly defined. Although the outcomes of interest—early detection of lung carcinomas and lung cancer mortality—are obvious and the intervention is clearly described, the article is less clear with regard to the population of interest and the standard of care. The study population was not recruited through a standardized protocol. Rather, it included anyone deemed by physicians at the participating sites to be at above-average risk for lung cancer. Nearly 12% of the sample were individuals who had never smoked nor been exposed to lung carcinogens in the workplace; these persons were included on the basis of an unspecified level of secondhand smoke exposure. It is impossible to know whether they were subjected to enough secondhand smoke to give them a lung cancer risk profile similar to that of a smoker. It is also not obvious what was considered the standard of care in the I-ELCAP study. Although it is common for screening studies to compare intervention programs with “no screening,” the lack of a comparison group in this study leaves the standard entirely implicit.

Second, is the study's design appropriate to the clinical question ? Depending on the nature of the treatment or test, some study designs may be more appropriate to the question than others. The randomized controlled trial, in which a study subject sample is randomly divided into treatment and control groups and the clinical outcomes for each group are evaluated prospectively, is the gold standard for studies of screening programs and medical therapies. 3, 4 Cohort studies, in which a single group of study subjects is studied either prospectively or at a single point in time, are better suited to assessments of diagnostic or prognostic tools 3 and are less valid when applied to screening or treatment interventions. 5 Screening evaluations conducted without a control group may overestimate the effectiveness of the program relative to standard care by ignoring the benefits of standard care. Other designs, such as nonrandomized comparative studies, retrospective studies, case series, or case reports, are rarely appropriate for studying any clinical question. 5 However, a detailed discussion of threats to validity arising within particular study designs is beyond the scope of this article.

The I-ELCAP study illustrates the importance of this point. The nature of the intervention (a population screening program) called for a randomized controlled trial design, but the study was in fact a case series. Study subjects were recruited over time; however, because the intervention was an ongoing annual screening program, the number of CT examinations they received clearly varied, and it is impossible to tell from the data presented how the number of examinations per study subject is distributed within the sample. With different study subjects receiving different “doses” of the intervention, it thus becomes impossible to interpret the average effect of screening in the study. In particular, it is unclear how to interpret the ten-year survival curves the report presents; if the proportion of study subjects with ten years of data was relatively small, the survival rates would be very sensitive to the statistical model chosen to estimate them.

The lack of a control group also poses problems. Without a comparison group drawn from the same population, it is impossible to determine whether early detection through CT screening is superior to any other practice, including no screening. Survival data in a control group of unscreened persons would allow us to determine the lead time, or the interval of time between early detection of the disease and its clinical presentation. If individuals in whom stage I lung cancer was diagnosed would have survived for any length of time in the absence of screening, the mortality benefit of CT screening would have been overstated. Interpreting this interval as life saved because of screening is known as lead-time bias. The lack of a comparable control group also raises the question of overdiagnosis; without survival data from control subjects, it cannot be known how many of the lung cancers detected in I-ELCAP would have progressed to an advanced stage.

… does the paper clearly define the population of interest, the nature of the intervention, the standard of care to which the intervention is being compared, and the clinical outcomes of interest?

The types of cancers detected in the baseline and annual screening components of the I-ELCAP study only underscore this concern. Of the cancers diagnosed at baseline, only 9 cancers (3%) were small cell cancer, 263 (70%) were adenocarcinoma, and 45 (22%) were squamous cell cancer. Small cell and squamous cell cancers are almost always due to smoking. Data from nationally representative samples of lung cancer cases generally show that 20% of lung cancers are small cell, 40% are adenocarcinoma, and 30% are squamous cell. The prognosis for adenocarcinoma is better even at stage I than the prognoses for other cell types, especially small cell. 6 The I-ELCAP study data suggest that baseline screening might have detected the slow-growing tumors that would have presented much later.

A third question is whether the study was conducted in a methodologically sound way . This point concerns the conduct of the study and whether additional biases apart from those introduced by the design might have emerged. A discussion of the numerous sources of bias, including sample selection and measurement biases, is beyond the scope of this article. In randomized controlled trials of screening programs or therapies, it is important to know whether the randomization was done properly, whether the study groups were comparable at baseline, whether investigators were blinded to group assignments, whether contamination occurred (ie, intervention or control subjects not complying with study assignment), and whether intent-to-treat analyses were performed. In any prospective study, it is important to check whether significant attrition occurred, as a high dropout rate can greatly skew results.

In the case of the I-ELCAP study, 2 these concerns are somewhat overshadowed by those raised by the lack of a randomized design. It does not appear that the study suffered from substantial attrition over time. Diagnostic workups in the study were not defined by a strict protocol (protocols were recommended to participating physicians, but the decisions were left to the physician and the patient). This might have led to variation in how a true-positive case was determined.

What Are the Results?

Apart from simply describing the study's findings, the results component of critical appraisal requires the reader to address the size of the treatment effect and the precision of the treatment-effect estimate in the case of screening or therapy evaluations. The treatment effect is often expressed as the average difference between groups on some objective outcome measure (eg, SF-36 Health Survey score) or as a relative risk or odds ratio when the outcome is dichotomous (eg, mortality). In cohort studies without a comparison group, the treatment effect is frequently estimated by the difference between baseline and follow-up measures of the outcome, though such estimates are vulnerable to bias. The standard errors or confidence intervals around these estimates are the most common measures of precision.

The results of the I-ELCAP study 2 were as follows. At the baseline screening, 4186 of 31,567 study subjects (13%) were found by CT to have nodules qualifying as positive test results; of these, 405 (10%) were found to have lung cancer. An additional five study subjects (0.015%) with negative results at the baseline CT were given a diagnosis of lung cancer at the first annual CT screening, diagnoses that were thus classified as “interim.” At the subsequent annual CT screenings (delivered 27,456 times), 1460 study subjects showed new noncalcified nodules that qualified as significant results; of these, 74 study subjects (5%) were given a diagnosis of lung cancer. Of the 484 diagnoses of lung cancer, 412 involved clinical stage I disease. Among all patients with lung cancer, the estimated ten-year survival rate was 88%; among those who underwent resection within one month of diagnosis, estimated ten-year survival was 92%. Implied by these figures (but not stated by the study authors) is that the false-positive rate at the baseline screening was 90%—and 95% during the annual screens. Most importantly, without a control group, it is impossible to estimate the size or precision of the effect of screening for lung cancer. The design of the I-ELCAP study makes it impossible to estimate lead time in the sample, which was likely substantial, and again, the different “doses” of CT screening received by different study subjects make it impossible to determine how much screening actually produces the estimated benefit.

… would my patient have met the study's inclusion criteria, and if not, is the treatment likely to be similarly effective in my patient?

Will the Results Help in Caring for My Patient?

Answering the question of whether study results help in caring for one's patients requires careful consideration of three points. First, were the study's patients similar to my patient ? That is, would my patient have met the study's inclusion criteria, and if not, is the treatment likely to be similarly effective in my patient? This question is especially salient when we are contemplating new indications for a medical therapy. In the I-ELCAP study, 2 it is unclear whether the sample was representative of high-risk patients generally; insofar as nonsmokers exposed to secondhand smoke were recruited into the trial, it is likely that the risk profiles of the study's subjects were heterogeneous. The I-ELCAP study found a lower proportion of noncalcified nodules (13%) than did four other chest CT studies evaluated by our group (range, 23% to 51%), suggesting that it recruited a lower-risk population than these similar studies did. Thus, the progression of disease in the presence of CT screening in the I-ELCAP study might not be comparable to disease progression in any other at-risk population, including a population of smokers.

The second point for consideration is whether all clinically important outcomes were considered . That is, did the study evaluate all outcomes that both the physician and the patient are likely to view as important? Although the I-ELCAP study did provide data on rates of early lung cancers detected and lung cancer mortality, it did not address the question of morbidity or mortality related to diagnostic workup or cancer treatment, which are of interest in this population.

Finally, physicians should consider whether the likely treatment benefits are worth the potential harms and costs . Frequently, these considerations are blunted by the enthusiasm that new technologies engender. Investigators in studies such as I-ELCAP are often reluctant to acknowledge or discuss these concerns in the context of interventions that they strongly believe to be beneficial. The I-ELCAP investigators did not report any data on or discuss morbidity related to diagnostic procedures or treatment, and they explicitly considered treatment-related deaths to have been caused by lung cancer. Insofar as prior research has demonstrated that few pulmonary nodules prove to be cancerous, and because few positive test results in the trial led to diagnoses of lung cancer, it is reasonable to wonder whether the expected benefit to patients is offset by the difficulties and risks of procedures such as thoracotomy. The study report also did not discuss the carcinogenic risk associated with diagnostic imaging procedures. Data from the National Academy of Sciences' Seventh report on health risks from exposure to low levels of ionizing radiation 7 suggest that radiation would cause 11 to 22 cases of cancer in 10,000 persons undergoing one spiral CT. This risk would be greatly increased by a strategy of annual screening via CT, which would include many additional CT and positron-emission tomography examinations performed in diagnostic follow-ups of positive screening results. Were patients given annual CT screening for all 13 years of the I-ELCAP study, they would have absorbed an estimated total effective dose of 130 to 260 mSv, which would be associated with approximately 150 to 300 cases of cancer for every 10,000 persons screened. This is particularly critical for the nonsmoking study subjects in the I-ELCAP sample, who might have been at minimal risk for lung cancer; for them, radiation from screening CTs might have posed a significant and unnecessary health hazard.

In addition to direct harms, Eddy 5 and other advocates of evidence-based critical appraisal have argued that there are indirect harms to patients when resources are spent on unnecessary or ineffective forms of care at the expense of other services. In light of such indirect harms, the balance of benefits to costs is an important consideration. The authors of I-ELCAP 2 argued that the utility and cost-effectiveness of population mammography supported lung cancer screening in asymptomatic persons. A more appropriate comparison would involve other health care interventions aimed at reducing lung cancer mortality, including patient counseling and behavioral or pharmacologic interventions aimed at smoking cessation. Moreover, the authors cite an upper-bound cost of $200 for low-dose CT as suggestive of the intervention's cost-effectiveness. Although the I-ELCAP study data do not provide enough information for a valid cost-effectiveness analysis, the data imply that the study spent nearly $13 million on screening and diagnostic CTs. The costs of biopsies, positron-emission tomography scans, surgeries, and early-stage treatments were also not considered.

… did the study evaluate all outcomes that both the physician and the patient are likely to view as important?

Using the example of a recent, high-profile study of population CT screening for lung cancer, we discussed the various considerations that constitute a critical appraisal of a clinical trial. These steps include assessments of the study's validity, the magnitude and implications of its results, and its relevance for patient care. The appraisal process may appear long or tedious, but it is important to remember that the interpretation of emerging research can have enormous clinical and operational implications. In other words, in light of the stakes, we need to be sure that we understand what a given piece of research is telling us. As our critique of the I-ELCAP study report makes clear, even high-profile studies reported in prominent journals can have important weaknesses that may not be obvious on a cursory read of an article. Clearly, few physicians have time to critically evaluate all the research coming out in their field. The Technology Assessment and Guidelines Unit located in Southern California is available to assist KP physicians in reviewing the evidence for existing and emerging medical technologies.

Acknowledgments

Katharine O'Moore-Klopf of KOK Edit provided editorial assistance.

• Unger M. A pause, progress, and reassessment in lung cancer screening. N Engl J Med. 2006 Oct 26; 355 (17):1822–4. [ PubMed ] [ Google Scholar ]
• The International Early Lung Cancer Action Program Investigators. Survival of patients with stage I lung cancer detected on CT screening. N Engl J Med. 2006 Oct 26; 355 (17):1763–71. [ PubMed ] [ Google Scholar ]
• Campbell DT, Stanley JC. Experimental and quasi-experimental designs for research. Chicago: Rand McNally; 1963. [ Google Scholar ]
• Holland P. Statistics and causal inference. J Am Stat Assoc. 1986; 81 :945–60. [ Google Scholar ]
• Eddy DM. A manual for assessing health practices and designing practice policies: the explicit approach. Philadelphia: American College of Physicians; 1992. [ Google Scholar ]
• Kufe DW, Pollock RE, Weichselbaum RR, et al., editors. Cancer Medicine. 6th ed. Hamilton, Ontario, Canada: BC Decker; 2003. (editors) [ Google Scholar ]
• National Academy of Sciences. Health risks from exposure to low levels of ionizing radiation: BEIR VII. Washington, DC: National Academies Press; 2005. [ PubMed ] [ Google Scholar ]