LibGuides: Knowledge Syntheses: A How-To Guide: Appraisal and Analysis

Why Critically Appraise Studies?

Not all studies are trustworthy. In order for your systematic review to represent the best possible evidence, your team must critically appraise each study thoroughly before it is included. The criteria by which you assess included studies should be clearly outlined in your report. You should also maintain a record of which studies have been left out, and the grounds for which they were excluded.

Below you can find a compilation of critical appraisal checklists and guides sorted by the type of study that they appraise.

Critical Appraisal Checklists

Checklist collections for multiple study designs

JBI Critical Appraisal Tools
JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers. Includes tools for Analytical Cross Sectional Studies, Case Control Studies, Case Reports, Case Series, Cohort Studies, Diagnostic Test Accuracy Studies, Economic Evaluations, Prevalence Studies, Qualitative Research, Quasi-Experimental Studies, Randomized Controlled Trials, Systematic Reviews, and Text and Opinion.
NIH National Heart, Lung, and Blood Institute Study Quality Assessment Tools
Includes checklists and guidance on use for various study types, including: Controlled Intervention Studies, Systematic Reviews and Meta-Analyses, Observational Cohort & Cross-Sectional Studies, Case-Control Studies, Befor-After (Pre-Post) Studies with no Control Group, and Case Series Studies.
Latitudes Network
LATITUDES is a library of validity assessment tools for use in evidence syntheses. Researchers can register tools under development. The website also provides access to training resources.
Repository of Quality Assessment and Risk of Bias Tools
A resource for finding and selecting a risk of bias or quality assessment tool for evidence synthesis projects. This project was created and is maintained by Duke University's Medical Center Library & Archives and is under continual development.
more...less...
Downloadable Excel sheet with resources links, descriptions, and characteristics included.

Randomized Control Trials (RCTs)

Cochrane Risk of Bias tool for Randomized Trials
Cite the tool as: Higgins JPT, Sterne JAC, Savović J, Page MJ, Hróbjartsson A, Boutron I, Reeves B, Eldridge S. A revised tool for assessing risk of bias in randomized trials In: Chandler J, McKenzie J, Boutron I, Welch V (editors). Cochrane Methods. Cochrane Database of Systematic Reviews 2016, Issue 10 (Suppl 1). dx.doi.org/10.1002/14651858.CD201601.
The Consolidated Standards of Reporting Trials (CONSORT) Statement
A detailed document that outlines an explanation and elaboration of the CONSORT statement for reporting randomized controlled trials. It also includes a critical appraisal flow diagram.
Jadad Scale
This article outlines the Jadad Scale for critically appraising individual studies. A three-question appraisal questionnaire in included in the Appendix.
Jadad, A.R.; Moore R.A., Carroll D., Jenkinson C., Reynolds D.J.M., Gavaghan D.J., McQuay H.J. (1996). "Assessing the quality of reports of randomized clinical trials: Is blinding necessary?". Controlled Clinical Trials 1996 17 (1): 1–12. PMID 8721797

Non-Randomized Studies or Observational Studies

Avoiding Bias in Selecting Studies
Methods Guide by the Agency for Healthcare Research and Quality (AHRQ)
AXIS
A critical appraisal tool used to assess the quality of cross-sectional (prevalence) studies.
The Case Series Studies Critical Appraisal Checklist
A quality appraisal checklist for case studies.
Critical Appraisal Tools
Includes checklists for various studies such as analytical cross-sectional studies, case-control studies, case reports, case series, cohort studies, diagnostic test accuracy studies, prevalence studies, quasi-experimental studies, randomized controlled trials, and systematic reviews.
The Critical Review Form - Quantitative Studies
MINORS
Methodological Index for Non-Randomized Studies (MINORS). A tool for assessing non-randomized interventional studies.
Newcastle Ottawa Scale for non-randomized studies
The Newcastle-Ottawa Scale (NOS) is an ongoing collaboration between the Universities of Newcastle, Australia and Ottawa, Canada. It was developed to assess the quality of nonrandomised studies with its design, content and ease of use directed to the task of incorporating the quality assessments in the interpretation of meta-analytic results.
ROBINS-E
Risk of Bias for non-randomized (observational) studies or cohorts of Exposures. Useful for conducting systematic reviews that include non-randomized studies.
ROBINS-I
The ROBINS-I tool (Risk Of Bias In Non-randomized Studies - of Interventions) was developed by members of the Cochrane Bias Methods Group and the Cochrane Non-Randomised Studies Methods Group.
Study Quality Assessment Tools
Study Quality Assessment Tools for Controlled Intervention Studies, Systematic Reviews & Meta-Analyses, Observational Cohort and Cross-Sectional Studies, Case-Control Studies, Before-After (Pre-Post) Studies With No Control Group, Case Series Studies.
Tools for Critical Appraisal
Duke University's guide to critical appraisal, including tools, articles, checklists, and templates for appraising different types of studies and reviews

Animal Research

ARRIVE guidelines
The Animal Research: Reporting of In Vivo Experiments (ARRIVE) guidelines. A checklist for improving the reliability and quality of reporting for animal research.
Critical Appraisal of Studies Using Laboratory Animal Models
Includes two tables covering the components of critical appraisal.
SYRCLE's RoB tool
Systematic Review Centre for Laboratory Animal Experimentation (SYRCLE). A risk of bias assessment tool for assessing animal studies.

Diagnostic Accuracy

CASP: Diagnostic Test Checklist
A methodological checklist that provides key criteria relevant to diagnostic test studies.
Diagnostic Accuracy Studies Worksheet
Helps assess the validity and applicability of included diagnostic tests.
QUADAS-2
A revised tool for the quality assessment of diagnostic accuracy studies. The current version of QUADAS and the tool that we recommend for use in systematic reviews to evaluate the risk of bias and applicability of primary diagnostic accuracy studies.
STARD 2015 Checklist
Standards for the Reporting of Diagnostic accuracy studies (2015). Helps determine the accuracy and completeness of reporting of studies of diagnostic accuracy. The STARD Statement consists of a 30 item checklist and flow diagram.

Systematic Reviews

(descriptions copy pasted from https://guides.library.utoronto.ca/c.php?g=713309&p=5121959)

AMSTAR-2
A critical appraisal tool for systematic reviews that include randomised or nonrandomised studies of healthcare interventions, or both.

Used to assess the methodological quality of a systematic review and as a guide to performing a systematic review. Two agreements are required during quality assessment ensuring lower risk of bias. AMSTAR has guidelines explaining each outlined item.
CASP Checklist
A methodological checklist that provides key criteria relevant to systematic reviews.
Mixed Methods Appraisal Tool
Developed by researchers at McGill University, the MMAT has been designed for the appraisal stage of complex systematic literature reviews that include qualitative, quantitative and mixed methods studies (mixed studies reviews).
ROBIS
Tool for assessing the risk of bias in systematic reviews (rather than in primary studies), offering information to help you complete a ROBIS assessment, and resources to help you present the results of your ROBIS assessment.
Systematic Review Critical Appraisal Worksheet
A checklist with questions for assessing the reliability, importance, and applicability of the studies included based on the alignment of the research question and eligibility criteria.

Grey Literature

AACODS
Jess Tyndall, Medical Librarian and Head of the Gus Fraenkel Medical Library at Flinders University, developed the AACODS checklist (Authority, Accuracy, Coverage, Objectivity, Date, Significance) as an evaluation and Critical Appraisal Tool specifically for use with grey literature sources.
more...less...
Tyndall, J. AACODS Checklist. Flinders University, 2010. Available from http://dspace.flinders.edu.au/dspace/

Tools and resources concerning appraisal of knowledge involving Indigenous communities

Assessing the quality of health research from an Indigenous perspective: the Aboriginal and Torres Strait Islander quality appraisal tool
Harfield, S., Pearson, O., Morey, K. et al. Assessing the quality of health research from an Indigenous perspective: the Aboriginal and Torres Strait Islander quality appraisal tool. BMC Med Res Methodol 20, 79 (2020). https://doi.org/10.1186/s12874-020-00959-3
Promoting cultural rigour through critical appraisal tools in First Nations peoples’ research
Lock, M.J., Walker, T. and Browne, J. (2021), Promoting cultural rigour through critical appraisal tools in First Nations peoples’ research. Australian and New Zealand Journal of Public Health, 45: 210-211.

Books about Critical Appraisal

Making Sense of Critical Appraisal by Olajide Ajetunmobi
Call Number: W 20.5 A312m 2002
Available online and in print
Evidence-based Practice Workbook by Janet Salisbury; Paul Glasziou; Chris Del Mar
Call Number: WB 102 G549 2007
Available online and in print
Studying a Study & Testing a Test: How to Read the Medical Evidence by Richard K Riegelman
Call Number: WA 950 R554s 2005
Available online and in print
Users' Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice by Gordon Guyatt et al.
Call Number: WB 102 U84 2015
ISBN: 9780071790710
Call Number: WB 102 U84 2015
Available online and in print
How To Read a Paper: The Basics of Evidence-Based Medicine by Trisha Greenhalgh
Call Number: W 20.5 G813h 2010
2014 edition available online
2010 Edition available in print

Tools for Analysis

Dalhousie subscribed software
NVivo (for qualitative analysis)
Available for download via Dalhousie subscribed software
RevMan (From Cochrane)
R for meta-analysis
Balduzzi S, Rücker G, Schwarzer G. How to perform a meta-analysis with R: a practical tutorial. Evidence-Based Mental Health 2019;22:153-160.
SPSS (Statistical Product and Service Solutions)
Available for download via Dalhousie subscribed software
STATA/SE (standard Edition)
Available for download via Dalhousie subscribed software