Skip to Main Content

Knowledge Syntheses: A How-To Guide

Overview of systematic review steps and resources to assist researchers conducting reviews

What is a Systematic Review?

What is it?

"Systematic reviews ask a specific question about the effectiveness of a treatment and answer it by summarising evidence that meets a set of pre-specified criteria." [1]

 

systematic review is a thorough compilation and analysis of all known evidence on a given subject. In order to be formally recognised by publishers and repositories, a systematic review must include the following elements:

  1. A clearly defined research question and protocol (research plan)
    • The research question is often developed after performing preliminary research on the subject, ensuring that it is viable for a systematic review
    • Thoroughly search the literature to ensure no other systematic review already exists on your topic
       
  2. Evidence of a rigorous search process
    • The reason why it is called a "systematic" review is because of the systematic search process that is required to uncover all of the evidence on a given subject
    • Systematic searching demands a carefully planned and thorough search strategy that will recall the maximum number of relevant results
    • Systematic reviews must include the exact search strategy used to find literature in each database
       
  3. Inclusion and exclusion criteria
    •  Not all evidence found during the search process will be relevant or appropriate for your research question.
    • Clearly define the criteria you use to decide which studies should and should not be included in your analysis
       
  4. Critical appraisal and bias assessment of all included studies
    • If a study is to be included in your review, the quality of its evidence must be critically appraised by each member of your research team
    • All studies carry an inherent risk of bias, studies should be thoroughly evaluated on their impartiality. This step ensures that your systematic review will represent the highest possible quality of evidence.
       
  5. An in-depth report outlining the process of finding and appraising literature, extracting data, measuring bias, and analysing results.

 

Outline of Stages

1.  Gathering your team (Minimum of two reviewers with a third to serve as a tiebreaker)

  • A systematic review must have a team of two or greater.
  • Choose team members wisely and based on areas of expertise.
  • The tiebreaker is used to resolve disagreements between the reviewers for stages of the review that are blinded (screening, data extraction, critical appraisal) and are completed by two independent reviewers
     

2.  Questioning (Define a narrow question, may use PICO)  

The PICO format is commonly used to define the research question into one that is a searchable question. In some cases, the PICO format may not work and another format can be used.

P: Patient/Problem or Person

I: Intervention/Exposure/Therapy or Treatment

C: (optional) Comparison (e.g. a placebo or another drug/therapy)

O: Outcome(s), what is the expected or anticipated outcome you will find in the literature?

A systematic review question should also be narrow in scope in order to be able to conduct the review in one year. The purpose of a systematic review is to draw conclusions based on the evidence to answer that one well-defined and narrow question. 
 

3.  Planning (Create a protocol, plan methods & strategies, register protocol)

  • Having a plan in place is essential to a good quality review and by spending more time planning before the review takes place, you could avoid issues or errors that may slow down the process or be detrimental to the review.
  • Evaluate if the review is feasible, checking to make sure there are no conflicting reviews and also ensuring that there is a plan to carry out each stage of the review.
  • Setting goals and timelines for the review is important as well as mapping out how the review project will be managed.

 

This is also put into a document called a protocol.

  • Registering the protocol is optional but highly recommended.
  • The protocol also includes defining a priori what the selection criteria will be for the review in terms of inclusion and exclusion criteria for what studies should be screened by for inclusion in the review. 
     

4.  Searching/Screening (Exhaustive, transparent & repeatable searching for evidence/selecting studies) 

  • Includes search in several places including:
    • multiple databases
    • grey literature/clinical trial registries
    • hand-searching of the literature (performed by the subject matter expert)
       
  • It is best practice to involve a librarian or an information specialist in:
    • creating the comprehensive search
    • translating the search for databases or grey literature
    • documenting the search
    • deduplicating the repeating references in a citation manager
    • writing the search methods for the review
       
  • The search stage may also include contacting other experts in the field to identify publications that have not been published yet
     
  • Include both published and unpublished literature to avoid a type of publication bias
    • Also called positive outcomes bias since positive outcomes are more likely to be published
       
  • Screening is done in two phases:
    • The first phase is screening titles/abstracts (together)
      • Screening is done independently by two reviewers, with a third reviewer serving as a tiebreaker
      • Reviewers should not move on to the full text screening phase until they have screened all of the titles and abstracts and each is a clear 'Yes' or 'No' without 'maybes' remaining
    • The second phase is screening full texts
      • Acquire and read all of the full texts and screen them based on the studies selection criteria
      • Only Yes's are included in the review but all No's must have a reason listed for exclusion
         
  • The new PRISMA 2020 requires reporting of study Near Misses too
    • Near misses are any studies that did not meet inclusion in the review but were very close to being included

There are tools designed specifically to assist with the systematic review screening phase
 

5.  Managing & reporting 

  • All methods must be fully reported, transparent and reproducible
  • The methods reported must also follow the recommended reporting guidance such as the PRISMA 2020
  • Reporting guidance can be identified by searching the Equator Network https://www.equator-network.org/
  • Reporting guidance may be modified for review types similar to the systematic review
  • Refer to the many PRISMA 2020 extensions http://www.prisma-statement.org/Extensions/ for more information
     

6.  Data Extraction/Synthesizing the evidence 

  • This stage includes:
    • appraising the evidence
    • interpreting results
    • performing a qualitative (narrative analysis) and/or a quantitative/meta-analysis
  • A meta-analysis is optional and is only done if it is feasible
    • A bio-statistician or advanced training in statistics is recommended if doing a meta-analysis
       
  • Evidence from studies are assessed using critical appraisal or Risk of Bias tools and/or checklists by study design. 
  • Data from all studies must also be extracted and put into tables/charts such as the Summary of Findings (SOF) table and is reported as a narrative synthesis.
  • Data is collected from all studies if conducting a meta-analysis and its numerical findings are reported

 

Here are some more detailed elaborations and examples:

Synthesis: Provide a narrative synthesis of the included studies individually and when combined (What are the differences and the commonality between studies?) or what can be demonstrated from the research when combining the studies together?

A meta-analysis is optional. Create a data abstraction/extraction form for the purposes of collecting data that is similar across all included studies, include a ‘Characteristics of Studies’ table to show this data (see table example).

Summary of Findings tables are provided starting on page 8 of the same document.

Data extraction must be done using data extraction forms and independently/blinded by two reviewers, with a 3rd reviewer serving as a tiebreaker. 

 

7.  Drawing Conclusions, Writing & Publishing

  • After completing these steps, the results of the review must be shared.
    • What is the level of evidence?
    • Is there evidence in support of the question or are more studies needed to draw conclusions?
    • What are your recommendations for future studies?
    • What are the limitations to your systematic review?
    • How do these findings from your review change what is known on the topic or question?

Conclusions/Recommendations:

  • Discuss what contribution this review makes and how your review answers or addresses the original question
  • Discuss any gaps found in the research
  • Make recommendations for needed research to address these gaps and the importance of addressing them
  • Discuss the overall strength of evidence in support of your original question (strong, moderate or weak)

Limitations

  • Systematic reviews with narrowly defined review questions provide specific answers to specific questions  
  • Alternative questions that have not been answered usually need to be reconstructed by the reader  

Source: Cochrane. Background to Systematic Reviews

This link will open a PDF document. 

https://ph.cochrane.org/sites/ph.cochrane.org/

files/public/uploads/Unit_One.pdf

 

Credit:
Adapted from Temple University Health Science Libraries

Systematic Review Methods Resources