systematic review research paper

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved August 26, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

• A-Z Publications

Annual Review of Psychology

Volume 70, 2019, review article, how to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.

• Andy P. Siddaway 1 , Alex M. Wood 2 , and Larry V. Hedges 3
• View Affiliations Hide Affiliations Affiliations: 1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected] 2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom 3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected]
• Vol. 70:747-770 (Volume publication date January 2019) https://doi.org/10.1146/annurev-psych-010418-102803
• First published as a Review in Advance on August 08, 2018
• Copyright © 2019 by Annual Reviews. All rights reserved

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Article metrics loading...

Full text loading...

Literature Cited

• APA Publ. Commun. Board Work. Group J. Artic. Rep. Stand. 2008 . Reporting standards for research in psychology: Why do we need them? What might they be?. Am. Psychol . 63 : 848– 49 [Google Scholar]
• Baumeister RF 2013 . Writing a literature review. The Portable Mentor: Expert Guide to a Successful Career in Psychology MJ Prinstein, MD Patterson 119– 32 New York: Springer, 2nd ed.. [Google Scholar]
• Baumeister RF , Leary MR 1995 . The need to belong: desire for interpersonal attachments as a fundamental human motivation. Psychol. Bull. 117 : 497– 529 [Google Scholar]
• Baumeister RF , Leary MR 1997 . Writing narrative literature reviews. Rev. Gen. Psychol. 3 : 311– 20 Presents a thorough and thoughtful guide to conducting narrative reviews. [Google Scholar]
• Bem DJ 1995 . Writing a review article for Psychological Bulletin. Psychol . Bull 118 : 172– 77 [Google Scholar]
• Borenstein M , Hedges LV , Higgins JPT , Rothstein HR 2009 . Introduction to Meta-Analysis New York: Wiley Presents a comprehensive introduction to meta-analysis. [Google Scholar]
• Borenstein M , Higgins JPT , Hedges LV , Rothstein HR 2017 . Basics of meta-analysis: I 2 is not an absolute measure of heterogeneity. Res. Synth. Methods 8 : 5– 18 [Google Scholar]
• Braver SL , Thoemmes FJ , Rosenthal R 2014 . Continuously cumulating meta-analysis and replicability. Perspect. Psychol. Sci. 9 : 333– 42 [Google Scholar]
• Bushman BJ 1994 . Vote-counting procedures. The Handbook of Research Synthesis H Cooper, LV Hedges 193– 214 New York: Russell Sage Found. [Google Scholar]
• Cesario J 2014 . Priming, replication, and the hardest science. Perspect. Psychol. Sci. 9 : 40– 48 [Google Scholar]
• Chalmers I 2007 . The lethal consequences of failing to make use of all relevant evidence about the effects of medical treatments: the importance of systematic reviews. Treating Individuals: From Randomised Trials to Personalised Medicine PM Rothwell 37– 58 London: Lancet [Google Scholar]
• Cochrane Collab. 2003 . Glossary Rep., Cochrane Collab. London: http://community.cochrane.org/glossary Presents a comprehensive glossary of terms relevant to systematic reviews. [Google Scholar]
• Cohn LD , Becker BJ 2003 . How meta-analysis increases statistical power. Psychol. Methods 8 : 243– 53 [Google Scholar]
• Cooper HM 2003 . Editorial. Psychol. Bull. 129 : 3– 9 [Google Scholar]
• Cooper HM 2016 . Research Synthesis and Meta-Analysis: A Step-by-Step Approach Thousand Oaks, CA: Sage, 5th ed.. Presents a comprehensive introduction to research synthesis and meta-analysis. [Google Scholar]
• Cooper HM , Hedges LV , Valentine JC 2009 . The Handbook of Research Synthesis and Meta-Analysis New York: Russell Sage Found, 2nd ed.. [Google Scholar]
• Cumming G 2014 . The new statistics: why and how. Psychol. Sci. 25 : 7– 29 Discusses the limitations of null hypothesis significance testing and viable alternative approaches. [Google Scholar]
• Earp BD , Trafimow D 2015 . Replication, falsification, and the crisis of confidence in social psychology. Front. Psychol. 6 : 621 [Google Scholar]
• Etz A , Vandekerckhove J 2016 . A Bayesian perspective on the reproducibility project: psychology. PLOS ONE 11 : e0149794 [Google Scholar]
• Ferguson CJ , Brannick MT 2012 . Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychol. Methods 17 : 120– 28 [Google Scholar]
• Fleiss JL , Berlin JA 2009 . Effect sizes for dichotomous data. The Handbook of Research Synthesis and Meta-Analysis H Cooper, LV Hedges, JC Valentine 237– 53 New York: Russell Sage Found, 2nd ed.. [Google Scholar]
• Garside R 2014 . Should we appraise the quality of qualitative research reports for systematic reviews, and if so, how. Innovation 27 : 67– 79 [Google Scholar]
• Hedges LV , Olkin I 1980 . Vote count methods in research synthesis. Psychol. Bull. 88 : 359– 69 [Google Scholar]
• Hedges LV , Pigott TD 2001 . The power of statistical tests in meta-analysis. Psychol. Methods 6 : 203– 17 [Google Scholar]
• Higgins JPT , Green S 2011 . Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0 London: Cochrane Collab. Presents comprehensive and regularly updated guidelines on systematic reviews. [Google Scholar]
• John LK , Loewenstein G , Prelec D 2012 . Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23 : 524– 32 [Google Scholar]
• Juni P , Witschi A , Bloch R , Egger M 1999 . The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 282 : 1054– 60 [Google Scholar]
• Klein O , Doyen S , Leys C , Magalhães de Saldanha da Gama PA , Miller S et al. 2012 . Low hopes, high expectations: expectancy effects and the replicability of behavioral experiments. Perspect. Psychol. Sci. 7 : 6 572– 84 [Google Scholar]
• Lau J , Antman EM , Jimenez-Silva J , Kupelnick B , Mosteller F , Chalmers TC 1992 . Cumulative meta-analysis of therapeutic trials for myocardial infarction. N. Engl. J. Med. 327 : 248– 54 [Google Scholar]
• Light RJ , Smith PV 1971 . Accumulating evidence: procedures for resolving contradictions among different research studies. Harvard Educ. Rev. 41 : 429– 71 [Google Scholar]
• Lipsey MW , Wilson D 2001 . Practical Meta-Analysis London: Sage Comprehensive and clear explanation of meta-analysis. [Google Scholar]
• Matt GE , Cook TD 1994 . Threats to the validity of research synthesis. The Handbook of Research Synthesis H Cooper, LV Hedges 503– 20 New York: Russell Sage Found. [Google Scholar]
• Maxwell SE , Lau MY , Howard GS 2015 . Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?. Am. Psychol. 70 : 487– 98 [Google Scholar]
• Moher D , Hopewell S , Schulz KF , Montori V , Gøtzsche PC et al. 2010 . CONSORT explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 340 : c869 [Google Scholar]
• Moher D , Liberati A , Tetzlaff J , Altman DG PRISMA Group. 2009 . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339 : 332– 36 Comprehensive reporting guidelines for systematic reviews. [Google Scholar]
• Morrison A , Polisena J , Husereau D , Moulton K , Clark M et al. 2012 . The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. Int. J. Technol. Assess. Health Care 28 : 138– 44 [Google Scholar]
• Nelson LD , Simmons J , Simonsohn U 2018 . Psychology's renaissance. Annu. Rev. Psychol. 69 : 511– 34 [Google Scholar]
• Noblit GW , Hare RD 1988 . Meta-Ethnography: Synthesizing Qualitative Studies Newbury Park, CA: Sage [Google Scholar]
• Olivo SA , Macedo LG , Gadotti IC , Fuentes J , Stanton T , Magee DJ 2008 . Scales to assess the quality of randomized controlled trials: a systematic review. Phys. Ther. 88 : 156– 75 [Google Scholar]
• Open Sci. Collab. 2015 . Estimating the reproducibility of psychological science. Science 349 : 943 [Google Scholar]
• Paterson BL , Thorne SE , Canam C , Jillings C 2001 . Meta-Study of Qualitative Health Research: A Practical Guide to Meta-Analysis and Meta-Synthesis Thousand Oaks, CA: Sage [Google Scholar]
• Patil P , Peng RD , Leek JT 2016 . What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspect. Psychol. Sci. 11 : 539– 44 [Google Scholar]
• Rosenthal R 1979 . The “file drawer problem” and tolerance for null results. Psychol. Bull. 86 : 638– 41 [Google Scholar]
• Rosnow RL , Rosenthal R 1989 . Statistical procedures and the justification of knowledge in psychological science. Am. Psychol. 44 : 1276– 84 [Google Scholar]
• Sanderson S , Tatt ID , Higgins JP 2007 . Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int. J. Epidemiol. 36 : 666– 76 [Google Scholar]
• Schreiber R , Crooks D , Stern PN 1997 . Qualitative meta-analysis. Completing a Qualitative Project: Details and Dialogue JM Morse 311– 26 Thousand Oaks, CA: Sage [Google Scholar]
• Shrout PE , Rodgers JL 2018 . Psychology, science, and knowledge construction: broadening perspectives from the replication crisis. Annu. Rev. Psychol. 69 : 487– 510 [Google Scholar]
• Stroebe W , Strack F 2014 . The alleged crisis and the illusion of exact replication. Perspect. Psychol. Sci. 9 : 59– 71 [Google Scholar]
• Stroup DF , Berlin JA , Morton SC , Olkin I , Williamson GD et al. 2000 . Meta-analysis of observational studies in epidemiology (MOOSE): a proposal for reporting. JAMA 283 : 2008– 12 [Google Scholar]
• Thorne S , Jensen L , Kearney MH , Noblit G , Sandelowski M 2004 . Qualitative meta-synthesis: reflections on methodological orientation and ideological agenda. Qual. Health Res. 14 : 1342– 65 [Google Scholar]
• Tong A , Flemming K , McInnes E , Oliver S , Craig J 2012 . Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med. Res. Methodol. 12 : 181– 88 [Google Scholar]
• Trickey D , Siddaway AP , Meiser-Stedman R , Serpell L , Field AP 2012 . A meta-analysis of risk factors for post-traumatic stress disorder in children and adolescents. Clin. Psychol. Rev. 32 : 122– 38 [Google Scholar]
• Valentine JC , Biglan A , Boruch RF , Castro FG , Collins LM et al. 2011 . Replication in prevention science. Prev. Sci. 12 : 103– 17 [Google Scholar]
• Article Type: Review Article

Most Read This Month

Most cited most cited rss feed, job burnout, executive functions, social cognitive theory: an agentic perspective, on happiness and human potentials: a review of research on hedonic and eudaimonic well-being, sources of method bias in social science research and recommendations on how to control it, mediation analysis, missing data analysis: making it work in the real world, grounded cognition, personality structure: emergence of the five-factor model, motivational beliefs, values, and goals.

Systematic Review

• Library Help
• What is a Systematic Review (SR)?

Steps of a Systematic Review

• Framing a Research Question
• Developing a Search Strategy
• Searching the Literature
• Managing the Process
• Meta-analysis
• Publishing your Systematic Review

Forms and templates

Image: David Parmenter's Shop

• PICO Template
• Inclusion/Exclusion Criteria
• Database Search Log
• Review Matrix
• Cochrane Tool for Assessing Risk of Bias in Included Studies

   • PRISMA Flow Diagram  - Record the numbers of retrieved references and included/excluded studies. You can use the Create Flow Diagram tool to automate the process.

   •  PRISMA Checklist - Checklist of items to include when reporting a systematic review or meta-analysis

PRISMA 2020 and PRISMA-S: Common Questions on Tracking Records and the Flow Diagram

• PROSPERO Template
• Manuscript Template
• Steps of SR (text)
• Steps of SR (visual)
• Steps of SR (PIECES)

Adapted from  A Guide to Conducting Systematic Reviews: Steps in a Systematic Review by Cornell University Library

Source: Cochrane Consumers and Communications  (infographics are free to use and licensed under Creative Commons )

Check the following visual resources titled " What Are Systematic Reviews?"

• Video  with closed captions available
• Animated Storyboard

• << Previous: What is a Systematic Review (SR)?
• Next: Framing a Research Question >>
• Last Updated: Aug 26, 2024 12:37 PM
• URL: https://lib.guides.umd.edu/SR

Systematic Reviews

Aims and scope.

The journal publishes high quality systematic review products including systematic review protocols, systematic reviews related to a very broad definition of human health, (animal studies relevant for human health), rapid reviews, updates of already completed systematic reviews, and methods research related to the science of systematic reviews, such as decision modelling. At this time Systematic Reviews does not accept reviews of in vitro studies. The journal also aims to ensure that the results of all well-conducted systematic reviews are published, regardless of their outcome. (You can now also prospectively register these reviews on PROSPERO.)

Trending articles

Click here to view which articles have been shared the most in the last month!

• Most accessed

Examining the effectiveness of food literacy interventions in improving food literacy behavior and healthy eating among adults belonging to different socioeconomic groups- a systematic scoping review

Authors: Arijita Manna, Helen Vidgen and Danielle Gallegos

Knowledge and practices of youth awareness on death and dying in school settings: a systematic scoping review protocol

Authors: Emilie Allard, Clémence Coupat, Sabrina Lessard, Noémie Therrien, Claire Godard-Sebillotte, Dimitri Létourneau, Olivia Nguyen, Andréanne Côté, Gabrielle Fortin, Serge Daneault, Maryse Soulières, Josiane Le Gall and Sylvie Fortin

Evaluating the effectiveness of large language models in abstract screening: a comparative analysis

Authors: Michael Li, Jianping Sun and Xianming Tan

Prevalence of human filovirus infections in sub-Saharan Africa: A systematic review and meta-analysis protocol

Authors: Christopher S. Semancik, Christopher L. Cooper, Thomas S. Postler, Matt Price, Heejin Yun, Marija Zaric, Monica Kuteesa, Nina Malkevich, Andrew Kilianski, Swati B. Gupta and Suzanna C. Francis

Patients’ experiences of mechanical ventilation in intensive care units in low- and lower-middle-income countries: protocol of a systematic review

Authors: Mayank Gupta, Priyanka Gupta, Preeti Devi, Utkarsh, Damini Butola and Savita Butola

Most recent articles RSS

View all articles

Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement

Authors: David Moher, Larissa Shamseer, Mike Clarke, Davina Ghersi, Alessandro Liberati, Mark Petticrew, Paul Shekelle and Lesley A Stewart

The PRISMA 2020 statement: an updated guideline for reporting systematic reviews

Authors: Matthew J. Page, Joanne E. McKenzie, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer, Jennifer M. Tetzlaff, Elie A. Akl, Sue E. Brennan, Roger Chou, Julie Glanville, Jeremy M. Grimshaw, Asbjørn Hróbjartsson, Manoj M. Lalu, Tianjing Li…

The Editorial to this article has been published in Systematic Reviews 2021 10 :117

Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study

Authors: Wichor M. Bramer, Melissa L. Rethlefsen, Jos Kleijnen and Oscar H. Franco

PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews

Authors: Melissa L. Rethlefsen, Shona Kirtley, Siw Waffenschmidt, Ana Patricia Ayala, David Moher, Matthew J. Page and Jonathan B. Koffel

Drug discontinuation before contrast procedures and the effect on acute kidney injury and other clinical outcomes: a systematic review protocol

Authors: Swapnil Hiremath, Jeanne Françoise Kayibanda, Benjamin J. W. Chow, Dean Fergusson, Greg A. Knoll, Wael Shabana, Brianna Lahey, Olivia McBride, Alexandra Davis and Ayub Akbari

Most accessed articles RSS

Sign up to receive article alerts

Systematic Reviews is published continuously online-only. We encourage you to sign up to receive free email alerts to keep up to date with all of the latest articles by registering here .

Peer review mentoring

The Editors endorse peer review mentoring of early career researchers. Find out more here

Article collections

Thematic series The role of systematic reviews in evidence-based research Edited by Professor Dawid Pieper and Professor Hans Lund 

Thematic series Canadian Task Force on Preventive Health Care Evidence Reviews Edited by Assoc Prof Craig Lockwood

Thematic series Automation in the systematic review process Edited by Prof Joseph Lau

Thematic series Five years of Systematic Reviews

View all article collections

Latest Tweets

Your browser needs to have JavaScript enabled to view this timeline

• Editorial Board
• Manuscript editing services
• Instructions for Editors
• Sign up for article alerts and news from this journal
• Follow us on Twitter

Annual Journal Metrics

Citation Impact 2023 Journal Impact Factor: 6.3 5-year Journal Impact Factor: 4.5 Source Normalized Impact per Paper (SNIP): 1.919 SCImago Journal Rank (SJR): 1.620

Speed 2023 Submission to first editorial decision (median days): 92 Submission to acceptance (median days): 296

Usage 2023 Downloads: 3,531,065 Altmetric mentions: 3,533

• More about our metrics

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Systematic Reviews

• Review Types
• Scoping Review Steps
• Before You Begin

P = Plan: decide on your search methods

I = identify: search for studies that match your criteria, e = evaluate: exclude or include studies, c = collect: extract and synthesize key data, e = explain: give context and rate the strength of the studies, s = summarize: write and publish your final report.

• Biomedical & Public Health Reviews

Congratulations!

You've decided to conduct a Systematic Review! Please see the associated steps below. You can follow the  P-I-E-C-E-S = Plan, Identify, Evaluate, Collect, Explain, Summarize  system or any number of systematic review processes available  (Foster & Jewell, 2017) .

P =   Plan: decide on your search methods

Determine your Research Question 

By now you should have identified gaps in the field and have a specific question you are seeking to answer. This will likely have taken several iterations and is the most important part of the Systematic Review process. 

Identify Relevant Systematic Reviews 

Once you've finalized a research question, you should be able to locate existing systematic reviews on or similar to your topic. existing systematic reviews will be your clues to mine for keywords, sample searches in various databases, and will help your team finalize your review question and develop your  inclusion and exclusion criteria. , decide on a protocol and reporting standard, your  protocol  is essentially a project plan and data management strategy for an objective, reproducible, sound methodology for peer review. the  reporting standard or guidelines  are not a protocols, but rather a set of standards to guide the development of your systematic review. often they include checklists. it is not required, but highly recommended to follow a reporting standard. .

Protocol registry:   Reviewing existing systematic reviews and registering your protocol will increase transparency, minimize bias, and reduce the redundancy of groups working on the same topics ( PLoS Medicine Editors, 2011 ). Protocols can serve as internal or external documents. Protocols can be made public prior to the review. Some registries allow for keeping a protocol private for a set period of time.

Cochrane Database of Systematic Reviews (UGA Login)  (Health Sciences)

A collection of regularly updated, systematic reviews of the effects of health care. New reviews are added with each issue of The Cochrane Library Reviews mainly of randomized controlled trials. All reviews have protocols.

PROSPERO  (General)

This is an international register of systematic reviews and is public. 

Campbell Corporation  (Education & Social Sciences)

Topics covered include Ageing; Business and Management; Climate Solutions; Crime and Justice; Disability;  Education; International Development; Knowledge Translation and Implementation; Methods; Nutrition and Food Security; Sexual Orientation and Gender Identity; Social Welfare; and Training.

Systematic Review for Animals and Food  (Vet Med & Animal Science)

Reporting Standards:  

Campbell MECCIR Standards  (Education & Social Sciences)

Cochrane Guides & Handbooks  (Health & Medical Sciences)

Institute of Medicine of the National Academies: Finding What Works in Healthcare: Standards for Systematic Reviews  (healthcare)

• PRISMA for Systematic Review Protocols  (General)
• PRISMA Checklist  (General)
• PRISMA for Scoping Reviews  (General)

Decide on Databases and Grey Literature for Systematic Review Research

Because the purpose of a SR is to find all studies related to your research question, you will need to search multiple databases. You should be able to name the databases you are already familiar with using. Your librarian will be able to recommend additional databases, including some of the following: 

• PubMed  (Health Sciences)
• Web of Science  
• Cochrane Database  (Biomedical)
• National and Regional Databases (i.e. WHO LILACS scientific health information from Latin America and the Caribbean countries)
• CINAHL  (Health Sciences)
• PsycINFO  (Psychology)

Depending on your topic, you may want to search clinical trials and grey literature. See this  guide  for more on grey literature.

Develop Keywords and Write a Search Strategy

Go   here for help with writing your search strategy  

Translate Search Strategies

Each database you use will have different methods of searching and resulting search strings, including syntax. ideally you will create one master keyword list and translate it for each database. below are tools to assist with translating search strings. .

Includes syntax for Cochrane Library, EBSCO, ProQuest, Ovid, and POPLINE.

The IEBH SR-Accelerator is a suite of tools to assist in speeding up portions of the Systematic Review process, including the Polyglot tool which translates searches across databases. 

University of Michigan Search 101 - SR Database Cheat Sheet

Storing, Screening and Full-Text Screening of Your Citations

Because systematic review literature searches may produce thousands of citations and abstracts, the research team will be screening and systematically reviewing large amounts of results. During  screening , you will remove duplicates and remove studies that are not relevant to your topic based on a review of titles and abstracts. Of what remains, the  full-text screening  of the studies will then need to be conducted to confirm that they fit within your  inclusion/exclusion criteria.   

The results of the literature review and screening processes are best managed by various tools and software. You can also use a simple form or table to log the relevant information from each study. Consider whether you will be coding your data during the extraction process in your decision on which tool or software to use. Your librarian can consult on which of these is best suited to your research needs.

• EndNote  Guide (UGA supported citation tracking software) - great for storing, organizing, and de-duplication
• RefWorks  Guide (UGA supported citation tracking software) - great for storing, organizing, and de-duplication
• Rayyan  (free service) - great for initial title/abstract screening OR full-text screening as cannot differentiate; not ideal for de-duplication
• Covidence  (requires a subscription) - full suite of systematic review tools including meta-analysis
• Combining Software  (EndNote, Google Forms, Excel) 
• Forms such as  Qualtrics  (UGA EITS software) can note who the coder is, creates charts and tables, good when have a review of multiple types of studies

Data Extraction

Data extraction processes differ between qualitative and quantitative evidence syntheses. In both cases, you must provide the reader with a clear overview of the studies you have included, their similarities and differences, and the findings. Extraction should be done in accordance to pre-established guidelines, such as PRISMA. 

Some systematic reviews contain meta-analysis of the quantitative findings of the results. Consider including a statistician on your team to complete the analysis of all individual study results. Meta-analysis will tell you how much or what the actual results is across the studies and explains results in a measure of variance, typically called a forest plot.

Systematic review price models have changed over the years. Previously, you had to depend on departmental access to software that would cost several hundred dollars. Now that the software is cloud-based, tiered payment systems are now available. Sometimes there is a free tier level, but costs go up for functionality, number or users, or both. Depending on the organization's model, payments may be monthly, annual or per project/review.

• Always check your departmental resources before making a purchase.
• View all training videos and other resources before starting your project.
• If your access is limited to a specific amount of time, wait to purchase until the appropriate work stage

Software list

Tool created by Brown University to assist with screening for systematic reviews.

• Cochrane's RevMan

Review Manager (RevMan) is the software used for preparing and maintaining Cochrane Reviews.

Systematic review tool intended to assist with the screening and extraction process. (Requires subscription)

• Distiller SR

DistillerSR is an online application designed specifically for the screening and data extraction phases of a systematic review (Requires subscription) Student and Faculty tiers have monthly pricing with a three month minimum. Number of projects is limited by pricing.

• EPPI Reviewer  (requires subscription, free trial)

It includes features such as text mining, data clustering, classification and term extraction

Rayyan is a free web-based application that can be used to screen titles, abstracts, and full text. Allows for multiple simultaneous users.

• AHRQ's  SRDR tool  (free) which is web-based and has a training environment, tutorials, and example templates of systematic review data extraction forms

"System for the Unified Management, Assessment and Review of Information, the Joanna Briggs Institutes premier software for the systematic review of literature."

• Syras Pricing is based on both number of abstracts and number of collaborators. The free tier is limited to 300 abstracts and two collaborators. Rather than monthly pricing, the payment is one-time per project.

Evidence Synthesis or Critical Appraisal

PRISMA guidelines suggest including critical appraisal of the included studies to assess the risk of bias and to include the assessment in your final manuscripts. There are several appraisal tools available depending on your discipline and area of research.

Simple overview of risk of bias assessment, including examples of how to assess and present your conclusions.

CASP is an organization that provides resources for healthcare professionals, but their appraisal tools can be used for varying study types across disciplines.

From the Joanna Briggs Institute: "JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers."

Johns Hopkins Evidence-Based Practice Model  (health sciences)

National Academies of Sciences, Engineering, and Medicine

Document the search; 5.1.6. Include a methods section

List  of additional critical appraisal tools from Cardiff University. 

Synthesize, Map, or Describe the Results

Prepare your process and findings in a final manuscript. Be sure to check your PRISMA checklist or other reporting standard. You will want to include the full formatted search strategy for the appendix, as well as include documentation of your search methodology. A convenient way to illustrate this process is through a  PRISMA  Flow Diagram. 

Attribution: Unless noted otherwise, this section of the guide was adapted from Texas A&M's "Systematic Reviews and Related Evidence Syntheses" 

• << Previous: Before You Begin
• Next: Biomedical & Public Health Reviews >>
• Last Updated: Aug 23, 2024 1:53 PM
• URL: https://guides.libs.uga.edu/SystematicReview

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

How to write a systematic review

Affiliations.

• 1 The Methodist Orthopedics and Sports Medicine Center, Houston, Texas [email protected].
• 2 Sports Medicine Center, The Ohio State University, Columbus, Ohio Department of Orthopaedic Surgery, The Ohio State University, Columbus, Ohio.
• 3 Department of Orthopaedic Surgery, The Ohio State University, Columbus, Ohio.
• 4 Sports Medicine Center, The Ohio State University, Columbus, Ohio Department of Mechanical and Aerospace Engineering, The Ohio State University, Columbus, Ohio.
• PMID: 23925575
• DOI: 10.1177/0363546513497567

Background: The role of evidence-based medicine in sports medicine and orthopaedic surgery is rapidly growing. Systematic reviews and meta-analyses are also proliferating in the medical literature.

Purpose: To provide the outline necessary for a practitioner to properly understand and/or conduct a systematic review for publication in a sports medicine journal.

Study design: Review.

Methods: The steps of a successful systematic review include the following: identification of an unanswered answerable question; explicit definitions of the investigation's participant(s), intervention(s), comparison(s), and outcome(s); utilization of PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines and PROSPERO registration; thorough systematic data extraction; and appropriate grading of the evidence and strength of the recommendations.

Results: An outline to understand and conduct a systematic review is provided, and the difference between meta-analyses and systematic reviews is described. The steps necessary to perform a systematic review are fully explained, including the study purpose, search methodology, data extraction, reporting of results, identification of bias, and reporting of the study's main findings.

Conclusion: Systematic reviews or meta-analyses critically appraise and formally synthesize the best existing evidence to provide a statement of conclusion that answers specific clinical questions. Readers and reviewers, however, must recognize that the quality and strength of recommendations in a review are only as strong as the quality of studies that it analyzes. Thus, great care must be used in the interpretation of bias and extrapolation of the review's findings to translation to clinical practice. Without advanced education on the topic, the reader may follow the steps discussed herein to perform a systematic review.

Keywords: PRISMA; PROSPERO; evidence-based medicine; meta-analysis; systematic review.

© 2013 The Author(s).

PubMed Disclaimer

Similar articles

• Systematic Reviews in Sports Medicine. DiSilvestro KJ, Tjoumakaris FP, Maltenfort MG, Spindler KP, Freedman KB. DiSilvestro KJ, et al. Am J Sports Med. 2016 Feb;44(2):533-8. doi: 10.1177/0363546515580290. Epub 2015 Apr 21. Am J Sports Med. 2016. PMID: 25899433 Review.
• Reporting and methodological quality of systematic reviews in the orthopaedic literature. Gagnier JJ, Kellam PJ. Gagnier JJ, et al. J Bone Joint Surg Am. 2013 Jun 5;95(11):e771-7. doi: 10.2106/JBJS.L.00597. J Bone Joint Surg Am. 2013. PMID: 23780547
• Evaluating characteristics of PROSPERO records as predictors of eventual publication of non-Cochrane systematic reviews: a meta-epidemiological study protocol. Ruano J, Gómez-García F, Gay-Mimbrera J, Aguilar-Luque M, Fernández-Rueda JL, Fernández-Chaichio J, Alcalde-Mellado P, Carmona-Fernandez PJ, Sanz-Cabanillas JL, Viguera-Guerra I, Franco-García F, Cárdenas-Aranzana M, Romero JLH, Gonzalez-Padilla M, Isla-Tejera B, Garcia-Nieto AV. Ruano J, et al. Syst Rev. 2018 Mar 9;7(1):43. doi: 10.1186/s13643-018-0709-6. Syst Rev. 2018. PMID: 29523200 Free PMC article.
• Evidence-based medicine, systematic reviews, and guidelines in interventional pain management: part 6. Systematic reviews and meta-analyses of observational studies. Manchikanti L, Datta S, Smith HS, Hirsch JA. Manchikanti L, et al. Pain Physician. 2009 Sep-Oct;12(5):819-50. Pain Physician. 2009. PMID: 19787009
• Research Pearls: The Significance of Statistics and Perils of Pooling. Part 3: Pearls and Pitfalls of Meta-analyses and Systematic Reviews. Harris JD, Brand JC, Cote MP, Dhawan A. Harris JD, et al. Arthroscopy. 2017 Aug;33(8):1594-1602. doi: 10.1016/j.arthro.2017.01.055. Epub 2017 Apr 27. Arthroscopy. 2017. PMID: 28457677 Review.
• Artificial Intelligence in Emergency Trauma Care: A Preliminary Scoping Review. Ventura CAI, Denton EE, David JA. Ventura CAI, et al. Med Devices (Auckl). 2024 May 23;17:191-211. doi: 10.2147/MDER.S467146. eCollection 2024. Med Devices (Auckl). 2024. PMID: 38803707 Free PMC article. Review.
• Patterns of sedentary behavior among older women with urinary incontinence and urinary symptoms: a scoping review. Leung WKC, Cheung J, Wong VCC, Tse KKL, Lee RWY, Lam SC, Suen LKP. Leung WKC, et al. BMC Public Health. 2024 Apr 30;24(1):1201. doi: 10.1186/s12889-024-18703-7. BMC Public Health. 2024. PMID: 38689284 Free PMC article. Review.
• The Functional Integrity of the Anterior Cruciate Ligament Can Be Objectively Assessed With the Use of Stress Radiographs: A Systematic Review. Schwartz J, Rodriguez AN, Banovetz MT, Braaten JA, Larson CM, Wulf CA, Kennedy NI, LaPrade RF. Schwartz J, et al. Orthop J Sports Med. 2024 Apr 25;12(4):23259671241246197. doi: 10.1177/23259671241246197. eCollection 2024 Apr. Orthop J Sports Med. 2024. PMID: 38680218 Free PMC article.
• Understanding Risk Factors for Suicide Among Older People in Rural China: A Systematic Review. Zhang Q, Li S, Wu Y. Zhang Q, et al. Innov Aging. 2024 Feb 17;8(3):igae015. doi: 10.1093/geroni/igae015. eCollection 2024. Innov Aging. 2024. PMID: 38618517 Free PMC article. Review.
• Writing a Scientific Review Article: Comprehensive Insights for Beginners. Amobonye A, Lalung J, Mheta G, Pillai S. Amobonye A, et al. ScientificWorldJournal. 2024 Jan 17;2024:7822269. doi: 10.1155/2024/7822269. eCollection 2024. ScientificWorldJournal. 2024. PMID: 38268745 Free PMC article. Review.

Publication types

• Search in MeSH

LinkOut - more resources

Full text sources.

• Ovid Technologies, Inc.

Other Literature Sources

• scite Smart Citations
• MedlinePlus Health Information

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Literature Reviews

• Getting started

What is a literature review?

Why conduct a literature review, stages of a literature review, lit reviews: an overview (video), check out these books.

• Types of reviews
• 1. Define your research question
• 2. Plan your search
• 3. Search the literature
• 4. Organize your results
• 5. Synthesize your findings
• 6. Write the review
• Artificial intelligence (AI) tools
• Thompson Writing Studio This link opens in a new window
• Need to write a systematic review? This link opens in a new window

Guide Owner

Contact a Librarian

Ask a Librarian

Definition: A literature review is a systematic examination and synthesis of existing scholarly research on a specific topic or subject.

Purpose: It serves to provide a comprehensive overview of the current state of knowledge within a particular field.

Analysis: Involves critically evaluating and summarizing key findings, methodologies, and debates found in academic literature.

Identifying Gaps: Aims to pinpoint areas where there is a lack of research or unresolved questions, highlighting opportunities for further investigation.

Contextualization: Enables researchers to understand how their work fits into the broader academic conversation and contributes to the existing body of knowledge.

tl;dr  A literature review critically examines and synthesizes existing scholarly research and publications on a specific topic to provide a comprehensive understanding of the current state of knowledge in the field.

What is a literature review NOT?

❌ An annotated bibliography

❌ Original research

❌ A summary

❌ Something to be conducted at the end of your research

❌ An opinion piece

❌ A chronological compilation of studies

The reason for conducting a literature review is to:

Literature Reviews: An Overview for Graduate Students

While this 9-minute video from NCSU is geared toward graduate students, it is useful for anyone conducting a literature review.

Writing the literature review: A practical guide

Available 3rd floor of Perkins

Writing literature reviews: A guide for students of the social and behavioral sciences

Available online!

So, you have to write a literature review: A guided workbook for engineers

Telling a research story: Writing a literature review

The literature review: Six steps to success

Systematic approaches to a successful literature review

Request from Duke Medical Center Library

Doing a systematic review: A student's guide

• Next: Types of reviews >>
• Last Updated: Aug 29, 2024 11:40 AM
• URL: https://guides.library.duke.edu/litreviews

Services for...

• Faculty & Instructors
• Graduate Students
• Undergraduate Students
• International Students
• Patrons with Disabilities

• Harmful Language Statement
• Re-use & Attribution / Privacy
• Support the Libraries

• Introduction
• Conclusions
• Article Information

The process to select eligible RCTs is shown.

Lines indicate point estimates; shaded areas, 95% CIs.

eFigure. Defining Eligible Randomized Clinical Trials

eTable. Factors Associated With Citation of Systematic Reviews

Data Sharing Statement

• Should a Systematic Review Be Required in a Clinical Trial Report? JAMA Network Open Invited Commentary March 23, 2023 Ivo Abraham, PhD; Matthias Calamia, PharmD; Karen MacDonald, PhD

See More About

Sign up for emails based on your interests, select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Get the latest research based on your areas of interest.

Others also liked.

• Download PDF
• X Facebook More LinkedIn

Jia Y , Li B , Yang Z, et al. Trends of Randomized Clinical Trials Citing Prior Systematic Reviews, 2007-2021. JAMA Netw Open. 2023;6(3):e234219. doi:10.1001/jamanetworkopen.2023.4219

Manage citations:

© 2024

• Permissions

Trends of Randomized Clinical Trials Citing Prior Systematic Reviews, 2007-2021

• 1 Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
• 2 Department of Public Health and Primary Care, School of Clinical Medicine, University of Cambridge, Cambridge, United Kingdom
• 3 Center for Diversity in Public Health Leadership Training, Kennedy Krieger Institute, Johns Hopkins University, Baltimore, Maryland
• 4 Department of Cardiology, Virginia Commonwealth University, Richmond, Virginia
• 5 M. Louise Fitzpatrick College of Nursing, Villanova University, Villanova, Pennsylvania
• 6 Section Evidence-Based Practice, Western Norway University of Applied Sciences, Bergen, Norway
• 7 School of Public Health and Primary Care, Chinese University of Hong Kong, Hong Kong
• 8 Department of Medicine, School of Medicine, Johns Hopkins University, Baltimore, Maryland
• Invited Commentary Should a Systematic Review Be Required in a Clinical Trial Report? Ivo Abraham, PhD; Matthias Calamia, PharmD; Karen MacDonald, PhD JAMA Network Open

Question   Has the citation of prior systematic reviews in reports of randomized clinical trials improved over time?

Findings   In this cross-sectional study of 4003 randomized clinical trials (RCTs), the percentage of RCTs citing systematic reviews increased from 35.5% in 2007 to 2008 to 71.8% since 2020, with an annual rate of increase of 3.0%. RCTs with 100 participants or more, nonindustry funders, and authors from high-income countries were more likely to cite systematic reviews than those with fewer than 100 participants, industry funders, and authors from low- and middle-income countries.

Meaning   These findings suggest that the citation of prior systematic reviews in reports of RCTs has improved over time but may need further improvement.

Importance   Systematic reviews can help to justify a new randomized clinical trial (RCT), inform its design, and interpret its results in the context of prior evidence.

Objective   To assess trends and factors associated with citing (a marker of the use of) prior systematic reviews in RCT reports.

Design, Setting, and Participants   This cross-sectional study investigated 737 Cochrane reviews assessing health interventions to identify 4003 eligible RCTs, defined as those included in an updated version but not in the first version of a Cochrane review and published 2 years after the first version of the Cochrane review was published.

Main Outcomes and Measures   The primary outcome was the citation of prior systematic reviews, Cochrane or others, as determined by screening references of eligible RCTs. Factors that may be associated with the citation of prior systematic reviews were also examined.

Results   Among 4003 eligible RCTs, 1241 studies (31.0%) cited Cochrane reviews, 1698 studies (42.4%) cited prior non-Cochrane reviews, and 2265 studies (56.6%) cited either type of systematic review or both; 1738 RCTs (43.4%) cited no systematic reviews. The percentage of RCTs citing prior Cochrane reviews, non-Cochrane reviews, and either or both types of review increased from 28 studies (15.3%), 46 studies (25.1%), and 65 studies (35.5%) of 183 RCTs before 2008 to 42 studies (40.8%), 65 studies (64.1%), and 73 studies (71.8%) of 102 RCTs since 2020, respectively; the annual increases were 1.9% (95% CI, 1.4%-2.3%), 3.3% (95% CI, 2.9%-3.7%), and 3.0% (95% CI, 2.5%-3.5%), respectively. The proportion of RCTs citating prior systematic reviews varied considerably across clinical specialties, ranging from 28 of 106 RCTs (26.4%) in ophthalmology to 386 of 553 RCTs (69.8%) in psychiatry ( P  < .001). RCTs with 100 participants or more (risk ratio [RR], 1.16; 95% CI, 1.03-1.30), nonindustry funding (RR, 1.43; 95% CI, 1.27-1.61), and authors from high-income countries (RR, 1.10; 95% CI, 1.03-1.17) were more likely to cite systematic reviews than those with fewer than 100 participants, industry funding, and authors from low- and middle-income countries, respectively. A journal requirement to cite systematic reviews was not associated with the likelihood of citing a systematic review.

Conclusions and Relevance   This study found that the citation of prior systematic reviews in RCT reports improved over time, but approximately 40% of RCTs failed to do so. These findings suggest that reference to prior evidence for initiating, designing, and reporting RCTs should be further emphasized to assure clinical relevance, improve methodological quality, and facilitate interpretation of new results.

Randomized clinical trials (RCTs) should be justified, designed, and interpreted in the context of prior evidence. 1 - 4 Failure to consider prior evidence may be associated with low clinical relevance, compromised methodological quality, and even redundant RCTs on clinical questions for which sufficient quality evidence is already available. Such studies may waste valuable resources, unnecessarily put patients at potential harm, and damage public trust in scientific research. 5 - 10

According to the Cochrane Handbook for Systematic Reviews of Interventions , a systematic review “attempts to collate all empirical evidence that fits prespecified eligibility criteria in order to answer a specific research question” and is characterized by a clear set of objectives, a reproducible methodology, a comprehensive search, an assessment of the validity of findings, and a systematic presentation. 11 A high-quality and up-to-date systematic review comprehensively synthesizes prior evidence and can help to ensure that new RCTs are worthwhile and informative. 1 - 4 For example, systematic reviews may help justify the necessity of a new trial, plan its sample size, and overcome major methodological problems of prior similar trials. When the new trial is completed, systematic reviews can help assess its outcomes among the overall evidence by comparing and synthesizing its findings with those of prior similar trials.

Making reference to or citing a relevant systematic review in an RCT report may be used as an indication of the use of prior evidence. In the past 2 decades, many stakeholders have endorsed the importance of and made great efforts to promote the citation of prior systematic reviews in RCT reports. For example, the Consolidated Standards of Reporting Trials ( CONSORT ) reporting guideline statement recommends that an RCT report include “a reference to a systematic review of previous similar trials or a note of the absence of such trials.” 12 In 2017, an international group of health research funders called for funding only research that had been set in the context of existing systematic reviews or had robustly demonstrated a research gap. 13 Further, some journals have explicitly required citation of prior relevant systematic reviews since 2010. 14 In addition, researchers have been advocating citing systematic reviews, including through international organizations, such as the Evidence-Based Research Network 15 and EVBRES (Evidence-Based Research) a Cost Action. 1 - 3 , 16

A 2022 study 17 suggested that researchers frequently failed to cite a relevant systematic review. However, it remains unclear whether the reference to prior evidence has improved in RCTs and what factors may be associated with this practice. 17 We thus conducted this study to address these research questions. Objectives were to assess the citation of systematic reviews in RCT reports, compare the citation of systematic reviews in RCT reports across clinical specialties, and assess the factors associated with citing systematic reviews in RCT reports.

In this cross-sectional study, we identified eligible RCTs from systematic reviews conducted by Cochrane (referred to as Cochrane reviews). The citation of prior systematic reviews was assessed in reports of these RCTs. All data in this study were obtained from open sources. Therefore, ethics review and informed consent exemptions were approved by the local Institutional Review Board of the Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences. We followed the Strengthening the Reporting of Observational Studies in Epidemiology ( STROBE ) reporting guideline in reporting this study.

All Cochrane reviews indexed in the Cochrane Database of Systematic Reviews up to October 2021 were screened. A Cochrane review was considered eligible if it included only RCTs or quasi-RCTs; it assessed the effectiveness, efficacy, or safety of a health intervention; it had been updated at least once (we included the first and latest versions); none of its versions had been withdrawn; and it included at least 1 meta-analysis to ensure similarity of RCTs. We used the Web of Science to assess the citation of systematic reviews by eligible RCTs. Therefore, we further excluded Cochrane reviews that were not indexed in the Web of Science.

An RCT cannot cite a prior systematic review if no prior systematic review is available. Therefore, we used a special design to include RCTs with at least 1 Cochrane review available to cite to reduce selection bias. Eligible RCTs were selected by comparing the first and latest version of an eligible Cochrane review. Specifically, an eligible RCT was included in the latest version of an eligible Cochrane review, not included in the first version of this Cochrane review, and published at least 2 years later than the first version of this Cochrane review was published. This 2-year grace period accounted for publication delays and ensured that the first version of the Cochrane review was available to cite when RCT reports were under development (eFigure in Supplement 1 ). Therefore, eligible RCTs could cite the first version of the Cochrane review.

In addition, we considered only RCT reports published in English journals as full articles. When multiple reports of an RCT were available, we considered only the primary report as decided by the authors of the Cochrane review. Eligible RCTs were identified by comparing references under the “References to Included Studies” section between the first and the latest version of an eligible Cochrane review. An RCT was counted only once if included in multiple Cochrane reviews.

Trialists may cite systematic reviews for various purposes, such as to justify a new RCT, inform its design, or put its findings in the context of prior similar RCTs. In this study, we considered only whether a prior systematic review was cited, regardless of the purposes.

A Cochrane review may be updated multiple times. Citing any prior version of the Cochrane review by the eligible RCT was considered a citation.

Eligible RCTs may cite prior, relevant non-Cochrane reviews, that is, systematic reviews conducted independently of Cochrane. Operationally, a non-Cochrane review was screened to confirm that it met the following methodological prerequisites: at least 2 bibliographic databases were searched; any form of quality or risk of bias assessment using validated tools or self-developed criteria was reported in the results section; and the review intended to include RCTs. We included non-Cochrane reviews that searched for RCTs but identified none and excluded those that considered only observational studies. We also included non-Cochrane reviews if the result of quality or risk of bias assessment was reported as a supplement with an explicit link in the results section. To be clinically relevant, a non-Cochrane review was expected to assess the efficacy, effectiveness, or safety of health interventions similar to eligible RCTs and assess health conditions similar to RCTs.

The title and abstract of references cited by each eligible RCT were obtained from the Web of Science. Then, titles of these references were screened for review or meta-analysis and abstracts were screened for PubMed, Medline, Embase, Cochrane, or Web of Science. Full texts of references with the keywords mentioned previously were obtained and manually screened by 2 epidemiologists (Y.J., Z.Y., F.L., and D.L.) for methodological prerequisites and 2 clinicians (including X.W.) for clinical relevancy. Systematic reviews published in journals other than the Cochrane Database of Systematic Reviews were considered Cochrane reviews if they were reprints of or directly derived from a Cochrane review.

The following data items were abstracted from eligible RCTs: year published, sample size, number of recruiting centers, funders, country of first and senior author affiliations, and journals where published. When co–first authors were reported, we considered the first 1. When co–senior authors were reported, we considered the last 1. When multiple affiliations were reported for the same author, the country of the first affiliation was considered.

Author guidelines of journals where eligible RCTs were published were reviewed for requirements about citation of systematic reviews in RCTs. The requirement was classified into 1 of 3 categories: explicit requirement on citing a systematic review, referring to the CONSORT statement, and no requirement. The data abstraction was conducted by 2 authors independently to reduce information bias, and a third researcher resolved any disagreements (Y.J., Z.Z., C.W., Q.J., and X.Y.).

We calculated the percentage of eligible RCTs citing systematic reviews by type (Cochrane review, non-Cochrane review, or either), year of publication, and clinical specialty. Clinical specialties included obstetrics and gynecology, psychiatry, pediatrics, neurology, urology, oncology, respiratory disorders, cardiovascular disorders, orthopedics, anesthesiology, dentistry, dermatology, ophthalmology, gastroenterology and hepatology, endocrinology, and others. In the analysis, specialties with fewer than 100 eligible RCTs were classified into the category of others. A Cochran-Mantel-Haenszel test was used to compare the citation of systematic reviews across clinical specialties.

The trend of eligible RCTs citing Cochrane reviews, non-Cochrane reviews, or either type of systematic review was assessed by linear regression models. A log-binomial model was used to control for confounding to assess the association between the citation of systematic reviews and characteristics of eligible RCTs. Factors included sample size (<100 vs ≥100 individuals), 18 - 22 number of recruiting centers (single center vs multicenter), funding (industry funded, not industry funded, no funding, or not reported), country of authors (high-income countries vs low- and middle-income countries), the requirement of journals (explicit requirement, referring to the CONSORT statement, or no requirement), year of publication, and clinical specialty. We considered an RCT to be conducted by researchers from high-income countries if its first author or senior author was affiliated with an organization in a high-income country. 23 No missing values existed for any variables extracted. Adjusted relative differences with 95% CIs were estimated. SAS statistical software version 9.4 (SAS Institute) was used for data cleaning and analysis. All statistical analyses were 2-sided based on P values, and the level of statistical significance was set at .05. Data analysis was conducted between February and May 2022.

A total of 8707 Cochrane reviews were identified and screened, of which 1642 reviews were considered eligible and searched for eligible RCTs. Eventually, 4003 eligible RCTs published in 1205 journals were identified from 737 Cochrane reviews ( Figure 1 ). No eligible RCTs were excluded from analyses due to missing values.

Characteristics of eligible RCTs are shown in Table 1 . Most eligible RCTs recruited at least 100 participants (2298 RCTs [57.4%]), recruited participants from a single center (2320 RCTs [58.0%]), had first or senior authors from high-income countries (2758 RCTs [68.9%]), were published in journals referring to the CONSORT statement (2515 RCTs [62.8%]), and were supported by nonindustry funders (1998 RCTs [49.9%]). Because too few eligible RCTs were published in 2007 and 2021, they were combined with those published in 2008 and 2020, respectively, in analyses.

We extracted 112 801 references from 3882 eligible RCTs using the Web of Science; of these, 10 457 references were identified as possible systematic reviews using keywords. From this methodological screening, we obtained 3074 systematic reviews, of which 209 reviews were clinically irrelevant and subsequently excluded. Eventually, 2865 references were considered non-Cochrane reviews.

We could not extract references from 121 eligible RCTs using the Web of Science. References of those RCTs were manually searched for non-Cochrane reviews. A total of 71 references were considered non-Cochrane reviews.

Overall, 1241 eligible RCTs (31.0%) cited Cochrane reviews, 1698 RCTs (42.4%) cited non-Cochrane reviews, and 2265 RCTs (56.6%) cited either type of systematic review. Thus, 1738 RCTs (43.4%) failed to cite systematic reviews. Table 1 and Figure 2 show the trends of citing systematic reviews in eligible RCTs.

The percentage of eligible RCTs citing Cochrane reviews increased from 28 of 183 RCTs (15.3%) in 2007 to 2008 to 42 of 102 RCTs (40.8%) in 2020 to 2021, with an annual increase of 1.9% (95% CI, 1.4%-2.3%). The percentage of eligible RCTs citing non-Cochrane reviews increased from 46 of 183 RCTs (25.1%) in 2007 to 2008 to 65 of 102 RCTs (64.1%) in 2020 to 2021, with an annual increase of 3.3% (95% CI, 2.9%-3.7%). The percentage of eligible RCTs citing either type of systematic review increased from 65 of 183 RCTs (35.5%) in 2007 to 2008 to 73 of 102 RCTs (71.8%) in 2020 to 2021, with an annual increase of 3.0% (95% CI, 2.5%-3.5%). Overall, the percentage of eligible RCTs citing Cochrane reviews was 12.6% (95% CI, 8.6%-16.7%) lower than that of RCTs citing non-Cochrane reviews.

As shown in Table 2 , the percentage of eligible RCTs citing systematic reviews varied considerably by clinical specialty ( χ 2  = 402; P  < .001). For citing Cochrane reviews, the percentage was highest in eligible RCTs in psychiatry (233 of 553 RCTs [42.1%]) and lowest in dermatology (10 of 109 RCTs [9.2%]). For citing non-Cochrane reviews, the highest and the lowest percentages were found in RCTs in orthopedics (108 of 193 RCTs [56.0%]) and ophthalmology (16 of 106 RCTs [15.1%]), respectively. For citing either type of systematic review, the highest citation percentage was in RCTs in psychiatry (386 RCTs [69.8%]) and the lowest percentage in ophthalmology (28 RCTs [26.4%]).

There were 3 factors associated with citing either type of systematic review. RCTs that recruited at least 100 participants were more likely (risk ratio [RR], 1.16; 95% CI, 1.03-1.30) to cite systematic reviews than those that recruited fewer than 100 participants. RCTs that were supported by nonindustry funders (RR, 1.43; 95% CI, 1.27-1.61), received no external funding (RR, 1.27; 95% CI, 1.08-1.50), and did not report funding sources (RR, 1.26; 95% CI, 1.08-1.46) were more likely to cite either type of systematic review than those supported by industry funders. RCTs with first or senior authors from high-income countries were more likely (RR, 1.10; 95% CI, 1.03-1.17) to cite systematic reviews than RCTs with first and senior authors from low- and middle-income countries. The number of recruiting centers and the requirement of journals were not associated with the citation of systematic reviews (eTable in Supplement 1 ).

A systematic review of prior research can help identify research gaps and ensure that new RCTs do not reexamine questions that have already been addressed. 24 - 26 In addition, a systematic review that critically assesses the quality of prior similar studies may provide details that improve the design of new RCTs. When new RCTs are reported, systematic reviews may be used to explicitly integrate results of new RCTs into the existing evidence base. Although this cross-sectional study found that the percentage of RCTs citing systematic reviews improved from 35.5% in 2007 to 2008 to 71.8% since 2020, more than a quarter of RCTs still failed to do so.

Approximately 56.6% of RCTs included in our study cited systematic reviews, an estimate similar to prior estimates, which were generally greater than 60%. 27 - 29 The slight difference may be associated with methodological differences. For example, prior estimates were generated from a single cohort of RCTs sharing similar health conditions and interventions under investigation, while our study sample included RCTs addressing various questions. In addition, the citation of systematic reviews improved in the last decade; therefore, estimates may be associated with the time when the RCTs were published.

RCTs with industry funding were less likely to cite systematic reviews. This is not surprising given that prior research has found that industry-funded RCTs were less likely to follow guidelines or recommendations on the design, conduct, or report of RCTs. For example, industry-funded RCTs were less likely to report results on trial registries 30 and more likely to report biased results in favor of sponsor products. 31 Although we expected that RCTs published in journals requiring citation of systematic reviews would be more likely to cite reviews, results did not support this assumption. This may be associated with a lack of enforcement of such requirements. Some reviews labeled as systematic reviews were not considered reviews per our criteria because of the failure to conduct a comprehensive search or quality assessment. 32 , 33 In addition, the citation of systematic reviews varied considerably by clinical specialty and thus should be especially prompted in clinical specialties with lower citation rates.

We hypothesized that RCTs were more likely to cite Cochrane reviews because the design of the study ensured the availability of at least 1 Cochrane review to cite, and the quality of Cochrane reviews is generally higher than that of non-Cochrane reviews. 34 However, the citation of Cochrane reviews was significantly lower than that of non-Cochrane reviews, especially in recent years. There may be steps that Cochrane can take to improve the citation of Cochrane reviews. For example, the term systematic review may be added to the title of Cochrane reviews to be more easily identified by researchers who search PubMed or Embase for systematic reviews rather than the Cochrane Library. In addition, stakeholders may consider transitioning all Cochrane reviews to open access. 35

Several obstacles may hinder the citation of systematic reviews. For example, high-quality and up-to-date systematic reviews are not always available to justify a new RCT and guide its design. We based our study sample on existing Cochrane reviews, but in many fields, the evidence has not been synthesized by existing systematic reviews to date. 36 Although we considered only non-Cochrane reviews that met some limited quality criteria, researchers should be aware that the quality of many existing systematic reviews is suboptimal. 37 - 39 To increase the citation of prior systematic reviews, more resources and researchers with expertise and experience are needed to improve the coverage of health topics by high-quality and up-to-date systematic reviews. 40 , 41 In addition to researchers and journals, more stakeholders in the research system, including academic institutions, institutional review boards, and funding agencies, 42 may play a role by mandating a systematic review of prior research as a prerequisite to approving new applications.

There are several limitations of our study. First, we considered reports of only RCTs and systematic reviews indexed in the Web of Science. Thus, the accuracy of our estimates depended on the coverage of journals and quality of indexing on the Web of Science. It was challenging to estimate the magnitude and direction of potential biases affecting estimates. Second, all eligible RCTs were identified from Cochrane reviews. It is unclear to what extent these RCTs could represent all published RCTs, although reviews covered topics across 15 specialties. Third, we used keywords to screen for possible systematic reviews from the reference list of eligible RCTs. There may be systematic reviews cited that we failed to identify. Fourth, although we developed criteria to define non-Cochrane systematic reviews, including a comprehensive search in 2 bibliographic databases and quality assessment of included RCTs, we did not formally assess the quality of these reviews. The quality of some non-Cochrane reviews may have been too low to guide the design of new RCTs. However, because we applied the same criteria to all non-Cochrane reviews, the trend of RCTs citing prior systematic reviews should not be significantly biased. Fifth, although the quality of Cochrane reviews is generally better than that of non-Cochrane reviews, 43 there is evidence showing that the quality of some Cochrane reviews is also suboptimal. 34 However, these low-quality Cochrane reviews are unlikely to be associated with estimates of RCTs citing prior systematic reviews.

Citing prior systematic reviews may imply that authors were more likely to consider prior evidence, and thus such citation was used as a proxy for considering prior evidence in this study. However, we should be aware that authors may cite a systematic review for many reasons; citing prior systematic reviews is not a direct measure of considering prior evidence. The agreement between citing prior systematic reviews and considering prior evidence remains unclear.

This cross-sectional study found that the citation of prior systematic reviews in reports of RCTs improved over time, but overall, approximately 40% of reports of RCTs failed to do so. These findings suggest that evidence-based research, such as through the use of a systematic review, should be promoted to inform the justification and design of new RCTs and to report their findings within the context of what is already known.

Accepted for Publication: January 5, 2023.

Published: March 23, 2023. doi:10.1001/jamanetworkopen.2023.4219

Open Access: This is an open access article distributed under the terms of the CC-BY License . © 2023 Jia Y et al. JAMA Network Open .

Corresponding Author: Jinling Tang, PhD, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Room D519, 1068 Xueyuan Ave, Shenzhen University Town, Shenzhen, China 518055 ( [email protected] ).

Author Contributions: Drs Tang and Robinson had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Jia, Yost, Lund, Tang, Robinson.

Acquisition, analysis, or interpretation of data: Jia, B. Li, Z. Yang, F. Li, Zhao, C. Wei, X. Yang, Jin, Liu, X. Wei, Robinson.

Drafting of the manuscript: Jia, C. Wei, X. Yang, X. Wei, Yost.

Critical revision of the manuscript for important intellectual content: B. Li, Z. Yang, F. Li, Zhao, Jin, Liu, Yost, Lund, Tang, Robinson.

Statistical analysis: Jia, B. Li, C. Wei.

Administrative, technical, or material support: Jia, B. Li, Z. Yang, Zhao, X. Wei, Yost.

Supervision: Yost, Lund, Tang, Robinson.

Conflict of Interest Disclosures: None reported.

Funding/Support: This study was supported by grant KQTD20190929172835662 from the Shenzhen Science and Technology Program from the Shenzhen Municipal Government, Guangdong Province, China.

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Data Sharing Statement: See Supplement 2 .

Additional Contributions: We thank Jennifer Dean Durning, MSN (Villanova University), for her kind help in data collection and manuscript review. We thank Yumen Wen, MD (Johns Hopkins University School of Medicine), for his kind help in data collection. We also thank Lance Rodewald, PhD (China Center for Disease Control and Prevention), for editing and polishing the manuscript. None of these individuals were compensated from this work.

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 27 August 2024

Charting sustainable urban development through a systematic review of SDG11 research

• Abdulaziz I. Almulhim   ORCID: orcid.org/0000-0002-5384-7219 1 ,
• Ayyoob Sharifi   ORCID: orcid.org/0000-0002-8983-8613 2 ,
• Yusuf A. Aina   ORCID: orcid.org/0000-0002-0763-9865 3 ,
• Shakil Ahmad 4 ,
• Luca Mora 5 , 6 ,
• Walter Leal Filho 7 , 8 &
• Ismaila Rimi Abubakar   ORCID: orcid.org/0000-0002-7994-2302 9  

Nature Cities ( 2024 ) Cite this article

369 Accesses

1 Altmetric

Metrics details

• Environmental studies
• Social policy

The Sustainable Development Goal (SDG) 11 underscores the imperative of creating inclusive, safe, resilient and sustainable cities and communities by 2030. Here we employ bibliometric techniques to assess the evolving landscape of SDG11 research. Using a comprehensive dataset of over 21,000 scholarly publications, we investigate publication trends, thematic focus areas, authorship patterns, keyword co-occurrences and citation networks related to SDG11 research. The results reveal a consistent increase in research output, reflecting the growing global interest in urban sustainability studies. We identify influential authors, organizations and countries shaping the research landscape, highlighting existing global collaborative networks and emerging research hubs. Core thematic areas emphasize critical topics and interdisciplinary connections. Citation networks underscore the impacts of disseminating research outputs, including seminal works. This study offers insights for policymakers, academics and practitioners to align their collective efforts toward sustainable, inclusive and climate-resilient urban development. Moreover, it advances SDG11 by noting opportunities for further research, knowledge dissemination and international collaboration.

Similar content being viewed by others

A big data approach to assess progress towards Sustainable Development Goals for cities of varying sizes

Examining the unsustainable relationship between SDG performance, ecological footprint and international spillovers

Sustainable Development Goals (SDGs): Are we successful in turning trade-offs into synergies?

The growing interest in sustainable urban development is driven by challenges posed by urbanization, socioeconomic activities and environmental issues 1 . Urban areas contribute 80% of the world’s gross domestic product 2 , but also account for around 75% of global resource consumption, 65% of energy use and over 70% of carbon emissions 3 . The ecological footprint of urban environments, which measures the resources required to sustain socioeconomic activities, has been increasing 4 , 5 , and the global urban extent is projected to double by 2030 6 . Similarly, the global urban population is projected to reach 68% by 2050 7 , which could surpass the capacity of most urban areas 8 . Africa and Asia will host most of the future urban populations despite housing and infrastructure inadequacies 7 . Rapid urbanization, poverty and climate change (CC) further intensify the vulnerability of urban dwellers 9 .

Sustainable urban development aims to balance economic production, environmental protection and social inclusiveness. It emerged as a response to the critique of modernist views that prioritized physical appearance and order in cities over context, equity and inclusion 6 . Due to the limited progress in achieving the Millennium Development Goals, the Sustainable Development Goals (SDGs) were established in 2015 to ensure that no country is left behind in achieving sustainable development by 2030 10 . Many of the SDGs are closely related to urban settings, where sustainability challenges are complex and interwoven 11 . SDG11 specifically focuses on urban challenges and aims to make ‘cities and human settlements inclusive, safe resilient and sustainable’ by reducing the negative effects of urban development while improving socioeconomic development 10 .

The importance of SDG11 stems from the principles of inclusive, safe and resilient city. An inclusive city is characterized by the idea that all individuals, irrespective of their economic status, gender, race, ethnicity or religion, have the ability and empowerment to actively engage in the social, economic and political opportunities available within urban environments 6 . It seeks to address environmental racism and promote inclusive and fair urban development through social justice and equitable distribution of environmental benefits and burdens. In such a city, everyone is afforded equal access and participation in the diverse aspects that cities provide. On the other hand, a safe city refers to a city that possesses the capacity to provide protection and security against potential dangers, harm or risks, while a resilient city denotes a city’s ability to recover and restore its fundamental functions and structures following natural disasters and crises caused by human activities 6 , 8 . SDG11 is significant because it aims to ensure that cities develop sustainably.

However, SDG11 has been criticized for its limited emphasis on urban inequalities, decentralization and funding for local authorities 6 . Other challenges include localizing the universal indicators 12 , governance issues 13 , data accessibility and comparability 14 and smart city development 12 , 15 . Nevertheless, SDG11 serves as a platform for directing and monitoring urban development, fostering socioeconomic development and ensuring equity, inclusion and environmental protection 16 . Therefore, it is crucial to assess the literature on progress toward SDG11 targets 10 , especially at the halfway point to the target year, to inform interventions necessary for their achievement 17 .

While SDG11 has attracted significant global research attention 18 , comprehensive reviews of SDG11 literature are limited. Existing studies have primarily focused on assessing all the SDGs 19 , 20 , which obscures specific challenges and makes it difficult to track progress or design targeted interventions for individual goals. Recent work has highlighted the insufficient achievement of the SDGs and the need for transformative governance and participatory approaches 21 . Other studies have underscored the gap between research and policies, the underutilization of cities as pivotal arenas for achieving SDGs 22 and the lack of indicators to measure progress toward implementing SDGs 15 . Some studies have assessed SDGs’ implementation in specific region 17 , their impacts on addressing risks 23 and crises 1 , and their implications for health and well-being 24 , environmental research 25 and private sector involvement 26 . Most of the SDG research emanates from developed countries, showing a gap in the coverage of developing countries 27 . The few SDG11 studies in the Global South have narrow focus. While one paper investigated the impact of SDG11 on forest-based livelihoods 28 , another study researched the challenges of SDG11 implementation using a single-country experience 6 . Therefore, an in-depth and broad review of SDG11 literature is necessary to bridge this knowledge gap and identify key challenges and opportunities as well as potential pathways for achieving the targets set in SDG11.

Therefore, this research aims to assess the SDG11 research trends and themes using a bibliometric technique. It is the first global and comprehensive scientometric study on the SDG11 domain. By focusing on research conducted since the formulation of the SDGs, the study addresses the following research questions: (1) what are the global trends in SDG11 research? (2) How has the thematic focus of SDG11 research evolved over time? (3) What are the challenges and priority areas for SDG11 research? The contributions of the study to theory and practice are to:

Identify significant thematic areas and trends in SDG11 research since the promulgation of the SDGs, which can inform researchers, policymakers and practitioners about the current state of knowledge within the field and highlight priority areas for SDG11 research.

Map research clusters, knowledge sharing and collaboration patterns, thereby providing insights into the dynamics of research networks and facilitating the formulation of strategies to foster research excellence, interdisciplinary and international collaborations and the effective allocating of research resources.

Underscore the knowledge gaps, emerging topics and challenges within SDG11 research, offering evidence-based insights to align urban development initiatives with SDG11 research frontiers, enhance the efficacy of interventions and contribute to the development of inclusive, safe, resilient and sustainable cities.

SDG11 research trends

Research on SDG11 has significantly grown in terms of annual publications and citations since 2016, indicating a rising interest in this field (Fig. 1 ). The number of publications has increased by 1.3-fold, and this upward trajectory is expected to continue. Notable emerging research areas include the institutionalization of SDGs within local and global settings 18 and the impact of smart cities on advancing the SDGs 12 , 15 . Previously, studies on the epistemology and challenges of urban population growth were prevalent 29 . However, SDG11 research has now evolved into multidisciplinary fields, driven by heightened attention to urban challenges such as CC, urbanization and population growth.

A total of 21,153 articles were published, receiving 229,182 citations. The number of publications rose from 9,238 in period 1 (2016–2019) to 11,915 in period 2 (2020–2022).

Source data

The increasing trend in SDG11 publications can be attributed to several factors, including the desire to improve institutional rankings, a supportive research environment, investments and endowments, faculty promotion requirements and advancements in information and communication technology. There are also socioeconomic factors, such as increasing urbanization rates and gross domestic product, urban expansion and transformation, a deeper understanding of urban dynamics and challenges. Additionally, the policy environments in different countries can influence academic interests and research in urban studies, shaping research priorities and collaborations. Other contributing factors include research challenges faced by low-income countries and research support by governments, the private sector, international development agencies and scholars, all focusing on sustainable urban development.

SDG11 research is further propelled by recent international summits and collaborations that highlight the urgency of protecting the ecosystem and ensuring human safety 1 . Since 2015, CC issues have received greater attention due to key factors. The adoption of the Paris Agreement raised awareness and urgency for action on CC, resulting in a greater focus on related issues in various sectors, including urban planning and policy 13 . Scientific consensus on CC impacts and the role of human activities has also strengthened over the years, with Intergovernmental Panel on Climate Change assessments emphasizing the significance of cities in addressing CC 23 . As a result, CC considerations are increasingly integrated into research, policy and planning processes.

Urban planning and development strategies have prioritized climate mitigation and adaptation measures, such as reducing greenhouse gas emissions, promoting renewable energy, enhancing resilience to extreme weather events and incorporating green infrastructure. The focus on CC has accelerated the transition toward low-carbon and resilient cities, with efforts directed toward sustainable transportation, energy-efficient buildings, green spaces and climate-responsive infrastructure 6 . Collaboration and international cooperation are essential in addressing climate change, with cities and countries sharing best practices, knowledge and resources to develop and implement climate action plans 24 . Initiatives such as the C40 Cities Climate Leadership Group facilitate knowledge exchange and collective action among cities 30 . The increased attention to CC signifies a shift toward more sustainable and resilient urban development, emphasizing the need for proactive measures to mitigate greenhouse gas emissions, adapt to climate risks and promote equitable and sustainable urban environments.

Thematic focus of SDG11 research

There is an imbalance in the attention given to research themes within SDG11 as revealed by co-occurrence map (Supplementary Fig. 1 ). The dominant themes are affordable housing (SDG11.1), urban transport (SDG11.2), policy and governance (SDG11.3) and access to public spaces (SDG11.7). Housing affordability issues have consistently remained a focal point in SDG11 research, with urban studies, policy development and community-driven efforts for finding solutions to these complex challenges 30 , 31 . These issues were highlighted in Habitat I (Vancouver, 1976), which emphasized the importance of shifting governance and planning paradigms to develop policies and strategies to address rapid urbanization challenges, including shelter shortages and urban inequalities, and promote affordable housing options 30 , 32 . Habitat I has laid the foundation for subsequent global efforts and policy frameworks, such as Habitat II (Istanbul, 1996) and the New Urban Agenda, which continue to prioritize housing as a pivotal component of sustainable urban development. The persistent focus on affordable housing shows that cities still face many challenges in providing adequate housing for all 30 .

Urban policy and governance are other significant terms, indicating scholarly focus on strategies for promoting inclusive and sustainable urban development, enhancing participatory, integrated and sustainable urban planning and management. However, many cities lack the capacity to address urban inequalities, provide adequate housing 31 , public spaces and other urban services, which disproportionately affect women and racial minorities 30 . Moreover, urban redevelopment practices that lead to gentrification exacerbate existing inequalities 32 . Governance-based approaches seek to improve collaboration between public agencies and civil society to prioritize the implementation of urban planning strategies that enhance livability standards while addressing challenges such as CC and sustainability 30 .

Urban transport, which is related to SDG11.2 aiming to ensure safe, affordable, accessible and sustainable transport systems for all, has emerged as a key research theme. Important issues related to mobility, transportation and urban form include increased automobile dependence amid growing urbanization and suburbanization, challenges faced by public transit systems, growing awareness of environmental concerns, shift toward sustainable and multimodal transportation, transit-oriented development, integration of technology in transportation systems and the relationship between transportation and urban densification, compact development, CC adaptation and resilience, equity and social inclusion, and shifts in policy and governance approaches 1 , 6 , 11 . This theme also emphasizes the importance of walkability, public transit infrastructure and their role in enhancing transportation accessibility and influencing mode choice 33 . The transportation cluster also suggests that improving accessibility through urban form and built environment interventions can impact the travel behavior of urban residents and offer cobenefits for human health and environmental sustainability 24 . Incorporating such cobenefits in SDG11.2 could provide more incentives for access to safe efficient, equitable and sustainable transport infrastructure and systems in cities.

The implications of urbanization and land-use changes for sustainability, resilience and CC adaptation and mitigation in cities are also major themes. SDG11.6 aims to reduce the environmental impacts of cities, particularly in relation to air pollution and waste. The literature suggests that regulating urban growth 6 , controlling land-use changes, conserving biodiversity 27 and promoting green infrastructure are essential for achieving this target 34 . These actions, when implemented within integrated planning frameworks, can also reduce vulnerability, enhance resilience and contribute to progress in CC adaptation and mitigation, as emphasized in SDG11.5 (ref. 6 ). Such integrated frameworks should recognize the interconnections between various urban systems, including water, food, energy, waste and transportation, to promote sustainable and resilient urban development 35 . Cities are adopting strategies to reduce their carbon footprint, enhance energy efficiency and prepare for climate risks.

Smart cities and innovation enabled by information and communication technologies have increasingly been utilized to tackle urban development challenges and facilitate innovative and transformative urban governance mechanisms that contribute to the SDGs 15 . The rapid development and integration of digital technologies, such as the Internet of Things, artificial intelligence, big data analytics and sensor networks, have opened new possibilities for improving urban services, infrastructure and quality of life 33 . Smart cities leverage these technologies to enhance efficiency, connectivity and sustainability. The interest in smart cities stems from the recognition that technology can play a transformative role in addressing urban challenges, improving quality of life, promoting sustainability and fostering economic growth 12 , 36 . However, it is important to ensure that smart city initiatives are inclusive, equitable and responsive to the needs and aspirations of all residents.

Comparing the co-occurrence maps of period 1 and period 2 reveals limited changes in key thematic areas, despite the emergence of the coronavirus disease 2019 (COVID-19) pandemic during period 2 (Fig. 2 ). The key thematic areas in period 2, including urban governance and policy, transportation, urban sustainability and resilience, and urbanization and urban growth, remain consistent with period 1, indicating the continued relevance of these topics in research, albeit with potential expansions. However, a closer analysis of the clusters reveals that COVID-19 has emerged as a new area of SDG11 research in period 2, as attention has shifted toward adapting to the pandemic’s detrimental effects on cities. The pandemic has triggered paradigm shifts in various SDG11 domains, including public health, remote work, digitalization, vulnerabilities, inequalities, resilience, sustainability, urban spaces, proximity-based planning approaches such as the 15-minute city and global cooperation 9 . These shifts have influenced work, health, social equity, environmental stewardship 2 and urban planning, shaping innovative approaches and priorities in the postpandemic world. Urban inequality terms, such as slums and informality, inadequate housing and poverty, are brought to the forefront by the pandemic. Controlling the pandemic and addressing the citizen demand in slums and informal settlements has received significant attention 37 , 38 , 39 , 40 . Mobility restrictions and lockdowns to curb the virus’s transmission have presented challenges for service accessibility, particularly in disadvantaged neighborhoods where vulnerable groups reside. Lastly, the connection between sustainability and resilience has strengthened in the postpandemic period. The pandemic has offered new insights into the susceptibility of cities to various stressors and highlighted the inseparable connections between urban resilience and SDG11 (ref. 28 ).

a , b , The key thematic areas in period 1 (2016–2019) ( a ) are urban governance and policy (red), transportation (blue), urban sustainability and resilience (green), and urbanization and urban growth (yellow), while period 2 (2020–2022) ( b ) primarily focuses on urban governance and policies (red), urban studies (red), transportation (blue) and urbanization (green), particularly after the pandemic.

However, three SDG11 targets are not well-represented in both periods. One such target is SDG11.4, which aims to enhance efforts in preserving and conserving natural heritage, vital for improving urban sustainability 41 . Another target, SDG11.a, which focuses on strengthening urban–rural linkages, is also not adequately reflected in Fig. 2 . The intrinsic connection between cities and their surrounding rural areas necessitates the incorporation and strengthening of ties between urban and rural regions to achieve SDG11 (ref. 6 ). Gaps related to rural–urban linkages include limited understanding of interdependencies, inadequate infrastructure and services in rural areas, weak governance and coordination mechanisms, and social and cultural disconnect 13 . These gaps hinder the development of integrated strategies, contribute to economic disparities, limit access to services, impact agricultural productivity and food security, and create environmental and social challenges. Lastly, there is a lack of research on SDG11.c, which aims to support least-developed nations in developing safe and resilient urban areas, which is not surprising as these countries are often underrepresented in urban studies research 30 .

Major contributors to SDG11 research

Various countries, institutions, journals and authors have contributed to SDG11 research between 2016 and 2022. China leads in terms of the number of publications and citations generated, followed by the United States and the United Kingdom (Supplementary Fig. 2 and Supplementary Table 1 ). Among the top 20 productive countries, 14 are from the Global North countries, with South Africa and Brazil as the sole representative of Africa and Latin America and the Caribbean, respectively (Supplementary Fig. 3 and Supplementary Table 2 ). Increasing research collaboration among the top countries (Fig. 3 ), research infrastructure and facilities, manpower and financial support significantly contribute to their high SDG11 research output.

China followed by the United States and the United Kingdom dominates SDG11 research collaborations. There are significant connections among European, North American and Asian institutions, while Africa is less connected with Asia and Latin America and the Caribbean. Freq, frequently.

A co-citation analysis (Supplementary Table 3 ) reveals that Chinese institutions, such as the Chinese Academy of Sciences, have the highest number of articles and citation counts, followed by University College London and the University of Melbourne. The leading affiliations have changed over time, highlighting the strengthening of research institutes and the correlation between research collaboration and societal impacts (Supplementary Table 4 ). In terms of influential journals for SDG11 research, ‘land’ followed by ‘cities and land use’ policy tops the list (Supplementary Tables 5 and 6 ), with a growing interest in fields related to smart and sustainable cities, transport policies, regional planning and environmentally conscious building practices (Supplementary Fig. 4 ). These journals also address multiple issues related to environmental concerns, technological advancements, economic benefits, quality of life, justice and public awareness, driving the development of smart and sustainable cities.

The 15 most published authors in both periods focused on urbanization and urban growth, and the implementation, challenges and achievements of SDG11 (Supplementary Fig 5 ). This indicates an increased recognition of the SDG11 targets and their implementation over time, with the contributions of these authors significantly increasing from 2002 to 2016. Supplementary Table 7 shows that Chinese authors dominate the SDG11 publications, which correlates with China’s lead in institutions, affiliations and collaborations related to SDG11 research. The most cited SDG11 articles are revealed in Supplementary Table 8 , while the prominent authors that influenced SDG11 research are reported in Supplementary Table 9 . The top cited papers by SDG11 research are presented in Supplementary Tables 10 and 11 .

Key facts from the bibliometric analysis

The research on SDG11 has gained significant prominence across various fields, including urban studies, environmental sciences, geography, transportation and urban governance (Supplementary Table 12 ). The increasing environmental concerns, urbanization and global economic growth have spurred academic interest in SDG11 research from disciplines such as human geography, transportation, forestry, CC and sustainability science (Supplementary Table 13 ). Key thematic areas within SDG11 research encompass urban governance, affordable housing, transportation, urban sustainability and resilience, smart cities, urbanization and urban growth, which align closely with SDG11 targets 18 , 20 , 42 , 43 . However, research focus on SDG11 has remained relatively stable, with limited attention given to urban inequalities, safeguarding cultural and natural heritage 41 and specific impacts of the COVID-19 pandemic on urban sustainability.

This study reveals a notable increase in the total SDG11 research output from 2016 to 2022, reflecting the growing emphasis on SDG11 research in recent years compared with earlier periods. China emerges as the leaders in terms of research outputs, citations, authors, institutions and collaborations, closely followed by the United States and the United Kingdom. These three countries contribute 47.71% of SDG11 research productivity within this period, which is higher than 31% reported in a previous similar study 28 .

The dominance of Global North countries in the top 20 countries with the highest number of publications and citations related to SDG11 research is expected given their strong institutional capacity, research funding, highly ranked universities and collaborations. China’s surge in publications on SDG11 can be attributed to rapid urbanization, economic growth, government support and active international collaborations 2 , 11 . Generally, the landscape of research on SDG11 demonstrates an Anglo–American hegemony, which may reinforce power asymmetries and have significant implications for sustainability and resilience 30 . It is concerning that while projections indicate that 90% of future urban population growth will occur in cities of the Global South, particularly Africa and Asia, there is limited research on urban development challenges in these regions 7 .

The debate about the politics of knowledge production in SDG11 research often revolves around the controls of knowledge production processes. Large, well-funded institutions in developed countries tend to dominate research agendas, focusing on themes and solutions relevant to their own contexts, overlooking the unique needs and challenges of the Global South, which perpetuate existing inequalities and privileging certain types of knowledge. Also, knowledge production involves recognizing and integrating diverse ways of knowing. While Western scientific paradigms have traditionally dominated SDG11 research, there is an increasing recognition of the importance of indigenous and non-Western knowledge systems. Integrating these diverse epistemologies enriches understanding and leads to more effective and culturally relevant solutions.

Additionally, SDG11 research is inherently interdisciplinary, involving fields such as urban planning, sociology, environmental science and public policy. However, interdisciplinary collaboration can be challenging due to differing terminologies, methodologies and research priorities. Navigating these differences becomes crucial in the politics of knowledge production to create cohesive and comprehensive research outputs. Finally, bridging the gap between knowledge production and its implementation faces political, economic and social barriers. Researchers and practitioners are increasingly considering how knowledge on urban sustainability can effectively influence policymaking and practice in diverse urban contexts. Mobilizing knowledge to address these barriers becomes a key consideration in the politics of knowledge production.

Challenges to achieving SDG11

There are several challenges to achieving SDG11 targets, including inadequate provision of affordable housing 31 , essential services 24 , green spaces 2 , 34 , efficient transportation 33 and conservation of cultural and natural assets 25 . Rapid urbanization 1 , 7 , CC impacts 44 , insufficient investment in public infrastructure 30 , poor governance 13 and widening livelihood, land and resources inequalities 43 further exacerbate these challenges. For example, rapid urbanization puts immense pressure on housing, infrastructure, services and resources, making it challenging to effectively manage urban growth and ensure sustainable urban development 11 . Inadequate urban planning and land-use policies lead to inefficient land utilization, urban sprawl and inadequate provision of basic services 7 , 21 . The existence of slums and informal settlements where a large portion of the urban dwellers live in substandard housing conditions without tenure security 14 and limited access to electricity, water, sanitation, education, healthcare and employment opportunities 23 , 37 , and marginalized and vulnerable populations facing social exclusion, add to the complexity.

Moreover, competing priorities and trade-offs, lack of integration among various urban sectors and agencies 35 , inadequate human, technical and material resources at local government levels 45 , and insufficient local indicators and methods for implementation and monitoring 46 often hamper the implementation of SDG11 targets. Additionally, limited awareness of SDG-related challenges for policy formulation and implementation hinders context-depended decision-making and targeted interventions 21 , 27 . Addressing social inequalities, ensuring inclusivity in urban development and synergy among multiple fields, including social, technical, environmental, policy and management are crucial for achieving SDG11 (refs. 14 , 26 , 46 ). A valuable lesson can be learned from the success of the framework for assessing the implementation of SDG11 targets at the local level in Japan 42 .

Conclusions

This study aims to enhance our understanding of urban sustainability and provide insights for future research, policies and actions needed to achieve SDG11 targets. By conducting a comprehensive bibliometric assessment of over 21,000 publications from 2016 to 2022, it significantly contributes to the existing body of knowledge, highlighting trends, thematic areas and knowledge gaps related to SDG11 research across countries, institutions, authors and journals. SDG11 research has evolved into a multidisciplinary field, encompassing diverse themes, such as transportation, housing, urban sustainability, smart cities, urbanization and urban governance and policy. However, there is a need to address the gaps in research on urban safety and inclusion, which are critical dimensions often overlooked in favor of environmental and economic aspects of sustainability. This imbalance in research thematic areas risks perpetuation of already existing disparities within SDG11 research and its goals.

China, the United States and the United Kingdom emerge as the top contributors to SDG11 research and collaboration. To foster more SDG11 research in low-income economies, it is essential to provide increased funding support, capacity building and training for scholars, promote collaboration and knowledge exchange, and improve research infrastructure and data collection. Despite global challenges such as armed conflicts, CC and the COVID-19 pandemic, progress toward achieving the SDGs will become apparent by 2030. However, there are still opportunities for further research, knowledge dissemination and international collaboration toward developing safe, sustainable and inclusive urban development. The following are priority areas for SDG11 research:

Urban policy and governance: reforms should focus on providing equitable access to basic services such as water, sanitation, electricity, healthcare and education; upgrading and formalizing informal settlements; and improving living conditions of over one billion people residing in slums 37 . Participatory governance, community engagement and empowerment can enhance social inclusion by considering the voices and needs of marginalized groups 13 , 23 . Urban policy should also prioritize preserving historic and natural resources, protecting vulnerable areas and implementing sustainable urban design principles 47 . Future studies can help understand the dynamics, challenges and opportunities and monitor progress toward SDG11 targets 15 .

Localizing SDG11 targets: spatial planning and land-use strategies should consider the needs of diverse urban populations, promote inclusive zoning and engage local communities and stakeholders in decision-making processes, crucial for fostering ownership, empowerment and social cohesion, leading to more sustainable and inclusive urban development 3 . However, enhancing the capacity for localizing SDG11 targets requires building the knowledge and skills of local governments, policymakers and practitioners. Capacity-building initiatives, such as training programs, workshops and knowledge exchange, can promote interdisciplinary understanding and sharing of best practices.

Concerted and collaborative efforts: the international community, academics, policymakers and stakeholders can work together to create inclusive, safe, resilient and sustainable communities. Collaborative efforts can facilitate a comprehensive understanding of urban challenges and potential solutions by integrating diverse perspectives, data and methodologies. Disseminating research findings contributes to evidence-based policy development and informed decision-making, enabling the learning of lessons and replication of successful interventions.

Breaking down silos: integrated and cross-sectoral approaches help narrow the gaps between sectors, local governments, policymakers and stakeholders, leveraging local resources and capacities while fostering communication, knowledge sharing and collaboration 31 . Cross-sectoral working groups, joint planning processes and integrated policy frameworks promote holistic and coordinated decision-making among various sectors, including urban planning, housing, transportation, health, education, environment and social welfare 47 .

Digitalization and smart city development: maximizing the benefits of digitalization and smart city solutions requires addressing challenges such as bridging digital divides and ensuring data access, privacy and security. Prioritizing citizen-centric approaches and public accessibility to technology 36 are essential for leveraging expertise and resources 15 . Interoperability, scalability, data-driven decision-making and inclusivity contribute to evidence-based planning and equitable access to smart city technologies 12 , 48 , 49 , 50 , 51 .

This study comprehensively assessed SDG11 research, emphasizing significant thematic areas, trends, challenges and suggestions for prioritizing SDG11, including effective urban policy and governance, localizing SDG11 targets, concerted and collaborative efforts, and digitalization and smart city development. To broaden the scope of SDG11 research, future bibliometric reviews should encompass non-Web of Science databases and gray literature, including publications from government and nongovernmental agencies. Despite its limitations, this study’s findings provide valuable references for further research on SDG11.

The present study utilized a bibliometric technique to analyze academic publication on SDG11, tracing the research trend, the evolving key themes and identifying contributing authors, institutions and countries. Bibliometrics is a quantitative technique that allows for the analysis of trends in scholarly publications, such as research articles, conference papers and books, and visualizes scholarly publication patterns 52 . This technique is instrumental in analyzing extensive literature sets by relying on statistical observations and text-mining capabilities, which qualitative review methods such as systematic reviews cannot accomplish 53 . Additionally, it presents a scientific landscape of authors, countries, organizations and collaborations that contribute to worldwide scientific literature.

Bibliometric analysis requires interpretation, introducing an element of subjectivity 54 . Therefore, a sensemaking approach was adopted to transition from describing the bibliometric results to interpreting them. Sensemaking helps derive insightful information from bibliometric analysis and can be integrated into systematic literature reviews 55 , 56 . It applies to various international indexing, abstracting and citation databases, such as Scopus, Web of Science, Dimensions, PubMed and Education Resources Information Center, which cover journals, books, reviews and conference proceedings from around the world and different regions. For this study, Web of Science was chosen as the database to obtain bibliographic data due to its wide range of topics in various fields of study such as natural sciences, health sciences, engineering, social science, computer science and materials sciences. It is one of the world’s largest peer-reviewed scientific literature databases, with 87 million indexed items.

Specialized bibliometrics software were employed, including VOSviewer (version 1.6.19) 52 , Biblioshiny (version 4.1.3) 55 and BibExcel (version 2017) 57 . VOSviewer, known for its user-friendly interface, was used to understand the thematic focus and evolution of research on SDG11. It generates networks of nodes and links, with node size representing the frequency of the studied item, and link width indicating the strength of connections between items. Clusters of intricately linked nodes are shown in distinct colors. The thematic focus was examined for two periods: period 1 (2016–2019) and period 2 (2020–2022), considering the time since the SDGs were introduced to the time of data collection in this study. Another reason for this categorization is that evidence shows that the pandemic has significantly affected progress toward achieving SDGs 58 . VOSviewer allows for various types of analysis, including term co-occurrence, co-citation, citation and bibliographic coupling 53 . A term co-occurrence analysis was used in this study to highlight key thematic areas. To ensure accuracy and avoid separate counting of synonyms, a thesaurus file was developed and added to the software before the analysis. A summary of the data, including the number of authors and journals, used in the analysis is presented in Table 1 and will be further explained below.

A comprehensive search query was formulated to retrieve relevant data on SDG11, and it was executed in the title, abstract and keywords fields (TS) in Web of Science on 5 July 2023. The initial query shown the following box resulted in a total of 334,224 documents. Co-citation analysis was employed to identify the most influential journals contributing to SDG11 research. Two works are considered co-cited when they are both mentioned in the works cited section of a subsequent publication 59 (Zhao, 2006).

TS = ((‘city’ OR ‘cities’ OR ‘human settlement*’ OR ‘urban’ OR ‘metropoli*’ OR ‘town*’ OR ‘municipal*’ OR ‘peri-urban*’ OR ‘urban-rural’ OR ‘rural-urban’) AND (‘gentrification’ OR ‘congestion’ OR ‘transport*’ OR ‘housing’ OR ‘slum*’ OR ‘informal settlement*’ OR ‘sendai framework’ OR ‘Disaster Risk Reduction’ OR ‘disaster’ OR ‘DRR’ OR ‘smart cit*’ OR ‘resilient building*’ OR ‘sustainable building*’ OR ‘building design’ OR ‘buildings design’ OR ‘urbani?ation’ OR ‘zero energy’ OR ‘zero-energy’ OR ‘basic service*’ OR ‘governance’ OR ‘citizen participation’ OR ‘collaborative planning’ OR ‘participatory planning’ OR ‘inclusiveness’ OR ‘cultural heritage’ OR ‘natural heritage’ OR ‘UNESCO’ OR ‘ecological footprint’ OR ‘environmental footprint’ OR ‘waste’ OR ‘pollution’ OR ‘pollutant*’ OR ‘waste water’ OR wastewater* OR waste-water* OR ‘recycling’ OR ‘circular economy’ OR ‘air quality’ OR ‘green space’ OR ‘green spaces’ OR ‘nature inclusive’ OR ‘nature inclusive building’ OR ‘nature inclusive buildings’ OR ‘resilient’ OR ‘resilience’ OR ‘healthy cit*’ OR ‘sustainable’ OR ‘sustainability’ OR ‘green’ OR ‘nature*’ OR ‘Green infrastructure*’ OR ‘nature-based solution*’ OR ‘nature based solution*’ OR ‘child*’ OR ‘wom?n’ OR ‘elderl*’ OR ‘disabl*’ OR ‘disabilit*’ OR ‘disabled’)) AND PY = (2016–2022) NOT PY = (2023)

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) framework was used to report document search and filtration process. The PRISMA framework is designed to help scholars transparently report why their review study is conducted, what activities are performed and what discoveries are made, ideal for both systematic reviews and bibliometric studies 60 . PRISMA presents the four stages of the above query’s overall searching and filtration process (Fig. 4 ). The identification stage yielded 334,224 records, which were then screened to select only article-type documents ( n  = 277,165). Subsequently, documents were further screened based on language, selecting only English documents ( n  = 257,374). In the final stage, documents were screened based on specific categories closely related to cities and SDG11, resulting in a selection of six major categories: urban studies, environmental studies, geography, urban and regional planning, architecture, transportation and physical geography ( n  = 21,168). Finally, 15 duplicated documents were removed, resulting in a final dataset of 21,153 documents.

A four-phase flow diagram of the data extraction and filtration process of SDG11 literature, adapted from Priyadarshini 57 . WoS, Web of Science.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Data availability

The data that support the findings of this study are available as supplementary information. The steps for curating the data from the Web of Science have been provided in the text. If there is a further need, data are available on figshare at https://doi.org/10.6084/m9.figshare.26360125 . Source data are provided with this paper.

Almulhim, A. I. & Cobbinah, P. B. Can rapid urbanization be sustainable? The case of Saudi Arabian cities. Habitat Int. 139 , 102884 (2023).

Article   Google Scholar  

Huang, W. et al. Evaluating green city development in China using an integrated analytical toolbox. J. Clean. Prod. 400 , 136703 (2023).

Feng, T. & Zhou, B. Impact of urban spatial structure elements on carbon emissions efficiency in growing megacities: the case of Chengdu. Sci. Rep. 13 , 9939 (2023).

Javeed, S., Siddique, H. M. A. & Javed, F. Ecological footprint, globalization, and economic growth: evidence from Asia. Environ. Sci. Pollut. Res. 30 , 77006–77021 (2023).

Rees, W. E. Cities, energy, and the uncertain future of urban civilization. Oxf. Dev. Stud. 51 , 11–17 (2023).

Abubakar, I. R. & Aina, Y. A. The prospects and challenges of developing more inclusive, safe, resilient, and sustainable cities in Nigeria. Land Use Policy 87 , 104105 (2019).

UN-Habitat World Cities Report 2022: Envisaging the Future of Cities. (UN-Habitat, 2022).

Almulhim, A. I. & Cobbinah, P. B. Framing resilience in Saudi Arabian cities: on climate change and urban policy. Sustain. Cities Soc. 101 , 105172 (2024).

Barbier, E. B. & Burgess, J. C. Sustainability and development after COVID-19. World Dev. 135 , 105082 (2020).

The Sustainable Development Goals Report 2018 (United Nations, 2018).

Chen, M., Chen, L., Cheng, J. & Yu, J. Identifying interlinkages between urbanization and Sustainable Development Goals. Geogr. Sustain. 3 , 339–346 (2022).

Clement, J., Ruysschaert, B. & Crutzen, N. Smart city strategies—a driver for localizing sustainable development goals? Ecol. Econ. 213 , 107941 (2023).

Hansson, S., Arfvidsson, H. & Simon, D. Governance for sustainable urban development: the double function of SDG indicators. Area Dev. Policy 4 , 217–235 (2019).

Leal Filho, W. et al. Using the sustainable development goals towards a better understanding of sustainability challenges. Int. J. Sustain. Dev. World Ecol. 26 , 179–190 (2019).

Parra-Dominguez, J., Gil-Egido, A. & Rodríguez-González, S. SDGs as one of the drivers of smart city development: the indicator selection process. Smart Cities 5 , 1025–1038 (2022).

Grainger-Brown, J., Malekpour, S., Raven, R. & Taylor, E. Exploring urban transformation to inform the implementation of the Sustainable Development Goals. Cities 131 , 103928 (2022).

Xu, Z. et al. Assessing progress towards sustainable development over space and time. Nature 577 , 74–78 (2020).

Salvia, A. L., Leal Filho, W., Brandli, L. L. & Griebeler, J. S. Assessing research trends related to Sustainable Development Goals: local and global issues. J. Clean. Prod. 208 , 841–849 (2019).

Indana, F. & Pahlevi, R. W. A bibliometric approach to Sustainable Development Goals (SDGs) systematic analysis. Cogent Bus. Manag. 10 , 2224174 (2023).

Yamaguchi, N. U. et al. Sustainable development goals: a bibliometric analysis of literature reviews. Environ. Sci. Pollut. Res. 30 , 5502–5515 (2023).

Pattberg, P. & Bäckstrand, K. Enhancing the achievement of the SDGs: lessons learned at the half-way point of the 2030 Agenda. Int. Environ. Agree. 23 , 107–114 (2023).

Keith, M. et al. A new urban narrative for sustainable development. Nat. Sustain. 6 , 115–117 (2023).

Pedersen, A. B. et al. SDGs at the halfway point: how the 17 global goals address risks and wicked problems. Ambio 52 , 679–682 (2023).

Sweileh, W. M. Bibliometric analysis of scientific publications on ‘sustainable development goals’ with emphasis on ‘good health and well-being’ goal (2015–2019). Global. Health 16 , 68 (2020).

Mihelcic, J. R. et al. Environmental research addressing Sustainable Development Goals. Environ. Sci. Technol. 57 , 3457–3460 (2023).

Palau-Pinyana, E., Llach, J. & Bagur-Femenías, L. Mapping enablers for SDG implementation in the private sector: a systematic literature review and research agenda. Manag. Rev. Q. 26 , 1–30 (2023).

Google Scholar  

Mishra, M. et al. A bibliometric analysis of sustainable development goals (SDGs): a review of progress, challenges, and opportunities. Environ. Dev. Sustain. 26 , 11101–11143 (2023).

Devisscher, T. et al. in Sustainable Development Goals: Their Impacts on Forests and People (eds. Katila, P. et al.) (Cambridge Univ. Press, 2019).

Brenner, N. & Theodore, N. Cities and the geographies of ‘actually existing neoliberalism’. Antipode 34 , 349–379 (2002).

Sharifi, A., Khavarian-Garmsir, A. R., Allam, Z. & Asadzadeh, A. Progress and prospects in planning: a bibliometric review of literature in urban studies and regional and urban planning, 1956–2022. Prog. Plann. 173 , 100740 (2023).

Olanrewaju, A., Tan, S. Y. & Abdul-Aziz, A.-R. Housing providers’ insights on the benefits of sustainable, affordable housing. Sustain. Dev. 26 , 847–858 (2018).

Rice, J. L., Cohen, D. A., Long, J. & Jurjevich, J. R. Contradictions of the climate-friendly city: new perspectives on eco-gentrification and housing justice. Int. J. Urban Reg. Res. 44 , 145–165 (2020).

Park, S., Choi, K. & Lee, J. S. To walk or not to walk: testing the effect of path walkability on transit users’ access mode choices to the station. Int. J. Sustain. Transp. 9 , 529–541 (2015).

Jayasooriya, V. M., Ng, A. W. M., Muthukumaran, S. & Perera, B. J. C. Green infrastructure practices for the improvement of urban air quality. Urban For. Urban Green. 21 , 34–47 (2017).

Hachaichi, M. & Egieya, J. Water–food–energy nexus in global cities: addressing complex urban interdependencies. Water Resour. Manag. 37 , 1811–1825 (2023).

Mora, L., Deakin, M. & Reid, A. Combining co-citation clustering and text-based analysis to reveal the main development paths of smart cities. Technol. Forecast. Soc. Change 142 , 56–69 (2019).

McFarlane, C. Rethinking informality: politics, crisis, and the city. Plan. Theory Pract. 13 , 89–108 (2012).

Harvey, D. From managerialism to entrepreneurialism: the transformation in urban governance in late capitalism. Geogr. Ann. B 71 , 3–17 (1989).

Peck, J. Political economies of scale: fast policy, interscalar relations, and neoliberal workfare. Econ. Geogr. 78 , 331–360 (2002).

Cervero, R. & Landis, J. Twenty years of the Bay Area rapid transit system: land use and development impacts. Transport. Res. A 31 , 309–333 (1997).

Guzman, P. C., Roders, A. R. P. & Colenbrander, B. J. F. Measuring links between cultural heritage management and sustainable urban development: an overview of global monitoring tools. Cities 60 , 192–201 (2017).

Yamasaki, K. & Yamada, T. A framework to assess the local implementation of Sustainable Development Goal 11. Sustain. Cities Soc. 84 , 104002 (2022).

van Zanten, J. A. & van Tulder, R. Towards nexus-based governance: defining interactions between economic activities and Sustainable Development Goals (SDGs). Int. J. Sustain. Dev. World Ecol. 28 , 210–226 (2021).

Londono-Pineda, A. A. & Cano, J. A. Assessments under the United Nations sustainable development goals: a bibliometric analysis. Environ. Clim. Technol. 26 , 166–181 (2022).

Biggeri, M. A. ‘Decade for Action’ on SDG localization. J. Hum. Dev. Capabil. 22 , 706–712 (2021).

Benedek, J., Ivan, K., Török, I., Temerdek, A. & Holobâcă, I. H. Indicator-based assessment of local and regional progress toward the Sustainable Development Goals (SDGs): an integrated approach from Romania. Sustain. Dev. 29 , 860–875 (2021).

Abubakar, I. R. & Alshammari, M. S. Urban planning schemes for developing low-carbon cities in the Gulf Cooperation Council region. Habitat Int. 138 , 102881 (2023).

Batty, M. Smart cities, big data. Environ. Plann. B 39 , 191–193 (2012).

Smith, N. Toward a theory of gentrification, a back to the city movement by capital, not people. J. Am. Plann. Assoc. 45 , 538–548 (1979).

Jacobs, J. The Death and Life of Great American Cities (Random House, 1961).

Smith, N. The New Urban Frontier: Gentrification and the Revanchist City (Routledge, 1996).

Donthu, N., Kumar, S., Mukherjee, D., Pandey, N. & Lim, W. M. How to conduct a bibliometric analysis: an overview and guidelines. J. Bus. Res. 133 , 285–296 (2021).

van Eck, N. J. & Waltman, L. Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics 84 , 523–538 (2010).

Hajek, P., Youssef, A. & Hajkova, V. Recent developments in smart city assessment: a bibliometric and content analysis-based literature review. Cities 126 , 103709 (2022).

Lim, W. M. & Kumar, S. Guidelines for interpreting the results of bibliometrics analysis: a sensemaking approach. Glob. Bus. Organ. Excell. 43 , 17–26 (2023).

Aria, M. & Cuccurullo, C. bibliometrix: an R-tool for comprehensive science mapping analysis. J. Informetr. 11 , 959–975 (2017).

Åström, F., Danell, R., Larsen, B. & Schneider, J. (eds) Celebrating Scholarly Communication Studies: A Festschrift for Olle Persson at His 60th Birthday Vol. 05-S (International Society for Scientometrics and Informetrics, 2009).

Priyadarshini, P. The COVID-19 pandemic has derailed the progress of Sustainable Development Goals. Anthr. Sci. 1 , 410–412 (2022).

Zhao, D. Towards all-author co-citation analysis. Inf. Process. Manag. 42 , 1578–1591 (2006).

Shamseer, L. et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 349 , g7647 (2015).

Download references

Acknowledgements

A.I.A. acknowledges Imam Abdulrahman Bin Faisal University in Dammam, Saudi Arabia, for their support in conducting this study. A.S. acknowledges the support of the Japan Society for the Promotion of Science KAKENHI grant number 22K04493. We appreciate Hiroshima University for supporting the open-access publication of this article.

Author information

Authors and affiliations.

Department of Urban and Regional Planning, College of Architecture and Planning, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia

Abdulaziz I. Almulhim

The IDEC Institute and Network for Education and Research on Peace and Sustainability, Hiroshima University, Hiroshima, Japan

Ayyoob Sharifi

Department of Geomatics Engineering Technology, Yanbu Industrial College, Yanbu, Saudi Arabia

Yusuf A. Aina

Directorate of Library Affairs, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia

Shakil Ahmad

The Business School, Edinburgh Napier University, Edinburgh, UK

Academy of Architecture and Urban Studies, Tallinn University of Technology, Tallinn, Estonia

Department of Natural Sciences, Manchester Metropolitan University, Manchester, UK

Walter Leal Filho

Research and Transfer Centre ‘Sustainable Development and Climate Change Management’, Hamburg University of Applied Sciences, Hamburg, Germany

College of Architecture and Planning, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia

Ismaila Rimi Abubakar

You can also search for this author in PubMed   Google Scholar

Contributions

A.I.A.: conceptualization, methodology, formal analysis, data curation, writing—original draft, writing—review and editing, investigation and project administration. A.S.: methodology, software, formal analysis, visualization and writing—original draft. Y.A.A.: conceptualization, writing—original draft, investigation and validation. S.A.: methodology, software, formal analysis, visualization and data curation. L.M.: writing—review and editing, and investigation. W.L.F.: writing—review and editing, and investigation. I.R.A.: writing—review and editing, investigation, supervision, validation and resources.

Corresponding authors

Correspondence to Abdulaziz I. Almulhim or Ayyoob Sharifi .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Nature Cities thanks Aliyu Salisu Barau, Oluwatobi Owojori, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information.

Supplementary Figs. 1–5 and Tables 1–13.

Reporting Summary

Source data fig. 1.

Statistical source data in Excel format.

Source Data Fig. 2

Source data fig. 3, source data fig. 4, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Almulhim, A.I., Sharifi, A., Aina, Y.A. et al. Charting sustainable urban development through a systematic review of SDG11 research. Nat Cities (2024). https://doi.org/10.1038/s44284-024-00117-6

Download citation

Received : 22 September 2023

Accepted : 30 July 2024

Published : 27 August 2024

DOI : https://doi.org/10.1038/s44284-024-00117-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Expanding the concept of sustainable interaction design: a systematic review.

Share and Cite

Besana, N.; Krogh, P.G.; Petersen, M.G.; Spallazzo, D. Expanding the Concept of Sustainable Interaction Design: A Systematic Review. Sustainability 2024 , 16 , 7486. https://doi.org/10.3390/su16177486

Besana N, Krogh PG, Petersen MG, Spallazzo D. Expanding the Concept of Sustainable Interaction Design: A Systematic Review. Sustainability . 2024; 16(17):7486. https://doi.org/10.3390/su16177486

Besana, Nicola, Peter Gall Krogh, Marianne Graves Petersen, and Davide Spallazzo. 2024. "Expanding the Concept of Sustainable Interaction Design: A Systematic Review" Sustainability 16, no. 17: 7486. https://doi.org/10.3390/su16177486

Article Metrics

Supplementary material.

ZIP-Document (ZIP, 76 KiB)

Further Information

Mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Wiley Open Access Collection

An overview of methodological approaches in systematic reviews

Prabhakar veginadu.

1 Department of Rural Clinical Sciences, La Trobe Rural Health School, La Trobe University, Bendigo Victoria, Australia

Hanny Calache

2 Lincoln International Institute for Rural Health, University of Lincoln, Brayford Pool, Lincoln UK

Akshaya Pandian

3 Department of Orthodontics, Saveetha Dental College, Chennai Tamil Nadu, India

Mohd Masood

Associated data.

APPENDIX B: List of excluded studies with detailed reasons for exclusion

APPENDIX C: Quality assessment of included reviews using AMSTAR 2

The aim of this overview is to identify and collate evidence from existing published systematic review (SR) articles evaluating various methodological approaches used at each stage of an SR.

The search was conducted in five electronic databases from inception to November 2020 and updated in February 2022: MEDLINE, Embase, Web of Science Core Collection, Cochrane Database of Systematic Reviews, and APA PsycINFO. Title and abstract screening were performed in two stages by one reviewer, supported by a second reviewer. Full‐text screening, data extraction, and quality appraisal were performed by two reviewers independently. The quality of the included SRs was assessed using the AMSTAR 2 checklist.

The search retrieved 41,556 unique citations, of which 9 SRs were deemed eligible for inclusion in final synthesis. Included SRs evaluated 24 unique methodological approaches used for defining the review scope and eligibility, literature search, screening, data extraction, and quality appraisal in the SR process. Limited evidence supports the following (a) searching multiple resources (electronic databases, handsearching, and reference lists) to identify relevant literature; (b) excluding non‐English, gray, and unpublished literature, and (c) use of text‐mining approaches during title and abstract screening.

The overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process, as well as some methodological modifications currently used in expedited SRs. Overall, findings of this overview highlight the dearth of published SRs focused on SR methodologies and this warrants future work in this area.

1. INTRODUCTION

Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the “gold standard” of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search, appraise, and synthesize the available evidence. 3 Several guidelines, developed by various organizations, are available for the conduct of an SR; 4 , 5 , 6 , 7 among these, Cochrane is considered a pioneer in developing rigorous and highly structured methodology for the conduct of SRs. 8 The guidelines developed by these organizations outline seven fundamental steps required in SR process: defining the scope of the review and eligibility criteria, literature searching and retrieval, selecting eligible studies, extracting relevant data, assessing risk of bias (RoB) in included studies, synthesizing results, and assessing certainty of evidence (CoE) and presenting findings. 4 , 5 , 6 , 7

The methodological rigor involved in an SR can require a significant amount of time and resource, which may not always be available. 9 As a result, there has been a proliferation of modifications made to the traditional SR process, such as refining, shortening, bypassing, or omitting one or more steps, 10 , 11 for example, limits on the number and type of databases searched, limits on publication date, language, and types of studies included, and limiting to one reviewer for screening and selection of studies, as opposed to two or more reviewers. 10 , 11 These methodological modifications are made to accommodate the needs of and resource constraints of the reviewers and stakeholders (e.g., organizations, policymakers, health care professionals, and other knowledge users). While such modifications are considered time and resource efficient, they may introduce bias in the review process reducing their usefulness. 5

Substantial research has been conducted examining various approaches used in the standardized SR methodology and their impact on the validity of SR results. There are a number of published reviews examining the approaches or modifications corresponding to single 12 , 13 or multiple steps 14 involved in an SR. However, there is yet to be a comprehensive summary of the SR‐level evidence for all the seven fundamental steps in an SR. Such a holistic evidence synthesis will provide an empirical basis to confirm the validity of current accepted practices in the conduct of SRs. Furthermore, sometimes there is a balance that needs to be achieved between the resource availability and the need to synthesize the evidence in the best way possible, given the constraints. This evidence base will also inform the choice of modifications to be made to the SR methods, as well as the potential impact of these modifications on the SR results. An overview is considered the choice of approach for summarizing existing evidence on a broad topic, directing the reader to evidence, or highlighting the gaps in evidence, where the evidence is derived exclusively from SRs. 15 Therefore, for this review, an overview approach was used to (a) identify and collate evidence from existing published SR articles evaluating various methodological approaches employed in each of the seven fundamental steps of an SR and (b) highlight both the gaps in the current research and the potential areas for future research on the methods employed in SRs.

An a priori protocol was developed for this overview but was not registered with the International Prospective Register of Systematic Reviews (PROSPERO), as the review was primarily methodological in nature and did not meet PROSPERO eligibility criteria for registration. The protocol is available from the corresponding author upon reasonable request. This overview was conducted based on the guidelines for the conduct of overviews as outlined in The Cochrane Handbook. 15 Reporting followed the Preferred Reporting Items for Systematic reviews and Meta‐analyses (PRISMA) statement. 3

2.1. Eligibility criteria

Only published SRs, with or without associated MA, were included in this overview. We adopted the defining characteristics of SRs from The Cochrane Handbook. 5 According to The Cochrane Handbook, a review was considered systematic if it satisfied the following criteria: (a) clearly states the objectives and eligibility criteria for study inclusion; (b) provides reproducible methodology; (c) includes a systematic search to identify all eligible studies; (d) reports assessment of validity of findings of included studies (e.g., RoB assessment of the included studies); (e) systematically presents all the characteristics or findings of the included studies. 5 Reviews that did not meet all of the above criteria were not considered a SR for this study and were excluded. MA‐only articles were included if it was mentioned that the MA was based on an SR.

SRs and/or MA of primary studies evaluating methodological approaches used in defining review scope and study eligibility, literature search, study selection, data extraction, RoB assessment, data synthesis, and CoE assessment and reporting were included. The methodological approaches examined in these SRs and/or MA can also be related to the substeps or elements of these steps; for example, applying limits on date or type of publication are the elements of literature search. Included SRs examined or compared various aspects of a method or methods, and the associated factors, including but not limited to: precision or effectiveness; accuracy or reliability; impact on the SR and/or MA results; reproducibility of an SR steps or bias occurred; time and/or resource efficiency. SRs assessing the methodological quality of SRs (e.g., adherence to reporting guidelines), evaluating techniques for building search strategies or the use of specific database filters (e.g., use of Boolean operators or search filters for randomized controlled trials), examining various tools used for RoB or CoE assessment (e.g., ROBINS vs. Cochrane RoB tool), or evaluating statistical techniques used in meta‐analyses were excluded. 14

2.2. Search

The search for published SRs was performed on the following scientific databases initially from inception to third week of November 2020 and updated in the last week of February 2022: MEDLINE (via Ovid), Embase (via Ovid), Web of Science Core Collection, Cochrane Database of Systematic Reviews, and American Psychological Association (APA) PsycINFO. Search was restricted to English language publications. Following the objectives of this study, study design filters within databases were used to restrict the search to SRs and MA, where available. The reference lists of included SRs were also searched for potentially relevant publications.

The search terms included keywords, truncations, and subject headings for the key concepts in the review question: SRs and/or MA, methods, and evaluation. Some of the terms were adopted from the search strategy used in a previous review by Robson et al., which reviewed primary studies on methodological approaches used in study selection, data extraction, and quality appraisal steps of SR process. 14 Individual search strategies were developed for respective databases by combining the search terms using appropriate proximity and Boolean operators, along with the related subject headings in order to identify SRs and/or MA. 16 , 17 A senior librarian was consulted in the design of the search terms and strategy. Appendix A presents the detailed search strategies for all five databases.

2.3. Study selection and data extraction

Title and abstract screening of references were performed in three steps. First, one reviewer (PV) screened all the titles and excluded obviously irrelevant citations, for example, articles on topics not related to SRs, non‐SR publications (such as randomized controlled trials, observational studies, scoping reviews, etc.). Next, from the remaining citations, a random sample of 200 titles and abstracts were screened against the predefined eligibility criteria by two reviewers (PV and MM), independently, in duplicate. Discrepancies were discussed and resolved by consensus. This step ensured that the responses of the two reviewers were calibrated for consistency in the application of the eligibility criteria in the screening process. Finally, all the remaining titles and abstracts were reviewed by a single “calibrated” reviewer (PV) to identify potential full‐text records. Full‐text screening was performed by at least two authors independently (PV screened all the records, and duplicate assessment was conducted by MM, HC, or MG), with discrepancies resolved via discussions or by consulting a third reviewer.

Data related to review characteristics, results, key findings, and conclusions were extracted by at least two reviewers independently (PV performed data extraction for all the reviews and duplicate extraction was performed by AP, HC, or MG).

2.4. Quality assessment of included reviews

The quality assessment of the included SRs was performed using the AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews). The tool consists of a 16‐item checklist addressing critical and noncritical domains. 18 For the purpose of this study, the domain related to MA was reclassified from critical to noncritical, as SRs with and without MA were included. The other six critical domains were used according to the tool guidelines. 18 Two reviewers (PV and AP) independently responded to each of the 16 items in the checklist with either “yes,” “partial yes,” or “no.” Based on the interpretations of the critical and noncritical domains, the overall quality of the review was rated as high, moderate, low, or critically low. 18 Disagreements were resolved through discussion or by consulting a third reviewer.

2.5. Data synthesis

To provide an understandable summary of existing evidence syntheses, characteristics of the methods evaluated in the included SRs were examined and key findings were categorized and presented based on the corresponding step in the SR process. The categories of key elements within each step were discussed and agreed by the authors. Results of the included reviews were tabulated and summarized descriptively, along with a discussion on any overlap in the primary studies. 15 No quantitative analyses of the data were performed.

From 41,556 unique citations identified through literature search, 50 full‐text records were reviewed, and nine systematic reviews 14 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 were deemed eligible for inclusion. The flow of studies through the screening process is presented in Figure  1 . A list of excluded studies with reasons can be found in Appendix B .

Study selection flowchart

3.1. Characteristics of included reviews

Table  1 summarizes the characteristics of included SRs. The majority of the included reviews (six of nine) were published after 2010. 14 , 22 , 23 , 24 , 25 , 26 Four of the nine included SRs were Cochrane reviews. 20 , 21 , 22 , 23 The number of databases searched in the reviews ranged from 2 to 14, 2 reviews searched gray literature sources, 24 , 25 and 7 reviews included a supplementary search strategy to identify relevant literature. 14 , 19 , 20 , 21 , 22 , 23 , 26 Three of the included SRs (all Cochrane reviews) included an integrated MA. 20 , 21 , 23

Characteristics of included studies

SR = systematic review; MA = meta‐analysis; RCT = randomized controlled trial; CCT = controlled clinical trial; N/R = not reported.

The included SRs evaluated 24 unique methodological approaches (26 in total) used across five steps in the SR process; 8 SRs evaluated 6 approaches, 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 while 1 review evaluated 18 approaches. 14 Exclusion of gray or unpublished literature 21 , 26 and blinding of reviewers for RoB assessment 14 , 23 were evaluated in two reviews each. Included SRs evaluated methods used in five different steps in the SR process, including methods used in defining the scope of review ( n  = 3), literature search ( n  = 3), study selection ( n  = 2), data extraction ( n  = 1), and RoB assessment ( n  = 2) (Table  2 ).

Summary of findings from review evaluating systematic review methods

There was some overlap in the primary studies evaluated in the included SRs on the same topics: Schmucker et al. 26 and Hopewell et al. 21 ( n  = 4), Hopewell et al. 20 and Crumley et al. 19 ( n  = 30), and Robson et al. 14 and Morissette et al. 23 ( n  = 4). There were no conflicting results between any of the identified SRs on the same topic.

3.2. Methodological quality of included reviews

Overall, the quality of the included reviews was assessed as moderate at best (Table  2 ). The most common critical weakness in the reviews was failure to provide justification for excluding individual studies (four reviews). Detailed quality assessment is provided in Appendix C .

3.3. Evidence on systematic review methods

3.3.1. methods for defining review scope and eligibility.

Two SRs investigated the effect of excluding data obtained from gray or unpublished sources on the pooled effect estimates of MA. 21 , 26 Hopewell et al. 21 reviewed five studies that compared the impact of gray literature on the results of a cohort of MA of RCTs in health care interventions. Gray literature was defined as information published in “print or electronic sources not controlled by commercial or academic publishers.” Findings showed an overall greater treatment effect for published trials than trials reported in gray literature. In a more recent review, Schmucker et al. 26 addressed similar objectives, by investigating gray and unpublished data in medicine. In addition to gray literature, defined similar to the previous review by Hopewell et al., the authors also evaluated unpublished data—defined as “supplemental unpublished data related to published trials, data obtained from the Food and Drug Administration  or other regulatory websites or postmarketing analyses hidden from the public.” The review found that in majority of the MA, excluding gray literature had little or no effect on the pooled effect estimates. The evidence was limited to conclude if the data from gray and unpublished literature had an impact on the conclusions of MA. 26

Morrison et al. 24 examined five studies measuring the effect of excluding non‐English language RCTs on the summary treatment effects of SR‐based MA in various fields of conventional medicine. Although none of the included studies reported major difference in the treatment effect estimates between English only and non‐English inclusive MA, the review found inconsistent evidence regarding the methodological and reporting quality of English and non‐English trials. 24 As such, there might be a risk of introducing “language bias” when excluding non‐English language RCTs. The authors also noted that the numbers of non‐English trials vary across medical specialties, as does the impact of these trials on MA results. Based on these findings, Morrison et al. 24 conclude that literature searches must include non‐English studies when resources and time are available to minimize the risk of introducing “language bias.”

3.3.2. Methods for searching studies

Crumley et al. 19 analyzed recall (also referred to as “sensitivity” by some researchers; defined as “percentage of relevant studies identified by the search”) and precision (defined as “percentage of studies identified by the search that were relevant”) when searching a single resource to identify randomized controlled trials and controlled clinical trials, as opposed to searching multiple resources. The studies included in their review frequently compared a MEDLINE only search with the search involving a combination of other resources. The review found low median recall estimates (median values between 24% and 92%) and very low median precisions (median values between 0% and 49%) for most of the electronic databases when searched singularly. 19 A between‐database comparison, based on the type of search strategy used, showed better recall and precision for complex and Cochrane Highly Sensitive search strategies (CHSSS). In conclusion, the authors emphasize that literature searches for trials in SRs must include multiple sources. 19

In an SR comparing handsearching and electronic database searching, Hopewell et al. 20 found that handsearching retrieved more relevant RCTs (retrieval rate of 92%−100%) than searching in a single electronic database (retrieval rates of 67% for PsycINFO/PsycLIT, 55% for MEDLINE, and 49% for Embase). The retrieval rates varied depending on the quality of handsearching, type of electronic search strategy used (e.g., simple, complex or CHSSS), and type of trial reports searched (e.g., full reports, conference abstracts, etc.). The authors concluded that handsearching was particularly important in identifying full trials published in nonindexed journals and in languages other than English, as well as those published as abstracts and letters. 20

The effectiveness of checking reference lists to retrieve additional relevant studies for an SR was investigated by Horsley et al. 22 The review reported that checking reference lists yielded 2.5%–40% more studies depending on the quality and comprehensiveness of the electronic search used. The authors conclude that there is some evidence, although from poor quality studies, to support use of checking reference lists to supplement database searching. 22

3.3.3. Methods for selecting studies

Three approaches relevant to reviewer characteristics, including number, experience, and blinding of reviewers involved in the screening process were highlighted in an SR by Robson et al. 14 Based on the retrieved evidence, the authors recommended that two independent, experienced, and unblinded reviewers be involved in study selection. 14 A modified approach has also been suggested by the review authors, where one reviewer screens and the other reviewer verifies the list of excluded studies, when the resources are limited. It should be noted however this suggestion is likely based on the authors’ opinion, as there was no evidence related to this from the studies included in the review.

Robson et al. 14 also reported two methods describing the use of technology for screening studies: use of Google Translate for translating languages (for example, German language articles to English) to facilitate screening was considered a viable method, while using two computer monitors for screening did not increase the screening efficiency in SR. Title‐first screening was found to be more efficient than simultaneous screening of titles and abstracts, although the gain in time with the former method was lesser than the latter. Therefore, considering that the search results are routinely exported as titles and abstracts, Robson et al. 14 recommend screening titles and abstracts simultaneously. However, the authors note that these conclusions were based on very limited number (in most instances one study per method) of low‐quality studies. 14

3.3.4. Methods for data extraction

Robson et al. 14 examined three approaches for data extraction relevant to reviewer characteristics, including number, experience, and blinding of reviewers (similar to the study selection step). Although based on limited evidence from a small number of studies, the authors recommended use of two experienced and unblinded reviewers for data extraction. The experience of the reviewers was suggested to be especially important when extracting continuous outcomes (or quantitative) data. However, when the resources are limited, data extraction by one reviewer and a verification of the outcomes data by a second reviewer was recommended.

As for the methods involving use of technology, Robson et al. 14 identified limited evidence on the use of two monitors to improve the data extraction efficiency and computer‐assisted programs for graphical data extraction. However, use of Google Translate for data extraction in non‐English articles was not considered to be viable. 14 In the same review, Robson et al. 14 identified evidence supporting contacting authors for obtaining additional relevant data.

3.3.5. Methods for RoB assessment

Two SRs examined the impact of blinding of reviewers for RoB assessments. 14 , 23 Morissette et al. 23 investigated the mean differences between the blinded and unblinded RoB assessment scores and found inconsistent differences among the included studies providing no definitive conclusions. Similar conclusions were drawn in a more recent review by Robson et al., 14 which included four studies on reviewer blinding for RoB assessment that completely overlapped with Morissette et al. 23

Use of experienced reviewers and provision of additional guidance for RoB assessment were examined by Robson et al. 14 The review concluded that providing intensive training and guidance on assessing studies reporting insufficient data to the reviewers improves RoB assessments. 14 Obtaining additional data related to quality assessment by contacting study authors was also found to help the RoB assessments, although based on limited evidence. When assessing the qualitative or mixed method reviews, Robson et al. 14 recommends the use of a structured RoB tool as opposed to an unstructured tool. No SRs were identified on data synthesis and CoE assessment and reporting steps.

4. DISCUSSION

4.1. summary of findings.

Nine SRs examining 24 unique methods used across five steps in the SR process were identified in this overview. The collective evidence supports some current traditional and modified SR practices, while challenging other approaches. However, the quality of the included reviews was assessed to be moderate at best and in the majority of the included SRs, evidence related to the evaluated methods was obtained from very limited numbers of primary studies. As such, the interpretations from these SRs should be made cautiously.

The evidence gathered from the included SRs corroborate a few current SR approaches. 5 For example, it is important to search multiple resources for identifying relevant trials (RCTs and/or CCTs). The resources must include a combination of electronic database searching, handsearching, and reference lists of retrieved articles. 5 However, no SRs have been identified that evaluated the impact of the number of electronic databases searched. A recent study by Halladay et al. 27 found that articles on therapeutic intervention, retrieved by searching databases other than PubMed (including Embase), contributed only a small amount of information to the MA and also had a minimal impact on the MA results. The authors concluded that when the resources are limited and when large number of studies are expected to be retrieved for the SR or MA, PubMed‐only search can yield reliable results. 27

Findings from the included SRs also reiterate some methodological modifications currently employed to “expedite” the SR process. 10 , 11 For example, excluding non‐English language trials and gray/unpublished trials from MA have been shown to have minimal or no impact on the results of MA. 24 , 26 However, the efficiency of these SR methods, in terms of time and the resources used, have not been evaluated in the included SRs. 24 , 26 Of the SRs included, only two have focused on the aspect of efficiency 14 , 25 ; O'Mara‐Eves et al. 25 report some evidence to support the use of text‐mining approaches for title and abstract screening in order to increase the rate of screening. Moreover, only one included SR 14 considered primary studies that evaluated reliability (inter‐ or intra‐reviewer consistency) and accuracy (validity when compared against a “gold standard” method) of the SR methods. This can be attributed to the limited number of primary studies that evaluated these outcomes when evaluating the SR methods. 14 Lack of outcome measures related to reliability, accuracy, and efficiency precludes making definitive recommendations on the use of these methods/modifications. Future research studies must focus on these outcomes.

Some evaluated methods may be relevant to multiple steps; for example, exclusions based on publication status (gray/unpublished literature) and language of publication (non‐English language studies) can be outlined in the a priori eligibility criteria or can be incorporated as search limits in the search strategy. SRs included in this overview focused on the effect of study exclusions on pooled treatment effect estimates or MA conclusions. Excluding studies from the search results, after conducting a comprehensive search, based on different eligibility criteria may yield different results when compared to the results obtained when limiting the search itself. 28 Further studies are required to examine this aspect.

Although we acknowledge the lack of standardized quality assessment tools for methodological study designs, we adhered to the Cochrane criteria for identifying SRs in this overview. This was done to ensure consistency in the quality of the included evidence. As a result, we excluded three reviews that did not provide any form of discussion on the quality of the included studies. The methods investigated in these reviews concern supplementary search, 29 data extraction, 12 and screening. 13 However, methods reported in two of these three reviews, by Mathes et al. 12 and Waffenschmidt et al., 13 have also been examined in the SR by Robson et al., 14 which was included in this overview; in most instances (with the exception of one study included in Mathes et al. 12 and Waffenschmidt et al. 13 each), the studies examined in these excluded reviews overlapped with those in the SR by Robson et al. 14

One of the key gaps in the knowledge observed in this overview was the dearth of SRs on the methods used in the data synthesis component of SR. Narrative and quantitative syntheses are the two most commonly used approaches for synthesizing data in evidence synthesis. 5 There are some published studies on the proposed indications and implications of these two approaches. 30 , 31 These studies found that both data synthesis methods produced comparable results and have their own advantages, suggesting that the choice of the method must be based on the purpose of the review. 31 With increasing number of “expedited” SR approaches (so called “rapid reviews”) avoiding MA, 10 , 11 further research studies are warranted in this area to determine the impact of the type of data synthesis on the results of the SR.

4.2. Implications for future research

The findings of this overview highlight several areas of paucity in primary research and evidence synthesis on SR methods. First, no SRs were identified on methods used in two important components of the SR process, including data synthesis and CoE and reporting. As for the included SRs, a limited number of evaluation studies have been identified for several methods. This indicates that further research is required to corroborate many of the methods recommended in current SR guidelines. 4 , 5 , 6 , 7 Second, some SRs evaluated the impact of methods on the results of quantitative synthesis and MA conclusions. Future research studies must also focus on the interpretations of SR results. 28 , 32 Finally, most of the included SRs were conducted on specific topics related to the field of health care, limiting the generalizability of the findings to other areas. It is important that future research studies evaluating evidence syntheses broaden the objectives and include studies on different topics within the field of health care.

4.3. Strengths and limitations

To our knowledge, this is the first overview summarizing current evidence from SRs and MA on different methodological approaches used in several fundamental steps in SR conduct. The overview methodology followed well established guidelines and strict criteria defined for the inclusion of SRs.

There are several limitations related to the nature of the included reviews. Evidence for most of the methods investigated in the included reviews was derived from a limited number of primary studies. Also, the majority of the included SRs may be considered outdated as they were published (or last updated) more than 5 years ago 33 ; only three of the nine SRs have been published in the last 5 years. 14 , 25 , 26 Therefore, important and recent evidence related to these topics may not have been included. Substantial numbers of included SRs were conducted in the field of health, which may limit the generalizability of the findings. Some method evaluations in the included SRs focused on quantitative analyses components and MA conclusions only. As such, the applicability of these findings to SR more broadly is still unclear. 28 Considering the methodological nature of our overview, limiting the inclusion of SRs according to the Cochrane criteria might have resulted in missing some relevant evidence from those reviews without a quality assessment component. 12 , 13 , 29 Although the included SRs performed some form of quality appraisal of the included studies, most of them did not use a standardized RoB tool, which may impact the confidence in their conclusions. Due to the type of outcome measures used for the method evaluations in the primary studies and the included SRs, some of the identified methods have not been validated against a reference standard.

Some limitations in the overview process must be noted. While our literature search was exhaustive covering five bibliographic databases and supplementary search of reference lists, no gray sources or other evidence resources were searched. Also, the search was primarily conducted in health databases, which might have resulted in missing SRs published in other fields. Moreover, only English language SRs were included for feasibility. As the literature search retrieved large number of citations (i.e., 41,556), the title and abstract screening was performed by a single reviewer, calibrated for consistency in the screening process by another reviewer, owing to time and resource limitations. These might have potentially resulted in some errors when retrieving and selecting relevant SRs. The SR methods were grouped based on key elements of each recommended SR step, as agreed by the authors. This categorization pertains to the identified set of methods and should be considered subjective.

5. CONCLUSIONS

This overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process. Limited evidence was also identified on some methodological modifications currently used to expedite the SR process. Overall, findings highlight the dearth of SRs on SR methodologies, warranting further work to confirm several current recommendations on conventional and expedited SR processes.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

Supporting information

APPENDIX A: Detailed search strategies

ACKNOWLEDGMENTS

The first author is supported by a La Trobe University Full Fee Research Scholarship and a Graduate Research Scholarship.

Open Access Funding provided by La Trobe University.

Veginadu P, Calache H, Gussy M, Pandian A, Masood M. An overview of methodological approaches in systematic reviews . J Evid Based Med . 2022; 15 :39–54. 10.1111/jebm.12468 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]