Home            Past Issues            About IP            About IAP           Author Information            Subscription            Advertisement              Search  

   
Research brief

Indian Pediatr 2011;48: 383-385

Quality Assessment of Systematic Reviews of Health Care Interventions using AMSTAR

 

Vanitha Jagannath, Joseph L Mathew, GV Asokan and Zbys Fedorowicz

From UKCC (Bahrain Branch) and Ministry of Health, Bahrain; Box 25438, Bahrain.

Correspondence to: Dr Vanitha A Jagannath, Specialist Pediatrician, Department of Paediatrics,
KIMS Bahrain Medical Center, Um al Hassam Ave, Adliya, PO Box 175829 Manama,
 Bahrain.
Email: [email protected]

Received: March 04, 2010;
Initial review: March 25, 2010;
Accepted: July 22, 2010.
Published online 2010 November 30.

 PII: S097475591000187-2
 

Abstract

Appraisal of the methodological quality of systematic reviews would reflect on their utility for the clinicians and policymakers. This study was done to assess the quality of systematic reviews published in five leading Indian medical journals using AMSTAR. 22 systematic reviews of healthcare interventions were identified. The scores ranged 0 to 10 (mean 3.77 and median 2.5), 9 reviews scored > 4/11. Most frequent ‘yes’ and ‘no’ scores were: publication status as an inclusion criterion (12 /22), respectively and duplicate study selection and data extraction (17 /22). Several suboptimal aspects of methodological quality were identified in the reviews evaluated.

Key words: AMSTAR, India, Medical journals, Systematic reviews.


Healthcare professionals would need to read a large number of original articles every day to keep track of the latest information in their field; systematic reviews are one way in which they can keep abreast with current medical literature. Systematic reviews provide a logical synthesis of the research base via a carefully formulated question and analysis of all available evidence. A comprehensive search of the literature utilizing predetermined inclusion and exclusion criteria is followed by a critical appraisal of the risk of bias in any included studies, and it may also include a synthesis of data for the outcomes of interest [1]. An increasing number of systematic reviews are being published in a wide range of medical journals across the spectrum of healthcare and the methodological quality of these reviews varies widely [2]. In spite of the care with which they are conducted, some may provide different answers to the same clinical question [3]. It is imperative that systematic reviews are appraised for robustness of methodological quality before being used for either healthcare policy or clinical decision making [4]. There is an increased recognition that the methodological quality and reporting quality of systematic reviews are two distinctly different aspects to be considered in the appraisal of reviews. Methodological quality represents how well the systematic review was conducted (literature searching, pooling of data, etc). The reporting quality, however considers how well systematic reviewers have reported their methodology and findings. Currently there are more than 24 instruments available to appraise the methodological quality of systematic reviews, but few have been developed systematically or validated empirically. AMSTAR (a measurement tool to assess the methodological quality of systematic reviews) was developed specifically to fulfill this requirement [5].

We conducted this study to assess the quality of systematic reviews published in five leading Indian medical journals using AMSTAR instrument.

Methods

We identified 5 of the leading Indian journals in MEDLINE by their citation ratings. MEDLINE was searched, using ‘systematic review’ sub set limit, for citations to articles published in the 5 journals from January 2000 to November 2009. Assessment of the searches against pre-specified inclusion criteria was conducted in duplicate by two of the authors. Full text copies of all included studies were assessed against AMSTAR and the scores were recorded. The time taken to complete the assessment and difficulties encountered, if any, while administering the test were also noted.

Box I Items in AMSTAR Tool to Assess Methodological Quality of Systematic Review
 
  1. Was an ‘a priori’ design provided? 
  2. Was there duplicate study selection and data extraction? 
  3. Was a comprehensive literature search performed?
  4. Was the status of publication (i.e. grey literature) used as an inclusion criterion?  
  5. Was a list of studies (included and excluded) provided?
  6. Were the characteristics of the included studies provided?
  7. Was the scientific quality of the included studies assessed and documented?
  8. Was the scientific quality of the included studies used appropriately in formulating conclusions?
  9. Were the methods used to combine the findings of studies appropriate?
  10. Was the likelihood of publication bias assessed?
  11. Was the conflict of interest stated? Reviews that achieve high scores indicate a higher methodological quality than those with low scores.

Results

Twenty-two systematic reviews of healthcare interventions were retrieved (Fig. 1). AMSTAR scores ranged from 0 to 10 (max score 11) mean 3.77 and median 2.5, only 9 reviews scored > 4/11. Most frequent ‘yes’ scores were: Was the status of publication used as an inclusion criterion (12/22)? The most frequent ‘no’ scores were: Was there duplicate study selection and data extraction (17/22)? The low yield of reviews retrieved in the searches precluded any comparisons of scores between individual journals and the year of publication of the reviews. The average duration to complete the test was 15 minutes and no specific difficulties were encountered while applying the test.

Fig. 1 Search strategy for AMSTAR Assessement of systematic reviews in Indian journals.

The systematic reviews published in the Indian journals varied widely in quality as assessed by AMSTAR. This tool illustrates some common trends in the conduct of systematic reviews published in these journals although these do not appear to be uniform across all of the journals that were considered. There were several reviews which did not achieve a Yes score and some managed only one Yes score. The standard practice of assuming acceptable quality of methodology if the AMSTAR score is equal to or more than 4/11 may suggest that several of these published reviews are methodologically unsound [6]. However, it would be logical to expect the presence of all the elements to be able to make a good review, as all of them are equally important.

Contributors: All the authors were involved in conceptualizing, designing, evaluating and writing this article.

Funding: None.

Competing interests: None stated.

What this Study Adds?

• Very few of the systematic reviews identified in this study were found to be of optimal methodological quality as assessed by AMSTAR.

References

1. Systematic Review definition in HTA 101 : Glossary; National Information Center on Health Services Research and Health Care Technology (NICHSR). Available at URL www.nlm.nih.gov/nichsr/hta101/ta101014.html. Accessed on 3 February, 2010.

2. Lau J, Ioannidis JPA, Schmid CH. Summing up evidence: one answer is not always enough. BMJ 1995;310:1085-6.

3. Moher D, Soeken K, Sampson M, Campbell K, Ben Perot L, Berman B. Assessing the quality of reports of systematic reviews in pediatric complementary and alternative medicine. BMC Pediatr. 2002:2;3.

4. Jadad A, Moher M, Browman G, Booker L, Sigouin C, Fuentes M. Systematic reviews and meta-analyses on treatment of asthma: critical evaluation. BMJ. 2000;320: 537-40.

5. Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z , et al. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS One. 2007:2: e1350.

6. Canadian Agency for Drugs and Technologies in Health (CADTH). Available on: www.rxforchange.ca. Accessed on 3 February, 2010.

 

Copyright© 1999 by the Indian Pediatrics (Disclaimer)