|
Indian Pediatr 2011;48: 383-385 |
|
Quality Assessment of Systematic Reviews of
Health Care Interventions using AMSTAR |
Vanitha Jagannath, Joseph L Mathew, GV Asokan and Zbys
Fedorowicz
From UKCC (Bahrain Branch) and Ministry of Health,
Bahrain; Box 25438, Bahrain.
Correspondence to: Dr Vanitha A Jagannath, Specialist
Pediatrician, Department of Paediatrics,
KIMS Bahrain Medical Center, Um al Hassam Ave, Adliya, PO Box 175829
Manama,
Bahrain.
Email: [email protected]
Received: March 04, 2010;
Initial review: March 25, 2010;
Accepted: July 22, 2010.
Published online 2010 November 30.
PII: S097475591000187-2
|
Abstract
Appraisal of the methodological quality of systematic
reviews would reflect on their utility for the clinicians and
policymakers. This study was done to assess the quality of systematic
reviews published in five leading Indian medical journals using AMSTAR.
22 systematic reviews of healthcare interventions were identified. The
scores ranged 0 to 10 (mean 3.77 and median 2.5), 9 reviews scored >
4/11. Most frequent ‘yes’ and ‘no’ scores were: publication status as an
inclusion criterion (12 /22), respectively and duplicate study selection
and data extraction (17 /22). Several suboptimal aspects of
methodological quality were identified in the reviews evaluated.
Key words: AMSTAR, India, Medical journals, Systematic
reviews.
|
Healthcare professionals would need
to read a large number of original articles every day to keep track of the
latest information in their field; systematic reviews are one way in which
they can keep abreast with current medical literature. Systematic reviews
provide a logical synthesis of the research base via a carefully
formulated question and analysis of all available evidence. A
comprehensive search of the literature utilizing predetermined inclusion
and exclusion criteria is followed by a critical appraisal of the risk of
bias in any included studies, and it may also include a synthesis of data
for the outcomes of interest [1]. An increasing number of systematic
reviews are being published in a wide range of medical journals across the
spectrum of healthcare and the methodological quality of these reviews
varies widely [2]. In spite of the care with which they are conducted,
some may provide different answers to the same clinical question [3]. It
is imperative that systematic reviews are appraised for robustness of
methodological quality before being used for either healthcare policy or
clinical decision making [4]. There is an increased recognition that the
methodological quality and reporting quality of systematic reviews are two
distinctly different aspects to be considered in the appraisal of reviews.
Methodological quality represents how well the systematic review was
conducted (literature searching, pooling of data, etc). The reporting
quality, however considers how well systematic reviewers have reported
their methodology and findings. Currently there are more than 24
instruments available to appraise the methodological quality of systematic
reviews, but few have been developed systematically or validated
empirically. AMSTAR (a measurement tool to assess the methodological
quality of systematic reviews) was developed specifically to fulfill this
requirement [5].
We conducted this study to assess the quality of
systematic reviews published in five leading Indian medical journals using
AMSTAR instrument.
Methods
We identified 5 of the leading Indian journals in
MEDLINE by their citation ratings. MEDLINE was searched, using ‘systematic
review’ sub set limit, for citations to articles published in the 5
journals from January 2000 to November 2009. Assessment of the searches
against pre-specified inclusion criteria was conducted in duplicate by two
of the authors. Full text copies of all included studies were assessed
against AMSTAR and the scores were recorded. The time taken to complete
the assessment and difficulties encountered, if any, while administering
the test were also noted.
Box I Items in AMSTAR Tool to Assess
Methodological Quality of Systematic Review |
- Was an ‘a priori’ design provided?
- Was there duplicate study selection and data extraction?
- Was a comprehensive literature search performed?
- Was the status of publication (i.e. grey literature) used as
an inclusion criterion?
- Was a list of studies (included and excluded) provided?
- Were the characteristics of the included studies provided?
- Was the scientific quality of the included studies assessed
and documented?
- Was the scientific quality of the included studies used
appropriately in formulating conclusions?
- Were the methods used to combine the findings of studies
appropriate?
- Was the likelihood of publication bias assessed?
- Was the conflict of interest stated? Reviews that achieve high
scores indicate a higher methodological quality than those with
low scores.
|
Results
Twenty-two systematic reviews of healthcare
interventions were retrieved (Fig. 1). AMSTAR scores ranged
from 0 to 10 (max score 11) mean 3.77 and median 2.5, only 9 reviews
scored > 4/11. Most frequent ‘yes’ scores were: Was the status of
publication used as an inclusion criterion (12/22)? The most frequent
‘no’ scores were: Was there duplicate study selection and data
extraction (17/22)? The low yield of reviews retrieved in the searches
precluded any comparisons of scores between individual journals and the
year of publication of the reviews. The average duration to complete the
test was 15 minutes and no specific difficulties were encountered while
applying the test.
|
Fig. 1 Search strategy for AMSTAR
Assessement of systematic reviews in Indian journals. |
The systematic reviews published in the Indian journals
varied widely in quality as assessed by AMSTAR. This tool illustrates some
common trends in the conduct of systematic reviews published in these
journals although these do not appear to be uniform across all of the
journals that were considered. There were several reviews which did not
achieve a Yes score and some managed only one Yes score. The standard
practice of assuming acceptable quality of methodology if the AMSTAR score
is equal to or more than 4/11 may suggest that several of these published
reviews are methodologically unsound [6]. However, it would be logical to
expect the presence of all the elements to be able to make a good review,
as all of them are equally important.
Contributors: All the authors were involved in
conceptualizing, designing, evaluating and writing this article.
Funding: None.
Competing interests: None stated.
What this Study Adds?
• Very few of the systematic reviews identified in this study
were found to be of optimal methodological quality as assessed by
AMSTAR. |
References
1. Systematic Review definition in HTA 101 : Glossary;
National Information Center on Health Services Research and Health Care
Technology (NICHSR). Available at URL www.nlm.nih.gov/nichsr/hta101/ta101014.html.
Accessed on 3 February, 2010.
2. Lau J, Ioannidis JPA, Schmid CH. Summing up
evidence: one answer is not always enough. BMJ 1995;310:1085-6.
3. Moher D, Soeken K, Sampson M, Campbell K, Ben Perot
L, Berman B. Assessing the quality of reports of systematic reviews in
pediatric complementary and alternative medicine. BMC Pediatr. 2002:2;3.
4. Jadad A, Moher M, Browman G, Booker L, Sigouin C,
Fuentes M. Systematic reviews and meta-analyses on treatment of asthma:
critical evaluation. BMJ. 2000;320: 537-40.
5. Shea BJ, Bouter LM, Peterson J, Boers M, Andersson
N, Ortiz Z , et al. External validation of a measurement tool to
assess systematic reviews (AMSTAR). PLoS One. 2007:2: e1350.
6. Canadian Agency for Drugs and Technologies in Health
(CADTH). Available on: www.rxforchange.ca. Accessed on 3 February,
2010.
|
|
|
|