|
Indian Pediatr 2017;54: 863-866 |
|
Evaluation and
Validation of a Model for Identifying Serious Bacterial
Infections among Children Presenting to the Emergency
|
Source Citation:
Irwin AD, Grant A, Williams R, Kolamunnage-Dona R,
Drew RJ, Paulus S, et al. Predicting risk of serious bacterial
infections in febrile children in the emergency department. Pediatrics.
2017;140(2):e20162853 (Epub ahead of print)
Section Editor: Abhijeet Saha
|
Summary
In this diagnostic accuracy study of clinical and
biomarker variables in the diagnosis of serious bacterial infections
(SBIs), including pneumonia, in febrile children (age <16 y), a
diagnostic model was derived by using multinomial logistic regression
and internal validity. External validation of a published model was
undertaken, followed by model updating and extension by the inclusion of
procalcitonin and resistin. There were 1101 children studied, of whom
264 had a SBI. A diagnostic model discriminated well between pneumonia
and no SBI (concordance statistic 0.84, 95% CI 0.78–0.90) and between
other SBIs and no SBI (0.77, 95% CI 0.71–0.83) on internal validation. A
published model discriminated well on external validation. Model
updating yielded good calibration with good performance at both
high-risk (positive likelihood ratios 6.46 and 5.13 for pneumonia and
other SBI, respectively) and low-risk (negative likelihood ratios 0.16
and 0.13, respectively) thresholds. Extending the model with
procalcitonin and resistin yielded improvements in discrimination. The
authors concluded that diagnostic models discriminated well between
pneumonia, other SBIs, and no SBI in febrile children in the emergency
department.
Commentaries
Evidence-based Medicine Viewpoint
Relevance: Fever is one of the most common
presenting complaints among children; and many children present to the
Emergency, with fever accompanied by other symptoms/signs [1,2]. These
symptoms and signs assist in localizing a cause for the fever in many
cases; however, investigations are often required to confirm the
presence of a specific focus. The results of some of these
investigations are not immediately available; hence children are often
administered empiric antibiotic therapy, pending the availability of
reports. The presence of clinical and/or laboratory features that could
predict bacterial infection (versus viral or non-infectious
causes) could facilitate the rational use of antibiotics; with
widespread benefits to individual children, institutions, and the
community at large. Unfortunately, to date there are no reliable
clinical or laboratory markers [3,4]. Numerous investigators have tried
to devise diagnostic models (factoring in symptoms, clinical signs and
laboratory parameters) with variable success [5,6]. Irwin et al.
[7] recently published a paper wherein they attempted the following: (i)
to identify clinical and laboratory characteristics in febrile children
that could predict the presence of SBI; (ii) to derive a model
and validate it internally; (iii) to perform external validation
of an existing external model; and (iv) enhance the external
model with additional laboratory parameters. A brief outline of the
methodology adopted is presented in Table I.
Table I Outline of Methodology Adopted
Objective 1: Prediction of SBI among febrile children presenting
to the Emergency. |
|
Study design:
Diagnostic accuracy study |
|
Inclusion criteria:
Children (<16y) with documented fever or history of fever, if
clinical management warranted blood sampling (criteria not
mentioned). |
|
Exclusion criteria:
Children with primary immune deficiency. |
|
Sample size
calculation: 2300 participants were estimated to be required,
but data from 532 was used. |
|
Reference standard:
SBI was determined as pneumonia or other SBI (bacteremia,
urinary tract infection, meningitis, osteomyelitis, and septic
arthritis) using pre-defined criteria. A category of probable
SBI included children who were given antibiotics for >72h even
with negative culture (basis unclear). Those failing to meet the
criteria for SBI were labelled as “no SBI”. |
|
Index test(s):
Clinical and laboratory characteristics (unspecified) identified
from literature review. |
Objective 2: Internal validation of a predictive model developed
from Objective 1. |
|
Predictor factors
underwent univariate, followed by multivariate analysis by
logistic regression. Using a stepwise method, a predictive model
was developed. The man outcomes were pneumonia, other SBI and no
SBI. Internal validation was carried out in 569 children
enrolled in the study. |
Objective 3: External validation of a pre-existing predictive
model. |
|
A previous model [8]
was explored; and updated by “refitting variables, and
estimating individual coefficients” (details not given).
The validation process was undertaken in the entire cohort of
1101 children for the same outcomes viz. pneumonia, other SBI
and no SBI. |
Objective 4: Updating the external model with additional
parameters. |
|
The
external model was updated by adding procalcitonin and resistin
(chosen on the basis of findings in Objective 1). |
Critical appraisal: Owing to the multiplicity of
objectives, complexity of methods (and statistics), and opacity of
presentation, this is a difficult study for critical appraisal. However,
in general, a reasonable sample size and step-wise attempt to identify
the best model for prediction add value to the study. Considerable
attention has been paid for robust statistical methodology. However,
some issues stand out for attention.
The sample size calculation used previous estimates
of sensitivity and specificity; without explicitly stating what these
were for; and against what. Although a sample size of 2300 was deemed
adequate, the actual number enrolled was 1101; which was further split
into two (almost equal) groups for the derivation and internal
validation phases. Therefore it is unclear whether the study was
adequately powered.
In studies of diagnostic test accuracy, the usual
pattern is to identify the clinical condition using a reasonable
reference standard and compare the index test(s) against this. Here, a
set of definitions for SBI was evolved and used as the reference
standard. Clearly, this was not a fool-proof method as the protocol
included classification by two personnel with a third stepping in to
resolve disagreement. Unfortunately, no data are presented showing the
magnitude and scope of disagreement with the reference standard. The
authors mentioned that all children were followed-up for four weeks to
decrease mis-classification. Although no details are provided on
what/how this was done, it highlights the scope for error with the
reference standard used.
Careful analysis of the definitions of the terms in
the reference standard shows that ‘bacterial pneumonia’ was defined by
clinical symptoms/signs plus focal consolidation on radiography. This is
an inappropriate definition as focal consolidation has been reported in
conditions such as viral pneumonia, aspiration syndromes and underlying
airway malformations [9-12], all of which can present with symptoms and
signs suggesting ‘bacterial pneumonia.’ Traditionally bacterial
pneumonia is defined on the basis of culture of lung aspirate, or
pleural fluid, or blood (although it is highly insensitive) [13]. Of
course, bacterial pneumonia can also have chest X-rays not
showing consolidation [14,15].
Likewise, the definition of bacteremia included
bacteria detected by culture or PCR. In this regard, it is important to
note that recent data from the PERCH (Pneumonia Etiology Research for
Child Health) project reported a nearly comparable yield of Pneumococci
among pneumonia cases and non-pneumonia controls (7.3% and 5.5%
respectively); suggesting that blood PCR has poor specificity for
Pneumococcal bacteremia. In fact, 6.3% children without confirmed
bacterial infection were PCR positive. Further, less than two-thirds of
the culture positive cases were PCR positive, suggesting poor
sensitivity as well. These observations confirm that molecular methods
in blood may not be appropriate for confirming bacteremia [16]. The
study [7] also diagnosed urinary tract infection by appropriate
criteria, but included the unclear phrase "in a normally sterile urine
sample". These issues limit the confidence in the reference standards
used in this study.
These could explain why the list of diagnoses
presented in a Supplementary Figure is quite different from the
reference standard definitions. The former included categories such as
‘lower respiratory, upper respiratory, viral, gastrointestinal and
other’, as outcome diagnoses – all of which had cases with SBI. This
suggests potential for misclassification. There could also be potential
misunderstanding between the terms ‘severe’ and ‘serious’.
Most of the odds ratios (OR) comparing pneumonia or
other SBI versus no SBI, were borderline. In fact, the OR for
resist in and NGAL included 1.0 (suggesting that the effect was akin to
a coin toss). Even for CRP, the OR was only 1.02. However, the authors
reported that these parameters also were associated with SBI.
Extendibility: There are several issues
that limit the extendibility of the study data to our setting. First,
the report [7] itself shows that even within developed countries, there
is diversity of clinical diagnoses, and findings amongst children
presenting with fever. Adding to this diversity, the entirely different
set of differential diagnoses of acute fever in our setting,
encompassing infectious causes (such as enteric fever, dengue, malaria,
viral meningoencephaitis) and non-infectious causes (toxic
encephalopathy, poisoning etc.)’ makes it difficult to use the data from
this study. Routine vaccination against the common childhood bacterial
pathogens in developed countries also makes the spectrum quite
different. Further, it appears that only 1872 of 7949 (23.5%) children,
who presented with fever, required blood sampling. This is a very low
proportion compared to our setting.
Conclusion: This study showed that a combination
of clinical features and laboratory results could be developed into a
model to predict serious bacterial infection, in the setting where it
was developed. However, there are several methodological issues that
limit its application in routine practice.
Funding: None; Competing interests: None
stated.
Joseph L Mathew
Department of Pediatrics,
PGIMER, Chandigarh, India.
Email:
[email protected]
References
1. Barbi E, Marzuillo P, Neri E, Naviglio S, Krauss
BS. Fever in children: Pearls and Pitfalls. Children (Basel).
2017;4.pii: E81.
2. Burokienë S, Raistenskis J, Burokaitë E,
Èerkauskienë R, Usonis V. Factors determining parents’ decisions to
bring their children to the pediatric emergency department for a minor
illness. Med Sci Monit. 2017;23:4141-8.
3. SanaeiDashti A, Alizadeh S, Karimi A, Khalifeh M,
Shoja SA. Diagnostic value of lactate, procalcitonin, ferritin,
serum-C-reactive protein, and other biomarkers in bacterial and viral
meningitis: A cross-sectional study. Medicine (Baltimore).
2017;96:e7637.
4. Yusa T, Tateda K, Ohara A, Miyazaki S. New
possible biomarkers for diagnosis of infections and diagnostic
distinction between bacterial and viral infections in children. J Infect
Chemother. 2017; 23: 96-100.
5. De Santis O, Kilowoko M, Kyungu E, Sangu W,
Cherpillod P, Kaiser L, et al. Predictive value of clinical and
laboratory features for the main febrile diseases in children living in
Tanzania: A prospective observational study. PLoS One. 2017;12:e0173314.
6. Matha SM, Rahiman SN, Gelbart BG, Duke TD. The
utility of procalcitonin in the prediction of serious bacterial
infection in a tertiary paediatric intensive care unit. Anaesth
Intensive Care. 2016; 44: 607-14.
7. Irwin AD, Grant A, Williams R, Kolamunnage-Dona R,
Drew RJ, Paulus S, et al. Predicting risk of serious bacterial
infections in febrile children in the emergency department. Pediatrics.
2017;140. pii: e20162853.
8. Nijman RG, Vergouwe Y, Thompson M, van Veen M, van
Meurs AH, van der Lei J, et al. Clinical prediction model to aid
emergency doctors managing febrile children at risk of serious bacterial
infections: diagnostic study. BMJ. 2013;346:f1706.
9. Hasan K, Jolly P, Marquis G, Roy E, Podder G, Alam
K, et al. Viral etiology of pneumonia in a cohort of newborns
till 24 months of age in rural Mirzapur, Bangladesh. Scand J Infect Dis.
2006;38:690-5.
10. Guo WL, Wang J, Zhou M, Sheng M, Eltahir YM, Wei
J, et al. Chest imaging findings in children with influenza A
H1N1. Saudi Med J. 2011; 32: 50-4.
11. Kocakoc E, Kiris A, Sen Y, Bozgeyik Z. Pediatric
idiopathic pulmonary hemosiderosis diagnosed by sputum analysis: plain
radiography and computed tomography findings. Med Princ Pract.
2003;12:129-32.
12. Soylu H, Koltuksuz U, Kutlu NO, Sarihan H, Sen Y,
Ustün N, et al. Morgagni hernia: an unexpected cause of
respiratory complaints and a chest mass. Pediatr Pulmonol.
2000;30:429-33.
13. Mathew JL, Singhi S, Ray P, Hagel E,
Saghafian-Hedengren S, Bansal A, et al. Etiology of community
acquired pneumonia among children in India: prospective, cohort study. J
Glob Health. 2015; 5:050418.
14. Fancourt N, Deloria Knoll M, Baggett HC, Brooks
WA, Feikin DR, Hammitt LL, et al. Chest radiograph findings in
childhood pneumonia cases from the multisite PERCH Study. Clin Infect
Dis. 2017;64:S262-S270.
15. Shah SN, Bachur RG, Simel DL, Neuman MI. Does
This Child Have Pneumonia?: The rational clinical examination systematic
review. JAMA. 2017;318:462-71.
16. Morpeth SC, Knoll MD, Scott JAG, Park DE, Watson
NL, Baggett HC. Detection of pneumococcal DNA in blood by polymerase
chain reaction for diagnosing pneumococcal pneumonia in young children
from low- and middle-income countries. Clin Infect Dis. 2017;64 :
S347-56
Pediatric Emergency Medicine Physician’s Viewpoint
Nijman and colleagues describe an elegantly conducted
study on development of a clinical prediction model to aid emergency
clinicians in determining risk for serious bacterial infections (SBIs)
in febrile children (age 1 mo to 15 y). The strengths of the study
include: (a) a logical extension of their previous work, where
they had initially developed a model with 26 variables, which has been
trimmed down considerably and now includes clinical examination
variables (age, duration of fever, tachycardia, temperature, tachypnea,
ill appearance, chest wall retractions, prolonged capillary refill time
(>3 s), oxygen saturation (<94%)) and C-reactive protein; (b) a
well designed multi-center prospective study with robust analytic
methodology; and (c) adherence to the guiding principles of an
ideal clinical prediction rule i.e. derivation followed by
validation (independent sample from a different hospital) and broad
validation (emergency department from a different country).
Unfortunately, the clinical applicability of this prediction model is
very limited for various reasons. First, the term SBI has been
used variably by researchers and in most instances has been limited to
bacteremia, bacterial meningitis, urinary tract infections and
pneumonia, while the authors have included septicemia and various other
bacterial infections. Second, most studies have been limited to
otherwise well-appearing febrile children/infants (i.e. in
children who do not have an obvious source for fever) and present a
diagnostic challenge to emergency clinicians, while this study included
children with co-morbidities and those who had evidence of clinical
signs and symptoms that would potentially identify source of fever such
as tachypnea for pulmonary infections. Third, the age range is
extremely broad and it is inconceivable that a prediction model can be
applied across the entire spectrum of pediatric age where the etiology
and pathogenesis of bacterial infections varies considerably. Fourth,
the absence of urinalysis, a screening test with excellent performance
characteristics that identifies the most common bacterial infection is
surprising. Fifth, there is no mention of procalcitonin, a
screening test with better performance characteristics than C-reactive
protein, complete blood counts and absolute neutrophil counts.
Finally, the model performs better for pneumonia, while not as well
for other SBIs. In summary, a well-designed study with excellent
analytic approach, but very limited clinical utility due to an
unnecessarily broad definition of SBI, superior performance for only one
of the many SBI (pneumonia), and inclusion of wide age range where a
comprehensive clinical examination may be sufficient in aiding the
clinician for risk stratification and patient management.
Funding: None; Competing interests: None
stated.
Prashant Mahajan
Department of Emergency Medicine and Pediatrics,
CS Mott Children’s Hospital of Michigan, Ann Arbor, MI. USA
Email: [email protected]
|
|
|
|