Home            Past Issues            About IP            About IAP           Author Information            Subscription            Advertisement              Search  

   
journal club

Indian Pediatr 2017;54:863-866

Evaluation and Validation of a Model for Identifying Serious Bacterial Infections among Children Presenting to the Emergency

Source Citation: Irwin AD, Grant A, Williams R, Kolamunnage-Dona R, Drew RJ, Paulus S, et al. Predicting risk of serious bacterial infections in febrile children in the emergency department. Pediatrics. 2017;140(2):e20162853 (Epub ahead of print)

Section Editor: Abhijeet Saha

 

Summary

In this diagnostic accuracy study of clinical and biomarker variables in the diagnosis of serious bacterial infections (SBIs), including pneumonia, in febrile children (age <16 y), a diagnostic model was derived by using multinomial logistic regression and internal validity. External validation of a published model was undertaken, followed by model updating and extension by the inclusion of procalcitonin and resistin. There were 1101 children studied, of whom 264 had a SBI. A diagnostic model discriminated well between pneumonia and no SBI (concordance statistic 0.84, 95% CI 0.78–0.90) and between other SBIs and no SBI (0.77, 95% CI 0.71–0.83) on internal validation. A published model discriminated well on external validation. Model updating yielded good calibration with good performance at both high-risk (positive likelihood ratios 6.46 and 5.13 for pneumonia and other SBI, respectively) and low-risk (negative likelihood ratios 0.16 and 0.13, respectively) thresholds. Extending the model with procalcitonin and resistin yielded improvements in discrimination. The authors concluded that diagnostic models discriminated well between pneumonia, other SBIs, and no SBI in febrile children in the emergency department.

Commentaries

Evidence-based Medicine Viewpoint

Relevance: Fever is one of the most common presenting complaints among children; and many children present to the Emergency, with fever accompanied by other symptoms/signs [1,2]. These symptoms and signs assist in localizing a cause for the fever in many cases; however, investigations are often required to confirm the presence of a specific focus. The results of some of these investigations are not immediately available; hence children are often administered empiric antibiotic therapy, pending the availability of reports. The presence of clinical and/or laboratory features that could predict bacterial infection (versus viral or non-infectious causes) could facilitate the rational use of antibiotics; with widespread benefits to individual children, institutions, and the community at large. Unfortunately, to date there are no reliable clinical or laboratory markers [3,4]. Numerous investigators have tried to devise diagnostic models (factoring in symptoms, clinical signs and laboratory parameters) with variable success [5,6]. Irwin et al. [7] recently published a paper wherein they attempted the following: (i) to identify clinical and laboratory characteristics in febrile children that could predict the presence of SBI; (ii) to derive a model and validate it internally; (iii) to perform external validation of an existing external model; and (iv) enhance the external model with additional laboratory parameters. A brief outline of the methodology adopted is presented in Table I.

Table I Outline of Methodology Adopted
Objective 1: Prediction of SBI among febrile children presenting to the Emergency.
Study design: Diagnostic accuracy study
Inclusion criteria: Children (<16y) with documented fever or history of fever, if clinical management warranted blood sampling (criteria not mentioned).
Exclusion criteria: Children with primary immune deficiency.
Sample size calculation: 2300 participants were estimated to be required, but data from 532 was used.
Reference standard: SBI was determined as pneumonia or other SBI (bacteremia, urinary tract infection, meningitis, osteomyelitis, and septic arthritis) using pre-defined criteria. A category of probable SBI included children who were given antibiotics for >72h even with negative culture (basis unclear). Those failing to meet the criteria for SBI were labelled as “no SBI”.
Index test(s): Clinical and laboratory characteristics (unspecified) identified from literature review.
Objective 2: Internal validation of a predictive model developed from Objective 1.
Predictor factors underwent univariate, followed by multivariate analysis by logistic regression. Using a stepwise method, a predictive model was developed. The man outcomes were pneumonia, other SBI and no SBI. Internal validation was carried out in 569 children enrolled in the study.
Objective 3: External validation of a pre-existing predictive model.
A previous model [8] was explored; and updated by “refitting variables, and estimating individual coefficients” (details not given).  The validation process was undertaken in the entire cohort of 1101 children for the same outcomes viz. pneumonia, other SBI and no SBI.
Objective 4: Updating the external model with additional parameters.
The external model was updated by adding procalcitonin and resistin (chosen on the basis of findings in Objective 1).

Critical appraisal: Owing to the multiplicity of objectives, complexity of methods (and statistics), and opacity of presentation, this is a difficult study for critical appraisal. However, in general, a reasonable sample size and step-wise attempt to identify the best model for prediction add value to the study. Considerable attention has been paid for robust statistical methodology. However, some issues stand out for attention.

The sample size calculation used previous estimates of sensitivity and specificity; without explicitly stating what these were for; and against what. Although a sample size of 2300 was deemed adequate, the actual number enrolled was 1101; which was further split into two (almost equal) groups for the derivation and internal validation phases. Therefore it is unclear whether the study was adequately powered.

In studies of diagnostic test accuracy, the usual pattern is to identify the clinical condition using a reasonable reference standard and compare the index test(s) against this. Here, a set of definitions for SBI was evolved and used as the reference standard. Clearly, this was not a fool-proof method as the protocol included classification by two personnel with a third stepping in to resolve disagreement. Unfortunately, no data are presented showing the magnitude and scope of disagreement with the reference standard. The authors mentioned that all children were followed-up for four weeks to decrease mis-classification. Although no details are provided on what/how this was done, it highlights the scope for error with the reference standard used.

Careful analysis of the definitions of the terms in the reference standard shows that ‘bacterial pneumonia’ was defined by clinical symptoms/signs plus focal consolidation on radiography. This is an inappropriate definition as focal consolidation has been reported in conditions such as viral pneumonia, aspiration syndromes and underlying airway malformations [9-12], all of which can present with symptoms and signs suggesting ‘bacterial pneumonia.’ Traditionally bacterial pneumonia is defined on the basis of culture of lung aspirate, or pleural fluid, or blood (although it is highly insensitive) [13]. Of course, bacterial pneumonia can also have chest X-rays not showing consolidation [14,15].

Likewise, the definition of bacteremia included bacteria detected by culture or PCR. In this regard, it is important to note that recent data from the PERCH (Pneumonia Etiology Research for Child Health) project reported a nearly comparable yield of Pneumococci among pneumonia cases and non-pneumonia controls (7.3% and 5.5% respectively); suggesting that blood PCR has poor specificity for Pneumococcal bacteremia. In fact, 6.3% children without confirmed bacterial infection were PCR positive. Further, less than two-thirds of the culture positive cases were PCR positive, suggesting poor sensitivity as well. These observations confirm that molecular methods in blood may not be appropriate for confirming bacteremia [16]. The study [7] also diagnosed urinary tract infection by appropriate criteria, but included the unclear phrase "in a normally sterile urine sample". These issues limit the confidence in the reference standards used in this study.

These could explain why the list of diagnoses presented in a Supplementary Figure is quite different from the reference standard definitions. The former included categories such as ‘lower respiratory, upper respiratory, viral, gastrointestinal and other’, as outcome diagnoses – all of which had cases with SBI. This suggests potential for misclassification. There could also be potential misunderstanding between the terms ‘severe’ and ‘serious’.

Most of the odds ratios (OR) comparing pneumonia or other SBI versus no SBI, were borderline. In fact, the OR for resist in and NGAL included 1.0 (suggesting that the effect was akin to a coin toss). Even for CRP, the OR was only 1.02. However, the authors reported that these parameters also were associated with SBI.

Extendibility: There are several issues that limit the extendibility of the study data to our setting. First, the report [7] itself shows that even within developed countries, there is diversity of clinical diagnoses, and findings amongst children presenting with fever. Adding to this diversity, the entirely different set of differential diagnoses of acute fever in our setting, encompassing infectious causes (such as enteric fever, dengue, malaria, viral meningoencephaitis) and non-infectious causes (toxic encephalopathy, poisoning etc.)’ makes it difficult to use the data from this study. Routine vaccination against the common childhood bacterial pathogens in developed countries also makes the spectrum quite different. Further, it appears that only 1872 of 7949 (23.5%) children, who presented with fever, required blood sampling. This is a very low proportion compared to our setting.

Conclusion: This study showed that a combination of clinical features and laboratory results could be developed into a model to predict serious bacterial infection, in the setting where it was developed. However, there are several methodological issues that limit its application in routine practice.

Funding: None; Competing interests: None stated.

Joseph L Mathew

Department of Pediatrics,
PGIMER, Chandigarh, India.
Email: [email protected]

References

1. Barbi E, Marzuillo P, Neri E, Naviglio S, Krauss BS. Fever in children: Pearls and Pitfalls. Children (Basel). 2017;4.pii: E81.

2. Burokienë S, Raistenskis J, Burokaitë E, Èerkauskienë R, Usonis V. Factors determining parents’ decisions to bring their children to the pediatric emergency department for a minor illness. Med Sci Monit. 2017;23:4141-8.

3. SanaeiDashti A, Alizadeh S, Karimi A, Khalifeh M, Shoja SA. Diagnostic value of lactate, procalcitonin, ferritin, serum-C-reactive protein, and other biomarkers in bacterial and viral meningitis: A cross-sectional study. Medicine (Baltimore). 2017;96:e7637.

4. Yusa T, Tateda K, Ohara A, Miyazaki S. New possible biomarkers for diagnosis of infections and diagnostic distinction between bacterial and viral infections in children. J Infect Chemother. 2017; 23: 96-100.

5. De Santis O, Kilowoko M, Kyungu E, Sangu W, Cherpillod P, Kaiser L, et al. Predictive value of clinical and laboratory features for the main febrile diseases in children living in Tanzania: A prospective observational study. PLoS One. 2017;12:e0173314.

6. Matha SM, Rahiman SN, Gelbart BG, Duke TD. The utility of procalcitonin in the prediction of serious bacterial infection in a tertiary paediatric intensive care unit. Anaesth Intensive Care. 2016; 44: 607-14.

7. Irwin AD, Grant A, Williams R, Kolamunnage-Dona R, Drew RJ, Paulus S, et al. Predicting risk of serious bacterial infections in febrile children in the emergency department. Pediatrics. 2017;140. pii: e20162853.

8. Nijman RG, Vergouwe Y, Thompson M, van Veen M, van Meurs AH, van der Lei J, et al. Clinical prediction model to aid emergency doctors managing febrile children at risk of serious bacterial infections: diagnostic study. BMJ. 2013;346:f1706.

9. Hasan K, Jolly P, Marquis G, Roy E, Podder G, Alam K, et al. Viral etiology of pneumonia in a cohort of newborns till 24 months of age in rural Mirzapur, Bangladesh. Scand J Infect Dis. 2006;38:690-5.

10. Guo WL, Wang J, Zhou M, Sheng M, Eltahir YM, Wei J, et al. Chest imaging findings in children with influenza A H1N1. Saudi Med J. 2011; 32: 50-4.

11. Kocakoc E, Kiris A, Sen Y, Bozgeyik Z. Pediatric idiopathic pulmonary hemosiderosis diagnosed by sputum analysis: plain radiography and computed tomography findings. Med Princ Pract. 2003;12:129-32.

12. Soylu H, Koltuksuz U, Kutlu NO, Sarihan H, Sen Y, Ustün N, et al. Morgagni hernia: an unexpected cause of respiratory complaints and a chest mass. Pediatr Pulmonol. 2000;30:429-33.

13. Mathew JL, Singhi S, Ray P, Hagel E, Saghafian-Hedengren S, Bansal A, et al. Etiology of community acquired pneumonia among children in India: prospective, cohort study. J Glob Health. 2015; 5:050418.

14. Fancourt N, Deloria Knoll M, Baggett HC, Brooks WA, Feikin DR, Hammitt LL, et al. Chest radiograph findings in childhood pneumonia cases from the multisite PERCH Study. Clin Infect Dis. 2017;64:S262-S270.

15. Shah SN, Bachur RG, Simel DL, Neuman MI. Does This Child Have Pneumonia?: The rational clinical examination systematic review. JAMA. 2017;318:462-71.

16. Morpeth SC, Knoll MD, Scott JAG, Park DE, Watson NL, Baggett HC. Detection of pneumococcal DNA in blood by polymerase chain reaction for diagnosing pneumococcal pneumonia in young children from low- and middle-income countries. Clin Infect Dis. 2017;64 : S347-56

Pediatric Emergency Medicine Physician’s Viewpoint

Nijman and colleagues describe an elegantly conducted study on development of a clinical prediction model to aid emergency clinicians in determining risk for serious bacterial infections (SBIs) in febrile children (age 1 mo to 15 y). The strengths of the study include: (a) a logical extension of their previous work, where they had initially developed a model with 26 variables, which has been trimmed down considerably and now includes clinical examination variables (age, duration of fever, tachycardia, temperature, tachypnea, ill appearance, chest wall retractions, prolonged capillary refill time (>3 s), oxygen saturation (<94%)) and C-reactive protein; (b) a well designed multi-center prospective study with robust analytic methodology; and (c) adherence to the guiding principles of an ideal clinical prediction rule i.e. derivation followed by validation (independent sample from a different hospital) and broad validation (emergency department from a different country). Unfortunately, the clinical applicability of this prediction model is very limited for various reasons. First, the term SBI has been used variably by researchers and in most instances has been limited to bacteremia, bacterial meningitis, urinary tract infections and pneumonia, while the authors have included septicemia and various other bacterial infections. Second, most studies have been limited to otherwise well-appearing febrile children/infants (i.e. in children who do not have an obvious source for fever) and present a diagnostic challenge to emergency clinicians, while this study included children with co-morbidities and those who had evidence of clinical signs and symptoms that would potentially identify source of fever such as tachypnea for pulmonary infections. Third, the age range is extremely broad and it is inconceivable that a prediction model can be applied across the entire spectrum of pediatric age where the etiology and pathogenesis of bacterial infections varies considerably. Fourth, the absence of urinalysis, a screening test with excellent performance characteristics that identifies the most common bacterial infection is surprising. Fifth, there is no mention of procalcitonin, a screening test with better performance characteristics than C-reactive protein, complete blood counts and absolute neutrophil counts. Finally, the model performs better for pneumonia, while not as well for other SBIs. In summary, a well-designed study with excellent analytic approach, but very limited clinical utility due to an unnecessarily broad definition of SBI, superior performance for only one of the many SBI (pneumonia), and inclusion of wide age range where a comprehensive clinical examination may be sufficient in aiding the clinician for risk stratification and patient management.

Funding: None; Competing interests: None stated.

Prashant Mahajan

Department of Emergency Medicine and Pediatrics,
CS Mott Children’s Hospital of Michigan, Ann Arbor, MI. USA
Email: [email protected]


 

Copyright © 1999-2017 Indian Pediatricsfont>