 |
|
Original Article Indian Pediatrics 2005; 42:1086-1091 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Evaluation of an Internet-Delivered Pediatric Diagnosis Support System (ISABEL®) in a Tertiary Care Center in India |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Sandeep B. Bavdekar and Mandar Pawar Department of Pediatrics, Seth GS Medical College & KEM Hospital, Parel, Mumbai 400 012, India.
Young medical graduates with clinical experience of 36 months or less, man the pediatric emergency rooms (ER) in public hospitals. They commonly face difficulties related to diagnosis and in such situations, these graduates approach their senior colleagues for opinion and advice. Although textbooks and online resources like the PubMed, the Cochrane database and clinical evidence series represent stockpiles of information and knowledge, they are of limited use to the attending doctor in the ER, as they do not provide patient-specific advice or help in prompt decision making that is required in the ER. ISABEL® (ISABEL Medical Charity, UK), a computerized differential diagnostic aid for pediatrics that is delivered through the Internet to the healthcare professionals, is a form of computerized clinical decision support system (CDSS)(1). The tool has undergone limited evaluation in the UK. At the time of conducting the study, it was available worldwide without any access fee. It was felt that ER doctors in resource-poor countries would use the tool increasingly. As the children in resource poor countries suffer from different ailments and as doctors in these countries receive different mode of training, we decided to evaluate the sensitivity of the ISABEL® diagnostic tool in a tertiary care hospital in India. Subjects and Methods Study design and conduct This study was carried out in the department of Pediatrics of an 1800-bedded tertiary care hospital after obtaining permission from the institution’s Ethics Committee. The department of Pediatrics has 90-bedded inpatient facility including a nine bedded pediatric intensive care unit (PICU). The hospital records of patients admitted to the Pediatric Intensive Care services from a pediatric unit in the 31-month period from January 2000 were retrieved. Resident doctors in the second or third-year of training and working in the capacity of Registrars wrote down the salient features (symptoms and clinical signs at presentation or results of emergency laboratory investigations carried out within 30 minutes of admission) on the basis admission notes in a Performa. They were blinded about the patient’s progress, outcome and results of investigations done during hospitalization and diagnosis at discharge or death. The registrars were told to use appropriate medical terminologies with correct spellings for describing salient features and to use only positive findings while filling up the Proforma. One investigator (MP) registered himself with the website www.isabel.org.uk. On accessing the tool, he provided information about the patient’s age (newborn, infant, child or adolescent) and then put in the termino-logies as provided by the registrars without changing any word or even spelling of a word. MP copied the list generated by the ISABEL® database on submission of the data. This process was followed for each and every subject record. After data of all the subjects was entered, the second investigator (SBB) accessed complete medical records. Wherever available, diagnosis written on the discharge summary, death certificate or autopsy report was considered as the final diagnosis. It was decided that if these were not available, the final diagnosis would be surmised on the basis of clinical data and results of investigations carried out during the hospital stay. The final diagnosis so obtained was deemed to be the "Gold Standard". Before entering the final diagnosis, SBB was blinded as to the list produced by ISABEL® for the patient. The investigators then compared the list of diseases and disorders generated for each subject by ISABEL® with the corresponding final diagnosis. The tool was considered to be successful if the final diagnosis was one of possibilities in the list generated by ISABEL®. Differential diagnostic tool ISABEL® (ISABEL Medical Charity, UK) is a computerized differential diagnostic aid for pediatrics that is delivered via the Internet and is powered by proprietary software called Autonomy. The engine works by matching patterns within the unformatted text with the text from standard pediatric textbooks. In other words, the clinician punches in the various manifestations noted in a patient in a textbox provided on the web page. The engine produces a list of 10-15 conditions by matching the text in the textbox after searching the standard pediatric texts that have been added to the ISABEL® database. During the period of this study, the textbooks that formed the ISABEL® database included Nelson’s textbook of Pediatrics (16th edition, 2000, WB Saunders), Forfar and Arneil’s textbook of Paediatrics (5th edition, 1998, Churchill Livingstone), Jones and Dragon Churchill’s pocket book of toxicology (2001, Churchill Livingstone) and Rennie and Roberton’s textbook of Neonatology (3rd edition, 1999, Churchill Livingstone)(2). Outcome Variables The presence of "final diagnosis" in the list provided by ISABEL® was the outcome measure studied. The tool was considered to be accurate for clinical diagnosis if the final diagnosis was one of possibilities in the list generated by ISABEL®. Sensitivity was calculated as the number of times the correct diagnosis was present in the list provided for each subject and was expressed as percentage. Results Records of 200 (111 boys and 89 girls, age range 28d-12years) subjects out of the 202 records retrieved were analysed. Records of two subjects were not included as these patients were discharged against medical advice before adequate diagnostic work-up could be undertaken. The age-wise distribution of subjects was as follows: 83 (41.5%) infants including 11 neonates, 65 (32.5%) children aged 1-5 years and 52 (26%) children aged 5-12 years. The final diagnosis was determined from the following sources: discharge summary 134 (67%), death certificate 44 (22%) and autopsy findings 22 (11%). As these documents were readily available, the option of inferring the final diagnosis on the basis of available clinical and laboratory data was never exercised. As shown in Table I, infectious diseases and disorders pertaining to cardiovascular system and nervous system predominated, together accounting for 161 (80.5%) subjects. In thirty-nine (19.5%) instances, the final diagnosis was absent from the list provided by ISABEL®. The final diagnoses more frequently missed by ISABEL® were septicemia, tuberculosis, including tuberculous meningitis, metabolic seizures, tetanus, renal failure and hepatic encephalopathy. Table I Sensitivity of ISABEL® in Diagnosing Various Conditions
The Table also depicts the ability of the tool to detect various individual and systemic diagnoses. The clinical accuracy was highest for malignancies, poisonings and bronchial asthma (100%). The tool had a sensitivity exceeding 80% for the three most common groups of disease encountered: infections and diseases of the cardiovascular and nervous systems. However, the sensitivity was low for conditions such as renal failure and a clutch of miscellaneous disorders. Discussion Relevant medical information is constantly needed for making clinical diagnoses and this information need has been mainly fulfilled through accessing textbooks. Advances in information technology and telecommunications have made it possible to share information through the worldwide network. Tools providing diagnostic aid use these technologies and ISABEL® is one such tool dedicated to pediatrics. Such tools have the potential to minimize medical error through avoiding misdiagnosis or missed diagnoses and help overcome difficulties that doctors face while extracting relevant information from medical literature, whether paper-based or in electronic format(3). The study represents the first attempt at determining the sensitivity of ISABEL® differential diagnostic tool in a patient population outside UK. The overall sensitivity of 80.5% demonstrated in the study is higher than that reported for computer-based diagnostic systems studied by Berner et al.(4) and Graber et al.(5) However, it is lower than the sensitivity of ISABEL® estimated in a UK-based study(2). The sensitivity varied widely amongst various disorders. Overall, the tool missed 27 diagnostic entities in 39 subjects. The prominent diagnostic entities missed were tuberculosis, septicemia, renal failure and anemia. The study indicates that the tool holds promise for use in resource-poor countries. However, the tool’s sensitivity needs to be improved to ensure that inexperienced doctors do not overlook critical diagnostic possibilities. This could probably be achieved by modifying the tool to take into consideration the geographical location of the patient. ISABEL® implemented this change while we were in the process of analyzing data. However, such a change could only improve the sensitivity pertaining to infectious diseases and might not have great impact in improving the overall sensitivity. ISABEL® should continue to encourage its users to report their experiences by providing hard data regarding its failure or sub-optimal results. Such a feedback mechanism will help the managers continually modify the tool with the aim of increasing its sensitivity. It is possible that the registrars did not use the correct terms. Nevertheless, the tool should be designed in such a manner that it is able to minimize the sensitivity-lowering effects of less than optimal inputs. This is vital since young doctors with different grades of abilities (pertaining to eliciting the history and performing clinical examination and choosing the right terms for submission to the database) would indeed be using it. At the same time, the institutions and consultants should train graduates so that they could extract maximal benefit from the diagnostic tool. In developing countries, physicians whose mother tongue is not English would use the tool. If wrong words or spellings were punched in, the result obtained would be sub-optimal and this could lower the sensitivity. It is easy to take care of such limitations The computer program could be altered so that it does not accept a wrongly spelt word and in addition offers suggestions of correct spellings and words. The study had its share of limitations: we retrieved the hospital records of subjects admitted to the PICU, which typically admits critically ill children requiring medical therapy. The rationale for choosing subjects from PICU was that it was necessary to ascertain sensitivity of the tool where it matters most, namely in critically ill children. This means the results of this study cannot be extrapolated to diagnosing children with non-critical illness. The PICU in our setting admits infants and children with medical illnesses. Hence, we have not been able to include children with surgically correctable disorders. We used hospital cases admitted in the past as subjects in the study, rather than carrying out the study on a prospective basis. Prospective enrolment would have resulted in the treating doctor spending his time with the computer when his patient required urgent attention. In addition, such time would have been spent when we were not sure if the tool provided reasonably acceptable results or if the doctors were sent on a "wild goose chase". Alternatively, we were not sure if withholding ISABEL®-generated list from caregivers would be ethically correct and operationally feasible. The methodology employed simulated the usual environment and also enabled the investigators to complete the study in a reasonable time frame. The resident doctors who provided the key words to be punched in were not given any special training. But, we thought that this would represent the usual situation in most institutions, where the young doctors would be expected to learn on their own and hone their skills for optimizing the yield from the website over time. Hence, the sensitivity noted in the study probably indicates the minimum sensitivity that could be expected under similar situations. Although the number of cases included is more than that studied in an earlier publication, the numbers in each diagnostic category are considerably small. The study, though noteworthy, represents only a preliminary assessment of the ISABEL® differential diagnostic tool. It indicates that the tool offers a reasonable sensitivity despite differences in patient population and user profile. More research needs to be done to define certain issues more clearly. It may be worthwhile to study if training resident doctors improves the clinical accuracy. It remains to be determined if the other diagnostic possibilities mentioned in the ISABEL® list are relevant or induce doctors to undertake unnecessary and expensive diagnostic evaluation. If the tool is determined to be appropriate for diagnostic purposes, its potential for teaching medical students could also be explored. Acknowledgements The authors thank Dr. Nilima Kshirsagar, Dean of Seth G.S. Medical College and K.E.M. Hospital for permitting them to publish this study. The authors are also grateful to Dr. Nithya Gogtay, Associate Professor of Clinical Pharmacology and Dr. Milind S. Tullu, Lecturer, Department of Pediatrics for offering constructive comments for improvement of the manuscript. Contributors: SBB helped in conception and design; collection of data and preparation of the first and final draft. MP was responsible for collection, analysis and interpretation of data and provided intellectual contribution in editing the first draft and preparation of the final draft. Funding: None. Competing Interests: None.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
References | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
