Pediatric Appropriate Evaluation Protocol for India (PAEP-India): Tool for Assessing Appropriateness of Pediatric Hospitalization

research paper

Indian Pediatr 2018;55: 1041-1045

Pediatric Appropriate Evaluation Protocol for India (PAEP-India): Tool for Assessing Appropriateness of Pediatric Hospitalization

Manoja Kumar Das¹, Narendra Kumar Arora¹, Ramesh Poluru¹, Anju Seth², Anju Aggarwal³,
Anand Prakash Dubey⁴, PC Goyal⁵, Geeta Gathwala⁶, Ashraf Malik⁷, Anil Kumar Goel⁸,
Aparna Chakravarty⁹, Sugandha Arya¹⁰, Amit Upadhyay¹¹, Madhur Gupta¹², Thomas Mathew¹³, Rajamohanan K Pillai¹⁴, John Mathai¹⁵, Sivamani Manivasagan¹⁵, S Ramesh¹⁵, Mahesh Kumar Aggarwal¹⁶, Chsirtine G Maure¹⁷ and Patrick LF Zuber¹⁷

From ¹The INCLEN Trust International, Okhla Industrial Area, Phase I, New Delhi; Departments of Pediatrics; ²Lady Hardinge Medical College, New Delhi; ³University College of Medical Sciences, New Delhi; ⁴Maulana Azad Medical College, New Delhi; ⁵North DMC Medical College and Hindu Rao Hospital, New Delhi; ⁶Pt BD Sharma Postgraduate Institute of Medical Sciences, Rohtak, Haryana; ⁷Jawahar Lal Nehru Medical College, Aligarh Muslim University, Aligarh, UP; ⁸All India Institute of Medical Sciences, Raipur, Chhattisgarh; ⁹Hamdard Institute of Medical Sciences and Research, Jamia Hamdard, New Delhi; ¹⁰Vardhman Mahavir Medical College and Safdarjang Hospital, New Delhi; ¹¹LLRM Medical College Meerut, Uttar Pradesh; ¹²WHO Country office India; ¹³Community Medicine, Government Medical College, Thiruvananthapuram, Kerala; ¹⁴Government Medical College, Thiruvananthapuram, Kerala; ¹⁵PSG Institute of Medical Sciences and Research, Coimbatore, Tamil Nadu; ¹⁶Ministry of Health and Family Welfare, Government of India, New Delhi; and ¹⁷World Health Organization, Geneva, Switzerland.

Correspondence to: Dr Narendra Kumar Arora, Executive Director, The INCLEN Trust International, F1/5, Okhla Industrial Area, Phase 1, New Delhi 110 020, India.
Email: [email protected]

Received: November 22, 2017;
Initial review: March 05, 2018;
Accepted: September 27, 2018.

Objectives: To develop and assess Pediatric Appropriateness Evaluation Protocol for India (PAEP-India) for inter-rater reliability and appropriateness of hospitalization.

Design: Cross-sectional study.

Setting: The available PAEP tools were reviewed and adapted for Indian context by ten experienced pediatricians following semi-Delphi process. Two PAEP-India tools; newborn (£28 days) and children (>28 days-18 years) were developed. These PAEP-India tools were applied to cases to assess appropriateness of admission and inter-rater reliability between assessors.

Participants: Two sets of case records were used: (i) 274 cases from five medical colleges in Delhi-NCR [£28 days (n=51); >28 days to 18 years (n=223)]; (ii) 622 infants who were hospitalized in 146 health facilities and were part of a cohort (n= 30688) from two southern Indian states.

Interventions: Each case-record was evaluated by two pediatricians in a blinded manner using the appropriate PAEP-India tools, and ‘admission criteria’ were categorized as appropriate, inappropriate or indeterminate.

Main outcome measures: The proportion of appropriate hospitalizations and inter-rater reliability between assessors (using kappa statistic) were estimated for the cases.

Results: 97.8% hospitalized cases from medical colleges were labelled as appropriate by both reviewers with inter-rater agreement of 98.9% (k=0.66). In the southerm Indian set of infants, both reviewers labelled 80.5% admissions as appropriate with inter-rater agreement of 96.1% (k= 0.89).

Conclusions: PAEP-India (newborn and child) tools are simple, objective and applicable in diverse settings and highly reliable. These tools can potentially be used for deciding admission appropriateness and hospital stay and may be evaluated later for usefulness for cost reimbursements for insurance proposes.

Keywords: Bed use, Cost, Hospital stay, In-patient, Utilization.

Efficient and rationale allocation of health resources in India and other developing countries is essential. The recent National Health Policy of India and Ayushman Bharat Yojana aim at healthcare universalization and improving both out-patient and in-patient accessibility [1,2]. Hospitalizations consume a major proportion of healthcare resources in India and 50.9% of the resources are utilized for secondary (34.8%) and tertiary (16.1%) level services [3]. There is severe shortage of in-patient beds in India (0.7 beds/1000 population vs. world average of 3.96) with long waiting period for hospitalization particularly at government hospitals [4]. Standardizing admission and discharge processes has improved utilization, patient-flow and waiting time in several countries, and appropriateness evaluation protocols (AEP) are in wide use [5-7]. At present no Pediatric Appropriateness Evaluation Protocol (PAEP) tool is available for use in India.

We describe the development and pilot-testing of the PAEP-India tool to determine the inter-rater reliability for appropriateness of two sets of hospital admissions: (i) 274 in-patients from five medical colleges from Delhi and surrounding states and (ii) 622 hospitalized infants from a southern Indian cohort.

Methods

Development of PAEP Tool

The tool development process used semi-Delphi technique [8]. The PAEP tool adaptation for India (PAEP-India) process followed the steps: (i) an expert group of ten experienced pediatricians from ten medical colleges, each with >10 years of clinical experience was constituted considering the differences in symptomology, diseases and threshold for hospitalization, the experts felt need for developing separate tools for newborns (£28 days) and children (>28 days-18 years); (ii) review of the available literature was done and seven PAEPs (from 6 high- and 3 middle-income countries) were sourced [9-19]; (iii) two rounds of review and pilot testing of the tools was done by the experts; (iv) a face-to-face meeting was held for finalization of tools.

Each tool has two sections; ‘admission criteria’ to assess the appropriateness at admission and ‘day of care’ criteria for the appropriateness of hospitalization duration. For the PAEP- India (child) tool, the group made some amendments in both the admission criteria and day of care criteria for children (Box 1).

Box 1 Admission Criteria and Day of Care Criteria for Development of PAEP-India*(Child) Tool

• "severity of illness" section, "any fever for >48 hours when a diagnosis has not been reached" was revised to ">72 hours".

• Considering accidents, "burns/inhalational injury" and "exposure to poison and snake/scorpion bite" were added.

• Under the "severe electrolyte/acid base/hematological abnormality" section, "hypocal-caemia", "raised creatinine", "thrombocytopenia", "increased respiratory rate" were added, modified "total leukocyte count cut-off to <5000/mm3" and added "raised diastolic blood pressure" to hypertension.

• Under the "intensity of services" section, revised "nebulisation use at least every 4 hours".

• Day of care criteria: added "lack of suitable care taker availability (for abandoned child)/protected place" under "patient condition" section; and clarified "unstable vitals in last 48 hours" under the "within 48 hours of the day reviewed" section.

* PAEP-India: Pediatric Appropriateness Evaluation Protocol

The PAEP-India (newborn) tool was drafted with reference to the available management protocols for newborns [21-23]. The group did not suggest any change in the draft and finalized it. Finally, ‘admission criteria’ section comprised of 44 items in both the tools, and the ‘day of care’ for newborn and child tools comprised of 27 and 29 items, respectively (Web Annexure I and II).

The group of ten pediatricians who participated in the development of the PAEP-India tools were invited as raters. Five pairs of raters were made and each pair was assigned the same set of case sheets in a blinded manner; raters in a pair were not from the same institution and did not know the other member of their pair. All the raters underwent orientation to have common understanding of using the PAEP reviewers’ manual.

We had two sets of case-records drawn from different settings for inter-rater reliability assessment: 274 pediatric cases records drawn from five medical colleges located in Delhi and surrounding states (admitted during July-September 2015), and 622 cases-records from Kollam and Coimbatore, who were part of a cohort of 30688 infants recruited and followed-up in another study [24]. The case-records were anonymized and assigned unique study numbers.

The Ethics Committees of participating Institutes reviewed and approved the study. Waiver of informed consent was granted for data collection from the case records from five medical colleges. Informed consent was obtained for recruiting the infant cohort from Kollam and Coimbatore.

Statistical analysis: Inter-rater agreement was evaluated by the Cohen’s kappa (k) statistic [25]. Landis and Koch guidelines were adopted as benchmark scales of k coefficients (moderate: 0.41-0.60, substantial: 0.61-0.80, and almost perfect: 0.81-1.0) [26]. Statistical analyses was performed using STATA version 15.0 (StataCorp LLC, Texas, USA). Overall agreement was the proportion of judgements in which two raters agreed on categorizing as appropriate, inappropriate and indeterminate. We assessed the inter-rater reliability only for the ‘admission criteria’; the ‘day of care criteria’ section could not be evaluated as complete clinical, nursing and laboratory assessment records were not available for most of the days in most of the case sheets in both datasets.

Results

The age strata of 274 case-records from Delhi and surrounding states were: £28 days (n=51), >28 days-12 months (n=48), 13-59 months (n=67), and >5-18 years (n=108). There were 54 surgical cases (20.8%). Out of 622 cases from southern India cohort, 471 (75.7%) were from Kollam and 151 (24.3%) were from Coimbatore. The median age at entry and exit were 48 days (range 32-175 days) and 153 days (range: 67-562 days), respectively. There were equal proportion of boys (50.3%) and girls (49.7%) in the cohort (Table I). Out of 622 cases hospitalized to 146 hospitals, 50% (n=311) were admitted to tertiary (level 3), 49% (n=304) to secondary (level 2) and 1% (n=7) were admitted to primary care (level 1) care facilities respectively.

TABLE I	Clinical Diagnoses in the Two Data  Sets Used for Inter-rater Agreement for Appropriateness of 
Admission  Using PAEP-India Tools

Characteristic	Delhi and	Kollam and
	Surroundings	Coimbatore
	(n=274)	(n=622)
Children >28 d -18 y
Infections	122	524
Acute respiratory infections	23	427
Acute gastroenteritis	26	33
CNS infections	20
Urinary tract infections	3	19
Acute febrile illness^@	28	25
Other infections^#	22	20
Congenital diseases^€	4	26
Other systemic diseases	43	68
Seizure/CNS disorders	15	25
Other medical disorders*	28	43
Surgical conditions	54	4
Gastrointestinal	19	4^
Urological	12	0
Other surgical conditions	23	0
Neonates (<28 d)		_
Medical problems	42	_
Neonatal sepsis	24	_
Other medical disorders^$	18	_
Surgical conditions	9	_
^@Acute febrile illness including malaria, dengue, enteric fever, urinary tract infection, and other for evaluation; ^#Multiple system infection involves infection of more than one organ, may be with features of sepsis; ^€Congenital diseases including cardiac and central nervous system (CNS) malformations and other parts; *Other medical conditions including malignancy, coagulopathy, severe acute malnutrition, drug reaction, poisoning, constipation, nephrotic syndrome and hematemesis; ^All 4 children had intussusception; ^$Other medical conditions in neonates included preterm care, respiratory distress syndrome, hyperbilirubinemia, hypo/hyper-glycemia, and birth asphyxia.

Kappa (k) coefficient was 0.66 (95% CI 0.30, 1.0) for the overall dataset for hospitalized cases in medical college. Both raters categorized 97.8% admissions as appropriate. The observed inter-rater agreements were >98% for aggregated and the disaggregated data according to age and gender. The agreement for PAEP appropriate cases and inappropriate cases were >98% with values of 0.66 and 0.49, respectively (Table II).

TABLE II	Inter-rater Agreement for Appropriateness of Hospitalization Using PAEP-India Tools

Reviewer 1	Reviewer 2
	Appropriate	Inappropriate	Indeterminate
Medical colleges in Delhi and surrounding areas (n=274)
Appropriate	268	0	1
Inappropriate	2	1	0
Indeterminate	0	0	2
Hospitals in Kollam and Coimbatore (infants) (n=622)
Appropriate	480	13	0
Inappropriate	9	68	2
Indeterminate	0	0	50

The k for overall dataset was 0.89 (95% CI 0.84, 0.93) for the Kollam and Coimbatore infants. Overall both raters categorized 80.5% admissions as appropriate. The appropriate admissions in public and private hospitals were 84.5% and 78.5%, respectively. The observed agreement was >90% in most categories except for the level 1 hospitals (85.7%) which had just seven admissions.

Discussion

This is the first effort to develop a tool for assessing the appropriateness of pediatric admission and hospitalization duration in India. The PAEP-India tools performed well for admission appropriateness assessment, both for newborns and children and across different levels of hospitals.

In the absence of a gold standard and true valid measure of appropriateness, the consensual validity is reflected through inter-rater agreement. Studies on admission appropriateness using the PAEP tools in different countries have reported variable levels of both observed agreement and kappa statistic (0.29-0.89) [9-12,14,16,19]. With PAEP-India tools, observed agreements were uniformly high with both the datasets. Despite a high observed agreement, lower kappa values may be observed, when the marginal values are imbalanced. On the contrary, higher kappa value may be observed for asymmetrical imbalanced marginal totals. Kappa is affected by prevalence and may not be reliable for rare observations. Thus very low kappa values may not necessarily reflect low overall agreement. Decision on performance of a tool should also consider the observed versus expected agreement, consistency across contexts, and suitability of the criteria for specific settings besides kappa statistics [27,28].

The reported proportion of appropriate pediatric admissions reported in literature range between 68%-89.5% and 59.3%-98% in high- and middle-income countries, respectively [9-12,14-16,19]. The proportion of appropriate admissions in the present study was high, particularly in medical colleges, in view of the higher demand and pressure for admission. A previous study reported that one-third of the adult patients overstay in hospitals [29]; which was triangulated by the perceptions of 83% of resident doctors and 43% of nurses in another study [30].

Awareness of the raters about the source of cases in first dataset might have influenced the high expected and observed agreements (>90%). For the second dataset (southern Indian infant cohort), the raters were neither aware about the hospitals nor involved in the study implementation or patient care. We could not assess the appropriateness of the duration of hospitalization due to lack of necessary information at the time of discharge. In the first dataset, the experts involved in development of PAEP-India tools also applied the tools in cases drawn from some medical colleges where they worked. These factors might have increased observed agreement and categorization of cases as appropriate. The findings for the level one health facilities from southern India may not be generalized as the number of hospitalizations were too small.

In conclusion, the PAEP-India tools performed consistently in two different settings demonstrating consensual validity. The advantages of the PAEP-India tools is their simplicity, objectivity and applicability in different hospitals. Further application and evaluation of these tools is required in diverse settings across India including health facilities of all levels for triangulating the evidence of its utility. Meanwhile the PAEP-India tools have potential application in insurance systems, quality assessment processes, and resource-allocation.

Contributors: MKD, NKA: planning, tool development, data analysis, manuscript writing; RP: data analysis, manuscript review; AS, AA, APD, PCG, GG, AM, AKG, AC, SA, AU, TM, RKP, JM, SM, SR: tool development, data collection, manuscript review; MG, MKA, CGM, PLFZ: planning, technical inputs, manuscript review.

Funding: World Health Organization.

Competing Interest: None. The authors Madhur Gupta, Chsirtine G Maure and Patrick LF Zuber are staff-members at World Health Organization and had no role in implementation, data collection and analysis. The findings and conclusions in this report are those of the authors and do not necessarily represent the official positions of the World Health Organization.

What is Already Known?

• Country- and context-specific Pediatric Appropriateness Evaluation Protocol (PAEP) tools are in use for rational use of in-patient facilities.

What This Study Adds?

• India-specific PAEP tools for the assessment of appropriateness of pediatric and newborn hospitalizations were developed and tested for reliability with two sets of hospitalized children.

References

1. Ministry of Health and Family Welfare, Government of India. Ayushman Bharat Yojana (2018-2022). Government of India; 2018. Available from: https://www.pradhan mantriyojana.in/ayushman-bharat-yojana/. Accessed March 26, 2018.

2. Ministry of Health and Family Welfare, Government of India. National Health Policy, 2017. Government of India; 2017. Available from: https://mohfw.gov.in/sites/default/files/9147562941489753121.pdf. Accessed July 19, 2017.

3. National Health Systems Resource Centre. National Health Accounts Estimates for India (2013-14). Ministry of Health and Family Welfare Government of India; 2016. Available from:https://mohfw.gov.in/sites/default/files/89498311 221471416058.pdf. Accessed July 19, 2017.

4. Planning Commission (Government of India). Report of the Working Group on Tertiary Care Institutions for 12th Five Year Plan 20122017. Government of India; 2011. Available from: http://planningcommission.nic.in/aboutus/committee/wrkgrp12/health/WG_2tertiary.pdf. Accessed July 19, 2017.

5. Ortiga B, Salazar A, Jovell A, Escarrabill J, Marca G, Corbella X. Standardizing admission and discharge processes to improve patient flow: A cross sectional study. BMC Health Serv Res. 2012;12:180.

6. Strumwasser I, Paranjpe NV, Ronis DL, Share D, Sell LJ. Reliability and validity of utilization review criteria. Appropriateness evaluation protocol, standardized medreview instrument, and intensity-severity-discharge criteria. Med Care. 1990;28:95-111.

7. Kemper KJ, Fink HD, McCarthy PL. The reliability and validity of the pediatric appropriateness evaluation protocol. QRB Qual Rev Bull. 1989;15:77-80.

8. McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int J Clin Pharmacy. 2016;38:655-62.

9. Werneke U, Smith H, Smith IJ, Taylor J, MacFaul R. Validation of the paediatric appropriateness evaluation protocol in British practice. Arch Dis Child. 1997;77:294-8.

10. Esmail A. Development of the paediatric appropriateness evaluation protocol for use in the United Kingdom. J Public Health Med. 2000;22:224-30.

11. Esmaili A, Seyedin H, Faraji O, Arabloo J, Qahraman Bamdady Y, Shojaee S, et al. A pediatric appropriateness evaluation protocol for Iran children hospitals. Iran Red Crescent Med J. 2014;16:e16602.

12. Smith HE, Sheps S, Matheson DS. Assessing the utilization of in-patient facilities in a Canadian pediatric hospital. Pediatrics. 1993;92:587-93.

13. Gloor JE, Kissoon N, Joubert GI. Appropriateness of hospitalization in a Canadian pediatric hospital. Pediatrics. 1993;91:70-4.

14. Henley L, Smit M, Roux P, Zwarenstein M. Bed use in the medical wards of Red Cross War Memorial Children’s Hospital, Cape Town. S Afr Med J. 1991;80:487-90.

15. Shafik MH, Seoudi TMM, Raway TS, Al Harbash NZ, Ahmad MMA, Al Mutairi HF. Appropriateness of pediatric hospitalization in a general hospital in Kuwait. Med Princ Pract. 2012;21:516-21.

16. Kreger BE, Restuccia JD. Assessing the need to hospitalize children: Pediatric appropriateness evaluation protocol. Pediatrics. 1989;84:242-7.

17. Werneke U, MacFaul R. Evaluation of appropriateness of paediatric admission. Arch Dis Child. 1996;74:268-73.

18. Bianco A, Pileggi C, Trani F, Angelillo IF. Appropriateness of admissions and days of stay in pediatric wards of Italy. Pediatrics. 2003;112:124-8.

19. Formby DJ, McMullin ND, Danagher K, Oldham DR. The appropriateness evaluation protocol: application in an Australian children’s hospital. Aust Clin Rev. 1991;11:123-31.

20. Black RE, Morris SS, Bryce J. Where and why are 10 million children dying every year? Lancet. 2003;361: 2226-34.

21. Agarwal R, Deorari A, Paul VK. AIIMS Protocols in Neonatology. 1st ed. New Delhi: CBS Publishers; 2015.

22. National Neonatology Forum, India. Evidence based clinical practice guidelines. National Neonatology Forum; 2010. Available from: http://www.nnfi.org/. Accessed July 19, 2017.

23. Ministry of Health and Family Welfare, Government of India. Facility based new born care training packages. Government of India; 2014. Available from: http://nhm.gov.in/nrhm-components/rmnch-a/child-health-immunization/child-health/guidelines.html.　Accessed April 16, 2016.

24. World Health Organization. Global Advisory Committee on Vaccine Safety, 15-16 June 2016. Wkly Epidemiol Rec. 2016;91:341-8.

25. Cohen J. A Coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:37-46.

26. Landis JR, Koch GG. The measurement of observer agree-ment for categorical data. Biometrics. 1977;33:159-74.

27. Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43:543-9.

28. Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol. 1990;43:551-8.

29. Chakravarty A, Parmar N, Bhalwar R. Inappropriate use of hospital beds in a tertiary care service hospital. Med J Armed Forces India. 2005;61:121-4.

30. Kanwar V, Salaria N, Khanduja P. How do nurses perceive hospital bed utilization: A prerequisite for quality improvement in nursing care. Int J Health Sci Res. 2014;4135-41.