Home            Past Issues            About IP            About IAP           Author Information            Subscription            Advertisement              Search  

   
research paper

Indian Pediatr 2015;52: 212-216

Diagnostic Accuracy of Indian Scale for Assessment of Autism (ISAA) in
Children Aged 2-9 Years


Sharmila Banerjee Mukherjee, Manoj Kumar Malhotra, Satinder Aneja,
*Satabdi Chakraborty and
#Smita Deshpande

From Department of Pediatrics, LHMC and Associated Hospitals, New Delhi; *Department of Psychiatry, Pt. BD Sharma PGIMS, Rohtak, Haryana; and #Department of Psychiatry, PGIMER – Dr. Ram Manohar Lohia Hospital, New Delhi, India.

Correspondence to: Dr Sharmila B Mukherjee, Department of Pediatrics, Kalawati Saran Children’s Hospital, Bangla Sahib Road, New Delhi 110 001, India.
Email: theshormi@gmail.com

Received: July 07, 2014;
Initial review: September 01, 2014;
Accepted: January 09, 2015.



Objective
: To determine the diagnostic accuracy of Indian Scale for Assessment of Autism (ISAA) in children aged 2-9 year at high risk of autism, and to ascertain the level of agreement with Childhood Autism Rating Scale (CARS).

Design: Diagnostic Accuracy study

Setting: Tertiary-level hospital.

Participants: Children aged between 2 and 9 year and considered to be at a high risk for autism (delayed development, and age-inappropriate cognition, speech, social interaction, behavior or play) were recruited. Those with diagnosed Hearing impairment, Cerebral palsy, Attention deficit hyperactivity disorder or Pervasive developmental disorders (PDD) were excluded.

Methods: Eligible children underwent a comprehensive assessment by an expert. The study group comprising of PDD, Global developmental delay (GDD) or Intellectual disability was administered ISAA by an investigator after one week. Both evaluators were blinded. ISAA results were compared to the Expert’s diagnosis and CARS scores.

Results: Out of 102 eligible children, 90 formed the study group (63 males, mean age 4.5y). ISAA had a sensitivity 93.3, specificity of 97.4, positive and negative likelihood ratios 85.7 and 98.7 and positive and negative predictive values of 35.5 and 0.08, respectively. Reliability was good and validity sub-optimal (r low, in 4/6 domains). The optimal threshold point demarcating Autism from ‘No autism’ according to Receiver Operating Characteristic curve was ISAA score of 70. Level of agreement with CARS measured by Kappa coefficient was low (0.14).

Conclusions: The role of ISAA in 3-9 year old children at high risk for Autism is limited to identifying and certifying Autism at ISAA score of 70. It requires re-examination in 2-3 year olds.

Keywords: Autism spectrum disorder, Certification, Diagnosis, Pervasive Developmental disorders.


Autism spectrum disorder (ASD) is the most recent nomenclature for developmental disorders characterized by persistently impaired social interaction and communication, with stereotypic behavior [1]. These have previously been also referred to as Pervasive developmental disorders (PDD) or Autism [2]. Western literature reports the prevalence of PDD in children as 0.67-1.2% [3,4]. According to a multicentric Indian community study, it is 0.8 - 1.3% in 2- to 9-year-old children [5]. Early identification of Autism is invaluable as timely intervention is known to improve outcomes [6]. Current standard protocols of evaluation recommend satisfying diagnostic criteria of International Classification of Diseases (ICD) or Diagnostic and Statistical Manual of Mental Disorders (DSM), followed by qualitative assessment with internationally validated instruments [1,2,7,8]. These include Autism Diagnostic Observation Schedule-Generalized (ADOS-G), Autism Diagnostic Interview-Revised (ADI-R), and Childhood Autism Rating Scale (CARS) [9-11]. Following this protocol is challenging in India as differences between Eastern and Western expectations of behavior influence parental appreciation of symptoms, leading to cultural bias and affecting instrument psychometric properties [12]. CARS, which also rates severity, is the only tool validated in the Indian population [13]. ADOS-G and ADI-R use is additionally limited by cost, and mandatory international accreditation.

An ideal Indian diagnostic tool for Autism requires accounting for variable literacy levels and heterogeneous culture and languages. It needs to be inexpensive, accurate, valid, reliable and easy to administer. It should also be able to fulfill multiple purposes; clinical (diagnosis, grading severity, planning intervention and monitoring), research and certification. The Indian Scale for Assessment of Autism (ISAA) was jointly developed by the National Trust, Ministry of Health and Family Welfare, and Ministry of Social Justice and Empowerment of the Government of India [14]. Its envisioned purpose was to establish diagnosis, and to rate severity (that was converted to extent of disability), so that it enabled certification and availing of benefits from ‘Welfare of Persons with Autism, Cerebral Palsy, Mental Retardation and Multiple Disabilities Act’ [15].

ISAA was validated in a multi-centric study involving 1124 participants aged 3-22 year with already diagnosed PDD, Intellectual Disability (ID), other disabilities, and normal intellect, who belonged predominantly to higher socio-economic strata with higher literacy levels [16]. Since manifestations are affected by effect of intervention, developmental age and chronological age, its ability to diagnose children, especially younger ones was questioned due to their underrepresentation. The present study was done to determine the diagnostic accuracy of ISAA in children aged 2-9 year, and measure the level of agreement with CARS.

Methods

This hospital-based study was conducted in the Pediatric Developmental Centre of a Medical College in Northern India from December 2011 to March 2013, after obtaining institutional Ethical Committee approval. Children between 2-9 years considered to be at high risk for Autism were consecutively recruited. These included children with parental concern regarding any one or more of the following: developmental delay, age-inappropriate cognition, speech delay and inappropriate social interaction, behavior or play. The sample size calculated was 85, assuming sensitivity, specificity and power of 80% each, alpha error 0.05, precision ±10% at 95% confidence interval and attrition 10% (software-N Master 2.0, CMC Vellore). Those without accompanying primary care giver, isolated hearing impairment, Cerebral palsy, or already diagnosed PDD or Attention Deficit Hyperactivity Disorder (ADHD) were excluded. Informed consent was obtained from all eligible children.

After evaluation by Brainstem Evoked Response Audiometry, the evaluation for autism was scheduled on two days, one week apart. Comprehensive assessment (reference standard) was done on the first day by a Pediatric consultant (with ³8 years experience in developmental pediatrics). This comprised of a parental interview with observation and examination of the child. Developmental Profile (DP-II) was administered to estimate Developmental quotient (DQ) and derived Intelligence quotient (IQ), and Vineland Adaptive Behavior Scale (VABS II) for adaptive function and maladaptive behavior indices [17,18]. DSM IV diagnostic criteria for PDD were applied, and CARS (DSM III-based) for assessing severity (total scores of <30, 30-37 and >37 indicate No autism, Mild to moderate autism and Severe autism, respectively) [2,11]. The study population was consecutively selected based on a standard diagnostic algorithm (Fig. 1). Children were categorized as (i) Global Developmental Delay (GDD)- younger than 5 years with DQ <70, not fulfilling DSM IV criteria for PDD (ii) Intellectual Disability (ID) with IQ <70 and Low adaptive levels (³2 SD of norms), not fulfilling DSM IV criteria for PDD (iii) PDD – fulfilling DSM IV criteria for PDD with or without GDD/ ID and (iv) Others- other diagnoses.

Fig. 1 Algorithm depicting evaluation and characterization of study subjects.

On the next visit, ISAA (test instrument) was administered by a trained pediatric resident. Test-retest (within 3 months) and inter-rater reliability (by an ISAA expert) was determined in 10% and 33.3% patients, respectively. ISAA comprises of 40 items covering 6 domains; Social relationship and reciprocity, Emotional responsiveness, Speech-language and communication, Behavior patterns, Sensory aspects and Cognitive. Individual items are scored on a Likert scale based on history and interviewer observation. Autism is diagnosed when the total score is ³170. Severity is categorized as mild, moderate and severe Autism based on scores of 70-108, 109-153 and >153, respectively. Both evaluators were blinded to the results of the other’s evaluation. Counseling and further management was done based on the expert’s diagnosis.

Statistical analysis: SPSS software (version 19.0) was used. Sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratio, validity, and reliability were measured. Kappa coefficient and Receiver Operator Characteristic (ROC) were determined for level of agreement.

Results

The primary presenting symptoms of the 102 recruited children were age-inappropriate behavior (64.4%), developmental or cognitive delay (60% and 48.9%), speech delay (40%), age-inappropriate play (32.2%) and age-inappropriate social interaction (25.5%). Five refused participation and 7 were excluded (4 cerebral palsy, 2 neuro-degenerative disorders and 1 hearing impairment). The study group comprised of 90 children (63 males) with mean age 4.5 years. Age-wise distribution was 2-3 years (26.7%), 3-5 years (28.9 %) and 5-9 years (44.4%). Most were from the Middle/Lower Middle Socio-economic strata with parental literacy till higher secondary level [19]. Expert diagnoses were PDD (77, 85.5%), isolated GDD (3, 3.3%), isolated ID (5, 5.5%), and others that included 1 Dravet syndrome, 2 ADHD and 2 Behavior problems (5, 5.8%). CARS scores indicated No autism in 12 (13.3%), Mild to moderate autism in 16 (17.7%) and Severe autism in 62 (68.8%). Co-morbid GDD/ID were observed in 87% of the children with PDD; moderate cognitive impairment (DQ/IQ 35-50) more in children with Mild to moderate autism, and severe cognitive impairment (DQ/IQ 20-35) more in severe autism.

ISAA administration: The average administration time was 17.4 minutes. During administration, it became apparent that the content of a few items were unsuitable for the younger children. On assessment of construct validity it was noted that Pearson correlation coefficient (r) was acceptable (0.8-0.89) in only Social and Emotional domains with sub-optimal values (£0.5) in the other four. Test-retest and inter-rater reliability was 0.93-0.99 and 0.99, respectively. ISAA scores ³70 (diagnostic of autism) was seen in 76 (84.4%) children, with rating of severity 53.9% mild, 46 % moderate, and none with severe Autism. Psychometric parameters are presented in Table I. Level of agreement of ISAA with CARS was low (Kappa coefficient 0.14, minimal acceptable value ³0.4); however, the ROC curve (Fig. 2a) showed the best cut-off point at a score of 70 with 0.92 sensitivity and 0.97 specificity. The scatter diagram plotted between ISAA and CARS total scores showed maximal clustering around ISAA scores of 70-80 (Fig. 2b).

Table 1 Psychometric Properties of Indian Scale for Assessment of Autism (ISSA)
Age group Number Sensitivity Specificity PPV NPV PLR NLR
2-9 years 90 92.3 97.4 85.7 98.7 35.5 0.08
2-3 years 24 100 92.3 100 100 12.9 0
3-9 years 66 90 96.4 81.8 98.2 25.0 0.11
Key: NLR- Negative Likelihood Ratio, NPV-Negative Predictive Value, PLR-Positive Likelihood ratio, PPV- Positive Predictive Value.
 

Fig. 2 (a) ROC-curve of Indian Scale for Assessment of Autism (ISAA) in children aged 2-9 years; (b) Correlation between total scores obtained on ISAA and Childhood Autism Rating Scale (CARS).

 

Discussion

This hospital-based study was conducted to determine the diagnostic accuracy of ISAA in 2-9 year old children presenting with features considered to be at ‘high risk’ for autism, and to ascertain its level of agreement in rating severity with CARS. The present study differed from the original validation study by multiple aspects: younger participant age (study objective), smaller sample size (albeit statistically adequate), lower socio-economic and literacy levels (hospital patient profile), undisclosed diagnosis (eliminating respondent bias), administration by a pediatrician, and use of comprehensive assessment as the reference standard instead of only CARS. This approach is considered superior to the use of a single tool, as it qualitatively and holistically assesses the multiple facets of ASD [7]. A major limitation realized post-hoc was failure to incorporate age-stratification and purposive sampling during patient selection. This resulted in skewed participant profile; lesser 2-3 year olds and children with isolated GDD/ ID.

Most children (87%) with Autism were low functioning (co-existent GDD/ID), which is higher than international data (40-80%) but close to a previous Indian study (90%) [20]. Reasons for this may be explained by the aforementioned drawbacks of using International psychometric tools in Indian children, i.e. cultural bias and non-validation. The use of tools designed for children with GDD/ID to assess DQ/IQ results in variable data when applied in ASD, scoring is based on the ability to perform, without considering unwillingness (frequently seen in autism). Adaptive function is a better reflector of ability as it considers frequency and quality of performance [21]. Evaluation of diagnostic accuracy of a tool entails critical examination of validity (the extent to which a test measures what it is supposed to measure), accuracy (psychometric properties) and reliability (the degree to which a test consistently measures whatever it measures) [22,23]. Some items demonstrated over-lapping content, ambiguous phrasing (i.e. ‘unable to grasp pragmatics of communication’), and scoring of features considered developmentally normal in young children as deviant (i.e. ‘unable to maintain peer relationships’, ‘inconsistent attention and concentration’). Manifestations of ASD are age-dependent; positive symptoms (overt behaviors) are easily identified irrespective of age, and negative symptoms (absence of pro-social symptoms) more often missed in younger children due to non-recognition. Both require inclusion when a single tool is used for a wide age range. The sub-optimal construct validity of ISAA may be due to these shortcomings.

The original sensitivity and specificity of ISAA was reported as 94.3 and 92, respectively [16]. In this study accuracy of ISAA was found acceptable, albeit specificity was marginally lower. Although figuratively acceptable, these parameters need to be interpreted with caution in 2-3 year olds due to aforementioned item unsuitability and smaller sample size. ROC curves are used to assess inherent validity. An area-under-the-curve (AUC) value approaching 1 indicates superior performance. Despite the aforementioned fallacies, the optimal threshold (point of maximum correct classification) was still 70 (the point that demarcated ‘No autism’ from ‘autism’ in the validation study). This implies that this ability remains consistent even in 2-9 year olds. Further categorization of severity was found unsatisfactory, evident by poor agreement with CARS and absence of clustering around ISAA scores of >153, which had been expected since most children had severe autism. The accuracy of ISAA is comparable to the INCLEN Diagnostic Tool for ASD (INDT-ASD), another validated Indian instrument designed to identify ASD without grading severity or disability [24]. Whether INDT-ASD also displays similar drawbacks when used in pre-school children is uncertain as an age-wise data comparison is unavailable [24,25].

To conclude, despite its many advantages (indigenous, free, availability in regional languages and requiring minimal training) and acceptable psychometric properties, the role of ISAA in 3-9 year old children is limited to only identifying autism and certifying disability of at least 40%. This requires further examination in 2-3 year olds. It may not be possible to use ISAA for assessing severity.

Contributors: SBM, SA: conceived the concept of the study, and were the neuro-developmental experts of the study; they will stand as guarantors; SBM: designed the study, and acquired clinical data related to comprehensive assessment of study subjects; MKM: collected ISAA related data of study subjects; SD, SC: trained MKM in administration of ISAA; SC: also helped in collection of ISAA related data; SBM, MKM: did the literature search and drafted the manuscript with important inputs from SA, SD and SC. All authors approved the final manuscript.

Funding: None; Competing Interest: None stated.


What Is Already Known?

ISAA is reported to be an accurate, valid and reliable Indian tool for diagnosing Autism and grading severity and disability among persons aged 3-22 year.

What This Study Adds?

ISAA is psychometrically acceptable and reliable but has sub-optimal validity in 3-9 year-old children.

ISAA can identify autism at a cut-off score of ³70 and thus certify disability of ³40% in 3-9 year-old children.


References

1. American Psychiatric Association. Autistic Spectrum Disorders. In: Diagnostic and Statistical Manual of Mental disorders. 5th ed. Arlington VA: American Psychiatric Association; 2013.p.50-9.

2. American Psychiatric Association. Autistic Disorder. In: Diagnostic and Statistical Manual of Mental Disorders. 4rth ed. Washington DC: American Psychiatric Association; 1994.p.70-5.

3. Centers for Disease Control. Prevalence of autism spectrum disorders-autism and developmental disabilities monitoring network, 14 sites, United States, 2008. Morbid Mortal Wkly Rep. 2007;56:1-28.

4. Raviola G, Gosselin GJ, Walter HJ, DeMaso DR. Pervasive Developmental Disorders and Childhood Psychosis. In: Kliegman RM, Stanton BF, St. Gene III JW, Schor NF, Behrman RE, editors. Nelson Textbook of Pediatrics. 19th ed. Philadelphia: Saunders, Elsevier; 2012.p.100-7.

5. Deshmukh VB, Mohapatra A, Gulati S, Nair M, Bhutani VK, Silberberg DH, et al. Prevalence of neuro-developmental disorders in India. Proceedings of International Meeting for Autism Research; 2013 May 4; Kursaal Centre, Donostia, San Sebastián, Spain. Available from: https//imfar.confex/imfar/2013/webprogram. Accessed April 5, 2014.

6. Chawarska K, Volkmar FR. Autism in Infancy and Early Childhood. In: Volkmar FR, Paul R, Klin A, Cohen D, editors. Handbook of Autism and Pervasive Developmental Disorders. 3rd ed. New Jersey: John Wiley and Sons, Inc; 2005.p.223-46.

7. Klin A, Saulnier C, Tsatanis K, Volknar FR. Clinical Evaluation in Autism Spectrum Disorders: Psychological Assessment with a Transdisciplinary Framework. In: Volkmar FR, Paul R, Klin A, Cohen D, editors. Handbook of Autism and Pervasive Developmental Disorders. 3rd ed. New Jersey: John Wiley and Sons, Inc; 2005.p.772-98.

8. World Health Organization. The International Classification of Diseases 10. Classification of Mental and Behavioral Disorders: Clinical Descriptions and Diagnostic Guidelines. Geneva: World Health Organization;1992.

9. Lord C, Risi S, Cook EH, Dilavore PC. The Autism Diagnostic Observation Schedule-Generic: a standard measure of social and communication deficits associated with the spectrum of autism. J Autism Dev Dis. 2000;24:659-85.

10. Lord C, Rutter M, LeCouteur A. The Autism Diagnostic Interview-Revised: a revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. J Autism Dev Dis. 1994;24:659-85.

11. Schopler E, Reichler RJ, Renner BR. The Childhood Autism Rating Scale (CARS). Los Angeles: Western Psychological Services, Inc;1988.

12. Freeth M, Sheppard E, Ramachandran R, Milne E. A cross-cultural comparison of autistic traits in the UK, India and Malaysia. J Autism Dev Disord. 2013;43:2569-83.

13. Russell PSS, Daniel A, Russell S, Mammen P, Abel JS, Raj LE, et al. Diagnostic accuracy, reliability and validity of Childhood Autism Rating Scale in India. World J Pediatr 2010;6:141-7.

14. Ministry of Social Justice and Empowerment. Scientific Report on Research Project for Development of Indian Scale for Assessment of Autism. New Delhi: Government of India; 2009.

15. Ministry of Social Justice and Empowerment. The National Trust for Welfare of Persons with Autism, Cerebral Palsy, Mental Retardation and Multiple Disability, The National Trust Regulations, 2001. Available from: www.social justice.nic.in/ntregu2001.php. Accessed June 23, 2014.

16. National Trust Web based Intervention Resource Centre. Indian Scale for Assessment of Autism. Available from: www.nationaltrust.co.in. Accessed May 6, 2012.

17. Alpern GD, Boll TJ, Shearer M. Developmental Profile II. Los Angeles: Western Psychological Services; 1989.

18. Sparrow SS, Balla DA, Cicchetti DV. Vineland Adaptive Behavior Scales. Circles Pines: AGS Publishing; 2005.

19. Kumar R, Dudala SR, Rao AR. Kuppuswamy’s socio-economic status scale- a revision of economic parameter for 2012. IJRDH 2013;1:2-4.

20. Juneja M, Mukherjee SB, Sharma S. A descriptive hospital based study of children with autism. Indian Pediatr. 2005;42:453-8.

21. Sugar M, Konstantareas M, Rampton G. The adaptive profiles of individuals with Autism spectrum disorders. J Dev Dis. 2010;16:72-6.

22. Greenhalgh T. How to read a paper: papers that report diagnostic or screening tests. BMJ. 1997;315:540-3.

23. STARD Steering Committee. Standards for the Reporting of Diagnostic Accuracy Studies Statement. Available from: www.stard-statement.org. Accessed August 3, 2012.

24. Juneja M, Mishra D, Russell P, Gulati S, Deshmukh V, Tudu P, et al. INCLEN diagnostic tool for Autism Spectrum Disorder (INDT-ASD): Development and validation. Indian Pediatr. 2014;51:359-65.

25. Wong CM, Singhal S. INDT-ASD: An autism diagnosis tool for Indian children. Indian Pediatr. 2014;51:355-6.   

 

Copyright © 1999-2015 Indian Pediatrics