Sensitivity and specificity are two components that measure the inherent validity of a diagnostic test for dichotomous outcomes against a gold standard. Receiver operating characteristic (ROC) curve is the plot that depicts the trade-off between the sensitivity and (1-specificity) across a series of cut-off points when the diagnostic test is continuous or on ordinal scale (minimum 5 categories). This is an effective method for assessing the performance of a diagnostic test. The aim of this article is to provide basic conceptual framework and interpretation of ROC analysis to help medical researchers to use it effectively. ROC curve and its important components like area under the curve, sensitivity at specified specificity and vice versa, and partial area under the curve are discussed. Various other issues such as choice between parametric and non-parametric methods, biases that affect the performance of a diagnostic test, sample size for estimating the sensitivity, specificity, and area under ROC curve, and details of commonly used softwares in ROC analysis are also presented.

Key words: Sensitivity, Specificity, Receiver operating characteristic curve, Sample size, Optimal cut-off point, Partial area under the curve.

Diagnostic tests play a vital role in modern medicine not only for confirming the presence of disease but also to rule out the disease in individual patient. Diagnostic tests with two outcome categories such as a positive test (+) and negative test (–) are known as dichotomous, whereas those with more than two categories such as positive, indeterminate and negative are called polytomous tests. The validity of a dichotomous test compared with the gold standard is determined by sensitivity and specificity. These two are components that measure the inherent validity of a test.

Two popular indicators of inherent statistical validity of a medical test are the probabilities of detecting correct diagnosis by test among the true diseased subjects (D+) and true non-diseased subjects (D-). For dichotomous response, the results in terms of test positive (T+) or test negative (T-) can be summarized in a 2×2 contingency table (Table I). The columns represent the dichotomous categories of true diseased status and rows represent the test results. True status is assessed by gold standard. This standard may be another but more expensive diagnostic method or a combination of tests or may be available from the clinical follow-up, surgical verification, biopsy, autopsy, or by panel of experts. Sensitivity or true positive rate (TPR) is conditional probability of correctly identifying the diseased subjects by test: S_N = P(T+/D+) = TP/(TP + FN); and specificity or true negative rate (TNR) is conditional probability of correctly identifying the non-disease subjects by test: S_P= P(T-/D-) = TN/(TN + FP). False positive rate (FPR) and false negative rate (FNR) are the two other common terms, which are conditional probability of positive test in non-diseased subjects: P(T+/D-)= FP/(FP + TN); and conditional probability of negative test in diseased subjects: P(T-/D+)= FN/(TP + FN) respectively.

Table I 



Diagnostic Test Results in Relation to True Disease Status in a 2×2 Table 

Standard errors (SE) are needed to construct a confidence interval. Three methods have been suggested for estimating the SE of empirical area under ROC curve [7,9-10]. These have been found similar when sample size is greater than 30 in each group provided test value is on continuous scale [11]. For small sample size it is difficult to recommend any one method. For discrete ordinal outcome, Bamber method [9] and Delong method [10] give equally good results and better than Hanley and McNeil method [7].

These are used when the statistical distribution of diagnostic test values in diseased and non-diseased is known. Binormal distribution is commonly used for this purpose. This is applicable when test values in both diseased and non-diseased subjects follow normal distribution. If data are actually binormal or a transformation such as log, square or Box-Cox [12] makes the data binormal then the relevant parameters can be easily estimated by means and variances of test values in diseased and non-diseased subjects. Details are available elsewhere [13].

Another parametric approach is to transform the test results into an unknown monotone form when both the diseased and non-diseased populations follow binormal distribution [14]. This first discretizes the continuous data into a maximum 20 categories, then uses maximum likelihood method to estimate the parameters of the binormal distribution and calculates the AUC and standard error of AUC. ROCKIT package containing ROCFIT method uses this approach to draw the ROC curve, to estimate the AUC, for comparison between two tests, and to calculate partial area [15].

The choice of method to calculate AUC for continuous test values essentially depends upon availability of statistical software. Binormal method and ROCFIT method produce results similar to non-parametric method when distribution is binormal [16]. In unimodal skewed distribution situation, Box-Cox transformation that makes test value binormal and ROCFIT method perform give results similar to non-parametric method but former two approaches have additional useful property for providing smooth curve [16,17]. When software for both parametric and non-parametric methods is available, conclusion should be based on the method which yields greater precision to estimate the AUC. However, for bimodal distribution (having two peaks), which is rarely found in medical practice, Mann-Whitney gives more accurate estimates compare to parametric methods [16]. Parametric method gives small bias for discrete test value compared to non-parametric method [13].

The area under the curve by trapezoidal rule and Mann-Whitney U are 0.9142 and 0.9144, respectively, of mid-arm circumference for indicating low birth weight in our data. The SE also is nearly equal by three methods in these data: Delong SE = 0.0128, Bamber SE = 0.0128, and Hanley and McNeil SE = 0.0130. For parametric method, smooth ROC curve was obtained assuming binormal assumption (Fig 2) and the area under the curve is calculated by using means and standard deviations of mid-arm circumference in normal and low birth weight neonates which is 0.9427 and its SE is 0.0148 in this example. Binormal method showed higher area compared to area by non-parametric method which might be due to violation of binormal assumption in this case. Binormal ROC curve is initially above the empirical curve (Fig 2) suggesting higher sensitivity compared to empirical values in this range. When (1–specificity) lies between 0.1 to 0.2, the binormal curve is below the empirical curve, suggesting comparatively low sensitivity compared to empirical values. When values of (1–specificity) are greater than 0.2, the curves are almost overlapping suggesting both methods giving the similar sensitivity. The AUC by using ROCFIT methods is 0.9161 and it standard error is 0.0100. This AUC is similar to the non-parametric method; however standard error is little less compared to standard error by non-parametric method. The data in our example has unimodal skewed distribution and results agree with previous simulation study [16, 17] on such data. All calculations were done using MS Excel and STATA statistical software for this example.

Total area under ROC curve is a single index for measuring the performance a test. The larger the AUC, the better is overall performance of diagnostic test to correctly pick up diseased and non-diseased subjects. Equal AUCs of two tests represents similar overall performance of medical tests but this does not necessarily mean that both the curves are identical. They may cross each other. Three common interpretations of area under the ROC curve are: (i) the average value of sensitivity for all possible values of specificity, (ii) the average value of specificity for all possible values of sensitivity [13]; and (iii) the probability that a randomly selected patient with disease has positive test result that indicates greater suspicion than a randomly selected patient without disease [10] when higher values of the test are associated with disease and lower values are associated with non-disease. This interpretation is based on non-parametric Mann-Whitney U statistic for calculating the AUC.

Figure 3 depicts three different ROC curves. Considering the area under the curve, test A is better than both B and C, and the curve is closer to the perfect discrimination. Test B has good validity and test C has moderate.

Fig. 3 Comparison of three smooth ROC curves with different areas.

Hypothetical ROC curves of three diagnostic tests A, B, and C applied on the same subjects to classify the same disease are shown in Fig 4. Test B (AUC=0.686) and C (AUC=0.679) have nearly equal area but cross each other whereas test A (AUC=0.805) has higher AUC value than curves B and C. The overall performance of test A is better than test B as well as test C at all the threshold points. Test C performed better than test B where high sensitivity is required, and test B performed better than C when high specificity is needed.

Fig. 4 Three empirical ROC curves. Curves for B and C cross each other but have nearly equal areas, curve A has bigger area.

The choice of fixed specificity or range of specificity depends upon clinical setting. For example, to diagnose serious disease such as cancer in a high risk group, test that has higher sensitivity is preferred even if the false positive rate is high because test giving more false negative subjects is more dangerous. On the other hand, in screening a low risk group, high specificity is required for the diagnostic test for which subsequent confirmatory test is invasive and costly so that false positive rate should be low and patient does not unnecessarily suffers pain and pays price. The cut-off point should be decided accordingly. Sensitivity at fixed point of specificity or vice versa and partial area under the curve are more suitable in determining the validity of a diagnostic test in above mentioned clinical setting and also for the comparison of two diagnostic tests when applied to same or independent patients when ROC curves cross each other.

Partial area is defined as area between range of false positive rate (FPR) or between the two sensitivities. Both parametric (binormal assumption) and non-parametric methods are available in the literature [13,18] but most statistical softwares do not have option to calculate the partial area under ROC curve (Table III). STATA software has all the features for ROC analysis.

Table III



Some Popular ROC Analysis Softwares and Their Important Features

Optimal threshold is the point that gives maximum correct classification. Three criteria are used to find optimal threshold point from ROC curve. First two methods give equal weight to sensitivity and specificity and impose no ethical, cost, and no prevalence constraints. The third criterion considers cost which mainly includes financial cost for correct and false diagnosis, cost of discomfort to person caused by treatment, and cost of further investigation when needed. This method is rarely used in medical literature because it is difficult to implement. These three criteria are known as points on curve closest to the (0, 1), Youden index, and minimize cost criterion, respectively.

We describe more prevalent biases in this section that affect the sensitivity, specificity and consequently may affect the area under the ROC curve. Interested researcher can find detailed description of these and other biases such as withdrawal bias, lost to follow-up bias, spectrum bias, and population bias, elsewhere [22,23].

It is difficult to rule out all the biases but researcher should be aware and try to minimize them.

Adequate power of the study depends upon the sample size. Power is probability that a statistical test will indicate significant difference where certain pre-specified difference is actually present. In a survey of eight leading journals, only two out of 43 studies reported a prior calculation of sample size in diagnostic studies [26]. In estimation set-up, adequate sample size ensures the study will yield the estimate with desired precision. Small sample size produces imprecise or inaccurate estimate, while large sample size is wastage of resources especially when a test is expensive. The sample size formula depends upon whether interest is in estimation or in testing of the hypothesis. Table IV provides the required formula for estimation of sensitivity, specificity and AUC. These are based on the normal distribution or asymptotic assumption (large sample theory) which is generally used for sample size calculation.

Table IV Sample Size Formula for Estimating Sensitivity and Specificity and Area Under the ROC Curve

Variance of AUC, required in formula 3 (Table IV), can be obtained by using either parametric or non-parametric method. This may also be available in literature on previous studies. If no previous study is available, a pilot study is done to get some workable estimates to calculate sample size. For pilot study data, appropriate statistical software can provide estimate of this variance.

Formulas of sample size to test hypothesis on sensitivity-specificity or the AUC with a pre-specified value and for comparison on the same subjects or different subjects are complex. Refer [13] for details. A Nomogram was devised to read the sample size for anticipated sensitivity and specificity at 90%, 95%, 99% confidence level [27].

There are many more topics for interested reader to explore such as combining the multiple ROC curve for meta-analysis, ROC analysis to predict more than one alternative, ROC analysis in the clustered environment, and for tests for repeated over the time. For these see [13,28]. For predictivity based ROC, see [6].

Contributors: Both authors contributed to concept, review of literature, drafting the paper and approved the final version.

Funding: None.

Competing interests: None stated.

1. Reddy S, Dutta S, Narang A. Evaluation of lactate dehydrogenase, creatine kinase and hepatic enzymes for the retrospective diagnosis of perinatal asphyxia among sick neonates. Indian Pediatr. 2008;45:144-7.

2. Sood SL, Saiprasad GS, Wilson CG. Mid-arm circum-ference at birth: A screening method for detection of low birth weight. Indian Pediatr. 2002;39:838-42.

3. Randev S, Grover N. Predicting neonatal hyperbilirubinemia using first day serum bilirubin levels. Indian J Pediatr. 2010; 77:147-50.

4. Vasudevan A, Malhotra A, Lodha R, Kabra SK. Profile of neonates admitted in pediatric ICU and validation of score for neonatal acute physiology (SNAP). Indian Pediatr. 2006;43:344-8.

5. Khanna R, Taneja V, Singh SK, Kumar N, Sreenivas V, Puliyel JM. The clinical risk index of babies (CRIB) score in India. Indian J Pediatr. 2002;69:957-60.

6. Indrayan A. Medical Biostatistics (Second Edition). Boca Raton: Chapman & Hall/ CRC Press; 2008. p. 263-7.

7. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29-36.

8. Campbell G. Advances in statistical methodology for evaluation of diagnostic and laboratory tests. Stat Med. 1994;13:499-508.

9. Bamber D. The area above the ordinal dominance graph and area below the receiver operating characteristic graph. J Math Psychol. 1975;12:387-415.

10. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the area under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837-45.

11. Cleves MA. Comparative assessment of three common algorithms for estimating the variance of the area under the nonparametric receiver operating characteristic curve. Stata J. 2002;3:280-9.

12. Box GEP, Cox DR. An analysis of transformation. J Royal Statistical Society, Series B. 1964;26:211-52.

13. Zhou Xh, Obuchowski NA, McClish DK. Statistical Methods in Diagnostic Medicine. New York: John Wiley and Sons, Inc; 2002.

14. Metz CE, Herman BA, Shen JH. Maximum likelihood estimation of receiver operating characteristic (ROC) curve from continuously distributed data. Stat Med. 1998;17:1033-53.

15. ROCKIT [Computer program]. Chicago: University of Chicago. Available from: www-radiology.uchicago. edu/krl/KRL_ROC/software_index6.htm. Accessed on February 27, 2007.

16. Faraggi D, Reiser B. Estimating of area under the ROC curve. Stat Med. 2002; 21:3093-3106.

17. Hajian Tilaki KO, Hanley JA, Joseph L, Collet JP. A comparison of parametric and nonparametric approaches to ROC analysis of quantitative diagnosis tests. Med Decis Making. 1997;17:94-102.

18. Pepe M, Longton G, Janes H. Estimation and comparison of receiver operating characteristic curves. Stata J. 2009;9:1.

19. Jiang Y, Metz CE, Nishikawa RM. A receiver operating characteristic partial area index for highly sensitive diagnostic test. Radiology. 1996;201:745-50.

20. Youden WJ. An index for rating diagnostic test. Cancer. 1950;3:32-5.

21. Perkins NJ, Schisterman EF. The inconsistency of ‘optimal’ cut points obtained using two criteria based on the receiver operating characteristic curve. Am J Epidemiol. 2006;163:670-5.

22. Whiting P, Ruljes AW, Reitsma JB, Glas AS, Bossuyt PM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy – a systematic review. Ann Intern Med. 2004;140:189-202.

23. Kelly S, Berry E, Proderick P, Harris KM, Cullingworth J, Gathercale L, et al. The identification of bias in studies of the diagnostic performance of imaging modalities. Br J Radiol. 1997;70:1028-35.

24. Malaria Site. Peripheral smear study for malaria parasites – Available from: www.malariasite.com/malaria/DiagnosisOfMalaria.htm. Accessed on July 05, 2010.

25. Thomas S, Srivastava A, Jeyaseelan L, Dennison D, Chandy M. NESTROFT as screening test for the detection of thalassaemia & common haemoglobino-pathies – an evaluation against a high performance liquid chromatographic method. Indian J Med Res. 1996;104:194-7.

26. Bachmann LM, Puhan MA, ter Riet G, Bossuyt PM. Sample sizes of studies on diagnostic accuracy: literature survey. BMJ. 2006;332:1127-9.

27. Malhotra RK, Indrayan A. A simple nomogram for sample size for estimating the sensitivity and specificity of medical tests. Indion J Ophthalmol.2010;58:519-22.