|
Y.K. Sarin
Meenu Khurana,
M.V. Natu,
Abraham G. Thomas,
Tejinder Singh
From the Christian Medical College, Ludhiana - 141 008, India.
Reprint requests: Dr. Tejinder Singh, Professor of Pediatrics Christian
Medical College,
Ludhiana - 141 008, India.
Manuscript received: December 15, 1997; Initial review completed: March 3, 1998;
Revision accepted: May 26, 1998
Evaluation has a profound effect on nature of learning and is considered as the single most important variable in directing the students to learn in a particular
way(1). Ever since the era of entrance examinations started, the market has been flooded with books containing a large number of multiple choice questions (MCQs). Students generally spend more time mastering the MCQs than they spend on reading a text book. Such a situation is likely to distort learning. A further problem arises due to 'quality' of MCQs contained in these. books. It goes without saying that if students learn from faulty MCQs, the learning may not be upto the mark. The present study was con- ducted to analyze the published MCQs in terms of three parameters (enumerated below) used for item analysis.
Subjects and Methods
Fifty final year medical students of Christian Medical College, Ludhiana took the test at a pre university preparatory
examination. The question paper consisted of 150 MCQs from all clinical disciplines. These were randomly picked up from some of the popular books of MCQs available in the market. All the questions were of one best response type and allowed 1 min per question for answering.
The question papers were scored using the key. There was no negative marking. The scored papers were arranged in order of marks obtained and were later subdivided
into 2 groups of 25 each, which were designated as Higher ability
group (H) and Lower ability group (L). Based on these groupings,
the following indices were calculated for all questions using standard methodology(2). (a) Facility value (FV); (b) Discrimination index (01); and (c) Distractor efficiency (DE).
Facility value is an indicator of difficulty of the item. Lower the facility value, more difficult is the item. In simple terms, it tells the percentage of students in a group who could answer that item
right. The preferred range of FV is 40-60; however, for
competitive examinations, items may have a lower facility value.
Discrimination index is a measure of the ability of an item to differentiate between good and poor students.
Items with a DI between 0.15
-
0.24 need to
be revised while those with value less than 0.15 are most likely to be discarded. Distractor efficiency tells, how efficient each alternative was in distracting poor students but not good students. A good distractor should be picked up by at least 5% of the students(3).
Results
The scores obtained by the students ranged from 26 to 133. The internal consistency of the test by split-half method was 0.59 (p < 0.05). The values of parameters have been shown in Table 1. There was a wide variation in the values obtained. Most items had a lower FV, which may be acceptable in competitive examinations. However, the discrimination index was poor for most items. The number of items with a DI value of less than 0.25 were 123 (82%). In few instances, there was negative discrimination, meaning thereby that more L group students were answering it correctly as compared to H group. The distractor efficiency of many distractors was poor (Table II). In 7 items, none of the distractors was ticked by any student. Put in another way, it means that out of 450 distractors used in these items (3 per item),
,
94 (20.8%) were not used at all. Even amongst the distractors that were used, 137
(38.4%) failed as effective distractors (<
5%
response).
Discussion
Evaluation is an important component of learning cycle. It is natural that it flows from the objectives of learning and has to be designed to test the attainment of objectives. It can't be taken as a separate entity in itself. Further, the evaluation tools have to be suited to a particular group of students and generalizations across the groups are generally poor.
TABLE I
Distribution of Parameters.
|
FV
|
DI |
|
Range |
No. (%) |
Range |
No. (%) |
|
0-20 |
27 (18.0) |
<0 |
21 (14.0)* |
|
21-40 |
59 (39.3) |
0-0.14 |
61 (40.6) |
|
41-60 |
18 (12.0) |
0.15-0.24 |
41 (27.3) |
|
61-80 |
35 (23.3) |
0.25-0.35 |
23 (15.3) |
|
81-100 |
11 (7.3) |
> 0.35 |
4 |
(2.6) |
*11 (7.3%) items had values less than -0.15
TABLE II
Distractor Efficiency
Efficiency parameter
|
Number of items (%)
(n=150) |
| One distractor not used |
35 (23.3) |
| Two distractors not used |
19 (12.6) |
| Three distractors not used |
7 (4.6) |
The increasing use of MCQs in post-graduate entrance examinations has distorted the sequence of learning. MCQ books have replaced the traditional sources of acquisition of knowledge. A perusal of
many of these MCQs reveals that they suffer from structural flaws(2) as well as
printing errors including a wrong key. Since the purpose of the student in reading these books is not to test his knowledge but to learn the key in case a question appears in the examination paper, he is likely to get factually incorrect knowledge.
The results of item analysis demonstrate an interesting pattern. Most of the items had a low FV, indicating that they were not correctly answered by majority of the students. In addition, the DJ for many of them was far below the acceptable range of 0.25 - 0.35. Taken together it raises doubts about the quality of an item. Negative discrimination was also seen with certain items which in other words means that more L group students were answering it correctly as compared to H group. These items are likely to distort learning.
The values obtained at item analysis strengthen our belief that not all questions can become a source of learning in this way. There are questions with negative discrimination and there are distractors which have failed to attract even a single student. It is therefore right to presume that a student who bases his learning on these MCQs is likely to learn sub optimally and possibly suffer at the examination also.
We do not want to convey the message that all published MCQs are of poor quality. In fact, some items had a very good DJ. However, there are items which need to be discarded, in addition to a large number of items which need revision.
The era of entrance examinations has made the students depend more on MCQ books(4). It has to be accepted as a necessary evil but efforts have to be made to overcome the negative aspects of this phenomenon. Tackling this problem may not be easy. In our opinion, the following points need to be considered:
(a) As teachers, we should impress upon students that these books should be used only to evaluate what has been learnt from standard texts and not as a primary source of learning.
(b) In case the key is at variance, it is better
.
to consult the respective subject expert
rather than to take it as correct.
(c) The publishers of these books should desist from publishing that such and such question appeared in such and such examination.
(d) It would be better to give some idea regarding FV and DI of each question. It may be more helpful to the student rather than knowing that this question was used by 'A' institution or 'B' institution.
|
1.
Miller GE, Fulop T. Educational strategies for health professions. Geneva, WorId
Health Organization Public Health Papers No. 61, 1974; pp 1-83.
2.
Singh T, Natu MV, Singh D, Paul VK. Test and item analysis. In: Better Pediatric Education 1st edn. New Delhi, lAP Edu- cation Centre, 1997; pp 73-83.
3.
Guilbert JJ. Test and measurement techniques. In: Educational Handbook for Health Professionals, 1st edn. Geneva, WHO Offset Publication No. 35, 1981; pp 4.01-4.65.
4.
Adkoli BV. Attributes of a good question paper. In: Sood R. Assessment in Medical Education. Trends and Tools; 1st edn. K.T. Wig Center for Medical Education and Technology, All India Institute of Medical Sciences, New Delhi, 1995; pp 67-81.
|
