Medical Education

Indian Pediatrics 2001; 38: 500-513  

A Practical Approach to Running an Objective Structured Clinical Examination in Neonatalogy for Formative Assessment of Medical Undergraduates

Piyush Gupta
Hema Jyoti Bisht

From the Department of Pediatrics, University College of Medical Sciences and GTB Hospital, Delhi 110 095, India.

Correspondece to: Dr. Piyush Gupta, Block R-6-A, Dilshad Garden, Delhi 110 095, India.

E-mail: [email protected]

Conventional clinical or practical examina-tion suffers from certain inherent deficiencies. In addition to in-built variability due to student, patient, or examiner factors that increase the subjectivity of marking, it has several other disadvantages. Assessment is usually global and not competency based; often only the end-result is tested and not the process of arriving at that conclusion. There is little scope for testing practical skills, communication with patients, or attitudes, and there is hardly any scope for a systematic feedback(1).

Objective structured clinical examination (OSCE) was evolved to overcome some of these problems. An OSCE consists of a circuit of stations through which each candidate must pass. These stations assess practical, commu-nication, technical, and data interpretation skills and there is a predetermined decision on the competencies to be tested(2,3). Only one center in India has reported their experience with OSCE as a tool for formative evaluation in Pediatrics(4,5); however, the operative details of the exercise were lacking.

In our institution, final year MBBS students are posted in Pediatrics by rotation for a period of four weeks. Considering that new-born care is an essential, integral and ‘must acquire’ skill for a primary care physician, the students are exposed to neonatology for around 7-8 days en block during this posting. We are conducting an OSCE for ward leaving assessment at the end of neonatology posting to assess the psychomotor, communication and data interpretation skills related to essentials of newborn care. The present paper summarizes the practical and applied details of one of such exercises.

Final year students are rotated through Pediatrics in batches of 12 to 14 students each. Approximately 21 to 24 hours (7-8 sessions of three hours each) of total posting time in Pediatrics is devoted to teaching-learning activities in the newborn care. The neonatal unit, after much deliberation, has already laid out a list of 20 specific objectives to be achieved by the learner following exposure to this training week. The objectives are framed with one eye on the undergraduate curriculum and the other on the needs of the community. The students are provided with a schedule of learning activities as well as the desired objectives at the start of their posting. The students are exposed to labor rooms, postnatal wards, well baby and high-risk clinics, rooming in area, and neonatal intensive care unit. Learning tools include an integration of structured teaching with practical demonstra-tions, tutorials, group discussions, and hands on experiences. Students are encouraged to observe practical procedures and communica-tion sessions. They are motivated to pursue independent study on issues derived either from tutorial sessions or self-generated. All throughout, a facilitator is earmarked to guide the students through a learning session. The student is also free to interact with faculty members who act as important resources for learning.

A critical consideration in planning an OSCE is the number of examination stations, the content of each station, and the number of observers required. For a batch of 12-14 students, we have devised an OSCE consisting of 15 stations. These are essentially of three types:

Procedure Station: The candidate is asked to perform a certain task in front of an observer who marks the student as per the provided checklist. Usually these tasks are in terms of taking history, performing examination, or demonstrating a procedure. The role of the observer is to assess how competently and comprehensively each candidate completes the given clinical task. Marks in the checklist are given for each itemized component and may not be necessarily equal for all items. Components considered to have performed adequately are awarded full marks, those performed inadequately/poorly or not at all receive no marks. Additional marks are incorporated in the checklist for performing the skill in a fluent and coordinated manner. Figures 1 to 3 each exemplifies a task and checklist for an observed station as used in the present evaluation.

Response Station: The candidate works by himself, not supervised by the observer. Response to certain questions is anticipated, either based on the previous procedure station or chosen to evaluate areas of knowledge, interpretation, problem solving etc. These may be in form of patient problems, case histories, laboratory or field data, radiographs, speci-mens, equipment, clinical photographs, etc. The stations are so designed that the student is unable to answer the questions correctly with-out using the data provided. The completed answer sheets are evaluated later by compar-ing with model answers and a key containing the distribution of marks. Two examples of this type of station along with the model answers and key are provided in Figs. 4 & 5.

OSCE not only deals with the ‘doing’ part but also takes into account the ‘what’ component of clinical competence. Question stations are a type of Response stations, meant exactly for this purpose, i.e., to test the results arrived at the previous stations(1). For example, the student may look for capillary refill time at a procedure station but provides the interpretation at the following question station. Due to overall less number of stations, separate question stations were not included in the present scheme. However, we tried to incorporate the question element within the checklist for procedure station.

Rest Station: These stations are included to give candidates, observers, and patients a break. They also allow time if required, to substitute patients at a clinical station or to complete the written left over task from the previous stations. The rest stations can be designed imaginatively; these may be planned with intent to relax the examinee and highlighted with some slogan or cliche for the student to ponder over. We prefer to have a combination of food for thought as well as for the stomach at the rest stations.

Critical Stations: These stations are designed to assess a candidate’s competence in a crucial skill, e.g., resuscitation. Passing at these stations is necessary in order to achieve overall success grades. Since ours is a formative assessment, we usually do not designate any station as critical in these examinations. However, we allot a higher share of marks to the particular station on resuscitation (Fig. 3).

The format of 15 station OSCE for each batch of students is decided upon, keeping in mind our constraints of manpower, time, space, economy and other logistics. The break-up of stations is as follows: (i) Observed stations – 6; (ii) Response stations – 6; (iii) Rest stations – 3; and (iv) Critical station – nil. The number of observed stations was decided after taking into view the probability of availability of committed and motivated observers on the day of examination. We purposefully have more rest stations than desired in a 15 station OSCE design, as it is usually a new experience for the candidates and warrants more breathers; no other clinical department in our Institute is conducting OSCE at present.

OSCE Station No.|__________|               Max Marks 10                                 Student Roll No.|________|

Demonstrate proper technique of hand washing and dressing prior to entering the Neonatal Intensive Care Unit.

The Checklist

Items

Max. marks

Adequate

Inadequate

Bares hands up to elbows and removes
watch/jewellery

½

Uses soap/solution and makes leather/foam

½

Cleans/scrubs

½

½

1

1

1

½

Scrubs vigorously for 2 minutes

1

Rinses properly (hand to wrist)

½

Dries hands

1

Wears gown and mask correctly

1

Performs fluently

1

Grand Total

10

Fig. 2. Example of an observed station related to clinical assessment. The first part delineates the task to be performed while the second component provides a candidate’s checklist including the mark sheet. The tasks to be performed are broken item-wise and scored accordingly by the observer. A separate checklist is used for each student. The observer is instructed to mark only one response out of two, i.e., adequate or inadequate, for each item respectively.

 

OSCE Station No.|__________|               Max Marks 10                                 Student Roll No.|________|

1. Examine this newborn. Classify as term/preterm and justify the same.
2. Record weight of this baby.
3. Ask for exact gestational age of this child from the observer.
4. Specify the weight for gestational age category of this child with the help of your data and provided chart.

Once the consensus is reached on the number and type of stations to be included, the next task is to formulate the questions, model keys, and checklists for each station. The principals of content validity and practicability govern individual station design. It is important that the test contains a representative sample of what the student is expected to have achieved and this is balanced with proposed expenditure, available equipment and the time allotted to each station. The duration of a station is usually determined by the task involved(6). We allot a uniform time for all stations, i.e., 5 minutes each for ease of performance as well execution; though schemes are available that adjust both 5 and 10 minute stations in a single OSCE examination. The questions are discussed and formulated in a meeting including the consultant and residents working in the neonatal unit. All the residents have been briefed on the OSCE concept and methodology; few of them also had the practical experience and had assisted the first author earlier in running an OSCE examination for undergraduate medical students.

Observed stations are related to crucial clinical skills such as resuscitation, weighing the baby, assessing the gestation, diagnosing low birth weight or growth retardation, neonatal examination, maintenance of asepsis, communication skills, breast feeding, and elicitation of danger signs. Figures 1 to 3 depict the design of three such stations (station numbers 6,8, and 13 concerned with asepsis, clinical examination, and resuscitation, respectively) specifying the task to be done, and the anticipated item wise responses. These responses are to be graded by the examiner based on the student’s level of performance, on a checklist. Students are not aware of the content of the checklist. We also include one knowledge based recall station (rapid fire), so as to incorporate a structured viva voce examination in this OSCE.

Response stations comprise of problem solving exercises pertaining to both data as well as clinical case histories, besides inter-pretation on radiographs, clinical photographs, and specimens/equipment pertinent with core neonatal care. Figure 4 demonstrates the content of an unobserved station involving a data problem used in the present examination (Station No. 1). Figure 5 presents an example of a clinical problem (Station 9).

All the station tasks are printed in big bold letters on an A4 size paper using a laser printer in clear and unambiguous terms. Similarly, all the checklists for observed stations should be readied and printed in excess of that required. Surplus can always be used later. A copy each of the tasks, as well as the checklist printouts is kept with two senior persons; in case, any one of them fails to show up on the exact date and time of examination.

OSCE Station No.|__________|               Max Marks 15                                 Student Roll No.|________|

A Term baby is born by normal vaginal delivery to a second gravida mother. Amniotic fluid is not stained with meconium. Go ahead with the process of resuscitation with provided dummy and equipment. You are free to ask for vital signs of the baby from the observer, wherever appropriate.

The Checklist

*Mark as "yes" or "no" in place of "adequate" or "inadequate" respectively.

OSCE Station No.|__________|               Max Marks 10                                 Student Roll No.|________|

The Problem

Yearly data (for the year 1999) pertaining to deliveries and their outcome in a community is as follows:

Calculate the Perinatal and Neonatal Mortality Rates for this community for the year 1999, demonstrating the steps taken to arrive at the results.

Model answers:
Perinatal mortality rate = (114 + 410)/12600 ´ 1000 = 41.6 per 1000 births.

Neonatal mortality rate = (410 + 280)/(12600 – 114) ´ 1000 = 55.2 per 1000 live births.

The key: Five marks each are for correct calculation of two mortality rates. The student gets full marks if the steps/formula used and the ultimate answers are correct. Eighty per cent marks are awarded if the steps/formula are correct but calculations are wrongly done. No marks are awarded, in case the formula used is incorrect, or the steps of calculation have not been demonstrated, even if the ultimate answer is correct.

OSCE Station No.|__________|               Max Marks 10                                 Student Roll No.|________|

A 26-year-old booked primigravida mother delivered a female baby weighing 1.2 kg at 32 weeks of gestation. Apgar score of the baby was 5 and 9 at 1 and 5 minutes, respectively. Baby was admitted to NICU. The stay in nursery and problems encountered are as follows:

Provide the most likely diagnosis for each of the listed problem.

Model answers: (1) Hyaline membrane disease/RDS, (2) ABO hemolytic disease, (3) Hypocalcemia, (4) Withdrawal bleeding, (5) Umbilical sepsis.

The key: Two marks are given for each correct answer.

Fig. 6. An outline of the Station Map for a 15 station OSCE in Neonatology including the layout of stations and position of time-keeper. Candidates enter through main entrance five minutes prior to start of examination. All of them are already briefed and supplied with this map. They initially proceed to the timekeeper’s room and as the first bell sounds, disperse to their respective stations as communicated to them earlier. The bell keeps on sounding at 5 minutes intervals. The arrows indicate the movement of the students from one station to another.

Conduction of OSCE requires a considerable amount of floor space. Stations for clinical examination require a larger space to accommodate the patient and the observer, in addition to the student. Ideally, the OSCE should be conducted in a large single room providing at least 2 sq. m. space for each station. Alternatively, it can be conducted in two or more adjacent rooms. The station transition time should be provided and adjusted for within the 5 minute slot. We utilize the NICU areas such as lying in room, the corridor, one of the cubicles that is unoccupied; and support services rooms, including those meant for the consultant, sister in charge, laboratory, pantry, doctors and nurses changing area. Fifteen stations are adjusted in this area taking care to screen them from one another to ensure privacy for both patient and the candidates. Position of stations is decided depending upon the nature of the station and available space. Finally, a station map is constructed; this serves as the most important tool and reference material for all further preparations (Fig. 6).

Once a station map has been constructed, the next step is to draw up a detailed list of all the equipment and manpower required (Table I). This should be done at least 4 to 5 days prior to the examination. Adequate supply should be ensured to avoid last minute problems.

Furniture and Equipment: Tables and chairs should not be broken or three legged. Examination couches should be with mattresses. Supply of clean and ironed table-clothes, towels and linen should be ensured. All specimens and equipment should be labeled and/or flagged as per the requirements of the task to be performed, at least a day before the examination. Electrical connections, gadgets and mannequins should be checked to be in good working order. Care should be taken that the disposables used are not thrown after the examination; these can be stored and recycled in subsequent OSCEs.

Recruitment and Training of Manpower: Examiners/observers chosen for the examina-tion session should be highly motivated and committed. We select them from the resident staff posted in the neonatology unit only. The group is systematically introduced to OSCE. The process of sensitization starts a month prior to the examination and includes several brief sessions on overview of the modern evaluation techniques with special reference to clinical examination. At the end of these sessions, the participants should be able to define OSCE, design OSCE based on teaching learning objectives, appreciate the usefulness of OSCE in formative and summative assessment, and be conversant of the difficulties, advantages and limitations of this method as compared to the traditional clinical evaluation. As per Table I, we required 6 observers/examiners and one time keeper for the examination session that was to last 1 hour and 15 minutes (15 station rotation for 5 minutes each). As it is not possible for the examiners to leave midway between the examination, those on emergency duty are spared. A time-keeper is also identified. One of our laboratory attendants/clerks is entrusted with this job.

A day before the examination, each examiner is finally allotted his/her station and provided the specific details of that station. They are also shown the checklists for their respective stations and format of marking is discussed. They are instructed to be objective observers and maintain only the appropriate interaction with the candidate relevant to the concerned task.

Patient Selection: The use of patients for clinical examination stations is regarded as crucial to ensure authenticity and to increase validity(7). Owing to the nature and duration of the examination, and fragility of concerned patient, i.e., newborn; it is desired that preferably healthy stable newborns should be selected. The newborns should always be accompanied with their mothers as not to deny the basic right of breastfeeding during an examination. For the same purpose, it is important to have an extra newborn available for each of the clinical station. Duplicate neonates should have similar weight/gestation and physiology so that they can be substituted when the original infant/mother needs a break. The mothers should be told that the infants might be examined repeatedly by different students. Both the mother and child should be kept well briefed, warm and comfortable. The mother should also be provided with nourishment during the examination. She should enjoy the experience and not be frightened by it. Patients should be selected and recruited at least a day before. Main-tenance of asepsis and temperature is of utmost important while dealing with clinical examina-tion of the newborns.

Simulated Patient: It may not be possible or practical to recruit patients for certain stations, such as those assessing communication skills. For this purpose, simulated patients (SP) are planted(8). In the west, the SPs are recruited from drama students or professional actor/actresses. In our scheme given in Table I and Fig. 6, Station No. 11 was concerned with the counselling of mother on breast-feeding. We trained one of our staff-nurse to act as the mother on this station. Simulated patient (i.e., the staff nurse who had to act out as the mother) was trained for her role prior to the examination. She was given an outline of her proposed role and a draft script. This was followed by a rehearsal of her individual role with a clinician. The clinician advised her on technique of communication and medical facts. After the OSCE, the SP was debriefed; criticism of the station was invited, which helps with future restructuring of station(9).

Food and Water: Eatables to be placed at the rest stations should be unperishable, non-sticky, and non-messy. We buy them on the day or an evening before the examination morning.

Table I__List of Stations with the Requirements* for Equipment, Patients and Observers for OSCE.

Station
No.

Station
description

Basic
equipment

Specificneeds
needs

Patient/observer
requirement

1.

Data interpretation

Table, 1 chair

Calculator

–

2.

Clinical
examination:
reflexes and
danger signs

Patient screen, examination
couch/warmer, 2 chairs,
heater/blower, handrub,
paper napkins

Weighing machine,
rectal thermometer

2 normal
newborns with
mothers,
1 Observer

3.

Equipment:
Phototherapy

Writing desk, 1 chair

Phototherapy equipment
with duly labeled
parts/components

–

4.

Rest station

Table, 1 chair

A tray with biscuits,
napkins

–

5.

Clinical
photographs

Mounting board, writing
desk, 1 chair

A chart with affixed
and labeled
photographs

–

6.

Asepsis: Hand
washing and
dressing up

Washing area with elbow
operated taps, running water,
dryer

Soap, Sterile gowns,
chittelle forceps,
disposable mask and
caps, stop watch

1 Observer

7.

Radiology-
X-ray interpretation

Table, viewboxes, electrical
connection, extension cord, 1
chair

X-rays

–

8.

Clinical
examination:
anthropometry,
gestation and
interpretation

Patient screen, examination
couch/warmer, 2 chairs,
heater/blower, handrub,
paper napkins

Weighing machine,
intrauterine growth chart,
infantometer, measuring
tape

2 IUGR
newborns with
their mothers,
1 Observer

9.

Problem solving

Table, 1 chair

Problem sheet

–

10.

Rest station

Stool, 1 chair

A box of Rewari/Gazak

–

11.

Counseling-
breastfeeding

Table 3, chairs

Syringe, blade, cup

Simulated
mother,
1 Observer

12.

Specimens

Table, 1 chair

Specimens to be
displayed duly labeled
and flagged

–

13.

Resuscitation

Table, 1 chair

Resuscitation manikin
with all supplies
including oxygen

1 Observer

14.

Rapid fire-viva
questions

Table, 2 chairs

 –

1 Examiner

15.

Rest station

Stool, 1 chair

A jar of toffees, jug of
water, glasses

–

* Additional requirements: Time keeper, stop watch, bell, checklists for Station numbers 2,6,8,11,13 and 14 and key for Station numbers 1,3,5,7,9 and 12.

All the stations are set up on the day of examination, though the furniture and station identification numbers had been arranged a day before. The authors reach the venue of examination, 2 hours prior to start of the examination. The observers/examiners and students are called in 1 30 minutes before the OSCE is scheduled to start. The tasks/questions are pasted on respective stations 30 minutes before ringing the first bell. Examiners are also handed over the checklists simultaneously.

Candidate’s Briefing: At the start of their posting, all the students are primed regarding OSCE as the mode of their ward leaving assessment. On the final day of assessment, they collect in the seminar room of the department where they are briefed on the methodology of the examination. Each one of them is handed over a station map (Fig. 6) along with an answer book to record their answers on unobserved stations. The first station for each student is marked on his/her respective station map. As the students had not encountered this mode of examination in the past, they are instructed on how to move between stations with the help of the map. The candidates are also informed about the type of stations and reminded about the rest station. They are asked to display their roll numbers prominently on the lapels of their coats/aprons for ease of marking by the observer on procedure stations. The candidates are then escorted to the examination area. As the first bell sounds, the students disperse to their respective allocated first stations.

The time-keeper rings the bell loud and clear enough to be heard at all stations. This should be pre-checked. The bell is sounded at intervals of 5 minutes and a record of the elapsed time and the number of bells sounded is maintained. In all, the time-keeper has to sound the bell 16 times.

PG supervised the whole process; in addition, he was prepared to step into examine at any station at any time, if needed.

After the OSCE, the group of candidates is taken around all the stations and a feedback is provided to them with a breakdown of how they have performed at each station, by the respective observer. PG provides them with feedback on the unobserved stations. The theme and aims of individual stations is outlined, correct response and/or procedure is informed, and common mistakes committed by the students are emphasized. Lastly, the candidates are asked to provide a written feedback (in the form of a paragraph) on the entire exercise, which includes their experiences, suggestions for improvement and future expectations. Ideally, the feedback should be structured to be more meaningful. All students participate and provide the feedback. A selection of feedback received from the students is provided in Table II.

After the examination is over, the observers are invited to comment upon the process and provide their valuable inputs and suggestions for identifying the lacunae and strengthen future such exercises. The feedback also promotes discussion to allow improve-ments in station design and teaching of particular skills.

A complete mark-list is prepared and circulated to the candidates a day later that also indicates the maximum, minimum, and the average marks scored by them at individual stations.

Table II__Students’ Feedback on OSCE as an Evaluation Tool in Neonatology

An OSCE provides a reliable, reproducible, and valid form of assessment. Used as a formative assessment tool, it provides a unique opportunity for effectively blending the teaching learning and evaluation methods. The ward-leaving examinations become more enjoyable rather than assuming psyche-threatening proportions. The emphasis can be diverted to achieve stimulation of thoughts and promote self-learning. Variety in stations maintains students’ interests. There is increased student faculty interaction and a positive feedback is immediately available that is useful both to the student and the facilitator. Moreover, the system is not rigid and is easily adaptable as per local needs to the subject and department concerned. Students seem to appreciate the challenge of OSCE and enjoy demonstrating clinical skills in that manner. However, it is to be kept in mind that the exercise is very labor intensive, requires very careful organization and meticulous planning. Only ingenuity and imagination limit the range of clinical skills that can be tested through OSCE.

OSCE also has its own limitations. While it may test specific skills, it does not evaluate the comprehensive understanding of the candidate. It tends to segregate the patients’ problem into components rather than testing him as a whole. This drawback can be overcome by combining OSCE with a traditional case presentation. Moreover, the scheme described here can only include stable and large neonates. The learning outcomes related to the care of sick newborns can be incorporated in an OSCE which may (and can) be conducted in the NICU setting.

Authors would like to thank the Senior Residents (Dr. Ashish, Dharmendra, Nisha, and Rajesh) and postgraduate students (Drs. Dinesh, Geetinder, and Kamal Pratap) posted in the neonatalogy unit for their kind help for smooth execution of the depicted exercise.

Contributors: PG conceived the idea of conducting OSCE and put it into practice. He provided the framework and overall concept for the exercise and drafted the manuscript. He will act as the guarantor of the paper. HJB helped in conduction and co-ordination of the entire exercise. She also helped in drafting the manuscript.

Funding: None.

Competing interests: None stated.

1. Singh T, Singh D, Paul VK. Principles of Medical Education. IAP Education Centre, Department of Pediatrics, All India Institute of Medical Sciences. New Delhi, 2000; pp 94-104.

2. Harden RM, Stevenson M, Downie WW, Wilson GM. Assessment of clinical compet-ence using objective structured examination. British Medical Journal 1975; 1: 447-452.

3. Harden RM, Gleeson FA. Assessment of medical competence using an objective struc-tured clinical examination (OSCE). Medical Education 1979; 13: 41-54.

4. Verma M, Singh T. Experiences with objective structured clinical examination (OSCE) as a tool for formative evaluation in pediatrics. Indian Pediatr 1993; 30: 699-702.

5. Verma M, Singh T. Attitudes of medical students towards Objective Structured Clinical Examination (OSCE) in pediatrics. Indian Pediatr 1993; 30: 1259-1261.

6. Van der Vleuten CPM, Van Luyk SJ, Swanson DB. Reliability of the Maastricht skills test. Proc Ann Conf Res Med Edu 1988; 27: 228-233.

7. Stillman P, Swanson D, Regan MB, Philbin MM, Nelson V, Ebert T, et al. Assessment of clinical skills of residents with standardized patients. Ann Inter Med 1991; 114: 393- 401.

8. Smee SE. Standardized patients: A trainer’s perspective. Ann Comm Orient Educ 1993; 6: 273-282.

9. Van der Vleuten CPM, Swanson DB. Assessment of clinical skills with standardized patients: State of the art. Teaching Learning Med 1990; 2: 58-76.