Letters to Editor

MCQ, EMSQ or multiple true/false questions ?

David A Bender

Dept of Biochemistry and Molecular Biology, and Academic Centre for Medical Education, University College London, Gower Street, London WC1E 6BT Email: d.bender@ucl.ac.uk

Recent articles in this journal have argued for Extended Matching Sets Questions to replace Multiple Choice Questions (Wood 2003) and for the retention of MCQ (Harper, 2003). There is a place for both types of question, as well as multiple true/false questions, possibly together with confidence assessment as part of the marking scheme.

Wood (2003) has made a convincing case for replacing simple Multiple Choice Question tests (MCQ), in which students select one correct answer from four or five, with the more modern Extended Matching Set Question (EMSQ) in which students match individual cases listed under a brief scenario with one from a list of (typically) 16 possible answers. Much of Wood’s case rests on the argument that MCQs test factual recall rather than application of knowledge, and with only four or five possible answers the question “cues” the student. Certainly, with poorly constructed MCQs it is easy to eliminate the unlikely answers, and guess the correct answer. Harper (2003) has argued that MCQs can be designed to test higher levels of understanding and application of knowledge. He is correct, but there are few good examples of MCQs that do so.

Harper also argues that the style of EMSQs, with an initial scenario or vignette, followed by a series of individual cases to be matched to the long list of possible answers, is likely to confuse the student. Part of his problem is with layout, either on screen or on paper, and he is correct that poorly laid out questions will indeed confuse students unnecessarily under the stress of examinations – but the same applies to traditional MCQs as well. One advantage of the EMSQ format is that the student has to evaluate several pieces of information, and indeed may have to decide what information in the scenario or vignette is relevant and what is not – surely the ability to sift information is an important skill that we should test.

Neither Wood nor Harper discusses a third type of computer-marked test, the multiple true/false question (MTFQ), in which the student is presented with a brief lead in, followed by four or five statements, each of which must be marked true (or correct) or false (or wrong). Any number of the possible answers may be correct or incorrect. This type of test has the advantage over MCQ in that the student must evaluate each possible answer, rather than selecting the one correct answer. This means that in the same space as an MCQ tests one fact, an MTFQ tests four or five. Unlike MCQs, there is little cuing of the student, since all possible answers must be evaluated. Furthermore, it is easy to change wording subtly, so that a true statement becomes false (or vice versa), so inhibiting students who rote learn from past papers. Like MCQs, MTFQs can be designed to test application of knowledge rather than simple recall, but again there are few good examples.

Gardner-Medwin (1995, 1996, 1998, and see http://www.ucl.ac.uk/~cusplap) advocates confidence assessment as part of the marking scheme for both MCQs and MTFQs. He has not yet applied it to EMSQs, but there is no reason why it should not be done. The student not only has to select the correct answer, or mark each statement true or false, but must also assess his/her confidence in the answer: unsure scores +1 if correct and zero if incorrect; fairly sure scores +2 or –2; a high level of confidence scores +3 if correct but attracts a penalty of –6 if incorrect. Gardner-Medwin’s rationale is that a student not only has to know something, but needs to know whether s/he is confident of that knowledge or is guessing. In a clinical setting this might translate to “Do I know the dose of this drug, or should I look it up to be certain?” – the most dangerous person is the doctor who is convinced s/he is correct, but is in fact wrong; such people kill patients!

As with much in teaching, learning and assessment, there is a great deal of fashion and opinion, but little hard evidence for either side. EMSQs are a more recent development than MCQs or MTFQs. Therefore, innovators will assume that they are better because they are newer, while traditionalists will argue that the older types of test, with larger banks of tried and tested questions are better.

We should not discard our banks of validated MCQs and MTFQs, and it makes little sense to convert them into EMSQs just to follow fashion. What we should be doing is to develop new banks of EMSQs to use alongside established MCQs and MTFQs. Ideally we would set two papers in an assessment, one using MCQs or MTFQs, and the other using EMSQs, then evaluate the discriminatory value of each. This would provide us with some evidence.

There is probably a case for using both types of assessment anyway: MCQs and MTFQs are certainly valuable for assessing factual knowledge, and students do need a basis of factual knowledge; EMSQs are better for assessing the application of that knowledge, but on their own are unlikely to distinguish between the student who has not learnt the basic facts and the student who cannot apply the knowledge.

References

Gardner-Medwin, A.R. (1995) Confidence Assessment in the Teaching of Basic Science. ALT-J (Association for Learning Technology Journal) 3:80-85.

Gardner-Medwin, A.R.  & Curton, N.A. (1996) Confidence Assessment in the Teaching of Physiology. J. Physiol. 494:74P (abstract).

Gardner-Medwin, A.R. (1998) Updating with Confidence: Do your students know what they don't know?  Health Informatics 4:45-46

Wood, E. J. (2003) What are Extended Matching Sets Questions? Bioscience Education e-Journal Volume 2 http://bio.ltsn.ac.uk/journal/vol1/beej-1-2.htm

Harper, R. (2003) Multiple Choice Questions – a Reprieve. Bioscience Education e-Journal Volume 2 http://bio.ltsn.ac.uk/journal/vol1/beej-2-6.htm