Article

What are Extended Matching Sets Questions?

E J WOOD

School of Biochemistry & Molecular Biology, University of Leeds, Leeds LS2 9JT, UK

Date received: 20/03/03 Date accepted:30/04/03

Introduction

In many, or even most, university science courses, multiple-choice tests (MCQs) are used for assessing students, at least in the early years of the course. In such tests, students pick one or more correct answers (options) from four or five choices given following a statement or stem. These tests are objective and easy to mark (using a marked-sense card reader and a computer) making them attractive for large classes, and students are familiar and reasonably comfortable with the format (Booth, 1984; Schuwirth & van der Vleuten, 2003). MCQs are deceptively easy to write (it is actually quite hard to write ‘good’ ones), but once written it is possible to have a bank of questions stored away and select from them at random when the next exam is to be set: for the pitfalls of MCQ writing, see Case & Swanson, 1994; 1998. The computer can also do statistics on the class results and calculate “discrimination indices” and other parameters that hopefully lead to improved question writing (Kehoe, 1995).

Doing it by elimination

In the possible answers to the questions set, there will be a number of incorrect answers or “distractors” and many students operate, at least partly, not by knowing the correct answer but by trying to eliminate the incorrect ones. Sites in the Internet offer suggestions on how to proceed if you don’t know the answer (Runté, 1995). Furthermore, many MCQs ask for the remembering of isolated pieces of information rather than testing the ability to use knowledge to solve problems. Remembering isolated facts and/or eliminating responses that are ‘obviously’ incorrect may be regarded as an unsatisfactory way of developing mastery of a subject. It would be preferable if students were required to apply knowledge rather than simply remember ‘facts’ and it would be good if we could set MCQs in a way that prevented students getting the correct answers by elimination.

Can MCQs be used to test the application of knowledge, therefore? A simple example would be a problem that requires a calculation that generates a numerical answer. Students could then be given a choice of five values and be asked to identify the correct one. But this does not offer much of a choice and it is usually quite easy to spot (and eliminate) distractors which are well outside the likely answer. However, if we extend this concept further and do not limit ourselves to five possible responses, we can start to develop the Extended Matching Set type of question, in which there could be up to 26 (e.g. all the letters of the alphabet) possible responses, only one of which is correct. It is a relatively trivial task to change the marked-sense cards and computer system to accept one of 20-odd responses rather than one from five. Now we have a situation where eliminating the incorrect answers becomes much more difficult or even impossible, thus requiring the student to do the calculation and then identify the answer from a long list. It might be noted here that with numerical answers it is always preferable to offer ranges rather than single numbers. For example, if the answer to the calculation is actually ‘5’, then it is best to offer ‘4.8-5.2’,say, to deal with the problem of the calculator coming up with the answer ‘4.98’.

Extended Matching Sets questions (EMSQs)

This principle of testing the application of knowledge rather than simple recall can easily be extended to deal with non-numerical questions, and this forms the basis of Extended Matching Set questions. Remembering that the aim is to get students to apply knowledge rather than simply recall isolated facts, the format of the question then takes the form of a small problem. Such problems are used in the medical education field where they are given as short cases called vignettes. In three to six sentences, a medical case is described giving various details such as the patient’s symptoms, the results of lab tests, etc, and the student is asked to arrive at a diagnosis. This will then be chosen from a long list rather than five choices. The spirit of testing in this way is that, having read and understood the vignette and the information given in it, the student should be in a position to produce the answer, i.e. the diagnosis, and choose this from the list.

The examples given in Box 1 show how a question of this type might appear, first in the standard MCQ format and secondly as an EMSQ.

The first of these questions requires the remembering of an isolated piece of information and the distractors would be fairly easy to eliminate. The second question requires use of information and deduction and then remembering what the cause of the disease is. In the second question, the list could obviously be a much longer list of diseases.

It may be argued that having to write about 20 responses for each question stem is very laborious. However, this is not as bad as it might seem. In the first place, the responses are short, usually one or two words: a list of diseases, microorganisms, drugs, etc: and in the second place, one list can be used for several question stems on the same general theme. The example given in Box 2 shows how EMSQs for an examination in medical microbiology might be constructed. Here again, a simple question (Question 3) may be used but with a list sufficiently long that elimination is all but precluded: alternatively, a short case may be described (Question 4) requiring the use of knowledge.

Although Extended Matching Set questions have mostly been used in the medical field where the vignette plus responses is easy to understand and the questions fairly easy to write, there is no reason why they should not be used in other subject disciplines. In biological science, the vignette might be the description of an experiment or a series of observations, with students having to make deductions from the data. Case & Swanson (1998) suggest that the steps for writing EMSQs might be as follows:

(1)   Identify the “theme” [disease, drug, metabolite, organism].

(2)   Write the “lead-in” (i.e. for each patient described select the most likely diagnosis).

(3)   Prepare the list of “options”. These should be single words or short phrases, and should be in alphabetical order (unless there are logical reasons for not doing this), and they should all be of the same sort, i.e. disease, drug, as in (1) above. One list may be used for several question stems.

(4)   Write the “items”. They should be broadly similar in structure.

(5)   Review what you have written and check that there is a single “best” answer for each question. Then get one or more colleagues to review the items.

How do students react?

It might be asked how students respond to this different mode of questioning. In our own experience, medical students previously familiar with the one-from-five type of question do relatively poorly the first time they are faced with the new-style questions, probably because at least some of them are still working in the “eliminate the distractors” mode. It has been said many times that assessments (or examinations) drive student behaviour. On the assumption that students will adopt whatever mode of working gives them the best chance of passing the examination, if we set MCQs that simply require the remembering and recall of isolated facts, then that is what they will give us. If, on the other hand, we set EMSQs that require them to use and evaluate data with the aim of making a deduction, then this is what we will get. We set the exams and, therefore, the answer is in our hands!

Conclusion

The EMSQ format retains many of the advantages of MCQ tests (objectivity, computer marking) but transforms the questions into items that can ask students to solve little problems rather than recall isolated pieces of information. They can also help to prevent students answering by elimination rather than actually knowing the answer. Although many departments have a bank of MCQ items and are naturally reluctant to change the format, one hopes that they may see the pedagogic benefits of EMSQs. Furthermore, it is often possible to change a question in MCQ format to one in EMSQ format simply by increasing the number of options (Box 3).

This then sets the scene for a more imaginative approach to this form of assessment. Although EMSQs have mostly been used in Medicine thus far (Alcolado & Mir, 2003), there is no reason why they should not be used in other subject areas too.

References

Alcolado, J. & Mir, M. A. (2002). Extended-matching questions for finals, Churchill Livingstone, Edinburgh.

Booth, A.G. (1984). The use of a microcomputer equipped with a mark-sense card reader for student assessment. In Microcomputers in Biochemical Education, ed Wood, E. J., pp 181-193. London: Taylor & Francis.

Case, S.M. and Swanson, D.B. (1994). Extended matching items: a practical alternative to free-response questions. Teaching and Learning in Medicine, 5, 107-115.

Case, S.M. & Swanson, D.B. (1998). A full description of the many pitfalls of traditional MCQs and the relative advantages of Extended Matching Set questions in the medical context can be found in Susan M. Case & David B. Swanson’s book, Constructing Written Test Questions for the Basic and Clinical Sciences (Second Edition, January 1998), published by the American National Board of Medical Examiners (3750 Market Street, Philadelphia PA 19104). This does not exist as a printed version but may be downloaded from the website: www.NBME.org. It is available for non-commercial, educational and scientific purposes only, may not be modified and requires that copyright and permission notices appear on all reproductions. A list of relevant references is also included. This text is highly recommended and the author of this article acknowledges deep gratitude to Susan Case &  David Swanson as the originators of the ideas expressed here.

Kehoe, J. (1995). Basic item analysis for multiple-choice tests. ERIC/AE digest. http://www.ericfacility.net/ericdigests/ed398237.html. Accessed 30th April, 2003.

Runté, R. (1995). Basic rules for taking a multiple-choice test, Strategies for taking multiple-choice tests, and Test wise tricks,  http://www.edu.uleth.ca/courses/ed3604/take/mc/how.html Accessed 30th April, 2003.

Schuwirth, L.W.T. and van der Vleuten, C.P.M. (2003). Written assessment. British Medical Journal, 326, 643 - 645.