Research article

Multiple-choice Questions – A Reprieve

Ray Harper

Department of Sport, Exercise & Biomedical Sciences, University of Luton, Park Square, Luton, LU1 3JU, UK

Date received: 28/08/03                     Date accepted:23/10/03

Multiple-choice tests are widespread and it is important that they are implemented with rigorous attention to quality by following “good practice” to avoid the pitfalls in their design. Contrary to some suggestions, such MCQ’s can be used to assess a variety of outcomes including all of the competences in Bloom’s taxonomy and the desire to assess the use of knowledge does not require more complex question structures such as EMSQ’s.  MCQ’s are a useful assessment tool, with a simple design, easy implementation (with appropriate staff development) and with clarity of focus.

Introduction

In a recent article in BEE-j, Wood (2003) has suggested that multiple-choice questions (MCQ’s) should be replaced by extended matching sets questions (EMSQ’s). This has been proposed previously by Wilson & Case (1993). It is the case that in some paper-based assessments EMSQ’s can have a useful place. However, in the context of changing MCQ’s to EMSQ’s, the problems associated with EMSQ’s may well outweigh any advantages. Wood identified two justifications for changing to EMSQ’s: (i) that MCQ’s assessed “remembering isolated pieces of information” rather than “the ability to use knowledge” and (ii) that they could be answered by eliminating the incorrect options (distractors). This paper will show that the use of MCQ’s is justified, can be used to assess the ability to use knowledge and that there may be pedagogic problems with the use of EMSQ’s.

Answering by Elimination

There are various texts outlining good practice for the design of MCQ’s (Brown, Bull & Pendlebury, 1997; Bull & McKenna, 2001; Pritchett, 1999). These indicate that the distractors should be “meaningful” and “equally possible”. The example referred to by Wood (2003), where it is possible to answer the question just by eliminating the incorrect options, does not follow these good practice elements of MCQ design. The question is shown below with the accompanying text:

"If the question asks you something you do not know, see if you can cross out any of the wrong answers before you guess:

1.The capital of Alabama is

          A) Montgomery

          B) Birmingham

          C) Edmonton

          D) Ottawa

You may not know anything about Alabama. But you do know that Edmonton and Ottawa are Canadian cities. So (C) and (D) must be wrong answers for this question. By crossing out Edmonton and Ottawa, you only have to guess between (A) or (B). That means you have a 50/50 chance of getting it right, just by guessing.” (Runté, 1995)

The distractors should at least be cities from the U.S.A. and possibly large towns/cities in Alabama depending on the background of the students. Appropriate MCQ distractors can only be eliminated by relevant knowledge of the subject being assessed and this makes the process of eliminating each of the distractors almost as useful to the assessment process as knowing the correct option. Indeed, eliminating 3 appropriate distractors from 4 options may demonstrate more knowledge of the subject than knowing the correct answer! McCabe & Barrett (2003) have promoted the use of scoring systems for objective tests that are designed to give credit for demonstrating this type of knowledge. This demonstrates that with an appropriate set of distractors, elimination should not be an issue and it indicates the importance of staff development for question designers and the promotion of good practice.

The long list of options provided in many EMSQ’s might well prevent the user from determining the correct answer by elimination. However, there is nothing to prevent the use of a long list of options in MCQ’s if this is the desired outcome. It is difficult to see how this can be a basis for not using MCQ’s. The advantage of the EMSQ would be the increased efficiency of only displaying the scenario, ‘lead-in’ and the list of options once. This is not a pedagogic argument for changing from MCQ’s to EMSQ’s and the pedagogic problems with the structure of EMSQ’s are discussed below.

Remembering Information

It is axiomatic that students need to know the basic terminology and concepts of a subject before they can understand more demanding interrelationships and applications of the basic knowledge. It would appear that MCQ’s might be ideal vehicles to assess this basic knowledge that necessarily to a large extent involves its recall rather than its application. I suggest that it is not unreasonable to check that a student can identify C=O as a carbonyl group, to remember that it is a polar bond and to remember that polar bonds can form hydrogen-bonds before looking at the application of this knowledge to the stabilisation of protein structures. Remembering information is still a part of the learning path and MCQ’s can readily assess this learning. Although EMSQ’s are commonly used to assess the use of knowledge, there is nothing inherent in the structure of an EMSQ to prevent it being used to assess recall.

The Ability to Use Knowledge

The first example of an EMSQ provided by Wood (2003) in Box 1, Question 2 is reproduced below and is actually a MCQ.

The inclusion of a ‘clinical vignette’ in the stem to a question does not change the structure from a MCQ to an EMSQ. Wood subsequently indicated how such questions could be converted into EMSQ’s by extending the list of options and adding further questions that use one set of options and some preliminary text. However, as Wood’s own examples clearly show, the ability to assess the use of knowledge is not inherent in the structure of the question but in its content and wording and an MCQ with an appropriate set of distractors can perform this task as well as any other style of question.

The above authors promoting the use of EMSQ’s appear to assume that MCQ’s cannot assess the ability to use knowledge without reference to an evidence for this. That MCQ’s can assess beyond the basic “knowledge” competence in Bloom’s taxonomy (Bloom, 1972) and include all 6 of Bloom’s competencies has been discussed previously (Bull & McKenna, 2001; Pritchett, 1999). Examples of MCQ’s at all of the 6 competence levels can be found in Bull & McKenna (2001). I anticipate EMSQ’s could similarly be used to assess all of these levels since the two question styles are based on fundamentally similar principles. The requirement to assess the use of knowledge does not appear to provide any rationale for a change from a MCQ to an EMSQ style of question.

Problems with EMSQ’s compared with MCQ’s

(i) Structure and Presentation

EMSQ’s have some initial scenario or ‘clinical vignette’ and ‘lead-in’, then the list of answer options that are available followed by several questions (items). This lengthy structure generates a large amount of text that with appropriate organisation and a readable font size may not fit onto a standard A4 page. In this case, where the set of questions cannot be presented on one page, the student is required to flick between pages to see the question and remind themselves of the scenario or the list of answer options, this would be distracting and may lead to underperformance by candidates as discussed below. Clearly such situations can be resolved by reducing the number of questions relating to the ‘vignette’ or by reproducing the vignette and answer options but this looses much of the advantage in efficiency of the EMSQ over an equivalent set of MCQ’s.

Even should the set of questions fit on one page, the student has to scan visually between the questions and the long list of answers. After the first question based on the ‘lead-in’, scenario & options, the subsequent questions become separated from the scenario, etc. by some now irrelevant text. This is distracting and may prevent the student from concentrating fully on determining the answer. This effect is minimised in MCQ’s that are relatively compact. The possibility of students being affected by the presentation of the questions should not be overlooked. Ricketts & Wilks (2002) found that MCQ tests presented on-screen in a scrolling format where students had to use the mouse and scroll bars to navigate produced an average decrease in median marks of 14% compared with questions presented on paper and 20% compared with questions presented on-screen singly. I am not aware of any comparison of EMSQ’s with an equivalent set of MCQ’s. However, the work of Ricketts & Wilks suggests that elements that distract the candidates may lead to underperformance. Until it is demonstrate that the more complex structure of EMSQ’s does not have a significant detrimental effect on student performance it may be safer to use MCQ’s. Alternatively, extensive practice by a cohort in formative tests containing EMSQ’s may be advised, before using EMSQ’s in summative tests. Generating demanding questions does not require questions with complex structures; MCQ’s that contain ‘vignettes’ when appropriate can suffice as discussed above.

Constructing a relatively long list of answer options that is suitable for several questions in an EMSQ is not straightforward even though the questions are all on the same topic. When editing the list care needs to be taken that a key for one of the questions is not omitted. In such a long list there is a high probability that there will be distractors that the student can easily eliminate and may in effect leave the student considering only those distractors that would be put in an equivalent well-designed MCQ for that particular question.

(ii) Flexibility

Although multiple-choice tests can be conducted in paper format with manual or computer-based marking through OMR, many such tests are now undertaken as computer-assisted assessments (CAA) using appropriate assessment software. It is useful to construct questions that can readily be used either on paper or in CAA. MCQ’s can easily be transferred into CAA software since this is one of the types of question structures available in all of the main CAA software packages. For EMSQ’s such a transfer is not straightforward since the structure cannot be readily replicated in CAA software and the EMSQ’s would normally need to be converted into MCQ’s.

Conclusion

Although MCQ’s can be subject to problems such as cueing and poor distractors that are easily eliminated, these should be eliminated by adherence to good-practice and do not require using alternative questions structures such as EMSQ’s. MCQ’s are most commonly used to assess the basic category of Bloom’s taxonomy – knowledge (recall of information). However, this does not prevent MCQ’s from being used for the more advanced categories that assess the use and application of knowledge, and again this does not require the use of more complex question structures, particularly since these structures may distract the user and lead to underperformance. EMSQ’s may have a place in objective test assessments but there does not appear to be any evidence that this should be for the replacement of MCQ’s. The quality of a question lies in its content and design and not in any particular structure. A tool should not be castigated due to the possibility of its poor implementation, particularly when poor implementation is possible with alternative tools.

Communicating author Dr. R. Harper, Dept. Sport, Exercise & Biomedical Sciences, University of Luton, Park Square, Luton LU1 3JU, UK: Tel: 01582 734111; Fax: 01582 743400;  ray.harper@luton.ac.uk

 

References

Bloom B. S. (1972) Taxonomy of Educational Objectives: Handbook 1, Cognitive Domain. New York: Makay.

Brown, G., Bull, J. & Pendlebury, M. (1997) Assessing Student Learning in Higher Education. London: Routledge.

Bull, J. & McKenna, C. (2001) Blueprint for Computer Assisted Assessment. London: RoutledgeFalmer.

McCabe, M. & Barrett D. (2003) CAA Scoring Strategies for Partial Credit and Confidence Levels. Seventh International Computer Assisted Assessment Conference Proceedings, Learning & Teaching Development, Loughborough University. http://www.caaconference.com/. Accessed 20 August 2003.

Pritchett, N. (1999) Effective Question Design. In Computer-Assisted Assessment in Higher Education (pp. 29-37), Ed's Brown S., Bull J. & Race P.: London: Kogan Page Ltd.

Ricketts, C. & Wilks, S.J. (2002) Improving Student Performance Through Computer-based Assessment: insights from recent research. Assessment & Evaluation in Higher Education, 27(5), 475-479.

Runté, R. (1995). Basic rules for taking a multiple-choice test. http://www.edu.uleth.ca/Courses/Ed3604/take/mc/how.html. Accessed 20 August 2003.

Wilson, R. B. &  Case, S. M. (1993) Extended Matching Questions: An Alternative to Multiple-choice or Free-response Questions. J. of Veterinary Medical Education 20(3). http://www.utpjournals.com/jour.ihtml?lp=jvme/jvme203/ExtendedMatchingQuestions.html. Accessed 20 August 2003.

Wood, E. (2003) What are Extended Matching Sets Questions?  BEE-j (1,1). http://bio.ltsn.ac.uk/journal/vol1/beej-1-2.aspx. Accessed 20 August 2003.