Evaluation: is an open book examination easier?

Dr Richard Brightwell¹, Ms Janine-Helen Daniel and Dr Angus Stewart²

School of Biomedical & Sports Science, Edith Cowan University, Perth
¹r.brightwell@ecu.edu.au  ²a.stewart@ecu.edu.au

Received; accepted:12/03/2004

In terms of the assessment of students, it is generally accepted that open book examinations create an enriched environment, offering the student an opportunity to better understand and respond to a particular question. The present study provided the opportunity to examine this assumption and test it in a controlled manner.

During the study of an introductory University Anatomy and Physiology unit, students were required to complete two assessments. The first assessment was an online multiple-choice examination with 50 questions either reviewing concepts or requiring critical thinking and clinical applications. This test was first undertaken by students (N=196) in class time. The results were analysed and the opportunity was offered for the students to repeat the test with the assistance of their textbooks.  Upon repeating the test under these conditions it was shown that there was no significant difference between the means of the scores.

It was concluded that a suitably constructed set of questions could be used to discriminate student abilities in either an open or closed book environment.

Introduction

The focus of many educators has been to maximise the usefulness of every academic activity, whether these activities cause high- or low-anxiety levels amongst partaking students. With this in mind, the current perception of assessments is that of a learning tool, both able to objectively rank students according to ability, and to enhance and enrich the learning environment (James, McInnis, and Devlin, 2002; Feller, 1994). Over an extended period of time, open book assessments have fulfilled both the requirements of an assessment tool and a student-centred approach to education, as a technique that reduces the level of anxiety experienced by participating students. This is thought to result in more comprehensive student examination preparation, and hence more consistent learning throughout the course of study, with students avoiding "cramming" (Theophilides and Dionysiou, 1996; Theophilides and Koutselini, 2000).

It is generally accepted that the perceived "worth" of an open book assessment is directly related to its ability to expose higher order "thinking" skills of candidates. For example, studies have suggested that assessments designed for open book completion prompt students to exhibit their levels of skill in analysing, synthesising, and evaluating course materials and designated learning outcomes in a measurable and mainly standardised fashion. However, possibly because the level of difficulty is raised, the marks achieved by students in open book examination have been shown to be no higher than those obtained under closed book conditions (Ioannidou, 1997; Francis, 1982; Theophilides and Dionysiou, 1996).

There are many possible explanations for these results, however, one study found that achievement levels at the upper end of the scale ('B' grade students) were greater for those assessed on the higher cognitive skills in an open book fashion than those achieved by students in the "traditional" manner (Francis, 1982).

When students are surveyed, they almost always prefer open book assessments as opposed to closed book assessments, despite the acknowledged understanding that open book examinations generally require the exhibition of higher order thinking skills. Studies have shown that students perceive five main benefits in open book examinations, namely practice in the creative use of course content, course content proficiency, increased opportunity for student self-evaluation and feedback, less examination stress and greater student regulation of content studied. The same study showed that educators perceive open book examinations as providing the opportunity to promote thinking rather than memorising. Therefore relaxing the environment within which students learn encourages student ownership of study, and increases self-evaluation with respect to the level of achievement of stated course outcomes (Theophilides and Dionysiou, 1996).

 In addition, open book tests are thought to more adequately represent real life situations, where problems are encountered and (virtually) limitless resources are available for the development of useful solutions to the dilemma (Feller, 1994).

Real life situations for which educators aim to prepare their students (Feller, 1994), can be competently managed by the use of the six classes of behaviour outlined in Bloom's Taxonomy (Anderson and Sosniak, 1994). These six classes are widely recognised and have been the stimulus for some of the greatest changes in education (including assessment protocols). The taxonomy is divided into six levels, progressing from knowledge-based to evaluation-based behaviours. In this way, educators seek to prepare students for life by facilitating practice opportunities where all Bloom's behaviours are integrated in the solving of problems similar to those encountered in real situations. It has been proposed that open book assessments achieve this aim by mimicking real life situations, hence these assessments would deserve a place in most assessment protocols (Feller, 1994). 

Subjects

The test population consisted of 196 first semester Bachelor of Science students at Edith Cowan University (ECU). ECU is a relatively new university which is located in the city of Perth, Western Australia. The university has more than 20 000 students undertaking study in over 160 courses at both undergraduate and postgraduate levels. In addition to Australian students, the university has over 1,600 international students, who originate from over 60 countries. Students participating in the study were located at the University's main campus, sited at Joondalup, a suburb in the northern corridor of Perth. The test students were completing an introductory Anatomy and Physiology unit, and received much of their course material on-line. Prior to the first assessment, the students had received at least 6 weeks of instruction. No revision sessions were offered, however students were provided with the opportunity to complete on-line course revision quizzes if they so chose.

On-line Course Materials

On-line course materials were offered to students using the Pearson Education, Inc. CourseCompassT facility, powered by Blackboard^©. Students received a student access card for Fundamentals of Anatomy and Physiology, Martini, 5^th Edition, Prentice Hall, during the first lecture of the course, and were instructed how to set up a CourseCompassT account. Students accessed the site www.CourseCompass.com and using the student access code each individual set up an account with a unique student logon and password.

CourseCompassT provided the unit coordinator with an in-built assessment facility, allowing the assessment for this study to be delivered and completed by students on-line. In addition, CourseCompassT recorded the time required (beyond the set limit of one hour) for each student to complete the assessment. Other CourseCompassT information helped to define student study patterns by providing information such as the number of times each student accessed the facility, which feature each student used most, and completion of practice quizzes.

Assessment

The assessment used for the present study consisted of an on-line multiple choice test, comprising 50 questions and worth 30% of the students' overall grades. The test consisted of 44 questions equivalent to the Bloom's Taxonomy level 2 (comprehension-style questions), and 6 questions equivalent to the Bloom's Taxonomy level 3 (application-style questions). (Appendix 1)

The test was administered under similar conditions for the closed and open book sittings. During normal tutorial times, students were taken to a computer laboratory where they were instructed to logon on to CourseCompassT. Students were then provided with a tutorial unique password (set by their normal tutor) to allow access to the assessment site. A time limit of one hour was set for students, within which they were expected to complete the test. This allowed for one minute per question, with 10 minutes revision time.  Each student first completed the assessment in a closed book setting, and then approximately one week later, completed the same test in an open book setting. After the first test the students were told that they would be asked the same set of questions, under the same conditions but with full access to any books they may require.

Results

The test results were collected using the automated assessment tool provided within the CourseCompassT program utilised by the School of Biomedical & Sports Science at ECU. The data were analysed using a t-test (two-sample assuming unequal variances, refer to Table 1 for a summary). There was no significant difference (p >0.05) between the mean scores achieved by students in the closed and open book sitting of the test (Figure 1, Table 1).

Table 1.  Student marks. Significance (p >0.05) was assessed using a two-sample students t-test assuming unequal variances.

	Closed Book	Open Book
Minimum	11	15
Maximum	44	44
Mean	27.46428571	28.2704816
SEM	6.27929034	6.096097267
Number Completed	196	196
p value (when comparing closed to open book results)	0.0989858528995594

Closed Book

The minimum and maximum scores for the closed book sitting of the test were 22% and 88% respectively, with a mean of 55%. 

Open Book

The minimum and maximum scores for the open book sitting of the test were 30% and 88% respectively, with a mean of 57%.

Time

The time taken by students to complete the open book assessment, over and above the time limit of 60 minutes, was recorded by CourseCompassT. One hundred and sixty-five students completed the test within the set time limit,

while 28 students required additional time to complete the assessment. Of the 196 students, the timing for 3 students did not correctly register with CourseCompassT. There was no significant difference between the mean score of those who completed the open book assessment within the time limit and those requiring more time to complete the assessment (Figure 2, Table 2).

Table 2. The influence of time on students marks in the open book sitting.

Significance (p >0.05) was assessed using a two-sample students t-test assuming unequal variances.

	<60 minutes		>60 minutes
	Marks	Marks	Time (sec)
Minimum	15	18	9
Maximum	44	39	614
Mean	28.10909091	28.78571429	97.71428571
SEM	6.126728907	6.087979309	142.7561654
Number Completed	165	28	28
p value (when comparing <60 to >60 min. results)	0.295101736478886

Students who completed the open book test in <60 minutes

The exact time taken for these students to complete the closed book tests was not recorded. The minimum and maximum scores for the closed book sitting of the test were 30% and 88% respectively, with a mean of 53%.

The minimum (30%) and maximum (88%) scores for the open and closed book sittings of this group of students were exactly the same. However, the mean score of the open book test (56%) was not significantly different to that of the closed book (55%). (Figure 3). Students who completed the open book test in >60 minutes

The minimum time over the 60 minute limit required to complete the open book test was 9 seconds and the maximum time was 10 minutes 14 seconds. The average time taken over the limit was 1 minute 37 seconds.

The minimum and maximum scores for the closed book sitting of the test were 22% and 78% respectively, with a mean of 56%.

The minimum score for the open book sitting of the test was 36%. The maximum score for the open book sitting of the test (78%) was exactly the same as the closed book score. However, the mean score of the open book test (58%) differed, non significantly ( p> 0.05) from that of the closed book test (56%). (Figure 4).

Discussion

The study is unique in that it utilises a recently developed multi-media instructional, support, and evaluation platform, namely the Pearson Education, Inc. CourseCompassT facility powered by Blackboard^©. A traditionally styled test (Feller, 1994; Ioannidou 1997) was used as a convenient mechanism for investigating the influence of open book testing (James et al., 2002). The results support previously published conclusions: that is in a test where questions cover up to Bloom's Level III cognitive skills, open book tests generally do not result in statistically significant differences in student marks and therefore their achievement levels (Ioannidou, 1997). 

The very similar overall results obtained in the closed book and subsequent open book test were surprising. However it is difficult to adequately compare open and closed book assessments, as the two styles generally test different metacognitive skills (Feller, 1994; Francis, 1982). Research has shown that generally students do not adequately prepare for open book tests (Boniface, 1985; Theophilides and Koutselini, 2000), which may explain why students' marks did not increase for the second sitting of this test. In addition, research has shown that students feel less anxiety when preparing for and completing open book assessments (Francis, 1982; Boniface, 1985; Feller, 1994; Ioannidou 1997; Theophilides and Dionysiou, 1996). Without further study, it is impossible to determine whether or not this may have contributed to a less concentrated approach to the second sitting of the assessment, however this cannot be overlooked as a reason. The data also reveal that an open book setting tended to allow a non-significant improvement in the marks of the weaker students.

It is interesting to note that students spending more than 60 minutes on the assessment did not demonstrate any significant improvement in marks. While there was an increase in the maximum mark achieved by students completing under this time. This outcome should be investigated in a further study, as it is possible that the increased time may be counter-productive for students who excessively revise their answers.

 Open book testing has been shown to result in more consistent and varied study, with students consulting a variety of sources to achieve their desired level of proficiency with regard to stated course outcomes (Francis, 1982; Theophilides and Koutselini, 2000). It is likely that this perceived benefit motivates course coordinators to include open book examinations in the assessment schedules for tertiary courses of study. The popularly held concept that the open book examination is "easier" than the closed book examination (Francis, 1982; Boniface, 1985; Feller, 1994; Ioannidou 1997; Theophilides and Dionysiou, 1996) are not supported by the present study. Therefore, reservations with regard to the potential objective and discriminatory assessment of students using the open book examination have been substantially allayed.

References

Anderson, L., and Sosniak, L. (1994) Bloom's Taxonomy: a forty year retrospective. Chicago: University of Chicago Press.

Boniface, D. (1985) Candidates' use of notes and textbooks during an open-book examination.
Educational Research, 27(3), 201.

Feller, M. (1994) Open-book testing and education for the future. Studies in Educational Evaluation, 20(2), 235-238.

Francis, J. (1982) A case for open-book examinations. Educational Review, 34(1), 13-26.

Ioannidou, M. (1997) Testing and life-long learning: open-book and closed-book examination in a university course. Studies in Educational Evaluation, 23(2), 131 -139.

James, R., McInnis, C., and Devlin, M. (2002) Assessing learning in Australian universities. [Online] Available: www.cshe.unimelb.edu.au/assessinglearning/index.html [2002, September].

Theophilides, C., and Dionysiou, O. (1996) The major functions of the open-book examination at the university level: A Factor Analytic Study. Studies in Educational Evaluation, 22(2), 157-170.

Theophilides, C., Koutselini, M. (2000) Study behaviour in the closed-book and the open-Book examination: a comparative analysis. Educational Research and Evaluation: An International Journal on Theory and Practice, 6(4), 379 (Abstract only.).

Copyright

© 2003 Dr Richard Brightwell_, Ms Janine-Helen Daniel and Dr Angus Stewart,

The authors assign to UniServe Science and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to UniServe Science to publish this document in full on the Web (prime sites and mirrors) and in printed form within UniServe Science 2003 Conference proceedings. Any other usage is prohibited without express permission of the authors.

Appendix 1

Sample questions from the online test.