Research Article

The use of computer-based assessments in a field biology module

Glenn K Baggott and Richard C Rayne

School of Biological and Chemical Sciences, Birkbeck, University of London

Date received: 30/03/2007        Date accepted: 21/05/2007

Formative computer-based assessments (CBAs) for self-instruction were introduced into a Year-2 field biology module. These CBAs were provided in ‘tutorial’ mode where each question had context-related diagnostic feedback and tutorial pages, and a self-test mode where the same CBA returned only a score. The summative assessments remained unchanged and consisted of an unseen CBA and written reports of field investigations. When compared with the previous three year-cohorts, the mean score for the summative CBA increased after the introduction of formative CBAs, whereas mean scores for written reports did not change. It is suggested that the increase in summative CBA mean score reflects the effectiveness of the formative CBAs in widening the students’ knowledge base. Evaluation of all assessments using an Assessment Experience Questionnaire indicated that they satisfied the ‘11 conditions under which assessment supports student learning’. Additionally, evidence is presented that the formative CBAs enhanced self-regulated student learning.

Introduction

There is a consensus in the environmental and biosciences that fieldwork has considerable merits as learning experience for students (Kent et al, 1997; The Higher Education Academy: Centre for Bioscience, 2004). As well as providing exposure to the ‘hidden curriculum’ of interpersonal skills and self-management (Andrews et al, 2003; Boyle et al, 2003) it also provides a rich source of experiential learning. Consequently, a current concern is its disappearance from the undergraduate curriculum (Smith, 2004; HUBS, 2006). With diminished opportunities for fieldwork it seems essential that fieldwork modules should be as productive as possible, indeed students themselves, whilst always positive about the experience (Boyle et al, 2003), recognize there are ways in which the learning experience could be further enhanced (Besenyei et al, 2003). Enhancements range from a staged sequence of enquiry-based exercises (Panizzon and Boulton, 2004) through use of mobile devices to utilise travel time effectively (Elkins and Elkins, 2006). In this paper we report on the introduction of computer-based assessments (CBAs) for formative purposes into a Year-2 undergraduate module in field biology. These formative CBAs were used for self-instruction so providing an automated ‘tutorial’ element to complement the extensive tutor-student and peer dialogue characteristic of a field-based module. In this way we introduced a greater element of self-regulation of student learning than previously (Nicol and Macfarlane-Dick, 2006).

In changing this module we integrated the formative CBAs fully with other assessments and ensured that the automated feedback provided by the CBAs was consistent, as far as practicable, with current recommendations for effective feedback (Gibbs and Simpson, 2004—05). Consequently, CBAs were intended to satisfy the relevant ‘conditions’ of the ’11 conditions under which assessment supports learning’ (Gibbs et al, 2003; See Appendix: Supplementary material 1) and all assessments were evaluated within the framework of Gibbs & Simpson’s (2003) Assessment Experience Questionnaire.

Module structure

This module is the students’ first introduction to ecology and data analysis, although all previously had completed a module about report writing. Students attend five teaching sessions one month before a residential field course, where they undertake fieldwork. Three sessions address the basic ecological theory underpinning the fieldwork; one session is used to practice data analysis techniques used for fieldwork; and one session is devoted to 5-minute oral presentations by pairs of students. Students devise agreed peer assessment criteria for these presentations. At the residential field course students undertake three field investigations (intertidal communities of the rocky shore, plant communities, freshwater communities) on three separate days and a project lasting 2 days. All fieldwork is undertaken in groups of 3—4 students with laboratory sessions every evening. In these sessions data collation and analysis is undertaken collaboratively within groups and with tutor-student discussions. The project also provides extensive opportunities for tutor and peer dialogue. Project outcomes are communicated orally to fellow students by 10-minute presentations; these are assessed by peers using the criteria developed in the pre-field course session.

The assessment has both formative and summative elements. In 2003—05, students were issued with a CD containing formative CBAs during the pre‑field course sessions; a second CD was issued at the residential field course. All CBAs were provided in a ‘tutorial’ mode where each question returned context-related diagnostic feedback and contained tutorial materials. The same CBAs were also delivered in a ‘self-test’ mode that returned a score without feedback. The CD’s also contained lecture handouts, field protocols and additional learning resources, including images.

From 2000 through 2005 summative assessments consisted of an unseen CBA and written reports. These reports, in the form of a scientific paper, were required for the three field investigations and the project. All required sampling methodology, graphical and statistical analysis as well as a discussion of the investigation in relation to the intended learning outcomes. Two months were allowed for preparation of the reports. The summative CBA took place two months after the residential field course and an individual analysis of answers was provided to each student as a diagnosis of strengths and weaknesses. After moderation by another examiner, the written reports were returned graded, with written feedback from the tutor, one month after submission. The feedback comprised annotations and a page of commentary identifying how effective each report was as a scientific communication and how well it had addressed the intended learning outcomes.

Methods

CBAs

CBAs were constructed using TRIADS software (CIAD, 2007). For each question students could access tutorial pages (Appendix: Supplementary material 2 and 3) and, on submitting their answer, diagnostic feedback was displayed providing hints for correct completion if an incorrect response had been given (Supplementary material 3). Model answers were not used. CBAs on the first CD were designed to promote learning of ecology basics and practice essential skills. CBAs on second CD were designed to support the field investigations. Assessment items were composed using the RECAP taxonomy (Imrie, 1995; Table 1) and grouped into CBAs by intended learning outcomes (Table 2). The items within each CBA on CD 1 were arranged to contain a preponderance of recall questions and distributed to address the learning outcomes of the pre-field course sessions. Similarly, CBAs on CD2 were distributed to cover the learning outcomes of the field investigations but used mainly comprehension and application items (Table 2). For the unseen summative CBA, assessment items covered all of the non-skills related learning outcomes and were distributed across RECAP categories and the module overall learning outcomes (Table 3).

Table 1 The criteria used to classify CBA items by cognitive type using the RECAP taxonomy (modified from Imrie, 1995)

Table 2 The number of CBA items on the CD’s categorized by the RECAP taxonomy in relation to intended learning outcomes. Each of the sets of specific learning outcomes indicated by the topic titles in bold, with gloss in italics, constituted a separate CBA.

Evaluation

A modified version of the Assessment Experience Questionnaire of Gibbs and Simpson (2003) was constructed using the questions about the examination, the quality, and the use of feedback. Only questions with high loadings in the main factors of their analysis were utilized (Gibbs and Simpson, 2003). The first questionnaire was administered anonymously immediately after the unseen CBA (Supplementary material 4) and the second with the return of the marked field reports and module grades (Supplementary material 5). The wording was altered where necessary to fit the assessments evaluated. Both questionnaires contained additional questions about learning resources. In the post-summative CBA questionnaire there were two extra questions about use of CBAs for examination practice; the second questionnaire also asked for views on the utility of assessments for future assessed work (Supplementary material 5). The responses are summarized as the proportions of respondents that agreed, disagreed or were neutral about the statements. Both questionnaires also asked what aspects of the ‘tutorial’ and ‘self-test’ CBAs were found most useful as a learning resource.

Table 3 The number of items, categorised by the RECAP taxonomy and overall learning outcomes, used in the summative CBA. Learning outcomes were: 1. to have an appreciation of the biological diversity of animals and plants; 2. to have gained specific knowledge about three types of ecological communities; 3. to have gained a working knowledge of how to analyse and present ecological data.

On the basis of the responses to this question answers were categorized into

1. responses indicating improvement of student learning — CBAs helped understanding of topics, CBAs helped respondent improve, respondent found diagnostic feedback useful, or respondent used the tutorial pages;
2. responses indicating preparation for the summative CBA — CBAs provided exam practice, respondent wanted model answers, respondent used CBAs for numeracy practice;
3. responses other than these categories - respondent valued 24x7 access; respondent liked interactivity and graphics.

Results

When compared with 2000—02, mean scores for the summative CBA increased after the introduction of formative CBAs, whereas mean scores for written reports did not change (Table 4). Mean scores for recall items were higher in 2004 (68.4±2.9%; mean±SEM) and 2005 (69.2±4.3%) than in 2003 (56.5±5.0%). Mean scores for comprehension and application items were higher than those for recall items (70.7±3.3% and 76.7±4.9%, respectively), and did not change from 2003 through 2005. Marks for the different types of written reports were all strongly correlated with each other (Table 5); although significant, coefficients were much smaller for the correlations between the summative CBA and written reports scores.

Table 4 Assessment scores (%) for summative CBA and written reports before and after the introduction of formative CBAs in 2003. Summative CBA means sharing superscripts differ at p<0.01 (Tukey post-hoc mean comparisons family error rate 0.05; one-way ANOVA for years F=3.92, p=0.003)

Table 5 Pearson correlation coefficients for summative assessments 2003 through 2005 (n=41). All significant at p<0.001.

A factor analysis of summative CBA item RECAP type (Table 3) and written report type was undertaken for 2003—05. With no rotation and eigenvalues greater than 1, this analysis identified two factors accounting for 69.3% of the variance, with the first four factors accounting for 89.0% of the total variance. For factor 1 the loadings were comprised of two groups. High scores for factor 1 were associated with coursework items and comprehension questions (Table 6). Other examination items contributed less to the factor 1 score. Factor 2 divided into coursework and examination items: high scores for factor 2 were associated with high coursework percentages and low exam item percentages. There were no differences between the cohorts 2003—05.

Table 6 Principal component factor analysis with item loadings for all forms of assessed written reports and summative CBA items categorized by the RECAP taxonomy, for 2003 through 2005 (n=41).

Evaluation

In all years all students used the formative CBAs. The majority, 73%, used them very frequently or frequently; there was no difference reported in usage at the two survey times. The responses to the Assessment Experience Questionnaire (Table 7) indicated that the majority of students felt both modes of summative assessment, unseen CBA and written field reports, helped to improve their understanding of content, expanded their knowledge (‘…brought things together for me’; ‘I understand things better…’), and was an authentic assessment of their understanding (‘…you can’t get away with not understanding and still get good marks’). A majority of students claimed to have utilized both CBA and written feedback (‘I read the feedback carefully…’), that feedback assisted understanding (‘The feedback helped me to understand…’), and that feedback helped them improve (‘I can see from the feedback what I need to do to improve’). In the case of ‘tutorial’ CBAs a majority of respondents claimed not to have understood most of the feedback, yet most said it had helped them ‘..understand things better’.

Table 7 Student views about different modes of assessment for year cohorts 2003—05. In all years questionnaires were administered immediately after the summative CBA and after return of marked field reports and final module grade.

AFTER SUMMATIVE CBA				AFTER RETURN OF MARKED REPORTS
n =35	AGREE	DISAGREE	NEUTRAL	n =22	AGREE	DISAGREE	NEUTRAL
Views about the computer-based examination				Views about the field reports
Doing the exam brought things together for me.	57a	11	31	Doing the field reports brought things together for me.	90	5	5
I learnt new things while preparing for the exam.	91	0	9	I learnt new things while preparing for the field reports.	95	5	0
I understand things better as a result of the exam.	56	11	33	I understand things better as a result of the field reports.	89	6	6
The electronic tutorials/self-tests helped me prepare for this examination.	86	3	11
Preparing for the exam was not mainly a matter of memorising.	56	28	17
After the exam I’ll probably remember most of what I learnt.	66	6	29
In exams I can’t get away with not understanding and still get good marks.	89	0	11	In writing field reports you can’t get away with not understanding and still get good marks.	86	0	14
Views on the feedback from electronic tests				Views on the feedback from field reports
The feedback provided by the electronic tutorials/self-tests helped me prepare for this examination.	85	12	3
I read the feedback carefully and tried to understand what the feedback is saying.	94	0	6	I read the feedback carefully and try to understand what the feedback is saying	100	0	0
The feedback prompted me to go back over material.	91	0	9	The feedback prompted me to go back over material	71	14	14
I did use the feedback for revising.	86	3	11
The feedback helped me to understand things better.	82	3	15	The feedback helped me to understand things better.	86	0	14
I understood most of the feedback.	14	74	11	I understood most of the feedback.	70	15	15
I can see from the feedback what I need to do to improve.	68	12	21	I can see from the feedback what I need to do to improve.	73	14	14

a. Percentage of responses. Students were asked to indicate whether they strongly agree, agree, disagree, strongly disagree or were neutral about each statement. In this table responses for strongly agree and agree have been aggregated, as have strongly disagree and disagree categories. For statements in italics the wording used was phrased negatively (Supplementary Material 1 and 2). In his Table these statements are rephrased positively so that ‘agree’ indicates that the respondent felt their learning experience had been improved.

Students were also asked what aspects of the ‘tutorial’ and ‘self-test’ CBAs did they find most useful as a learning resource. When surveyed after the summative CBA, the most frequent responses indicated that the formative CBAs provided exam practice (28%), followed by respondents liking diagnostic feedback (15%) and the opinion that CBAs helped understanding (13%). When surveyed after return of the reports, the most frequent responses indicated that formative CBAs helped the understanding of topics (36%), followed by the opinions that CBAs provided exam practice (13%) and that CBAs helped the respondent improve (13%). When these responses were categorised into either a) the formative CBAs helped understanding or b) they provided examination practice, and surveyed after the summative CBA, more students said the CBAs helped them prepare for the unseen CBA (61%) than assisting their understanding the topics (39%) (c² = 6.25, 1 d.f., P=0.01). In contrast, when surveyed after return of the reports, only 23% said CBAs were used in preparation for the summative CBA. Correspondingly, representative student comments emphasizing the utility of ‘tutorial’ CBAs in assisting understanding were,

‘the visual learning helped comprehension…was most useful especially the feedback where comprehension had been less than full’.

‘…learnt in my own time with tutorials and feedback errors…seems a more active way of learning — more enjoyable’.

In contrast, typical comments emphasizing the use of formative CBAs for summative CBA preparation,

‘Good preparation for the types of questions that would be asked..’

‘it would be helpful …to include the option of displaying the correct answer.’

Students also reported that both ‘tutorial’ and ‘self-test’ CBAs helped them regulate their own learning. Typical comments included,

‘The combination was good. I could do a self test and then go back over the tutorials.’

‘…showed me where I should concentrate…allowed me to test myself without the pressure of having a mark…’

‘…I could use them in my own time and go over tricky bits as often as I liked...’

‘…being able to do them whenever…were interactive — not just reading.’

‘…having a structured self-test...’

When asked, after the return of marked reports, whether the feedback from assessments would assist in future assessment tasks, 90% agreed it would help them to write laboratory reports, 63% agreed it would help prepare for examinations, and 76% felt it would help them write essays.

Discussion

The availability of formative CBAs in 2003—05 was associated with a significant increase in summative CBA mean score but not the mean score for written reports (Table 4). This was despite that the second set of CBAs specifically addressed the field investigations (Table 2) and student comments indicating their use of formative CBAs to master the field investigations. The increase in summative CBA mean score was due to higher scores for recall questions, although mean scores for these were still lower than for comprehension or application questions. Thus it is possible that formative CBAs were facilitating surface learning strategies through recall of facts. In contrast, the various types of written report required higher cognitive skills than recall, and indeed the summative CBA overall scores were poorly correlated with scores for the different types of report (Table 5). However, the comprehension questions of the unseen CBA had a factor loading similar to the written reports (Table 6) indicating the likelihood that these items assessed student cognitive skills akin to those assessed by written reports.

A reason for this disparity may be indicated by student comments about their use of formative CBAs. Whilst most students said they used formative CBAs to prepare for the summative CBA (Table 7), the comments suggested students were of two types: those who perceived the formative CBAs specifically as training for the unseen CBA, and those who found they assisted in understanding the module content. Conceivably, then, these two groups may correspond to those employing surface and deep learning strategies, as has been often described in the literature (e.g. Biggs, 1993; Beishuizen and Stoutjesdijk, 1999; Karabenick and Collins-Eaglin, 1997; Scouller, 1998). However, it is striking that one month after the summative CBA, when compared to immediately after it, significantly fewer students commented that the formative CBAs served as practice for the unseen CBA. We suggest that, rather than formative CBAs simply promoting surface learning, they were markedly effective in structuring the students’ access to disparate sources of information and so improving the students’ knowledge base. Certainly, the majority of students agreed that the feedback from all assessments, and specifically both ‘tutorial’ and ‘self-test’ CBAs, showed them how they could close the gap between the learning outcomes and their performance.

The automated feedback of the formative CBAs, as well as written feedback, were used as successful tactics to promote the ’11 conditions under which assessment supports student learning’ (Gibbs and Simpson, 2003). These tactics are summarized in Table 8. Student responses after the summative CBA indicated that the automated feedback had the desirable characteristics of being used, promoting understanding and helping students improve their learning (Table 7); written feedback was also perceived to have the same characteristics. The only condition apparently not fully achieved was condition 9 (Table 8). When surveyed just after the summative CBA, a majority of students felt they didn’t understand most of the automated feedback yet said it helped them understand things better and helped them improve (Table 7). This inconsistency arose, we suggest, from the structure of the automated feedback. Model answers to questions were deliberately not supplied; rather, hints for successful completion were provided, although by following the hints a correct completion would be achieved. The small number of comments requesting model answers would also be consistent with this interpretation. Thus we suggest the agreement with the question ‘I don’t understand some of the feedback’ (Supplementary material 1) reflects the formative structure of the feedback.

Table 8 The tactics used to attain the ‘11 conditions under which assessment supports student learning’ (after Gibbs & Simpson, 2003).

The formative CBAs and module activities proved successful in promoting the principles for self-regulation of student learning (Nicol and Macfarlane-Dick, 2006). As Table 9 indicates, formative CBAs helped promote five of the seven principles with the encouragement of teacher and peer dialogue relying on the main strength of field-based teaching — the extensive opportunities for group work and discussion. Whilst arguably fieldwork modules are the best opportunity to develop student self-regulation of learning, as an integral part of the ‘hidden curriculum’, the evidence from student comments suggests it is possible to enhance this by the appropriate use of formative CBAs. Indeed, digital technology can be very effective in promoting self-regulation of learning (Nicol and Milligan, 2006). Thus there is further scope to make field-based modules even more effective for learning in biology.

Table 9 The extent to which the module characteristics promoted student self-regulation of learning.

1. Nicol and Macfarlane-Dick (2006)

In summary, the introduction of formative CBAs with feedback into a field-based module was associated with an increase in mean score for the summative CBA; mean scores for written reports did not change. We conclude this reflects the increased opportunity for students to widen their knowledge base. The use of automated feedback was an effective tactic to achieve the ‘11 conditions under which assessment supports student learning’ (Gibbs et al, 2003) and there was evidence that the formative CBAs enhanced self-regulated learning, a common characteristic of field-based modules.

Acknowledgement

This study was conducted with funding from HEFCE FDTL4 for the ‘Online assessment and feedback’ (OLAAF) project. We thank Ellen Howey and Caroline Matute for constructing the TRIADS assessments. We also wish to thank Don Mackenzie of CIAD for his generous and rapid help in troubleshooting TRIADS software problems.

Communicating Author

Dr Glenn K Baggott, School of Biological and Chemical Sciences Birkbeck, University of London, Malet Street, LONDON WC1E 7HX. Tel:020 7631-6244 Fax:020 7631-6246 Email: g.baggott@bbk.ac.uk

References

Andrews, J., Kneale, P., Sougnez, W. , Stewart, M., and Stott, T. (2003) Carrying out Pedagogic Research into the Constructive Alignment of Fieldwork. Planet, issue 11, available at http://www.gees.ac.uk/pubs/planet/index.htm (accessed 10 February 2007)

Beishuizen, J. J. and Stoutjesdijk, E. T. (1999) Study strategies in a computer assisted study environment. Learning and Instruction, 9 (3), 281-301

Besenyei, L., Watkin, G., and Oliver, K. (2003) An evaluation of the educational effectiveness of fieldwork within environmental science awards at the University of Wolverhampton, CELT Learning and Teaching Projects 2003/04 available at http://hdl.handle.net/2436/3694 (accessed 10 February 2007)

Biggs, J. B. (1993) What do inventories of students' learning processes really measure? A theoretical review and clarification. British Journal of Educational Psychology, 63, 3-19

Boyle, A, Conchie, S., Maguire, S., Martin, A., Milsom, C., Nash, R., Rawlinson, S., Turner, A., and Wurthmann, S. (2003) Fieldwork is Good? The Student Experience of Field Courses. Planet, issue 11 available at http://www.gees.ac.uk/pubs/planet/index.htm (accessed 10 February 2007)

Centre for Interactive Assessment Development (CIAD). (2007) CIAD Home. http://www.derby.ac.uk/ciad/ (accessed 29 March 2007)

Elkins, J.T. and Elkins, N.M.L. (2006) Improving Student Learning During Travel Time on Field Trips Using an Innovative, Portable Audio/video System. Journal of Geoscience Education, 54 (2), 147-152

Gibbs, G and Simpson, C (2003) Measuring the response of students to assessment: the Assessment experience Questionnaire. 11th Improving Student Learning Symposium, pp 1-12

Gibbs, G and Simpson, C (2004-5) Conditions under which assessment supports students’ learning. Learning and Teaching in Higher Education Issue 1, 3-31

Gibbs, G., Simpson, C. and Macdonald, R. (2003) Improving student learning through changing assessment — a conceptual and practical framework. European Association for Research into Learning and Instruction, EARLI Conference, Padova

Heads of University Biological Sciences (HUBS) (2006) Curriculum and the Benchmark/Teaching Field Biology, available at http://www.biohubs.org.uk/ (accessed 2 February 2007)

Imrie, B. W. (1995) Assessment for learning and taxonomies. Assessment & Evaluation in Higher Education, 20 (2), 175-189

Karabenick, S. A. and Collins-Eaglin, J. (1997) Relation of perceived instructional goals and incentives to college students' use of learning strategies. Journal of Experimental Education, 65 (4), 331-341

Kent, M., Gilbertson, D.D., and Hunt, C.O. (1997) Fieldwork in geography teaching: A critical review of the literature and approaches. Journal of Geography in Higher Education, 21 (3), 313-332

Nicol, D.J. and Macfarlane-Dick, D. (2006) Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher education, 31 (2), 199-218

Nicol, D. J. & Milligan, C. (2006), Rethinking technology-supported assessment in terms of the seven principles of good feedback practice. In Innovative Assessment in Higher Education, eds. Bryan, C. and Clegg, K., pp 1-14. London, UK: Taylor and Francis.

Panizzon, D. L. and Boulton, A. J. (2004) Strategies for enhancing the learning of ecological research methods and statistics by tertiary environmental science students. Bioscience Education e-Journal, volume 4 available at
http://www.bioscience.heacademy.ac.uk/journal/vol4/beej-4-1.htm (accessed 14 February 2007).

Smith, D. (2004) Issues and trends in higher education biology fieldwork. Journal of Biological Education, 39 (1), 6-10

The Higher Education Academy: Centre for Bioscience (2004) Fieldwork, available at http://www.bioscience.heacademy.ac.uk/events/reports/fieldwork.htm (accessed 25 May 2007)

Appendix

An appendix of supplementary material is available as a PDF document only