Research Article

Measuring assessment: a methodology for investigating undergraduate assessment

A.C. Crook¹ and J.R. Park²

¹Centre for the Development of Teaching and Learning, The University of Reading, UK, ²School of Agriculture, Policy and Development, The University of Reading, UK

Date received: 27/09/2004 Date accepted: 29/10/2004

 

For many degree programmes, including those within the School of Agriculture, Policy and Development (SAPD) at the University of Reading, modularisation has resulted in increased flexibility of degree programmes and a wider choice of modules offered to students. An important consequence of this is that module co-ordinators and programme directors may no longer have the same detailed knowledge of the amount, timing and diversity of assessments (coursework) being undertaken by their students. This situation may be further compounded when students study modules that are co-ordinated by a number of different Schools. The first aim of this project was therefore to provide a systematic methodology for analysing the nature and timing of assessments from a range of degree programmes using 'electronic assessment diaries', which we designed for this research. These diaries, completed by undergraduate volunteers, recorded details of all coursework assessments, module by module, for the autumn and spring terms 2003/4. A further aim of the research was to provide information to Course Directors to enable a more pedagogic and informed approach to future changes in assessment practices. The discussion reflects on the use of the electronic diaries and the provision of assessment- related information to programme directors and lecturing staff.

Introduction

Assessment, defined as a form of evaluation or appraisal, is probably one of the most important aspects of the education process and has often been described as "the tail that wags the curriculum dog" (Hargreaves, 1989). It is recognised that assessment in its various forms will often determine a considerable proportion of the independent study students undertake during their undergraduate degree, yet students may sometimes view assessment as being "inauthentic, pointless and another hurdle to jump over.." (Ramsden; in Norton et al., 2001). Perhaps more significant is the fact that assessment can often drive what, when and how students learn (Brown et al., 1994; Biggs, 1999). However, despite this, in many disciplines, assessment is one of the most under-researched areas of higher education (Wakeford, in Fry et al., 1999).

What, when, how and how much to assess student learning are important questions to consider when designing degree programmes and modules, particularly those which may be taught by a number of academic staff, each of whom may wish to assess the topic(s)/skills they have taught. The move to modularisation at the University of Reading in which students have greater flexibility of module choices (and degree programmes) means that it is often difficult for programme directors to know the precise details of the overall assessment (coursework) 'profile' of different degree programmes. As a result, assessment issues may occur, including: an increased volume of assessment; repetitive assessments, i.e. the same skills being repeatedly assessed in different modules, whilst other skills are effectively 'ignored'; and 'bunching' of assessment deadlines, which is potentially problematic for students but also for staff in terms of finding time to provide quality, timely feedback.  Indeed, in a recent internal web-based survey at Reading, 49% of respondents (lecturers) stated that modularisation has had an impact upon the way in which they assess undergraduates (Crook, 2003); in particular this manifested itself in terms of an increase in the amount of assessed coursework and a less detailed knowledge of what (and how) students are being assessed because of the increased range of modules they are now able to study. A good example of this is the Environmental Sciences degree programmes at Reading in which students may study modules from five different Schools.

Further common issues relating to assessment, which may occur in higher education institutions include:

1. the belief that some students are being over-assessed whilst others are perhaps under-assessed;
2. that assessments may not always be appropriately aligned with module learning outcomes;
3. that students may not be experiencing a sufficiently diverse range of assessments, particularly in terms of encouraging them to become self-directed, independent learners.

This research investigates some of these issues in relation to students studying five different degree programmes based within the School of Agriculture, Policy and Development at the University of Reading.

Project Aims

The primary objective of this research was to provide a systematic methodology for more effective monitoring of assessment loads, timing, diversity and appropriateness (regarding module learning outcomes). In particular, the project was designed with an overall view to facilitating a more informed and pedagogic approach to future decision-making regarding undergraduate assessment and feedback provision. A further research aim was to analyse coursework assessment in the context of modularisation within and between modules for a number of degree programmes within the School of Agriculture Policy and Development (SAPD). The specific objectives were to evaluate:

1. the diversity of assessment experienced by students, i.e. methods of assessment, type of assessment (summative/formative, individual and group assessment);
2. timing of assessments (submission dates within/after the module);
3. the number of assessments (volume) within and between modules per degree programme.

It was envisaged that the generation of 'real' assessment data would not only lead to a greater awareness of student assessment but should also result in the rationalisation of assessments within and between modules, which in turn, has the potential to 'free-up' staff time.

Methods

There were two alternatives available for collecting undergraduate assessment data; the first was to make use of existing formal module descriptions, which are available on the University website, and the other was to actively involve students and ask them to record their assessment experiences for each module throughout the year. The latter method was utilised for two principal reasons:

1. module descriptions do not always provide specific details of all assessments, e.g. they do not necessarily provide information on either assessment submission dates and/or formative assessments that do not contribute to the module marks, nor do they necessarily provide details of group assessment opportunities within modules;
2. the requirement to collect data on the actual amount of time students spend preparing for assessments, which obviously cannot be ascertained from the module descriptions.

Programme directors within the School were asked to identify students whom they considered might be interested in participating in this study; as a result, thirteen Part two (second year) student volunteers representing five degree programmes (spanning science and management) were recruited for this research. The degree programmes represented were: Rural Environmental Sciences (RES), Agricultural Business Management (ABM), Agriculture (Ag), Rural Resource Management (RRM) and Applied Biology (AB). Part two students have at least one year's experience of University assessments and it was thought using this year group might encourage students to participate in the project because they might benefit in Part three from any changes to the School's assessment practices as a result of our research. Following an initial email contact, these students were invited to participate in a workshop (in early October 2003) and the following issues were discussed:

1. an outline of our proposed research plans and how the student data would contribute to the School's future review of assessment practices;
2. the issue of student anonymity and the fact that an individual's data would always remain anonymous in any subsequent dissemination of the project;
3. how the students would be rewarded for their contribution to the project (individual book tokens each term).

This meeting was also used as an opportunity to discuss the 'electronic assessment diary' with the students, which was designed in Excel specifically for this project (see Figure 1).

Figure 1: An outline of the electronic assessment diary created for this research project showing how students record individual module data.

Students were briefed on how to enter data into the assessment diary and were given annotated handouts to show the layout of the diaries. The workshop also provided an opportunity for students to ask questions concerning the use and implementation of the diary data. At the end of the workshop each student was given a copy of the electronic diary on floppy disk. We used floppy disks because i) it was logistically easier than making use of an Internet-based diary, which would have required students to remember access passwords, and ii) because all of the students had regular access to a PC with a floppy disk facility. On each student's disk an assessment diary template had been set up with his or her individual module information pre-entered; each module is represented by a single worksheet within a single Excel file (Figure 1). Students were required to keep a record of their assessments and to add the required data into the relevant module worksheet as they progressed through the term. For each individual piece of assessment the students were required to enter as much of the following data as possible:

1. the week the assessment was assigned;
2. the week the assessment was due to be submitted;
3. the total number of hours they spent preparing for the assessment (defined as time spent researching the assessment plus time spent actually preparing the assessment for submission);
4. the percentage contribution of the assessment to the final module marks (students were asked to enter 0% when assessments were purely for feedback purposes only, i.e. formative assessment);
5. whether or not the assessment was 'individual' or based on group work.

Inbuilt into each Excel diary were two additional worksheets, 'Time Plan' and 'Master Sheet' (Figures 2 and 3, respectively), which were invisible to students ('hidden' within the Excel file) but which were linked to the student's diaries so that every time a student entered new data, these were automatically collated and summed within the time plan and master sheets. This enabled a rapid review of a student's assessment profile during the various interim meetings that were held with them throughout the year and it also facilitated data collation and analysis. Additional information was collected centrally on the University entry grades of the students in the cohort and their performance in individual modules during the time period of this project. Data were analysed in Excel, and where appropriate, graphs include standard error bars.

Figure 2: The overall 'time plan' sheet, which indicates the submission dates for individual assessments per module across the term.

Figure 3: The assessment diary 'master sheet', which indicates the total number and diversity of assessments, plus the total time spent by students preparing assessments per module per term.

Results

In total, 10 of the 13 students submitted usable (i.e. complete) diary entries for both the autumn and spring terms and the analysis presented here is based on these data. Collectively these data represent assessment (coursework) information from 120, 10-credit Part two modules.

Number and diversity of assessments

A summary of the number and type of assessments experienced by the students in the autumn term is presented in Table 1[1]. These data illustrate the variation in volume and diversity of assessment experiences, including students taking the same degree programme. The most common assessments across all modules were essays, written reports, oral presentations and within-module tests/exams. These diaries suggested that students experienced a reasonable diversity of assessments, although many of these methods of assessment could be classified as being 'conservative'.

*Other report includes farm reports and practical exercises

**Miscellaneous includes calculations, debates, oral exams and critiques

Volume of assessment

There was considerable variation in the volume of assessment experienced by students taking the same number (but different combinations) of modules; this varied from 28 to 48 individual pieces of assessment across the two terms. Figure 4 illustrates the average number of assessments experienced per module by each student and the number of broadly different types of assessment they experienced during the year. It is worth emphasising that modules vary with respect to the proportion of mark derived from coursework as opposed to annual exams (which take place during week six of the summer term). It is therefore perhaps not surprising that a weak, but statistically insignificant (P=0.06), relationship was found between the percentage worth of coursework within a module and the number of assessments set within that module, with the number of assessments tending to increase within modules having higher proportions of assessed coursework. On average students were expected to complete approximately three pieces of assessed coursework per module; however, this analysis suggests nothing of the complexity or size of these individual pieces of assessment.

Figure 4: The mean number of assessments per module (+ SE) and the number of different types of assessment experienced by individual students

Number of assessments and student preparation time

There was no significant relationship between the time students spent preparing coursework assessments and the overall number of assessments they had to prepare (P=0.216). Given that all modules were of equal credit weighting (10 credits) there was considerable individual variation in assessment preparation time between students (see Figures 5 and 6). For example, one student spent a total of 441 hours preparing for 44 assessments over the autumn and spring terms (approx. 10 hours per assessment), whereas another spent 102 hours preparing for 29 assessments (approx. 3.5 hours per assessment). Whereas it is recognised that: i) not all assessments are equally 'demanding' (e.g. in terms of preparation), ii) students have different abilities, iii) students work at different rates and iv) assessments may differ in their relative contribution to the module's mark, this variation is still considerable. It should be noted that the range in the number of assessments completed by students is also considerable (varying from 28-48 across the two terms). This is mainly a function of degree programme and module selection although we acknowledge that some error may have occurred in respect to the way in which students classified individual pieces of assessment within their electronic diaries.

 

Figure 5: The total time (hours) spent by students preparing coursework assessments.

 

 

Figure 6: The number of assessments completed during the year by each student and the average preparation time (+ SE) spent by students per assessment.

Type of assessment and student preparation time

Overall, the students spent significantly more time preparing for essays and field reports than for other types of assessment (see Figure 7).

In common with other analyses of the student diaries (e.g. number of assessments and student preparation time) these data also revealed significant variation in individual student preparation time for different types of assessment. For example, students spent most time spent preparing for essays (10.5 ± 5.9 hours) and field reports (7 ± 5.5 hours) and least time preparing for within-module MCQ tests (3.9 ± 2 hours) and poster presentations (3.4 ± 2 hours).

Contribution to module marks and student preparation time

Interestingly, there was no significant relationship between the time students spent preparing for assessed coursework and how much that coursework contributed to the module's overall marks (P=0.370). In fact, students were in some cases observed to spend more time preparing for assessments that were cumulatively worth 20% or 30% than they were preparing for assessments within modules that were 100% coursework-assessed. As expected, there was a normal variation in the overall Part two mark achieved by students. However, if preparation time per percent mark achieved is calculated then there is considerable variation between students (ranging from 1.6 to 6.7 hours). This is again not surprising given the nature of individuals, but it does illustrate one of the difficulties of trying to allocate notional study hours to large and diverse (in terms of module choices) groups of students.

Volume of assessments and submission dates

There was considerable variation in the timing of submission dates for assessments between degree programmes. However, the overall trend was an increase in the number of assessments due in weeks 5, 7, 9, 10 and 10+ (i.e. after the module/term ended). To highlight variation in assessment submission dates and to identify areas of assessment 'bunching' we tracked a random sample of students (three students taking three different degree programmes). This analysis revealed highly variable patterns of assessment submission dates across weeks one to nine ranging from an approximately consistent weekly submission of a single piece of work (Rural Resource Management degree student) to that in which assessment loading varied on a weekly basis from zero to three assessments due on an irregular basis (Rural Environmental Science and Applied Biology); all students, however, experienced a consistent increase in the number of assessments due for submission in week 10 (end of term).

Discussion

Analysis of the assessment diary data for the academic year 2003/4 has revealed some significant and interesting patterns that illustrate the variation in assessment experiences of Part two SAPD students. Although our data represent only a small sample size and thus need to be appropriately interpreted, they have nevertheless provided useful information on assessment data collection methods and have afforded an insight into the student assessment experience within the SAPD. In particular, these data have highlighted the different approaches students take to their learning, which in turn, will hopefully contribute to teaching staff being better informed when reviewing assessment practices in the future.

An overview of the number and diversity of assessments for the volunteer students showed that these students were assessed in a number of different ways; the most common form of assessment was however, some form of written assignment (essay or report). The master sheet summaries of the assessment diaries provide an overview of the assessment experiences of students taking different combinations of modules and as such, they provide a framework for staff to quickly determine the diversity and volume of assessment being set for students within and between modules. This, in turn, may provide a mechanism to quickly gather an overview of the relative alignment between module learning outcomes and their assessments on an individual modular basis and on a programme basis (by analysing the master sheets of students taking different combinations of modules within a particular degree programme). The aim is to ensure that students have opportunities to be assessed on their ability to learn and develop a wide range of skills through an appropriate diversity of assessment methods, thus addressing the issue of diverse learning styles (Honey and Mumford, 1995).

There was no consistent pattern in assessment volume for students taking the same number (but different combinations) of modules. These data need to be interpreted in the context of assessment type (and the 'weighting' of the assessment), since some assessments are naturally more time-consuming (or 'demanding') to prepare than others and some may differ in the proportion they contribute to the module's overall marks (see below). What these data do suggest however, is that if students taking the same number of modules are being assessed with coursework that is similarly time-consuming and of equal weighting (e.g. essays of equivalent word length and percentage contribution to the module marks etc.), then a more detailed review of assessment volume should take place to ensure that these students have similar (if not identical) numbers of assessments to submit. This is especially important in the context of the University's  '10 study hours per module credit[2]' to ensure that modules of equal credit weighting do indeed have equitable assessment demands in terms of student preparation time. However, it is important to bear in mind that there will always be individual variation in the time students take to prepare the same types of assessments, a situation which was most clearly demonstrated in terms of the time taken for students to prepare written work (Figure 7).

It was interesting to note that there was no significant relationship between the time students spent preparing coursework assessments and the overall number of assessments they had to prepare. These data were particularly interesting because they revealed the wide variation in time students were spending on their assessments. It is difficult with such a small sample size of students to know whether or not these data are truly representative of the degree programme assessment 'regimes' as a whole. In light of these data, perhaps the aim should be to factor in 'natural' variation in preparation time when we are designing modules and their associated assessments. In addition, we need to ensure that students know exactly what is expected of them in their assessments, so that they can make more effective use of their study time.

We had assumed that students would spend more time preparing assessments that carried a larger proportion of module marks but we found no significant relationship between the average total preparation time and the proportion of module marks contributed by assessed coursework. In fact our data showed that in some cases, students were spending more time preparing coursework that contributed a relatively low percentage of marks than they were preparing assessments that were worth 100% of the module's marks. Without detailed knowledge of the types of assessments involved and the relative submission dates it is difficult to make generalisations from these data. However, we should consider the fact that students may naturally be more enthused to work on certain types of assessments than others (and these may just so happen to carry fewer marks) and/or that they are particularly inspired by their lecturers to invest time and energy into a particular assessment (and one that may not actually contribute most towards the module marks). Moreover, we could also reflect that perhaps students are not being as 'strategic'[3] as they could be in terms of their time management for assessments of different percentage weightings and/or that staff are designing assessments that are not 'aligned' with their relative contribution to the module's marks. In the light of these data, one possible plan of action is to spend more time with students discussing the contribution of coursework to the module's marks so that students can learn to organise and manage their study time more effectively.

The time plan data revealed variation in the assessment submission dates between the five different degree programmes; there was however, an overall trend which showed 'peaks' in submission dates from the mid to the end (and after) term. These peaks of assessment activity not only put pressure on students but of course also place considerable pressure on staff at these times to mark coursework and return feedback to students. By tracking a smaller number of students studying different degree programmes across the autumn term it was interesting to note the variation in assessment submission dates across the term, ranging from an approximately consistent weekly submission of a single piece of coursework to that in which assessment loading varied on a weekly basis from zero to three assessments due on an irregular basis. All of these students, however, experienced a consistent increase in the number of assessments due for submission in week 10. This meant that staff were frequently marking coursework and returning feedback to students after the module had ended, which may not necessarily be most supportive to student learning, particularly when there is a prolonged break between modules (i.e. a vacation). Indeed, in a review of 87 studies of what contributes most to student achievement, Hattie (1987) showed that feedback was the "most powerful single influence"; this reinforces the fact that feedback needs to be returned to students as quickly as possible if they are to receive effective learning benefits. This study has therefore provided some useful quantitative data that can be used as the basis for future assessment reviews to ensure that an appropriate balance of feedback provision (self/peer and lecturer provision, as appropriate) is facilitated throughout the course of a module.

On reflection, one weakness with the electronic diaries was that students did not enter those assessments which were worth 0% towards the module's marks but which were purely designed to provide them with feedback (i.e. formative assessments). This was disappointing, as we had hoped to compare the nature of formative versus summative assessments within and between modules. It is unclear whether this lack of information returned was because students did not know which assessments/exercises were formative or summative, or whether they simply forgot to enter any examples of formative assessment they had experienced. Similarly, students frequently forgot to label assessments as either 'individual' or 'group work', which meant that we were also unable to compare the levels of individual and group work within and between modules. Students also tended to classify the same assessment types in different ways, which meant that interpreting the electronic diaries and comparing assessment diversity was not always straightforward.  Moreover, it should of course be noted that the students who participated in this project were not chosen at random and were instead selected by programme directors as being those most likely to be interested in taking part in the study and thus most likely to complete the assessment diaries. It could therefore be argued that the data we collected are a non-representative sample. The research was also reliant on the accuracy of the information recorded by the students and their understanding of the project requirements. Despite this, our study has facilitated a number of action plans and has provided us with a valuable insight into the student perspective on assessment. We are currently revising our data collection protocol and updating the assessment diaries for an expanded university-wide study, which has recently received funding for the next academic year 2004/5.

Communicating author Dr Anne Crook, Centre for the Development of Teaching & Learning, Room 4 HumSS Building, Whiteknights Campus, University of Reading, Reading RG6 6AA, UK
Tel  0118 378 7948;  a.c.crook@reading.ac.uk
 

Acknowledgments

This research project was funded by a 2003/04 University of Reading Teaching and Learning Development Fund Award. The authors would like to thank the staff and students at the University of Reading who supported the project and Janet Pryse, from the University of Reading Centre for Staff Training & Development, who provided valuable feedback on an earlier draft of this paper.

Reference List

Biggs, J. (1999) Teaching for Quality Learning at University. Buckingham, UK: Open University Press.

Brown, S. Rust, C. and Gibbs, G. (1994) Strategies for Diversifying Assessment in Higher Education. The Oxford Centre for Staff Development, Oxford, UK: Oxonion Rewley Press.

Crook, A. C. (2003) A three part internal report for the University Board for Teaching and Learning: Part I: Undergraduate Assessment at the University of Reading.

Hargreaves, A. (1989) Curriculum and Assessment Reform. Milton Keynes, UK: Open University Press.

Hattie, J.A. (1987) Identifying the salient facets of a model of student learning: a synthesis of meta-analyses. International Journal of Educational Research, 11, 187 - 212.

Honey, P. and Mumford, A. (1995) Using Your Learning Style. Maidenhead, UK: Peter Honey Publications.

Norton, L., S., Tilley, A. J., Newstead, S. E. and Franklyn-Stokes, A. (2001) The pressures of assessment in undergraduate courses and their effects on student behaviours. Assessment & Evaluation in Higher Education 26(3), 269 - 284.

Wakeford, R. (1999). Principles of assessment. In A Handbook for Teaching and Learning in Higher Education, eds Fry, H., Ketteridge, S. and Marshall, S., pp 58 - 69. London, UK: Kogan Page Limited.