Research Article

Evaluation of Formative Computer-based Assessment by Cell Biology Students with Differing Entry Qualifications and Ethnicity

Christopher Bax¹, Glenn Baggott², Ellen Howey², Carolyn Pellet-Many², Richard Rayne², Maria Neonaki¹, Bridget E. Bax³, Christopher Branford White¹

¹Institute for Health Research and Policy, London Metropolitan University, ²School of Biological & Chemical Sciences, Birkbeck College, and ³Department of Child Health, St George’s University of London

Date received:24/01/2006      Date accepted:24/11/2006

This study was carried out to examine students’ responses to the use of on-line assessments that included feedback. First year BSc students taking a Cell Biology module undertook such an assessment and were then asked to evaluate the test by completing an anonymous questionnaire. Answers were analysed in light of the respondents’ ethnicity and also the academic route by which they gained entry to university. Statistical analysis of the data showed that students’ views of the utility of the test did not differ significantly either as a function of ethnicity or academic experience, since 80-90% of the students agreed/strongly agreed that regular tests and feedback assisted in their learning process. Generally, it was noted that non-Caucasian population, especially the Black African group strongly agreed that the test exercise benefited their earning experience. In conclusion, asynchronous on-line formative tests which provide feedback are valuable learning aids for students from a wide variety of backgrounds.

Introduction

Recent Government policy has focused on influencing the number of students in Higher Education (HE) and the backgrounds from which they are drawn. In particular, the White Paper The Future of Higher Education (DES, 2003)  expressed the ambition to increase the number of people (especially 18-30 year olds) that enter and succeed in HE and outlined the need to increase participation by individuals from groups which, for whatever reasons, have traditionally had limited engagement with HE.  Difficulty in achieving these goals is compounded by a fall in the number of young people studying science-related subjects at secondary school and thereafter at university.  In consequence, some institutions have felt it necessary to close science departments, especially Physics and Chemistry. This does not bode well for the government’s desire for a knowledge-based economy which is predicated on an increase in science graduates. 

Within this context, students from disadvantaged socio-economic backgrounds and ethnic minorities have been identified as target groups. There have been strenuous efforts by the HE sector to develop recruitment policies that enable the widening participation agenda to proceed (Higher Education Funding Council for England (HEFCE) Strategic Plan 2003-2008). Successful participation of minority ethnic groups in HE depends, however, on more than recruitment. It is recognised that students’ backgrounds need to be taken into account when analysing their participation, progression, retention and performance in HE. Many factors, notably ethnic origin, may influence not only educational outcome, but also have a bearing on possible feelings of discrimination and isolation (Bird et al, 1992). Connor et al (2004) have reported that on average all minority ethnic groups do not do as well in degree performance as white students.

Similarly, there is evidence that students who follow the third route (access courses) into HE may not do quite as well as school leavers or college entrants (Osborne et al, 1997), and the approach of such students to learning may differ from those with traditional entry qualifications (Hayes et al,1997).

Towards meeting the needs of expansion and consequent larger student class sizes, institutions have been given opportunities to develop new learning and teaching strategies through the HEFCE’s Fund for Development of Teaching Learning (FDTL) initiative. In 2002, Birkbeck College led a successful consortium bid as part of the FDTL round 4, to support the On Line Assessment and Feedback (OLAAF) project (www.bbk.ac.uk/olaaf/). The partners included the Universities of Birmingham, Brighton, Cardiff (Dental School/College of Medicine), Kent, Plymouth, and London Metropolitan University (previously University of North London & Guildhall University). Each institution linked to the project was committed to develop, evaluate and disseminate online learning and assessment materials. London Metropolitan’s involvement in the programme involves the development and evaluation of assessments in the biological and applied sciences. Partners have covered various subject areas including biochemistry, cell biology, civil engineering, dental pathology, field biology and nursing.

As part of the OLAAF project, we have authored an online assessment with feedback for use by students in a Year 1, BSc-level Cell Biology module at London Metropolitan. Over 40% of students on this module had previously studied on the University’s Foundation (Access) course, and over 60% of the students belonged to minority ethnic groups, thus providing an opportunity to discover whether these factors might have a bearing on students’ perceptions of the value of this type of formative assessment.

Students using the assessment tool completed a questionnaire inviting them to evaluate this test in relation to aspects of ease of access and use, question design, and impact on their learning of the course material. Answers were analysed in light of the respondents’ ethnicity and also the academic route by which they gained entry to university. The outcomes of our analyses show that the online assessments were perceived as easy to access and use, and as being useful to student learning, regardless of students’ academic route to university or of their ethnicity.

Methods

The assessment consisted of 30 recall-type multiple choice questions, together with feedback that was pre-written for each possible student response. Students undertook the test in their own time. The assessment was produced using TRIADS (www.heacademy.ac.uk/604.htm), an authoring and delivery system based on Macromedia Authorware. Data generated by students undertaking the assessment was automatically filed to a networked computer and stored in text files that were subsequently handled using Microsoft Excel.

The test was evaluated by a total of 37 first-year students on our BSc biosciences programmes. Each completed an anonymous questionnaire regarding question design, student learning and operational aspects similar to that of Bull and McKenna (2003), see Table 1. The evaluation form is given in Appendix 1.

Table 1 Questions used in evaluation of online assessment tool. Where applicable, students were asked to choose from strongly disagree, disagree, neutral, agree, strongly agree

Responses to questions were scored, with ‘strongly agree’ given a score of 5, through to ‘strongly disagree’ being allocated a score of 1, except for Question 5 which had only 3 response levels. Statistical analysis was performed on the responses to questions 3, 4, 8, 9, 10, 11 and 12, these being the questions relevant to students’ attitudes towards assessment design and perceived effects of this type of assessment on student learning. Statistical comparisons for overall analysis of responses to questions with regard to ethnicity and entry qualifications were made using a mixed (or split plot) ANOVA. SPSS for Windows version 12.0 was used to analyse the data.

Results

Background of the Respondents

Table 2A shows a breakdown of the ethnic origin of the cohort completing the questionnaire. The Black African and Asian students made up 47% of the group and this was 10% greater than the proportion of Caucasian/European (37%). Students not identifying with any of these groupings accounted for less than 3% of the total. The ethnic distribution of students taking the Cell Biology module is generally consistent with the overall ethic mix of bioscience students and of the University, where 61% of undergraduate students are from ethnic minorities (London Metropolitan HEFCE return, unpublished).

Table 2A Self-classification of 37 students in terms of (a) ethnicity and (b) pre-degree level educational qualifications

Table 2B Ethnic origins of non-Caucasian students by self-classification

With respect to entry qualifications, 58% of the cohort started their degrees with A levels or International equivalents, while 34% had come through an Access route (Table 2B).

Overall comparisons between responses according to respondents’ background

Using a mixed ANOVA there were found to be no significant differences in the responses to questions 3, 4, 8, 9, 10, 11 and 12 between the two entrance qualification groups, F (1, 34)=0.05; p>0.05, nor between those of the two ethnic groupings, F (1, 35)=0.78; p>0.05.  No significant result was found for question*entrance qualification interaction, F (5, 152)=0.93; p>0.05.

No significant result was found for question*ethnic group interaction, F (5, 169) = 2.16; p>0.05.

Responses to individual questionnaire items

Q3. The questions were clear / unambiguous

Ninety two percent of Caucasian and non-Caucasian respondents agreed with this statement, as did 90% students with A levels and 94% of  non-A level students (data not shown).

Q4. The subject range covered by the tests was suitable for the course

Again, the majority of students agreed with this, with 100% of Caucasian students and 70% of non-Caucasians responding positively, and 91% and 78% of students with and without A levels respectively agreeing (data not shown).

Q5. On the whole, the questions were: too easy / about right / too difficult

Caucasian and ethnic minority groupings reported that they felt that the degree of difficulty of questions in the assessment was about right for their level and understanding of Cell Biology (69% and 75% respectively; data not shown). From the perspective of entry qualifications, 68% of students with A levels were satisfied in this regard, compared with 78% of non-A level students.

Q6,7. The best / worst thing was…

Responses were varied, but 36% of 25 respondents found the best aspect of the test was the use of feedback, followed by the perception that the test was straightforward (20%) and that the questions were presented in a multiple choice format (20%). Of the 10 students who cited negative aspects of the test, 4 said the questions were too easy, while 4 found the questions confusing.

Q8.  The questions followed on well from the study materials

Figure 1  Evaluation of the relevance of the tests to the study materials

There was clear agreement that the questions followed on well from the study materials (Figure 1), though this appeared to be felt more by students with A levels (92%) than those without (64%); indeed 14% of this latter group disagreed. When analysed by ethnicity, there was little apparent difference between the two main groupings, although a small number (8%) of non-Caucasians also disagreed.

Q9.  Generally, I found the feedback from the test questions useful

Figure 2 Student evaluation of the utility of the feedback provided in the assessment

When asked to assess whether they had found the feedback in the on-line test useful (Figure 2), students were substantially in agreement with this, although it is noticeable that the Caucasian cohort tended only to agree, rather than agree strongly here.

Q10.  I was motivated by the feedback to refer back to the course materials

Figure 3 The effect of in-test feedback on students’ engagement with the course learning material

Students were somewhat less decided about whether they were sufficiently motivated by feedback to refer back to the course material (Figure 3), with a significant minority being neutral on the subject. Only 46% of Caucasians concurred with this statement compared with 79% of non-Caucasians. Overall there was a general view that the students, irrespective of their ethnic and academic backgrounds, were motivated, but a small number, 7-15% disagreed with this view.

Q11. Having in-course tests makes me study more than modules which do not have these tests

Figure 4 The effect of in-course tests on students’ motivation to study

By comparison, the ability of in-course tests to encourage students to study more than they would have done otherwise is demonstrated in Figure 4, with 70-80% of respondents agreeing or strongly agreeing, regardless of groupings. Again a minority disagreed with this assertion - but interestingly no one from the non-A level cohort.

Q12. On balance, the use of regular testing with feedback helps me to learn

80-90% of students agreed or strongly agreed with the assertion that the use of regular tests incorporating feedback helped them to learn (Figure 5).

Figure 5 Student evaluation of the utility of regular testing with feedback in helping them to learn

Discussion

Over 60% of the University’s students come from different ethnic groupings and a similar percentage currently have their fees paid for them by the local education authority. Over 50% of the student population in the University and biosciences are female. London Metropolitan University has an overarching learning and teaching strategy that encompasses e-learning. Furthermore, participation in the OLAAF project has enabled the Biosciences to develop the University’s learning and teaching strategy within the context of computer-based assessment that incorporates feedback.

Based upon the ethnic mix of the cohort studying the Cell Biology module, findings from this study may strategically influence future learning and teaching strategies in the discipline, particularly for those students who are at universities where widening participation and access is core to the institution’s mission.

Using a mixed (or split plot) ANOVA there was found to be no significant difference between evaluation scores from the Caucasian students and those from ethnic minorities in responses to questions relating to students’ attitudes towards assessment design and effects on student learning (p>0.05). It is possible that within the non-Caucasian cohort as a whole there are hidden differences between individual minority groupings; for example, recent published data indicates that there are clear differences in the proportions of different minorities gaining a 1st or 2(i) (Higher Education Statistics Agency, 2004), thus it would not be unexpected if there were also differences in students’ perceptions of what is helpful to their learning; however as this is a relatively small study such analysis is beyond the scope of this investigation and would require more data. However, a detailed analysis of how our Year 0  (Foundation) students subsequently perform in the first year of their degree courses is currently underway.

Likewise, there was no significant difference in scores from those students who gained entry to their degree course on the basis of A level results as compared with those who had gained entry via Access courses either at London Metropolitan or elsewhere (p>0.05). This is encouraging in the light of previous work which suggests that it cannot be assumed that such cohorts will behave in a similar fashion (Osborne et al, 1997; Hayes et al, 1997).

It is well known that students learn from their own experiences and also through engagement and interaction with others (Vygotsky, 1978). Integrating feedback into formative assessments increases the interactive nature of such tests, and helps learners to focus on the questions they are being asked, rather than simply determining whether their responses are right or wrong (Bangert-Drowns et al, 1991). Such an approach lends itself readily to e-learning and has been shown to be popular with Bioscience students (Chin and Anderson, 2005).

This work has provided first-hand information about the use and value of the online learning model as evaluated by students from differing ethnic and educational backgrounds. It is recognised that there are likely to be many additional factors that will influence students’ academic performance and their approaches to learning; this study addresses two of these, but age, and gender socio-economic class are also highly relevant areas for consideration (Connor et al, 2004), either as separate or inter-relating factors (Pawney, 1997). However, from this study we can conclude that students found the assessments useful regardless of their academic backgrounds or ethnicity thus supporting the hypothesis that asynchronous on-line formative tests which provide feedback are valuable learning aids for students with non-conventional academic backgrounds. We would welcome the opportunity to extend this study to a wider population of students following cell biology modules that are now integral to most bioscience degree programmes.

Acknowledgements

We are grateful to Peter Humphries for collating data.

Communicating author
Christopher Bax, Dept of Health & Human Sciences, London Metropolitan University, 166-220 Holloway Road, London, UK.
Telephone 020 7133 2044  Fax 020 7133 2096 E-mail c.bax@londonmet.ac.uk

References

Bull, J. and McKenna, C. (2003) A Blueprint for Computer-Assisted Assessment. Routledge Falmer

Bangert-Drowns, R.L., Kulick, J.A., and Morgan, M.T. (1991) The instructional effect of feedback in test-like events. Review of Educational Research, 61, 213-238

Bird J., ChingYee W., Sheibani, A. and Myler A. (1992) Rhetorics of access — realities of exclusion? Black Students into Higher Education. Journal of Access Studies, 7, 146-163

Chin, P. and Anderson, A. (2005) Using e-learning to promote peer learning and assessment. In: Proceedings of the science Learning and Teaching Conference, University of Warwick, pp15-19. Centre for Bioscience, the Higher Education Academy

Connor, H., Tyers, C., Modood, T. and Hillage, J. (2004) Why the difference? A closer look at Higher Education minority ethnic students and graduates. Research Report RR552. London, UK: Department for Education and Skills. London

Hayes, K., King, E. and Richardson, J. (1997) Mature students in higher education: III. Approaches to studying in access students. Studies in Higher Education, 22, 19-31

Higher Education Statistics Agency, 2004

Osborne, M. Leopold, J. and Ferrie, A. (1997) Does access work? The relative performance of access students at a Scottish university. Higher Education, 33, 155-176

Pawney, J. (1997) Trends in gender and attainment. SCRE Spotlights, 64, 1-4. The Scottish Council for Research Education

Vygotsky, L.S. (1978) Mind and society: The development of higher mental processes. Cambridge, MA, USA: Harvard University Press