Research Article

Supporting the Development of Undergraduates’ Experimental Design Skills and Investigating their Perceptions of Project Work

Jane MacKenzie¹ and Graeme Ruxton²

¹Learning and Teaching Centre, University of Glasgow, ²Institute of Biological and Life Science, University of Glasgow

Date received: 31/03/2006       Date accepted: 27/07/2006

Abstract

Project work represents a significant component of most Bioscience degrees. Conscious that students are not necessarily given adequate preparation for their final year project, we have investigated two core elements in the 3^rd year of a 4-year Honours programme. One element, an investigative project on aspects of insect biology, has run for several years. The second, an Experimental Design course, was run for the first time in 2003, in order to support the development of key skills for bioscience graduates. The present paper describes a multi-stranded evaluation of these core elements.

Significantly increased marks were seen in the Insect Projects in the years after the Experimental Design course was introduced. It is clear from our analysis that the students valued the Experimental Design course, believing that it had made them better able to design experiments and critique the experiments of others. We therefore conclude that the Experimental Design course is, at least in part, responsible for students’ improved performance in the Insect Projects. However, we do not believe that this is the only factor that has made the students engage with the Insect Projects more successfully than in previous years.

From in-depth student interviews we identify a number of features of project work that are likely to enhance student engagement and motivation. Finally, we present some of the students’ conceptions of their prior experiences of experimental work that, we believe, give an indication of why students do not always value the opportunities to undertake experimental work that their courses offer them.

Introduction

According to the Quality Assurance Agency’s subject benchmark statement for Bioscience “All honours degree students are expected to have some personal experience of the approach, practice and evaluation of scientific research” (QAA, 2002). Project work is certainly seen as an essential element of most Bioscience degrees, and a final year project often contributes significantly to a student’s ultimate mark. This is not to say that project work is free from controversy. For example, a special interest group hosted by the HEA Centre for Bioscience has discussed concerns such as the resourcing of projects, the impact on supervisors’ time, and the appropriate credit value to be placed upon these projects.

As pointed out at the Making the Most of Final Year Projects event in 2004, “Poor or poorly motivated students may just be throwing expensive chemicals down the sink” (Wood, 2004). Since the majority of students are unlikely to have already developed the skills, motivation and confidence necessary for independent project work, it is important that appropriate training is included in bioscience programmes. The breadth of skills required were succinctly expressed by a project supervisor in research by Ryder (2004, p5) where the supervisor stated what she wanted students to learn from their final year projects:

“[to] appreciate the overall strategy of scientific work, of identifying the problem, planning a strategy to solve it, designing the experiments (...) carrying them out, interpreting the results if any, troubleshooting if the results are not usable, interpreting the results if they are usable and then using them to assess the current state of the problem and going on to the next bit of work to solve it.

Such skills are not only essential for the successful biology graduate, but are also central to the practice of research biologists (Lederberg, 1995). Indeed, these creative and critical skills hold generic value for science graduates, no matter the field of employment that they ultimately enter. Furthermore, since we live in an age were the Internet and other media provide a great deal of information of varying quality, the ability to evaluate evidence can be seen as a key life skill.

Most bioscience degree programmes do contain elements intended to develop students’ experimental design and data analysis skills These often consist of lecture courses introducing elements of experimental design (e.g. the use of controls and appropriate sampling) and data analysis (with an emphasis on statistical analysis). However, most bioscience programmes offer students little opportunity to design and criticise experiments in a low risk setting. That is, most of students’ experience of practical work will be participating in controlled exercises and highly structured investigations (Hazel and Baillie, 1998) with little opportunity to truly ‘experiment’ prior to the final year project itself.

In 2003–04, in an attempt to address this perceived gap, a new course to aid the development of skills for effective project work was introduced into the 3^rd year curriculum of the Zoology/Aquatic Biology degree programmes at the University of Glasgow.

The Experimental Design Course

Since 2003, 3^rd year students (of a 4 year Honours programme in the Scottish system) have participated in a series of facilitated discussion sessions. The sessions are supported by a textbook on experimental design for biologists (Ruxton and Colegrave, 2003; now available as a second edition, 2006) and a specially designed series of supporting questions and experimental design problems. The students (approximately 60 in the cohort) do not attend any lectures on the topic. The discussion sessions take place early in the first semester and are delivered to groups of about 15 students. Within the sessions, the students work in smaller groups to design experiments to answer a number of ‘real-life’ research questions, or ‘mini-projects’. This form of teaching is not something that the students had previously experienced, and this is likely to be the first textbook that they read cover to cover. An example mini-project is shown in Box 1.

Box1. Sample mini-project used in the Experimental Design course

All of the questions and mini-projects used can be downloaded at

ftp://www.bioscience.heacademy.ac.uk/TDF/sessionsweb.pdf .

The Insect Projects

Throughout the academic year, these same students have a number of opportunities to test, and be assessed on, their experimental design skills. The last (and most challenging) of these is a short-term lab or field-based group project: the ‘Insect Project.’ As part of the Insect Biology element of the 3^rd year curriculum, the students choose from a range of topics and undertake projects on aspects of insect biology. The projects are supported by a series of lectures on insect taxonomy, anatomy and physiology. Students work in self-selected groups of two or three to design and carry out experiments addressing a research question or questions. Each group is assigned a supervisor who is there to support the group, but the students make the majority of decisions about the design of the project themselves. The projects are assessed by means of a group presentation and on their ability to design, carry out, analyse and present a scientific investigation. Group scores are modulated by assessment within groups to produce individual student scores (Goldfinch and Raeside, 1990). The Insect project has been running with few alterations for over a decade. In the past, the success of the projects has been highly variable with some student groups engaging successfully and operating in a fairly independent manner; whilst other groups have required a great deal of input and guidance from the supervisor.

Evaluation of the Experimental Design Sessions

The effectiveness of using small group discussions to teach experimental design skills has been evaluated over two sessions, 2003–04 and 2004–05. A number of methods have been employed. Firstly, a detailed questionnaire (Supplementary material 1) has been administered to students at the end of the course in both years.

Secondly, student marks in the Insect Projects have been gathered for the two years that the Experimental Design course has run; and were compared to marks from the previous years. In addition, the final year marks of the students who participated in the course in 2003–04 have now been collated and compared with those from the previous year.

Thirdly, at the conclusion of the Insect Projects in 2003–04, four of the six project supervisors were interviewed using a semi-structured protocol (Supplementary material 2). These supervisors were chosen because they had been involved with the projects for a minimum of 5 years. The interviews, which lasted 45–60 minutes, were recorded and transcribed and common themes identified. The findings from these interviews (discussed below) led us to the decision to interview students at the conclusion of the Insect Projects and their associated assessment in 2004–05 and this represents the fourth strand of our evaluation.

18 randomly-selected students were invited for interview; 14 of these consented to be interviewed and each interview lasted 30–45 minutes (Supplementary material 3). The students were asked a number of questions related to the influence of the Experimental Design course on their approach to the Insect Projects. They were also asked about other aspects of the Insect Projects not directly related to the Experimental Design course. Again, the interviews were recorded and transcribed and the authors analysed these transcripts independently to identify themes across the pool of data. These themes are presented below with illustrative quotes.

Results

Student Questionnaires

Responses to the questionnaire at the end of the first year, coupled with the reflections of one of the authors (GR, who facilitated half of the discussion sessions), and observations of the sessions by the other author (JM) were used to refine the course for the second year of its presentation. Refinements included:

• changing the room layout from a boardroom format to grouping students at smaller tables;
• formally allocating the students to sub-groups for the duration of the course;
• reduction of the number of sessions from 6 to 5, with sessions lasting 90 minutes instead of 60; and
• Inclusion in each session of at least one practical task where students work in their sub-groups of 3-4 to design and perform an experiment, feeding back their experiences to the larger group. This is to allow the students the experience of putting an experiment they’ve designed into practice. It involves the students leaving the building in which the sessions take place to gather some data through on-the-spot interviews or observations/classifications.

Despite these changes, the questionnaire data are not significantly different for the two years under investigation, and thus the data from both years were pooled. Here we report the student responses to four items of the questionnaire. In response to the question: “Overall, do the feel that this method of learning was effective?” 85.4% of students (76 out of 89) answered ‘yes’. An opportunity to comment on any aspect of the course was included in the questionnaire and these comments included the following:

“Experimental design has no set answers that are always right or wrong. Therefore it’s very important to be able to discuss the different uses of techniques and when they are more appropriate, etc. rather than learning their names in lecture and not fully understanding their use in context of each individual experiment.”

When asked “Would you have preferred this part of the course to have been delivered by lecture?” only 6.7% of students (6 out of 90) said ‘yes’, although a number of students commented that they would have liked to attend lectures in addition to the discussion sessions.

Finally, the students were asked to consider two related statements: “I feel that my ability to design experiments has been enhanced by participating in the experimental design sessions” and “I feel that I am now better able to evaluate other workers’ experiments (e.g. research papers and research talks).” The students were asked to respond to these statements by selecting one option from: strongly agree, agree, disagree or strongly disagree. 94.4% (84 out of 89) and 92.0% (81 out of 88) respectively, agreed or strongly agreed with these two statements.

Student Marks

The average mark for the Insect Projects in the two years during the study were 15.3 (out of 20) in 2004–05 and 15.5 in 2003 –04, compared to 11.0 in the preceding year. The score in 2002–03 was not considered to be unusually low (individual student scores are not available for earlier years so are not included in this analysis). This difference was highly statistically significant (Analysis of Variance, p<0.001). This did not correlate with a general increase in grade-score since final degree marks were unchanged (analysis only possible for the cohort of students who participated in the first presentation of the Experimental Design course). Nor has an increase in Honours project marks been observed for the cohort of students who have now completed their degrees.

Supervisor Interviews

The improved performance in the Insect Projects was borne out by the findings from our interviews of the supervisors of the projects in 2003–04. These interviews were designed to gauge the supervisors’ overall perceptions of their students’ performance. The supervisors were unanimous in believing that the majority of students engaged more successfully with the projects, required less direct guidance and supervision and that the outcomes of the projects were more successful than in previous years. Some of the comments from the interview with Insect Biology project supervisors are shown in Box 2.

Box 2. Insect Project supervisors’ comments in interview

Student Interviews

In the 2004–05 sessions, we interviewed a panel of students shortly after the conclusion of the Insect Projects. One strand of our interview protocol sought to examine the impact and influence of the Experimental Design course and associated textbook on the students’ engagement with their Insect Project.

The Value of the Experimental Design Course

In keeping with the results of the student questionnaires, most students interviewed (11 of 14) stated that participating in the Experimental Design course had made them more confident about designing experiments. Of the remaining students, some felt this was an area that they had covered previously in their academic careers:

“I didn’t find the experimental design sessions that useful because, I think, I did stats in first year, so obviously we did experimental design then and it was very similar to what I learnt at that.” Student 13

A number of the students who stated that they felt that the Experimental Design course made them feel more confident about designing experiments had not, at the time, seen the relevance of the course to their 3^rd year studies but thought it more relevant to 4^th year:

“I think it just got me more aware the fact you know you really need to start thinking about the honours project.” Student 3

Influence of the Experimental Design course on the Insect Projects

The students were asked to make specific links between the Experimental Design course and the Insect Projects. When asked whether they thought the conduct of their project was influenced in any way by taking part in the Experimental Design course, 5 stated that they didn’t believe so. However, on closer questioning it was clear that for at least 3 of these students the Experimental Design course had had an impact:

“I’d never really thought like that, that you can increase the number of animals in your control section to increase the power of an experiment and stuff like that. I hadn’t thought of anything like that before.” Student 5

Of the nine students who recognised that the Experimental Design course had had an influence on decisions they’d made about their projects, some recognised that participation in the course had made them more aware of the need for rigour:

“Because there’s so many different variables that even like three of you are just sitting here and you might be watching something and your interpretation of it’s different.” Student 6

“The more you think about it, the more you get, you say right no, that can’t be conclusive and it could be interpreted another way…” Student 8

Others made links between their own projects and one or more of the activities in the Experimental Design course:

“Because remember in one of the experimental design things, there was ... they were talking about if women found eye colour more attractive or something and I turned round there, somebody was going, ‘oh you just get a whole pile of fellas with different coloured eyes’ and I was like, do you? Are you stupid? You need the same face.” Student 9

Other students found that the Experimental Design course and its associated resources were reassuring:

“If I get stuck I can refer to the book em, it was almost like a safety net.” Student 14

Factors impacting on the Insect Projects unrelated to the Experimental Design course

In a further attempt to investigate why the students in the two years in which the Experimental Design course ran performed better in the Insect Projects, the second strand of our interview protocol asked questions about the projects which did not relate directly to the Experimental Design course.

All 14 students stated that they had enjoyed doing the Insect Projects. Every student interviewed also stated that the opportunity to choose their project had been a positive influence on the way they engaged with it:

“By choosing your own I think you’re more interested in the topic you choose, so you’re more likely to be motivated in your research.” Student 4

Most of the students (9 of 14) claimed that being able to choose the members of their group had also had a positive impact on their engagement and the effort that they put into the projects.

“Generally with projects with groups, I’ll work harder than I do for myself because I don’t like letting people down.” Student 7

For reasons beyond our control, a potentially confounding change in curriculum organisation coincided with the introduction of the Experimental Design course. In previous years, the Insect Biology projects were started before the Easter vacation and lasted for a total of five weeks, during which another element of course work ran simultaneously. In the two years under investigation in this study, the Insect Biology course started after the Easter vacation, lasted for 3 weeks, and was the only course element that the students were undertaking at that time. For the duration of the three weeks, a one-hour lecture was scheduled each day; the remainder of the time the students were free to undertake their Insect Project. This was also the last element of the 3^rd year curriculum. This curricular change might have had an influence on the students’ engagement with the Insect biology course. Indeed, all 14 students commented that the fact this was the only formal course element at the time had a positive impact on their engagement with the projects.

“I like to concentrate on one thing at a time. When things are overrunning you just, you don’t get the chance to put all your all into the one thing.” Student 8

Emergent Themes

Given the semi-structured nature of the interviews, a number of other themes emerged from the analysis of the student interviews that were not the topic of any question or questions. These include the impact that an element of independence has on engagement with project work, the effectiveness of project work for learning and the students’ realisation that the Insect Projects were different to their previous academic experiences.

Five of the students talked specifically about enjoying working independently. For example:

“I just don’t really like being spoon fed, you know. I think you end up learning more if you actually sit down and had to think about.” Student 9

“I was quite proud of how we did it just because, starting with nothing and then actually working up the, actually getting your own results and everything. It was just, that’s the sort of thing that I wanted to do in the first place you know.” Student 7

Four of the students talked specifically about the quality of the learning they experienced while engaged in the projects, not just in terms of the skills necessary for independent project work, but also about the organism or system they were investigating.

“I think you learn a lot more when you’re doing it by yourself and you’re out there, like, fending for yourself.” Student 3

An awareness that the Insect Projects were different to the students’ prior experience of experimental or practical work was the final theme that emerged from the interviews. In total, ten of the students made some comment that reflected this awareness. Some mentioned the lack of independence in the early years of their course, e.g.:

“In first and second year you know you weren’t, you weren’t doing it by yourself, you weren’t going out there and getting your, collecting your own data as such.” Student 3

Others talked about the value of seeing the whole process through, and regret that this had not happened before.

“And I think that was the proper, the first proper experiment we’ve done from start to the end. … We did everything ourselves. It was great. Yes, I really, cos this is what we’re supposed to do. I want to write a paper now or something. … I wish we’d done it before…” Student 8

Yet others expressed views that might explain why so many undergraduates do not value much of their laboratory experiences:

“Because you’re … just being force fed this experiment that’s been done over and over and it’s not you know sort of ‘don’t worry if it goes wrong we have class results from years ago that all went right’.

“… You’re like fair enough, yeah it’s good to have the actual hand-on experience of doing the experiment but what was the point in doing it if you’re just going to give us the result anyway.” Student 9

Discussion

It has been suggested that, “students cannot conduct meaningful enquiries in areas in which they have no background”(Hegarty, 1982, cited by Boud et al, 1989, p141). Courses should therefore offer opportunities for students to learn the necessary skills (both conceptual and practical) to undertake project work effectively (Boud et al, 1989). Many degree programmes do teach experimental design, including essential concepts like hypothesis generation and the use of appropriate sample sizes and controls. Rarely, however, do these offer students the opportunity to practice the skills and concepts prior to an exam or piece of assessed project work. It is known that learning in small groups develops students’ confidence and communication skills (Griffiths et al, 1996) and we believe that the structured discussions used in the Experimental Design course are an effective way, possibly the first the students have experienced, to allow students to practice their design skills, in a ‘low risk’ (i.e. unassessed) setting with immediate feedback whilst simultaneously allowing them to practice the ‘group’ skills which will be utilised later in the year. In questionnaires, the students reported that they believed that the methods of delivery of the course (small group discussions supported by a textbook) are an effective way to learn these skills, with only a handful of students preferring that the course had been lecture based. It is known that confidence or self-belief (self-worth) is highly linked with performance (Covington, 1984) and we feel that the opportunity to practice designing experiments in the discussion sessions will have allowed most students to engage with their projects with a higher than previous level of confidence.

We do not believe that the introduction of the Experimental Design course was solely responsible for the improved student performance in the Insect Projects. It is possible that the experience of working in small groups in the Experimental Design course prepared the students more effectively for the group work element of the Insect Projects; however, this is not borne out in the performance of the students in other elements of group work in the 3^rd year curriculum. Unlike previous years, the projects ran for a shorter period of time (three weeks as opposed to five weeks) and other than daily supporting lectures, the students were free to spend as much or as little time as they wished in undertaking their project work. Students appeared to appreciate this opportunity to concentrate on ‘one thing’ at a time. We propose that this change in the timing of the 3^rd year curriculum, along with the increased skill levels of the students, in terms of their ability to design experiments rigorously, and their reported increased level of confidence have contributed to the improved performance in the Insect Projects.

We believe that the Insect Projects as they currently run are, in themselves, engaging and meaningful for students. It is known that for students to be successfully motivated they need to “find challenge, stimulation, satisfaction and meaning in the work they do” (Seifert, 2004, p.143). It was clear from our interviews that, for most students, the Insect Projects were stimulating and satisfying; the students enjoyed engaging in the projects and reported satisfaction with the outcomes. It was also clear that, though challenging, the students felt supported and ‘safe’ working with their supervisors. It has long been recognised that experimental project work, where students have a level of autonomy, is highly motivational and that there is an additional level of engagement if the student attains a sense of ownership of their work (Bliss and Ogborn, 1977). The impact of independence and the freedom to make choices on students’ engagement and motivation emerged clearly from our interviews. In addition, the students reported an awareness of the quality of their learning through project work and a recognition that the Insect Project was different to all of their prior experiences at University.

We continue to refine the Experimental Design course. One aspect of the course that has not been popular with the students is the subject matter of the mini-project activities. As can be seen in the example in Box 1, the practical was designed to support the learning of experimental design skills rather than learning about a particular aspect of Zoology. This abstraction of the subject matter of the practical away from the students’ perceived area of scientific interest was not received favourably. Hence, the mini-project will be changed to focus on allowing the students to make enquiries that they feel are relevant to their degree course. For example, in future these Zoology students will carry out an investigation aimed at comparing avian biodiversity between different sites, rather than categorising human hair colour.

Further, in response to the frequent comment that the students valued the opportunity to see the ‘whole process through,’ the intention is to link experimental design and analysis of collected data more explicitly than has been done in the past. The data the students collect in their Experimental Design practicals will be used as the subject matter in a Data Analysis module that the students will engage in immediately afterwards. The intention of this course restructuring is to make the links between experimental design and data analysis more explicit, to tackle the common problem of low motivation (and confidence) of bioscience students towards statistical analysis and to give them the satisfaction of carrying out scientific enquiries all the way from designing an experiment to answer a specific question, to using the data they collect to come up with an answer to the original question.

There is no evidence that the introduction of a course to teach Experimental Design skills has had an impact on students’ performance in the final year project. A possible reason for this is the expectation of final year project supervisors that they (rather than the student) will design the data collection protocols for the project, in contrast to Insect Biology where the academic staff encourage the students to do this. However, it has been noticeable that in the years since the Experimental Design course was introduced, more students have taken advantage of the option to submit their own suggestion for the subject matter of their final year project, rather than picking ready-made ones from a list provided by potential supervisors. Hence, although grades have not increased, there is a suggestion that our students’ confidence in their ability to identify and tackle scientific questions has.

Acknowledgements

The work reported here was supported by an HE Academy Centre for Bioscience Teaching Development Grant.

We would like to thank the students for agreeing to participate in this study and the considerable help and stimulation we received from Sarah Brown, Richard Inger and Kevin Conway.

Communicating Author

Jane MacKenzie, Learning and Teaching Centre, University of Glasgow, Southpark House, 64 Southpark Avenue, Glasgow, G12 8LB. Email: jane.mackenzie@admin.gla.ac.uk

References

Bliss, J. and Ogborn, J. (1977) Student Reactions to Undergraduate Science. London, UK: Heinemann

Boud, D., Dunn, J., and Hegarty-Hazel, E. (1989) Teaching in Laboratories. Milton Keynes, UK: The Society for Research into Higher Education

Covington, M. (1984) The self-worth theory of achievement motivation: findings and implications. Elementary School Journal, 85, 5–20

Goldfinch, J. and Raeside, R. (1990) Development of a peer assessment technique for obtaining individual marks on a group project. Assessment and Evaluation in Higher Education, 15, 210–231

Griffiths, S., Houston, K., and Lazenbatt, A. (1996) Enhancing student learning through peer tutoring in Higher Education. Coleraine, UK: University of Ulster

Hazel, E. and Baillie, C. (1998) Improving Teaching and Learning in Laboratories. Milperra, Australia: Higher Education Research and Development Society of Australasia

Hegarty, E.H. (1982) The role of laboratory work in science courses: implications for college and high school levels. In Rowe, M.B. (Ed.) Education in the 80s: Science Washington, DC: National Education Association

Lederberg, J. (1995) Sloppy research extracts a greater toll than misconduct. The Scientist, 20, 13

Quality Assurance Agency for Higher Education (2002) Subject benchmark statements Academic standards — Biosciences http://www.qaa.ac.uk/academicinfrastructure/benchmark/honours/biosciences.asp (accessed 11 May 2006)

Ruxton, G.D. and Colegrave, N. (2003) Experimental design for the life sciences. Oxford, UK: Oxford University Press

Ruxton, G.D. and Colegrave, N. (2006) Experimental design for the life sciences (Second edition). Oxford, UK: Oxford University Press

Ryder, J. (2004) What can students learn from final year research projects? Bioscience Education E-journal volume 4, 1–8 http://www.bioscience.heacademy.ac.uk/journal/vol4/beej-4-2.pdf (accessed 31 March 2006)

Seifert, T.L. (2004) Understanding student motivation. Educational Research, 46, 137–149

Wood, E. (2004) Why offer final year projects. Higher Education Academy Centre for Bioscience Conference - Making the Most of Final Year Projects at ftp://www.bioscience.heacademy.ac.uk/events/cardiff/wood.pdf (accessed 31 March 2006)