Descriptive account of original electronic resource

Research skills training for undergraduate researchers: the pedagogical approach of the STARS project

John A Finn and Anne C Crook

Teagasc, Johnstown Castle, Wexford, Ireland  & Department of Zoology,  Ecology and Plant Science, University College Cork, Lee Maltings, Prospect Row, Ireland

Date received: 13/05/03                     Date accepted:01/07/03

Introduction

In mediaeval times, apprentices produced a 'masterpiece' as evidence that they had completed their training, and justification for their qualification in a particular craft. Similarly, the final year research project has come to be seen as the culmination of the research training of science-based undergraduate students e.g. Exley, (2000). Here, the terms 'science-based' and ‘scientists’ are used in an inclusive form to refer to a variety of subject disciplines, such as biology, agriculture, chemistry, engineering and psychology. In most higher education institutions, a considerable proportion (sometimes up to 20-30%) of the overall degree assessment is allocated to a final year research project and in some cases, the final project mark is awarded on the basis of the final written thesis alone. Thus, the ability to conduct research is a valued attribute of education in science-based disciplines. A competent and experienced use of the transferable skills associated with conducting scientific research is also highly valued by employers. An additional and significant proportion of science-based degrees is devoted to various combinations of literature reviews, oral and poster presentations, field-trip reports, practical laboratory reports, essays etc.

In the final year of a science-based undergraduate degree, performance in the research project can often make the difference of a whole grade in the overall tally of final results. However, supervisors know only too well the perennial problems and mistakes that arise in undergraduate research projects. These problems often include the following: poor project management; inability to clearly identify hypotheses; poorly designed experiments with no controls and/or inadequate replication; poor or inappropriate graphical and tabular presentation of data; incorrect, incomplete and inconsistent use of citations. Significantly, in many cases, it is probably only the intervention of the supervisor that prevents a lot more of these mistakes appearing in the final undergraduate thesis. Nevertheless, it is of concern that the culmination of undergraduate training so regularly reveals deficiencies in these types of basic research skills. Therefore, it seems pertinent to ask ‘is the importance of the final year research project reflected in the preparation that students receive for it?’ More generally, is the importance and expected standard of research skills in higher education reflected in the curriculum of undergraduate degrees? And perhaps most importantly, how can we improve the learning of research skills?

To this end, we report on the development of a computer-based learning resource that aims to improve the ability of research students to plan, design, manage and execute scientific research. The authors developed this resource as part of the Scientific Training by Assignment for Research Students (STARS) project that was funded by University College Cork (UCC), Ireland as an Award for Research on Innovative forms of Teaching and Learning. STARS will be implemented at UCC in the academic year 2003/2004. Here, we discuss the pedagogical approach adopted by STARS, and reflect more generally on the role of science education in higher education.

For the foreseeable future, the STARS learning resource is available at no cost to users outside UCC at http://www.ucc.ie/research/stars.

The development of STARS

In order to improve the research training provided to undergraduates and to design a resource that could provide opportunities for formative assessment and feedback, we first posed the question ‘what skills and abilities are needed to effectively conduct research?’

Most degree courses seem to recognise the essential role of experimental design and statistical analysis as part of a general scientific education. However, a more holistic and realistic identification of research skills might be expected to include:

·         Project management (planning, scheduling, goal-setting, time management, and effective communication between student and supervisor(s));

·         The ability to conduct a literature review;

·         The ability to generate and test hypotheses and to design appropriate experiments;

·         The selection of appropriate sampling methodologies;

·         Data analysis;

·         Report writing;

·         Data presentation (oral, poster, graphical, written);

·         Interpretation of data and critical thinking;

·         The ability to analyse, synthesise, and critically evaluate information.

This list is by no means exhaustive, nor have we attempted to include all of these elements in the STARS resource. However, by identifying the variety of skills required to conduct scientific research and through a consideration of training needs, we were able to define the scope of the STARS learning resource. As such, STARS has been designed to complement and reinforce other relevant modules or training, rather than as a stand-alone research training resource.

To effectively design the approach and content of the learning resource we were developing, it was assumed that potential users of STARS will have received the following training/experiences: i) an introduction to statistics and experimental design, ii) some field- or laboratory- based practical experience and iii) have written a range of scientific reports. An explicit aim of STARS was to provide training in research skills for undergraduates who would be expected to undertake a final year research project; ideally it is intended that students complete their interaction with the STARS learning resource before beginning their final year research projects. Note that the content of STARS may also be appropriate for postgraduate research students, particularly those in their first year.

STARS: resource content

The STARS learning resource consists of three main sections: ‘Useful Tips’, ‘Short Activities’ and ‘Case Studies’ and is designed in a hierarchical manner so that the skills developed in the Short Activities section can be integrated for use in the more demanding ‘Case Studies’ section. Many (but not all) of the examples and assessments in the STARS resource reflect the authors’ expertise in the environmental sciences; nevertheless, we believe that the research principles and skills examined in these should be applicable to a wide range of subject disciplines.

The ‘Useful Tips’ section (http://www.ucc.ie/research/stars/usefultips.html) provides advice and guidance on issues such as the viva voce, common problems in seminar presentations, and advice on creating effective posters. There is also a ‘Useful links’ section that provides links to a number of selected educational websites with high quality information related to a range of research skills.

 The ‘Short Activities’ section identifies some of the important research skills (http://www.ucc.ie/research/stars/shorttasks.html) and provides short, specific assignments for each. Examples of the research skills covered in this section include: experimental design, referencing, scientific writing, and data presentation. The assignments have been designed so that they contain intentional errors that students are expected to identify and rectify and which vary in their intellectual requirements. For example, simpler activities in the 'Data Presentation' section require students to identify and implement the correct formatting of graphs and tables that are provided in electronic format. Other activities require students to determine the most appropriate format for presentation of different types of raw data. The section on 'Referencing' begins by requiring students to proofread and correct the format of a list references that include intentional errors. The more demanding assignments require students to detect the improper use of references in text, coupled with formatting inconsistencies in the list of references. In addition, ‘open-ended’ questions are provided throughout the ‘Short Activities’ section to encourage the student to reflect further on the appropriate use(s) of a particular research skill.

The ‘Case Studies’ section requires students to implement, integrate (http://www.ucc.ie/research/stars/casestudies.html) and further develop the skills that they have developed using the ‘Short Activities’ section. The dominant type of case study consists of a short research report (3-4 pages long) written in the format of a standard scientific manuscript and which includes a variety of intentional errors. As an assignment, students are required to read the report and evaluate it by producing the equivalent of a referee’s report. In some case studies, data from the assigned research report are provided, and students are expected to analyse, graph or tabulate the data in an appropriate manner given the relevant background information to the study. Other case studies differ from this approach. For example, one case study focuses on an appraisal of a published paper given with the author’s permission (http://www.ucc.ie/research/stars/pathogens.html) whilst another provides extracts from a research report, on which the student’s understanding is examined (http://www.ucc.ie/research/stars/oyster.html). For all case studies, a critique is provided that identifies the errors in the research report, and students are expected to compare their evaluation with that of the ‘official’ critique after completing the assignment. Again, additional ‘open-ended’ questions are provided within the case studies that encourage the student to reflect further upon the presented research report. Thus, the overall aim of the ‘Case Studies’ is to foster the integration of research skills and to encourage critical and independent thinking.

For both ‘Short Activities’ and ‘Case Studies’ all assessments can be downloaded in ‘pdf’ file format and, in the majority of cases, feedback is immediate as the intentional errors are highlighted; these can also be downloaded. In this way, we provide opportunities for formative assessment for students, which may be particularly relevant where the research project mark is only assessed on the basis of the final written thesis.

Pedagogical principles underpinning STARS

Experiential learning

The role of experiential learning, most notably developed by Kolb (1984), is considered to be an important approach for effective learning to occur. Typically, it requires the following elements of an experiential learning cycle in which:

·         Learners are fully and freely involved in new experiences;

·         Learners must have time and space to reflect on experience from different perspectives. This is when learners may be most influenced by feedback;

·         Learners must be able to form and re-form ideas, process their ideas, take ownership of them and integrate their new ideas into sound, logical theories;

·         Learners have the opportunity to use theories to make decisions, problem-solve and test implications in new situations (adapted from Fry et al., 2000).

The design of STARS strives to adopt the above elements of an experiential learning cycle, and some examples of this are provided in Table 1. In particular, the STARS resource aims to create a series of problem-based learning resources that facilitate a deep understanding of the correct approaches to scientific research. Although difficult to articulate, there is great insight and understanding achieved by students when they are challenged with a problem and have to correctly choose and implement the appropriate research skill to address the problem: the educational equivalent of the ‘no pain, no gain’ principle!

Table 1.  Identification of elements of experiential learning that may be achieved with the STARS learning resource.

Variety of learning opportunities

We intend that a variety of class-based learning opportunities be provided using the STARS website as a learning resource. In practice, the effectiveness of STARS will be best achieved if it is incorporated into a supportive and interactive learning environment. For example, class-based learning opportunities may include:

·         Peer learning via small group co-operation with problem-solving;

·         Group discussion of the points indicated for further consideration;

·         Learning from tutor’s/lecturer’s expert opinion and relevant experience.

Computer assisted learning relies on sound pedagogy

The essential criterion in assessing whether one should use CAL or not is whether it improves the learning environment for students; the use of multimedia has no magic effect per se on the enhancement of student learning (e.g. Heinich et al., 1996). Brooks (1997) makes the point that 'While transforming some content to a multimedia format may be a “cool” and popular thing to do, it by no means ensures learning gains . . [and] . . any significant learning gains (in an active learning environment) are related to the active learning strategy carefully integrated with the multimedia rather than just the multimedia alone'. White (2000) recommends that ‘ . . . it is essential that practitioners build their use of IT upon the foundation of good teaching methods which have already been established’.

Thus, those pedagogical principles that instruct good practice in more traditional forms of learning and teaching can and should be applied to CAL. Such principles include:

·         Provision of clearly stated learning outcomes;

·         Use of a variety of learning and teaching methods;

·         Use of a variety of formative and summative assessments that assess how well students have learned/performed the learning outcomes;

·         Communication of expectations and standards to students;

·         Provision of timely, relevant and constructive feedback on assessments;

·         Encouragement of peer-assisted and student-centred learning.

Practical implementation of STARS

An important point is that although the STARS learning resource relies on online provision, this is very different to online instruction. As Hughes (2002) points out:

'To draw an analogy with a laboratory, we do not say to students: "There's a laboratory, go and do something in it." A practical class is linked to concurrent lectures, is assessed, requires a write-up and contributes to the final mark awarded. Similarly, software should be properly integrated into a module.'

The provision of electronic (or any other format of) learning resources as a 'bolt-on' or optional extra often results in minimal use, and can be very ineffective in the development of student learning (personal observations, JAF and ACC). Given the importance of summative assessment as a determinant of student learning behaviour, the successful use of the STARS learning resource is dependent on its integration into a taught module, and into the appropriate types of summative assessment for that module.

We considered a number of ways in which this type of assessment could be achieved. First, one could use some of the current Short Activities or Case Studies as continuous assessment. Students could complete an appraisal of a Short Activity or Case Study in their own time, and submit the appraisal at the next class. At present, one could only use Short Activities or Case Studies for which the critiques are not provided on the web site but lecturers could provide additional ‘unseen’ resources as appropriate to their class. However, because this approach may be susceptible to plagiarism, this is a less favoured option for summative assessment; nevertheless, it is likely to be an appropriate approach for formative assessment.

Secondly, students could be assessed on written appraisals of the existing Short Activities or Case Studies, which they would conduct in the class. This could be assessed on an individual or group basis (however, the latter raises the problem of how best to assess group work). This type of summative assessment could be implemented at the end of some classes, and would be suitable for continuous assessment. Related to this option, students could be assessed on their implementation of specific research skills covered by the STARS resource, e.g. performance in written, oral or poster presentations.

Thirdly, lecturers could develop new assignments (such as the Short Activities or Case Studies) that students must complete under exam conditions, thereby avoiding potential plagiarism problems, and which would assess the research skills developed using STARS. In particular, the current Case Studies provide an approach as to how skills integration and critical thinking might be assessed. The creation of new assignments would require extra effort on the part of the class tutor/lecturer, but an advantage is that the subject matter of the exam material could be customised to the degree subject of the students. Moreover, when creating new assignments, lecturers could make use of their own research publications, thereby minimising the effort involved whilst at the same time providing a clear link between their research and teaching.

As a fourth option, students could be assessed on their implementation of a range of research skills. For example, students could be set a short experimental or field-based task (with a specific aim and all training/ equipment etc. provided) and asked to conduct the task, either individually or in groups (with minimal lecturer input). Alternatively, students could be provided with the materials and methods and raw data from a research project, and would be asked, for example, to communicate the main findings of the research. For either of these two approaches, students would be assessed individually on the basis of the student’s ability to use the correct writing style, statistics, experimental design, data interpretation etc. This approach would test multiple skills and might show whether the learning outcomes of STARS are put into practice when the students are required to communicate something ‘real’. An additional possibility for assessing student skills could involve asking students to provide a written critique on a piece of work that they have completed (e.g. lab/field report) but which has yet to be submitted (i.e. there is no feedback for them to use as a basis for their critique). This has the advantage of training students to be critical of their own work and to learn how to realistically assess their own standards. Such assignments may only be practical when class sizes are small because of the high demand on a lecturer’s time to provide individual feedback not only on the original work but on the student’s critique of it.

It is probably unwise to attempt to be prescriptive in recommending an assessment strategy for the use of STARS within a taught module. In practice, we expect that some combination of the above options would be most suitable, with the precise combination depending on a number of factors, such as class size, availability of computers, computing expertise of students and the level of expertise of students’ research skills.

Another important consideration is that students may be suspicious or reluctant to use novel learning resources. However, there are a number of methods that can be adopted to encourage students to use an electronic resource and these include the following:

·         Provide a tour of the resource in class. Demonstrate the navigation and emphasise the aspects of the site that will be of particular value;

·         Fully integrate the resource into classroom teaching;

·         If the resource is an essential part of the student learning experience, one should base summative assessments on the content of the resource, or clearly test the learning that should be gained from use of the resource;

·         When students ask a direct question regarding a subject that is available on the resource, direct them to look it up themselves, thereby gaining confidence in experimenting with site navigation.

Discussion

Electronic learning resources

Computer-based learning in general has several advantages as a teaching resource. It can promote active learning, particularly when learning resources are interactive and problem-based (Brooks, 1997). CAL can foster an individualistic approach to learning in which students can progress at their own pace, and pursue their own interests within the relevant parameters of the course (Brooks, 1997). The individualistic approach would also be expected to enhance motivation, confidence and learner responsibility. Ultimately, it may help to foster deep learning. As well as enhancing learning of individuals on campus, the individualistic approach also makes CAL suitable for many distance learners, persons with learning disabilities or special needs (provided the learning resources are appropriately designed). There is the potential to save staff time by providing alternative forms of delivery, such as lectures or practicals (e.g. Brain et al., 1999), although this saving may be best realised when CAL replaces repeated class sessions. Educational websites can also provide hyperlinks to other high quality educational sites on the Internet, which can be an extraordinarily rich source of information. Unlike books, CD-ROMs and videos, CAL can be dynamic, and can quickly change to respond to students needs, or to update materials (Seal & Przasnyski, 2001), which is particularly relevant in the ever-changing world of science.

However, the use of CAL is not without its concerns. Most important of these, CAL is sometimes used simply for the modernisation of teaching materials in the classroom, with less regard for its contribution to student learning (Szabo & Hastings, 2000). For example, the use of PowerPoint as a replacement for overheads in lectures may only serve to better entertain, rather than better educate students (Szabo & Hastings, 2000). CAL may reduce contact time between students and teachers, which may adversely affect those students who prefer more personal interaction. Moreover, both staff and students may be worried about using such technology, and may require training to become familiar or competent in its use (McGowan & Sendall, 1997). At an institutional level, sufficient investment in computer technology is required for CAL to be viable. For some students, regular access to computers or up-to-date software may be a problem (particularly in developing countries). Even within developed countries, there may be issues of equity as some students may own PCs, and therefore have greater ease of access compared to others. There is also the consideration of the time it takes to design an effective CAL resource and the potential expense in terms of additional hardware/software needed for its design and implementation.

Science education

Designing a learning resource to foster research skills requires considered reflection of the need, importance and effective development of research skills. As such, this process has challenged our own views on science education and training in higher education. Fry et al. (2000) point out that learning is neither the addition of learning nor the accumulation of fact; rather, learning occurs by "fitting new understanding and knowledge into, with, extending and supplanting old understanding and knowledge" and that ” any learning of a higher order, involving understanding or creativity, for example, can usually only happen when the underlying schemata are themselves changed to incorporate this new understanding". In terms of science education then, perhaps there is a greater need for education that is centred on the process of science rather than the product of science? Put another way, science in higher education may focus too much on the teaching of knowledge (a body of facts) that is the end product of the ‘scientific method’. Instead, scientific training may be better integrated, and better served, by a more specific demonstration of how such knowledge is produced by the scientific method. As a result, the STARS resource has been designed to demonstrate both research practice and the scientific method using data collected by research-active lecturers. STARS, therefore, provides a clear link between teaching and research at a time when such links are becoming increasingly desirable in higher education.

Expectations and standards in research training

The STARS resource raises a number of important questions in relation to science teaching and training in higher education. For example, are the expectations and standards of performance in STARS appropriate for undergraduate science degrees? Do undergraduate degrees have an excessive emphasis on the ‘product’ of science? It is difficult to answer these questions without a clear understanding of what educational aims a particular degree wishes to achieve. This raises yet more fundamental questions, such as: What standards of research skills does one expect of a BSc degree programme?; What is it that distinguishes degrees awarded in science from degrees awarded in other disciplines?; What differences are there in the research ability required at BSc, MSc and PhD levels?

Table 2. Selected extracts from Subject Benchmark Statements for excellent performance (first class honours) across degrees in Earth Sciences, Environmental Sciences and Environmental Studies in the UK. For further details, see QAA (2000) or http://www.qaa.ac.uk/crntwork/benchmark/earthscience.pdf.

The Quality Assurance Agency (QAA) in the UK has produced Subject Benchmark Statements (http://www.qaa.ac.uk/crntwork/benchmark/index.htm) for a wide variety of undergraduate degrees, which offer a guide to the expected standards of student performance. Looking at extracts from the Benchmark Statements for First Class Honours degrees in Earth Sciences, Environmental Sciences and Environmental Studies (See Table 2), many of the performance levels and skills correspond closely to those targeted in the STARS learning resource. Although the wording varies among different Benchmark Statements, the importance of generic research skills (such as those in Table 2) pervades a variety of disciplines identified by the QAA e.g., Agriculture, forestry, agricultural sciences, food sciences and consumer sciences; Biosciences; Chemistry; Geography; Linguistics and Psychology.

Conclusions

The development of the STARS learning resource has raised a number of fundamental issues relating to science teaching and training in higher education. Perhaps most importantly, it has highlighted the different approaches to science education, in particular, the process versus product approach. In a world that is increasingly competitive, perhaps there is now a greater need to consider the skills (and level of skills) that we believe our science students should possess upon graduation. Many of these skills are not science specific and may be transferred to a variety of different contexts, thus potentially increasing the diversity of careers that graduates can consider. We believe that the design of the STARS resource can help to foster a variety of scientific and generic skills that may better equip our graduates for life after the undergraduate degree. However, it is important to remember that resources such as STARS should not be ‘stand-alone’ and are most likely to be maximally effective as learning aids if incorporated into appropriate module programmes where appropriate training and guidance can be provided. Only in this way can the potential learning benefits of these kinds of resources be truly quantified.

Acknowledgements

We are grateful to University College Cork, Ireland for an Award for Research on Innovative forms of Teaching and Learning that funded the STARS project and in particular to Professor Áine Hyland for her continued support of this work. JAF is grateful to Julian Park for useful discussion.

Communicating author John A Finn, Teagasc, Johnstown Castle, Wexford, Ireland.

Tel  00 353 (0)53 71273; Fax: 00 353 (0)53 42004; jfinn@johnstown.teagasc.ie

References

Brain, S., Dewhurst, D.G. and Williams, A.D. (1999) Evaluation of the usefulness of a computer-based learning program to support students learning in pharmacology, ALT-J, 7, 37-45.

Brooks, D.W. (1997) Web-teaching: a guide to designing interactive teaching for the world wide web. New York, USA: Plenum Press.

Exley, K. (2000) Key aspects in science and engineering. In A handbook for teaching and learning in higher education: enhancing academic practice, eds Fry, H., Ketteridge, S. and Marshall, S., pp 265-288. London, UK: Kogan Page.

Fry, H., Ketteridge, S. and Marshall, S. (2000) Understanding student learning. In A handbook for teaching and learning in higher education: enhancing academic practice, eds Fry, H., Ketteridge, S. and Marshall, S., pp 21- 40. London, UK: Kogan Page.

Heinich, R. Molenda, M., Russell, J.D. and Smaldino, S.E. (1996) Instructional media and technologies for learning, Englewood Cliffs: Prentice-Hall.

Hughes, I. (2002) TLRPs (Teaching and Learning Resource Packs) for teaching, LTSN Bioscience Bulletin 7 (Autumn), 10.

Kolb, D. A. (1984) Experiential learning, New Jersey, USA: Prentice-Hall.

McGowan, C. and Sendall, P. (1997) Using the World Wide Web to enhance writing assignments in introductory chemistry courses, Journal of Chemical Education, 74, 391-392.

QAA. (2000) Subject benchmark Statements for Earth Sciences, Environmental Sciences and Environmental Studies, Quality Assurance Agency for Higher Education, UK.

Seal, K.C. and Przasnyksi Z.W. (2001) Using the World Wide Web for teaching improvement, Computers and Education, 36, 33-40.

Szabo, A. and Hastings, N. (2000) Using IT in the undergraduate classroom: should we replace the blackboard with PowerPoint?, Computers and Education, 35, 175- 187.

White, S. (2000) Using information technology for teaching and learning. In A handbook for teaching and learning in higher education: enhancing academic practice, eds Fry, H., Ketteridge, S. and Marshall, S., pp 147-160. London, UK: Kogan Page.