Short Communication

A Pilot Evaluation of an Online Tool Designed to Aid Development of Basic Laboratory Skills

Paula Booth, Elizabeth Kebede-Westhead, Rose Heaney and Stephanie K Henderson-Begg

Health and Bioscience, University of East London, UK

Date received: 09/09/2009      Date accepted: 16/04/2010

Keywords: pilot, computer, simulation, laboratory

A pilot study was conducted using a cohort of 18 students, studying for their Bioscience Masters, and carrying out project work throughout the summer. On arrival for a laboratory class the students were asked to complete a baseline confidence log and answer a knowledge quiz. The confidence log and quiz were developed from ideas suggested in Draper et al., (1996). The forms assessed confidence and knowledge in terms of specific, basic laboratory skills which would be able to be practised on the UEL Virtual Lab (http://www.uelconnect.org.uk/hab/UELVirtualLab.html). The confidence log used a visual analogue scale and students were asked to rate how confident they felt in performing a basic laboratory procedures or calculations formulated from each of the sections in the UEL Virtual Lab. The quiz paper had 10 questions testing knowledge of basic laboratory skills.

The students were allocated into two, comparable groups on the basis of the results from the confidence log. Those allocated to the Flash group were asked to attend a compulsory Flash induction during the lunch time. Use of the UEL Virtual Lab was demonstrated and each student was able to spend time practising on the software and asking questions. At the end of the session each participant was given a diary log to mark down when and how often they used the software. It was explained that use of the UEL Virtual Lab was optional but that it would be beneficial to their subsequent performance if they used it.

Two weeks later all students were again requested to complete a confidence log and quiz, identical to those they had completed a fortnight earlier. Additionally the Flash group were asked to complete an evaluation questionnaire and to return their diary logs. The evaluation questionnaire (see Appendix) used a 5-point Likert scale to gather information on general computer usage ease and a visual analogue scale to measure satisfaction with the UEL Virtual Lab. Again questions were based on measures suggested by Draper et al, (1996) and used in Picciano (2002). There was also opportunity for the students to comment on the software. In addition, the Flash students were invited to a focus group the following week. Four students attended and so were self-selected. The focus group was semi-structured, had two facilitators and the students were asked to express their opinions about the UEL Virtual Lab.

Results

Mean quiz scores for both groups increased over the 2-week period, but there was no significant difference (p=0.52) in the improvement between the two groups (Table 1).

Mean confidence gains of each group for each task are illustrated in Table 2. Mean scores show that the Flash group achieved higher confidence gains than the control group and for the volume task this improvement was significant (p<0.05). However, none of the students practised on the UEL Virtual Lab after the initial presentation and there was a great deal of individual variation.

Table 1 Learning gains of the Flash and control groups following the pre and post-tests conducted either side of a two week interval.

	Pre-Test Mean Score	Post-Test Mean Score	p	Mean learning gain
Flash	4.1 (±2.03)	5.9 (±2.03)	0.031	1.8 (±1.83)
Control	4.3 (±1.77)	5.5 (±1.51)	0.051	1.2 (±1.69)
p	0.847	0.659		0.517

There was no statistically significant correlation between pre-test quiz and confidence scores for the Flash group but for the Control group a moderate correlation was found (r = 0.65, p<0.05). No statistically significant correlations were found between post-test quiz and confidence scores.

Table 2 Improvements in confidence for the Flash and control groups.

	FLASH	CONTROL
Test	Difference score (SD)	Difference score (SD)
Pipettes	11.62 (±15.15)	3.2 (±21.15)
Volume*	17.00 (±20.08)	-2.1 (±22.17)
Solutions	26.37 (±34.09)	12.6 (±24.62)
Weight	9.62 (±18.55)	-2.1 (±26.36)
Combined Mean*	16.16 (±14.70)	2.9 (±18.95)

* significant at p<0.05

Of the students who responded to the evaluation questionnaire all used a PC everyday and rated their skill level between competent and expert and between confident to very confident on a computer.

Figure 1 Visual Analogue Scale scores for satisfaction of the UEL Virtual Lab. (Participants marked their level of satisfaction ranging from 0% to 100% satisfied).

Results showing the level of student satisfaction with the package are given in figure 1. All seven students responded ‘Yes’ to the question ‘will you recommend UEL Virtual Lab to a friend.’

Lastly, students were asked to make suggestions for improvements or any other comments.

“Some basic equation is necessary to be included in the free play section to make the solution for problem easier”

“When doing dilutions, the programme should show the answer if the dilution is wrong.”

“While training, instructions could be more clear.”

“There should be one clear example for each topic. So that students can understand the question and its solution easily. After that he/she can do it easily without any effort.”

Focus group discussion illuminated the findings from the questionnaire. Two reasons were suggested for the lack of use of the Virtual Lab in their own time: familiarity with the material and lack of time. As an alternative, it was suggested by the participants (unanimously and enthusiastically) that undergraduates rather than masters students might be very keen to use this software.

Some good ideas were generated as to how the UEL Virtual Lab could be used more effectively. One idea suggested was that computers loaded with the software should be made available during class time

Discussion

This pilot study provided guidance to staff about how the UEL Virtual Lab might be used in the future, in particular the need to embed its usage within classroom teaching and potentially make it available to undergraduates. Encouragingly overall student satisfaction with the software was high and although the Flash group did not have a significantly higher learning gain than those in the control group this is perhaps not surprising, considering that the students only saw the UEL Virtual Lab once in the initial presentation.

The work also highlights some interesting questions about student perceptions and understanding. Clearly it was disappointing that none of the participants practised on the UEL Virtual Lab in their own time. The reasons claimed were familiarity with the material and lack of time. All the participants in this study were studying a masters’ programme and should have covered the subject matter during their bachelor degree. However, scores from the quiz results suggest that students may not have had as much knowledge of basic skills as they perceived. The average mean score for the whole group at baseline was 4 out of 10, with 44% of the group answering only 3 or less questions out of 10 correctly. At baseline there was a medium strength correlation between confidence and performance but no correlation was found at test either for the cohort as a whole or as two individual groups. The results suggest that masters’ students would still benefit from practising on the software.

Interestingly, despite the two groups being selected on the basis of confidence scores, the Flash group had higher gains in confidence than the control group and for the volume task this difference was significant. This overall increase in mean confidence in the Flash group is consistent with the findings of Rolfe (2009). Draper (1996) suggests that confidence logs are a diagnostic tool in which rises in confidence reflect student perception that they have learnt something between two time-points. This current study hints that, for some individuals at least, even short-term exposure to a teaching aid may substantially increase student confidence. However, the small student numbers and large degree of individual variation mean this suggestion must be viewed with caution.

Everyone in the Flash cohort answered that they would recommend the software to their friends, yet none of the students were sufficiently motivated to use the material in their own time. The possibility that students were giving the answer they thought the lecturer wanted to hear ought to be raised. The focus group was held in a classroom in the university and this may have contributed to desirable responding. Lack of use of the Virtual Lab in students’ own time may, therefore, have been due to a less desirable reason such as lack of interest which the students did not disclose.

The UEL Virtual Lab is a product that has value as it could be used by any school or university in which basic laboratory procedures take place. Additionally, development of a simulation of basic laboratory skills in a virtual reality such as Second Life is planned. Evaluations will be carried out to assess advantages and disadvantages of both Flash and virtual reality platforms.

Acknowledgements

We would like to thank Gemma Mclean and Gemixin Ltd. for development of the UEL Virtual Lab.

Communicating author

Dr S Henderson-Begg, University of East London, School of Health & Biosciences, Romford Road, Stratford, London E15 4LZ, Telephone: +44 20 8223 6470, Fax: +44 20 8223 4959
Email: S.Henderson-Begg@uel.ac.uk

References

Draper, S. W. (1996) Observing, Measuring, or Evaluating Courseware. www.psy.gla.ac.uk/~steve/Eval.HE.html (accessed 17 July 2009)

Draper, S.W., Brown, M.I., Henderson, F. P. and McAteer, E. (1996) Integrative evaluation: An emerging role for classroom studies of CAL. Computers & Education. 26 (19-3),17–32

Picciano, A.G. (2002) Beyond student perceptions: issues of interaction, presence and performance in an online course. JALN, 6 (1)

Rolfe,V (2009) Development of a Virtual Analytical Laboratory (VAL) multimedia resource to support student transition to laboratory science at university. Bioscience Learning and Teaching Case Study available at www.bioscience.heacademy.ac.uk/ftp/casestudies/VRolfe.pdf (accessed 20 October 2009)