Research Article

Report on the EMBER project — a European Multimedia Bioinformatics Educational Resource

Terri K Attwood¹, Ioannis Selimas¹, Rob Buis², Ruud Altenburg², Robert Herzog³, Valerie Ledent³, Viorica Ghita³, Pedro Fernandes⁴, Isabel Marques⁴ and Marc Brugman²

¹Faculty of Life Sciences & School of Computer Science, University of Manchester, UK;  ²Expert Centre for Taxonomic Identification, Amsterdam, The Netherlands; ³Université Libre de Bruxelles, Belgium; ⁴Instituto Gulbenkian de Ciência, Oieras, Portugal

Date received: 30/03/2005             Date accepted: 19/08/2005

EMBER was a European project aiming to develop bioinformatics teaching materials on the Web and CD-ROM to help address the recognised skills shortage in bioinformatics. The project grew out of pilot work on the development of an interactive web-based bioinformatics tutorial and the desire to repackage that resource with the help of a professional multimedia publisher. We report here on the completion of the European project and its current status: the website is now accessible from http://www.ember.man.ac.uk

Introduction

The field of bioinformatics emerged in the 1980s when developments in information technology were used to harvest the fruits initially of small-scale, and ultimately of large-scale, sequencing projects. As a discipline, it involves design, establishment and maintenance of databases, and design of interrogation, analysis and visualisation software to help interpret results of analyses in biologically meaningful ways. Consequently, it requires scientists with knowledge of both biology and computing. Initially, there were few or no courses available through which to provide bioinformatics training - biologists largely had to teach themselves how to access different databases and to use computational tools to solve genome-scale problems.

In the early 1990s, the emergence of the web provided new opportunities for teaching and, in particular, for bridging the bioinformatics skills gap. Web-based approaches have several advantages (they are easy to use, they allow students to study away from formal classes etc.); but they also have disadvantages (e.g., they can be so easy to use that students complete exercises without fully engaging with their content, students do not read web pages, students may avoid the lab by working at home, and maintenance is arduous). Despite the potential problems, the web has been widely used to deliver courses and primers in all aspects of science.

Web-based bioinformatics materials began to emerge in the mid 1990s. In 1995, an interactive tutorial (BioActivity) was developed to give students first-hand experience of practical protein sequence analysis on the Internet (Attwood, 1997, 1999, 2000). Within the practical (see Figure 1), each page carried brief header instructions, coupled with a more detailed commentary illustrated with embedded diagrams. Further information and suggestions for additional reading were provided in 'info loops', accessed via hyperlinked icons in the bottom right-hand corner of each page — to get the most from the practical, the commentaries urged students to consult as much of the background material as possible.

The practical content was devised around a scenario in which the student had sequenced an ‘unknown’ fragment of DNA. By means of real-time database searches, the protein it encoded, the family to which it belonged and its biological function had to be discovered and, where possible, placed in a structural context.

Figure 1 Example pages from BioActivity: each contains a header, beneath which are left- and right-hand frames. The left provides a menu; the right a commentary, giving the rationale for that part of the practical; 'info’ icons provide further explanations, background information etc.

Results of these searches were sent to additional left- and right-hand output pages (Figure 2). The central commentary therefore always remained visible, while allowing results from different database searches, to be viewed simultaneously in separate windows (this facilitated cutting and pasting of information between different pages, and comparison of different search outputs). The practical also provided built-in exercises designed to challenge students’ understanding at each stage before allowing them to move deeper into the practical.

 

Figure 2 An interactive page from BioActivity, illustrating how the results of database searches are sent to separate left- and right-hand output windows

As the practical was 'live' on the Internet, it offered the genuine experience of working with the web; inevitably, this was sometimes good and sometimes bad (during rush-hour, when links broke, etc.). On computers with small screens, another problem was that users would fill their screens with a given page — when BioActivity sent results to its left- and right-hand windows, these would be obscured by the browser, causing confusion. Management of several windows is not difficult, yet seemed to demand an unexpectedly high level of concentration from students.

BioActivity has been accessible for a decade and has gradually matured (http://www.bioinf.man.ac.uk/dbbrowser/bioactivity). Through questionnaires, students have rated the practical as a useful aid to learning, the overall experience providing a good foundation for future exploration of bioinformatics resources on the web. But the practical has its faults, not least for newcomers, for some of whom there is a steep learning curve. Furthermore, with increasing external web use (on some occasions more than 100,000 practical pages have been hit per day), it was becoming more important and increasingly burdensome to maintain.

To address some of these issues, in 2001 the European Commission funded a two-year educational project, EMBER, which aimed to develop a new suite of bioinformatics teaching materials based on BioActivity. EMBER comprises a self-contained, interactive web-tutorial in bioinformatics (with emphasis on protein sequence and structure analysis) and the equivalent course on CD-ROM; the CD-version was necessary to provide students in European domains for whom Internet availability is not optimal access to the same resource. The project involved several partners with different roles: some helped to develop the contents, others tested the package by conducting user trials. Collaboration with a professional multimedia publisher (the Expert Centre for Taxonomic Identification (ETI)) was also essential to ensure high-quality production of the web- and CD-ROM-based materials.

Methods & Results

The initial phases of the project aimed to tailor the new resource to the requirements of industrial and academic employers by identifying the nature of the current skills shortfall and defining a minimum standard of required knowledge. A questionnaire was therefore devised, with questions mapping onto potential areas to be covered in EMBER: these included core (e.g., biological databases, protein sequence and structure analysis) and advanced (e.g., phylogeny, ontologies, structure prediction) topics, and supplementary material (e.g., information theory, statistics).

The idea of the questionnaire was to establish if these topics were appropriate, and whether any had been missed. Although the questionnaire was kept simple, largely requiring only yes/no answers, feedback was poor (only 16% of contacts replied), but the responses we did get were useful. Topics considered most important included sequence and structure analysis, gene prediction, alignment algorithms and statistics. Knowledge of experimental  techniques, protein structure prediction, EST analysis, molecular evolution and programming languages were given less emphasis. Database management systems, biological ontologies and image analysis were generally considered more advanced subjects.

Content and format of the tutorial

The range of responses from the questionnaire indicated that a modular structure would be desirable for the tutorial, allowing students from different backgrounds to take only those modules that were relevant to their needs. As EMBER aimed to provide basic bioinformatics teaching materials within a fixed time-frame, we had to be pragmatic in implementing these results and to prioritise the topics to be included (Mabey and Attwood, 2001). Those we adopted are shown in Table 1: they include basic chapters dealing with translation of DNA sequences, searches of protein sequence and family databases, sequence alignment and protein structure classification; advanced chapters covering homology modelling and threading, and more challenging analyses with uncharacterised sequences; and supplementary chapters involving case studies with particular biological themes, building on methods from the basic chapters and introducing new ones. We also incorporated assessment tools, to evaluate both the product and student performance; results were then sent to ETI to develop the new web- and CD-ROM-based tutorials.

Table 1 Main themes covered in the EMBER tutorial

Basic tutorial Translation & identification Sequence database searches Protein family analysis Sequence alignment Property profiles Fold classification Advanced tutorial Homology modelling Threading Mystery sequences Case studies Investigating the human genome Investigating the inositol phosphatase family Investigating phylogeny relationships of ABC transporter domains Investigating sickle cell haemoglobin

The user trials

Once prototyped, the materials were tested in user trials. Within the framework of EMBER, trials were run in Portugal, Belgium and Manchester, but the tutorial was also opened to users outside the consortium (courses registered on the database at that time are shown in Table 2). The goal of the trials was to obtain feedback both on the ease-of-use, look-and-feel, interest of content, etc., of the material (via the Final Feedback section), and on the performance of the students before and after exposure to the material (via Initial and Final Multiple Choice Quiz (MCQ) sections). Some evaluators provided information on the materials, others provided feedback only on student progression, and some obtained feedback on both.

Table 2 Courses registered on the EMBER database, April 2004. Those asterisked were part of the EMBER assessment

Country	Course ID	Number of users
Belgium	EMBER0028	20 users*
Belgium	EMBER0034	20 users *
Belgium	EMBER0036	7 users *
Finland	EMBER0024	15 users
Hungary	EMBER0001	9 users
Manchester	EMBER0027	16 users *
Portugal	EMBER0025	28 users *
Portugal	EMBER0035	5 users *
Portugal	EMBER0037	7 users *
Slovakia	EMBER0038	10 users
Sweden	EMBER0020	19 users
Sweden	EMBER0021	9 users

In Manchester, we ran a trial with 16 students (from biological and computer science backgrounds) from MSc Bioinformatics. During the introductory module of the MSc, students receive 16 lectures and 8 2-hour practicals — the latter was formerly given by BioActivity, so we substituted this with EMBER. To meet the requirements of the MSc, only the basic chapters were completed. Students were then asked to fill in the Final Feedback form (Table 3), which was kept short to encourage feedback. For each question, answers could vary from very bad/negative, bad/negative, neutral, good/positive, to very good/positive. Figure 3 summarises the responses.

Table 3 The questions from EMBER’s feedback form

Figure 3 Summary of feedback following the Manchester trial

Viewed in isolation, the results suggest that EMBER was a positive experience for students who used it, and that there are some areas that would benefit from further development. However, a single trial could not tell us to what extent the feedback reflected on EMBER itself, to what extent on the type of students, or to what extent on the nature of the trial (i.e., the amount of time allocated, the number of chapters they were required to complete, the amount of assistance from demonstrators etc.). For this information, it was necessary to consider the results of other trials, conducted at different locations, with different students, using different scenarios.

To put the results of the Manchester trial into context, feedback from all registered users of the tutorial was collated, 52 in total (Figure 4). As before, the overall response was good/positive to very good/positive, as denoted by the preponderance of 4- and 5-star ratings. For some questions, the results appeared quite negative (i.e.,11, 16 and 17). However, these questions relate to the frequency of technical problems and the need of students to consult either additional material or a demonstrator — in all these cases, the apparent negative outcome is thus a positive result for EMBER (i.e.,there were not frequent technical problems, and students did not frequently have to consult material outside EMBER or a demonstrator).

Figure 4 Overall feedback from 52 registered users

Feedback about using EMBER is valuable, even if it is purely qualitative: it shows that most users enjoyed the experience and wanted more time with the tutorial. The feedback highlights the fact that the trial scenarios at the various locations were different: students were from different backgrounds and courses were run for different durations, with different levels of supervision. Owing to the breadth of material covered, it is clear that some of the trials could only cover part of the tutorial; or, for those that tried to cover all of the material, the time allocated was too short for the students to get into the subject in depth.

This result is not surprising: EMBER evolved from a practical that was itself designed to complement a lecture course; it has not only reproduced the original material but has added several chapters — a trial run over one or two afternoons would simply not allow sufficient time for students to assimilate all of the material. Hence, the feedback affirms that the tutorial very effectively complements traditional face-to-face (F2F) lecture courses, where sufficient time is given to the formal practical component. And, because the tutorial is already broken down into basic and advanced sections, and case studies, teachers may select those parts that best suit the students: e.g., beginners might only undertake the basic sections; more expert users might only attempt the case studies. In addition, the feedback suggests that, for more advanced students (i.e., those familiar to some extent with databases and search tools), EMBER is suitable for self-paced learning. In the Manchester trial, it was evident that the amount of consultation with demonstrators was significantly reduced by comparison with BioActivity, and several of the students elected not to come to the practical sessions, but chose instead to work from home.

Overall, the feedback helped us to make useful revisions to the tutorial: e.g. the Belgian trial offered several suggestions for improving the clarity of some of the chapters (importantly, they raised the issue of the difficulty of the English for non-native speakers). There were also some valuable ideas for future improvements: e.g., to allow teachers to select in advance only those modules of interest to them, the tutorial then only providing chapters and quizzes relating to the selected modules; another possibility would be to allow teachers to develop their own modules, and customise EMBER accordingly (clearly, extending the materials beyond protein analysis would significantly improve the scope of the tutorial).

To consider some of the more pedagogical issues, we now turn to the 3 trials for which information was captured on student performance in the Initial and Final MCQ sections: EMBER0025, which had 28 users; EMBER0035, which had 5 users; and EMBER0037, which had 7. The Initial and Final MCQs, at the beginning and end of the tutorial, are identical. They consist of 24 questions that cover basic bioinformatics concepts (all of which are explored in detail during the tutorial). The results of the Initial Quiz give an indication of the level of bioinformatics knowledge of the user prior to starting the tutorial; those of the Final Quiz give an indication of how the tutorial has performed as a didactic tool. This approach was overly simple, but the time-constraints of the project precluded the use of more sophisticated techniques.

Figure 5 Comparison of Initial and Final Quiz results for EMBER0025

The Initial and Final Quiz results from students who completed EMBER0025, EMBER0035 and EMBER0037 are illustrated in Figures 5 and 6.

Overall, of 40 students participating in 3 trials, 3 dropped their score in the Final Quiz; 2 achieved the same score; and 35 improved their score. Those whose scores deteriorated or stayed the same probably did not have time to complete the tutorial. Those whose scores increased constitute 87.5% of the users, achieving an average increase of 4.7 points. To some extent, the range of results reflects the different trial scenarios and different backgrounds of participants (some were advanced users, others were novices, some completed all chapters, others did not). Overall, however, almost 90% of participants improved their performance by the end of the tutorial (with a 40-50% average score increase relative to their initial scores). Further development of these naïve assessment tools is clearly necessary to properly appreciate the pedagogical impact of EMBER; nevertheless, this initial outcome is very positive.

The EMBER interface

Aspects of the tutorial interface are illustrated in Figures 7 and 8. Each chapter has a concise set of aims (Figure 7a), step-by-step instructions, background information, references and its own MCQ. A notepad is available for saving results between chapters, and a list of contents is always visible on the left-hand side of the page. Navigation may be effected either via the page tabs or by using the table of contents.

Figure 7 Screenshots  showing (a) an aims page and (b) an instructions page

The instructions pages are broken down into small steps; navigation arrows allow progression to subsequent steps once all tasks have been completed (Figure 7b). The info pages provide supporting information, broken down under sub-headings and illustrated with images, diagrams, etc; again, navigation arrows allow progression to and from different sub-headings (Figure 8a). A glossary is also provided, allowing information retrieval by glossary term, by definition, abbreviation and/or synonym.

Figure 8 Screenshots showing (a) an info page and (b) the Final Quiz

As mentioned above, EMBER contains various assessment tools, including MCQs in each chapter and at the beginning and end of the tutorial (Figure 8b). For each MCQ, answers are computed and relayed to the student, with information to indicate how many correct answers were obtained and, if mistakes were made, what the errors were. The results of the Final and Initial Quiz are related, so students may evaluate their performance relative to when they commenced the tutorial.

EMBER’s database management system

Possibly one of the most important aspects of EMBER is the implementation of a database management system (DBMS), which allows course organisers to track their courses and to manage student results. The system currently allows either a course organiser to create his own course, accessible only to him and his students, or an independent user to access the tutorial. Thus students must create an account with the DBMS (Figure 9) to register either for a given course or for independent use.

Figure 9 Access to EMBER via the DBMS (http://www.ember.man.ac.uk)

To register a new course, organisers must enter via the login page (username add_course, password r3gist3r) and complete the registration form (Figure 10a). This prompts for a password and course access code, and the system responds by providing further confidential details on how to manage the course. This process gives substantial security, requiring password access by the organiser and identification of courses with unique codes, without which students cannot register for those courses. The registration form has also been designed to gather information about the course being registered: the institution and country in which it is taking place, the number of students and demonstrators on the course, and the number of computers available to them. This information is valuable for the purpose of trial evaluation, allowing different trial scenarios to be compared.

Figure 10 (a) Registration form presented to organisers initiating new courses; (b) Profile form presented to students who register with a new course

A student may create an account within a new course by completing a registration form (Figure 10b), which asks him to define his profile (scientific background, qualifications, level of bioinformatics knowledge, etc.) by choosing from a number of description fields. This helps the course organiser to understand the students’ background and level of knowledge; it also renders results from MCQs and assessments during the trials more meaningful. The form also gathers information on where the student studies or works, which may differ from where the course is being run. At the end of the form, students must provide a username and password, and select the course for which they are registering by means of the ID and access code provided by their organiser, ensuring that they have registered for the right course. If a student is not part of a formal programme, he simply chooses ‘independent’ as his course.

Once courses have been created, the DBMS gives organisers the power to manage or monitor their courses in various ways. With the confidential details provided by the system, they may access the database and view all their registered students. Using the information provided by their students at registration, they can see their background subjects and qualifications, and their level of bioinformatics knowledge. They can also see the scores for the Initial and Final MCQs, together with the date these were taken and the answers students got wrong. In addition, for each student, they can view the results of each end-of-chapter quiz, again with the date they were completed and which answers they got wrong. Finally, they can export all this information in a number of different file formats, suitable for widely-used statistics and spread-sheet software.

Results in the DBMS are also available to the development team at Manchester. This can help us to analyse: how the tutorial is performing as a teaching tool; how individual users with specific backgrounds perform in different areas of the course; what questions might not be performing as expected and might need refinement; what is the level of bioinformatics understanding or training in different countries; and so on. Such information is also held for independent users, allowing similar analyses to be performed; it also facilitates comparison of the effectiveness of EMBER in traditional F2F, distance learning or personalised settings.

Concluding remarks

The EMBER tutorial requires students to tackle a range of concepts, including databases and software tools, many of which they may be meeting for the first time. They must become familiar with a variety of websites, and must not only understand the differences between different types of database, but also the differences between databases of the same type and their software interfaces and their search outputs. The tutorial therefore presents many challenges, but also provides a wealth of background information to guide users through the theoretical aspects of their work.

EMBER’s trials (and tribulations!) highlighted different strengths and weaknesses. The Portuguese trials, although limited in terms of student participation, confirmed that most students significantly improve their understanding of basic bioinformatics concepts by the end of the tutorial; the Manchester trial showed a far greater level of student engagement with the material and less dependence on demonstrators compared with its predecessor, BioActivity; the Belgian trials shed light on the comprehensibility of the tutorial and the need for better graphics. We appreciate its current limitations in terms of content, flexibility of use, accessibility for the visually impaired and evaluation tools, and are working to improve these. Nevertheless, in its current guise, having learned from our early work, EMBER represents a next step toward the provision of easy to use, self-contained bioinformatics teaching materials, suitable for conventional F2F delivery or for self-paced tuition.

EMBER has replaced BioActivity in Manchester’s Bioinformatics MSc and may do the same in Birkbeck College’s virtual course in protein structure analysis; hopefully, other users will also see the benefit of moving to the improved system. Future development will continue in collaboration with the North West Institute for BioHealth Informatics, where EMBER is now hosted (http://www.ember.man.ac.uk) and the European Bioinformatics Institute; the CD-ROM is available from ETI-IS (http://www.etiis.org.uk/aboutetiis.htm). Background project information is available from the homepage http://www.bioinf.man.ac.uk/dbbrowser/ember/; for general enquiries, contact

georgina.moulton@manchester.ac.uk, selimas@bioinf.man.ac.uk, or attwood@bioinf.man.ac.uk

Communicating Author: Terri Attwood, Faculty of Life Sciences & School of Computer Science, University of Manchester, M13 9PL
Tel: +44 (0)161 275 5766 Email: teresa.k.attwood@manchester.ac.uk

Acknowledgements

We are grateful to the European Commission for the initial funding, and to the EPSRC for bridging funds from Manchester’s platform grant, PARADIGM, which allowed us to complete the project. We thank Jane Mabey and Giles Velarde for their contributions to the early stages of the project, and all users of BioActivity and EMBER for their valuable feedback.

References

Attwood, T. K. (1997) BioActivity: An Interactive Bioinformatics Practical on the WWW. Life Sciences Educational Computing, 8(3), 10-13

Attwood, T. K. (1999) An interactive bioinformatics practical on the World Wide Web. In Laboratory Work in Science Education: an Evaluation with Case Studies. Ed. L. Jervis, SEED Publications, University of Plymouth

Attwood, T.K. (2000) An interactive practical at the interface of Web-based and conventional publishing. CAL-laborate. A collaborative publication on the use of Computer Aided Learning for tertiary level physical sciences, 4, 1-6

Mabey, J.E. and Attwood, T.K. (2001) EMBER: a European Multimedia Bioinformatics Educational Resource. CAL-laborate. A collaborative publication on the use of Computer Aided Learning for tertiary level physical sciences, 6, 13-16