Research Article

Teaching Critical Appraisal to Sport & Exercise Sciences and Biosciences Students

Anna C. Phillips

School of Sport & Exercise Sciences, University of Birmingham, UK

Date received: 11/08/2009     Date accepted: 17/12/2009

Abstract

Seminars were implemented to develop undergraduates’ critical appraisal skills and their effectiveness was evaluated. Participants were 140 undergraduate students consisting of 103 students from Sport and Exercise Sciences and 37 from Biosciences. Four seminars were employed to develop and reinforce critical thinking and provide an opportunity for practise and group work. Source material included research proposals and published journal articles. Two linked pieces of coursework assessed critical thinking skills. Teaching method effectiveness was examined using the students’ questionnaire responses and comparison of coursework grades across the module. Students reported finding the seminars useful and helpful, and their self-ratings of critical appraisal skills improved from pre- to post-seminar. However, this was not generally reflected in assessment grades across the group. Overall, there was a significant decline in grades from the first to the second piece of coursework. However, although Sport and Exercise Sciences students’ scored significantly lower on the second coursework, Biosciences students scored higher. It is possible that this type of teaching helps to boost performance in students who originally are new to such skills. Future studies would need to examine whether different methods or longer follow-up might also yield improvements in objective measurements of students’ critical appraisal ability.

Keywords: critical appraisal; seminars; research articles

Introduction

Critical thinking or critical appraisal is “purposeful, reasoned, and goal-directed thinking… it is the kind of thinking involved in solving problems, formulating inferences, calculating likelihoods, and making decisions” (Halpern, 1998). It is also a key skill required in many areas of employment (Pithers and Soden, 2000) and in the final year of many university degree courses which require an in-depth literature review. This entails both the synthesis and critical appraisal of the concepts presented in relevant published source material. Such evaluation is classed as a higher order thinking skill in terms Bloom’s of educational objectives (Bloom, 1956). More recently these objectives have been revised and the ability to create and generate new ideas or ways of viewing the world has been placed higher up the taxonomy, with evaluation relegated to second place (Anderson and Krathwhol, 2001). Nonetheless, in teaching undergraduate students to critique and evaluate source material the ability to create or generate solutions and new alternative approaches to that which they are criticising is a relevant and related skill. Creative thinking has also been emphasised as a key component of critical appraisal in university education (Bonk and Smith, 1998).

A framework for critical thinking in nursing education drew on the existing literature suggesting that it involves elements of interpretation, analysis, evaluation, inference, explanation, and self-regulation of one’s own thinking. They also emphasised the importance of logicalness, i.e., the ability of students to critique their own logic and spot the mistakes (Dexter et al., 1997). Others, also in nursing education, have defined critical appraisal skills as analysing, applying standards, discriminating, information seeking, logical reasoning, and predicting and transforming knowledge (Scheffer and Rubenfeld, 2000). Critical appraisal can also be considered as “ self-guided, self-disciplined thinking which attempts to reason at the highest level of quality in a fair-minded way” (Elder, 2007). Although the definitions of critical thinking vary, nearly all emphasise the ability to gather, evaluate, and use information effectively (Beyer, 1985). The present author considers many of these aspects to be relevant and defines critical appraisal as the ability to analyse and evaluate information, question its validity in a logical, evidence-based manner, and formulate inferences on the basis of the evidence.

Developing critical appraisal in university students is a challenge, given that such skills are generally not promoted in primary and secondary education, leaving many university students with relatively poor reasoning and problem-solving skills (Klimoviene et al., 2006). It is also clear that many university degree courses, although valuing the importance of these skills, do not employ specific teaching of critical appraisal skills but rely rather on incidental learning (Castle, 2006). It has also been suggested that more transparency is required in the links between what is being taught and its assessment (Biggs, 2000). This principle was applied to radiography students, who were rated their own critical appraisal skills on the basis of what they had been taught throughout their degree using a modified version of the Critical Thinking Questionnaire (Faccione, 2001); scores were then compared to their written assessment grades. However, although students’ rated their abilities highly, their assessed work showed “little attempt to use critical discussion…” (Castle, 2006). Abrami, et al., (2008) proposed that lecturers should align coursework and teaching with learning outcomes so that students are aware they are being taught critical appraisal, and Castle (2006) recommended only to assess one aspect of critical appraisal at a time.

A systematic review of the effectiveness of critical appraisal teaching for health care workers revealed that from 137 articles, 121 did not assess the effectiveness of this teaching (Hyde et al., 2000). In addition, several studies only implemented critical appraisal teaching with a small number of students, therefore making generalisation difficult (Hyde et al., 2000). These are common limitations, observed previously in a similar systematic review (Audet et al., 1993). Of the 16 articles identified by Hyde et al. (2000) which did assess the impact of teaching critical appraisal, it was found that most of the studies (n = 14) showed a benefit, mainly in terms of participants attitudes, although this may reflect ‘desirable responding’, and improved knowledge and skills. A previous systematic review of courses teaching critical appraisal to medical students showed an overall increase in students’ knowledge, but not an application of this knowledge to critical reviewing of the literature (Norman and Shannon, 1998). When these courses were used with medical residents, the gains in knowledge were smaller, and although participants reported reading literature more critically, this was not reflected in objective ability tests. Similarly, another systematic review concluded that the evidence for an improvement in knowledge was weak, and that “the ability of participants to appraise evidence critically … was not convincing” (Taylor et al., 2000).

Bensley and Haynes (1995) suggested students can improve their critical appraisal skills but to do so requires they have opportunities to practise these skills. Peters et al., (2002) implemented critical interactive thinking exercises over three years. Tasks included: the writing of a composition addressing a specific problem or little understood phenomenon; preparation of the defence of a hypothesis; presentation of the arguments supporting the hypothesis to a peer group; and group assessment of the quality of the arguments made. Although students were initially apprehensive, it was reported that the majority eventually reported the experience as being a positive one which enhanced their critical appraisal skills. Unfortunately, the effect on performance on assessments was not discussed by the authors.

It may be that particular aspects of teaching critical appraisal help to improve this skill in students, for example, the integration of group-based learning. Similarly, simple factors such as the frequency of attendance at critical appraisal courses and credits contingent on attendance were thought to be associated with the relative success critical thinking teaching in medical students and residents (Norman and Shannon, 1998).

Consequently, new approaches to teaching critical appraisal should incorporate elements of: transparency between skills teaching, learning outcomes, and assessment; opportunity for practice of the skills being taught; supportive group learning; and incentives to attend the teaching sessions such as teaching sessions linked to assessments. In addition, given the paucity of studies incorporating any assessment, particularly the quantitative objective assessment of teaching effectiveness, it is important to examine the effectiveness of a new approach using measurements of both students’ own views of the helpfulness of the skills teaching, ratings of their own critical appraisal ability, and their objective assessment grades. Further, much of the previous literature in this area has focussed on small numbers of students in the health sciences and medicine, thus it would seem important to incorporate and assess this type of teaching in other undergraduate settings with a relatively large group of students.

The present study aimed to examine the effectiveness of a set of teaching methods focused on teaching undergraduate Biosciences and Sport & Exercise Sciences students’ skills of critical appraisal as defined as the ability to evaluate information, question its validity in a logical evidence-based manner, and formulate inferences on the basis of the evidence. The methods mainly consisted of four seminars. Different types of source material were used to gradually increase in the depth of critical appraisal required and give multiple opportunities for practise. Seminars were planned with opportunity for facilitated group discussion. Coursework relying on critical appraisal skills was introduced once students had had the opportunity to practise these skills in a non-assessed context. Learning effectiveness was examined using student evaluation via qualitative questionnaires and rating scales, and the quantitative comparison of students’ coursework grades across the module. It was hypothesised that students would rate the teaching methods as helpful, that their confidence in their critical appraisal skills would increase over time, and that their coursework grades would improve following their implementation.

Method

Participants

Participants were 103 second year Sport and Exercise Sciences students and 37 Biosciences students opting to take the Behavioural Medicine module which commenced October 2007. Of the 113 who returned the initial questionnaires, the average age of 19.5 (+0.75) years, and 75 were female. The majority (88%) of participants were white Caucasian, with seven identifying themselves as Asian, and two as mixed race. Seven reported dyslexia.

Materials and Procedure

Critical analysis was developed by means of four 2-hour seminars. Each extended the skills raised in the previous seminar, and the subject matter was linked to that covered in the module. The second seminar was linked to the first piece of coursework required for the module, and the final two with the second piece of coursework. Each seminar began with an introduction by the seminar leader and group brain-storming followed by a task set in small group discussions, small group presentations, and finally whole-group discussion and feedback from the seminar leader. The seminar leader circulated around the small groups to facilitate discussion. The format and content of each seminar is shown in Figure 1.

Figure 1 Seminar format and content

In seminar 1, the grant proposals were designed to include some key flaws, and were based on real-life research questions in the area of Behavioural Medicine, for example, examining the influence of sleep deprivation on cytokine production in adults. The groups were asked to make a recommendation of whether to reject or accept the proposal and to discuss its flaws. They were then asked to imagine themselves as the researchers presenting the proposal and asked to redesign the proposal and improve on it. In seminar 2, the students discussed a published journal article, for example, the effects of caregiving stress on the antibody response to vaccination, and identify flaws in the science and its interpretation. In seminars 3 and 4, the discussion was based on two published studies aiming to examine the same phenomenon but with different findings, for example, the effects of watching a humorous film upon the secretion of salivary immunoglobulin A.

Questionnaires

Student perceptions of the seminars were quantitatively and qualitatively evaluated by anonymous questionnaire using Likert-type scales and open-ended format questions. The first two questionnaires collected basic demographic information and then asked participants to rate on an 8-point scale (where 0 = ‘not at all good’ to 7 = excellent) their pre-seminar critical appraisal ability. After the seminar they also rated: which parts of the seminar they had found to be the most interesting and useful; their critical appraisal ability now; and whether or not they thought the skills the seminar had taught would be useful for their coursework and more generally. For seminars 3 and 4, a questionnaire was administered at the end of seminar 4. This was more qualitative in nature and targeted specifically at the learning outcomes of the seminar. It asked first what the students thought the aims of the seminars were, then requested them to indicate on a 5-point scale (where 0 = ‘not at all’ and 4 = ‘very much’) the extent to which they thought the seminar helped them understand: what to look out for when critically appraising articles; the content of the articles; the key differences between the articles; what was required for the coursework; how the differences in studies could have led to different findings; and how to argue critically about the strengths and weaknesses of the two studies. They were also asked to indicate to what extent they found the seminars to be interesting and enjoyable and space made available for further comment.

Coursework

The first piece of coursework was a journal article review based on the critical appraisal of the journal article from seminar two. Marks were given for the critical discussion of the strengths, limitations and weaknesses of the study. The second coursework was a critical review of two journal articles which examined the same research question but had different findings. Marks were assigned for the critical discussion of why the results of the two studies differed.

Data Analysis

Data from the questionnaires and coursework grades were statistically analysed using SPSS version 15. Comparison of students’ ratings of their critical appraisal ability pre-seminar and post-seminar for seminars one and two was conducted using univariate Analysis of Variance (ANOVA). Analysis of covariance (ANCOVA) was used to assess whether the change in students ratings was influenced by the postgraduate seminar leader, gender, ethnicity, or dyslexia status. The difference between the two coursework grades across the year was examined using repeated measures ANOVA. Analysis of the change in grades over time by the type of student was assessed using a mixed model repeated measures ANOVA with type of student as the between-subject variable. Finally, a change in critical appraisal ability was created for seminars 1 and 2 by subtracting pre-seminar ratings of ability from post-ratings. Correlations were then run between students’ change in ratings of their critical appraisal ability, their ratings of the seminars, and their coursework grades to examine the association between any self-rated change in critical appraisal ability and objective coursework grades.

Results

Feedback Questionnaires

Students’ evaluation data from seminars 1 and 2 is shown in Table 1. Their ratings of critical appraisal ability increased from significantly (p < 0.001) pre- to post- seminar, p < 0.001, and this was not influenced by postgraduate demonstrator, gender, or dyslexia status, although the few Asian students rated their abilities as significantly higher overall than white or mixed race students (4.9±0.79, 4.2±0.95, and 3.4±0.18, respectively; p < 0.05). One hundred and six students (94%) felt that the seminar would help them in their coursework, and 99 (88%) agreed that it would help them in future academic and vocational work. Finally, students wrote further comments about the seminars, including: “Good group work — good test cases, lots to discuss & develop, made us consider lots of effects”, “it was a great way of interacting with others, and sharing people’s ideas”.

Table 1 Questionnaire results for the seminars 1 and 2

	Most interesting	Most useful	Mean (SD) pre-rating of critical appraisal (0-7)	Mean (SD) post-rating of critical appraisal (0-7)
Seminar 1 (n = 113 81% response rate)			3.4 (±1.06)	5.0 (±0.89)*
Critical appraisal of a grant proposal	66%	72%
Development of a revised proposal	54%	59%
Seminar 2 (n = 96; 69% response rate)			3.8 (±0.95)	4.0 (±0.89)*
Critical appraisal of the article	64%	36%

* p <0.001 significant increase from pre-rating to post-rating

NB. The students did not tend to rate one particular aspect of the seminar as the most interesting, but ticked all of the sections they found to be interesting, consequently the frequencies for each aspect of the seminar do not add up to 100%.

After seminar 2, the students again displayed a significant increase in self-rating of critical appraisal ability from the pre- to post-seminar, p < 0.001, again not influenced by postgraduate demonstrator, gender, ethnicity, or dyslexia status. The pre-seminar 2 rating was significantly higher than the previous pre-seminar rating for seminar 1, p < 0.05, but the post-seminar rating for seminar 2 was not significantly higher than the post-seminar rating for seminar 1. All of the respondents agreed that the seminar would help them with their coursework, and 83 (90%) agreed that it would help with future academic and vocational work. Finally, students wrote further comments about the seminar. These consisted of statements that the seminar was “Excellent!”, “Good & very helpful for the coursework” in terms of a “...clearer idea on how to do well on the coursework: “Good step by step take through of what we need”.

The responses to the question about the aims of seminars three and four resulted in many different responses. These were most commonly “to help with the coursework” or “to help us critically analyse journal articles”. Evaluations of the seminars are shown in Table 2, which shows that although the students rated the seminars as moderately successful in teaching them critical appraisal they did not necessarily find them interesting and enjoyable. Seminars taught by one particular demonstrator received significantly higher ratings (3.1±0.76) than those taught by the other two demonstrators (2.5±0.60, and 2.8±0.72) p < 0.05 with the exception of ratings for understanding the coursework. However, the mean differences in ratings between post-graduate demonstrators were relatively small, the largest being a difference of 0.87 for ratings of how well the seminars taught the skill of critical argument. Males also gave significantly higher ratings than females in response to the question about how well the seminars taught the skill of critical argument (2.8±0.82 and 2.5±0.86, respectively) and how interesting and enjoyable they found the seminars, (2.4±0.88 and 2.0±0.84, respectively) both p < 0.05. There were no differences in ratings for students of different ethnicities, or between those with and without dyslexia. Finally, 15 students also added suggestions in the section provided for other comments regarding ways to improve the seminars. The more helpful ones suggested that seminar 4 was more effective and perceived as more useful than seminar 3.

Table 2 Questionnaire results for the seminars 3 and 4

Seminars 3 and 4 N = 95 (68%)	Mean (SD) (scored 0-4)
Extent to which the seminars helped you to…
Know what to look out for when appraising journal articles	2.8 (±0.73)
Understand the content of the articles	2.7 (±0.76)
Understand the key differences between the articles	2.8 (±0.87)
Critically argue how differences might have led to different results	2.6 (±0.88)
Understand the coursework	2.8 (±0.85)
Extent to which you found the seminars interesting and enjoyable	2.2 (±0.87)

Coursework

Coursework numerical scores were compared across the module. The means and standard deviations are displayed in Table 3. When the grades for the first and second coursework were compared for the group as a whole, there was a significant decrease over time. Due to the intake of 37 Biosciences students not being familiar with the type of assessment used on this module, the grades are also summarised for Sport & Exercise Sciences and Biosciences students separately. Biosciences students scored lower overall across both pieces of coursework, p < 0.01, and a significant interaction effect was also observed between the change in grades over time from coursework 1 to coursework 2 and the type of student, p < 0.01, such that Sportex students’ grades significantly decreased from the first to the second coursework, but Biosciences students’ grades improved, although this was not quite significant at the 5% level (p=0.052).

Table 3 Mean (SD) for the coursework for the year group and split by type of student

		Journal Article Review	Critical Review of Two Papers
Group	N	Mean (SD)	Mean (SD)
2007-08	136	59.1 (±9.38)	56.8 (±10.44)*
2007-08 Sportex	103	61.0 (±8.94)	56.9 (±10.47)*
2007-08 Biosciences	33	52.9 (±8.06)	56.4 (±10.47)#

* p <.05 significant change in grade between the two pieces of coursework. # p = .052

There were no significant associations between students’ self-rated change in their critical appraisal ability from pre- to post-seminar (for seminars 1 and 2), their ratings of the seminars 3 and 4, or their grades for either piece of coursework.

Discussion

On the basis of the questionnaire feedback, the students seemed to find the seminars moderately enjoyable and useful. A positive finding was that, in general, students’ perceptions of the seminar teaching was not associated with their own gender, age, ethnicity, dyslexia status, or the particular teacher. In addition, the gender difference noted for two of the questions in relation to seminars 3 and 4 only appeared for these two variables, and the difference was relatively small (0.3, and 0.4 on 0-4 rating scales), indicating that on the whole the seminars had widespread appeal.

The lack of significant relationship between students’ objective assessment grades and their positive evaluation of their critical appraisal skills and the change in their self-rated critical skills due to the seminars may reflect a general tendency of students to overestimate their critical appraisal abilities. This mismatch has been observed by (Castle, 2006) who in developing critical thinking to student radiographers found that students seemed to over-estimate their critical skills. Another explanation and potential limitation of the present study is that the generally positive nature of the students’ responding on the seminar evaluation questionnaires could be, in part, due to socially desirable responding such that the students knew the aim of the seminars and responded in the way they felt the researcher wanted. Socially desirable responses have been cited previously as an explanation for the lack of relation between students’ self-report of their critical abilities and their actual grades (Hyde et al., 2000). However, (in an effort to minimise this type of responding) the questionnaires were anonymous and the researcher was not present during their completion.

An alternative explanation for the current finding of a lack of a positive association between students’ ratings of critical appraisal and their coursework grades is that it is simply not possible for certain students to progress to this level of critical thinking. Piagetian theory suggests that some individuals do not fully progress through the stages of development to formal operations, which incorporates deductive logic and critical hypothetical thinking (Piaget, 1936). However, this is perhaps less likely in students who have progressed as far as higher education.

Interesting differences were observed between the performance of the biosciences students who improved their coursework grades and those studying Sport and Exercise Sciences whose grades dropped between the two pieces of coursework (Table 3). It is possible that critical appraisal teaching may not be able to increase the grades of high ability students who are already in possession of many of these skills, but is capable of improving the grades of those who are less practised. Magnussen et al., (2000) observed for critical appraisal teaching taught through enquiry-based learning methods, a significant increase the mean critical thinking score of lower performing students, no change to the grades of those in the middle, and a drop in the grades of higher performing students. This is perhaps a result of the advantages of group and self-directed learning that enquiry-based approaches use (Magnussen et al., 2000). That the, originally higher performing, Sport and Exercise Sciences students’ grades dropped between the two pieces of coursework might also be explainable in this way. Alternatively, it is possible that the drop in grades by the Sportex students could reflect decreased effort following reasonable grades for the initial assessment, whereas Biosciences students might have increased their efforts given the lower grades for the journal article review. Unfortunately we have no data to test this.

For more challenging types of assessment, more practise at critical appraisal may be necessary before improvement in objective grades can be observed (Bensley and Haynes, 1995). In the present study, it was not possible to continue to follow the students to see whether or not their grades improved over subsequent years, but perhaps this type of strategy should be employed in future to examine the possibility that application of critical appraisal skills to written coursework is a competency that develops gradually.

The present study, naturally, has several limitations. First, coursework grades were the only objective measure of the success of the teaching, and it is debatable how well effective a measure these are. It remains possible that the critical appraisal teaching led to improvements in other ways which were not observed or measured in this project. Future developments may focus on the incorporation of some other type of follow-up such as objective assessment of the students’ critical appraisal and argumentation in the seminars themselves. This method has been used by others (Peters et al., 2002). Finally, it is possible that some students did not learn effectively the skills of critical appraisal. The ratings for the usefulness of seminars 3 and 4 were around the middle of the rating scale, indicating that there was room for improvement, although the extent to which ratings of usefulness indicate the extent of learning is debatable. Indeed, some of the students commented that the hypothetical listing of the possible differences between papers in seminar 3 was not particularly helpful, and that seminar 4 was far more useful in terms of the coursework and critical appraisal generally. It is likely that some changes should be made to these later seminars to improve their effectiveness. For example, seminar 3 could be used to summarise the key differences between the actual two studies being used. As a follow on, seminar 4 could then be used solely to discuss the differences previously identified which most might have contributed to the different findings, and logical arguments of how this might be the case.

Future developments emerging from this study may include additional assessments which are normative rather than summative, to help the students develop and practise their critical thinking skills and monitor their progress. In terms of the objectivity of the self-report data, it would be interesting to see whether students assessing their critical skills at the start and finish of the module were less affected by socially desirable responding within the seminar itself.

In conclusion, a series of seminars designed to gradually develop critical appraisal skills were used to increase undergraduate students’ critical appraisal. Students’ ratings of the critical appraisal skills showed improvements over time, although this was not reflected in or associated with their assessed coursework grades. However, it remains to be tested whether or not the use of more directly linked assessments, changes to the teaching methods, and longer follow-up over time might yield improvements in students’ abilities in terms of this important skill.

Acknowledgements

The author would like to acknowledge the invaluable help of the postgraduate seminar teachers: Dr Stephen Gallagher, Ms Elinor Olander, and Mr Aamer Sandoo.

Communicating Author

Dr Anna C. Phillips, School of Sport & Exercise Sciences, University of Birmingham, Birmingham, B15 2TT, UK. Tel: +44 121 4144398. Fax: +44 121 414 4121 e-mail: A.C.Phillips@bham.ac.uk

 

References

Abrami, P.C., Bernard, R.M., Borokhovski, E., Wade, A., Surkes, M.A., Tamim, R., et al. (2008). Instructional Interventions Affecting Critical Thinking Skills and Dispositions: A Stage 1 Meta-Analysis. Review of Educational Research, 78, 1102–1134
doi:10.3102/0034654308326084

Anderson, L., & Krathwhol, D. (2001). A Taxonomy for Learning, Teaching and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. New York: Longman

Audet, N., Gagnon, R., Ladouceur, R., & Marcil, M. (1993). [How effective is the teaching of critical analysis of scientific publications? Review of studies and their methodological quality]. Canadian Medical Association Journal, 148, 945–952

Bensley, D.A, and Haynes, C. (1995) The acquisitiong of general purpose strategic knowledge for argumentation. Teaching of Psychology, 22, 41–45
doi:10.1207/s15328023top2201_13

Beyer, B.K. (1985) Critical thinking: What is it? Social Education, 49, 270–276

Biggs, J. (2000) Teaching for Quality Learning at University. Buckingham: Society for Research into Higher Education & Open University Press

Bloom, B.S. et al. (1956) Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook 1: Cognitive Domain by a committee of college and university examiners. London: David McKay Co.

Bonk, C.J., & Smith, G.S. (1998) Alternative instructional strategies for creative and critical thinking in the accounting curriculum. Journal of Accounting Education, 16, 261–293
doi:10.1016/S0748-5751(98)00012-8

Castle, A. (2006) Assessment of the critical thinking skills of student radiographers. Radiography, 12, 88–95
doi:10.1016/j.radi.2005.03.004

Dexter, P., Applegate, M., Backer, J., Claytor, K., Keffer, J., Norton, B., & Ross, B. (1997) A proposed framework for teaching and evaluating critical thinking in nursing. Journal of Professional Nursing, 13, 160–167
doi:10.1016/S8755-7223(97)80067-1

Elder, L. (2007) Another Brief Conceptualization of Critical Thinking. www.criticalthinking.org (Accessed 23/09/2009)

Faccione, P.A. (2001) The teaching for thinking student course evaluation form. http://www.insightassessment.com/pdf_files/Eval%20Course%20Form%20CT.pdf Retrieved 1/06/2008

Halpern, D.F (1998) Teaching critical thinking for transfer across domains. American Psychologist, 53, 449–455
doi:10.1037/0003-066X.53.4.449
PMid:9572008

Hyde, C., Parkes, J., Deeks, J., & Milne, R. (2000) Systematic review of the effectiveness of teaching critical appraisal. Oxford: ICRF/NHS Centre for Statistics in Medicine.

Klimoviene, G., Urbonienė, J., & Barzdžiukienė, R. (2006) Developing Critical Thinking through Cooperative Learning. Studies about languages, 9, 77–84

Magnussen, L., Ishida, D., & Itano, J. (2000) The impact of the use of inquiry-based learning as a teaching methodology on the development of critical thinking. Journal of Nursing Education, 39, 360–364
PMid:11103974

Norman, G. R., & Shannon, S. I. (1998) Effectiveness of instruction in critical appraisal (evidence-based medicine) skills: a critical appraisal. Canadian Medical Association Journal, 158, 177–181
PMid:9469138    PMCid:1232690

Peters, M. W., Smith, M. F., & Smith, G. W. (2002) Use of critical interactive thinking exercises in teaching reproductive physiology to undergraduate students. Journal of Animal Science, 80, 862–865
PMid:11890425

Piaget, J. (1936) The origins of intelligence in children. New York: W.W. Norton & Company, Inc.

Pithers, R.T., & Soden, R. (2000) Critical thinking in education: A review. Educational Research, 42, 237–249

Scheffer, B. K., & Rubenfeld, M. G. (2000) A consensus statement on critical thinking in nursing. Journal of Nursing Education, 39, 352–359
PMid:11103973

Taylor, R., Reeves, B., Ewings, P., Binns, S., Keast, J., & Mears, R. (2000) A systematic review of the effectiveness of critical appraisal skills training for clinicians. Medical Education, 34, 120–125
doi:10.1046/j.1365-2923.2000.00574.x
PMid:10652064