Book Review

Bad Science

By Ben Goldacre

London: Harper Perennial ISBN 978-0-00-728487-0 (2009) £8.99

Since 2003, Medical Doctor Ben Goldacre has written the Guardian Bad Science column in which he dismantles the sort of pseudoscience and sloppy science which is all too prevalent in media coverage of biomedicine. This book is a systematic collation of several of the stories that have evolved during the first five years of his campaign. Readers familiar with Goldacre’s newspaper articles or his website (www.badscience.net) will recognise many of the villains presented here, including homeopaths, nutritionists, slack journalists, pharmaceutical companies and AIDS dissenters.

Goldacre is keen to emphasise, however, that his purpose is not simply to lambast the peddlers of error but rather to “teach good science by examining the bad” (p165), adding that “the aim of this book is that you should be future-proofed against new variants of bullshit” (p87). I believe he is sufficiently successful in his endeavour that this book ought to be prescribed reading for all Year One students on bioscience, medicine (and journalism!) courses.

In particular, Bad Science emphasises the fundamental importance of evidence in validating any experiment or treatment. Goldacre is concerned both that the evidence exists in the first place and that – as far as it is possible to confirm – the data shows what it is purported to show. By looking at examples of poor practice we are introduced to key aspects of trial design, of appropriate statistics and the perils of overinterpretation of otherwise valid experiments.

Trial design: the emphasis on medical applications of science inevitably means that research discussed in the book tends to focus on experiment involving human subjects. Although this is a nuanced subset of the types of experiments we might be encouraging bioscientists to design, it nevertheless highlights some broader issues in the generation of objective data, including the use of controls, appropriate randomisation and providing sufficient detail and transparency in and write-up such that any reader will have a sporting chance of spotting the glitches.

(Mis)use of science literature: Goldacre examines four ways in which science literature is routinely mishandled. These are: extrapolating and overinterpreting data (e.g. taking findings from in vitro work on isolated cells and assuming the equivalent is true in vivo in a whole organism); underplaying ‘confounding variables’ (e.g. when simply observing differences between research subjects and assuming this correlated with the factor under investigation, without conducting an appropriately controlled intervention study); cherry-picking only results that fit the hypothesis (whilst ignoring data, possibly the majority of findings, that don’t fit our model); and referring to studies that are not published in peer-reviewed journals (and frequently not published at all). Our students might themselves be guilty of the first three and need to be aware of the fourth. It is vital, therefore, that they understand not only how to conduct experiments, but also the safeguards in the publishing process that mean peer-reviewed work carries greater authority than conference presentations or over-enthusiastic press releases.

Lies, damned lies and statistics: Goldacre draws the reader’s attention to two distinct abuses of statistics. Firstly, there is habit much beloved of newspaper headline writers to report ‘relative risk increase’ (e.g. factor Y doubles your chances of developing condition X) rather than ‘absolute risk increase’ (e.g. factor Y increases your risk of developing condition X from 0.2% to 0.4 %). Both approaches may describe the same data, but the latter is less dramatic, and hence less likely to sell papers. Goldacre actually advocates a move towards quoting ‘natural frequencies’ as the must transparently intelligible form or numbers (e.g. “2 in every 1000 people will have condition X and this increases to 4 in every 1000 if they are exposed to factor Y”).

The second statistical ‘trick’ is the use of the wrong sort of test. There is, of course, the risk that we do this accidentally, but Goldacre targets those who do so deliberately because use of an inappropriate statistical test generates positive-sounding numbers. Some pharmaceutical companies are said to be guilty of this sleight of hand. This abuse is more subtle than the first, and more difficult to detect without a certain amount of statistical nous, raising the importance of maintaining sufficient maths within the bioscience curriculum, even if the students find it hard.

Using Bad Science to promote better science
As indicated above, I believe that Bad Science would be valuable reading for first year undergraduates. There would be much merit in having this as prescribed reading for a Year One skills or introductory module. In my role as an admissions tutor, I receive several e-mails each summer from students starting the following term and asking which textbooks to buy. My consistent response this time around has been to recommended that they read Bad Science now and wait until the course has started before they part with money for a chunky tome.

This is not to say that Bad Science is without faults. I do have a number of minor gripes about the book. Transposing a series of short newspaper articles into a longer format runs the risk of repetition and/or material being presented ‘out of sequence’. I think there are occasions when the book has fallen foul of both of these traps.

My edition (Harper Perennial, 2009) proudly boasts that it includes an extra chapter previously excluded due to a court case, now resolved. Although the index has been updated to reflect changes in page numbering resulting from the splicing in of the additional material, there are various in-text cross-references that have not been corrected.

I was also disappointed at times with the referencing. Granted this is a popular science book not an academic work, and it would be inappropriate to expect extensive footnotes. Nevertheless, since the author is frequently critical of the lack of referencing in media reports of science, I was disappointed on a couple of occasions to find no citation at the back offering indication of where I could follow up on the “huge amount of research” (p75).or find out the back-story prompting the statement “there is a lot of history here” (p.144). If a new edition is produced, please can these be added.

These are, however, minor quibbles. On balance Bad Science is an excellent and highly readable resource which I thoroughly recommend to you and your students.

[A longer version of this review is available on my blog; Journal of the Left-Handed Biochemist]

 

Reviewed by

Chris Willmott

Department of Biochemistry
University of Leicester
Leicester UK, LE1 9HN