Question:
Nothing comes for free: if you can cope with 400 words on statistics, wecan trash a front page news story together. "Cocaine floods theplayground," roared the front page of the Times last Friday. "Use of theaddictive drug by children doubles in a year."

Doubles? Now that was odd, because the press release for this governmentsurvey said it found "almost no change in patterns of drug use, drinkingor smoking since 2000". But the Telegraph ran with the story as well. Sodid the Mirror. Perhaps they had found the news themselves, buried in thereport.

So I got the document. It's a survey of 9,000 children, aged 11 to 15, in305 schools. The three-page summary said, again, there was no change inprevalence of drug use. I found the data tables, and for the questionabout using cocaine in the past year, 1% said yes in 2004, and 2% said yesin 2005. Except almost all the figures were 1%, or 2%. They'd all beenrounded off. By asking around, I found that the actual figures were 1.4%for 2004 and 1.9% for 2005, not 1% and 2%. So it hadn't doubled. But ifthat alone was my story, this would be a pretty lame column, so read on.

What we now have is an increase of 0.5%: out of 9,000 kids, about 45 morekids saying "yes" to the question. Presented with a small increase likethis, you have to think: is it statistically significant? Well, I did themaths, and the answer is yes, it is, in that you get a p-value of lessthan 0.05. What does that mean? Well, sometimes you might throw "heads"five times in a row, just by chance. Let's imagine that there wasdefinitely no difference in cocaine use, the odds were even, but you tookthe same survey 100 times: you might get a difference like we have seenhere just by chance, but less than five times out of your 100 surveys.

But this is an isolated figure. To "data mine", and take it out of itsreal world context, and say it is significant, is misleading. Thestatistical test for significance assumes that every data point, everychild, is independent. But, of course, here the data is "clustered". Theyare not data, they are real children, in 305 schools. They hang outtogether, they copy each other, they buy drugs off each other, there arecrazes, epidemics, group interactions.

The increase of 45 kids taking cocaine could have been three majorepidemics of cocaine use in three schools, or mini-epidemics in a handfulof schools. This makes our result less significant. The small increase of0.5% was only significant because it came from a large sample of 9,000data points - like 9,000 tosses of a coin - but if they're not independentdata points, then you have to treat it, in some respects, like a smallersample, and so the results become less significant. As statisticians wouldsay, you must "correct for clustering".

Then there is a final problem with the data. In the report, there aredozens of data points reported: on solvents, smoking, ketamine, cannabis,and so on. Standard practice in research is to say we only accept afinding as significant if it has a p-value of 0.05 or less. But like wesaid, a p-value of 0.05 means that for every 100 comparisons you do, fivewill be positive by chance alone. From this report you could have donedozens of comparisons, and some of them would indeed have been positive,but by chance alone, and the cocaine figure could be one of those. This iswhy statisticians do a "correction for multiple comparisons", which isparticularly brutal on the data, and often reduces the significance offindings dramatically, just like correcting for clustering can.

Mining is a dangerous profession - and data mining is just the same. Thisstory went from an increase of 0.5%, that might be a gradual trend, butcould well be an entirely chance finding, to being a front page story inthe Times about cocaine use doubling. You might not trust the pressrelease, but if you don't know your science, you take a big chance whenyou delve under the bonnet of a study to find a story.

Answer: In an inverse of the Condominium Rice pudding thread I'd call her *Ms*McKeith. She bought her PhD from the Clayton College of Natural Health(http://www.ccnh.edu/), a place whose "degrees" are accredited by theAmerican Association of Drugless Practitioners (http://www.aadp.net/) andthe American Naturopathic Medical Accreditation Board(http://www.anmab.org/pages/1/index.htm). I suppose quacks have to sticktogether.