6 ways to tell if a statistic is true

1. Consider the source

   1. Who are the authors of the study and who paid for it to take place? Who is likely to benefit from the results of the study? For example, imagine reading the statistic that “80 % of dentists recommend Brand A toothpaste” as it “kills 99 % of germs”. If the dentists here had been paid in some way by the toothpaste company, which paid for the study, can we view the results as reliable? Not really, as somebody looks to gain financially from the results. And so we can naturally expect there to be some kind of bias towards the company. 
       
2. Look for the impact

   1. If you’re looking at a scientific study in a journal, it can be helpful to check the “impact factor” of that journal. The impact factor measures the rank of a journal based on the number of times its articles have been mentioned (or cited) by other researchers in their work. Generally, journals with high impact factors are considered to be more trustworthy as they have gone through more careful review processes. That said, beware of journal bias. This is where journals have a bias toward publishing positive results i.e. where a real difference is found among samples. But no one tends to publish the many experiments that have been tried and didn’t show conclusive differences. It would be useful for researchers to know which experiments had been tested and failed so to avoid doing unnecessary work repeating them.
       
3. Check the sample

   1. The idea of statistics is to test, mathematically, the probability that the difference you see between samples is real. It also gives a specific level of confidence that differences seen in the sample should be seen in the greater population too, allowing us to draw conclusions about the greater population from our sample results. This is when the idea of sample size and representation becomes very important. Ideally, if you wanted to know something about a certain population, you would ask everyone or perform the experiment on every member of that population. But this is often impractical and expensive. So, where possible, researchers try to choose a sample which is representative of the population as a whole. You couldn’t try to predict the results of a national election by polling 20 people in south London. You’d have to have a much larger sample size, and they should be from all over the UK. So what’s the best sample size? The bigger the better, as there will be less bias in the data. Check, however, how the sample was chosen. Was it selected at random? Was it people who had volunteered to take part? Frequently, people who choose to answer a voluntary survey have strong (usually negative) opinions, and this will introduce bias into the data.
       
4. Check the error margins

   1. There’s usually some room for error in statistics. But it’s helpful if we can calculate the uncertainty that our result will be different from the real population if we were able to survey the entire population. This is called a ‘confidence interval’. Say you took 100 randomly chosen men and measured their height, and got an average of 180 cm with a standard deviation of 20 cm (standard deviation is just how much each member of the sample differs from the average value). You would then choose a confidence interval (usually 90, 95 or 99 %) and do some calculations to determine the margin of error based on that level of confidence. Let’s say the error is plus or minus 6.5 cm at 95 % confidence. What this means is that 95 % of the time the true average height of the male population will fall into our interval (173.5 – 186.5 cm), the other 5 % of the time (or 1 in 20 experiments), it won’t. It’s always worth looking out for this key information. It’s clear from this too, that the larger the sample size, the better an estimate of the true value for the population.
       
5. Question the connection

   1. Statistics is basically about probabilities. When working with probabilities, it’s useful to understand the difference between dependent and independent events. Dependent events are when one event influences the probability of another event happening. For example, winning the lottery is dependent on buying a lottery ticket, since if you don’t buy a ticket you have zero chance of winning. Independent events are when one has no effect on the probability of the other happening. Buying a lottery ticket and owning a blue car are independent events as they have no connection to each other. Closely related to these ideas are the concepts of correlation and causation. Correlation is when two events seem to follow the same pattern. For example, data records show that the amount of ice cream eaten in New York and the number of murders are positively correlated – as more ice cream is sold, more murders seem to occur! But just because there’s this rather odd correlation doesn’t mean that one caused the change in the other. In other words, correlation doesn’t always mean causation. Clearly, eating more ice cream doesn’t make people more likely to be killers!
       
6. Watch out for misleading numbers

   1. If something shocks you or goes against all common wisdom and other respected research, it’s worth digging deeper. Do some simple checks such as checking that the percentages in a pie chart all add up to 100 %. Some surveys or even experiments will give very precise numbers for something they can’t be that precise about. For example, if a national survey reports that 3,150,234 people in the UK are cat owners, you would be right to question this exact number. It’s unlikely that everyone in the UK would have been asked about cat ownership. This is an estimate based on a sample and should have been reported as around 3 million. Watch out for data that has been presented in a misleading way in a figure, for example by stretching the axes of a graph to make it appear as if there is a greater difference between samples. Also, remember that if you double a very small effect, you still have a very small effect, but the misleading claim could be made that a particular treatment was twice as effective!