{"id":456,"date":"2018-07-31T12:05:19","date_gmt":"2018-07-31T12:05:19","guid":{"rendered":"https:\/\/onemol.org.uk\/?page_id=456"},"modified":"2025-09-21T18:34:10","modified_gmt":"2025-09-21T18:34:10","slug":"papers-about-p-values","status":"publish","type":"page","link":"https:\/\/onemol.org.uk\/?page_id=456","title":{"rendered":"Some papers about p<\/em> values"},"content":{"rendered":"\n

Jump to follow-up<\/span><\/a><\/p>\n\n\n\n

These papers have nothing much to do with single molecule kinetics. They were written by David Colquhoun after his retirement from the world of single ion channels, as a way to keep him off the streets. They are listed here as a convenient place to keep a record.<\/p>\n\n\n\n

The papers concern the misinterpretation of tests of significance. Such tests were barely ever used in our single ion channel work. They represent a return to the interest of DC in statistical inference that he had in the 1960s, and which culminated on the publication of a textbook, Lectures on Biostatistics<\/em><\/a> (OUP, 1971). The textbook has aged quite well, with the exception of the parts on interpretation of p<\/em> values. In the 1960s, I missed entirely the problems of null hypothesis significance testing. But better late than never.<\/p>\n\n\n\n

The problem lies in the fact that most people still think that the p<\/em> value is the probability that your results occurred by chance -see, for example. Gigerenzer et al<\/em>.,(2006) [download pdf]<\/a>. It is nothing of the sort.<\/p>\n\n\n\n

The false positive risk (FPR) is the probability that a result that has been labelled as “statistically significant” is in fact a false positive. It is always bigger than the p<\/em> value, often much bigger.<\/p>\n\n\n\n

My recommendations<\/strong>. In brief, I suggest that p<\/em> values and confidence intervals should still be cited, but they should be supplemented by a single number that gives an idea of the false positive risk (FPR). The simplest way to do this is to calculate the false positive risk that corresponds to a prior probability of there being a real effect of 0.5. This would still be optimistic for implausible hypotheses but it would be a great improvement on p<\/em> values. The FPR50<\/sub>, calculated in this way is just a more comprehensible way of citing likelihood ratio (see 2019 paper<\/a>). And the likelihood ratio (not the p<\/em> value) is what sums up the evidence from your experiment; it does so in an entirely frequentist way, so avoiding the problem of priors that is the bane of Bayesian approaches.<\/p>\n\n\n\n

Please note<\/strong>: the term “false discovery rate”, which was used in earlier papers, has now been replaced by “false positive risk”. The reasons for this change are explained in the introduction of the 2017 paper<\/a>.<\/p>\n\n\n\n

If you prefer a video to reading, try this, on YouTube<\/a>.<\/p>\n\n\n\n

Original papers about the problem<\/h3>\n\n\n\n

<\/a>
Colquhoun, D. (2014)<\/strong>. An investigation of the false discovery rate and the misinterpretation of p<\/em>-values. Open access at Royal Society Open Science<\/em><\/a> This first paper looked at the risk of false positive results by simulation of Student’s t<\/em> test. The advantage of simulation is that it makes the assumptions very clear without much mathematics. The disadvantage is that the results aren’t very general.<\/em> #DC2014 <\/span><\/p>\n\n\n\n

<\/a><\/p>\n\n\n\n

Colquhoun, D. (2017)<\/strong>. The reproducibility of research and the misinterpretation of p<\/em>-values . Open Access at Royal Society Open Science<\/em><\/a>. This paper gives, in the appendix, mathematically exact solutions or the false positive risk, calculated by the p-equals method. This allows the false positive risk to be calculated, as a function of the observed p value, for a range of sample sizes. A web calculator is provided that makes the calculations simple to do.<\/em><\/span> #DC2017<\/p>\n\n\n\n

<\/a><\/p>\n\n\n\n

\n

Colquhoun, D. and Longstaff, C. (2017)<\/strong>. Web calculator<\/strong> for false positive risk https:\/\/fpr-calc.ucl.ac.uk\/<\/a><\/p>\n<\/div><\/div>\n\n\n\n

Update<\/em><\/strong>. Thanks to a cock-up by UCL, the web calculator was offline for a while. Thanks to a ingenious solution provided by the wonderful people at our web hosts, Positive Internet<\/a>, the original link, http:\/\/fpr-calc.ucl.ac.uk\/<\/a>, now works again. That link is actually redirected to an html page (http:\/\/fpr-calc.positive-dedicated.net\/) on the Positive Internet server in which is embedded the back up copy that is kindly hosted by Daniel Lakens in Eindhoven, Netherlands. The only difference is that this version on longer rearranges on a mobile phone. #fpr-calc <\/p><\/p>\n\n\n\n

You can also go directly the server at Eindhoven University of Technology, at https:\/\/shiny.ieis.tue.nl\/fpr_calc\/<\/a>.<\/span><\/p>\n\n\n\n

There is also a copy of the web calculator at https:\/\/davidcolquhoun.shinyapps.io\/fpr-calc-ver1-7\/<\/a><\/p>\n\n\n\n

<\/a><\/p>\n\n\n\n

Colquhoun, D. (2019)<\/strong>. The false positive risk: a proposal concerning what to do about p<\/em> values, The American Statistician<\/em>, 2019<\/a> (open access to full text). Also available on arXiv). This paper examines more closely than before the assumptions that are made in calculations of FPR. It makes concrete proposals about how to solve the problem posed by the inadequacy of p<\/em> values, with examples.<\/em><\/span> #DC2019<\/p>\n\n\n\n

In the same online edition, The American Statistician<\/em> published 43 papers<\/a> that were designed to say what should<\/em> be done about the problem of abuse of p<\/em> values. One of these makes suggestions that are similar to mine, but based on even simpler calculations -see Benjamin and Berger (2019)<\/a>. The only problem with their approach is that it doesn’t cater for the Jeffreys-Lindley phenomenon<\/a> and it is therefore not suitable for very large samples.<\/p>\n\n\n\n

At the same time, Nature<\/em> published a comment piece<\/a> on the p<\/em> value problem. The gist of this piece was a plea to abandon the term “statistically-significant”, because it involves the obviously silly idea that observing p<\/em> = 0.049 tells you something different from p<\/em> = 0.051. It was co-signed by 840 people<\/a> (including me). Nature<\/em> also published an editorial<\/a> which half-understood the problem and, sadly, said “Nature<\/em> is not seeking to change how it considers statistical analysis in evaluation of papers at this time”. This sums up the problem: it is in the interests of both authors, and of journals, to continue to publish too many false positive results. Until this problem is solved, the corruption will continue.<\/p>\n\n\n\n

<\/a><\/p>\n\n\n\n

Colquhoun, D. (2019b)<\/strong>. A response to critiques of \u2018The reproducibility of research and the misinterpretation of p<\/em>-values\u2019. Royal Society Open Science<\/em><\/a>. This one started life as a response to a critique of my 2017 paper. But it evolved into a more general discussion of the assumptions made in my approach, and concluded with a summary of my present views about what should be done about p<\/em> values. In brief, I now think that p<\/em> values and confidence intervals should continue to be given, but they should be supplemented by an estimate of the false positive risk. I suggest the notation FPR50<\/sub> for the false positive risk that’s calculated on the basis that the prior probability of a real effect existing is 0.5. <\/em><\/span><\/p>\n\n\n\n

Popular accounts of the problem<\/h3>\n\n\n\n

Colquhoun, D. (2015) False discovery rates and P values: the movie. On YouTube<\/a>. This slide show is now superseded by the 2018 version<\/a>.<\/em><\/span><\/p>\n\n\n\n

<\/a><\/p>\n\n\n\n

Colquhoun, D. (2015)<\/strong>. The perils of p-values. In Chalkdust magazine<\/em>. Available at http:\/\/chalkdustmagazine.com\/features\/the-perils-of-p-values\/<\/a>. Chalkdust is a magazine run by students at UCL. This article deals with the principles of randomisation tests as a non-mathematical way to get p values, plus a bit about what’s wrong with p values.<\/em><\/span><\/p>\n\n\n\n

<\/a><\/p>\n\n\n\n

Colquhoun,D.(2015)<\/strong>. Randomisation tests. How to get a P value with no mathematics. A short (6 slides, 15 min) video on YouTube<\/a>. Forget t tests. The randomisation test is at least as powerful and it makes no assumption of normal distributions. Furthermore it makes very clear the fact that random allocation of treatments is an essential assumption for all tests of statistical significance. Of course the result is just a p value. It doesn’t tell you the probability that you are wrong: for that, see the other stuff on this page.<\/em><\/span><\/p>\n\n\n\n

<\/a><\/p>\n\n\n\n

Colquhoun, D.(2016)<\/strong>. The problem with p<\/em>-values. Aeon magazine<\/a>. (This attracted 147 comments<\/a>.
This essay is about the logic of inductive inferencee<\/em>. It is a non-mathematical introduction to the ideas raised in my 2014 paper.<\/em> <\/span><\/p>\n\n\n\n

<\/a><\/p>\n\n\n\n

Colquhoun, D. (2017)<\/strong>. Five ways to fix statistics. State false positive risk, too. Nature<\/em>, volume 251<\/a>. A collection of short comments by five authors on what should be done about p values.<\/em><\/span><\/p>\n\n\n\n

<\/a><\/p>\n\n\n\n

Colquhoun, D. (2018)<\/strong>. The false positive risk: a proposal concerning what to do about p<\/em>-values (version 2). This video is a slightly extended version of a talk that I gave at the Evidence Live meeting, June 2018, at the Centre for Evidence-Based Medicine, Oxford. It supersedes my earlier 2015 video on the same topic. It is an exposition of the ideas that are given in more detail in the 2017 paper and in the 2019 paper. In November 2019 an new version was posted -the content is the same, but the volume of the sound track is better. <\/em> <\/span><\/p>\n\n\n\n