Errors in hypothesis testing

\(P\) is not the probability that the null hypothesis is true

The \(P\)-value is a probability; it is probability that the distance between \(\bar{X}\) and \(\mu\) is at least as big as the distance between the observed sample mean and \(\mu\), if the \(\mu\) corresponds to the null hypothesis. To find the probability we have assumed that the value of the population parameter corresponds to the null hypothesis. Sometimes the \(P\)-value is interpreted as quantifying the probability that the null hypothesis is correct or true. This is wrong. In our tuna example, it's wrong to say that "the probability that the true mean weight of the cans is 95 is 0.012". We don't assign a probability to the null hypothesis. The \(P\)-value arises from asking how unusual our observed sample mean is, given we assumed it arose from a distribution centred at the null hypothesis. Figure 2 reminds us of this.

\(1-P\) is not the probability that the null hypothesis is false

The mistake of interpreting the \(P\)-value as indicating the probability that the null hypothesis is true has a corollary, which is also an error in interpretation. Sometimes \(1-P\) is interpreted as measuring the probability that the null hypothesis is incorrect or false. Again, this is also wrong.

Next page - Errors in hypothesis testing - Large \(P\)-values do not prove the null hypothesis