A significance test is a formal procedure for using observed data to decide between two competing claims also called hypotheses. The claims are often statements about a parameter, like the population proportion p or the population mean μ.
Null Hypothesis H0 & Alternative Hypothesis Ha:
The claim we weigh evidence against in statistical tests is the null hypothesis H0. Often the null hypothesis is a statement of "no difference”.
The claim about the population that we are trying to find evidence for is the alternative hypothesis Ha.
One-sided alternative hypothesis and two-sided alternative hypothesis
The alternative hypothesis is one-sided if it states that a value is larger than the null hypothesis value or if it states that the value is smaller than the null hypothesis. It is two-sided if it states that the perimeter is different from the no hypothesis value (it could be either larger or smaller).
It is common to refer to a significance test with a one-sided alternative hypothesis as a one-sided test or one tailed test and a test with a two-sided alternative hypothesis as a two-sided test or two tailed test.
The null hypothesis has the form H0: parameter = value. The alternative hypothesis has one of the forms below: Ha: parameter < value (one tailed) Ha: parameter > value (one tailed) Ha: parameter ≠ value (two tailed)
To determine the correct form of Ha, read the problem carefully. Hypotheses always refer to a population, not to a sample. Be sure to state H0 and Ha in terms of population parameters. It's never correct to write a hypothesis about a sample statistic, such as p̂ = 0.64 or x̄ > 85.
Important note: To recognize that a problem is calling for a specific test, look for key phrases such as "has the true value changed?" Or “is there convincing evidence that the true value is different?” Also, a question might ask the students to "test the claim that μ = 10" without suggesting a direction for the alternative. In cases like this, use a two sided test.
Significance tests ask if sample data give convincing evidence against the null hypothesis in favor of the alternative hypothesis. A test answers a question, "how likely is it to get a result like this just by chance when the null hypothesis is true?" The answer comes in the form of a probability.
Statistical tests use an elaborate vocabulary, but the basic idea is this: an outcome that would rarely happen if the null hypothesis were true is good evidence that the null hypothesis is not true. The idea of stating a null hypothesis to find evidence against it seems odd at first.
It may help to think of a criminal trial. The defendant is "innocent until proven guilty.” That is, the null hypothesis is innocent and the prosecution must try to provide convincing evidence against this hypothesis in favor of the alternative hypothesis: guilt. That's exactly how statistical tests work; although in statistics, we deal with evidence provided by data and use probability to see how strong the evidence is.
The probability that measures the strength of the evidence against H0 (in favor of Ha) is called a P value.
The probability, computed assuming H0 is true, that a statistic such as p̂ or x̄ would take a value as extreme as or more extreme than one actually observed is called the P value of the test.
Small P values are evidence against each other because they say that the observed result is unlikely to occur when H0 is true. Large P values fail to give convincing evidence against H0 and in favor of Ha because they say that the observed result is likely to occur by chance alone when H0 is true.
Interpreting P-value
Assuming the proportion/mean that [variable] is [null hypothesis], there is [p-value]% of getting a difference of sample proportions/means [tail value] or greater OR -[tail value] or lower, purely by chance.
Statistical Significance
The final step in performing a significance test is to draw a conclusion about the competing claims you were testing. We will make one of two decisions based on the strength of the evidence against the null hypothesis (in favor of the alternative hypothesis)—reject H0 or fail to reject H0. If our sample result is too unlikely to have happened by chance assuming H0 is true, then we will reject H0 and say that there's convincing evidence for Ha.
Otherwise, we will fail to reject H0 and say that there is not convincing evidence for Ha. This wording may seem unusual at first, but it's consistent with what happens in a criminal trial. Once the jury has waived the evidence against the null hypothesis of innocence, they return one of two verdicts: "guilty “(reject H0) or "not guilty“ (fail to reject H0). A not guilty verdict doesn't guarantee that the defendant is innocent, just that there's not convincing evidence of guilt.
Likewise, a fail to reject H0 decision in a significance test doesn't mean the H0 is true. For that reason you should never “accept H0" or use language that you believe H0 is true.
P-Value small -> reject H0 -> convincing evidence for Ha (in context) P-Value large -> fail to reject H0 -> not convincing evidence for Ha (in context)
There is no rule for how small a p value is required to reject H0, but we can compare the value with a fixed value we regard as decisive, called the significance level a.
If we choose a = 0.05, we are requiring that the data give evidence against H0 so strong that it would happen less than 5% of the time just by chance when H0 is true. If we choose a = 0.01, we are insisting on stronger evidence against the null hypothesis, a result that would occur less often than 1 in every 100 times by chance alone when H0 is true.
P-Value smaller than a -> reject H0 -> convincing evidence for Ha (in context) P-Value larger than or equal to a -> fail to reject H0 -> not convincing evidence for Ha (in context)
0.05 is the most common significance level used. If the word problem does not already give a significance level to use, choose 0.05.
Type errors
You can make type errors if there is a problem in your experimental techniques or sampling process. Keep in mind that in certain situations, a specific type error would be more dangerous. For example, if you were testing for a certain chronic disease, a false negative (Type 2) is more dangerous than a false positive (Type 1).
In the analogy where H0 is innocence, Type 1 Error: reject H0 when H0 is true (convict innocent person) Type 2 Error: fail to reject H0 when Ha is true (let guilty go free)
