Many people often encounter p-value in life, however surprisingly some of them easily get confused by either p-value or
the alpha level, which are the two basic concepts in hypothesis testing. Today we try to give a very easy-to-get
version of these two terms.

So to understand p-value, we need first to know

What is an alpha value?

In hypothesis testing, there are usually two opposite hypothesis and by convention we would probably
set the one with more confidence as null hypothesis, set another again it. Without looking
at the data, we intuitively would assume null hypothesis is right, unless data strongly against it.
So null hypothesis commonly has to be the safe, conservative choice.

For example, we are researchers and we think that we made a
discovery of a real phenomenon. Then the null hypothis would be “we didn’t discover anything”.

But we insist on our theory, so let’s get some data to find out. Before analyzing the data
we need first set a standard for the testing. The standard is the alpha level, usually set of .05.
This means that assuming that the null hypothesis is true, we may reject the null only if the observed
data are so unusual that they would have occurred by chance at most 5 % of the time.
In our case, we would only be statistically sure that we indeed discover a new phenomenon
only if the data are so unusual that the chance of having them is lower than alpha value.
Thus the smaller the alpha, the more stringent the test (the more unlikely it is to find a statistically
significant result). If the chance of having observed data is higher than the alpha value, that means
it is nothing special, and could happen anytime, so we didn’t discover anything.

What is a p-value?

Once the alpha level has been set, a statistic (like the mean) is computed. Each statistic has an associated
probability value called a p-value, or the likelihood of an observed statistic occurring due to
chance, given the sampling distribution.

So alpha sets the standard for how extreme the data must be before we can reject the null
hypothesis. The p-value indicates how extreme the data are. More precisely, p-value means that probability
that we could have these data under null hypothesis. We compare the p-value with the
alpha to determine whether the observed data are statistically significantly different from the null
hypothesis:

• If the p-value is less than or equal to the alpha (p< .05), then we reject the null hypothesis, and
  we say the result is statistically significant.
• If the p-value is greater than alpha (p > .05), then we fail to reject the null hypothesis, and we say
  that the result is statistically nonsignificant (n.s.).