← All blogs

Confidence and Errors (Framework of Frequentist Hypothesis Testing)

Vineeth Bhat ·

  1. What’s there?
    1. Why do I write this?
  2. Errors
    1. Structure of a hypothesis test
    2. The errors themselves
  3. p-values
    1. First principles
    2. Theory
    3. (Mis)interpretations
    4. With an example
  4. Confidence intervals
    1. Fundamentals
    2. Theory
    3. Example
    4. (Mis)interpretations
  5. Sources

What’s there?

I wanted to note down Type I and II errors, p-values and confidence intervals.

The central challenge in science and data analysis is to distinguish a real effect from random chance.

I wish to understand this.

Why do I write this?

I wanted to create an authoritative document I could refer to for later. Thought that I might as well make it public (build in public, serve society and that sort of stuff).

Errors

Structure of a hypothesis test

  1. The Null Hypothesis (H_0): This is the default assumption, the hypothesis of “no effect” or “no difference.” It represents the status quo. For example, “H_0: The new drug has no effect on blood pressure.” The entire framework is designed to assess the strength of evidence against this hypothesis.
  2. The Alternative Hypothesis (H_A or H_1): This is the claim we wish to investigate. It represents a real effect or a true difference. For example, “H_A: The new drug lowers blood pressure.”

Based on the data we collect, we must make a decision: either reject the null hypothesis (in favor of the alternative) or fail to reject the null hypothesis.

The errors themselves

Given the true state of the world (which is unknown to us) and the decision we make, there are four possible outcomes, two of which are errors:

Decision: Fail to Reject H_0Decision: Reject H_0
Truth: H_0 is trueCorrect DecisionType I Error (False Positive)
Truth: H_A is trueType II Error (False Negative)Correct Decision (True Positive)

p-values

First principles

Theory

Formal Definition (p-value): The p-value is the probability, assuming the null hypothesis (H_0) is true, of observing a test statistic that is at least as extreme as the one actually observed.

p = P(\text{Test Statistic} \ge \text{Observed Statistic} | H_0 \text{ is true})

(Mis)interpretations

The Decision Rule: To make a decision, we compare the p-value to our pre-specified significance level \alpha.

With an example

You am a botanist (yay!). You have developed a new fertilizer. Does it make plants actually grow taller?

Your core question is: “If this fertilizer has no effect at all, how surprising is it that my fertilized plants grew so much taller than the unfertilized ones?”

To answer this, you set up an experiment:

After a month, you measure all the plants. You find that the fertilized plants are, on average, 2 cm taller. To standardize this result, you calculate a t-statistic. This single number boils down the difference in means, the variation in the data, and the sample size. Let’s say you calculate a t-statistic of 2.5.

Now, you need to know if t = 2.5 is a big deal. You turn to theory. Statisticians know that if the fertilizer had no effect (the null hypothesis is true), and you repeated this experiment many times, the t-statistics you’d get would form a specific bell-shaped curve called a t-distribution, centered at 0. This is your null distribution.

You check your null distribution to see how your result (t = 2.5) fits in. The p-value is the probability of getting a t-statistic that is at least as extreme as 2.5, assuming the fertilizer is useless. “Extreme” here means 2.5 or greater, OR -2.5 or less.

Let’s say the area under the tails of the curve for these extreme values is 0.012. So, your p-value = 0.012.

The p-value of 0.012 is a measure of surprise. It means that if the fertilizer had no effect, there would only be a 1.2% chance of seeing a height difference as large as the one you observed just by random luck. Because this chance is so small, you should be very surprised. This surprise leads you to strongly doubt your initial assumption that the fertilizer is useless.

This is the critical part. The p-value of 0.012 does NOT mean there’s a 1.2% chance that the null hypothesis is true (i.e., a 1.2% chance the fertilizer is useless). It’s a statement about your data’s rarity under the null hypothesis, not a statement about the hypothesis itself.

Before the experiment, you set a significance level, your threshold for surprise. Let’s use the standard \alpha = 0.05.

What if your p-value was 0.34?
In that case, since 0.34 > 0.05, you would fail to reject the null hypothesis. This doesn’t prove the fertilizer is useless. It just means this particular experiment didn’t provide strong enough evidence to convince you that it works.

Confidence intervals

Fundamentals

Theory

Formal Definition (Confidence Interval): A 100(1-\alpha)\% confidence interval for a parameter \theta is an interval [L, U], computed from the sample, such that if we were to repeat the sampling process an infinite number of times, 100(1-\alpha)\% of the thus-constructed intervals would contain the true, fixed parameter \theta.

P(L \le \theta \le U) = 1-\alpha

This probability statement is about the procedure of constructing the interval, not about the specific interval we have calculated.

Example

Construction (Example for a Mean): A confidence interval for a population mean \mu is typically constructed as:

\text{Point Estimate} \pm \text{Margin of Error}

\bar{x} \pm t^* \cdot \frac{s}{\sqrt{n}}

where:

(Mis)interpretations

Sources

  1. Mathematics for Machine Learning (link)