Hypothesis testing involves making decisions based on sample data, but errors can occur. Type I errors happen when we reject a true null hypothesis, while Type II errors occur when we fail to reject a false null hypothesis. Understanding these errors is crucial for interpreting statistical results.

The power of a test is the probability of correctly rejecting a false null hypothesis. Factors like sample size, effect size, and alpha level influence power. Balancing the risks of Type I and Type II errors is essential for designing effective hypothesis tests and drawing accurate conclusions.

Hypothesis Testing Errors and Outcomes

Type I vs Type II errors

• Type I error (False Positive) occurs when rejecting the null hypothesis even though it is actually true
  • Denoted by the Greek letter alpha ($\alpha$)
  • Leads to concluding an effect or difference exists when it does not (false drug efficacy)
  • Can result in unnecessary actions or changes based on incorrect conclusions (unnecessary medical treatment)
• Type II error (False Negative) happens when failing to reject the null hypothesis despite it being false
  • Denoted by the Greek letter beta ($\beta$)
  • Results in concluding no effect or difference exists when it actually does (missed cancer diagnosis)
  • Can lead to missed opportunities or failure to address important issues (untreated medical condition)
• Correct decisions in hypothesis testing involve
  • Rejecting the null hypothesis when it is false (True Positive) (correctly identifying a disease)
  • Failing to reject the null hypothesis when it is true (True Negative) (correctly identifying absence of disease)

Probabilities of hypothesis testing errors

• Alpha ($\alpha$) represents the probability of making a Type I error
  • Typically set by the researcher before conducting the test (0.05, 0.01)
  • Lower alpha values reduce Type I error risk but may increase Type II error risk (stricter significance level)
• Beta ($\beta$) represents the probability of making a Type II error
  • Depends on factors such as sample size, effect size, and alpha level
  • Can be calculated using statistical power (1 - $\beta$)
• Relationship between alpha and beta
  • Decreasing alpha (Type I error rate) generally increases beta (Type II error rate) if other factors remain constant
  • Balancing the risks of Type I and Type II errors is crucial in designing hypothesis tests (medical screening tests)

Power of the test concept

• Power of the test is the probability of correctly rejecting the null hypothesis when it is false
  • Calculated as 1 - $\beta$, where $\beta$ is the Type II error rate
  • Higher power indicates a greater likelihood of detecting a true effect or difference (drug effectiveness)
• Factors affecting power of the test include
  1. Sample size - larger sample sizes generally increase power by reducing sampling variability (clinical trial enrollment)
     • Increasing sample size can help detect smaller effects or differences
  2. Effect size - larger effects or differences are easier to detect and result in higher power (strong drug response)
  3. Alpha level - lower alpha levels (0.01) reduce power compared to higher levels (0.05)
• Power and Type II error rate ($\beta$) are inversely related
  • As power increases, the probability of making a Type II error decreases
  • Researchers aim to design studies with high power to minimize Type II error risk (well-powered clinical trials)

Statistical Decision Making

• Test statistic: A value calculated from sample data used to make decisions about the null hypothesis
• Critical value: The threshold that determines whether to reject or fail to reject the null hypothesis
• Decision rule: Guidelines for rejecting or failing to reject the null hypothesis based on the test statistic and critical value
• Statistical significance: When the test statistic exceeds the critical value, indicating strong evidence against the null hypothesis
• Confidence interval: A range of values likely to contain the true population parameter, providing a measure of uncertainty