Hypothesis testing and p-values are key tools for making decisions about population parameters based on sample data. They help researchers determine if observed results are statistically significant or likely due to chance.

P-values represent the probability of getting results as extreme as those observed if the null hypothesis is true. By comparing p-values to significance levels, we can decide whether to reject the null hypothesis and draw meaningful conclusions about our research questions.

Hypothesis Testing and P-Values

Interpretation of p-values

• Represents the probability of observing a sample statistic as extreme or more extreme than the one observed, assuming the null hypothesis is true
  • Small p-value indicates the observed sample result is unlikely to occur if the null hypothesis is true (p-value = 0.01)
  • Large p-value suggests the observed sample result is likely to occur under the null hypothesis (p-value = 0.8)
• Calculated based on the sampling distribution of the test statistic under the null hypothesis
  • Test statistic measures the difference between the observed sample statistic and the value expected under the null hypothesis (z-score, t-score)
  • Sampling distribution describes the variability of the test statistic when the null hypothesis is true (normal distribution, t-distribution)
• Affected by factors such as sample size, variability, and the magnitude of the difference between the observed sample statistic and the null hypothesis value
  • Larger sample sizes, lower variability, and greater differences between the observed and null values generally lead to smaller p-values (n = 100 vs n = 20)
  • Effect size quantifies the magnitude of the difference or relationship being studied

P-value vs significance level

• Significance level ($\alpha$) is a predetermined threshold for making decisions about the null hypothesis
  • Common significance levels are 0.01, 0.05, and 0.10 ($\alpha = 0.05$)
  • Choice of significance level depends on the consequences of making a Type I error (rejecting a true null hypothesis)
• Decision rule compares the p-value to the significance level to determine whether to reject or fail to reject the null hypothesis
  1. If $p \leq \alpha$, reject the null hypothesis in favor of the alternative hypothesis
  2. If $p > \alpha$, fail to reject the null hypothesis; insufficient evidence to support the alternative hypothesis
• Rejecting the null hypothesis suggests the observed sample result is statistically significant and unlikely to have occurred by chance alone under the null hypothesis (p-value = 0.02, $\alpha = 0.05$)
• Failing to reject the null hypothesis does not prove the null hypothesis is true; only indicates insufficient evidence to support the alternative hypothesis (p-value = 0.15, $\alpha = 0.05$)
• The critical value is the point on the test statistic distribution that corresponds to the significance level

Contextual conclusions for hypothesis tests

• State conclusions in the context of the original research question or problem
  • Avoid technical jargon and focus on practical implications
  • Relate conclusions back to the population of interest and variables being studied (college students, test scores)
• When the null hypothesis is rejected, conclude there is sufficient evidence to support the alternative hypothesis
  • "Based on the sample data, there is sufficient evidence to conclude that the average weight loss for the new diet plan is greater than 5 pounds."
  • "The survey results indicate a significant preference for Brand A over Brand B among consumers aged 18-34."
• When the null hypothesis is not rejected, conclude there is insufficient evidence to support the alternative hypothesis
  • "Based on the sample data, there is insufficient evidence to conclude that the proportion of defective products differs from the claimed value of 0.05."
  • "The study did not find a significant difference in job satisfaction between employees working remotely and those working in the office."
• Acknowledge limitations of the study and avoid overgeneralizing results
  • Consider sample size, representativeness of the sample, and potential confounding variables
  • "While the results suggest a significant difference in customer satisfaction between the two stores, further research with a larger sample size and more diverse locations is needed to generalize the findings to the entire chain."
  • "The experiment indicates a potential link between the new teaching method and improved test scores; however, additional studies are required to establish causality and control for other factors that may influence student performance."

Statistical Inference and Estimation

• Confidence intervals provide a range of plausible values for a population parameter
• Statistical power is the probability of correctly rejecting a false null hypothesis
• Standard error measures the variability of a sample statistic and is used in calculating confidence intervals and test statistics