Resampling methods are game-changers in statistics. They let you analyze data without making assumptions about its distribution. By repeatedly sampling from your original data, you can estimate important stats and test hypotheses more accurately.

Bootstrap and permutation tests are the two main types of resampling. Bootstrap samples with replacement to estimate sampling distributions, while permutation tests shuffle data labels to assess significance. Both are super useful when traditional methods fall short.

Resampling Methods in Inference

Principles and Applications

• Resampling methods involve repeatedly sampling from the original data to create new datasets
• Allows for the estimation of sampling distributions and the assessment of statistical properties without relying on parametric assumptions
• Particularly useful when the underlying population distribution is unknown, the sample size is small, or when the assumptions of traditional parametric methods are violated (normality, homoscedasticity)
• Can be used for a wide range of applications
  • Estimating standard errors
  • Constructing confidence intervals
  • Conducting hypothesis tests
  • Assessing model stability and robustness

Types of Resampling Methods

• The two main types of resampling methods are bootstrap and permutation tests
  • Bootstrap methods involve sampling with replacement from the original data, preserving the original sample size
  • Permutation methods involve shuffling the labels or assignments of the original data, preserving the structure within each group or condition
• Bootstrap and permutation methods differ in their approach to generating new datasets and the types of inferences they enable
  • Bootstrap methods are primarily used for estimating sampling distributions, standard errors, and confidence intervals
  • Permutation methods are primarily used for assessing statistical significance and testing hypotheses

Bootstrap Methods for Analysis

Estimating Standard Errors and Confidence Intervals

• The bootstrap method involves repeatedly sampling with replacement from the original data to create a large number of bootstrap samples, each of the same size as the original dataset
• The statistic of interest (mean, median, correlation coefficient) is calculated for each bootstrap sample
• The distribution of these statistics across the bootstrap samples is used to estimate the sampling distribution of the statistic
• The standard error of a statistic can be estimated by calculating the standard deviation of the statistic across the bootstrap samples, providing a measure of the variability of the statistic
• Bootstrap confidence intervals can be constructed using various methods
  • Percentile method
  • Bias-corrected and accelerated (BCa) method
  • Bootstrap-t method
• These methods take into account the distribution of the bootstrap estimates

Hypothesis Testing with Bootstrap

• Hypothesis testing can be conducted using the bootstrap by comparing the observed statistic to the distribution of the statistic under the null hypothesis
• The distribution of the statistic under the null hypothesis is obtained by bootstrapping the data under the constraints of the null hypothesis
• The p-value is calculated as the proportion of bootstrap samples with a statistic as extreme as or more extreme than the observed statistic

Permutation Tests for Significance

Conducting Permutation Tests

• Permutation tests involve randomly shuffling the labels or assignments of the original data to create a large number of permuted datasets
• The structure of the data within each group or condition is preserved during the permutation process
• The statistic of interest (difference in means, correlation coefficient) is calculated for each permuted dataset
• This creates a distribution of the statistic under the null hypothesis of no effect or no difference between groups
• The observed statistic from the original data is compared to the distribution of the statistic under the null hypothesis
• The p-value is calculated as the proportion of permuted datasets with a statistic as extreme as or more extreme than the observed statistic

Applications of Permutation Tests

• Permutation tests are particularly useful for assessing statistical significance in situations where the assumptions of traditional parametric tests are violated
  • Non-normality
  • Heteroscedasticity
  • Small sample sizes
• Can be applied to various research designs
  • Two-sample tests
  • Paired tests
  • Analysis of variance (ANOVA)
  • Regression models
• Permutation tests are conducted by appropriately permuting the data labels or residuals based on the research design

Bootstrap vs Permutation Methods

Comparison of Bootstrap and Permutation Methods

• Bootstrap and permutation methods are both resampling techniques, but they differ in their approach to generating new datasets and the types of inferences they enable
• Bootstrap methods are more versatile and can be used for a wider range of applications
  • Estimating standard errors
  • Constructing confidence intervals
  • Conducting hypothesis tests
• Permutation methods are more specifically focused on assessing statistical significance and testing hypotheses

Choosing Between Bootstrap and Permutation Methods

• The choice between bootstrap and permutation methods depends on the research question, the nature of the data, and the assumptions that can be made
• Bootstrap methods are often used when the goal is to estimate sampling distributions and confidence intervals
• Permutation methods are preferred when the focus is on hypothesis testing and assessing statistical significance
• In some cases, a combination of bootstrap and permutation methods can be used (permutation test with bootstrap-based confidence intervals)
  • Provides a more comprehensive and robust analysis of the data