Confidence and prediction intervals in multiple regression help us understand the uncertainty in our estimates. They show us the range of likely values for coefficients and future observations, giving us a clearer picture of how reliable our model is.

These intervals are crucial for making informed decisions based on our regression results. By quantifying uncertainty, they allow us to assess the practical significance of our findings and make more accurate predictions for new data points.

Confidence Intervals for Coefficients

Definition and Interpretation

• A confidence interval for a regression coefficient provides a range of values likely to contain the true population value of the coefficient with a specified level of confidence (typically 95%)
• Interpret a confidence interval as the range of plausible values for the true effect of a predictor variable on the response variable, given the observed data and the chosen confidence level
• Example: A 95% confidence interval for the coefficient of $X_1$ is (0.5, 1.2), suggesting that a one-unit increase in $X_1$ is associated with an increase in the response variable between 0.5 and 1.2 units, with 95% confidence

Calculation and Properties

• Calculate the confidence interval using the point estimate of the coefficient ($\hat{\beta}$), its standard error ($SE(\hat{\beta})$), and the critical value from the t-distribution with $(n-p-1)$ degrees of freedom
  • $n$ represents the sample size
  • $p$ represents the number of predictors
• The formula for a confidence interval is $\hat{\beta} \pm t_{\alpha/2, n-p-1} \times SE(\hat{\beta})$, where $\alpha$ is the significance level (e.g., 0.05 for a 95% confidence interval)
• A narrow confidence interval indicates a more precise estimate of the coefficient, while a wider interval suggests greater uncertainty
• If the confidence interval does not contain zero, the coefficient is considered statistically significant at the specified level of confidence

Prediction Intervals for New Observations

Definition and Interpretation

• A prediction interval provides a range of values likely to contain a future individual response ($Y$) for a given set of predictor values ($X_1, X_2, ..., X_p$) with a specified level of confidence (typically 95%)
• Interpret a prediction interval as the range of plausible values for a single new observation, given the observed data, the predictor values, and the chosen confidence level
• Example: A 95% prediction interval for a new observation with $X_1=10$ and $X_2=5$ is (75, 95), suggesting that the response value for this new observation is expected to fall between 75 and 95 with 95% confidence

Calculation and Properties

• Calculate the prediction interval using the fitted value ($\hat{Y}$), the standard error of the prediction ($SE(pred)$), and the critical value from the t-distribution with $(n-p-1)$ degrees of freedom
• The formula for a prediction interval is $\hat{Y} \pm t_{\alpha/2, n-p-1} \times SE(pred)$, where $SE(pred) = \sqrt{MSE \times (1 + h)}$
  • $MSE$ is the mean squared error
  • $h$ is the leverage of the new observation
• Prediction intervals are generally wider than confidence intervals for the mean response because they account for both the uncertainty in the estimated regression line and the variability of individual observations around the line
• The width of the prediction interval depends on the level of confidence, the sample size, the variability of the data, and the distance of the new observation from the center of the data

Confidence vs Prediction Intervals

Key Differences

• Confidence intervals estimate the range of plausible values for the true population coefficients, while prediction intervals estimate the range of values for a future individual response
• Confidence intervals are based on the standard errors of the coefficient estimates, while prediction intervals also incorporate the variability of individual observations around the regression line
• The width of a confidence interval depends on the sample size, the variability of the data, and the level of confidence, while the width of a prediction interval additionally depends on the distance of the new observation from the center of the data

Applications

• Use confidence intervals to assess the significance and precision of the estimated coefficients and to draw conclusions about the relationships between predictors and the response variable
  • Example: If the 95% confidence interval for a coefficient includes zero, the predictor is not considered statistically significant at the 0.05 level
• Use prediction intervals to provide a range of likely values for a new observation given specific predictor values and to quantify the uncertainty associated with individual predictions
  • Example: A manufacturer uses a prediction interval to estimate the range of product quality scores for a new batch based on the settings of the production process variables

Factors Affecting Interval Width

Data and Sample Characteristics

• Sample size: Larger sample sizes generally lead to narrower confidence and prediction intervals by providing more information and reducing the standard errors of the estimates
• Variability of the data: Higher variability in the response variable ($Y$) and the predictor variables ($X$) results in wider intervals due to increased uncertainty in the estimates
• Distance from the center of the data: For prediction intervals, observations further from the center of the data (i.e., with higher leverage) will have wider intervals due to less information available for precise predictions at the extremes

Model and Interval Specifications

• Level of confidence: Higher levels of confidence (e.g., 99% vs. 95%) result in wider intervals to capture the true parameter with the specified level of certainty
• Number of predictors: As the number of predictors ($p$) increases, the degrees of freedom decrease, potentially leading to wider intervals, especially when the sample size is small relative to the number of predictors
• Collinearity: High collinearity among the predictors can inflate the standard errors of the coefficient estimates, resulting in wider confidence intervals for the affected coefficients
• Example: Increasing the confidence level from 95% to 99% will widen both confidence and prediction intervals, as it requires a larger range of values to achieve the higher level of certainty