Least squares estimation is a crucial technique in multiple regression analysis. It helps find the best-fitting line by minimizing the sum of squared residuals, providing unbiased estimates of regression coefficients with the smallest variance among linear unbiased estimators.

Understanding coefficient interpretation is key to making sense of regression results. Coefficients show how much the response variable changes when a predictor changes, holding others constant. Their significance is determined through hypothesis testing using t-statistics and p-values.

Least Squares Estimation

Minimizing Sum of Squared Residuals

• The least squares method minimizes the sum of squared residuals to estimate the regression coefficients (parameters) in a multiple linear regression model
• The least squares estimates are obtained by solving a system of normal equations derived from the partial derivatives of the sum of squared residuals with respect to each coefficient
• The least squares estimates are unbiased and have the smallest variance among all linear unbiased estimators (BLUE) when the assumptions of the classical linear regression model are satisfied

Computing Least Squares Estimates

• The least squares estimates can be computed using matrix algebra, where the estimated coefficients are given by the formula: $\hat{\beta} = (X'X)^{-1}X'y$, where $X$ is the design matrix and $y$ is the vector of response values
  • The design matrix $X$ contains the values of the predictor variables for each observation
  • The vector $y$ contains the corresponding values of the response variable
• The standard errors of the estimated coefficients can be obtained from the diagonal elements of the variance-covariance matrix of the estimators, which is given by $\hat{\sigma}^2(X'X)^{-1}$, where $\hat{\sigma}^2$ is the unbiased estimator of the error variance
  • The standard errors provide a measure of the precision or uncertainty associated with the estimated coefficients
  • Larger standard errors indicate less precise estimates and suggest that the corresponding predictors may not be statistically significant in explaining the variation in the response variable

Coefficient Interpretation

Understanding Coefficient Estimates

• The estimated coefficients represent the change in the expected value of the response variable for a one-unit increase in the corresponding predictor variable, holding all other predictors constant (ceteris paribus)
  • For example, if the estimated coefficient for the predictor "age" is 0.5, it means that for every one-year increase in age, the expected value of the response variable increases by 0.5 units, assuming all other predictors remain constant
• The sign of the estimated coefficient indicates the direction of the relationship between the predictor and the response variable (positive or negative)
  • A positive coefficient suggests a direct relationship, where an increase in the predictor leads to an increase in the response variable
  • A negative coefficient suggests an inverse relationship, where an increase in the predictor leads to a decrease in the response variable
• The magnitude of the estimated coefficient depends on the scale of the predictor variable and should be interpreted in the context of the units of measurement
  • For instance, if the predictor "income" is measured in thousands of dollars, an estimated coefficient of 2.5 means that a $1,000 increase in income is associated with a 2.5-unit increase in the response variable

Hypothesis Testing and Significance

• The standard errors of the estimated coefficients provide a measure of the precision or uncertainty associated with the estimates
• The t-statistic, calculated as the ratio of the estimated coefficient to its standard error, can be used to test the hypothesis that the true coefficient is zero (i.e., the predictor has no effect on the response)
  • A large t-statistic (in absolute value) and a small p-value (typically < 0.05) suggest that the coefficient is statistically significant and that the predictor has a significant impact on the response variable
  • A small t-statistic and a large p-value indicate that the coefficient is not statistically significant and that the predictor may not be important in explaining the variation in the response variable

Model Goodness of Fit

Coefficient of Determination (R-squared)

• The coefficient of determination, denoted as R-squared, measures the proportion of the total variation in the response variable that is explained by the multiple regression model
• R-squared ranges from 0 to 1, with higher values indicating a better fit of the model to the data
  • An R-squared of 0 means that the model does not explain any of the variation in the response variable
  • An R-squared of 1 means that the model perfectly explains all of the variation in the response variable
• R-squared is calculated as the ratio of the explained sum of squares (ESS) to the total sum of squares (TSS), where:
  • ESS is the sum of squared differences between the predicted values and the mean response
  • TSS is the sum of squared differences between the observed values and the mean response
• R-squared can be interpreted as the square of the correlation coefficient between the observed and predicted values of the response variable

Adjusted R-squared and Model Selection

• The adjusted R-squared is a modified version of R-squared that accounts for the number of predictors in the model and penalizes the addition of irrelevant predictors
  • It is useful for comparing models with different numbers of predictors
  • The adjusted R-squared will only increase if a new predictor improves the model more than would be expected by chance
• While R-squared is a useful measure of goodness of fit, it should not be the sole criterion for model selection, as it does not account for the complexity of the model or the importance of individual predictors
  • Other factors to consider when selecting a model include the parsimony principle (preferring simpler models), the practical significance of the predictors, and the interpretability of the model

Regression Assumptions

Key Assumptions

• Linearity: The relationship between the response variable and the predictors is linear, meaning that the expected value of the response is a linear combination of the predictors
• Independence: The observations are independently sampled from the population, and the errors are uncorrelated with each other
• Homoscedasticity: The variance of the errors is constant across all levels of the predictors (i.e., the spread of the residuals is consistent)
• Normality: The errors are normally distributed with a mean of zero and a constant variance

Multicollinearity and Influential Observations

• No multicollinearity: The predictors are not highly correlated with each other, as this can lead to unstable and unreliable estimates of the coefficients
  • Multicollinearity can be detected using the variance inflation factor (VIF) or by examining the correlation matrix of the predictors
  • Solutions to multicollinearity include removing one of the correlated predictors, combining them into a single predictor, or using regularization techniques (ridge regression or lasso)
• No outliers or influential observations: The presence of outliers or influential observations can distort the least squares estimates and affect the validity of the model
  • Outliers are observations with unusually large residuals or extreme values of the predictors
  • Influential observations are those that have a disproportionate impact on the estimated coefficients or the model fit
  • Diagnostic plots (residual plots, leverage plots) and measures (Cook's distance, DFFITS) can be used to identify outliers and influential observations

Consequences of Assumption Violations

• Violations of these assumptions can lead to biased, inefficient, or inconsistent estimates of the coefficients and can affect the validity of hypothesis tests and confidence intervals
  • Non-linearity can be addressed by transforming the variables or using non-linear regression models
  • Non-independence can be addressed by using robust standard errors or modeling the correlation structure of the errors
  • Heteroscedasticity can be addressed by using weighted least squares or robust standard errors
  • Non-normality can be addressed by using robust regression methods or transforming the response variable