Probabilistic foundations of Multiple Linear Regression.
Multiple Linear Regression is the principal statistical engine for multivariate modeling, econometric forecasting, and observational data analysis. Extending bivariate regression to arbitrary dimensions via matrix algebra, projection operators, and the Hat matrix, it enables simultaneous estimation of partial regression slopes while controlling for confounding covariates. In this laboratory, we construct design matrices, perform matrix OLS inversion, diagnose multicollinearity through Variance Inflation Factors, and confirm the partial slope scaling gradient m = 0.250. In mathematical statistics, empirical data science, and modern computational engineering, understanding the asymptotic sampling properties and theoretical behavior of Multiple Linear Regression, Matrix Formulation, and Multicollinearity Diagnostics provides crucial guarantees. Rigorous experimental validation connects foundational probability laws with observable empirical phenomena, ensuring that statistical inference remains valid under finite sample sizes and real-world data collection constraints.
Viva voce questions & analytical theory
How is Multiple Linear Regression formulated in compact matrix notation?
Multiple Linear Regression models a scalar response Y as a linear function of p - 1 predictors: Y = X β + ε, where Y is an n × 1 response vector, X is an n × p design matrix whose first column is a vector of ones (for the intercept) and remaining columns contain predictor values, β is a p × 1 vector of unknown regression coefficients, and ε is an n × 1 vector of unobserved errors with E[ε] = 0 and Cov(ε) = σ^2 I_n. The OLS solution minimizes ||Y - X β||^2, yielding normal equations (X^T X) β̂ = X^T Y. If X has full column rank p, the unique solution is β̂ = (X^T X)^(-1) X^T Y.
What is the Hat Matrix H, and what are its geometric properties?
The Hat Matrix H = X (X^T X)^(-1) X^T is the orthogonal projection matrix that projects the observed response vector Y onto the p-dimensional column space of X, yielding fitted values: Ŷ = H Y. H possesses three vital geometric properties: (1) Symmetric: H^T = H; (2) Idempotent: H^2 = H; (3) Trace equals rank: tr(H) = rank(X) = p. The diagonal elements h_{ii} = x_i^T (X^T X)^(-1) x_i are the leverage values of individual data points, satisfying 0 ≤ h_{ii} ≤ 1 and ∑ h_{ii} = p. The residual projection matrix is I - H, such that residuals are e = (I - H) Y.
What is multicollinearity, and what mathematical problems does it create?
Multicollinearity occurs when two or more predictor variables in X are highly linearly correlated. In the extreme case of perfect collinearity, rank(X) < p, making X^T X singular and non-invertible. In near-collinear regimes, X^T X is ill-conditioned (small determinant, large condition number). While OLS estimates remain unbiased, their sampling variances Cov(β̂) = σ^2 (X^T X)^(-1) explode dramatically. High multicollinearity causes: (1) Wide confidence intervals; (2) Unstable coefficient signs; (3) Inability to isolate the partial effect of individual predictors; and (4) High overall R² paired with individually non-significant t-tests.
What is the Variance Inflation Factor (VIF), and how is it interpreted?
The Variance Inflation Factor for predictor j is defined as VIF_j = 1 / (1 - R_j^2), where R_j^2 is the coefficient of determination obtained from regressing predictor X_j on all other p - 2 predictors in the model. The variance of the estimated coefficient can be factored as Var(β̂_j) = (σ^2 / S_{jj}) · VIF_j. Thus, VIF_j measures how much the variance of β̂_j is inflated by collinearity relative to an orthogonal design. A common rule of thumb considers VIF > 5 moderate collinearity, and VIF > 10 severe multicollinearity requiring remedial action (such as ridge regression, PCA, or variable deletion).
Why is Adjusted R-squared (R²_adj) preferred over ordinary R² in multiple regression?
Ordinary R² = 1 - SSE / SST is mathematically guaranteed to increase (or remain unchanged) whenever a new predictor is added to the model, even if the predictor is completely irrelevant white noise. To prevent over-parameterization, Henri Theil introduced Adjusted R-squared: R^2_adj = 1 - [SSE / (n - p)] / [SST / (n - 1)] = 1 - (1 - R^2) [(n - 1) / (n - p)]. R^2_adj penalizes model complexity by dividing sums of squares by their respective degrees of freedom, increasing only if the added predictor improves model fit more than would be expected by pure chance.
How is the omnibus F-test for overall regression significance formulated?
The omnibus test evaluates the global null hypothesis H_0: β_1 = β_2 = ··· = β_{p-1} = 0 (all slope coefficients are simultaneously zero). The test statistic partitions total variance: F = [SSR / (p - 1)] / [SSE / (n - p)] = [R^2 / (p - 1)] / [(1 - R^2) / (n - p)]. Under H_0 and normal errors, F follows Snedecor's F-distribution with (p - 1, n - p) degrees of freedom. A significant omnibus F indicates that the model explains a significant proportion of variance, even if individual predictors suffer from collinearity.
What are Cook's Distance and DFBETAS in multiple regression diagnostics?
Cook's Distance D_i measures the aggregate influence of deleting the i-th observation on all p fitted values simultaneously: D_i = (e_i^2 / (p s^2)) · [h_{ii} / (1 - h_{ii})^2]. Points with D_i > 4/n or D_i > 1 are deemed highly influential. DFBETAS_{j(i)} measures the standardized change in a specific regression coefficient β̂_j when observation i is omitted: DFBETAS_{j(i)} = (β̂_j - β̂_{j(i)}) / [s_{(i)} √(X^T X)^{-1}_{jj}]. Values |DFBETAS| > 2/√n flag observations that exert disproportionate leverage on individual slopes.
Experimental precautions and statistical checks
- Check the condition number of the Gram matrix X^T X to ensure numerical stability during matrix inversion.
- Verify that sample size n strictly exceeds the number of parameters p (n >> p) to avoid rank deficiency.
- Compute Variance Inflation Factors (VIF) for all predictors to diagnose multicollinearity before interpreting coefficients.
- Inspect partial regression plots (added-variable plots) to assess the marginal contribution of each regressor.
Theoretical assumptions & limitations
- Multiple regression identifies predictive linear associations, but cannot prove causal relationships without experimental control.
- Severe multicollinearity inflates parameter standard errors without reducing the model's overall predictive accuracy.