Mathematical foundations of Least-Squares Curve Fitting.
When analyzing physical experiments, engineering measurements, or observational data, recorded values are inevitably contaminated by stochastic noise and instrumental uncertainties. In such settings, forcing a mathematical model to pass exactly through every point via interpolation is counterproductive, amplifying noise and creating spurious oscillations. The method of least squares, developed independently by Carl Friedrich Gauss and Adrien-Marie Legendre, seeks a model that minimizes the cumulative squared deviation between observations and predictions. In this laboratory, we construct the Vandermonde design matrix, assemble the normal equations, investigate the condition number squaring effect, and extract the optimal regression parameter β = 2.500.
Viva voce questions & analytical theory
Why is least-squares approximation preferred over exact interpolation for experimental data?
Experimental data contains random measurement noise. Exact interpolation forces the curve to pass through every noisy point, yielding wild unphysical oscillations. Least-squares regression filters out uncorrelated noise by finding a low-dimensional manifold that minimizes total squared error.
How are the normal equations derived geometrically?
The model response vector ŷ = Xβ lies in the column space col(X). The distance ||y - ŷ||₂ is minimized when the residual vector r = y - ŷ is orthogonal to col(X), meaning Xᵀ r = 0. Substituting r = y - Xβ yields Xᵀ (y - Xβ) = 0, or (Xᵀ X) β = Xᵀ y.
What is the primary numerical danger of forming the normal equations directly?
The condition number of the Gram matrix is the square of the condition number of the design matrix: κ(Xᵀ X) = (κ(X))². If X is ill-conditioned (as occurs with high-degree polynomials), squaring the condition number can cause complete loss of numerical precision.
What alternative matrix decomposition solves least-squares without squaring the condition number?
QR decomposition (X = Q R) solves R β = Qᵀ y directly, and Singular Value Decomposition (SVD, X = U Σ Vᵀ) solves β = V Σ⁺ Uᵀ y. Both work with κ(X) directly rather than (κ(X))².
What does the Gauss-Markov theorem guarantee about the ordinary least-squares estimator?
The Gauss-Markov theorem proves that if the noise terms have zero mean, are uncorrelated, and have constant variance σ² (homoscedasticity), the OLS estimator β is the Best Linear Unbiased Estimator (BLUE), having the smallest variance among all linear unbiased estimators.
How is the coefficient of determination R² interpreted?
R² = 1 - SSE / SST represents the fraction of total variance in the dependent variable explained by the regression model. R² = 1 indicates a perfect deterministic fit, while R² = 0 means the model performs no better than the sample mean.
How do orthogonal polynomials (Legendre, Chebyshev) improve polynomial regression?
Using orthogonal polynomials makes the Gram matrix Xᵀ X diagonal (or identity), completely eliminating cross-coupling between parameters and keeping condition numbers close to 1 regardless of degree.
Algorithm precautions and stability rules
- Center and scale input variables (subtract mean, divide by standard deviation) prior to forming polynomial design matrices to prevent ill-conditioning.
- Verify that the number of observations M strictly exceeds the number of regression parameters p (M > p) to ensure an overdetermined system.
- Avoid using standard normal equations for polynomial degrees d ≥ 5; transition to QR decomposition or orthogonal Legendre polynomials.
- Inspect residual plots for heteroscedasticity or systematic non-random curvature patterns indicating model misspecification.
- Ensure floating point precision remains 64-bit IEEE 754 during the evaluation of cross-product matrices Xᵀ X.
- Never extrapolate polynomial regression models far beyond the convex hull of the training data domain.
Theoretical limitations
- Ordinary least squares is sensitive to rogue outliers because squaring residuals gives disproportionate weight to extreme deviations.
- The normal equations Gram matrix condition number squares κ(X), making high-degree polynomial regression ill-conditioned.
- Standard OLS assumes errors are confined to the dependent variable y; errors in independent variables x require total least squares (orthogonal distance regression).