Chi-Squared Goodness of Fit theory

In 1900, Karl Pearson revolutionized mathematical statistics by introducing the goodness-of-fit test. Prior to Pearson, deviations between observed data and theoretical distributions were evaluated heuristically without formal probability metrics. Pearson considered a multinomial experiment with n observations categorized into K mutually exclusive classes with observed frequencies O_1, ..., O_K and hypothesized probabilities p_1, ..., p_K. He defined the test statistic χ² = ∑_{i=1}^K (O_i - E_i)^2 / E_i where E_i = n p_i, and proved that as n → ∞, this quadratic form converges in distribution to a Chi-Squared variable with K - 1 degrees of freedom.

The K observed frequencies are constrained by a single linear conservation condition: the total count must equal the sample size: ∑_{i=1}^K O_i = n. Because the expected counts also satisfy ∑_{i=1}^K E_i = n, the deviations sum to zero: ∑ (O_i - E_i) = 0. This single linear constraint removes one degree of freedom from the K-dimensional multinomial vector. Consequently, the covariance matrix of the cell deviations has rank K - 1, and the asymptotic distribution is χ²(K - 1).

If m unknown parameters of the target distribution must be estimated from the sample data using efficient methods (such as maximum likelihood on the binned counts), each independently estimated parameter imposes an additional linear constraint on the multinomial manifold. As proven by R. A. Fisher in 1922 (correcting Pearson's original assertion), the asymptotic degrees of freedom are reduced to df = K - 1 - m. For example, testing for normality with estimated mean and variance reduces df to K - 1 - 2 = K - 3.

William G. Cochran (1952, 1954) established empirical guidelines for when the continuous Chi-Squared distribution accurately approximates the discrete multinomial deviance: (1) For goodness-of-fit tests, no expected frequency E_i should be less than 1.0 (E_i ≥ 1); and (2) At most 20% of the bins should have expected frequencies less than 5.0 (E_i < 5). If these conditions fail, neighboring bins should be combined (pooled) to ensure sufficient expected counts, or exact multinomial tests should be used.

The Pearson test statistic can be expressed as the sum of squared standardized residuals: χ² = ∑_{i=1}^K r_i^2, where r_i = (O_i - E_i) / √E_i. Each r_i measures the deviation in units of standard errors. Under H_0, each r_i is asymptotically standard normal N(0, 1) (subject to the linear constraint ∑ √E_i r_i = 0). Examining individual cell residuals r_i allows practitioners to diagnose exactly which categories drive statistical significance.

The G-test uses the log-likelihood ratio statistic G = 2 ∑_{i=1}^K O_i ln(O_i / E_i). By expanding the natural logarithm in a second-order Taylor series ln(1 + x) ≈ x - x^2/2 where x = (O_i - E_i)/E_i, the G-statistic simplifies to G ≈ ∑ (O_i - E_i)^2 / E_i = χ². Thus, Pearson's test and the G-test are asymptotically equivalent under H_0, although Pearson's test often exhibits slightly superior accuracy in small-to-moderate sample sizes.

To test whether continuous data follows a distribution F(x), the continuous domain is partitioned into K intervals [a_{j-1}, a_j). The expected probabilities are p_j = F(a_j) - F(a_{j-1}). To maximize statistical power and ensure equal expected counts, equiprobable binning (where each bin has expected probability p_j = 1/K) is standard practice. The Kolmogorov-Smirnov test provides an alternative that avoids binning entirely.