Maximum Likelihood Estimation (MLE) theory

Let X_1, ..., X_n be independent, identically distributed random variables with probability density or mass function f(x; θ), where θ ∈ Θ is the unknown parameter vector. The likelihood function is the joint density viewed as a function of the parameter: L(θ; x) = ∏_{i=1}^n f(x_i; θ). The log-likelihood function is ℓ(θ; x) = ln L(θ; x) = ∑_{i=1}^n ln f(x_i; θ). The Maximum Likelihood Estimator θ̂_MLE is the parameter value that maximizes the likelihood over the parameter space: θ̂_MLE = argmax_{θ ∈ Θ} ℓ(θ; x).

The Score function S(θ) is the gradient (vector of first partial derivatives) of the log-likelihood function with respect to the parameter: S(θ) = ∇_θ ℓ(θ; x) = ∑_{i=1}^n ∇_θ ln f(x_i; θ). Under standard Cramér regularity conditions allowing differentiation under the integral sign: E[S(θ)] = ∫ [∇_θ f(x; θ) / f(x; θ)] f(x; θ) dx = ∫ ∇_θ f(x; θ) dx = ∇_θ ∫ f(x; θ) dx = ∇_θ (1) = 0. Thus, the expected value of the score function is identically zero at the true parameter value.

Fisher Information I_n(θ) quantifies the amount of information that an observable random sample carries about an unknown parameter θ. It is defined as the variance of the score function: I_n(θ) = Var(S(θ)) = E[S(θ) S(θ)^T]. Under regularity conditions where f(x; θ) is twice continuously differentiable, the Information Identity states that Fisher information equals the expected negative Hessian of the log-likelihood: I_n(θ) = - E[∇_θ^2 ℓ(θ; x)]. For i.i.d. observations, Fisher information is additive: I_n(θ) = n I_1(θ).

Under standard regularity conditions, as sample size n → ∞: (1) Consistency: θ̂_n converges in probability to the true parameter θ_0 (θ̂_n →^p θ_0); (2) Asymptotic Normality: √n (θ̂_n - θ_0) converges in distribution to N(0, I_1(θ_0)^(-1)); (3) Asymptotic Efficiency: θ̂ achieves the Cramér-Rao Lower Bound asymptotically, meaning no consistent estimator has smaller asymptotic variance; (4) Invariance: if g(θ) is a one-to-one function, the MLE of g(θ) is g(θ̂_MLE).

The 'Holy Trinity' of likelihood-based hypothesis testing comprises: (1) Likelihood Ratio Test (LRT): compares maximum likelihoods under restricted and unrestricted models via λ = 2 [ℓ(θ̂) - ℓ(θ_0)] ~ χ²(r); (2) Wald Test: evaluates the distance of the unrestricted MLE from the null hypothesis normalized by curvature: W = (θ̂ - θ_0)^T I_n(θ̂) (θ̂ - θ_0) ~ χ²(r); (3) Score (Lagrange Multiplier) Test: evaluates how close the score gradient is to zero at the null: LM = S(θ_0)^T I_n(θ_0)^(-1) S(θ_0) ~ χ²(r). All three are asymptotically equivalent under H_0.

When the likelihood equations S(θ) = 0 have no closed-form analytical solution (e.g., in logistic regression or Weibull models), iterative numerical optimization is used. Expanding S(θ) in a first-order Taylor series around current iterate θ^(t): 0 ≈ S(θ^(t)) + S'(θ^(t)) (θ^(t+1) - θ^(t)). Solving for θ^(t+1) yields the Newton-Raphson update: θ^(t+1) = θ^(t) - [∇^2 ℓ(θ^(t))]^(-1) ∇ ℓ(θ^(t)). In Fisher scoring, the observed Hessian is replaced by its negative expected value -I_n(θ^(t)), enhancing stability.

Standard asymptotic optimality of MLE breaks down when regularity conditions are violated: (1) Parameter-dependent support: For U[0, θ], the score equation is non-differentiable; the MLE is the sample maximum X_{(n)}, which converges at rate n rather than √n and has a non-Gaussian Weibull limit; (2) Boundary parameters: If true θ lies on the boundary of parameter space Θ, the score does not vanish and the asymptotic distribution is a mixture; (3) Unbounded likelihood: In Gaussian mixture models with unequal variances, the likelihood can diverge to infinity around single data points.