Maximum Likelihood Estimation (MLE)

Overview

1. Concept

Distinction:

• Probability: Given fixed parameters $\theta$, what is the probability of observing data $X$? ($P(X|\theta )$)
• Likelihood: Given observed data $X$, how likely is it that the parameters are $\theta$? ($L(\theta |X)$)

MLE seeks to find $\hat{\theta }$ that maximizes $L(\theta |X)$.

2. Procedure

1. Define the Likelihood Function:

   $$L(\theta )=\prod _{i=1}^{n}f(x_i;\theta )$$

   (Assuming independence).

2. Log-Likelihood:
   It is computationally easier to maximize the sum of logs than the product of probabilities.

   $$\ell (\theta )=\sum _{i=1}^n\ln f(x_i;\theta )$$

3. Differentiate: Take the derivative with respect to $\theta$ and set to zero.

   $$\frac{\partial \ell }{\partial \theta }=0$$

4. Solve: Find $\hat{\theta }$.

3. Properties of MLE

• Consistent: Converges to the true parameter value as $n\to \mathrm{\infty }$.
• Efficient: Achieves the lowest possible variance (Cramér-Rao lower bound) asymptotically.
• Invariant: Functional invariance (MLE of $g(\theta )$ is $g(\hat{\theta })$).

4. Related Concepts

• Binary Logistic Regression - Uses MLE.
• Simple Linear Regression - OLS is equivalent to MLE under normal errors.