Cramer-Rao Lower Bound

The Cramer-Rao lower bound (CRLB) sets a fundamental limit on how precisely you can estimate a parameter. For any unbiased estimator $\hat{\theta}$ of $\theta$: $$\text{Var}(\hat{\theta}) \geq \frac{1}{I(\theta)}$$ where $I(\theta)$ is the Fisher information. For $n$ i.i.d. observations: $\text{Var}(\hat{\theta}) \geq \frac{1}{n \cdot I(\theta)}$. Intuition: Fisher information measures how "informative" data is about $\theta$. The CRLB says: the more informative your data, the lower your estimation error can be — but there's a hard floor. No amount of cleverness in your estimator can beat this bound. An estimator that achieves equality is called efficient. Proof sketch: Apply Cauchy-Schwarz to $\text{Cov}(\hat{\theta}, \text{score})$. Since the score has mean 0 and variance $I(\theta)$, and $\text{Cov}(\hat{\theta}, \text{score}) = 1$ for unbiased estimators, we get $1 \leq \sqrt{\text{Var}(\hat{\theta}) \cdot I(\theta)}$. Concrete example: Estimating the mean $\mu$ of a normal distribution $N(\mu, \sigma^2)$ from $n$ samples. Fisher information per observation: $I(\mu) = 1/\sigma^2$. CRLB: $\text{Var}(\hat{\mu}) \geq \sigma^2/n$. The sample mean $\bar{X}$ achieves exactly this — it's efficient. When to use: Evaluating whether your estimator is any good (compare its variance to the CRLB). Proving that a particular estimator is optimal. In adaptive testing, the standard error from EAP estimation is bounded by $1/\sqrt{I(\theta)}$ — the CRLB tells you when you've extracted enough information to stop.