Fisher Information

Fisher information quantifies how much information an observation carries about an unknown parameter $\theta$. It is the variance of the score function, or equivalently the expected curvature of the log-likelihood: $$I(\theta) = E\left[\left(\frac{\partial}{\partial\theta} \ln f(X;\theta)\right)^2\right] = -E\left[\frac{\partial^2}{\partial\theta^2} \ln f(X;\theta)\right]$$ Intuition: Imagine the log-likelihood function as a landscape. If it has a sharp peak at the true $\theta$, a single observation pins down $\theta$ precisely — that's high Fisher information. If the peak is broad and flat, the data is ambiguous — low Fisher information. Fisher information measures the "sharpness" of that peak. Concrete example: Flip a coin with unknown bias $p$. The log-likelihood for one flip is $x \ln p + (1-x)\ln(1-p)$. The second derivative is $-x/p^2 - (1-x)/(1-p)^2$, so: $$I(p) = \frac{1}{p(1-p)}$$ At $p = 0.5$: $I = 4$. At $p = 0.01$: $I = 101$. Extreme coins are easier to detect because most flips are informative. For $n$ independent observations: $I_n(\theta) = n \cdot I(\theta)$ — information adds linearly. When to use: Deriving the Cramer-Rao lower bound on estimator variance. Designing optimal experiments (maximize information). In adaptive testing, selecting the next question that maximizes Fisher information at the current ability estimate. Central to Item Response Theory. Alternative approach: Fisher information is itself a wide tool in statistical inference. It connects to maximum likelihood, sufficient statistics, and optimal experimental design.