CS249 - Digital Communications and Signal Processing

An information source has entropy.

Minimal Length Coding

The average information is the entropy of an information source. The entropy is given by $\sum P(X_i)I(X_i)$ where $I$ is the Shannon information.

Shannon's first theorem: $L_s \geq H(X)$. $L_s$ = average length.