aka Shannon–McMillan–Breiman theorem

Uses Convergence of random variables. Video

Asymptotic equipartition property for i.i.d. processes

, using the Law of large numbers (the Weak law of large numbers in particular). See also Hoeffding's inequality for a non asymptotic result

We can also define a Typical set

This can also be understood using the proof I did of the entropy for non-uniform distributions, and using the law of large numbers

For more general stochastic processes

We define Entropy rate

The asymptotic equipartition property (AEP) was first stated by Shan- non in his original 1948 paper [472], where he proved the result for i.i.d. processes and stated the result for stationary ergodic processes. McMillan [384] and Breiman [74] proved the AEP for ergodic finite alphabet sources. The result is now referred to as the AEP or the Shan- non–McMillan–Breiman theorem. Chung [101] extended the theorem to the case of countable alphabets and Moy [392], Perez [417], and Kieffer [312] proved the L 1 convergence when {X i } is continuous valued and ergodic. Barron [34] and Orey [402] proved almost sure convergence for real-valued ergodic processes; a simple sandwich argument (Algoet and Cover [20]) will be used in Section 16.8 to prove the general AEP.