See Markov chain

Class structure

Communicating classes. Set of states that can communicate with each other (which constitutes an Equivalence relation).

Periodicity

Period: greatest common divisor of return periods.

E.g.: random walk on Z, has period 2.

For a periodic MC (with period > 1), equilibrium state depends on time index.

Recurrence

Reccurent vs transient states.

• If the probability of return or recurrence is 1, then the process or state is recurrent.
• If the expected recurrence time is finite then this is called positive-recurrent;
• if the expected recurrence time is infinite then this is called null-recurrent.

Recurrence. Check if sum of probabilities to return at times t equals infinity.

Convergence and ergodic theorems

Ergodic theorem

Reversibility

Global and detailed balance. Detailed balance means reversibility!

{{Markov_chain_stationarity_reversibility.pdf}

A transition matrix where the whole state space is a communicating class

Stopping time, strong Markov property.

Let $C$ be a communicating class. Then either all states in $C$ are transient or all are recurrent.

As steps size decreases, while changing other parameters in certain ways, we can get a Continuous-time Markov chain