Markov Chain Analysis and Stationary Distribution

Open Live Script

This example shows how to derive the symbolic stationary distribution of a trivialMarkov chainby computing its eigen decomposition.

Thestationary distributionrepresents the limiting, time-independent, distribution of the states for a Markov process as the number of steps or transitions increase.

Define (positive) transition probabilities between statesAthroughFas shown in the above image.

symsabcdefcCA建行positive;

Add further assumptions bounding the transition probabilities. This will be helpful in selecting desirable stationary distributions later.

assumeAlso([a, b, c, e, f, cCA, cCB] < 1 & d == 1);

Define the transition matrix. StatesAthroughFare mapped to the columns and rows1through6. Note the values in each row sum up to one.

P = sym(zeros(6,6)); P(1,1:2) = [a 1-a]; P(2,1:2) = [1-b b]; P(3,1:4) = [cCA cCB c (1-cCA-cCB-c)]; P(4,4) = d; P(5,5:6) = [e 1-e]; P(6,5:6) = [1-f f]; P

P =
                 
                   $(\begin{array}{c} a & 1 - a & 0 & 0 & 0 & 0 \\ 1 - b & b & 0 & 0 & 0 & 0 \\ cCA & 建行 & c & 1 - cCA - 建行 - c & 0 & 0 \\ 0 & 0 & 0 & d & 0 & 0 \\ 0 & 0 & 0 & 0 & e & 1 - e \\ 0 & 0 & 0 & 0 & 1 - f & f \end{array})$

计算所有可能的分析固定distributions of the states of the Markov chain. This is the problem of extractingeigenvectorswith corresponding eigenvalues that can be equal to 1 for some value of the transition probabilities.

[V,D] = eig(P');

分析特征向量

V =
                 
                   $\begin{array}{l} (\begin{array}{c} \frac{b - 1}{a - 1} & 0 & - \frac{(c - d) (建行 - b cCA - b 建行 + c cCA)}{σ_{1}} & 0 & - 1 & 0 \\ 1 & 0 & - \frac{(c - d) (cCA - a cCA - a 建行 + c 建行)}{σ_{1}} & 0 & 1 & 0 \\ 0 & 0 & - \frac{c - d}{c + cCA + 建行 - 1} & 0 & 0 & 0 \\ 0 & 0 & 1 & 1 & 0 & 0 \\ 0 & \frac{f - 1}{e - 1} & 0 & 0 & 0 & - 1 \\ 0 & 1 & 0 & 0 & 0 & 1 \end{array}) \\ where \\ σ_{1} = (c + cCA + 建行 - 1) (a + b - a c - b c + c^{2} - 1) \end{array}$

Analytical eigenvalues

diag(D)

ans =
                 
                   $(\begin{array}{c} 1 \\ 1 \\ c \\ d \\ a + b - 1 \\ e + f - 1 \end{array})$

Find eigenvalues that are exactly equal to 1. If there is any ambiguity in determining this condition for any eigenvalue, stop with an error - this way we are sure that below list of indices is reliable when this step is successful.

ix = find(isAlways(diag(D) == 1,'Unknown','error')); diag(D(ix,ix))

ans =
                 
                   $(\begin{array}{c} 1 \\ 1 \\ d \end{array})$

Extract the analytical stationary distributions. The eigenvectors are normalized with the 1-norm orsum(abs(X))prior to display.

fork = ix' V(:,k) = simplify(V(:,k)/norm(V(:,k)),1);endProbability = V(:,ix)

Probability =
                 
                   $\begin{array}{l} (\begin{array}{c} \frac{b - 1}{(a - 1) σ_{2}} & 0 & 0 \\ \frac{1}{σ_{2}} & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & \frac{f - 1}{σ_{1} (e - 1)} & 0 \\ 0 & \frac{1}{σ_{1}} & 0 \end{array}) \\ where \\ σ_{1} = \sqrt{\frac{{(f - 1)}^{2}}{{(e - 1)}^{2}} + 1} \\ σ_{2} = \sqrt{\frac{{(b - 1)}^{2}}{{(a - 1)}^{2}} + 1} \end{array}$

The probability of the steady state beingAorBin the first eigenvector case is a function of the transition probabilitiesaandb. Visualize this dependency.

fsurf(Probability(1), [0 1 0 1]); xlabelaylabelbtitle('Probability of A');

Figure contains an axes object. The axes object with title Probability of A contains an object of type functionsurface.

figure(2); fsurf(Probability(2), [0 1 0 1]); xlabelaylabelbtitle('Probability of B');

Figure contains an axes object. The axes object with title Probability of B contains an object of type functionsurface.

The stationary distributions confirm the following (Recall statesAthroughFcorrespond to row indices1through6):

StateCis never reached and is therefore transient i.e. the third row is entirely zero.
The rest of the states form three groups, {A,B}, {D} and {E,F} that do not communicate with each other and are recurrent.