Let Q be the transition matrix of a Markov chain on the state space {1, 2,...,M}, such

Chapter 11, Problem 20

(choose chapter or problem)

Let Q be the transition matrix of a Markov chain on the state space {1, 2,...,M}, such that state M is an absorbing state, i.e., from state M the chain can never leave. Suppose that from any other state, it is possible to reach M (in some number of steps). (a) Which states are recurrent, and which are transient? Explain. (b) What is the limit of Qn as n ! 1? (c) For i, j 2 {1, 2,...,M 1}, find the probability that the chain is at state j at time n, given that the chain is at state i at time 0 (your answer should be in terms of Q). (d) For i, j 2 {1, 2,...,M 1}, find the expected number of times that the chain is at state j up to (and including) time n, given that the chain is at state i at time 0 (in terms of Q). (e) Let R be the (M 1)(M 1) matrix obtained from Q by deleting the last row and the last column of Q. Show that the (i, j) entry of (I R) 1 is the expected number of times that the chain is at state j before absorption, given that it starts out at state i. Hint: We have I + R + R2 + = (I R) 1, analogously to a geometric series. Also, if we partition Q as Q = 0 B@ R B 0 1 1 CA where B is a (M 1) 1 matrix and 0 is the 1 (M 1) zero matrix, then Qk = 0 B@ Rk Bk 0 1 1 CA for some (M 1) 1