MATHEMATICAL PROBABILITY I
MATHEMATICAL PROBABILITY I STAT 581
Popular in Course
Popular in Statistics
This 26 page Class Notes was uploaded by Miss Sabina Grimes on Monday October 19, 2015. The Class Notes belongs to STAT 581 at Rice University taught by Staff in Fall. Since its upload, it has received 37 views. For similar materials see /class/225042/stat-581-rice-university in Statistics at Rice University.
Reviews for MATHEMATICAL PROBABILITY I
Report this Material
What is Karma?
Karma is the currency of StudySoup.
Date Created: 10/19/15
Summary of Stat 581582 Text A Probability Path by Sidney Resnick Taught by Dr Peter Olofsson Chapter 1 Sets and Events l2 Basic Set Theory SET THEORY The sample space Q is the set of all possible outcomes of an experiment eg ifyou roll a die the possible outcomes are Ql23456 A set is nitein nitely countable if it has nitely many points A set is countabledenumerable if there exists a bijection ie ll mapping to the natural numbers N l23 e g odd numbers integers rational numbers Q mn where m and are integers The opposite of countable is uncountable eg ER real numbers any interval ab If you take any two points in a set and there are in nitely many points between them then the set is considered to be dense eg Q ER Note A set can be dense and be countable OR uncountable The power set on is the class of all subsets of Q denoted 29 or PQ ie ifQ has n elements PQ has 2quot elements Note if Q is in nite and countable then PQ is uncountable SET OPERATIONS o Complement AS w e Q a E A 0 Union AUBmeQaerraeBorboth o Intersection A BmeQaeA and weB SET LAWS o AssociatiVity A U BU C A U B U C o DistributiVity B H UAt UltB H A ET 0 De Morgan s UAt Af ET ET The opposite is true for all ofthe above 13 Limits 0fSets inf A A ert k D k sup anAk gAk inf Ak ert 0 An occurs eventually liminfn A U U Ak a e Q a e A for all but nitely many n quot1 n1kn no su no no limsupquot Aquot k gtpnAk UAk w e Q a e Aquot for in nitely many n quot1 n1kn 0 An occurs in nitely often 0 liminfn Aquot g limsupquot Aquot o liminfn A 5 limsup A and Vice versa If liminf limsup A then the limit exists and is A l5 Set Operations and Closure is called a M or algebra if 1 Q 6 2 A 6 3 A 6 3 ABe 3AUBe A eld is closed under complements nite unions and intersections B is called a c eld or calgebra if 1 Q 6 2 B 6 3 B 6 3 B1B2 UB1 6 11 A c eld is closed under complements countable unions and intersections Note Q is the smallest possible G eld and 22 is the largest possible c eld The sets in a c eld are called measurable and the space Q is called a measurable space 1 6 The c field Generated by a Given Class C Corollary The intersection of c elds is a c eld Let C be a collection of subsets of Q The c eld generated by C denoted g9 is a c eld satisfying a 6C D C b IfB is some other c eld containing C then B D 6C 6C is also known as the minimal c eld Chapter 2 Probability Spaces I 21 Basic De nitions and Properties Fatou s Lemma probabilities lim inf liminf limsup limsup P A s PAns PAns P A n gt 00 n gt 00 n gt 00 n gt 00 A function F 5 gt 01 satisfying 1 F is right continuous 2 F is monotone nondecreasing 3 F has limits at i 00 is called a probability distribution function I 22 More on Closure P is a nsystem if it is closed under nite intersections That is A B e P 3 A H B e P A class of subsets L of Q that satisfies 1 Q e I 2 A e I A e I 3 nimAn Am An e z UnAn e r is called a Xsystem Dynkin s Theorem a If P is a nsystem and L is a Xsystem such that P CL then 6P CL b If P is a nsystem 6P P That is the minimal 6f1eld over F equals the minimal 7 system over F Proposition 224 If a class C is both a Xsystem and a nsystem then it is a 6f1e1d A class 5 of subsets Q is a semialgebra if the following hold i Q Q e S ii 5 is a psystem iii IfA e S then there exist some finite n and disjoint sets C1Cn with eachCl e S 3 A EPIC Chapter 3 Convergence Concepts 31 Inverse Maps Let Q and Q be two spaces and let X Q H Q For a set 3 g Q the set X391BV m e Q Xa 6 B39 is called the inverse map of B39 X391B39 is also called the c eld generated by X denoted 0X Proposition 311 If B is a G eld on Q then X 1 is a c eld on Q 32 Measurable Maps Random Elements Induced Probability Measures A pair set039 eld eg Q is called a measurable space X is a random variable if X Q H X is measurable if X391 is aset in V B 6 y ie X 1 39g Measurable sets are events Constant functions are always measurable Proposition 322 Let X Q1B1H Q2B2 and Y Q2B2 H Q3B3 be two measurable maps Then Z Y o X YXa Q1B1 H Q3B3 is also measurable The Borel G eld in SW is the class of open or halfopen or closed rectangles X is a random vector cgt X1X2 are random variables Corollary 322 If X is a random vector and g SRk H ER is measurable then gX is a random variable The distribution of X aka probability measure induced by X is denoted Px P o X39l Chapter 4 Convergence Concepts 41 Basic De nitions Independence events P A J HPltA1 for all I in 12 n IEI IEI Independent classes Classes C1 are independent iffor any choice A 6 C1 il n the events A1 Aquot are independent Theorem 411 If C1Cn are independent Irsystems then 0C1039Cn are independent 42 Independent Random Variables Independent random variables Random variables are independent if their c elds are independent Corollary 422 The discrete random variables X1 X k with countable range R are k independent iff PXl xwi lk HPXX x1 for all x ER ilk 11 45 Independence Zero One Laws Borel Cantelli Lemma Borel Cantelli Lemma 1 Let An be any sequence of events such that ZPltAn lt oo ie converges then PAn io 0 Borel Cantelli Lemma II If A1 An are independent events such that Z PAn 00 ie diverges then PAn io l A property that holds with probability l is said to hold almost surely as NOTE 0 is divergent n l o IS convergent Let Xn be a sequence ofrandom variables and de ne Fquot 0XMXM nl2 The tail c eld r is r Fn Events in r are called tail events Atail event is NOT affected by changing the values of nitely many X A 6field all of whose events have probabilities equal to 0 or 1 are called almost trivial at Kolmogorov s 0 1 Law If Xn are independent random variables with tail 6field r then A e r 3 PA 0 or 1 This in turn implies that the tail 6field is at 0 Note if we assume iid then more events have the 01 property Lemma 451 If g is an at 6field and X is measurable wrt g this implies that X is constant as An event is called symmetric if its occurrence is not affected by any permutation of nitely many Xk Notes 0 Tail events are not affected by the VALUES and symmetric events are not affected by the ORDER 0 All tail events are symmetric converse is false Hewitt Savage 0 1 Law If Xn are iid then all symmetric events have probability 0 or 1 Chapter 5 Integration and Expectation 51 Preparation for Integration A simple function is a random variable of the form X Z a I A where a e SEA 6 i l2n and A1An formapartitibln of Q Properties of Simple Functions 1 IfX is simple then aX Eat6111A where at is constant 11 N IfXY are simple then X Y 2w b IAWBJ 1v 3 If XY are simple then XY 2161le IAWBJ 1v IfXY are simple then MinX Y ZMz39nwl b IAWBJ or 1v MaxX Y ZMaxw b IMBJ 1v Theorem 511 Measurability Theorem Let X be a nonnegative random variable Then there exists a sequence of simple functions Xn 3 0 S Xquot T X I 52 Expectation and Integration Expectation of X with respect to P Let X be a simple function then Ex jx dP aIIIAIdP2 11 IAdP a1PAI 11 11 11 Note EIA PA Steps to show expectation 1 Indicators Let X 1A Ex EIA PA 2 Simple Functions Let X EricleA 11 3 Nonnegative Random Variables Suppose X Z 0 Xn a sequence of simple functions and 0 s x T x 4 General Functions X X X39 lim lim If xquot T x and Ym T x then EX1s welldefined 1f EXn EYm n gt 00 m gt 00 X is integrable if Eleltoo where W X X39 Therefore X is integrable 3 E X lt 0013 X lt 00 then EX will be nite n A Convergence Theorem for H 39 0 S Xquot T X 3 EXn T EX or E hm T Xquot hm T EXn 00 7 gt 00 n gt Inequalities a Modulus EX s EQXl b Markov Su X L F xgt0 PHX gt 1 lt M ppose e 1 or any l c Chebychev Assume E X lt oo VarXlt oo PHX EXX Z 11S VarX 22 d Triangle le m s lxll le e Weak Law of Large Numbers WLLN The sample average of an iid sequence approximates the mean Let Xquot n 2 1 be iid with nite mean and variance and suppose n X lim EXn u and VarX62 is nite Then for any 8 gt 0 P H u gt g 0 n gt 00 n 53 Limits and Integrals n A Convergence Theorem for series If X j Z 0 are nonnegative random variables for n 2 1 then Xj ZEXJ ie the expectation and in nite sum can be interchanged 11 11 Fatou s Lemma expectations lim inf lim inf lim sup lim sup E X s EX S EXnS E X n gtoo n gtoo n gtoo n gtoo 1 Convergence Theorem for H quot If Xquot gt X and there eXists a dominating random variable Z 6 L1 3 Xn S Z then EXn gt EX and E Xn X gt 0 NOT I BOOK 7 General Measure and Integration Theory A function u gt SR is called a measure if i y 2 0 ii y 0 iii A1A2 6 disjoint 2 yUA1 EMA PPN A TA 2 yAn gt uA If Q UCK Where Cquot C and uCn lt 00 V n then u is called c nite on the class C quot1 THM If in u on a n system P and m u are c nite on P then H1 u on 6P THM A c nite measure on a semialgebra S has a unique extension to 6P I 55 Densities Let X be a random variable If there eXists a function f 3 PX B L f xdx for all Borel sets B then f is called the density ofX THM If f exists then Egx Igxfxdx Note dFX fXdX LEMMA IfJAfd z39iAgd V A616 gt fg W 56 The Riemann vs Lebesgue Integral Lebesgue Measure 7 o Msingleton 0 o Minterval length of interval Steps to show integration Indicators Let f IA J QfdJ JQIAd 2 61 2 A N Simple Functions Let f Z alIAI 11 LA Nonnegative Random Variables Suppose f 2 0 f simple and 0 S f T f De ne from 1im Indy 4 General Functions f fl f and de ne Ifdy Ifdy Jf39dy 4 DNE if 00 oo Types of Integration 0 Riemann 7 approximate area under a curve Via rectangles o Lebesgue 7 approximates Via a measure This is the limit of simple functions Note Riemann diVides the domain and Lebesgue diVides the range of a function THM If Riemann exists then Lebesgue integral exists The converse is not necessarily true A property which holds everywhere except on a set A with MA0 is said to hold galmost eve where ptae THM g is Riemann Integrable cgt g is continuous ae Fatou s Lemma integrals 139 39f 139 39f 139 139 I mlnw jd snn fKWS lmsupl d sllmsup dy oo n gtoo n gtoo Dominated Convergence Theorem for integrals Let f f1 f2 be a sequence of functions 3 fquot gt f Ifthere exists afunction g 3 S g V n and Igdy lt oo then Ifndya jfdy Monotone Convergence Theorem for integrals Let f f1 f2 be a sequence of nonnegative functions 3 fquot T f n gt 00 Then Ifndy gt Ifdy Identity limsupnfn 1iminfn f 57 Product Spaces 21 X 92 w1a2a1 e Qla2 6 S22 is called the product space If A e A and B e z then the set A X B wla2a1 e Aa2 e B is called a measurable rectangle The c eld generated by the rectangles is called the product c eld denoted A 9 z For C e A 2 we de ne the sections as CX yxye C and Cy xxyeC 0 Sections of functions and sets are measurable De ne measures 723 2 on 21 X 22 A 9 z mo Jalultcxgtd ltxgt me Maw Both measures equal uAuB when C A X B o If uv are probability measures then 72391 72392 are probability measures THM If uv are c nite then there eXists a unique measure on A 9 z such that 7239A X B uA 0B NOTE I u X I I 59 Fubini s Theorem Fubini s Theorem If I g d7r exists then 1g dz dw ugt in g I 1 g Special Cases when Theorem Holds 1 g 2 0 Tonelli s Theorem 2 d7r lt oo ie g integrable often called Fubini s Theorem Chapter 6 Convergence Concepts Suppose QFP is a probability space then X QF gt SEE is a random variable The distribution of X is denotedP o X 391 PX If u is a measure on ELF then u o f 1 is the induced measure on AG AlmostSure o PXn gtX1 2 Xn gtXas o ZPan Xlgtcltoo v sgt0 3 Xn gtXas quot1 o PPN X gt X as amp Y gt Y 3 X Yn gt X Y this holds for Lp and therefore in probability and in distribution In Probability o PXn Xlgtgampgt0 V gt0 3 Xn PgtX o THM 631b Convergence in probability can be characterized by convergence as of subsequences P X gtX 3 Each th has a further subsequence iX w 3 Xnk COR 631 X gt X as amp g continuous 3 gXn gt gX as U gtX as Xquot PgtX amp g continuous 3 gXn PgtgX o COR 632 Version of DCT X gt X as Xn is Y amp EYlt so 3EXn gtEX o PPN Xquot PgtX amp Xquot PgtY 3 PX Y 1 therefore cannot have different limits in probability Same holds for as and LP InL 2 o Letpgt0 E Xn X P gt0XnigtX o PPN Xquot X g continuous amp bounded 3 gXn igtgX o PPN If X LgtX for some p and rltp then X X In Distribution 0 Let X X1 X2 with distribution function F F1 F2 where FXPXS X F1XPX1SX etc F x gt Fx Vx F continuous 3 X dgtX 39 quot 39 39 between thes of convergence 0 Xn gtXas3Xn PgtX o XKLgtX3Xn PgtX o Xn PgtXXn gtX Inegualities P Chebyshev s PG Y Z 5S E ii 1 for any pgt0 amp Sgt0 s Holder s prgt1qgt1amp il1then EHXY HS EhX I171 Y Iql P q Special case of Holder s is when pq2 ie Cauchy Schwartz 1 1 1 Minkowski s ForpZgt1 E X Y IPYF S E I X IPYF E I Y lplF For p1 this is the Triangle Inequality Chapter 7 Law of Large Numbers amp Sums of Independent Variables Weak Law of Large Numbers WLLN THM 721 where bnn amp Sn 2X 1 11 0 Let X1 X2 independent amp an ZEXJ X S n then if 1 1 PQXJ gt 0amp0 1 quot w 2 FZEk X s n gt0 k1 isquot aquot gtP 0 n LetXl X2 Lid amp an nElXj X s nJthen if 1 nPQXj gt nampgt0 1 2 7EXj2Xj s 4amp0 n Sn an P gt0 n LEMMA Pw amp in gt a 2 i Pm n n n Kolmogorov s Convergence Criterion THM 733 0 Let X1 X2 independent If ZVarXlt 00 then XXX EXj D converges as 11 11 0 The convergence of XX is determined by 11 11 Kronecker s Lemma 741 X 0 Let Xk and bu be sequences of real numbers b T 00 If converges then 11 J 1 quot H 72X gt0 bn 1 o COR 741 Suppose EXlt oo Vn bnTooamp EMlt00 3 11 1 n S bES gt 0 as Kolmogorov s Strong Law of Large Numbers SLLN THM 751 0 Let X1 X2 Lid with nite mean m ie EHX1lt 00 Then amp gt u 215 n Chapter 8 Convergence in Distribution CDF of a random variable is de ned as FX PX S X F is a possible cdfiff 1 0 s Fx 3 1 2 F is nondecreasing 3 F is rightcontinuous If F 00 0 and Foo l we call F proper or nondefective It is assumed that all cdf s are proper Convergence in Distribution X dgtX if F x gt F x Vx where F is continuous This is also called weak convergence and can be denoted Fquot x WgtF Lemma 811 If two cdf s agree on a dense set then they agree everywhere A dense set has no gaps LEMMA A cdf has at most countably many discontinuities PPN If Fquot WgtF then F is unique Skorohod s Theorem Suppose Xquot gt X Then there exists a random variable 1 d X 3 XfX on the probability space 01 BorelLebesgue 3 X X Xi Xn n 12 and Xi gt X as Continuous Mapping Theorem If X gt X and g continuous then gXn dgt gX Theorem 841 X gt X cgt EhXn gt EhX for all bounded and continuous functions h Chapter 9 Characteristic Functions and the Central Limit Theorem Moment Generating Function Mt Ee x Characteristic Function t Ee x EcostX iEsintX t E ER 0 Maps from the Real Numbers to the Complex Plane 0 Chf always exists whereas mgf does not Properties of CHF 0 1 0 XY independent 3 xw quY 60 iEX o In Generality lnEXn Theorem 951 d3 uniquely determines the distribution of X Theorem 952 Xn dgtX 3 xwt gt Xt Vt 2 it The chf of a standard normal distribution is bx I 2 e 4 Theorem 971 Central Limit Theorem iid case Let X1X2 be iidwithmean u and 0392 lt00 Let Sn ixk and let N N01 k1 S en n u dN ad PROOF IS SHOWN ON NEXT PAGE Th Proof of CLT giid case Suppose u0 and 0392 1 Let Q be the chf oka and dquot be the chf of n Then on r 139 J Ee ZXkW J1 Ee Want to show that 671 no la By Taylor s Formula X 2q3lk0 assume after k2 all functions are negligible ie H converge to 0 as n goes to in nity With X j o l0 z0 o 01 39 l0 I 0 since we supposed p 0 o qgt20 1 Delta Method S Sn ny i Let X1X2 be iid ByCLTwith Xi u dgtN Hence n 04 for large n i N lez This method is used to make inferences about u PPNIfg39 0 then 2 g b Nglag39l2 Xklgttsnampgt0 v rgt0 The X k are said to satisfy the Lindeberg condition if EX n m Theorem 981 Lindeberg Feller CLT Let Sn 2039 Var 2 Xk J VarSn Then the Lindeberg condition implies 161 k1 S 4 H N01 SH 391 EMX 26 Liapunov condition For some 5 gt 0 161 gt 0 PPN Liapunov 3 Lindeberg 1 Important relationship N log 161 Chapter 10 Martingales 101 Prelude to Conditional Expectation The RadonNikodym Theorem DFN If MB0 3 vB0 then v is absolutely continuous with respect to u ie v ltlt u Radon Nikodm Theorem THM 1012 If v ltlt u amp u is 6f1nite then El a measurable function fQ gtSR3VBJdeu VBeBorel d v o RN Derlvative f 7 61 dugout 0 Chain Rule va ltlt u amp ultltp then vltltp amp dp d dp 102 De nition of Conditional Expectation Conditional Probability 0 Let QBP be a probability space amp let g e B be a subofield of B Take A e B El a unique random variable 2 3 1 z is measurable with respect to g 2 PAnG sz dPVG e g PA n B Discrete I gtPAlB PltBgt 0 In general Continuous gt fx y M f y Conditional Expectation 0 Let zEXlg be the conditional expectation of X with respect to g Then it must ful ll the following 1 z is measurable with respect to g 2 6X dP sz dP v Ge g 0 Generally o If X is gmeasurable then EXlgX o If X is independent of g then EXlgEX fxY o EXlY xfxY dx x7 dx I l I N 103 Properties of Conditional Expectation Linearity If X Y 6 L1 and 1 6 SR we have EaX YXGria EX G EY G If X e G X 6 L1 then EXGa39S39X EXl 9 1300 Monotonicity If X Z 0 and X 6 L1 then EX GZ 0 as Modulus Inequality If X 6 L1 then EX GS EQXHG Monotone Convergence Theorem If X 6 L1 0 S Xquot T X then EXn Monotone Convergence implies the Fatou Lemma lim inf lim inf E X G s EX n n gtoo n gt 00 b ElimsupX GJZ limsupEXn n gt 00 GT EXG 8095 G n G n n gtoo 00 Fatou implies Dominated Convergence If Xquot 6 L1 lim 03 lim E Xquot G EX n gt oo n gt 00 Product Rule Let XY be random variables satisfying X YX 6 L1 If Y e G then as EXYG YEXG 10 Smoothing If G1 C G2 C B then for X 6 L1 a EEX GZG1 EX G b EEX G1GZ EX Gl 0 NOTE The smallest c eld always wins Xquot SZEL1 and Xquot gtXwthen G n 0 104 Martingales Martingales o DFN Sn is called a martingale with respect to Bu if a Sn is measurable with respect to En V n b ESn1 l 13quot Sn 0 If b is S then it is a supermartingale If Z then it is a submam39ngale o Submartingales tend to increase 0 Supermartingales tend to decrease IfSn is amartingale then ESnk l Bu Sn Vkl2 If Sn is a martingale then ESn constant 105 Martingales I 106 Connections between Martingales and Submartingales Any submartingale Xn Bquot n 2 0 can be written in a unique way as the sum of a martingale MWBK n 2 0 and an increasing process An 7 Z 0 ie Xquot Mn An I 107 Stopping Times A mapping 1 Q I gt 012oo is a stopping time if u n e Bquot Vn e N 012 To understand this concept consider a sequence of gambles Then I is the rule for when to stop and Bquot is the information accumulated up to time 71 You decide whether or not to stop after the 71m gamble based on information available up to and including the 71m gamble Optional Stopping Theorem If Sn is a martingale and 1 lt 2 lt 3 lt are wellbehaved stopping times then the sequence SDl SUI SUE is a martingale with respect to BUIBUZBUK Consequences ElSU J E180 J ElSU J o Ifwe let U1 0 and U2 uthen ES0 EISU U is a wellbehaved if a PU lt 00 1 b E SUIIlt 00 c EHSU 1 gt nI gt0 I 109 Examples Gambler s Ruin Basic concept Imagine you initially have K dollars and you place 1 bet You continue to bet 1 after each winloss Playing this game called Gambler s ruin the ultimate question is will you hit 0 or N rst Formally written suppose Zn are iid Bernoulli random variables ie winlose 1 satisfying PIZK i1 and let X0 jo ie initial balance Xquot ZZZI j0 n 21 ie balance after 11 71 bet This is a simple random walk starting at jo Assume 0 S jo S N Will the random walk hit 0 or N rst 0 with probability p i X N with probability 1 j o Branching Process Basic concept Consider a population of individuals who reproduce independently Let X be a random variable with offspring distribution pk PX k k 012 and let m EX Start with one individual easiest to think of a singlecelled organism who has children according to the offspring distribution The children then have children of their own independently based on the same distribution Let Zn be the number of individuals in the rim generation 20 E l The process Zn is called a simple branching process or GaltonWatson process znrl NOTE Zn 2X k where X k are the number of children of the km individual in generation k1 nl Xk are iid 9 EZi m Ele gt 0 m lt l extinction Hence EZn E 1 m 1 3 stagnation gt 00 m gt growth Extinction PE PUZn 01imn PZn 0 Let pk PX k k 012 The function s Z skpk is called the probability 160 generating function pgf of X Let q PE then you can solve for q via the equation 3 s m lt 13 PE l subcritical m 13 PE13 critical in gt 1 3 PE 0 sup ercritical Useful identity Let Y Z 0 with mean u and EY2 lt oo finite 2quotd moment Then 2 FY 02 gt EYz Z actual population size 7 Then WK 1s a mart1ngale w1th EZn mquot expected population size respect to the c eld Bquot 0ZOZn o PW0 is either 1 or q o If variance is nite then PW0q THM Let Wquot COR There exists an integrable random variable W such that W gt W as PPN In a martingale if 2quotd moments are bounded then Sn gt S 1010 Martingale and Submartingale Convergence Doob s Theorem If Sn is a submartingale such that sup Ele Jlt oo then there exists an integrable random variable S such that Sn gt S as PPN Nonnegative martingales always converge