Theoretical Statistics STAT 210A
Popular in Course
Popular in Statistics
This 3 page Class Notes was uploaded by Floy Kub on Thursday October 22, 2015. The Class Notes belongs to STAT 210A at University of California - Berkeley taught by Staff in Fall. Since its upload, it has received 38 views. For similar materials see /class/226735/stat-210a-university-of-california-berkeley in Statistics at University of California - Berkeley.
Reviews for Theoretical Statistics
Report this Material
What is Karma?
Karma is the currency of StudySoup.
Date Created: 10/22/15
Stat210B Theoretical Statistics Lecture Date February 8 2007 Lecture 7 Lecturer Michael I Jordan Scribe Kurt Miller 1 Properties of VCClasses 11 VC preservation Let C and D be VCclasses Le classes With nite VCdimension Then so are CE C e C o CUD1CECDED o C D1CECDED o Where lt1 is ll o CXD2CECDED 12 Half spaces Let Q be a nitedimensional vector space of functions Let C g 2 0 g E Q or more formally C w 9a 2 0 g E Q Then V0 S dimQ l 13 Subgraphs De nition 1 A subgmph of f X A R is the subset X X R given by 95 t t S A collection 7 is a VCsubgmph class if the collection of subgraphs is a VC class 2 Covering Number We now begin to explore a more powerful method of de ning complexity than VCdimension 21 De nitions De nition 2 Covering Number Pollard 1984 p 25 Let Q be a probability measure on S and f be a class of functions in L1Q ie Vf E f f lt 0 For each E gt 0 de ne the L1 covering number N1E Q 7 as the smallest value of m for Which there exist functions 91 gm not necessarily in f such that minj E 9quot S E for each f in 7 For de niteness set N1E Q 7 0 if no such m exists 2 Lecture 7 Note that the set 9 that achieves this minimum is not necessarily unique De nition 3 Metric Entropy De ne H1E Qf log N1 E Q 7 as the 1 metric entropy of 7 More generally HpE Qf uses the LpQ norm Write this as lg lm flglde1plt De nition 4 Totally bounded A class is called totally bounded if VE HpE Qf lt 00 Another kind of entropy De nition 5 Entropy with bracketing Let N BE Q 7 be the smallest value of m for which there exist pairs of functions gf such that Vj 179lepr lt E and Vf E f 3jf st 9 S f 3 gym Then we de ne the entropy with bracketing as prQE Qf log prQE Qf Finally using IgHoo supxgx gx let NOOE7 be the smallest m such that there exists a set g l such that supfef minj1wym 7 ngoo lt E Then HOOET lOgNoQE f 22 Relationship of the various entropies Using the de nitions above we have that 1 H1EQf s H BEQf Va gt 0 2 HPBEQ S HooE27 VE gt 0 Can these quantities be computed for normal classes of functions Yes but you would generally look them up in a big book We ll look at how to compute one of these quantities here 23 Examples Example 6 Let f f 01 l S l ie functions from 01 to 01 with rst derivatives bounded by 1 Then H00 E f where A is a constant that we will compute ll0 A Proof Let 0 10 lt 11 lt lt am l where we kE and k 0m Let Bl 10111 and Bk ak71ak For each f E 7 de ne N m flllc f 5 lBC k21 f takes on values in Ek where k is an integer We also have 3 2E because fak1 7 fak1 S E by construction and fac 7 fak1 S E since f is bounded by 1 We now count the number of possible f obtained by this construction At 10 there are lE 1 choices for fa0 since f only takes on values of Ek in 0 1 Furthermore combining previous results gives us Ware 7 aker Wale ak mate fak71llfak71i aker S 3 3E Lecture 7 3 Therefore having chosen fak1 f can take on at most 7 distinct values at ak Therefore Nooaaf s 1 71 which gives us that 1 Hoo25f S E log7 logl1Ej 1 so our constant can be chosen as any constant that gt log 7 El A seminal paper in this eld is by Birman and Solomjak in 1967 They present other examples of metric entropy calculations including Example 7 Let f f 01 a 01 ffltmgtx2dx g 1 Then Hagar g flailm Example 8 Let f f R A 01 f isincreasing Then H BEQ S A Example 9 Let R A 01 f S 1 the class of bounded variation Then HpBE Qf 3 Ag Lemma 10 Ball covering lemma A ball BAR in Rd of radius R can be covered by 4R E d E Proof Let Cj be a packing of size E Euclidean norm This implies that balls of radius E with centers at Cj cover BAR otherwise we could add more points Cj to the packing Let Bj be the ball of radius 54 centered at cj We must have that B O Bj is empty for i y j Therefore 3 are disjoint and balls of radius E Ujsj c BdRE4i A ball of radius p has volume Cdpd where Cd is a constant that depends on the dimension d Therefore the volume of the union Uij is MCdE4d and since it is a subset of BAR 54 we have E d E d lt i Mcd lt4 Cd RT 4 With a simple manipulation of this equation we get that a MS lt4REgt E References Pollard D 1984 Convergence of Stochastic Processes Springer New York