### Create a StudySoup account

#### Be part of our community, it's free to join!

Already have a StudySoup account? Login here

# DESIGN ANLYS EXPMTS STAT 502

UW

GPA 3.66

### View Full Document

## 9

## 0

## Popular in Course

## Popular in Statistics

This 51 page Class Notes was uploaded by Providenci Mosciski Sr. on Wednesday September 9, 2015. The Class Notes belongs to STAT 502 at University of Washington taught by Peter Hoff in Fall. Since its upload, it has received 9 views. For similar materials see /class/192504/stat-502-university-of-washington in Statistics at University of Washington.

## Reviews for DESIGN ANLYS EXPMTS

### What is Karma?

#### Karma is the currency of StudySoup.

#### You can buy or earn more Karma at anytime and redeem it for class notes, study guides, flashcards, and more!

Date Created: 09/09/15

Statistics 502 Lecture Notes Peter D Hoff November 247 2008 Contents 1 Principles of experimental design 11 Induction 12 Model of a process or system 13 Experiments and observational studies 14 Steps in designing an experiment 2 Test statistics and randomization distributions 21 Summaries of sample populations 22 Hypothesis testing via randomization 23 Essential nature of a hypothesis test 24 Sensitivity to the alternative hypothesis 25 Basic decision theory 3 Tests based on population models 31 Relating samples to populations 32 The normal distribution 33 Introduction to the t test 34 Two sample tests 35 Checking assumptions 351 Checking normality 352 Unequal variances 4 Con dence intervals and power 41 Con dence intervals via hypothesis tests 42 Power and Sample Size Determination 421 The non central t distribution 422 Computing the Power of a test CONTENTS ii 5 51 Introduction to ANOVA 60 51 A model for treatment variation 62 511 Model Fitting 63 512 Testing hypothesis with MSE and MST 66 52 Partitioning sums of squares 70 521 The ANOVA table 72 522 Understanding Degrees of Freedom 73 523 More sums of squares geometry 76 53 Unbalanced Designs 78 531 Sums of squares and degrees of freedom 79 532 ANOVA table for unbalanced data 81 54 Normal sampling theory for ANOVA 83 541 Sampling distribution of the F statistic 85 542 Comparing group means 88 543 Power calculations for the F test 90 55 Model diagnostics 92 551 Detecting violations with residuals 93 552 Checking normality assumptions 94 553 Checking variance assumptions 96 554 Variance stabilizing transformations 100 56 Treatment Comparisons 106 561 Contrasts 107 562 Orthogonal Contrasts 109 563 Multiple Comparisons 112 Factorial Designs 115 61 Data analysis 116 62 Additive effects model 122 63 Evaluating additivity 125 64 Inference for additive treatment effects 129 65 Randomized complete block designs 139 66 Unbalanced designs 145 67 Non orthogonal sums of squares 152 68 Analysis of covariance 154 Nested Designs 159 71 Mixed effects approach 166 List of Figures H H 21 22 23 24 25 26 31 32 33 34 35 36 41 42 43 44 51 52 Model of a variable process 2 Wheat yield distributions 12 Approximate randomization distribution for the wheat example 16 Histograms and empirical CDFs of the rst two hypothetical samples 20 Randomization distributions for the t and KS statistics for the rst example Histograms and empirical CDFs of the second two hypotheti cal samples 22 Randomization distributions for the t and KS statistics for the second example 22 The population model 27 X2 distributions 33 t distributions 34 The t distribution under Hg for the wheat example 39 Randomization and t distributions for the t statistic under H0 40 Normal scores plots 44 A tlo distribution and two non central thy distributions 52 Critical regions and the non central t distribution 55 y and power versus sample size7 and the normal approximation to the power 57 Null and alternative distributions for another wheat example7 and power versus sample size 59 Response time data 61 Randomization distribution of the F statistic 70 iii LIST OF FIGURES 53 54 55 56 57 58 59 510 511 512 513 514 515 68 69 610 611 612 613 614 615 616 617 iv Coagulation data 83 F distributions 87 Normal theory and randomization distributions ofthe F statistic 88 Power as a function of n for m 47 Oz 005 and 7202 1 92 Power as a function of n for m 47 Oz 005 and 7202 2 92 Normal scores plots of normal samples7 with n E 2050100 95 Crab data 97 Crab residuals 98 Fitted values versus residuals 99 Data and log data 101 Diagnostics after the log transformation 102 Mean variance relationship of the transformed data 107 Yield density data 110 Marginal Plots 117 Conditional Plots 118 Cell plots 119 Mean variance relationship 119 Mean variance relationship for transformed data 120 Plots of transformed poison data 121 Comparison between types I and H7 without respect to deliv ery 130 Comparison between types I and H7 with delivery in color 131 Marginal plots of the data 135 Three datasets exhibiting non additive effects 138 Experimental material in need of blocking 140 Results of the experiment 141 Marginal plots7 and residuals without controlling for row 142 Marginal plots for pain data 149 Interaction plots for pain data 150 Oxygen uptake data 155 ANOVA and ANCOVA ts to the oxygen uptake data 156 Potato data 161 Diagnostic plots for potato ANOVA 162 Potato data 163 Potato data Chapter 1 Principles of experimental design 11 Induction Much of our scienti c knowledge about processes and systems is based on induction reasoning from the speci c to the general Example survey Do you favor increasing the gas tax for public trans portation 0 Speci c cases 200 people called for a telephone survey 0 lnferential goal get information on the opinion of the entire city Example Women7s Health Initiative Does hormone replacement im prove health status in post menopausal women 0 Speci c cases Health status monitored in 16608 women over a 5 year period Some took hormones7 others did not 0 lnferential goal Determine if hormones improve the health of women not in the study CHAPTER 1 PRINCIPLES OF EXPERIMENTAL DESIGN 2 Process 9 Figure 11 Model of a variable process 12 Model of a process or system We are interested in how the inputs of a process affect an output Input variables consist of controllable factors x1 measured and determined by scientist uncontrollable factors 2 measured but not determined by scientist noise factors 6 unmeasured7 uncontrolled factors7 often called experimen tal variability or error For any interesting process7 there are inputs such that variability in input a variability in output If variability in an input factor x leads to variability in output y we say z is a source of variation In this class we will discuss methods of designing and analyzing experiments to determine important sources of variation 13 Experiments and observational studies Information on how inputs affect output can be gained from o Observational studies Input and output variables are observed from a pre eXisting population It may be hard to say what is input and what is output 0 Controlled experiments One or more input variables are controlled and manipulated by the experimenter to determine their e ect on the output CHAPTER 1 PRINCIPLES OF EXPERIMENTAL DESIGN 3 Example Women7s Health Initiative WHI 0 Population Healthy post menopausal women in the US 0 Input variables 1 estrogen treatment yesno 2 demographic variables age race diet etc 3 unmeasured variables 0 Output variables 1 coronary heart disease eg MI 2 invasive breast cancer 3 other health related outcomes 0 Scienti c question How does estrogen treatment affect health out comes Observational Study 1 Observational population 93676 women enlisted starting in 1991 tracked over eight years on average Data consists of z input variables yhealth outcomes gathered concurrently on existing populations 2 Results good healthlow rates of CHD generally associated with estro gen treatment 3 Conclusion Estrogen treatment is positively associated with health out comes such as prevalence of OHD Experimental Study WHI randomized controlled trial 1 Experimental population 373092 women determined to be eligible t gt 18845 provided consent to be in experiment t gt 16608 included in the experiment CHAPTER 1 PRINCIPLES OF EXPERIMENTAL DESIGN 4 z 1 estrogen treatment 16608 women randomized to either z 0 no estrogen treatment Women were of different ages and were treated at different clinics Women were blocked together by age and clinic7 and then treatments were randomly assigned within each agegtlttreatment block This type of random allocation is called a randomized block design age group 1 50 59 2 60 69 3 70 79 CllHlC 1 7111 7712 7713 2 7121 7122 7123 71M of women in study7 in clinic i and in age group j of women in block ij Randomization scheme For each block7 50 of the women in that block were randomly assigned to treatment x 1 and the remaining assigned to control x 0 Question Why did they randomize within a block 2 Results JAMA7 July 17 2002 Also see the NLHBI press release Women on treatment had a higher incidence rate of o CHD 0 breast cancer 0 stroke 0 pulmonary embolism and a lower incidence rate of o colorectal cancer 0 hip fracture CHAPTER 1 PRINCIPLES OF EXPERIMENTAL DESIGN 5 3 Conclusion Estrogen treatment is not a viable preventative measure for CHD in the general population That is7 our inductive inference is speci c higher CHD rate in treatment population than control suggests general if everyone in the population were treated7 the incidence rate of CHD would increase Question Why the different conclusions between the two studies Con sider the following possible explanation Let x estrogen treatment 6 health consciousness not directly measured y health outcomes correlation Association between z and y may be due to an unmeasured variable 6 randomization correlation causation CHAPTER 1 PRINCIPLES OF EXPERIMENTAL DESIGN 6 Randomization breaks the association between E and z Observational studies can suggest good experiments to run7 but cant de nitively show causation Randomization can eliminate correlation between z and y due to a different cause 6 aka a confounder No causation without randomization 14 Steps in designing an experiment H D 00 7 CT CT ldentify research hypotheses to be tested Choose a set of capsiimental units7 which are the units to which treat ments will be randomized Choose a responseoutput variable Determine potential sources of variation in response a factors of interest b nuisance factors Decide which variables to measure and control a treatment variables b potential large sources of variation in the units blocking variables Decide on the experimental procedure and how treatments are to be randomly assigned The order of these steps may vary due to constraints such as budgets7 ethics7 time7 etc CHAPTER 1 PRINCIPLES OF EXPERIMENTAL DESIGN 7 Three principles of Experimental Design 1 Replication Repetition of an experiment Replicates are runs of an experiment or sets of experimental units that have the same values of the control variables More replication a more precise inference Let yA response of the ith unit assigned to treatment A yg response of the ith unit assigned to treatment B i 17 7n Then ijA 31 133 provides evidence that treatment affects response7 ie treatment is a source of variation7 with the amount of evidence increas ing with n 2 Randomization Random assignment of treatments to experimental units This removes potential for systematic bias on the part ofthe researcher7 and removes any preexperimental source of bias Makes confounding the effect of treatment with an unobserved variable unlikely but not impossible 3 Blocking Randomization within blocks of homogeneous experimental units The goal is to evenly distribute treatments across large potential sources of variation Example Crop study Hypothesis Tomato type A versus B affects tomato yield Experimental units three plots of land7 each to be divided into a 2 gtlt 2 grid Outcome Tomato yield Factor of interest Tomato type7 A or B Nuisance factor Soil quality bad soil gt good soil CHAPTER 1 PRINCIPLES OF EXPERIMENTAL DESIGN 8 Questions for discussion 0 What are the bene ts of this design 0 What other designs might work 0 What other designs wouldn7t work 0 Should the plots be divided up further If so7 how should treatments then be assigned Chapter 2 Test statistics and randomization distributions Example Wheat yield Question ls one fertilizer better than another7 in terms of yield Outcome variable Wheat yield Factor of interest Fertilizer type7 A or B One factor having two levels Experimental material One plot of land to be divided into 2 rows of 6 subplots each 1 Design question How should we assign treatmentsfactor levels to the plots We want to avoid confounding a treatment effect with another potential source of variation 2 Potential sources of variation Fertilizer7 soil7 sun7 water7 etc CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONSlO 3 Implementation If we assign treatments randomly7 we can avoid any pre eXperimental bias in results 12 playing cards7 6 red7 6 black were shuf ed and dealt 1st card black a 1st plot gets B 2nd card red a 2nd plot gets A 3rd card black a 3rd plot gets B This is the rst design we will study7 a completely randomized design 4 Results B A B A B B 269 114 266 237 253 285 B A A A B A 142 179 165 211 243 196 How much evidence is there that fertilizer type is a source of yield variation Evidence about differences between two populations is generally measured by comparing summary statistics across the two sample populations Recall7 a statistic is any computable function of known7 observed data 21 Summaries of sample populations Distribution 0 Empirical distribution l5rab a lt yi S bn 0 Empirical CDF cumulative distribution function F61 y S yn Meow o Histograms o Kernel density estimates Note that these summaries more or less retain all the information in the data except the unit labels Location CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIB UTIONS11 1 n 0 sample mean or average y g 211 yi 0 sample median Athe value yv5 such that m S mm 212 m 2 y39an 212 To nd the median7 sort the data in increasing order7 and call these values ya 7110 If there are no ties7 then if n is odd7 then Mm is the median 2 if n is even7 then all numbers between Mg and y1 are medians Scale 0 sample variance and standard deviation 0 interquantile range 11251175 interquartile range 7102579975 95 interval Example Wheat yield All of these sample summaries are easily obtained in R gt yAlt7c11i47 2377 1797 1667 2117 196 gtyBlt7c2697 266 2537 2amp57 1427 243 edian yA 18 i 75 CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS12 Q H No 2 w o 3 c5 5 a 3 b9 4 8 mo Ln S SN D ace 0 N 0 Q N o c5 c2 8 8 o I I I O I I I I I O I I I I I 15 20 25 10 15 20 25 30 0 10 20 30 40 y y y 3 53 2 EC 0 at 5 0 Do c 39 39 w a 0 ca Scs 10 15 20 25 go In VA 5 o39 93 N 53 c5 0 ISO B o q o o o I I I I l l l I O I I I I I I 15 20 25 10 15 20 25 30 10 15 20 25 30 35 y YB Figure 21 Wheat yield distributions CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS13 gt sdyA 1 4234934 gt sdyB 1 54151699 gt quantileyAprobCi257475 25 75 164850 204725 gt quantileyBprobC4257475 25 75 24550 264825 So there is a difference in yield for these wheat elds Would you recommend B over A for future plantings Do you think these results generalize to a larger population 22 Hypothesis testing via randomization Questions 0 Could the observed differences be due to fertilizer type 0 Could the observed differences be due to plot to plot variation Hypothesis tests 0 H0 null hypothesis Fertilizer type does not affect yield 0 H1 alternative hypothesis Fertilizer type does affect yield A 4 4 4 1 Lat 4L 39 test vuluut D the plan il li rv Of H0 in light Of the data Suppose we are interested in mean wheat yields We can evaluate H0 by answering the following questions 0 ls a mean difference of 593 plausibleprobable if H0 is true 0 ls a mean difference of 593 large compared to experimental noise To answer the above7 we need to compare lg73 Al 5937 the observed difference in the experiment CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS14 to values of 1173 7 ijAl that could have been observed if H0 were true Hypothetical values of 1173 yTAl that could have been observed under H0 are referred to as samples from the null distribution Finding a null distribution Let 9YA7YB 9YLA7 7Y6A7Y1B7 7Y6B lYB YAl This is a function of the outcome of the experiment It is a statistic Since we will use it to perform a hypothesis test7 we will call it a test statistic Observed test statistic g1147 2377 7 1427 243 593 gobs Hypothesis testing procedure Compare gobs to 9YA7 YB for values of YA and YB that could have been observed7 if H0 were true Recall the design of the experiment 1 Cards were shuf ed and dealt B7 B7 B7 B7 and fertilizer types planted in subplots B A B A B B B A A A B A 2 Crops were grown and wheat yields obtained B A B A B B 269 114 266 237 253 285 B A A A B A 142 179 165 211 243 196 Now imagine re doing the experiment in a universe where H0 no treatment effect77 is true 1 Cards are shuf ed and dealt B7 B7 B7 B7 and wheat types planted in subplots CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS15 2 Crops are grown and wheat yields obtained B A B B A A 269 114 266 237 253 285 A B B A A B 142 179 165 211 243 196 Under this hypothetical treatment assignment7 YA7YB 114253211196 D73 7 YA 107 This represents an outcome of the experiment in a universe where o The treatment assignment is BABBAAABBAAB 0 H0 is true IDEA To consider what types of outcomes we would see in universes where H0 is true7 compute 9YA7YB under every possible treatment assignment and assuming H0 is true Under our randomization scheme7 there were 121 i 12 i 924 616 7 6 7 equally likely ways the treatments could have been assigned For each one of these7 we can calculate the value of the test statistic that would7ve been observed under H0 917927 79924 This enumerates all potential pre randomization outcomes of our test statis tic7 assuming no treatment effect Along with the fact that each treatment CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS16 l l l I 0 0 2 4 6 8 YB YA lYBYAl l quotf O O N 00 O O 30 z a a E E ow 99 Q o O O h 0 D1333 0 o O O 1 Figure 22 Approximate randomization distribution for the wheat example assignment is equally likely these value give a null distribution a probability distribution of possible experimental results if H0 is true gk S F95lH0 Pr9YA7YB S lHo 924 This distribution is sometimes called the randomization distribution be cause it is obtained by the randomization scheme of the experiment Comparing data to the null distribution ls there any contradiction between H0 and our data PrgYAYB 2 593lH0 0056 According to this calculation the probability of observing a mean differ ence of 593 or more is unlikely under the null hypothesis This probability calculation is called a p Ualue Generically a p value is The probability under the null hypothesis of obtaining a result as or more extreme than the observed result77 CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS17 The basic idea small p value a evidence against H0 large p value a no evidence against H0 Approximating a randomization distribution We dont want to have to enumerate all t possible treatment assign ments lnsteadl7 repeat the following S times for some large number S a randomly simulate a treatment assignment from the population of pos sible treatment assignments7 under the randomization scheme b compute the value of the test statistic7 given the simulated treatment assignment and under H0 The empirical distribution of 91 7gs approximates the null distribu tion PrgYA7YB 2 gobslHO The approximation improves if S is increased 95 2 gobs S Here is some R code ylt7c26l9314266337253285442419165211243196 Xlt7077B777 77A777 7713777 77A777 7713777 7713777 7713777 77A 77A777 WA 7 7713777 77A77 gr nulllt7real for s in 1210000 xsimlt7sample X g null slt7 abs meany xsim B 7 meanyxsim A 23 Essential nature of a hypothesis test Given H07 H1 and data y y1 Wyn CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONSlS 1 From the data compute a relevant test statistic gy The test statistic gy should be chosen so that it can differentiate between H0 and H1 in ways that are scienti cally relevant Typically gy is chosen so that small under H0 93 is prObably large under H1 E0 Obtain a null distribution A probability distribution over the possible outcomes of gY under H0 Here Y Y1Yn are potential experimental results that could have happened under H0 00 Compute the p Ualue The probability under H0 of observing a test statistic gY as or more extreme than the observed statistic gy pvalue Pr9Y 2 9ylHo If the p value is small evidence against H0 If the p value is large i not evidence against H0 Even if we follow these guidelines we must be careful in our speci cation of H0 H1 and gY for the hypothesis testing procedure to be useful Questions 0 ls a small p value evidence in favor of H1 0 ls a large p value evidence in favor of H0 0 What does the p value say about the probability that the null hypoth esis is true Try using Bayes7 rule to gure this out 24 Sensitivity to the alternative hypothesis In the previous section we said that the test statistic gy should be able to differentiate between H0 and H1 in ways that are scienti cally relevant77 What does this mean Suppose our data consist of samples yA and yB from two populations A and B Previously we used gyAyB lyB 7 ml Let7s consider two different test statistics CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIBUTIONS19 t statistic 9y y 7 lyTB yiAl t A7 3 i 7 51117114 1713 2 71A 7 1 2 773 7 1 2 5p i 7 DA i 7 03 71A 1nB 1 71A 1 This is a scaled version of our previous test statistic in which we com pare the difference in sample means to a pooled version of the sample standard deviation and the sample size Note that this statistic is where 0 increasing in lyB 7 ml 0 increasing in 71A and n3 0 decreasing in 517 A more complete motivation for using this statistic will be given in the next chapter KolmogorovSmirnov statistic 9K5yA7yB max 301 7 FAyl yElR This is just the size of the largest gap between the two sample CDFs Comparing the test statistics Suppose we perform a CRD and obtain samples yA and yB like those in Figure 23 For these data 0 71A 773 40 o y A 10051173 970 0 5A 087 SB 207 The main difference between the two samples seems to be in their variances and not in their means Now lets consider evaluating H0 treatment does not affect response using our two new test statistics We can approximate the null distributions of gtYA Y3 and gKSYA YB by randomly reassigning the treatments but leaving the responses xed CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIB UTIONS20 8 cgt 5 EM 50 D o O l l l I o 6 8 10 12 14 A0 b YA L V o be N 39Equot o 50 Q o O O O o39 I I I I I I 6 8 10 12 14 6 8 10 12 14 YB Y Figure 23 Histograms and empirical CDFs of the rst two hypothetical samples Gsimlt7NULL for s in 125000 xsimlt7sample X yAsimlt7yXsim A yBsimlt7yXsim B g1lt7 g1 tstat yAsim yBsim g2lt7 g1 ks yAsim yBsim Gsimlt7rbind Gsim7 c g1 g2 These calculations give t statistic gtyAyB 100 PrgtYAYB 2100 0321 KS statistic gKSyAyB 030 PrgKSYAYB 2 030 0043 The hypothesis test based on the t statistic does not indicate strong evidence against H07 whereas the test based on the KS statistic does The reason is that the t statistic is only sensitive to differences in means In particu lar7 if A 233 then the t statistic is zero7 its minimum value In contrast7 the KS statistic is sensitive to any differences in the sample distributions Now lets consider a second dataset7 shown in Figure 257 for which 0 nAnB40 CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIB UTIONS21 O 0 O 39139 3 3 w a T a s o s U U M Q Q l l O39 O O 12 O I 0 l 2 3 4 01 02 03 04 t statistic KS statistic Figure 24 Randomization distributions for the t and KS statistics for the rst example 79A 1011ny 1073 5A 175 SB 185 The difference in sample means is about twice as large as in the previous ex ample7 and the sample standard deviations are pretty similar The B samples are slightly larger than the A samples on average ls there evidence that this is caused by treatment Again7 we evaluate H0 using the randomization distributions of our two test statistics t statistic gtyA7yB 154 PrgtYAYB 2 154 0122 KS statistic gKSyAyB 025 PrgKSYAYB 2 025 0106 This time the two test statistics indicate similar evidence against H0 This is because the difference in the two sample distributions could primarily be summarized as the difference between the sample means7 which the t statistic can identify CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIB UTIONS22 O rq o o b a 2 5 o39 00 D o O O o39 l l l l I o 8 10 12 14 16 A0 b YA Lu lt1 0 b N E 039 Q o O O O o39 I I I I I I I I 8 10 12 14 16 8 10 12 14 16 YB Y Figure 25 Histograms and empirical CDFs of the second two hypothetical samples O 0 C5 In a a a V E O39 E U U m D Q N N O39 13 O L l I l l l l l 0 1 2 3 4 01 02 03 04 05 t statistic KS statistic Figure 26 Randomization distributions for the t and KS statistics for the second example CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIB UTIONS23 Discussion These last two examples suggest we should abandon 9 in favor of 9K5 if we are interested in comparing the following hypothesis H0 treatment does not affect response H1 treatment does affect response This is because as we found 9 is not sensitive all violations of H0 it is only sensitive to violations of H0 where there is a difference in means However in many situations we are actually interested in comparing the following hypotheses H0 treatment does not affect response H1 treatment increases responses or decreases responses In this case H0 and H1 are not complementary and we are only interested in evidence against H0 of a certain type ie evidence that is consistent with H1 In this situation we may want to use a statistic like gt 25 Basic decision theory Task Accept or reject H0 based on data truth action H0 true H0 false accept H0 correct decision type ll error reject H0 type I error correct decision As we discussed 0 the p value can measure of evidence against H0 0 the smaller the p value the larger the evidence against H0 Decision procedure 1 Compute the p value by comparing observed test statistic to the null distribution 2 Reject H0 if the p value S a otherwise accept H0 CHAPTER 2 TEST STATISTICS AND RANDOMIZATION DISTRIB UTIONS24 This procedure is called a level oz test It controls the pre experimental prob ability of a type I error or for a series of experiments controls the type I error rate Prtype l errorlHO Prreject HO HO Prpvalue S alHO 04 Single Experiment Interpretation If you use a level oz test for your ex periment where H0 is true then before you run the experiment there is probability 04 that you will erroneously reject H0 Many Experiments Interpretation lf level oz tests are used in a large population of experiments then H0 will be declared false in 100 gtlt oz of the experiments in which H0 is true PrH0 rejecteleO true Oz PrH0 accepteleO true i 1 7 Oz PrH0 rejecteleO false 7 PrH0 accepteleO false 7 PrH0 rejecteleO false is the power Typically we need to be more speci c than H0 false77 in order to calculate the power We need to specify how it is false Chapter 3 Tests based on population models 31 Relating samples to populations If the experiment is o complicated7 0 non or partially randomized7 or 0 includes nuisance factors then a null distribution based on randomization may be dif cult to obtain An alternative approach to hypothesis testing is based on formulating a sam pling model Consider the following model for our wheat yield experiment 0 There is a largein nite population of plots of similar sizeshapecom position as the plots in our experiment 0 When A is applied to these plots7 the distribution of plot yields can be represented by a probability distribution pA with expectation EDDA fypAydy MA variance VarYA EYA 7 1402 0 25 CHAPTER 3 TESTS BASED ON POPULATION MODELS 26 0 When B is applied to these plots7 the distribution of plot yields can be represented by a probability distribution p3 with expectation EYB fypBydy MB variance VarYB EYB 7 pg 0123 o The plots that received A in our experiment can be viewed as indepen dent sarnples frorn pA and likewise for plots receiving B Y1A7YnAA N pA Y1B7YnBB N p3 Recall from intro stats ElYAl Elizywl H u lM 7 11gt l 7 11gt We say that YA is an unbiased estimator of MA Eurtherrnore7 if YLAV 39 39 7YnAA are independent samples from population A7 then 1 VarYA VarlaZlLA 71 Z Var YLA 71 03 Ti71A This means that as 71A a 007 Varl7A EYA 7 MAY a 0 which7 with unbiasedness7 irnplies YA MA CHAPTER 3 TESTS BASED ON POPULATION MODELS 27 All possible39 A wheat yields 0 Experimental samples 3 00 O oo o O O 8 d 139 O ran om samp mg V V 3 3 8 8 O o O O O O O O l I l l l l I 5 10 15 20 25 30 35 yA g All possible39 B wheat yields Experimental samples 0 S o39 O O O 00 O V O O 0 52430 V 53515 N O Q o O O O O O O O Figure 31 The population model CHAPTER 3 TESTS BASED ON POPULATION MODELS 28 and we say that YA is a consistent estimator for MA Several of our other sample characteristics are also consistent for the corresponding population characteristics As 71 a 007 YA MA Si a 7 I 4149 a FAltzgt new 71A 700 Back to hypothesis testing We can formulate null and alternative hypotheses in terms of population quantities For example7 if MB gt MA we would recommend B over A7 and vice versa 0 Ho MA MB 0 H1 MA 3quot MB The experiment is performed and it is observed that 1837 y A lt 733 243 This is some evidence that MA gt MB How much evidence is it Should we reject the null hypothesis Consider evaluating evidence against Hg with our t statistic 7 7 7 lyB yAl 9tYA7yB spxlnA 1713 To decide whether to reject H0 or not7 we need to know the distribution of 9YA7YB under H0 Consider the following setup Assume Y1A7YnAA N pA Y1B7YnBB N p3 Evaluate H0 MA MB versus H1 MA 31 MB ie7 whether or not ypAydy ypBydy To make this evaluation and obtain a go value7 we need the distribution of 9YA7YB under MA MB This will involve assumptions aboutapproxima tions to pA and p3 CHAPTER 3 TESTS BASED ON POPULATION MODELS 29 32 The normal distribution The normal distribution is useful because in many cases7 0 our data are approximately normally distributed7 andor 0 our sample means are approximately normally distributed These are both due to the central limit theorem Letting PMU denote a population with mean M and variance 02 then X1 N 1310117 0i X N P 02 m 2 20m 2 i ZXZ amp normal iii7072 71 Xm N Pmva 71er Sums of varying quantities are approximately normally distributed Normally distributed data Consider crop yields from plots of land Y1 a1 gtlt seedi a2 gtlt soili a3 gtlt waterl a4 gtlt suni The empirical distribution of crop yields from a population of elds with varying quantities of seed7 soil7 water7 sun7 etc will be approximately normal M U7 where M and 039 depend on the effects 11 a2 a3 a4 and the variability of seed7 soil7 water7 sun7 etc Additive effects i normally distributed data Normally distributed means Consider the following scenario Experiment 1 sample 119 Experiment 2 sample 142 7y iid p and compute 732 7y iid p and compute 731 CHAPTER 3 TESTS BASED ON POPULATION MODELS 30 Experiment in sample yim y m iid p and compute m A histogram of 131 QM will look approximately normally distributed with sample mean y1 ym m M sample variance y1 ym 0271 ie the sampling distribution of the mean is approximately normali7 02717 even if the sampling distribution of the data are not normal Basic properties of the normal distribution 0 Y N normali702 aY b normalai b7 1202 0 Y1 normali1af7 Y2 normali2U Y1Y2 independent i Y1 Y2 normali1 270 0 oifyl of 52 Yn iid normaliz727 then 7 is statistically independent 7 How does this help With hypothesis testing Consider testing H0 MA MB treatment doesn7t affect mean Then re gardless of distribution of data7 under H0 YA amp normaliUinA YB amp normaliU23n3 37375714 amp normal0UiB where 033 Ti71A 01237113 So if we knew the variances7 we7d have a null distribution 33 Introduction to the t test Consider a simple one sample hypothesis test Y1 Yn iid P7 with mean M and variance 02 CHAPTER 3 TESTS BASED ON POPULATION MODELS 31 H0 3 M 0 H1 3 M 7 0 Examples Physical therapy 0 Y1 muscle strength after treatment muscle score before 0 H0 0 Physics 0 Y1 boiling point of a sample of an unknown liquid 0 H0 100 C To test H07 we need a test statistic and its distribution under H0 7 7 Mel might make a good test statistic o it is sensitive to deviations from H0 0 its sampling distribution is approximately known ElYl M Varl7 0271 7 is approximately normal Under H0 57 7 no N normal0UZn7 but we cant use this as a null distribution because 02 is unknown What if we scale 57 7 M0 Then Y 0 Y is approximately standard normal and we write fY normal01 Since this distribution contains no unknown parameters we could potentially use it as a null distribution However7 having observed the data y7 is fy a statistic o y is computable from the data and n is known CHAPTER 3 TESTS BASED ON POPULATION MODELS 32 o no is our hypothesized value a xed number that we have chosen 0 039 is not determined by us and is unknown The solution to this problem is to approximate the population variance 02 with the sample variance 52 Onesample t statistic Y M0 tY lt gt SW For a given value of no this is a statistic What is the null distribution of tY 177 177 50 so Am 0 WE av lfY1 Yn iid normalp0 02 then is normal0 1 and so it would seem that tY is approximately distributed as a standard normal distribu tion under H0 M 0 However if the approximation 5 m 039 is poor like when n is small we need to take account of our uncertainty in the estimate of 039 The X2 distribution Z1 Zn iid normal0 1 i Z Z N xi chi squared dist with 71 degrees of freedom ZltZi Z2 N X271 Y1 Yn iid normalpa Y1 7 M0 Yn 7 M039 iid normal0 1 5 2ltYi7ugt2xi 04 7 Y x24 Some intuition Which vector do you expect to be bigger Z1 Zn or Z1 7 ZZn 7 Z Z1 7 Z Zn 7 Z is a vector of length 71 but lies in an n 7 1 dimensional space In fact it is has a singular multivariate normal distribution with a covariance matrix of rank n 7 1 pX CHAPTER 3 TESTS BASED ON POPULATION MODELS 33 00 o o39 q o o39 o O O 0 5 10 15 20 25 30 X Figure 32 X2 distributions Getting back to the problem at hand7 1 7171 71712 7171 7 2 i 5702 N X2717 which is a known distribution we can look up on a table or with a computer 02 039 The t distribution If 0 Z N normal 01 g 0 X N xi 0 ZX statistically independent7 then Z 7 tm7the t distribution with m degrees of freedom 1 Xm How does this help us Recall that if Y1 Yn iid normalp7 02 pt CHAPTER 3 TESTS BASED ON POPULATION MODELS 34 v 0 0 l O g 0 O I I I I I I I 73 72 71 0 1 2 3 t Figure 33 t distributions o V507 7 M039 norrnal01 73182 N Xf 0 Y 52 are independent Let Z V7307 7 MU7 X 73152 Then Z W i W0 Xn 1 kHzn 71 This is still not a statistic because M is unknown However7 under a speci c hypothesis like H0 M no it is a statistic Y M0 s It is called the t statz39stz39c KY N tnil 0 Some questions for discussion CHAPTER 3 TESTS BASED ON POPULATION MODELS 35 o What does Xm converge to as m 7 oo o What happens to the distribution of tY when n 7 007 Why 0 Consider the situation where the data are not normally distributed 7 What is the distribution of V507 7 M039 for large and small 71 7 What is the distribution of 7273152 for large and small 71 7 Are V507 7 M and 52 independent for small 71 What about for large n Twosided onesample ttest H Sampling model Y1 Yn iid normali702 D Null hypothesis H0 M no 00 Alternative hypothesis H1 M 31 0 4 Test statistic tY V7307 7 s o Pre experiment we think ofthis as a random variable7 an unknown quantity that is to be randomly sampled from a population 0 Post experiment this is a xed number Cf Null distribution Under the normal sampling model and H07 the sam pling distribution of tY is the t distribution with n 7 1 degrees of freedom tY N tikl lf H0 is not true7 then tY does not have a t distribution If the data are normal but the mean is not no then tY has a non central t distribution7 which we will use later to calculate power7 or the type ll error rate If the data are not normal then the distribution of tY is not a t distribution CHAPTER 3 TESTS BASED ON POPULATION MODELS 36 6 p value Let y be the observed data p value PrltYl Z WWHHO PrlTn1lthYl 2 gtlt PrTn1 Z 7 2 17 pttobsn 71 ttestymu 111110 7 Level oz decision procedure Reject H0 if 0 p value S 04 or equivalently 0 Z tn11a2 for 04 tn11a2 2 The value tn11a2 2 is called the critical value value for this test In general7 the critical value is the value of the test statistic above which we would reject H0 Question Suppose our procedure is to reject H0 only when ty 2 tn11a ls this a level oz test 34 Two sample tests Recall the wheat exarnple B A B A B B 269 114 266 237 253 285 B A A A B A 142 179 165 211 243 196 Sampling model 11147 YnAA N iid norrnalpAUZ 113 7Ynng N iid norrnaluBUZ In addition to norrnality we assume for now that both variances are equal CHAPTER 3 TESTS BASED ON POPULATION MODELS 37 Hypotheses Ho MA MB HA HA 7e MB Recall that YB YANNMB MA7UZ 7117i Hence if H0 is true then 1 1 YBiYAN0702 77D 71A B How should we estimate 02 7 52 Z1yiA 1702 ZE yw QB p HA7177B1 71A 1 2 773 1 2 HA71HB1DA HA71HB1DB This gives us the following two sample t statistic YB 7 YA tYA7YB N tnAn372 527 a t E Selfcheck exercises 1 Show that 71A 713 7 212702 xiAJrnBiz recall how the X2 distribu tion was de ned 2 Show that the two sarnple t statistic has a t distribution with nAnB 7 2df Numerical Example Wheat again Suppose we want to have a type l error rate of Oz 005 Decision procedure 0 Level 04 test of H0 MA MB with 04 005 o Reject H0 if pvalue lt 005 0 Reject H0 gt t10y975 CHAPTER 3 TESTS BASED ON POPULATION MODELS 38 Data 0 y 1836 5 1793 nA 6 o 793 2430 5 2654 713 6 t statistic o 512 2224 517 472 o tyAyB 593472 16 16 218 Inference 0 Hence the p value PrlT10l 2 218 0054 0 Hence H0 MA MB is not rejected at level 04 005 77 77 gt t test yx77 77 yX var equalTRUE Two Sample titest data yx 771477 and yx 771377 t 7211793 df 10 pivalue 0105431 alternative hypothesisztrue difference in means is not equal to 0 95 percent confidence interval 711999621 01132954 sample estimates mean of X mean of y 18136667 24130000 Always keep in mind where the p value comes from See Figure 34 Comparison to the randomization test Recall that we have already compared the two sample t statistic to its mn domz39zatz39zm distribution A sample from the randomization distribution were obtained as follows 1 Sample atreatment assignment according to the randomization scheme 2 Compute the value of tYAYB under this treatment assignment and assuming the null hypothesis pT CHAPTER 3 TESTS BASED ON POPULATION MODELS 39 04 03 01 1 r 4 2 0 2 4 T Figure 34 The t distribution under Hg for the wheat example The randomization distribution of the t statistic is then approximated by the empirical distribution of tlt1gt MS To obtain the go Value7 we compare tabs tyAyB to the empirical distri bution oft1 tS t9 gt to s pvalue 1 151 b1 tistatiobslt7titest yx77 77yx B 7 variequa1T stat tistatisimlt7rea1 fors in 1210000 xsimlt7samp1e X tmplt7t 1 test yxsim B 7y xsim A 7var i equa1T t stat isimslt7tmpstat mean abstistatisim gt abstistatiobs Density CHAPTER 3 TESTS BASED ON POPULATION MODELS 40 139 o m o N o S I I I I I I I 6 74 72 0 2 4 6 ITYA YB Figure 35 Randomization and t distributions for the t statistic under H0 When I ran this7 I got tltsgt gt 218 W 0058 m 0054 PrlTnAnB2l 2 218 ls this surprising These two p values were obtained via two completely different ways of looking at the problem Assumptions Under H07 0 Randomization Test 1 Treatments are randomly assigned 0 t test 1 Data are independent samples 2 Each population is normally distributed 3 The two populations have the same variance Imagined Universes o Randomization Test Numerical responses remain xed7 we imagine only alternative treatment assignments CHAPTER 3 TESTS BASED ON POPULATION MODELS 41 o t test Treatment assignments remain xed we imagine an alternative sample of experimental units andor conditions giving different numerical responses Inferential Context Type of generalization o Randomization Test inference is speci c to our particular exper imental units and conditions 0 t test under our assumptions inference claims to be generaliz able to other units conditions ie to a larger population Yet the numerical results are often nearly identical Keep the following concepts clear tstatistic a scaled difference in sample means computed from the data tdistribution the probability distribution of a normal random variable divided by the square root of a X2 random variable ttest a comparison of a t statistic to a t distribution randomization distribution the probability distribution of a test statis tic under random treatment reassignments and H0 randomization test a comparison of a test statistic to its randomization distribution randomization test With the tstatistic a comparison ofthe t statistic to its randomization distribution Some history de Moivre 1733 Approximating binomial distributions T 21 Yi Y E 0 1 Laplace 1800s Used normal distribution to model measurement error in experiments Gauss 1800s Justi ed least squares estimation by assuming normally distributed errors CHAPTER 3 TESTS BASED ON POPULATION MODELS 42 GossetStudent 1908 Derived the t distribution GossetStudent 1925 testing the signi cance77 Fisher 1925 level of signi cance77 Fisher 1920s7 Fisher7s exact test Fisher 1935 It seems to have escaped recognition that the physical act of randomisation which as has been shown is necessary for the validity of any test of signi cance affords the means in respect of any particular body of data of examining the wider hypothesis in which no normality of distribution is implied77 Box and Andersen 1955 Approximation of randomization null distribu tions and a long discussion Rodgers 1999 If Fisher7s ANOVA had been invented 30 years later or computers had been available 30 years sooner our statistical procedures would probably be less tied to theoretical distributions as what they are today77 35 Checking assumptions For our t statistic YB 7 YA tYAYB 1 1 527 a t E we showed that if YLA YnAA and Y1BYnBB are independent samples from pA and p3 respectively and c pA and p3 are normal distributions then tYA7 YB N tnAn372 So our null distribution really assumes conditions a b and Thus if we perform a level oz test and reject H0 we are really just rejecting that a b c are all true CHAPTER 3 TESTS BASED ON POPULATION MODELS 43 reject H0 i one of a b or c is not true For this reason we will often want to check if conditions b and c are plausibly met If b is met c is met H0 is rejected then this is evidence that H0 is rejected because MA 31 MB 351 Checking normality Normality can be checked by a normal probability plot Idea Order observations within each group y1A S y2A S S ynAA compare these sample quantiles to the quantiles of a standard normal distribution 2mm S 22 S S 2a nA 2 nA 2 nA 2 here PrZ S 2 k 7 i 7 l The 71 is a continuity correction A 2 TM 2 2 If data are normal the relationship should be approximately linear Thus we plot the pairs 2amp72711M 7 27l79nAA 2 2 Normality may be checked roughly by tting straight lines to the probability plots and examining their slopes 352 Unequal variances For now we will use the following rule of thumb lf 14 lt 51245213 lt 4 we wont worry too much about unequal variances Sample Quantiles 2 Sample Quantiles 20 Sample Quantiles 05 05 Sample Quantiles 0 5 16 20 24 1 05 05 10 25 15 CHAPTER 3 TESTS BASED ON POPULATION MODELS 44 o v2 o 2 o E E E E 395 a 395 O E E E H I N In I O N O o O m 2 2 2 O ca 00 ca ca H m o o o o o I I I I I H I I I I I I I I I I I 10 00 10 10 00 10 10 00 10 10 00 10 Theoretical Quantiles Theoretical Quantiles Theoretical Quantiles Theoretical Quantiles 039 o o H m 39 E E 039 39 E 2 g E 3 E E 5 5 v2 5 O 2 3 2 cf 2 O E E E0 m o m n m o I o 239 o o I I I I I l I I I I I l I I I I I 10 00 10 10 00 10 10 00 10 10 00 10 Theoretical Quantiles Theoretical Quantiles Theoretical Quantiles Theoretical Quantiles o m m m g o H H o H o E E o E A O O O o O 2 O 2 2 o a a D o o m i E cf E 3 E O o o O i o I I I I I I I I I I I I l 10 00 10 10 00 10 10 00 10 1 Theoretical Quantiles Theoretical Quantiles Theoretical Quantiles Theoretical Quantiles o o E N E 3 E m 3 o 3 3 w a a a cquot 39 E r E 3 E o m o m m 2 o I I I I I I I I I I I I I I I l I 0 0 1 0 0 1 1 0 0 1 0 0 10 00 10 Theoretical Quantiles Theoretical Quantiles Theoretical Quantiles Figure 36 Normal scores plots 10 00 10 Theoretical Quantiles CHAPTER 3 TESTS BASED ON POPULATION MODELS 45 This may not sound very convincing In later sections7 we will show how to perform formal hypothesis tests for equal variances However7 this won7t completely solve the problem If variances do seem unequal we have a variety of options available 0 use the randomization null distribution 0 transform the data to stabilize the variances to be covered later 0 use a modi ed t test that allows unequal variance The modi ed t statistic is 173 A mm 13 f This statistic looks pretty reasonable7 and for large 71A and 713 its null dis tribution will indeed be a normal01 distribution However7 the exact null distribution is only approximately a t distribution7 even if the data are actually normally distributed The t distribution we compare tw to is a tym distribution7 where the degrees of freedom Vw are given by 2 A 71B 2 239 1 1 nAil nA ngil n3 This is known as Walsh s approzimatz39on it may not give an integer as the degrees of freedom Vw This t distribution is not7 in fact the exact sampling distribution of tdiffyA yB under the null hypothesis that MA MB and 0 31 723 This is because the null distribution depends on the ratio of the unknown variances7 0 and 723 This dif culty is known as the Behrens Fisher problem Which twosample t test to use o If the sample sizes are the same 71A 713 then the test statistics tw 31A7 33 and tyA7 313 are the same however the degrees of freedom used in the null distribution will be different unless the sample standard deviations are the same

### BOOM! Enjoy Your Free Notes!

We've added these Notes to your profile, click here to view them now.

### You're already Subscribed!

Looks like you've already subscribed to StudySoup, you won't need to purchase another subscription to get this material. To access this material simply click 'View Full Document'

## Why people love StudySoup

#### "Knowing I can count on the Elite Notetaker in my class allows me to focus on what the professor is saying instead of just scribbling notes the whole time and falling behind."

#### "Selling my MCAT study guides and notes has been a great source of side revenue while I'm in school. Some months I'm making over $500! Plus, it makes me happy knowing that I'm helping future med students with their MCAT."

#### "Knowing I can count on the Elite Notetaker in my class allows me to focus on what the professor is saying instead of just scribbling notes the whole time and falling behind."

#### "It's a great way for students to improve their educational experience and it seemed like a product that everybody wants, so all the people participating are winning."

### Refund Policy

#### STUDYSOUP CANCELLATION POLICY

All subscriptions to StudySoup are paid in full at the time of subscribing. To change your credit card information or to cancel your subscription, go to "Edit Settings". All credit card information will be available there. If you should decide to cancel your subscription, it will continue to be valid until the next payment period, as all payments for the current period were made in advance. For special circumstances, please email support@studysoup.com

#### STUDYSOUP REFUND POLICY

StudySoup has more than 1 million course-specific study resources to help students study smarter. If you’re having trouble finding what you’re looking for, our customer support team can help you find what you need! Feel free to contact them here: support@studysoup.com

Recurring Subscriptions: If you have canceled your recurring subscription on the day of renewal and have not downloaded any documents, you may request a refund by submitting an email to support@studysoup.com

Satisfaction Guarantee: If you’re not satisfied with your subscription, you can contact us for further help. Contact must be made within 3 business days of your subscription purchase and your refund request will be subject for review.

Please Note: Refunds can never be provided more than 30 days after the initial purchase date regardless of your activity on the site.