Appl Research Methods
Appl Research Methods ESP 178
Popular in Course
Popular in Environmental Science & Policy
This 13 page Class Notes was uploaded by Amber Homenick on Tuesday September 8, 2015. The Class Notes belongs to ESP 178 at University of California - Davis taught by Susan Handy in Fall. Since its upload, it has received 38 views. For similar materials see /class/187481/esp-178-university-of-california-davis in Environmental Science & Policy at University of California - Davis.
Reviews for Appl Research Methods
Report this Material
What is Karma?
Karma is the currency of StudySoup.
You can buy or earn more Karma at anytime and redeem it for class notes, study guides, flashcards, and more!
Date Created: 09/08/15
ESP 178 Applied Research Methods 221 Lecture and Exercise Measures and Scales Two types of scales Scales as responses to speci c question Likert scales 7 do you agree or disagree 3point 5point or 7point scale Variations 7 true or not true important or not important Scales as combination of items sometimes called an index 7 to create a single measure of a complex concept with multiple dimensions Ways of combining 7 usually average or add or count Factor analysis 7 to create or con rm distinct factors from list of items 139 I39 39 quotquot testing7 39 J of r to items in scale Level of measurement 7 ordinal or interval or ratio Examples from articles Gatersleben et a1 pg 339 pp 342345 Environmental behavior scale 7 based on 33 items asking how frequently they perform certain environmentally sound consumer behaviors factor analysis to identify 6 underlying dimensions used to create 2 scales scores range from 1never to 5always Environmental awareness 7 based on 12 items agreedisagree scale scores on items averaged to get scale scores range from 1 to 5 Grob pg 211212 Environmental behavior 21 items representing pro or antienvironmental behaviors plus one if do pro items if don t do anti items minus one if don t do pro item if do anti items for each item scores range from 21 to 21 E 39 39 7 factual 39 39 Ag six multiple choice questions one point for each correct answer scores range from 0 to 6 E 39 39 7 quot39 ofproblems six items truenot true scale scores on items averaged to get scale scores range from 1 to 7 Nordlund and Garvill pg 747 Proenvironment behavior frequency of performing 25 different proenvironmental behaviors e g recycling saving hot water scores on items averaged to get scale scores range from 1 regularly to 4 rarely Problem awareness 12 items agreedisagree scale scores on items averaged to get scale scores range from 1 to 7 Steps for constructing a scale Dimensions of concept Items to represent these dimensions use same format for each item Scales for items eg 5 point agreedisagree scale use same scale for each item Combining scores on each item into composite scale eg add average Note Researchers often use existing scales developed by other researchers ESP 178 Applied Research Methods 117 Lecture Sampling Key Concepts from Chapter 5 Sampling a way of studying a subset of the population but still ensuring generalizability vs census 7 study of entire population 7 does the study have external validity Term Definition Exam le Another Example Unit of Analysis Level of social life being Child CDdays playing Neighborhood CDaverage in Chapter 6 studied 7 individuals or outside per week lliving level of fitness of children groups of individuals on culdesac or not lshare of streets that are cul desacs Population The set of individuals or Sacramento area children Neighborhoods in the Target other elements to which Sacramento region study findings will be pOPU latlon generalized sample is drawn from the population Elements Individual members of the Charlie Lucy Linus Patty Midtown Natom as Land Park population Violet etc Laguna West Sampling Frame List of all elements or Public school rolls List of neighborhoods other units containing the Phone listings List of cities in Sacramento elements used for Marketing list of region drawing sample households with children Enumeration Units that contain one or Households contain Cities contain neighborhoods Units more elem ents useful if children sampling frame not available for elements themselves Cases Elements selected for the Charlie Patty Natomas Laguna West sample Generalizability Defmition Example 1 Example 2 Type Sample From sample to Results for sample of Results for sample of Generalizability population depends children the same as for all neighborhoods in Sacramento n sampling error children of Sacramento region region the same as for all neighborhoods in Sacramento region CrossPopulation Generalizability From one population to another Results for individuals in Sacramento region the same as for other regions Results for neighborhoods in Sacramento region the same as for cities elsewhere in California The US Sampling error Any difference between characteristics of a sample and the characteristics of population from which it was drawn Random sampling error inherent in process of sampling Systematic sampling error minimizable with good sampling plan Representative sample A sample that looks like the population from which it was selected in all respects that are potentially relevant to the study How do you know you have one Type De nition Use Issues Probability Know in advance how likely that any Allows for use of Systematic bias non sampling ie element of the population will be inferential statistics I respondents random selected for the sample random quantitative hypothes1s selection equal chance testing studies Nonprobability sampling ie nonrandom Do not know in advance how likely that any element of the population will be selected for the sample non random selection not equal chance Allows for use of descriptive statistics only qualitative exploratory studies Not representative so can t generalize Type Method De nition Example Probability Simple random Sample chosen strictly by Random digit dialing from phone sampling samE ling chance listings for Sacram ento region Systematic random Select first element randomly Every 10003911 number in the phone samEling the select every nth element listings Stratified random Population sorted into strata Households split into those living on sampling 7 according to key culdesacs those not random characteristic random sample drawn from each group if 1 proponlonate sampling within strata sample in 10 households live on culdesac for each stratum in proportion then 1 culdesac household for every to its size in population 9 nonculdesac households Stratified random Population sorted into strata Households split into those living on sampling according to ke culdesacs those not random characteristic random sample drawn from each group 500 dlspmponlonate sampling within strata sample households randomly selected in for each stratum intentionally each group even though only 1 in 10 not in proportion to its size in household live on culdesacs in population region Cluster sampling Cluster is naturally occurring Neighborhoods randomly selected grouping of elements draw a households within these random sample of clusters neighborhoods randomly selected then draw random sample of elements within cluster Non Availability Elements selected because Friends and coworkers of the probability sampling or they re easy to find research team who have children used as sample OR families at the samphng conv mence Farmer s Market asked to participate sampllng Quota sampling Quota set to ensure that Families at the Farmer s Market sample represents certain asked to participate until 10 families characteristics in proportion to on culdesacs and 90 not on culde their prevalence in population sacs recruited Purposive sampling Sample includes individuals 20 public health officials who work particularly knowledgeable on programs to get children to be about issues under study 7 more physically active key informants Snowball sampling Start with initial sample ask Find a few families with children that these individuals to live on culdesacs ask them if they recommend other individuals have neighbors with children or other and so on for hardto reach friends who live on culdesacs and 39 so on Inferential Statistics Basics for Probability Sampling Sample statistic statistic e g mean computed from sample data Population parameter true value for statistic e g mean for population we don t know this Sampling error population parameter 7 sample statistic we don t know this Con dence interval interval in which we can be con dent that true value lies based on sample statistic and its standard error 95 con dence interval is statistic 2 standard errors General rules Larger sample means more con dence in representativeness less sampling error narrower con dence intervals More homogeneous population means more con dence in representativeness less sampling error narrower con dence intervals What matters is sample size not share of population sampled Other things to think about in sampling Goal Example 1 Example 2 Ensuring that the Sample includes children who live on Sample includes neighborhoods that have independent variable culdesacs and children who don t lots of culdesacs and neighborhoods that have few culdesacs varres Ensuring that the Sample includes only children from Sample includes only neighborhoods with control variables don t moderateincome households average income in the moderate range vary Readings for 122 Review the sampling approach for each of the following articles CM Werner Feedback and Participants sections pg 1920 of reader Nord et al Data and Measurement section pg 29 Daneshvary et al Empirical Analysis section second paragraph pg 154 Gatersleben et al Two Studies and Selection of Respondents sections pg 53 and Selection of Respondents section pg 5657 Wallner et al Sampling section pg 139 Kaplowitz and Hoehn Research Design section second paragraph pg 148 Bengston and Fan Data and Methodology section rst paragraph pg 158 McComas et al Methods section rst two paragraphs pg 167 AbuGhazzeh Methods section pg 173174 For each article try answering the following questions What is the unit of analysis From what population were the cases selected What method e g simple random sampling availability sampling etc was used to select cases from this population Do the cases that were studied represent in the aggregate the population from which they were studied What information do you need to answer these questions that the authors don t provide ESP 178 Applied Research Methods Strati ed vs Cluster Sampling Strati ed Cluster Example 1 Divide city into districts lDivide city into districts clusters strata 2Draw random sample of districts 2 Draw random sample of 3Draw random sample of 39 39 J from each district 39 39 J from each district Reason To ensure desired number of To make it easier to do doortodoor for use 39 39 J in each district surveys Another 1 De ne strata for UCD students 1 De ne clusters as UCD classes Example by year e g 151 2 etc 2 Draw a random sample of classes 2 Draw a random sample of from the course schedule students from within each year 3 Draw random sample of students from within those classes 7 or survey all of them Reason To ensure desired number of To make it easier to distribute surveys for use students in each year in person Criteria Strata de ned by selected Cluster de ned by spatial or social for characteristic 7 key independent linkages 7 may not have anything to de ning variable e g what year they re do with study e g what class they re in in Notes Need to have sampling frame for Useful when surveys to be each stratum May be necessary when sampling frame for entire population is not available but separate sampling frames for strata are administered inperson Useful when sampling frame for clusters is available but not for elements within clusters ESP 178 Applied Research Methods 124 Lecture Causality Key Concepts from Chapter 6 both editions Types of Causal In 39 De nition Example Nomethetic Change in independent variable will be Converting through streets to culdesacs a general followed by change in dependent traffic calming technique is followed by variable all else equal ceteris parabis increases in street play for children Idiographic Specific series of events thoughts or When we moved to Davis we bought a house Speci c actions that result in particular outcome on a culdesac and now our children play for particular individual outside m ore Causal Internal Validity The observed relationships are real Criteria De nition Everml Association Relationship between cause The average frequency of street play is positively associated and effect is established with culdesacs living on a culdesac means more street play Nonspurious Spurious if apparent Observed association between culdesacs and street play can be association between two explained by association between parental attitudes and both variables caused by culdesacs and street play 7 so association is spurious extraneous third variable Timeorder Cause comes before effect Frequency of street play increases after children move to culde sac Causal Logical explanation for how Culdesacs have lower levels of traffic and neighbors tend to mechanism cause leads to effect know each other better so parents are more willing to let their children play outside Context Understand the conditions In some urban areas deadend streets are seen as potential crime under which the relationship magnets so parents are reluctant to let their children play holds outside there Treatment of Time Ensure that the cause comes before the effect Type Approach Key Features Example Cross One sample that varies on Sample of households surveyed on street sectional independent variable with play and traffic levels analysis looks at measurement at one point in di erences in street play associated with time di erences in traffic levels across households as unit of analysis Longitudinal Repeated Cross One sample that varies on Sample of households surveyed on levels Sectional independent variable with of street play and traffic levels a different Design 7 trend study measurement at one point in time second sample that varies on independent variable with measurement at second point in time sample of households surveyed five years later on street play and traffic levels analysis looks at change in street play associated with change in traffic levels across the samples as unit of analysis Panel Design FixedSample One sample that varies on independent variable with measurement at two points in tim e Sample of households surveyed on levels of street play and traffic levels same sample surveyed again two years later analysis looks at change in street play associated with change in traffic levels for households as unit of analysis ESP 178 Applied Research Methods Sampling Inferential Statistics n sample size Sample statistic statistic computed from sample data eg mean Population parameter true value of statistic for population Sampling error population parameter 7 sample statistic Standard deviation how close individual scores are to mean for the sample to what degree individual scores cluster around the sample mean Distribution of individual scores 7 centers around sample mean Number of individual sc ores Sample Scores mean Standard error SE how close mean scores from repeated samples will be to true population mean to what degree sample means cluster around the population mean Number of samples with mean of that value Distribution of sample statistics 7 centers around true value the population parameter Population Sample mean means We can estimate SE using SD and n SE of mean SDsquare root n We can use SE to calculate a 95 con dence interval CI statistic 196 X SE Implications More homogeneous populations mean tighter con dence intervals SD i gt SE i gt C1 i Larger sample sizes mean tighter con dence intervals n T gt SE i gt C1 i Example UC Davis vs CSU Chico Beer Drinking Let s say that we survey 20 students from each campus about their weekly beer drinking measured in pints The histogram for the hypothetical survey results looks like this Figure 1 Distribution of Pints per Week 45 40 35 30 25 20 15 10 05 00 Number of Students 12345678910111213 Pints per Week The average for UCD is 41 with a standard deviation of 254 The average for CSUC is 60 with a standard deviation of 332 Can we say with con dence that CSUC students drink more than UCD students Let s calculate 95 con dence intervals Table l Table 1 f 39 39 quot off quot 39 Intervals UCD CSUC Mean 41 60 Standard Deviation 254 332 Standard Error n20 126 136 95 CI low 157 334 95 CI high 653 866 Standard Error n200 018 024 95 CI low 370 554 95 CI high 440 646 The confidence intervals mean that for 95 of the samples of 20 students we could draw the sample average would be between 157 and 653 at UCD and between 334 and 866 at CSUC Another way we say this commonly but not quite correctly is that we are 95 con dent that the true average lies in these ranges So do CSUC students drink more Notice how the con dence intervals overlap This means that we can t conclude 7 statistically 7 that average beer drinking is different on the two campuses The average for the population could be equal or even higher at UCD But what if our sample was 200 students with the same mean and same standard deviation Notice how much the standard error drops with a larger sample the means of repeated samples are going to cluster more than with a small sample This means much tighter con dence intervals Now we can be pretty darn sure that CSUC students drink more than UCD students Hypothetically of course Data Collection Errors Errors ESP 178 Applied Research Methods 214 Lecture Survey Design Surveys Available data Indepth interviews Focus groups Measurement error street non Nonresponse to answer survey observation out Sampling error Survey Design Purpos e of questions To measure variables in conceptual model 7 include one or more questions per variable unless variables are to be measured using other methods Rules for question writing Design avoid confusing phrasing no double negatives no doublebarreled questions minimize risk of bias wording of question response alternatives offered avoid making I or 39 39 39 people like to agree minimize fencesitting neutral opinion and oating no real opinion maximize utility of response categories exhaustive and mutually exclusive From Dillman use simple words do not be vague keep it short be specific do not talk down to respondents avoid bias avoid objectionable questions do not be too specific avoid hypothetical questions ing questionnaires build on existing instruments e g borrow questions refine and test questions 7 pretest expert panel focus groups pilot tests add interpretive questions if needed openended explanations of responses maintain consistent focus don t ask too many irrelevant questions order the questions context effect partwhole question effect make the questionnaire attractive white space formatting etc Survey Administration Design How Setting Questionnaire Inperson Pros and Cons of each approach See Exhibit 814 in Schutt 810 in Red Edition Dillman s Total Design Method for mail surveys l Respondentfriendly questionnaire 2 Four contacts by rst class mail with additional special contact 1 Prenotice letter a few days before questionnaire is sent 2 Questionnaire with cover letter explaining importance 3 Thank you postcard a few days up to a week after questionnaire 4 Replacement questionnaire sent to nonrespondents 24 weeks after previous questionnaire mailing 5 Final contact by telephone a week or so after fourth contact or by fedex or express mail 24 weeks after previous mailing 3 Return envelopes with real rstclass stamps vs businessreply envelopes 4 Personalization of correspondence 5 Token prepaid financial incentives Incentives Things to think about Measurement Error Non response Bias Writing Questions AKIbigUOUS T00 long Biased Too sensitive Specific enough Designing the Instrument Order of questions logical Too thick Impact of order on answers Hard to read Hard to follow Not appealing enough Any unerfluou questions Administering the Survey Interviewer in uence Certain kinds of people less likely to respond Incentives to get more people to respond Readings for 219 There are two good articles on survey design and question writing in the reader the Fowler article and the Dillman article plus copies of two large environmental surveys one from Minnesota and one from Europe links to these surveys and others are available on the class webpage Read or at least skim the articles and take a look at the methodologies for these surveys plus the questions themselves For the next stage of the proposal assignment you ll be designing your own survey instrument or other type of data collection instrument ESP 178 Applied Research Methods 34 Lecture Historical Research and Existing Data Key Points from Chapter 12 Chapter 11 in Red Book Unobtrusive methods Allow us to investigate social processes at other times and in other places when the actual participants in these processes are not available Cross Sectional I 1 Single Case Historical events Historical process research research Multiple Cases Crosssectional Comparative historical comparative research research Features of qualitative historical research Caseoriented holistic conjunctural temporal historically speci c narrative idiographic inductive Features of quantitative comparative research Uses available data to examine relationships between variables across cities nations etc Secondary data publicly available data archives data from another researcher data from your own previous projects Examples US Census httpwww census vmainww cen m html American Time Use Survey BLS httpwwwblsgovtushomehtm General Social Survey httpwwwnorcuchicagoeduprojectsgensocasp Responsible use of secondary data 1 What were agency s goals in collecting the data 2 Who was responsible for data collection and what were their quali cations 3 What data was collected and what were they intended to measure 4 When was the information collected 5 What methods were used for data collection 6 How is the information organized What form are data available in Adequate documentation 7 e g variable labels 7 How consistent are the data with data available from other sources 8 What is known about the success of the data collection effort Documentation Methodological Complications Measuring across contexts missing data hard to test reliability and validity measures are inadequate measurement equivalence a problem necessary simpli cations Sampling across time and place interdependence among cases Identifying causes method of agreement comparison of cases in terms of similarities and differences on independent and dependent variable Ethical Issues FOIA crosscultural issues ESP 178 Applied Research Methods 311 Lecture Focus Groups and Other Qualitative Techniques Data Collection Other Qualities Surveys Available data Focus on Nomothetic explanation Focus groups Observation Focus on Idio graphic explanation Method De nition Example Participant observation Sustained relationship with people in order to observe them as they go about their normal activities I rent a house on a culdesac for a summer get to know the residents there talk to them about street play and observe what their kids do Intensive or indepth interviewing Openended relatively unstructured questioning to get in depth information on feelings e nerience perceptions I find a convenience sample of families living on culdesacs and visit their houses one at a time to interview them about street play Focus groups Group interviews led by a facilitator who asks questions and encourages discussion among participants on the topic of interest I find a convenience sample of 12 families living on culdesacs and invite one parent from each family to participate in a group discussion held at the nearby library on a weekday evening Observation type Systematic observation De nition Exam le Researcher uses standard form on which to record variation in the variables of interest in the observed setting 7 quantitative data I use a standardized form to record number of basketball hoops other toys and play equipment chalk drawings etc on each street Complete observation Researcher tries to see things as they happen without actively participating in events I rent a house on a culdesac for a summer tell the residents why I m there then sit in the front window and observe what their kids do Participation and observation Researcher informs some group members about research interests but then participate in group to gain direct sense of what group members experience I rent a house on a culdesac for a summer tell the residents why I m there get to know the residents over time talk to them about street play and observe what their kids do Covert or complete participation Researcher keeps purpose secret and tries to act like other participants I rent a house on a culdesac for a summer don t tell the residents why I m there let my kids do what the other kids are doing and generally see what it s like to live on a culdesac Participatory action research Researcher serves as resource to participants I rent a house on a through street to observe street play and work with residents to get the city to put in a barrier or questions and recording answers Focus groups Ethical Issues Voluntary participation Subject wellbeing Identity disclosure Con dentiality 7 e g ctitious names Data Analysis see Chapter 10 Chapter 13 in red book Document transcripts notes etc Develop codes look for themes patterns key points Train coders practice coding test for reliability Code transcripts Analyze findings quotesexamples summary counts Re ect Computerassisted qualitative analysis Content analysis Readings for 313 Skim the articles listed in the Qualitative Studies section of the reader Wallner Hunziker and Kienast indepth interviews Kaplowitz and Hoehn indepth interviews versus focus groups Bengston and Fan content analysis McComas Shanahan and Butler content analysis AbuGhazzeh observation For each think about What was the question the researchers were asking What data collection approach did they use Could they have used other approaches What were the advantages of the approach they used 3 How did they pick their sample and if relevant recruit participants How did they carry out their data collection 4 How did they analyze the data What did they learn from this analysis
Are you sure you want to buy this material for
You're already Subscribed!
Looks like you've already subscribed to StudySoup, you won't need to purchase another subscription to get this material. To access this material simply click 'View Full Document'