Methods Of Data Collection
Methods Of Data Collection SOC 2820
Popular in Course
verified elite notetaker
verified elite notetaker
verified elite notetaker
verified elite notetaker
verified elite notetaker
verified elite notetaker
Popular in Sociology
This 44 page Class Notes was uploaded by Giovanna Casper on Wednesday September 30, 2015. The Class Notes belongs to SOC 2820 at Western Michigan University taught by Staff in Fall. Since its upload, it has received 11 views. For similar materials see /class/216816/soc-2820-western-michigan-university in Sociology at Western Michigan University.
Reviews for Methods Of Data Collection
Report this Material
What is Karma?
Karma is the currency of StudySoup.
You can buy or earn more Karma at anytime and redeem it for class notes, study guides, flashcards, and more!
Date Created: 09/30/15
Chapter 1 Science Society and Social Research Chapter Overview Chapter 1 begins to differentiate social scientific knowledge from other forms of knowledge by calling attention to how nonscientific observation leads to erroneous conclusions It explains how the social science approach overcomes many errors in everyday reasoning Motivations for conducting social research are addressed This chapter introduces students to the four reasons for conducting social research description exploration explanation and evaluation as well as the differences between qualitative and quantitative research The chapter includes a lengthy discussion of validity and authenticity as goals of social research The different dimensions of validity measurement validity generalizability and causal validity are defined and examined The chapter closes on a discussion of the strengths and limitations of social research noting the importance of other research and other researchers in the scientific endeavor Lecture Outline 1 Errors in Everyday Reasoning a Errors in Observation i Selective observation Choosing to look only at things that are in line with our own preferences or beliefs ii Inaccurate observation Thinking that we have seen something that is not true Overgeneralization concluding that What was observed or What is known to be true for some cases is true for all cases Illogical reasoning prematurely jumping to conclusions or arguing on the basis of invalid assumptions Resistance to change reluctance to change ideas even in light of new information May occur because 0 39 Egobased commitments inability to admit we re mistaken in earlier conclusions Excessive devotion to tradition Uncritical agreement with authority ST 0 P if H The Logic of Science a Science a set of logical systematic documented methods for investigating nature and natural processes the knowledge produced by these investigations Social Science The use of scientific methods to investigate individuals societies and social processes the knowledge produced by these investigations How science and specifically social science overcomes errors in everyday reasoning 39 Systematic sampling representative observation overcomes overgeneralization by systematic procedure for selecting objects of study that guarantee a measurable representativeness of the groupindividuals about which will be generalize Speci c criten39afor causality overcomes illogical reasoning by using explicit criteria to identify causes and determining presenceabsence of said criteria Systematic observation and measurement overcomes selective and inaccurate observation through systematic sampling and measuring iv Use of empirical observable data overcomes resistance to change by being based in objective reality Peer evaluation reduced egobased commitments excessive devotion to tradition and unquestioning respect for authority ST 0 lt HI Motives for social research a Policy guidance or program management eg government decisions or planning b Academic concerns eg testing social theory c Personal interest IV Four types of social research a Descriptive Research research that defines and describes social phenomena eg National Geographic Survey 2000 that described Internet users around the world and identified differences between countries b Exploratory Research investigation of social phenomena without expectations eg electronic diabetes newsgroups were found to also be support and information networks a place where information could be assimilated to inform choices c Explanatory Research research that identifies causes and effects of social phenomena eg research that suggests that Internet use hurts or helps other forms of social interaction d Evaluation research that determines the effects of a social program or other type of intervention e g in the Toronto Ont Suburb that was wired with the Internet universal Internet access increased relations between residents V Quantitative and Qualitative Orientation a uantitative methods data collection methods such as surveys and experiments that record variation in social life in terms of categories that vary in amount i Data are numbers OR attributes that can be ordered in terms of magnitude ii Most often used for explanation description and evaluation b Qualitative methods data collection methods such as participant observation intensive interviewing and focus groups that are designed to capture social life as participants experience it rather than in categories predetermined by the researcher i Data are mostly written or spoken words or observations ii Data do not have a direct numerical interpretation i Exploration is the most often motive for using qualitative methods c Triangulation The use of multiple methods to study one research question VI Goals of Social Research a Validity When our statements or conclusions about empirical reality are correct 39 Measurement validity exists when a measure measures what we think it measures Generalizability exists when a conclusion holds true for the population group setting or event that we say it does given the conditions that we specify 1 Sample generalizability exists when a conclusion based on a subset of a larger population holds true for the entire population 2 Crosspopulation external validity exists when findings about one group population or setting holds true for another group population or sett iii Causal internal validity exists when a conclusion that A leads to or results in B is correct b Authenticity When the understanding of a social process or social setting is one that re ects fairly the various perspectives of participants in that setting ie a resolution of whether an objective social reality exists independent of actors interpretations VII Social research as a collective endeavor a Other researchers may find different results b Critical evaluation of previous work should guide current research c The accumulation of evidence is the goal of science d Social phenomena are complex one study will not necessarily capture everything e New studies should focus on weak points or controversial conclusions Chapter 2 The Process and Problems of Social Research Chapter Overview Chapter 2 begins with an outline of how to create and refine a social research question Next it provides instruction on how to conduct a review of the literature with specific instructions regarding social science journals and using the Internet in searches This chapter also addresses the relationship between social theory and social research methods The relationship between theory and data collection is described as a process shown in Exhibit 26 as the research circle Deductive research begins with a theory that generates a hypothesis data is collected to test this hypothesis and empirical generalizations are drawn from the data to connect back to the theory In this section the concepts of independent variables dependent variables and direction of association are explained Inductive research is explained as beginning with data collection from which empirical generalizations are induced and connected to extant social theory Inductive logic may also be used to make sense of anomalous and serendipitous findings in deductive research Chapter 2 includes a lengthy discussion of the consideration of ethics in research design presenting the ASA guidelines for research involving human subjects This chapter closes with a brief section on the constituent parts of a social research proposal Lecture Outline VIII The Social Research Question is a question about the social world that you seek to answer through the collection and analysis of first hand verifiable empirical data a Three stages of formulating a research question i Identifying social research questions A researcher s personal experiences 2 Social research literature 3 Social theory 4 Request from an agency Refining social research questions develop a list of possible questions as you go along narrow it to the most interesting and workable candidates repeat as necessa Evaluating social research questions I Feasibility given resources can be finished in time and within budget 2 Social importance makes a differences in the social world 3 Scientific relevance resolves contradictions in or advances social theory IX Social Research Foundations a Searching the social science literatur i Because of the more rigorous review process published social science journals must be consulted 1 Compile names of authors keywords and journal names iii Conduct searches through catalogs indexes widening or limiting search parameters as necessar iv Check the results by scanning abstracts for relevance v Use bibliographies to expand search vi Read articles for content and to get ideas about theory and methods b Searching the Web 39 Direct addressing if you know the URL Uniform Resource Locator you can go directly to a site ii Browsing subject directories such as Yahoo or through collections of websites maintained as lists in other websites iii Searching using a program that indexes web pages to locate sites related to keywords such as Google Alta Vista I Caveats about search engines Search engines vary in size Different search engines will produce different results Search engines identify and index sites differently Search engines have unique syntax and ranking systems i Phrases often have to be in quotation marks ii Rankings are usually by relevance No search engine covers more than one third of the pages on the Web 9 57 an f It takes most search engines six months or more to locate new pages 2 Hints for searching on the Web mph 99 ST c Reviewing Research 1 ii Assess credibility Clarify your goals lest you get too many results Quality is NOT guaranteed Anticipate change and impermanenc Different search engines produce different results Be concerned about generalizability based on Web availability Use Appendix H to help evaluate websites Avoid following irrelevant paths surfing Cite sources Record URL name of information provider and date from which you obtain information of each research article see Appendix B ow was the report reviewed prior to its publication 2 What is the author s reputation and previous work 3 Who fun 4 Example a OrgPST 239 Compare articles relevant aspects 0 integrated review ded and sponsored the research Pate and Hamilton 1992 Research question What is the deterrent effect of arrest on domestic violence cases with an additional focus on the role of informal social control Type of research explanatory Theory deterrence rational choice Replication of Sherman and Berk 1984 Research design i Experiment ii Clearly defined concepts iii Hypothesis clearly stated iv Sampling technique identified v Data tested with original hypothesis Findings and conclusion i Informal social processes are important in terms of employment but not marriage ii Discussion of addition to literature and weaknesses of study to assess implications of the entire set of articles for the f your research question and procedures and write an for your own article or proposal Rank reports in order of credibility 2 Identify overarching framework that links all articles together How does each individual article contribute to that framework 4 Do not include irrelevant articles Social Research Strategies connecting theory and empirical data a Theory is a logically interrelated set of propositions about empirical reality39 ii iii iv lt Makes sense of m Predicts behavior any interrelated phenomena or attitudes likely to occur given certain conditions Helps identify what to look for in a study Connects implications of finding to other research Examples deterr interactionism ence theory labeling theory rational choice theory symbolic b Theory connects to empirical data in two manners see Exhibit 26 for a graphic depiction of the research circle that connects theory and data 39 Deduclive research a specific expectation is deduced from a general theoretical 1 premise and then tested with data that has been collected for this purpose i Hypothesis a tentative statement about empirical reality involving a relationship between two or more variab es Variables a characteristic or property that can vary take on different values or attributes a Dependent variable a variable that is hypothesized to vary depending on or under the in uence of another variable b Independent variable a variable that is hypothesized to cause or lead to variation in another variable Direction ofAssociaiion between variables in a hypothesis a Positive as independent variable goes up so does the dependent variable OR as the independent variable goes down so does the dependent variable b Negative or Inverse as independent variable goes up dependent variable goes down OR as independent variable goes down dependent variable goes up c If variables are categorical there is NO direction of association 4 An example of deductive research Sherman and Berk 1984 a Test a hypothesis generated by deterrence theory i Punishment reduces recidivism ii Arrest for spousal abuse reduces risk of repeat offenses b Set up experiment c Of those arrested 13 repeated while 26 without arrest did empirical generalization d Move back to support for deterrence theory e Replications with improvements and in more locations 5 Deductive research in explanatory and evaluative research a State hypothesis clearly b Design research to test hypothesis c Use data to test hypothesis empirical generalizations are patterns in the data that connect back to theory d Replications repetitions of research design ii Inductive research begins with specific data that are then used to develop induce a theory to account for patterns in the data Can be intentional as in exploratory research 2 Inductive logic can be used when analyzing empirical generalizations discovered while hypothesis testing a Anomalous patterns don t fit the proposed theor b Serendipitous patterns are new unexpected patterns Example Starting with an empirical generalization people who have a stake in conformity such as a job or marriage seem to be less likely to recidivate and develop a theory to account for it The adequacy of an explanation formulated after the fact is necessarily less certain than an explanation presented prior to the collection of data Inductive explanations are more trustworthy if subsequently tested with deductive research Exploratory research is often inductive a More authentic less generalizable b Identifies themes or patterns in research iii Descriptive research does not connect with theory but is an interim between data collection and the generalizations based on them N E E 4 V39 0 1 Starts with data and proceeds only to the state of making empirical generalizations 2 Much research for governments and organizations in primarily descriptive 3 Descriptive research can stimulate more ambitious inductive and deductive research XI Social research ethics a Scientists must be honest and reveal their methods i Honesty openness and ethical behavior with peers and subjects ii Know the difference between honest errors and frau b Scientists much consider uses to which findings will be put 39 Critical traditions question ability of researcher to set aside values in research and use research to advance specific goals ii Traditional social scientists think personal values should be left out iii When to publish and how to publish iv Who decides timing and format of publication or if publication will occur at all c Concern for human subjects i Federal mandates required that Internal review boards IRBs review research proposals for ethical considerations Professional associations such as the ASA may provide ethical guidelines iii ASA code of ethics No harm should come to subjects Voluntary participationinformed consent Researchers should fully disclose their identity Anonymity and confidentiality must be maintained for individual research participants unless it is voluntarily and explicitly waived 5 The benefits of a research project should outweigh any foreseeable risks iv Informed consent requires 1 Respondents are competent to give consent a Children are not competent to give informed consent but they must assent to participate in research to which their legal guardians have consented 2 Respondents consent voluntarily a Consent is suspect if researcher has some power over potential respondents such as a professor conducting research on students 3 Respondents are fully informed of research 4 Respondents understand what they ve been told v In certain circumstances deception may be acceptable That is researchers may withhold some inform ation l Consent forms may bias research by altering participants behavior 2 In cases where subject deception may be defensible debrie ng is mandatory the researcher explains to the subject what happened during the research and allows respondent to ask questions vi Confidentiality A confidentiality statement should be included in the informed consent agreement about how each subject s privacy will be protected 2 Standards of confidentiality do not apply to observation in public places and for information made available in public records HeP Nf XII Social research proposals PartI a Every proposal should have at least these six sections i An introductory statement of the research problem ii A literature review iii A methodological plan iv A bu v An ethics statement vi A statement of limitations b If a proposal is under competitive review include these 39 A compelling rationale for un ing ii Results from a pilot study Chapter 3 Theories and Philosophies of Social Research Chapter Overview Chapter 2 begins with an outline of how to create and refine a social research question Next it provides instruction on how to conduct a review of the literature with specific instructions regarding social science journals and using the Internet in searches This chapter also addresses the relationship between social theory and social research methods The relationship between theory and data collection is described as a process shown in Exhibit 26 as the research circle Deductive research begins with a theory that generates a hypothesis data is collected to test this hypothesis and empirical generalizations are drawn from the data to connect back to the theory In this section the concepts of independent variables dependent variables and direction of association are explained Inductive research is explained as beginning with data collection from which empirical generalizations are induced and connected to extant social theory Inductive logic may also be used to make sense of anomalous and serendipitous findings in deductive research Chapter 2 includes a lengthy discussion of the consideration of ethics in research design presenting the ASA guidelines for research involving human subjects This chapter closes with a brief section on the constituent parts of a social research proposal Lecture Outline XI The Origins of Social Science a Sociology was first conceived as a distinct social science during the Industrial Revolution in Europe b Shift from community in which people knew each other intimately to society in which people know one another through specialized social roles i Gemeinschaft societies 1 based on community 2 homogenous 3 social relations based on kinship 4 often based on a common religion ii Gesellscha societies 1 based on association 2 individualistic 3 competitive 4 have a developed division of labor XIV Theoretical perspectives in Social Science a unctiona ism i Durkheim Is there anything that can replace the power of traditional social bonds in a modern societ 7 1 Mechanical solidarity based on bonds of likeness traditional societies 2 Organic solidarity while the division of labor weakens bonds based on likeness it strengthens bonds based on interdependence 3 Organic solidarity serves the function of bonding people together in a society with a developed division of labor ii Functionialism a social theory that 1 explains social patterns in terms of their consequences for society as a 2 emphasizes interdependence of social institutions 3 r 39 comm 39 39 39 interestin 39 39 39 of social order iii Key concepts of Durkheim 1 division of labor 2 solidarity 3 strength of social bonds 4 propensity to commit suicide suicide rates 5 societal functions iv Contemporary example Does the amount of interaction between children and adults in a neighborhood differ for children of different racial groups Sampson Morenoff and Earls 1999 1 Exhibit 32 shows that white kids have a spatial advantage in their neighborhoods while black kids have spatial vulnerability in theirs b Con ict Theory i Marx Social classes were the key groupings in society and con ict between them is not only the norm but the engine of social chan e materialist assumption social change can be explained in terms of the material conditions in society especially technology the economic system is the primary structure of society while other structures help acclimate people to economic conditions ie the economic system is the independent variable that shapes ideas and other social processes 3 economic class is the primary source of social stratification ii Con ict Theory a social theory that 1 identifies con ict between social groups as the primary force in society 2 assumes that understanding the bases and consequences of con ict is the key to understanding social processes iii Weber ideas namely the Protestant Ethic were the independent variable that shaped the modern economic system 1 Social status political power and economic class are all sources of stratification not just class iv Contemporary example Bradshaw and Wallace 1996 multinational corporations have greater power than many less developed countries based on a comparison of corporate sales to gross domestic products c Rational Choice Theory i Adam Smith rational individual action results in a larger social good ii Rational Choice Theory explains social processes in terms of rational costbenefit analyses that shape individual behavior iii Contemporary example Gottfredson and Gottfredson 1988 the likelihood of reporting theft to police depends on whether the victim had insurance d Symbolic Tnteractionism i Cooley re ective consciousness is social consciousness ii Mead the self re ects the general systematic pattern of social or group behavior in which it and the others are involve iii Symbolic interactionism a theory that focuses on the symbolic nature of social interaction 2 focuses on how interaction conveys meaning and promotes socialization iv Contemporary example Skoll 1992 an unstated goal of treatment programs is to domesticate female residents male and female sexuality is treated and constructed very differently N XV The relationship between Research and Theory a General theoretical perspectives provide alternative frames of reference i Encourages different questions about the social world ii No single theoretical approach has rallied most social scientists ST Social research should 39 Seek to extend challenge or specify a single theory ii Test implications of one theory against others iii Combine aspects of different theories c Middle Range Theories a term coined by Merton that refers to type of theories that are the general focus of primary research concerns d Kuhn s The Structure of Scientific Revolutions Normal science gradual incremental work that adds to the body of scientific knowledge operating within and not challenging scientific paradigms Scienti c paradigm a set of beliefs that guide scientific work in an area includes unquestioned prepositions accepted theories and exemplary research findings 2 changes only when a large body of contradictory evidence accumulates and an alternative perspective appears Paradigm shift a change in paradigms brought about by a body of contradictory evidence Scienti c revolution an abrupt transition from one theoretical paradigm to another that causes scientists to begin to accept new ways of thinking No scientific revolution has occurred in the social sciences 1 There is no single paradigm 2 There is no accumulation of evidence of one paradigm over another 3 Theoretical paradigms are alternatives E lt XVI Social Research Philosophies ositivist research philosophy A belief that there is a reality that is external to us that we can understand through empirical research i Positivists believe that there is an objective reality that exists apart from the perceptions of those who observe it39 the goal of science is to better understand this reality Positivism The believe shared by most scientists that there is a reality that exists quite apart from our own perceptions of it although our knowledge of this reality may never be com lete Postpositivism The belief that there is an empirical reality but that our understanding of it is limited by its complexity and by the biases and other limitations of researchers 2 Tntersubj ective agreement an agreement by different observers on what is happening in the natural or social world iii Positivist Research Guidelines 1 Test ideas against empirical reality without becoming too personally invested in a particular outcome Plan and carry out all investigations systematically Document all procedures and disclose them publicly Clarify assumptions Specify the meaning of all terms Maintain a skeptical stance toward all current knowledge Replicate research and build social theory 8 Search for regularities or patterns iv Advancing knowledge is a positivist research goal 1 The goal of traditional positivist social science is to advance scientific knowledge achieved by publication in academic journals or presentation at academic conferences 2 Value considerations are beyond the scope of science and should be held in abeyance during social research b Interpretivist research philosophy since all empirical data come through the sense and are interpreted by the mind we can never be sure we have understood reality properly SQEJ HeP N Interpretivism the belief that reality is socially constructed and that the goal of social scientists is to understand what meanings are given to that reality Verstehen a term coined by Max Weber to explain understanding of meaning in social circumstances iii Constructivist paradigm A perspective that emphasizes how different stakeholders in social settings construct their beliefs iv Hermeneutic circle constructivist research technique in which a variety of individuals are elicited challenged and exposed to new information and new more sophisticated ways of interpretation until some level of consensus is reached 1 The researcher conducts an openended interview with Respondent 1 to learn hisher construction and then asks himher to name another respondent who feels differently 2 The second respondent is then interviewed in the same way but is also asked to comment on the themes raised by the previous respondent 3 The process continues until all major perspectives are represented and then may be repeated again with the same respondents 4 Case report the final product characterized by thick description of constructed realities underlying motives feelings or rationale that allow reader to vicariously experience it v Interpretivist research guidelines 1 Identify stakeholders and solicit their claims concerns and issues 2 Introduce the claims concerns and issues of each stakeholder group to the other stakeholder groups and ask for their reaction 3 Focus further information collection on claims concerns and issues about which there is a disagreement among stakeholder groups 4 Negotiate with the stakeholder groups about the information collected and attempt to reach consensus on the issues about which there is disagreement vi Creating change is the interpretivist research goal 1 Social science as a public philosophy Bellah 2 Participatory action research William Foote Whyte the researcher involves some members of the setting studies as active participants c Sociology of Knowledge an integrated philosophy that studies the process by which people make themselves as they construct society 39 Individuals internalize social order through socialization Research approaches and interpretations are shaped by social background of researcher iii Emotions can influence us before reason because of biology what something reminds us of may be far more important than what it is iv Subjective aspects of human experience cannot be ignored or expunged entirely from the data collection process Fir XVII Extending Social Science Investigations a Multiple methods and multiple contexts b Attention to social context i Contextual effect When relationships among variables differ across geographical units like counties or across other con ex ii Example Dannefer and Shutt 1982 and Schutt and Dannefer 1988 found that harshness in sentencing juvenile offenders varied in counties with different levels of racial polarization and underclass poverty c The Natural World The natural environment in which people live and the biology of the human body interact with social processes in many ways eg Udry 1988 found that a combination of hormones and social context predict sexual behavior among teens Chapter 4 Conceptualization and Measurement Chapter Overview Chapter 4 discusses the process of conceptualizing and operationalizing concepts After defining these terms the chapter turns to different methods for measuring social reality The following methods are considered in detail use of existing data constructing questions single questions indexes and scales direct observation and unobtrusive measures These sections are followed by a brief discussion on the superiority of using mixed measurement methods Next the chapter addresses level of measurement nominal ordinal interval and ratio and dichotomies Chapter 4 next addresses measurement validity in detail distinguishing face validity content validity criterion validity and construct validity as well as different means for assessing reliability testretest interitem alternative forms and interobserver reliability The chapter concludes on guidelines for improving reliability and validity and a brief section on measurement in qualitative research Lecture Outline XVIII It is only when we conclude that a study used valid measures of its key concepts that we can have some hope that its conclusions are valid a Concept a mental image that summarizes a set of similar observations feelings or ideas in order to be useful in research it must be defined because not all people share a particular definition b Conceptualization is the process of specifying what is meant by a term 39 In deductive research conceptualization helps to translate portions of abstract theory into testable hypotheses involving specific variables In inductive research conceptualization is an important part of the process used to make sense of related observations Review definitions from social theory and previous research iv Distinguish subconcepts dimensions v How does conceptualization fit with theoretical framework vi What assumptions underlie the theoretical framework 39 The definition of any one concept rests on shared understandings of other terms used in the definition 2quot S XIX Operationalization The process of specifying the operations that will indicate the value of cases on a variable identify specific observations that will indicate the concept in empirical reality a Variable some aspect of a concept that varies from which we select even more concrete indicators b Constant a concept that does not vary Meausrement The process of linking abstract concepts to empirical indicants Operation Aprocedure for identifying or indicating the value of cases on a variable i Measurement can be based on diverse methods of data collection Variables and operations should be consistent with the research question iii Time and resource limitations must be taken into account when selecting variables and devising operations 90 XX Using Available Data a How appropriate are data for concepts of interest b How are indicators constructed c How accurate are data d Are original questions sufficiently close to the measures needed for the new research question e Maj or data sources i nterUniversity Consortium for Political and Social Reserach ii General Social Survey GSS l Administered by the NORC at the University of Chicago Sample of 1500 Americans Annual until 1994 biennally since Core questions do not vary but each year additional questions are added that change from year to year 39 XXI f If existing data is inappropriate researchers may construct their own surveys Constructing questions Asking questions is the most common and most versatile operation for measuring social variables a Single questions i Closedended or xed choice questions offer respondents explicit responses from which to choose Most large surveys Easy to process with computers Easy to analyze with statistics Reduces ambiguit Makes it easier for respondent to answer Can obscure what people really think if choices do not match the range of possible responses 7 Responses must be Exhaustive all respondents can find an appropriate response except in Check all that apply questions b Mutually exclusive all respondents will find only one appropriate responses 8 To make response choices exhaustive and mutually exclusive include at least one option with room for ambiguity eg other please explain ii Openended questions lack explicit response choices and allow respondents to fill in their own answers 1 Small number of people 2 Preferable when full range of responses cannot be anticipated a Questions not used in previous surveys b Questions asked of new groups 3 Lessens confusion about meaning of responses involving complex questions 4 Questions should be reviewed carefully for clarity before they are used Question sets indexes and scales reduce idiosyncratic variation in which individual responses to single questions vary because of reactions to particular words or ideas in the question 1 Index a composite measure based on the sum or average of the responses of several questions that measure the same concept a If a common concept is being measured people s responses should be correlated Reliability measures statistics that help determine whether measures are consistent c Many preexisting indexes already exist such as the CESD these have been proven reliable and promote comparability Cautions when using indexes i Our presupposition that each component question is measuring the same concept may be mistaken ii Combining responses to specific questions can obscure important differences in meaning among the questions iii The questions may cluster together in a subset such as in a multidimensional index in which several aspects dimensions of a concept are measured e Calculating an index i Arithmatic average or sum of responses with all questions counting e uall ii Scale Assigning different weights to different questions before summing or averaging 9939 i S7 P l Weights should ideally be determined by empirical testing 2 Weights can be determined by logic Guttman scaling weighting so that heavier weights predict responses on lighter weights 9 XXII Direct Observations are another way to operationalize variables a Good as a supplement to interviews and survey ata b Measurement of choice for research in natural settings c Avoids problems of poor recall and selfserving distortions d Drawbac s i Observers can t observe everything ii Observations are filtered by observer iii Presence of observer may change behavior XXIII Unobtmsive Measures collecting data about individuals or groups without their a Types of unobtrusive measures i Physical trace evidence 1 When behavior cannot be directly observed because it is hidden or in the past 2 When behavior has not been recorded in a source of available data ii Archives available data including media and documents iii Simple observation iv Contrived observation using hidden recording devices or manipulation to elicit a response XXIV Combining Measurement Options a Questioning can be a poor approach for measuring behaviors that are i Highly socially desireable ii Socially stigmatized iii Illegal b Triangulation is the use of two or more different measures of the same variable i Strengthens measurement ii Especially strengthens measurement when based on different methods of data collection XXV Levels ofMeasurement refer to the mathematical precision with which the values of a variable are expressed a Level of measurement is not inherent in the variable but must be chosen by the researchers i Ideally measurement should be taken at the most precise highest level of measurement ii Respondents may be unwilling to respond to some specifics eg income so lower levels ofmeasurement can be use b Nominal level of measurement qualitative with no mathematical interpretation i Categorical or qualitative level of measurement ii Varies in kind or quality but not by amount iii Nominal categories must be exhaustive and mutually exclusive each case corresponds to one and only one attribute l exhaustive every case can be classified 2 mutually exclusive every case can have only one attribute c Ordinal level of measurement specifies only the order of cases 39 Greater thanless than operations ii Indicates relative position of cases iii Ordinal categories must be exhaustive and mutually exclusive iv Assumes that respondents have similar interpretations of the terms used to designate the ordered responses v Providing explicit anchor points for respondents can improve the comparability of responses to ordinal ranking questions d Interval level of measurement numbers represent fixed measurement units but have no absolute zero point 1 Addition and subtraction are possible ii Ratios are meaning ess iii Must be exhaustive and mutually exclusive iv There are few true interval level measures in the social sciences v Indexes often create interval levels of measurement if the range of the index doesn t include a zero e Ratio level of measurement numbers represent fixed measurement units where zero means absolutely no amount of whatever the variable indicates an absolute zero point 39 Multiplication and division are possible so ratios are meaningful ii Highest level of mathem atical precision iii Must be exhaustive and mutually exclusive iv For most statistical analysis interval and ration levels of measurement can be treated as equivalent f Dichotomies variables having only two values attributes 1 Can be treated as ordinal variables presence ofabsence of XXVI Evaluating Measures a Measurement Validity the extent to which measures indicate what they are intended to measure A valid measure of a concept is one that is closely related to other apparently valid measures of the concept and to the known or supposed correlates of the concept but that is not related to measures of unrelated concepts irrespective of the methods used for the other different measures Face Validity confidence gained from careful inspection of a concept to see if it s appropriate on its face if a measure more obviously pertains to the meaning of the concept being measured more than to other concepts Content Validity establishes that the measure covers the full range of the concept s meaning ie covers all dimensions of a concept Criterion Validity establishes that the results from one measure match those obtained with a more direct or already validated measure of the same phenomenon the criterion 1 Concurrent validity exists when a measure yields scores that are closely related to scores on a criterion measured at the same time 2 Predictive validity exits when a measure is validated by predicting scores on a criterion measured in the future Construct Validity established by showing that a measure is related to a variety of other measures as specified in a theory used when no clear criterion exists for validation purposes 1 Convergent validity achieved when one measure of a concept is associated with different types of measures in the same concept this relies on the same type of logic as measurement triangulation Discriminant validity scores on the measure to be validated are compared to scores on measures of different but related concepts and discriminant validity is achieved if the measure to be validated is NOT strongly associated with the measures of different concepts 19 b Reliability A measurement procedure yields consistent scores when the phenomenon being measured is not changing or that the measured scores change in direct correspondence to actual changes in the phenomenon ii 1 Reliability is a prerequisite for measurement validity Ways to assess reliability l TestRetest Reliability testing unchanging phenomenon at two different times the degree to which the two measurements are related to one another a When ratings are by an observer rather than the subjects themselves this is called intraabserver reliability or intrarater reliability b Answers about the past are less reliable when they are very specific because the questions may exceed the subjects apacity to remember accurately 2 Interitem Reliability Internal consistency the association of answers to a set of questions designed to measure the same concept a The stronger the association among individual items and the more items included the higher the reliability of an index b Cranbach s alpha is a statistic commonly used to measure interitem reliability 3 Alternate Farms Reliability comparison of subjects answers to slightly different versions of survey questions a SplithalvesReliability sample is split in two and receives slightly different forms of the questions the correspondence between subsamples assesses splithalves reliability 4 InterobserverReliability correspondence between measures made by different observers c lmproving Reliability and Measurement Validity 39 Some selfreporting may be reliable but invalid Don t choose the first measure you think of Conduct a pretest with a small sample iv Provide careful training of personnel v Use previously tested measures Use multiple measures vii When devising new measures 1 Engage potential respondents in group discussions about the questions to be included in the survey 2 Conduct cognitive interviews 3 Audiotape test interviews during pretest phase of a survey 5 XXVH Measurement in qualitative research a Concepts and variables emerge inductively b Conceptualization operationalization and validation are ongoing interrelated and iterative Chapter 5 Sampling Chapter Overview This chapter explains how and why samples are drawn in social research After first defining key terms such as population sample elements the chapter explains how generalizability is established through representative sampling Next this chapter introduces sampling methods First probability sampling methods are identified and detailed simple random systematic random cluster and stratified random sampling The chapter next explains nonprobability methods availability quota purposive and snowball sampling These sections are followed by a review of the implications of sampling on assessing research and a brief statement of how qualitative research approaches the idea of sampling The chapter presents a nonmathematical description of sampling distributions with an explanation of how sampling error is related to confidence intervals The chapter closes on a brief discussion of how a researcher determines an appropriate sample size Lecture Outline XXVIH Sampling methods are the procedures that primarily determine the generalizablity of research findings a Population The entire set of individuals or other entities to which study findings are to be generalize i The aggregation of elements that we actually sample from NOT some larger aggregation that we wish we could have studied ii In cases where population is not bounded by geography or membership a clear conceptual definition must be specified b Sample A subset of a population used to study the population as a whole c Elements The individual members of the population whose characteristics are measured and therefore constitute the sample d Sampling Frame A list of all elements or other units containing the elements in a population e Enumeration Units Units that contain one or more elements and that are listed in a sampling frame f Sampling Units Units listed at each stage of a multistage sampling design primary secondary etc XXIX Generalizability a Sample generalizability Ts sample generalizable to population Cr pp39d ng 139 quotquot Tssam le quot tootherl 1 quot quot i Cannot be determined without further study ii Target population a set of elements larger than or different from the population that was sampled and to which researchers would like to generalize any study findings c Sampling error any difference between the characteristics of a sample and the characteristics of a population from which it was i The larger the sampling error the less representative the sample d Assessing the diversity of the population i Sampling is unnecessary if the elements in a population are identical ii Representative sample a sample that looks like the population from which it was selected in all respects that are potentially relevant to the study ie the distribution of characteristics is the same as in the total population iii Unrepresentative sample a sample in which some characteristics are over represented or underrepresented relative to the total population iv Census a study of the entire population 1 usually too expensive and time consuming to be practical 2 assumes a perfect response rate which is difficult at best XXX Sampling methods a Probability Sampling Methods sampling methods that allow us to know in advance how likely it is that any element of a population will be selected for the sample i Probability ofselection the likelihood that an element will be selected from the population for inclusion in the sam le a census the probability of selection is 10 2 If half of all elements will be selected the probability of selection is 05 3 As the size of the sample as a proportion to the population decreases so does the probability of selection ii Relies on random or chance selection procedures in which cases are selected only on the basis of chance 1 Not the same as haphazard or availability convenience sampling see below 2 Causes for concern in random sampling An incomplete sampling frame does not allow a true random sample H Nonresponse rates greater than 30 can not usually be generalized to the population because nonrespondents are likely to differ systematically from those that agree to participate b Probability Sampling Methods i General characteristics 9 5quot 0 Probability of selection is known Probability of selection is not zero No systemch bias nothing but chance determines which elements are included in the sample More generalizable All samples will have some sampling error some deviation from the characteristics in the population As size of sample increases error goes decreases lT As homogeneity of population increases error decreases The proportion of the population that the sample represents does NOT affect sample representativeness l The larger the sample the more confidence in sample representativeness raw size is more important than proportion of population H The more homogenous the population the more confidence in sample representativeness H1 The fraction of a population that a sample contains does NOT affect sample representativeness unless sample fraction is very large ii Simple Random Sam ling l 2 3 4 5 7 Identifies cases strictly on the basis of chance Methods of simple random selection a Random number table see Appendix E a list of random numbers used to select elements in a population see Exercise 1 in Doing Research for stepbystep method b Lottery procedure each element is written on a scrap of paper mixed up and selected as in a lotter c Computers can generate a random selection of elements d Randomdigit dialing a machine dials randomly selected numbers within a phone prefix Probability of selection is the same for all elements samplepopulation Also known as Equal Probability Selection Method EPSEAI Replacement sample each element is returned to the sampling frame after it is selected so that it may be sampled again a Simple random may be done with or without replacement sampling b Random sampling with replacement sampling is rarely used iii SystematicRandom Sam ll 2 3 4 5 I7 quotg F1rst element is selected randomly and then every nth element is selected Convenient when populations are arranged sequentially Periodicity An error that results when a sequence varies in some periodic manner such that systematic random sampling doesn t work because it selects a specific type ofelement Sampling interval the number of cases from one sampled case to another Steps a Population is divided by number of cases required by sample yielding the sampling interval b A number less than or equal to the sampling interval is selected randomly to identify the first case to be sampled Every nth case is selected where n is the sampling interval 0 iv Strati edRandom Sampling 1 All elements in the sampling frame are distinguished according to their value on some relevant characteristics 2 That characteristic forms the sampling strata 3 The size of each stratum in the population must be known 4 Elements are randomly selected from within the strata 5 Each element may belong to one and only one strata 6 Proportionate strati ed sampling Each sampling stratum represents exactly its proportion in the population 7 Disproportionate strati ed sampling The proportion of sampling strata are intentionally varied from what it is in the population I Probability of selection is known but differs by strata lT Used to ensure that cases from smaller strata are included in sufficient number to use statistics and make comparisons lTl Facilitates comparisons between strata 8 Weighting can be used in stratified samples to account for different stratum size a In disproportionate stratified samples as a means for making generalizations about the population as a whole b Can reduce the lack of representativeness of a sample due to nonresponses if the proportions in the total population are known v Cluster sampling 1 Useful when sampling frame is not available 2 Cluster a naturally occurring mixed aggregate of elements in a population with each element occurring in one and only one cluster 3 Procedure 1 Stage 1 Draw a random sample of clusters lT Stage 2 Elements within randomly selected clusters are randomly selected lTl Multistage clustering sample is common a Clusters at first stage are primary sampling units b Clusters at second stage are secondary sampling units etc 4 The more clusters selected with the fewest individuals in each the more representative the sample 5 The more internally hom ogenous the clusters the fewer elements needed per cluster 6 Sampling error is greater than in simple random sampling 0 Non Probability Sampling Methods sampling methods that do not let us know in advance the likelihood of selecting each element i General characteristics Often used in qualitative research 2 Used in quantitative studies when probability selection is not possible 3 Does not yield representative samples lowers ability to generalize findings Useful in preliminary exploratory studies ii Availability sampling Also known as convenience sampling accidental sampling and haphazard sampling 2 Selection of elements is done by what is available andor convenient to researcher 3 Useful when in a new setting and in exploratory studies 4 Not rigorous or representative 5 Much popular research is based on convenience sampling eg reader olls iii Quota sampling 1 Quotas preset number of elements based on characteristics in a population to ensure that the sample represents those characteristics in proportion to their prevalence in the population 2 Sample may be representative on characteristics specified by the quota but it is probably not representative in any other wa 3 Relevant characteristics must be known of entire population in order to set quotas 4 If you can t draw a random sample it may be better to draw a quota sample than to use no quotas iv Purposive samplin Each element is selected for a purpose usually because of unique position of the sample elements 2 Selecting informants 1 Knowledge of areaarenasituation under study H Willingness to talk HI Representativeness of range of points of view 3 Continue to select interviewees unti l Completeness overall sense of meaning of a concept theme or process has been achieve H Saturation confidence that you are learning little that is new from subsequent interviews v Snowball sampling 1 Identify one member of a population and speak to himher and then ask that person to identify others in the population and speak to them then ask them to identify others and so on 2 Useful for hard to reach and hard to identify populations for which there is no sampling frame but the members of which are somewhat interconnected 3 Useful for sociometric studies that chart relationships between members 4 Useful for exploring populations of interest before developing a formal sampling plan 5 Useful for developing a census of informal leaders 6 No confidence about how representative the sample is 7 Initial contacts may shape entire sample 8 Respondentdriven sampling respondents are given an incentive to recruit peers into the sample XXXI Lessons About Sample Quality a Sample quality cannot be evaluated if population is not specified b Sample quality cannot be evaluated if selection technique is not specified c Sample quality is determined by sample actually obtained not by the selection method itself d Crosspopulation generalizations are conjecture no matter how strong the sample generalization e A sample that allows for comparisons involving theoretically important variables is better than one that does not allow for such comparisons XXXH Generalizability in Qualitative Research Lack of generalizability may be the goal or may be defined as a strength of the research b Multiple sites can be used to increase generalizability c Studying the typical can improve generalizability XXXIH Sample Distribution a hypothetical distribution of a statistic across all the random samples that could be drawn from a population used to estimate sampling error 37 ST 0 P W XXXIV Det a b F Lecture Outline XXX V Tim a Infereniial statistics a mathematical tool for estimating how likely it is that a statistical result based on data from a random sample is representative of the population from whic the sample is assumed to have been selected The sample statistic computed from the sample data that occurs most frequently is identical to the true value of that statistic in the population ie it is the population parameter Sample distributions are normal symmetrical bellshaped if produced by random sampling error centered on the population parameter In a normal distribution a predictable proportion of cases falls within certain ranges from the population parameter known as con dence inlerva s By convention statisticians use only 95 99 and 999 as confidence intervals Sample distributions are more compact when based on larger samples ermining a Sample Size The less sampling error desired the larger the sample size must be Samples with more homogenous populations can be smaller than samples of more diverse populations i Stratified random samples can be smaller than simple random samples If the only analysis planned for a survey sample is to describe the population in terms of a few variables a smaller sample is required than if a more complex analysis involving subgroups is planned i If much of the analysis will focus on estimating the characteristics of subgroups within a sample the size of the subgroups is more important than the size of the total sam le When detecting very strong relationships among variables a smaller sample is required than if testing weaker relationships Statistical power analysis can provide more precise estimates of sample size but require advanced statistical training Current practices as guidelines 39 National surveys 10002500 ii Local or regional a few hundred Chapter 6 Causation and Research Design Chapter Overview Chapter 6 establishes the difference between cross sectional and longitudinal designs repeated cross sectional fixed sample panel and eventbased for establishing time order Following a discussion of unit of analysis ecological and reductionist fallacies are explained The difference between nomothetic and idiographic causal explanations are explained Next the chapter covers the concept of causality in socia order nonspuriousness mechanism and context The discussion centers around how each of these criteria are established when using experimental and nonexperimental research designs 1 research Five criteria for identifying causal effects are identified association time e Order and Research Design designing to determine what happened first Crosssectional research design al data are collected at one point in time 39 High risk of detecting a spurious relationship a relationship that appears to be connected but is not Time order can be established using cross sectional data 1 Independent variable is fixed at some point prior to variation in the dependent variable such as gender We believe that respondents can give reliable reports about the past retrospective data Measures are based on records that contain information on cases in earlier periods ii 4 We know that cases were equivalent in the dependent variable prior to the treatment b Longitudinal research design data are collected at two or more points in time i Repealed CrossSectional Design data are collected from different samples within the same population also known as Trend studies 1 Goal To determine whether a population has changed over time 2 Process a Sample is drawn and data collected at time 1 b Time passes some individuals leave and others join the population c At time 2 another sample is drawn and data collected ii Fixed sample panel design data are collected from the same individuals the panel also known as a panel design Goal to know whether individuals in a population have changed over time 2 Because a panel design follows the same individuals it is better than a repeated cross sectional design for testing causal hypotheses 3 Process a A sample panel is selected and data collected at time 1 b Time passes some members of the panel may become available and population may chan e c At time 2 panel is contacted and data collected 4 Difficulties with panel designs a xpense b Attrition panel members may drop out of study c Subject fatigue panel members may tire of repeated interviews and drop out or give thoughtless answers iii Eventbased Design data are collected from different samples within a cohort also known as cohortstudies l Cohort individuals or groups with a common starting point eg birth year year of graduation years of seniorit 2 An eventbased design can be a type of repeated cross sectional or a type of panel design XXXVI Unit of analysis the level of social life on which the research question is focused such as individuals groups towns or nations a In most sociological and psychological studies the units of analysis are individuals b Units of observation the level of social life at which data is collecte c Conclusions about processes at the individual level should be based on individual level data39 conclusions about grouplevel processes should be based on data collected about groups i Ecological Fallacy a group level premise is used to draw conclusions about individuals drawing conclusions about individuallevel processes from group level data ii quot 1 39 39 Fallacy 1 39 39 or 39 139 39 1 139 fallacy Making inferences about group processes from individual data39 reduces individual behaviors to being entirely caused by individual attitudes and preferences XXXVH The Meaning of Explanation a Cause An explanation for some characteristic attitude or behavior of groups individuals other entities or for events b Nomolheiic causal explanation an explanation involving the belief that variation in the independent variable will be followed by variation in the dependent variable when all other things are equal ceteris paribus i Ts probabilistic Counterfactual the situation as it would have been in the absence of variation in the independent variab e iii Causal e ect nomothetic perspective The finding that change in one variable leads to change in another variable other things being equal ceteris paribus Idiographic causal explanation the concrete individual sequence of events thoughts or c actions that resulted in a particular outcome for a particular individual or that led to a particular event also known as an individualist or historicist explanation 39 Causal e ect idiographic perspective the finding that a series of events following an initial set of conditions leads in a progressive manner to a particular event or outcome ii Narrative reasoning iii High concern for context iv Holistic explanations v Deterministic d CaseOriented Understanding attempt to understand a phenomenon from the standpoint of the participants i Phenomenological ii Not causal but illustrative XXXVIH Criteria for Identifying a Causal Effect i ii and iii below are most critical for establishing causality i Empirical Association Empirical observed correlation between the independent and dependent variables ie they must vary together ii Appropriate Time Order The independent variable comes before the dependent variable ie the temporal priority of the independent variable iii Nonspuriousness The relationship between the independent and the dependent variable must not be due to a third variable ie the relationship must not be spurious or false In an experiment randomization reduces the risk of spuriousness 2 Randomization or random assignment the use of random techniques to designate subjects into treatment or comparison groups 3 ln nonexperimental designs statistical control can reduce the risk of spuriousness Statistical control a technique used in nonexperimental research to reduce the risk of spuriousness The effect of one or more variables are removed for example by holding them constant so that the relationship between the independent and the dependent variables can be assessed without the influence of variation in the control variables iv Identifying a causal mechanism the process that creates a connection between the variation in an independent variable and the variation in the dependent variable it is hypothesized to cause Intervening variables Causal mechanism39 the variables that explain the relationship between the independent and dependent variables Extraneous variables third variables that do not appear in the initial hypothesis and produce a spurious relationship between independent and dependent variables v Context Identification of other variables that allow for the relationship between the independent and dependent variable to occur Chapter 7 Experiments r e 19 Chapter Overview Chapter 7 centers on experiments as a method for collecting data The characteristics of a true experiment are discussed followed by an analysis of how the true experiment established causality Then quasiexperimental designs nonequivalent control group and beforeandafter designs are described in detail Nonexperimental designs are introduced including the ex post facto control group and oneshot case study designs The chapter then turns to problems of validity associated with experiments Sources of internal invalidity are detailed noncomparable groups selection bias mortality endogenous changes history contamination treatment misidentification Problems with external invalidity are also presented The Solomon Four Group Design is introduced as a means to overcome the potential interaction of testing and treatment The final section of the chapter covers ethical issues related to experiments especially deception and the distribution of benefits Lecture Outline Experim ents a The most powerful design for testing causal hypotheses because experiments establish association time order and nonspuriousness b Experimental research is most appropriate for answering research questions about the effect of a treatment or some other variable whose values can be manipulated by the researcher 90 XL The most powerful design for testing nomothetic causal hypotheses Greater confidence of validity of causal conclusions than in other designs True Experiments a Two comparison groups to establish association 1 ii iii iv Experimental Group The group of subjects in an experiment that receives the treatment or experimental manipulation Comparison group The group of subjects that is exposed to a different treatment from the experimental group or that has a different value on the independent variable Control group A comparison group in an experiment that receives no treatment In many experiments the independent variable is the presence of absence of something b Independent variable variation must be collected before assessment to establish time order i ii iii Posttest A measurement of the outcome dependent variable in both groups after the experimental group has received the treatment Pretest A measurement of the dependent variable taken prior to experimental intervention 1 Exactly the same as posttests but at a different time 2 Not necessary for true experiments because random assignment assumes that groups will be initially similar 3 Direct measure of how much groups have changed over time 4 Allows verification that randomization has been successful 5 Provides a more complete picture of the conditions in which the intervention did or didn t have an effect Randomized comparative change design aka pretestposttest control group design a randomized experimental design with a pretest and a post test c Randomization or Random assignment to grou s 1 Random assignment randomization The random assignment of subjects into experimental and control groups 1 Identifies effect of treatment 2 Systematic bias does not affect assignment 3 Does not ensure subjects are representative of some larger population 4 Creates two or more equivalent groups to ensure internal validity Matching The assignment of subjects into experimental and control groups in which subjects are paired based on some similarity eg gender age and individuals in the pair are randomly assigned to each group 1 Matching can only be done on a few characteristics and therefore may not truly distribute characteristics between the two groups 2 When matching is used design is term ed quasiexperim ental Using randomization techniques after matching pairs can approximate randomization E iii Randomized comparative posttest design aka posttest only control group design a randomized experimental design with no pretest d Limitations to True Experimental Designs i Doesn t usually identify causal mechanism ii Doesn t guarantee control over conditions to which subjects are exposed after r up assignments this is especially problematic for field experiments e Meeting criteria for nomothetic causes i Association is unambiguous with comparison between grou s ii Time order is established by comparisons based on pretests and posttests as well as between comparison groups iii Nonspuriousness established by randomization iv Some ambiguity in establishing causal mechanism v Context is more controllable in laboratory than field experiments XLl QuasiExperimental Designs research designs in which the comparison group is predetermined to be comparable to the treatment group in critical ways but are not randomly assigned to groups a Non Equivalent Control Group Designs groups are designated before the treatment occurs but are not created through random assignment or matchin i Control group is selected to be as similar as possible to experimental group through matching of either individual or aggregate characteristics ii To be quasiexperimental individuals must be able to choose whether to be in experimental or control groups b Before andAfterDesigns designs that have no control group basis of comparison is between pretest and posttest i Fixed sample panel designs are not quasiexperimental because comparing subjects to themselves at just two points in time does not provide adequate control ii Multiple Group Before andAfter Designs several before and after comparison groups are made involving the same variables but different groups iii RepeatedMeasures Panel Designs several pretest and posttest observations of the same group iv Time Series Designs many 30 observations are made in pretest and posttest eriods c Causality in quasiexperiments i s unambiguous as true experiments in establishing association ii Time order is well established in beforeandafter designs but less clear with nonequivalent control group designs iii Nonspuriousness cannot be completely established but considerable nonetheless iv Mechanisms are usually not identified but repeated measures designs can test hypotheses about causal mechanisms v Better at addressing context provided the researcher measures contextual variables XLH Nonexperim ental designs a ExPost Facto Control Group Designs groups are designated AFTER experimental treatment i Somewhat like nonequivalent control group desing ii Potential for selfselection into experimental and control groups may create systematic bias b Oneshot Case Studies Crosssectional designs in the experimental design literature c Causality inNonexperiments 39 Clear evidence of association ii Longitudinal designs better at establishing time order than cross sectional designs iii Nonspuriousness is the major weak point of nonexperimental designs iv No particular advantage or disadvantage in terms of causal mechanisms but qualitative research does facilitate identification of mechanism Because of the ability to survey large numbers one shot case studies facilitate investigation of contextual effects lt XLHI Validity in experiments a True experiments are well suited to produce internal validity but much weaker in generalizability b Quasiexperiments may be more generalizable but more prone to problems with internal invalidity c Sources of Internal lnvalidity in experimental designs i Non comparability of groups 1 Selection bias subjects in experimental and control groups are initially different a Reduced through randomization b Weaker in field experiments because of potential for non random assignment c Differential attrition or deselection alters content of groups d High chance of selection bias in nonequivalent control group desi ns e Pretesting helps researchers to determine and control for selection bias 2 Endogenous change natural developments in the subjects independent of experimental treatment account for some or all of the change between pre and posttest scores a Testing Pretest can in uence posttest scores b Maturation Changes may be caused by the aging of subjects c Regression e ect when subjects are selected because of scoring on an extreme on future testing they will tend to regress back to the avera e 3 History e ects external events Things that happen outside the experiment that may change subjects scores a ka The History E ect 4 Contamination control and experimental groups somehow affect one another a Compensatory Rivalry The John Henry E ect when groups know they re being compared they may increase their efforts just to be more competitive b Demoralization the control group may feel they ve been left out and perform worse than expected c Contamination can be limited by i No contact between groups ii Relatively brief treatment period ii Treatment misidenti cation some process of which the researcher is not aware is responsible for the apparent effect of treatment 1 Expectancies of experimental sta staff actions and attitudes change the behavior of subjects a Selffu lling prophecy staff actions encourage expected response Double blind designs neither the subject nor the staff knows who s getting the treatment and who s not to eliminate staff 9 bias 2 Placebo e ect Subjects change because of expectations of change not because of treatment itself 3 Hawthorne E ect Participation in the study may change behavior simply because subjects feel special for being in the stu y d Generalizability i Sample generalizability is low xperimental subjects are unlikely to be representative of any larger population 2 The more artificial the setting the less generalizable the results 3 Field experiments are better at approximating generalizability than laboratory experiments with volunteers ii External validity Crosspopulation is weak in experimental research The body of experimental research can give confidence than one experiment 2 Replication is the best tool for establishing generalizability from experiments 3 Solomon Four Group Design lessens the impact of pretest sensitizing because it includes two sets of control and experimental groups I Experimental Group 1 Pretest Treatment Posttest lT Control Group 1 Pretest Posttest lTl Experimental Group 2 Treatment Posttest IV Control Group 2 Posttest XLIV Combining methods a Qualitative methods can help understand experimental effects b Experimental techniques can be added to surveys to allow stronger hypothesis testing c Process Analysis a technique for understanding how an experiment effects the dependent variable by taking periodic measures throughout an experiment to assess whether treatment is being delivered as planned d Factorial surveys Randomly selected subsets of survey respondents are asked different questions representing in effect different experimental treatments i is to determine the impact of different questions on answers to other questions ii Only indicate what respondents say they will do doesn t measure behavior XLV Ethics a Deception occurs when subjects are misled about research procedures in order to determine how subjects would react if they were not research subjects 39 Necessary to simulate realworld in a lab settin i1 Deception is acceptable only if experiment does not pose greaterthan everyday risks to subjects iii Debriefing is required 1 Research is justifiable because of value of findings 2 Equally effective alternative procedures are not feasible 3 Researchers have obtained approval from an IRB or similar board 4 Deception will not involve aspects that might affect participants willingness to participate b Debrie ng explaining the true nature of the experiment to subjects after participation is a good idea when using deception c Distribution of bene ts How much are subjects harmed by the way treatments are distributed in the experiment i Random distribution of benefits is justified when the researchers do not know whether some treatment is actually beneficial OR ii There are insufficient resources available to fully fund a benefit for every eligible person Chapter 8 Survey Research Chapter Overview This chapter provides details about survey research It explains why surveys are such a popular form of data collection Students are instructed on how to write survey questions and how to design a questionnaire or interview schedule Next different forms of survey administration are considered mailed surveys groupadministered surveys phone surveys inperson surveys and electronic surveys Each is described and compared with other methods of administration in terms of cost response rate and appropriate content The chapter closes on a discussion of ethics in surveys centered largely on the importance of maintaining confidentiality and considering anonymity Lecture Outline XLVI Survey Research collects information from a sample of individuals through their responses to standardized questions a One third of all published research in sociology economics and social psychology are surveys b Three advantages i Versatility can enhance understanding of just about any social issue ii Efficiency many respondents at a relatively low cost and relatively quickly iii Generalizability survey methods lend them selves to probability sampling from large populations c Omnibus survey a survey that includes a range of topics of interest to a social scientist or other sponsor such as the GSS d Splitballot design some questions are asked of only a randomly selected subset of respondents i More questions without substantially increasing costs ii Facilitates comparison of question wording XLVH Errors in Survey Research a Errors of observation poor measurement of cases that are surveyed b Errors of nonobservaiion omission of cases that should be surveyed i Nonresponse can distort the sample when individuals refuse to respond or cannot be contacted 70 is a minimum response rate to speak to representativeness 2 Nonresponse rates are increasing in the US and Western Europe 3 Benefits of participation must outweigh the costs if you expect people to participate 4 Clear and interesting questions and presenting them in a well organized uestionnaire will improve response rate ii Coverage of the population may be inadequate due to a poor sampling frame iii Sampling error XLVIH How to write survey questions a Selecting good questions is the single most important concern for survey research b Begin with brainstorming and review of previous surveys c Be clear and avoid confusing phrases 39 Simple direct questions are best Use shorter rather than longer words and sentences iii Keep number of words per question less than 20 and number of commas less than 4 iv Don t abbreviate so much as to confuse respondents v Avoid negatives especially double negatives vi Avoid double barreled questions questions that ask two questions but allow only one answer Identify clearly what kind of information each question is to obtain eg thoughts feelings behaviors characteristics Use ller questions questions that identify relevant respondents and mark them clearly Fr 5 lt E 1 skip patterns order of questions and skips established by filter questions 2 contingent questions questions answered only by those who answered yes to a filter question d Mnimize bias i Words should not trigger biased responses ii Don t use biased or loaded words iii Phrasing may make questions more or less attractive iv Make sure response choices re ect full range of possible sentiment v If responses fall on a continuum it is important that the number of positive and negative categories are balanced vi Push polls polls that try to influence respondents are not surveys but are really propaganda e Avoid making either disagreement or agreement disagreeable 39 Subjects tend to agree just to avoid disagreeing Present both sides of an attitude scale in the question itself Response choices should be phrased to make each choice as agreeable as the others iv Add a counterargument in favor of one side to balance an argument in favor of the other v When illegal or socially disapproved behavior is the focus be vigilant about wording as respondents will be reluctant to admit it v1 Asking about a variety of behaviors that range from socially acceptable to socially unacceptable will soften the impact of agreeing with socially unacceptable or illegal thoughts or actions f Allow for uncertaint i Fencesitters respondents who see themselves as being neutral and who may skew results if you force them to choose between opposites ii Floaters respondents who choose a substantive answer when they really don t know iii Don t know responses may be chosen more often by less educated respondents and lazy respondents iv Omit no opinion or don t know categories ifyou feel that people have an opinion but are reluctant to express it v Forced choice questions questions without a don t know option vi Adding an openended question in which respondents are asked to discuss their opinions can shed light on floaters vii Interviewers can omit neutral and don t know options from questions but record noncommittal responses g Maximize the utility of response categories i Responses should be mutually exclusive and exhaustive ii Filter questions may tell respondents to skip questions so responses need not be exhaustive 1 Skip in should be minimized on selfadministered surveys iii Check all that apply questions do not have to be mutually exclusive 1 Check all that apply questions should be minimized iv Avoid vagueness in response choices 1 refer to specific times or events 2 Don t make unreasonable demands of your respondents memories 3 Limit questions about past experiences toiat mostithe past month unless the focus is on major events v Adding questions on same issue may correct vague answers 1 itive interviewing using a series of questions to imprve memories about the event of real interest vi Number of responses varies in appropriateness so pretesting is key XLLX Design of a Questionnaire a Questionnaire the survey instrument containing the questions in a selfadministered survey Interview schedule the survey instrument containing the questions asked by the b interviewer in an inperson or phone survey c The context created by a questionnaire has a major impact on how individual questions are interpreted or if they are answered at a d Use previously designed measures if i They measure the concepts of concern in present research ii They seem appropriate for the population in present research e Refine and test questions in pretests to make sure respondents will understand the questions i Ask colleagues to review questions ii Review prior research on topic iii Have a panel of experts review questions iv Conduct a focus group of potential respondents in which they are asked to explain their responses v Conduct cognitive interviews the researcher asks a question and then probes with followup questions to learn how the question was understood vi Behavior coding a researcher observers several interviews or listens to taped interviews and codes according to strict rules the number of times that difficulties occur with each question vii Pilot study of questionnaire 1 Prepare questionnaire complete it yourself and revise 2 Try revision on friends and colleagues and revise 3 Draw a small sample from population and administer revision and revise viii Add interpretive questions where respondents are asked to explain what they mean by their responses xtremely important if questionnaire has not been thoroughly pretested 2 Five issues to consider when developing interpretive questions a What do respondents know b What relevant experiences do respondents have c How consistent are respondents attitudes and do they express some larger perspective or ideology d Are respondents actions consistent with their expressed attitudes e How strongly are attitudes held f Maintain a consistent focus i Except in omnibus surveys ii Don t include irrelevant questions iii Measure all independent and dependent variables iv Respondents may tire of long lists of redundant or unimportant questions v Researchers tend to error on the side of extraneous questions to avoid missing something g Order the questions carefully i Questions should be sorted into thematic categories ii Question order must make logical conversational sense iii First question is very important connects to the primary purpose of the study 2 should interest the respondent 3 should be easy to answer 4 should apply to everyone in the sample 5 should NOT be a sensitive question iv Filter questions may appear early in order to identify which sections a respondent should skip L v Context e ects one or more questions in uence how subsequent questions are interpreted be sure to check for this especially if early questions commit respondent to later responses 1 Partwhale question e ect if a general question is preceded about a question in specific respondents are more likely to report differently on the general question 2 Context effects are greater when two or more questions address the same subject 3 Context effects are greater for general summary questions 4 Reversing question order on a split ballot design can identify context effects 5 Matrix format a series of questions that concern a common theme and that have the same response choices written so a common initial phrase es to each one a h Make the questionnaire attractive neat clear clean and spacious Lots of white space ii List responses vertically iii Response choices are distinguished clearly and consistently iv Indicate skip patterns with graphics Make instructions very clear on skip patterns vi Use distinctive formatting to identify instructions vii Instructions should explain how to answer each type of question viii Booklets are better than multipage questionnaires Organizing surveys a b Features of survey desi ns i Manner of administration selfadministered or interviewer led ii Questionnaire structure highly or loosely structured iii Setting one or multiple respondents at a time iv Cost low to high Survey designs i Mailed surveys questionnaire is mailed to respondent who fills it out and returns it self administered usually highly structured one respondent at a time low cost 1 Central concern is maximizing response rates 2 Low response rate hurts representativeness 30 or less in first mailin 3 Procedure for increasing response rate a Send a preliminary introduction letter a few days before the questionnaire is mailed b Senda H p A envelope if possible a token monetary award and a personalized cover letter tha 39 Establishes credibility of researcher and research Personalize greeting and contains an original signature ii39 Is interesting to read iv Explains voluntary participation and informed consent Send reminder postcard after 2 weeks Send new cover letters and replacement questionnaires 24 quot 39 a f 4439 r stamped 9 68 weeks after initial mailing mail new cover letters and replacement questionnaires through a different mode of delivery or a different survey design 4 Other ways to increase response rates F Use clear and understandable questions Use few open ended questions Use a credible research sponsor Write an identifying number on the questionnaire to determine who nonrespondents are Include a token incentive Include a selfaddressed stamped envelop with each copy of questionnaire Use presurvey advertising Response rates may mask selection bias so make every effort to contact nonrespondents i Phone surveys of nonrespondents help reduce bias j Poorly designed questionnaires tend to produce more incomplete surveys ii Group administered surveys completed by individual respondents gathered in a group selfadm inistered usually highly structured multiple respondents at a time low cost 9 57 W FV IQ 1 High response rates 2 Seldom feasible because it requires a captive audience 3 Administrator must not bias answers in introductory statement 4 Before distribution researcher should explain purpose of survey express appreciation and describe steps on survey 5 A cover letter should be distributed 6 Distribute envelopes with the survey to ensure confidentiality 7 If respondents feel coerced eg students they may not answer honestl iii Telephone surveys interviewers question respondents over the phone staff conducts interviews usually highly structured one respondent at a time moderate cost Obtaining a sampling frame a Phone directories for local surveys b Commercial firms provide area codes and exchanges for national surveys c Commercial firms provide national files based on local directories 2 Coverage is not complete in any sampling frame 3 Random digit dialing allows capture of unlisted numbers 4 Interviewers must always establish that they are talking to the appropriate person 5 Maximizing responses to phone surveys a Multiple call backs often 20 or more b Interviewers must be able to verbally convey all information about response choices and skip patterns c Interviewers must be prepared for distractions interruptions and incompletes i Rapport is lower than in face to face interviews ii Sprinkle interesting questions throughout d Careful interviewer training e ComputerAssisted Telephone Interviews CATI The questionnaire is programmed into a computer with relevant skip patterns only allowing legal entries thus incorporating the tasks of interviewing data entry and some data cleaning f Telemarketers have tainted phone surveys people will decline to participate or screen calls with an answering machine 6 Phone surveys are appropriate for short surveys of the general population 7 response rates tend to be high but refusal rates are increasing iv Inpersan interviews facetoface interviews in which interviewer asks questions of respondents staff conducts interviews can be loosely structured one respondent at a time most expensive Highest response rates and highest cost Questionnaires can be longer and more complex Question order can be controlled on site Physical and social context can be monitored Respondents answers can be probed and clarified Interviewers may bias results so careful training is necessary Problems a Standardization is more difficult b Rapport increases quality of responses but changes consistency 39 Project professional image as interviewer Establish rapport at outset by explaining interview and consent form iii Ask questions from a distance that is close but not intimate iv Try to maintain consistency v Repeat questions if respondent is confused SQEJ HeP NE 1 v1 Use nondirective neutral probes for open ended questions c Presence of an interviewer makes sensitive material more difficult d If respondents are similar on social characteristics respondents will answer differently than if they are different e Respondents may alter responses based on presence of other people at the time of the interview 8 ComputerAssisted personal Interviewing CAPI interviewers carry a laptop computer that is programmed to display the interview questions 9 Maximizing response rates a ontact rates are lower in central cities and for singleperson households b Contract rates are higher among elderly and households with young children c Less educated respondents are more likely to refuse or say don t know d Wealthy people avoid questions about money v Electronic surveys respondents complete surveys mailed through email or available on website selfadministered usually highly structured one respondent at a time lowest cost 1 Email surveys are sent to respondent email addresses respondents complete them and then email them bac a Very cheap b Cumbersome for even moderately long surveys 2 Web surveys are stored on a server Respondents visit the website and answer questions 3 A large fraction of households are not connected to the Internet so this is not a good way to conduct research on the general population 4 Surveys should not be too technical or fancy for respondent or for average computer capacities 5 requires programming skills 6 Interactive voice response systems respondents receive automated calls and answer questions by pressing numbers on their touchtone phones or speaking numbers that are interpreted by computerized voice recognition software Short surve s b Highly motivated respondents vi Mixed mode surveys l Strengths of one survey mode can compensate for weaknesses of another 2 Maximize likelihood of securing data from different types of respondents 3 Respondents may give different answers because of mode of survey Unimade design substantially reduces mode effect on responses a Same question structures response choices and skip instructions across modes b Small number of response choices r e Ll Comparison of survey designs see Exhibit 810 a Mailed surveys have lowest response rate but good for reaching large dispersed populations at relatively low costs Phone surveys require greater funds and increasingly bad on response rtes c In person surveys allow greatest length and complexity as well as ability to monitor conditions while questionnaire is complete d Mailed surveys are preferable for sensitive questions but inperson interviews can give respondents a separate sheet of sensitive questions S7 e Electronic surveys are limited by technology and access f In person interviews are the strongest design and generally preferable when sufficient resources and a trained staff are available g Telephone surveys have many of the advantages of inperson interviews but at a lower cost h A decision about the best survey design for any particular study must take into account the unique features and goals of the stu y LH Combining methods a Adding qualitative data provides information on quality of records and quantitative measures as well as meaning of fixed responses b Records compiled by streetlevel bureaucrats officials who serve clients and have a high degree of discretion require close inspection by multiple methods L1H Ethics in Survey Research a Fewer ethical dilemmas than in experimental or field research i refusal is easy methods are straightforward Captive audiences must be free to decline to participate b Confidentiality is greatest concern i wers may damage respondents if identities are discovered ii Limit knowledge of respondents identities to staff and then only for specific research purposes eg followup mailings iii Only numbers should identify numbers on surveys iv Electronic surveys should keep information encry ted c Anonymity no identifying information is ever recorded to link respondent to responses 39 Precludes followups Possible only if followups are not needed Not possible in inperson interviews rv To maintain anonymity in mailed surveys include a postcard that can be mailed separately to identify respondents 3955 Chapter 9 Qualitative Methods Observing Participating Listening Chapter Overview This chapter considers What constitutes qualitative research and in doing so contrasts this with experimental survey and other more quantitative research methods Qualitative researc encompasses three principal methods participant observation intense interviewing and focus groups Participant observation is described in detail with special attention to the role of the researcher complete observer participant observer or covert participant Interviewing is discussed with particular attention to the selection of informants Focus groups are described as a third general type of social research Lecture Outline LIV Qualitative Methods refer to participant observation intense interviewing and focus groups a Features of Qualitative Methods i Collection of primarily qualitative rather than quantitative data ii Typically begin with exploratory research question iii Inductive approach iv Data gathering and analysis proceed together v A focus on previously unstudied processes and unanticipated phenomena vi Orientation to social context and interconnection vii Focus on human subjectivity and meanings given by participants viii Sensitivity to the subjective role of the researcher ix Use of idiographic rather than nomothetic causal explanations x Re exive research design develops as research progresses b Origins of qualitative research 39 Anthropology and sociology did fieldwork ii Direct participation in activities of subjects c Field Research research conducted Within the context of everyday activities i combines various forms of qualitative research ii involves huge investment in time iii avoids artificiality of laboratory designs iv Emphasizes complex context v Good for understanding mechanism vi Often referred to as participant observation but see below LV Participant observation a qualitative method for gathering data in which natural social processes are studied as they happen in the field rather than in the laboratory and left relatively undisturbed a Characteristics i observing and interacting with subjects in the course of their normal activities ii avoid the artificiality of experiments and the structured questioning of surveys iii context is better understood b Choosing a role i Considerations 1 specifics of the situation 2 researcher s background and personality 3 larger sociopolitical context 4 ethnical concerns ii Complete observation researchers try to see how things happen Without actively participating l Reactive e ects because observation is unnatural the presence of a researcher can alter the social situation iii Mixed partiCipatianabservatian researchers try to see how things happen While participating in some activities under stu y 1 Some members of the group at least should be informed of role as researcher 2 Participation in group builds rapport with group members iv C lt d Develo E E e Sampli i ii iii 3 Ethical advantages roup members can choose to keep some information hidden b Researcher can decline to participate in some activities 4 As rapport builds artificiality and reactive effects decline Covert partictpation complete participation to gain entry into otherwise inaccessible settings adopting the role of covert participant in which the researcher keeps hisher identity secret and tries to act like other participants Low reactive effects 2 Accessibility to hard to access settings 3 Difficulties Cannot take notes or use obvious recording devices Cannot ask questions that may arouse suspicion Have to model act spontaneous reactions to be consistent with other participants Must keep up act all of the time in the field Ethical minefield informed consent is not secured in advance Going native going along with unethical activities in the course of research Research may cause unintended consequences for research subjects Research may reduce trust in social scientists a b c wot1 FV IQ Entering the field Before entering the filed 1 have a clear understanding of likely research question 2 learn appropriate dress and behavior 3 find informants to make introductions 4 get formal permission as necessary Informal at first to build trust and rapport Formal direct approaches may secure entry to public figures Researcher should be ready with a rationale for participation and some sense of potential benefits to participants Gatekeepers key informants who can determine access to the field andor other respondents ping and Maintaining Relationships Interaction early in research is key to later success Key informants knowledgeable insiders who knows the group s culture and is willing to share access and insights Don t fully immerse in the field eg William Whyte did no research on the family from whom he rented his room Other things to keep in mind Develop a plausible and honest explanation for yourself and your study Maintain the support of key individuals in groups or organizations understudy Be unobtrusive and unassumin Don t be too aggressive in questioning others Ask sensitive questions only after strong rapport is established Be selfrevealing to a pint Don t fake a social similarity Avoiding giving or accepting money or gifts but without violating norms of reciprocity Be prepared for intergroup con ict if multiple groups are involved ng people and events Sampling decision are based on need to intensively study people places or phenomena of interst Small number of sites or programs Sample must be appropriate and adequate 2 993 O LVI 1 Fr iv Not representative 1 critical case unusually rich in information pertaining to research question 2 typical case judged to be typical 3 deviant case provides a useful contrast v Studying more than one case almost always strengthens causal conclusions and makes findings more generalizable vi Nonprobability sampling methods can be employed vii Theoretical sampling when a researcher focuses investigation on particular processes that seem to be important and select instances to allow comparisons or checks with which perceptions can be tested adding members to sample t include additional critical typical or deviant cases viii Experience sampling method ESAI The experiences thoughts and feelings of a number of people are sampled randomly as they go about their daily activities 1 Participants carry an electronic pager 2 Participants fill out reports when they are beeped Taking notes i Notes are the primary means of recording participant observation data ii Jottings brief notes taken in the field iii Field notes written after observations based on jottings Write immediately after observation period or as soon as possible Takes up to three times as long as observation period Distinguish direct quotes from paraphrased quotes Surrounding context should be detailed Record own thoughts and feelings in the field 6 Su plement notes with visual material as appropriate iv Daily log daily record of activities sentiment and perceptions Managing the Personal Dimension personal relationships invariably develop with informants but maintain ethical standards and distance i Varies with depth of researcher involvement ii Increased contact brings sympathy which hampers criticism iii Researchers social attributes and those of subjects shape personal relationships iv Guidelines proposed by Whyte 1955 1 Take time to consider how you want to relate to potential subjects as people Speculate about what personal problems might arise and how you will respond to them 3 Keep in touch with other researchers and personal friends outside the research settin 4 Maintain standards of conduct that make you comfortable as a person and that respect the integrity of your subjects Systematic observation developing a standard form on which to record variation within the observed setting in terms of variables of interest i Tncreases reliability of observational data ii Good for multiple methods investigations V eP Nf 19 Intensive interviewing relies on openended questions to develop a comprehensive picture of the interviewee s background attitudes and actions mm ho 99 79 Open ended questions May be highly unstructured or structured Tend to be lengthy Becomes like a directed conversation Requires active probing of responses Learn about setting before interviewing Plan an outline of topics to cover in an interview Grand tour questions open ended questions that are designed to elicit lengthy narratives W 3 1391 LVH Random selection of informants is rare Select interviewees W o 39 Are knowledgeable on the subject of interest ii Are willing to talk iii Represent a range of perspectives from within a group Selection of new interviewees should continue until a saturation point is reached that is until new interviews yield little additional information Establishing and maintaining a partnership in an interview i Do not violate standards of social behavior ii Treat interviewees with respect iii Show interest in interviewee early in the interview iv Early in the interview explain clearly the purpose of the interview v Be mindful ofpace Asking questions and recording answers i Plan questions around an outline but be exible in order ii Make questions short and to the point iii Use nondirective probes iv Follow up questions can be tailored to answers v Tape recorders are a good idea are routinely ignored however some respondents will give away good information when the tape recorder gets turned off vi Constant notetaking is a distraction combining participant observation and intensive interviewing can deepen understanding Focus groups groups of unrelated individuals that are formed by a researcher and then led in a group discussion of a topic a b 099 corvs Fr LVIH OFT Marketing researchers were the first to adopt focus groups as a widespread methodology Selection of participants 39 710 people Usually strangers to one another Unrepresentative of population iv Have time to participate v Have some knowledge of to 1c 39 Share key characteristics of target population Homogenous groups are more willing to talk but heterogenous groups may stimulate new ideas Researcher must not favor any particular perspective or participant Open ended questions are posed by group leader Interview guide researcher asks specific questions and guides conversation but is exible Discussions tend to be unstructured and exible No procedure for generalization i Does not produce generalizable results ii Better to conduct several focus groups on the same topic and compare results to a saturation point Good for i Developinghypotheses ii Developing survey questions iii Investigating the meaning of survey results iv Quickly assessing the range of opinions about an issue Ethical Issues in Qualitative Research a Voluntary participation a huge issue in participant observation especially in covert participation Subject wellbeing concern over feelings and reputation of informants Identity disclosure researcher should endeavor to fully disclose his or her identity P Confidentiality make every effort to expunge all possible identifiers from published F Online research Does use of billboards discussion groups and listservs require informed consent Chapter 10 Evaluation Research Chapter Overview This chapter introduces evaluation research which grew out of the need to assess social programs After a brief history of evaluation research key terms associated with evaluation research are introduced Evaluation research is compared with more traditional forms of social science research Next the chapter addresses the potential foci of evaluation research needs assessments evaluability assessments process evaluation impact analysis and efficiency analysis Several considerations are addressed when designing evaluations Whether how the program works is important black box or program theories to Whom the evaluation is oriented other researchers or stakeholders Whether quantitative or qualitative methods should be employed and Whether to measure simple or complex outcomes Ethical considerations are addressed for evaluation research including an overview of federally mandated ethical practice criteria and ideas to less detrimental impacts on participants The chapter conclusion includes a discussion of the obstacles associated with evaluation research Lecture Outline Evaluation Research is conducted for a single purpose to investigate social programs a Began after expansion of federal government during the Great Depression and WWII b Became more important with Great Society programs of 1960s because program evaluation became a requirement c The New Jersey lncome Maintenance Experiment was the first large scale experiment to test social policy in action d Decline of evaluation research firms in early 1980s as Great Society programs also declines e American Evaluation Association was founded in 1986 as a professional organization for evaluation researchers f Government Performance and Results Act of 1993 required some type of evaluation for all government programs LX Parts of social program a In puts resources raw materials clients and staff that go into a program Program process the complete treatment or service delivered by the program Outputs the services delivered or new products produced by the program process Outcomes the impact of the program process on the cases processed Feec 2ack information about service delivery system outputs outcomes or operations that is available to any program inputs i Evaluation research itself is really just a systematic approach to feedback Stakeholders individuals and groups who have some basis of concern with the program often setting the research agenda and controlling research findings LXI Types of evaluation research a Needs assessment an attempt to determine if a new program is needed or an old one is still required i Defining needs is tricky and political ii It is a good idea to use multiple indicators of needs Evaluability assessment a determination if a program may be evaluated Within available time and resources 39 May be used to sell an evaluation Tends to rely on qualitative methods in Often has action research approach iv Why can t a program be evaluated 1 Management only wants superior performance confirmed nap057 239 7 t 2 Staff doesn t trust evaluation to be fair or balanced 3 program has no clear sense of intent 4 Program is not distinct from other services produced by the agency c Process Evaluation implementation assessment evaluation research that investigates the process of service delivery i Important for complex programs ii Can identify the specific aspects of a program that have an impact iii Formulative evaluation process evaluation that is used to shape and refine program operations iv Quantitative data is useful but qualitative data are used to elucidate and understand internal program dynamics unanticipated outcomes and interaction among program participants d Impact analysis evaluation research that compares what happened after a program was implemented with what would have happened had there been no program at all i Also known as impact evaluation and summative evaluation ii Experimental design is preferred method for maximizing internal validity 1 Program is independent variable treatment 2 Outcomes are dependent variables 3 Random assignment allows evaluation of programs impact independent of the types of people in different groups iii Selection bias in nonexperimental designs is a common problem because people selfselect to participate in programs e E iciency analysis costbenefit and costeffectiveness evaluation 1 Costbene t analysis Are programs financial benefits sufficient to offset the programs costs 1 specifies costs and benefits which are often contingent on definition andor perspective 2 measures costs and benefits 3 monetarizes costs and benefits ii Cost e ectiveness analysis How much does it cost to achieve a given effect LXH Design alternatives in evaluation research a Black box or program theory is it important how the program gets results i Blackbox if program results are of primary importance how it works may be of secondary importance ii Program theory a descriptive or prescriptive model of how a program operates and produces effects 1 specifies how a program is expected to operate 2 identifies which program elements are operational 3 specifies how a program is to produce effects 4 improves understanding of the relationship between the independent variable the program and the dependent variable outcomes a identifies conditions required for the program to have intended effect b descriptive program theory specifies impacts that are generated and how this occurs suggesting a causal mechanism intervening factors and context generally empirically based c prescriptive program theory specifies what ought to be done by the program but has not yet been empirically tested or observed i how to design or implement treatment ii expected outcomes iii how performance should be judged iii Theorydriven evaluation when a researcher has sufficient knowledge before the investigation begins outlining a program theory can help to guide the investigation of a program process in the most productive directions b C d LXIH b 0 Researcher or Stakeholder Orientation is the primary evaluator of research a set of social scientific peers or a funding agency i Stakeholder approaches encourage researchers to be responsive to stakeholders l utilizationfocused evaluation the evaluator forms a task force of program stakeholders Who help to shape the evaluation project so that they are most likely to use its results 2 action research participatory research program participants are engaged with the researchers as coresearchers and help to design conduct and report the research 3 appreciative inquiry eliminates the professional researcher altogether in favor of a structured dialogue about needed changes among program participants them selves ii Social science approaches emphasize the importance of researcher expertise and maintenance of some autonomy in order to develop the most trustworthy unbiased program evaluation 1 goal free evaluation researchers do not goals of program but assess and compares needs of participants to a Wide array of program outcomes to identify unanticipated outcomes iii Integrative approaches attempt to cover issues of concern to both stakeholders including participants and evaluators balancing stakeholder concern Wi scientific credibility Quantitative or Qualitative methods 39 Evaluation research that attempts to identify the effects of a social program typically use quantitative methods ii Qualitative methods useful for l investigating program process 2 learning how individuals react to treatment 3 understanding actual operation of programs 4 understanding more complex social programs Simple or complex outcomes 39 Most evaluation research attempts to measure multiple outcomes ii Unanticipated outcomes can be missed iii Single outcomes may miss the process of how a program works Ethical issues in evaluation research a The direct impact on participants and their families through social programs heightens the attention to hum an subjects concern Needs assessments evaluability assessments process analysis and costbenefit analysis have few special ethical considerations When program impact is focus human subjects problems multiply 39 Distribution of benefits Preservation of confidentiality is difficult i Decisions about burdening participants with evaluation research iv Research designs are often shaped by politics v Sharing of findings among all stakeholders or just some v Uncertainty in effectiveness evaluations vii Randomization and scientific credibility Will the results actually be used I E v d Federally mandated lRBs must assess all research for adherence to ethical practice guidelines 39 Minimize risks to participants Reasonable risks in relation to benefits Equitable selection of individuals randomization implies this iv Informed consent Monitorin of data Privacy and confidentiality EiFir slt e To lessen any detrimental program impact 39 39 39 39 number in control group ii Use minimum sample size iii Test only new parts of the program not the entire program iv Compare treatments that vary in intensity rather than presence and absence Vary treatments between settings rather than among individuals in a single s t in lt LXIV Obstacles to realizing potential of evaluation research valuation research can miss important outcomes or aspects of the program process Researchers can be subjected to crosspressures by stakeholders Answering to stakeholders can compromise scientific design standards Researchers may be pressured to avoid null findings or find their research findings 9 57 1gnored e Evaluation reports might need to be overly simplified for a lay audience and thus subject to some distortion Chapter 13 Qualitative Data Analysis and Content Analysis Chapter Overview This chapter begins with an evaluation of the features of qualitative comparison which has an implicit comparison with quantitative analysis from the previous chapter Key ideas about qualitative analysis are defined such as case study thick description progressive focusing and grounded theory The phases of qualitative data analysis are defined and explained Three types of qualitative data analysis are examined with some detail ethnography qualitative comparative analysis and narrative analysis Next the chapter reviews the potential of computer assisted qualitative data analysis programs such as HyperRESEARCH and NVivo The chapter then turns brie y to explaining content analysis a quantitative technique that shares much in common with qualitative analysis The chapter closes on a brief statement about ethics and guidelines for assessing the quality of qualitative reports Lecture Outline LXV Features of Qualitative Analysis a Data is text not numbers b Absence of predefined variables and hypotheses c Tends to be inductive d Emic focus i Emic focus Representing a setting with the participants terms ii Etic focus Representing a setting with the researchers terms e iterative and reflexive f progressive focusing the process by which a qualitative analyst interacts with data and gradually refines his or her ocus g Tends toward interpretivist perspectives h Guidelines Know yourself your biases and your preconceptions ii Know your question iii Seek creative abundance iv Be flexible v Exhaust the data vi Celebrate anomalies vii Get critical feedback viii Be explicit i Three modes ofreading text i Literally focus on its literal content and form quot Reflexively focus on how researcher s own orientation shapes interpretation and focus iii lnterpretively attempt to construct interpretation of what the text means Tends to be exploratory u Focus attention on interrelated aspects of setting or group the case Ethnography a study of the cultural life of a group Openended research questions rather than hypotheses The case study a setting or group that the analyst treats as an integrated social unit that must be studied holistically and its particularlity 39 The case may be a single organization community social group family individual etc ii Must be understood in its entiret iii Thick description a description that provides a sense of what it is like to experience that setting from the standpoint of the natural actors in that setting F BHT LXVI Techniques of Qualitative Data Analysis a Phases of qualitative data analysis i Documentation and collection of data 1 Analysis begins in the field with jottings field notes daily log transcriptions margin notes etc 2 Keeps track of a rapidly growing volume of data 3 Develop an outline for the analytic process ii Categorization into concepts ay begin with a simple observation 2 Analytic insights are tested against new observations and modified an iterative process A matrix or chart facilitates coding and categorization iii Connection between concepts 1 Centerpiece of the analytic process move from description to explanantion 2 Can also be diagrammed in a chart or matrix 3 Causal models in networks not lines iv Corroboration Authenticating conclusions 1 set standard exists for evaluating the validity of conclusions in a qualitative study 2 Assess in terms of a Credibility of informants b Was text spontaneous or prompted by researcher c Does the research in uence actions or statements of subjects 3 Tacit knowledge the largely unarticulated contextual understanding that is often manifested in nods silences humor and naughty nuances 4 Compare conclusions with other research 5 Reflexive reporting a natural history of the development of the evidence enables others to evaluate findings as well as how the researcher thinks she personally affected the report v Reporting the account LXVH Alternatives in Qualitative Data Analysis a Ethnography the study of a culture or cultures that some group of people share i Long term immersion in field setting Multiple methods of data collection and impressionistic materials it Researcher attempts to maintain some objectivity iv Rich description of life within culture v Subject to distortion by bias and artificiality of having a researcher present b Qualitative Comparative Analysis i Identifies qualitative variables that are present or absent in a series of cases ii Constructs multiple pathways to outcomes through combinations of resenceabsence variables c Narrative analysis 39 ocus on the story itself P F ii Seeks to preserve the integrity of personal biographies or a series of events that cannot be adequately understood in terms of discrete elements iii Coding strategies revolve around narratives as wholes Grounded Theory To build up inductively a systematic theory that is grounded in or based on the observations i Observations are summarized into conceptual categories ii Categories are tested with more observations iii Conceptual categories are refined and retested Computer Assisted Qualitative Data Analysis programs designed to speed up the analysis process eg NVivo HyperRESEARCH 39 Make it easier to experiment with different codes 11 Make it easier to test different hypotheses about relationships iii Make it easier to diagram emerging theories iv Make it easier to prepare final report v General process and ability of qualitative data analysis programs 1 Scan or enter data 2 Code data in program a Autocoding by key words must be checked b Viewing coded text embedded in original document c Codes can be revised d Multiple codes can be assigned e Comments can be linked to text segments f Hyperlinks can be placed in text 3 Analysis focuses on review of cases or segments with similar codes and examining relationships between codes 4 For reports a Use text to illustrate codes b Generate code frequencies LXVlTl Content Analysis The systematic objective quantitative analysis of message characteristics that facilitate making inferences from text LXLX 090579 Coding and categorizing text Identifying relationships among constructs defined in text Units surveyd are newspapers books shows persons in shows etc Words or other features are coded and counted Steps 39 Identify population of documents or other textual sources for study etermine unit of analysis iii Select a sample of units from population iv Design coding procedures for variables to be measured 1 Coding schemes 2 Codebook 3 Coding form Test and refine coding procedures 1 Tntercoder reliability 2 Context and measuring not must count of words 3 Validity assessed through construct validation 12 lt Evaluating qualitative analysis IQH FDPOUP Does it illuminate the phenomenon as lived experience Is it based on thickly contextualized materials Is it historically and relationally grounded Does it describe research process and researcher interaction with the setting Is it situated in prior research Does it acknowledge the researcher s own orientation upon starting the investigation Content analysis is judged by standards of validity applied to other quantitative measures LXX Ethics a Research integrity and quality depend heavily on researcher b Ownership of data and conclusions must be established c Use and misuse of results must be considered
Are you sure you want to buy this material for
You're already Subscribed!
Looks like you've already subscribed to StudySoup, you won't need to purchase another subscription to get this material. To access this material simply click 'View Full Document'