Research Design & Analysis: Lecture 15
Research Design & Analysis: Lecture 15 PSY 3392
Popular in Research Design and Analysis
verified elite notetaker
Popular in Psychlogy
This 8 page Class Notes was uploaded by Kimberly Notetaker on Monday April 25, 2016. The Class Notes belongs to PSY 3392 at University of Texas at Dallas taught by Noah Sasson in Summer 2015. Since its upload, it has received 14 views. For similar materials see Research Design and Analysis in Psychlogy at University of Texas at Dallas.
Reviews for Research Design & Analysis: Lecture 15
Report this Material
What is Karma?
Karma is the currency of StudySoup.
You can buy or earn more Karma at anytime and redeem it for class notes, study guides, flashcards, and more!
Date Created: 04/25/16
QUASI-EXPERIMENTAL DESIGNS “Quasi”: To some degree, in some manner, virtual, in some significant sense or degree; an “almost” experience - Quasi-experiments are used most often in applied (i.e., natural) settings The True Experiment A manipulation (only one level of the IV gets it) An appropriate comparison group (the other level of the IV) High degree of control o Randomization o Control of all conditions of the experiment o Lack of full control distinguishes the quasi-experiment from the true experiment The Quasi-experiment Has an IV and a DV Includes a comparison group, but lacks full control o No Randomization – uses “nonequivalent control group” If at all possible, use a true experiment Problems with Conducting a True Experiment in Real World Settings - Obtaining permission from people in authority - Obtaining participants o Typically, not all individuals want to participate Self-selection bias - Randomization o Often difficult in applied settings Can be disruptive Perception that randomization is “unfair” Ethics of the no-treatment control If a no-treatment control group is unfair, what are the alternatives? - Alternate order of treatments o One group gets tx1 then tx2, other gets tx2 then tx1 - Wait-list control o One group gets tx; if beneficial, control group gets tx later - Use of established treatments as a control - Treatment as usual control (the least you can do; free to pursue whatever treatment you’d like) Why Use a Quasi-Experiment? Randomization not always feasible External validity – does the intervention “work” in the real world Some hypotheses can’t be practically tested with a true experiment o Examples: 1. Testing differences between natural groups 2. Testing the efficacy of public program (e.g., a new recycling initiative) or social policy (e.g., abstinence-only education) Threats to Internal Validity Confounds that provide alternative explanations for a research finding o Controlled by the true experiment…but not always a quasi- experiment The quasi-experiment is a compromise Eight main “threats to internal validity” o a.k.a., confounds or alternative explanations o All of these threats are present in the pre-test/post-test designs that I detest so vehemently The Eight Common Confounding Effects: 1. History: some non-tx event produces change Ex: treating alcoholism; solution: control group 2. Maturation: people change over time Ex: treating depression; solution: control group 3. Testing: people get better when tested again Ex: measuring IQ in an intervention for a learning disability; solution: space them far apart; use equivalent but different measures; control group (what a placebo in a drug trial does) 4. Instrumentation: measure of your DV changes Ex: mechanical changes (scale), or interviewers get better; solution: ensure reliability and validity of instruments, control group 5. Regression to the Mean Any individual score may be invalidly high or low due to chance and not by an accurate measure of performance People at extreme ends of a measure tend to move toward the middle over time Ex: improving memory; baseball batting averages o Regression to the mean can easily be mistaken for a treatment effect! Solution: don’t choose samples based on the “extreme” scores on a pre-test; use reliable measures 6. Subject Attrition: participants are lost over time Ex: people lose interest in study, move away, etc.; final sample then differs from the beginning sample Solution: careful follow-up procedures; statistically compare those who dropped out to those who remain (look at the pre-test to those who dropped out…) 7. Selection: One group systematically differs from the other in ways unrelated to the intervention Ex: one floor of a mental health hospital gets the tx, the other is the control group. Solution: randomization; if not possible, remain aware of this threat and try to match groups to minimize confounds 8. Addictive Effects with Selection: when any of the first 6 threats exists for 1 group but not the other (#1 problem in a quasi-experiment) Limitations of the True Experiment: While true experiments control for these 8 threats, they still can’t control everything o There are a few universal threats to internal validity that can affect even true experiments! Contamination – when groups communicate o Diffusion of treatments – tx transfers to conrols All controlled if groups are kept SEPARATED (not easy) o Resentment and Rivalry Experimenter Expectancy o Controlled through double blind studies More Limitations… Novelty Effects – the “newness” of the tx has an effect rather than the tx itself o Example of pink prison cell study Evaluating Public Programs and Policy One-group pretest-posttest o You know how much I hate this one! No control group at all o Just two observations, with the treatment in the middle o Design diagram: O 1 X O2 A very, very bad choice! o No control group = no definitive conclusions! o All threats to internal validity may be present! Nonequivalent Control Group Design Compare intervention to a “like” control group, but without randomization » “Like”-ness determined by pretest match on the DV If groups truly are comparable, it controls for many threats to internal validity (but not additive effects with selection) Still Vulnerable to “Additive Effects with Selection”! Interrupted Time Series Design Comparison of baseline before and after intervention Better than pretest-posttest » because it uses many observations before (baseline) and after the intervention » Good for one time interventions where long-term effects are desired BUT no control group! So not an ideal design. Time Series with Nonequivalent Control Group - A better option - Multiple pre and post tests - Intervention plus Comparison group - Rules out history and instrumentation effects COMMUNICATING RESEARCH What is different about this course than all the other ones? It’s an application course! Conducting Research - Idea or Question o Hypothesis - Devise way to investigate this idea *this class covers o A method to test your hypothesis - Examine and interpret what you find o Have your results answered your questions? - Tell other people! o The most important! The Research Process (8 Steps) Disseminating Research Presenting at research conferences o Lectures o Poster presentations Colloquia and Lectures o Center for Children and Families Lecture series o DCS: Developmental, Cognitive, and Social/Personality brownbag series Research reports o Peer reviewed – provides validation Writing Research Reports Purpose of the Research Report: o Standardized format for a scientist to explain to readers about their research o Why you did this? Make a case for your study o What did you do? Describe your methods o What did you find? Results of the study o Does any of this matter? Explain the findings and what they mean APA Style o Purpose: Common organizational format Guidelines for how to present research in a clear and unbiased manner Readers and writers alike benefit from a standardized format Structure of the Report: o Abstract Summarize Issue Method Findings Conclusions and Implications 100 – 150 words (…or longer depending on journal rules) Note: Although this is the first section read, it is often written last o Introduction (answers the question: why you did this study) Brief – comprehensive but not exhaustive Do not bore your readers with excess information Why is your study different/important Describe the problem and why it is important Make a specific case for your study, not just a summary of previous research o Selective presentation to support your specific argument based on past literature with citations Present hypotheses Must be justified by the literature o Method Participants How many? o Sample size: N = ? Who/what? What were the characteristics of the participants? o Human Age, ethnicity, SES, other relevant variables o Animal: Species, source, etc. o Archival: Source Measures/Materials What kind of data was collected? o Questionnaires/Survey Data What do they measure? Are there sibscales? Types of scales and reporting Some example items o Observational data Describe where and how Coding systems? What is being coded? Who is doing the coding? Operational definitions of all relevant variables Independent/Dependent Procedure o How were the data collected? Data collection protocol; step by step procedure Recruitment o SONA, flyers, craigslist, etc. IRB approval and informed consent Participant tasks Data Analysis (sometimes) o Only if analytic techniques were particularly complex o Explain and justify the reasons for using such complex analyses Goals of Method Section: Reader knows exactly what happened Detailed enough that other researchers could replicate o Results – what happened Present and summarize analyses – don’t interpret them Include: Purpose of the analysis Type of analysis Summary of analytic results Present confidence intervals, effect sizes, etc. Summary statement Results for each hypothesis plus post hoc tests Tables and Figures Use sparingly o Only when information cannot be conveyed easily or in another way Do not repeat information Refer the reader to the important results in a table/figure Be careful of misleading scale in figures o Discussion – what it means Summarize results briefly Which hypotheses were or were not supported? Were the results as expected or were there surprises? Discuss the meaning/interpretation of results What is the meaning of these specific results? How do these results generalize beyond the sample (if they do)? *samples generalize What do the results mean for science and theory more broadly? Be conservative; don’t go beyond the data Causal statements Excessive theorizing Limitations: All studies have them! Note limitations Explain how the results can be generalized and how they can’t Discuss how future research might correct current limitations Purpose of discussing limitations? What next? Conclusions and Implications Brief (2 – 3 paragraphs at most) Broad importance of the study Application and implications for future research o References List everything you cited! Be careful of citing presentations, dissertations, and non peer- reviewed material o Why? o Your work may be seen as weaker Flaws in theory or ability to find solid support for your hypotheses Full Circle Explained purpose and rationale of study based on previous literature, and outlined hypotheses to be tested o Intro Detailed how the study was conducted in order to answer the questions posed in the intro o Methods Describes what was found using the data gathered through procedures outlined in the Methods o Results Interprets the Results and relates them to current knowledge and the real- world o Discussion
Are you sure you want to buy this material for
You're already Subscribed!
Looks like you've already subscribed to StudySoup, you won't need to purchase another subscription to get this material. To access this material simply click 'View Full Document'