New User Special Price Expires in

Let's log you in.

Sign in with Facebook


Don't have a StudySoup account? Create one here!


Create a StudySoup account

Be part of our community, it's free to join!

Sign up with Facebook


Create your account
By creating an account you agree to StudySoup's terms and conditions and privacy policy

Already have a StudySoup account? Login here

Evaluating Evidence

by: Buster Heller

Evaluating Evidence ISS 305

Buster Heller
GPA 3.91

Norbert Kerr

Almost Ready


These notes were just uploaded, and will be ready to view shortly.

Purchase these notes here, or revisit this page.

Either way, we'll remind you when they're ready :)

Preview These Notes for FREE

Get a free preview of these Notes, just enter your email below.

Unlock Preview
Unlock Preview

Preview these materials now for free

Why put in your email? Get access to more of this material and other relevant free materials for your school

View Preview

About this Document

Norbert Kerr
Class Notes
25 ?




Popular in Course

Popular in Integrative Studies Social Sci

This 18 page Class Notes was uploaded by Buster Heller on Saturday September 19, 2015. The Class Notes belongs to ISS 305 at Michigan State University taught by Norbert Kerr in Fall. Since its upload, it has received 79 views. For similar materials see /class/207752/iss-305-michigan-state-university in Integrative Studies Social Sci at Michigan State University.


Reviews for Evaluating Evidence


Report this Material


What is Karma?


Karma is the currency of StudySoup.

You can buy or earn more Karma at anytime and redeem it for class notes, study guides, flashcards, and more!

Date Created: 09/19/15
October 4 2011 Q The behavior of the workers at the Hawthorne Electric plant is often cited as evidence that A Anecdotal evidence is not very informative B Your expex C The act of observing can strongly alter the behavior of those observed Q ESPN asks viewers to call in to vote for the greatest basketball player of all time In announcing the results the announcer says that their poll was quotnot scientific What makes it unscientific A it wasn t empirical B one can t do science on something frivolus D the observations were not careful or systematic enough Must science always reveal the truth 0 Some eg stanovich says yes 0 quotScientist try to describe the world as it really is as opposed to what our prior beliefs dictate it should be like P 29 Stanovich 0 Given the essentially skeptical nature of scientific inquiry science will quotcontinually challenge previously held beliefs by subjecting them to empirical tests in such a way that they can be shown to be wrong p 26 Stanovich 0 quotPsychology is thus like other sciences in reiecting the idea that people need to be shielded from the truth p29 0 Some behavioral scientists eg Shelley taylor Jonathan Brown Martin Seligman have recently argued that people are sometimes happier and healthier if quotshielded from the truth that is quotignorance can be bliss o For example blaming failure on temporary factors outside your personal control even when they re not tends to prevent depression in the face of failure 0 Others like Stanovich eg Robyn Dawes Albert Ellis argue that effective human functioning usually or always is promoted through accurate knowledge of the world that is quotignorance is not bliss Miniquiz Question Our overview of the question quotdoes science quotreject the idea that people need to be shielded from the truth suggests that in the scientific community A There is agreement that the answer is YES B There is no agreement on an answer C There is agreement that the answer is quotNoquot D There is agreement that this is an empirical question Introduction 0 We ve stressed need for operational definitions to o A satisfy the empirical requirements of testable amp falsifiable sensory experience and o B to do so in a public amp hence replicable way 0 But there might be many ways of operationalizing a concept or variable 0 Eg take the variable from abnormal psychology of depression 0 Q How might we operationalize it Self report Behavioral symptom otology Physiologically Peer reports etc o How do we choose among them what makes one any better than another 5 Problems of measurement Basic concepts A Variable B Measurement Levels of measurement ll Errors of measurement A Introduction 0 Basic Concepts 0 Variable o any attribute which can assume different values among the members of a class of subjects or events but which has only one value for any given member of that class at any time 0 Variables are the way that things can differ and for which we can observe differences Examples Physical variables height weight eye color mass Psychological variables Depression intelligence need for achievement aggression Observations should be recorded so have a precise meaning have the same meaning for all Words often inadequate eg very smart quite smart above average Numbers usually better meaning is usually more precise IQ of 120 vs quotvery smart meaning is usually shared 120 means the same thing to different people Numerical summaries can be Levels of measurement 0 Some numbers carry more information than others 0 There are 4 levels of measurement 0 Nominal or categorical numbers as labels I Eg males1 females2 o Ordinal numbers permitting ordering o Eg 1st place 2nd place 3rd place finisher in a race 0 lnterval numbers permitting computingcomparing differences 0 Eg 0 vs 10 vs 20 degrees Fahrenheit 0 Ratio numbers permitting ratios 0 Eg 0 vs 10 vs 20 degrees Kelvin o Eg 0 vs 15 vs 30 in charitable contributions 0 So all else being equal better operational definitions are those which 0 Result in numeric values permit measurements 0 Result in higher level Miniquiz Question We asserted that numbers are preferable to words in summarizing our observations Which of the following is NOT among the reasons why A Numbers have a more precise meaning B Numbers can express feelings better than words C Numbers have a more widely shared meaning D Numbers can be combined mathematically to summarize our observations lfl use grade point average to measure scholastic abilty im measuring INTERVAL measurement Assume grade is 00 and the highest is 40 A Nominal B Ordinal C Interval D ratio Desirable qualities Measurements that have little error 0 Useful to distinguish 2 kinds of error 0 Random error 0 Low Random Error Random Error Sources of error which bounce randomly around the underlying true value on the variable 0 Eg the quotflutterquot or noise of a speaker o Across repeated measurements they would tend to cancel out so that the mean random error is zero 0 Eg there is random Examples of sources of random error when measuring 0 length 0 Lining up object carefully 0 Angle of eye to instrument 0 Rounding errors 0 Mistakes in recording 0 Scholastic aptitude eg with the SAT 0 Mood o Guessing 0 Errors filling out the answer sheet 0 Mistiming Desirable qualities Low Random Error High Reliability 0 Reliability of a measure is an index of how well random noise has been controlled 0 Perfect reliabilityno noise I Rarely if ever achieved 0 A reliable measure measures something Precisely October 6 2011 ISS 305 Desirable Qualities Low systematic error There are many sources of systematic error or bias including o Experimenter expectancy effects selection biases testing effects demand characteristics 0 Systematic response biases o Moderation response biases o Acquiescence response bias 0 Eg social desirability response biases good not to respond accurately I When a problem I Solutions 0 Increase anonymity eg no names randomized response techniques 0 Add inducements for accurate reporting eg bogus pipeline Cross check self reports eg behaviorally Avoid self reports altogether 0 Physiological measures eg guilty knowledge test 0 Unobtrusive measures webb Campbell et al Note that as a skeptical consumer you need to check if such solutions have been used when the bias is likely Random Response technique 0 What if I wanted to estimate the of gay people at MSU o I give you a wheel like the one on the left You spin the wheel in private Points to one group only you know which I ask quotdo you belong to that group yes or no 0 From distribution of answers I can figure out Gay o If all were gay should get 75 Yes o If none were gay should get 25 Yes 0 So can estimate gay by actual of Yes answers Low systematic errorHigh Validity o Validity of a measure is an index of how well systematic error has been controlled or Validity of a measure is how well your measuring what you want to measure and not something else viz a source of bias Perfect validityno systematic error 0 A measure with much systematic error is invalid 0 Q What s the minimum level What would a measure with quotno validity look like 0 A one which was all bias totally unrelated to measured variable eg 0 IQ with a bathroom scale 0 Ability as a soldier with inquiry on sexual orientation Miniquiz Question A measure which has low validity A Also have low reliability B May have high reliability C Will have high reliability Example 0 Let s suppose its 1993 and you re a parent trying to help your 18 year old daughter decide where to go to college 0 Suppose during your visit to MSU your see this news report sexual assaults 22 felonies and 7misdimeaners o Is this a fair conclusion o Is this a valid measure of MSU s safety for women 0 Sources of bias I Using total number rather than per capita rates of assault 0 How to eliminate this bias 0 Using assaults reported to police rather than some more direct measure o How to eliminate this bias I eg anonymous surveys 0 Ways of establishing Reliability of a measure or scale 0 Eg a scale to measure weight 0 How could I ahow that it has a little random error I Hint science assumes nature is consistent under same conditions nature behaves same and true value of variable weight is same 0 1 Temporal consistency of measurement 0 lfyou could measure the same people at two points in time the greater the reliability the more similar the measurements should be Could look at some measure of the variability of repeated observations I Eg what of IQ testings varies by more than 15 points from one to next 0 Usually assessed by computing a testretest correlation 0 Reliability as temporal consistency of measurement 0 Assumes that people ie true value does not change from first to second measure 0 When is this assumption likely to be a problem 0 Long intervals for most variables lots of time for change I Eg testing weight before and after the Xmas holidays 0 Even very short intervals for unstable traits eg moods skills If there are carryover or testing effects simply being tested at Time 1 affects your true score at immediate time 2 I Eg testing effect 0 Using the same exam two days apart to test the reliablilty of the test as a measure of knowledge of l55305 material I Eg fatigue effect 0 Eg a marathon in the morning and the afternoon to test the reliability of marathons as measure of long distance running ability 0 Ways of establishing Reliability Of a measure or scale using human judges o For example 0 Judging sobriety by police officers 0 Judging gymnastic performance 0 How could we show that random error was low that the observations were reliable o 2 lnterjudge agreement of measurement o If two human judges are measuring the same thing the greater the reliability of theirjudgments the more similar their two measurements should be Assessed with various statistics 0 Ways of establishing reliability of multiitemsummative measure or scale 0 For example 0 SAT as a measure of scholastic aptitude o ISS 305 midterm as measures of knowledge of course material 0 3 Internal Consistency o If a multiitem measure a summative scale is reliable the measures obtained with different items ought to be similar 0 Most common way of thinking about reliability I But only one of several ways of doing so Assessed using various statistics I Eg alternate forms correlation I Partwhole correlation 0 Assessing reliability What s a research consumer to do 0 Best 0 Look for direct empirical evidence of the reliability of the measures as used in the project in question I Eg testtetest internal consistency correlation 0 Next Best 0 Look for indirect evidence of the measures reliability 0 These notes are a hit and miss he goes so fast through the slides and talks so much I have started to put SLIDE in when there is a new slide and usually do if I was unable to finish the notes The notes to start the Final exam or 2nd exam October 18 2011 lss 305 6 Problems of Description Introduction Suppose we want to observe and describe whether herpes patients receiving a new drug lmpervion 39339 Slide How would I report these data to the FDA Q Would I send them a printout of the whole matrix of numbers Q Why Or why not 0 Often one either isn t able or won t want to describe a full set of observations completely O Lack means eg time opportunity space O Target of message lacks capacity O May not be needed in many instances Often sufficient to summerarize large sets of observations Ultimate summarization is to reduce a large set of observations 39339 Slide Q We sometimes summarize a large number of observations with a few descriptive statistics All of the following are good reasons to do EXCEPT A No information is lost in such summarization B people cannon keep too much information in their heads C it is hard to present a large Averages Measures of central Tendency Arithmetic Mean 0 Mean X 11X 12X50073500 0 But what does the Mean actually mean 0 Can use a physical modelmetaphor to clarify O Suppose we have just 5 scores 33112 O Suppose we put the people on a wightless board at the point of their score that is Q If you do the calculatioin you find that the mean2113 35O8 That is also the point that would put the board 9 39339 Slide Properties of the mean Means o Are east to compute o Lend themselves to useful inferences O because means have certain useful statistical properties 0 Are sensitive to every score especially to extreme scores O Generally changing a score will change the mean O Adding or subtracting a score will affect the mean except when O Adding or subtracting an extreme score can powerfully affect the mean gt For example one can insure that all scores but one are below or above the mean with a wsufficiently extreme single score gt If fatigue scores could be more extreme 30 to 30 I A single extra scoregt22 would result in a mean gt30 even though 5 of 6 scores are lt30 I A single extra score lt14 would result in a mean lt30 39339 Slide Q Which of the following is untrue about the arithmetic mean as a measure of central tendancy a IT CAN PRODUCE A SUMMARY THAT IS FAR MORE PRECISE THAN THE The Median or Md o Is the quotmiddlemostquot score 0 Divides the top and bottom half of the observations o Is the 50th percentile O Percentile rank of a score the percentage of all people who scored lower XX percentile is that score for which XX scores below gt So 50 or half score below the 50th percentile What is the median the 50th percentile of the 5 scores show below Whats the 60th percentile in the scores below I More Practice 39339 Slide 9 O O Other properties of the median 0 Q When will meanmedian I A1 Anytime the distribution is symmetric 0 Q First what do we mean by a distribution of scores 0 A1 A graph or plot that summarizes the scores 0 A2 for us we could stack folks up at each score and then draw lines from the tops of each stack O For example in the set of 5 scores below the distribution would look like this O The plot is a picture of the full set of scores or the distribution of scores 0 Then what do we mean by a distribution being symmetric O That there is some point that divides all the scores up into mirror imags of one another O For every scores above that point there 39339 Slide Q Unlike the mean the median is a sensitive to every available score B Less sensitive to extreme cases C More sensitive to extreme cases D none of the above the median acts like The mode or Mo o Is the quotmost popular score the one that occurs the most 0 What is the mode of the distribution below 0 Other even more exotic statistics eg O 39339 Slide Slippery averages 0 Q why would one choose the mean vs the median o A sometimes to create an impression that scores are especially high or low Q Eg See fig on pg 33 of Huff text showing salaries for all 25 employees of a small company gt Note that the book was first published in 1951 when these were typical salaries gt Multiply by 10 to get more current salaries Q Who might prefer to use the mean Q Who might prefer to use the mode 39339 Slide 0 Are NBA players paid a lot Or 0 Whats the average salary in the NBA 0 In 1998 here were the figures O 411 players O Meanarithmetic average2600000 O Median 1300000 O Mode 272250 0 Why so different Q There are a few very highly paid superstars gt In 1998 Michael Jordan made 126 M more than the next highest paid player gt Now Kirk Nowitski makes 164M Q But many players are making at or near the leagues minimum 272250 in 1998 gt In 1998 80 players made less than 300000 Q You get 39339 Slide Q Sun City is suppose to be primarily for older people but a few residents have infant grandchildren living with them lfl were the advertiser trying to convince buyes that the average age of sun city residents is very high which common central tendency statistic would I be LEAST likely to use a the mean b the median C the mode d the hyperactive mode e the hypoglycemic mode 39339 A basic problem is trying to boil all the useful information in a set of observations down to a single summary number 39339 You often lose vital information 39339 For example gt For a diver deciding if it is safe to dive I Two swimming pools may have the same average depth 10ft but differs importantly in how safe they are to dive into 39339 Slide Variability 3920 There are many useful ways of summarizing variability Range O Different between highest and lowest score Q Can be too sensitive to a single extreme score lnterquartile range 9 Difference between the 75th and 25th percentiles 9 Less sensitive to most extreme scores Variance 9 Mean of squared deviation from the mean 9 When is variance0 9 When is it at its maximum Standard Deviation 9 Square root of variance O September 29 2011 N Kerr Scientific evaluation of empirical statements Empiricism what evidence is consideration Empiricism also requires that we evaluate empirical statements in light of all available and relevant observations notjust our own immediate or preferred observations We can t pick and choose which facts ie carefully and well established observations we will pay attention to and which we ignore EG some creation science advocates ignore evidence that o Converging evidence shows age of the earth is gt 4 billion years old 0 The fossil evidence documents the appearance of new species over geological time A statementideatheory eg Publicness Science doesn t rely on observations by any one person at any one time Science assumes that if something occurs once it will occur again if the conditions are the same 0 Our confidence in any reported observation increases with the number of times that others have made the same observation Science only trust and rely on observations which are replicable or repeatable Thus science disregards observations which 0 Only one person has had eg a unique personal experience I Eg single reports of an alien abduction 0 Others have tried but consistently failed to replicate eg cold fusion Pubicness Who s the public Q Who is the public whose observational consensus establishes the validity of observation A Notjust the general public but 0 Those who have the ability trainging skill opportunity and motivation to make good valid Pubicness Peer Review Scientists place high value on having the evidence for any empirical statement carefully evaluated by other qualified trained scientists o Implicit in publication in a professional peer reviewed journal 0 quotHave the findings been published in a recognized scientificjournal that employs some type of peer review procedure The answer to this question will almost always separate Claims from the real thing p 11 stanovich A Scientific AttitudeMindset Empirical statements can be false and observations may differ from person to person This requires a certain way of thinking for science The scientist must always be a skeptic one who habitually doubts questions or suspends judgment Nothing is considered proved in science until all significant doubts have been laid to rest Controlledsystematic observation October 11 2011 lss 305 What s a research consumer to do 0 Least best 0 quotFace validity of the measure I Is it reasonable as a measure of the variable I What are measures with low face validity of 0 Physical attractiveness o IQ o quotValidity by assertion or by authority 0 Someone who should know asserts that the measure is measuring what it is supposed to measure 0 BUT I Fallible I Risky o Eg 1 use symptoms of mental illness as a way to measure demonic possession o Eg 2 use number of interviews as measure of quality of an investigation 0 Eg 3 use tourist numbers as a measure of city safety 0 Eg 4 use parent satisfaction as measure of effectiveness of Head Start 0 A last resort What s a research consumer to do 0 Next best 0 Rational but nonempirical ways of establishing validity 0 Content validity I The content of the measure is a fair representative sample of all that should go into a measure of the variable 0 For example how would one content validate a final course exam I Consensual validity o The agreement consensus of experts that the measure is valid 0 Problems 0 Need to identify legitimate experts 0 Only useful when a consensus exists 0 Best content may not be the content which is easiest to obtain OR to reach agreement on I Eg teaching Whats a research consumer to do Best 1 Concurrent validity o The measure distinguishes between groups known to differ on the variable 0 Example how might Assessing Validity Whats a research consumer to do N Predictive Validity o A valid measure should predict future behavior that should be affected by the variable 0 Examples 0 Racial prejudice IQ 0 Depression Whats the hidden assumption in a predictive validity test that the variable actually does predict 39339 3 Convergent Validity gt A valid measure of variable ought to produce similar scores ie be correlated with other presumably well validated measures gt Such convergent validation depends on what assumption I The validity of the quotwell validated measure


Buy Material

Are you sure you want to buy this material for

25 Karma

Buy Material

BOOM! Enjoy Your Free Notes!

We've added these Notes to your profile, click here to view them now.


You're already Subscribed!

Looks like you've already subscribed to StudySoup, you won't need to purchase another subscription to get this material. To access this material simply click 'View Full Document'

Why people love StudySoup

Jim McGreen Ohio University

"Knowing I can count on the Elite Notetaker in my class allows me to focus on what the professor is saying instead of just scribbling notes the whole time and falling behind."

Allison Fischer University of Alabama

"I signed up to be an Elite Notetaker with 2 of my sorority sisters this semester. We just posted our notes weekly and were each making over $600 per month. I LOVE StudySoup!"

Bentley McCaw University of Florida

"I was shooting for a perfect 4.0 GPA this semester. Having StudySoup as a study aid was critical to helping me achieve my goal...and I nailed it!"

Parker Thompson 500 Startups

"It's a great way for students to improve their educational experience and it seemed like a product that everybody wants, so all the people participating are winning."

Become an Elite Notetaker and start selling your notes online!

Refund Policy


All subscriptions to StudySoup are paid in full at the time of subscribing. To change your credit card information or to cancel your subscription, go to "Edit Settings". All credit card information will be available there. If you should decide to cancel your subscription, it will continue to be valid until the next payment period, as all payments for the current period were made in advance. For special circumstances, please email


StudySoup has more than 1 million course-specific study resources to help students study smarter. If you’re having trouble finding what you’re looking for, our customer support team can help you find what you need! Feel free to contact them here:

Recurring Subscriptions: If you have canceled your recurring subscription on the day of renewal and have not downloaded any documents, you may request a refund by submitting an email to

Satisfaction Guarantee: If you’re not satisfied with your subscription, you can contact us for further help. Contact must be made within 3 business days of your subscription purchase and your refund request will be subject for review.

Please Note: Refunds can never be provided more than 30 days after the initial purchase date regardless of your activity on the site.