Sply Chain ModLogistics
Sply Chain ModLogistics ISYE 3103
Popular in Course
Popular in Industrial Engineering
This 0 page Class Notes was uploaded by Maryse Thiel on Monday November 2, 2015. The Class Notes belongs to ISYE 3103 at Georgia Institute of Technology - Main Campus taught by Staff in Fall. Since its upload, it has received 8 views. For similar materials see /class/234207/isye-3103-georgia-institute-of-technology-main-campus in Industrial Engineering at Georgia Institute of Technology - Main Campus.
Reviews for Sply Chain ModLogistics
Report this Material
What is Karma?
Karma is the currency of StudySoup.
Date Created: 11/02/15
Published in Technology Review JuneJuly 1980 1624 The SeerSucker Theory The Value of Experts in Forecasting J Scott Armstrong People are willing to pay heavily for expert advice Economists are consulted to tell us how the economy will change stock analysts are paid large salaries to forecast the earnings ofvarious companies and political experm command large fees to tell our leaders what the future holds The available evidence however implies that this money is poorly spent But because few people pay attention to this evidence I have come up with whatl call the quotseersucker theory quotNo matter how much evidence exists that seers do not exist suckers will pay for the existence of seersquot One would expect experts to have reliable information for predicting change and to be able to utilize the information effectively However expertise beyond a minimal level is of little value in fore casting change This conclusion is both surprising and useful and its implication is clear Don39t hire the best expert hire the cheapest expert This is not to say that experts have no value they can contribute in many ways One particularly useful role of the expert seems to be in assessing a current situation And although estimates of current status play an important role in forecasting I will deal only with the role of expertise in forecasting change Value of Experts The Evidence Man studies have been done on the value of expertise in a given subject area Most evidence comes from the field of finance but studies have also been done in psychology economics medicine sports and sociology The relationship of accuracy to expertise in a particular field has been measured in various ways education experience reputation previous success andselfidentification Expertise above a very low level and accuracy are unrelated Exhibit 1 and accuracy may even drop after a certain level This minimal expertise can be obtained quickly and easily For example in 1955 R Taft surveyed 81 psychological studies that examined predictions made by experts and nonexperts He concluded that nonpsychologists were more capable of making predictions about people39s attitudes and behavior In one typical study done by Austin Grigg PhD s trainees in psychology and naive subjects undergraduates each listened to 10minute interviews with three clients and then predicted how each client would fill out three different personality tests There was no difference in accuracy between the PhD s and the trainees but both these groups did significantly better than the naive subjects Thus a small amount of expertise was useful More recently Bernard Levy and E Ulman asked professional mental health workers students and people with no mental health experience to distinguish 48 normal people from 48 psychiatric patients by looking at paintings the subjects had done All the subjects39 predictions were significantly better than chance but here accuracy did not correlate with experience Exhibit 1 Some expertise seems to lead to a higher level of accuracy in forecasting change beyond a minimal level however additional expertise does not improve accuracy and there is even some evidence that it may decrease accuracy Forecast Accuracy High Low E Level of Expertise High The performance of experts and novices in fore casting prices of stocks was first examined by Garfield Cox in 1930 He found no advantage for expertise In 1933 Alfred Cowles examined 255 editorials by Hamilton an editor of the Wall StreetJournal who had gained a reputation for successful forecasting During the period from 1902 to 1929 Hamilton forecast 90 changes in the market 45 were correct and 45 were incorrect Cowles also found that a sample of 20 insurance companies did slightly worse in their investmenm than the market averages from 1928 to 193139 16 financial services did slightly worse than the market average from 1928 to 193239 and forecasts in 24 financial publications were slightly worse than the market average over this same period Other studies some done as recently as the late 1970s have reinforced these conclusions Roy Johnson and BF McNeal had 12 health care professionals 7 5 staffpsychologists 6 social workers and a physician predict the length of hospital stay for 379 mental patients over an 18month period The scores ranged from 63 percent to 86 percent correct The professionals with more experience in psychology were no more accurate William Avison and Gwynn Nettler examined predictions in nine public opinion polls from 1959 to 1971 Experts as judged from the amount of schooling were no better at forecasting change In three studies in which quotexpert forecastsquot were more accurate the gain was small A small but statistically significant correlation for a sample of 26 experts in the social and natural sciences in forecasting 123 events in their fields was found by Kaplan Skogstad and Girshick in 1950 Similar findings were obtained in 1976 by Wise who examined 1556 predictions published in the United States between 1890 and 1940 For predic tions related to social technological economic and political changes people with experience in the rele vant field seemed to do slightly better than those outside the field In a 1971 study by Robert Winkler spormwriters did a little better than graduate students and faculty members in forecasting scores of collegiate and pro fessional football games The bookmakers39 fore casts in turn were slightly better than those of the sportswriters Overall the evidence suggests there is little benefit to expertise And because improved accuracy shows up only in large samples claims of accuracy by a single expert would seem to be of no practical value Surprisingly I could find no studies that showed an important advantage for expertise This evidence does not include every area however and further studies may show that the seersucker theory cannot be generalized Another possibility is that researchers find it easier to publish evidence refuting than confirming the common notion that expertise is useful However in light of a 1977 study by Michael Mahoney this possibility seems remote Mahoney asked 75 reviewers to referee a paper Two versions of the paper were presented to randomly selected subsamples of reviewers The papers differed only in the results one version had results favoring the common wis dom of the day and the other refuted it A strong bias was found toward accepting the study that agreedwith a commonly held hypothesis and rejecting the one that contradicted this hypothesis Is Accuracy Irrelevant Assume for a moment that the seersucker theory is true that expertise is useless in forecasting change Is there any rational explanation for why clients continue to purchase worthless information One explanation is that the client is not interested in accuracy but only in avoiding responsibility A client who calls in the best wizard available avoids blame if the forecasm are inaccurate The evasion of responsibility is one possible explanation for why stock market investors continue to purchase expert advice in spite o overwhelming evidence that such advice is worthless The avoidance of responsibility is illustrated in a 1978 study by Joseph Cocozza and Henry Steadman In New York psychiatrists are asked to predict the dangerousness of mental patients patients diagnosed as dangerous are then placed in involuntary confinement Although numerous studies have shown that psychiatrists cannot predict who is dangerous the expert39s diagnosis was accepted by 87 percent of the courts in this study Cocozza and Steadman suggest that their finding may illustrate a belief in magic that secret knowledge of the specialist can control the unpredictable The expert advice seems to relieve the court of further responsibility Cases involving risk and uncertainty seem most likely to lead to avoidance of responsibility An example is provided in a study of longrange forecasts of bed requiremenm for six Michigan commu nity hospitals Clients were satisfied only when the forecasts matched their preconceptions When differences arose the hospital administrators followed their preconceptions anyway ignoring the advice of experts The preconceived forecasts exceeded e experts39 forecasts for five of the six hospitals and led to decisions that resulted in a 50 percent oversupply of beds Ineffective Learning by Seers quotExpertise breeds an inability to accept new viewsquot Laski 1930 The continued inclination for people to consult expert advice has been the subject of much study In 1948 BF Skinner experimented with a pigeon in a cage Food was given to the pigeon on a random time schedule What happened The bird concluded that a counterclockwise movement produced the food since it was doing that the first time food appeared It repeated this behavior whenever it was hungry This initial learning proved to be highly re sistant to change even though it had absolutely nothing to do with the appearance of food In 1958 Lloyd Strickland found that people do a goodjob of simulating pigeons He had subjects act as managers of two subordinates whom I will call Stan and Ned The manager could see Stan39s work and communicate with him easily Communication with Ned was poor However both Stan and Ned produced the same amount and quality of work The manager trusted Ned but thought Stan required constant supervision he had concluded that his management efforts were responsible for Stan39s output In a more recent study Daniel Kahneman and Amos Tversky discussed a ight school training pro gram in which trainers adopted a recommendation from psychologists that they use only positive reinforcement they praised successful work and said nothing otherwise After a time the instructors concluded that positive reinforcement did not work when they praised someone for successfully completing a series of complex maneuvers the trainee would often do worse the next time That happens because learning involves mistakes A student cannot consistently perform well an exceptionally good trial will usually be followed by a more average trial and conversely for an exceptionally poor trial The ight school trainers noticed this phenomenon and attributed it to their actions As a result they quotlearnedquot that what works is punishment for bad behavior because the odds are that the next trial will be better Rewards they concluded just lead to overconfidence ofthe learner In these studies subjects first are assuming that their own actions control the situation This effect shows up even in studies ofgamblers Second they are looking for evidence to support their hypotheses that is they are looking for confirming evidence and avoiding disconfirming evidence This latter phenomenon can occur even in the absence of any notion of control or emotional involvement as shown in the following experiment PC Wason presented subjects with a three number sequence 246 The subjects were told that this sequence had been generated by a rule that the experimenter had in his head The subjects were then asked to learn the rule by generating additional threenumber sequences eg 81012 After each try the experimenter told the subject ifthe new sequence agreed with the rule The subject could generate as many threenumber sequences as she wished39 when she felt confident of the rule she wrote it down The correct rule was quotthree numbers in increasing order of magnitude that is a lt b lt c Despite its simplicity only about 25 percent of the people tested discovered the correct rule Subjects usually selected a hypothesis eg quotadd two to each successive numberquot and looked only for evidence to confirm this hypothesis they did not attempt to refute it In other words most people refuse even to entertain the possibility that they are wrong The story gets worse Subjects who wrote the wrong rule were allowed to try again to generate additional sets of numbers to obtain more evidence About half these subjects continued to search for confirmation for the ru e they had been told was wrong It is not clear whether subjecm failed to accept disconfirming evidence because they were unable or be cause they were unwilling When asked how they would find out whether their hypothesis was wrong however few recognized the need to look for dis confirming evidence by generating a sequence of numbers inconsistent with their hypothesis ls Wason correct that people avoid disconfirming evidence lfyou believe quotyesquot I can present confirming evidence to make you happy If you believe quotnoquot I cangive a prima facie argument and you may get upset at me So I leave it to you to seek disconfirming evidence The evidence just cited however implies why one might expect expertise to reduce accuracy as suggested by the dotted line in the chart on page 20 The greater one s feeling of expertise the less likely that disconfirming evidence will be used Loren Chapman and IP Chapman studied this issue by asking 32 subjects with high expertise to examine data from homosexual and heterosexual subjects The information was contrived so that there were no relationships among variables that previous literature had found to be irrelevant Nevertheless the practicing clinicians saw the relationships that they expected to see which incidentally were the same invalid relationships expected by a group of nonexperts and they had great difficulty seeing valid relationships even when these were dramatic In a related study George Strickler found that although people with high expertise rejected valid disconfirming evidence subjects with much less expertise improved their accuracy by using disconfirming evidence Salvaging the Expert The seersucker theory implies that clients will continue to depend upon experm It is important then to consider whether experts can improve their ability to forecast change The prospecm are not good evidence reviewed by Nisbett and Wilson shows that experm are often unaware of how they make judgmenm and predictions For example a 1964 study by EC Webster showed that decisions in employment interviews are typically made in the first 30 seconds of the interview Moreover the reasons for the decision are not usually understood by the in terv1ewer Still there is hope Detailed instructions for improving judgmental forecasting are provided by Hillel Einhorn and Robin Hogarth and additional suggestions are given in my book LongRange Forecasting Probably the key is to make an active search for disconfirming evidence Without this search disconfirming evidence is often ignored mis interpreted or misused The advice to seek disconfirming evidence is not new it is the principle behind quotobjectivequot scientific experiments Unfortunately it is not often used even by scientists and training does not seem to help In a study using Wason39s 24 6 problem Mahoney and DeMonbreun found that the aversion to disconfirming evidence is just as prevalent among physical scientists as it is among psychologists 1 have recommended an old solution for the problem of finding disconfirming evidence The method of multiple hypotheses first suggested by TC Chamberlin in 1890 can be used to change one s role from advocate of a particular belief to arbiter among various competing viewpoints When using multiple hypotheses disconfirming evidence for one hypothesis could be confirming evidence for another hypothesis Although experts are poor at forecasting change this does not mean that judgmental forecasting is useless However since all available evidence suggests that expertise beyond an easily achieved minimum is of little value in forecasting change the most obvious advice is to hire inexpensive experts Also look for unbiased experts those who are not actually involved in the situation Finally there is safety in numbers Robin Hogarth has suggested using at least three independent experts and preferably six to ten The conditions under which the seersucker theory holds are not well known it may or may not apply to all areas of forecasting However in view of the evidence it seems wise to put the burden of proofupon the experts to show that their expertise in a given area is valuable Further Reading Armstrong 1 Scott LongRange Forecasting From Crystal Ball to Computer New York Wiley Interscience 78 Avison William R and Nettler Gwynn quotWorld views and crystal ballsquot Futures 8 1976 11 21 Chamberlin TC quotThe method of multiple workinghypothesesquot Science 148 1965 754759 Reprinted from Science 1980 Chapman Loren I and Chapman P quotIllusory correlation as an obstacle to the use ofvalid psychodiagnostic signs Journal ofAbnormalPsychology 74 1969 271 280 Cocozza Joseph I and Steadman Henry 1 quotPrediction in psychiatry An Eexample of misplaced confidence in expertsquot SocialProblems 25 1978 265276 Cowles Alfred quotCan stock market forecasters forecast Econometrica I 1933 309324 Cowles Alfred Stock market forecasting Econometrica 12 1944 206214 Cox Garfield V An Appraisal ofAmerican Business Forecasts Chicago University of Chicago Press 1930 Einhorn Hillel I and Hogarth Robin M Confidence injudgment Persistence of the illusion ofvalidityquot PsychologicalReview 85 1978 394 416 Griffith IR and Wellman BT quotForecasting bed needs and recommending facilities plans for community hospitals Medical Care 17 1979 293303 Grigg Austin E quotExperience of clinicians and speech characteris tics and statements ofclients as variables in clinical judgment Journal ofConsultingPsychology 22 1958 315319 Hartsough W Ross quotlllusory correlation and medicated association A findingquot Canadian Journal ofBehavioral Science 7 1975 151154 Hogarth Robin M quotA note on aggregating opinionsquot Organizational Behavior andHuman Performance 21 1978 121129 Johnston Roy and NcNeal BF Statistical vs clinical prediction Length of neuropsychiatric hospital stayquot Journal ofAbnormalPsychology 72 1967 335340 Kahneman Daniel and Tversky A On the psychology of predic tionquot Psychological Review 80 1973 237251 Kaplan A Skogstad AL and Girshick MA The prediction of social and technological evensquot Public Opinion Quarterly 14 Spring 1950 93110 Langer Ellen I and Roth Jane quotHeads 1 win Tails it39s chance The illusion of control as a function of the sequence of outcomes in a purely chance taskquot Journal ofPersonality and SocialPsychology 32 1975 951 955 Laski Harold quotLimitations of the expertquot Chemical Technology 4 April 1974 198 202 Originally from Harper s 1930 Levy Bernard 1 and Ulman E quotJudging psychopathology from paintingsquot Journal ofAbnormalPsychology 72 1967 182187 Mahoney Michael 1 quotPublication prejudices An experimental study of confirmatory bias in the peer review system Cognitive Therapy andResearch I 1977 161 175 Mahoney Michael 1 and DeMonbreun BG quotPsychology of the scientist An analysis of problemsolving bias Cognitive Therapy andResearch I 1977 229238 Nisbett Richard E and Wilson TD quotTelling more than we can lnow Verbal reports on mental processesquot PsychologicalReview 84 1977 231259 Skinner BF Superstition in the pigeon Journal ofExperimentalPsychology 38 1948 168172 Strickland Lloyd H Surveillance and trustquot Journal ofPersonality 26 1958 200215 Strickler George quotActuarial naive clinical and sophisticated clinical prediction of pathology from figure drawings Journal ofConsultingPsychology 31 1967 492494 Taft R quotThe ability tojudge people Psychological Bulletin 52 1955 128