New User Special Price Expires in

Let's log you in.

Sign in with Facebook


Don't have a StudySoup account? Create one here!


Create a StudySoup account

Be part of our community, it's free to join!

Sign up with Facebook


Create your account
By creating an account you agree to StudySoup's terms and conditions and privacy policy

Already have a StudySoup account? Login here


by: Aliyah King


Aliyah King
GPA 3.97


Almost Ready


These notes were just uploaded, and will be ready to view shortly.

Purchase these notes here, or revisit this page.

Either way, we'll remind you when they're ready :)

Preview These Notes for FREE

Get a free preview of these Notes, just enter your email below.

Unlock Preview
Unlock Preview

Preview these materials now for free

Why put in your email? Get access to more of this material and other relevant free materials for your school

View Preview

About this Document

Class Notes
25 ?




Popular in Course

Popular in Dance

This 9 page Class Notes was uploaded by Aliyah King on Saturday September 12, 2015. The Class Notes belongs to Dance 40 at University of California - Irvine taught by Staff in Fall. Since its upload, it has received 60 views. For similar materials see /class/201918/dance-40-university-of-california-irvine in Dance at University of California - Irvine.


Reviews for STU WKSHP IN MOD I


Report this Material


What is Karma?


Karma is the currency of StudySoup.

You can buy or earn more Karma at anytime and redeem it for class notes, study guides, flashcards, and more!

Date Created: 09/12/15
A Solution to Plato39s Problem The Latent Semantic Analysis Theory of Acquisition Induction and Representation of Knowledge Or a really longtillelo an equally longr Winded article Thomas Landauer and Susan Dumais L A big problem 0 Inductive Paradox How do we know so much given how little information there appears to be in the world This is the Poverty of the Stimulus broadly construed Modern theories of knowledge acquisition We must accept some constraints that greatly narrow the solution space of the problem that is to be solved by induction Plato the knowledge is innate we simply need contemplate and infer from hints that we encounter in the world LampS Focus on Learning Vocabulary oThe Problem If we know 40000 10000 words by age 20 we would need to have learned 715 words every day for 18 years beginning at age 2 Question Do you know even one more word today than you did yesterday Do you recall learning 7 words a day every day every week Wouldn t this kind of learning imply effort on your part Wouldn t you remember how hard it was Or perhaps you cannot remember perhaps you learned the words without deliberate effort at apprehension Maybe there is a computational model that can with some specified constraints show how people learn vocabulary at the astonishing rate mentioned above The Inductive Value of Dimensionality Optimization Problem Jack and Jill are on the phone Jack tells Jill I can see three houses A B and C A appears to be 5 units from B and C B and C are about 8 un ts apart Using these estimates Jill plots the houses as a triangle Then Jack says the houses are on the same street This new constraint enables Jill to correctly plot the relationship a n between the three houses Whereas knowing the distance between three objects sets up the first model adding one more constraint enables reduction to one dimension while still conveying the necessary information Semantic similarity bw 2 words Hovv dirnensionaiity appiies to word reiat ons 1 The distance between words is inyerseiy reiated to their sirniiarity 2 We wouid Want to parsetext into Windows so tnat we capture discursive context words tnat appear in the sarne Window or discourse phrase sentence paragraph etc tend to cornerrorn nearby iocat ons in sernant c space 3 Tnis wouid aiiow us to estirnate the reiatiye sirniiarity or any pair or words by observing the reiatiyerreouency or their ioint occurrence in suc Windows 4 But rnany words do not appear together directiy even though tney may be sernant caiiy reiated How to account rortnis indirect reiation7 e g purplelavender overweightcorpulent use oniy one Sentence b Gearsbrakes eraseriead parts or an obiect car pencil 9 r m u m O comfortable friarl where is my lord I do remember well where I should be And there I am Where is my Romeo LSA experiment g3 iuhi ekt s a espearean Input Romeo 0 76 maw gm 0 73 corne y 0 73 playwrights 0 73 Shakespeare 0 71 rarna 0 71 actors 0 70 theater 0 70 burroonery 0 70 soiiioquy pia s 0 70 y 0 70 actor 0 70 narniet 0 69 rnacbetn 0 67 playgoing 0 67 theatr cai Database Matters Heart DB heart disease beat courage heart 1 0 31 0 16 0 05 disease 031 1 001 004 beat 0 16 0 01 1 0 02 courageO 05 0 04 0 02 1 Literature DB heart disease beat courage heart 1 0 26 0 59 0 28 disease 0 26 1 0 23 beat 0 59 0 23 1 0 39 courage 0 28 0 37 0 39 1 But can LSA really get subtle distinctions Not even ten ears ago you could buy a house forfifty thousand dol ars Even ten years ago you could not buy a house forfifty thousand dollars 0 LSA calculates cosine of these sentences as 1 O or identical 0 Exarngle 2 0 1 Manchester United is a soccer team 95 0 2 A soccertearn united Manchester 0 3 United a soccer team defeated Manchester 1 O Solution Take all local estimates of distance into account at once Selecting appropriate dimensionality for pairwise estimates will be critical to achieving correct results based on mutual constraints Technical overview Word meanings are represented as vectors in if dimensional space Estimates of airwise meaning similarities and of similarities among related pairs never ob together can be improved if fitted simultaneously into a space of the same if dimensionalit dea is similar to factor analysis or multidimensional sca ing MDS LSA in Psychological Terms 0 Words exist as points in high dimensional space 0 Sender chooses words located near each other when generating a string In a s ort time window contiguities in the output of words reflects closeness in the sender s semantic space eceiver can make firstorder estimates bw pairs by their relative frequency 0 occurrence in t e same temporal context eg a paragraph Receiver can reconstruct sender s space by estimating similarities between observed and unobserved words ie reconstruct dimensionality sender used How the LSA Model Works 0 Psychological similarity between any two words is reflected in the ay the cooccur in small subsamples of languagethe source of language samples produces words in a way that ensures a mostly orderly stochastic mapping between semantic similarity and output distance The model then fits all of the pairwise similarities into a common space of high but not unlimited dimensionality 0 1 Word frequency in a particular context transformed into logfrequency o Compressive function yields a Spacing effect association of A a B is greater if both appear in 2 different contexts tnan if they appeartvvice in the one context 0 2 All cell entriesfora given Word are divided by the entropy fort at Word Result Makes rirnar e informative relation bvv the entities rather than the rnere fact that they occurred together o lnverse entro rneasure estirnates degreeto Wnicn opserVing occurrence of a component specifies What context it is in The larger entropy the iess inforrnation its observation transrn ts about the piaces it nas occurred so the iess usagede ned meaning it acquires an conversely the iess the meaning of a particuiar context is dete rnined bythe Word DJ tn tn 0 Q 3 E 8 o a tn 0 3 LSA model may be similar to associative processes in inlormation retrieval Goal Retr eve tne textsirom memory that person has in mind Question Do we remember things 1 because we think that they are similar or 2 because there is a general logic in the informat on is processed that makes them similar LSA offers a condensed representaton of the relationship between data points by capturing nignerrorder associations lt a particularstimulus X e g a word has been associated with some other stimulus V by beingfrequentlyfound in ioint context i e contiguity and Y is associated with Z then the condensation can cause X and Z to have similar represen ations However the strength of of the indirect XZ associat on depends on much more than a combination of the strengths of XV and YZ lt also depends on the relation of each of the stimuli X Y Z to every other ent ty in the space AXA induction of a latent higher order similar ty structure among representations of a large collection of events LSA adjusts with new input Updating occurs throughout the space since each object is related in some way more or less similar to every other The relation between any two representations depends not only on direct experience with them but with everything else ever experienced LSA may also be analogized to a Three Layer Neural Net Layer 1 One node for every word t Layer 3 One node for every text window ever encountered Layer 2 Several hundred nodes number to be determined This number is one which maximizes accuracy in a least squares sense with which activating any Layer 3 node activates the Layer 1 nodes that are its elementary constituents a pattern of activation across Layer 2 nodes Singular Value Decomposition SVD I SVD is a linear rnetnodfor decomposing a matrix into independent principal components factor analysis is an example oftnis I From Appendix Fundamental proof of SVD shows that there always exists a decomposition of this form sucntnat matrix rnultiplicat on oftne tnree derived matrices reproducestne original matrix exactly so long ast ere are enough factors wnere enough is alwa s less than or equal to the smaller of the number of rows or columns of the original matrix Cunlext Testing LSA Four Questions I Can a simple linear model acquire knowledge of humanlike word meaning similarities given a large amount of natural text I Would its success depend strongly on the dimensionality of its representation I How would its rate of acquisition compare with a human reading the same amount of text I How much knowledge would come from indirect references that combine information across samples vs direct access from local contiguity Test 1 TOEFL Performance 0 LSA model first trained on 4 6 mill on words from Grolier s Academic American Encyclopedia 30743 art cles parsed as first 2000 characters or entire text entry gt 151 words a long paragrap 0 Text data cast as 30743 columns where each column represents one tegtlt sample 60768 rows each row represents a unique word type that appeared in at least2 samples Cells contained reduency With wh ch a particular word type appeared in a particular tegtltt sample 0 SVD performed and 300 dimensions retained 0 80 retired items from TOEFL Test of English as a Fore gn Language 0 Choice 1 stem problem word and four cho ces Need to choose one most similar to the stem word LSA Performance I TOEFL Test I Model 515 out of 80 644 correct normalized to 525 when accounting for guessmg I Actual foreign applicants to US colleges 51680 645 52 7 corrected I LSA appears to mimic human performance I Question Do foreign students know as much English as represented in 46m words from an encyclopedia Q2 Dimensionality 0 Correct dimensional ty is critical to success how many dimensions to re ain7 nsy El 3 A 0 2 dimensions retained 13 correct 0 No reduction all dimensions 16 correct 0 quest on We can f gure out t e correct dimensionality through tr al and error rhow Proportion canon an Synunynl Tesl w ld as igningthe correct dimensionality work in real life7 And do different quot39 i 10 um Inoo 10050 datasets reduire different d i me nsions Number nl Dimension in LsA log Q3 Learning Rate I Caveat LSA model learns similarities between words as units not for their syn actic grammatical properties spelling sounds morphology etc LSA is also not concerned with production I How well does LSA learn when compared with children at various age levels Assume a range of 715 words learned per day through high school Children have to learn words b reading sincet ey now more wor s d 0 HS than available in speech spoken vocab Estimated to be 25 of print vocab Plus there is very little direct instruction Learning rate estimates LSA to the Rescue 0 Children learn about one new word eve flve 0 Hypothesis Children rely on indirect as Well as ParagraPhS baSed one eStlmateS of reading Wei direct learning LSA captures the indirect portion reading Speed How do children learn7 0 Target Given text input similar to what a child Expel me tg to mpmve eammg receives LSA should learn close to 10 words per o Jenkins Stein Wood51984 5tr graders read paragrapns day thus accountingfor natural rate containing 18 low requency Words x eacn over Several ays When tested fore cnoice defl ltlo students scored Implicit Idea Learning about a wordis meaning between 510Corre0 from a tex ual encounter de ends on knowtng the o Elley 1989 Nagy 1985 Learn by reading exposure only meaning of the other wordsii LSA also captures this notion The re uced dimensional vector for a word is a linear combination of information about all other wor s 5 vvor s per day 50 paragrapns at a rate of 05 learned h o Conclusion if tnese rnetnods don t work how do chlldren acouire tneir vocabulary7 Simulating a school setting Other issues 0 Assume a child has read 38m words equivalent to 0 Does context window size matter LampS control for 25000 0f enCyCIOPEdl S mPIES window shrinking it from 2000 characters to 500 o LampS estimated that direct effect was 00007 words Ram s Were 3W the same n TOEFL test 53 gained per word encounter vs 01500 words VS 52 0 Bags of words problem 0 d words gaine j per paragratpilix 5toplaoragradphs 0 LSA ignores grammar In theory LSA could learn rea per y on average y a S u en wor S from nominally scrambled sentences that in fact do Ieamw a day ot make sense 0 Thus LSA appears to account for the 10 words learned per day Summary of Vocabulary Simulations 0 1 LSA learns word similarities at level similar to moderately competent English readers 0 2 About 75of word knowledge is due to indirect computation 0 There is enough information in language learners uire vocab tests LSA solves Plato s problem Lexical knowledge Reference versus usage Word meaning involves ootn usage and reference Use words lri context Referto ldeally an idea concept semantics L A nas a more narrow interpretation of reference words refer only to otner words and to sets of words Pemaps reference can be seen in tnis i gnt in LSA word meaning is enerated by a statistical process operating o samples o ata meaning is fluldJarld one person s usage and referent for a word is sligntlv lffererll from tne next person s tnat Orle s understanding of a word cnanges witn ime Garbage ln Garbage Out 0 Cont d But what if someone were to write gibberish say LSA autoimported text from the Web No one can understand what the batch of text samples means but its inclusion affects all the words in the dimensional space The point people need to use words correctly problem 0 production and at least somewhat attached to conventional meanin LSA cannot discriminate proper usage and proper relationships without grammar Real World Referents oan LSA learn pragmatic reference7Thl5 would require added dimensions for context visual perceptual etc LampS argue tnat Qulrle s gava ai problem can be solved insorar as knowledge of iavagal wou d be solved once tne leamer tassuming an L8 model is exposed to enougn text of tne anguageto be acquired sucnt at tne relationsnips petween tne words encountered constrain tne target word tnrougn lrldll ect assoc atlorl roplem But for someone wno needs to xnow now and nas zero vocabulary of tne target language Wouldn t it seem lixe tnis is a nign cost wav to learn gavagai tnat vou naveto absorb a lot7 More generally can LSA worx witn meager input to simulate age 2 eamer7 How LSA zeroes in on meaning 0 Even in the absence of external referents LSA can by resolving the mutual entailments of a multitude context similarities lead to agreement on the usage of a word or make a word highly similar to a context even if it never occurs in that context Contextual Disambiguation Skilled readers disambiguate as they go 0 How does LSA handle Words that have multiple meanings7 E g line fly bear man 0 For example With the online LSA online at lsa Colorado edu you have to choose the database first before running LSA Othervvise you may not get m aningful results humans load full parameters which then disambiguate7 How would ou accountfor lowfrequency Words that you encounter or the first time in a given Ex The player caught the high fly to leftfield 0 LSA can capture local meaning if 0 1 The input is a 3way matrix of word phrase paragraph The phrase vector holds the local context Or 0 2 A local rerepresentation process occurs such that a secondary SVDlike condensation or some other mutual constrain satisfaction process using the global cosines as input that would have more profound meaningrevision effects than simp e provision o gt Only components of meaning that it shares with the context after averaging comprise its disambiguated meaning ConstructionIntegration Theory How to use LSA to represent the meaning of text strings l e sentences or paragraphs7 0 Goal Estimate a measure of coherence that derives from the overlap of meaning in sentences as they build on each other 0 Experiment 27 encyclopedia articles subjected to SVD in 00 dimensions Eac sen ence in experimental paragraphs represented as the average of the vectors it contained 0 Paragraph coherence measured as average cosine bvv successive sentences Figure shows measured comprehensibility for LSA r93 and word level r17 LSA r 09 9 E a n I p m Ward lavI r m1 0 Casino Eeweell Sentences a 3 p s E Cnmprehnnslon iv LSA Simulation of Till et al 1988 I Theoretically I 1 larger cosines bw homographic word and its related words than between it and control words I 2 vector average of the passage words before homographic word should have higher cosine with contextrelevant word related to than contextirrelevant word I 3 vector average of words in a passage should have higher cosine with the word related to the passages inferred situational meaning than to control words I 28 pairs of passages 112 target words Cosines reveal LSA accomplishes correct inference Table 1 L171 Simulation 0f ml m um i D 39 1quot Sam Hugh39s Inference nrgclx Right Wrnng Right Wmng Unrclmcd anc A E C 1 normal Homogmph nlnn 20 21 09 05 07 pvx A m lt mom r z 39 ull pamgc wun hanmgmph 24 2i 2i l5 mos ss M pvs Canons pvs c 1 Full passage Wuhan z 59 immogmpli 21 is 21 u 15 p vs A ans p u c a bum p vs a nm 1 4x 1 59 z 2 46 Problems with Target Selection I Were the target homographs handselected Does it matter whether words were prescreened for variable meaning Can LSA find the homographic words on its own or is this process a secondary step after running LSA once and getting ambiguous results I How would LSA work with compoundform words in Enlish Chinese Navajo


Buy Material

Are you sure you want to buy this material for

25 Karma

Buy Material

BOOM! Enjoy Your Free Notes!

We've added these Notes to your profile, click here to view them now.


You're already Subscribed!

Looks like you've already subscribed to StudySoup, you won't need to purchase another subscription to get this material. To access this material simply click 'View Full Document'

Why people love StudySoup

Bentley McCaw University of Florida

"I was shooting for a perfect 4.0 GPA this semester. Having StudySoup as a study aid was critical to helping me achieve my goal...and I nailed it!"

Janice Dongeun University of Washington

"I used the money I made selling my notes & study guides to pay for spring break in Olympia, Washington...which was Sweet!"

Jim McGreen Ohio University

"Knowing I can count on the Elite Notetaker in my class allows me to focus on what the professor is saying instead of just scribbling notes the whole time and falling behind."


"Their 'Elite Notetakers' are making over $1,200/month in sales by creating high quality content that helps their classmates in a time of need."

Become an Elite Notetaker and start selling your notes online!

Refund Policy


All subscriptions to StudySoup are paid in full at the time of subscribing. To change your credit card information or to cancel your subscription, go to "Edit Settings". All credit card information will be available there. If you should decide to cancel your subscription, it will continue to be valid until the next payment period, as all payments for the current period were made in advance. For special circumstances, please email


StudySoup has more than 1 million course-specific study resources to help students study smarter. If you’re having trouble finding what you’re looking for, our customer support team can help you find what you need! Feel free to contact them here:

Recurring Subscriptions: If you have canceled your recurring subscription on the day of renewal and have not downloaded any documents, you may request a refund by submitting an email to

Satisfaction Guarantee: If you’re not satisfied with your subscription, you can contact us for further help. Contact must be made within 3 business days of your subscription purchase and your refund request will be subject for review.

Please Note: Refunds can never be provided more than 30 days after the initial purchase date regardless of your activity on the site.