### Create a StudySoup account

#### Be part of our community, it's free to join!

Already have a StudySoup account? Login here

# 247 Class Note for CS 59000 with Professor Qi at Purdue

### View Full Document

## 13

## 0

## Popular in Course

## Popular in Department

This 31 page Class Notes was uploaded by an elite notetaker on Friday February 6, 2015. The Class Notes belongs to a course at Purdue University taught by a professor in Fall. Since its upload, it has received 13 views.

## Similar to Course at Purdue

## Reviews for 247 Class Note for CS 59000 with Professor Qi at Purdue

### What is Karma?

#### Karma is the currency of StudySoup.

#### You can buy or earn more Karma at anytime and redeem it for class notes, study guides, flashcards, and more!

Date Created: 02/06/15

CS 59000 Statistical Machine learning Lecture 9 Yuan Aura j PUMME C3 3ampth 23 200 Outline Review of model comparison and Fisher s linear discriminant Linear classification Discriminant functions Probabilistic generative models Probabilistic discriminative models Likelihood Parameter Posterior amp Evidence Likelihood and evidence palm fpltvlwmigtpltwwigtdw Parameter posterior distribution and evidence PltDiW7MiPWiVli MPHIi 39 19Wi737Mi I Evidence penalizes overcomplex models Given Mparameters InpD 2 1npDlWMAp A1 In lt Maximizing evidence leads to a natural trade off between data fitting amp model complexity MD M1 Evidence Approximation amp Empirical Bayes Approximating the evidence by maximizing marginal likelihood Wit plttlw6gtpltwltamlet dwdad A Wit 2 palta plttlw6gtpltwlta3gtdw Where hyperparameters a maximize the evidence Wlm Known as Empirical Bayes or type2 maximum erhhood Model Evidence and CrossValidation Fitting polynomial regression models 1 0 2 4 7 8 M Rootmeansquare error Model evidence Classification Approaches Discriminant functions Directly assigns an input vector in a specific class Probabilistic generative models Model the data generation process MXICk and use Bayes rule PXCIPCA MK 39 Probabilistic discriminative models Model the class conditional densities pltCtlx directly PCle Distance from X to decision surface Fisher s Linear Discriminant find projection to a line st samples from different classes are well separated Example in 20 bad line to project to good line to project to classes are mixed up classes are well separated A na39ive choice of separation measure Let in and z be the means of projections of classes 1 and 2 Let u and 2 be the means of classes 1 and 2 39 i111 2l seems like a good measure Problem of Na39ive Separation Criterion How good is l 1 2l as a measure of separation The larger I111 7 2 the better is the expected separation 1 Hui a2 I A i 1 12 toooI I I gt I the vertical axes is a better line than the horizontal axes to project to for class separability however 1 tzl gt l 1 quot 172i Scatter of Data in Each Class Define their scatter as s z z M i1 Thus scatter is just sample variance multiplied by n scatter measures the same thing as variance the spread of data around the mean scatter is just on different scale than variance 0 0 o o larger scatter 0 smaller scatter if Solution Normalization by Scatter Fisher Solution normalize lil1 ml by scatter Let y vtxi ie y s are the projected samples Scatter for projected samples of class 1 is 12 2Yi 12 y ie Class 1 Scatter for projected samples of class 2 is g5 2yi39 22 y 5 Class 2 Fisher Linear Discriminant Thus Fisher linear discriminant is to project on line in the direction vwhich maximizes want projected means are far from each other r H 111 U2 2 J V s a want scatter in class 1 is as want scatter in class 2 is as small as possible ie samples small as POSSI39be ie samples of class 1 cluster around the Of Glass 2 cluster around the projected mean A projected mean z Cost Function 2 M Z1 172 s1 52 If we find vwhich makes Jv large we are guaranteed that the classes are well separated projected means are far from each other 111 172 ulu odo gt H4 small g1 implies that small 2 implies that P 0190t9d samples 0 projected samples of class 1 are Clustered class 2 are clustered aI OUNd PFOI39GCted mean around projected mean Within Class and Between Class Scatter Matrices Define the separate class scatter matrices S1 and 2 for classes 1 and 2 These measure the scatter of original samples x before projection 51 2 xi 1xinu1t xe Class 1 2Xi1u2xi1u2t xie Class 2 Now define the within the class scatter matrix SW S1 S2 S2 Define between the class scatter matrix SB 2 11 2u1 2 t Generative eigenvalue problem 2 t M 3 73 3 s 52 v SWV Minimize Jv by taking the derivative wrt ve setting it to O gt st zswv Fisher s Linear Discriminant SBV ZSWV If SWhas full rank the inverse exists can convert this to a standard eigenvalue problem SWSBV 1v But SBX for any vector x points in the same direction as g 2 a 58x 1lu2zu1u2tx 11 u2 111 112 Thus can solve the eigenvalue problem immediately V Sm we IJ O 6 Projection that maximizes mean separation u w 391 FLD Projection Perceptron Realvalue inputs x weights w 1 otherwise Generalized Linear Model h inhr ze Z WTgbntn REAl where Mdenotes the set of all misclassified patterns Stochastic Gradient Descent Wet1 We UVEMW WW mint o x u K 39 39 o o n o u v M O a M H 39 V 39D o n H u n 39 a 4m 4 o w 39 u t 4 a quot H quot 71 4H 1 M I 4 7 1 quot 5 n 5 39 71 4w 5 n n x Probabilistic Generative Models PltXiC1PC1 13XiC1PC1 19XiC2PC2 1 1 eXp a 0a PC1iX 19XiC1PC1 19Xic219c2gt a111 Gaussian ClassConditional Densities Conditional densities of data 1 1 1 exp Mk12 1X The posterior distribution for labelclass 17C1iX 0WTX 100 W 2 1H1 H2 1 T 1 1 T 1 M61 quot2 72 v 2 t 1 O 2H1 1 2M2 2 11C2 Maximum Likelihood Estimation Let W31 W so that pC2 1 7r tn 1 denotes class C1 and tn 0 denotes class C2 Likelihood function N Pm 39a H1 H21 2 I H WJVX W1 2Vquot 1 7TNXn pj2 EMF t n 1 Maximum Likelihood Estimation N 1 A N 7T Y Z tn 1 Z l V 77 1 AT 4 T1 JV AI 1 N 1 N 1 Z tuxquot 2 N Z tnxn in 711 1 r S1 Z Xn 11Xn i 11 quot nECl 1 m S2 N2 Z X71 H2Xn M2I REC2 Discrete features Na39ive Bayes classification D mm mm 11 D aux Z 13 111 m 1 171lI11 190 111pCf i1 Probabilistic Discriminative Models Instead of modeling XlCLlt Model CHX directly Logistic Regression Let Mamas m a ww PC2V 31 39PC1W Ukehhoodfunc on N 39 1 71 MW H 1 ya n 1 gm Maximum Likelihood Estimation N 111ptiw Z tn 111312 1 tn 1111 171 711 N VEw 2m tum n 1 Note that LU 01 a la NewtonRaphson Optimization Let H denote Hessian matrix W11ew Wold N VEw Zm i an tn lt1gtTqgtw ltIgtTt n 1 N H VVEW 2 any 2 ltITltIgt 111 W11ew Z Wold 7 T 1 Tqwold 7 o l orlq t t converges in one iteration for linear regression

### BOOM! Enjoy Your Free Notes!

We've added these Notes to your profile, click here to view them now.

### You're already Subscribed!

Looks like you've already subscribed to StudySoup, you won't need to purchase another subscription to get this material. To access this material simply click 'View Full Document'

## Why people love StudySoup

#### "There's no way I would have passed my Organic Chemistry class this semester without the notes and study guides I got from StudySoup."

#### "I used the money I made selling my notes & study guides to pay for spring break in Olympia, Washington...which was Sweet!"

#### "Knowing I can count on the Elite Notetaker in my class allows me to focus on what the professor is saying instead of just scribbling notes the whole time and falling behind."

#### "Their 'Elite Notetakers' are making over $1,200/month in sales by creating high quality content that helps their classmates in a time of need."

### Refund Policy

#### STUDYSOUP CANCELLATION POLICY

All subscriptions to StudySoup are paid in full at the time of subscribing. To change your credit card information or to cancel your subscription, go to "Edit Settings". All credit card information will be available there. If you should decide to cancel your subscription, it will continue to be valid until the next payment period, as all payments for the current period were made in advance. For special circumstances, please email support@studysoup.com

#### STUDYSOUP REFUND POLICY

StudySoup has more than 1 million course-specific study resources to help students study smarter. If you’re having trouble finding what you’re looking for, our customer support team can help you find what you need! Feel free to contact them here: support@studysoup.com

Recurring Subscriptions: If you have canceled your recurring subscription on the day of renewal and have not downloaded any documents, you may request a refund by submitting an email to support@studysoup.com

Satisfaction Guarantee: If you’re not satisfied with your subscription, you can contact us for further help. Contact must be made within 3 business days of your subscription purchase and your refund request will be subject for review.

Please Note: Refunds can never be provided more than 30 days after the initial purchase date regardless of your activity on the site.