### Create a StudySoup account

#### Be part of our community, it's free to join!

Already have a StudySoup account? Login here

# 618 Note 2 for CS 59000 with Professor Qi at Purdue

### View Full Document

## 8

## 0

## Popular in Course

## Popular in Department

This 56 page Class Notes was uploaded by an elite notetaker on Friday February 6, 2015. The Class Notes belongs to a course at Purdue University taught by a professor in Fall. Since its upload, it has received 8 views.

## Similar to Course at Purdue

## Reviews for 618 Note 2 for CS 59000 with Professor Qi at Purdue

### What is Karma?

#### Karma is the currency of StudySoup.

#### You can buy or earn more Karma at anytime and redeem it for class notes, study guides, flashcards, and more!

Date Created: 02/06/15

CS 59000 Statistical Machine learning Lecture 2 man warn Q a anqc purduaedu Fat 2008 Review Polynomial Curve Fitt39ng 0 1 yaww0u71xw2m2wMa E 70ij SumofSquares Error Function A tn 1St Order Polynomial 3rOI Order Polynomial 9th Order Polynomial 1 Ove rfitting e Training 6 Test RootMeanSquare RMS Error ERMS 2EwN Polynomial Coefficients M 0 M 2 1 M 3 M 9 mg 019 082 031 035 101 127 799 23237 0 2543 532183 w 1737 4856831 25 23163930 w 64004226 mg 106180052 w 104240018 0 55768299 w 12520143 Regularization Penalize large coefficient values mm 5 Z ynw tn 511w n21 Regularization lnA 18 Regularization 111A 0 Regularization ERMS vs lnA Training Test 35 25 20 30 IDA Polynomial Coefficients lnAz oo lnA 18 lnA0 103 035 035 013 101 23237 474 005 203 532183 077 006 mg 4856831 3197 005 11 23163930 389 003 w 64004226 5528 002 102 106180052 4132 001 w 104240018 4595 000 u 55768299 9153 000 w 12520143 7268 001 Data Set Size N 10 9th Order Polynomial 1 Data Set Size N 15 9th Order Polynomial Training Data the more the better 9th Order Polynomial Review Probability Theory Marginal Probability 1102 Joint Probability Conditional Probability U pltYyjlxxigt i X Y Probability Theory Sum Rule 0 1 L y 2 T Z 9 7 j L 101 ZpXaZYyJ Product Rule pltXwyjgt 4 39 The Rules of Probability Sum Rule pX ZpX Y Product Rule pX Y pYXPX Bayes Theorem 19X iYPY pYX poo pX I ZPXIYpY Y posterior oc likelihood X prior Probability Density amp Cumulative Distribution Functions A Transformed Densities Expectations Em Emma Em pxf dz Efifiyi Z pxy f 15 2322 Expectation gtm Approximate Expectation N 2 Z f33n discrete and continuous n21 Variances and Covariances mm IE x 1Efv2 1Ef2 1Efrv2 cov33 y Em 33 EM 9 EMH Exyy E5 3Ey c0VX y Exy X EXYT MYTH 1ExyXyT The Gaussian Distribution N ml102 m exp Zi m MP A a 02 W 39 ARM102 gt 0 A OONF 702 1331 Gaussian Mean and Variance 00 Nxma2 xdx y Ex2 00 N sclpmz 512 da 2 02 vara 1Ea2 1E12 a2 The Multivariate Gaussian Nltxlu2gt 275 mill2 exp ltx ugtT2 1ltx m Gaussian Parameter Estimation 1933 Likelihood function NOB I 02 12 pxiua2 Nltnu02 Maximum Log Likelihood Properties of LLML and a IL Curve Fitting Revisited Maximum Likelihood N ptxw HNtnlynaWaB 1 n1 5 N N N 1nptixw 5 221 yznw tn2 3 ln 31n27r 65w Determine WML by minimizing sumofsquares error N Z 953mWML tn2 ZIH 1 BML Predictive Distribution Pti7WMLa6ML N tyWMLv 1 Ii MAP A Step towards Bayes pwia NW0oz11 aM12 exp ngW pwixta 0lt ptixw6pwa E 2 N aw Zyanw tn2 ngW Determine WMAP by minimizing regularized sumofsquares error Bayesian Curve Fitting ptxxt ptawpw xt dw Ntma52c N mac mTS Z qbltasngttn 3 31 ltmTs n1 N T 8 1 aI Zq xnqbxnT n xx 4 n21 Bayesian Predictive Distribution ptic x t N timm 5263 Model Selection via CrossValidation E run 1 run 4 Curse of Dimensionality 232 172 gt D 13 331 133 C81 Curse of Dimensionality Polynomial curve fitting P 1469 6 D D D 9X7W we Z w z Z Z wij z ilij i1 i1 j1 z E wijkximjxk D I 1 D 1J Gaussian Densities in higher dimensions Decision Theory Inference step Determine either ptix or pxt Decision step For given determine optimal w Minimum Misclassification Rate A H pmistake pX 6 72162 px E R2C1 px C2 dx px C1 dx R1 R2 Minimum Expected Loss Example classify medical images as cancer or normal Decision cancer normal cancer lt 0 1000 gt C 5 normal 1 0 Minimum Expected Loss ijpx Ck dx Regions Rj are chosen to minimize EiL Z ijpwkix k Reject Option menus p62m 10 reject region Decision Theory for Regression Inference step Determine pXt Decision step For given make optimal prediction for w Loss function EL Ltyxpxtdxdt The Squared Loss Function Minimize EL t2pxt dx dt The Squared Loss Function Minimize EL t2pxt dx dt Entropy Important quantity in coding theory statistical physics machine learning Entropy Coding theory discrete with 8 possible states how many bits to transmit the state of All states equally likely 1 1 8 X g log2 g 3 bits x a b C d e f g h M 6 i 6 614 6 614 6 code 0 10 110 1110 111100 111101 111110 111111 1 1 1 1 1 1 1 1 4 1 H 1 1 1 1 1 m 2 0g 2 4 Og2 4 8 0g 8 16 OgZ 16 64 0g 64 2 bits averaecodele th 1gtlt11gtlt21gtlt31gtlt44gtlt1gtlt6 I1 g g 2 4 8 16 64 2 bits Entropy In how many ways can Q identical objects be allocated P bins W H77 Entropy maximized when Vi pi i E Entropy probabilities 05 025 H 177 probabilities 05 0 J Ln H 309 EDDUDUHUHHHHHHHUUUDDMD Differential Entropy Put bins of width f along the real line 1310 ZpxzAlnpxi p1npdx Differential entropy maximized for fixed 02 when WI Mirirwg in which case g 1 11127i390392 Conditional Entropy HyX pyxlnpYXdy dx HX y HMX HX The KullbackLeibler Divergence KLltqugt pltxgt1nqltxgtdx pltxgt1npltxgtdx N KL29q 2 Z 1nqXn0 1npXn KLODHQ 2 0 KLPHQ KLUJHP How to prove KLpilq 2 0 Hint Convex function amp Jensen s inequality

### BOOM! Enjoy Your Free Notes!

We've added these Notes to your profile, click here to view them now.

### You're already Subscribed!

Looks like you've already subscribed to StudySoup, you won't need to purchase another subscription to get this material. To access this material simply click 'View Full Document'

## Why people love StudySoup

#### "There's no way I would have passed my Organic Chemistry class this semester without the notes and study guides I got from StudySoup."

#### "I bought an awesome study guide, which helped me get an A in my Math 34B class this quarter!"

#### "Knowing I can count on the Elite Notetaker in my class allows me to focus on what the professor is saying instead of just scribbling notes the whole time and falling behind."

#### "Their 'Elite Notetakers' are making over $1,200/month in sales by creating high quality content that helps their classmates in a time of need."

### Refund Policy

#### STUDYSOUP CANCELLATION POLICY

All subscriptions to StudySoup are paid in full at the time of subscribing. To change your credit card information or to cancel your subscription, go to "Edit Settings". All credit card information will be available there. If you should decide to cancel your subscription, it will continue to be valid until the next payment period, as all payments for the current period were made in advance. For special circumstances, please email support@studysoup.com

#### STUDYSOUP REFUND POLICY

StudySoup has more than 1 million course-specific study resources to help students study smarter. If you’re having trouble finding what you’re looking for, our customer support team can help you find what you need! Feel free to contact them here: support@studysoup.com

Recurring Subscriptions: If you have canceled your recurring subscription on the day of renewal and have not downloaded any documents, you may request a refund by submitting an email to support@studysoup.com

Satisfaction Guarantee: If you’re not satisfied with your subscription, you can contact us for further help. Contact must be made within 3 business days of your subscription purchase and your refund request will be subject for review.

Please Note: Refunds can never be provided more than 30 days after the initial purchase date regardless of your activity on the site.