Computer Vision EECS 442
Popular in Course
verified elite notetaker
Popular in Engineering Computer Science
This 240 page Class Notes was uploaded by Ophelia Ritchie on Thursday October 29, 2015. The Class Notes belongs to EECS 442 at University of Michigan taught by Silvio Savarese in Fall. Since its upload, it has received 19 views. For similar materials see /class/231518/eecs-442-university-of-michigan in Engineering Computer Science at University of Michigan.
Reviews for Computer Vision
Report this Material
What is Karma?
Karma is the currency of StudySoup.
Date Created: 10/29/15
I EECS 442 Computer vision Shape from reflections Special ropic A taxonomy Recovering the shape of an obiect Visual cues texture o shading co shadows quotV A taxonomy Recovering the shape of cm obiect Visual cues texture o shading o contours o A taxonomy Recovering the shape of an obiect Visual cues o shading o contours o shadows A taxonomy Recovering the shape of an obiect Visual cues texture o o contours o shadows gov A taxonomy Recovering the shape of an obiect Visual cues texture o shading o contours o shadows Number of observers multiple views camera A taxonomy Recovering the shape of an obiect Visual cues texture o shading o contours o shadows Number of observers monocular 0 multiple views camera 2 Recovering the shape of an obiect Visual cues texture o shading o contours o shadows Number of observers monocular 0 multiple views Active lighting laser stripes o structured lighting patterns 0 shadows A taxonomy camera Laserprojectorlight Recovering the shape of an obiect Visual cues texture o shading o contours Number of observers monocular 0 multiple views Active lighting laser stripes o structured lighting patterns 0 shadows I Limitation Assumptions on surface reflectance function A taxonomy Specular Transparent Lamber ricm surfaces 1 Recovering the shape of an obiect Visual cues texture o shading o contours Number of observers monocular 0 multiple views Active lighting laser stripes o structured lighting patterns 0 shadows l Limitation Assumptions on surface reflectance function A taxonomy Specular Transparent Recovering the shape of an obiect Visual cues Itexturelo lshadingl o lcontours I Number of observers monocular 0 multiple views Active lighting laser stripes o structured lighting patterns 0 shadows 2 0 No intrinsic surface features 0 textures shading 1 0 No clear contours U imi L Recovering the shape of an obiect Visual cues texture o shading o contours Number of observers monocular 0 multiple views Active lighting laser stripes o structured lighting patterns 0 shadows Epipolar geometry doesn t work IOI IS imitat L Stereo matching algorithms fail Recovering the shape of an obiect Visual cues texture o shading o contours Number of observers monocular 0 multiple views Active lighting laser stripes o structured lighting patterns 0 shadows imitations L 7 Johann Erdmann Hummel The Granite Bowl in the Berlin Lustgarten 1831 A new perspective Use specular reflections as additional visual cue to estimate the surface s shape Savarese et aI CVPR 01 IJCV O7 Applications Ecommerce Diagnostic of space structures Digital archival a Industrial metrology Digital models of anatomical parts at ote sensing of liquid surfaces Recover local shape of mirror surface by measuring the deformation of a reflected scene A new perspective MC Escher Still Life with Spherical Mirror1934 A new perspective c t u39j c Mirror surface A new perspective Mirror surface 1 Smooth surface 2 Calibrated scene Explicit mapping local scene patch 9 reflected image patch Goal study this mapping Mirror stirface Pt P0 tAP fteR gtfte R3 ft fiteRZ How to find I undersnood A background note ngemw wggmmeumann proved D that me prob em 5 nso ub e usmg a N WR RVWH H construcuon b WWVV mWO HWSWQ extracuon of a cube root Axhazen was an raman Ph osopher e WW n W wgmswmw FBEEQErcMSFv Q39 wl ew8 conswdered as the father of modem How to find f0 om For spherical mirror Alhazen s problem Mirror surface PtP tAP Assume f0 is known how about t Ht at to Why ft m f0 ft0t7 toftot7to2 Mirror surface Pt P0 t AP TOOLS GOAL to to lagrunge Multiplier theorem Implicit Function Theorem Chen unclArva 2000 Summary Mapping fzt e 91 f0 6 93 39GOAL ftoftoquot39 Pt P0 tAP A new reference system u vw w Normal N V J POfo plane LIVXW Pt P0 tAP Summary Mapping f t e 91 f0 6 m3 GOAL fto Hfo w Normal N V J P O f0 plane u v x w Surface expanded in Monge form W au2 20uv bv2eu3 fu2v guv2 hv3 Summary Mapping f t 6 SR gtfo e 923 Suvw 0 GOAL m0 fare w Normal N v J POfo plane uvxw Surface expanded in Monge form W u2 2guv Qv2gu3 Lu2V guv2 v3 Pt Po tAP O W to39A wquotNunI an an f0 0 PO AP anac Observation Summary Mapping f t 6 SR gtfo e 923 GOAL fto litto w Normal N v J POfo plane uvxw Monge form w au2 20uv bv2 1 3 2 2 3 5eu fu vguv hv Similarly for ftoB 1 7 Local surface scene geometry Mirror surface 1 Image C f uveSR gtfiuve93 F 1313 2 13 2 it fuvfo fu u fv vafudju 2fwsv fmauvaquot u j 2w v Local surface scene geometry Mirror surface fuveSR gtfiuve93 Conclusion image paich Fscene patch local shape up to second order Tangent as function of 2mI and 3ml order parameters Monge form 1 2 2 1 w 2au 20uvbv ftoA ieu3fu2vguv2hv3 3 f0 0 PO Observation f0 0 a b C Tangent as function of 2mI and 3ml order parameters Monge form 1 2 2 1 w 2au 20uvbv ftoA ieu3fu2vguv2hv3 3 f0 0 PO Observation f0 0 a b C Singular specular surfaces foAquot3 foOPoAP fteSKfte R3 Observation f0 0 a b C no unique Theorem Convex parabooids are always nonsing uar I Singuldr A16t configurations blem Inverse pro Mirror surface Inverse problem From the mapping reflected image patch 9 local shape W Measurements Unknown Scene grid of lines Calibrated camera Known quantities Scene Camera i 39 Mirror surface b w 9 Scene grid of lines Calibrated camera Measured quantities Point position Q Orientation of curves at Q s ntersecting Calibrated camera through points Measured quantities Point position Q Orientation of curves at Q Scale distribution of points in a neighborhood of Q Equivalent to in km at Q Calibrated camera Local geometry we wish to recover Position ofthe re ection point M Normal of the surface at M Curvature ofthe surface at M Third order surface parameters around M l l Mirror surface gbctvvvi V Image plane quot 0 Oc Calibrated camera Image plane surface Point Tangent Surface Th39rd Order geometw surface Measu posrtlon plane curvature rements parameters up to 1 up to 1 pomt unknown unknown point Qo 39 Orientation up to 1 unknown along lines at Qo points points Mirror surface p i Calibrated camera Image plane Point Tangent Surface Thgd order Measu position plane curvature 5 ace rements parameters up to 1 up to 1 pomt unknown unknown 1 point Qo Orientation along lines at Qo up to 1 unknown points W points N N We know PO scene point OC center of the camera Q0 measured reflect point unknown surface knee Mon I Summary We don t know Mo reflect point But Mo o line of sight constraint Summary unknown 00 surface distance 8 OC Mo Q0 Q Q line of sight constraint 0 Mo SQo How about N Reflection constraints N E OC Mo Po plane Summary c OCVOPO pIane distance S OC Mo line of sight constraint Mo SQAO The principal plane defined by P0 00 Q0 How about N Reflection constraints oNE OC Mo Po plane ogizgr gtNFs Summary OC principal distance 8 Oc Mo plane A Q0 N v r I line of sight constraint Mo SQ The principal plane defined by P0 00 Q0 NFs CONCLUSION Surface position Mo and orientation N up to 1 unknown l Mirror surface b 1 a Calibrated camera Orientation along lines at Qo up to 1 unknown Image plane Point Tangent Surface Thgd order Measu position plane curvature 5 ace rements parameters up to 1 up to 1 pomt unknown unknown 39 point Qo points points Summary quot SOo Mo Q0 quot0 QO oMoszo NFS ff v 0 39 r o W 2au2 20uvbv2 Pt PO tAP p0 We need a further constraint e gt 8 unknown I I 1 00 Po 1 2 Observation 3 39 l b a b C unknown 4 unknowns known Summary SOc Mo o M5800 o NFs W 2au2 20uvbv2 F abcs P1t P0 t AP1 P2t P0 t AP2 951 fil PNt P0 tAPN A1 l8 Iquot 1 T I 2 fl 2 A1 B2 H 3 g 0 detH H 39 3720 s L a Summary o M5800 o NFs W 2au2 20uvbv2 F abcs P1tPotAP1 39 ngzo P2t P0 t AP2 PNt P0 t APN Rank THEOREM d tltHTHgt for ss rankH2 e 0 s actual value of s u 2m 43L ma m was we Summary M 2800 NFS W 2au2 20uvbv2 F abcs H mg 0 det TH P1t P0 t AP1 P2t P0 tAP2 0 9 s 88 PNt P0 t APN CONCLUSION Surface position M0 and orientation N can be computed using at least 3 orientations Summary MOZSQO NFs Hg O Rank THE OREM P1t PC t AP1 for ss rankH2 P2t P0 t AP2 If s s in general rankH3 detHTHSS o PNt P0 t APN From the rank theorem rankH2 a 2 a0 391 H3x3 9 O O 1 t C 2 Co 1 2 Ns gabcs P1t P0 t AP1 P2t P0 t AP2 PNt P0 t APN Summary Mo SQO NFs Rank THE OREM for ss rankH2 If s s in general rankH3 detHTHSS o Theorem there exists a family of 2nd order surface parameters for which the observed tangents are invariant w r t surface curvature aaoa1 bboQb1 ccoQ1 Calibrated camera Image plane Point Tangent Surface Thgd order Measu position plane curvature su ace rements parameters 39 up to 1 up to 1 pomt unknown unknown 39 point Qo Orientation along lines at Qo up to 1 unknown points points Some experimental results 3 orientations case positions and normals L lt gtigtr 02 error in position estimate lt llt 7 7 O 390 5 D E 2 error in the curvature estimate y quot y V Experimental results 2 orientations scale case Teapot with checkerboard pattern l r w 9 10 1100 1200 1300 1100 Portion of car fender we wish to recover Car fender with checkerboard pattern I700 H300 1201 1403 15m 1000 HBO quotCarrecon 39t39r39 c on 45 View re constructed normals Conclusions Explicit local relationship between scene shape and reflections Local surface shape can be recovered up to third order if and only if the scene is calibrated at least three five points are available in a neighborhood O rher pa rh minimization approaches Dense reconstruction of mirror surfaces 5 Rozenfeld ef al Dense mirroring surface recovery from 1D homographies and sparse Correspondences CVPR 07 w W n Such pairs are used to estimate a 1D homography Dense reconstruction of mirror surfaces Recover geometry of mixed specular diffuse obiec rs from two frames 5 Roth and M Black Specular Flow andfhe Recovery of Surface Sindune CVPR 06 Specular flow Recover radius and location of the sphere Further reading Volumetric approach I Voxel Carving for Specular Surfaces T Bonfort and P Sturm Shape from Distortion 3D Range Scanning of Mirroring Obiects M Tarini HLensch MGosle HPSeidel Reconstructing Curved Surfaces From Specular Reflection Patterns M Halstead B Barsky S Klein R Mandell PDE Based Shape from Specularities J Solem H Aanaes A Heyden Acquiring a Complete 3D Model from Specular Motion JY Zheng and A Murata Structured Highlight Inspection of Specular Surfaces AS Sanderson LE Weiss SK Nayar On Refractive Optical Flow S Agarwal S Mallick D Kriegman and S Belongie EECS 442 Computer vision Volumetric s rereo 39Defini rion Shope from Con rours Shope from Shadows 39Voxel coloring Today s lecture is a special ropic quotTraditionalquot Stereo P Vi39 D x O 02 Goal estimate the position of P given the observation of P from two view points Assumptions known camera parameters and position K R T quotTraditionalquot S rereo l x I x q 0 Subgoak 1 Solve the correspondence problem 2 Use corresponding observations to rriangula re Volumetric stereo IIIIIIIIIII I I I A a r Scene volume i Hypothesis pick up a point within the volume 2 Proiect this point into 2 or more images 3 Validation are the observations consistent Assumptions known camera parameters and position K R T Consistency based on cues such as Contourssilhouettes Shadows Colors Contours are a rich source of geometric information Apparent contour DEFINITION projection of the locus of points on the surface which separate the visible and occluded parts on the surface sato amp cipola apparent contour Camem SHhoue es silhouette Camera Why contours are interesting visual cues 00 Provide information in absence of other visual cues No texture No shading Why contours are interesting visual cues 00 Relatively easy to detect Easy contour segmentation Camera How can we use contours Object I 39 quot Vlsual cone Object apparent quot I contour Camera How can we use contours ijem N N N M x if quotquotquot At 7 p I mage v a N j j ibj t quot39 x mn u mur llmage V39sual cone Cam r3 Image object 5 contour in 2D Camera Image View point 1 I Object V4 The views are calibrated Object estimate visual hull View point 2 T I how to perform visual cones intersection 00 decompose visual cone in polygonal surfaces among others Reed and Allen 3999 Using contourssilhouettes in volumetric stereo also called Space carVing Martin and Aggarwal 1983 voxel Computing visual hull in 2D Computing visual hull in 2D Computing visual hull in 2D Computing visual hull in 2D j 39i gi 5 L7 Visual hull an upper bound estimate Consistency A voxel must be proiected into a silhouette in each image Space Carving has complexity ON3 39339 Octrees Szeliski 93 Complexity reduction octrees Complexity reduction octrees Subdiving volume in voxels of progressive smaller size Complexity reduction octrees Complexity reduction 2D example Complexity reduction 2D example 16 voxels analyzed 39 Complexity reduction 2D example 52 voxels analyzed 39 Complexity reduction 2D example 16x34 voxels analyzed 14165234x16 617 voxels have been analyzed in total rather than 32x32 1024 Advantages of space carving 00 Robust and simple 00 No need to solve for correspondences Limitations of space carving 00 Accuracy function of number of views Notagood estimate What else Limitations of space carving I I Concavity v ConcaVItIes are not modeled u 1 1351 i 1quot r74Lr I 7J I l39 I For 3D objects Are all types of concavities problematic Limitations of space carving Concavity v ConcaVItIes are not modeled Visual hull 2 Laurentini 1995 Closest approximation hyperbolic regions are ok Conservative Space carving a classic setup Camera Object Courtesy of seitz amp dyer Turntable Space carving a classic setup Space carving Experiments Z 24 poses 150 Z vogtlte size 2mm Space carving Experiments 2 24 poses 150 2 voxel size 1mm Space carving Conclusions 00 Robust 00 Produce conservative estimates 00 Concavities can be a problem 00 Lowend commercial 3D scanners blue backdrop turntable Contours in the computer vision literature Analyzing changes in apparent contours 00 Giblin and Weiss 1987 00 Cipolla and Blake 1992 00 Vaiant and Faugeras 1992 00 Ponce 92 Zheng 94 00 Furukawa et al 05 Picture from of Sato amp Cipolla Volumetric s rereo O a mmmm 0 p35 mWWEGWWNJG 39Shape from Shadows J gxa r kymmg Selfshadows are visual cues for shape recovery Selfshadows indicate concavities no modeled by contours Shape from shadow in the literature 2 Shafer amp Kanade 83 2 Hatzitheodorou amp Kender 89 2 etal 11 ght H atzi 39zeodm39ou amp Kender 89 00 shape as a 25D terrains ie surface modeled as fx 00 accurate shadow detection ie xb and xe Shape from shadow in the literature 2 Shafer amp Kanade 83 2 Hatzitheodorou amp Kender 89 2 etal H atzi 39zeodm39ou amp Kender 89 9 AIAAAA AA A n L39l39 1AAAA I A Al39AAA AAAAIAIAAI AA l39 46255 it Using shadows in volumetric stereo Shadow Carving Savarese et al 01 Selfshadows 00 Robust with respect to shadow estimates 00 Object with arbitrary topology no 25D terrains Object s upper bound Shadow carving setup Object Light Image 5 Camera Let s define a 2D model 2D slice model Object r 39 Camera 2D slice model Dbject Iquot Light 0 From 3D 39 39 We to 20 quot 9 Object Image line Camera Theidea Object s upper bound Image line o u uquot n I 0 39 390 39 t o 39 s 39 u 39 39 39 39 39 o o 39 u o o u o a Camera I Theidea Object Object s upper bound gt Shadow Image line shadow Camera I Object s upper bound Image line I39 o 39 Camera Theidea Object Object s upper bound Image line I39 o 39 Camera Theidea Object Theidea Object Object s upper bound Image line Camera Ught Theidea Object Object s upper bound Image line Camera Ught Theidea Reconstruction after 3 iterations What is known Camera line SOUFCG 00 Light known 00 Camera known 0 Object unknown 00 Upper bound object estimate What can we carve Camera Light source Shadow 00 shadow cone 00 upperbound shadow 00 light cone 00 carvable area What can we carve Camera Light source Image shadow Shadow 00 shadow cone 00 upperbound shadow 00 light cone 00 carvable area The upperbound estimate is refined What can we carve Camera Light source Image shadow Shadow What can we carve Camera Image shadow 00 No further volume can be removed Shadow v Carvmg process always conservative Proof of correctness Algorithm step k Camera Upperrbuund rmm step w Algorithm step k dua Wage Algorithm step k Camera dua Wage Wage Uppepbuunu rmm step w Consistency Avoxel must be proiected into both image shadow and dual image shad w Algorithm step k Camera ugm Wage shaduw Wage shaduw Uppepbuunu rmm step w Complexity 02N3 Shadow carving the setup Object Array of lights Turntable Shadow carving the setup Am equot 393 at Sgt I Am era Object Array of lights Turntable Experimental results Simulating the system with 3D studio Max 24 positions 4 lights 72 positions 8 lights 24 positions 4 lights 72 positions 8 lights 16 positions 4 lights Shadow carving summary 3 Produces a conservative volume estimate 3 Accuracy depending on view point and light source number 3 Limitations with specular amp low albedo regions Volumetric s rereo o ri n g Voxel Coloring Seitz amp Dyer 97 R Collins Space Sweep 96 39239 colorphotoconsistency 39239 Jointly model structure and appearance Basic idea View1 Is this voxel in or out Basic idea Uniqueness Multiple consistent scenes Uniqueness Multiple consistent scenes Tractability 7 mp Mcoors 3 Combinatorial number possible assignments 3 Exhaustive search not feasible 3 Model occlusions 3 Use visibility constraint The algorithm uuwuurmu za mucmumuwwm Jnrlanamuam aanauumauunnm C Algorithm complexity 3 voxel coloring visits each N3 voxels only once 3 project each voxel into L images 9 OL N3 NOTE not function of the number of colors Photoconsistency test Lcorr 00 000 Image 1 Image 2 If L lt Thresh voxel consistent A critical assumption Lambertian surfaces I100 1p 2p 2p Non Lambertian surfaces Experimental results Dinosaur 39239 72 kvoxels colored 39239 76 M voxels tested 39239 7 min to compute on a 250MHz Experimental results Flower 00 70 k voxels colored 00 76 M voxels tested 40 7 min to compute on a 250MHz Experimental results Room weird people Voxel coloring conclusions Good things 00 model intrinsic scene colors and texture 00 no assumptions on scene topology Voxel coloring conclusions Good things 439 model intrinsic scene colors and texture 0 no assumptions on scene topology Limitations 00 Constrained camera positions 00 Lambertian assumption space carving 00 Space carving is a binary voxel coloring I out El El in Camera quot39 iliililliiill Shadow carving 39o g 5 39 n i no visibility constraint needed Light SOUFCG Further contributions 3 A Theory of Space Carving Kutulakos amp Seitz 99 02 Voxel coloring in more general framework 02 No restrictions on camera position st Probabilistic Space Carving Broadhurst amp Cipolla ICCV 2001 Bhotika Kutulakos et al ECCV 2002 EECS 442 Computer vision Detectors part II Descriptors Blob detectors Invariance Descriptors Some slides of this lectures are courtesy of prof F Li prof S Luzebnik and various other lecturers Goal Identity interesting regions from the images edges corners blobs Matching Indexing Recognition Repeatability The same feature can be found in several images despite geometric and photometric transformations Saliency Each feature is found at an quotinterestingquot region of the image Locality A feature occupies a quotrelatively small area of the image Repeatability Illumination invariance Scale invariance V C y n h m d I C G O S L o o Harris De rec ror Invariance Detector titth quot Harris partial Yes No No corner H Eovm d fig Edge defection Sigma50 Edge 6 Derivative E of Gaussian 40 60 80 100 1200 140 1 O 150 2 0 g Edge maximum g of derivative c o Source 8 Seitz Edge detection as zero crossing Slqma 50 I 0 200 no 600 see 1000 1200 1400 1500 1300 2000 d2 1 1 K I Second derivative i 39 39 i of Gaussian d 2 g E 39 Laplacian x g 1 Juu 1 s39uu 1 duo zuuu who 12km Edge zero crossing of second derivative 1000 120a 14UU 1mm 1 uuu zuuu Source 8 Seitz Edge detection as zero crossing Keme From edges to blobs Blob superposition of nearby edges A blob n E an w 20 quot K m 2 maximum Ok greof buf who if The blob is slighfly fhicker or slimmer From edges to blobs Blob superposition of nearby edges Ell signal L zn 7 a in in 7w 2 Laplacian o 1 a a rzn 7 a 1n rzn 4 l 10 No longer maximum maximum Spalial selection magnilude of The Laplacian response will achieve a maximum al The cenler of lhe blob provided lhe scale of lhe Laplacian is llmalchedquot lo lhe scale of lhe blob Scale selection 0 We want to find the characteristic scale of the blob by convolving it with Laplacians at several scales and looking for the maximum response However Laplacian response decays as scale increases I 2Z 20 2E B C 8 20 20 original signal radius8 2C 2C El 6 2DD increasing a Why does this happena Unnormalized Laplacian res p0 This should give the max response Scale normalization The response of d derivative of Gaussian filter to 1 perfect step edge decreases as 6 increases m 0427 Scale normalization To keep response the same scale invarian r mus r mul riply Gaussian derivative by 6 Laplacian is the second Gaussian derivative so it must be multiplied by 52 Effect of scale normalization Original Signal Unnormalized Laplacian response 25 2m 20 2 m n n a n n n 72D rzn 72a 720 72D in in n 29 7 71 in n a 2n n A up 5 a an 2n a m an n 2n 7m an a 20 5100 g2nn Fm Scalenormalized Laplacian response 2 2n 0 m n o n a o n 72 4n rm rm em in a m 1 n 1 72 5 21 7m n m 72 n m was uznn Hm Hun mm Maximum Blob detection in 2D 0 Loplocion of Gaussian Circulorly symmetric operator for blob detection in 2D 62g 62g 6x2 632 Vzgz Blob detection in 2D 0 Loplocion of Gaussian Circulorly symmetric operator for blob detection in 2D wiw 2 azgazg 6x2 6y2 2 Scalenormalized Vnormg oquot Scale selection 39 For a binary circle of radius r the Laplacian achieves a maximum at azrxE Laplacian response mE scale 2 Image Characteristic scale 39 We define the characteristic scale as the scale that produces p ak of Laplacian response quotmn 1000 characteristic scale T Lindeberg 1998 quotFeature detection with automatic scale selectionquot International Journal of Computer Vision 30 2 pp 77116 Scalespace blob detector Convolve image with scalenormalized Laplacian at several scales Find maxima of squared Laplacian response in scalespace I I quotmm quotmy quotgym I I I Iquot Sm e 39Igg a my quotWquot Iquot quotI quotIquot quotmg quotmg um quotIquot Scalespace blob detector Example Scalespace blob de rec ror Example sigma 119912 Scalespace blob detector Example DOG David G Lowe quotIquot quot quot e imuqe feufures from scaleinvariant 39 39 CV60 2 04 39 Approximating the Laplacian with a difference of Gaussians L 02 Gmx y039 nyx y039 Loplocicm 190G GX yJw GX y Difference of Goussicms nGacian Gm kg am y 0 a A 10392L DOG gt Scale gt rst octave gt Difference of Gaussian Gaussian DOG Output location scale orientation more later Example of keypoin r defection Invariance D ete C39I39O r H H H igl m m Harris Yes No corner Lowe 3999 Yes Yes Yes No DOG HarrisLaplace Mikolaiczyk amp Schmid 3901 Collect locations xy of detected Harris features for 6 61 62 the sigma is here comes from gngy For each detected location xy and for each a reject detection if Laplacianxy c is not a local maximum w Output location scale Invariance D 919 cfo r H H U Ml f x 1 if l r Ha rris Yes Yes No No corner Lowe 3999 Yes Yes Yes No DOG Mi koloiczyk Yes Yes Yes N o amp Sch mid 390 1 Illumination invariance Repeatability Scale invariance Pose invariance Rotation Aftine Affine invariance K Mikolaiczyk and C Schmid Scale and Affine invarianf inferesf poinf defec rors IJCV 6016386 2004 Similarly to characferis ric scale selec rion defec r fhe characferis ric shape of rhe local fea rure Affine adaptation 13 1ny 2124 A o 2 R X 1y 1y 0 12 M XVI9w We can visualize M as an ellipse with axis lengths determined by the eigenvalues and orientation determined by R dkec on ofthe fa estchange Ellipse equation I I dwechon ofthe dowe change u v M qzconst v N The second moment ellipse can be viewed as the llcharacteristic shape of a region ewroe U39I Affine adaptation Detect initial region with Harris Laplace Estimate affine shape with M Normalize the affine region to a circular one Redetect the new location and scale in the normalized image Go to step 2 if the eigenvalues of the M for the new point are not equal detector not yet adapted to the characteristic shape Affine adaptation Output location scale affine shape rotation more later Scaleinvariant regions blobs Affine adaptation example 7 4 Affineadapted blobs Invariance De rec ror Harris corner Lowe 3999 Yes Yes Yes No DOG Mikoloiczyk Yes Yes Yes No amp Schmid 3901 Mikoloiczyk Yes Yes Yes Yes amp Schmid 3902 De rec ror H I Umm Harris Yes Yes No No corner Lowe 3999 Yes Yes Yes Yes DoG Mikoloiczyk Yes Yes Yes Yes amp Schmid 3901 02 Tuy reloors Yes Yes No Yes 00 Yes 3904 Kodir amp Yes Yes Yes no Brody O1 Mates 3902 Yes Yes Yes no Descriptors Goal Identity interesting regions from the images edges corners blobs Matching Indexing Recognition Matching Features s ri rching images De rec r feature points in both images Matching Features s ri rching images De rec r feature points in both images 0 Find corresponding pairs Matching Features stitching images Detect feature points in both images 0 Find corresponding pairs Use these pairs to align images Matching Features estimating F Use these pairs to estimate F Matching Features recognizing obiects 39Use these pairs to match different obiect instances Challenges Depending on the application a descriptor must incorporate information that is 0 Invariant wrt 39lllumination 39 Pose 39Scale 39 lntraclass variability distinctive allows a single feature to find its correct match with good probability in a large database of features Illumination normalization A ine I39nfensify cha nge I I b Make each patch zero mean gt 211 l b 1 Z07 y Ma y 7 Then make unit variance 1 a r Z Z39r y r1 ZNUM X image coordinate Pose normalization Keypoints are transformed in order to be invariant to translation rotation scale and other geometrical parameters Change of scale pose illumination 9M0 39G to Aseunog Pose normalization View 1 NOTE location scale rotation amp affine pose are given by the detector or calculated within the detected regions The simplest descriptor l x NM vector of pixel intensities wm W W Q Makes the descriptor invariant with respect to affine n transformation of the illumination condition Why can t we iust use this Sensitive to small variation of 0 location 0 Pose Scale 0 intraclass variability 39 Poorly distinctive S re reo sys rems Normalized Correlation WWW39W39 Wu Wn W WW W H Detector PATCH Bank of filters Image filter bank is L X lter responses intt t More robust but still quite sensitive to pose variations 39 Q 39 descriptor Bank of filters Steeroble filters Gaussian derivatives up to 4th order The remaining derivatives can be computed by rotation of 90 degrees De recfor PATCH FILTERS SIFT descriptor David G Lowe quotDistinctive image features from scaleinvariant kevgoints CV60 2 04 0 Alternative representation for image patches 0 Location and characteristic scale s given by DoG detector A L r SIFT descriptor David G Lowe quotDistinctive image features from scaleinvariant kevpoints CV60 2 04 0 Alternative representation for image patches 0 Location and characteristic scale s given by DoG detector Compute gradient at each pixel N x N spatial bins 0 Compute an histogram of M orientations for each bean A61 A61 ABM SIFT descriptor David G Lowe quotDistinctive image features from scaleinvariant kevpoints CV60 2 04 0 Alternative representation for image patches 0 Location and characteristic scale s given by DoG detector Compute gradient at each pixel N x N spatial bins 0 Compute an histogram of M orientations for each bean 0 Gaussian centerweighting A61 A61 ABM SIFT descriptor David G Lowe quotDistinctive image features from scaleinvariant keypoints CV60 2 04 0 Alternative representation for image patches 0 Location and characteristic scale s given by DoG detector Compute gradient at each pixel N x N spatial bins 0 Compute an histogram of M orientations for each bean 0 Gaussian centerweighting 0 Normalized unit norm Typically M 8 N 4 l x 128 descriptor SIFT descriptor Robust wrt small variation in llumination thanks to gradient amp normalization Pose small affine variation thanks to orientation histogram 0 Scale scale is fixed by DOG Intraclass variability small variations thanks to histograms Rotational invariance 0 Find dominant orientation by building smoothed orientation histogram Rotate all orientations by the dominant orientation C x 1 gt A 51 3 2quot L x This makes the SIFT descriptor rotational invariant Rotational inva ria nce Ro ra rional invariance Matching using SIFT David G Lowe quotDistinctive ima e features from scaleinvariant ke oinfs CV60 2 04 Matching using SIFT David G Lowe quotDistinctive ima e features from scaleinvariant ke oinfs CV60 2 04 De recfor PATCH FILTERS SIFT Shape con rex r Belongie et al 2002 Shape Context Count the number of points inside each bin eg Count 10 Compact representation of distribution of points relative to each point 0mm nf lifumm Berkeley Computer Vision Group Sh ape Lumen Unime ofCahfmmz Berkeley Computer Vision Group EECS 442 Computer vision Next lecture Mid level representations O rher in reres r poin r defectors Scale SaI39ency Kaa l39ramp Brady3901 3903 Other interest point detectors Scale Saliency Kaa ir amp Brady Ol 03 Uses entropy measure of local pdf of intensities HDSX Ipdsx10g2 pa sxdd deD Takes local maxima in scale Weights with change39 of distribution with scale Epdsxdd as WDSX S I deD To get saliency measure YDSXHDSXgtltWDSX O rher in reres r poin r de rec rors Scale SaI39encyKaciramp Brady3901 3903 o f in 2 a in 2n 3 mm Just using Using HDsx YDsx HDWD c Most salient parts detected Other interest point detectors maximum stable extremal regions matas et al 02 Sweep threshold of intensity from black to white Locate regions based on stability of region with respect to change of threshold Creating features stable to viewpoint change 0 Edelman Intrator amp Poggio 97 showed that complex cell outputs are better for 3D recognition than simple correlation l s impl e RF quotl c o s 3 r e simple RF sin oriented energy Feature stability to noise 0 Match features after random change in image scale amp orientation with differing levels of image noise 0 Find nearest neighbor in database of 30000 features Correctly matched we lmage noise Feature stability to affine change Match features after random change in image scale amp orientation with 2 image noise and af ne distortion Find nearest neighbor in database of 30000 features 3 E so 2 m E 3 s 40 2 8 Key oinllocalion A Location amp orieniaiion 7777777 n 20 Nearesldesciiplor k 0 0 10 20 3 40 50 o Viewpoini angle degrees Geometric blur Exttact Apply Spatse Spatially Channels Varying BIIII Su bsa rn ple GED 2m Blur descnptur Sparse Genmetru am NunrNegatwe Bf each Channels channel 29 unarmed Edge energev Berg et al 2001