Midterm 3 03/16/2017 This is a picture of the world (in this case a picture of flowers.) A Picture is necessarily a projection of three dimensions, so a picture of the world is going to be a projection of a three dimensional projected into a 2D space. Someone took a picture of flowers, result is 2d projection of three dimensional world. What does this meaDon't forget about the age old question of → YESTERDAY 1X-Me STUDY place - Why were you at the library?
We also discuss several other topics like asl notes
Don't forget about the age old question of notes asl
If you want to learn more check out adam reed unr
If you want to learn more check out how transpiration works
If you want to learn more check out chm 458 class notes
n? That means this used to have a dimension of depth but When I took a picture of it that dimension got flatten, so I can project the image in 2D. When we take a picture of 3D world we are basically getting rid of extra information. Lets say I want to reconstruct a true 3D image from 2D projection? So now I want to take this image and make it 3D dimensions. Going from 3d to 2d is really easy. You are just losing information. But going from 2d to 3d is really hard. How do I go from 2D to 3D? How do we reconstruct 3D information from 2D limited input. That problem is really really challenging. how do we do it? What kinds of things would we have to know to do that? We are basically looking through one eye at this image and there is a lot of information that is lost. Where things are relative to each other. What is the distance here between these two flowers? So going from 2d to 3d, Who has built ikea furniture? You take a 2d diagram to build a 3d product. How to reconstruct a 3D object from a 2D image. This is peoples full time judge for us to understand how to go from a 2d image to a 3d image. What you end up with is a lot of extra information. So in this case we might do it by adding dimensions, doing this, drawing that. So a causal direction of how things should happen. But of course in the real world our 2d don’t come with all those extra tidbits. Our projection of 2D worlds does not come with inches of length and width. So what is this object? It is a cube. How do you know this is a cube. A cube is a 3d object. This image is a 2d projection of a cube. What are the things that tell you that this is a cube in 3d? This image is consistent with an infinite amount of potential views. This could be a 2d object. This the problem that the visual system faces. When we have 3d and compress that into 2d we come uo with a problem that is basically like solving xy=36. What is the solution to xy=36. there is accountability an infinite amount of solutions. but you cant really solve that problem. it is really really hard problem. We have a 2d image on our retinas, and we have to take that image and reconstruct a 3d environment so that we can build our behaviorally relevant units that we want to interact with. How do solve this problem? How do we go from 2d dimensions and reconstruct a 3d world from 2d retinal images. Fortuanelty, even though we have compressed this image into 2d from 3d. there is still a lot of information in the image that we can use to reconstruct the three dimensional world. not only that but we have two eyes, two views of the world. We are not only looking at the world through one perspective, but actually we have two 2d images of the world. the information that is on the retinal image itself and the difference that is on the two retinal images in the two eyes, not to mention how we move our eyes to look at the world, all those things will come together to provide really powerful cues for reconstructing a 3d world from a compressed 2d image. so what are these cues? Today and Tuesday we will be talking about depth. How does the visual system do depth? how do we reconstruct a 3d world from 2d images on the retina. We are going to be talking about all the different cues that are in movement of our eyes, in the retinal image, and the difference in the retinal image in the two eyes. all of these are cues that we can use to arrive at a 3 dimensional percept. So depth cues are any information that specifies 3d position of the object or help us build a representation of a 3d world. depths cues can be grouped into two major categories. the first are oculomotor cues. oculomotor just means eye movements. So these are cues that we can glean from movements of our eyes, that give us some sense of object depth. the other cues is based on the retinal image itself. so information content that is in the retinal image. they can be monocular (so can be gleamed at looking through the world from one eye) or they can binocular (gleaned from looking at the world with both eyes). so we are going to talk about each of these in turn. what we will see is all of them provide some fairly accurate information about depth .ocular motor cues to some extent at least with some proximity to you and monocular pretty good at giving some depth but some binocular motor cues are really result in the richest and most precise representation of depth. but all of these things that come together to build a 3d dimensional world from 2d. so lets talk about the oculomotor cues. so these are the eye movement cues. so the two cues we are going to talk about to day are lens accommodation and eye convergence. What muscles are involved in lens accommodation? the cilliary muscles in conjunction with the zonules of zinn allow for changing the shape of the lens. What muscles would be involved in eye convergence? the rectus muscles and potentially the oblique muscles. Lets take that information and think about what kinds of information it can provide about depth. so lets take the first thing. lens accommodation. so what happens when you look at something up close. zonules of zin slacken and the ciliary muscles slackens. the sin and muscles works together to fatten the lens as something approaches the eye. the lens gets fatter in relation to the distance of the object from the eye. so you can imagine, ad the object approaches your eye, the lens has to get fatter. so if we want to know the distance we are fixating of something, in some respect we can monitor the status of the ciliary muscles (so how contracted the ciliary muscles are) that can give us a sense of how far the object is from us but of course this has a limit. what is the limit of this? how is this limited? so how do I figure out how far away someone is that sitting in the back? would I be able to use this cue? who thinks I can use this cue of motoring lens accommodation to figure out how far that person is far back from me? you cannot use this cue. it is not that reliable after a certain point. so beyond two meters it is not reliable. objects that are really far from me are outside the ranger where lens accommodation is going to affect my vision. it isn’t the case that if an object is really far away from me I need to bend that light to focus on my retina because the rays of light that are bouncing off the people in the back are coming in parallel. so we don’t need to use lens accommodation to see them. therefore, this is not a reliable cue beyond two meters. so convergence of the eyes (So now we are in both eyes, not just looking at the lenses in each eye but the what is happening in the two eyes when the object is coming closer to me so I fixate on that object I might move my eyes to point both of my fovea on the object.) As the strawberry approaches my eye are start converging both of my eyes, so the strawberry falls within the fovea of my two eyes. but again a similar principle happens. we can monitor the ocular motor muscles to see if something is close. am I moving my eyes to see if something is close off? if my eyes are converging that might suggest that something is closer to my eye so something is closer to me. but similiarly with accommodation, this cue is not reliable beyond 4 meters (convergence is not reliable beyond 4 meters). That means the convergence cue doesn’t really work. So the oculomotor cues are pretty useful if the object is within 4 meters, beyond that I cannot really rely on these cues. So the ocular motor cues allow us to monitor those eye muscles that responsible for bringing close object into focus but again they are not reliable past about 4 meters (2 meters for accommodation, 4 for convergence). The other limitation of the oculaor motor cues is that they only really tell us about the thing we are fixating on. so if I wanted to know how far away my hand is form me. the ocular motor cues can give me a sense of how far my hand is from me. but it can only really give me a sense, but they only play a role on the object I am fixating on. they do not tell us about anything that is going on the world. that stuff is almost just as important as the object I want to fixate on. the only thing I am getting information about is the object I want to fixate on. but it doesn’t tell me is this just an object floating around in space, or how far away is everything else in the world. what is the total picture of depth outside this fixation point. ocular motor cues do not give us that. in order to do that we need to be getting information about depth from the retinal image itself, so the rest of the stuff that is going on. so there are two ways we can do that from monocular cues and binocular cues. the monocular cues are depth cues that can perceived from just one eye. when you cover one eye the world does not look flat, and it does not make it look like they are pooping out (do things the really deeps, are things pooping out) looks kinds of the same, harder to gage depth but things do not look flat. that is because there is a lot of information that we use in the retinal image that doesn’t really rely on the difference between the two eyes, which is what we usually think about when we rely on depth perception. so these are the monuclar cues and they have two to distinct categories, static and dynamic cues. static cues can be gleamed from a glance of the world. static cues are glance of the world, snapshot of the world. dynamic cues are how objects move in the world in relation to each other. so lets talk about static cues first. some of these cues can be categorized as position based cues, which are how objects are positioned in relation to each other. so one these cues are partial occlusion. what this means is that if an object seems to be partially occluded by another object it is probably farther away form the object that is occluding it. that makes sense. if the stove is partially occluded by this table the stove is not likely to be in front of table but probably behind the table or counter. similiarly if the counter is partially occluded (behind it) by the stool it is likely to be behind the stool. the stool is likely to be in front. these are cues about partial occlusion of objects giving you information about where they are in depth. so this is a very simple example. these are two rectangles. they could be rectangles or they could be polygon that has this many side. but no we see it as one rectangle behind another rectangle because we are using these cues of partial occlusion that we have talked about perception interpolation, filling in missing parts. that is giving us a sense based on the relative position of objects that one object is behind or in front of the object. Another position based cue we can use is the relative height of images in the retinal image. so what does that mea? so lets say I am fixating on a particular point, so in this case I am fixating on the corner of the countertop (corner of the wall). I can look at the relative height of the image relative to my fixation point. So relative to a particular fixation. so things that are close to the fixation point are going to end up being further from me if im fixating on something in the distance. it is not always going to be reliable. in doors these things are going to be less reliable. outdoors they are a little more reliable because there are constrains in the naturalistic environment. the sky is open and up, not much above us and there is much below us. so if we are fixating on things in the distance, we can rely on things that are lower, lower away from our fixation point are going to be closer to us. Things that are higher, here are closer to the fixation point are going to be further away from us. that is relative height in the retinal image. more reliable below the horizon in the outside. what about size based cues? they rely on some extent on the size distance relation. so the size distance relation, that the farther away the object is the farther away the image is on the retina. we can kind of think of this as a rule of thumb. the further away an object is the smaller the image is on the retina. that makes sense. if I look at this podium it is taking up most of this visual field. but if I ran all the way over here it is taking up a smaller amount of space in my visual field. so as you can see it is the size distance relatuonhip. the further away an object is the smaller its image on the retinae. but you can also see that this does not only depend on the distance but also on the size of the object. so these are trees with long and short trunks. so lets say you have two pieces of chalk that are the same size, and show you the chalk like this. so the retinal image of this piece of chalk is going to be larger than this piece of chalk. this piece of chalk is farther away from you so its retinal image is going to be smaller. so now what happens when you take two pieces of chalk one being smaller one being larger and I again put them like this. the retinal image of this object of this object is the same as the smaller object since it is farther away. this object is bigger but it is further away. now we have an ambiguity. we have a situation where the retinal image of this object is the same exact size as this object. it is further away, so the retinal image is going to be smaller but it is larger than this object so the retinal image ends up being the same size. it is just based on the way it is positioned in space. that is what the diagram shows. this presents an ambiguity that we have to figure out. so the retinal project depends on the size of the object and the distance from you. how do we figure out ambuguites like these? What are other cues we can use. so imagine this is a golf ball, this a baseball, and this is a basketball. these three balls at difference distance from you, they are in a particular position that they project the same size. but we are not confused because we are familiar with these objects. so we know what the size of the basketball, baseball is ect. using the featural identity of these objects, we can say that even though the retinal images are the same it is not likely that what I am looking at is a golf ball in the shape of a basketball, it is probably a basketball in the distance as opposed to a basketball looking golf ball. similiarly if I show you this car maybe you are not sure if it is areal or toy car but if I put an object next to it that you are familiar with, now the object looks like a toy car. so you can use familiar size to figure out retinal images that are somewhat confusing based on the size distance relationship. so relative sizes is another principle we can us. so familiar size has to do with objects we are familiar with. relative size are objects that are related but do not have to be familiar with. relative size cues is that we assume objects are the same size so any difference in the image must be due to depth. we will assume these two gentleman are the same size which is safe to assume since there is not a lot of variability in the human form. so any variability in the retinal image must be due to depth. that one gentleman is sitting further away from the other gentleman. guy comes up to hi shin. when we are making these assumptions about relative depth and we have these difference in the retinal image of these two gentleman and it is the case that this kind of positioning is not inconsistent with what is going on the world. so something is up here and down here. so we are making this assumption that this guy here even though his retinal image is tiny he looks like a normal sized guy. when we move him down here on the same plane and cues conflict we end up with a tiny man. you do not need to be familiar with an object to understand relative size. don’t have to be familiar with index cards to think they are the same size. any difference in retinal image must be due to depth. similar based principle is applied to size based cues which is called texture gradients. small elements that are roughly all the same size. we can look at the frequency of these elements can change as we move across the retinal image. and what we can see if we make the assumption that this is a texture the elements of which are all roughly the same size, we get changes of spatial frequency across the texture, so higher frequency and lower frequency. the higher frequencey regions will be indicative of the texture being further away. so as we move up in this image, the elements become smaller, and since we are assuming al of the objects are the same size that means that they are moving back in depth. we can do this with textures that are grain of sand or even just dots. so here we have the density of the dots increasing and in this case the intensity of the dots are increasing into the horizon it looks like we are moving back in depth. doesn’t have to be pure textures. so this is a choir stall in a church. so you can assume all the candlesticks are the same size. they create a texture or image and as the retinal image decreases in size you can assume we are moving back in depth. we can do it with crowds. everyones heads looks like they are getting smaller, smaller heads mean they are further away. so those are cues that depend on the relative size of objects. another cue is linear perspective. you are probably familiar with this from art class. so parallel lines will converge as they recede. as you are driving along the road, the sides of the roads are parrellel, so as you look at a certain distance they look like they are converging to a single point. but of course we know the road is not getting narrower it is just recesfding in depth. parallel lines converge is a powerful monocular depth cue. so we can have parallel lines of the building as the top make it look like we are looking down an alley. so those are the size based cues: familiar size, relative sixe, texture gradients and linear perspectives. Monocular depth cues with how light interats with objects. so lighting based cues are very powerful. how do you know something has depth? well it is casting a shadow. so depth from shading can be a powerful monocular cues. so have dots popping out and popping in. why do they look different? because of light soruce usually comes from above bcs that are how things are typically lit. that means object is popping out. can change percept by tilting your head. can make them pop in and out. visual system assumes light is coming from above. hallow mask illusion shading cues that can give us information about depth. mask is just a white mask from the outside. the light source is providing normal shading cues. but on the inside of the mask shading cues have been drawn in to make it look like it is popping. this example shows that lighting cues ae very powerful cues of depth. we can be tricked to thinking that something is very powerfully popping out in three dimensions. another lighting based cues is depth from cast shadows. here are two disk. they both have cast shadows tell you about the surface they are on. tells depth, size and shape can tell you about the object and location of light source. atmospheric distancethings in the distance look fuzzier than things are close up because there are things in the air that makes. doesn’t happen with things that are like kids sitting in the back of the room but things like mountains. It look it fuzzier because of between you and the object there are water vapor, pollution, anf fog. things in the distance look fuzzier.even thought the air is the same acroos the scene but you don’t have the same interference. lighting cues of things in distance are giving us information about how far things are. When cues conflict. interesting ways to trip monocular depth cues. Ames room and buechet chair (taking a picture through one eye camera) ames room has a point that project out it is not really a rectangle even though it looks like a rectangle which means one guy can stand further back, texture gradient of floor makes it seem like a rectangle (makes it look like a painted flat floor) buechet chair legs of chair are much farther in front of man. platform of chair is actually in back chalk painting tht only works from one perspective so here is a 2d picture of 3d world. what type of monocular clues can we gleam? ∙ texture gradients of the grass ( gets more ) and ripples on water ∙ linear perspective (stream seems to narrow) ∙ atmospheric pressure (mountains seem fuzzier) ∙ depth from shading ∙ familiar size for mountains and tress (trees are somewhat similar size but retinal images of some of the trees seem smaller bcs they are father away) ∙ partial occlusion (bank in front of stream and stream in front of mountain) ∙ relative height (thing lower in image are closer to us) ∙ ∙ static cues (cues that can be gleamed from a single glance) but there is a whole other set of monocular cues that we can use with just one eye but these cues are dynamic. they rely on how the world moves. the first one we will talk about are motion parallax. you guys might have notice when you are a passanger in car or train and look at fixation point on the horizon. as we move, things that are in front of th point look like they are going past us and things on the other side of the fixation point look like they are going with us. this is the idea of motion parralx. things that are in front of point look like they are flying past you and things beyond that point look like they are traveling with you. and the further from fixation, the faster they seem like they are moving (the greater distance they travel). optic flow as looking in point in distance all the points around it look like they are radiating out of the fixation point. and things look they are moving faster as they farther away form fixation point. the faster they seem and more distance they cover. deletion and accretionsimilair to partial occlusion, circle moves across square part of suare gets deleted and as the circle keeps moving part of square gets accreded. it shows us on object is going in front of another object. this is just a dynamic partial occlusion cue. do something with just one eye for 15 min. 710 min That is what the visual ___ is facing. We have to take that image and build a 3D environment. How do we reconstruct a 3D world from a 2D retinal image? Even though we have compressed this image in 2D. There is still information in the image that can still help us recreate the image. We have two eyestwo images. The information on the retinal image itself and the difference of the retinal image on the two eyes. All this information can come together to help reconstruct a 3D world from a 2D image. What are these cues? Cues that help us arrive a 3D percept. Depth cues help us build a specific representation of a 3d world. Ocular movements ____. Cues in the Retinal image itself (binocular or monocular). All of them provide failry accurate info about depth. monocular cues are pretty good at depth but binocular give the richest depth. All these things come to build a 3d world from 2d. Oculomotor cues eye movement cues the ciliary muscles with zinules of zin allow for the change of the lens muscles involved in eye convergence are the rectus muscles and potentionally oblique what kinds of information about depth? lens accomadtion when we look up close. The zonules interact with ciliary miuscles to fatten the lens when something is closer. the lens gets fatter in relation to distance from the eye. as object gets closer, object has to get fatter. who thinks I can use the cue of lens accommodation as a cue? You cannot use this cue. not reliable cue beyond 2 meters. What is happening between these two eyes? Strawberry come closer, I start pointing my eyes o center. start converging my eyes. Beyond 4 meter I am not converging my eyes to keep the object in focus. beyond that I cannot really rely on these as cues. the other motivation of ocular cues is that they only tell us about the object we are fixating on. they don’t tell me anything about the rest of the stuff that is going on in the world. in order to do that we need to get information about the image from the retinal image itself?through monocular cues. Depth clues that can be in one eye. things don’t look flat when just using one eye. static cues are a single snapshot of the world. dynamic cues are how objects move in relation to each other. static cues position based cues how are objects positioned in relation to each other. When object is partially occluded it is probably behind the other object. giving us information about where they are in depth point. position based cues: relative height in the retinal image. can look at relative height of an object realte to the point of fixation. things that are closer from the fixation point are farther away from me. things that are lower are closer to us, things are higher are farther away from us. size based cue: rely to some extent on the size distance relation: the farther awau an object is, the smaller its image on the retinae. ∙ retinal project depends on size and distance ∙ two pieces of chalk are the same size. the retinal image for the one farther away is smaller. ∙ when bigger object is farther away, the retinal image of the farther but bigger image can be the same size of object that is closer ∙ that presents an ambiguity that we have to figure out. ∙ how do we solve ambiguity? o size based cues: familiar size o can use familiar object to clarify size of objects o relative size we assume the objects are the same size so any difference in the retinal image must be due to depth o texture gradients o linear based cues parallel lines converge as image is receding gin depth Lighting based cues: Depth from shading Hallow Mask illusion ∙ inside of mask looks like outside of mask ∙ outside of mask lighting gives cues of shading ∙ then shading of inside of mask makes it look like outside of mask ∙ lighting cues are very powerful for the dimension of depth ∙ atmospheric perspective thinks in the distance tend to look fuzzier than things that are closer because there are things in the air that make it look fuzzier. Things such as water pollution, fog, air is an interference When cues conflict Here is a 2D image of 3D world? ∙ what information can we use here? o texture grass, ripples on the water o linear perspective o partial occlusion of trees of mountain; bank partially occludes the stream Dynamic cues: Motion parallax Dynamic cues: optic flow dynamic cues: deletion and accretion shows us one object is behind another we talked about ocular motor cues, momocular cues (Static cues and dynamic cues) March 21, 2017 Monocular most be other cues Binocular disparity or retintal 5 min Furthermore because our eyes don’t move in our heads, we can rely on the distance. We can always rely on the distance of the difference between those eyes. They eyes will always be stuck in our skull in the same exact space. We have perspective on the world from two different places. 7min What does it mean to have two different pictures of the world? Hold up your thumb in arms distance. Close right eye. Then close their left eye. When you close your right eye the picture of the star moved in respect to your thumb. The two eyes have two different pictures of the world. This is monocular disparity. Differences in the images actually allows us to compute depth. Depth from Retinal Disparity ∙ How does the visual system achieve this? ∙ First, lets establish how points in a scene fall on the two retinae ∙ Falling on the fovea of one eye and falling on the fovea of the other eye ∙ There are a whole set of corresponding points that fall on the eye ∙ Zero disparity corresponding points fall on the same position on the two retinae relative to fixation ∙ This is called the horopter Noncorresponding Points ∙ Anything inside or outside of horopter falls on noncorresponding points on two retinae ∙ This yields retinal disparity different from 0 ∙∙ Three types of Retinal Disparity ∙ zero disparity: object falls on corresponding points in the two eyes (on the heropter) ∙ Crossed disparity: objects bearer the fixation point ∙ uncrossed disparity: object further from the fixation point Magnitude of Retinal Disparity The magnitude of disparity: The distance between the images of the object on the two retinae. The farther the object from the horopter, the greater the disparity between its images on the two retinae hypothesis two: do not need to recognize the object to see depth March 21 closing one eye world didn’t look flat but harder ot get around You have the experience of depth but something is missing. somewhat intalligable. diffuclt ot naviage and interact with 3d world. all of those cues we take about can only approximate depth. cant make it easuly and readily interact with objects. walking around or cooking is difficult to do when we are only relying on monocular cues. so there has to be something else that we can rely on expect these monocular cues. with just mono something is lacking. Today we will be talking about What it is the provides the more precise accurate depth information that actually allows us to take our fingers and grab something or something that allows us to interact with our 3d environment. That is binocialr dospaorty or retinal disparity. That is a cue to depth that is based on retinal image in both eyes. the disparity within the two eyes. we have two eyes and they are not overalapped in space. physically two separate eyes. they have two seperte views of the world. two images of the world are slightly different because different images. the images are slightly offset because of seeing from different angles. of you take a picture with a camera an then move it over 6 incees those images will not be the same, bcs you took two slightly different view points. two different copies of the visual scene. furthermore, bcs our eyes do not move in our eyes (not physically move in the skull) we can rely on the distance. we can always rely on disantce bcs our eyes will always be stuck in our skull in the exact same place. we know the distance between the two eyes. it is not going to change, distance bwtween the two eyes is consistent. we can use this information. critical info. perspective of the world from two places. use the information to figure out distance from objects with a lot of precision .line up thumb with star both eyes open, close lieft eye, then right eye. what happens to the image of the star? when close the right eye and left it moved with respect to the thumb. depends on the distance. the two eyes have two different pictures of the world. this is retinal disparity. difference of images allows us to compute depth. how do points in the scene fall on the two retinae. so fixate the star with both eyes. star is on the fovea in both eyes. falling on the same position of both eyes (falling on fovea of both eyes). so when we fixate the star there is no difference on the position of both retinaes. stimuli like this that fall on the fovea of both eyes fall on corresponding points in the two eyes. correspodong pints do not exhibit disparity which means there is no difference on location in mage of eye 1 and eye two. there are a whole set of points that fall on the corresponding points in both eyes. lets say you fixateon something in the distance, lets say me and you draw an invisble sphere through the point of fixation so it makes a cirle sphere of the same distance from you as the point of fixation so fixating on something, draw a sphere that is the same distance from me) those points ends up falling on the same postion on the two point. they are called xorrespodning points. they have zero disparity corresponding points falls on the same position on the two retinae relative to fixation zero disparity. this invisiable sphere is called the horopter. horopter. points that fall on the horopter fall on corresponding points on the two eyes. that means they exhbit zero disparity. no difference in location in the two eyes. so this is obviously just a small portion of the image, so fixation and horopter (invisible sphere around fixation) so any image falling on the sphere, any object falling on that sphere is going to exhibit zero disparity. anything outside the horopter, either closer to you from fixation or further from you are non corresponding points. non corresponding means image of the object in the world falls on different positions in the two eyes. non corresponding means the image falls on different locations in the two retinae. this yields a retinal disparity that is different from zero. so lets take a few examples. the book has a lot of good diagrams. lets say you are fixating the red pill. we all know the image of red object falls on the fovea when we fixate on it. it exhibits zero disparoity. the blue pill on the horopter projecting to point b on the left eye. and point b prime on the right eye. that means they are falling on corresdspong inf ppoints. that is what falling on the horropter means. if you see an object that is farther from you or closer to you you see they are falling on different parts of the retinae. the image would be on two different positions on the retina. that what it means to have zero disparity. that’s what happens when points on the outside and inside of the horopter fall on the retinae. when I am fixating on something. things that are in front of what I am fixating on up crossing over. things that are in front of my fixation are going to jump over. things that are in front if im looking with my right eye will jump to the left and things that I am fixating on with my left eye will jump to the right eye. Right eye: what’s behind fixation is on the right, what’s in front crosses over to the left. Left eye: what’s behind fixation is on the left, what’s in front crosses over to the right. when I close my left eye, left eye looks to the left of the right thumb. closing my right eye, the star is to the left of the front thumb. when looking with left eye looks to the left. when look with the right eye looks like it is to the left. @24 min Three types of retinal disparity Zero disparity: objects falls on corresponding points in the two eyes (on the horopter) Crossed disparity: objects nearer the fixation point Uncrossed disparity: object further form the fixation point Not only need to know if the object is closer or farther away from fixation, but also need to know how far away from fixation an object is (not enough to say oh it in front behind). Need to know the magnitude of the difference between the images on the two of eyes. The magnitude of disparity: The distance between the images of the object on the two retinae. The farther the object from the horopter, the greater the disparity between its images on the two retinae. You can measure the magnitude of the disparity by taking the difference between the two retinal locations that the image falls on. we can do that by dividing the spherical object of the eye into angular distances. zeroe is the fovea and have degrees. measure the magnitude of the disparity of the green pill: right eye minus left eye. right eye it falls at about 10 degrees. in the left eye it falls at about negative 10 degress. 10 10= 20 degrees. un corssed disparity is always going ot be greater than zero. brown pill: rught (50) left (60)= 5061=110 that is the magnitufde of the disparity. crissed disparity is always going to be less than zero. Putting it all together ∙ Images on the two retinae can fall on locations that correspond exhibiting zero disparity, or locations that do NOT correspond exhibiting nonzero disparity ∙ The type (crossed/uncrossed/zero) and magnitude of the disparity tells us about the depth of the scene (how big is the difference between the two retinae) ∙ But when we have disparity, how do we figure out which points match up? When we have disparity, how do we figure out which points lines up. corresponding points make a lot of sense. where it is in one eye is where it is in the other eye.non corresponding points (points that respond in two different positions in the two retinae) now we have a little bit of a problem. how do we know that the image here on this retinae should be matched up with a different location on the other retinae. this is the correspondence problem. how do we figure out which non corresponding points match up with each other. how do we solve this problem? We can try to figure this out by very tightly controlling information that we get from information we put into one eye and information we put into another eye. we will do this by using stereograms. this is an early entertainment device where you take a viewing thing like this. viewing master. divides the two eyes so this eye can only see what is over here and this eye can see what is over there. two slightly offset images to the two eyes (two 2d images to the two eyes) artificial disparity where one eye is seeing one image and the other eye is seeing a slightly altered image and this barrier is making it impossible for the eyes to see the other image. what you end up with is a situation where images start to pop out at you. the modern version of this is analyglyphs. align the images side to side, add filter to each image (filter out red in one and cyan in another) and what you get is an image that can pop out if people look at it using 3d glasses. The problem that we are dealing with is the correspondence problem. how does the brain figure out with noncorresponding points figure out match up with each other. to solve this problem we need have to control over the two different images the eyes get. Kind of see the image popping out. What does this mean for the correspondence problems? Optimal ways to solve the correspondence problem. Option 1: visual system does 2d object recognition for each retina, labeling each object in the scene, and once it has those labels It says the coffee mug in one eye has to match up to the coffee mug in the other eye so I will match those points together, once I have matched those points together and labeled those objects in two dimensions I can then compute disparity. Visual system first recognizes the object in a 2d plain and then figures out disparity by matching up those 2d labels (identified objects) in each retinal image. Option 2 is that the visual matches part of the visual image maybe based on very simple properties such as color, orientation, spatial frequencies, ect. once it does that than it computes depth disparity based on those very simple features (things like spatial frequency), then object recognition processed from there. so once we have the depth info than we can detect the identity of the objects of the scene. so this depth first hypotheseis. under this hypothesis you do not need to recognize objects in order to see depth. you can process depth first and then recognize the object. these are our two hypothesis that we are working with. so does object recognition come before correspondence matching or vice versa. so do we need to recognize objects in order to see depth or can we compute depth and then detect object later. hBela Julesz pionerring visual scientist who figured this out through a clever method cyclopean vision we see the world as if we have only a single eye we never sense that we have two eyes but the two eyes give us slot of really important information. and he had an idea of how to solve how the brain solves this corresponding problem by using stereograms but of random dots. he would generate a series of random dots so in this case it is just pixels and then he took a subset of the dots, one is white, two are black and in another image shifted the subset of dots and random dots were filled in from where they were shifted. so you can imagine what is happening here. he Is sort of creating a disparity situation, what he did was showed one of these images to one eye using a stereoscope and another image to another eye. what he predicted was that if it was the case that we can see depth without recognizing objects first we should be able to perceive a little square that’s popped out in from of the background because this box is offset in each of these images. it will be in different positions on these two retinae mimicking the disparity in 3d space. notice that there was no actual object here. so if this works, if it the case that we a square floating in front of another square, that would suggest that we can compute depth without actually identifying or recognizing other objects. square, x, heart What does this mean for the Correspondence problem? How does the visual system solve the correspondence problem? Which of our hypotheses was correct. option 2. we do not need to recognize objects in order to see depth. this is proved because we were only able to recognize which object it was once the depth was computed. once we actually computed the disparity information from this random mess by titling controlling what goes into each eye. only then were we able to recognize the object. we do not need to recognize objects to see depth. so what the brain is most likely doing matching up the two images on very low things, things such as spatial frequency. but how does the brain actual solve the correspondence problem? Cells that in the ocular dominance columns that respond to one eye. what type of cells fall in between those ocular dominance columns. cells that fall between both eyes. binocular cells. cells that get their input from both eyes. what does it mean for a cell respond to respond from input from both eyes and how does this solve how the brain solves the correspondence problem. this is a diagram of the binocular cell. the RF of the cell. cell has its RF in each eye *one in one eye and in a different location in RF of other eye). So you can see the distance between the RF and the fixation point are different for each eye. when we are fixating an object and there is another object in the visual scene hat is falling somewhere on the two retinae. cell only responds when the image of the object in the world, when the image falls on both RF of that one cell. and the image will only fall on the RF on that cell in each eye when it is at a particular distance from the fixation point there is only one way in the world that that object can be that that same image can fall the RF A and B of this cell. that is when the binocular cell responds. RF of cell is photocrecpeotrs for this eye and photrecpeotrs for that eye. BC only respond when image falls on their retinae which might be different locations in the two eyes. BC might have RF in the same location of the two eyes. that means they are responding to an image that falls on the horopter. BC are found all over the brain. We have talked about how we take an image like this and actually compute death. If we were actually standing in this market there are different cues that we can use to compute depth, monocular cues and binocular cues. monocular cues, oculomotors cues, lens accommodation and convergence, it also includes cues that were based on the optimal image so things like texture gradients, relative size, linear perspective, and dynamic cues (how objects move relative to the world). that gives a little bit of depth. but in a real 3d world the intangible quality or precision of depth is that we have two eyes (retinal disparity) computing depth through binocular disparity or retinal disparity gives us very precise where things are relative to the fixation point. Up until this point we see have been talking about how we see a static image, an image that is sort of a snapshot, we have been talking about static image. next time we will be talking about motion. so not just the world we see it but how do we compute an object that has moved across our visual field. how do we track that object. how do we track objects in motion 3/28 Physical properties of sound ∙ sound results when particles in a medium crash ∙ sound waves are transmitted through air and other media ∙ sound becomes weaker further from the source, because the same energy has to spread out over a greater areas ∙ the frequency amplitude, and form of the waves with the sound source Physical properties of sound how do they relate to perception? ∙ frequency measured in hertz (hz), cycles/sec perceived as pitch ∙ amplitude measured in decibels d(B) perceived as loudness ∙ waveform perceived as timbre Amplitude (loudness) ∙ Decibel: A unit of measure for amplitude (sound pressure level) ∙ damage zone! prolonged exposure can cause hearing loss External ear to middle ear ∙ in response to tympanix membrane the 3/23There are two images of the world on the two eyes that don’t necessarily match up. There is disparity between the images on the two eyes. From that disparity we can compute depth. Images that fall on the fovea of the two eyes exhibit zero disparity. Images that fall on the invisible sphere of fixation exhibit zero disparity. this is called the horopter. objects that are closer than us then the fixation on the horopter exhibit corssed disparity and further away exhibit uncrossed disparity. We also need to know about the magnitude of the disparity (how different the location of image 1 is in eye 1 or eye 2) to tell us about depth. WE talked about binocular cells that we can find all over the brain which have field that spans in two eyes. so a location in one eye and a location in the other eye. so these binocular cells have a disparity preference. they will respond when a preferred stimulus falls at a particular depth with relation to the fixation point. cells with RF that spans two eyes. ∙ Up until this point we talked about perceiving the world, but all of the principles we talked about with the exception of a very few really apply to just a single snapshot of the world. opening your eyes and just glancing. we havnt talked about the dynamic nature of the world. lets say you are selena Williams and your task is to hit a tennis ball that is approaching you. with just a snapshot you have depth color. but if you need to move your body hopefully hit the ball you have to not only track the object but you are moving also. so things are moving across your retina and things are also moving in the world. as you move things move across your retinae and things in the world move across your retinae. This is a big challenge. Imagine trying to figure out what is moving, what is stiationy, and how do I interact with it. That is the challenge of motion detection. we have to identify that the object we saw in one point in time is the same object we saw another location at another point in time. tennis ball is the same object and is coming torward me. So that is the challenge. and we have to do this not only as objects move in the world but also as we move in the world. How do we do this? We will talk about this today. Some of the initial works starts in the early parts of processing, especially V1. you might remember us talking about motion selective neurons in v1. cells in V1 that is tuned not only orientation but particular direction of motion. vertical lines that is moving horizontally across the visual field. how does motion perception happen in these neurons? how do these neurons actually detect motion by going back to single cell recording technique to see how one of these V1 neurons will respond to a moving stimulus. we want to map motion selectivity of this neuron by showing in the RF of the neuron showing various moving stimuli and measuring the response of the neuron, this is the classic single cell recoding technique. so we are going to vary the direction and speed of motion and see what happens. first stimulus, and we measure activity of the neuron and there are a few spikes but no real difference from baseline activity. when we do it from the other direction instead of having the ball move to the right we have the ball move to the left in the diagnol fashion we have baseline activity. Speeding up the motion of the light (the dot) downward in a diagnol fashion towards the left but faster. we end up observing an increase of activity from this neuron. if we put stationary stimulus in RF we find again baseline activity. this tells us. That means this neuron is tuned to motion with a particular direction and speed. So we are putting this stimuli in a certain part of the RF. Photoreceptors make up the RF of this neuron. How does this RF have to be organized in order to detect motion? since there is something specific and dynamic happening. this particular cell only likes something from the top right to bottom left at a particular speed. doesn’t like a stimulus that goes from the top right to bottom left at a slow speed. doesnt like a sstimulus that is just sitting in the middle of the RF stationatry. doesn’t like a stimulus moving in a different direction. how does the RF of the cell need to be organized in order for it to be selective to a particular direction of motion and a particular speed of motion. The neuron needs to be able to respond to an object that is not necessarily always in the same place, but moving in a particular direction in a particular speed. so lets say we have a ball that is moving across the visual field vertically at a particular speed going from point a to point b. we have the image of the dot moving across the retina moving from location to b prime. so the image of the object is moving across the retinae. that means the image of the object (Since the object is moving at a particular speed and direction) is also moving at a particular speed and direction. we can not be a motion and speed detecting neuron if we only have a RF that is only in one location on the retinae. one undiferentioned set of photrecepotrs will not be able to tell use about motion and speed. what we need is something that can give us a sense of where the object is located over time and when as the image moves across the retina. so if we are a neuron that wants to detect motion we need at least two locations that make up the RF. we need to be able to be listening to cells in location one and listening to the cells in location 2, in order to find out if the stimulus appears in both locations and when does the stimulus appear in both locations. if the stimulus does appear in both locations it would suggest that it is going in the right direction, and if it appears in the two locations at the right times it will suggest that it was going at the right speeds. how is a motion selective RF wired up? One possibility is a circuit, we have seen these kinds of things before when we talked about the RF of RGC, LGN, V1. seen this diagrams before where M is our motion selective neuron that is getting its signals from neurons that are listening to photoreceptors in the RF. So here is an example of a neuron that has two locations for its RF (RF1 and RF2). We want to see if this particular neuron can detect motion in a rightward direction. when we pass a stimulus through the RF of this motion detector. we will see if this type of circuit is enough to illicit a response from this neuron. not giving us a single that is illiciting a great response from baseline. the stimuli can also be moving in another direction. we would illicit the same exact response if the stimulus was moving in the other direction. Pattern of a little increase, drecrease, increase, decrease is very similar to when stimulus is going in the other direction. so it is not sufficient to just have two receptive fields wired up to the M neuron. we need something else. this does not give us direction speed. in order to make the circuit directionally to be able to identify the direction of the motion and the speed of the motion all we have to do is add something to one end of the circuit. and that is a delay in the signal. when added a delay here to the circuit in one side. so we have our stimulus again passing through RF1, then the delay keeps the stimulus delayed in RF1, then the stimulus to RF2, this neuron just listening to RF2 sends an excitatory signal. meanwhile the signal with the delay has finally caught up from this signal from this side. the signal from RF1 and RF2 arrive to M at the exact same time. so now M is getting to excitatory signals from its two RF at the same time. it is now going to signal the presence of its preferred stimulus. and that signal is going to be easily discrimitable from its baseline firing rate. so having the delay solves the problem of the signal not really being different from baseline. now you can see that it is different from baseline. the delay also takes care of the problem that this circuit was not sensitive to direction of motion since a stimulus going in the other direction would illicit a similar response. here if we do the same thing from the other side, send an exciatory signal from RF2, M increases firing a little bit, goes back to baseline, stimulus appears in RF1, hits the delay, after a delay, the signal increases a little bit. so when we go in this direction, stimulus appears in RF1, it sends an excitatory signal which is held up by the delay, stimulus is held up by RF2, sends exciatory signal no delay, signals arrive at M at M exact same time and get a big excitatory signal. not the opposite diection something different happens. a little excitation, back to baseline, caught up in delay, back to baseline. we have a different signal being sent by M when it is being sent in the preferred direction versus the opposite direction. so now we have a circuit that can discriminate direction of motion. now what about speed? how can we vary the speed of motion this neuron is sensitive to? very the delay on RF1. delay longer for slower stimulus and delay shorter for faster stimulus. so the direction of motion sensitivity is define by which neuron has the delay and the speed is defined by the length of the delay. So we have a simple circuit that is capable of detecting stimulus motion in a particular direction with a particular speed. now the question is, is this how we do it? is this how the visual system achieves motion detection? it is a possibility. in order to answer this question we will have to a psychophysical experience ourselves. what is our sense of motion? so fixate on this point in the center. when you look away everything is warping. one thing to notice is your perception of motion in the absence of motion, it was in the opposite direction in the stimulus we just saw. we are perceiving motion in the absence of motion and it was in the opposite direction of the motion we watched over and over and over. what is that? it is an after affect. where else have we seen an after affect? in color. why do we get color after affects. so why do we get color after affects? we have color opponent circuits. let say we have a red green color opponent circuit. continued stimulation of the RF of the neuron with red green color opponent circuit, when we put red light on the red green color opponent RF, we get an exciatory signal. and if we put a green light we inhibitory signal of the neuron in either direaction. that signals the presence of the red and the green. when we stare at red over and over and over we get color fatigue, and then we look at a white light, we then see green. why do we see green? because the red and green are in opponent process. so the same circuit that detects red with a excitatory signal detects green with an inhibitatory signal. that allows us to see color. what it means is that fatigue of the red portion (the red cells) in this case the photoreceptors result in an after image of the green. That is an opponent process. how do we explain motion after effect? we have out imple circuit. but it is not enough to explain motion after effect because what we have here is not an opponent process. this stimulus, this set up only detects one type of stimulus. a stimulus going in a particular direction in a particular speed. in order to get an after effect we need to get a particular circuit that is wired up to detect opposites. direction and speed in one direction and direction and speed in other direction. we need an opponent circuit. an opponent circuit explans motion after affection. we can build our opponent circuit with a simple circuit, which is what we discovered that we need a delay on a typical side of the circuit to detect a particular direction of motion and the delay needs to be a particular length to detect the speed of motion. so here is our motion opponent circuit. for our first example, we will go RF1 to RF2. so we have a horizontal dot moving across the 2 RFs of this detector neuron. we have overlapping simple circuties. we have 2 RF, we have neurons that are reciving signals from the photoreceptors in these RF. and those are subsequently sending their signals together to motion detecting neurons. there is a delay on one of the paths and no delay on the other path. this part of the diagram is two simple circuits. stimulus appears in RF1. neuron 1 sends excitatory signal to this motion detecting signal. excitatory signal ML and excitiory singlet to Mr with a delay. D ultimate motion detector neuron. so we get a little increase. the signal with the delay will get held up, now the stimulus moves to RFs to 2 and sned to Ml with a dekay and Mr with no delay. the signal of 1 and 2 are arriving at Mr at the same time. Mr now send inhibitory singal to D. D gets a little excitory dip, than big inhibitaroy dip from Mr. now the signal on this delay, we get a little excitory peak in D. what we see is little excition, big dip from baseline, and then a little excition. that signals motion in the rightward direction. righwatd motion activity in D goes down from baseline. ∙ for opponent ciruit to really be an opponent circuit it must detect a signal from one direction and the other direction. signal appears in RF2. 2 sends excitatory signal to Mr which directly sends inhibitory signal to D which causes a little dip. 2 also sends excitatory signal to Ml with a delay, then stimuli moves to RF1. 1 sends excitory signal directly to Ml that signal arrives at the same time from the signal from 2. now we get a big excitory signal from D (big increase in the activity in D). Meanwhile, ou excitory signal from 2 is on a delay it gets a little blip in D. We get overall net increase in D for motion to the left. Decrease activity goes down for rightward motion and it goes up for leftward motion. (around 38 min) ∙ lets think about what happens if I show this neuron (measuring RF of the neuron) presenting a stimulus in the RF of the neuron a left ward motion, so staring at leftward motion. what starts to happen? what happens when we stare are red light when we stare at over and over? photoreceptors become fatigued. here we are talking about neurons. but neurons can get fatigus. showing leftward motion over and over. the leftward direction circuit will start to get fatigued. so this pathway is getting fatigued. what is happening when nothing is goin on in this circuit, baseline activity? what happens when we have stimulus falls in the two RF at the same time (uniform stimulation)? what happens? we migh have excitory signals form ml and in inhintory signals. they cancel each out. when we fatigue one side (by showing leftward motion over and over) what happens? the left side stop sending the singal. Mr will send inhibitory signals. for our perception , when we watch something moving left war over and over and then we watch something stationry we perceive rightward motion. this is because it is an opponent circuit. if we fatigue, the side that perfecrs leftward motion perceives rightward motion. ∙∙ we have been talking about detecting an object in a particular speed in direction. so a tennis ball moving at a particular speed horizontaly. but we have neglected an important part of motion. we have come up with one direction. we have been working in 2d dimensions. we need to detect depth. important for motion. add that dimension to make sure nothing is flying at our face. motion and 3d vision go hand in hand. you can imagine that the problem we are dealing here. we have to detect motion in partocualr field in two directions. what if ball is not coming across viuals field but towar us. the problem is ..ball moving toward is at a particular angular is same distance like moving across retinae in a particular direction. how do we solve this ambiguity? we need two eyes and cells that have RF that tuned to direction, speed and motion, and to disparity. as we learned last time disparity tuned cells. disparity tuned cells are present in v1. so are motion tuned cells. cells that are tuned to motion and disparity (toward or away from us) that work starts to happen already in V1. we have been talking a lot about V1. V1 got disparity and motion tuned cells. RF in V1 are small relative to IT. V1 has small RF bcs V1 getting building blocks of objects. but this can pose a problem if we are only seeing snippets of motion. watching figure skater lots of motion on the same figure but only viewing it through peephole.s how do I put that infor together into a cohesive figure? this is the aperture problem. V1 is only getting snipets of info. not enough to put cohesive figure together, such as a figure skater in motion. we put all of this together to gleam motion on whole figure in middle temporal area (MT). we move out of V1 into dorsal pathway (where/how oberjects are and how we intereact with them). V1 (direction, spped of motion, depth) motion and disparity signals get into dorsal pathway specifically ending up in are MT. motion signals get sent to MT from V1. dsparity signals have stop n V2 anf then MT. MT has larger RF. putting together little appertures to recive large cohesive image of motion. ∙ ∙ how do we know mt is responsible for motion (the only thing it does is preccsing motion)?blank screen, moving doits, stationrory dots V1 repsonds to moving and station dots (that makes sense to stimuli , oriention, motion) ∙ in mt the only thing mt responds to is moving dots. only job mt has is motion. ∙ we can look at inficiudla MT cells, this cell responds best to motion in 135 degress (MT respod to direction and degree) ∙ how sensitive MT neurons are to motion? how much coherent motion is needed to get response from mt? how sensitive mt is to sensing change in noise? varying percentage of dots moving in coherent manner. at certain point can detect direction of motion. map out motion coherence. when is it enough to illicit response of the neuron? that tells us the sensituty of the neuron. that is the motion cohernence experiment. ∙ monkey just need 13 percent to detect motion coherence. ∙ MT is very sensitive to motion. only 13 percent. not very much. MT starts put motion together.∙ lesions to MT strobe light situation (image ot image ot image) people that have lesions in MT describe the world looks like there (looks like snapshots) how do you pour a cup of tea. move up the tea. ppl with lesions they are pouring they are seeing images not continuous. very debillaitng. cant drive or interact with the world ∙ We have neurons tuned to motion direction, speed, and disparity in V1 ∙ These cues combine in MT to allow us to see continuous motion across time and space o – MT has larger RFs than V1, and can combine information from many V1 cells ∙ BUT – there is still one potential confusion: Are WE moving, or is the WORLD moving? The problem is lets say I am station moving object. when I have stationary object and moving y eye. I also get an image of object moving acors the retia. same thing happens when I move and when the object. same image when we are both image. how do we distinguish our own motion or object motion? one way is the corollary discharge theory. when we move our eyes a signal is sent ot areas of the brain that controls eye movments to areas of the eye to tell the muscles to move our eye. superior colliculus is responsible for sending signals to ocular muscles ot move the eye . SC says move your eye what happens? we want the motion detecting parts of the brain to know the SC told the eye to move. SC sends copy of the signal to areas of the brain that are responsible for motion. by the time the signals from the eye arrive MT already knows the eyes have executed a movment. take it into account when get inout from the world. MT is getting information about what is moving in the world from V1, visual pathways . MT is gettinginfo about how the eyes are moving thorugh the corollary discgareg signal from SC. Signals the eyes to move, then sends a copy of the signal to a region near MT. MT takes that info into account from the info it gets from earlier processing areas. bcs it takes the eyes a while to executew eye movments, MT gets it before visual stimuli get it. MT is prepared. how we distinguish between things that moving in the world and things that are moving because we are movung at 2 min This isn’t the whole sotry. what if our eyes are station but we are in car. things are moving around us in a car, we aren’t moving but car is. can fix eyes. we have figure out the possible ways we can detect objects we see in time 1 space 1 is the same object we see in time 2 space2, how do we do that, how does the motion system detect motion, and how do we explana motion after affect, and how do we do this when we are ourselves are moving in the world. 3/28/17Up until now we have only been dealing with perceputal modality, vision. so taking in a scene like this we have been wondering how do we take a scene like this and carve it into behaviorally relevant units that we can interact with. then how do we tract those movements through space and time as we move in depth. so things like boats and buildings. we have followed this trajectory from the physical stimulus in vison which is light, which then gets channeled to the sights of transduction. how does the physical stimulus get funneled to the sights of transduction? cornea and lens funnel to the sights of transduction. the cells that do the transducing are photoreceptors. so the light gets transduced into neural signals that then get sent via RGC then to the brain for further processing where which we eventually have perceotural experience of buildings and fireworks and so forth. we have talked about how we do this in the visual domain to see things such as color, depth, motion, ect. which gives us a really rich experience of the world. but of course that is not our only experience of the world. when we are sitting on espinade seeing a scene like this we do not only want to see the fireworks but we also want to hear the fireworks bang. all of those things are critical for our perceptual experience. not only are we going to be able to detect those sounds. we need to be able to identify those sounds and figure out where they are coming from. it is not enough to say that I hear a bang or boom. we need to be able to know that the boom is coming from here. the symphony is coming from somewhere off this scene. the fog is coming from this boat. so we need to be able to identify the sounds and localize them. so this is very similar to the problem we were dealing with in vision. we have to take a visual stimulus that is coming from the world and we have to detect that visual stimulus. we have to transduce that physical stimulus into signals that the brain can understand and then we need to make sense of those signals. we need to figure out what we are hearing and where that sound is coming from. so that is the task we are going to start thinking about today. so first of all what is the physical stimulus, how does the auditory system detect and transduce it into signals, and then how does the brain start to make sense localizing and identifying sounds in the world to achieve our perceptual experience of watching the fireworks. so lets start with, just like we did with vision, what are the properties of light. so the physical properties of sound. what are they? Sound results when particles in a medium crash into each other, transmitting a vibration, resulting in sound waves. we have a sound source like this blast from a laptop. When we have sound source that sound source is vibrating. so lets take the case of a tunning fork. you take a tunning fork, you hit it on something, and it vibrates/ osscilates back and forth and creates a tone. so that vibration displaces particles in the air. so as it vibrates in one direction is compresses the air particles together. and is it ossiliated in the other direction it creates more space for the particles creating rarefractions, just the particles spreading out in space. so compression is the crashing of particles. what that means is that these particles that are crashing into each other are transmitting that vibration to other particles that are in the medium, in this case just air, resulting in a wave of particles crashing into each other. this means that in order to transmit sound we need to have a physical medium that contains stuff. that means air is made up of particles. water is made up of particles. we need to have those particles for those sound energies to ransfer things. so sound waves are transmitted through air and other mediums. depending on the medium that the sound is traveling through the speed of the sound can change. sound travels at different speeds through air, water, wood, iron, stone, ect. so depending on the desnity and rigidity of medium we get different speed of sound, some medium is not very dense but rigid like rubber or foam. sound travels very slowly for those substances. which makes sense because we use stuff like rubber and foam to insulate. sound does not really travel through those mediums. this means that unlike light sound cannot travel through a vaccum. there needs to be some substance to the medium through which sound is traveling in order to transfer those occislatiosn from particle to particle. this is unlike light which can travel theoriteically infiintlity through vaccums. this enables us to see things billions of light years of away. we cant hear things that are billions of light years away. unlike light, sound cannot travel in a vaccum. it needs a physical. sound can be reflected or absorded just like light. reflection of sound sounds like an echo. you can hear the reverberation happening in this room as I increase my voice. my voice is being reflected off of these somewhat dense surfaces. although they try to mitigate it by having these clothes covered. but my sound is reflected off of these surfaces resulting in a reverberation. but it can also be absorbed by media. so unlike light sounds cannot travel through a vaccum it needs to travel through a medium. but like light it can be reflected or absorbed by surfaces. another property of sound that is somewhat different than light that sound becomes weaker further from the source we go. that is because the same energy as we move away from the source spreads over a greater and greater area. so here is the sound source. as we move away form the sound source, the same energy twice as far from the source has to spread out at 4 times as much area. that means the energy must get weaker over space. so the further and further away we go from the sound dource the weaker the single is. that makes intuitive sense. we know from experience that the farther away we are from a sound the harder it is to hear. sound travels in a wave. the frewucny, amplitude, form of wave varies with the sound source. depending on the sound source the wave is going to look different physical properties, look different. the amplitude of the wave can vary. amplitude is the distance between the peaks and valleys of the wave, how high up versus how low. frequcy is how much the wave go up and down over a particular interval of space. amplitude and freuwncy can change and form can change can very by differences in the source of the sound. how does that impact perception? how does the physical dimension of sound wave varies physical perception of the sound. the frewuency of the sound corresponds to our perception of pitch. higher frequency, higher pitch, lower frequency, lower pitch. and the frequency of sound is typically measured in hertz. hert is unit of cycles per sec. more cylses per unit of time, the higher the frequcney. amplitude corresponds to our perception of perceived loudness. so here we have air pressure, on the y axis. and looking at how the wave travels and change air pressure (inc/dec). measure amplitude in decibels. finally wave form corresponds to the timbr or quality of the sound. so wave forms can be very simple. so in the case of a pure tone that may have been created by a computer program. a really pure pure tone. we make it a very simple wave. which is just a very simple increase and decrease. this is very atypical of nature. in nature we typically have complex wave forms. complex wave forms are made up of simple wave forms that come together to produce a complex sound. complex sounds are pretty much anything that is not just generated by a computer program. my voice producing a tone, a flute producing a tone, ect. depending on the wave form we will gave different perceptual experience of the sound. the flute and the violin producing the same tone, may have different complex wave forms resulting in different sound quality. frequency responds to perceived pitch, amplitude corresponds to perceived loudness, waveform is to the timbre or the quality of the sound. for light we talked about the entire the electromagnetic spectrum and how we can only perceive a small part of the electromagnetic spectrum. that is the physical portion of light. simillailry with sound we cannot perceive an entire range of frequencies that can possibly be produced. we only have access to a subset to all the potential frequencies that can be used. so in green this is the range of frequencies that can commonly be heard by a healthy young adult (20 to 32,000 hertz). jus tlike light, different anmals have difffernt access to different portions of sound. dolphins bats and moths can hear the higher frequency sounds. snake has a very narrow portion of sounds. there is variability in animals because of what would be relevant bcs what tyoes of sounds would be relevant to their survival. snake is on the ground, the type of vibrations that are transmitted through the ground, we are getting vibrations transmitted through the air, difference sources of sound, anatomical differences, evolutionary needs have shaped the range of sound that we can hear. I also want to point out that human speech sound represent only a small portion of all the sounds we can hear. we can produce the sounds around this range. we can hear many more sounds. we produce sounds that are only a small portion of the sounds we can hear. that is true for a lot of different creatures as well. amplitude. measure amplitude in decibles. we are going to measure amplitude in decibles. decibles is just a connvient way of representing the perceptual properties od sound. we do not only want ot measure the change of sound in just change in pressure. this increases in sound pressure might yield different increases in perceived amplitude of sound or perceived loudness of sound. so we talked about this kind of thing when we did physiscs and webers law. decibels represents the increase in perceived loudness as opposed to increase of total sound pressure. it is not quite as simple. it is a better capture of our perceptual experience of sound loudness. so as you can see down here we have things such as background noise, a whisper, normal human speech, passenger car passing down the street, up here we have a medium tuck, heavy truck, jack hammer, a rock band, airplane taking off. prolonged exposure to above 85 decibles can result in damage to your hearing. wearing earplugs at concert bcs it can result in damage. when on airplane and have to crank up headphones to hear over the sound of the airplane. and when you get off the airplane and press play think about what is going on in your ears. this is sound related hearing loss. putting together ideas of frequency and amplitude together. that fact that we can hear a range of limited frequencies that we can hear. what we can do is we can figure out how loud the sound needs to be at a particular frequency for us to be able to hear or detect. can do an experiment or use psychophysical methods , and we can play poure tunes to a subject of a range of frequency . and we can gradually increase the amplitude of the tune to ask the subject do you hear it. was It detectleable or not detectable. by doing this we can attain the absolute threshold for detection of a tone at a particular frequency and plot it on the audinility curve. we are not equally good at hearing all frequencies? no. as we vary frequency, increase it, we see we need different amplitudes to be able to minimally detect that tone. at 125 hertz we need to crank up that amplitude to about 22 decibels to just minally detect that tone. if we look at 1000 hertz, amplitude just needs to be Acutally 0). 16000 heart tone has to be 35 decibels in amplitude to minimally detect it. above audibility curve sounds are audible. below audibility curve sounds are not audible. just on this curve sounds are barley detectable. 0 db is defined as the min amplitude required for perception of a 1000 Hz tone for a young person with typical hearing. what frequencies are we most sensitive to? the middle frquences bcs that is range of himan voice is in this range. we are not so good at detecting really really low and really relaly high pitched sound. we need a lot more amplitude to detect those sounds. Waveform corresponds to our perception of timbre, which is just the quality of the sound. any natural sound producer is going to produce a complex wave form. complex waveform produced by a flute, produced by a violin. we can look at the range of frequencies and amplitudes produced by an instrument or any natural sound source. when they are both producing the same pitch. in this case both producing a g. both of these instrumetns are producing a g. the fundamental frequency corresponds to what the note sounds like to us. in addition of producing a fundamental frewucny of g these instruments also produce a set of harmonics that vary also in frequency and amplitude. two different instruments producing the same fudnametal frewuecny will prodice the same set of haromoncs with the same frequnceis but with different amplitudes. so the flute playing the g will produce fundamental frequency g, with a set of harmonics, and set of amplitudes. the violin will lay the frequency will produce a set of harmonics of the same frewucney as the flute but will vary in amplitude. that variation in amplitude of harmoncis results in a change in the quality of the sound. that is why a flute playing g will sound different than a violin playing g. you can tell the difference between the flute and the violin. that voice singing g and ur voice singing g will sound different and I will be able to tell the differences between our sounds. even though we are the same type of instruments (humans) we will be producing different amplitudes of harmonics resulting in a change of quality in the sound. in the real world it is extremely hard to encounter a really pure one, every sound we encounter is pretty much going to be a complex wavelength. Fundamentals and Harmonics Different physical producers of sound, playing the same fundamental frequency, will produce the same set of harmonics but with differentamplitudes. This affects the Embre, or quality, of the soundSo we understand the physical stimulus. so how do we detect, transduce, and then ultimately hear. we understand physical stimulus, copponents of physical stimulus that correspond to a perceptual experience (freiqucney to pitch, amplitude to loudness, waveform corresponds to timbre/quality of sound). those are the key critical pieces of our perceptual experience of sound that enable us to tell the difference between a violon and flute, human voice, ect. when we keep all of those things in mind what happens in the perceptual system in all those different stages. we talked about the basic steps of perceptual processing. the physical stimulus, sense organs funnel physical stimulus to receptor cells in sensory surface which transduce the physical stimulus into neural signals the brain can understand. nfomation from the physical stimulu s in the world is encoded and transmittd which eventuall gives rise to perceptual experience. as the brain processes thoise signals. we talked about how we do it with light in vision. how do we do with in sound? First we have to capture and detect sound vibrations. Be able to detect differences in frequncy, amplitude, and waveform. We have to keep this info organized. We don’t want to lose info about frequncy, amplitude, and waveform bcs these are the crital pieces of our perceptual experience. so we need an autdioty system that can detet differences in frequncy, amplitude, and waveform and keep that information organized as we send that onto the brain. finally, we have to transduce the signals into the neural signlas that the brain can understand and process further. ultimately giving rise to our experience of what it is and where it is coming from. Step 1: how do we capture and detect sound vibrations? diagram of periperial audoio system. consists of part of the ear you can see, mainly the pinna, auditoral canal, middle ear, and inner ear, paritlaly imbedded in the skull. so for a sound wave which our goal is to be detected by a human observer, what we eventually want to end up in the cochles. in the cochlea are the cells that respond and move forward with transduction. so the sites of trandcution are in the cochlea. if we are a sound wave and we want to be detected by a human observer we need to make it to the cells in the cochlea that are responsible for transduction. how do we get there? first steptis to be caputraed by the obserbed, so it can get funnled torward the sites of transduction. the pinna playing a pivotal role in that process. does the quality of the sound change as I move my pinna? notice how your perceptual experience changes. there are predators who can move there pinna to hear sounds of their prey. humans cant do that. Pinna capture the sound. funnel it to external interla auditoy cannal. external interlal audioty canal ends at tympanic membrane, other wise known as the ear drum. tympanic membrane boundry of middle ear. sepeartes external ear to middle ear. very important function. tympanic membrane is to transmit those sound ossicliaiton or sound vibbrations of the air from external audiotry canal to the coclea. how does the tympanic membrane do this? tympanic membrane lloks like this, can be variable and can be scared by ear infections. in respnd to sound vibration the tympanic membrane osscilates. it ossilates in the same freuwncy of the sound vibration it detects. tympanic membrane Is attached to a series of tiny bones attached to the middle ear. middle ear is air filled chamber betwenn tympanic membrane and coclea. tympanic membrane is occilating with sound vibrations. this oscialtion causes the three bones inside the middle ear to start moving in a particle way. the three bones are connected togrther that makes them act like a lever. the first bone attached the tympanic membrane, malleus/hamer, anvil/incus, and stirrup/stapes. the foot plate of the stapes sits right over opening in the coclea called the oval window. as the tympanic membrane occilates the bones in the middle brain move in a piston like amplyinf the vibration from tympanic membrane to the tiny surface of the oval window. so transmitting the oscikation of the vibration of tympanic membrane transmitting vubraton vua oval window to coclea and also amplinfinf sound. inside coclea is filled with fluid. going frim air to fluid chamber can result in sound energy as we go from one medium to another. in order ot mitage, the function of the oscialces to amplify the sound vibration so that it can be transmitted into the fluid of the coclea. function of the ossciles is to transmit the vibration form the tympanic membrane to the obal window of the cochle and ot also amplify that signal so we don’t have a big loss of sound energy. from going large surface are to small window confined with this position like motion creates this amplification. the ossicles with conjection with tiny muscles act as buffer to protect inner ear with very loud noices. the tensor tympani and stapedius in respond to loud sounds, those muscles stiffen up restrcitng the motion of the ossicles results in reduction of amplification of the vibration of tympanic membrane into the cochlea. so we are resticitng the motion of the bones so they cannot amplify them as much. very good with extremely loud sound to pull back transmitting that loud sound stops loud vibration. phycial procces. bigger amplitude means bigger motion. loud sound will make tympanic membrane ossicalte back and forth with greater amplitude occupying large space. protect inner ear from large movements to procect delicate cells. when you leave concert your hearing feels muffled. it is because of accousitc reflex. muffling is due to reduction of movment of tensing up of the two musles. it is a reflex. takes a little bit of time for it to happen so it does not protect from sudden loud noise. Finally, we have an opening the thorat that serves to equaly pressure between inner and middle ear. bcs this is a closed sytem there is a big change in airpressure outside. need to change pressure in the middle ear to equarte the pressure to the outer ear. other wise you get a build up pressure on one side (Affects eardrums ability to ossilate and transmit that soundlike on a plane). so opening the estachian tube by yawning can equal out the pressure (of outer ear and middle ear). medium on both sides need tobe the same so we don’t get a loss of enery so that means pressure must be the same. so we have funneled the stimulus, capture it with our pinae, we have funneled it through the auditory cannal, tympanic membrane is ossilating in response to sound vibration, ossilatin is transmitted and amflied by the ossciles into the coclea, this is where the transduction, the coclea is where we take the physical stimulus and transduce it into neural signals. so the cochlea is a coiled structure embedded in the skull. when we uncoil cochlea. it is a tube. what is going on inside this structure? the inside consist of multiple chambers filled with fluid. essentially connected chamber fille with fluid. between those chamber we have the cocllear duct, at between of choclear duct we have the basilar membrane. ssiting at the bottom of choclear duct is the baslare membrane divides the tympanic canal with the bottom portion and cochlear duct. sititng on top of baslar member is the organ of corti, which contains the cells that are going to be doing the transducing. running along the bottom portion of coclear duct is basar memebrane. it is frewucny selected. as a sound vibration is transitted into the cochlea the piston like action of osciccles is creating a traveling wave ossiltion through the fluid inside the inner ear. travelling osscialtion travelling wave through the fluid is causing the membrane to respond to the traveling wave. when laying on top of ocean, your body is moving up in down with the wave. that is what is happening with basilar membrane. because it is this long membrane It has particular physical proeprteis, it has a thicker narrower base and a thinner wider end base. the structure of baslar membrane and physical properties of sound make it so that the baslar membrane is tonotopically organized, which means different areas of the osilaor membrane will respond to tones of different frwincies. base osscialtes to higher frequnciy tone.s apex to lower frewuncy, middle resonds to middle frequency tones. if have a high freuwncy tone it means you can realy on only one area of the baslar membrane responding (the base). baslar membrane structure is thicker at the base, narrower and wider, and thiner at apex makes it optimally tuned to other tones. that means I can count on a positon (retintopic organization is organized that we can count on something in the world being on a particular place in the retina ) we can count on tones of a particular frequency activating in motion a a prticualr area of the baslar membrane. this is important when transdusing the frequencies. March 30, 2017Last time we were talking about our introduction to our audiotry system. WE talked about what is the typical stimulus. Havent gotten to transduction. We have just gone from physical stimulus and how physical stimulus gets funneled into the sites of transduction. so we have talked about sound traveling in waves. particles in a medium crashing into each other and we talked about the physical propeteis of sound that correspond to our perceptual experience. so sound waves have a frequency which corresponds to pitch. we have sound waves with amplitude which corresponds to loudness. sound waves of a wave form which corresponds to the timbre of the sound quality. we were talking about talking those three qualities and getting that information to the brain so that we can process things like a flute, voices. so the physical sound wave travelling through a medium in this example the medium is air it gets captured by the pinae, which is the outer part of the ear, funneled into the audiotiry canal where we get to the tympanic membrane. the tympanic membrane is the ear drum, which ossilates in respond to vibrations in the air, those sound waves. then the tympanic membrane is osscilating. tympanic membrane is attached to one of the three ossciles. one of tiny teeny bones, tiniest bones in our body. the function of the ossicles are to take the occilation of the tympanic membrane, which is in response to a sound source outside, a physical movement of the air particles that ossilaiton of the tympanic membrane, and amplify that signal and transfer it into the cochlea, so that is here at the oval window. oval window is the membrane covered opening to the cochlea. the middle ear is full of air. cochlea is full of fluid. so that amplification from the tympanic membrane to the tiny little foot plate, that amplification allows there to be no signal lose from going from air to water fluid. so that piston like action of the ossicles causes that vibration to be transferred into the cochlea. inside the cochlea (which is the coiled structure), transfer of sound vibration through the ossiclles causes travelling wave through the fluid inside the cochlea, specifically in the outer chamber of cochlea. canals containg perilymph, the liqud, that traveling wave of ossilation traveling thorugh the liquid in the cochle, hear diving bottom chamber here with this middle chamber cochlear duct, we have the basilar membrane. this is a membrane that is essentially siting ontop of the lower canal. and as the fluid moves through the choclea, as it moves with the traveling wave of ossiliation, the basilar membrane is moving with the fluid, just like a raft on the ocean. as you encounter a wave, if you are laying on a raft in the ocean, you move up and down with the wave similar to the baslar membrane. traveling wave travels through that perilymph, that liquid, causing the basilar membrane to ossilate in a particular way. basilar membrane has a thick narrow base it has a thick narrow base, and a wide thick apex. and if we look at the peak of ossiliation that travels through the fluid, the ossilation peaks at the base for high frequency sounds and peaks at the apex for low frequency sounds. then of course somewhere in the middle for everything in the middle. this is the demonstration I did with my scarf when I was the base. high frequcny osilation hapeened at the peak of the scarf , low frequcney ossilation peak happened way at the end. this means that the basilar membrane exhibits tonotopic organization. it is organized in respect to frequency. lower frequencies at one end, higher frequencies in the other end. and everything else in between. so the frequencies are represented in order. high frequencies medium and low. this is important because we want to make sure that these properties of sound correspond to our perceptual experience are captured by the external auditory system and eventually transferred to the brain. so we want to make sure we are getting information about frequency, frequency responding to pitch. baslaar membrane is otnotopically organized. that amount of displacement of the basilar membrane, how much it goes up and down, how much it ossilates is going to orrespond to amplitude. again just another physical property of sound. physical stimulus ossilates or vibrates because of frequcney of a particular amplitude. that information (physical process) gets transferred into the chochlea into the ossicels causing the basal membrane to ossiliate at the same frequency amplitude, ect of the sound source. so up until this point we are still talking about physical ossilaiton, so physical vibrations how do we get them into neural signals. so now that we have this membrane that has this nice organization dependenong on the frequency and different location on the basilar membrane will ossiliate. and depending on the amplitude it will ossilate with a particular amplitude. so how do we go from their to neural signals? now we need to get to transduction that is going to happen by . sitting on top of basilar membrane is the organ of corti. this where the cells that do the transduction are located. basalr membrane ossilates in response to the traveling wave through the fluid. the basilar membrane is tonotopically organized. running along the length of the basilar membrane is the organ of corti. what happens when the baslar membrane ossilates in a particular location? sitting on top of organ of corti is the tectorial membrane. imbedded here within this structure are two different types of cells. these cells are called hair cells. because they have little cilia on top that look like little hair cells. there are two different types of hair cells. outer hair cells and inner hair cells. these cells have two different functions. what happens to the basalar membrane as it starts to ossilate in response to the travelling wave through the fluid in the cochlea. what happens? we have a physical property here. the baslar membrane keeps moving up and down (Physically moving up and down). outer hair cells are touching upon this tacturial membrane, which is just another flap hagging out in the fluid of the choclear duct. as the baslar memnrane moves up and down with that travel wave ossilation the outer hair cells move acroos the tactural membrane that cause the outer hair cells to have a particular response. what happens is outer hair cells move up and down in a electromotil response. so stimulation of these outer hair cells by ossilaiton of the basalar membrane pressing hair cells against the tacturial membrane stimulate the outer hair cells. just like if you jump with a tampoline you will go up higher. that is what is happening with the outter hair cells. they start to move up and down. essentially amplifying the signal at that point. they are causing the overall up and down motion in that location to be (9 min). you can think of the outer hair cells as amplifying just like if you were to jump on a trampoline when it was also in motion. so the function of the outer hair cells is to amplify the ossialiton that are happening in that location. so what with the inner hair cells? as that motion is happening it created movement in the fluid of the inner hair cells (this is all fluid filled area). the cilia on top of the inner hair cell, movement of the cilia causes the inner hair cell to activate to send a signal that will eventually leave the cochlea via the auditory nerve. that signal is a neural signal. so the inner hair cell responds to osilation in that location on the basilair membrane in response to that ossilation generates a neural signal, releases NT. that means that the inner hair cells are the sight of transduction, they are transducing the physical ossilaltion into a signal the brain can understand. outer hair cells amplify the signal and inner hair cells do transduction. SO we have frequency information based on where on the basalar membrane this is happening. where on the basalar membrane are inner hair cells sending signals. the inner hair cells depending on the location on the basalar membrane, if you encounter a 16,000 hertz tone the inner hair cells on the region of the baslair membrane will have a peak ossilation at 16,000 hertz. those inner hair ccells are going to be signalting that they have a vibration happening. if I hear a 4,000 hertz tone these inner hairs on that postion on the basalar membrane that is getting peak ossilation will be signaling that they detected an ossilation. those inner hair cells are going to send signals out of the ear via spiral ganglia cells, axons which bundle together to the auditory nerve. the auditory nerve is what leaves the peripheral audiotry system and goes into the brain. each of these auditory nerve fibers have characteristic frequcny. they will only signal in response to a tone a sound of a particular frequcny. that makes sense. if this auditory nerve fiber is signaling its sending neural signals what frewucny/ tone is it hearing. 16khz, 1000kz, all activated then all three. so this is how frewucny can be encoded. how does frequcny get organized as it leaved the peripheral auditory system. each audotoory nerve fiber has a characteristic frequcny corresponding to the inner hair cells where those fibers are getting signals. which kind of cells make up the RF of the auditory spital ganglia cell? inner hair cells! as we move out of the peripheral auditory system keep that in mind that the RF of all the cells we are going to be made up of the sensory cells in the sensory surface, always the sites of transudcutin in this case the inner hair cells. we see that each audiotry nerve fiber has a characteristic frequency. how do we know that amplitude or loudness ? experiment: take a variett nerve fibers and record their activity as we play tones of a particular frequcny as increasing amplitudes. play a tone at 500 hertz. and going to look how various auditory nerve fibers respond att 10 , 20 , 30, decibels ect. these are neighboring auditory nerve fibers. we know they are neighbors because we can look at their charcterisitic frequency. what they respond to best. what they need the lowest volume amplitude to detect. and we can see that they are next to each other, they are neighbirs because the basalar membrane is toneotpically organized. low frewucny tg, high frew tg, ect. what does this mean for how we encode amplitude? when we look at the activity of a single auditory nerve fiber that doesn’t tell us much about amplitude only abour freuncy. but if I look ath the activity of the neighbors that will give me a hint about amplitudes, if I have a quite tone at 1000 hertz and I don’t have any other nerve fibers responding. that tells me I have a 1000 hertz tone and it is pretty quite. if I crank up the volume to 70, c, b, and d responds. the neighboring autity nerve fibers also respond when it is played really loud. if we have a big hige osilaiton you might start to trick your ossilation. so when we get the amplitude of the tone by looking at the groups of fibers together. as the amplitude of the sound decreases more neighboring fibers will respond. so we get frequncey and the activation of auditory nerve fibers at a particular location. and we get amplitude by looking a the overall groups of fibers. perceptually as the vokume fgoes up (amplitude) it starts to be diffiuclt to discrimate the tone from other tones but can tell how loud this is. so we are leaving the peripheral auditory system, we got a physical stimulus which causes the tympanic membrane to ossialte , the ossialtion is transferreded into the chochela by the ossicles which amplify the sound. the rtraveling waves in the fluid of the cochle cause the basalar membrane to ossialte at particular points givent he frequency of the tones that are being played. that causes inner hair cells to send signals via the spial ganglion cells. freq and amplitude of the sound we are hearing. waves forms are represented on all the different locations on the basalar membrane. pure tone might activate one location on the baslar membrane, more complex tones might activate multiple areas on the basalar membrane. wave form is set of all audotry nerve fibers and their response when leaving the periphery auditory system. so relieiving the audiotry system via the nerve =. the axons of the spiral gangil cells are leaving the peripheral audotry system. now we are in the cental auditory system. here is the chohela. audotry nerve fibers leave th ecochlea heading toreard the brain stemateh central ausotry sytem lets us do two things. Central audotury sytem lets us do 2 things ∙ 1. localize sound o localization invoves figuring out where the sound falls on three dimensions. the distance from us, the elevation of sound (up and down), and location of sound on the horizontal plane and the azimuth. o how do we figure out the elevation, location of horizontal plane,and the distance? distance with respoec to the whole sound and enrgy that coms to us. but we know as sound sources move away from us the sound signal becomes weaker. but of course some signals close will be weak and some signals far will be loud. that is not the whple story but that is one way to think about distance. we really need both of these other cues to start putting this stuff together o first we can go back to our outer ear, the pinna, the pinna is very crital for elevation localization (figuring out where something is up and down). the way we know this is by doing a little experiment. they have created a false pinna which changes the shapes of all these little bumps. shapes of bumps and ridges gives us a lot of information about localization. changing the shape of the pinna can have a serious impact on our ability to do this. so they have people play tones in different elevations and they have us localized those tones. where did those tones come from. red wehere they play the tone (actual location of tone) ,blue where people say the tone is, so people are not perfect.what happens when we stick the modiefied pinna in the ear. everything sounds like everything is coming form one elevation. make this person wear this thing at modified elevation for twenty days. then you start toget used to it. they start to get used to this. this is crticial for the locatliation of elevation. o localize source of sound on the horizontal plane where is something located on the horizontal plan. two sets of cues that allow us to localize the surce of sound parraele to the horizon 1. interaural intensity/level differences differences in the ampitdue of the sound that reaches one or the other ear from a sound source lets say I have a sound that is coming from the right side. the sound travels through air, gets to my right ear, then it has to travel through bone flesh brain (all the solid stuff betwenn the air) to get to left ear. this casts an acuotic shadow (head casts acustic shaow) that makes intensity and level of sound on one side of your head lower than the other side of your head where the sound is coming from. muffles the sound (makes it a little bit quitter). head casts acouic shadow that you can use to figure out where something is in the horizontal plane. different frequcny of sound cast different shadows. less of difference in interlevel frequency tones. when sound is directly in front or behind no acoustic shadow cast so no interlevel difference 2. interaural time differences – differences in times it takes a sound to arrive . just like the two eyes giving us different views of the world . two ears give us different sources of the world. two ears that are far apart from each other , separated by a big fat head. a big batch a stuff between this ear and this ear. the difference in ears , the stuff in the middle of two years, allows us to localize things on the horizontal plane is a product of the fact that our ears are some distance apart. sound has to travel through space to get ot you. if it is coming right in front of you, the sound needs to travel the same distance to get to the both ear. get to two ears at the same time. if sound is to the right if it has to travel a certain distance to get to the right ear, and travel this much more to get to the left ear. subtle but significant time difference between arrival to ears. leave ear and head into central nervous system. cues will be processed into the brain steam. here is the cochlea. separate halves of the superior olive respond selectively ∙ 2. hear different frewucnies and amplitude and tranalting it into pitch and loudness so we can figure out where the firework is going on ∙ look at title 1 and 2. then bullet below. ∙ so how do we take the information from the peripheral auditory system, code it in those peripheral audotry fibers and localize sounds and intensify the sounds. ∙ now look at bullets under 1so the interarual time difference and level difference both provide very powerful consisten and localizing objects in the world. if you are trying to figureout where sound is coming form you can move your head to make the ITD or ILD greater and start to help you localize something. you might move your head to get more information about where that sound is by changing ITD or ILD. these are the two cues: ITD or ILD. how do we use those cues to localize sound? leave the ear and head to the central nerovuous system. and these cues are beginning to be processed here in the brain stem. so here is the cochlea. here is the authodirty nerve fibersleaving the cochlea . their first stop is the choclear nucleus in the brain . from there they proced to separate halves of respond selecltlicly to ITD or ILD. so here is superiou lovie complex. right and left chockus nucleus , right left ear, and righ tad left sruperiou olive. separate halves of respond selecltlicly to differently to the cues. the medial nueclus of the superior olive (MSO) are computing are computing different time differene, and later superior olive(LSO) is computing intelevel differ. (MSO) interlevle timing difference difference of the arrival sound of the two ears given that the ears are separated out in space. looking at the righ mso. sound torward the left chocle, sound reached the left ear first. traveling along a path to the righ mso. because it is the elft ear. the path is kind of long. it takes a long time to get from the left ear all the way to the right.the physical sound has gotten to the righ ear. the right ear send signal via the aufotry nerve fibers, the superios nucls, into right mso. the signals arrive at the same time causing the particular neuron to fire. so the signal from the longer path and the signal form the shorter path arrive at the same time causing the singal to fire. this means that this particular neuron responds when sound is to the left at 60 degrees. now if the sounds moves, if it is in a different location. move it further to the right. the interwaorld timong difference will be less. c will respond slightly to the left and closer to 20 degrees from you. so think of these as little coincidne detecotrs. sound is to the right (left mso) activity fromn right chokear and left chokeor nucleus. (don’t get the degrees). when the sound is to the right of you cells in left mso respond, and sound to the right of you left mso respond. particular cells respond to particular loctions in space. LSO repsonds to interlelve difference. important thing to know if the sound is straight ahead of you you have equal activity between the right and left LSO. if sound is to the right, right LSO will be more active. left LSO will be less active. the difference between the activation between the right and left LSO tells you about the location of the soud. so when the sound is on the right , right lso is more active than left. (talking about ILD) sound luder on the right than the left . louder sound is activatin lso more on the side that the sound is on. LSO neurons respond to ILD. we got freq info, amplitude info, ebconded in neurons in the chcolear nucleus. audtoy nerve fiber s with characteristic frequency sending their signal to chokor nuclus , chockear nucleus sends some signals to tbe superios olive (olivatory ocmplecP) which computes these lcoatlaiton cues that helps us localize size. chochlear nucleus also helps us send sound up through the midbrain by inferious colliculus. by medial geniculs into audotry cortex. similarly superior olive are also being send upper in aiudotiry cortex. we got info info abpu frwucny and amplitude by the chocke buckus and inf o about localiziton superiou oliver ocmplec. those singalas being sent up to audoty cortex. whe we look into audoty cortex, in it there is an area called aufotry core region where the signals end up. this core region has a aprtcular organization, along th e outsie it is calle d caudal belt. cells that get their signals from superious oliver cortect (hgettig localziont) . inside we have cells getting singles form chocleat nicles (so getting freq). caudal belt refion this whole belt area are active for locatioaion form single cell recording (take moneky, single cell reocridng, differ t caudal belt neruons ) we are looking at the activity of one caudal belt neuron , we can see this neuron responds perfeertianly to sa sound at 90 degress to the right of the monkey. if we do this for a bunch of caudal belt neurons we will see they are tuned to different locations. they are getting their singles from superior olgate complex , ITD and ILD, and putting that info tg to compete localization. cells inside audorty core rgion are attuned to freuwncy in particular they cell are tonotopically organized . basalar emmnran is also tonotopically organized (freuwcny orer, low to high high to low) . similar in the aurodrty cortex, first stop in cortec are tonotopically organized. they cells have a tonotopic frewucny, indiiudal cells will respond differently to difffernt frewuncies. not only that but they are very slecitnve. how do different cells repsind in the audotu cortex to freq and tones abd amplitude . aufoty nerve fibers these cells have charateris freq, if sound is liud enough it will respond ot all diff types of freq butit it is not true for audoty cortex. they are very specific tuned for freq. charxteristic frq . respond a little to tones are few. wont eposnd even if they get really loud. we start to see real selevitity. in some, sound wave in the world gets funneled through audotruy canel to tympanic memrnae which ossialted in response to the sound in a particular freq and particular amplitude. that osslaiton is transferred to the glue of chocle by the ossicle which amplify the sound, traveling wave trabels through the fluid , causing outer hair cells to give elctomotal response ampluyfiyng the osslaiton in cells that in that location signal tone of a partical frequency and amplitude. that info gets sent out of the peripheral nerocis system via audoty nerve fibers to other side of choclea bucli. choclea bucli. send singlas to supeiro oliver cortex which cues ITD ITL. choclea ncleaus also sinde info up. all that eventually get intergrated up to the temporal region to figure out which tone from which friwncy is coming form where in loation is space. next task: take that info and put it into audoty objects. sound wave comes is, pinna funnels it so it foes through the auf