Consciousness Live! with Rachel Denison (S6 E2)

0:00 begin with right because if you have no constraints on your ability to increase the gain in these populations why not just increase the gain at the beginning of the trial leave the gain on really high you know the whole time and then you you would be able to have maximal performance on both targets and that you would have no incentive there be no incentive in the model to re you know increase the gain more at one time

0:36 than in another time so the real question is what the constraint is that makes it necessary to attend more at one time than at another time um and in the in this first version of the model that we've published the the way that we implemented this constraint was by was by depositing that well maybe there's some limitation in our in in the in the gain that's available to be allocated across

1:09 these short time intervals so maybe you have a sort of you know limited resource of gain that you can allocate across time and if you allocate it all at one moment in time at the ne next moment in time you don't get it back again immediately it takes some time for that gain to recover again to its kind of Baseline levels and that's how in you know in this version of the model we could um correctly predict the attentional trade-offs that we observe

1:46 empirically okay so I'm so sorry but unfortunately while you were just talking I slowly came to realize that uh the stream was had been dropped and I we were not on the we were not being broadcasted so but I only just now figured it out um because of how much skill I have at YouTube broadcasting oh my gosh okay so I apologize for that um yeah anyway so I'm here live now I think

2:21 we actually are live see I have it's very bad okay so sorry so um we're going to have to take it from the top I think okay holy moly I can't believe that actually happened I'm very sorry we're live now I'm here with Rachel Dennis Rachel I'm so sorry but can you reintroduce yourself for those who may not know who you are yeah of course um so now your audio is do do I sound a bit broken up to you audio

2:53 wise oh you sound okay to me okay do I sound okay you sound a little bit robot to me but it may just be because I it says we have an excellent connection everything's fine so I think it's temporary sorry about that Rachel go ahead and okay no problem so uh my name is Rachel Dennison and I am an assistant professor of psychological and brain Sciences at Boston University uh my lab studies visual perception

3:26 attention decision- making and awareness uh using tools from cognitive and computational Neuroscience and vision science okay and for those are who are just joining us we've been talking for half hour already where I thought we were live and unfortunately there was one extra button to click so it did not go through um but we were discussing Rachel's work on attention and we were talking about temporal attention so unfortunately that discussion maybe I don't want to make us repeat it but if I

3:58 can just sort of some suiz what I was taking away from from what you were saying and then maybe you could add to it if I'm missing anything was that when we're talking about a temporal attention we're talking about attention to a particular point in time when a task may be coming up when something is relevant um like crossing a street uh Etc um when you really need to pay attention at a particular moment and what what Rachel's work is uncovered in her lab is that uh when you attend to a specific moment in a short window you become better able to

4:33 perform that task but worse off right after and just before on a very short time scale uh and then we were talking about a model a mathematical model which would dis which would describe that um in computational terms and and I was asking for the specific explanation for how that was actually going to work and that's where we just started actually live broadcasting um and basically what I was taking from what you were saying Rachel was that the the way that this works is by actually adjusting the gain

5:07 of these things sort of on the fly live as it's happening so that there's something that's monitoring the activity of those neurons and then adjusting how sensitive they are to the external world as it deems necessary is that too much of an oversimplification or did I miss anything important in what we were just talking about I think that's exactly right so you know the the very simplest hypothesis is that um visual attention in terms of when we're attending to specific visual

5:42 information um is implemented via some top- down gain modulation which just means there's some modulatory signal uh onto the visual cortex that's changing the sensitivity of the visual neurons and when we think about temporally attention which is you know being able to prioritize some visual information at a specific moment in time that's very relevant to your task um our working hypothesis is that the gain of of neurons is actually dynamically

6:15 adjusted so that at the moment when you um when you have to attend and you really have to see that information very well uh the gain increases and at other moments uh before and after the gain is the gain is reduced yeah good okay so and I was going to ask you then about the relationship that you think uh between the mathematical model um and the set of equations that you developed and the

6:48 neural implementation of it because I know you guys say in your paper that there's couple of different ways that this could go actually um so you're not I don't know how committed you are to any particular neural implementation or if you have a favorite or something like that but I I am wondering about how you think that the the brain might actually be doing this sort of thing yeah I mean this is a question that we're very interested in and the nice thing about having a a Model A computational model that's implemented at the level of neural populations uh is that it actually makes

7:21 predictions for what the Dynamics of visual cortical life ity would would be if this model were correct um and so you know we can actually uh just look at the sensory layers of this mathematical model which is kind of formulated as a recurrent n network we can look at the sensory layers we can see what the Dynamics of those layers are we can look at the voluntary attention layer of the model

7:56 we can see what those modulatory Dynamics are um and that's a prediction for what we might measure in a neuroscience experiment where we measure activity from visual cortex and where we measure the degree of uh modulation of that activity in a top- down fashion and so what we've actually been doing um in my lab for the last couple of years and this was work that I started when I was a Posta with Marissa Cresco and David heager NYU and we continued doing it

8:29 with um students in my lab is we're doing these Neuroscience experiments so we're you know measure brain activity and use different techniques to track the modulatory influences on the visual cortex as the um you know as the trial is unfolding and people are seeing these sequences of to stimuli usually is what we do to be very you know to be very simp Le um and we ask them to attend to the first stimulus we ask them to attend

9:03 to the second stimulus or we sometimes ask them to attend to both stimulus and then we can measure what's happening in the visual cortex and we can measure these modulatory influences and we can ask you know does it match what the model predicted um and if so you know that's a point for the model and if not that's our opportunity to revise the model so more generally I am curious about your view about how models relate to the wet wear in the brain so so do you think that like by finding this kind of evidence that you're finding that you're

9:37 well increasing your own Credence in thinking that this is the actual computation the brain uses or is this just a so I guess what I'm having in mind here is the the difference between you know taking a brain's point of view and experimenters point of view so is this something that you think of as useful for describing the data we have or do you think this is actually what the neurons are doing the kind of computation they may be implementing so my my goal is to develop models that really are the computations that the brains are that the brain is implementing um at least with this kind of model you know that's the goal we

10:11 have the model is specified at level of neural populations exactly in order to be able to make these links between the Model Behavior and the me the neural measurements um and and and of course the psychophysical measurements because the model can also output um behavioral decisions um and so that that's the general strategy that my lab is taking for this problem of temporal attention and for you know mo most of the problems that we're working on is you know let let's have this interaction between models uh

10:46 computational models which are mathematically defined but that models that can make predictions about neural activity and models that can make predictions about behavior and then let's measure the neural activity and let's measure the behavior and let's update our models and let's iterate so that that's really the general strategy of of the of of our of our research yeah and that that's a perfect implementation of the strategy of uh cognitive Neuroscience um as approach to these topics so it's interesting by the way

11:18 one thing I forgot to mention that we were talking during the time when we weren't live but thought we were was how there's a lot of interesting work to be done on temporal attention and how I'm wondering how many of the things that we've discovered about spatial attention will kind of carry over to temporal attention um so it will be interesting to tune in to see I know um some of these questions are really empirically tractable it seems um absolutely yeah um so I want to ask about some related work but it's slightly different so this had to do with your uh work on the M&P

11:52 pathway uh the Magna and parvo uh Pathways and the different contributions they might make I mean I think it's sort of indirectly Rel Rel ated because ultimately if you're going to have a successful like realistic model uh of attention then it's going to have to include these different contributions if if there are and I think your work suggests that there is um so so first I want to ask you about that and then how you think these May relate but as far as I understand your work there you guys are really showing that there's a difference in contribution which reconciles the kind of debate um that people have been having in in the uh you

12:26 know what is that I know I'm blanking on the term but you know the you'll tell us what it is shortly I'm sure the perceptual blinking between the face in the house what is that called again oh are you are you referring to the binocular rivalry that's the one yeah yeah yeah um okay yeah so um so this is work I think the work that you're referring to is the work some work that I did when I was a PhD student with um with Michael silver UC

13:02 Berkeley and and you know I I it actually fits really well with the uh what I think is a a general theme that I keep returning to in my research and that I um you know and that I'm really excited about for my for my lab's direction which is really um thinking about the Dynamics of visual perception and thinking about the Dynamics of neural activity um and

13:36 trying to understand how they relate to each other and how we how we construct a visual experience that is actually structured in time um is actually kind of seamlessly structured in time and folds in this you know cinematic way um given the fact that the

14:09 brain takes time to process information that information has to travel from you know one point to the other and across many many different areas of the brain to analyze different aspects of the visual scene and then that information has to be all stitched together um and you know all these different steps take different amounts of time and occur at different latencies and this problem gets even more striking

14:43 once you start thinking about a multisensory context when you know you know auditory information is processed you know at some latency and visual information is processed at some other latency and there's all these different steps through the hierarchy um any yet you know our our actual perceptual experience is beautifully temp temporally structured and tends to match the temporal structure of the world and so um the the studies that we were doing uh that that I did with um with Michael

15:16 silver at Berkeley on the M&P uh uh roles of M&P systems in binocular rivalry we're looking at this interesting um this interesting phenomenon in binocular rivalry which is called uh some people call it interocular switch rivalry way it works is you um you have these uh so so the classic setup of binocular rivalry is you present you know one image to the left eye one image to the right eye and you know usually when there are different images you know we always have slightly

15:48 different images presented to the two eyes and that's what gives us our uh depth perception um but in binocular rivalry those images are very different and and they're incompatible they can't be integrated in the normal binocular vision systems and so what happens instead which is you know very cool is that you uh you see first one image like the image in the left eye for a little while and then your percept changes and you start seeing the image in the right eye and then it changes again you start

16:21 seeing the image in the left eye um and this happens kind of outside of your voluntary control for the most part yeah um that's what I was talking about the face in the house that's the classical demonstration of that was a face in one eye and a house or a place in the other and you get that popping back and forth right exactly right so so that's a that the face in the house is a really good way to to look at neural Corel it's a binocular rivalry because you can you can measure activity in FFA for the face and you can measure activity in PPA for the house you can see how that how that changes um but yeah what what happens uh

16:56 when you when you swap these images back and forth between the two eyes you can start asking is the percept that you're experiencing linked to what's in one eye or is it really um at a higher level linked to a particular um perceptual interpretation um and the reason that that is a kind of elegant experimental strategy to to look at that question is that if it's just linked to one eye every time you swap the images back and forth between between the two eyes you

17:29 should have a perceptual swap MH um but if it's linked to a higher level perceptual inference swapping the images back and forth might not make any difference and you might be able to have this sustained percept of just one of the um one of the images for longer you know across multiple swaps and um and and and indeed under certain uh stimulus conditions it is is the case that you get these longer sustained percepts um and under other conditions it seems that

18:05 you get these more abrupt changes that are linked to the eye and so you know a question is well what what determines when you have this more kind of eye based peret versus this more stimulus based peret right so to and to address the particular contributions of the pathway you use a different stimulus so you're using these gradings that are tilted in one way or or another but being presented to each eye so that you either get a train a switching of the gradings that coincide with the stimulus or you get this I forget what the

18:38 terminology that Ed but this random this greater this larger pattern um and what you're what you guys were looking at in that I didn't realize how old the paper or if it was older then you're more I didn't realize it wasn't curage so sorry but it was very interesting when I read it what you guys basically showed was that you can see that there's two different things going on here one related to the expectation about uh to how reliable the switch is or what's going to be there and the other related to um maybe attention or to something else so um yeah I mean in in that work

19:14 we were we were very interested in the uh you know it had been found before that a variety of really seemingly low-level properties of the um of the visual stimulus could have dramatic influences on whether you saw this kind of ey based percept or this stimulus based percept um and we wanted to understand the reason the reason for that and we kind of noticed that some of these um some of these differences seem to map on some that had been observed before seem to map on to these two um

19:50 early parallel Pathways in the visual system which uh which are the the manoc cellular or M pathway and the pver cellular or P pathway and these Pathways um start very very early in visual processing um they're completely separated at the at the level of the lateral geniculate nucleus which is the camic nucleus that sends visual information to the cortex um and these pathways are very interesting when we're thinking about Dynamic Vision because

20:23 they have very different um spatial temporal and and uh color uh sensitivities uh so the the M pathway has these kind of very transient responses to to um visual changes and it responds very fast um and it tends to respond to big uh kind of low spatial frequency you know coar scale uh fast changes um whereas the ke pathway um is

21:00 much more sustained it tends to respond to you know detailed color vision that's more stable and it's and its responses are more sustained as well um and you know to make a long story short what we found was that um when we actually designed experiments to manipulate the stimuli so that they would be more targeted to the m pathway or to the P pathway we could um change the uh rate of the you know I based uh percept versus the simulus based percept

21:37 um and uh in a way that was kind of consistent with activity in the M pathway perhaps signaling this transient change that was associated with the ey swap which would then let you actually perceive this transient uh eye swapping pattern um um and if we reduce the activity in the M pathway maybe kind of reducing the sensitivity of the system to these transient changes um people were more likely to see these more

22:11 sustained percepts so almost like you know so even though um I've been using the terms I based and stimulus based um and those are terms that other people have have used in literature um it you know these results suggest that it it might not be uh really competition at a monocular level versus competition at a higher level but you know an alternative way of thinking about it is there's these different um Dynamic patterns of input and when you're doing perceptual

22:45 inference uh maybe if you have more of this Transit Information you're more likely to see this rapidly switching pattern um you know you most like be signal that like oh something has has just changed my inference needs to reflect that data yeah I mean I found that one of the reasons that I found this was interesting just because I you don't see even though there is this well-known anatomical difference in in the visual system that everybody talks about in every introductory class it's not studied at the at this level as much

23:20 as you might think um so it's interesting to see you guys doing that and then that you uncover this interesting result that make different contributions perhaps um so so that it it does make me wonder how if you think that these are the sorts of things we would need to account for in a successful model which is what originally I was I was going for when I brought this up and now I just realized I was mixing up some other pay-per-views that I was talking about when I was talking about the expectation and attention um so I want to ask about that after that but I got flustered when I realized I made such a egregious error

23:54 earlier so I was rattled but anyway so I'm sorry about that as as well but uh so I wonder if if you think that well how much does this affect generally speaking our our models of of vision that we kind of neglect this or or not neglect it but it's not formally accounted for in many of our models the difference between right I mean I mean um you know the um the the the area that I think there's the most

24:28 you know the most room for some interesting interesting advancement of our of our vision models is indeed in the in the realm of Dynamics um is indeed in how we are able to process this Contin continuous visual input and construct our you know perceptual experience as it unfolds over time um and I think that uh our foundational knowledge about the anatomy

25:04 of the visual system and the low lowlevel um processing in early parts of uh you know the visual stream uh will probably be quite important for developing detailed models of that kind uh because what's happening you know what's happening in the r what's happening in the lgn is really determining the kind of input that the visual cortex gets and the kind of input that um you know then undergos all those hierarchical

25:41 stages of processing to sort of form some perceptual some perceptual inference um but you know throughout the visual system there's a lot of information about Dynamics that's encoded and that is something that we need we need a we really need to work on a more complete picture of how that Dynamic information gets translated into our Dynamic perceptual experience yeah I think that's really an important point and and one that a lot of people um will

26:15 kind of pay lip service to I guess that you know the the brain is a dynamical system unfolding in time but then a lot of our models like you were saying we were talking about earlier they're static um and even the way we discussed a visual system it's in term of the static hierarchies and we're like yeah there's a lot going on there but here's V1 then V2 then you're like uh so it I think it's your work in part the overall approach that you're taking and that we've been talking about is this to bring in the Dynamics part of it and to take seriously the idea that experience is something that unfolds in time and that therefore takes time um and

26:49 understanding the processes that lead up to the kind of experience we have uh is is extremely important and hard to do actually but the the the studies that you do are very interesting and clever and that's what it takes is to get at these kind of questions is really interesting and clever experiments um and a lot of the work like it just one simple thing can make a big difference like what you were saying earlier is that you guys in your work you present two stimuli and then you can cue the first stimuli or the second stimul you can play around with that that Paradigm a lot um and it's a simple

27:25 seemingly simple Innovation that um that leads to a lot of experimentally Rich ideas so I I really think that's interesting um I was gonna ask about sorry go ahead oh I I was just gonna say to that point that um we we really do try try to be as simple as possible and I think this is a I think this is a more General Point uh to make about when we're trying to develop especially when we're trying to develop quantitative models of um visual process processing and perception and

28:00 attention uh you know these are these are complicated um you know attention is a complicated cognitive process um you know perception is its own thing uh we we're trying to build models that we as humans can understand I mean that's part of my S philosophy about models is that they're they're for they're for us you know they're for humans uh to understand how these things work um and so I think we you know for that reason we re we try to simplify our our experimental

28:36 designs as much as we possibly can um while still retaining the manipulation that will let us answer our scientific question you know and and that's um that's important for this strategy of then relating it back to the model and helping it helping us to advance the model the experimental design is too complicated um or the simul or too complicated and so on it's harder it's harder for us to to make models that we can interpret and so um that's just kind of a meta you know metascience Point yeah which is um

29:10 you know when we're trying to make this these links between models and behavioral and neural activity uh we often find that it really helps to simplifi you know simplify as much as we as much as we can you know not too much that we lose the The Cure off the baby with the bath water yeah exactly the key simplif cuz you're right sometimes you get the feeling that there's too much trying to be I mean too many Innovations to that spoil the design not you read some of these things like okay this is too complicated to address the question could be done

29:43 simpler so you want to make sure you don't leave it out but not over complicated and that's hard to do as a researcher uh there's a lot of pressure to be clever yeah it's a fine line you know and I mean but modeling is a really humbling Enterprise you when you Tred to build these models that are going to actually explain explain the data and be understandable to you um it's it's you know it really forces you really forces you to think really hard about you know what ex what exactly the experimental design is and and you know what what exactly all the ingredients are so when you say understandable to you are you

30:19 meaning to indicate not deep neural networks or something like that or uh like the opposite of that approach something that you could see how it works is that what you mean by that or do you mean something yeah that's what I that's what I mean um um and you know I certainly think that there's a you know there's a there's a lot of fascinating work being done in deep neural networks and it's an important neural development that's uh for sure but um but yeah I think um you know different

30:52 kinds of models have different purposes in science and um the kind of models that we were just talking about these normalization models of attention the um normalization models of dynamic attention that we've been developing um the purpose of these models as far as I'm concerned is for us to understand the um the mechanism and for us to be able to make predictions about uh the brain and about behavior and if um you know it's not to say that when we build these models we already

31:27 know in our minds how they're going to behave because sometimes we're surprised um but it should be we we do find it helpful to build models that we can all we can ultimately even when they surprise us sometimes we can we can have a chance of figuring out oh this this is why you know this is why it had that kind of behavior in the situation right supp to just adjusting some parameter and then hoping it works and then going right interesting okay okay so um uh I I wanted to ask about the the the stuff

32:02 that I was confusing when I was flustered a second ago which was the relation between expectation and attention um and how you can separate those out uh because what I'm trying to ultimately get it to get us to talk about eventually is the idea of probabalistic perception and whether or not the perceptual systems use uncertainty and whether that entails that there is a probabilistic aspect to perception or not um so I I think this is a way to get into that topic uh but so I I wonder if first you could tell us

32:35 a little bit about the work that you did um where you try to separate out this difference between the the contributions of the expectation versus you're attending to the topic and then we'll kind of go from there yeah um I think this is a very important distinction um between expectation and attention um this is this is a distinction that has been made um in the literature you know first in the kind of spatial and feature-based attention expectation domains for example by Summerfield and eggner um and the the

33:11 way that I would Define the difference between expectation and attention is that expectation relates to the probability of something happening the probability of a particular thing happening um so you know if you have an expectation about you know um uh I I predict that the next thing that appears is going to be you know a lion or something like that right um uh that would be a terrifying prediction but you

33:44 know you might be able to make that prediction um uh attent attention uh is is uh what is actually what uh information would actually be relevant to your task so you know um in the example of a lion a lion is obviously quite relevant to your task um but you know you might be able to make a different kind of prediction like oh I predict that there's going to be a um toad you know um and I can predict with you know great deal of

34:21 certainty that there's going to be a toad coming coming around the but it doesn't matter I don't care you know let the toad be the toad um it's not particularly relevant for my task so there's no reason for me to actually attend to that to that toad when it appears um so you so these things are orthogonal in that way you can um have you can you can predict something that will occur and that thing can be either relevant or task relevant and and meanwhile there can be something quite task relevant for you but you cannot predict uh you know when it when it will

34:54 happen and so on um and so we've we've uh been doing work making this distinction um this distinction has come up in a few different places in you know in my previous research I'm not sure specifically which which one you're referring to but um recently uh this distinction has been really important to us when studying temporal attention um and temporal expectation so in the time domain um the way this expectation attention distinction gets cashed out is is uh temporal attention

35:30 involves a task relevant moment in time so um you know there's a a time that you know there's going to be something that happens there that's very important for your task whereas temporal expectation is um there's a time that you know something may happen something is likely to happen but it may not be relevant for your task so like an example I like to give and the temporal domain is let's say you're watching someone go up an go up an

36:01 escalator um you can watch someone go up an escalator and you can know when they're going to reach the top and so you can predict the moment when they're going to reach the top but it doesn't matter for you you know you're just a bander like that is a randomy on an escalator it doesn't matter um but if you're the one on the escalator Like You can predict the moment you're going to reach the top so that would be temporal expectation um but that moment is also test relevant for you because you're going to have to you know look at that moving edge of the escalator and prepare to step off the escalator and not get

36:33 your shoelaces CAU in the escalator and all that stuff right so so uh that would be an example of temporal attention which this task relevant moment and so in a lot of the work that we've been doing recently with um temporal attention it's been very important for us to make that distinction between expectation linked to predictability and attention linked to task relevance um because a lot of um previous Studies have um used um tasks where those two things are kind of U merge together so you might have a

37:08 stimulus occurring uh at some more or less predictable time um but the stimulus is always task relevant so you have both prediction and attention occurring together um and so what what it means is we don't really know whether the mechanisms of that process you know those processes that have been observed before are really related to the predictability of the stimulus like the timing predictability you know your ability to predict this interval over over which things are happening or are really related to the relevance of the

37:42 moment in time that you know you're trying to use for your behavior and so the way that we have dissociated this in our uh in our experiments is by presenting these kind of two sequential targets where both targets are completely predictable they always occur at the same fixed time relative to an in initial cue um so you so in every trial you know exactly when those uh stimula are going to occur they're completely predictable um but we tell you oh the first one's going to be more relevant or

38:14 the second one's going to be more relevant and that's our way of sort of controlling temporal expectation while manipulating uh temporal attention interesting so and so in the uh studies that I was reading this is the the most recent one I think was you present the two stim uh stimul but then you can present a a cue that's either accurate or inaccurate so you get the predictability measure uh because the stimul they can expect it there um but since the queue is kind of going to be reliable and you could adjust either how

38:47 much or not then you could really check to see whether they you give them some incentive to attend then you expect that there'll be you know attending there um so then you could check when they were not preparing to attend but still expecting stay there so that's where you get that crucial distinction exactly and so what do what do we uncover when we do that right so um so the fir the first thing that we've found is that and I think we we we alluded to this a little bit at the beginning but um I I'll just say it once again is that um when people

39:23 are attending to specific Moment In Time their perception improves at that time um but it also is impaired at you know preceding and following times and so what this means is that attention actually changes visual processing over and above uh predictability over and above specific predictability you can you know Zone you know hone in on that task relevant moment and it will change your perception um we've Al we've also found that the behavior of uh uh your microad so these little tiny eye

39:59 movements that you're always making even when you're not aware of it and even when you're fixating at a a single location those are suppressed those are suppressed and it they're suppressed before these predictable times which had been observed before but also the exact timing of the microcytic suppression um depends on the precise moment when you're attending so again it's this another example of how um temporal attention is you know sort of building over and above mere

40:32 predictability to adjust the precise timing of these processes um and now you know we have some new neural data that also shows uh changes in visual cortical activity that are specifically linked to temporal attention you know again over and above this temporal predictability or temporal expectation um so that's um uh you know so so that's what we're really working on now is trying to isolate these mechanisms of voluntary or gold directed temporal attention when

41:06 you know while we're controlling for predictability interesting so when you found this trade-off thing it was still at the same time scale of 300 milliseconds or so still at that very short time scale oh right right so the the tradeoff uh the trade-off between these two simly it it happens yeah the peak is around 250 or 300 milliseconds and then by the time the two stimul are if you place them you know even a second apart um the trade-off disappears and it seems that you can actually you know maximally

41:39 process both stimul if they're far enough apart and you know the way I like to think about it is you know this has to be true in some way like you can attend today and you can attend tomorrow and that's fine um and the question is then really you know how close to do these stimul have to be to each other when they start to interfere right um and on the time scale of perception a few hundred milliseconds is what I would consider an intermediate time scale you know it's close but it's not that close um it's you know it's not something that's as simple as uh you know masking

42:16 or that might happen in a very very early stage of visual processing it's not something that's mediated by you know very short time scale interactions that might happen um ear you know early in the visual system this has to be something that involves um uh temporal constraints that happen on a somewhat longer time scale um and I'm very interest you know this is something that we're really interested in in in our in the lab right now is thinking about this time scale you know this time scale of a few hundred milliseconds which is um

42:52 really I would say the time scale of a lot of our uh perception and cognition and behavior you know when we think about the rate of ptic eye movements you know we're making catic eye movements every you know 300 milliseconds or so 300 500 milliseconds um when we think about the the rate of speech you know syllables it's all it's also on that order you know a couple hundred 100 milliseconds you know per little chunk and and so

43:26 this is um I think of this hundreds of milliseconds time scale as like really core time scale to the way that our brain works and the way that our kind of um cognitive processing and perception unfold um and it's a question it's a question is why that time scale right you know imagine an organism that went 10 times faster um you know or 10 times slower you know um it's seems uh that you know those those would

44:00 all be sort of theoretically possible um so there must be something about the way that our our brains are built um that makes it so that we kind of tick along at roughly this time scale to sort of execute these like chunks of processing yeah I I find that stuff very interesting I I've been interested in questions like this since uh grad school for me when I was reading a work by Rudolph olos and sort of the idea of a perceptual Quant and all that stuff is very interesting um do you have any guesses as to the answer I mean my

44:34 preferred guess is has something to do with the waveforms um which are important in brain processing like Thea waves and so forth uh but uh do you have any of your favorite guesses well you know the I think there are there are a number of possibilities um for the kind kind of things that could lead to temporal constraints on on this time scale um you know one one you just mentioned is um these um

45:08 rhythmic features of neural data like Theta or Delta consolation um you know again it's sort of like is that begging the question like you know why why why you know why Theta and why not alpha or you know infra slow you frequencies like that um but um then we talk about tuning of the neurons right well I mean I think there's like something about Theta that helps the neurons sort of in tune to it or some because of the way that they're

45:41 built so it could just come down to those kinds of principles yeah there could be some sort of biomechanical reasons um that certain time scales work like biophysical reasons uh related to um single neurons or circuits um and so forth um so that's very interesting um you know there could be reasons re related to something about um communication across the cortical networks or cortical thalamic networks

46:16 um in terms of like how long does it take information to go here and there and uh you know to um on the one hand kind of converge on coherent interpretations or to you know do some readout processes that are necessary when you're doing actual tasks when you're you know making decisions and um guiding behavior and so on um and then you know there's other things about uh competitive competitive processes in the in the brain which you

46:50 know we often think about as happening between um neurons uh that are simultaneously active like you know Mutual inhibition or something like that but also given you know given the amount of time that it takes for information to kind of flow through the brain um you know there there are many nms that have fairly longtime scales of activity and so you know this raises all kinds of questions about what kind of interference across time

47:22 happens uh and um in the because of some of these temporal properties of these you know these systems these circuits and so on um yeah so I think there are a lot of um options I don't think you know they're mutually exclusive even um and that and part of our you know part part of why we're we're actually really focused on temporal attention right now in my lab is because this seems like a seems like an experimentally a really neat way of getting at these temporal constraints

47:55 you know we see the temporal constraints in behavior when you tend to one point in time you better at that better at that time worse at surrounding times okay nailed it that's the temporal constraint now you know we can also manipulate it you know we can manipulate the temporal constraint by let letting uh someone be cued to one point in time or another point in time uh this causes the trade-offs you know this changes the prioritization of one time over another time good uh so this is a you know this is an experimental strategy where we now

48:27 have experimental control over how the brain is handling these temporal constraints and so the hope is that now by you know looking at the neural activity and kind of tracking the information about these different stimul as they flow through the the brain we can start to understand um you know what what are the compensatory strategies that the brain is using to to Prior I ize one stimulus in the presence of these temporal constraints um which hopefully will also

49:02 give us insight into what those temporal constraints actually are yeah do do you think this time scale has effect at the like everyday level so I'm I'm wondering if you misjudge the timing of something you think it's going to happen at like a fraction of a second earlier than you may be attending too early and that's going to make you do worse a fraction of a second later when it actually shows up um you know like a crossing the street and you misjudge how far the car is away or something like that or and you're reaching for a ball trying to catch a ball that someone's throwing in you misjudge the angle or I don't know these

49:34 sorts of things or is that too short of a time scale to have these kind of practical impacts do you think the 300 milliseconds you know I I I think I think that it's possible um and I think that uh the the the place where Mo we're most likely to see really significant effects of uh this time scale are are cases where people are being you know put actually push their to their performance limits you know we think about these high high high demand high performance activities that we you know that we value a lot like um uh people who are you know top level athletes or

50:09 musicians or um uh fighter pilots or you know but even in simple even a more simple situation of driving a car like usually um the the way that our roads are constructed and that the spacing of the signs is you know put out on the highway it it it respects it respects the temporal constraints that we have and tries not to overload them um you know but in certain situations you may get into a demanding setting where you have a lot of simile at coming coming at

50:42 you um and your uh the fact that you could um not process all of that information uh could could really lead to some trouble you know like if some someone pull pulls out in front of you unexpectedly and there's something you know something else you're like that you're were already paying attention to I mean this is the kind of situation where your those temporal constraints become quite apparent and um you know can really matter is this the sort of thing that if we were to build an

51:14 artificial agent is this the type of constraint we would want to build into that agent or is it the type of thing we'd want to engineer out as much as possible that's a good question and and I I think it's hard to it's hard to say because we don't I think we don't really know why the temporal constraint exists now um you know um and I think the answer to that question probably depends on the reason you know if the temporal if the if the exact magnitude of the temporal constraint that we have in our

51:47 biological brains has some um has some good reason from an engineering perspective for kind of like processing and making inferences about the world like that's that's one thing um if it comes from you know the way that it it has to be implemented in the brain using biology using neurons using circuits you know then maybe artificial systems um you know would not be subject to the same temporal

52:19 constraints interesting um so I want to ask I know we're running out of time here uh but I want to ask about uh phenomenology or experience more generally which is what motivates a lot of your work um and what understanding how the you know perceptual experience is constructed in time um the how the process actually unfolds is a central concern but we haven't really talked too much about it as of yet a lot of the stuff has been the Forefront of been the mechanics of the mechanisms computations the neurons and stuff so I am wondering

52:52 about your um maybe your background like what you think about perception generally um and then we'll sort of dig in on how it relates to these models and things we've been talking about so what is your view of perception a open-ended question for you Rachel open-ended question yeah um the way I think about the way I think about perception is very much in the tradition of um you know the kind of Helm holian tradition of uh perceptual

53:26 inference so um you know which has which these days we might we might think about Innovation framework um so the idea that uh we're getting data on our sensors on our on our retinas on you know the sensors and other um sensory modalities and the the brain has to interpret that data to make an inference about

53:59 kind of what was it out there in the world that caused that pattern of light to fall on our retina for example um so you know this inference process where you're getting sensory input and you're constructing um an interpretation of what was what would have been what would have had to be in the world to cause that visual information so on this way of talking about perception it's okay to say there's unconscious perception yeah I mean I think you know

54:32 um perception uh is a a term that people use in different ways just like most of our terms um and and yeah I think uh uh you could you could use the term unconscious perception to refer to um you know the the visual processing that kind of um going on under the hood that that leads ultimately to these perceptual inferences is there anything that you would call a p a perception that is uh

55:07 of the same kind as like the thing we experience but not conscious or do not allow that I'm just curious like how you use the terms I'm not really don't have anything that I'm aiming for I'm just trying to figure out how you use the terms um I I'm not sure if I'm too dogmatic about the use of perception per se I think it's just I think it's usually more useful to just say conscious perception or um you know I think usually when people use the term unconscious

55:40 perception they're talking about some you know maybe stages of the perceptual inference that are happening um uh which you know those stages May uh may not be may not be count but the way the way I think about it is you know there's some visual processing that happens um and it's doing some perceptual inference it you know it it may involve a lot of sophisticated computations um and then it kind of

56:14 produces this result which is like the inference um and that inference is what we consciously experience okay interesting um so yeah because I I think some people use the term unconscious perception and what they mean is something of the same psychological kind of the thing which is conscious but just not so happen to be conscious so um as opposed to some lower level processing but I agree with you that these terms are thrown all over the place which is why I wanted to kind of to get clear about this so when you say um because now I'm thinking about this

56:49 paper that you co-authored with uh a few people but one of them was um block Ned block and Doby and one other person whose name is escaping me uh but it was on probable yo you ha yes sorry about that yanica um so uh this block on probabilistic perception and so when you are arguing for that this is the same definition of perception you're using there so it's supposed to be conscious perception the output of the inference uh what the thing which we experience which is probabilistic in some

57:21 sense or could so so so my yeah so my my view there is that um the the computations going into perceptual inference uh may be probabilistic um and you know we could talk about there are different ways of being probabilistic and what does it mean and how probabilistic and and that's where I think you know probably most of the scientific meat is is sort of answering these more graded

57:54 questions um but uh I I wouldn't I my personal view is that the kind of output of those computations you know the computations may be probalistic but um uh the output into our conscious experience sort of the character of our conscious experience um I don't see that as probabilistic okay um and the reason and and and the way the way I would explain that view is

58:30 that um to me probabilistic implies many simultaneous possible interpretations um many Alternatives that are simultaneously experienced um and I think uh conscious perception actually really kind of notably doesn't have that property right um and that goes back to some of the work we were talking earlier about binocular rivalry which is one of these classic examples where it's not that you have an experience of both options at

59:05 the same time but you know one feels more probable and the other feels less probable you you really just have a single uh percept a single perceptual interpretation and the other option is completely unavailable to your conscious experience um even though it you know is clearly represented somewhere in the visual system and you know could gain steam again and later yeah that was confusing me because I I wasn't sure uh so so when when you say that perception could be

59:39 probabilistic what you mean is that the processes that lead up to what we call the conscious perception might make use of or might somehow embody the axioms of formal probability theory in the way that they work yeah yeah exactly so um I I I think there are two I think there are two ways in which I think perception uh you know or perceptual processing you know could be probabilistic um one is in the course of perceptual inference so you know classic example um would be I'll give you two

1:00:15 examples one is um the uh you know the famous Checker Shadow illusion where you have this cheer board and there's an object sitting on the checkerboard it casts a shadow over the checkerboard and um you see the the square uh the squares that are covered by the Shadow the perceptual interpretation is that the color of those squares is like what the color of the paint on the checkerboard would be um even though the actual um luminance

1:00:51 of that part of the checker board is is quite is quite different um you don't see the you don't consciously experience the luminance you consciously experience the pain color um but experiencing you know having arriving that perceptual inference seems to involve some uh you know it could be explained by a basian inference process where you have you know multiple causes you have the cause of the Shadow you have the cause of the paint and your um you know you're interpreting the data

1:01:26 which is the luminance and making inference about oh you know it must be that this you know this Shadow has to be discounted because um it contributes in some way to the lumit okay so that kind of perceptual inference process I think is happening all the time in our visual systems uh it's you know it's a very sophisticated uh uh computations in my understanding and um it's uh it would make sense uh and there's various pieces of evidence for

1:02:00 the brain to be using um some kind of probabilistic computations in order to solve that inference problem um so that's kind of one you know one way that you could cash out um prob of perception or uh you know in in in that in that inference process and then and then the other way people often talk about probabalistic perception which I also agree with is going from um uh you know what the question is kind of what information is available to Downstream computations that might be

1:02:36 involved in perceptual decision making decision- making for example so um um so for example uh if uh if the output of perceptual inference is just this point estimate what we would call this point estimate meaning like a single valued interpretation of some visual feature um like oh the the cup is exactly you know 8 Ines for me or

1:03:11 something like that um if that were the only information that were passed on to Downstream computations you you wouldn't be able you you would have certain patterns of behavior that you know that are limited to being able to make use of that limited information um but it doesn't seem like that's what's happening it seems like uh uh in decision- making we have access to much more information than that it seems like we use that information in a statistically

1:03:44 appropriate way so according to you know um you know what what you would want to do if you had access to probability distributions and so so um so I think you know there's plenty of data that we have access to more than a point estimate in these Downstream computations that we use for perceptual decision-making uh we have access to that information when we make our confidence judgments or confidence reports um the question is you know how how much information do we have access

1:04:17 to later on um at what points in the system is information lost what kind of information is retained and what kind of information is lost and then what are the specific kinds of computations that are actually happening because um uh you could have computations that are perfectly basian and statistically perfect but they're making use of degraded information so you know so the answer isn't perfect um or you could have kind of rough approximations

1:04:51 of basan computations that make use of really good information and again the answer would still not be perfect so I think that's where that's the you know where a lot of the real kind of empirical work needs to be done just figuring out you know what information is available and when and what are the actual computations that are being used to make these perceptual decisions so the so the probabilistic view you said this earlier but it's committed to this idea that there's multiple representations uh that are competing in a sense sort of or that you have to be

1:05:22 summed over actually as it is but uh so that uh that they're all there but I take it that the alternative is that there's just like a point a single representation with a probability estimate attached to it um maybe something like you know red there 75% maybe or something like that so that you have the the probability that the statistical information but also the single representation without multiple representations are is that what the difference between your view and block

1:05:57 and Doby's view was in this uh paper or did I misinterpret that yeah so I so maybe it would be helpful to kind of spell out a few different options for yeah these used um so starting with the really uh no there's just no probabilistic information at all that would be like just the point estimate there's only a point estimate at the very other end of the extreme is there's a full probability

1:06:30 distribution uh present and when we say probability distribution here we really mean uh with respect to some visual features so for example let's say again we're thinking about like the distance of something from us um the full probability distribution would mean um there's information that could let us read out um the probability that the cup is exactly 8 you know inches away and also the probability that it's exactly 11 in away

1:07:03 and also the probability that it's exactly 3.5 in away so there's there's information in uh in the representation that could let us extract the likelihoods of different um all the different possible distances of the of the cup and so this would be like um we we we might think of this as like a posterior probability distribution over the distance that that cup is away from us um so that's the that's the extreme example on the probabilistic end um um

1:07:37 and you know you could add and you know that full probability distribution's worth of information is used probabilistically like it's used in terms of that information gets combined with other information in according to the rules of probability and so forth um so I think of I think of the full probabilistic view as like a combination of the nature of representation like there's a lot of information uh that could be used to get probabilities for any any possible

1:08:13 outcome you might want um and there's specific computations you know there's spe specific ways that that information is used that respects um uh statistics right so so it could go wrong in a couple ways right it could not do that empirically it that's right so it could it could it could not do that empirically or there might not be enough information so the so one intermediate view that you just suggested and this is one that um you know Doby and Ned have suggested although I will say that um I think the I think you're you you uh the

1:08:47 preprint that you're referring to is one that we're still um we're we're actually revising and this is uh you know this has been this has actually been a really interesting paper because um yanica and Doby and Ned uh we all had you know a lot a series of conversations trying to you know really work out these different Alternatives in in terms of what what would it mean to you know to say that perception is probabilistic and and what is the current current evidence that you know um and how does it relate to those

1:09:21 options so like the Intermediate view that you just said uh would be well maybe there's just a point of submit plus um some other single number that indicates the level of uncertainty uh uh in terms of maybe a standard deviation of a gussian distribution or something else um so that would be a much that would be a much sparser representation um in some cases you could use that kind of representation to read out

1:09:54 probabilities of all the different options like when the underlying distribution was gausian um but in some cases you know you couldn't you couldn't represent more complex probability distributions for example and so that kind of intermediate representation would be a you know would represent some loss of information and so you think that's a counts against it um I well are you just thinking that it's not so that's an empirical question whether there is all that information there or some was lost that's right I think it's an empirical question and I

1:10:27 think and I think the answer will probably end up being uh fairly nuanced because I think it may um depend on you know at what stage in neural processing are we talking about you know how much information is there and um and what um you know what approximations needed to be made uh in order to represent that information and so on so I think I think I think the question question really comes down to you know how much information is available how much information is lost at What stages

1:11:01 do all those things happen and then what are the computations that then make use of the information that is available do those computations operate in a probabilistic fashion um or do those computations really just do something more heuristic um uh that usually gets the job done but doesn't really respect the um you know rules of probability interesting and so just one question back to relate this to the phenomenology or to the conscious perception so I understand what you what we're talking about now

1:11:33 but do you think there's any aspect of conscious perception or phenomenology which is maybe not probabilistic but uh generic or abstract or not precise or somehow represents possibilities in a schmeared out way or do you think that perception is always like precisely defined phenomenologically wise or do you not have an opinion yeah I mean I think this is a really interesting question um and I yeah I I I dealt with some some of these

1:12:07 issues in the the one the one philosophy paper I I have I have ever written um which uh you know was thinking about um the notion of perceptual Precision so um I think uh in terms of our phenomenal experience we have a single interpretation of what's happening but various aspects of our of that interpretation are associated with

1:12:42 uncertainty um and and so you know one one example that I think about is if I you know if I hold up some object uh in my periphery um my interpretation on the one hand is that it is occupying one specific location um but I also have a sort of smeared out sense of what that location is like I don't have very good uh resolution about

1:13:17 what that location is and so the question is you know how can we how can we characterize that aspect of phenomenology you know um one one view which I which I don't like is to say well we see it we actually see the thing in multiple locations and attach different probabilities to those locations um that doesn't quite seem right to me in terms of you know just

1:13:50 from in introspection um but another view what which I would prefer is um well we we interpret it in one location but but we also have a sense that the information we have about its location is is imprecise we kind of you know it's a bit coar grained it's a bit indeterminate um so that's um yeah does that answer your question yeah so that would be

1:14:25 phenomenologically then you would capture that by not saying that it's representing different probabilities of things being here or there or there but that is just kind of like a more vague or more of a sense of something being there as opposed to it actually being located there I would say that you see it in a location but with a low Precision but with a low Precision but what does the low Precision part mean the low Precision means that um your

1:15:00 um your phenomenology yeah in the sense of your phenomenology um I mean it's interesting right because you can describe you can describe your phenomenology at different at different levels um so you know if I were going to make a impressionist painting of what I of what I saw I could probably do that fairly exactly by um you know kind of putting bits of paint all around this General region to to capture the

1:15:33 imprecision of my experience right um at the same time if I were going to make an interpretation about where the object is so an inference about the location of the object um I would not say it's in all those locations you know I would not say it's in multiple locations I would say it's in one one location um but I just can't tell very well what that location is yeah and do you think that the probabilistic if there are probabilistic processes that lead up to perception that

1:16:07 they influence conscious perception in any way or have they by that point disappeared completely in your view yeah I mean I think that the um I think that this uh yeah there are different ways of thinking about this and I do think it's helpful to sort of maintain the idea of the hierarchy of um of visual processing and think about okay what is the representation at

1:16:42 multiple levels and what is the phenomenology at multiple levels because I also think our phenomenology has you know multiple simultaneous um attributes uh that we can access you know we can access like I said this kind of like potois you know rendering of our experience or we can access the interpretation about you know what like where is that thing and like what color is it you know and stuff like that um I think that um the uh the information that's present in the neural

1:17:17 representations in the visual system you know gets translated into those aspect of our our phenomenology um and so it might be the case for example that um you know at a at a fairly you know low level of the visual system we have you know we have a population activity of neurons that um has uh uh has information in it about you know that could be could be read out as probabilities by decision processes and so forth uh you know some kind of

1:17:52 distribu of activity which is imprecise you know which has um stronger firing for location for you know 8 cm away and like less fire you know you can think of different ways of of cashing this out um but so that pattern of activity May contribute to this sort of like potois uh aspect of our phenomenology which feels imprecise um but then in a later step reading out that activity could give like a best guess as to where the where the thing actually is and that could to

1:18:28 the you know to the kind of singular aspect of our inference about like well I think it's I think it's there um you know but we might have access to you know even if even if in our phenomenology um you know our phenomenology has these sort of like singular interpretations because we have these multiple levels of representation um we may be able to use that to actually you know to to make guesses and things about where things are to to make richer

1:19:02 descriptions of where things are likely to be sort of given our full phenomenology and that would be a cognitive process right you know be a decision process to say well let me inspect this phenomenology and all its different levels and I can make some you know actually I can actually give some fairly Rich information that would be like you know likelihood of where this thing is and in fact there's some you know it's a nice recent study from um darl's lab that shows that if you if you ask people to make sort of multiple

1:19:35 guesses about you know what shape they think something has um you know they they can give you more information in subsequent guesses than in just the first guess and I think that's consistent with like um you know a lot of the perceptual decision-making literature that shows that we have access to pretty rich information or C certainly more than a point estimate's worth of information um about about things and I think there's this sort of open question of you know when we make those decisions is it that we are um unconsciously accessing the neural

1:20:11 representations that are very rich um and reading out Rich information from those representations um or is that is it that we are um you know consciously inspecting our our phenomenology and sort of using the multiple uh layers that we that we have to you make uh um make these kinds of perceptual or like metacognitive decisions yeah interesting huh whole new can of worms just got opened up there well uh that that's super interesting I mean there's so much

1:20:45 fascinating stuff in your work I could talk to you forever but I know time is running out and I have to let you go but at the end I just want to ask you if there's anything that you were hoping would come up during this discussion which hasn't yet come up which you want to talk about and the answer can be no that is fine well you know there is there is something that I wanted to mention on your Consciousness podcast uh which is um yeah which is just something that I would really love to see um in the field of

1:21:18 Consciousness science uh which is and it's sty we yeah we didn't we didn't talk too much about this at all so I'll bring it up now we I would love to see um a list a list of uh properties of consciousness of conscious experience of human human conscious experience um that we would that we would as a field like like to explain um and um the analogy that I think of is you know um

1:21:53 uh Consciousness sometimes feels like such a you know big problem to tackle um if we think about something else like uh the problem of you know what is life how do like what is life how does it work and I and I know people give this example but I think it's a useful example um because you know biology now we have some understanding about various properties of life right like properties like um oh how does how do uh how does energy work and um how does things grow and reproduce and move and you know things like that so you know once you

1:22:28 break it down into those different properties you can really start to make progress and I think um in the science of Consciousness it it could be really useful um for for us as a community to get together and make a list you know a list of of the of the kinds of properties that uh that we want to be able to explain uh about phenomenal experience um and uh and then you know and I I think this is uh PE different

1:23:01 researchers currently are tackling a lot of a lot of these properties but maybe it would be helpful to have a more sort of unified um effort to kind of uh just write some down yeah maybe even have like a a joint statement you know like they've had animal sentience or whatever important issue a bunch of scientists come together and say these are the things we think are important so it be it would be I think very important to have such a list now getting all these people to agree on anything like a list would be take a conference and a millennia but what but

1:23:34 I agree with you this is something because there's a lot of work on Consciousness and then it's like okay some that it's like and then We're Off to the Races everyone using their own methodology their own preferred approach but there is no agreed uh I mean this is a nonse but I'm sort of thinking it reminds me of Thomas Nagel's call for an objective phenomenology um and you don't even need to go that far some like way to describe experience without using experiential terms or something but but something about the Dynamics characteristics of human experience that we would agree if you could predict explain and manipulate these things that would go that would

1:24:07 count as having something to do with experience because there is not really an agreement on that as a matter of fact especially amongst per per uh protagonist of specific theories of Consciousness yeah let's make a list yeah I agree let's do it okay well thank you very much I think that's an important call for action for the field um Rachel it's been a great pleasure I really want to apologize for the for the mixup at the beginning of the Stream So the conversation at the beginning feels a bit um uh part of it's

1:24:41 missing is because it is but uh nonetheless I still I I want to thank you for your time this has been a a great conversation I myself have learned a lot um from it and I highly recommend your work to anyone interested in this in these questions and um I I really like it when you step out of the the kind of particular questions you're interested in and address these bigger questions so maybe maybe a book will be in the work uh where you're putting all this together that'd be that'd be pretty cool um but uh we'll have to stay tuned for that I suppose um in the meantime let me just say thank you and stick

1:25:15 around for one second I'll say bye out there but once again thanks for joining me okay well thank you thank you so