Consciousness Live! with Michael Graziano (S6 E3)

Season 6, Episode 3 · January 22, 2024 · 1:18:23

Listen: Apple Podcasts · Spotify · YouTube

Where the big topics come up

  • 59:26 Attention: “…may develop on their own I think you I mean well so we come back to the attention schema Theory and and I always try to point out it's a specific Theory within a larger framework…”
  • 3:24 Physicalism vs. Dualism: “…about Consciousness were you uh would you have thought of yourself as a physicalist or would you not have thought of these issues at all or um just I think that I I had range…”
Cite this episode

To cite a particular moment, add the timestamp (for example, “at 1:02:15”). Links to this page with ?t= and a number of seconds open the video at that point.

APA
Brown, R. (Host). (2024, January 22). Consciousness Live! with Michael Graziano (Season 6, Episode 3) [Video podcast episode]. In Consciousness Live! https://onemorebrown.com/consciousness-live/s6e3-michael-graziano/

Chicago
Brown, Richard, host. “Consciousness Live! with Michael Graziano (Season 6, Episode 3).” Consciousness Live!, January 22, 2024. Video podcast, 1:18:23. https://onemorebrown.com/consciousness-live/s6e3-michael-graziano/.

Transcript

Automatic captions from YouTube, lightly cleaned. Names and technical terms are sometimes misheard. Click a timestamp to jump there in the video.

Show / hide transcript

0:05 okay so I think we're back now all right um yes let me just redouble check this uh to be sure um okay yeah yes indeed we are live after all of that uh um unfortunate beginning so I apologize for that I don't know why

0:38 the uh software did that so I'll take full responsibility that for for that but uh we are now live so joining me today once again Michael Graziano so can you say again for those who might not know you who you are and what you do sure well my name is Michael Graziano and I am a professor of Neuroscience and Psych ology here at Princeton University I've been here a long time many years more than two decades I've done a lot of work on a lot of different areas of

1:12 Neuroscience uh motor control sensory processing but uh more recently really in the last 12 to 15 years I've worked on this question of of consciousness of what is it why why do we think we have really a whole collection of properties that people call Consciousness and can we understand it can we build it artificially and is that a good or bad thing for us to do and these are the the questions that uh my lab and I study

1:47 very cool okay so and you've been so when you were uh working on your on the body scheme of stuff that you did previously were you interested in consciousness I mean it didn't really come up until like you said there was a certain point at which you're kind of the focus kind of changed but I was wondering if there was a a lingering or background interest during those days as well yeah I think everyone in Neuroscience probably everyone in Psychology at some point is interested in Consciousness you can't help it actually everyone on Earth with any

2:21 ounce of self-reflective uh interest is interested in that question so yeah of course I was always interested in it I think one of the things that happened in studying um the body schema and uh and other aspects of movement movement control and representing the body is this realization of how important self models are for the brain and um that morphed into this uh question of Consciousness and you

2:55 know we walk around claiming to have lots of interesting consciousness properties and um I think that's a self model that's us trying to come to grips with understand model what's going on in there and so I think there was a there was a point to which all the previous work kind of set me up to transition to the the Consciousness question interesting um and so I guess I was wondering a little bit about if your background views on Consciousness before you

3:27 uh I mean you say in your book that you think and it's other places well you think there's a kind of inevitability to the attention schema Theory which what we're going to get to eventually at some point I think but uh um I I wonder if like your background before you started theorizing about Consciousness were you uh would you have thought of yourself as a physicalist or would you not have thought of these issues at all or um just I think that I I had range of ideas um so I I think had you asked me this

4:02 question 20 years ago I would have had a totally different answer and I would not have thought deeply about it and I would have said oh yeah it's this thing it's the feeling inside you and this and that and I nobody understands it and I would have more or less followed the same kinds of tropes that one hears a lot that it's not understandable nobody understands it it's a feeling it's not the physical processing of information it's some adjunct attaches to the information that's not physically measurable and so on and so I would have said all those things that everyone else

4:34 says um I I really think it was the the act of intensively studying how the brain models itself and how those models are never accurate right they're not supposed to be they're supposed to be useful and they're supposed to be simplified so that they're easy to compute on the Fly uh and what that means in terms of our understanding of ourselves or the world in general you know all our understanding everything we know everything we think we know is based on models that the brain constructs uh and those models are never accurate uh and and I think it was that

5:09 one had to think deeply about that to um arrive at that um that uh that understanding yeah so yeah okay so because it I mean you talk a lot about the meta problem and that's what sort of what it sounds like you were grappling with in your own uh coming to study the brain and try reconcile it with maybe these ideas that you had before was like why do we think Consciousness is so mysterious so it's that's what I was kind of curious about was exactly what you

5:40 know your pretheoretical con conception of Consciousness might be because I think that's going to shape a lot the way that you think the theory of Consciousness needs to shape up um afterwards I don't know if I was it's very it's very hard for me to fully grasp what it means to be a physicalist uh and I think there may be some deep contradictions in in those categories when you start thinking about them really carefully um so what does it mean to say Consciousness is a physical

6:15 thing I don't I don't know um arguably the most physicalist theory of all is the most absurd theory that Consciousness is like a ghost material like ectoplasm magic ghost material right that a physical thing that's a physicalist theory of Consciousness uh and um whereas you know the the kinds of things that I deal with now are are more questions of um what is it that we think we have what is it that the brain how

6:49 how does the brain model itself and and in what ways is that model accurate or inaccurate in what ways is that model really useful and important uh for our um survival are um are you know functioning so the the more philosophical questions of is it a physicalist theory or not is it an Illusionist Theory or not actually they start to become less important to me yeah and I can see why a lot some of it's really semantics in in a certain sense but H sometimes the semantics

7:23 matter and and what you know to to try to clearly POS I The View we want to you know maybe it doesn't fit neatly into these categories but one would like to know um well is there anything more than the brain and the way the brain functions necessary to explain uh Consciousness and I think on your your answer is no um that's right so that if that's what you that's sort of what I

7:54 mean by physicalism of course you can talk about basic particles and all that sort of stuff and that's great but I think at the at the fundamental level at the psychological level sorry not at the fundamental level but at the psychological level the question is can ear science come up with something that tells us about Consciousness yeah um and so the people who aren't physicalists want to say no and I take it that you want to say well either it's a confused question or yes but I wasn't sure which one of those is the right one to get you to say maybe what you're saying is you don't care

8:28 which one well yeah it's a it is partly a confused question but part of what I'm saying is the confusion is really interesting because the confusion is not so much scientists sitting around failing to grasp it the confusion is buil in um it's sort of like the as I've often used this analogy it's sort of like understanding color right we processed color and Newton figured out that color is a mixture of all all different well he didn't understand wavelengths but uh

9:02 there's a whole Spectrum but color is constructed by the the eye and the Brain something Newton figured out that there's um three primary colors which are constructs of the eye and the brain and the color that we see is not what's actually out there uh and Newton is the one who figured out that white that looks so pure and clean is actually a mixture of all these different things um and so you know if you ask the question if you imagine the hard problem of color uh how does white light get scrubbed clean of all

9:37 contaminants and turned pure right the answer is that's an incoherent question right however it's really interesting why that question comes up and um it says something quite deep about how the brain represents uh the eye and the Brain uh process and represent wavelengths and the spectrum of light so it's not you don't throw it away you don't say ah it's incoherent forget about it what you do is say well it's really interesting that it's incoherent in this complicated weird way why why does the brain process it that way um

10:11 what's the benefit of an incorrect but really simplified and elegant model of that aspect of the universe and I think that's kind of my perspective on Consciousness I I'm happy with I love the fact that it's so complicated and incoherent and magical list CU I think those are baked into us that's like how how the brain evolves to model itself MH so do you really think that it's like really part of the Common Sense notion of Consciousness that it's kind of not physical or magical or something like

10:44 that yeah uhuh when go ahead you y sorry I I was going to say when you press people on it so we did a survey study um a little while ago we just asked people online um in your own words what is consciousness and we got this huge range of responses and we sort of sorted them and categorized them and most people think of Consciousness as a perceptive kind of thing it takes in and understands um but it itself the The

11:21 Experience itself is not physically describable so you can't measure its weight you can't measure its heart Ness or you know it's um uh combustion temperature or things physical attributes of it you can kind of locate it in space because it's kind of located inside people these are the kinds of Concepts people attach to it so it's nonphysical in the sense it doesn't have a list of physical attributes one can measure about it right but people think of it as there so there are there are these Concepts that seem very Universal

11:54 that people have about about Consciousness yeah so when people I wonder about this because you know I think probably before I started studying philosophy I was a naive duelist my default position was probably that it was if I ever thought about it that the mind was not the brain I'm now not that way and I sort of um consider myself leaning more towards physicalism if you know depending on if anyone cares about that but nonetheless um so I sort of seize both sides uh um but I wonder like when I would when I naively my old self

12:28 thought oh the mind isn't physical um it's sort of like the way that a lot of people say humans aren't animals um and then you cuz everyone says that but they don't really mean that they they can't really mean that because humans are animals um it's just that they a special kind of animal which usually contrasted with the rest so we find it laborious to say non-human animals and so just say animals and then in the common mind I don't know that kind of just distinction gets rifi or something but couldn't

13:00 something like that be happening here too that when when people are thinking oh that's not a physical thing what they mean is that they're contrasting with all these other things you know like uh I mean you ask people if the wind is physical and they they they come around the saying yes but not automatically yeah because you know they have to think about it for a second like but H okay so what part of physical I feel it I guess but I can't see it taste it Etc or maybe you can't taste it I don't know but anyway so is that do you think it's more than that in the like General Common Sense view

13:34 um I think you're getting at something that sounds right to me so when there there's a whole list of attributes that we think of as physical and if something has all those attributes okay that's a physical thing like a rock has hardness and it occupies space and you can bash it onto things and so on and you can heat it up and cool it down and do all kinds of other things to it and measure all these properties wind in people's experience is missing some of those properties but it has some of them and so it's not quite as physical seeming in

14:10 people's cognitive categories uh so what kind of representation do people have of their own Consciousness effectively their own um highlevel attentive processing of things in the world what what kind of representation do they have the attributes they tend to attached to it uh well they think of it as having a location usually because it's kind of floating inside me they think of it as doing stuff like it allows me to understand and to make decisions but it lacks all the other physical properties

14:44 in people's concept right so there's there's neurons now we know people know there's neurons that might be related to it but neurons aren't it it is something floaty it doesn't have a a hardness and you can't knock on it and hear the sound and you can't measure its weight and you can't do all these other things so it's lacking almost all physical properties other than location right um and the ability to to affect things in the world in the sense that it it affects our Behavior so I I think you

15:20 know the brain builds these representations of things in the world and some of those representations include a lot of physical property like for a rock and so when we build a representation of wind it has fewer physical properties attached to it and this is something that has even fewer physical properties so it's it's hard for people to say this is a physical thing it's it's really in a sense a non-physical thing I think that's what's going on and then of course on top of that there's cultural layers and layers of belief in magic or belief in you know um uh whatever properties when attaches

15:57 to the immortal soul but those are culturally learned um I think there's some things that are not they're really built into us about how the brain understands its own internal processes right and I mean we'll get I know we keep circulating around the attention schema but I think that's an important point is that just like the body schema it's the the model that it builds is automatic out of our control um and I think the Phantom limb example is a really good example uh and not to mention the rubber hand illusion which I like the way you describe it but I never

16:29 I had that strong of a I never got the finger moving version of it so um these things really are out of our control and impress us with their like reality uh no matter what is there or not and and that gets I mean brings me to something I was curious about your view because you talk a lot about there being representations information in the brain um and that so like how would the brain know like brain know that it was conscious or something like that or what would if you asked it they it should say

17:01 there's an apple but it wouldn't say you know there's something that's like for me to see apple or anything like that unless you did these other things and so part of me is curious to know whether you think there could be even in principle um a creature that was conscious that couldn't answer that question that didn't have that extra thing attached or do you just think is that conceptually not something that could happen like a first order Theory yeah I think that um if I'm right if this theory is right then what the creature would have is

17:35 attention like it would have that property but it wouldn't know that it has it because it wouldn't have a representation of it right right and so it would have the kind of Base physical reality that gets recoded and represented in the brain as Consciousness so you could say of a creature with no body schema does it have arms and legs uh it might but then you could ask does it know that it has arms and legs well no it doesn't have a body schema so

18:09 it didn't represent those aspects of itself and so in this attention schema Theory um you could ask does a creature have attention does it have an ability to focus its mind and deeply process things and the answer might be yes but then you could ask well does it have a schema or mod a representation of that process does it know that it's doing those things and in that case if it is no attention schema the answer is no so um and and the the distinction the The Quirk here is that the schema is not

18:44 terribly accurate right so you take the body schema uh the creature with arms and legs no body schema it doesn't know it has arms and legs but you could also take a creature that does have a body schema but it's not a very accurate one and then you start asking it well do you have arms do you have legs and it says yes I have a you know a giant arm that's the strongest thing in the universe I don't know whatever it's weird body schema is telling it um and and uh you know now you have something that thinks it as magical attributes that it doesn't

19:18 have but it thinks those things because they're a model of something it actually does have right so anyway that's what I would say of this whole Consciousness thing yeah people can have creatures can have that base reality an attentive mind that can process things in depth creatures can also have a representation or model of that and US humans at least have a model that's so simplified and inaccurate that when we report it we describe it as yes

19:52 I have a magic Essence inside of me the Consciousness Essence but so like for example the Frog I think a a frog is an example of a creature that you say probably doesn't have the attention schema I is that yeah okay I would think I would think not okay so but they they do they have attenion um uh and they so from the point of view of the attention schema if they don't have it then they're not conscious in the sense of that they're not they're not going to have this extra thing which is attached or whatever

20:26 which has some functional benefits by the way so it's good to have around um but from the point of view of the Frog there is no so there's no subjectivity or no I mean cuz some people might want to say there's something that's just like to be the Frog uh it's just not very complicated or very exciting but it's all this other stuff about the world out there and the way it represents it um you could be a physicalist about the Frog and then say so why why not and I'm not saying this cuz I think that in fact

20:58 you know I Harbor suspicions that you probably need more than just that anyway but what what is your reason for thinking that that's not enough to say there's some kind of Consciousness there something that it's like to be the Frog snapping at the fly yeah I think what it really comes down to there's a kind of a heart to this whole concept this whole way of thinking it's kind of like the the center of that gordian knot yeah um that never mind the Frog for the moment

21:32 we'll come back to the Frog but for a person everything we think we know about ourselves everything including our absolute certainty that we have the experience uh and the Consciousness and the this and the that everything we know derives from information in the brain or we wouldn't know it we wouldn't say it we wouldn't think it we wouldn't believe it um all of that deres from information or if you will to put it differently um uh almost like a conservation law if it's coming out of your mouth that information must have been in in here

22:06 otherwise it can't come out right so uh it's all information everything has to derive ultimately from information in the brain so when you talk about your experience you know I I like apples um a favorite food so I always write about them but you look at this apple uh and you say it's an apple and it's red and it's there and I have a subjective experience of of what it's like to hold and look at the Apple uh what you're doing is um the your un your belief there's an apple

22:40 that obviously depends on representations of apples but your belief that you have a what it feels like that you have a subjective experience that there's something more than just processing the Apple that belief must also depend on information of a type con constructed in the brain I consider that logically necessary um and if you didn't construct that information if you could go in the brain and sort of suck out all the neurons and parts that process that other kind of information you would have this person with um For Whom the whole Consciousness

23:16 thing becomes irrelevant like it's just gone right uh so um that's one of the the The crucial points is that it's all it all derives from information but could that person still be conscious though so they wouldn't know it but why couldn't so I guess what I'm I like this is like the the you're right the archimedian main argument that you present is that if you're going to say your conscious has got to derive from some information in the brain so there's got to be something in there on which the belief

23:48 is based um okay so I agree there but why couldn't it be Consciousness on which it's face it could be but here's so here here's the second part okay um so now we we provisionally accept at least in following this argument that the claims we make whether about Consciousness or otherwise about anything derived from information let's call this information model building yeah um in the loosest sense that's what the brain does it builds models of

24:21 itself in the world and this and that okay here's the second part A A A contention that or a proposition that I offer all models in the brain are inaccurate they're all wrong um they're all simplifications all of them um or as the statistician George Fox is supposed to have said um all models are wrong some are useful right and I really like that quote so the brain builds models they're

24:55 always simplified cut coin one or approximate always and so if that's so then we have this simple conclusion that all the Consciousness stuff that we say and believe we have is derived from a model the model or maybe multiple models the models are not perfectly accurate so whatever we do have and let's presume we do have something in there whatever we do have is not really quite the same as the thing we think we have and claim to have

25:29 right right so now we are as scientists faced with this much par down question what is the thing in the brain that we really have that leads to this inaccurate model which leads to our claims of a kind of non-physical magical Consciousness and once you straighten that out you realize the thing we actually have does not need to have all the properties that that we think it has right right it doesn't need to have the non- physicality it doesn't need to have

26:04 the mystery it doesn't need to have um all these impossible to explain hard problem attributes it could be something else it could be something simple we don't know what it is uh I propose that it's attention the brain's attention process that's the thing we really have that gets represented in a mystical simplified form that leads us to claim to have Consciousness that's a that's an hypothesis and you know it might be wrong uh but um I think the logic of the situation forces one into a particular

26:41 perspective there is something it's different from how it's how the brain models it right and what we think we have is based on how the brain models it uh and so that's to me the only the only workable logical answer to Consciousness that General framework if you wanted to keep it within the confines of what a brain can do um well if you believe that our contention my my my contention that

27:17 I'm conscious and my belief that I'm conscious depends on the brain If you believe that everything else follows from that yeah so I mean not that I believe it but I would say some people would believe that there's a kind of you know mental quality which you can know about in a non causally information based way so you're have to at least deny that there is such a thing like that and I I mean I'm fine denying I'm just saying that uh not everyone's going to be fine with that AB absolutely so it all comes down you know of these two initial

27:52 propositions the first one is that everything we believe and claim to be true everything no matter how certain we are of it derives from information in the brain what that's principle one and there are people who don't believe that uh but I do as a neuroscientist I think that's um obviously true and it takes um disc connection from scientific truth to believe otherwise that's what I would say that number one this is true and proposition true two is the brain's models are always simplifications and you put those two together and you get

28:25 this General framework uh for for a theory and then one can you know try to fill in the details which is what we've tried to do but yes that's right so I wonder what you would would say to someone and here I'm just curious about what your response would be to someone who's going to be like you know uh more of a fan of introspection so they might say look you know when I I taste this tea drop on on uh a it's gone you can't see what it is but this winter tea is quite delicious the certain nutmeg make cinnamon

28:58 combination that's quite ever so delightful and when I focus on that um like when I'm enjoying the taste of it and I'm really just like wow this's taste and it's there uh it seems like everything about the taste is or from from the point of view of it being an experience of taste um is there it's right there for me and so what you're saying implies that I'm mistaken about that um that it's really the basis that the brain presenting a simplified model of something but there

29:31 is no like quality of tastiness that mixes this cinnamon and nutmeg in a certain way that that's not really there at all and so first I wonder if like do you think that phenomenologically not cognitively but like experientially seems that way and if there's any Merit to this um line of critique uh I think that probably the most common argument in Consciousness studies philosophically

30:06 especially uh that I run into that I've ever read that I've ever seen the most common argument goes something like this I know it's there because I'm experiencing it right um and you know whether it's the tea or some people say well I can poke myself with a pin and it hurts it's there I know it tea is better than poking tea is much better than poking uh so um I know it's there cuz I'm experiencing it uh right I'm experiencing it right now so I know it's there the problem is the it that we're talking about is

30:39 experience right right and so now you're stuck in a a pathology I know experience is there because I am experiencing it I know a is true because a is true I it's true because it's true and um I think that I mean that's not false toies are true yeah I think what's happening is um cognition is accessing deeper information always deeper information is

31:13 supplying it with the same answer over and over and over and over again and it can't get out of that and if you I think it's in in a sense similar I don't want to compare Consciousness to a delusion because I think a delusion is a patholog iCal State and Consciousness is a really useful important adaptation but in a delusion uh people can get stuck in the same thing you know when the wiring is messed up where squirrel in the head um the squirrel in the head for example yeah right that the guy and you know that's a bizarre wonderful and horrible

31:47 example for the guy who actually believed that but yeah uh in but just in general delusions there are people who are absolutely at a gut level certain that something is true and when they introspect uh their cognition is basically accessing deeper information constructed uh automatically and that information is always the same always coming back up into cognition and always telling them yep it's true yep it's true and they're stuck in that they can't get out of that one one can't escape you know one is trapped by the information

32:20 available in the brain so I think that's what's going on um that that uh it's to Circular uh and uh tautological to say it's true because it's true experience is there because experience is there uh and I think one can't even think the thought without the brain having built some rich representation of what it means by experience and that it's happening right now and so on so I think it's self models I think the brain is

32:52 Building self models um but I hesitate to I I I um I want to make sure it's clear I'm uh many many people will say aha so you think there's nothing there there's no there there uh but what I'm saying is there is there's presumably something there that the brain is building a model of it's just that the model is not super accurate right and so the thing we think is there is really different in many ways from the thing that's actually

33:26 there it's as though um you know in we had you know two we bipedal like we really are but the body schema represents us as being like a stump or something like that and we're convinced that we're stumps even though like what's there is completely other than uh there is a body it is a certain way but if it was represented in a really weird way that's how you would expect it to be and You' behave in in that way yeah and there are people who you know have that exact problem whose body schemas are really distorted uh and so they can't I mean there's so many examples of that of

34:01 very very weirdly distorted body schemas that people are absolutely convinced that's what they have yeah but objetive I mean so part of me likes this argument part of me wers like isn't it sometimes true that it's true because it's there so if I mean just ordinarily if I can can I you use experience to justify that that I mean why do I believe that there's tea on the table in front of me because I can see it is part of the reason and then there's other stuff about reaching and gra but here's the problem um cuz you can see it therefore there's tea there's

34:38 something right but what you're not saying is how do I know I can see the te because I can see the tea right that would be much closer to a topology of so uh the problem with the Consciousness thing is that it's not a separate evidence it's I um the evidence that I have experiened is simply that I have experience and that I have a problem with that I think is just a a um a logic loop it's a machine stuck in a

35:13 logic Loop that's what I think it is I mean I certainly agree with you that a logic machine could get stuck in that Loop but I wonder why you think that I or you are stuck in that Loop when exper itially and not cognitively I mean so it it just really seems that way that there's something experiential there and I know I believe it as well and so you know I I think you're focusing on the cognitive aspect which is good but isn't there something on which that cognitive

35:45 thing is based and you say yes it's just not the way we think that it is and even there I would say yeah you're I agree which you know in the business we would call weak illusionism for amongst the philosophers weak illusionism I think think almost everyone is committed to the idea that Consciousness is not the way we ordinar think yeah ordinarily think it is but that's different than a strong illusionism which says there's no nothing that it's like and I think that's what gets people riled up is thatz like you know there is something that it's like what it's like is this thing that we've been describing this

36:18 simple quality thing and what what you're saying isn't really there so in a way um why are we focused on the lower level as opposed to where all the seeming stuff is located which is where the Consciousness business would you know that isn't we should be talking about well uh the seeming business I mean the fact that you are so convinced that there is a seeming and that there is a phenomenology and that there is an experience again I would argue if you

36:52 believe in Neuroscience that must derive from information in the brain the fact that you believe those things that you even think those things exist that you um are so certain they exist certainty of course is a computation uh so all of that deres from information it's not I I think the mistake people make is when they look at a phenomenon like sipping tea and having a taste and they think that they they the image the concept is that there's um a phenomenal experience that just

37:27 sort of appears and is part of the tea and what I'm suggesting is no if you're saying that if you think it if you believe it in your gut no matter how uncogent and then attached it to the tea uh and if it didn't do that you wouldn't have any of that you wouldn't have any of those thoughts or beliefs and it'd be irrelevant to your life and we wouldn't be having a conversation about it so

38:01 something must be Computing that information in there right and uh so I I think it comes I think it comes to that uh so I mean I'm I'm partially on your side I'm mostly just trying to draw you out here but I'm also generally struggling with this because let me try from a different point of view I wonder um if we can bring our good Frenchman decart into the picture so what do you feel about this kind of idea of radical doubt I have you ever encountered any cartisian

38:33 before I I have thought a bit of I mean you know I have my own views of of Bal Dart uh one one thing I note just superficially Dart actually doesn't talk about Consciousness he talks about thought exactly and um if you think if if you imagine if you categorize thought as a type of information processing then yeah sure we think of course we think all lots and lots of thoughts going on in there uh and dayart actually never really asked

39:06 the question about subjective experience per se right because that really wasn't framed in that way back then uh so that's that's one point that makes it a little more difficult to try to interpret dayart in a modern sense uh but you ask about this radical doubt um I I would ask this question if I could of cart uh so you you of course know the the by now extremely famous thought experiment to try to capture the cartisian Viewpoint uh imagine your brain is in a vat and aliens are

39:40 manipulating with electrodes uh and everything you think you know is not really true how would you know the difference um here's what I would say to dayart what if you think that you've come up with a perfect airtight logical syllogism but every time you think through it the alien decides to screw with you and sends a little spark of electricity to your logic circuits and makes you leap to some totally illogical conclusion but you would never know it right and so what if it's true that

40:14 everything you think is true is false what if your logic is wrong what if it's true he he does he sort of deals with that but he tries to anyway not in a good good way but his you know namely I don't want to think about it way but uh I agree that's a that's a point if any case anytime there step-by-step reasoning you're subject to the Evil Genius coming in and messing with you deart pretty much that's right so he says I I think therefore I am sounds good sounds logical but what if it isn't right if you're really going to introduce radical doubt then the truth

40:47 is we know nothing of course maybe we don't know nothing we don't even know if we know nothing right if once you basically what I I see dayart doing is is saying if you accept no assumptions what can you conclude and the answer is nothing because conclusions always rest on assumptions yeah so I I have a bit of problem with dayart which strips everything out and then there's nothing left and then we're nowhere so yeah interesting um and I want to follow up on that but also I want to keep us sort

41:20 of on track but uh because I was thinking of more of the more modern interpretation uh of daycard where you know this could be where where you sort of think yourself to the point where right now you could be in a dream um and you so okay so if I'm going to dream right now then there's not really a cup of tea here that I've been talking about then you think to yourself okay well we replace the dream with some brain in the vat scenario um or even further the ultimate step replac that with there's nothing physical at all just just uh Minds um and then dayart says Aha and

41:55 something interesting is supposed to follow from that or carians who like this more modern interpretation of them um but it sounds like what you're saying is that you don't trust that that step anywhere along the process that you don't think there's so you can't get yourself to the point where you're like thinking seriously that this right now could be uh not physical I so I think that um there is a sense in which I not certain of anything and can't be because of arguments like the

42:30 dayart argument or the brain in the bat and yeah that's true and I can I don't really know if I walk out the door maybe I'll just see a giant weird desert landscape with desert worms everywhere I don't know I'm last time I looked or Beetle exactly exactly Beetle Juice you know it looked like a corridor the last time I looked I think if I even did that and if the memory wasn't implanted right I don't really know anything but if you take some minimal assumptions which I think we all have to do to get anywhere you know there's a level of doubt that's

43:01 just not useful right and so the world view in which yeah that that I exist in is essentially one where I trust evidence and basic logic and I've decided that that's okay and within within that world then I'm perfectly willing to accept that this is not illusion I mean my my brain is built a model of the world that's not accurate I'm sure that this is not the way the world really is in some objective sense but it's a

43:35 representation of it that's reflective of the real thing and um and so I accept basic reality of the outside world and the inner world right yeah and the only reason I was bringing that up was to kind of get you to talk about whether you think that there is a sense in which at that minimal level we can talk about experience because the way that I would interpret or lots and not I mean who cares about deart but a lot of people would interpret the you know Kito I think that I am as really something to do with experience that I experience therefore I am that's really what's at

44:07 the bottom of it is that you know I'm I'm seeing I'm thinking I'm willing I'm doubting those are all experiential terms um and so there is a sense in which even though I'm you know lean towards physicalism I sort of feel like yeah right now could this all be could I imagine a movie where this was happening in some sort of non-physical realm and I think uh that's not conceptually absurd but it sounds like and there still be experiences of tables and chairs and so forth and it would be you know there was a Robin William movies where he sort of almost had this as a plot he goes into a

44:40 painting I think it was called what dreams make up that's kind of irrelevant however the point is it's not conceptually absurd but it sounds like on the attention scheme of you like on your view of Consciousness that that we can't even get to that point where we can kind of start from this EXP experience first sort of position yep and maybe that's a good thing on your view but a lot of people would see that as like starting from a start backwards or inside out or something so I wonder how it could be it could be but I think uh the brain processes a massive amount of information only a small amount of

45:14 which ever enters or touches what we call experience and um why do we say that we have these experience es of some things and not others and I always come back I know I'm just repeating myself here but I always come back to that same proposition if you believe in neuroscience and basic science and logic which I accept our assumptions but I I decide to Tak in those assumptions if you accept that basic platform to build on

45:48 then the our our certainty that we have experiences derives from information in the brain and everything else flows from that there's got to be something in there I I've been very interested in the link in particular to social cognition right because we evolved to look at other people and interpret them as having Minds with experiences and thoughts and if you sucked out all the social cognition parts of the brain all

46:21 the theory of Mind Parts I wonder does that mean you have a person who just know longer even comprehends um mind for which the very concept of mind and experience is just like irrelevant gone nothing yeah you know what when you ask people tell me about experience your experience itself what part of the brain are you asking I mean you're asking the parts of the brain that know what that is that know what that concept is so you're asking theory of Mind circuitry in the brain and so it gives you an answer it gives you the answers that it

46:54 knows that it's evolve the process yeah so I mean I yeah but couldn't the I mean I wonder why you resist saying that the thing of which it's a model is consciousness and it's just that Consciousness is different because of we think of it as being different than it is because of the way it's modeled in the brain which would be a sort of straightforward physicalist interpretation of your view but in in many places you seem to resist that and say no it's it's yeah there's

47:27 something there but it's it's not Consciousness um well I don't know what words to put on it um so to give you an example a really concrete example uh you sit in a room you look at a chair and your brain builds a model of chair and you say there's a chair and we all know that the model is not perfectly accurate and that the real chair is different from the model and the thing that you say is there is not really quite the same as the thing that is there right um and we've all collectively decided to

48:00 call them all chairs like the real thing is a chair the thing being modeled in the brain is a chair and the thing we say is there is a chair and one could dice it and slice it philosophically and say no they're all different from each other which is true uh but we still call it all the same word and so if you wanted to Shir you could call it all Consciousness you could say yeah we have Consciousness in the sense that the brain is capable of focusing in and deeply processing some things over other things like the global workspace style pull all this stuff

48:34 together uh the most essential pieces of information into one Central processor that can then influence the rest of the brain the brain can do that that's real that's there well let's call that Consciousness then there's the thing the brain thinks it has because it's built a model of that process and the thing it thinks it has has all these bells and whistles and properties that we report having well let's call that Consciousness too so if you want to we can attach the word Consciousness to it um I'm just saying that the as far as I know there's pretty good evidence

49:08 actually that the I think at least when I look at the evidence that when we talk about our Consciousness the things we're conscious of and the things we're not conscious of um they map on really well to the things that the brain is paying attention to and the things the brain's not paying attention to right so it just kind of looks to me like the thing that's really there is attention and the thing that we say we have that is the model that's been built

49:44 of it that's the thing that we call Consciousness that's that's what I that's what I think is going okay I mean yeah it's interesting because it sounds like you you sort of have like an attention Theory Of Consciousness with like a attention schema theory of the meta Theory Of Consciousness for why we think Consciousness is a certain way when it isn't um uh but okay so I I I mean I still can't myself personally I can't really I mean I I see how if I'm going

50:17 to say or believe something it's got to be based on information in the brain and that information may just be what Consciousness is but I also of think that we should allow that there could be just Consciousness without this other thing attached to it or at least that that's should be possible in some sense um that it should be an open question whether the frog is having a a uh an experience in the full-fledged sense um not as complicated as mine but in a phenomenal sense uh regardless of its status or not but it sounds like it almost comes down as conceptually

50:50 like defined by your theory that the answer has to be no because of this like a TI logic kind of case which sort of seems to seem I don't know it's it's delivers the wrong seems like it should be an empirical result rather than like a logical result that the frog doesn't have experience or something well let's think of it this way let's say the frog has experience um but we want to understand let's say we want to understand how it is that people think they have experience and claim to have experience and talk about

51:23 it and introspect and say I have it and then they report on it okay um there's what's a good way to put this um go back 500 years before people knew anything about the brain and you can ask did people have serotonin or let's say you know um uh glutamate uh you know in the in in their brain neurotransmitter

51:55 um yes um they had it there it was um but people didn't say they had had it they didn't know they had it they didn't talk about having it um they didn't report having it so there is a disconnect between having it and that whole ability to know you have it think you have it believe you have it and talk about it right right so there's a kind in that instance there's a disconnect um so if we want to try to understand how people say have Consciousness believe they have Consciousness talk about argue about it

52:29 like we're doing what I'm saying is and I think you've said this makes sense that it derives from bundles of information in the brain that are models self models right once you've got that explanation you don't need the actual feel maybe it's there maybe it's not but there's no proposed mechanism for how that actual feeling specifically influences the neural circuitry such that the brain builds uh information

53:04 about it and then talks about it and so I would put it this way um there's a view of for example o of how objects behave for example you take a um a drinking glass and you drop it on a hard tile floor and it shatters uh there's kind of two classes of explanation one is uh the last got mad and was so emotional about it that it decided to break up all right so it's the um intentionality hypothesis of the behavior and the other one would be a more Newtonian you know the forces from

53:39 the strike exceeded the Integrity the um forces holding the glass together um now you could successively argue that those two are not incompatible just because you've explained that dropping the glass Newtonian forces shatter just because you've traced it all back to that doesn't mean the glass didn't actually feel angry maybe it felt angry okay and I would say that's true you have a point but that is not a necessary part of the explanation for any of the Behavior now

54:15 I prefer to use aam's razor and say forget about that you don't have to you can believe the glass had a feeling but I would say logically you can't believe that the glass is feeling caused it to break because the breaking is already fully explained and so I would say in the Consciousness debate maybe we have this conscious experience and feeling but it is not what's causing us to say we have it believe we have it or claim we have it that is information in the brain uh in the form of these models that are automatically built that are

54:50 self-descriptive and those models are there to describe real processes that need to be predicted so that the brain can successfully control itself okay and so uh go ahead believe that there's the conscious feeling that's great all I'm saying is it's not what's causing us to say we have it and think we have it and believe we have it which is a totally different process and that's related to the attention schema on your view that that other process that's right so this is the hypothesis we put forward is that in specific the The crucial model that

55:25 leads people to say they have Consciousness is modeling attention a kind of high level attention so I wonder if I if I uh switch topics sort of slightly but not hopefully too much do you have any thoughts on how the models in the brain get implemented because some people might um and again I'm not necessarily one of these people but I'm curious about what your response is they might think look we're helping ourselves to too much stuff here we're talking about models we're talking about information but what we really have are neurons so

56:00 aren't these things already of like of different class so to speak um that that that uh we need a story about how they get implemented in the brain so they were trying to eat a cake and have it as well yeah um I think that the answer to that will be um artificial networks Neal networks uh and that's why this is such an interesting topic to me and something that um my lab is working toward how can we build these things into artificial

56:33 systems what does it even mean to build uh models in neural network architectures um and do you think that our current AI systems have models like I know there's some debate about chat gbt and those type things where they even are generating models of of things or not I think yes these things have models they um you know when when you look at chat GPT I mean it's a big mystery box and uh somewhere in there there's you know um implicit distributed information in the form of uh connection

57:07 weights that amounts to predictive models of things in the world uh and it's learning it's learning these information structures and so I think chat GPT has certain kinds of models one of the things we've done is asked how good are its models of other people's minds in other words how good is its theory of mind uh and it turns out it's really good um uh chat gp4 has theory of mind that exceeds most people wow so yes it's have you done the

57:42 kind of experiments that you done with human subjects on chat gbg4 yes so we do have a um a paper that I've been uh collaborating with that um looks at standard tasks tests of social cognition applied to human populations and applied to chat GPT and a couple others of the of these um wow machines and um some of them do a little less well Than People some do terribly some do much better but chat GPT 4 has um

58:19 extraordinary uh ability and it I don't think it can do that unless somewhere be bued in there in an implicit form it has good predictive models of uh what a mind is how a mind contains information you know what does it mean for another person to think this and therefore be able to act this and that way so it's building models um we don't think I don't think that it has a model of itself in the same way that people do because it's not trained on that kind of thing so interesting so do you think that uh while these models are going to

58:54 result result in a kind of implicit or explicit even beliefs uh which might lead to these machines saying or thinking that they have Consciousness in a way that we do yeah I think it's entirely possible but I think you probably have to train it in especially right I do I am not I do not subscribe to this very old longstanding idea that if you just make it complicated enough or train it enough then eventually it'll spontaneously develop the Consciousness thing I think it's much more specific the Star Trek Voyager Theory Of Consciousness right

59:29 right exactly yeah got enough information it became conscious okay yeah exactly but couldn't happen spontaneously though I mean because in our case it did and for and per your argument that if it has to control something a model is useful for control then it seems like these sorts of things may sort of develop on their own if we're not careful yep I think um they may develop on their own I think you I mean well so we come back to the attention schema Theory and and I always try to point out it's a specific Theory within a larger framework and when I

1:00:04 talk about inevitability and you know logical certainty on my ends at least I'm talking about the larger framework the specific Theory says that it's um it's a focused highlevel um selective attention in particular that is being predictively modeled by the machine which leads us as machines to think that we have all kinds of properties like Consciousness okay so that's that's the framework um I think right now um artificial agents don't

1:00:40 have the same kind of attention that we have uh and um therefore they were probably never as they stand right now would not develop the same kind of self models that we have in in that respect uh we could build that stuff um I I think that it's probably easily buildable it's you know within 10 years we could have this stuff or sooner probably um it's just a matter of building it and I also think that because one of the main uses of

1:01:15 Consciousness for people and I think this is not fully appreciated one of the main uses of Consciousness is that we attribute it to each other and it's part of our social Matrix that we live in our our you know the social atmosphere the world that we we swim in an ocean of perceived Consciousness and it allows us to get along with each other and it's one of the tricks I think Evolution came up with to make us a pro-social species and so there may actually be a benefit to building some of the stuff into our

1:01:48 machines so that they're not just all big sociopaths yeah so so could is it really the attention schema rather than a certain sort of schema that that matters for your view because I'm wondering now as I hear you thinking I was thinking um couldn't chat GPT 4 have a a different kind of schema modeling something other than attention whatever it has but in a way that describes it in a shorthand which convinces it that it has this non-physical Essence which we

1:02:21 would then say I guess that's so it could be any I mean even you couldn't you have a body schema just a body schema that described the body as a non-physical thing and then you would think you a ghost or something so it seems like it's a schema Theory rather than attention like it's not really attention that matters as much as the way the schema describes whatever its modeling yes uh I I I hypothesize that the attention is crucial to the experience claim right to experience something is the brain's short hand of saying I'm

1:02:57 paying attention to it that's the theory but you're right you can have schemas of all kinds of other things in your brain um and the body scheme is a perfect example because the Phantom limb shows that the body schema uh you know you lose your arm your body schema says no you still have an arm it's a ghostly thing but it's there it's you know immaterial non-physical arm that's still there Al so you're right you can have schemas that tell us things that are physically incoherent um that are ghostly that are magical so there's

1:03:32 all kinds of possibilities in there uh and you're right uh artificial agents could develop their own effectively their own form of Consciousness I I'm not sure that it would be experience in the same sense that people uh claim to have experience mhm so what do you mean by experience the that the specific like phenomenal of it yes the yes but you know I think that's all buildable I think or trainable let's say

1:04:07 uh and I think that that'll get sort out people will build the stuff for better do you think it has any moral implications for how we treat these machines um I think we start getting into the real details of morality this way when you're really start building things right and then you get into these questions like what even is morality uh I mean there's wonderful theories that link it to emotions and um evolution of emotions uh and now you have these machines that have no emotions uh that

1:04:41 we know of because we don't even know how to we don't understand emotions in the human brain and we don't know how to build them into machines uh and so you start asking questions like okay maybe we make it conscious but it it has no emotional association with anything so it doesn't care whether it lives or dies is it okay to kill it um I it certainly becomes ethically complicated let's just say okay but and you might wonder whether it's ethically complicated even start working towards creating a something which we thought could be a

1:05:14 conscious uh artificial agent yes I think that's G to happen anyway you think can't people can't resist I think PE yeah I think there's that's technology for you okay so I want to just make sure that we connect what we were just talking about back to how it got brought up which was um how models get implemented in the brain and so what I take to be your answer is that for anyone who thinks we're helping ourselves to too much we can see that we aren't by looking at what the what these systems are doing by building something

1:05:47 like a brain getting these sorts of properties out of it strengthens our idea that the brain itself can have these sorts of things going on is that the general idea yeah that's right okay so I wonder then what you might say to someone who thought look what's really going on is well I mean you know you build the physical basis of it and then the non-physical thing happens um so that there's some sort of lawlike connection between the uh you know so I'm channeling someone like David chers or someone who's who's perfectly fine with all the information stuff 100 100% happy with it but also thinks there's

1:06:20 some law of physics or n not physics law of nature which connects information states to these abstract quality states that are not physical and then I guess what if I would sort of guess what your feeling would be it' be like what's the point of those things um they're not doing anything for us uh but do you is that the kind of critique you're going to yeah they're nonexplanatory parts of the theory I compared it once to you know the medieval humors Theory uh so you know I don't know if you read that one article on on fle

1:06:53 fleem yeah yeah that's you know people you know my website has the tagline the usual fleem and philosophy from which yeah it's not from your article but I'm a I love the humor Theory it's hilarious yeah there you go so yeah the theory was or part of the theory was people get sluggish because they have too much Flem and fleem fleem is a sluggish flowing fluid so therefore they must get sluggish and that's a really kind of analogical kind of thinking but you have to ask yourself what is the mechanism by which having too much of a particular kind of liquid makes you psychologically

1:07:26 more like that liquid uh and that specific connection is not apparent and so one has to ask the question having conscious feeling what's the connection between that and specifically neurons processing information such that you think and believe and claim to have conscious feeling what's the connection from one to the other right we spend so much time trying to explain how neurons produce the conscious feeling but nobody ever addresses what I call aob how does the conscious feely thing

1:07:59 turn around and affect the neurons in such a specific way that we can say that we have it right that remains totally unaddressed and you don't need it you don't need the extra feely thing to explain the full behavior of the system yeah and then I guess what they would say is you need the extra feely thing to explain like my introspective access to it and then you would say yeah but that's already like a simplified illusion sort of thing anyway we don't need to explain that yeah my theory explains why you meaning David shamers are so certain that my theory is wrong

1:08:34 right I mean it does sort of seem that you get to a point where there's just kind of like a Bedrock impass here and each of each side has a sort of diagnosis of the other side and why they don't accept you know their View and then there's nothing really you can do past that I mean you get to the point where you can say look you say that cuz you're stuck in a loop and then he's going to say that you look you're saying that because you're too committed to physicalism and they trying to explain way this the obvious thing and then it's like where do we go from there it's it's not really clear I mean for from the science point of view it doesn't matter because all the science continues as it

1:09:06 was anyway but like is there do you think that this whole debate ultimately ends up panning out in kind of a frustrated like harumph or is there's a way to progress past this yeah here's how I think it will progress uh when it becomes of everyday pragmatic importance to the average person in other words two things one when AI becomes so humanik that we automatically attribute Consciousness to it and people already do all over the world they can't help

1:09:40 attributing some degree of Consciousness to AI whether it has it or not uh um just like kids attribute Consciousness to their stuffed animals uh but there's there there will come a point I think when AI is at such a level that people people just say yeah okay it's conscious uh and then whatever you know went into the AI people will have to accept Okay that must be the Consciousness thing um this the second development I think ultimately will be mind uploading which is much farther in our future and a real science

1:10:13 fiction kind of proposition but when you get to the point when you can copy the data structures in a human brain and put them in an artificial system and then the person seemingly in this artificial world attests to still being conscious uh there comes a point when it's sort of like the flat earth round earth there comes a point when trade Roots got so important for everyone's everyday life that people threw up their hands and said okay fine it's round right yeah I mean you'd be surprised

1:10:46 Flat Earth is sort of making a comeback but uh that's true that is true but you know that's a different topic let's not go down that road but yeah so you think it's going to be a pragmatic thing that there's not going to be some like dramatic headline Consciousness illusion it's just going to be oh okay you know we can't help but treat that thing conscious people already are doing this to the roombas I mean God help us when these things have faces and are talking to us we don't stand a chance I feel like so of course we're going to assume it's conscious um and then you know mind

1:11:20 uple being fully I think you're you're right if that ever becomes you know my my secret last Refuge is that I wish that Consciousness were biological in some sense but it's hard to really maintain that view um but uh you know anyway so uploading if that's possible um that's one thing but even less dramatically just any partial sort of neural augmentation I mean I think it's going to be when all it takes is for Grandma to come home with an artificial hippocampus and cuz my grandfather had Alzheimer's and if he'd ever come home

1:11:52 from the hospital and suddenly looked at me like he knew me and said oh Richard I recognize your voice I would have been you would have made it would have floored me so I think that when that stuff starts happening that's going to be not I mean you don't need to go all the way into the computer simulation before you have real questions about is Grandma seeing my face does she really have visual experience and when she says oh starts crying I haven't seen your face after having an cortical implant so you know whether those things pan out I don't know but I like this idea that it's going to be sort of practic practical reasons that Inspire us yeah

1:12:25 yeah unfortunately because I would love it if there was some sudden headline that said the magical lists are all wrong it's an illusion I'd be like yeah I could New York Times say it but that that won't happen I don't think yeah you never know I mean it does sort of feel like at the end it's just these are metaphysical questions and not science questions so you know there's going to be one side that says look everything you say is compatible with me still thinking there's this extra stuff and you're going to say everything you say it's

1:12:57 compatible with me thinking it's not there um and then I don't know so it does it does seem a bit frustrating uh I wonder if like I know I can't keep you forever although I have a million things I want to Badger you about but I am thinking that um when you just really quickly when you mentioned the chat GPT thing there would be no evidence obviously or no way of this thing coming up with this like the beam theory of vision right because this depends on this this view that you have that

1:13:30 there's some evidence um that people have an intuitive idea that Consciousness is like a force emanating out of the mind um that's not something that chat gbt is going to come up with is it so uh right so just to give a little larger context uh for viewers um yeah one of the things we've done is looked at some of the weirder more irrational um Concepts or not Concepts perceptions people have of each other's

1:14:03 Consciousness and one of them is the beams coming out of the eye which we hypothesize is a actually a useful simplifying trick for keeping track of who is attending to what to model attention as though it were like an invisible fluid that flows out of the source to the thing that's being attended to uh and indeed there now we've collected a lot of evidence that shows that people do this um at an unconscious level uh and um the visual motion areas are crackling away representing attention as a flowing thing coming out of people's eyes right

1:14:37 so now you ask would chat GPT ever come up with that and here's the thing chat GPT has access to all human writing basically uh and and probably more at this point uh images and so on uh and the I beam are baked into a lot of our our writing they're everywhere culturally um so we talk about you know the light of love coming out of people's eyes and there's Oodles of stuff WR written about how you feel the pressure of someone stare on the back of your

1:15:10 neck right or he stared so hard at you that you had to take a step back or you know Superman yeah looks kill or Superman has um x-ray vision and and it burns holes and things which makes no sense because you know x-ray vision really should be the x-rays go into your eye not coming out as in a beam right so I think that chat GPT probably already has some ability to grasp this concept of eye beams whether you know H H if you asked it about that I'm presuming it

1:15:47 would be able to give you answers like yes people sometimes believe it uh so I think it knows about I think chat GPT knows about these uh fictitious ey beams but but would you think that it would be useful for it to employ that as part of its model of what other people are doing no because I think it's not interactive in the same way I think you got to build a machine that's in real time looking at a person and predicting where is that person paying attention to right now and how is that going to affect that

1:16:18 person's Behavior right right and and when you build a system that's inter active Moment by moment in real time in real space then it becomes useful for it to build those kinds of models yeah interesting okay well there's many things like I said I want to ask you but uh I got to be mindful of that you probably better things to do than talk about Consciousness with me all day n so abely nothing better okay good well then I want to know if like in the course of this discussion anything that you were expecting to come up or wanting to talk about hasn't come up yet so if there the

1:16:53 answer could be no but in case there was something that you thought might come up but hasn't I was wondering no no I don't think so I we covered a lot of ground I'm sure there's a lot more that could be covered but nothing specific pops into mind okay very cool well I want to thank you for joining me today and I want to also say that I really like your approach to trying to find a common framework um you know there's disagreements about how to do it from the scientific point of view trying to move past all all the infighting between the theorists and trying to provide

1:17:27 something that looks like a combination or u a building an understanding Community um which is something that's been lacking so I I really notice that that's a a theme of your work and whether you agree with how you put all the ingredients together or not you know that's a separate issue but the the idea that we should be working together on this I think is very important um and so I I really admire that in your work and I know it's not um Prime some resistance with that but I think we need people out there trying to say look how do these ideas all fit together enough harboring our own little safe harbors so uh keep

1:18:02 up that good work hopefully and um uh I would like to talk to you more about these experiments one day if they ever come out with the chat GPT so if you'd be willing have you back for that but other than that thanks again and hang around just for one second and I'll say buy off air but uh all righty well thank you very

← All episodes · Search all episodes · All guests

Latest from the blog

All posts [ -> ]