Consciousness Live! with Adrien Doerig (S4 E13)

0:00 Here, we're going live. And we are live. All right. Joining me this morning, a very, very special guest, Adrian Doerig. Adrian, thank you for joining me. I wonder if you could say a little bit about who you are and what you do for those who might not know you. All right. Well, thanks for having me. My name's Adrian Doerig. I'm Swiss. I studied life sciences and physics at the EPFL, which is the Swiss Institute of Technology, basically. And then I did a PhD in neuroscience

0:34 with Professor Michael Herzog, with whom I started working on consciousness. Sort of I wouldn't say as a side project, but sort of as a second project next to my more psychophysics modeling main line of work. Uh-huh. I got my thesis last year and since then I've been working with Tim Kietzmann in the Netherlands at the Donders Institute. And there I'm focusing a bit more on the modeling of our vision. And I hope I get to get there at some point because

1:06 I still haven't been because of the the COVID. So, that's it. Interesting. So, um what is this project that you're working on in the in the psychophysics? Is that you said you use as your main project? Well, it turned out that my consciousness stuff worked fairly well, so it ended up taking more space than I thought it would, but let's say that I was hired for being a modeler and trying to model psychophysical results initially. I see. Okay. And so, you're still doing

1:42 that though. Yeah. Yeah. Now I moved to Professor Kietzmann's lab and he's all about recurrent deep neural networks and how they process information and trying to gain some insights about uh, human visual processing through this uh, computational approach. So, yeah, that's what I'm doing now, mainly. And I Very interesting. I'd like to keep going with the consciousness stuff as well. Yeah, that's I mean it's it's a bit interesting. I mean I was going to ask about this a little bit later, I guess, but since maybe I'll ask about it now. I mean, given the what you're saying as

2:16 your sort of entry into the field, it sounds like what you were what what you were thinking was that as a kind of young beginning scholar, you want to have like a stable day job, so to speak, and then you like consciousness, so you'll work on that, but uh, you you weren't really thinking that that would pay the bills, so to speak, when you were when you were planning at the early stages. Is that sort of like the right way of characterizing what you're saying? Well, not exactly. It's a more the opposite, actually. I was doing physics, and then I read Roger Penrose's book, which got me fascinated with consciousness. Now, I don't think it's a

2:48 very fruitful approach anymore, but it was a very good book. So, I went to Michael Herzog and asked if I could work on consciousness, and he said yes, and I did my master's thesis with him on this, and then he said he said to me, "Well, maybe you need a day job." So, I'm going to hire you to do computational modeling. I see. Okay. Yeah, so that's like the big question that got me into neuroscience in the first place. Okay, interesting. So, was the Penrose book The Emperor's New Mind, or was it Yeah, okay. It is a good book, I agree. I'm not really don't think the approach is fruitful,

3:19 but it's definitely a great book. One of the ones that rekindled the interest. All right, that's interesting. Okay, so I want to come back to that, cuz I have some questions about what you think about the status of consciousness re sciences, or something like that. But I guess first I want to get to talk about your views about the science of consciousness. So, is you this recent paper that you I guess you co-authored, um, the hard criteria paper. I wonder if you could tell us a little bit about what the project is there, and then I want to ask you some stuff about it. Mhm. So,

3:54 it's an attempt to try and understand better how we can approach consciousness scientifically. When I read um Chalmers' book, I was flabbergasted and I started a lots of questioning about how we can study consciousness at all and I was discussing this with Michael Herzog and then we ended up trying to narrow down how we can be purely empirical about it. Mhm. Maybe because we I was

4:27 sort of in an impasse when it came to more theoretical stuff. I was I couldn't know how to decide between the more let's say Dennett approach or the more Chalmers approach. So, I was puzzled and we tried to get the empirical approach to consciousness straight and we tried to not worry too much about uh the philosophical underpinnings, the metaphysical concerns. So, we start off by saying um well, even though it's hard to define

5:00 consciousness from a metaphysical or I don't know how to say philosophical point of view, we have lots of phenomenology that we can use. For example, if you're in a coma or if you're awake, there's an obvious difference. Well, there is many obvious differences, but that's some phenomenology that we can use uh in many paradigms like rivalry, masking, and so on and so forth. Right. Um we can investigate changes in consciousness. We can have depend uh independent variables that we manipulate and we can see how

5:32 consciousness as a dependent variable changes. And this has been going on for a long time now. And it's puzzling. It's puzzling because we have lots of data, but also uh a huge number of theories. So, it's as though the data is not constraining the theories. So, we thought, well, we have lots of data and that's a good starting point, but we need to understand better how the data constrains theories and that's basically what we tried to put forward in this contribution. We tried to

6:07 understand what kind of phenomenology is is useful for the science of consciousness and we tried to lay out the set of criteria Yeah. to approach the problem. Can I Can I ask you just really quickly be in what you what the way you were setting up the problem, you said there's a contrast between the Dennett and the Chalmers approach and clearly there is, but then you also said that you want to set aside the metaphysical issues. So, I would have thought that the primary one of the primary contrasts between the Dennett

6:39 and Chalmers approaches was metaphysical. So, when you if you set that aside, what what do you mean to be contrasting in the approaches then? Well, yeah, so the the point is to not settle on that question before doing science. Ah, I see. So, that's yeah. So, I was so puzzled with these big questions that we tried to sort of bring it down to a more like mundane level and say maybe

7:14 we'll understand this better later on and we just like tried to deal with the data we have at the moment and how can we do this? Yeah, I see. Okay. In that In that sense to me it's a bit interesting. I mean, the whole thing's interesting obviously, but one thing that's that's striking me right now is that it's like almost like a uh I don't know how to put it. It's like the same thing happening over again because if you look back like historically when Crick and Koch first come up, they say, "Look, you know, these philosophical problems, we're

7:47 going to get nowhere. We got to just we we'll talk about just the neural correlates of consciousness and then we'll just bypass all that metaphysical stuff." And then people made some progress. And now it looks like you're coming back and you're saying, "Well, now here we are what like, you know, what uh um 40 years later or whatever or 30 years later and we're we need to set it still they're they're coming back. These big metaphysical problems are coming back. We need to set them aside still and make progress on how the data relates to the theory. Yeah, there's this it's starting to be well known now that in consciousness studies we sort of go

8:19 in circles. Uh Matthias Michel has written about this. And I like to think that we go in circles but sort of there's a third dimension in the circles. We're spiraling upwards, right? So, in a sense you're you're of course correct. It's uh the same kind of approach. I think that the approach they uh promulgated has been quite helpful actually and has led to a lot of interesting experiments. And I think that today we have a lot of experiments but we still don't know really well, how they constrain theories and a

8:53 lot of my work has been on trying to figure this out. You know, I'm still not convinced 100% about everything I've written but all of the papers try to say, "Well, here is some data. Can we use it to sort of disentangle different theories from each other and maybe show that some approaches are less fruitful than others." Yeah, no, I think the project overall is extremely important. Especially it's frustrating because that a lot of these theories will appeal to the same data but for opposite points. Like split-brain

9:25 patients or whatever, you know, it's like uh contrary theories will invoke split-brain data as evidence for their theories like um well, something bad is happening there if it can't be evidence for both. So, I there is this like it's not clear even what would count as evidence for particular theories. So, I think the whole project's one that's direly needed. Mhm. Um so, but nonetheless, there's some metaphysical questions in there. For ex- for example, you know, the difference between a theory like IIT and global workspace, that there's metaphysical differences uh

9:57 between those theories. One's a structural theory, one's a computational theory. Um so, you're still wanting to be ultimately dealing with those kinds of questions, but I guess you're I'll Yeah. focus on what empirically would separate those sorts of things from each other. Yeah. Exactly. We'd like to say We we actually I we think that it's the two approaches are complementary, you know, maybe if from a more metaphysical perspective, you can gain some kinds of insights. It's just that our business is more on the side of saying, "Well, uh when it comes to addressing empirical

10:30 data about consciousness, what are the strengths and weaknesses of, for example, IIT versus global workspace?" Right. Exactly. I I wouldn't discard the whole metaphysical discussion as fruitless. It's just not the business I'm in. Right. Exactly. But also, I mean, but but you would agree, I think, that by having empirical evidence that could possibly adjudicate between, like for example, IIT and global workspace theory, we also would thereby have some evidence that would help us decide between metaphysical Mhm. Yeah, yeah. schemes. Um in in one sense anyway. Although, it's

11:03 with IIT, it's it's tricky because it can be interpreted in different ways metaphysically itself. So, anyway, maybe it won't work there. Um but uh so, what do you think some of these empirical criteria are? Uh can you can you tell us like how can we differentiate some of these theories? Yeah. So, the we have a list of four criteria. The first one is that we need to have uh both conscious and unconscious alternatives when we have a phenomenon we want to study, and we call a phenomenon that has two of these um

11:37 paradigm case, and that's a term borrowed from uh Scott Aaronson. So, the idea is that um if I show you five fingers and then I remove two fingers, then the content of your consciousness has changed, but the consciousness itself has not really been manipulated, we would say, because there is no unconscious alternative. On the other hand, if I show you a masking stimulus where 50% of the time you can see the target and 50% of the time you

12:10 cannot see, and then we contrast these two, since you have a conscious and an unconscious alternative, then we're actually targeting consciousness. And we phrase each of our criteria as sort of a challenge, because we're not saying like this if you don't satisfy this, your theory is garbage, but we say for if you have a theory that doesn't really have an unconscious alternative, for example, you say um learning is important for consciousness, something like this, Right. well, the challenge in that case, if you want to address this kind of empirical

12:43 data, is that you need to formalize, you need to explain uh in which cases you should observe the conscious and in which cases you should observe the unconscious alternative. And I think, to be honest, that's a widely agreed-upon criterion. So, that's the the first one, but I think that a lot a lot of talk about consciousness, in fact, sort of would like to accept this criterion, but in reality doesn't really. Ah, okay. So, give us an example of that. So, when you

13:19 say, for example, um I will use the rubber hand illusion to study consciousness. Yeah. Well, sure you feel like the rubber hand illusion is I'll just explain what it is. You put your real hand on the table and the experimenter puts a rubber hand right next to it. And then we hide your real hand and we stroke your hand and the rubber hand at the same time and you sort of feel like the rubber hand becomes your own hand. And this has been studied quite often to you know as a as a

13:54 paradigm that tells us something about consciousness. No. And here there is no unconscious alternative. So sure it tells you something about body representations and things like that, but I don't think this particular line of work tells us anything about consciousness. Our illusions are the same thing. If you have a subjective illusion, for example, uh you see something uh you see black dots on a Hermann Herring grid. Uh-huh. It's an illusion where you see black dots that are not really there. Well, these black dots can perfectly well

14:26 arise from sort of unconscious processing that happens in your retina, for example. So Right. you know, naively you would say, well, that's part of conscious studies to study this, but we would say, well, as long as you don't have an unconscious uh alternative where you can dissociate which parts of the processing are about consciousness or not, then you have to convince us that it is really about consciousness. Yeah, I mean, I think this is a really an important I mean, all the criteria I think you bring up are interesting and important, but I think this one's especially important cuz a lot of people

14:59 just conflate like you say, well, yeah, this is I I think a binocular rivalry is also an example of this kind of thing where people say, well, look, you know, like no report paradigms, you can do this, but that begs the question because you haven't really shown the contrast between that and the unconscious case. And there's evidence that you can do these unconsciously actually get these kind of binocular rivalry. Mhm. So on the other hand though, so I think on the one hand what you're saying is very important, but on the other hand is it a bit too strong to say that anybody who wants to work on consciousness or talk

15:32 about consciousness has to have this kind of contrasting unconscious case. Um or or is it is the claim only that if you want to say something's foundational about consciousness, like what it is or something, then you have to have these contrasted cases? Mhm. I think I would go so far as to say that if you want to claim this experiment targets the difference between consciousness and unconsciousness, you need this contrast. Uh-huh. Then depending on how you on what you mean by

16:06 consciousness, if for example, you mean, I don't know, moral intuitions or something that is sometimes associated with consciousness, but it's not really the same thing, then maybe you don't need that. Or free will, you know, these things like sometimes come up in consciousness conferences, and there we're talking about something different than I think what we're talking about now. And so in this case, maybe you don't need that sort of contrast. Well, I wasn't I I hear you right about that. I I agree, but I it wasn't me who was thinking about free will. I was thinking

16:38 more of like the rubber hand illusion case. Mhm. So, can't are they could couldn't you be using a paradigm like that to to say something about consciousness um even if you didn't have the contrasting case? Or do you think that um it any experiment that is involved with like saying anything about consciousness should have a contrasting just as like a methodological issue, if it if it's involving consciousness at all, does it need this contrasting case? Or only in certain only if your goal is

17:10 experimentally a certain thing, do you need the contrasting case? Can you tell me what you have in mind? Like what the rubber hand illusion can tell you about consciousness without a contrast. What do you have in mind? Um Uh well, it gives us a sense that um the the awareness that we have of bodily ownership is malleable. Mhm. And that um you know, it's somewhat surprising to think that you can mistake a something that isn't part of your body as part of your body. And so, it tells

17:44 us something about the way we experience um the the body even if it's not um telling us something fundamental about the nature of consciousness, it does give us some insight into sort of what it's like to have a body and how it can be altered and um so that was sort of sort of the contours or shape of the phenomenology of experience or something like that can be No, yeah. I think that but that's fine. It's just that, you know, as you say, it's not really telling us something fundamental about about consciousness. I think it's

18:17 telling us maybe what it can be like under different circumstances to have a body. I mean cuz cuz of what that was so the contrast that I had in mind with what I was just saying was uh would be along the lines of um someone who might say, "No, no, even in that case, uh you might be getting the same thing unconsciously. So, maybe you could have, you know, an unconscious uh rubber hand illusion." And so, just saying, "Ah, rubber hand illusion, therefore you're telling me something about consciousness." So, it is the second So, it does sound like it's a

18:48 more general thing that you think that even even if I So, I just do an experiment with the rubber hand illusion and I publish my data and I say, "Aha, so something about consciousness the blah blah blah of experience." And then you say, "Nope, you didn't have a contrasting case, so you can't even say anything about whether this was about consciousness or not." Is that the kind of point Well, to me it's kind of the same as saying, "Here's an experiment about consciousness. Now, your conscious percept is five fingers and now your conscious percept is three fingers. Right. If you don't show that you can have an unconscious um

19:23 uh rubber hand illusion, basically, then the for all I know the the difference between when you feel your real hand as yours and when you feel the rubber hand as yours is a distal difference that happens all before consciousness has anything relevant to say. Right. I think this is the main problem is that you maybe there is a way to show that it is not a distal uh process that happens before anything related to consciousness. In that case, fine, but if you can't show it, then I think you're not warranted to say that you studied

19:55 consciousness per se. I see. Then we might be inclined to say that body ownership is sort of consciousness-y, but it's all not very well defined, in my opinion. Yeah, exactly. I see. Okay. So, does this um I mean, I have two two more things I want to ask you about this one uh if I may. So, the the first is does this imply a kind of strong thesis that every mental state can be unconscious? That or, you know, there are no no states which are just by themselves the kind by the state they are conscious.

20:28 Um because if you're if you're someone like Ned Block and you think there's like phenomenological overflow, I'm not saying I agree with them, I'm just saying if you're a theorist of that bent, then you're going to think that some some states when they occur, they are just intrinsically conscious by their nature. There there's something that's like to have them. Um and so you're not going to get an unconscious contrast with a state like that. Uh so is this then I don't think uh you're going to use this as an objection against those kinds of views? Uh if not necessarily, I don't think there's an entailment there. It doesn't

21:01 entail though that you're going to have a hard time studying these um So, if you want to say "Look, I experimentally manipulated consciousness, and here is the outcome of my experiment, and the thing you're observing can only occur consciously." You're going to have a hard time like targeting it experimentally, but it doesn't mean that it can't happen. Yeah. So, it just means more difficult. Cuz I I think I think Block's whole like approach here is like a one of the

21:34 abduction, the mesh argument, that there's no like experimentally zoning in on these. We we sort of have to sort of look at a whole bunch of data and see how the different levels fit together. So, how does the psychological psychophysical data fit with the neurological data? And then we try to see how they mesh together. In other words, that give us an argument to best explanation. Yeah, so that's an approach that could work. Let's say you have a bunch of contrasts on different levels of experimentation, and you sort of mesh them, and you find out

22:07 that the conclusion is that Block is correct. Then that's fine. Okay. Uh or you find out, just for the record, that he's not correct. Yeah, that could that could also Let's see about that. Let's just put a plug in for that. Uh some people might think that comes out the other way, but still get a good match. Okay, so that's interesting. Um I mean, that that's if if what you're saying is right, then there's a lot of work which alleges to be about consciousness, which really isn't.

22:40 Yeah, I think I would be prepared to bite that bullet. Yeah. I mean, that that may Mhm. Say again? Uh I It's It's not that it has nothing to do whatsoever about consciousness. You know, if I study vision in general, of course it'll tell me something about how the conscious precepts are formed, and what it looks like, what it feels like to have vision and things like that. But it's not telling us about really consciousness itself, like per se. It's only telling us about about the content about content, which

23:14 may or may not be conscious. Yeah. Like I agree. Um so the final thing here is I wonder what you're going to say about uh you know, the kind of critique which has arisen recently of unconscious perception generally, you know, coming out of things like people like Megan uh Peterson uh Lau. Yeah. The problem of the criterion and all this kind of kerfuffle that's gone up around unconscious perception. So uh so how do you think that affects the kind of uh argument you're making here? Yeah. So to summarize this concern, uh I think the first

23:48 paper I know of that addressed it is uh Peterson and Lau, who showed that if you really control your experiment very well, it's very hard to find unconscious process evidence of completely unconscious processing. Right. So that's quite a big concern. I have a few points in response. The first is that if it's impossible to study unconscious processing, I don't think we're in a very good position as

24:20 researchers. I think I would still stand by the argument that you need some sort of contrast. So if it's very hard to find, then it's just going to be harder to do science, but Uh-huh. So that's the first thing. And the second thing is that um it's it's well possible that we need to do some conceptual work and some experiment a couple of experimental breakthroughs to figure out how to isolate conscious from unconscious

24:56 trials, but I think I think it's possible and uh I know that some people are working on trying to show this at the moment and I I yeah, my hunch is that although this is a concern, it's a it's a fixable one. Do you think it's possible? I mean, now this may be something that you don't want to pronounce on, so if if so, just say so, but I I'm listening to this, I'm just correct me. Uh is it possible like in principle do you think to have strongly unconscious

25:30 stimuli in the sense that the a lot of what we do to make the conscious the the stimuli conscious weaken the signal, um you know, Is it possible to have like a really like a strong signal, but that's unconscious experimentally? Do you think that's something that we could get or do you think that's we'll always have to depend on these kind of techniques? No, I think it can work. It's just Mhm. Yeah. that seems to be the Lau Peters assumption, right? The Peters and Lau assumption is that there's you'd have d' prime equal zero and then you would

26:03 still have Yeah, yeah, yeah. Well, one line of research that I kind of like has come from Hakwan Lau where he's shown not this exactly, but he's shown that if you make the task as difficult in the conscious and unconscious cases, you can still find a contrast there. Because the problem the issue is that if you make the unconscious task very difficult and the conscious one very easy, then you're not you're comparing apples and oranges, but he showed that you can make them both as difficult as one another and you can still get some signals and difference between the two.

26:36 So, I think that's encouraging and maybe we can talk about that later on, but we conducted some experiments using vernier stimuli where in certain cases, you respond correctly 75% of the time because that's how we calibrate it. And then if you change the stimulus 400 milliseconds later, you change something that comes much much later, the the first element becomes completely impossible to report. Uh-huh. So, I

27:09 would say that in this kind of case and everything is counterbalanced so that there is no bias in the experiment that I see. And in these cases, you have like a change in performance from 75% correct to 50% and I think that's a pretty strong difference. Yeah. So, this is a retro effect that that things presented after the stimulus by how far? So, what happens is that um we show it's called the vernier stimulus and it's two two vertical bars with an

27:42 offset. So, that would be a vernier. And the task of the subjects is to focus on this vernier and say if the bottom bar is to the right or the left. And now I'm a bit puzzled by mirror effect in the camera, but for me the correct response would be right here. I don't know for you what it is. For me it's left. Okay, so left. So, you would say left like this and right like this. Yeah. And so, you present a vernier and you present it when you present it alone, it's super easy to see. And then in a par- in a paradigm that's called metacontrast masking, what you can do is after the vernier, you insert

28:15 straight lines like this that show up one after the other. Okay, so you see diverging streams. And if you just show this, interestingly, the you see the vernier as moving like this. You see the offset as being transported even though it's represented only it's presented only in the central stimulus. Right, interesting. And this shows you that in this case, uh the vernier is well represented in the brain and there's no like no problem to detect it. But the interesting thing is if

28:47 after 400 milliseconds of presenting these vertical bars, you present the vernier with the the opposite offset, Uh-huh. then the two verniers integrate, and you can only you cannot report any of the offsets anymore. Neither the second one nor the first one. What you see is just a straight moving bar. Interesting. And if the second vernier is the same offset direction as the first one, they sort of add up, and it becomes even easier. And everything is counterbalanced so that you cannot predict the orientation of one

29:20 based on the other. And so, what this shows is that the first element is well represented. Mhm. It's represented so well, in fact, in your brain that you can report the correct orientation 75% of the time. Right. And changing something 400 milliseconds later makes it completely impossible to report. Yeah, interesting. So, how how do we I mean, how do you interpret that result as So, you're I noticed you said you can't report it. Um is there still an experience of the

29:53 offsets or um a representation of it that's not accessible, or do do we have any evidence about that? Yeah. Yeah, so we wrote a paper about that with Matthias Michel, where we argued that this is a big big big problem for local theories of consciousness that basically argue that when a stimulus is represented adequately in the sensory cortices, in the visual cortex, for instance, then that's what gives you conscious access. Uh-huh. And the main local theory is the local recurrence

30:26 theory by Victor Lamme, and it says, basically, that when you have reentrant processing in the sensory and visual cortex, then that's when consciousness kicks in. Mhm. reentrant processing takes about 100 milliseconds to kick in. So, this experience experiment, I mean, the one I just described shows that in fact, you can represent something for 400 milliseconds, but not perceive it. Right. That's a big problem. So, now we have two

30:59 uh different comebacks that localists could uh use. The first one would be to say, "Well, uh in fact, the stimulus is not well represented." Uh-huh. But that would be puzzling because Yeah, that would be very it it would mean that the brain has to predict 400 milliseconds in advance what the second element will look like in order to choose how to process the first element. Right, exactly. So, that's unlikely. The second the second the second argument would be what you

31:32 just said. Well, maybe it's consciously perceived, but not accessed. Right. Uh yeah, exactly. So, like in the case of what we call phenomenal overflow. And so, there's two kinds of two flavors of this overflow hypothesis. There's a weak one and the strong one. And I have no problem with the weak one. The weak one is to say, "Well, now I'm experiencing a scene, and if you ask me just to to report the

32:04 color of the wall in front of me, I will say white, but of course, I have a lot more stuff in my conscious percept that I'm just not, you know, paying attention to it or trying to report it." You mean like the shadows on the wall and those sorts of things? the color of this jug, or if there is a cup in the scene, or the orient- the relative orientation of uh I don't know, my window with, you know, there's a lot of detail that I'm not accessing and that I'm just I could report it if I was focusing

32:37 my attention on it and trying to report it, but I'm not. Right. And I think uh So, that's the weak sense of overflow. That's the weak sense of overflow, and it can buy you a lot of stuff, you know. For example, in this Sperling uh paradigm, I think you don't need the strong overflow to save Block's theory. You can use the really quickly tell us what the strong overflow is on your way of So, the strong overflow is saying that even when you focus all your attentional resources on a feature, Mhm. you cannot report that feature.

33:13 And so, I'll just read um a a quote by Victor Lamme that we have in our paper with Mattias on this to show you that have it up. Yeah. Well, I was reading I was reading the paper to remember what we said. Okay, good. So, he says, "It is safe to assume invisibility and masking because there is no conceivable reason that could prevent the subject from reporting the visual percept had the subject had one.

33:46 The subject is sitting there, focusing his full attention on the target location, ready to push the button as soon as the target comes up. The not seeing can therefore not be attributed to the absence of attention, to the lapse of memory, or to any other cognitive function sitting between potentially conscious sensation and the report." Okay, so the idea is that in this case where we tell the subjects that there is going to be one element like this in the center, and they have to just report it, and they focus all of their attention on it, and you still say

34:21 that they perceive it, but they can't report it. Basically, you're stuck in a weird position where it's impossible to study the precept. Right. So, for this reason, I think that this paradigm that I just mentioned tells us goes a bit further than just saying, "Well, maybe it's not accessed." It tells you if it's conscious but not accessed, then it has to be the strong overflow version of of the phenomenon, and this is problematic because then

34:56 basically all bets are off, you know? Lamme, for example, relies a lot on masking studies, and he says, "Well, in this case you see it, in this case you don't." And I can say, "Well, okay, but I think you In the case where you don't see it, it's strongly overflowing." Right. It becomes very problematic. Yeah, exactly. No, I see the I see the kind of argumentative strategy here. Um So, so the issue is is more uh about um Well, yeah, it can can't be empirically

35:30 tested given So, why I I I guess what I'm wondering is Well, I have one or two things. One, I thought that the argument was aimed at a different target, so I want to get to that in a second. But the second thing is I wonder is why can't someone like Block respond to this kind of line of argument and say, "But look, that kind of misses the point of the mesh argument. The mesh says um you know, that's uh there's no way to experimentally get at these things, so what you do is you argue it to the best explanation mesh, blah blah blah, homunculus, all that stuff to get at it." Uh but um

36:04 Oh, Mattias is over here also chiming in. He says, "It's also important to mention that they don't ask whether they see the veneer or not, so they uh ask to discriminate the offset of the veneer, which is an objective task." I see. Okay. Um But uh thanks for that chiming in over there. Uh but I wonder But anyway, so why can't Ned respond in this way by with the mesh argument thing? Mhm. So, you say, "Look, you can't experimentally test this." And he goes, "Yeah, I know. That's the whole point of the mesh argument is like you can't, but there's another We have this other

36:36 mythology called argument to the best explanation mesh." So, is the Is your argument aimed even at a target that he would or do not care about like his Yes, anyway, I wonder what you think about that. Yeah, so it's been a while, to be honest, since I considered the mesh argument very seriously. I know that Matthias, who's here, has a paper on this, I think. Where he argues that it doesn't work. And so the mesh argument is To me, it's very

37:15 hard to defeat because you can sort of conjure it up whenever Yeah, right. you have you have a problem, you know. That's That's the benefit of a mesh. You get stuck in it. It's an Yeah, um No, I know what you mean, though. It's It's It can be wielded in multiple ways at any point. Yeah, I think if I think you would Okay, I think the mesh argument would work if you have like very strong evidence that you can trust it. Uh-huh. And not trust alternative

37:50 hypotheses, right? So, let's say you do the mesh thing and it sort of clearly rules out other theories and you're left with just this, then the inference to the best explanation works because that's the best explanation. But, I don't think that the mesh mesh argument really supports strong overflowing contents because when you look at the rest of the empirical data and you do the mesh thing, you don't end up with this as the best explanation. Maybe it's a preferred explanation, but that's not the case. Right. So, I mean, I agree with

38:24 everything you were just saying right there, but that's why I was thinking thinking that this this uh line of argument that you were just pushing uh was a bit maybe too strong because not too strong, but just maybe aimed at a different target or um because uh yeah, he's I think the real debate is going to be over what's the best mesh. Yeah. and not as opposed to your preferred one as as you were just saying. And uh as opposed to the thing that you were talking about before about well, here's something that you can't experimentally

38:56 confirm or deny um because that's what the point of the mesh is designed to do. And then the debate moves over there and goes, "Well, what's the best way that these things hook up together?" Yeah, one one thing I would like to mention though is that that's fine. If that's all you say, that's fine. But then, you can't you're not allowed to say, "And by the way, the masking supports my theory." Right. Because I can make the same argument for the masking. Mateus just made the same point over in the chat, by the way. Oh. Well, good thinking, Mateus. Yeah, you you are the you guys are

39:29 uh on the same wavelength. That's collaborative. That's good. We're on the same page, on the same YouTube page. Yeah, exactly. Um okay, so I I got I want to ask about I This does all weave into everything is sort of related to the first criteria, which as we can see is extremely worthy of attention and and weighty and and all that stuff. Anyway, so I want to come we move to second one, which if I remember right is the unfolding argument. Is that the second one? Uh if I recall, yeah. I think so. Okay. Uh Before we move on, I would just like to point out that I think this kind of

40:02 conversation is the reason why I think there's a lot of interesting work to be done in the sort of domain that interests me. Thinking about how empiric empirical data constrains theories because I think that all we just said is far from trivial and really needs some careful work, you know, to understand how this experiment with the various constraints theories. You need to take a lot of things into considerations. And you maybe some of the criteria we're putting forward are wrong, but I think at least the approach of trying to

40:35 disentangle these things is important. And I think that was a good illustration of it. Yeah, I couldn't agree more actually. No, I'm glad you I'm glad you put it that way. I couldn't agree more. Um Okay, so yeah, I mean since we could I could spend the whole time talking to just about this, but I did want to talk about the unfolding stuff. Um because I I really have mixed feelings about the unfolding stuff. I want to like it and I think I do, but then sometimes I don't, but then sometimes I sometimes I think it's very profound and like earth-shattering and then other times I'm like, "No, it can't be." And then so

41:08 yeah, anyway, um I mean, I who cares what I think? Anyway, that's just sort of my feelings about it. It's like I feel I sort of I'm ambivalent about about the argument. So, I wonder if you could tell us a little bit about how how you arrived at it and how it came about and then why why you think it does and doesn't do. Mhm. Well, the way I I reached the idea was that I was thinking about this big consciousness paper that we are talking about the one the hard criteria. And I was trying to think about how empirical data constraint

41:40 theories. And I was working at my university and my girlfriend was in the cafeteria working with me. And all of a sudden I had this idea about the unfolding argument. I just didn't say anything. She was packing her stuff and we're going to leave. And I left back to my office and I wrote it down for 30 minutes. And then when I came back, she was like, "Then?" It was like It was like Newton and the apple where you know, it just hits you at some point. So, that's how I came up with it. But so, the argument is as follows. There's different approaches you can

42:14 take for having a theory of consciousness. And one of them is to say that what matters for consciousness is the way parts of a system interact. And I mean, a priori, I thought that was sort of a promising approach because one of the naive ideas you can have to explain consciousness is just saying, "Well, the brain is very complicatedly wired and so it just creates consciousness for this reason." That's what I used to think, you know, when I was a teenager or things like

42:47 this. It's a very intuitive. I'm not trying to ridicule it. I'm just saying it's quite intuitive. Yeah. And so, that's what we call a causal structure theory. It's a theory that says it doesn't matter what the system does, what matters is how the system does it, how the parts of the system causally affect each other. And information integration theory is the most famous of all these causal structure theories. Right. The unfolding argument is an argument proposing that if you take seriously the kind of empirical data that we've been

43:20 talking about, then um causal structure theories such as IIT are out of the window. That's what the argument tries to show. And the way it shows it is quite simple. It's simply to say, "Well, all of the experiments that we were talking about up to now can be seen as some sort of input-output function." Right. Right? I show you a stimulus, let's say, that's the input, and then I observe your response, and that's the output. But we know, and that's not like a thought experiment or anything fuzzy

43:52 like that. It's a logical truth. It's a mathematical theorem. We know that the causal structure doesn't matter for uh the input-output function, right? So, you can have the same input-output function in many, many different ways. And in particular, you can have the same input-output function in a recurrent or a feedforward network. Okay? So, now you have a dissociation between what the causal structure theories or IIT. We have a dissociation between what these theories think consciousness is

44:25 and all of the data that you can ever gather about consciousness. So, you have Yeah, and that's a big problem. Yeah. It's a big problem. And what makes the argument a bit difficult to to make is that proponents of IIT, for example, can still say, "Yeah, but nevertheless, only the one that has the right causal structure is conscious." And in fact, uh I they were saying this before we wrote the paper. They already were aware of that you could have a feedforward

44:58 network that does all the same things as a recurrent one, but only one of the two is conscious. Uh. And so, the argument now is to say, "Well, here are all the experiments in the world you can do about consciousness if you had a billion years, and here are two systems that respond identically to all of these experiments, but you're telling me that one is more conscious than a human and the other one is not conscious at all, for example. And you can never know. So, either you say therefore the theory is wrong because none of the observe observations

45:33 you make can tell the difference between the two systems, or the theory is unfalsifiable." That was the more difficult uh part of the argument was to say, "If IIT says only what the one with the right causal structure is conscious is conscious, then it's circular, basically, because you have no independent criterion to show this. Right? You can I can say, "Look, they behave identically in all cases." And they can say, "Yeah, but this one has the right causal structure, so it's

46:05 conscious." And I say, "Well, that's circular because you haven't showed me this. You're just assuming it." Right. Exactly. Yeah, so I I mean, it's It's a very interesting argument. The story about how you came to it is funny, too, by the way. Um I I hopefully you got a nice girlfriend if she forgave you for that. Let's just uh give a shout-out for that. But, um it's so Here's a flat-footed response, and you know, I don't think it's

46:38 a great response, but it's like a flat-footed one. I just want to get your reaction to it before we go on. So, one one thing you might say is, "Look, you know, it's a little bit like saying you want to study how birds fly, and then you you come up with this big theory about birds flying, and then someone else comes along and says, 'Oh, hey, you know, jerk off over here, we got planes. They don't flap their wings, so you're wrong about how birds fly.' Now, obviously, something's wrong there. Now, if you're studying flight in general, then yeah, there it's a mistake. But, if you're studying how

47:09 birds fly, then, you know, knowing about airplanes is sort of irrelevant. Maybe for understanding general aero- principles of aerodynamics is important, but understanding what birds do requires understanding the mechanisms and their causal structures and how they causal powers the bird has. Um so, why aren't Why isn't that sort of like what you're saying? Like, what the bird flying We want to study the bird flying. That means, you know, how we're conscious creatures like us are conscious. So, uh we you know, we want a theory about that that works for us. Um and to show that there's these other

47:44 kinds of things which could do similar things, like the flying is similar in both cases, but I'm not sure why that's supposed to be an objection to like an a theory of how the bird does it. And if, you know, that maybe with the way that you know, I'm not a big fan of IIT, so don't get me wrong. I'm I'm playing devil's advocate here just to get your response. Um I mean, I'm not the biggest enemy of IIT either. I just I I I am somewhat persuaded by these objections, but I am just curious about like why isn't that what you're doing is is unfairly saying. So, is is your question why should we trust the reports of the unfolded

48:18 system? Yeah. That's the question. Yeah, exactly. Yeah. So, well, there's there's a couple of things to replay here. The first thing is that maybe that's a bit too theoretical, but I'll I'll still say it anyway. The first thing is not that I'm waiting to see the unfolded system and say, "Aha, look, now you're falsified." You know, I'm just saying this is an illustration of a fundamental contradiction in the theory.

48:52 Mhm. It's an illustration that the theory is not able to link the things that a system reports to its consciousness. They're like the two different things. So, it's not a falsification. It's not that if I said, "Oh, here's my theory of how birds fly." and then I show you an airplane and I say, "Aha, you didn't expect that one, did you? Uh here's a different way of flying that falsify You know, it's not like this." It's saying, "Here's an argument showing that there is some sort of inconsistency inside the

49:25 theory." But maybe that's a bit too far-fetched and not very convincing. Now, there's I think a few other ways we can make the point uh stand. The first is that when we explain the argument to make it sort of straightforward and easy to understand, we say, "Here is one system with this causal structure, and here is another system with a completely different brain, you know, like just it does it's completely different. And then you say, "Wow, so much so many differences between these two systems. I

50:00 don't really want to trust the the unfolded one." Yeah. But of course, the argument can be made on a scale as small as we want. For example, in the paper we say, "Well, why don't we replace just a little part of the visual system, let's say with a feedforward system, right? We replace the uh and making a caricature here, but let's say we replace the color part of the brain with a little implant that has the same input-output structure. And uh now none of the reports changed, so the subject says, "Well, I still see

50:35 red and it still reminds me of of sunsets and so on." Uh-huh. Right? And so now it's a smaller thing, and this can go as small as you want. Okay? You can even say, "Well, there's presumably many many ways of changing a few connections in the brain and that changes what the IIT would claim to be the the the percept." Yeah. Right? And we don't even need to do it ourselves. I mean, if you fall on your head, your causal structure changes. If you're hydr- if you're a hydranencephalic

51:10 child, you have a if you have a stroke, Yeah. these things can change the causal structure quite drastically. And so we don't want to say that we don't trust the stroke patients when they talk about consciousness, I think. And Yeah. I mean, so it sounds a little bit like I mean, maybe I'm getting this wrong here, but or misunderstanding what you're saying, but it sounds a little bit similar to me to the like dancing fading qualia argument from Chalmers, which I don't know if you're familiar with that that part of his work or not, but but basically the idea is that he

51:42 sets up this kind of continuity uh uh between the biological system on the and a digital system on the other hand and they're replacing neurons with with um silicon chips. But it sounds like what you're doing is a version of that, but instead of replacing biology with silicon chips, we're replacing one causal structure with another kind of causal structure. Um so we imagine like replacing the visual cortex with an unfolded visual cortex. Yeah. And all the input outputs are the same, so you still have to go around saying things like I this tastes delicious and that looks wonderful. But

52:14 according to IIT, consciousness must be missing in in those cases. Mhm. And so it sounds like what you're saying is that that's so implausible that we ought to expect that consciousness stays the same under this transformation, which is sounds similar. Okay. Not exactly, but it's close to this. It's close to this. Okay. It's saying if you want to argue that you shouldn't trust systems that have a different causal structure, then you can't trust much. Because your brain is changing a lot, right? And so phi is changing. And

52:49 one last thing that last ingredient that is needed to make the argument is that sure, IIT might say yeah, your brain changes and uh consciousness changes too because the causal structure of your brain changes. Right. But here there is a hidden assumption. IIT has to assume that your brain can never ever change so as to change phi without changing the input output behavior. And we know that there are many many ways of doing so. So I'll reiterate the point. Uh-huh. Your brain is constantly

53:24 changing its causal structure because I don't know stuff happens in your brain, you fall on your head, you get a stroke, you there's many things that can happen. Right. Right, you get drunk, whatever. And you say some of those may change my brain to a feed forward system. as long as long as the change in your brain is mirrored by a change in consciousness, IIT is sort of fine because they can say, "Well, yeah, the change in the brain occurs and consciousness changes." Right. But we know, like, it's a mathematical fact that there are many,

53:57 many, many changes that will change the causal structure without changing the input-output function. Right. And IIT needs to say that this can never happen, otherwise I cannot trust your reports. I cannot trust my own reports, in fact, because phi is changing and the percept predicted by IIT is changing, but it doesn't change my reports. So, the argument I'd like to make But is that I mean, is that fair though? Um So, what you're saying is that I mean,

54:29 they have to rule something out, which doesn't sound all that implausible to rule out, which is that hitting you on the head is going to change your brain into a feed forward system. No, that doesn't seem like something we need to worry too much about. It doesn't have The change doesn't have to be take a super complicated recurrent system and unfold it. It can just be take a recurrent system and change it a little bit, and phi predicts that now you don't have phi of 10, but phi of 9, phi of 9, and also you're not experiencing redness, you're experiencing blueness. It's like just a

55:03 different recurrent system. Right. The The point is that in general, well, I I understand the argument that we shouldn't trust systems that are too different from humans. Okay? But I think that this if you try to take this to rule out the unfolding argument, you have to basically trust pretty much nothing at all. And Yeah, I guess I don't see why where the slope is that slippery that you slide all the way down to that to that point because Because because so What they're

55:36 saying is you don't trust things which are un un causally their causal structure is doesn't have a certain integrated you know, metaphysical integrated blah blah blah Mhm. mix homina I don't want to get into all of it but you know what it's that stuff that they care about so it doesn't like So what you're saying is they have to rule out the this idea that the brain will change into a system that isn't like that. And granted there are many many systems which aren't like that which are behaviorally input-output equivalent to

56:09 the brain but why do we have to worry that that's happening to me right now? So so of natural processes. Yeah, so I guess that deserves more more examples but so if you have like a neuron here, two neurons here that talk to each other and a neuron here and then like these two neurons change the way they communicate. Yeah. it can be on the very super local level and changing this local thing shouldn't you wouldn't expect it to have any impact on the

56:43 function but it can impact the causal structure. Even if it's less drastic than taking your whole brain and unfolding it. I see. So what you're saying is that I mean that you already said that but I guess now I understand what you're trying to say is that the phi changes but my reports don't change. Yeah. Because you haven't changed the structure enough to change the function. Yeah, yeah, exactly. But you're going to change the functionality. So I mean so if you change a synapse like you get four more dendrites or dendrites spine grows or you know

57:15 that's going to change something functionally. Maybe not dramatically as far as reporting stuff like that goes but it's going to change something. Well, not necessarily because we know that there's a lot of degeneracy when when comes to implementing an input-output function. Uh-huh, I see. And so, maybe for some reason that I don't see, there is something avoiding all of these changes happening. Yeah. Then, basically, IIT would need to

57:49 guarantee this either experimentally or as an axiom or something like this. Right. Yeah, no, I see I see better what you're saying. the more easy to the more easy to make point is that you need sort of a criterion that you're going to choose to say what you trust and what you don't trust. Right. And you're not really allowed to trust experiments on Drosophila and then say that you think that it's not fair to to have a little implant in your brain

58:21 that changes the the causal structure a little bit, Right. But maybe there is a way out of Maybe there is a way out for IIT here. Maybe they can find a criterion that satisfies everyone and they say, "We trust this. We don't trust this. We don't trust this." And it's all good. But I just don't see it at the moment. Yeah. I mean, it's interesting because I mean, I never really understood, to be honest, what like phi is measuring. I know they want to say like level of consciousness, but I never really understood that. Like something has higher phi than me,

58:54 what does it mean? Like the grid. This grid has higher phi than me. Like than me? Like are you kidding? It's phi is higher than mine. It's more conscious than me, but And then so, but when you talk to these guys and you ask them this, what they say is, "Well, the have the high phi means you can make a lot of distinctions in your experience. Doesn't mean you're making them. It means you're capable of experiencing a lot of distinctions." And so, the greatest has that dispositional capability. It's not exercising it. So, so, so similarly in the kinds of cases that

59:26 you're talking about, when the phi changes from nine to eight or whatever like you know whatever you're talking about but the reportability doesn't change. I'm not sure if that really is a problem for the theory because what they're saying is like if the phi changes that means you've lost the distinction. If it goes down then there was something which you could have distinguished in your experience before which you can't now. Yeah. If it goes up then there's distinctions that you can make in your experience that you couldn't make before. But it's not clear why that would necessarily have to be reported. Now I get what you're saying if it's not reported then you can't experimentally

59:58 study it or something. reportable. Yeah. between the two is probably impossible to notice even if you could ask a hundred million questions to the system. Like there's it's not it's not just like the the weak overflow thing where you're not paying attention to it. It's like Right. It's the hard so once again you're like forcing them back to the hard the the the the strong overflow. Yeah. Yeah. As the only way out here. Yeah. And don't put me in a position of defending the strong overflow but uh Um I mean it's I was just to defend it

1:00:33 really quickly. I mean some people think you know what's the big deal? You have some bit of consciousness there that you you can't report it if it's not reportable. Um I mean Well they cuz the the really the conclusion of the unfolding argument. I'm trying to sorry I'm being so dense here but the conclusion is that one or the other of two things is true. Either the theory is falsified already or it's like outside of the realm of

1:01:08 science. It's not a scientific theory. So those are the those are the two conclusions. And so it now what we're we're talking about is okay so in order to meet the criterion of the unfolding argument wanted to like bite the bullet they've got to go to the strong overflow and say there's experiences that are not reportable and so then you say aha but now you're unscientific and then they go aha but now we argue about the mesh argument and that is a perfectly scientific argument Yeah but I

1:01:41 think what it's so they the kind of objection I'm considering they would say to you is that you know you have way narrow definition of what science is as only involving like a experiment like if you can't distinguish it in this experiment then it's not a scientific theory but that's the whole point of this other stuff is like well there's a whole bunch of other That's that's that's maybe an approach that you know for example Block could take to uh counter the paper that Matthison and I wrote with the the Valius But I don't

1:02:15 think this is available to IIT because what the unfolding argument shows is that basically the reports it's not that in one particular experiment the reports can change it's that you can make the conscious state of a system be whatever you want and the both the content and the state actually and preserve the reports so you could for example create a system that is constantly saying that it's I don't know

1:02:50 enjoying a concert of the Rolling Stones even though what it's doing what it's really experiencing like according to IIT is constantly smelling ham you know or something like that Yeah exactly it's completely dissociated so you can't mesh anything you know they say okay but then maybe in this experience experiment uh it doesn't work so but I'll do a mesh with this experience this one and this one and then we say but no but the situation is the same for each and every experiment." Right, exactly. So, there's no mention dissociated. That's our claim.

1:03:23 Let Let me try one more thing. And remember, I'm not the best the best guy to defend IIT cuz I agree with you secretly. So, you know, but I'm trying my best to be impartial. So, I wonder like so, one thing that the you know, when I when I talked to Julio, um I was at a workshop where he was they were talking about all this stuff and one of the things that I kept hearing him say um was that, "Look, you guys are all missing the point because IIT starts with phenomenology. And so, we don't have this problem. I mean, I didn't hear him say this specifically

1:03:55 about unfolding, but it was in relation to things which are in this general area. But the his idea was, "Look, we start with phenomenology and then we make a some uh so, we have these axioms, you know, blah blah blah. And then we have the postulates. And the postulates allegedly are ways of going from the abstract axioms to an implement something that would implement those things. So, the phenomenology is already built in. And so, the reason why you trust the the uh IIT system but not the unfolded system is because we got to IIT starting with phenomenology,

1:04:29 but we didn't get to the unfolded system starting the same way. And so, that this stuff about why would you trust one or the other is, you know, misses the whole the whole point because in in in one sense, one gets the sense that Julio is one of these guys who is like, "Look, David Chalmers just Dude, stop accusing us of the hard problem and both missing all the things. Like, we're [ __ ] doing it. Like, okay, we're starting with phenomenology over here. Like, this is what we're doing." That's kind of the attitude he has. And this the argument you're presenting sort of um one can get the feeling that, "Oh, you're bystepping sidestepping that

1:05:01 whole starting point and saying, 'Look, you have this problem.'" When really his whole approach was designed to meet the problem. Yeah. I see. Um So, I think this is a nice place for me to just flag that when I first encountered IIT, I thought it was really cool. And I really respect the approach because as you say it tries to really tackle the problem head-on and it's admirable. I still think it doesn't work though. And here's why. And here's why. So, you can try, I agree, if you can show me that you start

1:05:38 with phenomenology, uh and you can do it in an unambiguous way, then maybe you don't need third-person data and maybe the unfolding argument doesn't apply in this case. Or even for sure, if you don't need the third-person, you know, experiment, then there's no problem with the unfolding. But the problem is that the axioms really cannot be just assumed to be true. For one thing, it sounds a little bit Cartesian to say, "Well, I just know all

1:06:12 this." You know? Just like that and I don't need to prove it. But if at least everyone in the world was like, "Yeah, these axioms, man, they're totally correct." Maybe we could, you know, start from them and then if everyone was like, "Ah, here is the axiom and here is the mathematical formulation and it's clear as water that this is how you go from one to the other." Maybe we could, you know, accept it. But the problem is first that many people don't accept the axioms. They can be criticized in many

1:06:47 different ways as for example, Bain has shown. Some researchers like flatly deny that the axioms are true. For example, Zeki doesn't think that uh consciousness is unified or some illusionists would perhaps deny that it exists. And so, you just have to convince these people and to do this you need experiments. On top of this it has been shown many times that there's lots of degeneracy. Like there's not a single straightforward way to go from the axioms to the mathematical computations. And so for

1:07:21 all of these things you need third-person data. And I think proponents of IIT agree with this and they say, "Well, here we need third-person data." But it's like the snake biting its tail, right? You say "Oh, no, the unfolding argument is here but doesn't matter we rely on first-person data then but we justify the axioms with third-person data. Oh, but then the unfolding argument kicks in. So, we save it by going to first-person data but then Right. we cannot use this to get to phi so we use third-person data and just goes round and round. So, that's why I think

1:07:54 it doesn't work. Even though it's kind of a a neat idea. Yeah, interesting. And I yeah, like I said, I find it persuasive for the most part. So, I I think it's really interesting and it's got people talking a lot. It's caused a lot of conversations. So, that's also exactly what we need is more conversations like this to happen. So, that's fantastic. I agree with this and I would in a sense I would feel better about about all these papers if we had a few other people trying to lay out their own

1:08:28 criteria and Right. you know, and we could have a more multi-sided conversation because now I sometimes feel like I'm nagging and saying, "Yeah, but you can't do this." And I don't want to be the police kind of Oh, well, I kind of want to be the police You're the consciousness police now. Too but too late. Uh yeah, no, it is starting that conversation though. So, I think more people should do that. I agree. Um but but I also wonder to what extent

1:09:01 you think that this argument generalizes cuz I I've heard I've seen like places where I think you've said maybe it applies to global workspace theory too in certain versions of it, maybe like the global neuronal workspace. I've seen like other people try to generalize the argument. I'm thinking of like the whole argument and and that and that kind of thing. So do you think they're successful cuz they generalize it to every like every like it's like universal asset all of a sudden. So do you think that's successful? So anyway, yeah, I'm wondering about those kinds of sort of things. Yeah. So you're talking about the paper by Kleiner and Hoel

1:09:34 Yeah. where they generalize the argument and claim that it applies to all theories except like purely behaviorist theories that basically say consciousness is your report. Right. Exactly. So then it then it wins. Yeah. Well, no because they say that this is circular. So it doesn't win. Okay. Dang, so close. But um I'm sort of half I'm not really convinced

1:10:06 that it has the same I see the problem but I don't think it has the same impact for other theories as it has for causal structure theories. And here is the difference. In the case of the unfolding argument as applied to causal structure theories, you can just like logically show that IIT has this sort of inherent tension, this sort of contradiction where the reports are double dissociated from the purported contents and level. You can just show that as a matter of

1:10:39 logic. And the other cases, they say there exists transformation that mean that the global workspace is conserved No, it's destroyed but the reports are the same. And I see the point but to me this is a restatement of the hard problem saying that there is a sort of deep deep concern. Right. Right. That there could be a zombie or the concern that there could be a transformation. And depending on how we interpret the global workspace theory

1:11:13 and depending on what this particular transformation is, you can discuss. I agree that it's sort of a concern, but it's not as extreme as what happens with the unfolding argument where you can actually show it and the proponents of the theories say, "Yes, it's true. That's the right interpretation of our theory and there is this problem." And you know, So so you don't think it generalizes. I mean, you sort of uh it does generalize, but in a harmless way, right? It kind of just generalizes to the uh It doesn't Yeah, yeah. It generalizes to the same and so the generalization is

1:11:48 sort of as impactful as the problem posed by the hard problem to me. Okay. Right. And you can always tell Stan Dehaene that the global workspace cannot work because there could be in another world, you know, a global workspace with no consciousness. And he can say, "Well, yeah, I see the problem, but I'm still going to keep keep on working on my theory." Right. Uh but but you don't think that like the cuz that was sort of like the same general line of response I was pretending to have, right? That the IIT people could have. But you So you think

1:12:21 it doesn't work for them why why it could work for Stan. Well, the difference is that Stan can say, "Yeah, maybe maybe there are some transformations that pose this problem to my theory. And when you will show me one, I will consider and maybe say, 'Well, okay, I need to change my theory in this way or I need to revise this assumption or bite the bullet and then have a big problem.' Or you know, things like this." Right. But so it's it's responsive to empirical

1:12:53 evidence. Yeah, I think so. And a big part of the problem is that IIT like is sort of pushed in a corner where they have to accept things that they don't want to accept. And I don't think that for example, global workspace workspace theory is in that corner quite yet. Okay. So but what about what about recurrent processing like uh you know, Lama? Is he in the corner? I guess it depends. It depends how he wants to interpret his

1:13:28 theory. If he wants to say, well, recurrence per se I think we have a section about this in the paper. If you say recurrence per se is necessary and sufficient for consciousness, then he's in the corner. If he says, well, no because I mean, you need the all these background conditions, you know, for this to work, then he's not in the corner, but he doesn't have a complete theory because Then other systems has to specify the the background condition. Right. Okay, interesting. So the only one who's purely hit by it is IIT or any I mean, maybe there are other purely causal structure theories, but

1:14:00 that's like the only one that we know of right now that's like purely a causal structure theory. Yeah, I think the I think the unfolding argument hits you with full strength if you accept to be a purely causal structure theory. And if you don't, I see where Kleiner and Hoel are going with their argument, but I don't think it's as strong because it's just like a thought experiment. It's just Right. It's not like an actual demonstration that this is the logical

1:14:33 problem in your theory. Right. Whereas whereas for IIT, you think that's just sort of like an entailment of their theory. And in fact, they inadvertently admit it and then go on and say, but who cares? And that's the problem. Mhm. Yeah. So I I I I guess I would one like one day like to ask a proponent of global workspace if they think that two neurons here and two neurons that are recurrently connected here and one of them has a winner take all and gets broadcast in these two neurons is conscious. Right. And if they say yes, they're a causal structure theory and they have a

1:15:05 problem. And if they say, well, no, then I mean the whole question will revolve about why they say no and how you interpret the theory and and so on. Interesting. I mean, cuz that almost sounds like you're switching to the small systems problem to make the point. To sort of say that they they push them away, I think. Uh-huh. It It's related in the sense that because I thought the small systems problem was like its own problem. Yeah. But now it sounds like what you're saying is you can use it to show that they have a problem in this other genre

1:15:37 over here. Well, in the case of global workspace, I guess it's causally related because I mean the small network with two neurons like this and two recurrent neurons and there is a winner take all mechanism and one of the two gets broadcast. Mhm. That's the global workspace causal structure. So, if you interpret the global workspace theory as a causal structure, then this is an example of it. Mhm. And I don't think that this is what the

1:16:12 global workspace theory people want to hold. I don't think Yeah. I don't think they would say, yeah, sure, it's conscious. So, it was just a way to point out that all of these problems that Kleiner and Hoel point out depend heavily on how global workspace theory will respond. I see. And they haven't really responded yet or not I mean it it's No, they haven't. They haven't. Yeah, exactly. But it's hard for them to respond because it's all very vague. They don't they can't really say yeah,

1:16:45 we accept it or not. They just say well, yeah, maybe I don't know. Yeah, exactly. Yeah. I mean, I know like for instance like Bernie bars, he thinks global broadcasting is a necessary condition for consciousness but not sufficient. So there's got to be other conditions that are met. He's like very clear about that. So then But I don't know about Stan's view about that. It's not as clear to me about what the Dehaene and kind of you global neuronal workspace take it is on on those sorts of questions. Yeah, I don't know either. But that's the point. We can't really tell Yeah, exactly.

1:17:17 the problem is until It doesn't mean their theory is perfect, right? That's like a hole in the theory for sure. But Yeah. I mean, what what one of the nice things about this whole kind of discussion that you're bringing out is that well, as you said at the beginning, we have a lot of data, we have a lot of theories, but actually when you actually get to looking at the connection between the two, we start to realize, "Gee, the theories aren't really worked out in all the detail we might have thought. They're sort of like very hand-wavy." When you get to the nuts and bolts, like, "Okay, how does it it deal

1:17:49 with this piece of this experiment, this interpretation?" So this is the sort of work I think where the field needs to to be focusing on. It's like yeah, there's this myth that if we just get it like if we just get a connectome, then we'll solve consciousness or the mind. It's like, "No, that's that's not even half the story." So I I think it's very cool that we're bringing that you're bringing these lights and that we're starting to have these kinds of conversations. Um okay. Uh let's see. I Oh, I see we're we're getting on time. I can't keep you forever, but we're only halfway through. So I I mean, I guess I would you want to get to the other two

1:18:22 criteria here if we have time. Um I know you have somewhere you have to be, too. Uh so I don't want to get in conflict with that. But So we already talked about the small and large network systems problem a little bit. Um but uh I mean, is is is this related in your mind to I will Hawkman sometimes, you know, calls the hard hard enough problem or what's he call it the calls it something. Um what you know, trying to say like does your theory apply to bees or does your theory apply to computers? Um sort of just looking at different

1:18:54 enough problem again? It was That's Scott Aaronson, I think uh he came up with that. So But anyway, so just saying like which systems are conscious basically is is is that what this large small network stuff is about? So the small and large network argument is just to say basically that for many theories you can satisfy all of the you know, conditions posed by the theory for a system to be conscious and like

1:19:27 using five neurons or 10 neurons or something like that in a small network and you can also realize it if you consider the galaxy or something like that. Right. Right. And so that poses basically to my mind what this poses as a challenge is to say, well, probably you're opening the doors too wide and something is missing. Some constraint is missing from your theory. And that doesn't mean that your theory is crap at all

1:19:59 but it sort of explains why we have so few conflicts between theories because everybody is opening the door wide and everybody is sort of under constraint and I think that's a neat you know, hunch that the field can have and try to work out a little bit better what they think should be added. Right? Right. and then the other the second part of this argument is to say, okay, you can also maybe accept that uh five neurons are conscious and you can but then you

1:20:32 have to then the problem is that you need a criterion for saying which of these small networks are conscious and this criterion is sort of under determined by empirical data. So maybe it's true, but we're going to have a hard time figuring out if it's true or not. Um but I to me that it's more interesting to view it not as a weird, you know, panpsychist claim and more as a challenge for theories to try and get out of it. Right.

1:21:07 Right. Yeah, and so how successful do you think like the theories we have are at getting out of it? Like most of them are because it's like this can the widest net of all the argue criteria to capture the most theories you think. I'm not sure. I think for example, the global workspace theory is pretty susceptible to this because broadcasting is such a general computational mechanism that as we said before, you could realize it's a tiny network. So it

1:21:40 seems like, you know, something is missing. For other theories, it depends a bit more on how you want to interpret the theory. Right. Right. So for example, higher order thought theory, I don't think you know, in any of the forms that people hold succumbs to this argument, but No, good. So you say, so what's a thought? And if you say, well, it's just what's a higher order thought? If you say, well, it's just, you know, reading out from some representation, then the small network argument applies, but nobody really

1:22:14 believes this. Uh-huh. Uh I guess Uh but then the problem is with the that the problem is that the mechanism for hot is pretty vague. So you can't really know which systems implement the hot and again it's a challenge like saying you a nice way to develop hot would be to find a mechanism where it's precise enough so that we know what the hell a higher order thought is and which systems have it but also it's not opening the doors wide enough that

1:22:49 the stupid little you know memory readout system Right. implements it. Uh I mean so do you do you use uh you don't don't I mean isn't there a worry here for for some versions of higher order theory maybe um like uh like Hakwan's perceptual reality monitoring theory which just says basically that there is a simple kind of system that's reading out or that not reading out but pointing at that that's that seems like you could get a simple system like that could be implemented in relatively I mean maybe not four neurons but you know

1:23:22 100 or so could probably get you could get something like that. Uh so I don't think that that's if if the mechanism simple enough depending on what it is uh the argument might apply. Yeah. So the way I understand Hakwan's theory is that the argument would apply to it except that he has one clause if which if I understand correctly is something like you need a complex enough No, what was it again? complex enough I don't remember. At some yeah.

1:23:59 metacognitive capacity or something like that. And that's very ill-defined in my opinion. Right. Uh but so it sort of shuts the door to that argument. Yeah. it opens the door to what do you mean? Yeah, exactly. It opens the door to the incredulous stare. By the way, someone in the chat over here is asking about O R a a Um so uh I don't know I guess I didn't really notice that you were talking about that a lot in the paper, but do you mention Orch OR as favorably or just Yeah, we mentioned it. Okay, so in under what category?

1:24:33 Uh, we So, we mention it as uh, associating consciousness with insight because that's the argument Penrose comes and says um, here's something that only humans can do and machines can't do. It's having insight and basically finding a way out of Gödel's incompleteness theorems. Right. And so, this therefore is what we should

1:25:08 look for when we look for consciousness and then he finds a quantum mechanical process that sort of is supposed to solve uh, Gödel's insight. Uh, No, Gödel's Well, so supposed to explain how a system can sort of get out of the incompleteness thing. Can sort of realize that Gödel's theorem is true. And we say, well, uh, you're just identifying consciousness with insight and you're

1:25:40 First of all, the argument that you need quantum mechanics for this, you can accept it or not, but let's assume that it is true. And if we accept that it is true, uh, then maybe you have a theory of insight, but you have no unconscious alternative because it's the the the problem mainly is the unconscious alternative thing in our opinion because you have like quantum collapse all the time everywhere and it happens when consciousness is there and when consciousness is not there experimentally and that's the problem for us. Yeah, okay. You know, it's interesting.

1:26:13 Now I realize I've never had anyone defending Orch OR come and talk to me for a minute. I should get someone. Yeah. But, uh I've never found it very convincing. I mean, I guess I mean, cuz I I don't know if I understand the uh objection that you're raising though. But, uh as far as I understand, it's like, okay, you have these two equally wacky ideas which get joined together. Um on the one hand, you have Penrose's idea that uh um the collapse of the wave function is consciousness. Uh that's that's Penrose's ideas I

1:26:46 understand it. And then, you have Stuart Hameroff's idea that microtubules are are important somehow. Yeah, or directly involved. combine the two, and what you get is that the brain is like a conductor which organizes a bunch of all of these random collapse events into a coherent picture, but that the stuff is out there all the time collapsing and there's little pops of consciousness here and there or whatever. So, I mean, that's my novice understanding of Oakor. I think that's the basics of it. But, so now, what you

1:27:19 were saying was that they have a problem with this unconscious comparison case. Is that Is that because of the fact that consciousness is like always popping into existence even when no one's around? Yeah. Yeah, so as far as I understand, the idea is that consciousness occurs when you have the sort of auto collapse of the of the quantum state. And this happens all the time, everywhere. Exactly. Even when there is no experimental sign of consciousness.

1:27:52 Then, maybe maybe the Hameroff part, you know, you could try and measure how much quantum entanglement you have when you're awake versus when you are asleep. Right. And if that's the full extent of the theory, just saying, well, you have quantum entanglement when you're awake, but not when you're asleep to a large degree. If that's it, then okay, it's fine and we can check it out. I'm just I just think it's seems a bit weak as a theory. Yeah. Um I'm not I'm not sure it's really like one of the

1:28:26 main contenders. Uh I know Stuart wants it to be, but I'm not sure it's really I don't Yeah. Because I think for exactly this problem, it's First of all, it's at odds with a lot of mainstream neuroscience in the sense that this idea that the way neurons function is just like the computations going on in the microtubules is way beyond what I mean, Stuart says it's a self to self to self to self, right? That neuroscience can matter because he's saying each neuron has like the computational power of like what we generally think of as a whole brain, basically. So, it's like astronomically under-estimated if he's right.

1:29:01 Um but there just really doesn't seem to be a good way to really test empirically the kind of claims that they're making. I mean, um except for the more general one that you were just saying. That's why I think it's as far as we know, if you want to understand what the brain does, you need to look at the neurons, you know? Yeah. And not at the microtubules, even if the microtubules may influence it, but you know, the relevant part are are the neurons and also, yeah, as far as we know, neurons are sort of linked to As far as we know, yeah. So, the argument is as far as we know, if you

1:29:34 want to understand consciousness, you need to understand the brain. And as far as we know, if you want to understand the brain, you need to understand neurons and microtubules are sort of not really relevant, but maybe maybe yeah. Yeah. Maybe this is wrong, yeah. I mean, someone in the in the chat is saying someone named Queerdo, I think, is saying that uh they're testing it currently and they claim it's the most falsifiable of theories of consciousness. Okay. Yeah, so as I Yeah, so as I said, if the extent of the theory is to say they will like consciousness is quantum

1:30:06 entanglement, and if we find no quantum entanglement during wakefulness then it will be falsified. Then fine. It's just that it's not going to be a very to my mind interesting theory if if it's true because you have quantum entanglement everywhere all the time. Yeah, exactly. I mean that's why it's kind of like a panpsychist or you know, consciousness is everywhere sort of view. Um but you know, my worry is more like you have to modify the equations of quantum mechanics to to make it work.

1:30:39 But I mean it's still consistent with experimental stuff, you know. I'm always a little bit Once you start going in yeah. It's a little bit sketchy. But anyway, so um I forget what I was where I'm seeing the time is ticking away here. So I want to ask about the fourth criteria which we haven't talked about but can you tell us briefly what the fourth one is before we Yeah, the The fourth criterion is simply saying can you tell me which systems apart from humans

1:31:13 are conscious? And if not, that's not necessarily devastating but you don't have a full theory of consciousness. Right. And if so, which which systems are conscious? And so this is typically meant to target theories like the thalamocortical loop theory that says that basically what matters for consciousness is having thalamocortical loops. Right. And this is an interesting theory and I think it led to cool research. But is then the idea that consciousness only

1:31:46 occurs in the thalamocortical loops or is a function performed by the thalamocortical loop important? Right. What What is important within the thalamocortical loop which which does billions of things? So, that's the the criterion is to say, well, if you have a theory like that uh it would be interesting if you could tell us a bit more about what's important. Yeah, exactly. I mean and a lot of I mean, you know, you take someone like John Searle and his biological naturalism, which I'm

1:32:19 somewhat sympathetic to. I wish it were true, but it's like this is a big problem for them. They say, "Well, what is it about the biological system that you can't do in a functional duplicate?" And they're like, uh "I don't know. Is it Maybe it's calcium. Yeah, yeah, it's it's definitely calcium ions." Right? Like, there's nothing really you can say that's plausible at that point. Yes. Yeah. So, this is a big problem, I think, for theories of consciousness is saying when it applies um and when it doesn't. And so, I think one tricky thing is that for many theories, the problem is kind

1:32:51 of in between the small systems argument and this problem. Because either you keep it kind of vague and you evade the small systems argument, but then you can't really explain what is truly important for consciousness, or you explain what's truly important for consciousness, like broadcasting, but then you fall into the other problem. Right. Many theories, I wouldn't say are ruled out or wrong or even in a bad position, but none of them make a convincing case, in my opinion, to escape both of

1:33:25 them. I know that, for example, Matthias Michel thinks that hot can escape both, but the Uh-huh. I haven't Why do you think it doesn't escape it? convinced yet. I don't know. Because Yeah, I would need I need to think about it more. I'm just Suffice it to say, I haven't been convinced. Yeah, fair enough. Um interesting. So, that's what I was going to ask you, actually. Do you think any theory can kind of navigate through all these And what what saying is currently you don't think that there's there's any theory we

1:33:57 have that I mean like you just said I mean I think it's very modest a conclusion. Not none of these theories are falsified or should be thrown in the garbage or not trash but they're definitely none of them has satisfactory answers to every one of these considerations. Yeah and these are considerations that deserve satisfactory answers in my opinion. Yeah and but so are you optimistic that as a group we could get to such a point where we have a theory that has at least you maybe not a universal acceptance but

1:34:29 the majority of people think it's like you know like a standard model in particle physics could we ever get to that point in consciousness? So I I think in principle it's possible yeah. Uh-huh are you doing all right? But I think that unless we sort of get to grips with the kinds of criteria that we propose and potentially a few others we will never get consensus because if you're too vague or open the door too wide you're not going

1:35:02 to be mutually exclusive with other theories. So that's why it's really important. The the goal is not to get everyone to be really specific so that we can tell them all they are wrong. The goal is more to say well here are a few important things to try and make theories mutually exclusive so that then we can say oh this theory is good for this but it's wrong for that and this one is good for this and then combine them and then stop you know the current status is kind of everyone thinks about their own theory and does experiments and since

1:35:35 they're vague enough or open the doors wide enough they can explain the experiments in their framework but then the other people from the other theory are doing the same. Right exactly. So so the bottom line is that I think it's there's hope for improvement and I think that the approach we suggest is one important part of it but also other people keeping doing experiments even developing their own theories is equally important. Right, exactly. No, it's an exciting time

1:36:07 because lots and lots going on sort of sort of the like before we become a normalized science. Um, so it's interesting and these here I mean yeah anyway I'm getting distracted by what you're just saying but I'm trying to stay focused. It's too too interesting Adrian stop it. But so do you think like and you're just as a as a young researcher someone is starting out making already interesting contributions and shaping the discussion and all that stuff. Do you think that consciousness science will still be a recognizable field 10 15 years from now?

1:36:40 Um, or you know 20 or whatever or do you think it's going to be like a fad and then people look back on it and be like oh look at that but you know now we just do you know there's perception there's attention there's you know and then it just goes back to standard neuroscience. Or do you think that we've made a kind of interdisciplinary area that will last. I mean maybe maybe you don't know but I'm just curious like what your sense as a person who's entering. Yeah, I I've never asked myself this question but if I had to you know say something off the top of my head I think we will stay a a field

1:37:16 for a long time because um well we have interesting philosophical questions and conceptual things to think about. We also have good enough empirical access to consciousness so that science can you know focus on consciousness and not just on perception. And so I don't see why it should all break down. I yeah. So you're not cuz some people are pessimistic that you know like maybe you know, as a

1:37:50 philosopher I don't have to care as much cuz no one I'm not asking for money to do what I'm doing. I mean, I am, but not constantly. I sort of asked once and they said yes, and now I'm like, all right. But I don't have to constantly go back and be like, can I do more? Can I do more? Yeah, yeah. Uh, so it's different in science, obviously, than philosophy because you're constantly wondering about who's giving me that that funding and how my lab's going to survive. Um, and and I know that that's obviously a very big difference in the culture and the way we think about things because of that, but, um, but some people are thinking that, you know, things like panpsychism, which as a philosopher I don't have to That's my

1:38:23 kind of my segue there. I don't have to worry about too much, but like as a scientist, some people worried about that. So we have panpsychism, we have IIT, which has, you know, if you take your line seriously, it's kind of unfalse, it's either false or like not real science. Um, panpsychism is also something that's unfalsifiable and therefore not scientifically So some people think, well, and these are big, kind of getting popular, you know, depending on who you talk to. IIT is a leading theory. Yeah, panpsychism is being taken more seriously by scientists. Mhm. Um, so some people are

1:38:57 pessimistic for the longevity of the field uh, because of those kinds of influences, and then they they don't see like if those try to get associated with like legitimate science of consciousness, now they're thinking that we're all doing that. Mhm. Um, then the field itself sort of is besmirched in a way that no one wants to fund the experiments and they're like, oh, that's just BS and we're not going to give you money for that. Mhm. So I'm not saying I agree with that critique 100%, but I I do think that as a scientist it carries a different weight than as me as a philosopher I don't have to worry as

1:39:29 much. So I wonder like, do you worry about that kind of stuff or do No, because I don't because when I go to, for example, uh, the ASSC, which is a conference about, uh, the science of consciousness, I see a lot of sensible experiments and I also don't want to make it seem like I I think all IIT scholars are bad scientists. I think their theory is wrong, but they make good science, too, you know? They do experiments that are worth doing. And I really don't have the feeling when I

1:40:03 go to ASSC that it's it's weird and that it should be regarded badly by other scientific fields. I did go to the Yeah, there is this other conference. Tucson one once and there it was more I have to admit it was my feelings were more mixed. What year did you go to Tucson? I It wasn't there, actually. It was in Switzerland. It was in Interlaken. Oh, right. It was one of those other years. 2 years ago, I think. Uh-huh. I still had a good time, but

1:40:35 Yeah, of course. It's a great time, but that was in 2019, was that uh Yeah, essentially. I think so. Uh-huh. Uh yeah, I mean this this if the field was constantly always like this, I would be more more concerned. But I think there's a lot of great research being done and even though we have to get to grips with some conceptual issues with how we compare theories, I think I think it's going to be nice. Yeah, good. Okay, I I want to be optimistic, so I I'll accept that for now.

1:41:08 I I see the danger that you mentioned, but I don't think it's getting out of hand quite yet. Okay, good. Yeah, no, I share that I mean, I share that sentiment, but then again, like I said, I have different different acculturation, different expectations, and so I'm interested to hear your your point of view. Yeah, maybe my my my point of view is tainted by the fact that I'm in parallel doing mostly, actually, computational neuroscience, so Right. And I use deep learning and that's like the most fundable thing these days, so Yeah, exactly. That is the most

1:41:39 less of a concern for me. That's a good point, yeah. Um All right, so I I see we're getting on in time. I really do have to let you go, but I I wonder at the end here if there was anything you were expecting would come up uh which hasn't. The answer can be no, but I always like to check at the end. Um is there something you were Yeah, it's nice of you to check. One thing that we didn't really talk about is that is a paper uh we wrote with Michael Herzog and Leila Drissi Daoudi about uh the possibility that human consciousness

1:42:12 might be a discrete process. Ah. Which we didn't really have time to go into. And I think it's going to maybe take a bit too long to go into it now, but I just wanted to flag this. Um and it's another When when I give an when I try to give examples of how focusing on empirical data as we do and trying to not go into uh strong overflow and all of these things, how this can be fruitful, I like to point to this paper because I think we have a nice review of many empirical phenomena and we try to say something

1:42:45 theoretically novel with it. And I would I would direct people to this paper called Okay. All in good time uh and then I don't know the rest. I'll just find it so I can say the the title. Okay. Yeah, I've read that paper, it's good. It uh thank you. It's in Trends in Cognitive Sciences. And it's called All in Good Time: Long-Lasting Postdictive Effects Reveal Discrete Perception. So I just wanted I was talking to uh Evan Thompson about that that paper. We were talking uh Yesterday? I didn't watch

1:43:17 Yeah, it was didn't come up yesterday, but in our previous conversation, which I think was September. Um when I just I think that's when the paper came out, right? It was around September of last year or something. So it just came out, I think. But uh cuz I'm very interested in that topic. But yeah, so if you're interested, I'll have to harass you to come back and talk to me about that stuff cuz uh that's what glad to. Yeah, cool. I find that very interesting. Well, on that note, let me thank you for joining me because I do have to wrap this up. But it's been great fun talking to you. I learned a lot from your work and thinking through these issues. I think they're very important. So, I hope more people read these papers and all that stuff and uh Yeah, hang out for a second. I'll say

1:43:50 bye after the air, but thanks again for joining me. Okay. Thank you for having me. All right, we're off. Clear. Well, I'm glad