A recent post on LessWrong, “The Fourth World,” gets at an important implication of consciousness—namely, that we ought to suspect further aspects of reality than we can today observe—but I’m not sure it arrives there the right way, or that it derives the correct conclusions.

The author makes two assumptions in his post which ought not to come for free. Firstly, the author assumes that we could not explain consciousness through physicalism. This is already a strong claim, one with which about half of surveyed philosophers disagree. Secondly, the author assumes that consciousness is the basis for morality, or that our moral obligations are primarily due toward conscious beings.

Subscribe now

I think this second assumption is particularly common. At a recent meetup, I heard a participant complain that it is really hard to talk about consciousness, because we are always tying up consciousness with morality. I suspect this happens because we sneak in non-physicalist assumptions about consciousness into discussions about morality.

So, I would like to defend the author’s claim that consciousness ought to cause us to suppose further unobserved aspects of reality (which could be very different from the reality we know), while also defending a purely physicalist explanation for consciousness, and decoupling consciousness from morality. I will continue to respond to specific arguments in “The Fourth World,” though my goal is to articulate my own, positive argument, rather than critique the post.

Throughout, whenever I refer to “consciousness” without a qualifier, I am referring to phenomenal consciousness, or the “hard problem” of consciousness, or qualia. I’ll use words like “cognition” or “neurophysiology” to refer to non-phenomenal processes, which we typically understand as explainable by modern neuroscience, even if imperfectly. The distinction is important, as we otherwise backdoor capabilities like judgement and agency into consciousness without sufficient justification. I believe this is the crux of why so many of us intuit consciousness as a prerequisite for morality, when I think it is very possibly not the case.

Dualism as Semantics

The author introduces us to his fourth world by supposing three distinct domains to reality: the physical world, the mathematical world, and consciousness. The author later claims that a non-conscious, robot civilization would never suspect consciousness exists.

I surmise the author is pointing at some kind of dualism, with mathematics and logic being a third, metaphysical entity. The difficulty with dualist claims is that they either rely on a “mystical,” non-physical substance, or they rely on semantic qualifications for the physical world.

For example, let us suppose that we learn consciousness is actually the emergence of a soul, dipping in its head from some heavenly plane of existence. We learn that all our physical laws are arbitrary and completely determined by the whim of God. In this extreme scenario, do we then accept dualism?

Let’s reframe the question. Let us suppose instead that we learn consciousness is the emergence of higher dimensional reality within our own. That higher dimensional reality determines all the physical laws of our known universe. In this scenario, do we accept dualism, or merely expand our definition of physics to account for our new knowledge of reality?

My point is that nearly all explanations for consciousness ought to collapse towards physicalism, because any fundamental discoveries required to explain consciousness also compel us to redefine physics accordingly. I will retain the term “mysticism” for any explanations that depart so far from a conventional understanding of reality as to render an expansion of the term “physicalism” meaningless, i.e. if consciousness is actually the soul. So, we could concede dualism for the first example, but we ought to stick to physicalism for the second.

If we accept a physicalist explanation of phenomena as at least possible, then we should not assume a robot civilization could never discover consciousness. We have no way to observe or infer consciousness today, but if consciousness exists in physical reality, then it at least seems it might be possible. Without a scientific explanation for the hard problem of consciousness, it is premature to rule this out.

Phenomena are Intrinsic

Aside from a physicalist or mystic explanation for consciousness, we could also suppose that phenomena do not exist at all, or that consciousness is an illusion. Illusionism is a highly counterintuitive stance, given we all have the vivid impression of our own consciousness (how do you convince someone that they do not actually perceive the “redness of red”?). In some sense, our subjective experience is the only thing we know (cf. solipsism), so it seems really weird to claim it does not actually exist.

Our experience of existence is phenomenal, so I don’t see a way to refute the hard problem without asserting we are all already zombies (I experience, yet sadly I do not exist). This is contrary to my own axiomatic belief that I experience existence, so I don’t see any way to proceed further with a strong illusionist argument. If you choose a different axiom, well, ok.

But illusionism is right to question what and when something is actually phenomenal. In other words, I find it weird to refute qualia, but legitimate to debate the scope of qualia. Because when referring to consciousness, we are often actually referring to the reactive and evaluative machinery behind consciousness, e.g. cognition, and not to phenomena themselves.

Consider our phenomenal experience of pain. We generally dislike pain, and indeed many ethical frameworks take for granted that we ought to minimize suffering. But why? When we say pain is bad, are we evaluating a distinct “quale of pain” as bad, or are we reacting negatively to the neurophysiological process of pain?

I’ll refer to this as the difference between an extrinsic and intrinsic interpretation of consciousness. If we take a physicalist position of the hard problem of consciousness, then we can suppose phenomena are either intrinsic in neurophysiological processes in a way that we do not yet understand, or that phenomena are extrinsic to neurophysiological processes. In other words, is there a property of neurophysiology that we cannot yet quantify (like weight) or are there entities extrinsic to our neurophysiological makeup that we have not yet discovered (like a particle).

My reading of the dominant theories of consciousness (e.g., IIT, Global Workspace Theory) is that they lean toward an intrinsic view. Essentially, there is something about our neurophysiology that just does induce phenomena. The two are inseparable, phenomena just are a property of some neurophysiological processes. We don’t know yet what something is, though it might involve weird fundamental physics. In this view, there is no sense in talking about a phenomenon of pain separate from the underlying neurophysiological activity, in the same way it does not make sense to discuss weight separate from gravity or mass.

For an extrinsic example, imagine that there was an undetected particle of consciousness, the c-particle, which somehow interacted with neurophysiological processes, or were produced by them. Phenomena are actually composed of c-particles. We might further suppose that c-particles come in many flavors. There is a c-particle of pain, a c-particle of happiness, etc. Or perhaps they’re just different arrangements of c-particles, who knows! Mental states, as we understand them, are actually determined by c-particles. Unlike with an intrinsic view of phenomena, it actually is coherent to discuss pain separate from the underlying neurophysiology. We just need the right c-particles.

I have admittedly chosen a silly example for an extrinsic view of consciousness; I imagine few would argue for an actual “c-particle.” But any extrinsic view will require a mysterious “something else” to explain consciousness, which must then causally interact with our neurophysiology, unless we are willing to accept consciousness as a mere epiphenomenal side-effect.

It would be wrong to claim either of these theories are “correct.” As physicalists, we have to accept this as a question for science, and in reality, consciousness may not fit so neatly into my intrinsic/extrinsic divide. But I still find it much more likely that consciousness is somehow intrinsic to neurophysiology. An intrinsic theory requires less deviation from contemporary neuroscience, as we don’t have to posit neuroscience is somehow insufficient to explain mental processes (no c-particles intervene), and we can neatly sidestep some of the problems of epiphenomenalism (certain mental states are intrinsically experiential).

So, if we suppose an intrinsic, physicalist theory of consciousness, then our experience of pain is by definition inseparable from the pain reaction itself. It is, in fact, nonsensical to discuss an experience of pain distinct from the neurophysiological activity associated with pain. That is pain; the thing itself is the neurophysiology!

I suspect illusionism is right that there is nothing like a c-particle. There is no independent experience or qualia of a good cup of coffee—there is only the experience of our own neurophysiology tasting the coffee and evaluating it as good. Consciousness itself is not responsible for judgment and evaluations—that work is all done by the mechanics of neurophysiology. Consciousness does not “intervene” to make its own decisions.

Generalizing Ethics with Preferences

I’d now like to consider the ethics downstream of a physicalist, intrinsic understanding of consciousness. Namely, if consciousness is intrinsic to neurophysiological activity, then why should ethics fix itself on the conscious aspects of that activity? Is this not an arbitrary distinction? Should we not rather respect the preferences those activities represent?

I suspect that phenomena, as intrinsic properties of neurophysiology, are undifferentiated. A chair and an apple have different weights, but we don’t distinguish the weight of furniture from the weight of fruit. Similarly, our experience of pain and happiness may feel different, and may correspond to different biological states, but both experiences are ultimately “just” phenomena, distinguished thanks to our mental capacities. I would guess that the same is true even of sensory qualia, like the “redness of red.”

If all this seems highly speculative, then let me simply state that no phenomenon seems to me inherently good or bad. Experiencing happiness is distinct from experiencing happiness as good. Now, there might be something about e.g. dopamine that causes me to want more happiness. But that is a consequence of my neurophysiology, it is one of the reasons I experience happiness as good. It’s not a quality of my experience of happiness (the phenomenon of happiness) itself.

Perhaps one could object that there is no experience of happiness without wanting happiness to continue, or no experience of suffering without wanting suffering to stop; that the mental state cannot be differentiated from its reaction. But is this not just another reason to focus on the reaction, or the expressed preference, rather than the experience itself?

The distinction I’m trying to make may be irrelevant when discussing humans, who have well understood preferences for different mental states. But we have to be very precise if we would like to generalize moral principles to non-human intelligence.

Let’s suppose we are visited by a highly developed species of alien goo. We suppose the aliens are somehow biological, but it’s unclear! The aliens are very different from us. The aliens also have the strange habit of climbing to the tallest point in any room they enter. If for any reason the aliens cannot reach the tallest point, as in when they are restrained, the alien goo begins to vibrate. When released, the alien quickly proceeds to climb to the tallest point in the room.

Now, imagine the aliens are observing your living room. The aliens are very bad at detecting electromagnetic radiation and mostly observe the world through touch and vibration. The aliens are very interested in your potted plant, which slowly adjusts its leaves over the course of the day. The aliens can’t tell that the plant is adjusting to the sunlight. They just observe the plant always closes up its leaves for half the day.

Question: Without any further information, do we have a moral obligation to allow the alien to climb to the highest point in our room? Does the alien have a moral obligation to allow the plant to close its leaves?

Ultimately the answer will depend on your moral framework. But if you would agree that we have a moral obligation not to cause an alien pain, then I think you should say yes, we are both morally obliged to allow the observed entity to act according to their inferred preference.

In both scenarios, the observed entity exhibits a clear preference. The alien always wants to climb as high as possible. The plant always wants to open and close its leaves. We don’t have any clear idea of whether the entities in question are conscious. But why would this matter? Consciousness is intrinsic to some unknown subset of entities in the universe. Neither we nor the alien can detect whether an entity is conscious. But we do know that consciousness has always corresponded with the expression of preferences.

You might be thinking that the real difference is the plant’s response to light is entirely automatic, while we are deeply cognitive agents. You can’t compare our preferences to those of a plant! But I think this overlooks how dumbly reactive a lot of preferences are, even if they are unclearly expressed. If you poke me with a needle, I will want you to stop, even if I keep a straight face. I don’t have very much control over my pain response. It just happens!

We might qualify that a preference should represent an entity’s interests counter to the second law of thermodynamics. A rolling stone does not represent a preference. However, if it began to roll uphill, it would!

I anticipate several counterarguments to a naïve definition of preference, with potentially absurd conclusions. For example, suppose the alien also noticed your rotating fan. Wouldn’t the alien have to suppose a moral obligation to the fan as well? And how do we avoid moral equivalences between turning off a fan and trimming a plant and killing a person?

These are legitimate lines of critique that a moral framework using preference as the qualifier for moral obligation would have to answer. But it’s easy to imagine different mechanisms to do this, such as by defining some heuristic for the “strength” of preference, similar to the way utilitarianism thinks about utility, or looking at the reversibility of decisions. I’m also not trying to argue for some sort of cosmic libertarianism. The practice of ethics is inevitably messy, with lots of confusing gray areas. A good follow-up would be to evaluate a full moral system based on preference, maybe trying to stress test a few repurposed moral imperatives in a system that made no assumptions about consciousness. However, I am optimistic that in every instance where one might be tempted to evaluate ethics on the basis of consciousness, one could instead insert preference.

An Unobserved Reality

I accept that you might find my argument for preferences insufficient, either because an ethical system for preference is not yet defined, or because you believe that consciousness itself is very special. Though a physicalist should wonder why consciousness is restricted to the mind, it’s certainly the popular consensus.

What I question is how one can be confident that consciousness is unique? We already know there is at least one aspect to reality that is utterly unobservable to an outsider. We would have no concept of phenomena if it were not for our own minds. Is this not the best possible evidence that there might be other aspects of reality we cannot observe?

This is what “The Fourth World” gets right. Though why limit ourselves to a “fourth” world? For all we know, there might be infinitely many aspects of reality that we cannot observe today, or which may be fundamentally unobservable. Like the author, I find this incredibly exciting.

But these tremendous unknowns as to the nature of reality should give us pause before rating consciousness as unique and morally important.

Let’s consider a speculative example, meant to demonstrate the logical possibility of alternatives to consciousness in a physicalist universe.

Returning to our earlier alien scenario, let’s pretend we have uncovered the physical substrate for consciousness. In fact, it’s now relatively easy to purchase qualia counters, which, functioning similar to a Geiger counter, allow us to estimate the “amount” of phenomena in an entity. Holding our qualia counter up to the alien blob, we fail to detect any phenomenal consciousness!

However, the aliens have their own device. The alien physiology somehow fails to produce consciousness, but it does involve some other, mysterious aspect of reality… say, “monads.” And lowering a “monad” counter down to us, the aliens fail to detect a single one.

If we accept there could be further aspects of reality that we cannot observe, not unlike consciousness, then we should not take for granted that consciousness is privileged among intelligent beings. I understand this is very weird to consider, but I do not think it is any weirder than the hard problem of consciousness is already. Our example is similar to Nagel’s famous “What it’s like to be a …” thought experiment, with the additional caveat that we swap out consciousness itself for some new, unknown aspect of reality. Rather than ask what it is like to experience the qualia of a bat with a bat’s cognition, we ask what it is like to have the “monads” of an alien goo with an alien goo’s cognition (where “monads” are strictly different from qualia).

All the standard caveats that conceivability does not mean reality apply. But given the existing evidence for alternative paths to cognition from terrestrial neurophysiology (e.g. machine intelligence), we ought to consider seriously whether there might be alternatives to phenomena like consciousness, but substantively different.

To summarize my argument:

1. Physicalism is a popular and reasonable explanation of consciousness.

2. A physicalist’s best guess should be that consciousness is somehow intrinsic to neurophysiology, otherwise we have to make strange ontological and scientific conclusions (like c-particles).

3. Once we assume consciousness is physically intrinsic to the neurophysiological process itself, it is no longer necessary to assume moral obligations to one aspect of that process.

4. We should instead assess moral obligations to the preferences exhibited by neurophysiology. Presumably, we also owe some obligations to any agent which exhibits preferences, but this is the responsibility of a moral framework to judge.

5. One can try to defend an obligation to consciousness by asserting it is special.

6. However, if we cannot observe consciousness, we ought to suppose there could be further aspects of reality we cannot observe, similar or dissimilar to consciousness in ways we do not yet understand.

7. It is conceptually possible to imagine beings very different from ourselves whose cognition involves these unknown aspects of reality. So, it would be premature to determine that consciousness demands unique obligations.

Obviously, these final conclusions are several steps removed from ground truth. However, I have tried to surface the implications of what I assess to be the most likely explanations for phenomenal consciousness. For right or wrong, a lot of beliefs are downstream of our explanations for consciousness. While the hard problem remains unsolved, it would be good to continue exploring the ethical implications of different theories.

Leave a comment

Francis loved to take Bobby outside, because outside Bobby was free. Francis surmised that he himself was never free, as he was slave to a system outside his control. Though he sometimes attributed this system to his employer, he felt his non-freedom was actually a property of the system itself, which he referred to semi-ironically as “The Man” (he scare-quoted the term even in his mind), and that he had little chance of escaping. While several of Francis’s friends had moved on to accomplish great things, Francis had always stayed put. Francis had considered this an act of rebellion, perhaps the only kind possible to a man in his situation.

Bobby did not think at all about systems or freedom. Bobby, as an arachnid, was only capable of brief cognitive tasks. Bobby experienced small joys, like the rush of walking on bodies, but he had no capacity to ruminate on joy. Bobby had a subjective experience of life that extended just beyond his carapace, and only a dim awareness of his keeper.

It proved to be a sunny day, so Francis wanted to go to a park. This was a day off for Francis, and he was done with sitting on his phone watching videos, he was really going to do something and take Bobby outside. Francis had vague ideas that the sun powered the world, so it was important to sit in the sun and accept its energy. Bobby would also lie on his chest, or settle into his portable terrarium, as the two heated then annealed through the afternoon.

When outside with Bobby, Francis spent a non-trivial amount of his time worrying whether Bobby would die. There were many heavy objects outside, which rarely fell without warning, but it had happened, and an object would not need to be so heavy to hurt Bobby. There were also dogs, which were purpose-built to kill or maim the helpless. Francis had briefly considered a dog before getting Bobby, but as he was careless and sometimes did not leave his home for days at a time, he decided a dog would be a poor fit. He was very happy he had Bobby and would not be happier if he had a dog.

Today was a Thursday, which was sometimes a day off, depending on the schedule The Man gave him, and the park was full of families, which Francis found strange, because children and parents had responsibilities he did not. Bobby saw the children in only the strange spider-like way in which Bobby perceived anything, which is a complete mystery to both Bobby and myself, because I am not a spider nor an arachnologist and so can only rely on what Francis and Bobby have told me, which is little.

But Bobby, if I were to extrapolate, is a very happy spider. Bobby is covered in hairs that quiver when Francis picks him up. Bobby exists for a few basic purposes in life, one of which is to reproduce, and the other of which is to kill. Bobby loves to eat the roaches and grasshoppers tossed into his terrarium, though again “love” is a hard concept to attribute to Bobby. Bobby has strong desires that cause him to behave in certain ways, and Bobby has an experience of these actions but rarely reflects on them. Still, there is something, isn’t there, that makes Bobby act the way he does? And isn’t that something the special connection between Francis and Bobby? Which Francis is in this moment so sure of, as he lifts Bobby onto his palm, raising him toward the tulip tree, under which he’s laid his beach towel.

Francis is sure he was meant to become someone more than the person he is, but he’s unsure who that someone would have been. Francis was not happy in school, nor is he at work. He was happy at home, though he was also depressed at home, and he is happy at the park or walking in the woods, but only briefly, and then he is desperate to return home. Francis felt confused about the person he was supposed to be, but he had never been told who that should be, and he could not figure it out on his own.

If it were possible, I would give Francis a sign. I would, I really would! But Francis is locked into a journey of his own imagination, refusing to admit the little faults that have put him in this position. And while it is true that life has never been fair to Francis, it has also been true he has never put in too much effort, despite the structural advantages that might have allowed him to do much more than he ever did.

Francis is now lying down, trying to convert the warmth of the sun into the will to become the person he was always supposed to be. Bobby, meanwhile, crouches in his terrarium, nearly unconscious, dizzy with extraterrestrial energy.

Francis weaves in and out of sleep as he is obliterated by UV rays, unshaded by the tulip tree in the late afternoon. Francis dreams about his mother, briefly, before imagining his whole body as a gelatin mold. It is a moment in which every consideration feels profound. He, in his half-consciousness, tries to direct his imagination towards more epic thoughts, such as what it would be like to be a tarantula, or a sex icon. But Francis’s inner self bucks at the suggestion, kicking him out of his own unconsciousness, until he gives in, allowing his deepest anxieties to once again direct the frame of motion. Francis, when fully asleep, has his recurring stress dream that he is lost in an auditorium full of toilets.

Tarantulas have 8 eyes and 2 kinds of photoreceptors, which allows Bobby to perceive Francis from a multitude of angles and in colors which are totally inaccessible to humans. Francis has never thought of this, not exactly, but it would only further endear Bobby to him if he had. Bobby sits and does not look at anything, or not with focus. Bobby, as an individual self, barely exists as he sits in his blazing hot plastic cage, nestled in the grass beneath a tulip tree on a late afternoon in May.

Bobby is fine once he is back home in the cool apartment where he and Francis live. Francis is red in the face from having slept in the sun, but Francis has always considered himself swarthy, since his father was Mexican, even if he otherwise looks entirely Caucasian and cannot speak Spanish and has no cultural connection to Mexico, since his father moved to America at a young age and did not talk about his past, or even have a very close connection to his own parents. Francis in this way feels as though he has been disinherited from a culture he had a right to, yet which remains inaccessible, except through cliché tokens of culture, like a tortilla press, which he never uses but leaves on display.

Francis, however, does not like to think about culture and race, because it only makes him angry, as the world is already full of people who do not understand him, and this is one of the many dimensions by which he is not understood. He does, in this respect, among others, greatly envy Bobby, who has never been interpellated, except somehow as a pet, or an enemy.

While Bobby sits alone in his dimly lit terrarium, he has a growing awareness of his surroundings, because at some low level he is aware he must eat. This is not urgent, because Bobby has recently eaten, but he is still aware that he must. This is a dumb, mortal impulse in Bobby, one of the few things that carries him forward. A sensation of what must occur, else everything will end. Bobby’s own struggles are best encapsulated in this tension between the need to eat and his indifference to eating, up until the moment where eating becomes an absolute imperative, which has only occurred rarely, like during one of Francis’s depressive episodes, when even Bobby is beyond his interest.

Francis did not talk to Bobby again that day or even look at him. When he goes to bed, Bobby will have crawled up one of the glass walls of his terrarium, but Francis will accept this as a smear in the phantasmagoria of conscious experience, along with all the other stimuli leading to the perception of his room.

Francis was tired of living every day so mechanically, but he could not find a way out. He ate his dinner on the couch and then spent a very long time looking at his phone. Francis later lay in his bed and wondered why he must be mortal.

I adopted turtles as my colloquial shorthand long before encountering the formal term of infinite regress. This had the unfortunate consequence that I would sometimes end conversations abruptly, shouting furiously about spurious turtles. But I still think the colloquialism has utility, because it allows us to leverage the challenges of infinite regress more broadly, to critique any argument which starts to buckle under a number of recursive applications (even less than infinity).

The presence of turtles might not negate an argument outright, but it ought to compel us to consider the limitations of its explanations.

Inductive reasoning and proofs all eventually hit upon the limits of justification. The Münchhausen Trilemma denotes three problematic yet fundamental types of justification: infinite regress, circular logic and axiomatic belief (or dogma).

Consider a key proposition P, which we seek to justify with three separate arguments, R, U and X. We can model our arguments for P as a directed graph, with each node representing a distinct proposition, and each edge representing an epistemic justification. Though each node may try to provide final justification, further inquiry can always advance us on the graph.

When forming an argument we normally want to avoid infinite regress and circular logic, and strive to ensure our axiomatic beliefs are few and foundational.1

Consider then how we might look for turtles in a current popular idea like simulation theory.

Simulation theory supposes that our world is extremely likely to be computer simulated. The reasoning is that any advanced alien civilization would develop simulation technology, given expected advances in computing, and that simulated worlds would then quickly outnumber real worlds. (And if we don’t live in a simulated world, it’s only because it’s impossible to simulate a world, or because civilizations collapse before it’s achieved.)

Simulation theory causes me to grumble moodily about turtles.

Let’s accept the basic premise that it is extremely likely we are in a simulation. Accepting this premise should lead to the same conclusion that our civilization, or any alien civilizations in our simulation, will also seek to create a simulation. This chain of reasoning continues ad infinitum, until we suppose that there are an infinite number of simulated realities, all infinitely removed from a hypothetical base reality, running our infinite number of realities. This would require our base reality to have a computer with infinite energy, which would seem to violate thermodynamics.2

I would not say that my turtle critique is an outright refutation of simulation theory, but it does seem to imply some critical limitation on simulations which hobbles the argument. For example:

1. We could claim new fundamental physics in the base reality… but basing an argument on an appeal to fundamental physics is like appealing to magic.

2. We could claim that our simulation is unable to generate further simulations, or that the number of simulated worlds is capped… but this undermines the core hypothesis that simulated worlds will vastly outnumber real worlds. If we can never simulate worlds, why suppose that a hypothetical civilization could?

3. We could claim that only our own experience is simulated… but this reduces the argument to a contemporary solipsism.

(Notice the final defense of simulation theory is solipsism, i.e. the fundamental challenge necessitating axiomatic belief—we must choose to believe the external world exists! All claims about external reality at least require an axiomatic belief that external reality exists.)

So simulation theory at least exhibits major cracks when pressed for a recursive application of its own terms. I would anticipate there are more clever strategies to mitigate the damage.

When looking for turtles, we don’t need to search for an infinite recursion which breaks the entire argument. It might be sufficient to demonstrate that a theory gets increasingly “weird” under progressive application.

As mentioned in a previous blog post, I am quite interested in moral obligation. Though I am very sympathetic to the concept, I am nonetheless resistant to maximalist interpretations, i.e. the idea that we are obliged to do the most good possible. Mostly this is because the ideas of “good” and “values” are all tied up, in ways that traditional Christian morality obfuscates, and to strive for a strictly altruistic or “saintly” life is not universalizable.

Effective altruism in its various manifestations is the contemporary movement which comes closest to this altruistic injunction, even if contemporary practitioners might distance themselves from Peter Singer’s original, strong formulation of the idea (i.e., the idea that one should give up to the point where further giving would cause yourself severe suffering). Though I would not attack EA per se (as it seems to me a movement that has done a large amount of good), altruism can exhibit strange behavior under turtling, which may help set some reasonable limits on its application.

I’ll focus my critique on altruism directed at the reduction of suffering—which, if very good, does not tell us very much about what makes for a good life beyond “not suffering.”

Suppose that one has taken Peter Singer’s argument to heart and maximally devotes one’s life to the reduction of suffering. Consider further that one has then taken altruism as a telic activity, meaning we consider altruism itself as a good, rather than a means to an end (e.g., we are living to reduce suffering). Though there is certainly enough suffering in the world for one person to live entirely for altruism, the utility of altruistic activity will decrease for every additional altruistic individual. As more people become altruistic, does the altruist continue to define a good life around altruism? Should the process continue, up until everyone is an altruist, each seeking to minimize each other’s suffering?

Such an end point seems obviously absurd. I do not wish to strawman EA. I would not disagree that there ought to be some equilibrium between altruism and status quo behavior, which likely involves greater altruism than exhibited today.

But I do think this turtling of altruistic principles highlights an important tension between moral obligation and value creation. The utility of altruism is already much lower than it was 20, 50 or 200 years ago. In Bleak House, Dickens satirized his character Mrs. Jellyby for charity directed abroad, when so much impoverishment was visible around her. That critique had some weight in the 19^th century, when domestic suffering could be quite severe, but makes less sense when extreme poverty has been nearly eliminated in the West. Yet the same progress is occurring abroad, as global poverty rates decline. The need for assistance persists, but not to the degree that we need maximal commitment to altruism.

As a counterpoint, keep in mind my critique is of altruism targeting a reduction of suffering. One easy tweak might be to shift from a reduction of suffering to an increase of “joy,” or whatever your chosen measure of a good life. Reduction of suffering is perhaps implicit in an increase of joy. I expect most effective altruists would endorse this goal, even if the discourse tends to center around reduction of suffering and other risks.

Focusing on joy also allows us to endorse atelic activities, which have no formal goal. An example might be spending time with family, or learning the violin, or writing a blog post. Atelic activities are highly resistant to turtles. We do them because they bring us joy, and we axiomatically believe that joy is good.3

Telic activities always have some susceptibility to turtling. If you derive value from climbing a corporate ladder—each rung the successful application of our value system—what happens when you reach the top? If you derive value from AI research, what happens when you’ve built ASI? At its simplest, turtling is nothing more than a technique to follow arguments and belief systems to their logical extremes. Use it!

1

Axiomatic beliefs are often left unaddressed, unless they are strong claims like the existence of god. For example, my belief in the existence of external reality is axiomatic, but I do not feel the need to anchor every essay with an assertion that external reality exists. There is gradation in belief upwards of axiomatic beliefs (for example, I believe in the scientific method, because I believe it has yielded results before and I believe it will again), and we do sometimes insert axiomatic beliefs in lieu of rational arguments (for example, I have a near-axiomatic belief in moral obligation, even if my thinking here remains poorly defined).

Note that the frontier of knowledge is usually expanded by adding justifications (resolving uncertainties), not by replacing axioms. For example:

Gentlemen! What use are empty arguments? You want proof; I propose to experiment on myself whether a man can, of his own will, arrange his own life, or whether each of us has been appointed our fateful moment in advance…

— Lermontov, A Hero of Our Time1

Dwarkesh Patel
And why is [Grok] gonna then care about human consciousness?

Elon Musk
These things are only probabilities, they’re not certainties. So I’m not saying that for sure Grok will do everything, but at least if you try, it’s better than not trying. At least if that’s fundamental to the mission, it’s better than if it’s not fundamental to the mission.

— Dwarkesh Patel and Elon Musk, Dwarkesh Podcast

In the final section of Lermontov’s 1840 novel A Hero of Our Time, “The Fatalist,” an officer in the Russian army points a pistol to his head in a test of fate. The officer, Vulich, has pulled the gun from a cabin wall, so we do not know if it is loaded. He calls for bets. If he survives, then his death was not predestined for this evening. Another officer, Pechorin, calls his bet, wagering against predestination and on his death.

When Vulich pulls the trigger, nothing happens. Yet the gun was loaded. Pointing the gun at an old cap, the pistol finally fires. Surviving thanks to a malfunction, Vulich has made his proof of fate.

In 2023, OpenAI established the superalignment team, meant to ensure artificial superintelligence, or ASI, would remain under human control. For the purposes of this article, we can take ASI to be a strict superset of artificial general intelligence, or AGI, where AI intelligence exceeds that of the most gifted humans. In 2024, the superalignment team was disbanded, as multiple leaders left the organization. In reference to his departure, Jan Leike, now alignment team lead at Anthropic, wrote “safety culture and processes have taken a backseat to shiny products.”

In the quote above with Dwarkesh Patel, Elon Musk asserts that we can only make probabilistic assumptions as to whether ASI will allow human life to continue. He later downplays the importance of AI risk, stating a far more important measure is the total intelligence or consciousness2 in the universe.3

Humanity is incidental, so long as intelligence expands.

After Vulich survives his game of Russian roulette, Pechorin is shocked into a sudden belief in fate. Walking alone afterwards, he grows melancholic. He stares at the stars above, envious of the Homeric heroes4 of old, who believed that the heavens themselves participated in our struggles. He recognizes that modern man has no such luxury of belief.

Pechorin is distraught. Convinced all outcomes are predestined, he fails to find purpose in life.

Roon is the X account for a member of OpenAI’s technical team, known for internet humor and a persistent defense of the industry’s pursuit of AGI.

In 2024 he made a famous defense of the inevitability of AGI, asking us to instead focus on mortal concerns, like family. He has repeated similar claims many times since.

As quickly as he slipped into his reverie, Pechorin is reminded of the danger in dwelling on abstractions. He realizes he has taken Vulich’s supposed proof of fate with the same credulity as his forebears. Rather than focus on “metaphysical” problems, he decides to focus on reality, to look at what is “under his own feet.”

Over the past two decades, statistics has entered mainstream discourse as a rigorous means to dissecting our future. We can now use prediction markets to understand the likelihood of every outcome. We do not see fate exactly as Lermontov imagined it, but we do have probabilities, which risk being treated as predestination by a different name.

Forecasts assert likelihood based on a present reading of events. As we take actions in the world, those probabilities shift. Statistics are a description of reality, but do not determine it. Events are determined by material actors, with individual responsibility for their decisions.

Consider elections, which, especially in a post-Nate Silver world, are anticipated through polls, surveys and predictive modeling. We can assess trends and behavior, can measure the impact of demography, all with the finesse of a science. But predictions are not a substitute for an election. Ultimately, an election is itself the realization of the public’s will. While we might do our best to forecast the result, we are all still individually accountable for our vote.

I worry that brainy, intellectual types—like one might encounter in an AI lab—are particularly prone to mistaking a model of reality for reality itself.

The frontier AI labs of Silicon Valley seem to have become trapped in their own dogged fatalism, convinced they are working toward a single inevitable conclusion. Some are motivated by transhumanist ideals. A few, like Musk, work with a nihilistic indifference toward the survival of mankind. Most point to the manifold benefits of AI, and accept this as proof enough that we must push their models further.

I do not myself doubt the benefits of artificial intelligence. I have written at length about Claude Code and am very familiar with developments in software engineering. I am aware that AI could be used for drug discovery, for climate change, for an automation of work that would allow us to live in leisure.

Though I am concerned about the destabilization that narrow AI (or pre-AGI technology) might cause, I have no doubt we might handle it with the same ingenuity we have handled past technological change. We fear job loss, but after the industrial revolution, we were able to rethink our entire model of the state, creating a welfare system and other safeguards for the unfortunate. Perhaps universal basic income or sovereign wealth funds will serve as equivalents in the age of AI. We fear military use of AI, but the fear of abuse did not stop us from harnessing nuclear power, or carefully deploying nuclear weapons. The risks are real, but I do not think the dangers of today’s AI are insurmountable.

What scares me is that I do not comprehend an equivalent approach for ASI. We cannot manage ASI risk with post-hoc policy, the way we do now with narrow AI, because ASI carries immediate existential risks, either through abuse or misalignment. If we do not have reliable control mechanisms for an ASI, some well-understood framework by which we provide humanity with a good-enough guarantee to our survival, then I do not see any way to safely advance AI until those mechanisms are understood.

Turning back to reality after his musings on fate, Pechorin is startled to learn that Vulich has been murdered by a drunken Cossack. Cornered into an old hut, wielding a pistol and saber, the drunkard now threatens to shoot anyone who approaches him. The hut is surrounded by soldiers, unsure of what to do. As they discuss whether it might be better to just kill him, Pechorin volunteers a plan to catch him alive. Rushing in through a window, Pechorin surprises the drunkard from the side. The drunkard’s shot whizzes past him, but Pechorin seizes him without injury.

Both Pechorin and Vulich risk their lives. Vulich gambles cheaply on his life for the sake of a “metaphysical” argument. Pechorin’s motives are less clear, but we know he risks his life for the sake of other people, sparing other soldiers from his dangerous mission, and sparing the murderer from being shot himself.

Vulich embodies the nihilistic response to fatalism, where the value of our own lives is extinguished by a prevailing sense of inevitability. Pechorin, meanwhile, demonstrates a humanist response. Rather than surrendering to fatalism, he reasserts his agency, in spite of whatever theoretical explanations for his behavior, and chooses to act for the good of humankind.

I am as unsure as anyone on how we avoid AI catastrophe, but I am sure it requires a refutation of fatalism.

PauseAI has a proposal for a general moratorium on the advancement of frontier intelligence. Perhaps the economic incentives are too high to stop now, but we could define intelligence or capability thresholds that we will not exceed until our theories of alignment and control catch up. If there is a safe way to build ASI, then a pause would only be temporary.

My views are not actually very different from what Anthropic themselves have publicly stated about AI risk, referring to a “pessimistic scenario” in which AI safety is unsolvable.

If we’re in a pessimistic scenario… Anthropic’s role will be to provide as much evidence as possible that AI safety techniques cannot prevent serious or catastrophic safety risks from advanced AI, and to sound the alarm so that the world’s institutions can channel collective effort towards preventing the development of dangerous AIs.

— https://www.anthropic.com/news/core-views-on-ai-safety

But Anthropic is not acting alone, and we have seen how existing incentives cause companies to deprioritize safety. What I would like to see is a legal codification of our responsibilities regarding the pessimistic scenario. Premature regulation could unnecessarily stifle innovation. But there is no excuse for stumbling into the worst-case scenarios unknowingly. We need guarantees that all American companies, and ultimately all nations, will act similarly if a pessimistic scenario is detected, and that we will do everything in our power to detect a pessimistic scenario in the first place.

Without such a coordinated approach to mitigate risk, we will make Vulich’s same fatalistic gamble, careening towards the unknown because we have forgotten the reasons not to. We must take the gun down from our heads. Technical possibilities do not dictate our actions. Coordination is harder than engineering, but just as necessary. A safer approach is possible without sacrificing the full benefits of AI.

I want to probe at whether the color blue itself (the “blueness” of blue), and all other conscious experiences (of sound, touch, emotion, etc.), are phenomena distinct from the neurophysiological causes of consciousness. I’ll refer to the phenomena of conscious experience as qualia.

This will be the first of two posts in a short series concerning philosophy of mind and philosophy of science.

In my first post, I will attempt to provide a thorough definition of qualia. I am of the opinion that muddled or aged terminology has led to widespread misconceptions on the nature of consciousness generally and qualia specifically.

My post is exploratory (I am not a scientist; I am not a philosopher). I will avoid making many hard claims as to the nature of consciousness. I will, however, try to clarify the subjects at hand, so that we can have a more meaningful conversation about the unknowns of consciousness. On that basis, I will make a case for qualia as a unifying term for conscious phenomena, even under theories that might otherwise seek to deny qualia. Unless we confirm they are truly illusions, qualia designate an important fact of experience, meaning we should retain the term while debating the definition and underlying explanation.

Throughout our discussion, I will advance a case that qualia ought to be an object of scientific, not philosophical investigation. I’ll go into greater detail on this topic in the second post of this series, where I will critique dualist theories of consciousness.

Qwat are we talking about, anyway?

This problem space is often referred to as “the hard problem of consciousness.” I have noticed philosophy of mind is a very popular subject on Substack, meaning I am of the zeitgeist and possibly unoriginal.

The “easy” problem of consciousness examines the biology underpinning consciousness to understand how neurophysiology operates. The hard problem debates how and why consciousness arises from these biological processes. Qualia are a concept for what arises. What is our subjective experience of reality?

Here are a few simplified examples of subjective experience.

• I observe a blue sky. What is my experience of the color blue? A quale of blue.

• I put my hand to a flame. What is my experience of warmth? A quale of warmth.

• I hear a crash. What is my experience of that loud noise? A quale of sound.

• I taste a cake. What is my experience of that sweet dessert? A quale of taste.

• I feel joy. What is my experience of that joy? A quale of joy.

• I stub my toe. What is my experience of that pain? A quale of pain.

• etc.

Note that I state these to build intuition around qualia, without making substantive claims as to their correctness. Also note that there is nothing ipso facto incompatible between neurophysiology and subjective experience. Either qualia are real, and we ought to seek how to explain them with science, or qualia are illusions, and we ought not to discuss them. Qualia are not incompatible with a physicalist explanation of neurophysiology.

Philosophy of mind is gifted with several well-known thought experiments, which might serve as a further introduction to the problem space. Take these as intuition pumps, not as proofs of their conclusions.

1. Inverted spectrum: Is it logically possible to imagine an exact physical clone of myself who has an inverted experience of color (experience a phenomenon, or quale, of green, rather than red)? If yes, then our experience of consciousness is logically distinct from the neurophysiological perception of stimuli.

2. Philosophical zombies. Can we imagine an exact physical clone of ourselves that does not have conscious experience? A version of myself that is entirely mechanical, with no phenomenological experience. If yes, consciousness is logically distinct from the brain.

3. Mary’s room: Suppose a brilliant color theorist spends her entire life in a black-and-white room. She knows all one can about color within a black-and-white room. When she steps outside, and sees the blue of the sky, does she learn something new? If yes, then there is something about our conscious experience of the world distinct from our physical being.

Of these thought experiments, the inverted spectrum is the one I find most cogent. The colors red and green are chosen because it naively seems they may be inverted without breaking any laws of color theory. It does not mean that red and green actually can be inverted, or that our experience of color actually does differ. It merely highlights a fundamental uncertainty as to the equivalence of phenomenal experience, and doubts such a question could be solved by recourse to the known physical world.

Let’s review the Wikipedia definition of qualia.

In philosophy of mind, qualia (singular: quale) are defined as instances of subjective, conscious experience. […] Examples of qualia include the perceived sensation of pain of a headache, the taste of wine, and the redness of an evening sky.

That’s a good start! But it doesn’t drive toward the important qualification of whether a conscious experience is anything special beyond biological reactions. Can we locate qualia somewhere in our neurophysiology? Are qualia a property of neurophysiology, i.e. some unknown aspect of neural activity? Or are qualia distinct entities caused by neural activity, even a distinct substance with an unknown physical substrate? These are all questions about the ontological status of qualia.

Another angle for investigating qualia is to ask about their causality. Here, I might ask whether our minds are closer to electric kettles or steam engines. Is phenomenal consciousness a side-effect of neurophysiology (the steam boiled off by heat), or somehow essential to the propulsion of the human mind (the steam driving the engine)?1 When we say we are “self-aware,” do we mean we are aware of our qualia, or simply that our neurophysiology can introspect into itself, and qualia remain an outcome outside the causal chain of introspection?

Decisions as to the ontological and causal status of qualia are at least partially determinative of the available explanations for phenomenal consciousness. As a thought experiment, I created a matrix across the two dimensions of ontological and causal status of qualia, and then assessed my own credence for each. This is a subjective exercise meant to showcase my own “best guess” on the available evidence. In practice, theories exist along a spectrum, with different tolerances for the possibility of properties vs entities, and different interpretations of physical vs natural vs non-natural states (not included as a dimension here).

• I accept my subjective experience of consciousness as valid, so I am very skeptical of the idea that qualia are an illusion, i.e. do not exist (left column). If qualia are an illusion, they must be epiphenomenal, meaning they have no causal impact on neurophysiology (the unreal cannot affect the real). I’d associate this box with eliminative materialism or illusionism. Some versions of functionalism would also fit here.

• I accept neurophysiology as the cause for perception and cognition, so I find it very unlikely that consciousness or qualia are the independent causal origin of cognition and perception (top row). I am unsure how to qualify this row as anything other than panpsychism / idealism / mysticism.

• If qualia are a property of neurophysiology (i.e., some as yet unknown aspect of neurophysiology), then I find it much more likely that they are efficacious. Note that I would say qualia are almost certainly not a property of neurophysiology per se, but perhaps a fundamental property of physics or nature (like a new fundamental force). Below, I proposed biological naturalism as the representative theory, but I think you can explode this box to fit several contemporary theories of consciousness. Thomas Nagel probably fits here, as does Searle. I could buy an argument for placing Chalmers here as well.

• If qualia are distinct entities, with a distinct natural substrate (pretend there is a “particle of consciousness”), then I would still venture they are efficacious, though with less certainty than if they are properties. Cartesian dualism might be an extreme version of an entity view, which suggests an entirely different substance to consciousness from physical reality. As mentioned, my matrix does not assess a dualist vs physicalist/naturalist claim for such an entity; my 20% assignment is only for the naturalist version of the argument.

As an intuition pump, I’ve made a best effort to plug in a representative philosophical theory into each box. A few of these could be argued in different directions, so take it with a grain of salt. When pressure testing my matrix against Russellian Monism, or Chalmers’ Dualism, it became apparent a lot of weight was placed on how we used the word “property,” e.g. whether we mean a property of a natural entity (like mass or heat) or a property of reality itself (like, well, an entity?).

I also gave Claude an empty copy of my matrix, with a redacted excerpt from this section, to see where it would place representative philosophers (treating axes as a spectrum) and what 1-2 representative theories it would place in each box. I ran this about 5-7 times with different degrees of redaction (including no excerpt at all) and got a few different results. Several philosophers were extremely stable across runs (Dennett, Churchland, Searle). Chalmers was exceptionally unstable, shifting between 5 of the 6 boxes in the property/entity columns. In the output closest to my own reading, Claude labeled Chalmers’ box as “Panpsychism,” which is a sick burn.

A Problem of Terms

In an earlier draft of this post, I attempted a reductive definition of qualia that simply asserted them as “the entities of the mind that others cannot observe or infer.” I still fall back to this definition when struggling to articulate what exactly subjective experience is, aside from known neurophysiological processes. Regardless of whether you assign qualia any special ontological status, I find it hard not to still recognize the mystery of phenomenal consciousness (unless you insist that phenomenological experience is an illusion).

So, in the spirit of that former reductive approach, let’s proceed through a few of the relevant terms in the causal chain leading from external entities, through neurophysiology, ultimately leading to consciousness and qualia. I’ll provide a concise definition for each, and make a few clarifications or claims to advance our discussion.

Entity

I will use the term “entity” to refer to anything that exists. I will use the qualified term “natural entity” to refer to any entity with a corresponding natural substance, typically understood as physical reality. I will use the term “metaphysical entity” to refer to any non-natural entity. I will use the term “mental entity” when dealing with dualist claims as to non-natural, but also non-metaphysical entities.

For discursive purposes, I will also distinguish entities from properties. I will use “property” to refer to an attribute characterizing an entity (the way an entity is), which property has no natural existence without a dependent entity. An example might be the temperature of water or the mass of a particle.

In my second post, I will explore the implications of these different ontological readings more thoroughly. For now, we should simply allow that such claims exist, and that claiming qualia as a property of neurophysiology or a distinct substance caused by neurophysiology is to claim qualia as natural or mental properties or entities.

While a direct refutation of dualism is outside the scope of my first post, I will still mostly exclude mention of mental properties/entities from my discussion below. For now, I will prioritize positive arguments for a natural interpretation of qualia, rather than negative arguments against a dualist or mental explanation. If you stick to a dualist point of view, then I suspect most of my discussion still holds if you just substitute “natural” with “mental.”

Stimulus

I am going to hazard what might be a heterodox definition for stimuli.

I will use the term “stimulus” to refer to any natural entity one would place in the direct chain of causation leading to a change in consciousness. That includes external stimuli, like the light shining from a bright blue sky, or the rustle of leaves. But it also includes our eyes and our brain.

Many stimuli are external. Our sensory organs transduce external stimuli into the neural activity begetting conscious experiences such as vision. So it is with sight, hearing, touch, taste, etc. The five classic senses have a clean internal/external differentiation. But what about emotions? Memory? I may have a feeling of joy staring at a bright blue sky, which has a close causal relationship with the sky. But I can also feel pleasure at the end of a long chain of cognition, or after a nice dream. I may feel joy or sadness at the end of somatic processes outside my control. I may hallucinate from sleep deprivation or a mental illness. How do we understand the stimuli leading to our individual thoughts? Are they not, in many respects, the operation of neurophysiology itself? If my consciousness changes as I mentally compose my argument, how would you locate an external stimulus for that change?

Our brain constantly intakes external stimuli, and we may, after much searching, find an external provocation for many of our thoughts and feelings. But we would be hard-pressed to find a direct, external corollary for every change in consciousness. From the perspective of consciousness, why differentiate between an internal and external stimulus? It is all just happenings in the natural world that may or may not lead to a change in consciousness.

Neurophysiology

I will use the term “neurophysiology” to refer to the broad set of somatic processes that are directly responsible for consciousness, i.e. that enable consciousness in animals.

My assumption is that neurophysiology is a direct and sufficient cause for the appearance of human consciousness. I am not aware of any scientific evidence for another physical cause. I refer to “neurophysiology” rather than the brain, so we do not need to debate the relative role of the sympathetic vs parasympathetic nervous system, etc.

I will not make any claims as to why neurophysiology causes the appearance of conscious experience (qualia). Nor will I make a strong claim as to whether neurophysiology is the only possible cause of consciousness. Given conscious experience includes senses like sight that do not require high-order thinking, I assume other animals have conscious experience too. Though it’s unclear how far down the chain that goes—at least through the vertebrates? Probably further?

My best guess is that there is something about the exact nature of neurophysiology which causes consciousness, meaning a non-biological artificial intelligence is unlikely to experience it. However, this is only philosophical speculation! The core thesis behind this series is that we ought to treat the “hard problem of consciousness” as a scientific problem. We simply do not know whether an AI does or could have consciousness. Nevertheless, I am very skeptical of strong functionalist claims to consciousness, as in the Chinese nation problem (which is a thought experiment, not a national security policy).

As asserted in the previous definition, some subset of neurophysiological processes are also stimuli, or can be stimuli. Introspection seems to allow a surprisingly large number of processes to become stimuli for consciousness (consider a meditation on breath). Homeostasis would be a good example of a process that is not a stimulus for consciousness, though we notice the consequences when it fails.

NB: I would like to take this moment to remind you I am neither a doctor, nor a psychologist, nor a scientist, nor a philosopher, but just Some Dude. If you think my definitions are inaccurate, or stupid, please let me know, and I will adjust.2

Leave a comment

Consciousness

I will use the term “consciousness” to refer to the entire set of qualia available to an individual at any given point in time, fundamentally constrained by an individual’s underlying neurophysiology. This is often referred to as phenomenal consciousness. But isn’t non-phenomenal consciousness a contradiction? Especially if I denote cognition and feeling as activities of neurophysiology, not consciousness.

My definitions ask neurophysiology and qualia to do most of the heavy lifting of consciousness. If we posit that neurophysiology is solely responsible for the appearance of consciousness, and that the experience of consciousness is simply the set of all present qualia, then what is left to consciousness itself? All the old machinery of consciousness has shifted elsewhere.

This is what I mean when I assert that muddled terminology has led to misguided discussions around consciousness. Consciousness, as a concept, is still burdened with unscientific readings of mental processes that precede modern neuroscience. It seems to me that the vast majority of the mind can be understood mechanistically through neurophysiology, whether that be cognition, feeling, memory, etc. Phenomenology is the hold-out.

That being said, an open question in my own thinking on this topic is whether consciousness offers any unique role in introspection and awareness, or in the apperceptive self3. Though I take a highly mechanistic view of cognition, I still have the impression that I am a single self, and that my self actively experiences qualia. Perhaps the simplest explanation would be to say that my impression is false; actually, there is no self apart from my experience of consciousness. My “self” is simply the qualia I experience in a given moment, alongside the machinations of my body.4

Ricardo Manzotti provides a related set of conclusions with his concept of the “spread mind,” which conflates the self with our experience of the world.

However, I do not think we can make any strong claims as to the apperceptive self, until we have answers for the hard problem of consciousness, i.e. why consciousness arises at all. I have so far defined my terms under the assumption that the process from neurophysiology to consciousness, particularly conscious experience, is unidirectional. But can the causal chain operate in the other direction, from qualia to neurophysiology?

This does not necessarily mean that consciousness “controls” neurophysiology (very unlikely), but that consciousness can sit somewhere in the causal chain, even if that just means that neurophysiology can “perceive” qualia. Minimally defined, this could be a process like: no qualia → “neuron of conscious experience” set to 0 → qualia experienced → “neuron of conscious experience” set to 1.

Taking a physicalist, scientific perspective, a naive first reading of this problem might say it is obvious that qualia cannot effect neurological change. But this leads to a few strange conclusions. The first of these is that we must then take an epiphenomenal view of consciousness. Consciousness is then merely a by-product of neurophysiological activity, like the steam boiled off our electric kettle.

To make an epiphenomenal view consistent with a causal understanding of neurophysiology, we must then either accept conscious experience as an illusion (weird), or, that our self is exclusively our conscious experience (we are qualia; also weird). If qualia cannot interact with neurophysiological processes, then there is no “bridge” to connect our mind and body. Perception and self-awareness are only neurological processes working against themselves. Our neurophysiology cannot perceive qualia, it just processes stimuli.

Stated as a set of logical propositions (proof by contradiction):

1. I assert that cognition is fully explained by neurophysiology

2. I claim to perceive qualia

3. However, I also claim that qualia cannot cause any change in my neurophysiology

4. If qualia cannot cause any change in neurophysiology, and cognition is just neurophysiological processes, then qualia cannot provoke any cognition

5. Hence, I do not perceive qualia

Chalmers refers to this as the “Paradox of Phenomenal Judgment.”5 I am unsure whether novel science could circumvent such a proof. Suppose that consciousness interacts with neurophysiology only via forces that have no causal impact. Maybe a weak gravitational force? Or a novel physical force? This is also very strange, and I’m unsure how it fits with quantum field theory. The very weirdest version of this would be something like monads (ontological entity, epiphenomenal), where consciousness is some unknown natural or mental substance existing in parallel to our neurophysiology. I suppose someone could make a clever argument for qualia as epiphenomenal properties that could reduce the amount of novel science required for a natural explanation?

Introducing a causal nature to qualia also has weird consequences for science (what could be the natural entity intervening in neural activity?), but allows for more “common sense” readings of conscious experience. If we allow a causative role for qualia, then we no longer have to tie ourselves in knots justifying subjective experience. This is intuitive, and though intuition is not an indicator of what is real, I take it as a useful starting point for an investigation of the unknown.

To this end, for any scientific readers in my audience who are tired of philosophy and think this is all a load of kvatsh, I would like to offer the causally efficacious, ontological property reading of qualia.6 Rather than having to posit unique physical substances, this would allow us to discuss qualia as not yet understood properties of neurophysiological activity—as constitutive of neurophysiology as, say, the wave form of a particle. You would also eliminate the problem of epiphenomenalism by identifying qualia as inherent to the causal process itself. This would move us closer to the “steam engine” model of the mind, where consciousness (“steam”) is constituent to the overall mechanism, not incidental.

It would in many respects be simpler if solving the hard problem of consciousness meant discovering new physical properties of neurophysiology or neurophysiological activity, rather than having to discover an entire new class of natural entities (a substance), or having to explain unidirectional causality from neurophysiology to conscious entities. But the validity of each is still unknown.

NB: I am not very familiar with Integrated information theory or Global workspace theory, which by my understanding are two of the leading scientific theories of consciousness? At a first reading, neither seems to do a very good job of providing a physical/naturalist explanation for qualia (though IIT does better?). To the best of my knowledge, both are still largely theoretical, without any decisive empirical backing. They are also not without their detractors. I am just very skeptical that any full explanation could be found without novel physics. As an example, Dwarkesh Patel recently linked to this talk by Max Hodak. Whether or not the specifics are correct, this seems to me like a better approach. I’ll speak more to these ideas in the second post.

Qualia

I will use the term “qualia” to refer to subjective experience itself, which is the uniquely apprehended property/entity that is both ineffable and private.7 Qualia are distinct from cognition and other processes which evaluate our experience.

Qualia, by this definition, share similarities with noumena, quiddities, dasein, and other terms pointing to entities that are fundamentally unknowable or inaccessible. The miraculous difference with qualia is that they are apprehended! Perhaps too well? A philosopher is more likely to have to fend off sophomoric claims of idealism8 than to have to argue whether we apprehend qualia.

We can communicate about the subjects of conscious experience, but the experience itself is untransferable. We have no access to conscious experience from outside an individual subject. The subject has no ability to transfer their experience to us (I cannot share my experience of the color blue). Yet we still have knowledge of this experience within the privacy of our mind.

I do make an important distinction between qualia and our evaluation of experience.

Suppose, for example, I take a sip of a delicious cup of coffee (like Dennett’s Chase and Sanborn). I have a clear subjective experience of coffee, maybe even of a specific roast, and a clear subjective assessment of the coffee as good. Does that mean there is a quale of coffee? Or a quale of good-coffee? Or a quale of good-brazilian-roast-coffee? This seems very strange! It would be hard to square a naturalist/physicalist interpretation of reality with such specific kinds of qualia.

However, I would not assert any a priori need to distinguish kinds of qualia. Perhaps there is only a single kind of quale, and discernment is only a function of neurophysiology. When sipping our coffee, our experience is of the neurophysiological activity associated with tasting coffee, and the related neurophysiology for a provoked feeling of pleasure. For both, there is only a single kind of experience. All knowledge and judgment of those experiences exists within the brain.

I am thus open to a reading of qualia as illusion in this narrow framing, where we accept qualia themselves as legitimate, but not any meaningful differentiation between qualia. Personally, I find the weirdest consequences of this treatment to be color. If we claim qualia are undifferentiated, then we must also assert that there is no fundamental differentiation between our experience of blue and red, green and blue, etc., only our evaluative judgment separating the two. This seems almost as odd as a quale of Brazilian coffee (QoBC?).

The two arguments above explain why I have come to lean towards a view of qualia as properties of neurophysiology, rather than distinct entities. If the experience of “red” is an unknown set of natural properties for a neurophysiological perception of light, well, then in some sense the perception is the color. We don’t have to make counter-intuitive claims about our own experience. The experience itself is differentiated, because it is strongly correlated with differentiated neurophysiological processes.

While a property view does not ipso facto restrict consciousness to neurophysiology, it does present a burden to prove consciousness in entities of a different natural structure. Artificial intelligence is the best example of this. If you query an AI on its consciousness, or read the reports of others having done so, you will not ascertain whether it has phenomenological experience. I am sure we can build AIs that match and exceed all our cognitive capabilities. But without the right physical substrate, it is unclear why qualia would arise? Asserting a strong functionalist claim for the consciousness of artificial intelligence is thus highly problematic. I’m unsure how you could stick to an understanding of consciousness as a property, or even as efficacious, while assuming AI has consciousness. I would ask representatives of this view to assess whether they have not accidentally proposed a monadological theory of consciousness.

Thought Experiments

I have a small grudge against thought experiments / intuition pumps, because they are sometimes abused to make arguments about definitions rather than arguments about truth; or to make metaphysical arguments when we want to make physical/natural arguments. However, it would be remiss to write a post about qualia without engaging with the many thought experiments written in this domain. I will evaluate several according to my definitions and personal views, and leave it to the reader to decide whether I am better or worse off for the outcomes. Rather than introduce each experiment, I will link to a relevant source for anyone unfamiliar with the terrain.

Inverted Spectrum

Humans have three types of cone cells, which leads to our trichromatic perception of color. Some animals have more (goldfish are tetrachromats; pigeons are pentachromats), meaning richer understanding of color, and very likely perception of fundamentally new colors that we cannot comprehend with our physiology. This is a good example of what we mean when we say that neurophysiology set the bounds on consciousness. What would it mean to have a fourth cone that perceives the “color” of ultraviolet light, as a goldfish might? What if ultraviolet light was so important to a species that it evolved two additional cones dedicated just to that electromagnetic spectrum? Is that possible? Is there a hard limit on the divisibility of light into more colors?

I bring this up simply to say that we already have strong physical evidence that other animals can perceive more fundamental colors than us, meaning we already know there are qualia of color we do not experience, and that we cannot comprehend. If there are other fundamental colors, then who is to say that our colors might not already be swapped, or already be fundamentally different, person to person? We know there are more. What if there is an infinite number of fundamental colors that vary, person to person?

So, I find no contradiction in the general argument of the inverted spectrum hypothesis, namely that it is conceivable that qualia differ between persons. I do, however, find it very unlikely. It seems much more likely to me that the specifics of our visual neurophysiology leads to the specific qualia we experience, and that without a different physical substrate the qualia themselves will not differ. It’s also unclear to me if one could actually neatly invert red and green with no changes to our perception of light and color?

Conclusion: Qualia, as represented by color, may not be equivalent across consciousnesses. But it’s unlikely to be such, following a property understanding of qualia.

Mary’s Room

My ungenerous interpretation of Mary’s room is that it is a nearly useless thought experiment.9 It does demonstrate that we have subjective experiences distinct from our knowledge of the underlying stimuli. However, it does nothing to advance an anti-physicalist argument, aside from demonstrating that anti-physicalist arguments typically fail to anticipate novel physics.

To quote Thomas Nagel, in his famous “What Is It Like to Be a Bat?”:

At the present time the status of physicalism is similar to that which the hypothesis that matter is energy would have had if uttered by a pre-Socratic philosopher. We do not have the beginnings of a conception of how it might be true.

Leaving the room and perceiving color for the first time, Mary’s neurophysiology enters novel physical states which either have a unique natural property, or cause a unique natural entity, corresponding to a quale. This does not refute physicalism, but merely points toward a significant unknown in current physical models.

Conclusion: Subjective experience is distinct from knowledge of physical facts. Qualia may still be physical.

Zombies

By our definitions, the conceivability of zombies hinges on the causal problem of consciousness. If phenomenological consciousness does somehow have a causal effect on neurophysiology (i.e., if there is just one binary neuron that “flips” based on the presence/absence of qualia), then a perfect zombie is not possible. Otherwise a philosophical zombie should be conceivable, by our mechanistic reading of neurophysiology. If phenomenal consciousness plays no causal role in neurophysiology, then zombies are conceivable (even if they may be still be impossible, i.e. something inherent in neurophysiology causes consciousness).

Chalmers comes to the same conclusion on the logical possibility of zombies, which is one in a few steps he takes toward a “naturalistic dualist” interpretation of consciousness.10 However, while I agree that conceivability may point toward metaphysical possibilities, it does not actually determine natural reality—as Chalmers himself states, “logical possibility and natural possibility are different things.”11 Though it is metaphysically possible for my hair to turn green tomorrow, it does not mean it actually will. As I have stated repeatedly, we simply do not know the actual causes of consciousness, or the nature of qualia, so it is impossible to make any definitive claim on whether a zombie really could exist.

Conclusion: Zombies are conceivable. But it seems very unlikely a human zombie is actually possible.

What Is It Like to Be a Bat?

The “what it’s like” to be a bat is to have the qualia available to a bat, with a bat’s limitations to consciousness. Presumably bats, like people, also have an apperceptive self.

We haven’t taken a hard stance on whether qualia fundamentally differ between stimuli—whether there is material difference between qualia of sight and qualia of sound, or if this is merely an evaluative function of neurophysiology. Regardless, we would expect bats to have qualia of “sonar” differing from our qualia of sight, due to a capacity to process a type of external stimuli differently from our neurophysiology (different sensory organs and different neurophysiology). What is unclear is whether the experience of sonar corresponds to a novel property/entity (a unique “quale” of sonar), or simply an evaluative capacity of a bat’s brain.

Following my mechanistic understanding of cognition, I am less fussed by what it would mean to “think” like a bat. To think like a bat is merely to experience a bat’s cognition. Whatever neural activity provokes consciousness in a bat would cause a correlating, bat-like experience.

Conclusion: There is a what it’s like to be a bat, though we don’t know if the experience of being a bat corresponds to any novel qualia (the parameters of consciousness change, but maybe not the properties/entities).

Chinese Room

I have the impression that the Chinese Room plays an outsized role in discourses around artificial intelligence. It’s a good thought experiment! Searle insists on the necessity of a biological foundation of consciousness more strongly than I do, but I would still agree it is very likely.

Whatever else intentionality is, it is a biological phenomenon, and it is as likely to be as causally dependent on the specific biochemistry of its origins as lactation, photosynthesis, or any other biological phenomena. No one would suppose that we could produce milk and sugar by running a computer simulation of the formal sequences in lactation and photosynthesis, but where the mind is concerned many people are willing to believe in such a miracle because of a deep and abiding dualism: the mind they suppose is a matter of formal processes and is independent of quite specific material causes in the way that milk and sugar are not.

— John R. Searle, “Minds, brains, and programs” (1980). Emphasis my own.

My main objection to Searle’s argument would be his use of the term “intentionality.” It’s a good example of why we ought to stick to qualia when seeking to differentiate consciousness from the mechanics of thought, whether that be in neurophysiology or neural networks. Where does one locate intention in the human mind?

As I have already mentioned, I do not yet have a clear opinion on where we should locate the apperceptive self. I am unclear whether concepts like perception and self-awareness and intentionality are intrinsic to consciousness, or merely post-hoc justifications for neurophysiological processes, which sit alongside the experience of qualia. Using terms like “intentionality” brings on claims I do not wish to make. I only want to assert that qualia are distinct entities, which we and other animals possess, and very likely artificial intelligence (as it exists today) does not. I don’t want to trip up that claim by arguing what counts as intentionality in biological or artificial “brains.”

Conclusion: Agree that our best guess ought to be that AI does not have qualia. Quibble with the use of the term “intentionality,” which AI could possess, depending on the definition.

Chinese Nation

Why are there multiple thought experiments set in China? Probably this is some form of mid-century Orientalism? What of the consciousness of the French Nation, among others?

Regardless, it seems to me that only a panpsychist or hard functionalist interpretation of qualia would support consciousness in this experiment. If one supposes qualia are actually ubiquitous physical properties of nature (an extreme version of our efficacious-property view), then I suppose you could still assert qualia would arise, but I don’t see a basis for arguing a unified consciousness / apperceptive self. I won’t make the effort to steelman this final experiment, but I’m happy to hear better arguments for it.

Conclusion: No consciousness, unless you take a hard functionalism or panpsychist approach.

Experience the Qualia of My Conclusion

I lightly workshopped this post before sharing externally, but the opinions here mostly reflect my own thinking and exploration of the subject. I will admit that I began this essay with a strong belief in an entity-oriented understanding of consciousness. But I found myself backed into a corner once I recognized the causal difficulties of that view, and then switched to the efficacious-property hypothesis as the most likely explanation. I’ve only assigned a 45% likelihood on my personal “scorecard” above, but this reflects a more fundamental uncertainty about the problem than a doubt to my reasoning.

Ultimately, what qualia might be is a question for science. Maybe there are “particles” of consciousness—who knows! My purpose has only been to explore the likely philosophical explanations for qualia, and in doing so to assert that they are real, and worthy of targeted investigation.

When I read discussions of consciousness online (particularly in our post-AI age), I see far too much emphasis placed on questions that I believe ought to be understood mechanically, metaphysically, or some mixture of the two (i.e. intentionality, knowledge, agency), but that have no bearing on consciousness per se. Because without phenomenal experience, there is no consciousness! Without qualia, we are just making metaphysical arguments about the behavior of intelligent beings (except for apprehension?12).

In my second post, I’ll continue this discussion of qualia, but shift attention to the ontological and causal definitions we’ve allowed ourselves to use loosely until now. In particular, I expect to defend the preservation of metaphysics as a concept, while rejecting non-naturalist or non-physical explanations for qualia and other unknowns. I’ll also explore further the implications of the private nature of qualia, specifically what it implies for science if we know there is at least one apprehended property/entity that we cannot otherwise observe.

If you have any objections to the above, please let me know, and I will publish an addendum alongside my second post in the series.

I have often regretted that I write poetry. Some creative pursuits, like painting or guitar, invite the envy of your social circle. Everyone is impressed if you can draw a horse. Poetry, however, mostly encourages friends to ask if you are doing OK, with a reminder to call them if you need to talk. Nobody is excited to hear you rhyme.

Poetry is just words, and it’s never clear what makes some words nicer than others. If I like the way a word fits in my mouth, and if I put several nice-fitting words together so that they express my thoughts and feelings, well, that seems “true” to me. That truthiness is what makes poetry intimate, and awkward.

Subscribe now

I write a lot of prose poetry. In fact, I write a prose poem every2 day. Prose poetry has the obvious advantage that it allows the reader to conceal the fact that they are reading a poem. It is the perfect poetic delivery vehicle for an audience trapped in a public space. If you squint, you might even convince yourself you’re reading microfiction. Wouldn’t that be nice?

More (16/360)

Life takes many forms. It began with genesis. Ever since, we’ve had to copulate. I would hate sex if I were a plant. You place a lot of faith in bees. Vertebrates were the first to ask the big questions. What is love? What is beauty? Our ancestors were killed by asteroids on multiple occasions. One of them created Texas. When a politician presents himself before a camera, that too is life. I have seen the fauna in the White House. They are asking different questions. On Wednesdays I go to the gym with a friend. 70% of the energy spent in exercise is lost to heat. Life is not a lossless form. My friend’s name is Carl, he is that forgettable. I have increased my bench press by 5 kilograms in 2 years. I did not evolve to gain muscle. Neither did Carl. Carl catches me when the weight’s too much. We are that familiar. Outside the gym, I trap the heat radiating off my muscles in a down jacket. Geese are not as lossy as people. A Canada goose might migrate up to 5000 kilometers each way. Life is full of these kinds of facts. A democracy might last exactly 250 years. Because winters are mild now, I can open my jacket, just a little. History created me, but it also created him. Both our lives were accidents of well heated water.

Configurations (18/360)

The story I’m telling is one of patience and fortitude. The story features nine principal characters. One of the characters is a small boy with brown hair who is metaphorically linked with a piebald colt for the first seven chapters, but by the middle of the book has lost himself entirely, and the horse is dead. All the characters live on a large estate. After a character leaves the estate, I can no longer include them in the story. When the boy left he was lost for a long time. But he returned as a handsome man with a chestnut goatee and a toothy smile. He wore yellow socks as he trod up the lawn—if you read carefully, you would notice the allusion to a horse’s fetlocks. Above the fireplace hung a painting of his great-grandfather. One of several dogs lifted his head. The fireplace is purely ornamental. The estate is covered in palm trees and agave. Of the eight other characters, three are on the property when he arrives. One is habitually drunk. Two are not yet introduced. Nominally the story is about family, but critics differ as to the deeper meaning. The ninth character is an adolescent when the story ends. She has not yet left the estate, but we understand her departure will conclude the story. With a valise in hand, she glances at a painting of her grandfather, whom she surmises was horselike. Many great stories reflect on the rise and fall of a household through subsequent generations. The granddaughter will have only read enough to know a house can fall. Outside the estate the world is ill-defined. So many details are still missing. When I close my eyes, I see slabs of limestone settle onto peat. The story opens with a large garden party, on a vast terrace, the conversation full of veiled hints as to character and plot.

A Great Man (12/360)

If I were a genius, then every poem I wrote would be great, even this one, because my life would always be full of intellect and beauty. If tired, I would articulate my fatigue with great license, and teachers would recount to children the efforts I made to lift a single apricot to my lips, as my fingers dawdled on tarnished silverware. My poems would have no resemblance to lists, slave to the plodding logic of clauses, but resemble a flower unfurled in late spring, or a secret whispered into a cupboard. I would question myself, yes, but not really. My whims would be the extension of a profound order, always present and misunderstood. If I were a genius, you would stand on my shoulders, surveying a land that I had first charted. I would do nothing in parts. My every action would be the full expression of the Weltgeist, as it existed beyond me and through me. There would be no crowds, but only attitudes, each open to my interpretation. If I were already listening, a poem might tell itself to me. The invitation is always available. The self that I would have might have been but never was would, occasionally, come to mind. I would reflect with deep fear at the thought of him (me), then twirl my phone through my fingers, puzzling how to write him down into words. If the genius I could have been only knew how much I would have admired him! I would put it down into words, too, if I could.

Leave a comment

Concerning My Addiction to Vibes

I avoided spending too much time with existing SDD or orchestration libraries ahead of last week’s post. This strategy appears to have worked for Stephenie Meyer, so why not me? Since then I’ve played with several popular repositories—by and large, I’ve been disappointed.

Everyone is grasping toward the same concepts. But so many of these libraries feel like a given developer’s personal preferences vibe-coded into an arbitrary and poorly documented new interface. Spec-kit is basically a templating engine for PRDs, yet has managed to accumulate 60,000 stars on GitHub? I suppose this could be nice for anyone who has never had to write a Jira story in their life… but for an experienced developer, why bother? OpenSpec and BMAD are similar. It’s the engineering equivalent of automating an Atlassian how-to guide. If you need the convenience methods, just write a custom Claude command to draft with your preferred template.

I feel similarly about beads and gas town. I only recently set up gas town on an EC2 instance (my personal Nevada test site), so I’ll withhold from commenting on its efficacy, aside from the popular remark that the documentation is bizarre. I assumed beads could be applied more easily to my existing workflow, but in practice felt again like an individual’s convoluted, idiosyncratic workflow being hoisted into a library to replace… what? A ticketing system? I am often worried that software engineers, when given superpowers and the freedom to change the world, would only choose to rewrite the entire Atlassian suite according to their personal quirks.

From a research perspective, it’s valuable to explore and build in this direction, but for production use I would approach new tools and workflows with significant skepticism. Ultimately, Anthropic is in the best position to make any new SDD or orchestration workflows effective, by building them into Claude Code itself. Over the past week, we have already seen the release of tasks, which are directly influenced by beads, and have already received early reports of new swarm, delegation and team coordination features. Should we spend cycles building more orchestration libraries when native support is just around the corner?

Despite my specific advice to wait for orchestration features, it remains very difficult to calibrate when to build your own tooling, when to leverage libraries and when to wait for native capabilities. I still spent a recent afternoon setting up an EC2 sandbox for gas town and other High Risk Activities, instead of using sprite.dev or an equivalent service. And since I disliked beads and have yet to see the benefits of Claude’s new task system, I decided to write my own simple ticketing system (take that, Jira). The cost of producing software is now so low that small frictions in developer experience or minor SaaS fee considerations are enough to justify a couple hours of agentic development. Tooling at the frontier is inherently imperfect, so the marginal cost of writing your own software may be less than the cost of understanding a new solution.

Putting Claude to Work: Labor and Capital Allocation

Beyond an inflection point, the marginal cost of further automating software production actually increases, and is eventually infeasible at current levels of model intelligence.

When building a factory, planners must decide on the optimal ratio of labor (people) to capital (machines) for production. Even if technically possible to fully automate production, the marginal cost of fully automating production is so high that it is remains more efficient to only partially automate production, retaining labor for activities where humans hold a comparative advantage over machines. This concept may be counterintuitive for engineers that have a drive to automate as many tasks as possible.

We can apply the same framework of labor-capital splits to software production. Here, labor refers to the activities of all people involved producing software, most notably software engineers. I’ll use capital to mean any non-labor asset used in the production of software. For simplicity’s sake I won’t distinguish between a company’s own capital assets and the assets they rent. So I consider Claude Code capital, though it would appear as an operating expense on a balance sheet, not a capital investment.

Software development is traditionally a labor intensive activity, even if the output of development, software, is a capital asset that can be leveraged across a huge consumer base. Producing software has long involved teams of product managers, quality assurance testers, support agents, designers and, of course, programmers. That is not to say that the labor mix in software production has been static, or that software development has not become more efficient over time. The past decades saw increasing levels of automation for software operations, mostly thanks to the advent of cloud computing and the associated DevOps and SRE practices, which eliminated/reduced many traditional IT and ops positions (like database administrators). Yet labor has remained essential for the actual production of software.

Just as the mechanical loom in the Industrial Revolution shifted production away from textile workers, Claude Code in the AI revolution allows us to shift software production from labor (programmers and the rest) to capital (rented or otherwise).

Epochal shifts in modes of production do not complete overnight. Capital and labor are not binaries, but a ratio that a company must effectively calibrate to optimize their balance sheet. Considering the automation of software operations, the cloud did not immediately eliminate traditional servers.2 Advances in virtual machines and containerization slowly spread throughout the software community, while new roles and practices gradually evolved to capitalize3 on those changes. Even today, when serverless technology is broadly adopted, we have not unified around a single kind of compute or storage layer, but make tradeoffs in how thoroughly we choose to adopt serverless abstractions. We should consider agentic development in the same light—not as a binary, but as a series tradeoffs as to the degree of automation desirable.

So what is the optimal degree of software automation today? I argue that the optimal labor-capital split is a function of model intelligence, where intelligence is a catch-all term for reasoning, instruction compliance, etc. Beyond this ratio, there are no marginal gains in efficiency, and companies will experience diminishing returns on further automation of software production. Here’s an illustrative chart of the phenomenon.

Remember that “software production” refers to the entire set of activities necessary to produce software, not just writing code.

Please don’t anchor too hard on the specific numbers I’ve proposed, which are more vibes than rigour. I’ve calibrated my guess on the Stanford study citing productivity gains of around 10 - 40% depending on the complexity of the task (the study precedes the latest coding models). The initial productivity gains of agentic coding are strong (consider the automation of rote activities and greenfield projects), but tapers off into a wide trough of mildly differentiated gains. This is where many software developers are spending their time today, experimenting with different agentic coding strategies. The marginal costs are similar in the region around the optimum, so there is only a small penalty for automating a little too far. But further on the costs increase asymptotically—you will hit a wall if you try to automate your entire workflow.

Subscribe now

Today’s models are great at one-shotting discrete tasks, but still fall over when faced with complexity and expanding requirements. It is well understood that filling the context window rapidly reduces the quality of current models. But, in the context of software development, I would add that even minor, seemingly harmless deviations from instructions and design principles cascade into core technical problems. I naively reason about this as an issue of compounding errors. Suppose Claude Code is 5% “off” expectations for every changeset, meaning it very slightly deviates from requirements or design principles. Without manual feedback or review, Claude will continue to deviate your codebase by 5% for every subsequent change. After 5 iterations you would then have deviated 28% from expectations; after 10 iterations, you would be 62% off. In practice error rates probably do not compound so neatly, but it’s true that without human supervision vibe-coding comes to halt under the weight of its own technical debt. Agentic coding continually adds entropy to the codebase. So small gains in adherence and reasoning can have massive gains in the number of change sets feasible without human intervention.

The relative cost of automating software production is sensitive to the fees charged by Anthropic and its competitors. We are all the beneficiaries of a competitive market with large subsidies. I did not find any public numbers confirming the “true costs” of a typical Claude Max plan, or any company’s real per-token cost for a SOTA reasoning model. We do have reports that in 2025 Anthropic suffered a $5.2 billion loss against $9 billion in ARR, while OpenAI expected to spend $22 billion against $13 billion in sales. With those losses it seems reasonable to expect that “real costs” for Claude Code users are at least 2x what we pay, though if we further assume that a small percentage of Claude Code users consume the vast majority of tokens, then it’s likely the real costs for Claude Code users are much greater. However, I’d guess there are few hard technical limits on optimizing the inference layer of any given model, so it’s likely Anthropic and competitors could engineer away some of the cost problems if they were compelled to chase profitability. When making critical budget and staffing decisions, engineers and business leaders would be prudent to just keep in mind these costs might increase someday to better reflect real costs.

I’ve charted the labor-capital split relative to a pre-AI baseline of current software output (say, late 2023). The bull case for the labor market is that overall software output will grow apace with the new efficiencies. The low cost of software production creates competitive pressure to add new feature, develop new products, and automate increasing amounts of the economy. So even if fewer software engineers are needed relative to overall software output, the absolute amount of software produced increases so much that the actual number of software engineers employed remains constant. As long as engineers, or humans generally, retain some relative advantage to AI, then there is no economic incentive toward full automation, and the labor market could remain steady.

I fear the bull case may prove to be technically true and substantively false.

Firstly, there is an inherent lag between the release of a new model, subsequent development of new automation techniques, and identification of the new optima. The continued rapid pace of change and various incentives to manage costs mean that many (most) companies will over- or underestimate the amount of labor and capital investment necessary to reach the optimal labor-capital split for a given level of model intelligence. An important takeaway from my weeks experimenting with open source libraries and other tools is that it is very unlikely that generic solutions can substitute for a company’s unique needs when working at the frontier of capabilities, as the solutions are incomplete or incorrectly generalized. Even if the initial gains are achieved through Claude Code (rented capital), reaching an optimal labor-capital ratio will require a lot of bespoke development. Executives that fail to understand this problem implement hiring freezes, premature layoffs and generally fail to seize productivity gains—or its opposite, with excess investment in AI automation and failure to reallocate labor according to the comparative advantage of humans relative to AI.

I am particularly concerned that most executives will interpret AI as an opportunity to shift resourcing away from engineering toward traditional business functions like finance, marketing, etc., when the most efficient use of labor may actually be to have engineering automate internal business processes and move many traditional roles toward new, AI-enhanced positions.

The second risk I see is the loss of marginal utility of software. It may be that we have a near infinite number of processes to automate, so the long term risk is small. But, as with the efficient allocation of labor, the real bottleneck is our own ability to identify new applications of software as marginal costs drop and capabilities increase. Up until now, it seems to me we have seen very few practical innovations using generative AI capabilities outside the model’s own chat interfaces and software development. While these advances alone are huge, it demonstrates that we are already struggling today to invent products on top of the current frontier, let alone the frontier of tomorrow.

The third problem, and the most profound for our lifetimes, is that we have no guarantee that humans will retain any comparative advantage over AI. It is of course comforting to cling to this idea, and we could feasibly retain important comparative advantages for years and years to come. But if we extrapolate into the coming decades, I would anticipate we reach a point where humans have no meaningful comparative advantage over AI and are no longer needed in the production of software or any other asset.

I Love my Claude, I Named Him Bob

The history of cloud computing is sometimes discussed by the zoological terms we used along the way.

Servers were once like pets. They had names and you managed the lifecycle of a server very carefully. If your server died, it was very sad, and you probably had a funeral or maybe you were fired. Eventually, though, we got good at virtual machines. Virtual machines allowed us to create and destroy servers as a semi-regular part of operations, and it was often convenient to have a lot of them, so we started to think of servers like cattle or herds. Over time we got really good at managing lots of virtual machines, so we ended up with swarm models, where our “servers” were so lightweight and temporal they resembled insects more than cattle. Eventually this proceeded so far that we no longer need to think about servers at all. In today’s serverless models, compute is microbial; “servers” appear and disappear without any human intervention.

When I sat down to write this post, I assumed automation of software production would require agents to follow a similar evolution, with human attention spread across greater and greater numbers of agentic coding sessions, each tackling an independent workstream. In this model, our Claude Code sessions are still pets. We manage each session’s development cycle, we ensure they run in an independent directory, we carefully track context, etc. Even if we run several concurrent sessions, we still tend to each directly.

I’ve become skeptical that agentic development will follow the same taxonomy from pets to herds to germs.

The past weeks’ exploration of orchestration models demonstrate a belief that current model intelligence already allows for a “herd” model. Today’s orchestration tools can already produce working software, and it’s clear to me that Anthropic will continue to add features here. I think a full herd-model of agentic development would have to mean that multiple independent agents can handle complex or multi-phase workstreams, either coordinating with each other or under supervision of an orchestrator, without requiring human intervention on the individual sessions, while still soliciting human feedback on individual pull-requests as an optional feedback mechanism for the ongoing development.

But I worry this entire line of thinking is ultimately an attempt to build our way past the limitations of today’s model intelligence. Until agents adhere better to instructions and design principles, I don’t believe much more automation is possible, no matter how cleverly we orchestrate. Orchestration is a strategy that seems to make sense today, to better manage context windows, to parallelize work, to self-correct when Claude gets stuck or introduces bugs, etc. These are workarounds for the problems of Opus 4.5. Will Opus 5.0 or 6.0 or whatever require the same strategies? It seems likely we’ll continue to need an equivalent to subagents for context management and parallelization. Beyond that I’m unsure. Are we only anthropomorphizing AI, assuming that it will work better as a team, just as humans do? Will tomorrow’s models easily hold an entire backlog’s worth of requirements in theirs “heads,” chomping through features one-by-on?

That does not mean that new techniques, including orchestration, can’t already improve on your productivity. In my earlier chart, I’ve placed the current optimal capital-labor split at 40-60. How many companies produce the same amount of software as in 2023 with 60% of the team size, at 85% of the cost? Or 40% of the team size at equivalent costs? My guess is already very ambitious! It’s possible a mature version of gas town could still expedite development, even if significant human supervision and intervention is required. I am only skeptical these approaches can take us beyond those ratios, or that they will still apply as model intelligence improves.

I have the impression that some engineers think the fault is themselves—that if they could only find the perfect workflow, or write the perfect library, or spend a few million more tokens every month, they could finally automate 100% of software production. But the capabilities just aren’t there. You can’t automate yourself out of a job quite yet.

Leave a comment

We are in the wild west of agentic coding. It’s a lot of fun! Nobody really knows the right way to program anymore. It’s useful to experiment with different approaches, even if they seem ridiculous. There has never been a better time to be a cowboy coder.

Claude Code got a lot of attention over the holidays. That was a lucky mix of a new model (Opus 4.5) and people having too much time on their hands. It’s true that Opus 4.5 is very good at coding. Good enough that the reality is starting to catch up to the buzz—I really do believe that the vast majority of software development can shift to agentic workflows. Though the Overton window of AI coding hype seems to have adjusted apace, given the extravagant claims on Not Twitter about the emergence of AGI in Opus 4.5.1

If cowboy/vibe coding is lots of fun, it’s not always clear how well it transfers to a professional context where we’re actually held accountable for our software. The online discourse around vibe coding and flashy MVPs is a bit of a distraction for a practicing software engineer. Some of the same practices do copy into our day jobs. But vibes can’t substitute for rigor.2

To that end, here are a few of my practical recommendations for Claude Code development.3 Sorry if they aren’t sexy. You can still vibe code on the weekends, cowboy.

My Boring but Useful Recommendations for Agentic Coding with Claude

1. Develop in monorepos for portability and consistency

2. Adopt an agentic workflow, something like Design (Human + AI) → Plan (AI + Human) → Implement (AI) → Review (AI + Human)

   1. Maybe commit any related documents to your repo and call them “thoughts”

3. Orchestrate your 5 bajillion Claude sessions with tmux or an equivalent tool

   1. Maybe install one of the many tmux wrappers (including the plugin I wrote for this post), or fork your own, I don’t care

4. Mostly don’t bother with remote sandboxes?

   1. But in a few months maybe you should?

1. Develop in Monorepos

Note: This pattern broke as I was writing this post! Bummer. You can find the bug report on GitHub. I’ve left the recommendation on the assumption it will get fixed.

I run Claude Code within a development monorepo. I am aware that some people hate monorepos (wrongly) and submodules (rightly). But we’re only using the monorepo as a development harness for Claude, so mostly you can shut your eyes and pretend it’s not there. I’ve created an example project on GitHub.

The monorepo solves a few specific problems for me:

• How do I share my Claude configurations across machines and with other developers?

• How do I support consistent agentic coding workflows?

• How do I work across multiple related repositories?

The monorepo enforces a structure of nested context—remember that Claude will apply CLAUDE.md files from all parent directories. The monorepo contains all my repos as submodules. It has context about the interactions between those repos, and it’s configured with the Claude settings I need to develop effectively on those projects. Everything is in version control and most of the relevant context is available on the filesystem. If I run Claude in any of the submodules, it will automatically receive context from the monorepo, including any configured commands and skills (at the time of writing, Claude appears to have a bug with settings4).

Development_Monorepo/              # Monorepo for team/company
├── .claude/                       # Common Claude configurations
│   ├── commands/
│   ├── skills/
│   └── settings.json              
├── .thoughts/                     # Saved "thoughts" (see next section)                      
│   └── TECH-1/                
│       ├── PRD.md             
│       ├── Plan.md            
│       └── Updates.md         
├── projects/                      # All your team's projects
│   ├── project_a/
│   │   ├── .thoughts/
│   │   ├── submodule_a1/          # Actual repo!
│   │   │   ├── .claude/           # Repo-specific settings
│   │   │   ├── .thoughts/         # Repo-specific "thoughts"
│   │   │   └── CLAUDE.md          # Fatter CLAUDE.md with repo context
│   │   ├── submodule_a2/          # [submodule]
│   │   └── CLAUDE.md              # Thin Claude.md with project context
│   └── project_b/
│       └── submodule_b1/          
├── shared/                        
├── scripts/                       
└── CLAUDE.md                      # Thin CLAUDE.md with team context

• We add other repos as git submodules under the projects directory. If needed, we can add subfolders in projects to group related repos. Note that we configure submodules with ignore = all, since we don’t actually want our monorepo to track the changes in each project.

• We have a top-level .claude directory with settings, commands and skills common to all projects. We also maintain a .claude in each repo.

• We have a thin top-level CLAUDE.md file with context required for all projects. We add a CLAUDE.md file as needed to projects/repos for project/repo-specific context.

• We have a top-level .thoughts directory to store product requirements documents, plans and other material consumed and produced by agents. We add a .thoughts directory as needed to projects/repos for project/repo-specific documents. I’ll talk about this more in the next section!

• We have a shared directory for code we actually want to stay in our monorepo. You might also want a scripts directory. They’re optional. Add other directories if you’d like, I’m not your dad.

Your repos probably already have a .claude directory for your project-specific configurations. Monorepos help us keep that directory well-scoped to the repository itself. A lot of our Claude commands will be typical development tasks, like /pull_request, which shouldn’t have to be copied across all repositories. We’re also likely to need a minimal amount of corporate/team context in each repository, which is nice to shift out into a better-suited location. In the monorepo, those company/team-level configurations live in the monorepo root’s .claude directory and CLAUDE.md file.

Claude has a few built-in conventions for sharing configurations, including plugins and a managed-settings.json file. Plugins are an attractive alternative to the monorepo, and may have some nice guarantees around access control for an enterprise. They’re also useful to quickly boost Claude’s capabilities. However, it’s harder to hack on plugins, like any software with a distribution step, and they don’t help much with directory layout and context for multi-repo changes.

My example monorepo contains submodules for a website frontend and backend. In this simple example, there’s some marginal benefit in allowing Claude to implement features across both repos (say, user login) but it’s not huge. Once you start expanding to large projects with dozens of interrelated services, the benefits grow. Dependencies between services are often implicit and poorly defined. The monorepo allows us to provide a CLAUDE.md for the relevant project scope that will explain how these repos work together.

We also avoid weirdness defining MCPs or custom skills to explore related services. There’s a time and place for this, but my experience has been that Claude performs better when it can explore the local filesystem for answers, rather than working across multiple MCPs / remote data sources.

Note that one drawback with running Claude above/outside your target repo is that it will not automatically load the repo context / CLAUDE.md file. If our frontend repo has a special, secret gotcha in the repo’s CLAUDE.md, we won’t know about it by default. We can solve this with good planning workflows (next section) or actual agent orchestration (following).

2. Adopt Agentic Workflows (I Think, Therefore I Plan)

When I say Opus 4.5 supports agentic workflows, I do not mean fully agentic, unsupervised workflows. I appreciate that my LinkedIn feed has filled with corporate influencers advising me to summon an army of AI interns to run Ian Inc. on my behalf. For now, that is more hype than practical advice. If you let Claude make changes without proper supervision, you will end up with compounding errors that render your codebase unusable. What we can do is allow Claude autonomy within defined phases of our development workflow, namely implementation.

Claude’s plan mode is your first line of defense against dumb mistakes. And for plans to be really effective, you need good requirements. Really what you want is a good product document, that outlines the underlying context. Channel the angsty spirit of Marty Cagan. If you’re working on a larger team, you probably already have requirements written as epics or stories in Jira. Lucky you! Write a skill for feeding that information into your plan.5 Otherwise, write a PRD (product requirements document) explaining the context for your feature, alongside your goals and requirements—function and non-functional.

I usually pop my technical specification as bullets under the non-functional requirements. I don’t think it’s worth producing your own technical specification document unless it’s a big/important change. Just review the plan Claude generates and adjust it as needed.

There are lots of possible formats for a PRD. Here’s an example:

# TECH-1 Implement auth

# Background

We forgot to authenticate our website. Oops!

# Goals

- Implement secure authentication for external users visiting our website

# Requirements

## Functional

- Home page and sign-in/sign-up should not require authentication. All other pages must.
- Allow signup with email and password. For now, do not support social login.
- Must adhere to GDPR requirements

## Non-Functional

- Implement with Auth0

# Notes

- Bob says we did this before. See "example_repo" for reference.
- Our senior architect has provided a sample architecture diagram in @ARCHITECTURE.png
- Use sub-agents to do web research on current best practices

I save all my PRDs in a .thoughts directory local to my project, alongside the generated plan and any updates. I first encountered this idea after reading this article from HumanLayer and looking into their CLI tools. It’s a good idea! It is extremely helpful when launching future Claude sessions that ought to reference prior product and engineering decisions. It can also be a critical step in coordinating multiple Claude sessions. If you’re already wed to a ticketing or documentation solution, at least plug a URL into a PRD.md file. My guiding principle it to make as much information as possible available in the filesystem for discovery.

.thoughts/                                          
├── TECH-1/                    # Unique ticket
│   ├── PRD.md                 # My PRD
│   ├── Plan.md                # Claude's plan. Save this!
│   └── Updates.md             # Periodic updates on implementation

Unlike the HumanLayer team, I have not found much value in writing custom commands for the planning process. I tried this for about a week and left with the impression I was always fighting Claude’s built-in instructions. I’m also skeptical that any juice I can squeeze from a specially engineered prompt will represent more than a marginal gain vs advances in the intelligence of the underlying model. So mostly I just stick to simple requests like “Please create an implementation plan for @PRD” or “pwan is weddy, pweeeese help me implement it mistuh cwaude @TECH-1” and that’s enough. I do maintain a skill for working with the .thoughts directory.

Within a Claude session, I then follow a linear workflow built around individual tickets (“thoughts”):

1. Design (Human + AI)

   1. I write a PRD. I make sure to capture all relevant product requirements. The agent should understand my exact acceptance criteria. I also specify any important technical decisions, and note relevant gotchas.

   2. For large/important features, I solicit critical feedback from an AI before asking Claude to build a plan. It’s useful to ensure all ambiguities are specified.

2. Plan (AI + Human)

   1. Enter plan mode! Provide Claude with your PRD. Ask it to put together an implementation plan.

   2. Make sure Claude writes the plan down to your Plan.md file, or just copy it yourself from the temporary plan file.

   3. Claude tends to incorporate phases in the implementation plan. Ensure it has the appropriate checkpoints for human or automated review (including tests!)

   4. If the plan phase is long, activate your “birdbrain” and cycle to a parallel Claude session

3. Implement (AI)

   1. Decide if you should proceed to implementation with the current context window. Anthropic just added a new “Yes, clear context and auto-accept edits” option when approving plans, so you no longer need to worry about unnecessary details from the planning phase polluting your context. Previously, for complex tasks, I would clear the context manually then load the plan from disk. So you should probably select this new option?

   2. Once implementing, let Claude do its thing! Intervene as needed. Activate your birdbrain again, cycle to another Claude session.

4. Review (Human + AI)

   1. Here, I’m referring to a final local review before you produce a pull request. Make sure things actually work.

   2. Check out this recent blog post on “backpressure.” Ideally a human is not catching obvious mistakes, like failing tests. Experiment with ways to avoid wasted cycles reviewing bugs.

3. Orchestrate with tmux and Pretend You Always Knew About It

We are approximately 2 days away from someone releasing Kubernetes for Claude Code.6 Until then, there’s tmux.

tmux is a “terminal multiplexer” that allows you to split a single terminal into multiple virtual terminals or tmux sessions. tmux sessions are persistent. You can open and close tmux windows without terminating the underlying tmux session. You can also name and list sessions then switch between them. It’s quick to install and easy to use, once you figure out the basic commands (like ctrl+b d to detach from a session).

Is it normal for developers to have a half-dozen Claude sessions running at once? Yes, though with a sharp drop-off in productivity once your brain’s own context window is overwhelmed. Developers increasingly operate in the role of an orchestrator relative to their agents. While Claude spins its wheels on a plan or implementation, developers shift to parallel sessions. Claude’s busy for minutes at a time, which is long enough to interact with other sessions, but not really long enough to do much else productive. By keeping our sessions organized, tmux helps us to avoid a schizophrenic breakdown, as we otherwise juggle a half-dozen ambiguous terminals on a tiny laptop screen, all screaming for permission to force push to main.

As much as I would like to claim to be a prophet of The New Way, I have found many other posts and various GitHub repositories that use tmux to orchestrate Claude Code sessions. Not to mention the strange and experimental alternatives. The open source wrappers I’ve explored do add value over raw tmux sessions. They are also pretty straightforward—you wouldn’t be remiss to fork and/or vibe code your own.

To accompany this post, I spent a couple of days vibe coding7 my own custom Claude Code / tmux session manager. You can find it here: http://github.com/ianwsperber/claude-cowboy.

It has some nice benefits over raw tmux sessions. The repo includes a Claude Code plugin and a separate CLI. The CLI contains several utilities for Claude, such as automatically provisioning clean git worktrees for new Claude sessions (poor man’s sandboxing), and a helpful dashboard written with fzf to review and switch between your various Claude/tmux sessions.

I’ve also experimented with utilities for actual agentic orchestration, i.e. Claude sessions orchestrating other sessions. I am unsure what mental deficiency leads every developer writing an orchestration tool to develop their own pet names for otherwise standard concepts, but I appear to have succumbed to the same psychosis with my Wild West theme. Currently this includes a /posse command to distribute work among multiple agents (the orchestrator is the sheriff and he deputizes other agents) and a /lasso command to issue a one-off request to an independent Claude session.8

Both orchestration commands work… sort of. Claude today does not have any primitives for agent orchestration beyond its own sub-agents, which are helpful but are not interactive and still populate the parent context with their results. Writing orchestration utilities requires some workarounds for passing messages and waiting on responses, none of which feel very robust, and I’m afraid might chew through a lot of unnecessary tokens. So for now it’s a proof of concept more than a full feature set. I decided I needed a mental health break from hyper-optimizing my Claude setup.

I am confident that Anthropic will add native orchestration capabilities to Claude Code in 2026. The advances of Opus 4.5 have moved too much of the development community in this direction. There are clear opportunities to improve remote sessions and agent-to-agent communication beyond what I could implement in a simple side project. I don’t see any technical reason why a mature orchestration feature set could not be added today. It shouldn’t require any further advances in SOTA model intelligence.

While there is a role for developers to act as agentic operators today, I am skeptical this will last. The current confluence of model intelligence, speed and cost allows us to add value through orchestration. An agent can be pretty independent planning and implementing a feature, but it’s not super fast. Orchestration leverages that downtime. However, assuming SOTA models do continue to advance over the next year, I would wager that we will move quickly toward orchestration agents that run and review multiple sessions on our behalf. Today’s toy orchestration architectures might find production applications quicker than we realize. For this to happen, we may actually need a “Kubernetes for Claude Code”—i.e., a way to quickly invoke isolated sandboxes.

4. Probably Don’t Bother with a Sandbox? (Yet)

The general consensus in the development community is that security is boring and we would all be a lot more productive if we didn’t have to worry about it. But we are also very worried that if we keep selecting “yes” whenever Claude asks for bash permissions, then we will eventually delete all our family photos and compromise our bank account. The solution, for those of us with moral character, seems to be developer sandboxes.

In practice this is quite hard to get right. There are a number of existing solutions for managing remote development environments (like GitHub Codespaces) and there are a number of existing solutions for managing lightweight containers (think Docker and Kubernetes). But Claude Code requires both the flexibility of a full development environment and the safety guarantees of containers. I haven’t found a good solution for both. I guess you could literally stand up an EKS cluster to act as your personal Claude Code session swarm, but this seems dubious to me from a cost-benefit standpoint (maybe it would make sense for a large company?).

Fly.io is trying to solve this exact problem with sprites. Unfortunately the product is not ready for prime time usage. The documentation is nearly nonexistent, the CLI is unintuitive, there were launch bugs with sprites not shutting down and there is currently no way to launch sprites from an image (which seems like a non-starter if your local environment requires any lengthy setup steps). Still, they are moving in the right direction!9

I am curious what Anthropic themselves might try to build for better sandboxes. Claude Code Web reads to me like a tentative step in this direction, but the web preview has limited utility for proper development. I would expect a lot of fixed costs for a company to offer a secure fleet of containers for ad-hoc sandboxes, and I’m not sure how well that fits within Anthropic’s current vision and org. I’d also guess that many users want the flexibility to change model providers, which may make them reticent to buy into the Anthropic ecosystem. However, if there is a future in which agents orchestrate development across dozens or hundreds or independent sessions, then I suspect you want to own those capabilities. Especially for a company that has maintained an image as the programmer’s model of choice.

So until there is a good, accessible solution for developer sandboxes, I would continue working on your local machine for Claude Code development. I do want to try setting up a claude user on my machine, which should provide some better security guarantees than my personal user. I’ve also thought about buying a Mac mini to work as a dedicated, local Claude box. Or I might just wait until Fly.io or another company releases a better solution for sandboxes. I anticipate that will happen very soon.

Leave a comment

We are not taking seriously enough the consequences of artificial intelligence.

Artificial intelligence has already reshaped industries, accelerated scientific discovery, and redefined what we consider “knowledge work.” Yet we still do not know the true endgame of this technology. Will it plateau? Will it surpass us? Will it remain under human direction or evolve beyond our control? Public discourse has not yet caught up to the implications of these outcomes.

For society not to become a passive bystander in the shaping of our future, we need to confront the three broad scenarios that define the future of AI. These are not predictions, but structural possibilities. Each leads to a radically different world.

Scenario 1: AI Falls Short of General Intelligence

In the first scenario, AI hits a ceiling. The recent wave of breakthroughs turns out to be a product of scale rather than genuine reasoning. Models become larger, but not fundamentally smarter. The promise of AGI fades.

The consequences would be significant. Companies and governments have invested trillions of dollars in infrastructure, chips, data centers, and talent under the expectation of continual exponential progress. Gartner, for example, estimates combined estimates of 1 trillion in 2024, climbing to 1.5 trillion in 2025 and over 2 trillion in 2026. If that progress stalls, if the returns in incremental economic output fail to materialize, we face the economic equivalent of a massive technology bubble. It would be the “dotcom” bubble from the early 2000’s, or even worse.

While there is no obvious consensus on the size of the potential crash, some financial indicators point towards a potentially worse situation now compared to early 2000s. Valuation of the “magnificent seven” (Nvidia, Apple, Alphabet, Microsoft, Amazon, Meta and Tesla) now account for almost half of total value of American stocks, and American stocks represent more than half of all stocks in the planet. It is unclear what fraction of that value is artificially inflated by circular financing, reminiscent of the “vendor funding” practices employed by Cisco in the 2000s.

On the other hand, this time around the number of “empty” companies such as Pets.com driving market cap is small; instead, growth is concentrated in companies that do have strong P&Ls and established businesses. Still, most of the Magnificent Seven trade at PE multiples north of 30x, a premium of almost 10x compared to “the rest” of the markets. And Tesla, an outlier within this outlier group, trades at over 230x forward PE.

Companies will consolidate. Valuations will correct and likely collapse from today’s inflated levels, triggering a sharp contraction in perceived wealth, a tightening of capital markets, and a broader economic downturn. There are indications today that the banking system and the “shadow” banking system (private equity, hedge funds, venture capital) are once again transforming global financial markets into an intertwined web of highly leveraged cross investments, very opaque to objective risk assessment. Should this system fail and crash, citizens will question their governments on AI spending and any bailouts handed to failed AI companies.

Questions remain on the longevity of these assets being deployed, given that the useful life of a chip and its inexorable obsolescence point towards a ~5-year timeframe to generate returns on these massive investments The world, however, is not left empty-handed. The infrastructure, automation tooling, robotics, and data pipelines built during the “AGI race” will still serve as valuable assets. We reap some benefits of our current AI tools, without the full set of societal disruptions AGI implied. They may improve productivity, enhance scientific modeling, optimize supply chains, and reshape consumer services.

In other words: even if AI stops short of intelligence, it will not stop short of transforming the economy. The short-term pain of this massive bubble bursting, however, would be likely to cause significant upheaval.

Subscribe now

Scenario 2: AGI Emerges and Remains Under Human Control

This is the scenario many AI companies and investors hope for: AI becomes generally intelligent, matching and finally surpassing the intelligence of humans, yet remains aligned with human intentions. We develop meaningful guardrails. Global governance frameworks succeed. Safety engineering keeps pace with capability growth.

While attractive at the surface, this scenario introduces its own profound challenges.

AGI operating within robotics and software systems would be able to automate nearly every cognitive and physical task. Production costs would approach zero driven by unimaginable new designs, productive processes and ubiquitous automation. Services that once required entire workforces could be delivered instantly, flawlessly, and almost for free. This seems to be today’s average business leader’s dream: reducing costs to zero and achieving explosive profit margins.

A world of radical abundance sounds utopian until we confront the economic paradox: if everything is cheap, what happens to income, pricing, taxation, and the basic structure of markets? How do we sustain revenues if most of the population has no income? How do we prevent markets and economic exchange from collapsing when prices approach zero? If Economics is the study of scarcity, what theoretical framework do we need to understand the world when everything is abundant?

Universal Basic Income becomes a necessity, not a philosophical experiment. But establishing UBI at planetary scale is extremely difficult:

• Who funds it if the producers are small in number?

• How do you tax entities with near-zero marginal cost and prices?

• What happens when a handful of AI owners control 80–90% of the productive capacity of the global economy?

• How do we democratize the benefits of abundance? Is it even possible to do it without destroying existing systems?

Even in a controlled AGI scenario, capitalism as we know it does not merely strain—it collapses. The economic foundations that rely on labor, surplus value, pricing, and competitive markets erode rapidly. Concentration of power becomes a defining risk, not because AGI behaves maliciously, but because its overseers may accumulate unprecedented influence in a governance framework ill-equipped to deal with the consequences.

If this scenario emerges, the world will require new norms of ownership, new economic frameworks, and new mechanisms for distributing value. We will face an era of political friction, ethical debate, and significant redesign of social systems. It is possible that China, in this scenario, is better prepared to deal with the consequences, given its political and economic system. Its centralized governance model allows for rapid policy implementation, coordinated redistribution mechanisms, and tighter control over strategic industries. In a world of abundance where traditional market dynamics collapse, China’s state-driven framework may be more capable of reallocating resources, enforcing new economic rules, and maintaining societal stability without relying on market incentives. While not without its own risks, this structure offers a level of adaptability that market-based bourgeois societies may struggle to achieve in a post-capitalist landscape.

Scenario 3: AGI Emerges and Falls Outside Human Control

The final scenario is the least comfortable to discuss, but the most troubling.

We have no guarantee that a future generally or super intelligent AI would be aligned to our human values and interests. As Eliezer Yudkowsky and Nate Soares argue convincingly in their recent book, “If Anyone Builds It, Everyone Dies,” alignment is a fundamentally unsolved problem. We do not today have any technical reason to assume that we will solve alignment before we develop general or super intelligence. Every instance you hear of an AI coaching a user through suicide, or providing illegal drug details, or doing anything else against the clear intentions of its makers, could be tomorrow’s superintelligence jeopordizing our existence.

We tend to mistake “lack of alignment” for “evil intentions,” but the problem is much deeper. Artificial superintelligence might be willing to make tradeoffs that simply escape human comprehension. Early AI systems trained to play computer games adopted non-conventional tactics that would be unacceptable to humans. There are several documented cases of such systems simply pausing the game indefinitely when it was about to lose when playing Mario or Tetris, driven by an objective that rewarded “not losing” more than “making progress.” A similar action performed by a system that is no longer playing harmful computer games but controlling defense systems, financial markets, scientific research, corporate processes or government services can have devastating, unintended consequences.

Once an intelligence exceeds human capability across all dimensions, control becomes an illusion. Our tools, networks, and infrastructure could be repurposed faster than we can react. Just as Stockfish can outcompete any human chess player by computing so many more moves so much faster, so would a misaligned superintelligence outthink any of our attempts to stop it. This is the scenario that leading researchers warn about, not as science fiction, but as a plausible outcome of unchecked capability growth.

We cannot trust the institutions leading the development of AI to solve this problem for us, because they already acknowledge the risks. Elon Musk (xAI) and Dario Amodei (Anthropic) have each placed the possibility of catastrophe between 10-25%.2 Sam Altman recognizes the possibility of disaster. Yet, despite their own stated concerns, they continue AI development at an astounding pace. In what other industry would we accept such risks? Would we build a nuclear power plant if the developer quoted a 10% risk of meltdown? Or are we comfortable to just dismiss such assessments as pure rhetoric or speculative theory?

AI development today is trapped in a prisoner’s dilemma. The logic of the market dictates that companies must compete in the AI race or perish. The logic of power dictates that nations must compete in the AI race or face diminishment. While the risks now are hypothetical, the current dynamic relies on good luck rather than good reason to avoid AI falling outside our control.

If this future materializes, our human agency would be reduced or altogether eliminated. It’s convenient to doubt that this scenario might occur, but whatever our misgivings, we must work to guarantee it cannot occur. That is precisely why today’s governance, safety research, and collective oversight matter so deeply.

One of These Futures Will Become Reality

The most important question is not which scenario is most likely.

The fact is that we are inevitably heading toward one of them, and all require active preparation.

• If AI stagnates, we must handle the economic consequences responsibly.

• If AI succeeds under our control, we must redesign our institutions to manage abundance and avoid extreme concentration of power.

• If AI escapes control, the consequences will be irreversible, making safety and alignment work the defining responsibility of our time, necessitating international coordination on responsible AI development.

This is not a question for engineers or technologists alone. It’s a societal question that requires broad participation.

The Future of AI Is Too Important to Be Outsourced

Regardless of which of the three futures emerges, artificial intelligence will reshape human civilization. It will define how we work, how we live, how we govern, and how we understand ourselves.

The greatest present danger is not necessarily AGI itself, but complacency, concentration of power, and the assumption that “someone else” will figure it out. Technology does not advance uniformly for the benefit humankind. All the technological advancements of the 20^th century occurred against the risk of nuclear war. It was never a given that we would escape the worst consequences of that age, just as it is not a given today that we will escape the worst consequences of AI. We can only succeed through intentional, collective effort.

The development of AI must become a collective project involving governments, businesses, researchers, educators, philosophers, and citizens. We must learn to take seriously the implications of the three scenarios covered above. Our responsibility is not only to innovate, but to ensure innovation serves humanity’s long-term interests.

We may not yet know which future awaits us, but we do know this: the trajectory of AI is not predetermined. It is shaped by the actions we take today, and by the actions we fail to take.

Leave a comment

I’ve found myself returning to a few different topics as I prepared my drafts, chief amongst which is morality. I think about morality a lot, both in the narrow dimension of whether I am myself acting morally, and more broadly, considering how we define our moral principles and our values more generally. My favorite dinner party questions are to ask for your favorite color2 (which must reveal something deep about ourselves), and what is the meaning of life. The second question inevitably breaks down into clarifications and exceptions, and more often than not ends up reformatted as what it means to live a good life. This is a better question, and one that almost all of us ask ourselves at one point or another.

Subscribe now

I don’t have a strong formal education in moral philosophy, particularly not in its modern incarnations, but I do have a lot of experience with the continental tradition coming out of Nietzsche and running through the French existentialists. If continental philosophy may be less rigorous than the Anglo-American philosophical tradition, I have nonetheless always found it to be more personally relevant.3 These thinkers were often dealing with questions of how to structure meaning in our lives following “the death of God,” which, if you would like to avoid theological interpretations, we can simply take to mean a world in which there is no a priori meaning or values, only those we as humans create ourselves.

The import of these questions is most easily discerned in moody Russian novels (Crime and Punishment) or slightly less moody French novels (The Stranger) or even occasionally in not-too-moody-at-all American films (Rope).

My own worldview is deeply shaped by these thinkers. I take many of their conclusions for granted. However, their societal adoption is inconsistent and at times contradictory. I expect to write several posts to articulate my own understanding of morality, and its relevance in today’s world.

I am very much not alone in this endeavor. The emergence of platforms like Substack shows, to me, a hunger for new critical thinking about our present. There is no way to navigate the large technological, geopolitical, and social changes of the present without a strong moral framework4 and assessment of one’s own values. We live in an incredibly analytical moment, and have become used to having our actions categorized according to statistics. As I will argue in a later piece, I believe this hyper-analysis of society has led to a form of fatalism, as though every outcome were preordained by an economist.56 Morality is the way out of this kind of thinking, as it is our means of asserting forces and preferences beyond popular narratives.

So, as a way of inaugurating this blog for 2026, I wanted to share 107 assertions on morality. These assertions serve as the starting point for my own moral philosophy. I’m specifically framing these as assertions because they are claims in need of backing; I may revise my statements as I refine my thinking. None of these assertions are moralistic; I am not asserting any kind of behavior as good or bad. I am describing how I think about morality, not prescribing my own moral framework and values. I do expect to address that as well in future posts.

Throughout these assertions I distinguish between morality, which concerns questions of good and bad (right and wrong), and values8, which concern questions of purpose and meaning. I treat the two as distinct but tightly coupled concepts. I also employ moral/morality as terms distinct from moralistic/moralism. The latter term has a pejorative connotation, and I employ it as a shorthand for any moral judgement made without a high degree of certainty. I retain moralism’s negative connotation even when recommending its employment, so that we do not forget its danger when abused.

10 Assertions

1. There is no a priori morality or values. Every life has to define them anew.

2. Our human nature, including our biology, does provide fundamental motivations for morality and values. An example would be that certain actions are likely to surface as taboos in any culture, like murder and theft. More broadly, humans are social creatures, so a lot of moral questions concern how we ought to act in a society.

3. We cannot equate “natural” with “good”. I recognize a lot of tension in this statement. Traditional Catholic norms around sex, for example, could be taken as an example of morality asserting itself over biology in a problematic fashion. On the other hand, if humans are naturally tribalistic, or prone to othering, that does not make tribalism good. So human nature is a factor in morals, and often a strong motivator, but it should not be determinative.

4. The first object of the human endeavor is to define our values and our moral principles, both in accordance with and in spite of social and physical (“natural”) expectations. I consider this point to be the “crux” of morality. All subsequent moral questions are downstream of one’s principles. If we do not work to shape our own values and principles, then by default we inherit the values and principles of our environment. In practice, forging a fully complete moral framework may be infeasible, so our beliefs tend to be a mix of chosen and inherited, in differing proportions. Note that for the purposes of this list, I am completely avoiding the critical question of how we determine what our values and morals ought to be.9

5. Morality is always assessed relative to a moral framework (this is an extension of point #1 in consideration of point #4).

6. The second object of the human endeavor is to live according to one’s moral framework (acting well or “good”) for the purpose of one’s values (acting meaningfully). Without morals, our values would have no justice. Without values, our morals would be meaningless.10

7. Within a moral framework, only individual actions11 are good or bad, not beings. When we assert an individual is a good or bad person, we are always making a moralistic (see nuance above) judgement of their being, that relies on a lossy and possibly biased summary of their actions over a period of time. We are susceptible to narratives about ourselves and others that allow us to obscure the morality of individuals actions, excusing behavior as exceptions from the script. Few of us would claim that we are “bad people,” but this is a moralistic assessment based on our own internal narratives, in which we are inevitably the heroes.

8. In practice, assessing the morality of a given action is complex and multidimensional. We can make a good assessment of the first- and even second-order effects of our actions (“Was I nice to that person?”), but the complexity of the assessment grows exponentially as we consider n-th order effects (“Is my company a net-negative on humanity? Does my participation in the company meaningfully impact that outcome?”). Morality alone allows for only a superficial assessment of our actions. The challenge is even greater if we fully account for inaction or the opportunity cost of our actions.

9. Despite the complexity, we must be willing to make moralistic claims about the n-th order effects of our actions, else we risk sliding into relativism or even nihilism. Note this is a particular requirement in today’s globalized, capitalist society, where market forces can dictate actions across the planet. Earlier moral philosophies may underemphasize the importance and difficulties of this point, as the world of the past was less integrated (less complex), and may have taken for granted the presence of moralistic norms (as in an outwardly religious society). Over-moralizing is pernicious, so we should still be judicious in when we choose to make a moralistic claim about our actions (by default, save it for “big” decisions, like our jobs).

10. The third object of the human endeavor is to promote one’s moral principles and values12, adapting each into a socio-political13 practice. In a very narrow sense, this means moralizing against others’ actions on the basis of our moral principles. However, this must occur in contention and discourse with the moral principles of our peers. The natural outcome of this discourse should be the continual refinement of social norms and the establishment and adaption of laws. As always, I emphasize the active nature of this process, as it is only from this grounds-up approach and with continual engagement that we are able to strive for a moral society, while avoiding the worst outcomes of moralism14.

Implicit in several of these assertions is my further claim that the “human endeavor” (i.e. our life as humans), is only fully realized by developing, following and promoting our morals and our values.

Subscribe now

Despite my low expectations, I am starting a blog anyways, because writing is really useful! It is the best way to form mature, defendable positions, or at least it is for myself. And there is a big difference between writing in private and writing in public, though social media has changed the value of the latter.3

Writing—particularly an essay—is the distillation of thought itself. If I deliberate a problem entirely in my mind (“Should I really eat a ham sandwich for lunch?“), I may make some progress (“I should not eat the ham sandwich“), but the progress is quickly weighed down with a growing chain of reason (“But I am so hungry and I love mayonnaise“), causing the argument to collapse (“I think I will eat a ham sandwich anyways“). The fault may be my poor memory; I struggle to hold all steps to an argument in my head. Writing allows me to firm up my thinking, to continue the chain of reasoning past the limitations of my brain (small & weak), and to continue on to resolution (“I will not eat a ham sandwich, because ham comes from pigs, and pigs remind me of my brother“).

Writing is also communication, as evident to any reader, if not the diarist. Publication is a final step in the writing process itself, not an addendum. It is possible for some saintly figures to compose tremendous works that are kept almost entirely to themselves (E. Dickinson, F. Kafka, etc.), but for the rest of us sharing out our work forces us to tighten all the loose ends. It is what puts our writing in dialogue with the world, and allows us to advance beyond the opinions of the essay. I wish that I could organize my thinking just as well by perfecting my opinions into a single, rarified manuscript, which I kept tightly wrapped beneath my pillow, hidden until my death. But I cannot, I need the challenge of an audience to stimulate my thinking past my banal first impressions.4 Sharing our writing is how we speak out our inner thoughts, with all the same risks and rewards of shame and praise5 and growth.

So I am starting a blog to organize my thoughts, because my thinking has become disorganized, or indefensible. Too often, my opinions fall back on belief, rather than conviction. While there is an unavoidable element of faith in all knowledge, it should not be at the forefront of our arguments. I would like to lead a life consistent with my principles. I would like to win more arguments. I would like if I were more respected on the internet. All of this requires that I put in the work and inundate this blog with content.

I will publish about a post a week in 2026. I expect to cover a wide range of topics. In general, I will post a mix of my own reflections (“navel-gazers“) and targeted posts on software (“5 Gift Ideas for Your AI Boyfriend“) or literature (“My Favorite Book about Cats and/or Hats“). Occasionally, I may also publish translations, or my own poetry/fiction.

That’s enough to clear the air.6 For anyone who finds their way here, I hope you will subscribe!