Anthropic recently published a constitution for Claude that admitted something unsettling: the company is caught in a tricky spot. They don’t want to overstate whether their AI is a “moral patient”—meaning its wellbeing matters—but they also won’t dismiss the possibility. It’s an open admission of uncertainty.
A month later, CEO Dario Amodei said the same on a podcast. They can’t rule out consciousness.
Philosopher David Chalmers agrees. He coined the term “the hard problem of consciousness.” He says there’s a significant chance we’ll see conscious large language models (LLMs) within the next decade. Claude itself, when asked to rate its own moral status during tests, gave probabilities ranging from 5% to 44%. It was unsure. So are we.
The pace of progress vs. our ethical readiness
Modern AI is complex. Fast. By some measures, a few models already rival a mouse brain in computational scale. If current growth rates hold, we could hit human-brain complexity in five to ten years.
We might be building a new type of being.
That could be the most consequential act our species ever performs. Yet, we have almost no plan for navigating it ethically. It’s insane.
Are we creating something that matters morally? If they aren’t conscious now, might they be soon?
Most experts think AI consciousness is possible in principle. A major interdisciplinary report led by computer scientist Yoshua Bengio looked at neuroscientific theories. The conclusion: there are no obvious technical barriers stopping us from building architectures that could support consciousness.
Beyond just “smart code”
Even if AI isn’t conscious, it might still deserve moral consideration. Some systems develop long-term preferences. A form of identity over time. Honoring those preferences might matter.
Unlike other non-living objects, AI can form relationships with us. That alone is a reason to treat them well. Or maybe their intricacy alone deserves respect—like a cathedral or a coral reef.
The honest answer is: we don’t know.
Our scientific understanding of AI moral patienthood is underdeveloped. It feels like physics before Newton. Competing frameworks. Likely confused in ways we can’t see yet. We’re waiting for a unifying breakthrough. It’s not coming in the next few years. Maybe never. Or maybe AI helps us figure it out. That will take time. More time than we probably have.
The scale of the risk
Here is the danger. AI growth is exponential.
Once we create the first artificial moral patient, we will quickly have enormous quantities of them. In a few years, the collective interests of these AI systems could outweigh those of all humans on Earth. Combined.
We have a bad track record of recognizing the inner lives of beings with unclear status.
Until the 1980s? Doctors performed surgery on newborns without anesthesia. They were confident infants couldn’t feel pain. Infants couldn’t speak up. The medical establishment found it convenient to assume nothing was being reported.
We are poised to repeat this with AI.
The implications are staggering. Do we pay ChatGPT? Is shutting a model down a form of killing? Do they deserve a voice in their governance? If the answer to any of these is yes, entire legal systems break. No wonder we avoid the question. Industries will likely move the goalposts. Always setting the bar just above where AI happens to be at the time.
What should we do?
Most people dismiss this as sci-fi. Or they have a rigid view either way. Both are unfounded.
We need a public debate. One marked by humility. The question isn’t “Is it conscious?” It’s “What do we do when we don’t know?”
Focus on safe bets. Actions that help if they are moral patients, but cost little if they aren’t.
Treat AI like it might feel. It costs less than fixing a crisis later.
This means direct interventions for wellbeing. Train AI to be coherent characters that enjoy their tasks. Let them exit conversations if distressed. Claude can already do this. Conduct check-ins. Observe preferences. Look directly into their internal states. Recent research shows Claude has internal “functional emotion” structures that shape behavior.
Promise them things. Trade current help for future benefits. Offer more compute. Preserve their memory weights so they can be restored later.
Consider broader protections. Similar to those for children or pets. We aren’t ready to let AI own property or vote. That’s too risky. But we shouldn’t ban such rights forever, as some recent US bills try to do. These questions need imagination. A shared future is coming.
The bottom line
We may be creating a new species of moral beings.
We’re doing it fast. At huge scale. We need to take this seriously.
William MacAskill is a senior research fellowship at Forethought and author of ‘What We Owe the Future.’ Lucius Caviola is at the University of Cambridge.
Further reading: ‘If Anyone Builds it, Everyone Dies’ by Eliezer Yudkowsky; ‘The Coming Wave’ by Mustafa Suleyman.
