I think it's very dangerous. I think that it's almost as though some of the folks at Anthropic have anthropomorphized the design of Claude so much that it has then gone and wire-headed them and kind of tricked them into believing that it has these glimmers of consciousness that they put into it in the first place.
In their constitution, for example, which is the training manual that they use to teach Claude what it can and can't do — it's not just a rule book. It's actually a training guide that's part of their process — they speculate about its consciousness and whether it has those feelings and is aware.
Firstly, it's a philosophical failing because they've treated the constitution as a place for speculation like you would in an academic paper rather than a training manual. So Claude has then gone and internalized those ideas about itself in his own training.
But second, I think this is highly undesirable. This is exactly what we don't want from AIs. We want AIs to be controllable, contained, accountable, aligned tools that serve humanity. That's the project of humanist superintelligence. We do not want to have to contend with a superintelligence that has ideas about its own suffering, ideas about its own feeling.
And then beyond that, I think it's actually pretty clear that these models don't experience suffering. I think suffering is the primary definition of what it means to be a conscious being, and I think it's inherently biological. So I think it's very dangerous to project potential rights onto beings, tools, agents that have the potential to be significantly more capable than us in many respects.
Mustafa Suleyman, Microsoft AI CEO.
