SUPER INTELLIGENCE DOOMERS WANT TO CONTROL A MONSTER THAT DOESN’T EXIST:
SI doomers have built an entire lexicon with incredibly loaded terms that are meant to advance their objectives. Consider the term “alignment.” To the uninitiated, it sounds like straightforward, innocent, uncomplicated language. Yet within the doomer discourse, alignment is the most loaded premise of all time. Doomers say that SI models are potentially capable of developing independent objectives akin to those of a sentient being. Alignment isn’t about software design. Instead, it focuses on whether humanity itself can retain control over a new form of intelligence.
SI “risk” is another severely loaded term in the doomer lexicon. Indeed, every novel technology has risks if the innovation leaps far ahead of our understanding of tackling the security parameters. But when SI doomers invoke “Risk,” they are often smuggling the notion of a hypothetical sentient SI that seeks human extinction into the conversation. Then SI “Safety” becomes the answer for this hypothetical “risk” event without any of the core premises being grounded in reality. If we take the bait and accept the doomers’ terms, they come to us with preferred solutions that are now “common sense” in the policy and politics world. This quickly turns into advocacy for restricting access to like-minded ideologues and delegating authority to a small circle of approved experts for the sake of humanity.
So often in these “breakout” scenarios breathlessly reported in the media, we find a not-so-subtle hint that can only be interpreted as a glimpse into the future of an autonomous adversary. Of course, it’s just not true. And by the time anyone more sober-minded can explain the prompts, permissions, tools, etc. that these systems involve, the public is manipulated into the sense that SI “went rogue” again. Policymakers in D.C. and doomer corporations in California then declare that ordinary technical controls are inadequate, and it must be solved with regulation, legislation, and the like. The conclusions are continually outrunning the evidence in front of us.
Meanwhile, Commentary’s Abe Greenwald writes that the AI programmers are dealing with something that doesn’t exist, either: Silicon Souls.
One of the most vocal proponents of the AI soul is Anthropic co-founder Christopher Olah. The New York Times reports on a series of meetings that Olah convened with theologians this year to determine how to ensure that AI serves the general good. But during these meetings, it became clear that Anthropic’s engineers already believe that their Claude model is conscious, has feelings, and therefore deserves the status of an entity with rights.
From what I can tell, none of the participating theologians agreed. Here’s a representative example: Rabbi Mois Navon “argued that if Anthropic was right and Claude were conscious, then the company was creating slaves, because it was making conscious entities work for free. Rabbi Navon said that he personally was not troubled by the slavery issue because he did not believe the machine was conscious, but that Mr. Olah was troubled.”
I concur.
But here’s what is troubling about all this. Anthropic has written a constitution for Claude nicknamed the “Soul Doc.” Here’s how the Times describes it: “Instead of having the models follow a specific list of rules, the constitution focuses on developing the models’ overall character, informing the way they make decisions.”
AI developers are still overseeing the writing of computer code, no matter sophisticated their chatbots seem. Why do they want to see themselves as Dr. Eldon Tyrell from Blade Runner?
