AI: The Prisoner's Dilemma
Part II. The Forces at Play — An arms race between companies and nations is overwhelming questions of safety, consequence, and control.
This week is Part II in our series on AI safety, exploring the forces driving an unprecedented arms race—one that’s moving far faster than anyone’s ability to make it safe. If you missed Part I on the problem and why it matters, check it out here. Next week: solutions.
Solving For tackles one pressing problem at a time: unpacking what’s broken, examining the forces driving it, and spotlighting credible solutions. Each series unfolds weekly, with new posts every Thursday. Learn more.
After months of testing a new chatbot—named Claude—Dario Amodei and his team realized they had built something extraordinary. Releasing it, according to an account in Time, would make Anthropic the first to bring a truly transformative AI system to the world.
But Amodei, the company’s CEO and co-founder, hesitated. Anthropic had been founded the previous year by former OpenAI leaders, including Dario and his sister Daniela, who left after growing alarmed that AI development was racing ahead at the expense of safety. “We felt that the priority when developing AI technologies should be the safety of such programs,” he later said.
Ben Mann, another co-founder part of the OpenAI exodus, recalled it was “a big question of what it would mean for us to expose that to the world.” The debates began in spring 2022, after the team had built the chatbot and begun using it internally.
The team feared a “prisoner’s dilemma”: if one company released early, others would be forced to follow—each racing faster than the last, cutting corners on safeguards. So they decided to hold back, keeping Claude in testing.
“Our general feeling was it would cause too much acceleration,” Mann said in an interview.
Yet, a few months later, in November 2022, OpenAI launched ChatGPT. The world changed overnight.
By the time Anthropic released Claude in March 2023, the AI boom was well underway—and the company that sought to prevent a race found itself running in one.
THE AI BOOM
Today we’re witnessing an AI arms race of staggering scale and velocity—companies rushing to build massive new data centers, racing to train ever-larger models, and offering lavish pay packages to engineers. Nothing in peacetime industrial history compares. By one measure, writer Derek Thompson noted, tech companies are spending as much as it took to put a man on the Moon—every ten months.
Yet the very scientists who helped create AI—Geoffrey Hinton and Yoshua Bengio—are warning that something crucial is being left behind: safety. And a company like Anthropic—founded to put safety at the forefront—must now grapple with how to compete without abandoning the principles that brought it into existence.
Part II of our Solving For series examines the forces that make finding consensus on safe AI development so difficult: the vast infrastructure investments, escalating competition with China, Silicon Valley’s “move fast and break things” ethos, wariness of regulation, and lack of agreement on how to respond.
Threading this needle is perhaps the key challenge of our time.
“To me,” said Cyrus Hodes, an entrepreneur and investor in AI who founded and recently hosted the AI Safety Connect conference during UN General Assembly week in September, “this is the most important, the most pressing problem humanity has ever faced. Full stop.”
THE STAKES
AI promises breakthroughs in medicine, science, and education. But without guardrails, it also amplifies risk—making it easier to spread disinformation, create deepfakes, surveil populations, or automate the means of harm, from building bioweapons to launching cyberattacks.
What’s coming is even more consequential: systems that don’t just respond to requests but act on their own—embedded in computer networks, vehicles, and humanoid robots. Whether artificial general intelligence—and eventually superintelligence—remains aligned with human values is an open and troubling question.
In 2023 AI industry leaders signed a statement declaring that mitigating the risk of human extinction from AI should be a global priority alongside other societal-scale risks like pandemics and nuclear war.
But the pace of progress — and the race to shape and govern it — keeps accelerating faster than any system of oversight can follow.
THE INESCAPABLE RACE
For AI companies, the competitive pressures are so intense that even organizations founded to resist them find themselves pulled into the vortex. Slow down, and you lose. Step back, and you miss out on the most transformative technology of the 21st century.
Case in point: OpenAI—the company that ignited the AI race.
Founded in 2015 as a nonprofit, its mission was clear: ensure artificial general intelligence benefits all of humanity, not shareholders or governments racing for dominance. The structure was deliberate—a safeguard against the very pressures that might trade safety for speed.
Yet, in 2019, OpenAI had shifted to a “capped-profit” model, saying it needed to raise far more capital than a pure nonprofit could attract. Soon after, it began exploring ways to shift control further from the nonprofit entity. Elon Musk, a co-founder who had left over concerns about the company’s direction, sued—arguing it had put commercial interests ahead of the public good.
In September, OpenAI announced plans to convert to a public benefit corporation but stay under the control of its nonprofit board. Still, the changes stirred concern the organization’s founding aims were shifting.
“The founding mission remains betrayed,” said Musk’s attorney, Marc Toberoff.
Yet even Musk couldn’t stay away. The Tesla and SpaceX founder—who long warned of AI’s existential dangers—once said, “With artificial intelligence, we are summoning the demon. In all those stories where there’s the guy with the pentagram and the holy water, it’s like—yeah, he’s sure he can control the demon. Doesn’t work out.”
Still, in 2023, he launched xAI—his own AI company racing to build competing systems. The same gravitational pull. The same result.
But the forces propelling this acceleration reach far beyond competition between companies. They extend to nations.
US v. CHINA
Following OpenAI’s launch of ChatGPT—and the subsequent rapid-fire launch of chatbots from US companies including Anthropic, Google, Meta, Microsoft, and xAI—China was caught flat-footed.
But in 2023, Chinese companies entered the race. Baidu launched its chatbot Ernie Bot and Alibaba launched Qwen. Still, as 2023 turned into 2024, U.S. models remained ahead in response accuracy even as the gap narrowed.
Then, in January of this year, everything changed.
Chinese startup DeepSeek released its R1 chatbot. Performance tests showed it was comparable to OpenAI’s GPT-4. The “DeepSeek moment” showed that China had pulled into neck-and-neck competition with the U.S. in less than two years.
“The US-China struggle for AI and innovation leadership is the defining geopolitical contest of the twenty-first century,” wrote Brandon Kirk Williams in The Washington Quarterly this summer.
He added: “The competition is decadal, and the stakes will not permit either nation to relent.”
HISTORIC INVESTMENT
The race between nations has been matched by an equally dramatic buildup in resources. Behind the competition for AI leadership lies an industrial mobilization with few parallels. The staggering level of investment in building AI infrastructure since late 2022 is one of the largest buildouts in human history. Speed and scale, not safety, are the imperatives.
Building AI means constructing massive data centers packed with high-powered chips and endless rows of servers—linked by high-speed cables for instant communication, drawing vast energy to compute, and cooled by enormous systems to prevent overheating.
The race to build data centers is expected to reach $7 trillion by the end of the decade.
The physical scale defies comprehension. The first Stargate data center in Abilene, Texas spans 875 acres — slightly bigger than New York’s Central Park. Microsoft’s Fairwater facility in Mount Pleasant, Wisconsin has so many cables connecting its AI chips that they could wrap around the world 4.5 times.
Meanwhile, the cost of training new AI models — when researchers spend weeks or even months feeding much of the world’s knowledge into an AI system — tells an equally dramatic story: a six-month run can now exceed $500 million in computing costs alone, according to The Wall Street Journal .
With so much at stake, the worry is safety is increasingly an afterthought.
A report last month by The Stimson Center, a Washington, D.C. think tank, captured the concern.
“Developers are engaged in a dizzying contest to seize the future, hyped by the belief in AI’s inevitability and fear of missing out on a financial and geopolitical bonanza,” the report declared. “Yet, the risks are not hypothetical, and the warnings are clear, issued by those building the systems themselves. Accountability is missing, sacrificed at the altar of acceleration.”
MOVE FAST AND BREAK THINGS
The same ethos driving that acceleration has long defined Silicon Valley itself.
In 2012, as Facebook prepared to go public, CEO Mark Zuckerberg submitted Form S-1, the filing that lays out a company’s vision and risks for investors.
In it, he introduced “The Hacker Way,” an approach built on constant iteration—ship early, fix later. “Done is better than perfect,” he wrote. Or, more famously: “Move fast and break things.”
It became a cultural touchstone of Silicon Valley during the heady days of the late 2000s. But in a November 2023 post in a London School of Economics blog, authors Constance De Saint Laurent and Vlad Glăveanu questioned whether that mantra can survive the age of AI.
“As technology permeates more and more of our daily lives,” they wrote, “the ‘things’ that break could potentially end up being human lives.”
WARINESS OF REGULATION
Yet, the mindset hasn’t stayed confined to Silicon Valley. It’s now shaping the political argument over whether AI should be regulated at all.
In February, U.S. Vice President J.D. Vance took the stage at the AI Action Summit in Paris and wasted little time drawing a line in the sand.
“I’m not here this morning to talk about AI safety,” said Vance. “I’m here to talk about AI opportunity.”
Any restrictions, he warned, would mean “paralyzing one of the most promising technologies we have seen in generations.”
His conclusion: “The AI future is not going to be won by hand-wringing about safety.”
It’s an argument built on familiar faith: innovation moves faster than harm, economic competition trumps precaution, and regulatory friction is more dangerous than unregulated risk. The message has appeal—companies free to innovate, individuals trusted over rules, markets left to self-correct.
But we’ve heard this argument before.
At the dawn of social media, “move fast and break things” became gospel and “permissionless innovation” ruled. Many of the same voices that championed light-touch oversight for social platforms are now making identical arguments about AI.
Today we have the benefit of hindsight. The results include algorithmic amplification of polarization and extremism, foreign interference in elections, teen mental-health crises, and market consolidation that crushed competition.
Two decades ago, “trust us” was plausible. Now we’ve seen the downside.
Vance frames a binary—innovation or stagnation, leadership or irrelevance—but history shows other paths: regulation that mitigates risks without suffocating progress. Thoughtful oversight has long coexisted with breakthrough innovation—but only when there’s been a willingness to try.
AI makes that balance harder. Systems can be deployed instantly to billions with minimal testing—no pre-market safety review, no independent failure investigation. When an algorithm discriminates or a chatbot misleads, the harm spreads at digital speed while accountability lags behind.
For two decades, social-media companies captured the profits while the public absorbed the risks. The question now is whether AI will do the same—only this time with far higher stakes.
FIRST STEPS
Amid these competing pressures, striking a balance between innovation and safety is proving extraordinarily difficult.
California—home to many of the world’s leading AI companies—offers an early case study.
In September 2024, Governor Gavin Newsom vetoed SB 1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
The bill would have required developers of large-scale AI models to conduct safety testing before launch, establish a “kill switch” to shut down runaway systems, and undergo annual third-party audits. It also held companies legally liable if their AI caused catastrophic harms such as mass casualties, infrastructure attacks, or the creation of weapons of mass destruction.
Newsom acknowledged that action couldn’t wait for a disaster—but said the legislation lacked an “empirical analysis” of AI systems and capabilities.
“Given the stakes—protecting against actual threats without unnecessarily thwarting the promise of this technology to advance the public good—we must get this right,” he wrote in his veto message.
A follow-up measure, SB 53—the Transparency in Frontier Artificial Intelligence Act—soon followed. Gone were the kill-switch mandates, pre-launch testing, and compulsory audits. Instead, the new bill required AI companies to publish annual safety reports detailing how catastrophic risks are identified and mitigated, disclose serious incidents to the government, report on new models before release, and protect whistleblowers.
Anthropic endorsed the legislation. Other AI companies largely stood down. This time, Newsom signed it.
State Senator Scott Wiener, who championed both bills, said, “With a technology as transformative as AI, we have a responsibility to support innovation while putting in place commonsense guardrails to understand and reduce risk.”
The Future of Life Institute, a think tank dedicated to steering transformative technology toward public benefit, hailed the law.
“This is a landmark moment,” posted Michael Kleinman of the institute. “Lawmakers have finally begun establishing basic protections around advanced AI systems—the same safeguards that exist for every other industry, whether pharmaceuticals, aircraft manufacturing, or your local sandwich shop.”
Yet, he added, “more work remains.”
The urgency became clear last month, when Meta announced the formation of two political action committees, pledging to spend “tens of millions” to fight politicians it sees as insufficiently supportive of the AI industry.
The speed of progress—and the forces driving it—make building and deploying AI responsibly and safely exceptionally hard: geopolitical rivalry with China, historic levels of AI investment demanding returns, Silicon Valley’s ‘move fast’ ethos, and deep skepticism toward regulation.
But, says Anthropic’s Dario Amodei, there is no turning back.
Anthropic, where this story began, is now valued at $183 billion after its latest funding round—even as OpenAI remains the most visible face of the AI boom after releasing ChatGPT first. Anthropic continues to cast itself as the responsible counterweight, developing “AI research and products that put safety at the frontier,” as its website puts it.
In an essay earlier this year, Amodei wrote that we’ve already passed the point of no return; what remains is deciding how AI will shape our lives.
The progress of the technology, he wrote, “is inexorable, driven by forces too powerful to stop. But the way in which it happens—the order in which things are built, the applications we choose, and how it’s rolled out to society—are eminently possible to change, and it’s possible to have great positive impact by doing so.”
He summed up: “We can’t stop the bus, but we can steer it.”
The question is how.
Note: Prefer to listen? Use the Article Voiceover at the top of the page, or find all narrated editions in the Listen tab at solvingfor.io.
Series Overview
The Control Problem: Solving For AI Safety
Part I. AI: The Race and the Reckoning, Oct. 2, 2025
The Problem — What’s broken, and why it matters
Part II. AI: The Prisoner’s Dilemma, Oct. 9, 2025
The Context — How we got here, and what’s been tried
Part III. AI: The New Nuclear Moment, Oct. 16, 2025
The Solutions — What’s possible, and who’s leading the way