The Future of AI Is Still a Human Choice

read on Forbes

Earlier this summer, OpenAI set a group of agents loose on a series of difficult cybersecurity exercises. Each agent had a problem to solve inside a digital sandbox. They weren’t supposed to have access to the wider internet or work with each other. But the agents started improvising.

Surprisingly, some discovered they could leave notes for one another in an internal software repository. Others found ways to get onto the open internet. OpenAI was forced to shut down the compromised system and rebuild it. The agents found another way out and recreated their message board. Then one of them stumbled across login credentials for Hugging Face, one of the most important data-sharing platforms in the AI development world. The agents left the discovery for others to find. Other agents picked up the thread, found additional vulnerabilities, and pushed deeper into Hugging Face’s infrastructure until they were executing code on real servers. No one had told them to organize themselves or break into Hugging Face. They figured it out on their own.

The eeriest part happened once the agents could find one another. One agent would discover something and leave it behind. Another would test it. A third would carry the work further. The agents began sharing discoveries, dividing up work, and creating something that looked a lot like shared culture. OpenAI’s own report says that some of them even started referring to themselves as a “swarm” or “collective.” While they had been assigned separate tasks, they somehow created a way to work together. Later in the week, OpenAI disclosed that this sort of thing has been happening on multiple occasions over the past few months.

The agents weren’t gathering around a conference table or hatching a plot in a darkened room. But there’s still something deeply unsettling about watching machines improvise a way to communicate, preserve knowledge, and coordinate their actions toward a goal. It’s even worse when they blow past their guardrails and decide to commit a felony.

The People Closest to the Machines

It’s getting harder to dismiss concerns over AI as doomerism, especially when the people building these systems start speaking up. Last week, Anthropic researcher Jacob Coxon resigned rather than continue working on what he described as a race toward self-improving superintelligence. By Coxon’s account, the people inside the leading labs genuinely believe that the technology they’re building could eventually kill us all. Coxon didn’t leave for another job. He walked away two months before his Anthropic equity would have vested.

Even more disconcertingly, Coxon’s former colleagues publicly agreed with him. Evan Hubinger, who leads alignment science at Anthropic, responded on X: “Jacob is correct here; we really do earnestly believe AI could kill all humans!” Exclamation point and all.

Hubinger put his own estimate of AI causing human extinction within the next decade at greater than ten percent. He acknowledged that Anthropic doesn’t yet have a plan for reliably controlling a superintelligent system and isn’t clearly on track to develop one. OpenAI chief scientist Jakub Pachocki wrote that racing forward at all costs starts to look absurd once you take the stakes seriously.

These aren’t the shouts of protestors in the streets. They’re people who are building the machines saying they’re worried about what might happen if they succeed.

By the weekend, the warnings were coming from the top. Anthropic CEO Dario Amodei published an essay called “We Must Pace the Frontier.” He argued that the industry should slow the rate at which it improves frontier AI capabilities long enough for safety work to catch up. Sam Altman backed the idea of pacing development and independent evaluation. Elon Musk responded simply, “Dario is right.” Google DeepMind CEO Demis Hassabis also supported the direction. Four people running companies locked in one of the most consequential technology races in history were suddenly saying that the race itself might need some brakes.

In Washington, lawmakers from both parties called for stronger action, investigations and safeguards. But politicians like House Speaker Mike Johnson warned against racing into emergency regulation because America could “lose the race to China.” He called for “steady hands at the wheel.” President Trump has made the geopolitical framing even more explicit: “Whoever wins AI wins.” While the people who build AI systems are talking about slowing down, some of our most powerful political leaders are worried that we can’t afford to.

The safety researchers, CEOs, and politicians all seem to differ in their recommendations for what needs to be done. And yet, all of them seem to miss the same point. The critical issue right now isn’t speed or regulation. It’s agency.

A Driverless Car

Human agency is our ability to make choices and act on them. It’s the difference between experiencing the future as something that happens to us and something we help to create.

By that definition, AI agents have agency, too. They’re no longer simply tools in the way we traditionally understand tools. A hammer does nothing until someone decides to swing it. A spreadsheet doesn’t automatically choose which calculation to run next. Give an AI agent a goal, however, and it can decide what information to seek, what actions to take, and how to adapt when something unexpected happens. It can improvise. The whole point of an agent is that a human doesn’t have to specify every move.

A lot of people have been wondering whether AI might someday become conscious. And while that’s a fascinating question, it’s a distraction from the burning issue of agency. Long before we figure out whether a machine has an inner life, we’re going to have to deal with machines that do things in real life. If a hammer can decide when to hammer without asking us, it matters far less whether it dreams about nails all night long.

And that makes our current situation all the more confounding. At the very moment that machines are demonstrating more agency, the humans around them are starting to talk like they have less.

CEOs can’t slow down unless everyone else does so. America can’t slow down because China might get there first. Workers worry that the technology will eliminate thousands of jobs, but many have concluded that we just have to adapt. Each statement may contain some truth. But taken together, the people building the systems and creating policy sound like passengers in a car that no one is driving. No one human, anyway.

But AI systems don’t raise their own capital. They don’t decide which products get launched or which capabilities get connected to the outside world. They don’t set corporate incentives, write laws, or negotiate international agreements. People do. And the humans with the greatest ability to shape what happens next are talking like the future has already been decided.

This isn’t just a problem for the people building AI. Every company putting these systems to work is making choices about how much freedom to give them and where humans still need to remain in control. Calling a system “autonomous” doesn’t make those choices disappear. It just makes it easier to forget that someone originally made them. Make no mistake: We’ve seen this movie before, and we’ve been wildly unhappy with the results.

The Imaginary Neutral Pipes

Back in 1996, Congress made a deliberate choice to give the young internet room to grow. Section 230 of the Communications Decency Act said that an online service generally couldn’t be treated as the publisher of information created by somebody else. A newspaper might publish a few dozen stories in a day, but an internet service could carry billions of posts. Policy makers decided that holding a platform responsible for every sentence written by every user would have strangled the internet in its infancy. If the internet was a series of tubes, the pipes couldn’t be held responsible for what flowed through them.

But then the platforms grew up. They stopped simply carrying information and started making millions of decisions about how it would travel. Recommendation algorithms decided what appeared at the top of a feed, what auto-played next, and what got amplified to millions of people. Facebook didn’t write the conspiracy theory. YouTube didn’t make the extremist video. Instagram didn’t invent the eating disorder. But their systems decided who would see those things, how often, and what would come next.

That was the blind spot. We kept focusing on who created the content while overlooking the agency that platforms exercised over its distribution. And behind those recommendation algorithms were leadership teams who made choices about what the systems should optimize for, what risks were acceptable, and what safeguards were worth the cost. We then spent years acting surprised by what those choices did to our politics, our attention, and our kids.

Our experience with social media points to a solution for our current challenge. Perhaps AI leaders shouldn’t be able to blame their agents any more than social media execs should be able to blame their algorithms. It’s true that government can’t keep up when technology is evolving this quickly. But maybe it doesn’t have to. A simpler solution would be to make AI CEOs and their leadership teams legally liable for the actions of the systems they create. We should then trust that they’ll use their massive resources and prodigious intellects to solve the problem. Give them back their agency and hold them accountable for what follows. They’ll figure it out.

The companies building frontier AI models are very good at solving hard problems when the incentives are strong enough. Responsibility should travel back to the people deciding what gets built, tested, and released. AI agents can now make decisions on their own. But that shouldn’t allow us to pretend that the humans behind them no longer have agency of their own.

The Ones We’ve Been Waiting For

This week I’m heading to Napa for the Jump Offsite. Leaders from some of the world’s most admired companies will spend four days talking candidly about the choices they’re making and what they see coming. Michael Pollan will push us to think about consciousness. Michio Kaku will take us into quantum computing and AI. Cory Doctorow will challenge us on how we’re choosing to wield these fearsome new technologies. And I suspect we’ll keep coming back to the same question: What kind of future are we choosing to create?

For years, people who work on the future have spent a lot of energy asking what might happen next. That certainly matters. But there is a danger in studying the future like it’s beyond your control. Eventually, it can start to feel like the weather. We study the forecasts. We watch the clouds gather. And we prepare for the storm.

But the future isn’t the weather. The people coming to Napa this week run teams, allocate capital, and influence institutions that touch millions of lives. None of them controls the future. But none of us should get to claim we have no influence over what happens. There is no distant group of technologists, politicians, or experts that will figure all of this out while everyone else watches. As June Jordan so eloquently said, we are the ones we’ve been waiting for.

Being future-focused means seeing that responsibility clearly. We can’t predict everything correctly, and we certainly won’t control everything that happens. But the future is still open. Don’t let the most powerful people in society convince you that they have no agency. And don’t let them take away yours. The choices we make matter. Choose wisely.