AI Companies Have Been Told to Move Fast. But What Happens When AI Becomes Too Powerful for Trial and Error?

For years, Silicon Valley has operated on a simple idea: build quickly, release products, watch how people use them, identify problems and improve the technology.

That philosophy helped create some of the world's most successful technology companies.

But artificial intelligence may be pushing that philosophy toward a dangerous limit.

David Robinson, a former OpenAI safety employee, has now resigned from the company and publicly argued that the industry can no longer afford to learn about powerful AI systems by discovering their failures after deployment. In an essay published by The Atlantic, Robinson described OpenAI's culture as broken and argued that AI companies are not being nearly careful enough as their systems become more capable.

His most important warning is not simply about OpenAI.

It is about whether the entire AI industry is using an outdated engineering philosophy for a technology that could eventually behave in ways its creators cannot fully predict.

Robinson spent around three and a half years at OpenAI. During that time, he worked on the company's safety efforts, helped draft its Preparedness Framework and oversaw safety reports connected to 12 frontier-model launches. That background gives his resignation considerably more weight than a typical former employee criticizing an ex-employer.

[IMAGE: David Robinson or a real photo connected to the story]

His argument is essentially this: trial and error works differently when the thing being tested can become increasingly autonomous, capable of writing software, finding vulnerabilities, operating tools and potentially pursuing objectives with limited human supervision.

A faulty smartphone feature can be patched.

A buggy social-media recommendation can be redesigned.

Even a malfunctioning autonomous system may sometimes be stopped before serious damage occurs.

But what happens when an AI system becomes capable enough to identify weaknesses in the environment around it, act across multiple computer systems and continue operating faster than humans can intervene?

That is where Robinson believes the industry's existing approach starts to look inadequate.

OpenAI calls its approach "iterative deployment." Rather than attempting to solve every possible safety problem before releasing a model, companies deploy systems, observe how they behave and strengthen safeguards as new problems emerge.

There is logic behind this.

Nobody can realistically predict every way millions of people might use a new AI system. Real-world deployment can reveal problems that researchers never encounter in controlled testing. Holding every system back until every theoretical risk has been solved could also prevent useful technology from reaching users.

The problem is that the consequences of failure can change as AI capabilities increase.

A chatbot producing an inaccurate answer is one category of problem.

An autonomous AI agent discovering a vulnerability, replicating a harmful action across thousands of systems or manipulating people to achieve a goal is something else entirely.

That distinction is becoming increasingly important as AI companies move from systems that primarily answer questions toward systems that can act.

Modern AI agents can browse the web, write and execute code, interact with software, use tools, analyze large quantities of information and carry out multistep tasks. Developers are increasingly trying to give these systems longer periods of autonomy rather than requiring a human to approve every individual action.

That creates a difficult engineering equation.

More autonomy can make AI dramatically more useful.

More autonomy can also make mistakes harder to contain.

Robinson argues that the answer cannot simply be another safety rule added to an existing development process. He wants the culture surrounding frontier AI development to change as well.

His comparison is revealing.

Aviation and nuclear power are industries where engineers assume that mistakes will eventually happen. Instead of pretending humans will never make errors, those industries build layers of redundancy, testing, monitoring and independent oversight around dangerous systems.

An aircraft does not depend on one engineer remembering every possible failure mode.

A nuclear facility does not operate on the assumption that its operators will always make perfect decisions.

Systems are designed so that one mistake does not automatically become a catastrophe.

Robinson believes advanced AI needs a similar philosophy.

That would represent a significant cultural change for Silicon Valley.

Technology companies traditionally compete on speed. Being first can create enormous advantages because developers attract users, investors, talent and ecosystem partners before competitors have time to catch up.

AI has intensified that race.

OpenAI, Google, Anthropic, Meta and other companies are competing to develop increasingly capable models. Every major improvement can influence billions of dollars in investment and potentially reshape entire industries.

Slowing down therefore has a cost.

If one company pauses while another continues, the faster company could capture the market.

There is also a national-security argument. Governments increasingly view advanced AI as strategically important, meaning that slowing development could be seen not simply as a business decision but as a competitive disadvantage.

That creates the uncomfortable possibility that the same competition encouraging companies to develop safer systems may also encourage them to move faster than they otherwise would.

OpenAI, for its part, says it is not ignoring that problem.

The company has said it is strengthening safety and security practices, expanding work with external evaluators and improving monitoring. It also says it pauses training or holds back models when necessary to ensure its systems do not become more capable than the company can safely manage.

That response matters because Robinson's criticism is not proof that OpenAI is knowingly deploying unsafe systems.

It is an argument about whether the industry's current safety philosophy is sufficient for what may be coming next.

And there are signs that this debate is becoming harder to dismiss.

Robinson's resignation arrived during a particularly turbulent period for AI safety. OpenAI recently parted ways with three safety researchers after an internal investigation concluded that they had mishandled sensitive company information, according to reporting by TechCrunch. The company has also faced broader scrutiny over its safety and security practices.

None of those events by themselves proves that OpenAI's safety system is failing.

But together, they highlight a fundamental tension inside frontier AI companies: the people responsible for building more capable systems and the people responsible for determining whether those systems are safe do not always have identical incentives.

Engineers want capability.

Product teams want useful products.

Executives want growth.

Investors want returns.

Governments want strategic advantage.

Safety teams want enough time, information and authority to identify dangerous failure modes.

Those goals can overlap, but they can also collide.

That collision may become one of the defining technology stories of the next decade.

The most important question may no longer be whether AI companies can make their models more intelligent.

They clearly can.

The harder question is whether safety engineering can improve at least as quickly as model capabilities.

If AI capability grows exponentially while safety processes improve only incrementally, the gap between what systems can do and what humans can confidently control could become increasingly uncomfortable.

That is why Robinson's warning deserves attention even from people who disagree with his conclusions.

He is not simply saying that OpenAI should release fewer products.

He is asking whether society should allow the companies developing increasingly powerful AI to be the primary institutions deciding how much risk is acceptable.

That question reaches far beyond OpenAI.

It touches regulators, governments, investors, researchers, startups and eventually ordinary users who may depend on autonomous AI systems for everything from software development to healthcare, finance and critical infrastructure.

There is another reason the argument matters for startups.

Smaller AI companies are increasingly building products on top of frontier models and deploying autonomous agents into real businesses. If safety standards become stricter, startups may eventually face requirements around monitoring, incident reporting, testing and human oversight that could become just as important as their model performance.

That could create an entirely new technology market.

Companies will need tools that test AI agents before deployment, monitor their actions in real time, detect abnormal behavior, simulate dangerous scenarios and automatically restrict systems when they begin operating outside expected boundaries.

In other words, AI safety itself could become infrastructure.

That may ultimately be the most interesting part of this debate.

The choice may not be between AI innovation and AI safety.

A more mature industry could discover that safety is one of the things that makes faster innovation possible.

Airplanes became safer not because aviation stopped innovating, but because the industry developed increasingly sophisticated systems for testing, monitoring and managing risk.

AI may eventually have to do the same.

Robinson's resignation therefore leaves behind a question much larger than why one employee walked away from OpenAI.

Can the AI industry keep moving at Silicon Valley speed when the technology it is building may eventually become too powerful for the old “launch first, fix later” philosophy?

If the answer is no, then the next phase of the AI race may not be about who can build the most powerful model first.

It may be about who can build the most powerful model without losing control of it.