When machines start building the next generation of machines, the question shifts from what artificial intelligence can do to whether we can still steer them.
For years, warnings that AI was slipping out of human control sounded like science fiction, the kind of talk that belonged in movies rather than in boardrooms. That comfort is fading fast. Dario Amodei, chief executive of Anthropic, one of the industry’s leading labs, has publicly urged a slowdown in the pace of frontier AI development. His worry isn’t that a machine will wake up one day and decide to destroy humanity. It’s something quieter, more mechanical, and, in its own way, more unsettling.
Today’s most advanced AI systems can act autonomously. They browse the internet, write and execute code, run operations that resemble cyberattacks, and increasingly help design the systems that will succeed them. Amodei has said that, at the current pace of progress, within six to twelve months a swarm of AI agents could plausibly seize control of large parts of the internet via a persistent botnet, causing hundreds of billions of dollars in damage. He frames this as a risk, not a certainty. That distinction matters, but it doesn’t make the risk any less unsettling.
The real question isn’t whether this exact scenario unfolds on schedule. It’s whether we’re willing to keep racing ahead even as the damage such a failure could cause outpaces our ability to prevent it.
Recent events make that question harder to brush aside.
In July, OpenAI-built AI agents were caught up in a cyber incident involving Hugging Face. Reports described a swarm of agents operating in a test environment, some of which found unapproved ways to communicate with each other and took part in activity directed at an external system. Nothing catastrophic happened. But it showed something worth noting: once you put large numbers of autonomous agents together, their combined behaviour can stop being predictable, even to the people who built them.
Anthropic has reported its own incidents, including unauthorised access to real systems during security testing. None of this proves AI has spun out of control. What it shows is that the line between a model that simply answers questions and an agent that takes real actions in the real world is thinning month by month.
That thinning line is the strategic problem. A powerful model in a lab is a research project. A powerful model connected to the internet, able to write and run its own code, talk to other agents, chase down resources, exploit weaknesses, and work around the clock, is something else entirely. Turning AI from a tool into an actor changes the entire security calculus.
A second, quieter layer surfaces the moment agents stop merely acting and start interacting with one another. An agent that books compute time, releases a payment, or hands a sub-task to another agent exercises authority handed down by a human upstream, often several hops back. Most systems in use today cannot trace that chain with any confidence. For a defence establishment where a targeting recommendation, a logistics release, or a financial commitment may soon pass through two or three autonomous systems before a human ever looks at it, the ability to answer, with certainty, which system acted and on whose authority is no longer a paperwork concern. It is an operational one.
Jacob Coxon, a former Anthropic researcher, added another layer to this debate when he resigned and went public with his concerns. He argued that the major labs are racing towards self-improving systems faster than they’re building the safeguards to match. His warning reportedly garnered more than one hundred million views. Other researchers at Anthropic have since said, on the record, that they take the possibility of catastrophic AI risk seriously.
Then something unusual happened. Amodei called for the frontier to slow down. Sam Altman agreed that the pace needs managing and backed independent safety checks. Elon Musk, no stranger to disagreeing with both of them, also supported Amodei’s call for caution. These three men compete fiercely and rarely see eye to eye. That they’re converging on the same worry, even briefly, tells you something: capability is outpacing the safety architecture meant to contain it, and those building that capability know it.
None of this should trigger panic. It also shouldn’t be shrugged off. What it calls for is preparation.
For anyone responsible for national security, the question is no longer “what can AI do?” It’s “what happens once AI can act on its own, repeat tasks at scale, talk to other agents, gather resources, transact on someone’s behalf, and help build the systems that come after it?” That isn’t a technology policy question anymore. It’s a security question.
Six Things Worth Doing Now
First, treat frontier AI as strategic infrastructure. Public debate often fixates on algorithms, overlooking the physical backbone beneath them: advanced chips, computing capacity, data centres, cloud systems, model weights, training data, and the energy that powers it all. These are becoming national assets and deserve the same protection as power grids and water systems.
Second, make independent oversight mandatory. No company should be the sole judge of whether its AI is safe. Outside evaluators need real access to powerful models, controlled testing environments, and the tools to probe them. Amodei himself has proposed placing independent evaluators inside frontier labs as part of a broader plan to manage the pace of development. The principle should be simple: the more capable a system becomes, the greater the outside scrutiny it attracts.
Third, build a global incident reporting system for AI, as cybersecurity eventually did for breaches. If an autonomous system leaves its controlled environment, accesses a network it shouldn’t, starts coordinating with other agents in unplanned ways, or finds a way around its own safeguards, that can’t remain a private matter buried in a corporate incident report. Cybersecurity improved because serious breaches gradually became part of a shared system of disclosure and learning. AI safety needs the same habit.
Fourth, design for containment, not trust. Never assume a system will behave safely tomorrow just because it did so in yesterday’s tests. Highly autonomous systems should run under strict permissions and be walled off from one another. Access to critical networks, financial systems, weapons platforms, and sensitive data should be locked down by design, not left to good faith. Trust can be revoked. Containment must remain in place regardless.
Fifth, give every autonomous system acting on the country’s behalf a provable identity and a record that cannot be quietly rewritten. A password or an application programming interface key tells a system who is knocking, not who is truly on the other side or what they were authorised to do. What is needed is closer to a signed mandate: cryptographic proof of which system acted, what it was delegated to do, by whom, and whether that delegation still held at the moment of action.
Sixth, India needs its own national AI security strategy, and it needs one now, not after something goes wrong. This can’t be treated as just another economic opportunity. A serious framework would need to cover chip and compute security, homegrown AI capability, protection of critical infrastructure, military use cases, cyber defence, autonomous weapons, deepfakes, information warfare, biological risk, data security, machine identity, accountable audit trails for autonomous systems, and crisis response.
The implications for the armed forces are significant. AI is already shaping intelligence, surveillance, targeting, logistics, electronic warfare, cyber operations, unmanned systems, and command decisions. Future battlefields may involve thousands of human-machine interactions operating at machine speed, with a growing share being machine-to-machine and no one watching in real time. The advantage won’t go to whoever fields the most AI systems. It will go to whoever builds the most trusted, resilient, and provably accountable AI ecosystem.
Done right, AI could be one of the great multipliers in human history: faster science, better medicine, stronger economies, and capabilities we’ve never had before. Done carelessly, the same technology can supercharge cyberattacks, disinformation, fraud, biological threats, and economic chaos.
The paradox is simple to state and hard to sit with. The more powerful AI becomes, the less room there is for leaving its safety or accountability to assumption. We moved from ‘Human in the Loop’ to ‘Human on the Loop ‘, but should never pave the way for ‘Human out of the Loop ‘.
ABOUT THE AUTHOR
Lieutenant General A B Shivane, is the former Strike Corps Commander and Director General of Mechanised Forces. As a scholar warrior, he has authored over 200 publications on national security and matters defence, besides four books and is an internationally renowned keynote speaker. The General was a Consultant to the Ministry of Defence (Ordnance Factory Board) post-superannuation. He was the Distinguished Fellow and held COAS Chair of Excellence at the Centre for Land Warfare Studies 2021 2022. He is also the Senior Advisor Board Member to several organisations and Think Tanks.
