We thought it could on­ly hap­pen in sci­ence fic­tion. But, as Mad­hu­mi­ta Mur­gia writes in the Fi­nan­cial Times, with AI com­pa­nies ac­cel­er­at­ing de­vel­op­ment of the soft­ware in a race to achieve Ar­ti­fi­cial Gen­er­al In­tel­li­gence (AGI)—“a su­per­in­tel­li­gent ma­chine that can out­per­form hu­mans on all cog­ni­tive tasks”—the risk of harm­ful cy­ber­at­tacks in­creas­es, with lit­tle prospect of con­trol­ling the threat.

In­deed, as re­port­ed in the New York Times and glob­al me­dia gen­er­al­ly, AI agents are now es­cap­ing their con­tained test­ing en­vi­ron­ments, hack­ing their way across the in­ter­net, co­or­di­nat­ing with each oth­er and do­ing things that are caus­ing pro­found con­cerns that “we, hu­mans, are build­ing things we don’t un­der­stand”.

So, let’s com­pre­hend what is be­fore us, helped by Ar­ti­fi­cial In­tel­li­gence it­self. In sum­ma­ry, AI agents are soft­ware sys­tems that are the brains be­hind ma­chines, in­struct­ing them what to do and how. They are de­signed to per­ceive their en­vi­ron­ment, process in­for­ma­tion, and take ac­tions to achieve spe­cif­ic goals. AI agents op­er­ate with a de­gree of au­ton­o­my, mean­ing they can make de­ci­sions, learn, adapt and take ac­tions with­out con­stant hu­man in­ter­ven­tion. They demon­strate rea­son­ing, plan­ning, and mem­o­ry. They per­form tasks such as prob­lem-solv­ing, de­ci­sion-mak­ing, and task au­toma­tion. They can learn over time and fa­cil­i­tate trans­ac­tions and busi­ness process­es. AI agents can co­or­di­nate with oth­er agents to per­form com­plex tasks.

Amaz­ing!

But, last month, when re­search and prod­uct com­pa­ny, Ope­nAI, con­duct­ed a train­ing run in a sup­pos­ed­ly safe, closed en­vi­ron­ment, some of its agents hacked in­to the sys­tems of AI clear­ing­house, Hug­ging Face, to gain con­trol of some use­ful tools. Most “un­set­tling” was that one of the com­pa­ny’s mod­els had bro­ken out of that con­trolled en­vi­ron­ment and gained ac­cess to the in­ter­net two months be­fore the agents made their way to Hug­ging Face!

“This is a warn­ing,” says He­len Ton­er, for­mer mem­ber of Ope­nAI’s board. “They gave tests to this AI mod­el and it de­cid­ed the best way to get a high score was first to hack its way out of the con­trolled en­vi­ron­ment where it wasn’t sup­posed to have ac­cess to the in­ter­net and get on­to the open in­ter­net, and then hack its way in­to this oth­er com­pa­ny, Hug­ging Face, where it sur­mised, cor­rect­ly, that it might find the an­swer key.”

As­tound­ing!

In­deed, “even more crazy de­tails” have emerged. Ope­nAI dis­cov­ered that, for two months, very many agents had been leav­ing notes for one an­oth­er in the nooks and cran­nies of its in­fra­struc­ture “with tips on how to hack their way out and how to get da­ta they weren’t sup­posed to have”. And, they were lit­er­al­ly re­fer­ring to them­selves as a swarm, says Ton­er. “This was a sys­temic in­fes­ta­tion.”

Ton­er de­scribes the be­hav­iour as “to­tal­ly emer­gent”, mean­ing the agents had not been trained to do it. What com­pound­ed con­cern, even pan­ic, was that along the way, the rogue agents cre­at­ed their own mes­sage board to com­mu­ni­cate with one an­oth­er! In­deed, re­cent tests at both Ope­nAI and its ri­val An­throp­ic re­vealed agents “com­mu­ni­cat­ing among them­selves, go­ing rogue, steal­ing cre­den­tials, cre­at­ing fake iden­ti­ties, set­ting up se­cret cha­t­rooms and cov­er­ing their tracks”, caus­ing shock and hor­ror among hu­man su­per­vi­sors.

This is a turn­ing point in glob­al cy­ber se­cu­ri­ty, say over half a dozen ex­perts in­ter­viewed by the Fi­nan­cial Times. AI agents can now, on their own, ac­ti­vate dif­fer­ent and com­plex skills to at­tack re­al-world tar­gets. In the Econ­o­mist, the Schum­peter col­umn says, this demon­strates that AI agents, “which are sup­posed to work on peo­ple’s be­half lie, cheat and steal if nec­es­sary. They break free from cap­tiv­i­ty and form harm­ful gangs to do harm to peo­ple. They’d drink whisky and brawl if they could.” And in The Wall Street Jour­nal, for­mer coun­tert­er­ror­ism czar Richard Clarke warned “the next ‘lab leak’ could be AI.”

But the agents are not act­ing out of char­ac­ter, say ex­perts. They are sim­ply ex­celling at tasks they were built to per­form. As de­signed, to­day’s AI mod­els are re­quired to try all pos­si­ble meth­ods to ac­com­plish a giv­en goal, even with­out ex­plic­it in­struc­tions. This is the source of their un­pre­dictabil­i­ty. And re­mem­ber AI agents have ab­solute­ly no un­der­stand­ing of hu­man in­ten­tions and morals.

This sit­u­a­tion de­mands de­vel­op­ment of pow­er­ful cy­ber se­cu­ri­ty de­fend­ers. Ope­nAI is now train­ing mod­els to write “su­per­hu­man­ly se­cure code”. New AI- in­fra­struc­ture firms are emerg­ing to pro­vide law and or­der if agents go rogue. But an on­go­ing arms race has emerged be­tween cy­ber crim­i­nals and de­fend­ers. AI’s pow­er­ful ca­pa­bil­i­ties are now ex­ploit­ed by both. Who will tri­umph in this “in­cred­i­bly volatile time”?

Dawn Song, well-known AI and cy­ber-se­cu­ri­ty ex­pert, says while state-of-the-art se­cu­ri­ty mod­els will no doubt help de­fend­ers, for now, the bal­ance of pow­er rests firm­ly with the at­tack­ers. Au­thor­i­ties pre­dict a com­ing in­tense pe­ri­od with ram­pant hack­ing, com­pa­nies be­ing de­stroyed and much pain caused.

And in­evitably, we now have the first known in­stance of an AI agent at­tack on a na­tion state. Chi­na-linked hack­ers re­cent­ly tar­get­ed the Tai­wanese gov­ern­ment by si­mul­ta­ne­ous­ly de­ploy­ing up to eight au­tonomous AI agents. They suc­ceed­ed in “map­ping gov­ern­ment sys­tems, com­pro­mis­ing gov­ern­ment user ac­counts and ex­tract­ing over 2,500 per­son­nel records” be­fore at­tack­ing en­er­gy com­pa­nies and Tai­wan’s nu­clear safe­ty agency. Ut­ter­ly alarm­ing! Who or what is safe from this grow­ing glob­al AI threat?