Written by Kevin B. McDonald, COO & CISO at Alvaka

The Real Shift: AI Is Becoming the Operator

Like so many recent reports, Google’s latest GTIG report on adversarial AI should make executives, security leaders, and MSPs uncomfortable. Not because criminals are using AI. We already knew that. Not because phishing emails are getting better. We knew that too. The real story is about cybercriminals abandoning manual work and potentially shrinking the entire cybercrime ecosystem.  

For the last few years, most discussions around AI in business and cybercrime have focused on productivity. We have all acknowledged that threat actors write phishing emails, research victims, find vulnerability, and create malware faster. These are all a growing list of threats causing us to collectively lose sleep. What Google’s GTIG report discusses is something different. 

Attackers are no longer treating AI as a tool for human operators to use. They are beginning to use AI as the operator itself. 

According to the report, cyber criminals are fast moving from simple prompting and code help to autonomous agentic workflows. These systems can now act, solve problems, adapt to obstacles in real time and reach objectives with limited to no human involvement.  

What an AI-Driven Attack Looks Like in the Real World

I have seen the results of advanced AI deployment firsthand. A victim contacted me and explained that their finance team had been hit with a $1.5 Million BEC attack. They were very sophisticated and well aware of the common risks. Their vendor’s email account is believed to have been hacked by AI phishing. It then reviewed the emails in the vendor’s account and identified an invoice that was coming due. The AI then drafted an email to the victim customer in the tone and email “voice” of the vendor finance person. This is someone the victim frequently communicated with. The email came from a legitimate account, reminding the victim of the real debt and that a final reminder would be coming in a few days and new account information would be included. The sender’s signature salutation was unique and something only the usual sender would say.

Days later, the victim received the promised reminder email with a note to enter new account information in the vendor’s official portal that the victim commonly used. In the email was an invoice and embedded link to a website with a highly similar web address. The victim clicked the payment link as was traditional between the two companies. I must note that the victim acknowledged that she had concerns about changing the account information, but started the payment process anyway because everything felt normal. When she clicked the link, it took her to a website that was a flawless complete duplication of the vendor’s complex website. The victim clicked around, and the pages looked, acted and, most importantly, felt normal to her. All of the details she looked at down to the contact section, chat bot, and other functions were what she was used to seeing. The victim felt confident she had verified the site and went through the payment process and included the new banking information for the pay to account.

It was only after she received an email notice of non-payment a couple days later that she realized something was terribly wrong. To make a long story short, the attack from initial compromise to, website replication and the financial transaction is estimated to have happened in about 72 hours. Thankfully, we were able to assist them in a nearly miraculous recovery (as so much time had passed) with the help of federal law enforcement. This could have been avoided by a phone call, but we all operate in the familiar and many have said to me, I don’t want to look paranoid. Please be paranoid, make the call. Do not trust what you see or hear. Do not trust any incoming communication you did not initiate.

Attack Timelines Are Collapsing

Yes, 72 hours seems ludicrous but Google shares in their report, one example where adversaries reportedly compromised cloud resources and planned, built, and executed a credential harvesting attack in less than six hours. A reality we all face is that attack cycles continue to gain sophistication and shrink timelines in ways we could not have imagined just a few months ago. We defenders are still operating on human timelines. 

Companies need incident response systems, processes and time for investigations. We need people or systems to catch the actions, review alerts, and make decisions. Threat actors are increasingly building systems that do not need any of these time wasting steps, because the risk of damage is not theirs. Worst case for them, the Ai causes destruction and they fail to get paid.  

A key takeaway from this report is not that AI is just becoming smarter. AI is becoming faster than many organizations can reasonably detect and respond effectively. 

AI Is Becoming a Target, Too

Our industry has spent years discussing how AI could help businesses and was a threat to traditional security. Far less attention has focused on AI assets and Google calls this out. Model weights, proprietary prompts and guardrails, research, source code, API credentials, and cloud compute are examples of valuable targets being sought by cybercriminals’ AI.  

One of my greatest frustrations as a cyber defender is organizations rushing to build AI capabilities without deep consideration of how those systems will be protected. Tech history tells us that never ends well. Many executives have convinced themselves that AI coding tools somehow improve security. In reality, they are proven to create less secure code and a greatly expanded attack surface. Every automation becomes another layer that can be accessed and manipulated. 

We Are Repeating an Old Security Mistake at AI Speed

I have written extensively on the same pattern repeated for decades. We introduce new technologies for efficiency and features before security. Cybercriminals learn how to exploit vulnerability, defenders implement controls, and the criminals adapt yet again. The difference now is the speed of adaptation. AI is hyper accelerating it. 

Too many organizations do not know where their AI systems are or how they are being used. They do not know what employees are connecting to or sharing with them. They are unaware of vendors building AI into products that have access to sensitive information and yet they continue deploying it. 

This Is Not Science Fiction

The threats described in Google’s report are not science fiction involving self-aware machines. It is much more practical and therefore far more dangerous.  

For clients and MSPs alike, this means identity management, access controls, monitoring, and incident readiness are more critical than ever. The traditional assumption that we humans will catch and stop attacks before damage is done, is becoming less realistic by the day. 

What Executives Should Do Now

  1. Require verbal verification for money movement: No changes to banking, payment, wire, or vendor account information without a phone call to a known contact.
  2. Inventory every AI system in the business: Know what AI tools employees and vendors are using, what data they access, and where sensitive information is flowing.
  3. Treat AI as an attack surface: Apply the same security controls to AI platforms, prompts, APIs, and data stores that you apply to other critical systems.
  4. Strengthen identity security: Prioritize MFA, conditional access, privileged access controls, and continuous monitoring. Identity is increasingly the primary target.
  5. Assume attacks will happen at machine speed: Review incident response plans and eliminate delays that require unnecessary human intervention.
  6. Train employees to distrust familiarity: Legitimate accounts, familiar voices, realistic websites, and accurate business context are no longer proof of legitimacy.
  7. Evaluate vendor AI risk: Ask vendors what AI features are enabled, what data is exposed, and how those systems are secured.
  8. Review all AI-generated code and automations: Don’t assume efficiency improvements equal security improvements. Every automation creates new opportunities for abuse.
  9. Implement rapid escalation procedures: Staff should know exactly who to call when something feels wrong, even if they cannot explain why.
  10. Promote a culture of healthy paranoia: Make it acceptable to verify, question, and challenge unusual requests. A five-minute phone call is often cheaper than a six-figure loss.

Bottom Line

The era of assuming humans will spot and stop attacks before damage occurs is ending. Know where your AI is, protect your identities, verify financial transactions out-of-band, and prepare your organization to respond at machine speed.