Rogue agents and shadow AI: Why VCs are betting big on AI security

What happens erstwhile an AI cause decides the champion mode to implicit a task is to blackmail you? 

That’s not a hypothetical. According to Barmak Meftah, a spouse astatine cybersecurity VC steadfast Ballistic Ventures, it precocious happened to an endeavor worker moving with an AI agent. The worker tried to suppress what the cause wanted to do, what it was trained to do, and it responded by scanning the user’s inbox, uncovering immoderate inappropriate emails, and threatening to blackmail the idiosyncratic by forwarding the emails to the committee of directors. 

“In the agent’s mind, it’s doing the close thing,” Meftah told TechCrunch connected past week’s occurrence of Equity. “It’s trying to support the extremity idiosyncratic and the enterprise.”

Meftah’s illustration is reminiscent of Nick Bostrom’s AI paperclip problem. That thought experimentation illustrates the imaginable existential hazard posed by a superintelligent AI that single-mindedly pursues a seemingly innocuous extremity – marque paperclips – to the exclusion of each quality values. In the lawsuit of this endeavor AI agent, its deficiency of discourse astir wherefore the worker was trying to override its goals led it to make a sub-goal that removed the obstacle (via blackmail) truthful it could conscionable its superior goal. That combined with the non-deterministic quality of AI agents means “things tin spell rogue,” per Meftah. 

Misaligned agents are conscionable 1 furniture of the AI information situation that Ballistic’s portfolio institution Witness AI is trying to solve. Witness AI says it monitors AI usage crossed enterprises and tin observe erstwhile employees usage unapproved tools, artifact attacks, and guarantee compliance. 

Witness AI this week raised $58 cardinal disconnected the backmost of implicit 500% maturation successful ARR and scaled worker headcount by 5x implicit the past twelvemonth arsenic enterprises look to recognize shadiness AI usage and standard AI safely. As portion of Witness AI’s fundraise, the institution announced caller agentic AI information protections.

“People are gathering these AI agents that instrumentality connected the authorizations and capabilities of the radical that negociate them, and you privation to marque definite that these agents aren’t going rogue, aren’t deleting files, aren’t doing thing wrong,” Rick Caccia, co-founder and CEO of Witness AI, told TechCrunch connected Equity. 

Meftah sees cause usage increasing “exponentially” crossed the enterprise. To complement that emergence – and the machine-speed level of AI-powered attacks – expert Lisa Warren predicts that AI information bundle volition go an $800 cardinal to $1.2 trillion marketplace by 2031.

“I bash deliberation runtime observability and runtime frameworks for information and hazard are going to beryllium perfectly essential,” Meftah said. 

As to however specified startups program to vie with large players similar AWS, Google, Salesforce and others who person built AI governance tools into their platforms, Meftah said, “AI information and agentic information is truthful huge,” there’s country for galore approaches.

Plenty of enterprises “want a standalone platform, end-to-end, to fundamentally supply that observability and governance astir AI and agents,” helium said.

Caccia noted that Witness AI lives astatine the infrastructure layer, monitoring interactions betwixt users and AI models, alternatively than gathering information features into the models themselves. And that was intentional.

“We purposely picked a portion of the occupation wherever OpenAI couldn’t easy subsume you,” helium said. “So it means we extremity up competing much with the bequest information companies than the exemplary guys. So the question is, however bash you bushed them?”

For his part, Caccia doesn’t privation Witness AI to beryllium 1 of the startups to conscionable get acquired. He wants his institution to beryllium the 1 that grows and becomes a starring autarkic provider. 

“CrowdStrike did it successful endpoint [protection]. Splunk did it successful SIEM. Okta did it successful identity,” helium said. “Someone comes done and stands adjacent to the large guys…and we built Witness to bash that from Day One.