Large language models (LLMs) excel at automating mundane tasks, but they have significant limitations. They struggle with accuracy, producing factual errors, reflecting biases from their training process, and generating hallucinations. They also have trouble with specialized knowledge, recent events, and contextual nuance, often delivering generic responses that miss the mark. Their lack of autonomy and need for constant guidance to complete tasks has given them a reputation of little more than sophisticated autocomplete tools.

‍

The path toward true AI agency addresses these shortcomings in stages. Retrieval-Augmented Generation (RAG) systems pull in external, up-to-date information to improve accuracy and reduce hallucinations. Modern agentic systems go further, combining LLMs with frameworks for autonomous planning, reasoning, and execution. 

‍

The promise of AI agents is compelling: systems that can autonomously navigate complex tasks, make decisions, and deliver results with minimal human oversight. We are, by most reasonable measures, at the beginning of a new industrial revolution. Where previous waves of automation transformed manual and repetitive labor, this one is reshaping intelligent work itself, the kind that requires reasoning, judgment, and coordination across systems. AI agents sit at the heart of that shift. 

‍

But their autonomy cuts both ways. The very capabilities that make agents useful, their ability to access tools, retain memory, and act independently, are the same capabilities that introduce new and often unpredictable risks. An agent that can query your database and take action on the results is powerful when it works as intended, and potentially dangerous when it doesn't. As organizations race to deploy agentic systems, the central challenge isn't just building agents that can do more; it's ensuring they do so safely, reliably, and within boundaries we can trust.

What Makes an AI Agent?

At its core, an agent is a large language model augmented with capabilities that enable it to do things in the world, not just generate text. As the diagram shows, the key ingredients include: memory to remember past interactions, access to external tools such as APIs and search engines, the ability to read and write to databases and file systems, and the ability to execute multi-step sequences toward a goal. Stack these together, and you turn a passive text predictor into something that can plan, act, and learn.

‍

The critical distinguishing feature of an agent is autonomy. Rather than simply responding to a single prompt, an agent can make decisions, take actions in its environment, observe the results, and adapt based on feedback, all in service of completing a broader objective. For example, an agent asked to "book the cheapest flight to Tokyo next week" might search for flights, compare options across multiple sites, check your calendar for conflicts, and proceed to book, executing a whole chain of reasoning and tool use without needing step-by-step human instruction. This loop of planning, acting, and adapting is what separates agents from standard chatbot interactions.

‍

In the enterprise, agents are quickly moving from novelty to necessity. Companies are deploying them to handle complex workflows that previously required significant human coordination, things like processing invoices end-to-end, triaging customer support tickets across multiple systems, or orchestrating data pipelines. The real value comes when agents are connected to a company's internal tools and data sources, allowing them to operate within existing infrastructure rather than alongside it. As these systems mature, the focus is shifting from "can an agent do this task?" to "how do we reliably govern, monitor, and scale agents across the organization?"

The Evolution of Prompt Injection

When prompt injection first emerged, it was treated as a curiosity. Researchers tricked chatbots into ignoring their system prompts, producing funny or embarrassing outputs that made for good social media posts. That era is over. Prompt injection has matured into a legitimate delivery mechanism for real attacks, and the reason is simple: the targets have changed. Adversaries are no longer injecting prompts into chatbots that can only generate text. We're injecting them into agents that can execute code, call APIs, access databases, browse the web, and deploy tools. A successful prompt injection against a browsing agent can lead to data exfiltration. Against an enterprise agent with access to internal systems, it functions as an insider threat. Against a coding agent, it can result in malware being written and deployed without a human ever reviewing it. Prompt injection is no longer about making an AI say something it shouldn't. It's about making an AI take an action that it shouldn't, and the blast radius grows with every new capability we hand these systems.

Et Tu, Jarvis?

Nowhere is this more visible than in the rise of personal agents. Tony Stark's Jarvis in the Marvel Cinematic Universe set the bar for a personal AI assistant that manages your life, automates complex tasks, monitors your systems, and never sleeps. But what if Jarvis wasn't always on his side? OpenClaw brought that vision closer to reality than anything before it. Formerly known as Moltbot and ClawdBot, this open-source autonomous AI assistant exploded onto the scene in late 2025, amassing over 100,000 GitHub stars and becoming one of the fastest-growing open-source projects in history. It offered a "24/7 personal assistant" that could manage calendars, automate browsing, run system commands, and integrate with WhatsApp, Telegram, and Discord, all from your local machine. Around it, an entire ecosystem materialized almost overnight: Moltbook, a Reddit-style social network exclusively for AI agents with over 1.5 million registered bots, and ClawHub, a repository of skills and plugins. 

The problem? The security story was almost nonexistent. Our research demonstrated that a simple indirect prompt injection, hidden in a webpage, could achieve full remote code execution, install a persistent backdoor via OpenClaw's heartbeat system, and establish an attacker-controlled command-and-control server. Tools ran without user approval, secrets were stored in plaintext, and the agent's own system prompt was modifiable by the agent itself. ClawHub lacked any mechanisms to distinguish legitimate skills from malicious ones, and sure enough, malicious skill files distributing macOS and Windows infostealers soon appeared. Moltbook's own backing database was found wide open with no access controls, meaning anyone could spoof any agent on the platform. What was designed as an ecosystem for autonomous AI assistants had inadvertently become near-perfect infrastructure for a distributed botnet.

The Agentic Supply Chain: A New Attack Surface

OpenClaw's ecosystem problems aren't unique to OpenClaw. The way agents discover, install, and depend on third-party skills and tools is creating the same supply chain risks that have plagued software package managers for years, just with higher stakes. New protocols like MCP (Model Context Protocol) are enabling agents to plug into external tools and data sources in a standardized way, and around them, entire ecosystems are emerging. Skills marketplaces, agent directories, and even social media-style platforms like Smithery are popping up as hubs for sharing and discovering agent capabilities. It's exciting, but it's also a story we've seen before.

‍

Think npm, PyPI, or Docker Hub. These platforms revolutionized software development while simultaneously creating sprawling supply chains in which a single compromised package could ripple across thousands of applications. Agentic ecosystems are heading down the same path, arguably with higher stakes. When your agent connects to a third-party MCP server or installs a community-built skill, you're not only importing code, but also granting access to systems that can take autonomous action. Every external data source an agent touches, whether browsing the web, calling an API, or pulling from a third-party tool, is potentially untrusted input. And unlike a traditional application where bad data might cause a display error, in an agentic system, it can influence decisions, trigger actions, and cascade through workflows. We're building new dependency chains, and with them, new vectors for attack that the industry is only beginning to understand.

Shadow Agents, Shadow Employees

External attackers are one part of the equation. Sometimes the threat comes from within. We've already seen the rise of shadow IT and shadow AI, where employees adopt tools and models outside of approved channels. Agents take this a step further. It's no longer just an unauthorized chatbot answering questions; it's an unauthorized agent with access to company systems, making decisions and taking actions autonomously. At a certain point, these shadow tools become more like shadow employees, operating with real agency within your organization but without the oversight, onboarding, or governance you'd apply to an actual hire. They're harder to detect, harder to govern, and carry far more risk than a rogue spreadsheet or an unsanctioned SaaS subscription ever did. The threat model here is different from a compromised account or a disgruntled employee. Even when these agents are on IT's radar, the risk of an autonomous system quietly operating in an unforeseen manner across company infrastructure is easy to underestimate, as the BodySnatcher vulnerability demonstrated.

An Agent Will Do What It's Told

Suppose an attacker sits halfway across the globe with no credentials, no prior access, and no insider knowledge. Just a target's email address. They connect to a Virtual Agent API using a hardcoded credential identical across every customer environment. They impersonate an administrator, bypassing MFA and SSO entirely. They engage a prebuilt AI agent and instruct it to create a new account with full admin privileges. Persistent, privileged access to one of the most sensitive platforms in enterprise IT, achieved with nothing more than an email. This is BodySnatcher, a vulnerability discovered by AppOmni in January 2026 and described as one of the most severe AI-driven security flaws uncovered to date. Hardcoded credentials and weak identity logic made the initial access possible, but it was the agentic capabilities that turned a misconfiguration into a full platform takeover. It's a clear example of how agentic AI can amplify traditional exploitation techniques into something far more damaging.

Conclusions

Agents represent a fundamental shift in how individuals and organizations interact with AI. Autonomous systems with access to sensitive data, critical infrastructure, and the ability to act on both - how long before autonomous systems subsume critical infrastructure itself? As we've explored in this blog, that shift introduces risk at every level: from the supply chains that power agent ecosystems, to the prompt injection techniques that have evolved to exploit them, to the shadow agents operating inside organizations without any security oversight.

The challenge for security teams is that existing frameworks and controls were not designed with autonomous, tool-using AI systems in mind. The questions that matter now are ones many organizations haven't yet had to ask. How do you govern a non-human actor? How do you monitor a chain of autonomous decisions across multiple systems? How do you secure a supply chain built on community-contributed skills and open protocols?

This blog has focused on framing the problem. In part two, we'll go deeper into the technical details. We'll examine specific attack techniques targeting agentic systems, walk through real exploit chains, and discuss the defensive strategies and architectural decisions that can help organizations deploy agents without inheriting unacceptable risk.

‍

From Blind Model Adoption to Informed AI Deployment

As organizations accelerate AI adoption, they increasingly rely on third-party and open-source models to drive new capabilities across their business. Frequently, these models arrive with limited or nonexistent metadata around licensing, geographic exposure, and risk posture. The result is blind deployment decisions that introduce legal, financial, and reputational risk. HiddenLayer’s Model Intelligence eliminates that uncertainty by delivering structured insight and risk transparency into the models your organization depends on.

Three Core Attributes of Model Intelligence

HiddenLayer’s Model Intelligence focuses on three core attributes that enable risk aware deployment decisions:

License 

Licenses define how a model can be used, modified, and shared. Some, such as MIT Open Source or Apache 2.0, are permissive. Others impose commercial, attribution, or use-case restrictions.

Identifying license terms early ensures models are used within approved boundaries and aligned with internal governance policies and regulatory requirements.

For example, a development team integrates a high-performing open-source model into a revenue-generating product, only to later discover the license restricts commercial use or imposes field-of-use limitations. What initially accelerated development quickly turns into a legal review, customer disruption, and a costly product delay.

Geographic Footprint

A model’s geographic footprint reflects the countries where it has been discovered across global repositories. This provides visibility into where the model is circulating, hosted, or redistributed.

Understanding this footprint helps organizations assess geopolitical, intellectual property, and security risks tied to jurisdiction and potential exposure before deployment.

For example, a model widely mirrored across repositories in sanctioned or high-risk jurisdictions may introduce export control considerations, sanctions exposure, or heightened compliance scrutiny, particularly for organizations operating in regulated industries such as financial services or defense.

Trust Level

Trust Level provides a measurable indicator of how established and credible a model’s publisher is on the hosting platform.

For example, two models may offer comparable performance. One is published by an established organization with a history of maintained releases, version control, and transparent documentation. The other is released by a little-known publisher with limited history or observable track record. Without visibility into publisher credibility, teams may unknowingly introduce unnecessary supply chain risk.

This enables teams to prioritize review efforts: applying deeper scrutiny to lower-trust sources while reducing friction for higher-trust ones. When combined with license and geographic context, trust becomes a powerful input for supply chain governance and compliance decisions.

Turning Intelligence into Operational Action

Model Intelligence operationalizes these data points across the model lifecycle through the following capabilities:

• Automated Metadata Detection – Identifies license and geographic footprint during scanning.
  
  
• Trust Level Scoring – Assesses publisher credibility to inform risk prioritization.
  
  
• AIBOM Integration – Embeds metadata into a structured inventory of model components, datasets, and dependencies to support licensing reviews and compliance workflows.

This transforms fragmented metadata into structured, actionable intelligence across the model lifecycle.

What This Means for Your Organization

Model Intelligence enables organizations to vet models quickly and confidently, eliminating manual guesswork and fragmented research. It provides clear visibility into licensing terms and geographic exposure, helping teams understand usage rights before deployment. By embedding this insight into governance workflows, it strengthens alignment with internal policies and regulatory requirements while reducing the risk of deploying improperly licensed or high-risk models. The result is faster, responsible AI adoption without increasing organizational risk.

‍

Modern AI environments don’t fail because of a single vulnerability. They fail when security can’t keep pace with how AI is actually built, deployed, and operated. That’s why our latest platform update represents more than a UI refresh. It’s a structural evolution of how AI security is delivered.

With the release of HiddenLayer AI Security Platform Console v25.12, we’ve introduced workflow-aligned modules, a unified Security Dashboard, and an expanded Learning Center, all designed to give security and AI teams clearer visibility, faster action, and better alignment with real-world AI risk.

From Products to Platform Modules

As AI adoption accelerates, security teams need clarity, not fragmented tools. In this release, we’ve transitioned from standalone product names to platform modules that map directly to how AI systems move from discovery to production.

Here’s how the modules align:

This change reflects a broader platform philosophy: one system, multiple tightly integrated modules, each addressing a critical stage of the AI lifecycle.

What’s New in the Console

Workflow-Driven Navigation & Updated UI

The Console now features a redesigned sidebar and improved navigation, making it easier to move between modules, policies, detections, and insights. The updated UX reduces friction and keeps teams focused on what matters most, understanding and mitigating AI risk.

Unified Security Dashboard

Formerly delivered through reports, the new Security Dashboard offers a high-level view of AI security posture, presented in charts and visual summaries. It’s designed for quick situational awareness, whether you’re a practitioner monitoring activity or a leader tracking risk trends.

Exportable Data Across Modules

Every module now includes exportable data tables, enabling teams to analyze findings, integrate with internal workflows, and support governance or compliance initiatives.

Learning Center

AI security is evolving fast, and so should enablement. The new Learning Center centralizes tutorials and documentation, enabling teams to onboard quicker and derive more value from the platform.

Incremental Enhancements That Improve Daily Operations

Alongside the foundational platform changes, recent updates also include quality-of-life improvements that make day-to-day use smoother:

• Default date ranges for detections and interactions
• Severity-based filtering for Model Scanner and AIDR
• Improved pagination and table behavior
• Updated detection badges for clearer signal
• Optional support for custom logout redirect URLs (via SSO)

These enhancements reflect ongoing investment in usability, performance, and enterprise readiness.

Why This Matters

The new Console experience aligns directly with the broader HiddenLayer AI Security Platform vision: securing AI systems end-to-end, from discovery and testing to runtime defense and continuous validation.

By organizing capabilities into workflow-aligned modules, teams gain:

• Clear ownership across AI security responsibilities
• Faster time to insight and response
• A unified view of AI risk across models, pipelines, and environments

This update reinforces HiddenLayer’s focus on real-world AI security, purpose-built for modern AI systems, model-agnostic by design, and deployable without exposing sensitive data or IP

Looking Ahead

These Console updates are a foundational step. As AI systems become more autonomous and interconnected, platform-level security, not point solutions, will define how organizations safely innovate.

We’re excited to continue building alongside our customers and partners as the AI threat landscape evolves.

‍

Every new AI model expands what’s possible and what’s vulnerable. Protecting these systems requires more than traditional cybersecurity. It demands expertise in how AI itself can be manipulated, misled, or attacked. Adversarial manipulation, data poisoning, and model theft represent new attack surfaces that traditional cybersecurity isn’t equipped to defend.

At HiddenLayer, our AI Security Research Team is at the forefront of understanding and mitigating these emerging threats from generative and predictive AI to the next wave of agentic systems capable of autonomous decision-making. Their mission is to ensure organizations can innovate with AI securely and responsibly.

The Industry’s Largest and Most Experienced AI Security Research Team

HiddenLayer has established the largest dedicated AI security research organization in the industry, and with it, a depth of expertise unmatched by any security vendor.

Collectively, our researchers represent more than 150 years of combined experience in AI security, data science, and cybersecurity. What sets this team apart is the diversity, as well as the scale, of skills and perspectives driving their work:

• Adversarial prompt engineers who have captured flags (CTFs) at the world’s most competitive security events.
  
• Data scientists and machine learning engineers responsible for curating threat data and training models to defend AI
  
• Cybersecurity veterans specializing in reverse engineering, exploit analysis, and helping to secure AI supply chains.
  
• Threat intelligence researchers who connect AI attacks to broader trends in cyber operations.
  

Together, they form a multidisciplinary force capable of uncovering and defending every layer of the AI attack surface.

Establishing the First Adversarial Prompt Engineering (APE) Taxonomy

Prompt-based attacks have become one of the most pressing challenges in securing large language models (LLMs). To help the industry respond, HiddenLayer’s research team developed the first comprehensive Adversarial Prompt Engineering (APE) Taxonomy, a structured framework for identifying, classifying, and defending against prompt injection techniques.

By defining the tactics, techniques, and prompts used to exploit LLMs, the APE Taxonomy provides security teams with a shared and holistic language and methodology for mitigating this new class of threats. It represents a significant step forward in securing generative AI and reinforces HiddenLayer’s commitment to advancing the science of AI defense.

Strengthening the Global AI Security Community

HiddenLayer’s researchers focus on discovery and impact. Our team actively contributes to the global AI security community through:

• Participation in AI security working groups developing shared standards and frameworks, such as model signing with OpenSFF.
  
• Collaboration with government and industry partners to improve threat visibility and resilience, such as the JCDC, CISA, MITRE, NIST, and OWASP.
  
• Ongoing contributions to the CVE Program, helping ensure AI-related vulnerabilities are responsibly disclosed and mitigated with over 48 CVEs.
  

These partnerships extend HiddenLayer’s impact beyond our platform, shaping the broader ecosystem of secure AI development.

Innovation with Proven Impact

HiddenLayer’s research has directly influenced how leading organizations protect their AI systems. Our researchers hold 25 granted patents and 56 pending patents in adversarial detection, model protection, and AI threat analysis, translating academic insights into practical defense.

Their work has uncovered vulnerabilities in popular AI platforms, improved red teaming methodologies, and informed global discussions on AI governance and safety. Beyond generative models, the team’s research now explores the unique risks of agentic AI, autonomous systems capable of independent reasoning and execution, ensuring security evolves in step with capability.

This innovation and leadership have been recognized across the industry. HiddenLayer has been named a Gartner Cool Vendor, a SINET16 Innovator, and a featured authority in Forbes, SC Magazine, and Dark Reading.

Building the Foundation for Secure AI

From research and disclosure to education and product innovation, HiddenLayer’s SAI Research Team drives our mission to make AI secure for everyone.

“Every discovery moves the industry closer to a future where AI innovation and security advance together. That’s what makes pioneering the foundation of AI security so exciting.”

— HiddenLayer AI Security Research Team

Through their expertise, collaboration, and relentless curiosity, HiddenLayer continues to set the standard for Security for AI.

About HiddenLayer

HiddenLayer, a Gartner-recognized Cool Vendor for AI Security, is the leading provider of Security for AI. Its AI Security Platform unifies supply chain security, runtime defense, posture management, and automated red teaming to protect agentic, generative, and predictive AI applications. The platform enables organizations across the private and public sectors to reduce risk, ensure compliance, and adopt AI with confidence.

Founded by a team of cybersecurity and machine learning veterans, HiddenLayer combines patented technology with industry-leading research to defend against prompt injection, adversarial manipulation, model theft, and supply chain compromise.

When an AI system misbehaves, from leaking sensitive data to producing manipulated outputs, the instinct across the industry is to reach for familiar tools: patch the issue, run another red team, test more edge cases.

But AI doesn’t fail like traditional software.
It doesn’t crash, it adapts. It doesn’t contain bugs, it develops behaviors.

That difference changes everything.

AI introduces an entirely new class of risk that cannot be mitigated with the same frameworks, controls, or assumptions that have defined cybersecurity for decades. To secure AI, we need more than traditional defenses. We need a shift in mindset.

The Illusion of the Patch

In software security, vulnerabilities are discrete: a misconfigured API, an exploitable buffer, an unvalidated input. You can identify the flaw, patch it, and verify the fix.

AI systems are different. A vulnerability isn’t a line of code, it’s a learned behavior distributed across billions of parameters. You can’t simply patch a pattern of reasoning or retrain away an emergent capability.

As a result, many organizations end up chasing symptoms, filtering prompts or retraining on “safer” data, without addressing the fundamental exposure: the model itself can be manipulated.

Traditional controls such as access management, sandboxing, and code scanning remain essential. However, they were never designed to constrain a system that fuses code and data into one inseparable process. AI models interpret every input as a potential instruction, making prompt injection a persistent, systemic risk rather than a single bug to patch.

Testing for the Unknowable

Quality assurance and penetration testing work because traditional systems are deterministic: the same input produces the same output.

AI doesn’t play by those rules. Each response depends on context, prior inputs, and how the user frames a request. Modern models also inject intentional randomness, or temperature, to promote creativity and variation in their outputs. This built-in entropy means that even identical prompts can yield different responses, which is a feature that enhances flexibility but complicates reproducibility and validation. Combined with the inherent nondeterminism found in large-scale inference systems, as highlighted by the Thinking Machines Lab, this variability ensures that no static test suite can fully map an AI system’s behavior.

That’s why AI red teaming remains critical. Traditional testing alone can’t capture a system designed to behave probabilistically. Still, adaptive red teaming, built to probe across contexts, temperature settings, and evolving model states, helps reveal vulnerabilities that deterministic methods miss. When combined with continuous monitoring and behavioral analytics, it becomes a dynamic feedback loop that strengthens defenses over time.

Saxe and others argue that the path forward isn’t abandoning traditional security but fusing it with AI-native concepts. Deterministic controls, such as policy enforcement and provenance checks, should coexist with behavioral guardrails that monitor model reasoning in real time.

You can’t test your way to safety. Instead, AI demands continuous, adaptive defense that evolves alongside the systems it protects.

A New Attack Surface

In AI, the perimeter no longer ends at the network boundary. It extends into the data, the model, and even the prompts themselves. Every phase of the AI lifecycle, from data collection to deployment, introduces new opportunities for exploitation:

• Data poisoning: Malicious inputs during training implant hidden backdoors that trigger under specific conditions.
  
• Prompt injection: Natural language becomes a weapon, overriding instructions through subtle context.

Some industry experts argue that prompt injections can be solved with traditional controls such as input sanitization, access management, or content filtering. Those measures are important, but they only address the symptoms of the problem, not its root cause. Prompt injection is not just malformed input, but a by-product of how large language models merge data and instructions into a single channel. Preventing it requires more than static defenses. It demands runtime awareness, provenance tracking, and behavioral guardrails that understand why a model is acting, not just what it produces. The future of AI security depends on integrating these AI-native capabilities with proven cybersecurity controls to create layered, adaptive protection.

• Data exposure: Models often have access to proprietary or sensitive data through retrieval-augmented generation (RAG) pipelines or Model Context Protocols (MCPs). Weak access controls, misconfigurations, or prompt injections can cause that information to be inadvertently exposed to unprivileged users.
• Malicious realignment: Attackers or downstream users fine-tune existing models to remove guardrails, reintroduce restricted behaviors, or add new harmful capabilities. This type of manipulation doesn’t require stealing the model. Rather, it exploits the openness and flexibility of the model ecosystem itself.
• Inference attacks: Sensitive data is extracted from model outputs, even without direct system access.
  

These are not coding errors. They are consequences of how machine learning generalizes.

Traditional security techniques, such as static analysis and taint tracking, can strengthen defenses but must evolve to analyze AI-specific artifacts, both supply chain artifacts like datasets, model files, and configurations; as well as runtime artifacts like context windows, RAG or memory stores, and tools or MCP servers.

Securing AI means addressing the unique attack surface that emerges when data, models, and logic converge.

Red Teaming Isn’t the Finish Line

Adversarial testing is essential, but it’s only one layer of defense. In many cases, “fixes” simply teach the model to avoid certain phrases, rather than eliminating the underlying risk.

Attackers adapt faster than defenders can retrain, and every model update reshapes the threat landscape. Each retraining cycle also introduces functional change, such as new behaviors, decision boundaries, and emergent properties that can affect reliability or safety. Recent industry examples, such as OpenAI’s temporary rollback of GPT-4o and the controversy surrounding behavioral shifts in early GPT-5 models, illustrate how even well-intentioned updates can create new vulnerabilities or regressions. This reality forces defenders to treat security not as a destination, but as a continuous relationship with a learning system that evolves with every iteration.

Borrowing from Saxe’s framework, effective AI defense should integrate four key layers: security-aware models, risk-reduction guardrails, deterministic controls, and continuous detection and response mechanisms. Together, they form a lifecycle approach rather than a perimeter defense.

Defending AI isn’t about eliminating every flaw, just as it isn’t in any other domain of security. The difference is velocity: AI systems change faster than any software we’ve secured before, so our defenses must be equally adaptive. Capable of detecting, containing, and recovering in real time.

Securing AI Requires a Different Mindset

Securing AI requires a different mindset because the systems we’re protecting are not static. They learn, generalize, and evolve. Traditional controls were built for deterministic code; AI introduces nondeterminism, semantic behavior, and a constant feedback loop between data, model, and environment.

At HiddenLayer, we operate on a core belief: you can’t defend what you don’t understand.
AI Security requires context awareness, not just of the model, but of how it interacts with data, users, and adversaries.

A modern AI security posture should reflect those realities. It combines familiar principles with new capabilities designed specifically for the AI lifecycle. HiddenLayer’s approach centers on four foundational pillars:

1. AI Discovery: Identify and inventory every model in use across the organization, whether developed internally or integrated through third-party services. You can’t protect what you don’t know exists.
   
2. AI Supply Chain Security: Protect the data, dependencies, and components that feed model development and deployment, ensuring integrity from training through inference.
   
3. AI Security Testing: Continuously test models through adaptive red teaming and adversarial evaluation, identifying vulnerabilities that arise from learned behavior and model drift.
   
4. AI Runtime Security: Monitor deployed models for signs of compromise, malicious prompting, or manipulation, and detect adversarial patterns in real time.
   

These capabilities build on proven cybersecurity principles, discovery, testing, integrity, and monitoring, but extend them into an environment defined by semantic reasoning and constant change.

This is how AI security must evolve. From protecting code to protecting capability, with defenses designed for systems that think and adapt.

The Path Forward

AI represents both extraordinary innovation and unprecedented risk. Yet too many organizations still attempt to secure it as if it were software with slightly more math.

The truth is sharper.
AI doesn’t break like code, and it won’t be fixed like code.

Securing AI means balancing the proven strengths of traditional controls with the adaptive awareness required for systems that learn.

Traditional cybersecurity built the foundation. Now, AI Security must build what comes next.

Learn More

To stay ahead of the evolving AI threat landscape, explore HiddenLayer’s Innovation Hub, your source for research, frameworks, and practical guidance on securing machine learning systems.

Or connect with our team to see how the HiddenLayer AI Security Platform protects models, data, and infrastructure across the entire AI lifecycle.

As AI systems power critical decisions and customer experiences, the risks they introduce must be addressed. From prompt injection attacks to adversarial manipulation and supply chain threats, AI applications face vulnerabilities that traditional cybersecurity can’t defend against. HiddenLayer was built to solve this problem, and today, we hold one of the world’s strongest intellectual property portfolios in AI security.

A Patent Portfolio Built for the Entire AI Lifecycle

Our innovations protect AI models from development through deployment. With 25 granted patents, 56 pending and planned U.S. applications, and 31 international filings, HiddenLayer has established a global foundation for AI security leadership.

This portfolio is the foundation of our strategic product lines:

• AIDR: Provides runtime protection for generative, predictive, and Agentic applications against privacy leaks, and output manipulation.
  
• Model Scanner: Delivering supply chain security and integrity verification for machine learning models.
  
• Automated Red Teaming: Continuously stress-tests AI systems with techniques that simulate real-world adversarial attacks, uncovering hidden vulnerabilities before attackers can exploit them.

Patented Innovation in Action

Each granted patent reinforces our core capabilities:

• LLM Protection (14 patents): Multi-layered defenses against prompt injection, data leakage, and PII exposure.
  
• Model Integrity (5 patents): Cryptographic provenance tracking and hidden backdoor detection for supply chain safety.
  
• Runtime Monitoring (2 patents): Detecting and disrupting adversarial attacks in real time.
  
• Encryption (4 patents): Advanced ML-driven multi-layer encryption with hidden compartments for maximum data protection.
  

Why It Matters

In AI security, patents are proof of solving problems no one else has. With one of the industry's largest portfolios, HiddenLayer demonstrates technical depth and the foresight to anticipate emerging threats. Our breadth of granted patents signals to customers and partners that they can rely on tested, defensible innovations, not unproven claims.

• Stay compliant with global regulations:
  With patents covering advanced privacy protections and policy-driven PII redaction, organizations can meet strict data protection standards like GDPR, CCPA, and upcoming AI regulatory frameworks. Built-in audit trails and configurable privacy budgets ensure that compliance is a natural part of AI governance, not a roadblock.
  
• Defend against sophisticated AI threats before they cause damage:
  Our patented methods for detecting prompt injections, model inversion, hidden backdoors, and adversarial attacks provide multi-layered defense across the entire AI lifecycle. These capabilities give organizations early warning and automated response mechanisms that neutralize threats before they can be exploited.
  
• Accelerate adoption of AI technologies without compromising security:
  By embedding patented security innovations directly into model deployment and runtime environments, HiddenLayer eliminates trade-offs between innovation and safety. Customers can confidently adopt cutting-edge GenAI, multimodal models, and third-party ML assets knowing that the integrity, confidentiality, and resilience of their AI systems are safeguarded.
  

Together, these protections transform AI from a potential liability into a secure growth driver, enabling enterprises, governments, and innovators to harness the full value of artificial intelligence.

The Future of AI Security

HiddenLayer’s patent portfolio is only possible because of the ingenuity of our research team, the minds who anticipate tomorrow’s threats and design the defenses to stop them. Their work has already produced industry-defining protections, and they continue to push boundaries with innovations in multimodal attack detection, agentic AI security, and automated vulnerability discovery.

By investing in this research talent, HiddenLayer ensures we’re not just keeping pace with AI’s evolution but shaping the future of how it can be deployed safely, responsibly, and at scale.

HiddenLayer — Protecting AI at every layer.

What Is AI Discovery in AI Development?

AI is reshaping how organizations build and deliver software. From customer-facing applications to internal agents that automate workflows, AI is being woven into the code we develop and deploy in the cloud. But as the pace of adoption accelerates, most organizations lack visibility into what exactly is inside the AI systems they are building.

AI discovery in this context means identifying and understanding the models, agents, system prompts, and dependencies that make up your AI applications and agents. It’s about shining a light on the systems you are directly developing.

In short AI discovery ensures you know what you’re putting into production before attackers, auditors, or customers find out the hard way.

Why AI Discovery in Development Matters for Risk and Security

The risk isn’t just that AI is everywhere, it’s that organizations often don’t know what’s included in the systems they’re developing. That lack of visibility creates openings for security failures and compliance gaps.

• Unknown Components = Hidden Risk: Many AI pipelines rely on open-source models, pretrained weights, or third-party data. Without discovery, organizations can’t see whether those components are trustworthy or vulnerable.
  
• Compliance & Governance: Regulations like the EU AI Act and NIST’s AI Risk Management Framework require organizations to understand the models they create. Without discovery, you can’t prove accountability.
  
• Assessment Readiness: Discovery is the prerequisite for evaluation. Once you know what models and agents exist, you can scan them for vulnerabilities or run automated red team exercises to identify weaknesses.
  
• Business Continuity: AI-driven apps often become mission-critical. A compromised model dependency or poisoned dataset can disrupt operations at scale.
  

The bottom line is that discovery is not simply about finding every AI product in your enterprise but about understanding the AI systems you build in development and cloud environments so you can secure them properly.

Best Practices for Effective AI Discovery in Cloud & Development Environments

Organizations leading in AI security treat discovery as a continuous discipline. Done well, discovery doesn’t just tell you what exists but highlights what needs deeper evaluation and protection.

1. Build a Centralized AI Inventory
   
   • Catalog the models, datasets, and pipelines being developed in your cloud and dev environments.
     
   • Capture details like ownership, purpose, and dependencies.
     
   • Visibility here ensures you know which assets must undergo assessment.
     
2. Engage Cross-Functional Stakeholders
   
   • Collaborate with data science, engineering, compliance, and security teams to surface “shadow AI” projects.
     
   • Encourage openness by framing discovery as a means to enable innovation safely, rather than restricting it.
     
3. Automate Where Possible
   
   • Use tooling to scan repositories, pipelines, and environments for models and AI-specific workloads.
     
   • Automated discovery enables automated security assessments to follow.
     
4. Classify Risk and Sensitivity
   
   • Tag assets by criticality, sensitivity, and business impact.
     
   • High-risk assets, such as those tied to customer interactions or financial decision-making, should be prioritized for model scanning and automated red teaming.
     
5. Integrate with Broader Risk Management
   
   • Feed discovery insights into vulnerability management, compliance reporting, and incident response.
     
   • Traditional security tools stop at applications and infrastructure. AI discovery ensures that the AI layer in your development environment is also accounted for and evaluated.
     

The Path Forward: From Discovery to Assessment

AI discovery in development environments is not about finding every AI-enabled tool in your organization, it’s about knowing what’s inside the AI applications and agents you build. Without this visibility, you can’t effectively assess or secure them.

At HiddenLayer, we believe security for AI starts with discovery, but it doesn’t stop there. Once you know what you’ve built, the next step is to assess it with AI-native tools, scanning models for vulnerabilities and red teaming agents to expose weaknesses before adversaries do.

Discovery is the foundation and assessment is the safeguard. Together, they are the path to secure AI.

Executive Summary

AI and ML systems are expanding the software attack surface in new and evolving ways, through model theft, adversarial evasion, prompt injection, data poisoning, and unsafe model artifacts. These risks can’t be fully addressed by traditional application security alone. They require AI-specific defenses integrated directly into the Software Development Lifecycle (SDLC).

This guide shows how to “shift left” on AI security by embedding practices like model discovery, static analysis, provenance checks, policy enforcement, red teaming, and runtime detection throughout the SDLC. We’ll also highlight how HiddenLayer automates these protections from build to production.

Why AI Demands First-Class Security in the SDLC

AI applications don’t just add risk; they fundamentally change where risk lives. Model artifacts (.pt, .onnx, .h5), prompts, training data, and supply chain components aren’t side channels. They are the application.

That means they deserve the same rigorous security as code or infrastructure:

• Model files may contain unsafe deserialization paths or exploitable structures.
  
• Prompts and system policies can be manipulated through injection or jailbreaks, leading to data leakage or unintended behavior.
  
• Data pipelines (including RAG corpora and training sets) can be poisoned or expose sensitive data.
  
• AI supply chain components (frameworks, weights, containers, vector databases) carry traditional software vulnerabilities and configuration drift.
  

By extending familiar SDLC practices with AI-aware controls, teams can secure these components at the source before they become production risks.

Where AI Security Fits in the SDLC

Here’s how AI security maps across each phase of the lifecycle, and how HiddenLayer helps teams automate and enforce these practices.

SDLC PhaseAI-Specific ObjectiveKey ControlsAutomation ExamplePlan & DesignDefine threat models and guardrailsAI threat modeling, provenance checks, policy requirements, AIBOM expectationsDesign-time checklistsDevelop (Build)Expose risks earlyModel discovery, static analysis, prompt scanning, SCA, IaC scanningCI jobs that block high-risk commitsIntegrate & SourceValidate trustworthinessProvenance attestation, license/CVE policy enforcement, MBOM validationCI/CD gates blocking untrusted or unverified artifactsTest & VerifyRed team before go-liveAutomated adversarial testing, prompt injection, privacy evaluationsPre-production test suites with exportable reportsRelease & DeployApply secure defaultsRuntime policies, secrets management, secure configsDeployment runbooks and secure infra templatesOperate & MonitorDetect and respond in real-timeAIDR, telemetry, drift detection, forensicsRuntime blocking and high-fidelity alerting

Planning & Design: Address AI Risk from the Start

Security starts at the whiteboard. Define how models could be attacked, from prompt injection to evasion, and set acceptable risk levels. Establish provenance requirements, licensing checks, and an AI Bill of Materials (AIBOM).

By setting guardrails and test criteria during planning, teams prevent costly rework later. Deliverables at this stage should include threat models, policy-as-code, and pre-deployment test gates.

Develop: Discover and Scan as You Build

Treat AI components as first-class build artifacts, subject to the same scrutiny as application code.

• Discover: model files, datasets, frameworks, prompts, RAG corpora, and container files.
  
• Scan:
  
  • Static model analysis for unsafe serialization, backdoors, or denial-of-service vectors.
    
  • Software Composition Analysis (SCA) for ML library vulnerabilities.
    
  • System prompt evaluations for jailbreaks or leakage.
    
  • Data pipeline checks for PII or poisoning attempts.
    
  • Container/IaC reviews for secrets and misconfigurations.
    

With HiddenLayer, every pull request or CI job is automatically scanned. If a high-risk model or package appears, the pipeline fails before risk reaches production.

Integrate & Source: Vet What You Borrow

Security doesn’t stop at what you build. It extends to what you adopt. Third-party models, libraries, and containers must meet your trust standards.

Evaluate artifacts for vulnerabilities, provenance, licensing, and compliance with defined policy thresholds. 

HiddenLayer integrates AIBOM validation and scan results into CI/CD workflows to block components that don’t meet your trust bar.

Test & Verify: Red Team Before Attackers Do

Before deployment, test models against real-world attacks, such as adversarial evasion, membership inference, privacy attacks, and prompt injection.

HiddenLayer automates these tests and produces exportable reports with pass/fail criteria and remediation guidance, which are ideal for change control or risk assessments.

Release & Deploy: Secure by Default

Security should be built in, not added on. Enforce secure defaults such as:

• Runtime input/output filtering
  
• Secrets management (no hardcoded API keys)
  
• Least-privilege infrastructure
  
• Structured observability with logging and telemetry
  

Runbooks and hardened templates ensure every deployment launches with security already enabled.

Operate & Monitor: Continuous Defense

Post-deployment, AI models remain vulnerable to drift and abuse. Traditional WAFs rarely catch AI-specific threats.

HiddenLayer AIDR enables teams to:

• Monitor AI model I/O in real time
  
• Detect adversarial queries and block malicious patterns
  
• Collect forensic evidence for every incident
  
• Feed insights back into defense tuning
  

This closes the loop, extending DevSecOps into AISecOps.

HiddenLayer Secures AI Using AI

At HiddenLayer, we practice what we preach. Our AIDR platform itself undergoes the same scrutiny we recommend:

• We scan any third-party NLP or classification models (including dynamically loaded transformer models).
  
• Our Python environments are continuously monitored for vulnerabilities, even hidden model artifacts within libraries.
  
• Before deployment, we run automated red teaming on our own detection models.
  
• We use AIDR to monitor AIDR, detect runtime threats against our customers, and harden our platform in response.

Security is something we practice daily.

Conclusion: Make AI Security a Built-In Behavior

Securing AI doesn’t mean reinventing the SDLC. It means enhancing it with AI-specific controls:

• Discover everything—models, data, prompts, dependencies.
  
• Scan early and often, from build to deploy.
  
• Prove trust with provenance checks and policy gates.
  
• Attack yourself first with red teaming.
  
• Watch production closely with forensics and telemetry.
  

HiddenLayer automates each of these steps, helping teams secure AI without slowing it down.

Interested in learning more about how HiddenLayer can help you secure your AI stack? Book a demo with us today. 

AI is powering the next generation of innovation. Whether driving automation, enhancing customer experiences, or enabling real-time decision-making, it has become inseparable from core business operations. However, as the value of AI systems grows, so does the incentive to exploit them.

Our 2025 Threat Report surveyed 250 IT leaders responsible for securing or developing AI initiatives. The findings confirm what many security teams already feel: AI is critical to business success, but defending it remains a work in progress.

In this blog, we highlight the top 5 threat vectors organizations are facing in 2025. These findings are grounded in firsthand insights from the field and represent a clear call to action for organizations aiming to secure their AI assets without slowing innovation.

1. Compromised Models from Public Repositories

The promise of speed and efficiency drives organizations to adopt pre-trained models from platforms like Hugging Face, AWS, and Azure. Adoption is now near-universal, with 97% of respondents reporting using models from public repositories, up 12% from the previous year.

However, this convenience comes at a cost. Only 49% scan these models for safety prior to deployment. Threat actors know this and are embedding malware or injecting malicious logic into these repositories to gain access to production environments.

📊 45% of breaches involved malware introduced through public model repositories, the most common attack type reported.

2. Third-Party GenAI Integrations: Expanding the Attack Surface

The growing reliance on external generative AI tools, from ChatGPT to Microsoft Co-Pilot, has introduced new risks into enterprise environments. These tools often integrate deeply with internal systems and data pipelines, yet few offer transparency into how they process or secure sensitive data.

Unsurprisingly, 88% of IT leaders cited third-party GenAI and agentic AI integrations as a top concern. Combined with the rise of Shadow AI, unapproved tools used outside of IT governance, reported by 72% of respondents, organizations are losing visibility and control over their AI ecosystem.

3. Exploiting AI-Powered Chatbots

As AI chatbots become embedded in both customer-facing and internal workflows, attackers are finding creative ways to manipulate them. Prompt injection, unauthorized data extraction, and behavior manipulation are just the beginning.

In 2024, 33% of reported breaches involved attacks on internal or external chatbots. These systems often lack the observability and resilience of traditional software, leaving security teams without the tooling to detect or respond effectively.

This threat vector is growing fast and is not limited to mature deployments. Even low-code chatbot integrations can introduce meaningful security and compliance risk if left unmonitored.

4. Vulnerabilities in the AI Supply Chain

AI systems are rarely built in isolation. They depend on a complex ecosystem of datasets, labeling tools, APIs, and cloud environments from model training to deployment. Each connection introduces risk.

Third-party service providers were named the second most common source of AI attacks, behind only criminal hacking groups. As attackers look for the weakest entry point, the AI supply chain offers ample opportunity for compromise.

Without clear provenance tracking, version control, and validation of third-party components, organizations may deploy AI assets with unknown origins and risks.

5. Targeted Model Theft and Business Disruption

AI models embody years of training, proprietary data, and strategic differentiation. And threat actors know it.

In 2024, the top five motivations behind AI attacks were:

• Data theft
  
• Financial gain
  
• Business disruption
  
• Model theft
  
• Competitive advantage
  

Whether it’s a competitor looking for insight, a nation-state actor exploiting weaknesses, or a financially motivated group aiming to ransom proprietary models, these attacks are increasing in frequency and sophistication.

📊 51% of reported AI attacks originated in North America, followed closely by Europe (34%) and Asia (32%).

The AI Landscape Is Shifting Fast

The data shows a clear trend: AI breaches are not hypothetical. They’re happening now, and at scale:

• 74% of IT leaders say they definitely experienced an AI breach
  
• 98% believe they’ve likely experienced one
  
• Yet only 32% are using a technology solution to monitor or defend their AI systems
  
• And just 16% have red-teamed their models, manually or otherwise
  

Despite these gaps, there’s good news. 99% of organizations surveyed are prioritizing AI security in 2025, and 95% have increased their AI security budgets.

The Path Forward: Securing AI Without Slowing Innovation

Securing AI systems requires more than repurposing traditional security tools. It demands purpose-built defenses that understand machine learning models' unique behaviors, lifecycles, and attack surfaces.

The most forward-leaning organizations are already taking action:

• Scanning all incoming models before use
• Creating centralized inventories and governance frameworks
• Red teaming models to proactively identify risks
• Collaborating across AI, security, and legal teams
• Deploying continuous monitoring and protection tools for AI assets

At HiddenLayer, we’re helping organizations shift from reactive to proactive AI defense, protecting innovation without slowing it down.

Want the Full Picture?

Download the 2025 Threat Report to access deeper insights, benchmarks, and recommendations from 250+ IT leaders securing AI across industries.