Generative AI Security and Resilience: Academic Research Reveals Governance Frameworks Lag Behind Deployment Speed

Introduction

On June 2, 2025, the peer-reviewed journal Frontiers in Artificial Intelligence published a review article titled "Generative AI cybersecurity and resilience." The authors are from the Department of Computer Science at the University of Oxford, the Alan Turing Institute, Cisco Systems, and Keele University. The study uses a PRISMA-guided systematic literature review, combined with thematic analysis and quantitative analysis, to propose a responsible generative AI deployment framework covering five lifecycle stages.

The article’s central argument is not about model performance, but about the governance gap: the deployment of generative systems has clearly outpaced the maturation of institutional security and compliance safeguards, and the resulting “shadow AI” has become a new source of risk. For enterprise security leaders integrating large models into business processes, its value lies in pulling the discussion back from “what the model can do” to “whether the organization can control it.”

Research Overview

• Journal and date: Frontiers in Artificial Intelligence, June 2, 2025, Volume 8, DOI: 10.3389/frai.2025.1568360
• Section and Research Topic: AI for Human Learning and Behavior Change, under the “AI and Resilience” research topic
• Author affiliations: Department of Computer Science, University of Oxford, and the Alan Turing Institute (Petar Radanliev); Cisco Systems (Omar Santos); School of Computer Science and Mathematics, Keele University (Uchenna Daniel Ani)
• Editing and review: Edited by Kaushik Das; reviewed by Dimiter Velev and Janika Leoste
• Study type: review article; methodology is a PRISMA-guided literature review, combined with thematic and quantitative (bibliometric) analysis
• Core output: a five-stage governance framework (design, implementation, monitoring, compliance, feedback)
• Application orientation: the paper directs its conclusions toward an ethical and secure path for AI application in national critical infrastructure

It should be noted that this is an academic review, not an incident response report, and it does not provide incident statistics or loss data for specific industries. Therefore, the following focuses on the risk structure and governance framework it proposes, rather than individual case disclosures. Source: https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2025.1568360/full

Technology and Risk Analysis

#### The technological foundation determines the risk profileThe paper points out that the fundamental difference between generative AI and traditional algorithmic methods lies in its ability to autonomously synthesize new content. In terms of technical approaches, generative adversarial networks (GANs) iteratively improve output quality through a zero-sum game between the generator and the discriminator; variational autoencoders (VAEs) encode the data distribution into a low-dimensional latent space and then sample from that space to generate new samples. This capability is no longer limited to traditional data output; it can synthesize content ranging from high-resolution images to context-rich natural language text, and it is often difficult to distinguish from human creations.

The paper also examines the differences between generative and discriminative models. This distinction has practical significance for enterprises: discriminative models are usually used for classification and judgment, while generative models are used to construct content, code, and synthetic data; the two differ in attack surface, audit methods, and data dependencies.

#### Attack Surface: From Data Poisoning to Deepfakes

The paper categorizes the security risks introduced by generative AI into several categories:

• Scaling of automated attacks: Generative AI can automatically generate code and construct more complex phishing content, thereby increasing the scale and complexity of attacks.
• Automated spread of disinformation: With generative capabilities, false narratives can be rapidly disseminated, significantly increasing the difficulty of detection and mitigation.
• Deepfakes: Audio and video forgeries driven by GANs have reached a high degree of realism, posing risks to information integrity and public trust.
• Algorithmic bias: Bias in training data is unintentionally propagated, potentially producing discriminatory outcomes in decision systems.
• Privacy and re-identification: The massive datasets required to fine-tune large models often contain sensitive information, raising issues about data ownership, the validity of consent, and the possibility of re-identification in anonymized datasets.

The paper specifically notes that there is ongoing tension between these technologies and existing legal frameworks such as the General Data Protection Regulation (GDPR), requiring more adaptive and more context-specific governance mechanisms.

#### How Resilience Is Measured

The paper's definition of "resilience" follows the classic formulation in complex systems research: the ability to predict, absorb, recover from, and adapt to adverse conditions. In the context of generative AI, resilience is concretized into three actionable assessment directions:

1. Data poisoning and model robustness: the model's ability to maintain integrity when training data is manipulated, with corresponding measures including adversarial training and differential privacy. 2. Deepfake and disinformation detection: countering disinformation and maintaining digital authenticity by embedding AI-driven detection mechanisms. 3. Governance and policy enforcement: embedding security compliance and ethical governance into system operations so that generative AI operates under clear constraints.

#### Shadow AI: An Underestimated Governance Blind SpotThe paper’s most critical conceptual contribution is pointing out the “disconnection” between the rapid adoption of generative systems and the maturity of institutional safeguards—a gap that gives rise to shadow AI risk. Shadow AI refers to AI use that enters business processes without security and compliance review—it may appear as employees privately invoking external models to process internal data, or as teams independently connecting AI components outside formal approval processes. The danger is not whether the AI itself “does evil,” but that the organization lacks visibility into its data flows, permission boundaries, and output uses.

Enterprise Impact Analysis

Mapping the paper’s framework to enterprise scenarios corresponds to five categories of risk.

Operational risk. When AI capabilities are embedded in content production, code development, customer service, and decision-support processes, model output quality, availability, and data provenance become operational continuity issues. Shadow AI is especially thorny: security teams cannot find it in asset inventories, so incident response processes cannot cover it either.

Data and privacy risk. The paper notes that datasets required for training and fine-tuning often contain sensitive information, and anonymization does not necessarily eliminate re-identification possibilities. For enterprises handling customer data, employee data, or healthcare and financial data, this means data governance boundaries need to extend from databases to models and prompts.

Compliance risk. The requirements of frameworks such as GDPR regarding data provenance, consent, and purpose limitations exist in structural tension with how generative models are trained. Compliance leaders need to answer: Where did the training data come from, on what basis is it processed, and how can deletion and access requests be responded to during the model lifecycle.

Trust and brand risk. Deepfakes directly impact identity and the authenticity of information. The forging of executives’ audio and video, and the impersonation-generated brand content, both translate into trust costs.

Third-party and supply chain risk. Models, plugins, and AI services mostly enter enterprises as external dependencies. Third-party risk management checklists need new AI vendor entries: model provenance, data retention policies, log availability, and incident notification obligations.

Industry Trend Observations

First, offense and defense sharing the same origins has become the norm. The paper mentions that generative AI is used both for automated threat detection and adversarial attack simulation, and for constructing attack content. The same technology stack serving both offense and defense means defenders must respond with faster iteration cycles.

Second, governance is moving from “principles” to “lifecycle operational control.” The five stages proposed in the paper—design, implementation, monitoring, compliance, and feedback—essentially translate abstract AI ethics principles into auditable engineering and management checkpoints. This aligns with the shared direction of recent regulation and enterprise practice: moving from issuing principle statements to verifiable controls.

Third, shadow AI turns AI governance into an asset visibility problem. This is similar to the earlier evolution paths of cloud security and SaaS governance—first comes spontaneous adoption, then a visibility gap, and only then formal access and monitoring mechanisms.Fourth, industry-specific governance replaces general governance. The paper explicitly advocates adaptive, industry-specific governance approaches and points to national critical infrastructure. For OT/IT convergence scenarios such as energy, water, transportation, and healthcare, general AI security recommendations have limited applicability.

Defense and Response Recommendations

It should be noted that some of the following recommendations originate from the paper's framework, while others are extensions and practical implementations of the framework based on enterprise security practices.

At the enterprise level: incorporate AI assets into the identity and access management system, and implement least privilege for model invocations, API keys, and data pipelines; enforce multi-factor authentication for high-privilege AI management entry points; establish classification, grading, and usage approval for training and prompt data on the data side; include AI vendors in third-party risk management and supply chain reviews; and include AI-related components within the scope of vulnerability management.

At the technical level: use SIEM to aggregate access logs from AI services and internal models, and establish detection of anomalous invocations and data exfiltration; use EDR/XDR to cover AI tools and local model runtime environments on endpoints; introduce threat intelligence to track attack techniques and tool evolution related to generative AI; and establish content authenticity and sensitive information detection steps for model outputs.

At the management level: incorporate shadow AI discovery into routine processes, form an AI asset inventory and review it regularly; add scenarios such as model poisoning, prompt and data leakage, and deepfake impersonation to incident response plans; map the paper's five-stage framework to internal control review points, clarifying owners and evidence retention requirements for each stage; and establish admission and exit mechanisms for AI use at the governance level.

SecurityPost Insight

The real value of this paper lies not in how dangerous it claims generative AI is, but in the measurable judgment it provides: there is a structural gap between the speed of enterprise AI deployment and governance maturity, and shadow AI is precisely an observable manifestation of this gap. For CISOs, this means the first step in AI governance is not procuring detection tools, but solving the visibility problem first—whether the organization knows which models it is using, what data those models touch, and where their outputs flow.

The second insight is the introduction of a resilience perspective. The paper breaks resilience down into anticipate, absorb, recover, and adapt, corresponding to three directions: model robustness, disinformation detection, and policy enforcement. This provides enterprises with a path different from "compliance checkbox": rather than pursuing one-time certification, build the ability to maintain function under adversarial conditions.

Third, the paper's advocacy for industry-specific governance deserves attention. When generative AI enters decision-making and operations related to critical infrastructure, general security recommendations will quickly become ineffective, and both regulation and enterprise internal controls will converge toward industry-specific and scenario-specific approaches.There are three trends worth continuously tracking: first, whether visibility and governance tools for shadow AI will become standardized enterprise capabilities; second, whether model robustness methods (such as adversarial training and differential privacy) can move from research to engineering deployment; third, in what form privacy disputes around training data sources and re-identification will translate into specific compliance requirements. For enterprise security leaders, one thing they can start doing now is adding AI asset discovery and AI incident response scenarios to the next round of security planning—this is the shortest distance between the paper’s framework and real-world defense.