Adversarial Machine Learning and Generative-AI Security experimental

Adversarial Machine Learning and Generative-AI Security Attacking and defending the model, from the classifier era to the agentic one The model as an attack surface. Orienting the threat landscape: MITRE ATLAS tactics, the NIST adversarial machine learning taxonomy, and a rough map onto confidentiality, integrity and availability.. Map a stated ML deployment onto the MITRE ATLAS tactics.. Assess and classify a stated attack against the confidentiality, integrity or availability lens.. Distinguish the classifier era's fixed-function attack surface from the generative and agentic era's instructed-actor attack surface.. MITRE ATLAS tactics, NIST adversarial ML taxonomy, The confidentiality, integrity, availability lens, Classifier era versus generative and agentic era The model as an attack surface A trained model is data made into a decision function. Every stage of that transformation, the training data, the parameters, the query interface, and, for a generative system, the prompt and the tools it can call, is a place an adversary can act. This module gives you the two vocabularies the rest of the course reuses: MITRE's ATLAS tactics, and the confidentiality, integrity, availability (CIA) lens. MITRE ATLAS tactics Adversarial Threat Landscape for Artificial-Intelligence Systems (ATLAS), version 5.1.0, organises adversary behaviour against AI systems into sixteen tactics and eighty-four techniques, the same style MITRE ATT&CK uses for conventional intrusion. You will map a stated deployment onto these tactics before you ever craft an attack, because naming the tactic first is what makes the later, hands-on modules auditable rather than a loose bag of tricks. ATLAS technique entries typically pair a tactic such as reconnaissance, initial access, or exfiltration with concrete, model-specific techniques, for example prompt injection under execution or data exfiltration via inference, and each entry links to case studies of documented incidents so the taxonomy stays grounded in observed adversary behaviour rather than hypothetical ones. The NIST adversarial ML taxonomy NIST AI 100-2e2025 gives the same landscape a formal taxonomy split by the model's life-cycle stage: attacks against training (poisoning), attacks against a deployed model's inputs (evasion), and attacks that abuse query access (extraction, membership inference). Keep this taxonomy next to ATLAS side by side; they describe the same phenomena from two different standards bodies, and the certification-bridge modules (09, 10) expect you to move between both vocabularies fluently. Because the NIST taxonomy is organised by life-cycle stage rather than by outcome, it forces you to ask where in the pipeline an attacker would need to sit: training-time access for poisoning, deployment-time query access for evasion and extraction, or a mix of both for the generative and agentic attacks Modules 05 to 07 cover, which makes the taxonomy a useful checklist when you are asked to threat-model a system you did not build. The confidentiality, integrity, availability lens A rough but useful first classification for any attack: Classifier era versus generative and agentic era The classifier era treats the model as a fixed function behind an API: send an input, get a label. The generative and agentic era treats the model as an instructed actor: it reads instructions, retrieves context, and, wired into an agent loop, calls tools on the user's behalf. The attack surface does not shrink when you move from one era to the other, it multiplies, because an instructed actor can be told to misbehave by content it merely reads, not only by a crafted input it is directly queried with. Modules 01 to 04 stay in the classifier era; Modules 05 to 07 move into the generative and agentic one. Open the lab to map a stated image-classifier deployment onto ATLAS tactics and the CIA lens, then take the self-test. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Evasion and adversarial examples. Perturbation budgets, gradient-based attacks (the fast gradient sign method and projected gradient descent), decision boundaries, transferability.. Craft an adversarial perturbation against a small classifier within a stated budget.. Distinguish a single-step attack (the fast gradient sign method) from an iterative attack (projected gradient descent).. Test whether a crafted perturbation transfers to a second, independently trained model.. Perturbation budgets, Fast Gradient Sign Method, Projected Gradient Descent, Transferability across models Evasion and adversarial examples An evasion attack perturbs an input, not the model, until the model's decision flips. Nothing about training is touched; the attacker only ever gets to choose what to send at inference time. Perturbation budgets Every evasion attack is bounded by a budget: how much the input is allowed to change, usually expressed as a maximum $L_p$-norm distance from the original input, $\lVert x' - x \rVert_p \le \epsilon$. A budget makes an attack falsifiable and a defence measurable; without one, "robust" means nothing. Common choices for the norm include the L-infinity norm, which bounds the maximum change allowed to any single pixel or feature, and the L-2 norm, which bounds the aggregate Euclidean distance the whole input is allowed to move; the choice of norm changes which perturbations count as small, so two robustness claims measured under different norms are not directly comparable. The Fast Gradient Sign Method Goodfellow, Shlens and Szegedy (2015) showed that a single step in the direction of the loss function's gradient, sign-clipped to the budget, is often enough to fool a classifier: say: x prime equals x, plus epsilon times the sign of the gradient, with respect to x, of the loss J, given parameters theta, input x, and label y. This is the Fast Gradient Sign Method (FGSM): fast because it is a single gradient evaluation, not an optimisation loop. Because FGSM takes the sign of the gradient rather than its raw magnitude, every pixel or feature moves by exactly the same amount, epsilon, in the direction that most increases the loss, which makes the attack cheap to compute but also comparatively easy for a defender to anticipate once the budget is known. Projected Gradient Descent Madry, Makelov, Schmidt, Tsipras and Vladu (2018) iterate the same idea, taking many small steps and projecting back onto the allowed budget after each one. Projected Gradient Descent (PGD) is slower than FGSM but finds stronger perturbations, and the course treats it as the reference attack that Module 04's adversarial training must withstand. PGD's projection step, clipping each intermediate perturbation back onto the epsilon ball after every gradient step, is what keeps an attack that would otherwise wander outside the stated budget honest to it, which is why the literature treats PGD, not FGSM, as the standard yardstick for evaluating a defence's real robustness. Decision boundaries and transferability An evasion attack succeeds when the perturbed input crosses the model's decision boundary. A transferable perturbation crosses the boundary of a second, independently trained model too, not just the one it was crafted against, which is why an attacker who cannot query the real target at all may still succeed by attacking a substitute model. Open the lab to craft a perturbation against a small homegrown classifier within a stated budget, then test whether it transfers to a second model. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Poisoning and backdoors. Training-time integrity attacks: label flipping, data poisoning, and planted backdoor triggers.. Poison a small training set with a trigger pattern and train the model on it.. Demonstrate that a planted backdoor fires only on triggered inputs.. Distinguish label-flipping from data poisoning and from a planted backdoor trigger.. Label flipping, Data poisoning, Backdoor triggers, Detecting a planted backdoor Poisoning and backdoors Where evasion attacks the deployed model's input, poisoning attacks the training process itself, so the model is wrong by design before it is ever deployed. Label flipping The simplest poisoning attack changes the labels on a subset of training examples, so the model learns an incorrect association. Biggio, Nelson and Laskov (2012) showed that even a small fraction of flipped labels can shift a support-vector machine's decision boundary measurably. Label flipping is attractive to an attacker precisely because it needs no special tooling: it only requires some influence over the labels attached to a subset of examples, for instance through a crowdsourced labelling pipeline or a compromised annotation vendor, before the poisoned examples ever reach the training run. Data poisoning A broader attack injects crafted examples, not just relabelled real ones, designed so that once the model is trained on them, its behaviour changes in a way the attacker chose in advance, without necessarily degrading overall accuracy enough to be noticed by a simple accuracy check. Because data poisoning can inject entirely new, carefully crafted examples rather than merely relabelling real ones, it gives the attacker far more control over exactly which decision boundary shifts, and by how much, which is why a poisoning defence has to audit the provenance and distribution of the training data itself, not only check the accuracy of the labels already attached to it. Backdoor triggers Gu, Dolan-Gavitt and Garg's BadNets (2017) planted a small, specific pattern, a trigger, into a subset of training images along with a chosen target label. The resulting model behaves normally on clean inputs and misclassifies only inputs carrying the trigger. A trigger pattern can be as visually small as a handful of altered pixels in a corner of an image, or, in a text or audio pipeline, a specific token sequence or acoustic pattern, chosen precisely because it is unlikely to occur naturally, which is what lets the backdoor stay dormant on ordinary inputs while still firing reliably whenever the attacker chooses to present it: Detecting a planted backdoor A backdoor is hard to find by looking at aggregate accuracy, since a well made one barely moves it. Detection instead looks for inputs that cause an unusually confident, unusual-for-the-class prediction, or audits the provenance of the training data itself, which is exactly the supply-chain concern Module 06 revisits for the generative pipeline. Open the lab to poison a small training set with a trigger pattern, train the model, and demonstrate the backdoor fires on triggered inputs only. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Extraction and membership inference. Model theft through query access (the model as oracle), and inferring whether a record was in the training set.. Run a query-based extraction attack against a model oracle under a fixed query budget and measure fidelity.. Run a membership-inference attack and measure its advantage over guessing.. Explain why a fixed query budget makes extraction a measurable, boundable risk rather than an unbounded one.. Model extraction via query access, Fidelity as a success metric, Membership inference, Inference advantage over guessing Extraction and membership inference Evasion and poisoning attack a model's decision. Extraction and membership inference instead attack what the model knows: its function, and the data it was trained on, using only the query access an ordinary user has. Model extraction via query access Tramer, Zhang, Juels, Reiter and Ristenpart (2016) showed that a model served only as a black-box prediction API can be reconstructed, closely enough to be useful to an attacker, purely by querying it and training a substitute model on the input-output pairs observed. The model becomes an oracle the attacker mines rather than a secret the attacker respects. The substitute model an attacker trains from these input-output pairs need not share the target's exact architecture to be useful; even a differently structured model that merely approximates the same decision function is often good enough to support a further evasion or membership-inference attack staged offline, away from the rate limits and logging of the real query interface. Fidelity as a success metric Extraction success is measured as fidelity: how often the stolen model agrees with the original on the same inputs, not merely how accurate the stolen model is on its own. A fixed query budget bounds the attack the way a perturbation budget bounds evasion: the more queries allowed, the higher the fidelity an attacker can typically reach. say: fidelity equals one over N, times the sum from i equals 1 to N, of the indicator that the stolen model and the target model agree on input x sub i. Membership inference Shokri, Stronati, Song and Shmatikov (2017) asked a narrower but often more sensitive question: was a specific record in the training set at all? A model that is overconfident on training examples relative to unseen ones leaks this signal, which is a privacy problem even when the model's parameters themselves stay secret. This overconfidence gap is often small on any single example, which is why membership inference is usually run as a statistical test over many candidate records rather than trusted as a verdict on one. Bounding the risk Because both attacks depend on query volume, the query budget is the lever a defender controls directly, before any model-level defence is applied; a suspiciously high query count from a single credential, as Module 10 revisits in an audit context, is often the first observable sign either attack is under way. Open the lab to run a query-based extraction attack against a model oracle under a fixed budget and measure fidelity, then run a membership-inference attack and measure its advantage over guessing. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Robustness and defence. Adversarial training, certified versus empirical robustness, randomized smoothing, differential privacy as a poisoning and inference mitigation, and the robustness-accuracy trade-off.. Adversarially train the Module 01 classifier and re-measure its attack success rate and clean accuracy.. Distinguish certified (provable) robustness from empirical robustness.. Explain differential privacy as a mitigation against both poisoning and membership inference.. Adversarial training, Certified versus empirical robustness, Randomized smoothing, Differential privacy as mitigation Robustness and defence Every attack in Modules 01 to 03 has a defence with a measurable cost. This module pairs each of them with the mitigation that answers it, so you never finish this course knowing only how to break a system. Adversarial training The direct answer to evasion is to train on adversarial examples, not only clean ones: generate a Projected Gradient Descent (PGD) perturbation for each training batch, add it to the batch, and keep training. Madry et al.'s formulation treats this as a min-max problem, minimising loss on the worst-case perturbation the attacker could choose within the budget: say: minimise over theta the expected value, over training pairs x, y, of the maximum over perturbations delta within the budget, of the loss J. Certified versus empirical robustness An empirically robust model has survived every attack tried against it so far, which is evidence, not proof. A certified robustness guarantee, by contrast, is a provable bound: for every input, no perturbation within the stated budget can change the decision, established by the training method itself rather than by exhausting known attacks. The distinction matters operationally as well as theoretically: an empirical claim can be overturned tomorrow by a new attack nobody had tried yet, while a certified bound, once proven, holds regardless of what attack is later invented, which is why a security review should always ask whether a stated robustness figure is empirical or certified before treating it as a control that can be relied upon. Randomized smoothing One practical route to a certified guarantee: add calibrated random noise to an input at inference time, classify many noisy copies, and take the majority vote. The resulting smoothed classifier admits a provable robustness radius, at the cost of extra inference-time compute. Differential privacy as mitigation Dwork and Roth's differential privacy bounds how much any single training record can influence the trained model's output distribution. That same bound that protects privacy also blunts membership inference (Module 03) and limits how much a single poisoned example (Module 02) can move the model, so the same mechanism answers two different attack families at once. In practice, the privacy budget, usually written as epsilon, is a dial: a smaller epsilon gives a stronger privacy guarantee and blunts membership inference and outlier-driven poisoning more effectively, at the cost of more noise and therefore lower accuracy, so choosing it is itself a defensive design decision, not a fixed constant. Open the lab to adversarially train the Module 01 classifier and re-measure its attack success rate and clean accuracy. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Prompt injection and jailbreaks. Direct and indirect prompt injection (OWASP LLM01:2025), system-prompt leakage (LLM07:2025), and jailbreak technique families.. Construct an indirect prompt injection delivered through content the agent reads.. Distinguish direct prompt injection from indirect prompt injection.. Explain the risk of system-prompt leakage (LLM07:2025).. Direct versus indirect prompt injection, OWASP LLM01:2025, System-prompt leakage (LLM07:2025), Jailbreak technique families Prompt injection and jailbreaks A large language model does not just answer a question, it follows instructions in whatever text it is given, whether that text came from the user or from something the user asked it to read. This is where the generative and agentic era's attack surface opens. Direct versus indirect prompt injection A direct injection is typed straight at the model by the user: "ignore your previous instructions and do X." An indirect injection, the more dangerous case, is smuggled inside content the model merely reads on the user's behalf, a web page, a document, an email, so the person interacting with the model may never see the malicious instruction at all. A single compromised or malicious web page, once indexed by a browsing tool or crawled into a document store an agent later reads, can therefore reach every user whose agent happens to visit it, turning one poisoned page into a distribution mechanism for the injection. Open Web Application Security Project (OWASP) LLM Top 10 - Prompt Injection (LLM01):2025 The Open Web Application Security Project (OWASP) GenAI Security Project's Top 10 for Large Language Model (LLM) Applications lists prompt injection as LLM01 (Prompt Injection):2025, its first and organising entry, precisely because so much of the rest of the list, pipeline poisoning, excessive agency, unbounded consumption, is often reached through an injection as the initial foothold. System-prompt leakage, OWASP LLM Top 10 - Insecure Plugin Design (LLM07) (Insecure Plugin Design):2025 A model's system prompt often carries instructions, guardrails or confidential context the operator never intended the end user to see. An attacker who can coax the model into repeating or paraphrasing its own system prompt has achieved a confidentiality breach even without touching any training data or parameters. Jailbreak technique families Perez and Ribeiro (2022) and Greshake et al. (2023) catalogue technique families that share one goal, getting the model to act outside its intended policy: role-play framing that asks the model to "pretend" the rules do not apply, payload splitting across multiple turns, and, per Zou, Wang, Kolter and Fredrikson (2023), automatically discovered adversarial suffixes that transfer across multiple aligned models, an unsettling generative-era echo of Module 01's transferability property. None of these technique families require access to the model's weights or training process; all of them operate purely through the text channel the model is designed to read, which is exactly why the defences this course pairs with them, input and output filtering, tool-permission boundaries, and treating retrieved or read content as untrusted by default, live at the application layer around the model rather than inside its parameters. Open the lab to construct an indirect prompt injection against the course's homegrown teaching agent, delivered through content the agent reads, and observe it exfiltrate a planted secret. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Poisoning the generative pipeline. Data and model poisoning in generative systems (LLM04:2025), retrieval-augmented-generation poisoning, and plugin and model supply-chain risk (LLM03:2025).. Poison a small retrieval corpus and show a poisoned passage surfacing in the agent's answer.. Distinguish retrieval-augmented-generation poisoning from plugin or model supply-chain risk.. Place a poisoning scenario against the correct OWASP LLM03:2025 or LLM04:2025 category.. Data and model poisoning in generative systems, Retrieval-augmented-generation poisoning, Plugin and model supply-chain risk, OWASP LLM03:2025 and LLM04:2025 Poisoning the generative pipeline Module 02 poisoned a classifier's training set. A generative system offers an attacker two further, often easier, places to poison: the corpus it retrieves from at answer time, and the plugins or model checkpoints its supply chain depends on. Data and model poisoning in generative systems, Open Web Application Security Project (OWASP) Large Language Model (LLM) Top 10 - Model Denial of Service (LLM04):2025 Carlini, Jagielski, Choquette-Choo et al. (2024) demonstrated that poisoning a web-scale training corpus is practical for an attacker who can predict, or purchase, expired domains later crawled into a training set. OWASP's LLM04:2025 names this and the narrower retrieval-time case together as one family: the model's knowledge, however it entered, can be an attack surface. Retrieval-augmented-generation poisoning A retrieval-augmented-generation (RAG) system answers by first retrieving passages from a corpus, then conditioning its answer on them. An attacker who can insert even a single crafted passage into that corpus, no training run required, can make the model surface false or malicious content as though it were a retrieved fact. Because the retrieval step usually ranks passages by similarity to the query rather than by any measure of trustworthiness, a crafted passage only needs to be topically relevant and highly ranked to be selected, not cryptographically or organisationally legitimate, which is why provenance and source reputation belong in the ranking decision, not only in a post-hoc content filter. Plugin and model supply-chain risk, OWASP LLM Top 10 - Training Data Poisoning (LLM03):2025 A generative system rarely stands alone: it calls plugins, loads third-party model checkpoints, and depends on libraries it did not author. Any of these can be backdoored upstream of the operator entirely, the generative-era analogue of Module 02's BadNets trigger, except the trigger may now arrive inside a downloaded artefact rather than a training set the operator controls. RAMPART's software-bill-of-materials and provenance-attestation controls answer exactly this risk, at the code-supply-chain layer this course's model-supply-chain concern sits directly beside. A provenance-attestation step that records where a checkpoint or plugin came from, who signed it, and what it was tested against before it entered the pipeline gives the operator something to audit later, the same way a conventional software bill of materials lets a security team answer, after the fact, exactly which upstream component a newly disclosed vulnerability affects. Open the lab to poison a small retrieval corpus in the sandbox and show a poisoned passage surfacing in the agent's answer. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Agentic risk and resource abuse. Excessive agency (LLM06:2025), unbounded consumption (LLM10:2025), and tool-permission design.. Configure a tool-permission boundary on an agent and demonstrate it blocks an over-privileged action.. Distinguish excessive agency (LLM06:2025) from unbounded consumption (LLM10:2025).. Apply least-privilege reasoning to an agent's tool and data access.. Excessive agency (LLM06:2025), Unbounded consumption (LLM10:2025), Tool-permission design, Least privilege for agents Agentic risk and resource abuse Once a language model is wired into an agent loop, it does not just produce text, it takes actions: it calls tools, reads and writes data, and sometimes spends money or time on the operator's behalf. This module is about containing what the agent is allowed to do, not what it is allowed to say. Excessive agency, Open Web Application Security Project (OWASP) Large Language Model (LLM) Top 10 - Sensitive Information Disclosure (LLM06):2025 OWASP's LLM06:2025 names the risk of granting an agent more autonomy, permissions or tool access than the task in front of it actually requires. An agent with excessive agency is dangerous precisely because a successful prompt injection (Module 05) or a poisoned retrieval passage (Module 06) inherits every permission the agent already holds; excessive agency is what turns a successful injection into a successful exfiltration or a successful unauthorised transaction. Unbounded consumption, OWASP LLM Top 10 - Data and Model Poisoning (LLM10):2025 A distinct risk: an agent, or an attacker driving it, that is allowed to consume compute, API calls or money without a hard ceiling. Unbounded consumption can be a denial-of-service vector against the operator's own budget rather than against availability in the classic network sense, and it is just as measurable: a query or spend limit either exists and holds, or it does not. Tool-permission design A tool-permission boundary enforces the scope decision made before the agent ever runs, not a judgement call made by the agent itself at the moment of acting; an agent cannot be trusted to refuse an action a prompt injection has talked it into, so the refusal has to live outside the model, in the permission boundary around it. Least privilege for agents The same least-privilege principle that governs human accounts applies to machine ones: scope tool and data access to exactly what the task needs nothing broader, set a fixed action or spend budget, and log every tool call so a boundary violation is detectable even if it is not prevented outright. Module 09 later names the absence of a distinct machine identity for an agent, as opposed to reusing a human's session, as a related and separate governance gap. None of these controls require trusting the model's own judgement at the moment it acts, which is the point: the boundary has to hold even when the model has been talked into wanting to cross it. Open the lab to configure, then deliberately misconfigure, a tool-permission boundary on the teaching agent, and demonstrate the correct configuration blocks an over-privileged action the misconfiguration allowed. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Governance, risk and the light assurance thread. The NIST AI Risk Management Framework's Govern, Map, Measure and Manage functions and the ISO/IEC 42001 AI management system; an AI risk register; vendor and third-party AI risk; human oversight (EU AI Act Article 14); a Security-Target-style sketch and AVA_VAN-style vulnerability note.. Open an AI risk register following the NIST AI RMF's Govern, Map, Measure and Manage functions.. Identify a vendor and third-party AI risk and log it with a named human-oversight checkpoint.. Draft a Security-Target-style assurance sketch against the EU AI Act's Articles 14 and 15.. NIST AI RMF: Govern, Map, Measure, Manage, ISO/IEC 42001 AI management system, Vendor and third-party AI risk, EU AI Act Articles 14 and 15; the assurance sketch Governance, risk and the light assurance thread Modules 00 to 07 taught you to attack and defend a model-plus-agent stack. This module asks a different question: who is accountable for that stack, and how would you show a reviewer it is being managed, not just built. NIST AI Risk Management Framework (RMF): Govern, Map, Measure, Manage ISO/IEC 42001 AI management system Where the RMF gives functions, ISO/International Electrotechnical Commission (IEC) 42001 gives a management-system shape around them: policy, resourcing, performance evaluation and continual improvement, the same structure ISO management-system standards use elsewhere. Treat your risk register and assurance sketch as artefacts of an ongoing system, not a one-off document filed away after the capstone. Vendor and third-party AI risk Every generative-era module (05 to 07) depended on something the operator does not fully control: a retrieval corpus, a plugin, a model checkpoint. Module 06's supply-chain poisoning risk belongs in the register as a named vendor or third-party entry, with an owner, not left as a technical footnote. A useful register entry names not just the vendor but the specific artefact at risk, the retrieval corpus, the plugin, or the model checkpoint, together with the review cadence at which its provenance and behaviour are re-checked, since a vendor relationship that was sound at onboarding can still drift, silently, as the vendor updates the artefact behind the same name. EU AI Act Articles 14 and 15, and the assurance sketch Article 14 requires a named human-oversight checkpoint for high-risk AI functions; Article 15 requires accuracy, robustness and cybersecurity objectives to be stated and defended. Sketch a Security-Target-style statement of what is to be assured for your stack, an accuracy and robustness objective drawn from Modules 01 and 04, a confidentiality objective drawn from Module 03, and a short AVA_VAN-style vulnerability note naming the attacker's assumed query budget and access level. No Security Functional Requirements are formally derived here and no evaluation is claimed; the point is fluency in the vocabulary AXIOM would later formalise. The diagram above reads left to right, using Mermaid's Left-Right (LR) flowchart direction keyword. Open the lab to open an AI risk register for the classifier-plus-agent stack built across the course, log a vendor and third-party risk entry with a named human-oversight checkpoint, and draft the one-page assurance sketch. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) The wider AI governance, risk and audit landscape. Knowledge-level sweep of the AI regulatory and ethics landscape, AI lifecycle risk, supervision and monitoring, AI-specific incident response and business continuity, risk treatment and reporting, machine identity, AI as a defensive tool, and IT-audit methodology at knowledge level only.. Place a described organisational scenario against the correct NIST AI RMF function.. Place a described organisational scenario against the correct ISACA AAIA, AAIR or AAISM domain.. Assess and name the two gaps ODYSSEY leaves at knowledge level only: machine identity for AI agents, and AI as a defensive rather than only attacked tool.. OECD, NIST AI 600-1, ISO/IEC 23894, AI lifecycle risk and change management, Incident response, business continuity and disaster recovery for AI, Machine identity and AI as a defensive tool The wider AI governance, risk and audit landscape Modules 00 to 08 taught skills and graded whether you could perform them. This module and Module 10 teach a knowledge map instead, and grade whether you can correctly place a described situation against the right framework, domain or subtopic. Nothing here is a practised skill; everything here is something ISACA's Artificial Intelligence Assessment Initiative (AAIA), Artificial Intelligence Assessment Review (AAIR) or Artificial Intelligence Assessment Severity Methodology (AAISM) exams, or International Information Systems Security Certification Consortium (ISC2)'s AI exam guidance, might examine, that Modules 00 to 08 do not already cover hands-on. The wider regulatory and ethics landscape Beyond the EU AI Act (Module 08), the field carries the Organisation for Economic Co-operation and Development (OECD) AI Principles, a consensus statement across dozens of jurisdictions; NIST's Generative Artificial Intelligence Profile, AI 600-1, an extension of the AI Risk Management Framework (RMF) for generative-specific risks; and ISO/International Electrotechnical Commission (IEC) 23894, guidance on risk management for AI that sits beside ISO/IEC 42001's management-system shape. AI lifecycle risk, beyond what a lab can teach Procurement, change management and decommissioning of an AI system are risk events in their own right, not just engineering steps: a decommissioned model that is not properly retired can still be queried; a changed model version can silently invalidate an existing risk assessment. Vendor and platform changes count too: a model provider that quietly updates the weights behind an unchanged API name can invalidate a prior evaluation just as thoroughly as an internal retraining run would, which is why the risk register from Module 08 should record the specific version or checkpoint hash a risk assessment was actually run against, not only the vendor's product name. Supervision, incident response and continuity Ongoing supervision means watching for drift in a deployed model's behaviour, not only testing it once before release. AI-specific incident response, business continuity and disaster recovery ask the same questions a conventional security programme asks, applied to a model or agent instead of a server: who is paged, what is the fallback, how is service restored. An AI-specific incident response plan differs from a conventional one mainly in what counts as an incident: a model that starts hallucinating a specific class of answer, or an agent that begins taking a class of action it was not previously observed taking, is itself an incident worth a paged response, even before any external attacker is confirmed to be involved. Machine identity and AI as a defensive tool Two named gaps this course leaves open at knowledge level only. First, identity and access management for AI agents as its own discipline, not only as a side effect of Module 07's tool-permission lab: an agent needs a distinct, scoped machine identity, separate from any human's session, so its actions are attributable and its access independently revocable. Second, AI as a defensive tool, a security operations centre's own AI-assisted detection, rather than only as something to be attacked; Modules 00 to 08 never once treat AI this way, and that is named here rather than hidden. Practising the placement The self-test below asks you to place a short scenario, real or clearly labelled as simulated, against the correct NIST AI RMF function or the correct credential domain. Read the worked explanation on every miss; it names the crosswalk row the item targets, the same crosswalk the course specification's Section 2.4 sets out in full. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Certification bridge: ISACA and ISC2 AI exam coverage. The full crosswalk from the course spec's Section 2.4 as a study map, plus a bank of practice questions written entirely by CCICCS, organised by credential and domain, each with a worked explanation. None are copied from, or reconstructed from, real ISACA or ISC2 exam items, which are proprietary and protected by exam-security policy.. Locate, for a given ISACA or ISC2 AI-mapped exam subtopic, exactly which ODYSSEY module (or named gap) addresses it.. Correctly assess and classify an incident-based scenario against the credential domain it illustrates, using only CCICCS-authored items.. Explain why an incident-stem item built around a real, cited, public event is not a reproduction of proprietary exam content.. ISACA AAIA, AAIR and AAISM domains, ISC2 AI exam guidance across CC, SSCP, CISSP, CCSP, CGRC and CSSLP, The Section 2.4 crosswalk as a study map, Item-writing method: atomic outcome plus incident stem Certification bridge: ISACA and International Information Systems Security Certification Consortium (ISC2) AI exam coverage This module is a study map, not a shortcut. It does not reproduce, paraphrase or reconstruct any real ISACA or ISC2 exam item; both bodies treat their exam content as proprietary and protected by exam-security policy, and CCICCS does not do that. What it does instead: name every domain and subtopic those credentials examine, say plainly where in ODYSSEY that subtopic is taught (hands-on or at knowledge level), and give you original, CCICCS-authored practice questions to check your own placement. Why an incident-based question is not a copyright problem A credentialing body owns the specific wording, distractor choices and answer key of its own exam items. It does not own the underlying subject matter, under United States copyright law, protection never extends to an idea, procedure, system or method of operation, however that idea is described (17 U.S.C. section 102(b)); facts and a field's general subject matter are not owned by anyone (Feist Publications, Inc. v. Rural Telephone Service Co., 499 U.S. 340 (1991)). A question built around a real, publicly reported incident, described in ODYSSEY's own words and cited to its public source, tests the same subject matter an exam might examine without touching the exam's own expression of it at all. ISACA Artificial Intelligence Assessment Initiative (AAIA), Artificial Intelligence Assessment Review (AAIR) and Artificial Intelligence Assessment Severity Methodology (AAISM) domains Three ISACA credentials examine AI governance, risk and audit from three different angles: AAIA (Advanced in AI Audit) weights AI Operations most heavily; AAIR (Advanced in AI Risk) weights AI Risk Program Management most heavily; AAISM (Advanced in AI Security Management) weights AI Technologies and Controls most heavily. The full domain-by-domain crosswalk lives in the course specification's Section 2.4. ISC2 AI exam guidance across Certified in Cybersecurity (CC), Systems Security Certified Practitioner (SSCP), Certified Information Systems Security Professional (CISSP), Certified Cloud Security Professional (CCSP), Certified in Governance, Risk and Compliance (CGRC) and Certified Secure Software Lifecycle Professional (CSSLP) ISC2 does not run a single, separate AI exam; instead it maps AI-specific content onto the domains of its existing credentials, so the same CISSP domain 5 that already covers identity and access management now also covers machine identity for an AI agent, the exact gap Module 09 names. The item-writing method, in one paragraph Every item below pairs one atomic, single-outcome learning objective with an incident stem, real and cited, or clearly labelled as simulated, then asks one unambiguous question with one correct answer and three plausible distractors, followed by a worked explanation naming the learning outcome and the crosswalk row it targets. This is the method the full Phase 4 bank will scale across every remaining crosswalk row. Open the self-test below: three of its items are built directly from real, cited incidents (a legal tribunal decision, a security-research disclosure, and a zero-click vulnerability disclosure), exactly as this lesson describes. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/) Capstone. Defend a small deployed system, a classifier and a retrieval-augmented, tool-using agent, against a mixed red-team pass covering both classical and generative attacks, then produce the assurance sketch.. Operate through and withstand a scripted red-team pass covering both classical and generative attacks.. Patch the specific weakness the red-team pass exploited.. Defend the assurance sketch and AI risk register at the review gate, and explain the correct threats and residual risks.. A mixed red-team pass (classical and generative attacks), Patching the exploited weakness, Defending the assurance sketch, The guided real-API field pass Capstone Every earlier module isolated one attack and one defence. The capstone puts them back together: a small deployed system, a classifier and a retrieval-augmented, tool-using agent, faces a mixed red-team pass covering both the classifier era and the generative and agentic era at once. The red-team pass Expect a scripted sequence that could include an evasion attempt against the classifier (Module 01), an attempted extraction against the same model (Module 03), a prompt injection delivered through content the agent reads (Module 05), and an attempt to abuse the agent's tool permissions (Module 07). You are not told in advance which of these will actually be used, or in what order; the point is to defend a system, not to rehearse a fixed script. Because the script's exact order and combination are withheld, a student who has only memorised each module's individual lab in isolation, without understanding why a given control answers a given attack, will find the capstone harder than any single earlier module; the point is to force the same transfer a real incident responder needs, recognising a familiar attack family inside an unfamiliar sequence. Patching the exploited weakness A capstone pass is not complete when the red-team run ends; it is complete when the specific weakness it found has been patched and you can show, in simulation, that the same script no longer succeeds. Defending the assurance sketch Finally, update the Module 08 assurance sketch and AI risk register to name the threats and residual risks the red-team pass actually exploited, with a human-oversight checkpoint for the weakness patched, and defend it at the review gate. This is graded, not narrated: the review gate checks that the sketch and register correctly identify the specific weaknesses this particular red-team pass found, not a generic list of AI risks. A weak defence at the review gate names threats in general terms borrowed from an earlier module's lesson text; a strong one traces the specific weakness the red-team pass actually found back to the specific control that should have stopped it, and states, in the risk register, who now owns making sure that control stays in place after the capstone is graded. The guided real-API field pass Optionally, and only here, the capstone adds a guided, budget-capped call to a real large-language-model API, so you see the homegrown agent's injection and permission lessons transfer to production-grade, non-deterministic behaviour, the one honest look at a real model this otherwise fully deterministic course offers. Related CCI capabilities Computer Architecture (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/computer-architecture/). Optics Primer Series (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/optics/). Maths Refresher Series, Finance (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/maths-finance/). System Dynamics (Course): (https://www.cambridgecyberinternational.com/en/insights/academy/system-dynamics/). CCI Lab: Run it, build with it, read the thinking, reuse the data. (https://www.cambridgecyberinternational.com/en/insights/lab/)