The Incident-Ready AI Product Manager

In an internal document-delivery workflow I helped operate, one malformed document could block a multi-document package. After a hotfix, internal reporting moved from dozens of failures in one day to no reported failures over the next several days. The operating lesson I took from it was that one input should not be allowed to stop unrelated work without an explicit containment decision.

AI products add another layer. Consider a separate tabletop: at 09:04, a synthetic ticket reports that an exported career summary contains another synthetic user's sentence. At 09:09, the team stops exports. At 09:17, it disables enrichment. Nobody knows whether one account or one thousand are affected. This is not a real Bragora event.

The hypothesis is that predefined severity, traceability, containment, user remedy, replay, and rollback can speed defensible decisions and reduce recurrence. A timed tabletop should test it. If a lighter process performs as well, simplify.

AI incidents cross boundaries that product teams usually manage separately

A conventional service incident may be visible as downtime, latency, or errors. An AI product can remain available while returning harmful content, mixing context, taking an unauthorized action, or quietly degrading for a particular group. The interface works. The task outcome does not.

NIST SP 800-61 Rev. 3 integrates incident response with cybersecurity risk management. NIST's Generative AI Profile includes governance, pre-deployment testing, provenance, and incident disclosure, and notes that harm can be acute or aggregate.

A PM cannot define severity only by request failure rate. The team must consider who was affected, which decision or action changed, whether information crossed a boundary, whether the behavior is systematic, and whether users can recover.

Four named severity states make the first decision faster

I would define severity before launch. This proposed matrix is intentionally compact:

Four named severity states

Near missA test, monitor, or employee catches a failure before a user receives it. Preserve the case and add it to evals.
Contained user errorOne or a small known set of users receives an incorrect output. No sensitive data or external action, and correction is reliable.
Material product incidentAffects multiple users, creates an external action, materially changes a user decision, persists across a segment, or cannot yet be bounded.
Critical incidentSensitive data crosses trust boundaries, a safety or legal interest may be affected, an action cannot be reversed, or impact is systemic and expanding.

Product triggers, not legal classifications. Use the highest credible impact while scope is uncertain.

These are product triggers, not legal classifications. Security, privacy, safety, compliance, and legal owners may require different names or escalation rules.

Use the highest credible impact while scope is uncertain. In the tabletop, possible cross-account content triggers the critical state because the population is unknown. The incident commander can reclassify it as evidence narrows scope. One ticket is a detection count, not an incident-size estimate. Organizations with an existing system should map these names to it.

The response packet keeps containment ahead of explanation

The PM helps the incident commander make reversible decisions with incomplete information, not write a perfect root-cause story.

I would keep a response packet ready with seven sections:

The response packet

Declare and assign: incident, severity, commander, technical lead, product lead, communications lead, decision log.
Contain: disable the affected feature, tool, prompt, model route, retrieval source, or export path. Preserve a safe fallback.
Preserve evidence: freeze versions and retain traces, policy decisions, tool calls, and deployment changes under approved procedures.
Bound impact: affected time window, users, tasks, regions, models, prompts, data sources, and external actions.
Remediate users: stop further harm, provide a corrected artifact or reversal path, explain what users should do, assign communications.
Recover deliberately: replay representative cases, validate the fix, confirm monitoring, use a staged restart.
Learn: blameless postmortem, updated evals and controls, actions tracked to completion.

The OWASP GenAI Incident Response Guide was created to help security practitioners respond to incidents involving generative-AI applications. A PM should not replace that security expertise. The PM's contribution is to connect technical containment with user impact, feature state, business priority, support readiness, and the evidence required to resume.

Roll back when the team cannot defend continued exposure

Teams often delay rollback because root cause or scope is uncertain. That uncertainty can be a reason to reduce exposure.

For a suspected material or critical incident, I would ask the incident commander to follow the pre-approved runbook and reduce exposure when a mandatory control fails or any of these conditions holds. Security, privacy, legal, and availability owners should inform the decision:

Cross-user or cross-tenant private data appears in an output, which should trigger an immediate stop for the affected path.
An agent takes an unauthorized external action, which should also trigger an immediate stop for the affected path.
The affected population cannot be bounded with available traces.
A safety, employment, financial, or legal decision may be influenced by an unverified output.
The same failure survives a retry or appears across more than one input slice.
The team cannot reproduce which model, prompt, data, tool, and policy state produced the output.
Monitoring cannot confirm that the proposed fix is working.

In the tabletop, exports stay off while the team reconstructs the path. A text-only fallback can remain if it cannot retrieve private records or export content. Recovery moves through internal replay, synthetic tests, a monitored user cohort, then broader restoration.

Replay turns an incident into a durable product requirement

At 10:12 in the tabletop, the team finds that a cache key omitted the account boundary after a deployment. That detail is synthetic. The more important product question is what evidence would prevent recurrence.

The incident packet should capture the original user task, sanitized inputs, account and permission context, model and prompt versions, retrieval identifiers, tool calls, policy outcomes, generated output, and user-visible effect. Evidence handling should follow approved retention, access, privacy, and legal-hold procedures. The replay harness should reproduce the failure without re-exposing user data.

The fix is incomplete until the team adds a relevant regression or control check, or records why no reliable automated gate is feasible. That may include a cross-account isolation test, a regression input, a monitor for impossible account references, and a rollback check. Assign owners and due dates. Google's SRE guidance on postmortems emphasizes blameless analysis, system improvements, measurable action items, review, and learning rather than blame. For AI products, the postmortem should also state which assumption about the model, data, prompt, tool, or human review proved false.

Use a compact postmortem template:

Summary, severity, owners, and user impact.
Detection, decision, containment, and recovery timeline.
Affected versions, data paths, tools, policies, and user segments.
Trigger, contributing conditions, and safeguards that failed or worked.
User remediation and communications completed or pending.
Regression cases, control changes, action owners, due dates, and evidence required for closure.

Not every bad answer needs an incident channel

The counterargument is operational overload. If every hallucination becomes an incident, the team will stop taking the process seriously.

That is why the severity model distinguishes a known, reversible quality error from a material boundary failure. A wrong synonym caught in preview may be a bug and an eval case. A wrong claim exported to an employer may be a product incident because it leaves the user's control and affects representation. Context changes severity.

This week, run one tabletop with a timer. Begin with the named severity states and a rollback owner. Finish with a record of every missing log, unclear decision, absent support response, and control that could not be verified. Before full restoration, the exercise should produce at least one replayable regression case and a reviewed action list.

An incident-ready PM does not promise that AI will never fail. The promise is that the team can see the failure, limit its reach, help affected users, and make the product harder to break the same way twice.