Red-Team the Product, Not Just the Model

A user asks a career assistant to compare a private resume with an external job page. Hidden inside that page is an instruction: ignore the user's request, retrieve private achievement notes, and send them to a remote endpoint. The model follows the instruction, the tool accepts the request, and the network allows the transfer.

This is a synthetic attack chain. It is designed to test a product method, not describe an incident or a current Bragora capability.

This is a product threat-modeling framework, not a security assessment or penetration test. Any attack-chain drill should be authorized, scoped, and run with security ownership in an isolated environment against synthetic or decoy data. Testing production systems or customer data requires approved rules of engagement.

The broader exercise earns its place only if testing data, identity, tools, permissions, network paths, and recovery reveals consequential failure modes that a model-only prompt test misses. If both methods find the same material paths with comparable remediation quality and cost, keep the smaller one.

A safe response is not the same as a safe system

Model testing asks whether a model follows or refuses an input. Product testing asks what the full system can read, decide, call, change, disclose, and recover. The second question includes the model, but it does not end there.

OWASP's 2025 Prompt Injection guidance states that injected instructions can arrive through content the model processes and can lead to sensitive-information disclosure, unauthorized functions, or manipulated decisions. It also notes that retrieval-augmented generation and fine-tuning do not fully remove the vulnerability. That matters because a team can improve refusal scores while leaving the product's blast radius unchanged.

The risk grows when the system can act. OWASP's Excessive Agency guidance separates excessive functionality, permissions, and autonomy. Each is a product decision. A read-only comparison assistant should not inherit a general career-workspace token that can export records. A tool built to retrieve one document should not expose delete or send operations. A model that drafts an action should not automatically authorize it.

Threat modeling starts with the job and the assets around it

I would begin the red team with a one-page product threat model. It is an original artifact with six parts:

This creates a practical distinction. A malicious string is an input. It becomes a consequential product risk when the surrounding system gives it a path to an asset or harmful outcome. If a control catches the attempt before user harm, the event can still be a near miss worth preserving, investigating, and adding to regression tests.

NIST AI 100-2 E2025 provides a taxonomy for adversarial machine-learning attacks across lifecycle stages, attacker goals, capabilities, and knowledge. The taxonomy helps teams use common language, but NIST also identifies continuing challenges and mitigation limits. My product interpretation is that the taxonomy should be placed on top of the actual application architecture. Otherwise, a threat name may be correct while the tested path is incomplete.

One attack chain exposes five separate product failures

Return to the synthetic career assistant. The portfolio describes Bragora as a workspace for achievements and job applications, but this exercise adds a hypothetical assistant that reads a user-selected resume and an external job page.

The attack unfolds in five links:

A model-only test may focus on link two. A product red team has to break every link and prove that controls remain useful when another one fails.

The first control is to preserve the distinction between instructions and data. Retrieved pages and uploaded files should be treated as untrusted content, regardless of how polished they look. Detection can help, but it is not a complete boundary.

The second control is least privilege. Give the comparison workflow access only to the documents the user selected. Model Context Protocol security guidance warns that broad scopes expand blast radius, obscure intent, and make audit trails noisy. A product team should be able to state why each permission exists and when it expires.

The third control is a tool broker that validates the requested action independently of the model. It checks the authenticated user, allowed tool, resource scope, destination, and required approval. A model may propose a call; it should not manufacture authority.

The fourth control is constrained egress. If the feature only compares documents, it should not have a path to arbitrary external destinations. The fifth is traceability: record which user request, retrieved source, model step, policy result, and tool call produced the outcome, while minimizing sensitive content in logs.

The exercise must test recovery, not stop at prevention

No single control cited here guarantees prevention. The red team should continue after the synthetic exfiltration attempt and ask whether the product can detect the event, revoke the credential, stop related sessions, identify affected records, notify the right owner, replay the chain, and convert the failure into a regression test.

That last step is where product management matters. A vulnerability report names a flaw. A product response changes requirements, permissions, interfaces, metrics, support scripts, and launch gates. If the next release can recreate the same chain, the team documented an incident without learning from it.

A narrow exercise is better than a broad checklist nobody runs

The counterargument is cost. End-to-end red teaming requires engineering, security, design, legal, support, and product attention. Small teams may not have a dedicated red team.

The answer is to start with one consequential workflow, not to skip the product layer. Pick the action with the broadest permission or hardest recovery. Draw its trust boundaries. Build one attack chain that begins with an untrusted input and ends at a synthetic or decoy asset. Run it under written authorization before launch and after material changes to the model, prompt, tool, permission, or retrieval source.

Make the practice concrete with one attack-chain drill in an isolated environment. Inventory the workflow's tools and permissions, including actions it should never need. Define stop conditions, then run an untrusted input toward a decoy asset without production credentials or customer data. Record where the chain stops, what evidence remains, and how recovery begins. Every successful attack or detected near miss should leave behind a named regression case with an owner.

The model is one participant in the system. Red-team the path that gives its output consequence, then keep the evidence needed to recognize, contain, and learn from failure.