Guardrails for Generative AI: Keeping Output Safe and On-Policy

The model is not the control point

A generative model is a probabilistic text engine. Prompt it well and it usually behaves; prompt it in an unusual way, feed it hostile input, or simply roll the dice enough times, and it will eventually produce something off-policy — a leaked record, an unsafe instruction, a confidently wrong claim, or an answer to a question it should have refused. You cannot make this impossible by asking the model nicely in a system prompt. Instructions inside the prompt are suggestions the model mostly follows, not controls it must obey.

Guardrails are the controls that live around the model, enforced in code, so that safety does not depend on the model's cooperation. This is a defense-in-depth problem: no single filter is sufficient, and the reliable systems layer several cheap, independent checks on both sides of the model. This article walks the layers.

Guard the input before the model sees it

The first line of defense inspects what goes in. Several checks matter:

• Topic and scope enforcement. Decide what the system is allowed to discuss and reject the rest. A support assistant should not be answering questions about unrelated topics, and a classifier on the input can turn those away before they cost a token.
• PII and secret redaction on egress to the model. Sensitive data — customer PII, PHI, credentials, source code — should be detected and redacted or tokenized before the prompt leaves your boundary for a third-party model. This is exactly the job of a privacy layer such as the intSignal AI Gateway, which redacts sensitive fields and enforces policy before a prompt reaches the model.
• Prompt-injection screening. When input includes untrusted content, screen it for override attempts. This is imperfect — injection has no clean fix — so it is one layer among several, not the whole defense.
• Abuse and rate controls. Basic input-length and rate limits blunt both cost attacks and attempts to brute-force a jailbreak through sheer volume.

Input guards are cheap relative to a model call, so run them first — they save money and stop many problems before generation even begins.

Constrain what the model can produce

Shaping the output space is often more effective than filtering it afterward.

• Structured output. When the result feeds another system, demand JSON against a schema and validate it. A response that must conform to a schema has far less room to go off the rails than free-form prose, and invalid output is caught deterministically.
• Constrained choices. For classification or routing, restrict the model to a fixed set of allowed values and reject anything outside it. Do not let the model invent a category.
• Grounding requirements. For factual tasks, require the answer to be drawn from retrieved source passages and, where possible, to cite them. An answer that cannot point to a source is a candidate for rejection, not display.

Constraining the output does double duty: it improves reliability and it shrinks the attack surface at the same time.

Check the output before it reaches the user

The second line of defense inspects what comes out, before anyone sees or acts on it.

• Content safety classification. Screen generated text for the categories your policy prohibits — harmful instructions, harassment, disallowed advice — and block or route to a human before display.
• PII leakage checks. Verify the output is not returning sensitive data the requester is not entitled to, especially in retrieval systems where a permissions gap can surface someone else's records.
• Faithfulness / grounding verification. For grounded answers, check that the response is actually supported by the retrieved context and is not a fluent fabrication. A second model pass or a rules check can catch the worst hallucinations.
• Format and policy validation. Confirm the output meets structural and business rules — no broken JSON, no forbidden claims, no missing disclaimer — and fail closed if it does not.

The design principle is fail closed: when a guard is uncertain or a check fails, the safe default is to withhold, refuse, or escalate to a human — never to pass the questionable output straight through.

Prompt injection and the untrusted-content problem

The hardest guardrail problem deserves its own treatment. Prompt injection is when untrusted content the model reads — a web page, an email, a PDF, a support ticket — carries hidden instructions that hijack the model's behavior. It sits at the top of the OWASP Top 10 for LLM Applications, and it has no complete fix. It is mitigated, not solved. What actually reduces the risk:

• Separate instructions from data. Keep the untrusted content clearly delimited and never let it silently become part of the system's instructions.
• Least privilege on tools and data. An injected instruction can only do damage through the permissions the model holds. An agent that cannot delete data cannot be tricked into deleting data. Scope every credential to the task.
• Human confirmation on consequential actions. Sending, paying, deleting, and publishing stay behind explicit approval, so a hijacked prompt cannot act unilaterally.
• Output filtering as backstop. Even a compromised generation is caught if it tries to exfiltrate data past your output PII and content checks.

The lesson is architectural: assume the model can be manipulated, and design so that manipulation cannot reach anything valuable.

Layer the controls into a policy you can enforce

Individually these checks are simple. Their strength comes from being layered and centralized rather than scattered across every application. Routing all AI traffic through a common policy layer — an AI gateway — lets you enforce input redaction, output filtering, logging, and access rules consistently, and change policy in one place instead of re-implementing it in every app. Pair that enforcement point with your existing data loss prevention classification so AI destinations are governed by the same rules as email and file transfer, and consult the gateway documentation for the policy model.

The result is a system whose safety does not rest on any single model behaving — it rests on independent controls that hold even when the model does not.

Where to start

Begin with the two highest-value guards: redact sensitive data on the way in, and validate structure and content on the way out. Add topic scoping and grounding checks next, and reserve the heaviest work — injection defense and least-privilege tool design — for systems that read untrusted content or take real-world actions. Log everything so you can see what your guards catch and tune from evidence.

Guardrails are not a feature you bolt on before launch; they are the architecture that makes generative AI safe to deploy at all. If you want help designing that architecture and enforcing it consistently across your AI estate, talk to our team.