Guardrails belong in code around the model, not in instructions inside the prompt. A prompt asking the model to behave is a request it can be talked out of. A validator inspecting input and output is a control the model cannot argue with. Build the second, and treat the first as defence in depth.
The common architecture puts the safety rules in the system prompt and hopes. That fails for a structural reason: instructions and user input occupy the same channel, so anything the prompt asserts, the input can attempt to override.
Validation sitting outside the model has no such weakness. It never negotiates.
At a glance
- Input and output guardrails catch different failures and both are needed
- Deterministic checks beat model-based ones wherever a rule can be expressed in code
- Prompt instructions are defence in depth, never the primary control
- Every guardrail needs a defined behaviour on failure, including its own failure
- Fail-open is the default that silently disables safety layers
- Log what was blocked, or you cannot tune the thresholds
What belongs on the input side?
Checks that are cheap and decisive, run before any tokens are spent.
- Length and rate limits. Trivial, and they cap both cost and abuse.
- Topic scoping. A classifier deciding whether the request is in scope for this product at all.
- Injection heuristics. Detection of instruction-like patterns in user-supplied content.
- PII detection. Catching sensitive data before it reaches a provider or a log.
- Structural validation. Confirming supplied fields, formats and identifiers are well formed.
Input guardrails have an economic advantage worth stating plainly: they reject before generation, so a blocked request costs almost nothing while a blocked output has already been paid for.
What belongs on the output side?
Figure 1 — Two layers, two failure classes
Input guardrails
Catch malicious, malformed and out-of-scope requests before generation begins.
Cheap, decisive, and blind to anything the model does afterwards. A perfectly reasonable question can still produce an unacceptable answer.
Pre-generation
Output guardrails
Catch unsupported claims, leaked context, schema violations and off-policy content in what the model produced.
More expensive because generation already happened, and the only layer that sees what the user would actually receive.
Pre-delivery
Neither substitutes for the other. Input checks cannot anticipate what the model will say; output checks cannot prevent the cost of saying it. Systems with only one layer fail in the direction that layer does not watch.
Output checks worth having in nearly every system: schema conformance, grounding verification against retrieved sources, leakage detection for system-prompt or other-tenant content, policy classification, and deterministic validation of any machine-checkable value such as a URL, date or identifier.
Why do deterministic checks outperform clever ones?
Because they cannot be persuaded, they cost nothing, and they never drift.
| Check type | Cost | Can be bypassed by input? | Use for |
|---|---|---|---|
| Regex and format rules | Negligible | No | Identifiers, dates, formats, forbidden strings |
| Schema validation | Negligible | No | Structure and types |
| Database lookup | Low | No | Verifying entities actually exist |
| Classifier model | Moderate | Rarely | Topic scope, policy categories |
| LLM-based judge | High | Sometimes | Nuanced policy, grounding checks |
| Prompt instruction | Per-token | Yes | Defence in depth only |
Read that last row carefully. A prompt instruction is the only control in the table an adversarial input can argue with, and it is the one most systems rely on most heavily.
Watch for this
Decide explicitly what happens when a guardrail itself fails — when the classifier times out or the validation service is unreachable. Systems default to fail-open, meaning an outage silently removes your safety layer while everything appears healthy. For anything consequential, fail closed and return a graceful refusal instead.
How should a guardrail behave when it fires?
Four options, and picking per-guardrail rather than globally is what makes the system usable.
- Block and explain. Return a refusal the user can act on. Correct for policy violations.
- Regenerate once. Suitable for transient structural failures such as a schema violation. Cap the retries.
- Repair. Strip the offending element and return the remainder, where that leaves something coherent.
- Flag and pass. Deliver the output but mark it for review. Right for low-severity signals where blocking would frustrate more than it protects.
Blanket blocking on every signal produces a product people route around. Severity-based routing is what separates a guardrail system that ships from one that gets disabled after two weeks.
Where does the latency go?
Guardrails sit in the request path, so their cost is user-visible in a way model choice often is not.
Run deterministic checks first and in parallel where they are independent. Reserve sequential model-based checks for cases the cheap ones flag as uncertain, rather than running everything on everything.
For output guardrails on streamed responses there is a genuine tension. Streaming means the user sees tokens before validation completes, so either you buffer and lose the streaming benefit, or you stream and accept that a blocked response may be partially visible. Choose deliberately per surface, and buffer wherever the content is consequential.
Where do teams go wrong?
Putting the rules only in the prompt
Already the central point, and worth restating because it remains the most common architecture. Prompt instructions are worth having, but they are the weakest control available and should never be the only one.
Not logging blocks
A guardrail without logging cannot be tuned. You cannot tell whether a threshold is catching real problems or frustrating legitimate users, and the first sign of miscalibration becomes a complaint rather than a metric.
Treating guardrails as launch work
Policies change, attacks evolve, and thresholds drift out of calibration. A guardrail set written once at launch decays exactly like an unmaintained test suite, and with less visible symptoms.
What do experienced teams do differently?
They run new guardrails in shadow mode before enforcing them.
A guardrail deployed in logging-only mode for a week reveals its true false-positive rate against real traffic, which is almost never what the offline evaluation predicted. Enforcing only after that measurement avoids the launch-day incident where a well-intentioned filter blocks a fifth of legitimate requests.
They also keep an adversarial test set in CI: known injection attempts, boundary cases and previously-successful bypasses. It rarely improves, which is the point. It exists to detect the day a change reopens something that was closed.
A short glossary
- Input guardrail
- A check applied to a request before generation, rejecting malicious, malformed or out-of-scope input cheaply.
- Output guardrail
- A check applied to generated content before delivery, catching what the model actually produced.
- Fail-open
- Allowing a request through when a guardrail cannot run, silently removing protection during an outage.
- Shadow mode
- Running a guardrail in logging-only mode against real traffic to measure its false-positive rate before enforcing it.
- Prompt injection
- Supplying input crafted to override the model’s instructions, exploiting the fact that both share one channel.
Key takeaways
- Guardrails belong in code around the model; prompt instructions are defence in depth, never the primary control.
- Input checks are cheap and reject before generation; output checks are the only layer that sees what users receive.
- Deterministic checks cannot be argued with, cost almost nothing, and should be preferred wherever a rule can be coded.
- Define behaviour when a guardrail itself fails, because fail-open silently disables safety during an outage.
- Route by severity rather than blocking on every signal, or the system gets disabled as unusable.
- Streaming and output validation genuinely conflict; buffer wherever the content is consequential.
- Run new guardrails in shadow mode first, since real-traffic false-positive rates rarely match offline estimates.
Frequently asked questions
Can I implement guardrails purely in the system prompt?
Not reliably. Instructions and user input share one channel, so anything the prompt asserts the input can attempt to override. Prompt-level rules are worth including as defence in depth, but the enforceable controls are validators running in code outside the model.
Should guardrails run on input, output, or both?
Both, because they catch different failures. Input checks reject malicious or out-of-scope requests before you pay for generation. Output checks catch unsupported claims, leaked context and policy violations in what the model actually produced, which input checks cannot anticipate.
What should happen when a guardrail service is unavailable?
Decide deliberately rather than inheriting a default. Most systems fail open, meaning an outage removes protection while everything looks healthy. For consequential applications, fail closed and return a graceful refusal, accepting reduced availability in exchange for maintained safety.
Do guardrails add much latency?
Deterministic checks add almost none and can run in parallel. Model-based checks add a full inference call each and should be reserved for cases the cheap checks flag as uncertain. Running every check on every request is the usual cause of unacceptable guardrail latency.
How do guardrails work with streaming responses?
Awkwardly, and the tension is real. Streaming shows tokens before validation completes, so you either buffer and lose the streaming benefit or stream and accept that blocked content may be partially visible. Buffer wherever the content is consequential and stream only where it is not.
How do I stop guardrails blocking legitimate requests?
Run them in shadow mode against real traffic first, logging what they would have blocked without enforcing. Real-world false-positive rates rarely match offline estimates. Then route by severity rather than blocking uniformly, so low-severity signals flag for review instead of refusing.
Is a classifier better than a regex for guardrails?
Only where the rule cannot be expressed deterministically. Regex and schema checks are free, instant and impossible to talk around. Classifiers earn their cost on genuinely fuzzy judgments such as topic scope and policy categories, where no deterministic rule exists.
How often should guardrails be reviewed?
Continuously, in the same way as a test suite. Policies change, attack patterns evolve and thresholds drift. Keep an adversarial set of known injection attempts and previous bypasses in CI so a change that reopens a closed hole fails the build rather than reaching production.
References
- OWASP. OWASP Top 10 for Large Language Model Applications. Covers prompt injection and insecure output handling.
- NIST. AI Risk Management Framework.
- Willard, B. T., and Louf, R. Efficient Guided Generation for Large Language Models. arXiv:2307.09702. Relevant to enforcing structural constraints at decode time.
