What are guardrails?
The checks around an AI model that stop bad requests going in and bad answers coming out.
A model will try to answer almost anything. It can be wrong, it can be tricked, and it can say things a company cannot stand behind. Guardrails are the checks placed around it.
Checks that run before a message reaches the model and before its reply reaches you. If a check fails, the message is blocked, changed or handed to a person.
A new support agent has their replies reviewed before they go out, and some requests, like another customer’s address, are refused however they are worded. Guardrails are that review, done by software on every message.
Where they sit
- 01Your message
- 02Input check
- 03Model
- 04Output check
- 05You
- Off-topic requestsA bank bot asked to write a poem
- Prompt injectionText that tries to override the instructions
- Personal dataCard numbers, passwords
- Harmful requestsThings the product should never help with
- Made-up factsClaims that are not in the source documents
- Leaked dataAnother person's details, internal notes
- Promises outside the policyRefunds or discounts that do not exist
- Broken formatA reply the app cannot read
How they are built
- 1Rules — fixed patterns such as the shape of a card number or a list of blocked words. Fast and cheap, and easy to get around
- 2Classifiers — a small model trained to spot one thing, such as abusive language
- 3A second LLM call — another model is asked whether the reply follows the policy. Flexible, but slower and it costs more
- 4A person — for actions that matter, such as payments or deletions, a human approves
An example
Their limits
- They lower the risk, they don’t remove it. Unusual wording can still get past a check.
- They block good messages too. A strict check will sometimes refuse a fair question.
- They add time. Every check runs before you see the reply, and some of them cost money.
A common misunderstanding
“The system prompt is the guardrail.” Instructions in the prompt are requests to the model, and a model can be talked out of them. A guardrail in the stricter sense is a separate check that runs outside the model, so it still works when the model has been fooled. Some people use the word for both, and careful products use both.
In January 2024 the parcel company DPD switched off part of its customer service chatbot after a customer got it to swear and write a poem criticising the company. An output check for tone and topic is the kind of guardrail that catches that.
Related buzzwords
Prompt · Hallucination · AI agent · Harness · Prompt injection · Evals