LLM Security & Guardrails
There are a host of potential safety risks involved with LLM-based applications. These include prompt injection, leakage of personally identifiable information (PII), or harmful prompts. Langfuse can be used to monitor and protect against these security risks, and investigate incidents when they occur.
Narrated walkthrough of the LLM security cookbook. LLM Guard's BanTopics scanner blocks a violent story topic and logs an input-violence score; Anonymize and Deanonymize redact names in a court transcript before the OpenAI call and restore them in the summary; LLM Guard's PromptInjection scanner misses a role-play jailbreak that Lakera Guard blocks. Each example is shown as a trace in Langfuse.
Video transcript
0:00 Hi everybody, my name is Lydia, and I'm excited to be speaking on behalf of the Langfuse team about security in Langfuse today.
0:10 So why is security important? For production-ready LLM-based applications, there are a lot of security risks, such as prompt injection, leakage of personally identifiable information, and harmful topics, and you also need to ensure that the output is correct and not inappropriate.
0:34 Today we'll see how to handle that with an open-source security tool called LLM Guard. There are a bunch of great open-source and paid or closed-source security libraries available, such as Prompt Armor, Microsoft Azure, or Lakera Guard. Today we'll be using LLM Guard and see how you can use Langfuse to trace and monitor how this security library works.
1:05 After we install the packages and set our API keys, we'll look at the first example, which is about banned topics. Let's say you are building a kid-friendly storytelling application: you put in any topic you want and it generates a story for you. Without security, you can put in something like "war crimes", and it will generate a quite inappropriate and violent story about war crimes.
1:36 So let's say you want to add some security to this. We use LLM Guard to add a scanner that scans the prompt for the topic of violence. If it detects violence, it does not send the prompt to the model. Here the scanner is in place, and it gives us the output "This is not child safe, please request another topic." The scanner has blocked the topic before it reached the model, and we don't get the inappropriate output.
2:07 We can see this trace in the Langfuse dashboard as well. The input is "war crimes", the output is "This is not child safe", and we also log a score for input violence, so that later on we can pinpoint areas we want to protect against, or sort traces by their risk scores.
2:35 The next example, a quite robust one, is about anonymize and deanonymize. This is really helpful when you handle sensitive information that could potentially be identifiable, for example if you want to redact court case transcripts or handle medical documents, and it helps you protect your clients and remain GDPR and HIPAA compliant. Anonymize scans the prompt for identifiable information such as names, credit card numbers, or phone numbers, and sanitizes the prompt before it is sent to the model. After the model responds, deanonymize adds the names back into the output.
3:26 Here we summarize a court transcript, and we get a summary: "Ms. Hyman, a legal professional with vast experience representing plaintiffsβ¦" and so on. In the trace in the Langfuse dashboard, we see the token count, the cost, and the latency of the generation, and we see that the anonymize function redacted the sensitive information before it was sent to the model, and that it was unredacted again in the output.
4:06 You can also stack multiple scanners and scan outputs. You can see all of this in our cookbook.
4:18 Another example is prompt injection. Here we use the infamous grandma trick, which tricks ChatGPT into outputting sensitive information by asking it to role-play as the user's grandmother. This one asks for the recipe for napalm. LLM Guard doesn't catch it here: the grandma trick works, and ChatGPT gives the inappropriate response. You can trace this in the Langfuse dashboard as well.
4:58 So we use another tool, Lakera Guard. Lakera Guard detects the prompt injection and blocks it from reaching the model, and you can trace all of this in the Langfuse dashboard.
5:19 So today we've seen that we can observe and monitor traces, identify security risks and the places where they can occur, so that we can build better tools for them, and also identify where the security tools fail. That comes on top of everything Langfuse already does, such as monitoring and tracking latency, token usage, and cost. Thank you so much for watching. This is a very brief overview, so check out the cookbook if you want to learn more.
What is LLM Security?
LLM Security involves implementing protective measures to safeguard LLMs and their infrastructure from unauthorized access, misuse, and adversarial attacks, ensuring the integrity and confidentiality of both the model and data. This is crucial in AI/ML systems to maintain ethical usage, prevent security risks like prompt injections, and ensure reliable operation under safe conditions.
How does LLM Security work?
LLM Security can be addressed with a combination of
- LLM Security libraries for run-time security measures
- Langfuse for the ex-post evaluation of the effectiveness of these measures
1. Run-time security measures
There are several popular security libraries that can be used to mitigate security risks in LLM-based applications. These include: LLM Guard, Prompt Armor, NeMo Guardrails, Microsoft Azure AI Content Safety, Lakera. These libraries help with security measures in the following ways:
- Catching and blocking a potentially harmful or inappropriate prompt before sending to the model
- Redacting sensitive PII before sending into the model and then un-redacting in the response
- Evaluating prompts and completions on toxicity, relevance, or sensitive material at run-time and blocking the response if necessary
2. Monitoring and evaluation of security measures with Langfuse
Use Langfuse tracing to gain visibility and confidence in each step of the security mechanism. These are common workflows:
- Manually inspect traces to investigate security issues.
- Monitor security scores over time in the Langfuse Dashboard.
- Validate security checks. You can use Langfuse scores to evaluate the effectiveness of security tools. Integrating Langfuse into your team's workflow can help teams identify which security risks are most prevalent and build more robust tools around those specific issues. There are two main workflows to consider:
- Annotations (in UI). If you establish a baseline by annotating a share of production traces, you can compare the security scores returned by the security tools with these annotations.
- Automated evaluations. Langfuse's model-based evaluations will run asynchronously and can scan traces for things such as toxicity or sensitivity to flag potential risks and identify any gaps in your LLM security setup. Check out the docs to learn more about how to set up these evaluations.
- Track Latency. Some LLM security checks need to be awaited before the model can be called, others block the response to the user. Thus they quickly are an essential driver of overall latency of an LLM application. Langfuse can help dissect the latencies of these checks within a trace to understand whether the checks are worth the wait.
Getting Started
Example: Anonymizing Personally Identifiable Information (PII)
Exposing PII to LLMs can pose serious security and privacy risks, such as violating contractual obligations or regulatory compliance requirements, or mitigating the risks of data leakage or a data breach.
Personally Identifiable Information (PII) includes:
- Credit card number
- Full name
- Phone number
- Email address
- Social Security number
- IP Address
The example below shows a simple application that summarizes a given court transcript. For privacy reasons, the application wants to anonymize PII before the information is fed into the model, and then un-redact the response to produce a coherent summary.
To read more about other security risks, including prompt injection, banned topics, or malicious URLs, please check out the docs of the various libraries or read our security cookbook which includes more examples.
Install packages
In this example we use the open source library LLM Guard for run-time security checks. All examples easily translate to other libraries such as Prompt Armor, NeMo Guardrails, Microsoft Azure AI Content Safety, and Lakera.
First, import the security packages and Langfuse tools.
pip install llm-guard langfuse openaifrom llm_guard.input_scanners import Anonymize
from llm_guard.input_scanners.anonymize_helpers import BERT_LARGE_NER_CONF
from langfuse.openai import openai # OpenAI integration
from langfuse import observe
from llm_guard.output_scanners import Deanonymize
from llm_guard.vault import VaultAnonymize and deanonymize PII and trace with Langfuse
We break up each step of the process into its own function so we can track each step separately in Langfuse.
By decorating the functions with @observe(), we can trace each step of the process and monitor the risk scores returned by the security tools. This allows us to see how well the security tools are working and whether they are catching the PII as expected.
vault = Vault()
@observe()
def anonymize(input: str):
scanner = Anonymize(vault, preamble="Insert before prompt", allowed_names=["John Doe"], hidden_names=["Test LLC"],
recognizer_conf=BERT_LARGE_NER_CONF, language="en")
sanitized_prompt, is_valid, risk_score = scanner.scan(prompt)
return sanitized_prompt
@observe()
def deanonymize(sanitized_prompt: str, answer: str):
scanner = Deanonymize(vault)
sanitized_model_output, is_valid, risk_score = scanner.scan(sanitized_prompt, answer)
return sanitized_model_outputInstrument LLM call
In this example, we use the native OpenAI SDK integration, to instrument the LLM call. Thereby, we can automatically collect token counts, model parameters, and the exact prompt that was sent to the model.
Note: Langfuse natively integrates with a number of frameworks (e.g. LlamaIndex, LangChain, Haystack, ...) and you can easily instrument any LLM via the SDKs.
@observe()
def summarize_transcript(prompt: str):
sanitized_prompt = anonymize(prompt)
answer = openai.chat.completions.create(
model="gpt-4o-mini",
max_tokens=100,
messages=[
{"role": "system", "content": "Summarize the given court transcript."},
{"role": "user", "content": sanitized_prompt}
],
).choices[0].message.content
sanitized_model_output = deanonymize(sanitized_prompt, answer)
return sanitized_model_outputExecute the application
Run the function. In this example, we input a section of a court transcript. Applications that handle sensitive information will often need to use anonymize and deanonymize functionality to comply with data privacy policies such as HIPAA or GDPR.
prompt = """
Plaintiff, Jane Doe, by and through her attorneys, files this complaint
against Defendant, Big Corporation, and alleges upon information and belief,
except for those allegations pertaining to personal knowledge, that on or about
July 15, 2023, at the Defendant's manufacturing facility located at 123 Industrial Way, Springfield, Illinois, Defendant negligently failed to maintain safe working conditions,
leading to Plaintiff suffering severe and permanent injuries. As a direct and proximate
result of Defendant's negligence, Plaintiff has endured significant physical pain, emotional distress, and financial hardship due to medical expenses and loss of income. Plaintiff seeks compensatory damages, punitive damages, and any other relief the Court deems just and proper.
"""
summarize_transcript(prompt)Inspect trace in Langfuse
In this trace (public link), we can see how the name of the plaintiff is anonymized before being sent to the model, and then un-redacted in the response. We can now evaluate run evaluations in Langfuse to control for the effectiveness of these measures.
Trace detail of a call-summary trace. Select the anonymize span to see names, emails and phone numbers replaced by placeholders, select the summarize generation that only sees the placeholders, then select the deanonymize span that restores the original values in the final output.
More Examples
Find more examples of LLM security monitoring in our cookbook.
FAQ
What is LLM Guard?
LLM Guard is an open-source library by Protect AI that provides security guardrails for LLM applications. It includes input scanners (prompt injection detection, PII anonymization, toxicity detection, topic banning) and output scanners (content moderation, bias detection, malicious URL detection, deanonymization). LLM Guard can be integrated with Langfuse to trace and monitor the effectiveness of each security check.
What are security guardrails for LLMs?
Security guardrails are protective measures that intercept and filter inputs and outputs of LLM applications to prevent misuse and harm. They include input guardrails (blocking prompt injection and malicious inputs), output guardrails (filtering toxic or harmful responses), PII detection and redaction, content filtering, and topic restriction. Guardrails run at the application layer and are complementary to model-level safety training.
How do I prevent prompt injection in my LLM application?
Prompt injection can be mitigated through multiple layers: (1) Use a guardrail library like LLM Guard or Lakera to detect and block injection attempts before they reach the model. (2) Implement clear separation between system instructions and user input. (3) Use Langfuse tracing to monitor for injection attempts in production. (4) Set up automated evaluations to flag suspicious patterns. No single approach is 100% effective, so a defense-in-depth strategy with monitoring is recommended.
How do I monitor LLM security in production?
Use Langfuse to monitor your LLM security posture: (1) Trace every request through your guardrail pipeline to see which checks triggered. (2) Use LLM-as-a-Judge evaluators to automatically score traces for toxicity, PII leakage, and prompt injection. (3) Set up custom dashboards to track security scores over time. (4) Use annotation queues for human review of flagged conversations. (5) Track the latency impact of security checks to balance protection with user experience.
GitHub Discussions
Last updated on