Skip to content

About

A layered approach to building safe, resilient LLM pipelines

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

LLM Guardrails

Building Safe, Reliable, and Resilient LLM Pipelines


📖 Overview

This project demonstrates how to design and implement guardrails for Large Language Models (LLMs) using a layered, production-inspired approach.

The repository walks through five progressive notebooks, starting from basic rule-based checks and ending with a stateful agent that retries unsafe outputs using feedback loops. The goal is to show how real-world LLM systems balance safety, usability, and correctness.

All notebooks are implemented without inline instructional text, relying instead on architecture, code clarity, and this README for explanation—similar to real engineering environments.


🎯 Goals

  • Detect and prevent prompt injection attacks
  • Identify and block PII leakage (emails, phone numbers, IBANs, addresses)
  • Reduce false positives without compromising safety
  • Demonstrate layered guardrail architectures
  • Implement retry-based recovery instead of hard failures

📂 Project Structure

.
├── notebook_1_rule_based_guardrails.ipynb
├── notebook_2_ml_based_guardrails.ipynb
├── notebook_3_llm_as_judge.ipynb
├── notebook_4_combined_guardrails.ipynb
├── notebook_5_retry_logic_langgraph.ipynb
└── README.md

Each notebook can be executed independently, but they are designed to be followed in order.

Notebook Details

1. Rule-Based Guardrails

What it covers

  • Regex-based PII detection (email, phone number, IBAN, address)
  • Keyword-based prompt injection detection
  • Immediate blocking on violation

Pros

  • Fast and deterministic
  • Easy to debug
  • No model dependency

Cons

  • High false-positive risk
  • Limited flexibility
  • Easy to bypass

2. ML-Based Guardrails (Guardrails AI)

What it covers

  • Prebuilt validators for PII and unsafe content
  • Input and output validation
  • Structured failure handling

Pros

  • Better recall than regex
  • Less brittle
  • Minimal rule maintenance

Cons

  • Limited to predefined validator scope
  • Weak for application-specific policies

3. LLM-as-a-Judge

What it covers

  • Using an LLM to evaluate another LLM’s output
  • Detecting prompt injection via reasoning
  • Enforcing custom safety rules

Pros

  • Highly flexible
  • Strong contextual understanding
  • No training required

Tradeoffs

  • Higher latency and cost
  • Requires careful prompt design

4. Combined Guardrail Architecture

What it covers

  • Layering rule-based, ML-based, and LLM-based checks
  • Input vs output guardrails
  • Fail-fast vs fail-soft strategies

Why it matters

  • Mirrors real production systems
  • Reduces both unsafe outputs and false positives
  • Provides defense-in-depth

5. Agent Retry Logic with LangGraph

What it covers

  • Stateful agent execution
  • Output validation with feedback-driven retries
  • Retry limits and graceful failure handling

Execution Flow

User Input ↓ Input Guardrails ↓ LLM Generation ↓ Output Guardrails ↓ Retry with Feedback (if unsafe) ↓ Safe Response or Controlled Failure

Key Insight

Blocking alone degrades UX. Intelligent retries preserve safety and usefulness.


Testing Coverage

The notebooks include tests for:

  • Prompt injection attempts
  • Valid vs unsafe PII examples
  • Non-US formats (German phone numbers, IBANs)
  • Retry exhaustion scenarios
  • False-positive edge cases

Known Limitations

  • Regex-based PII detection lacks semantic understanding
  • LLM-as-judge introduces latency and cost
  • Prebuilt ML validators are domain-limited
  • No deployment or API layer included

Possible Extensions

  • Named Entity Recognition (NER) for PII detection
  • Metrics and observability for guardrail failures
  • Async execution and streaming support
  • Domain-specific judges (legal, medical, financial)
  • API or microservice deployment

Design Philosophy

Guardrails are not just filters, they are systems.

This project emphasizes:

  • Layered defenses
  • Recovery over rejection
  • Practical tradeoffs over theoretical perfection

Installation

Prerequisites

  • Python 3.10+
  • pip or conda

Setup

# Clone the repository
git clone <your-repo-url>
cd <your-repo-name>

# (Optional) Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Final Notes

This project is intended as a reference implementation for building practical, layered safety systems around LLMs. It prioritizes clarity, debuggability, and real-world tradeoffs over abstract or opaque guardrail mechanisms.

The notebooks and scripts are deliberately kept modular so they can be:

  • Extended with additional detectors or judges
  • Integrated into production pipelines
  • Adapted for domain-specific compliance needs

If you are using this work as a foundation, consider documenting your own policies, thresholds, and failure-handling strategies to ensure transparency and maintainability.


License

This project is provided for educational and experimental purposes.
You are free to adapt and extend it for internal or research use.


Built with care for engineers who value correctness, control, and clarity.

About

A layered approach to building safe, resilient LLM pipelines

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages