Building Safe, Reliable, and Resilient LLM Pipelines
This project demonstrates how to design and implement guardrails for Large Language Models (LLMs) using a layered, production-inspired approach.
The repository walks through five progressive notebooks, starting from basic rule-based checks and ending with a stateful agent that retries unsafe outputs using feedback loops. The goal is to show how real-world LLM systems balance safety, usability, and correctness.
All notebooks are implemented without inline instructional text, relying instead on architecture, code clarity, and this README for explanation—similar to real engineering environments.
- Detect and prevent prompt injection attacks
- Identify and block PII leakage (emails, phone numbers, IBANs, addresses)
- Reduce false positives without compromising safety
- Demonstrate layered guardrail architectures
- Implement retry-based recovery instead of hard failures
.
├── notebook_1_rule_based_guardrails.ipynb
├── notebook_2_ml_based_guardrails.ipynb
├── notebook_3_llm_as_judge.ipynb
├── notebook_4_combined_guardrails.ipynb
├── notebook_5_retry_logic_langgraph.ipynb
└── README.md
Each notebook can be executed independently, but they are designed to be followed in order.
What it covers
- Regex-based PII detection (email, phone number, IBAN, address)
- Keyword-based prompt injection detection
- Immediate blocking on violation
Pros
- Fast and deterministic
- Easy to debug
- No model dependency
Cons
- High false-positive risk
- Limited flexibility
- Easy to bypass
What it covers
- Prebuilt validators for PII and unsafe content
- Input and output validation
- Structured failure handling
Pros
- Better recall than regex
- Less brittle
- Minimal rule maintenance
Cons
- Limited to predefined validator scope
- Weak for application-specific policies
What it covers
- Using an LLM to evaluate another LLM’s output
- Detecting prompt injection via reasoning
- Enforcing custom safety rules
Pros
- Highly flexible
- Strong contextual understanding
- No training required
Tradeoffs
- Higher latency and cost
- Requires careful prompt design
What it covers
- Layering rule-based, ML-based, and LLM-based checks
- Input vs output guardrails
- Fail-fast vs fail-soft strategies
Why it matters
- Mirrors real production systems
- Reduces both unsafe outputs and false positives
- Provides defense-in-depth
What it covers
- Stateful agent execution
- Output validation with feedback-driven retries
- Retry limits and graceful failure handling
Execution Flow
User Input ↓ Input Guardrails ↓ LLM Generation ↓ Output Guardrails ↓ Retry with Feedback (if unsafe) ↓ Safe Response or Controlled Failure
Key Insight
Blocking alone degrades UX. Intelligent retries preserve safety and usefulness.
The notebooks include tests for:
- Prompt injection attempts
- Valid vs unsafe PII examples
- Non-US formats (German phone numbers, IBANs)
- Retry exhaustion scenarios
- False-positive edge cases
- Regex-based PII detection lacks semantic understanding
- LLM-as-judge introduces latency and cost
- Prebuilt ML validators are domain-limited
- No deployment or API layer included
- Named Entity Recognition (NER) for PII detection
- Metrics and observability for guardrail failures
- Async execution and streaming support
- Domain-specific judges (legal, medical, financial)
- API or microservice deployment
Guardrails are not just filters, they are systems.
This project emphasizes:
- Layered defenses
- Recovery over rejection
- Practical tradeoffs over theoretical perfection
- Python 3.10+
piporconda
# Clone the repository
git clone <your-repo-url>
cd <your-repo-name>
# (Optional) Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
This project is intended as a reference implementation for building practical, layered safety systems around LLMs. It prioritizes clarity, debuggability, and real-world tradeoffs over abstract or opaque guardrail mechanisms.
The notebooks and scripts are deliberately kept modular so they can be:
- Extended with additional detectors or judges
- Integrated into production pipelines
- Adapted for domain-specific compliance needs
If you are using this work as a foundation, consider documenting your own policies, thresholds, and failure-handling strategies to ensure transparency and maintainability.
This project is provided for educational and experimental purposes.
You are free to adapt and extend it for internal or research use.
Built with care for engineers who value correctness, control, and clarity.