Your personal AI memory assistant via WhatsApp - Store, search, and retrieve memories using natural language through text and voice.
MemVault is a multi-tenant digital memory system that integrates with WhatsApp to provide seamless memory storage and retrieval. Send text or voice messages to store memories, and query them using natural language—all encrypted and secure.
- 🔐 Multi-Tenant Authentication - Per-user isolation with WhatsApp phone verification
- 🎤 Voice-to-Memory - Automatic speech-to-text conversion using OpenAI Whisper
- 🧠 Intelligent Intent Detection - LLM-powered classification of store vs. query intents
- 🔍 Hybrid Search - BM25 + Vector search for optimal retrieval accuracy
- 🔒 Encryption at Rest - AES-256-GCM encryption for all stored memories
- ⚡ Scalable Architecture - Designed for 1000s of concurrent users
┌─────────────────────────────────────────────────────────────────────────────┐
│ MEMVAULT ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────────┐ ┌──────────────────────────┐ │
│ │ WhatsApp │────▶│ Twilio API │────▶│ FastAPI Backend │ │
│ │ Client │◀────│ (Webhook) │◀────│ │ │
│ └─────────────┘ └─────────────────┘ │ ┌────────────────────┐ │ │
│ │ │ Authentication │ │ │
│ │ │ (Phone-based) │ │ │
│ │ └────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌────────────────────┐ │ │
│ │ │ LangGraph Flow │ │ │
│ ┌─────────────────────────────────────┐ │ │ │ │ │
│ │ LANGGRAPH WORKFLOW │ │ │ ┌──────────────┐ │ │ │
│ │ │ │ │ │ Voice→Text │ │ │ │
│ │ ┌─────────┐ ┌─────────────┐ │ │ │ │ (Whisper) │ │ │ │
│ │ │ Input │───▶│ Intent │ │ │ │ └──────┬───────┘ │ │ │
│ │ │ Process │ │ Classifier │ │ │ │ │ │ │ │
│ │ └─────────┘ │ (LLM) │ │ │ │ ▼ │ │ │
│ │ └──────┬──────┘ │ │ │ ┌──────────────┐ │ │ │
│ │ │ │ │ │ │ Intent │ │ │ │
│ │ ┌────────────┴───────┐ │ │ │ │ Classifier │ │ │ │
│ │ │ │ │ │ │ │ (LLM) │ │ │ │
│ │ ▼ ▼ │ │ │ └──────┬───────┘ │ │ │
│ │ ┌──────────┐ ┌─────────┐│ │ │ │ │ │ │
│ │ │ STORE │ │ QUERY ││ │ │ ┌────┴────┐ │ │ │
│ │ │ Memory │ │ Search ││ │ │ │ │ │ │ │
│ │ └────┬─────┘ └────┬────┘│ │ │ ▼ ▼ │ │ │
│ │ │ │ │ │ │ Store Search │ │ │
│ │ └────────┬───────────┘ │ │ │ Memory Memory │ │ │
│ │ ▼ │ │ └────────────────────┘ │ │
│ │ ┌─────────────┐ │ └──────────────────────────┘ │
│ │ │ Response │ │ │
│ │ │ Generator │ │ │
│ │ └─────────────┘ │ │
│ └─────────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────────────────┐ │
│ │ DATA LAYER │ │
│ │ │ │
│ │ ┌──────────────────┐ ┌───────────────────────────────────┐ │ │
│ │ │ PostgreSQL │ │ Qdrant │ │ │
│ │ │ │ │ │ │ │
│ │ │ • User accounts │ │ • Vector embeddings (encrypted) │ │ │
│ │ │ • Auth tokens │ │ • BM25 sparse vectors │ │ │
│ │ │ • Audit logs │ │ • Multi-tenant collections │ │ │
│ │ │ • Usage metrics │ │ • Hybrid search indexes │ │ │
│ │ └──────────────────┘ └───────────────────────────────────┘ │ │
│ │ │ │
│ └───────────────────────────────────────────────────────────────────┘ │
│ │
│ │
└───────────────────────────────────────────────────────────────────────────┘
memvault/
├── backend/
│ ├── app/
│ │ ├── api/ # FastAPI routes & webhooks
│ │ │ ├── __init__.py
│ │ │ ├── routes.py # API endpoints
│ │ │ └── webhooks.py # WhatsApp/Twilio webhooks
│ │ ├── core/ # Core configuration
│ │ │ ├── __init__.py
│ │ │ ├── config.py # Settings & environment
│ │ │ ├── security.py # Encryption & auth
│ │ │ └── exceptions.py # Custom exceptions
│ │ ├── db/ # Database layer
│ │ │ ├── __init__.py
│ │ │ ├── postgres.py # PostgreSQL models & queries
│ │ │ └── qdrant.py # Qdrant vector DB operations
│ │ ├── llm/ # LangGraph & AI components
│ │ │ ├── __init__.py
│ │ │ ├── graph.py # LangGraph workflow
│ │ │ ├── embeddings.py # Text embeddings
│ │ │ └── whisper.py # Speech-to-text
│ │ ├── services/ # Business logic
│ │ │ ├── __init__.py
│ │ │ ├── memory.py # Memory CRUD operations
│ │ │ └── search.py # Hybrid search implementation
│ │ └── main.py # FastAPI application entry
│ ├── tests/ # Test suite
│ │ ├── __init__.py
│ │ ├── conftest.py # Pytest fixtures
│ │ ├── test_api.py # API endpoint tests
│ │ ├── test_graph.py # LangGraph workflow tests
│ │ ├── test_search.py # Search functionality tests
│ │ └── test_security.py # Encryption tests
│ ├── Dockerfile
│ ├── requirements.txt
│ └── pyproject.toml # Kubernetes manifests
├── docs/ # Documentation
│ ├── api.md # API documentation
│ ├── deployment.md # Deployment guide
│ └── architecture.png # Architecture diagram
├── scripts/ # Utility scripts
│ ├── setup.sh # Development setup
│ └── migrate.py # Database migrations
├── .env.example # Environment template
├── .gitignore
├── LICENSE
└── README.md
- Python 3.11+
- Docker & Docker Compose
- Twilio account with WhatsApp sandbox
- OpenAI API key
- Qdrant Cloud or local instance
git clone https://github.com/yourusername/memvault.git
cd memvault
# Copy environment template
cp .env.example .envEdit .env with your credentials:
# API Keys
OPENAI_API_KEY=sk-...
TWILIO_ACCOUNT_SID=AC...
TWILIO_AUTH_TOKEN=...
TWILIO_WHATSAPP_NUMBER=
# Database
POSTGRES_URL=postgresql://user:pass@localhost:5432/memvault
QDRANT_URL=http://localhost:6333
QDRANT_API_KEY=your-qdrant-key
# Security
ENCRYPTION_KEY=your-32-byte-base64-key
JWT_SECRET=your-jwt-secret
# Observability
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
PROMETHEUS_PORT=9090# Start all services (dev mode)
docker-compose up -d
# Check logs
docker-compose logs -f backend- Go to Twilio Console
- Set webhook URL:
https://your-domain.com/api/v1/webhook/whatsapp - Join the sandbox: Send "join " to the WhatsApp number
Send a message to your WhatsApp bot:
- Store a memory: "Remember that my favorite restaurant is Sushi Ko on Main Street"
- Query memories: "What's my favorite restaurant?"
- Voice notes: Send a voice message and it will be transcribed and processed
| Variable | Description | Required |
|---|---|---|
OPENAI_API_KEY |
OpenAI API key for embeddings & LLM | ✅ |
TWILIO_ACCOUNT_SID |
Twilio account SID | ✅ |
TWILIO_AUTH_TOKEN |
Twilio auth token | ✅ |
TWILIO_WHATSAPP_NUMBER |
WhatsApp-enabled number | ✅ |
POSTGRES_URL |
PostgreSQL connection string | ✅ |
QDRANT_URL |
Qdrant server URL | ✅ |
QDRANT_API_KEY |
Qdrant API key | ❌ |
ENCRYPTION_KEY |
32-byte base64 key for AES-256 | ✅ |
JWT_SECRET |
Secret for JWT tokens | ✅ |
LOG_LEVEL |
Logging level (DEBUG/INFO/WARNING) | ❌ |
MAX_MEMORIES_PER_USER |
Memory limit per user | ❌ |
OTEL_EXPORTER_OTLP_ENDPOINT |
OpenTelemetry endpoint | ❌ |
python -c "import secrets; import base64; print(base64.b64encode(secrets.token_bytes(32)).decode())"# Run all tests
pytest
# Run with coverage
pytest --cov=app --cov-report=html
# Run specific test file
pytest tests/test_api.py -v
# Run integration tests
pytest tests/ -m integration# Build production image
docker build -t memvault:latest ./backend
# Run with production compose
docker-compose -f infra/docker-compose.prod.yml up -d| Environment | Config File | Notes |
|---|---|---|
| Development | docker-compose.yml |
Hot reload, debug logging |
| Staging | docker-compose.staging.yml |
Test webhooks, sandbox |
| Production | docker-compose.prod.yml |
Full HA, autoscaling |
- At Rest: AES-256-GCM encryption for all memory content
- In Transit: TLS 1.3 for all connections
- Keys: Environment-based key management (use Vault in production)
- Phone number verification via Twilio
- JWT tokens with configurable expiry
- Rate limiting per user
- Complete data isolation per user
- Separate Qdrant collections per tenant
- Row-level security in PostgreSQL
- PostgreSQL: Connection pooling with PgBouncer
- Qdrant: Distributed mode with sharding
- Stateless FastAPI workers behind load balancer
- Redis for session state (if needed)
- Kubernetes HPA for auto-scaling
- Fork the repository
- Create a feature branch (
git checkout -b feature/great-feature) - Commit changes (
git commit -m 'Add great feature') - Push to branch (
git push origin feature/great-feature) - Open a Pull Request
This project is licensed under the MIT License - see the LICENSE file.
- LangGraph - Workflow orchestration
- Qdrant - Vector database
- FastAPI - Web framework
- Twilio - WhatsApp integration
vibe coded with ❤️ for the future of personal memory