Merge pull request #4 from webstackdev/feature/wp4-embeddings-and-qdrant-integration

Feature/wp4 embeddings and qdrant integration

Author: webstackdev

.env.example

@@ -8,7 +8,13 @@ REDIS_URL=redis://localhost:6379/0
 QDRANT_URL=http://localhost:6333
 
 OPENROUTER_API_KEY=
+OPENROUTER_API_BASE=https://openrouter.ai/api/v1
+OPENROUTER_APP_URL=
+OPENROUTER_APP_NAME=newsletter-maker
+EMBEDDING_PROVIDER=sentence-transformers
 EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2
+EMBEDDING_TRUST_REMOTE_CODE=false
+OLLAMA_URL=http://localhost:11434
 REDDIT_CLIENT_ID=
 REDDIT_CLIENT_SECRET=
 REDDIT_USER_AGENT=newsletter-maker/0.1

.vscode/settings.json

@@ -10,6 +10,7 @@
     "mday",
     "noinput",
     "nomic",
+    "OLLAMA",
     "PRAW",
     "PYTHONDONTWRITEBYTECODE",
     "PYTHONUNBUFFERED",

IMPLEMENTATION.md

@@ -32,7 +32,9 @@ Set up the Django project, Docker infrastructure, and development environment.
 - `.env.example` with all required environment variables:
   - `DATABASE_URL`, `REDIS_URL`, `QDRANT_URL`
   - `OPENROUTER_API_KEY`
-  - `EMBEDDING_MODEL` (default: `nomic-embed-text` or `all-MiniLM-L6-v2`)
+    - `EMBEDDING_PROVIDER`, `EMBEDDING_MODEL`
+    - `OLLAMA_URL` for local Ollama embeddings
+    - `OPENROUTER_API_BASE` for hosted OpenAI-compatible embeddings APIs
   - `SECRET_KEY`, `DEBUG`, `ALLOWED_HOSTS`
 - Health check endpoints: `GET /healthz/` (system health), `GET /readyz/` (DB + Qdrant reachable)
 - `justfile` with commands: `dev` (docker compose up), `test`, `migrate`, `seed`, `shell`
@@ -218,7 +220,18 @@ What is not built yet is the full done state from the plan, especially live feed
 
 Compute embeddings for all ingested content and store them in Qdrant for similarity search.
 
-**Embedding model:** Local, free — `sentence-transformers/all-MiniLM-L6-v2` (384 dimensions) or `nomic-embed-text` via Ollama (768 dimensions). Configurable via `EMBEDDING_MODEL` env var.
+**Embedding backend:** Configurable via `EMBEDDING_PROVIDER` plus `EMBEDDING_MODEL`.
+
+- `sentence-transformers`: local Hugging Face / SentenceTransformers model loading
+- `ollama`: local model served over Ollama's embedding API
+- `openrouter`: hosted embeddings through the OpenRouter `/embeddings` API
+
+Examples:
+
+- `EMBEDDING_PROVIDER=sentence-transformers`, `EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2`
+- `EMBEDDING_PROVIDER=ollama`, `EMBEDDING_MODEL=nomic-embed-text`
+- `EMBEDDING_PROVIDER=ollama`, `EMBEDDING_MODEL=qwen3-embedding-8b`
+- `EMBEDDING_PROVIDER=openrouter`, `EMBEDDING_MODEL=openai/text-embedding-3-small`
 
 **Qdrant setup:**
 
@@ -230,8 +243,9 @@ Compute embeddings for all ingested content and store them in Qdrant for similar
 
 1. Plugin fetches article → `Content` record created in Postgres
 2. Post-save signal (or inline in ingestion task) computes embedding
-3. Embedding + metadata upserted into tenant's Qdrant collection
-4. `Content.embedding_id` stores the Qdrant point ID for later retrieval
+3. The configured embedding provider returns a vector for the content text
+4. Embedding + metadata upserted into tenant's Qdrant collection
+5. `Content.embedding_id` stores the Qdrant point ID for later retrieval
 
 **Reference corpus seeding:**
 
@@ -246,8 +260,17 @@ Compute embeddings for all ingested content and store them in Qdrant for similar
 - `search_similar(tenant_id: int, query_vector: list[float], limit: int) -> list[ScoredPoint]` — find similar content
 - `get_reference_similarity(tenant_id: int, vector: list[float]) -> float` — average similarity against reference corpus
 
+**Operational usage:**
+
+- `just embed-all` — backfill embeddings for all content rows
+- `just embed-tenant <tenant_id>` — backfill one tenant's content
+- `python3 manage.py sync_embeddings --content-id <id>` — re-embed one record
+
 **Definition of done:** Every ingested content item has an embedding in Qdrant. `search_similar` returns semantically related articles. Reference corpus is seeded for test tenant.
 
+- Run `just embed-smoke` to confirm Django can talk to Ollama.
+- If that works, run `just embed-all` or `just embed-tenant <tenant_id>` to backfill real content.
+
 ### WP5: AI Skills + LangGraph Pipeline
 
 Implement the three Phase 1 skills and wire them into a LangGraph orchestrator.

README.md

@@ -92,6 +92,47 @@ python3 -m pip install -r requirements.txt
 
 For host-based development without Docker, install `requirements.txt`, then use `python3 manage.py migrate` and `python3 manage.py runserver`. The default `.env.example` is host-safe; Docker Compose overrides the service URLs inside containers.
 
+### Embedding Backends
+
+The embedding layer is provider-based. Configure it with `EMBEDDING_PROVIDER` and `EMBEDDING_MODEL`:
+
+- `sentence-transformers`: loads a Hugging Face / SentenceTransformers model inside the Django process
+- `ollama`: calls a local Ollama server for embeddings
+- `openrouter`: calls OpenRouter's embeddings API using the configured model id
+
+Common examples:
+
+```dotenv
+EMBEDDING_PROVIDER=sentence-transformers
+EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2
+```
+
+```dotenv
+EMBEDDING_PROVIDER=ollama
+EMBEDDING_MODEL=nomic-embed-text
+OLLAMA_URL=http://localhost:11434
+```
+
+```dotenv
+EMBEDDING_PROVIDER=openrouter
+EMBEDDING_MODEL=openai/text-embedding-3-small
+OPENROUTER_API_KEY=...
+OPENROUTER_API_BASE=https://openrouter.ai/api/v1
+```
+
+For SentenceTransformers models that require custom remote code, set `EMBEDDING_TRUST_REMOTE_CODE=true`.
+
+### Embedding Commands
+
+Use these commands to backfill or refresh embeddings for existing content:
+
+```bash
+just embed-all
+just embed-tenant 1
+python3 manage.py sync_embeddings --content-id 42
+python3 manage.py sync_embeddings --references-only
+```
+
 When `just dev` is running, Django admin uses the Postgres database inside Docker, not the host SQLite database. That means host commands like `python manage.py createsuperuser` create users in SQLite and will not let you log into the Docker-backed admin site.
 
 Create or update an admin user for the running Docker stack with:

core/admin.py

@@ -23,8 +23,8 @@ class EntityAdmin(admin.ModelAdmin):
 
 @admin.register(Content)
 class ContentAdmin(admin.ModelAdmin):
-	list_display = ("title", "tenant", "source_plugin", "published_date", "relevance_score", "is_active")
-	list_filter = ("tenant", "source_plugin", "is_active")
+	list_display = ("title", "tenant", "source_plugin", "published_date", "relevance_score", "is_reference", "is_active")
+	list_filter = ("tenant", "source_plugin", "is_reference", "is_active")
 	search_fields = ("title", "author", "url")
 
 

core/embeddings.py

@@ -0,0 +1,221 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from functools import lru_cache
+from uuid import uuid4
+
+from django.conf import settings
+from django.utils.dateparse import parse_datetime
+import httpx
+from qdrant_client import QdrantClient
+from qdrant_client.models import Distance, FieldCondition, Filter, MatchValue, PointStruct, VectorParams
+from sentence_transformers import SentenceTransformer
+
+from core.models import Content
+
+
+class EmbeddingProvider(ABC):
+    @abstractmethod
+    def embed_text(self, text: str) -> list[float]:
+        raise NotImplementedError
+
+    def get_embedding_dimension(self) -> int:
+        return len(self.embed_text("dimension probe"))
+
+
+class SentenceTransformerEmbeddingProvider(EmbeddingProvider):
+    def __init__(self):
+        self.model = SentenceTransformer(
+            settings.EMBEDDING_MODEL,
+            trust_remote_code=settings.EMBEDDING_TRUST_REMOTE_CODE,
+        )
+
+    def embed_text(self, text: str) -> list[float]:
+        return self.model.encode(text, normalize_embeddings=True).tolist()
+
+    def get_embedding_dimension(self) -> int:
+        return int(self.model.get_sentence_embedding_dimension())
+
+
+class OllamaEmbeddingProvider(EmbeddingProvider):
+    def embed_text(self, text: str) -> list[float]:
+        normalized_text = normalize_text(text)
+        response = httpx.post(
+            f"{settings.OLLAMA_URL.rstrip('/')}/api/embed",
+            json={"model": settings.EMBEDDING_MODEL, "input": [normalized_text]},
+            timeout=30.0,
+        )
+        if response.status_code == 404:
+            legacy_response = httpx.post(
+                f"{settings.OLLAMA_URL.rstrip('/')}/api/embeddings",
+                json={"model": settings.EMBEDDING_MODEL, "prompt": normalized_text},
+                timeout=30.0,
+            )
+            legacy_response.raise_for_status()
+            return legacy_response.json()["embedding"]
+        response.raise_for_status()
+        return response.json()["embeddings"][0]
+
+
+class OpenRouterEmbeddingProvider(EmbeddingProvider):
+    def embed_text(self, text: str) -> list[float]:
+        if not settings.OPENROUTER_API_KEY:
+            raise RuntimeError("OPENROUTER_API_KEY must be set when using the openrouter embedding provider.")
+        headers = {
+            "Authorization": f"Bearer {settings.OPENROUTER_API_KEY}",
+            "Content-Type": "application/json",
+        }
+        if settings.OPENROUTER_APP_URL:
+            headers["HTTP-Referer"] = settings.OPENROUTER_APP_URL
+        if settings.OPENROUTER_APP_NAME:
+            headers["X-OpenRouter-Title"] = settings.OPENROUTER_APP_NAME
+        response = httpx.post(
+            f"{settings.OPENROUTER_API_BASE.rstrip('/')}/embeddings",
+            headers=headers,
+            json={
+                "model": settings.EMBEDDING_MODEL,
+                "input": normalize_text(text),
+                "encoding_format": "float",
+            },
+            timeout=30.0,
+        )
+        response.raise_for_status()
+        return response.json()["data"][0]["embedding"]
+
+
+def collection_name_for_tenant(tenant_id: int) -> str:
+    return f"tenant_{tenant_id}_content"
+
+
+@lru_cache(maxsize=1)
+def get_qdrant_client() -> QdrantClient:
+    return QdrantClient(url=settings.QDRANT_URL, timeout=10.0)
+
+
+@lru_cache(maxsize=1)
+def get_embedding_provider() -> EmbeddingProvider:
+    provider_name = settings.EMBEDDING_PROVIDER
+    if provider_name == "sentence-transformers":
+        return SentenceTransformerEmbeddingProvider()
+    if provider_name == "ollama":
+        return OllamaEmbeddingProvider()
+    if provider_name == "openrouter":
+        return OpenRouterEmbeddingProvider()
+    raise ValueError(f"Unsupported embedding provider: {provider_name}")
+
+
+def get_embedding_dimension() -> int:
+    return get_embedding_provider().get_embedding_dimension()
+
+
+def embed_text(text: str) -> list[float]:
+    return get_embedding_provider().embed_text(normalize_text(text))
+
+
+def upsert_content_embedding(content: Content) -> str:
+    client = get_qdrant_client()
+    ensure_tenant_collection(content.tenant_id)
+    embedding_id = content.embedding_id or str(uuid4())
+    vector = embed_text(build_content_embedding_text(content))
+    client.upsert(
+        collection_name=collection_name_for_tenant(content.tenant_id),
+        points=[
+            PointStruct(
+                id=embedding_id,
+                vector=vector,
+                payload={
+                    "content_id": content.id,
+                    "tenant_id": content.tenant_id,
+                    "url": content.url,
+                    "title": content.title,
+                    "published_date": serialize_published_date(content.published_date),
+                    "source_plugin": content.source_plugin,
+                    "is_reference": content.is_reference,
+                },
+            )
+        ],
+        wait=True,
+    )
+    if content.embedding_id != embedding_id:
+        content.embedding_id = embedding_id
+        content.save(update_fields=["embedding_id"])
+    return embedding_id
+
+
+def search_similar(
+    tenant_id: int,
+    query_vector: list[float],
+    limit: int = 10,
+    *,
+    is_reference: bool | None = None,
+    exclude_content_id: int | None = None,
+):
+    if not tenant_collection_exists(tenant_id):
+        return []
+    query_filter = build_search_filter(is_reference=is_reference, exclude_content_id=exclude_content_id)
+    return get_qdrant_client().search(
+        collection_name=collection_name_for_tenant(tenant_id),
+        query_vector=query_vector,
+        limit=limit,
+        query_filter=query_filter,
+        with_payload=True,
+    )
+
+
+def get_reference_similarity(tenant_id: int, vector: list[float], limit: int = 5) -> float:
+    scored_points = search_similar(tenant_id, vector, limit=limit, is_reference=True)
+    if not scored_points:
+        return 0.0
+    return sum(point.score for point in scored_points) / len(scored_points)
+
+
+def ensure_tenant_collection(tenant_id: int) -> None:
+    client = get_qdrant_client()
+    collection_name = collection_name_for_tenant(tenant_id)
+    if tenant_collection_exists(tenant_id):
+        return
+    client.create_collection(
+        collection_name=collection_name,
+        vectors_config=VectorParams(size=get_embedding_dimension(), distance=Distance.COSINE),
+    )
+
+
+def tenant_collection_exists(tenant_id: int) -> bool:
+    try:
+        get_qdrant_client().get_collection(collection_name_for_tenant(tenant_id))
+    except Exception:
+        return False
+    return True
+
+
+def build_content_embedding_text(content: Content) -> str:
+    return "\n\n".join(part for part in [content.title, content.content_text] if part)
+
+
+def normalize_text(text: str) -> str:
+    normalized_text = text.strip()
+    if not normalized_text:
+        return "empty content"
+    return normalized_text
+
+
+def serialize_published_date(value) -> str:
+    if hasattr(value, "isoformat"):
+        return value.isoformat()
+    if isinstance(value, str):
+        parsed_value = parse_datetime(value)
+        if parsed_value is not None:
+            return parsed_value.isoformat()
+        return value
+    return str(value)
+
+
+def build_search_filter(*, is_reference: bool | None = None, exclude_content_id: int | None = None):
+    conditions = []
+    if is_reference is not None:
+        conditions.append(FieldCondition(key="is_reference", match=MatchValue(value=is_reference)))
+    if exclude_content_id is not None:
+        conditions.append(FieldCondition(key="content_id", match=MatchValue(value=exclude_content_id)))
+    if not conditions:
+        return None
+    return Filter(must=conditions if exclude_content_id is None else conditions[:-1], must_not=conditions[-1:] if exclude_content_id is not None else None)