context-mcp

A retrieval-augmented knowledge service exposed as a Model Context Protocol server. It ingests a small corpus, chunks and embeds it into pgvector-backed Postgres, and answers queries through two MCP tools — search(query) and answer(query) — the second returning an answer grounded in, and citing, the chunks it retrieved.

The focus here is to understand and exercise LLM observability and offline evals, the chatbot itself its just a means to an end. Every search()/answer() call is traced (LangSmith, when configured); a hand-labeled dataset with three scorers runs on every pull request and fails the build on a quality regression — a real CI gate, not a report someone might read later. Everything else — the corpus, the retrieval algorithm, the chunking strategy — is the simplest thing that works. See docs/ARCHITECTURE.md for the full pipeline and docs/adr/ for the reasoning behind the two decisions that shape this repo most.

How it works

flowchart LR
    corpus[corpus/*.md] -- "ingest" --> store[(pgvector / Postgres)]
    client[MCP client] -- "POST /mcp or stdio" --> tools["search / answer tools"]
    tools -- "embed + top-K search" --> store
    tools -- "answer()" --> llm[LLM client]
    llm -- "answer + citations" --> client
    tools -. "traceable()" .-> langsmith[LangSmith, if configured]

Loading

1. Ingest — chunk the bundled corpus, embed each chunk, upsert into the configured vector store.
2. Search — embed the query, retrieve the top-K most similar chunks.
3. Answer — hand the retrieved chunks to an LLM client, which returns an answer plus the citations it drew from.
4. Trace — every call is wrapped in a tracing decorator that's a no-op unless LANGSMITH_API_KEY is set.

Every model-backed component (Embedder, LLMClient, VectorStore) has a deterministic default — a hashing-trick fake embedder, a purely extractive stub LLM, an in-memory store — and a real, opt-in implementation that activates only when its corresponding secret (OPENAI_API_KEY, DATABASE_URL) is set. This means the full pipeline, the test suite, and the offline eval suite all run with zero network calls and zero API cost, every time, including in a fresh clone. See ADR-0002.

Operated by

This repo is operated day-to-day the same way as the rest of this account's AWS workloads: fleetctl discovers it by its user:project=context-mcp tag, tails /context-mcp/<env>/mcp-server logs, and checks its FinOps tag compliance. See FINOPS_TAGGING.md.

Local development

Zero AWS needed — a docker-compose stack runs the MCP server against a real pgvector-enabled Postgres container, with the bundled sample corpus, entirely offline:

make local-up   # docker compose -f local/docker-compose.yml up --build

Then call search/answer over the HTTP transport:

curl -s http://localhost:8080/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"answer","arguments":{"query":"How long should cold brew coffee steep?"}}}' | jq

make local-down tears the stack down. To run the CLI directly instead (useful for iterating on ingestion or the answer chain without Docker):

cd service
pip install -r requirements-dev.txt && pip install --no-deps -e .
DATABASE_URL="" context-mcp ingest      # in-memory store; falls back automatically
context-mcp serve --stdio               # or --http, for the HTTP transport
context-mcp evals                       # run the offline eval suite locally

Common tasks

• skills/add-a-retriever-or-datasource.md — add corpus documents, or swap in a new embedding/LLM/vector-store provider.
• skills/add-a-langsmith-eval.md — extend the offline eval dataset or add a new scorer.
• AGENTS.md — where things live and the constraints that don't bend, for an agent asked to change this repo's code or infrastructure.

Hosting behind mcp-gateway

context-mcp has no auth of its own — the MCP server is designed to be usable either standalone (as this README describes) or fronted by mcp-gateway, an authenticating, RBAC-enforcing reverse proxy for MCP servers. This service satisfies every requirement that gateway's register-a-backend-mcp-server.md skill lists: POST /mcp answering initialize/tools/list/tools/call as one JSON request in, one JSON response out (no SSE), GET /healthz returning 200, a single shared listen port, no auth of its own.

Registering context-mcp behind a gateway deployment is an integration pattern, not a code or infrastructure dependency: this repo's iac/ provisions its own VPC, ECS cluster, and RDS instance, entirely independent of any gateway — no shared Terraform state, no cross-repo module reference. An operator wiring this service into a gateway follows that skill's ECS steps in the gateway's repo, pointing at this service's ECR image and ECS service.

Deploy

make validate   # terraform init -backend=false && terraform validate
make test       # pytest + the offline eval suite (fakes only, no AWS creds needed)
make lint       # ruff check, tflint

Real deploys are CI-only, authenticated via GitHub OIDC against the role iac/main.tf's module.deploy_role creates — no static AWS keys, ever. .github/workflows/terraform.yml plans on PR and applies on merge to main; .github/workflows/deploy-app.yml builds and pushes the MCP server image and force-deploys the ECS service; .github/workflows/secrets-sync.yml (manual dispatch) writes real secret values from GitHub Environment secrets into SSM.

iac/ provisions one VPC (public subnets only, no NAT Gateway), an ECS cluster with a Fargate service for the MCP server, one RDS Postgres instance with a pgvector-eligible engine version, one ECR repository, the SSM config/secret surface below, an OIDC deploy role, and a budget alarm — every module consumed remotely from terraform-modules at ?ref=v0.4.0. No vendored iac/modules/ in this repo. See docs/ARCHITECTURE.md for the full inventory.

Config & secrets

Runtime config lives in SSM Parameter Store at /context-mcp/<environment>/<KEY>.

Key	Kind	Notes
EMBED_MODEL	String	Which OpenAI embedding model to use if OPENAI_API_KEY is set; ignored otherwise
VECTOR_TABLE	String	Table name PgVectorStore reads/writes
TOP_K	String	Default retrieval depth for search/answer
LANGSMITH_PROJECT	String	LangSmith project traces land in, when tracing is active
OPENAI_API_KEY	SecureString	Unset by default — real embedder/LLM stay off, fakes stay on
LANGSMITH_API_KEY	SecureString	Unset by default — tracing stays a no-op
DATABASE_URL	SecureString	Postgres connection string; assembled by an operator from the RDS-managed master-user secret, then synced the same way as every other secret

String parameters are Terraform-owned (iac/ssm.tf's module.string_config). Secret parameters are name-only in Terraform — a placeholder value with lifecycle.ignore_changes — and get their real value only from secrets-sync.yml. No secret value ever appears in code, .tfvars, or Terraform state.

Cost

Everything scales to a small, predictable footprint: one Fargate task, one db.t4g.micro RDS instance, no NAT Gateway. The fake embedder and stub LLM being the default means a fresh clone never accidentally incurs an OpenAI bill just by running tests or the eval suite. A budget alarm (module.budget) alerts on spend against the user:project tag. The whole stack is destroyable and safe to bring up in a fresh AWS account — terraform destroy removes everything iac/ created.

License

MIT