feat: mcp pipedream integration - #5578
Draft
ehayes2000 wants to merge 3 commits into
Draft
Conversation
Replace the fragile parts of the in-house MCP OAuth machinery with Nango (nango.dev) as the managed auth provider, so users can connect arbitrary spec-compliant MCP servers to the agent: - Nango's generic MCP integration handles OAuth endpoint discovery, dynamic client registration (DCR/CIMD), the consent flow (hosted Connect UI), token storage, and refresh. We store only the Nango connection ID (new nullable column on mcp_servers) and fetch fresh access tokens at connect time, cached in-process. - New endpoints: POST /mcp/servers/nango/session mints a Connect session for the current user (optionally pre-filling the server URL so the UI goes straight to consent); POST /mcp/servers/nango/complete verifies the finished connection against Nango (existence + end-user ownership) before upserting the server row and firing the auth-completed hook. DELETE /mcp/servers now also revokes the Nango connection. - NangoResolvingStore decorates the Pg server store, resolving tokens into records at load time, so the chat toolset and the import pipeline work with Nango-authorized servers unchanged. - Connections settings page and onboarding connector steps connect through the Nango Connect UI (vendored minimal iframe client), with the legacy in-house OAuth flow kept as fallback for deployments without NANGO_SECRET_KEY and for servers that need pre-registered clients (GitHub, Slack). Config (all optional, doppler): NANGO_SECRET_KEY, NANGO_API_URL (default https://api.nango.dev), NANGO_MCP_INTEGRATION_ID (default mcp-generic). Unset means Nango endpoints answer 501 and the legacy flow keeps working.
Advertise every connectable MCP server instead of a hard-coded preset list. New GET /mcp/servers/catalog merges two sources: - A curated priority list (domain/service/catalog.rs) — the connectors we actively promote, pinned to the top with product-voice taglines and flagged `priority` so clients can render them as a featured section. Adding a connector there is all it takes to advertise it. - The public MCP registry (registry.modelcontextprotocol.io), searched live (substring on server names, cursor pagination, version=latest), filtered to active servers with a plain streamable HTTP remote, and deduplicated against the priority tier. Pages are cached in-process for 5 minutes. MCP_REGISTRY_URL overrides the base URL if ever needed. The catalog route gets its own small router state (registry + auth) since it needs none of the per-user store/OAuth/Nango machinery. The Connections settings page now shows connected servers and the featured tier (served from bundled presets until the catalog answers, so nothing jumps), plus a searchable "all connectors" card with debounced search and load-more pagination. Connect buttons reuse the existing Nango flow with the legacy fallback; icons come from our bundled SVGs when we have them, the registry's icon otherwise. mcpUrlSupportsNango now keys off a static URL set so GitHub/Slack route to the legacy flow even when reached via the catalog rather than the (environment-gated) presets.
One way to connect MCP connectors, always Pipedream — no fallback paths.
Deletes the Nango integration and the entire legacy in-house OAuth
machinery (PKCE flow, AES-encrypted credential storage, Redis OAuth state,
pre-registered Slack/GitHub clients, OAuth callback endpoints).
How it works now:
- Pipedream Connect owns the account lifecycle: hosted consent UI,
provider OAuth apps (no per-provider client registration on our side),
credential storage, and refresh. We store only {app_slug, account_id}
per user — the mcp_servers table is rebuilt around that (existing rows
dropped; old grants have no forward path into Pipedream, users
reconnect through the new flow).
- Tool calls go through Pipedream's remote MCP server
(remote.mcp.pipedream.net), scoped per user + app via headers, in
tools-only mode; account credentials are injected by Pipedream
server-side and never transit our systems. Toolsets take an
McpConnection port; unconfigured deployments come up empty.
- Connect flow: POST /mcp/servers/pipedream/token mints a Connect token,
the frontend opens the hosted Connect UI (vendored minimal iframe
client), and POST /mcp/servers/pipedream/complete verifies the reported
account against Pipedream (existence + ownership by the calling user)
before persisting and firing the auth-completed hook. DELETE revokes
the Pipedream account.
- The connector catalog is backed by Pipedream's app directory (~2,500
apps, searched with cursor pagination, auth-less apps filtered out),
still merged with the curated priority tier pinned on top.
- Frontend: connections page and onboarding connector steps run the
Pipedream flow; connected rows are enable/disable + remove (the
unauthenticated half-state no longer exists). The custom add-by-URL
dialog is gone — connectors come from the catalog.
- Import/onboarding gathers match connectors by app slug instead of
server URL.
Config (doppler, document-cognition): PIPEDREAM_CLIENT_ID,
PIPEDREAM_CLIENT_SECRET, PIPEDREAM_PROJECT_ID; optional
PIPEDREAM_ENVIRONMENT (defaults production in prd, development
elsewhere), PIPEDREAM_API_URL, PIPEDREAM_MCP_URL. Unset means the MCP
connect/catalog endpoints answer 501 and toolsets are empty. Obsolete:
NANGO_*, MCP_REGISTRY_URL, SLACK_MCP_CLIENT_ID/SECRET,
GITHUB_CLIENT_ID/SECRET, MCP_CREDENTIALS_KEY_SECRET_NAME.
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.