🎞️ fix: Deliver Opted-In Custom Endpoint Media Safely

[danny-avila]

Summary

I completed custom-endpoint audio/video delivery and added upload-time rejection for unsupported direct-audio formats. Previously, explicitly allowed media could be hidden by the picker, routed away from the provider, or discarded by the encoder; unsupported audio could upload successfully and then abort message submission.

• Enable picker, drag-and-drop, unified routing, and encoding for custom OpenAI-compatible endpoints with an explicit matching supportedMimeTypes allowlist.
• Preserve inherited defaults and built-in provider behavior; keep explicit transcription, OCR, and code destinations unchanged.
• Canonicalize audio formats and reject unrepresentable direct audio with HTTP 415 before processing, with localized conversion/retry guidance and pending-attachment cleanup.
• Reuse the request-cached agent lookup instead of adding database reads.

Replaces #15709 and addresses #15708. Preserves the original contributor's commits and authorship. Documentation: LibreChat-AI/docs#761.

How it works

The existing endpoint MIME allowlist is the opt-in; no new configuration key is required.

fileConfig:
  endpoints:
    MyGateway:
      supportedMimeTypes:
        - "image/.*"
        - "application/pdf"
        - "video/.*"
        - "audio/.*"

upload route
  resolveEffectiveToolResource
    resolveUploadEndpoint (request-cached)
    resolveUploadLLMDeliveryPath
    getAudioFormat → reject incompatible direct audio with 415
  processAgentFileUpload (only after preflight passes)

The encoder emits OpenAI-format video_url / input_audio parts only for compatible, opted-in custom endpoints. Google/Vertex retain their media-block behavior.

Configuration limitations

Do not allow audio/video on a custom endpoint declared provider: anthropic: the unified route rejects this incompatible combination, but the legacy explicit-provider destination can still accept and discard it. Remove those MIME types from that endpoint's allowlist. Custom endpoint names matching built-in provider identifiers retain the resolver's existing built-in-provider interpretation.

Change Type

• Bug fix (non-breaking change which fixes an issue)
• This change requires a documentation update

Testing

I verified that this branch's complete Git tree is identical to 87bd5cffb2fead9e4d9121c26af208943f955b45, the tested head of #15709. Only commit ancestry changed: six scoped commits on current dev, with no integration merge commit.

• Passed 593 focused tests on that identical source tree: 150 API helper/encoder tests, 299 data-provider tests, 103 client picker/upload tests, and 41 HTTP file-route tests.
• Passed npx tsc --noEmit in packages/api and packages/data-provider, scoped ESLint, Prettier, import ordering, and git diff --check.
• Covered JSON and SSE-requested 415 responses, no processing on rejection, temporary-file cleanup, localized client failure, pending removal, and replacement upload.
• Covered accepted aliases and filename fallback through the real upload resolver and encoder, plus preserved transcription/code and Google/Vertex paths.
• Attempted local client typechecking and Lighthouse; stale local frontend dependencies and unavailable @tsdown/css prevented completion. Both passed in the original head's clean-install CI, alongside all 31 executed checks; five optional checks were skipped.

The original contributor reported a successful configured vLLM video upload and unchanged behavior without opt-in. I did not repeat that live-provider manual test.

New PR CI is pending. The original head received a clean Codex result; that is not a review of this new SHA. No additional review is requested, per maintainer instruction.

Test Configuration:

Node 24.16.0. Base: b88d70bf63598af19e9f599a0b16d6fe9f4aef8f (origin/dev). Configure a custom OpenAI-compatible endpoint using the example allowlist; verify supported media reaches the model, incompatible direct audio returns 415, and a converted replacement can be uploaded.

Checklist

• My code adheres to this project's style guidelines
• I have performed a self-review of my own code
• I have commented in any complex areas of my code
• My changes do not introduce new warnings
• I have written tests demonstrating that my changes are effective or that my feature works
• Local unit tests pass with my changes
• A pull request for updating the documentation has been submitted: 🎞️ docs: Video/Audio Opt-In for Custom Endpoints via supportedMimeTypes docs#761