Pubmatic multi inference

[Shantanu1058]

Thanks for submitting a PR to Triton!
Please go the the Preview tab above this description box and select the appropriate sub-template:

• PR description template for Triton Engineers
• PR description template for External Contributors

If you already created the PR, please replace this message with one of

• External contribution template
• Internal contribution template

and fill it out.

[greptile-apps]

Greptile Summary

This PR adds a POST /v2/multi_infer endpoint that fans out an array of inference requests across multiple models in parallel, deduplicates identical (model, version, inputs) pairs to avoid redundant work, and aggregates the results into a single JSON response. Several supporting refactors are included: EVBufferAddErrorJson is extracted to a shared header, response macros are moved to http_server_macros.h, evthr_defer is replaced by a retry-capable wrapper, and a new ScheduleInferAsync helper centralises request dispatch.

• POST /v2/multi_infer (src/multi_infer.cc, 1 049 lines): four-phase pipeline — validate slots, deduplicate by fingerprint, fill Triton requests, schedule in parallel; uses MultiInferAggregator (multi-slot) and MultiInferSingleSlotRequest (single-slot fast path) to handle completion callbacks and reply assembly.
• WriteDataToJson change (src/http_server.cc): applies RoundToScorePrecision (6 decimal places) to every FP32 and FP64 output, affecting all /v2/infer endpoints, not just multi_infer — a global precision change for existing clients.
• Dockerfile adds an ODBC-enabled derivative image that installs MySQL Connector/ODBC and replaces only the Triton binary and shared library from a local build.

Confidence Score: 2/5

• Three confirmed correctness bugs need to be fixed before merging: a double-free that crashes the process on compressed single-slot requests when scheduling fails, malformed JSON when evbuffer_pullup returns null in the dedup fanout path, and a global precision change that silently truncates FP64 outputs for all existing inference clients.
• The double-free in the single-slot error path and the global rounding of all FP32/FP64 outputs are both confirmed defects on code paths that are reachable today. The malformed-JSON issue in BuildHttpResponse, while triggered only by OOM, still produces an invalid HTTP response body with no error status. Together they affect both new (multi_infer) and existing (/v2/infer) flows.
• src/multi_infer.cc and src/http_server.cc need the most attention — the double-free, the JSON comma placement, and the global rounding change are all in these two files.

Important Files Changed

Filename	Overview
src/multi_infer.cc	New 1 049-line file implementing POST /v2/multi_infer. Contains a double-free on decompressed body when scheduling fails in the single-slot fast path, malformed JSON output when evbuffer_pullup returns null in the dedup fanout path, and a transaction-policy check that uses model name only (not model+version). The core parallel-fan-out logic and deduplication are structurally sound, but these three correctness issues need to be resolved.
src/http_server.cc	Adds EvthrDeferWithRetry, ScheduleInferAsync, FillMultiInferSlotTritonRequest, and a FP32 fast-path parser. Also applies RoundToScorePrecision to ALL FP32/FP64 outputs in WriteDataToJson — a global behavior change that silently truncates precision for every existing /v2/infer endpoint, including FP64 outputs where 9 significant digits are lost.
src/http_server.h	Adds pause_http_request and register_fini_cancel_hook parameters to InferRequestClass and CreateInferRequest, declares EvthrDeferWithRetry, ScheduleInferAsync, and HandleMultiInfer, and changes FinalizeResponse signature to accept an optional evbuffer* for JSON-only aggregation. Changes look correct and backward-compatible with default arguments.
src/http_error_json.h	Extracts EVBufferAddErrorJson into a shared header using RapidJSON for fast serialization. Clean refactor; removes duplicated implementations from http_server.cc and sagemaker_server.cc.
src/http_server_macros.h	Moves the three HTTP response macros (RETURN_AND_RESPOND_IF_ERR, RETURN_AND_RESPOND_WITH_ERR, RETURN_AND_RESPOND_IF_RESTRICTED) into a shared header so multi_infer.cc can reuse them. Clean extraction, no logic changes.
src/sagemaker_server.cc	Switches from direct evthr_defer to EvthrDeferWithRetry and removes the now-duplicate EVBufferAddErrorJson. Both are mechanical cleanups that improve resilience on high-load evhtp queues.
src/common.h	Adds RoundToScorePrecision (inline, 6 decimal places) and includes cmath. The helper itself is correct, but it is applied globally in http_server.cc WriteDataToJson rather than being scoped to the multi_infer fast-path serialization where it is semantically appropriate.
Dockerfile	New derivative Dockerfile that installs MySQL Connector/ODBC and replaces the tritonserver binary and libtritonserver.so. The ODBC symlink loop is a reasonable workaround; the DEB download uses curl with -fsSL from a trusted MySQL repo and checksums are not verified.

Sequence Diagram

sequenceDiagram
    participant C as HTTP Client
    participant H as HandleMultiInfer
    participant A as MultiInferAggregator
    participant T as Triton Core

    C->>H: "POST /v2/multi_infer {requests:[…]}"
    H->>H: evhtp_request_pause(req)
    H->>H: "Phase 1 — validate slots & policy"
    H->>H: Phase 2 — BuildUniqueInferGroups (dedup)
    H->>A: new MultiInferAggregator(unique_count, fanout)
    loop For each unique infer group
        H->>T: TRITONSERVER_InferenceRequestNew
        H->>T: FillMultiInferSlotTritonRequest
        H->>T: "ScheduleInferAsync (MultiInferShardRequest*)"
    end
    T-->>A: InferResponseComplete (shard u done)
    A->>A: OnUniqueInferDone(u, buf)
    Note over A: When done_count == unique_count
    A->>H: EvthrDeferWithRetry → SendReplyThunk
    H->>H: BuildHttpResponse (assemble JSON)
    H->>C: evhtp_send_reply + evhtp_request_resume

Loading

_{Reviews (4): Last reviewed commit: "Removed unwanted files and restored" | Re-trigger Greptile}

[greptile-apps]

Password embedded verbatim in ODBC connection string

BuildMySqlDriverConnectString embeds dsn_user_password directly as PWD={...}. If this string is ever logged or captured in a crash dump, the plaintext credential is exposed. Audit any future logging paths that might capture this string.