vLLM4j

JVM native binding for vLLM via Java FFM (Foreign Function & Memory API) and jextract.

Embeds CPython in-process, drives vLLM's synchronous LLMEngine directly, and exposes Jinja2 chat-template rendering -- all without HTTP, Ray, or AsyncLLMEngine.

How it works

┌──────────────┐     FFM (Panama)     ┌──────────────┐     Python C API     ┌──────────┐
│   Java 25    │ ──────────────────── │  libpython   │ ──────────────────── │   vLLM   │
│  (your app)  │    jextract bindings │   3.12       │   LLMEngine.step()   │  engine  │
└──────────────┘                      └──────────────┘                      └──────────┘

• No HTTP server -- vLLM runs in the same process as the JVM
• Continuous batching -- multiple requests processed in parallel via VllmIterator
• Token classification -- Java-side FSM detects reasoning (<think>) and tool-call tags in generated text

Targets vLLM 0.31.0 across all backends. The version is pinned in VLLM_VERSION at the top of scripts/setup-venv.sh.

Requirements

Component	Version
Java	25 (with --enable-preview)
Python	3.12
Maven	3.9+
uv	For venv management

Supported backends:

Backend	Platform	Notes
Metal	macOS Apple Silicon	via vllm-metal
CUDA	Linux x86_64	NVIDIA GPU
CPU	Any	Fallback

Quick start

# Build (macOS Apple Silicon + Metal)
JAVA_HOME=/opt/homebrew/opt/openjdk@25/libexec/openjdk.jdk/Contents/Home \
  mvn clean package -P macosx-aarch64,metal -DskipTests

# Run interactive REPL (macOS)
JAVA_HOME=/opt/homebrew/opt/openjdk@25/libexec/openjdk.jdk/Contents/Home \
  $JAVA_HOME/bin/java --enable-preview \
       --enable-native-access=ALL-UNNAMED \
       -Dvllm4j.venv=.venv \
       -jar target/vLLM4j-*.jar \
       --model Qwen/Qwen3-0.6B \
       --reasoning_tags "<think>|</think>"

# Run interactive REPL (Linux / CUDA)
# The build symlinks libpython to a stable path inside the venv
export LD_PRELOAD=$PWD/.venv/lib/libpython3.12.so
export TOKENIZERS_PARALLELISM=false
export VLLM_LOGGING_LEVEL=WARNING
export VLLM_WORKER_MULTIPROC_METHOD=spawn
java -Dvllm4j.venv=.venv \
     -jar target/vLLM4j-*.jar \
     --model Qwen/Qwen3-0.6B \
     --reasoning_tags "<think>|</think>"

---

Usage as a library

Important

vLLM4j is no longer published to Maven Central, and we are working on a new way to distribute it. In the meantime, build it from source and run mvn install with the profiles from Quick start, then depend on it from your local Maven repository:

<dependency>
    <groupId>io.gravitee.vllm</groupId>
    <artifactId>vLLM4j</artifactId>
    <version><!-- the <version> in this repository's pom.xml --></version>
</dependency>

To serve models rather than embed the library, try Gravitee Singularitee locally: an inference server that runs vLLM through vLLM4j (opt-in, Linux/CUDA first). See its Getting started.

Creating an engine

// Minimal — uses defaults (Qwen/Qwen3-0.6B, dtype=auto)
try (var engine = VllmEngine.builder().build()) {
    // ...
}

// Full configuration
try (var engine = VllmEngine.builder()
        .model("Qwen/Qwen3-0.6B")
        .dtype("float16")
        .venvPath(Path.of("/path/to/.venv"))
        .gpuMemoryUtilization(0.85)
        .maxModelLen(4096)
        .maxNumSeqs(32)
        .maxNumBatchedTokens(8192)
        .enforceEager(true)
        .trustRemoteCode(false)
        .quantization("awq")
        .swapSpace(4.0)
        .seed(42)
        .kvCacheDtype("auto")
        .enablePrefixCaching(true)
        .enableChunkedPrefill(true)
        .build()) {
    // ...
}

Chat templates

Use ChatTemplate to render conversations via the model's Jinja2 template:

var template = new ChatTemplate(engine);

String prompt = template.render(List.of(
        ChatMessage.system("You are a helpful assistant."),
        ChatMessage.user("What is the capital of France?")
), true);  // true = add generation prompt

Override the model's template if needed:

var template = new ChatTemplate(engine, "{% for m in messages %}..{% endfor %}");

Blocking generation

The simplest way to get a complete response:

try (var sp = new SamplingParams().temperature(0.0).maxTokens(128)) {
    var request = new VllmRequest("req-1", prompt, sp);
    RequestOutput result = engine.generate(request);

    String text = result.outputs().getFirst().text();
    FinishReason reason = result.outputs().getFirst().finishReason();
    int promptTokens = result.numPromptTokens();
    int generatedTokens = result.numGeneratedTokens();
}

Streaming with VllmIterator

Stream tokens as they're generated:

try (var sp = new SamplingParams().temperature(0.7).maxTokens(256)) {
    var request = new VllmRequest("req-1", prompt, sp);
    var iterator = new VllmIterator(engine);
    iterator.addRequest(request);

    iterator.stream().forEach(output -> {
        System.out.print(output.delta());   // incremental text
        // output.text()         — cumulative text so far
        // output.finished()     — true on last output
        // output.finishReason() — "stop", "length", etc.
        // output.tokenIds()     — all token IDs (only on last output)
        // output.logprobs()     — logprobs (only on last output, if requested)
    });
}

Parallel requests (continuous batching)

Submit multiple requests — vLLM's scheduler interleaves them:

var iterator = new VllmIterator(engine);

try (var sp1 = new SamplingParams().temperature(0.0).maxTokens(64);
     var sp2 = new SamplingParams().temperature(0.0).maxTokens(64);
     var sp3 = new SamplingParams().temperature(0.0).maxTokens(64)) {

    iterator.addRequest(new VllmRequest("france",  promptFrance,  sp1));
    iterator.addRequest(new VllmRequest("england", promptEngland, sp2));
    iterator.addRequest(new VllmRequest("poland",  promptPoland,  sp3));
}

// Stream interleaves outputs from all three requests
Map<String, StringBuilder> outputs = new HashMap<>();
iterator.stream().forEach(output ->
        outputs.computeIfAbsent(output.requestId(), k -> new StringBuilder())
               .append(output.delta()));

SamplingParams — full API

try (var sp = new SamplingParams()
        // Core sampling
        .temperature(0.8)
        .topP(0.95)
        .topK(50)
        .minP(0.05)
        .seed(42L)

        // Length control
        .maxTokens(512)
        .minTokens(10)

        // Repetition control
        .repetitionPenalty(1.1)
        .frequencyPenalty(0.5)
        .presencePenalty(0.3)

        // Stop conditions
        .stop("User:", "\n\n")
        .stopTokenIds(128001, 128009)
        .includeStopStrInOutput(false)

        // Multi-sequence
        .n(3)
        .bestOf(5)

        // Token handling
        .skipSpecialTokens(true)
        .spacesBetweenSpecialTokens(true)
        .truncatePromptTokens(4096)

        // Logprobs (see Logprobs section)
        .logprobs(5)
        .promptLogprobs(1)

        // Guided decoding (see Guided Decoding section)
        .guidedDecoding(GuidedDecodingParams.json(schema))) {

    // use sp...
}

Token classification (reasoning / tool calls)

Classify generated tokens into reasoning, tool-call, and answer regions using a tag-based FSM:

var state = new ConversationState()
        .reasoning("<think>", "</think>")
        .toolCall("<tool_call>", "</tool_call>");

var iterator = new VllmIterator(engine);
iterator.addRequest(request, state);

iterator.stream().forEach(output -> {
    GenerationState segment = output.state();  // REASONING, TOOLS, or ANSWER
    // output.delta() is already marker-free — see "How markers are matched" below

    switch (segment) {
        case REASONING -> System.err.print(output.delta());   // internal reasoning
        case TOOLS     -> handleToolCall(output.delta());      // tool invocation JSON
        case ANSWER    -> System.out.print(output.delta());    // user-facing answer
    }
});

// After generation — token breakdown
ConversationState result = iterator.conversationState("req-1");
System.out.println("Reasoning tokens: " + result.reasoningTokens());
System.out.println("Answer tokens:    " + result.answerTokens());
System.out.println("Tool tokens:      " + result.toolsTokens());
System.out.println("Total output:     " + result.totalOutputTokens());
System.out.println("Total (w/ input): " + result.totalTokens());
System.out.println("Finish reason:    " + result.finishReason());

Markers can have alternatives on either end, for dialects that enter or leave a channel more than one way:

var harmony = new ConversationState()
        .reasoning(
            List.of("<|channel|>analysis<|message|>"),
            List.of("<|end|><|start|>assistant<|channel|>final<|message|>",
                    "<|call|><|start|>assistant<|channel|>final<|message|>",
                    "<|channel|>final<|message|>"))   // the bare header, after a tool call
        .toolCall(
            List.of("<|channel|>commentary to=functions.",
                    "<|channel|>analysis to=functions."),
            List.of("<|call|>"));

How markers are matched

Markers are syntax, not content: a confirmed marker emits nothing, and its tokens are billed to the channel it opened. Streaming delta() is therefore safe to forward verbatim — there is nothing to strip. Four rules follow, each of which is a bug it prevents:

• Matched at the start of a run, longest first. A marker matches only where a run begins — after the previous one resolved, or at the start of generation. A variant you did not configure never matches: it refutes, and the whole header lands in the current channel as visible text.
• A split marker is buffered, never leaked. Text that is a strict prefix of a candidate is withheld (empty delta) until the marker completes or the text diverges; on divergence the buffered text is released into the current channel, in order. ConversationState#flush() releases anything still held when generation stops mid-marker, so no token is lost — call it at the end of a turn.
• A close marker arriving in ANSWER is suppressed too. No span is open for it to close, so it can only be stray syntax — and models emit it: after a tool call the next turn starts fresh and Harmony still prefixes its reply with the final-channel header. List that bare header among the closes, or the reply reads <|channel|>final<|message|>DONE.
• A turn that ends inside the tool channel reports TOOL_CALL. When the close marker is itself an EOS token (<|call|>), generation halts on it and no transition is ever seen mid-stream. An empty span never counts.
• A channel occurs once unless it says otherwise. TagBounds.repeatable defaults to true for TOOLS and false elsewhere — the ChatML rule, and what stops a literal <think> in an answer from re-opening reasoning. Harmony breaks it: one generation can run analysis, return to the final channel, then open a commentary preamble, and a channel that cannot re-open stops matching. Use reasoning(opens, closes, true) there, or the second header reaches the client as raw text with its tokens billed as answer.

Channels chain rather than nest: a tool call opening straight out of a reasoning span closes it implicitly. initialize(promptTokenCount, prompt) also seeds the starting state from a prompt that left a span open — a chat template ending in <think> — since the model never re-emits that opening marker.

Tool-use (multi-turn)

Define tools in OpenAI format, render them into the chat template, and handle multi-turn tool-call conversations:

// 1. Define tools
var weatherTool = Tool.function(
        "get_weather",
        "Get the current weather for a location",
        Map.of(
                "type", "object",
                "properties", Map.of(
                        "location", Map.of("type", "string",
                                "description", "City name, e.g. 'Paris'"),
                        "unit", Map.of("type", "string",
                                "enum", List.of("celsius", "fahrenheit"))
                ),
                "required", List.of("location")
        ));

var tools = List.of(weatherTool);

// 2. Render prompt with tools
var template = new ChatTemplate(engine);
var messages = new ArrayList<>(List.of(
        ChatMessage.system("You are a helpful assistant with tool access."),
        ChatMessage.user("What's the weather in Paris?")
));

String prompt = template.render(messages, tools, true);

// 3. Generate — model emits tool call
try (var sp = new SamplingParams().temperature(0.0).maxTokens(256)) {
    RequestOutput result = engine.generate(new VllmRequest("tc-1", prompt, sp));
    String response = result.outputs().getFirst().text();

    // 4. Parse tool call from response and build tool-call message
    var toolCall = ToolCall.function("call_001", "get_weather", "{\"location\": \"Paris\"}");
    messages.add(ChatMessage.assistantWithToolCalls(null, List.of(toolCall)));

    // 5. Add tool result
    messages.add(ChatMessage.toolResult(
            "{\"temperature\": 18, \"unit\": \"celsius\", \"condition\": \"cloudy\"}",
            "call_001",   // tool_call_id
            "get_weather" // function name
    ));

    // 6. Re-render and generate final answer
    prompt = template.render(messages, tools, true);
    result = engine.generate(new VllmRequest("tc-2", prompt, sp));
    System.out.println(result.outputs().getFirst().text());
}

Guided decoding

Constrain generation output to match a schema, regex, grammar, or choice list:

// JSON schema
var jsonGuide = GuidedDecodingParams.json("""
        {
          "type": "object",
          "properties": {
            "name":  {"type": "string"},
            "age":   {"type": "integer"},
            "email": {"type": "string", "format": "email"}
          },
          "required": ["name", "age"]
        }
        """);

// Regex pattern
var regexGuide = GuidedDecodingParams.regex("[A-Z]{2}-\\d{4}");

// Structural tag — constrain ONLY what a trigger opens, leaving prose free.
// A whole-request schema would force the model to answer in JSON even when it is
// just talking; this arms the schema at the tool-call header and releases it at
// "end", which is what makes constrained tool arguments possible.
var toolGuide = GuidedDecodingParams.structuralTag(structuralTagJson);
// structuralTagJson:
// {"structures": [{"begin": "<|channel|>commentary to=functions.write<|constrain|>json<|message|>",
//                  "schema": { ...the tool's parameter schema... },
//                  "end":   "<|call|>"}],
//  "triggers":   ["<|channel|>commentary to=functions."]}

// Choice from list
var choiceGuide = GuidedDecodingParams.choice(List.of("positive", "negative", "neutral"));

// EBNF grammar
var grammarGuide = GuidedDecodingParams.grammar("root ::= 'yes' | 'no'");

// Freeform JSON object (no specific schema)
var jsonObjectGuide = GuidedDecodingParams.jsonObject();

// Use with SamplingParams
try (var sp = new SamplingParams()
        .temperature(0.0)
        .maxTokens(256)
        .guidedDecoding(jsonGuide)) {

    RequestOutput result = engine.generate(new VllmRequest("guided-1", prompt, sp));
    String json = result.outputs().getFirst().text();  // guaranteed valid JSON
}

Logprobs

Request log-probabilities for generated and/or prompt tokens:

try (var sp = new SamplingParams()
        .temperature(0.0)
        .maxTokens(64)
        .logprobs(5)          // top-5 logprobs per generated token
        .promptLogprobs(1)) { // logprobs for prompt tokens

    RequestOutput result = engine.generate(new VllmRequest("lp-1", prompt, sp));

    // Generated token logprobs
    CompletionOutput completion = result.outputs().getFirst();
    for (Map<Integer, LogprobEntry> step : completion.logprobs()) {
        step.values().forEach(entry ->
                System.out.printf("  token=%s id=%d logprob=%.4f rank=%d%n",
                        entry.decodedToken(), entry.tokenId(),
                        entry.logprob(), entry.rank()));
    }

    // Prompt token logprobs
    for (Map<Integer, LogprobEntry> step : result.promptLogprobs()) {
        step.values().forEach(entry ->
                System.out.printf("  prompt token=%s logprob=%.4f%n",
                        entry.decodedToken(), entry.logprob()));
    }
}

To score a fixed set of tokens (classifier labels, A/B/C option codes), ask for exactly those with logprobTokenIds(...) instead of the top N; they are reported even when they are unlikely:

List<Integer> labels = List.of(engine.encode("A", false).getFirst(), engine.encode("B", false).getFirst());
try (var sp = new SamplingParams().temperature(0.0).maxTokens(1).logprobTokenIds(labels)) {
    Map<Integer, LogprobEntry> next = engine.generate(new VllmRequest("score-1", prompt, sp))
        .outputs().getFirst().logprobs().getFirst();   // one entry per label (plus the sampled token)
}

When streaming via VllmIterator with logprobs requested, each output carries the token ids generated in that step and their logprobs, one map per token id; join the steps for the whole sequence. Without logprobs, steps carry no token ids:

try (var sp = new SamplingParams().temperature(0.0).maxTokens(64).logprobs(5)) {
    var iterator = new VllmIterator(engine);
    iterator.addRequest(new VllmRequest("lp-stream", prompt, sp));

    iterator.stream().forEach(output -> {
        System.out.print(output.delta());
        List<Integer> tokenIds = output.tokenIds();                    // this step's tokens
        List<Map<Integer, LogprobEntry>> logprobs = output.logprobs(); // one map per token id
    });
}

Note: logprobs is null unless SamplingParams.logprobs(n) is set with n >= 1.

Tokenizer

Access the model's tokenizer directly:

// Encode text to token IDs
List<Integer> tokens = engine.encode("Hello, world!");

// Decode token IDs back to text
String text = engine.decode(tokens);

// Get vocabulary size
int vocabSize = engine.vocabSize();

Encoding data as data

vLLM tokenizes a text prompt with the tokenizer's defaults, which match added tokens anywhere in the string. That is right for a chat template's own markup and wrong for everything interpolated into it: a literal <|channel|> in a user message, a tool argument, or a tool result is promoted to the real control token (id 200005 for gpt-oss). The content is silently rewritten, and untrusted output — a file, a web page, a tool result — can inject conversation structure.

Encode each segment for what it is, then submit the ids:

List<Integer> ids = new ArrayList<>();
ids.addAll(engine.encode("<|im_start|>user\n", true));    // markup: specials parsed
ids.addAll(engine.encode(untrustedToolResult, false));     // data: specials split
ids.addAll(engine.encode("<|im_end|>\n<|im_start|>assistant\n", true));

engine.addRequest(VllmRequest.ofTokens("req-1", renderedPrompt, ids, sp));

encode(text, false) asks the tokenizer for split_special_tokens, so the marker encodes as ordinary text and round-trips through decode unchanged. Segments never add special tokens of their own (add_special_tokens=False), because a segment is a fragment of one prompt — concatenating them is the prompt. ofTokens still carries the rendered text, which is what seeds the generation state and what gets logged.

Multimodal (image and audio)

Send images and audio alongside text prompts (requires a vision/audio model):

// From byte arrays
var mmData = new MultiModalData()
        .addImage(imageBytes)
        .addAudio(audioBytes);

// From file paths
var mmData = new MultiModalData()
        .addImage(Path.of("photo.jpg"))
        .addAudio(Path.of("speech.wav"));

// Use in a request
try (var sp = new SamplingParams().temperature(0.0).maxTokens(256)) {
    var request = new VllmRequest("mm-1", prompt, sp, mmData);
    RequestOutput result = engine.generate(request);
    System.out.println(result.outputs().getFirst().text());
}

Engine stats

Query live engine state:

EngineStats stats = engine.getStats();
System.out.println("Model:        " + stats.model());
System.out.println("Dtype:        " + stats.dtype());
System.out.println("Max len:      " + stats.maxModelLen());
System.out.println("In-flight:    " + stats.numUnfinishedRequests());

Request metrics

Timing information returned with each completed request:

RequestOutput result = engine.generate(request);
RequestMetrics metrics = result.metrics();

System.out.printf("TTFT:       %.1f ms%n", metrics.ttftMs());
System.out.printf("Total time: %.1f ms%n", metrics.totalTimeMs());
System.out.printf("Queue time: %.3f s%n",  metrics.timeInQueue());

Request priority

Prioritize requests in the scheduler:

// Higher priority value = higher scheduling priority
var urgentRequest = new VllmRequest("urgent-1", prompt, sp, null, 10);
var normalRequest = new VllmRequest("normal-1", prompt, sp, null, 1);

var iterator = new VllmIterator(engine);
iterator.addRequest(normalRequest);
iterator.addRequest(urgentRequest);  // scheduled first despite being added second

Decision models (vLLM Semantic Router)

Decision models answer typed questions about a state in one forward pass, without generating tokens: pick an option (choice), give the probability that a condition holds (noul), or place it on an ordered rubric (score). vLLM's LLMEngine does not run them; vLLM Semantic Router serves them with its own model runtime, vllm_srun. VllmDecider drives that runtime in the embedded CPython, the way VllmEngine drives LLMEngine: no HTTP server, no extra process.

It needs vllm-srun in the venv, installed on the engine's torch:

VLLM4J_DECISION=1 mvn clean package -P macosx-aarch64,metal -DskipTests   # or: scripts/setup-venv.sh -b metal -r

try (var decider = VllmDecider.builder()
        .model("vllm-sr/Decision-2.0-Kai-0.6B")   // any built-in Decision 1.0, 2.0 or 3.0 model
        .device("auto")                            // cpu, cuda, mps, rocm...
        .build()) {                                // loads, verifies, checks golden answers

    Map<String, String> intents = new LinkedHashMap<>();   // options keep their order
    intents.put("billing", "Payment or invoice issue");
    intents.put("account", "Login or access issue");

    DecisionResponse response = decider.decide(
        DecisionRequest.builder(Map.of("request", "I was charged twice this month."))
            .choice("intent", "What does the user want?", intents)
            .noul("upset", "Is the user upset?")
            .score("urgency", "How urgent is it?", List.of("Not urgent", "Somewhat urgent", "Urgent"))
            .build());

    var intent = (Answer.Choice) response.answer("intent");   // choice, probabilities, confidence
    var upset = (Answer.Noul) response.answer("upset");       // probability
    var urgency = (Answer.Score) response.answer("urgency");  // score, probabilities, legend
}

• Answers are the runtime's POST /v1/decisions answers. A question that cannot be answered comes back as an Answer.Failed while its siblings are answered; a request refused as a whole throws DecisionException with the contract's status and code.
• Decision 3.0 (vllm-sr/d3, d3-flash, d3-mini, d3-nano, d3-lite, d3-edge) also reads images and videos: add them with .image("data:image/png;base64,...") and .video("data:video/mp4;base64,..."). On Apple Silicon run it with device("mps"): the runtime's CPU golden answers were recorded on x86, and the BF16 CPU path here rounds outside their tolerance, so the model refuses to load on cpu.
• Raw bodies: decide(Map<String, Object>) takes and returns the /v1/decisions JSON as Java maps, for what the typed API leaves out (states, set and span questions).
• Concurrency: decide is thread-safe and releases the GIL while it waits, so concurrent calls are batched by the runtime's scheduler.
• Closing: close() refuses new calls with IllegalStateException. A call already waiting gets its answer if the runtime gives it while stopping, or a DecisionException (503 not_ready); close() returns once every such call has returned.
• Next to an engine: a decider and a VllmEngine share one interpreter, so two runtime settings reach the engine too:
  • Loading a model freezes the Python heap (gc.freeze()): reference cycles among objects that existed then, the engine's included, are not collected until the decider is closed, which unfreezes it.
  • In a container whose CPU quota is below PyTorch's thread count, the runtime lowers torch.set_num_threads for the whole process. Set threads(...) on the builder, or OMP_NUM_THREADS, to keep your own.
  • On a shared GPU, cap the decider with memoryBudgetGib(...) and the engine with gpuMemoryUtilization(...).

---

Project structure

vLLM4j/
├── pom.xml                           # Two-axis Maven profiles: OS/arch x backend
├── .mvn/jvm.config                   # --enable-preview
├── scripts/
│   ├── download-jextract.sh          # Downloads jextract EA build
│   ├── setup-venv.sh                 # Creates .venv, installs vLLM per backend
│   └── generate-sources.sh           # Resolves Python.h, runs jextract
└── src/
    ├── main/java/io/gravitee/vllm/
    │   ├── Main.java                 # Interactive REPL with ANSI-styled output
    │   ├── Freeable.java             # free() + isFree() interface
    │   ├── platform/                 # OS, arch, backend detection
    │   ├── runtime/                  # CPython lifecycle (PythonRuntime, PythonRef)
    │   ├── binding/                  # FFM helpers (PythonCall, PythonTypes, PythonErrors)
    │   ├── engine/                   # VllmEngine, SamplingParams, RequestOutput, ...
    │   ├── decision/                 # VllmDecider (vLLM Semantic Router decision models)
    │   ├── iterator/                 # VllmIterator (continuous batching), VllmOutput
    │   ├── state/                    # ConversationState, StateEvaluation FSM, TokenTracking
    │   └── template/                 # ChatTemplate (Jinja2), ChatMessage, Tool
    └── test/java/io/gravitee/vllm/
        ├── decision/                 # Decision request/response mapping; VllmDecider integration
        ├── engine/                   # Unit tests for records, FinishReason, RequestMetrics
        ├── iterator/                 # Integration tests (real engine, parallel capitals)
        ├── platform/                 # Unit tests for platform detection
        ├── state/                    # Unit tests for FSM, token tracking, ConversationState
        └── template/                 # Unit tests for ChatMessage, Tool, ToolFunction

Tests

# Unit tests only (fast, no CPython needed)
mvn test -P macosx-aarch64,metal
# 162 tests, ~4s

# Integration tests only (loads real model)
# Each test class runs in its own JVM fork (reuseForks=false)
# so GPU memory is fully reclaimed between classes.
mvn test -P integration,macosx-aarch64,metal

# Integration tests on Linux + CUDA
# Requires: LD_PRELOAD (see Linux / CUDA notes). .venv/bin + CUDA_HOME are
# wired up automatically. On pre-Ampere GPUs (sm < 80) the profile sets
# VLLM4J_ATTENTION_BACKEND=TRITON_ATTN; see Linux / CUDA notes.
mvn test -P integration,linux-x86_64,cuda

# Decision model tests (vllm-srun in the venv, Decision-2.0-Kai-0.6B on CPU)
mvn test -P decision-integration,macosx-aarch64

Linux / CUDA notes

/dev/shm permissions

vLLM 0.16+ creates a multiprocessing.Lock() during executor initialisation (used by the shared-memory multimodal cache). On Linux this requires write access to /dev/shm. If /dev/shm is owned by root with mode 755 (common in restricted containers) the engine will fail with:

PermissionError: [Errno 13] Permission denied
_multiprocessing.SemLock(...)

Fix on your dev machine (survives until next reboot):

sudo chmod 1777 /dev/shm

Fix permanently (add to /etc/fstab):

tmpfs /dev/shm tmpfs defaults,size=8g 0 0

The linux-x86_64 Maven profile runs sudo chmod 1777 /dev/shm automatically during the validate phase, so mvn will prompt for your password once per session if /dev/shm has wrong permissions.

GPU memory and LoRA on small GPUs

CPython can only be initialized once per process, and GPU memory held by an engine is not released until the process exits. To avoid memory contention, the integration test profile uses reuseForks=false so that each test class runs in its own JVM fork. The _exit(0) shutdown hook terminates each fork cleanly and the OS reclaims all GPU memory before the next class starts.

This means each test class creates its own engine with the settings it needs:

• VllmIteratorTest — base model only (no LoRA), gpuMemoryUtilization(0.85)
• LoraTest — LoRA enabled, same GPU utilization (LoRA infrastructure takes its share, but the full GPU is available since the prior fork exited)

On cards with limited VRAM (8 GiB or less), both engines are configured with:

• enforceEager(true) — skips CUDA graph capture, which pre-allocates dummy LoRA adapters and OOMs on small cards
• maxModelLen(4096) (or 2048 for LoRA) — Qwen3-0.6B defaults to 40960, but the KV cache for that length doesn't fit in limited VRAM. Tests only need short contexts.
• maxNumSeqs(4) — default 128 causes sampler warm-up OOM on small cards; 4 is enough for the parallel/interleaving tests.

The /no_think suffix is appended to system prompts so Qwen3 skips its internal reasoning chain, producing shorter and faster test outputs.

ninja and the CUDA build toolchain

vLLM's flashinfer backend JIT-compiles CUDA kernels at first use via ninja and nvcc. vLLM4j wires this up automatically at interpreter startup (PythonRuntime.configureBuildToolchain): it prepends .venv/bin (where setup-venv.sh installs ninja) to PATH, and when the venv bundles an NVIDIA CUDA toolkit it exports CUDA_HOME pointing at it. No manual PATH export is required.

setup-venv.sh also pins the CUDA build-toolchain wheels (nvidia-cuda-nvcc, nvidia-nvvm, nvidia-cuda-crt, nvidia-cuda-cccl) to the CUDA version torch was built against. vLLM's CUDA wheel pins the CUDA runtime but lets these build wheels float to newer patches, which leaves flashinfer unable to compile its kernels (mismatched compiler/headers/ptxas); pinning them keeps the toolkit coherent.

Environment variables

The following variables are set by the example launch scripts and are worth knowing when running vLLM4j outside of them:

Variable	Recommended value	Why
LD_PRELOAD	/path/to/.venv/lib/libpython3.12.so (symlink created by the build)	Prevents undefined symbol: PyTuple_Type from Python extension modules (e.g. _ctypes) that are loaded after the JVM has already resolved libc symbols. Required on Linux.
VLLM4J_ATTENTION_BACKEND	TRITON_ATTN	vLLM 0.31.0 no longer reads an attention-backend env var, so vLLM4j forwards this one (or the -Dvllm4j.attentionBackend system property) to the attention_backend engine arg. On GPUs with compute capability < 8.0 (e.g. RTX 2070 = sm75) vLLM auto-selects FlashInfer, whose paged-prefill kernel fails at runtime (BatchPrefillWithPagedKVCache ... invalid argument); set TRITON_ATTN on those cards. Honored on the CUDA backend only.
TOKENIZERS_PARALLELISM	false	Suppresses the HuggingFace tokenizers deadlock warning that is emitted when the tokenizer is used in a forked subprocess.
VLLM_LOGGING_LEVEL	WARNING	vLLM's Python side logs at INFO by default, producing verbose scheduler and profiling output on every request. WARNING keeps the Java log clean.
VLLM_WORKER_MULTIPROC_METHOD	spawn	Ensures GPU worker processes are started with spawn rather than fork. Forking after torch.cuda.init() causes NCCL deadlocks in multi-GPU setups.

One-liner to export all of them before launching with java -jar:

# vLLM4j's build symlinks libpython into the venv (.venv/lib/libpython3.12.so).
# If you built the venv yourself, point LD_PRELOAD at the base interpreter's
# libpython instead.
export LD_PRELOAD=/path/to/.venv/lib/libpython3.12.so
export TOKENIZERS_PARALLELISM=false
export VLLM_LOGGING_LEVEL=WARNING
export VLLM_WORKER_MULTIPROC_METHOD=spawn

java -Dvllm4j.venv=/path/to/.venv -jar your-app.jar

For GPUs with compute capability < 8.0 (check with nvidia-smi --query-gpu=compute_cap --format=csv,noheader), also add:

export VLLM4J_ATTENTION_BACKEND=TRITON_ATTN

Maven profiles

Two orthogonal axes:

OS/arch (pick one):

• macosx-aarch64 -- macOS Apple Silicon
• macosx-x86_64 -- macOS Intel
• linux-x86_64 -- Linux

Backend (pick one):

• metal -- Apple Silicon Metal/MLX GPU
• cuda -- NVIDIA CUDA GPU
• cpu -- CPU-only fallback

Test control:

• integration -- runs @Tag("integration") tests (excluded by default). Uses reuseForks=false so each test class gets its own JVM fork with a clean GPU.
• decision-integration -- runs @Tag("decision-integration") tests (excluded by default): VllmDecider against a real decision model. Needs vllm-srun in the venv.

Example: mvn clean package -P macosx-aarch64,metal

CLI options

Model:
  --model <id>                HuggingFace model id (default: Qwen/Qwen3-0.6B)
  --dtype <type>              Torch dtype: auto, float16, float32 (default: auto)
  --venv <path>               Path to Python venv (default: .venv)

Prompt:
  --system <msg>              System prompt
  --chat_template <jinja2>    Jinja2 chat template string (default: from model)

Sampling:
  --temperature <float>       Sampling temperature (default: 0.7)
  --max_tokens <int>          Max tokens to generate (default: 512)
  --top_p <float>             Top-P nucleus sampling (default: 0.9)
  --top_k <int>               Top-K sampling (default: -1)
  --repetition_penalty <f>    Repetition penalty (default: 1.0)
  --frequency_penalty <f>     Frequency penalty (default: 0.0)
  --presence_penalty <f>      Presence penalty (default: 0.0)

Token Classification:
  --reasoning_tags "<open>|<close>"   e.g. "<think>|</think>"
  --tool_tags "<open>|<close>"        e.g. "<tool_call>|</tool_call>"

Design decisions

• No PyConfig -- uses setenv(PYTHONHOME) + Py_InitializeEx(0) (jextract can't generate PyConfig)
• GIL acquired per call -- each FFI crossing takes and releases the GIL (GIL.acquire() in a try-with-resources), so the JVM side never holds it between calls
• VLLM_ENABLE_V1_MULTIPROCESSING=0 -- in-process mode required for embedded CPython
• Delta streaming -- the iterator flips the request's SamplingParams to RequestOutputKind.DELTA, so vLLM hands back only the new fragment each step; the cumulative text is accumulated Java-side and materialised on the final output only
• Per-step tokenIds/logprobs -- a streamed VllmOutput carries only the token ids and logprobs of its own step, and only when the request asked for logprobs, so nothing accumulates across steps
• State classification is Java-side -- a tag-based FSM classifies generated text (following llamaj.cpp's pattern)
• Tools are template-level -- passed as a tools variable to the Jinja2 chat template
• Decision models run on vllm_srun, not LLMEngine -- vLLM cannot load their heads; VllmDecider drives vLLM Semantic Router's runtime in the same interpreter instead. Its exit_on_device_error is forced off, as it would os._exit the JVM on a device failure; the model reports degraded instead
• /dev/shm fix on Linux -- vLLM 0.16+ uses multiprocessing.Lock() which requires /dev/shm write access. The linux-x86_64 Maven profile runs sudo chmod 1777 /dev/shm during validate to fix misconfigured permissions

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.