Repository navigation
Probe manual nodes for llmman on 17434 as an Ollama-API engine - #15
Open
ericcurtin wants to merge 1 commit into
Open
ericcurtin wants to merge 1 commit into
ericcurtin wants to merge 1 commit into
Conversation
Collaborator
|
Thanks for opening this up and for your interest in PAIR! We're going to look into this and decide whether or not we want to bring this in. |
Author
@Noah-Tervalon-Nvidia cool, any feedback let me know |
llmman (https://github.com/llmmanorg/llmman) serves the Ollama API on 17434, but manual nodes were only probed for Ollama on 11434, so a node running llmman showed no inference engine. Retry the probe on 17434 and report the answering port in ollama_port; the broker and ollama-proxy already honour that field, so nothing else changes. Ollama wins when both answer. Signed-off-by: Eric Curtin <eric.curtin@docker.com>
Author
|
Rebased onto develop. |
Author
|
@Noah-Tervalon-Nvidia @ckelseynv Rebased and added the release intent block, PTAL. Thank you! |
Author
|
@nv-pgoode @kjlubick PTAL when you get a chance. Thank you! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Changelog title
Probe llmman on port 17434 for manual nodes
Changelog body
Bumps
llmman is a local model runner that serves the Ollama API on port 17434. Manual nodes were probed for Ollama on
11434only, so a device running llmman was reported as having no inference engine even thoughollama-proxycan route to it unchanged.manager.go:probeOllamaAPItries11434then17434and reports the answering port inollama_port; Ollama wins when both answer.11434literals become anollamaPortconstant.ollama_portis already forwarded toollama-proxy, which already dials it. llmman is not added as a new engine type, per CONTRIBUTING.manager_test.go:TestProbeOllamaAPIFallsBackToLLMMancovers Ollama-wins, llmman-only and neither.Scope
In: probing llmman on its default port, docs, test, version bump. Out: llmman as a managed local engine, mDNS-discovered nodes, non-default ports.
Validation
Testing:
gofmt -l .,go vet ./...,go test ./... -count=1inservices/nvpair-manual-nodesandservices/nvpair-ui-broker- clean (macOS arm64, Go 1.26). Not run against a live llmman/Ollama through the full broker/proxy path.Risk
IPC shapes unchanged;
ollama_portmay now be17434. One extra plain-HTTP probe per node per cycle when11434does not answer.Checklist
git commit -s), certifying the Developer Certificate of Origin.services/versions.jsonis written by automation, not edited by hand.