Skip to content

Feat: Add model-compatibility.json registry for self-hostable models #329

Description

@fuleinist

Problem

Issue #235 asks which smaller self-hostable models work with browser-harness (e.g. qwen3.6 35b-a3b). There is no documented compatibility matrix. Users deploying on limited hardware or air-gapped environments have no way to know what will actually work before spending hours on setup.

Proposal

Add a model-compatibility.json file at the root of the repository that lists tested models with compatibility metadata:

[
  {
    "model": "qwen3.6 35b-a3b",
    "provider": "ollama",
    "status": "verified",
    "notes": "Works well for simple click/type tasks. Struggles with complex multi-step flows.",
    "last_tested": "2026-05-01"
  },
  {
    "model": "mistral-small",
    "provider": "anyscale",
    "status": "verified",
    "notes": "Good balance of speed and capability for headless deployments.",
    "last_tested": "2026-05-01"
  }
]

Status values: verified | works | unknown | broken

Also add a lightweight CLI command:

browser-harness models list          # show all models
browser-harness models list --min-size 35b  # filter by parameter size
browser-harness models list --status verified  # only verified

And a --model-info <name> flag to get details on a specific model.

Alternatives considered

  • Wiki/docs page: harder to keep in sync, no machine-readable format
  • Just answering Supported/verified models lists #235 inline: does not scale, the question will keep being asked
  • Rely on user reports: no structured record of what has been tested

Before submitting

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions