Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ for the identity and profile model.

## Vector Search

msgvault can search your archive semantically using vector embeddings in addition to the default FTS5 keyword search. Point it at a self-hosted OpenAI-compatible embedding endpoint (Ollama, llama.cpp, LM Studio) and three surfaces accept either pure semantic search or BM25+vector fused via Reciprocal Rank Fusion:
msgvault can search your archive semantically using vector embeddings in addition to the default FTS5 keyword search. Point it at a self-hosted OpenAI-compatible embedding endpoint (Ollama, llama.cpp, LM Studio) or a hosted gateway such as [OrcaRouter](https://www.orcarouter.ai), and three surfaces accept either pure semantic search or BM25+vector fused via Reciprocal Rank Fusion:

- **CLI:** `msgvault search "..." --mode vector` or `--mode hybrid`
- **HTTP:** `GET /api/v1/search?q=...&mode=vector` or `mode=hybrid`
Expand Down
7 changes: 7 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,13 @@ All notable changes to msgvault, grouped by release.
empty one, and the daemon marks vector search stale when the configured
accounts resolve to a different source set than it was started with.

- Named [OrcaRouter](https://www.orcarouter.ai) provider for the embedding
and people-sweep inference surfaces: `[vector.embeddings] api_format =
"orcarouter"` and `[people.sweep.provider] kind = "orcarouter"` each fill
in the gateway endpoint, model, dimension, and `ORCAROUTER_API_KEY`
environment variable so a minimal config block selects the gateway by
name.

**Bug fixes**

- Everything and Files now page narrow analytical metadata before enriching
Expand Down
28 changes: 24 additions & 4 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -682,10 +682,11 @@ External OpenAI-compatible embedding endpoint used to convert message text into

| Key | Default | Description |
|---|---|---|
| `endpoint` | (required) | HTTP(S) base URL for an OpenAI-compatible embeddings API. msgvault appends `/embeddings` (for example, set `http://localhost:11434/v1`, not `.../embeddings`). |
| `model` | (required) | Model name to pass in each request (e.g., `nomic-embed-text`). |
| `dimension` | (required) | Vector dimension. Must match the model's output dimension. |
| `api_key_env` | — | Name of an environment variable containing the API key. Omit for anonymous endpoints. |
| `api_format` | `openai` | Request contract: `openai` (any OpenAI-compatible endpoint), `orcarouter` ([OrcaRouter](https://www.orcarouter.ai) gateway), or `voyage-contextual` (Voyage contextualized embeddings). Selecting `orcarouter` fills in the gateway defaults below for any omitted endpoint/model/dimension/api_key_env. |
| `endpoint` | (required) | HTTP(S) base URL for an OpenAI-compatible embeddings API. msgvault appends `/embeddings` (for example, set `http://localhost:11434/v1`, not `.../embeddings`). With `api_format = "orcarouter"` defaults to `https://api.orcarouter.ai/v1`. |
| `model` | (required) | Model name to pass in each request (e.g., `nomic-embed-text`). With `api_format = "orcarouter"` defaults to `openai/text-embedding-3-small`. |
| `dimension` | (required) | Vector dimension. Must match the model's output dimension. With `api_format = "orcarouter"` defaults to `1536`. |
| `api_key_env` | — | Name of an environment variable containing the API key. With `api_format = "orcarouter"` defaults to `ORCAROUTER_API_KEY`. Omit for anonymous endpoints. |
| `batch_size` | `32` | Embedding inputs per HTTP call. Long messages can contribute multiple chunk inputs. |
| `timeout` | `30s` | Per-request timeout. |
| `max_retries` | `3` | Retries per batch on transient failures. |
Expand Down Expand Up @@ -758,6 +759,25 @@ Optional background scheduling for the embed worker inside `msgvault serve`. Emp
| `cron` | — | 5-field cron expression. Empty string disables the standalone cron. |
| `run_after_sync` | `false` | When `true`, an embed pass runs after every successful scheduled sync. |

### `[people.sweep]`

Consent-gated, model-backed maintenance of durable person profiles. Disabled by default. When enabled, every outbound request requires an exact, previously granted consent for the effective policy fingerprint, and the provider boundary is disclosed by `msgvault person provider`. The provider speaks the OpenAI-compatible Chat Completions protocol.

| Key | Default | Description |
|---|---|---|
| `enabled` | `false` | Enable the people-sweep inference provider. Safe default: disabled. |
| `provider.kind` | `openai_compatible` | Provider contract. `openai_compatible` (any OpenAI-compatible endpoint) or `orcarouter` ([OrcaRouter](https://www.orcarouter.ai) gateway). Selecting `orcarouter` fills in the gateway defaults below for any omitted endpoint/model/api_key_env. |
| `provider.endpoint` | `https://api.openai.com/v1` | HTTP(S) base URL. With `provider.kind = "orcarouter"` defaults to `https://api.orcarouter.ai/v1`. |
| `provider.model` | (required) | Model name. With `provider.kind = "orcarouter"` defaults to `orcarouter/auto`. |
| `provider.api_key_env` | `OPENAI_API_KEY` | Name of an environment variable containing the API key. With `provider.kind = "orcarouter"` defaults to `ORCAROUTER_API_KEY`. Anonymous loopback mode omits it. |
| `provider.allow_anonymous` | `false` | Permit anonymous inference to a loopback endpoint only. |
| `provider.retention_posture` | (required) | Confirmed provider retention posture. |
| `provider.training_posture` | (required) | Confirmed provider training posture. |
| `provider.allowed_sources` | (required) | Source classes the provider may receive, e.g. `conversation_text`, `meeting_text`. |
| `provider.source_since` / `provider.source_until` | — | Date range for consented source material. |
| `provider.allow_sensitive` | `false` | Allow sensitive source material. |
| `provider.request_timeout` | `1m` | Per-request timeout. |

## Overriding the Home Directory

By default, msgvault stores everything under `~/.msgvault` (macOS/Linux) or `C:\Users\<you>\.msgvault` (Windows). To use a different location, you have two options:
Expand Down
14 changes: 14 additions & 0 deletions docs/usage/vector-search.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,20 @@ run_after_sync = true # run a pass after every successful sch
message_types = ["teams"]
```

For a hosted gateway, [OrcaRouter](https://www.orcarouter.ai) serves the same
OpenAI-compatible `/v1/embeddings` contract. Selecting `api_format =
"orcarouter"` fills in the endpoint, model, dimension, and key environment
variable so the block can stay minimal:

```toml
[vector]
enabled = true
backend = "sqlite-vec"

[vector.embeddings]
api_format = "orcarouter" # fills in endpoint/model/dimension/api_key_env
```

The `[vector]` section only takes effect when `enabled = true` **and**
the binary was built with the needed vector backend. If either is
missing, msgvault behaves as before. Disabled vector search returns
Expand Down
2 changes: 1 addition & 1 deletion internal/api/settings.go
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ var settingsCatalog = []settingDefinition{
stringSetting("vector.backend", "search", []string{"sqlite-vec", "pgvector"}, func(c *config.Config) string { return c.Vector.Backend }),
stringSetting("vector.db_path", "search", nil, func(c *config.Config) string { return c.Vector.DBPath }),
boolSetting("vector.skip_extension_create", "search", func(c *config.Config) bool { return c.Vector.SkipExtensionCreate }),
stringSetting("vector.embeddings.api_format", "search", []string{"openai", "voyage-contextual"}, func(c *config.Config) string {
stringSetting("vector.embeddings.api_format", "search", []string{"openai", "orcarouter", "voyage-contextual"}, func(c *config.Config) string {
return string(c.Vector.Embeddings.EffectiveAPIFormat())
}),
testableStringSetting("vector.embeddings.endpoint", "search", func(c *config.Config) string { return c.Vector.Embeddings.Endpoint }),
Expand Down
2 changes: 1 addition & 1 deletion internal/api/settings_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ func TestGetSettingsUsesAllowlistETagAndSecretStates(t *testing.T) {
require.NotNil(byKey["vector.embeddings.api_format"].Value)
require.NotNil(byKey["vector.embeddings.api_format"].Value.String)
assert.Equal("openai", *byKey["vector.embeddings.api_format"].Value.String)
assert.Equal([]string{"openai", "voyage-contextual"}, byKey["vector.embeddings.api_format"].Options)
assert.Equal([]string{"openai", "orcarouter", "voyage-contextual"}, byKey["vector.embeddings.api_format"].Options)
require.NotNil(byKey["server.trusted_proxies"].Value)
assert.NotNil(byKey["server.trusted_proxies"].Value.Strings)
assert.NotContains(byKey, "unsupported.private_value")
Expand Down
21 changes: 21 additions & 0 deletions internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -764,6 +764,27 @@ func decodeConfig(cfg *Config, path string, explicit, homeOverride bool, content
// explicitly configured key remains visible to validation and is rejected.
cfg.People.Sweep.Provider.APIKeyEnv = ""
}
if cfg.People.Sweep.Provider.Kind == peoplesweep.ProviderOrcaRouter {
// The pre-decode defaults fill in the OpenAI-compatible
// endpoint/model/key before the TOML file is read. When the file
// names the OrcaRouter gateway without spelling out those fields,
// clear the unset ones so the named provider's defaults apply
// instead of silently retaining the OpenAI defaults.
if !metadata.IsDefined("people", "sweep", "provider", "endpoint") {
cfg.People.Sweep.Provider.Endpoint = ""
}
if !metadata.IsDefined("people", "sweep", "provider", "model") {
cfg.People.Sweep.Provider.Model = ""
}
if !metadata.IsDefined("people", "sweep", "provider", "api_key_env") {
cfg.People.Sweep.Provider.APIKeyEnv = ""
}
}
// Re-apply the people-sweep provider defaults over any zero-valued
// fields that survived decode. A config that names the provider
// (`kind = "orcarouter"`) without spelling out endpoint/model/key gets
// those filled here, mirroring the vector re-apply below.
cfg.People.Sweep.ApplyDefaults()
if err := cfg.validateFastmailSources(fastmailSourceIDConfigured(content)); err != nil {
return nil, err
}
Expand Down
25 changes: 25 additions & 0 deletions internal/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1680,6 +1680,31 @@ strip_signatures = false
assert.True(cfg.Vector.Preprocess.StripQuotesEnabled(), "StripQuotesEnabled() should be true (unset → default)")
}

func TestLoadVectorOrcaRouterEmbeddingFormatFillsDefaults(t *testing.T) {
require := require.New(t)
assert := assert.New(t)
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.toml")

configContent := `
[vector]
enabled = true

[vector.embeddings]
api_format = "orcarouter"
`
require.NoError(os.WriteFile(configPath, []byte(configContent), 0o644), "WriteFile")

cfg, err := Load(configPath, "")
require.NoError(err, "Load")

assert.Equal("orcarouter", string(cfg.Vector.Embeddings.EffectiveAPIFormat()))
assert.Equal("https://api.orcarouter.ai/v1", cfg.Vector.Embeddings.Endpoint)
assert.Equal("openai/text-embedding-3-small", cfg.Vector.Embeddings.Model)
assert.Equal(1536, cfg.Vector.Embeddings.Dimension)
assert.Equal("ORCAROUTER_API_KEY", cfg.Vector.Embeddings.APIKeyEnv)
}

func TestLoadWithNamedOAuthApps_RelativePaths(t *testing.T) {
require := require.New(t)
tmpDir := t.TempDir()
Expand Down
25 changes: 25 additions & 0 deletions internal/config/people_sweep_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,31 @@ source_since = "2025-01-01"
assert.ErrorContains(t, err, "allowed_sources")
}

func TestLoadOrcaRouterPeopleSweepProviderFillsDefaults(t *testing.T) {
assert := assert.New(t)
require := require.New(t)
path := filepath.Join(t.TempDir(), "config.toml")
require.NoError(os.WriteFile(path, []byte(`
[people.sweep]
enabled = true

[people.sweep.provider]
kind = "orcarouter"
retention_posture = "zero_retention"
training_posture = "no_training"
allowed_sources = ["conversation_text"]
source_since = "2025-01-01"
`), 0o600))

loaded, err := Load(path, "")
require.NoError(err)
provider := loaded.People.Sweep.Provider
assert.Equal(peoplesweep.ProviderOrcaRouter, provider.Kind)
assert.Equal(peoplesweep.OrcaRouterDefaultEndpoint, provider.Endpoint)
assert.Equal(peoplesweep.OrcaRouterDefaultModel, provider.Model)
assert.Equal(peoplesweep.OrcaRouterDefaultAPIKeyEnv, provider.APIKeyEnv)
}

func TestLoadDoesNotReplaceExplicitEmptyPeopleProviderKeyEnv(t *testing.T) {
path := filepath.Join(t.TempDir(), "config.toml")
require.NoError(t, os.WriteFile(path, []byte(`
Expand Down
27 changes: 26 additions & 1 deletion internal/peoplesweep/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,15 @@ import (

const (
ProviderOpenAICompatible = "openai_compatible"
ProviderOrcaRouter = "orcarouter"

// OrcaRouter defaults for the [people.sweep.provider] block. The
// gateway serves the OpenAI-compatible Chat Completions contract, so
// the named provider shares the OpenAI-compatible transport.
OrcaRouterDefaultEndpoint = "https://api.orcarouter.ai/v1"
OrcaRouterDefaultModel = "orcarouter/auto"
OrcaRouterDefaultAPIKeyEnv = "ORCAROUTER_API_KEY"
OrcaRouterSignupURL = "https://www.orcarouter.ai"

SourceConversationText SourceClass = "conversation_text"
SourceMeetingText SourceClass = "meeting_text"
Expand Down Expand Up @@ -101,6 +110,20 @@ func (c *Config) ApplyDefaults() {
if c.Provider.Kind == "" {
c.Provider.Kind = ProviderOpenAICompatible
}
if c.Provider.Kind == ProviderOrcaRouter {
// Selecting the gateway by name fills in its defaults so an
// operator can point people-sweep at OrcaRouter with a minimal
// block. Explicit settings always win.
if c.Provider.Endpoint == "" {
c.Provider.Endpoint = OrcaRouterDefaultEndpoint
}
if c.Provider.Model == "" {
c.Provider.Model = OrcaRouterDefaultModel
}
if c.Provider.APIKeyEnv == "" && !c.Provider.AllowAnonymous {
c.Provider.APIKeyEnv = OrcaRouterDefaultAPIKeyEnv
}
}
if c.Provider.Endpoint == "" {
c.Provider.Endpoint = "https://api.openai.com/v1"
}
Expand All @@ -116,7 +139,9 @@ func (c *Config) ApplyDefaults() {
// disabled policy is permitted, but any configured structural value must be
// well formed.
func (c Config) Validate() error {
if c.Provider.Kind != ProviderOpenAICompatible {
switch c.Provider.Kind {
case ProviderOpenAICompatible, ProviderOrcaRouter:
default:
return fmt.Errorf("invalid [people.sweep.provider] kind %q", c.Provider.Kind)
}
if c.Provider.RequestTimeout <= 0 {
Expand Down
51 changes: 51 additions & 0 deletions internal/peoplesweep/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,57 @@ func TestConfigDefaultsStayDisabled(t *testing.T) {
require.ErrorContains(err, "disabled")
}

func TestOrcaRouterProviderDefaults(t *testing.T) {
require := require.New(t)
assert := assert.New(t)
config := peoplesweep.Config{Enabled: true, Provider: peoplesweep.ProviderConfig{
Kind: peoplesweep.ProviderOrcaRouter,
RetentionPosture: "zero_retention",
TrainingPosture: "no_training",
AllowedSources: []peoplesweep.SourceClass{
peoplesweep.SourceMeetingText,
peoplesweep.SourceConversationText,
},
SourceSince: "2025-01-01",
}}
config.ApplyDefaults()

assert.Equal(peoplesweep.ProviderOrcaRouter, config.Provider.Kind)
assert.Equal(peoplesweep.OrcaRouterDefaultEndpoint, config.Provider.Endpoint)
assert.Equal(peoplesweep.OrcaRouterDefaultModel, config.Provider.Model)
assert.Equal(peoplesweep.OrcaRouterDefaultAPIKeyEnv, config.Provider.APIKeyEnv)
require.NoError(config.Validate())

profile, err := config.Profile()
require.NoError(err)
assert.Equal(peoplesweep.OrcaRouterDefaultEndpoint, profile.Endpoint)
assert.Equal(peoplesweep.OrcaRouterDefaultModel, profile.Model)
assert.Equal(peoplesweep.OrcaRouterDefaultAPIKeyEnv, profile.APIKeyEnv)
}

func TestOrcaRouterProviderKeepsExplicitSettings(t *testing.T) {
require := require.New(t)
assert := assert.New(t)
config := peoplesweep.Config{Enabled: true, Provider: peoplesweep.ProviderConfig{
Kind: peoplesweep.ProviderOrcaRouter,
Endpoint: "https://proxy.example.test/v1",
Model: "my-model",
APIKeyEnv: "MY_KEY",
RetentionPosture: "zero_retention",
TrainingPosture: "no_training",
AllowedSources: []peoplesweep.SourceClass{
peoplesweep.SourceMeetingText,
},
SourceSince: "2025-01-01",
}}
config.ApplyDefaults()

assert.Equal("https://proxy.example.test/v1", config.Provider.Endpoint)
assert.Equal("my-model", config.Provider.Model)
assert.Equal("MY_KEY", config.Provider.APIKeyEnv)
require.NoError(config.Validate())
}

func TestProviderProfileHasStableCanonicalPolicy(t *testing.T) {
assert := assert.New(t)
profile, err := validConfig().Profile()
Expand Down
41 changes: 38 additions & 3 deletions internal/vector/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,28 @@ type EmbeddingAPIFormat string
const (
// APIFormatOpenAI uses the OpenAI-compatible embeddings contract.
APIFormatOpenAI EmbeddingAPIFormat = "openai"
// APIFormatOrcaRouter uses the OrcaRouter OpenAI-compatible embeddings
// gateway. It shares the plain OpenAI-compatible wire contract; the
// named format only changes the configuration defaults so an operator
// can select the gateway by name.
APIFormatOrcaRouter EmbeddingAPIFormat = "orcarouter"
// APIFormatVoyageContextual uses Voyage contextualized embeddings.
APIFormatVoyageContextual EmbeddingAPIFormat = "voyage-contextual"
// contextPolicyVersion identifies the contextual document assembly policy.
contextPolicyVersion = 2
)

// OrcaRouter defaults for the named embeddings format. The gateway serves
// OpenAI-compatible /v1/embeddings and mirrors the text-embedding-3-small
// contract (1536-dim).
const (
OrcaRouterDefaultEndpoint = "https://api.orcarouter.ai/v1"
OrcaRouterDefaultModel = "openai/text-embedding-3-small"
OrcaRouterDefaultDimension = 1536
OrcaRouterDefaultAPIKeyEnv = "ORCAROUTER_API_KEY"
OrcaRouterSignupURL = "https://www.orcarouter.ai"
)

// preprocessVersion identifies the embed/preprocess.go implementation
// generation. Bump whenever a change to that file produces a different
// Preprocess() output for the same PreprocessConfig flags — for
Expand Down Expand Up @@ -327,10 +343,10 @@ func (c *Config) Validate() error {
return fmt.Errorf("vector.backend: unknown backend %q (supported: \"sqlite-vec\", \"pgvector\")", c.Backend)
}
switch c.Embeddings.EffectiveAPIFormat() {
case APIFormatOpenAI, APIFormatVoyageContextual:
case APIFormatOpenAI, APIFormatOrcaRouter, APIFormatVoyageContextual:
default:
return fmt.Errorf("vector.embeddings.api_format: unknown format %q (supported: %q, %q)",
c.Embeddings.APIFormat, APIFormatOpenAI, APIFormatVoyageContextual)
return fmt.Errorf("vector.embeddings.api_format: unknown format %q (supported: %q, %q, %q)",
c.Embeddings.APIFormat, APIFormatOpenAI, APIFormatOrcaRouter, APIFormatVoyageContextual)
}
if c.Embeddings.EffectiveAPIFormat() == APIFormatVoyageContextual &&
c.Embeddings.Model != "voyage-context-4" {
Expand Down Expand Up @@ -362,6 +378,25 @@ func (c *Config) ApplyDefaults() {
if c.Backend == "" {
c.Backend = "sqlite-vec"
}
if c.Embeddings.APIFormat == APIFormatOrcaRouter {
// Selecting the gateway by name fills in its defaults so an
// operator can enable vector search with a three-line block:
// [vector.embeddings]
// api_format = "orcarouter"
// An explicit endpoint/model/dimension/api_key_env always wins.
if c.Embeddings.Endpoint == "" {
c.Embeddings.Endpoint = OrcaRouterDefaultEndpoint
}
if c.Embeddings.Model == "" {
c.Embeddings.Model = OrcaRouterDefaultModel
}
if c.Embeddings.Dimension == 0 {
c.Embeddings.Dimension = OrcaRouterDefaultDimension
}
if c.Embeddings.APIKeyEnv == "" {
c.Embeddings.APIKeyEnv = OrcaRouterDefaultAPIKeyEnv
}
}
if c.Embeddings.BatchSize == 0 {
c.Embeddings.BatchSize = 32
}
Expand Down
Loading