Skip to content

feat: add data onboarding CLI skill (SLMD-1717) - #78

Draft
rtiman-spotify wants to merge 1 commit into
mainfrom
slmd-1717
Draft

rtiman-spotify wants to merge 1 commit into
mainfrom
slmd-1717

Conversation

@rtiman-spotify

Copy link
Copy Markdown
Collaborator

Summary

Draft implementation of SLMD-1717: /confidence:onboard-data in the Confidence plugin.

  • Add the CLI command, canonical skill and references for domain-grouped review, optional goals, 20-table paging, bounded revision-specific approval and partial completion.
  • Define status/resume, selected retry, mapping resolution, inclusion of excluded tables, dependency-checked replacement planning and read-only refresh.
  • Register confidence-data for supported plugin hosts and update routing/documentation. Authentication, API composition and validation polling remain in MCP.
  • Add 14 synthetic decision-evaluation cases, paired with-skill/baseline runs, deterministic regression tests and a PR workflow for tests/typechecking.
  • Fix a pre-existing missing URLSearchParams declaration in the Metric Explorer scorer that blocked repository typechecking; include a regression test.

Validation

  • npm run typecheck
  • npm run test:onboard-data — 63 tests passed
  • git diff --check
  • Independent technical/safety review; fixed installed-plugin path resolution and successful zero-evaluator exits, with regression tests
  • Model-quality evaluation — not passing: with skill 2/14 (14.29%), baseline 5/14 (35.71%)
  • Live MCP/local-progress integration and real CLI acceptance journey

The model runs are synthetic single-turn action-plan evaluations, not executed warehouse operations. Failures include action-vocabulary/state-convention mismatches and recovery/decision-preservation checks; they need investigation. Unit test success does not establish acceptable model behavior or end-to-end safety.

Why this is a draft / remaining work

Not ready to merge or release.

  • Integrate the delivered data MCP contracts (SLMD-1921 and related discovery/import work) and durable local progress (SLMD-1931). The skill currently fails closed when required interfaces are missing; it does not implement an ad hoc journal or fabricate tool signatures.
  • Keep context import gated on implemented ownership/storage/read-only agent access (SLMD-1858).
  • Verify authoritative agent availability separately from registration/ACTIVE.
  • Diagnose model/harness mismatches and genuine semantic failures; improve and rerun the paired evaluations without dropping safety assertions.
  • Run the real CLI acceptance scenario: three available tables, one permission failure, one unresolved mapping and one exclusion, plus recovery and the 300-table review matrix.

No real warehouse imports, permission changes, analytical queries or remote data mutations were performed. No successful end-to-end recording exists; this CLI-only draft does not claim browser/video acceptance.

refresh is defined conservatively as read-only rediscovery and review, preserving previous decisions. Its V1 versus P1 launch priority remains undecided.

Add domain review, finite approval, recovery guidance, MCP registration and synthetic onboarding evaluations. Keep unsupported integrations fail-closed and disclose failing model-quality results in the draft PR.

Co-Authored-By: Xirp Agent (gpt-6.1-sol) <xirp-agent@users.noreply.ghe.spotify.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant