Skip to content

ADR 0061 cut 2: relation regression cases and fresh reevaluation #1103

Open

Description

@virtue039

Scenario

A person accepts that a source statement should map to a relation property and a direction. Editing the property's definition, domain or range can later change that interpretation, but there is no stored, human-confirmed case and no report showing what regressed.

I would like to take the relation-property slice of ADR 0061 cut 2, under #725. This is an implementation of the accepted regression-case direction, not a claim of ownership of the whole ontology roadmap.

First deliverable

  • An Editor selects a real open statement and its evidence, an existing relation property, a direction and a rationale, explicitly confirming the expected interpretation.
  • Store the reviewed statement/evidence snapshot and the expected interpretation, with KB isolation and attribution. Keep cases and evaluation history when their original source or target becomes unavailable.
  • Reevaluate relevant cases after property edits. Reuse production candidate filtering and alignment judgments through a fresh read-only evaluation path; do not read a cached human binding as the answer and do not modify formal bindings or facts.
  • Report expected versus actual target/direction, with pass, mismatch, inconclusive, error, pending and stale/unavailable states. A changed ontology during a run invalidates that run's applicability.
  • Show the result and evidence on the ontology workbench. A failure informs the person and never undoes their edit or rewrites the expected interpretation.

Proposed engineering contract for review

The case is a human-confirmed expectation over a preserved exemplar. Initial expected values are not copied from a model's current output. Source retirement/deletion and target deletion are explicit unavailable states; snapshots remain auditable. Evaluation reuses production alignment logic but does not write its result to production bindings. Current results carry case generation, ontology/model input identity and evaluation version; stale completions cannot replace a newer request.

First slice: entity-to-entity relation cases, manual case creation, property definition/domain/range edits and explicit reruns. Classes, broad dependency analysis, automatic capture at adoption, larger-domain competency benches and retirement of old Suggest are later slices of 0061.

Reviewable cuts and acceptance

  1. Schema/store/case APIs with provenance, scoped permissions, isolation, retirement and generation tests.
  2. Fresh evaluator and jobs, with scripted-model tests that exercise wrong direction, missing candidates, disagreement, malformed output, call failures and edit-during-run.
  3. Workbench integration and an executable small relation-case corpus.

Acceptance: save a reviewed case -> edit its property -> see a fresh evaluation -> open the before/after comparison and original quote. Deliberately changing the definition or reversing the reply must expose a mismatch. Mocked-model tests demonstrate lifecycle behavior; live-model runs are reported separately and are not described as proofs of general semantic accuracy.

Are these engineering boundaries consistent with 0061 cut 2? I will keep the implementation reviewable while the details are discussed.

Activity

  1. WaylandYang commented on Oct 8, 2026

    @WaylandYang
    Contributor

    Thank you for the proposal, and for reading 0061 closely.

    Yes, please take it, in a smaller shape than the one you describe. Decision 4 of 0061 does not need a separate evaluator: an edit to a property already makes alignment re-decide its signatures, and a case only compares that outcome with what was expected.

    First PR, and only this:

    • ontology_regression_cases: a statement, the property and direction it is expected to bind to, who added it and how (adoption or person).
    • Cases written on adoption from the proposal's example statements, and one API to add a case from any statement.
    • After alignment re-decides following an edit, record for each case whether it still binds, and when.

    Please leave out the evaluation jobs, the extra states and the version identities. A case whose signature a person bound by hand always passes; say so in the result instead of working around it.

    The workbench view is a second PR. @Maya-Kid, this was offered to you after the adoption work on #725; please say if you had already started.

    Thanks again.

  2. Maya-Kid commented on Oct 15, 2026

    @Maya-Kid
    Contributor

    Sorry for the late reply, @WaylandYang. We had not started on 0061 cut 2, and I'm glad @virtue039 has taken it.

  3. WaylandYang commented on Oct 22, 2026

    @WaylandYang
    Contributor

    Checklist update: the store/API slice (#1178) and the workbench view (#1184) are on dev; the separate evaluator with its extra states was dropped at my request. What is left of cut 2 is the negative-expectation slice specified in #1203 and implemented in #1334, which has been revised to drop the adoption veto and is awaiting re-review. This closes when #1334 lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions