Skip to content

[OPIK-7371] [BE] feat: auto-suffix duplicate automation rule names on create - #7568

Merged
awkoy merged 15 commits into
mainfrom
awkoy/opik-7371-rule-name-uniqueness
Jul 28, 2026
Merged

[OPIK-7371] [BE] feat: auto-suffix duplicate automation rule names on create#7568
awkoy merged 15 commits into
mainfrom
awkoy/opik-7371-rule-name-uniqueness

Conversation

@awkoy

@awkoy awkoy commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Details

Automation / online-evaluation rule names have no uniqueness, so re-running an SDK script that creates the same rule produces multiple rules that look identical in the UI (the list surfaces the name; the UUID is the real identifier). This adds best-effort, project-scoped auto-suffixing on create and on actual rename: when a name already exists in the target project(s), the rule is stored as name-1, name-2, … so rules stay distinguishable — without a DB uniqueness constraint or data migration (existing installs already contain colliding names, which the ticket rules out enforcing).

How it works

  • AutomationRuleDAO.findCandidateNames fetches existing names in the target project(s) sharing the requested name's prefix (name LIKE concat(?, '%') ESCAPE '!'), scoped via the automation_rule_projects junction table (authoritative post-backfill; the legacy project_id column is intentionally not consulted). One query serves create and update — update passes excludeRuleId so a rule never collides with its own current name.
  • AutomationRuleNames (new helper) resolves the final name over that candidate set: smallest-free -N suffix; comparison case-, accent- and trailing-space-insensitive to approximate MySQL's utf8mb4_unicode_ci collation; suffixed candidates are truncated to the 150-char column on a code-point boundary. The LIKE prefix escapes metacharacters (!, %, _ — escape char is ! because the query crosses three backslash-eating layers: Java text block → StringTemplate → MySQL) and is capped at 138 chars so truncated candidates of max-length names still match on re-run.
  • resolveUniqueName (service) is shared by save() (all 6 evaluator types) and update(); a non-name edit (sampling rate, enabled, filters, projects) never renames the rule — the stored name is compared first via findNameById.
  • @Size(max = 150) on the create and update models: an over-long name is now rejected with 422 at the API boundary instead of failing the insert with a 500 (minor API contract change; OpenAPI spec sync is automated).
  • Migration 000092 adds a (workspace_id, name) index. Open question for reviewers: measured with EXPLAIN on MySQL 8.4 at 50k rules, the optimizer never selects it — the project filter (junction index) is what bounds the query. Keep (in case optimizer stats change) or drop (write amplification while unused)? Documented in findCandidateNames Javadoc.

Intentional scope / accepted limitations

  • No DB unique constraint (per the ticket's constraints), so a rare concurrent create of the same name in the same project can still both persist — best-effort disambiguation, no worse than today.
  • Suffixing changes the stored name vs the client-supplied one; a name-based reconcile/upsert loop should read the create response's Location rather than assume the name is unchanged. This is the ticket's chosen direction (suffix, not hard-uniqueness/upsert).
  • Collation folding is approximate: exotic expansions (ßss) and non-default install collations (e.g. utf8mb4_0900_ai_ci) are not replicated exactly.
  • A rare un-backfilled legacy rule (no junction row) may be missed — degrading to a duplicate name, not an error.

Change checklist

  • User facing
  • Documentation update

Issues

  • OPIK-7371

AI-WATERMARK

AI-WATERMARK: yes

  • Tools: Claude Code
  • Model(s): Claude Opus 4.8 (initial implementation), Claude Opus 5 / Fable 5 (review rounds)
  • Scope: full implementation (helper, DAO query, create + update wiring), unit + integration tests, multiple rounds of adversarial self code-review with fixes applied, live browser E2E, and an EXPLAIN-based query-plan audit on seeded data (50k rules)
  • Human verification: reviewer to confirm behavior against the SDK re-run scenario in OPIK-7371

Testing

  • mvn test -Dtest=AutomationRuleNamesTest — 23/23 (parameterised resolve cases incl. case/accent/trailing-space folding, gap-fill, freed-base reuse, prefix lookalikes; likePrefix escaping + long-name cap incl. surrogate-pair boundaries; 150-char truncation and re-run distinctness).
  • mvn test -Dtest='AutomationRuleEvaluatorsResourceTest$DuplicateNameHandling' — 21/21 end-to-end (MySQL/ClickHouse/Redis testcontainers): same-project suffixing, cross-project isolation, multi-project shared-project collision, cross-evaluator-type collision, rename-to-existing with self-exclusion, non-name edits never rename, 422 for >150-char names (create + update), LIKE-metacharacter names (create + rename), a rule literally named % not over-matching, SQL-payload names stored literally, max-length names staying distinct across re-runs, accent-insensitive collision end-to-end.
  • Query-plan audit: EXPLAIN on MySQL 8.4 with 50k rules / 500 projects (typical + skewed project shapes) — findings documented in the DAO Javadoc and the review threads.
  • Live browser E2E against a locally-run backend+frontend: created the same rule name repeatedly in one project (→ -1, -2) and confirmed cross-project isolation. GIFs attached below.
  • Not run: full backend suite (scoped to the affected classes; CI runs all 16 integration groups — green); no frontend changes in this PR.

Documentation

No user-facing docs change. Behavior is transparent (rule names simply gain a -N suffix on collision). Note the 422-for-over-long-names contract change above.

… create

Rule names are not unique, so re-running an SDK script that creates the
same rule produced multiple rules that look identical in the UI. On
create, when a name already exists within the target project(s), the new
rule is now stored as name-1, name-2, ... so rules stay distinguishable
without a DB uniqueness constraint or data migration.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@awkoy
awkoy requested a review from a team as a code owner July 22, 2026 12:40
@github-actions github-actions Bot added java Pull requests that update Java code Backend tests Including test files, or tests related like configuration. labels Jul 22, 2026
@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

⏱️ pre-commit per-hook timing

Hook Description Result Duration
☕ spotless — java backend Format Java code 4.92s
Total (1 ran) 4.92s
⏭️ 40 skipped (no matching files changed)
Hook Description Result
🐍 trim trailing whitespace — python sdk Strip trailing whitespace ⏭️
🐍 fix end of files — python sdk Ensure files end in a newline ⏭️
🐍 ruff — python sdk Lint + autofix Python (ruff) ⏭️
🐍 ruff-format — python sdk Format Python code (ruff) ⏭️
🐍 mypy — python sdk Static type check ⏭️
🤖 trim trailing whitespace — optimizer Strip trailing whitespace ⏭️
🤖 fix end of files — optimizer Ensure files end in a newline ⏭️
🤖 check yaml — optimizer Validate YAML syntax ⏭️
🤖 check json — optimizer Validate JSON syntax ⏭️
🤖 check toml — optimizer Validate TOML syntax ⏭️
🤖 check for added large files — optimizer Block large files (>1MB) ⏭️
🔐 detect private key — optimizer Block committed private keys ⏭️
🤖 check for merge conflicts — optimizer Block merge-conflict markers ⏭️
🤖 check for case conflicts — optimizer Block case-only name clashes ⏭️
🤖 pyupgrade — optimizer Modernize Python syntax ⏭️
🤖 ruff — optimizer Lint + autofix Python (ruff) ⏭️
🤖 ruff-format — optimizer Format Python code (ruff) ⏭️
🤖 mypy — optimizer Static type check ⏭️
📓 nbstripout — optimizer notebooks Strip notebook output ⏭️
📝 markdownlint — optimizer Lint Markdown ⏭️
🔤 codespell — optimizer Fix common misspellings ⏭️
📊 radon cc — optimizer Cyclomatic-complexity gate ⏭️
📊 radon raw — optimizer Raw size metrics gate ⏭️
📊 xenon — optimizer Fail on complexity thresholds ⏭️
📊 lizard — optimizer Cyclomatic-complexity gate ⏭️
🧹 vulture — optimizer Find dead code ⏭️
🛡️ trim trailing whitespace — guardrails Strip trailing whitespace ⏭️
🛡️ fix end of files — guardrails Ensure files end in a newline ⏭️
🛡️ ruff — guardrails Lint + autofix Python (ruff) ⏭️
🛡️ ruff-format — guardrails Format Python code (ruff) ⏭️
🛡️ mypy — guardrails Static type check ⏭️
⚓ helm-docs Regenerate Helm chart README ⏭️
block non-public FE plugins Block non-public FE plugins ⏭️
🧪 pre-commit wrapper smoke tests Self-test the wrapper scripts ⏭️
🌐 eslint — frontend Lint + autofix JS/TS ⏭️
🌐 typecheck — frontend Whole-project tsc type check ⏭️
📘 eslint — typescript sdk Lint + autofix JS/TS ⏭️
📘 typecheck — typescript sdk Whole-project tsc type check ⏭️
⚙️ actionlint — github workflows Lint GitHub Actions workflows ⏭️
🐳 hadolint — dockerfiles Lint Dockerfiles ⏭️

…suffixing

Address Baz review: collision comparison now strips diacritics (approximating
utf8mb4_unicode_ci) so accented duplicates collide, and truncateToFit no longer
splits a surrogate pair on non-BMP names. Split the blank-name parameterized test
into explicit cases and added accent/surrogate coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address Baz review: stable literal prefix first, values grouped at the end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
awkoy and others added 2 commits July 22, 2026 16:55
…ions

- Collision query now also considers the legacy automation_rules.project_id
  column (LEFT JOIN + OR), so pre-junction rules on older installs are deduped.
- canonicalKey strips trailing spaces to match MySQL PAD SPACE comparison.
- Add unit (trailing space) and integration (multi-project shared collision,
  cross-evaluator-type collision) coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ng self)

Final review flagged the uniqueness invariant as half-enforced: create()
auto-suffixed but update() wrote the name verbatim, so a PATCH rename could
re-introduce a collision. update() now resolves a unique name against the
target project(s) excluding the rule itself (via findNamesByProjectsExcludingRule),
so an unchanged name or a non-name edit is never spuriously suffixed. Adds an
integration test for rename-collision and the self-exclusion no-op.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r commit

Baz review round 3:
- update() only resolves a unique name when the name actually changes vs the
  stored name (findNameById); a non-name edit (sampling rate, enabled, filters)
  no longer silently renames a rule to name-1 when a same-named rule already
  exists in the project scope (e.g. legacy duplicates or a shared-project edit).
- Move the "applied suffix" log out of the write transaction so a rolled-back
  write can't leave a misleading success line.
- Add integration test: a non-name edit that introduces a scope collision keeps
  the name unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
awkoy and others added 2 commits July 23, 2026 11:52
…missing 404

Self-review: findNameById returned a bare String, deviating from the codebase's
universal Optional<...> lookup convention and relying on JDBI's ambiguous
zero-row behavior. Updating a non-existent rule would call it before
updateBaseRule; return Optional<String> so the missing row is handled cleanly
and the update still 404s (not 500). Add a regression test for that path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve conflict in AutomationRuleEvaluatorService.update(): keep the OPIK-7371
name dedup (pass resolved uniqueName to updateBaseRule) while adopting main's
new trigger_scope parameter (OPIK-6741). save() and the DAO auto-merged (both
name-suffix queries and trigger_scope coexist).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Backend Tests - Unit Tests

2 867 tests   2 865 ✅  1m 33s ⏱️
  388 suites      2 💤
  388 files        0 ❌

Results for commit fbcb493.

♻️ This comment has been updated with latest results.

@awkoy

awkoy commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

ℹ️ Note on the failing Backend Tests - Unit Tests / Unit Tests check

The 5 failures are not related to this PR — they are a pre-existing breakage on main that this branch inherited by merging main in.

Failing tests: AnthropicClientGeneratorTest

  • SamplingParamsCapability.supportsSamplingParamsReturnsFalseForAdaptiveThinkingModels (claude-sonnet-5, claude-opus-4-7, claude-opus-4-8)
  • TemperatureGating.temperatureIsNotForwardedForAdaptiveThinkingModels (claude-sonnet-5, claude-opus-4-8)

Why it's not from OPIK-7371:

  • This PR's diff (origin/main...HEAD) touches nothing Anthropic/LLM-client related — only AutomationRuleNames, AutomationRuleDAO, and AutomationRuleEvaluatorService name-suffixing.
  • The failures reproduce deterministically on plain main (run locally on the merged tree: AnthropicClientGeneratorTestTests run: 32, Failures: 5). AnthropicModelName.supportsSamplingParams(...) now returns true for those models, where the test expects false.
  • Root cause looks like a "green-PRs → red-main" integration break: the adaptive-thinking judge PR ([issue-7526][BE] fix: support Anthropic adaptive-thinking models as online-rule judges #7531) added these assertions, and the later provider model-definition sync ([NA] [BE][FE] chore: sync provider model definitions #7560) shifted the underlying model list. It affects any PR that merges current main, not just this one.

Everything OPIK-7371-specific is green: all new unit tests (AutomationRuleNamesTest 14/14), the DuplicateNameHandling integration suite (7/7), all 16 Backend Integration Groups, CodeQL/Analyze/Baz.

This should clear once main's AnthropicClientGeneratorTest is fixed (owned by the Anthropic/model-sync change authors) and this branch re-merges main.

@andrescrz andrescrz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR has major issues that we should address before moving forward. I left guidance to fix them. Please take a look.

…ookup

Address @andrescrz review:
- Replace the fetch-all-names-in-workspace + Java-match approach with a bounded,
  index-backed prefix query (name LIKE 'base%'), so it no longer loads every rule
  name into memory.
- Add index automation_rules(workspace_id, name) (migration 000091).
- Scope collisions per project via the junction table only (authoritative after the
  AutomationRuleProjectMigration backfill); drop the legacy project_id OR branch.
- Collapse the two near-identical DAO queries into one StringTemplate query with an
  optional <if(excludeRuleId)> clause.
- Escape LIKE metacharacters (\ % _) in the prefix so free-form names match literally.
- update(): return the resolved name from the transaction (drop the String[] holder);
  keep the non-name-edit short-circuit; log after commit, no longer dumping projectIds.

Optimistic by design (no DB unique constraint; rare un-backfilled legacy rule or
concurrent create may duplicate) per ticket scope. Tests: unit 15, integration 10.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
awkoy and others added 3 commits July 24, 2026 10:58
… main merge

main introduced its own 000091 (000091_add_filter_guardrail_type_config_type);
renumber the automation-rules name index migration to 000092 to avoid a
duplicate changeset number.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The query returns prefix-filtered candidates; the precise collision check
happens in AutomationRuleNames.generateUniqueName. Name now says what it does.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@awkoy
awkoy requested a review from andrescrz July 24, 2026 11:37

@andrescrz andrescrz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has addressed many of the previous comments.

Let's keep iterating on this.

Please make sure all Baz comments are either addressed or replied.

…E escaping, shared resolution

Review fixes:
- Reject names >150 chars with 422 at the API boundary (@SiZe on both the
  create and update models); previously an over-long name reached the insert
  and failed with a 500 (pre-existing, surfaced by review)
- Declare the LIKE escape character explicitly and switch it from '\' to '!':
  the query passes through three escaping layers (Java text block ->
  StringTemplate -> MySQL), so ESCAPE '\' silently rendered as a syntax error;
  '!' is inert in all three
- Fix duplicate suffixes for max-length names on re-run: suffixing a 150-char
  name truncates its base, so the stored "name-1" no longer shared the full
  search prefix and the next run regenerated it; the LIKE prefix is now capped
  at 138 chars (150 minus the worst-case suffix reserve)
- Extract resolveUniqueName() shared by save() and update() so the suffixing
  rules cannot drift between them
- Hoist the \p{M} diacritics Pattern to a static final; use
  CollectionUtils.isEmpty for the null-safe check
- Rework the suffix log line: stable greppable prefix, all values trailing,
  no longer claims to print a "suffix" when it prints the full name
- Correct the "index-backed" claim in findCandidateNames docs: measured on
  MySQL 8.4 with 50k rules, the (workspace_id, name) index from migration
  000092 is never selected; the project filter is what bounds the query

Tests:
- DuplicateNameHandling: centralise helpers into two primitives with
  delegating overloads; add 422 length tests (create + update), wildcard
  '%'-named rule non-overmatch, SQL-payload literal round-trips, and the
  max-length re-run regression test
- AutomationRuleNamesTest: parameterise the (requested, existing) -> expected
  cases into one @MethodSource test (18 named cases); keep truncation and
  likePrefix cases standalone; cover the new prefix cap incl. surrogate-pair
  boundary

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@awkoy

awkoy commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up on my CI note above (23 Jul): obsolete — the AnthropicClientGeneratorTest breakage was fixed on main, and after merging it in, Unit Tests and all 16 Backend Integration Groups are green on this branch. Nothing pending from that thread.

Latest push (45f1981) addresses the remaining review round: name-length validation (422 instead of 500), explicit LIKE ESCAPE, a shared resolveUniqueName path, the test-helper/parameterisation cleanups, and one real bug the audit surfaced (duplicate suffixes for max-length names on re-run — failing test first, then fixed). Per-thread details in the inline replies.

Two claims previously rested on unit tests or shared code paths alone:
- rename-to-metacharacter-name: the update path shares resolveUniqueName
  with create, but escaping on rename had no end-to-end lock
- accent-insensitive collision (Café/Cafe): the in-Java canonical-key fold
  was unit-tested, but nothing proved the LIKE prefix fetch (running under
  utf8mb4_unicode_ci) actually returns the accented stored name as a
  candidate

Both are now integration-tested in DuplicateNameHandling (21 tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@awkoy
awkoy requested a review from andrescrz July 28, 2026 15:06

@andrescrz andrescrz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together. LGTM.

@awkoy
awkoy merged commit 948996c into main Jul 28, 2026
73 checks passed
@awkoy
awkoy deleted the awkoy/opik-7371-rule-name-uniqueness branch July 28, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Backend java Pull requests that update Java code 🔴 size/XL tests Including test files, or tests related like configuration.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants