Skip to content

Simplify bundle JSON schema annotation system#4670

Draft
shreyas-goenka wants to merge 8 commits intomainfrom
simplify-schema-annotations
Draft

Simplify bundle JSON schema annotation system#4670
shreyas-goenka wants to merge 8 commits intomainfrom
simplify-schema-annotations

Conversation

@shreyas-goenka
Copy link
Contributor

@shreyas-goenka shreyas-goenka commented Mar 5, 2026

Summary

  • Read OpenAPI descriptions directly from the spec at runtime instead of pre-extracting them into YAML files. The OpenAPI spec is now a requirement for schema generation.
  • Remove annotations_openapi.yml and annotations_openapi_overrides.yml which flip-flopped on every SDK bump. Merge manual overrides into a single annotations.yml.
  • Use bundle paths (e.g. resources.jobs.*) as annotation keys instead of Go type paths.
  • Remove bundle/docsgen. It was never used beyond the initial documentation scaffolding for our public docs.

Test plan

  • All schema tests pass
  • JSON schema output is identical (0 descriptions lost, 81 gained)
  • Schema generation is idempotent

🤖 Generated with Claude Code

@eng-dev-ecosystem-bot
Copy link
Collaborator

eng-dev-ecosystem-bot commented Mar 5, 2026

Commit: 9a16182

Run: 22858443739

Env 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
🔄​ aws linux 4 8 7 264 781 12:05
🔄​ aws windows 5 7 7 266 779 7:07
🔄​ aws-ucws linux 2 7 7 364 696 7:49
🔄​ aws-ucws windows 2 7 7 366 694 7:22
💚​ azure linux 2 9 271 779 5:55
💚​ azure windows 2 9 273 777 5:01
🔄​ azure-ucws linux 2 1 9 369 692 7:57
🔄​ azure-ucws windows 2 1 9 371 690 7:00
💚​ gcp linux 2 9 267 782 5:21
💚​ gcp windows 2 9 269 780 5:23
24 interesting tests: 11 flaky, 7 SKIP, 6 RECOVERED
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
🔄​ TestAccept 💚​R 💚​R 🔄​f 🔄​f 💚​R 💚​R 🔄​f 💚​R 💚​R 💚​R
🔄​ TestAccept/bundle/deploy/files/no-snapshot-sync ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/deployment/bind/job/generate-and-bind 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/deployment/bind/job/generate-and-bind/DATABRICKS_BUNDLE_ENGINE=direct 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/resources/postgres_branches/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/update_protected 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/without_branch_id 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/synced_database_tables/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🔄​ TestAccept/selftest/record_cloud/pipeline-crud ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/selftest/record_cloud/pipeline-crud/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/ssh/connect-serverless-gpu 🙈​s 🙈​s 🔄​f 🔄​f 🙈​s 🙈​s 🔄​f 🔄​f 🙈​s 🙈​s
🔄​ TestAccept/ssh/connection 💚​R 🔄​f 💚​R 💚​R 💚​R 💚​R 💚​R 🔄​f 💚​R 💚​R
Top 20 slowest tests (at least 2 minutes):
duration env testname
4:10 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:53 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:46 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:43 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:40 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:39 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:32 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:10 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:05 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:59 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:59 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:56 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:46 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:45 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:22 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:16 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:15 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:14 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:10 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:06 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct

shreyas-goenka and others added 4 commits March 9, 2026 13:26
Read OpenAPI descriptions directly from the spec at runtime instead of
pre-extracting them into annotation YAML files. This makes the OpenAPI
spec a requirement for schema generation and eliminates annotation files
that flip-flopped on every SDK bump.

Changes:
- Remove annotations_openapi.yml and annotations_openapi_overrides.yml
- Merge manual overrides into a single annotations.yml
- Use bundle paths (e.g. resources.jobs.*) as annotation keys instead
  of Go type paths
- Add path_mapping.go to resolve bundle paths to Go types via reflection
- Remove bundle/docsgen (was the only other consumer of annotations)
- Makefile schema targets now require DATABRICKS_OPENAPI_SPEC

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix copyloopvar lint warnings in annotations.go
- Fix gofmt issue in path_mapping_test.go
- Use maps.Copy instead of manual loop
- Restore permission level enums, lifecycle descriptions,
  markdownDescription, and SDK type overrides from the deleted
  annotations_openapi_overrides.yml into annotations.yml
- Update getOpenApiAnnotations to check embedded types for
  promoted field descriptions from OpenAPI spec
- Update TestNoDetachedAnnotations to handle Go type path keys

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…qlWarehouse, and SDK types

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ter rebase

Add a "Download OpenAPI spec" step to the validate-generated-is-up-to-date
CI job so make schema can find .codegen/openapi.json. Also regenerate schema
and python codegen to sync with latest main (removed grant types, updated
SDK descriptions).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@shreyas-goenka shreyas-goenka force-pushed the simplify-schema-annotations branch from 963126c to a508ab4 Compare March 9, 2026 13:38
When DATABRICKS_OPENAPI_SPEC is set, OpenAPI-derived descriptions,
deprecation info, and output-only markers are now persisted into
annotations.yml. This makes the schema fully reproducible from
annotations.yml alone, so CI can run `make schema` without downloading
the OpenAPI spec.

- Change OutputOnly from *bool to string for clean YAML serialization
- Add schema-openapi Makefile target for generating with the spec
- Revert CI download step; make schema now works without the spec
- Merge annotations from both Go type paths and bundle paths

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants