Skip to content

feature: add optimize() call to snapshot materialization (#1335)#1341

Merged
sd-db merged 3 commits intodatabricks:mainfrom
aarushisingh04:feature/snapshot-optimize
Mar 6, 2026
Merged

feature: add optimize() call to snapshot materialization (#1335)#1341
sd-db merged 3 commits intodatabricks:mainfrom
aarushisingh04:feature/snapshot-optimize

Conversation

@aarushisingh04
Copy link
Contributor

partially resolves #1335

Description

  1. added optimize() call to the snapshot materialization, matching existing behavior in table and incremental materializations. when a snapshot is configured with liquid_clustered_by, zorder, or auto_liquid_cluster, the OPTIMIZE query will now execute automatically after each snapshot run.

  2. the change is a single-line addition to snapshot.sql, placed after persist_constraints and before post-hooks (again, consistent with the pattern in table.sql and incremental.sql).

note: the original issue also mentioned default clustering columns and ALTER TABLE support for config changes. these are intentionally excluded from this PR to keep scope minimal and can be addressed in follow-up PRs if needed. i believe they need a design decision that should be discussed by/with maintainers first due to having a larger scope.

Checklist

  • I have run this code in development and it appears to resolve the stated issue (unit tests pass : this change mirrors the identical optimize() call pattern already used in table.sql and incremental.sql.)
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

Signed-off-by: aarushisingh04 <aarushi07.singh@gmail.com>
@sd-db
Copy link
Collaborator

sd-db commented Mar 5, 2026

Overall the change looks good, one think I would look to add is: a functional test for the snapshot + liquid clustering path. Since this changes default behaviour for all liquid-clustered snapshot tables, we should verify the full lifecycle (create + re-snapshot with updates) works correctly with clustering enabled. The "clustering off" path is already covered by the existing TestSnapshot and TestSnapshotCheck tests.This should be straightforward — just extend BaseSimpleSnapshot with liquid_clustered_by configured:

# In tests/functional/adapter/simple_snapshot/test_snapshot.py

@pytest.mark.skip_profile("databricks_uc_cluster", "databricks_cluster")
class TestSnapshotLiquidClustering(BaseSimpleSnapshot):
    @pytest.fixture(scope="class")
    def project_config_update(self):
        return {"snapshots": {"+liquid_clustered_by": "id"}}

This inherits the full snapshot lifecycle test and confirms the behavior is preserved with optimize running

Signed-off-by: aarushisingh04 <aarushi07.singh@gmail.com>
@aarushisingh04
Copy link
Contributor Author

@sd-db agreed, added the required functional test in test_snapshot.py. thanks for the review!

…stering test

Signed-off-by: aarushisingh04 <aarushi07.singh@gmail.com>
Copy link
Collaborator

@sd-db sd-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, thx for the PR !!

@sd-db sd-db merged commit a871acd into databricks:main Mar 6, 2026
sd-db added a commit that referenced this pull request Mar 7, 2026
## Summary

- Followup to #1341; I realised the table v2 materialization path in
`table.sql` was missing the `optimize()` call that exists in all other
materialization paths (table v1, incremental v1, incremental v2)
- When `use_materialization_v2` was enabled with `liquid_clustered_by`,
`zorder`, or `auto_liquid_cluster` configured, `OPTIMIZE` would not run
after table creation/replacement
- Adds a functional test for table v2 + liquid clustering

## Test plan

- [x] Unit tests pass (717/717)
- [x] Functional test `TestTableV2LiquidClustering` passes — confirms
`OPTIMIZE` runs in the v2 path
- [x] Existing liquid clustering tests still pass (no regressions)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run the optimize on snapshots with liquid clustering

2 participants