direct: fix drift detection for jobs with >100 tasks#4675
Merged
Conversation
…server task pagination Add two new invariant configs that use pydabs to define jobs: - job_pydabs_10_tasks: one job with 10 tasks (basic pydabs coverage) - job_pydabs_1000_tasks: one job with 1000 tasks (exposes pagination bug) Each config uses a venv init script to install databricks-bundles before the test runs. The Python resource loaders read UNIQUE_NAME, DEFAULT_SPARK_VERSION, and NODE_TYPE_ID from the environment. Also update the testserver JobsGet handler to paginate tasks at 100 per page, matching the real Databricks API behavior. This makes the 1000-task test fail locally the same way it fails on cloud. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GetByJobId only returns the first 100 tasks (one API page). Replace it with Get, which handles pagination and returns the complete job with all tasks merged. This fixes false drift detection for jobs with >100 tasks. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
jobs.get returns at most 100 tasks per response. Without pagination, bundle plan incorrectly detected drift for jobs with >100 tasks because the remote state only contained the first page. Fix DoRead to loop through all pages and append tasks, job_clusters, environments and parameters until NextPageToken is empty. Also fix the testserver to match the real API: on pages after the first, return empty arrays for properties that were fully included on page one. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
jobs.get returns at most 100 tasks per response. Without pagination,
bundle plan only compared the first 100 tasks against the desired state,
producing false drift for jobs with more tasks.
Jobs.Get handles pagination internally, so switch from GetByJobId
(single-page convenience wrapper) to Get.
Also update the testserver to match real API behaviour: on pages after
the first, return empty arrays for properties already returned on page
one ("arrays without additional results will be empty on later pages").
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
denik
requested review from
andrewnester,
anton-107,
pietern,
shreyas-goenka and
simonfaltum
as code owners
Collaborator
|
Commit: b0bf5e5
25 interesting tests: 8 FAIL, 7 KNOWN, 7 SKIP, 2 flaky, 1 BUG
Top 19 slowest tests (at least 2 minutes):
|
andrewnester
approved these changes
Mar 6, 2026
pietern
approved these changes
Mar 6, 2026
Contributor
pietern
left a comment
pietern
left a comment
There was a problem hiding this comment.
What happened before the fix?
Did the 1k task job drift?
Contributor
Author
Yes, added this to PR desc:
|
Collaborator
|
Commit: 74899eb
53 interesting tests: 18 FAIL, 17 RECOVERED, 9 flaky, 7 KNOWN, 1 BUG, 1 SKIP
Top 50 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Use jobs.Get() instead of jobs.GetById() to properly paginate tasks.
Why
Fixes plan for jobs with more than 100 tasks.
Tests
New pydabs configs for invariant tests for 10 and 1000 jobs. The 1k config failed no_drift test because remote state only had 100 tasks and the reset were considered 'update'.
Testserver for jobs is extended to read all jobs.