TEMP DO NOT MERGE: chore(nimbus): KINTODIAG instrumentation for kinto approve - #16995
Closed
jaredlockhart wants to merge 3 commits into
Closed
TEMP DO NOT MERGE: chore(nimbus): KINTODIAG instrumentation for kinto approve#16995jaredlockhart wants to merge 3 commits into
jaredlockhart wants to merge 3 commits into
Conversation
Because * The selenium teardown called approve() on the kinto_client factory, so it raised AttributeError and was swallowed by the bare except. * end_experiment() still ran, leaving a pending review that a later test signs as if it were its own, cascading until a test finds nothing pending and raises "Unable to approve kinto review". * approve() retries for 120s, so activating the teardown unbounded would burn that budget per test whenever no review is pending. This commit * Calls the factory so the teardown approves via a KintoClient instance. * Adds an optional retries argument to approve(), leaving the default budget unchanged for in-test callers. * Bounds the teardown to a short explicit budget. Fixes #16978
Because * the constraint was capped at <15, resolving to 14.0, which predates pytest 9.1 support (added in 16.4) * reruns of integration tests with chained fixtures die during setup instead of retrying, so the session ends in error with nothing recorded as failed * the first attempt's traceback never reaches the CI log, leaving the real failure readable only from the uploaded report artifact This commit * bumps pytest-rerunfailures to >=16.4,<17 and re-locks * adds --rerun-show-tracebacks to the integration test jobs that pass --reruns 1 Fixes #16979
… approve TEMP DO NOT MERGE. Throwaway diagnostic branch for the residual "Unable to approve kinto review" intermittency. Not to be merged. Because * `Desktop Nimbus UI` still fails ~9.5% of runs with `Unable to approve kinto review` after both the `--dist loadgroup` and selenium-teardown fixes, and the current code logs nothing about what was pending * `KintoClient.approve()` is slug-blind and its `KintoException` branch is a silent `pass`, so cross-worker signing of another test's review would be invisible * the selenium teardown swallows every exception, so its outcome is unobservable This commit * logs `KINTODIAG` lines from `approve()` on entry, per poll, on the `KintoException` branch, on success and on exhaustion, with the xdist worker, the caller from `PYTEST_CURRENT_TEST`, wall clock and elapsed time * names the slugs a pending review carries by diffing the workspace bucket against the signed main bucket * logs the selenium teardown's slug, rollout decision and approve outcome, including swallowed exceptions * adds `--count 3` to the `nimbus_ui` job to provoke the failure
Collaborator
Author
|
Closing: the instrumentation belongs in #16981 itself, where the fix under test lives. Moved there as |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TEMP DO NOT MERGE. Throwaway diagnostic branch. Closes no issue, must never be merged, and will be deleted once the log evidence is captured.
Because
Desktop Nimbus UIstill fails ~9.5% of runs withUnable to approve kinto reviewwith both the--dist loadgroupfix (fix(nimbus): stop rollout UI tests waiting on presence and colliding on kinto #16906) and the selenium-teardown fix (fix(nimbus): Approve only our own Kinto review in the selenium teardown #16981) applied, and the current code logs nothing about what was pending when it gave up.KintoClient.approve()checks only the collection status, so it signs whatever is pending regardless of slug, and itskinto_http.exceptions.KintoExceptionbranch is a silentpass. A teardown on one worker signing a review belonging to a test on another worker would be entirely invisible.seleniumfixture teardown wraps everything inexcept Exception: pass, so its outcome is unobservable.This commit
KINTODIAGlines fromapprove()on entry, on every poll, on theKintoExceptionbranch, on success and on exhaustion, each carrying the xdist worker, the caller fromPYTEST_CURRENT_TEST, wall clock and elapsed time.main-workspacebucket against the signedmainbucket (get_record_data()alone returns every workspace record, signed or not).seleniumteardown's slug, rollout decision and approve outcome, including swallowed exceptions.--count 3to thenimbus_uijob to provoke the intermittent failure within one run.Base is deliberate:
origin/main+ #16981 + #16980, so the failure is reproduced with both fixes and with first-attempt tracebacks printing.