Abort projector state update on lock timeout#519
Conversation
Waiting for an exclusive access lock on the `projector_states` table causes all projectors to be stuck, so only try to acquire the lock with a short timeout. Abort the entire update if it takes more than `projector_replayer_total_lock_timeout` to acquire the lock.
When queries are still running on view schema tables that need to be activated a lock timeout or deadlock can occur. Retry multiple times to acquire an exlusive lock on these tables.
There was a problem hiding this comment.
Pull request overview
This PR introduces a bounded lock-acquisition strategy for projector state updates and replay activation, aiming to abort/stop retrying when locks can’t be obtained within a configurable total timeout.
Changes:
- Add
projector_replayer_total_lock_timeoutconfiguration with a default value. - Update projector state writes and replayer view-schema table locking to use short per-attempt lock timeouts with retry-until-total-timeout logic.
- Adjust concurrency specs to use per-transaction lock timeouts (
SET LOCAL) and the new total timeout configuration.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| spec/lib/sequent/core/event_publisher_spec.rb | Updates concurrency specs to use the new timeout configuration and local lock timeout behavior. |
| lib/sequent/migrations/projectors_replayer.rb | Adds retry-with-total-timeout behavior when locking view schema tables and adjusts transaction boundaries in prepare/activate steps. |
| lib/sequent/core/projectors.rb | Adds retry-with-total-timeout behavior when locking/upserting ProjectorState. |
| lib/sequent/configuration.rb | Introduces a new configuration option for total lock timeout. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add helper method to set and restore a parameter for the duration of a block.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
erikrozendaal
force-pushed
the
fix/abort-projector-state-update-on-lock-timeout
branch
from
fc965f6 to
cba3bad
Compare
erikrozendaal
deleted the
fix/abort-projector-state-update-on-lock-timeout
branch
Uh oh!
There was an error while loading. Please reload this page.