Repository navigation
feat(queue): support per-queue pg-boss work options - #5
Merged
Merged
Conversation
Add an optional `workOptions` map to `withQueues()` so consumers can tune individual queues (concurrency, batch size, polling) instead of every queue sharing pg-boss defaults. Entries are keyed by queue name and forwarded to `boss.work(name, options, handler)`; queues omitted from the map keep the previous behavior (a single worker fetching one job at a time). This lets a reactive, high-volume queue run several jobs concurrently (via `localConcurrency`) so one long-running handler no longer head-of-line-blocks the rest of that queue, while low-traffic queues stay single-worker. The change is additive: `QueueRegistry`, `QueueJobHandler`, and `JobDataFor` are unchanged, so existing `withQueues(registry)` calls compile and behave exactly as before. `WithQueueOptions` is now generic over the registry so the `workOptions` keys are checked against the registered queue names.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional
workOptionsmap towithQueues()so a consumer can tune individual queues (concurrency, batch size, polling) instead of every queue sharing pg-boss defaults. Entries are keyed by queue name and forwarded toboss.work(name, options, handler). Queues omitted from the map keep the current behavior: a single worker fetching one job at a time.Motivation
Today
registerQueueWorkerscallsboss.work(name, handler)with no options for every queue, so each queue runs a single serial worker (batchSize: 1). A reactive, high-volume queue whose handler is long-running (e.g. one that launches and polls an external task) can be head-of-line-blocked by a single slow or stuck job: everything behind it waits until that handler returns. Meanwhile low-traffic queues are fine as single workers.localConcurrency(pg-boss spawns N independent workers per node for that queue) is exactly the right lever, but there was no seam to pass it. This adds one.Changes
src/queue/consumer.tsregisterQueueWorkerstakes an optionalworkOptionsmap and forwards a queue's entry to the three-argboss.work()overload; falls back to the two-arg overload when absent. Adds exportedQueueWorkOptionstype.src/builder.tsWithQueueOptionsis now generic over the registry and gains aworkOptionsfield (keys checked against queue names);withQueues()accepts it;buildConsumer()forwards it.src/queue/__tests__/register-queue-workers.test.tssrc/queue/__tests__/consumer-builder.test.tswithQueues(registry, { workOptions })builds a consumer.Backward compatibility
Additive and non-breaking.
QueueRegistry,QueueJobHandler, andJobDataForare untouched, so existingwithQueues(registry)calls compile and behave exactly as before. The newWithQueueOptionstype parameter defaults toQueueRegistry, so existing references to the type keep working.Testing
pnpm build:check(tsc --noEmit): cleanpnpm test:unit: 273 passed (21 files)pnpm lint: clean (2 pre-existing warnings intests/e2e/error-handling.test.ts, unrelated)Follow-up
A minor release (e.g.
1.16.0) is needed to consumeworkOptionsdownstream. Left out of this PR to match the repo's convention of separatechore: bump versioncommits.