feat: implement crawlee v4 RequestQueueClient over apify-client - #643
Open
B4nan wants to merge 2 commits into
Open
feat: implement crawlee v4 RequestQueueClient over apify-client#643B4nan wants to merge 2 commits into
B4nan wants to merge 2 commits into
Conversation
Crawlee v4 (apify/crawlee#3729) redesigned `RequestQueueClient` into a stateful, pull-based interface (`addBatchOfRequests`, `fetchNextRequest`, `markRequestAsHandled`, `reclaimRequest`, `isEmpty`, `isFinished`) modeled on the Python SDK — replacing the old thin REST wrapper. The old name-remapping proxy can't satisfy it, so this adds a real `ApifyRequestQueueClient` implementing the interface on apify-client's REST API: - `fetchNextRequest` locks the head server-side (`listAndLockHead`) so the queue is safe to share across consumers; `markRequestAsHandled` / `reclaimRequest` update the request and release the lock. - request ids are derived from the unique key the same way as the platform. - `getMetadata`/`drop`, no-op `purge`. Also adds `ApifyStorageClient.setStatusMessage` (Crawlee v4 calls it on the storage client to surface crawl progress) and bumps `@crawlee/*` to `^4.0.0-beta.66`. Validated end-to-end on the Apify platform: a CheerioCrawler run through Apify Proxy completed 29/29 requests driven entirely by the new request-queue client. Note: published `@crawlee/types@beta.66` still ships the pre-redesign interface in its `.d.ts` even though `@crawlee/core`'s runtime calls the new methods, so the client implements a local copy of the new interface (`RequestQueueClientV4`) until crawlee republishes regenerated types.
Crawlee v4 (apify/crawlee#3740) renamed the storage abstraction — `StorageClient` → `StorageBackend`, `RequestQueueClient` → `RequestQueueBackend`, `createXClient` → `createXBackend` — moved the in-memory client into `@crawlee/core` (`MemoryStorageBackend`, dropping the `@crawlee/memory-storage` package), and further refined the request-queue signatures (`fetchNextRequest`/`getRequest` now return `UpdateRequestSchema | undefined`, `null` → `undefined`, async `setExpectedRequestProcessingTimeSecs`). - `ApifyStorageClient` now implements `StorageBackend` (`createDatasetBackend` / `createKeyValueStoreBackend` / `createRequestQueueBackend`). - `ApifyRequestQueueClient` implements the real `RequestQueueBackend` directly — the temporary local interface copy is gone. - `StorageBackend` also dropped `setStatusMessage`, so `Actor.setStatusMessage` now sets the run status directly via apify-client. - Bumps `@crawlee/*` to `^4.0.0-beta.71`. Compile, 123 unit tests, and a local (memory-backend) smoke run on crawlee@71 all pass. The request-queue logic itself was validated end-to-end on the Apify platform against crawlee@66; on-platform validation on @71 is currently blocked by a crawlee packaging gap (its `@crawlee/fs-storage` native binding is unpublished for musl).
janbuchar
requested changes
Jul 21, 2026
janbuchar
left a comment
janbuchar
left a comment
Contributor
There was a problem hiding this comment.
The description seems outdated, e.g.:
ApifyStorageClient.setStatusMessage— Crawlee v4 calls this on the storage client to surface crawl progress (it was warningsetStatusMessage is not a function).
is no longer true since apify/crawlee#3818
| export interface OpenStorageContext { | ||
| config: Configuration; | ||
| client?: StorageClient; | ||
| client?: StorageBackend; |
Contributor
There was a problem hiding this comment.
I guess the field deserves a rename too?
Comment on lines
+210
to
+211
| // Crawlee v4's RequestQueueBackend is a stateful, pull-based interface that | ||
| // can't be satisfied by name-remapping; it's implemented on apify-client. |
Contributor
There was a problem hiding this comment.
Suggested change
| // Crawlee v4's RequestQueueBackend is a stateful, pull-based interface that | |
| // can't be satisfied by name-remapping; it's implemented on apify-client. |
ok thx bye
| * ``` | ||
| */ | ||
| export class ApifyStorageClient implements StorageClient { | ||
| export class ApifyStorageClient implements StorageBackend { |
Contributor
There was a problem hiding this comment.
Rename the class and file please
| } | ||
|
|
||
| async createRequestQueueClient(options?: CreateRequestQueueClientOptions): Promise<RequestQueueClient> { | ||
| async createRequestQueueBackend(options?: CreateRequestQueueBackendOptions): Promise<RequestQueueBackend> { |
Contributor
There was a problem hiding this comment.
The options don't make it possible to toggle between single and shared modes (no locking vs locking) — is this tracked in an issue? Or shall we fix it right here right now?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Crawlee v4 (apify/crawlee#3729) redesigned
RequestQueueClientinto a stateful, pull-based interface (modeled on the Python SDK) —addBatchOfRequests,fetchNextRequest,markRequestAsHandled,reclaimRequest,isEmpty,isFinished— replacing the old thin REST wrapper. The SDK's old name-remapping proxy can't satisfy it (addBatchOfRequests is not a functionat runtime), so this adds a real implementation.Changes
ApifyRequestQueueClient(new file) — implements the new interface on apify-client's REST API:fetchNextRequestlocks the head server-side (listAndLockHead) so the queue is safe to share across consumers;markRequestAsHandled/reclaimRequestupdate the request and release its lock (deleteRequestLock).sha256 → base64 → strip → 15 chars).isEmptyvialistHead;isFinishedviapendingRequestCount(counts in-progress, so 0 ⇒ truly done).ApifyStorageClient.setStatusMessage— Crawlee v4 calls this on the storage client to surface crawl progress (it was warningsetStatusMessage is not a function).@crawlee/*to^4.0.0-beta.66; removes the now-deadinitialCounthack.Validation
Verified end-to-end on the Apify platform: a
CheerioCrawlerrun through Apify Proxy completed 29/29 requests, 0 failed, driven entirely by the new request-queue client (addBatchOfRequests→fetchNextRequest→markRequestAsHandled→isEmpty/isFinished), withsetStatusMessageworking and no warnings. Unit tests (123) + build + lint clean.Published
@crawlee/types@beta.66still ships the pre-redesign interface in its.d.ts, even though@crawlee/core@beta.66's runtime already calls the new methods (and@crawlee/memory-storage@beta.66doesn't implement them — so localMemoryStoragecrawls are broken on published 66). beta.66 is effectively a broken transitional release. This PR targets the new interface (correct for the runtime, validated on-platform) and implements a local copy of it (RequestQueueClientV4) until crawlee republishes regenerated types — at which point that copy is deleted and weimplements @crawlee/types'RequestQueueClientdirectly.🤖 Generated with Claude Code