Repository navigation
Make header-generator compatible with Cloudflare workers (edge runtimes) #411
Description
Currently its not possible to use header-generator HeaderGenerator class inside Cloudflare workers as we get a the following error:
Error generating random headers with lib: ReferenceError: __dirname is not defined
I would be nice to use this library inside Cloudflare workers (or other runtimes that don't have all of NodeJs's FS APIs).
Looking at the code there are two places which are causing this issue;
- Loading JSON files which could be resolved by adding
resolveJsonModuletotsconfig.json.
https://github.com/apify/fingerprint-suite/blob/master/packages/header-generator/src/header-generator.ts#L215 - Which is a tricky problem, is with passing in a string path to BayesianNetwork that uses another library called adm-zip, maybe this code can be made optional so it can work in Cloudflare workers??
https://github.com/apify/fingerprint-suite/blob/master/packages/header-generator/src/header-generator.ts#L236
Activity
barjin commentedon Aug 13, 2025
Hello and thank you for your interest in this project!
As you've correctly pointed out, the broken
__dirnameresolution is only the first of many problems.The
HeaderGeneratorclass loads the probabilistic models from.zipfiles bundled in the package (in thedata-filesfolder). These are then internally unzipped usingadm-zipand used to generate the random header sets.Since Cloudflare Workers don't provide the
fsAPI, there is no way of loading the probabilistic model for us. This feature cannot be made optional, as it stands in the center of what this package is doing.I see some ways through (e.g. making the
JSONandzipfiles into Javascript code exporting the contents as constants), but we currently don't have capacity for such changes (and it might be a bit more complicated than this, especially with the automatic model updates). Thinking about it, it might also introduce performance issues with parsing such large files (maybe not). All in all, this is sadly not on our immediate roadmap.Community PRs welcome, though!
labithiotis commentedon Aug 13, 2025
@barjin I did look at making a PR, but quickly ran into an issue where the library loads zip files as you mentioned. In the end, I just made a simple GCP Cloud Functions that returns headers for CF worker.
I should have closed the issue when I realised it was a trivial change. I will close the issue now.
barjin commentedon Aug 13, 2025
Thank you for your detailed insight. I'll reopen this issue, as this question actually keeps popping up quite often (see e.g. here or various attempts at running Crawlee in different serverless environments. If we want to make our dependents bundleable and runnable in such environments, we'd likely need to solve this first.