Local downloading of optimization statistics for the build by mu-hun · Pull Request #1186 · AdguardTeam/FiltersRegistry

mu-hun · 2026-04-03T12:17:02Z

Enables local download and reuse of optimization stats, supporting filter creation based on a pinned stats snapshot.

This feature is managed entirely through build flags in FiltersRegistry.

The workflow

yarn download-stats (--download-stats) downloads per-filter stats.json files to temp/optimization/stats/filters/<filterId>/stats.json. Use --include or --skip to scope which filters are downloaded.
yarn build:local: If downloaded optimization stats exist, stats are read from there; if any required filter is missing its stats.json, the build fails with a message pointing to yarn download-stats. If stats doesn't exist, stats are fetched live from the remote server instead.

Strip volatile metadata lines (Checksum, Diff-Path, TimeUpdated, Version) from compiled filter files after build. Reduces noise when comparing outputs between compiler versions.

- one-liner => helper function - .txt extension to the constant in strip_generated_meta.js

mu-hun · 2026-04-08T16:10:15Z

Please specify how to perform the test.

I have just aligned with the flags you created (#1181). In the PR description,

Both --generate-cache and --use-cache flags now include optimization config handling alongside the existing filter source caching.

mu-hun · 2026-04-27T10:21:57Z

Crash fixed

7691c68. The cache dir is wiped with force: true before generate() runs, so re-running on an existing cache no longer errors.

Co-authored-by: Copilot <copilot@github.com>

mu-hun · 2026-08-26T03:44:14Z

If there are no further suggestions or change requests, I'm ready to merge 👀

maximtop

Mostly small stuff inline. The one I'd genuinely chase is the e2e percent.json mock — it doesn't intercept the compiler's fetch, so the new CI workflow hits the live network on every PR. The beta compiler pin and the script placement are already tracked in open threads.

maximtop · 2026-08-26T12:58:21Z

+
+// compile() always fetches percent.json remotely, even under use(). Mock only
+// that call so the suite below stays offline; everything else passes through.
+vi.mock('child_process', async (importOriginal) => {


Worth another look at the offline claim here: the compiler's ESM build fetches percent.json via createRequire(import.meta.url)('child_process').execFileSync('curl', ...) (checked the pinned 3.3.0-beta.1 dist, index.js:1000/1025), and since vitest externalizes @adguard/filters-compiler, this vi.mock('child_process') never intercepts that call — the real curl runs. So every yarn test (both Node legs of the new test.yaml on every PR) curls the live chrome.adtidy.org/optimization_config/percent.json and depends on it listing filterId 1. On a network hiccup the compiler's curl --retry 5 backoff (~31s) also blows past the 30s beforeAll timeout and fails the whole file. Inlining the compiler into vitest's transform pipeline (server.deps.inline), having the compiler accept a local percent.json, or stubbing curl via PATH would each fix it.

Including #1186 (comment), I'll take this issue as adding an option to load percent.json locally from #1222.

But I want to finalize this PR with no more additional patches, so I'd like to handle that issue after this feature is released. Therefore, I just dropped the "offline" keyword from the PR body and main issue description as follows:

Enables local download and reuse of optimization stats, supporting filter creation based on a pinned stats snapshot.

maximtop · 2026-08-26T12:58:51Z

 `template.txt` with a single `@include "./filter.txt"` directive, and compiles
 from that copy. The original `filters/` directory is never modified.

+Optimization stats are picked up automatically: if `temp/optimization/stats`


This says the local stats are "used as-is", but getOptimizableFilterIds() in the pinned compiler always fetches percent.json via curl, even after localOptimizationStatistics.use() — only the per-filter stats.json content comes from disk. So yarn build:local still needs network (offline it hangs through curl retries and dies with a raw curl error, no --download-stats hint), and the snapshot isn't fully pinned since the live percent.json decides which filters get optimized. Either document the remaining network dependency here (and drop "offline" from the PR body), or have the compiler read percent.json from the cache dir under use().

answered in #1186 (comment)

maximtop · 2026-08-26T12:58:51Z

+echo "Branch (reference): master          @ $MASTER_SHA"
+echo "Branch (feature):   $CHANGED_BRANCH @ $CHANGED_SHA"
+if [ "${BUILD_LOCAL:-false}" = "true" ]; then
+  BUILD_CMD="generate-cache && yarn build:local --no-patches-prepare --strip-generated-meta"


This script uses 2-space indentation and no shebang, while the repo's other shell scripts (scripts/auto_build.sh, the translations scripts) start with #!/bin/bash and use 4-space indentation. Harmless at runtime since DEVELOPMENT.md invokes it via bash, but worth matching when it moves in the child PR.

Okay, I'll update to use 4-space indent in one-level child PR.

maximtop · 2026-08-26T12:58:51Z

        };
    }

+    if (flags.useCache && flags.downloadStats) {


The three mutual-exclusion blocks here are identical except for flag names, and this PR adds the second and third. The neighboring incompatibility checks were just deduplicated via buildIncompatibilityError; the same treatment fits (a buildMutualExclusionError(a, b) helper or a small table of exclusive pairs), so a wording tweak isn't edited three times.

maximtop · 2026-08-26T12:58:51Z

+        });
+    });
+
+    it('--download-stats: downloads stats for all filters, compile skipped', async () => {


The three --download-stats tests below are line-for-line identical (~17 lines each) — set argv, import build.js, destructure the mocked compiler, vi.waitFor the same download assertion; only the extra flag and expected include/exclude arrays differ. A single it.each keeps the coverage without the copy-paste and would also absorb the repeated mock destructure recurring in nearly every test of this block.

Document that --download-stats swaps the entire local stats directory rather than merging per-filter, so scoped downloads must be paired with an equally scoped build. I've removed description "--include or --skip are not both at once." in AGENTS.md and add description rejects --report flag in both files. Also, "scoped downloads must be paired with an eqaully scoped build — sholud I mentioned the same for the `--generate-cache` flag? Commented in - #1186 (comment) - #1186 (comment) - #1186 (comment) - #1186 (comment)

Suggested in #1186 (comment) This reverts commit ec47a3a.

The top-level buildFilters() call had no .catch, so any rejection surfaced as an UnhandledPromiseRejection trace where the actionable hint (e.g. "run --download-stats") is easy to miss. Now matches the arg-validation error pattern: print the message in red and exit(1). Commented in #1186 (comment)

The top-level buildFilters() call had no .catch, so any rejection surfaced as an UnhandledPromiseRejection trace where the actionable hint (e.g. "run --download-stats") is easy to miss. Now matches the arg-validation error pattern: print the message in red and exit(1). Commented in #1186 (comment) About updated test case: The missing-stats.json case previously asserted on an unhandledRejection. Now that buildFilters() catches rejections and exits 1, the test spies console.error and process.exit, waits for exit(1), and checks the printed output contains the --download-stats hint plus the original error message.

#1186 (comment)

'optimization_config' base path name to 'optimization/stats': build.js now uses optimizationStatsDir = path.join(tempDir, 'optimization', 'stats'). Mirror that. Suggested in #1186 (comment)

--use-cache, --generate-cache, and --download-stats are one mutually exclusive group. Replace the three near-identical pairwise blocks with a single table + count check, so the wording lives in one place. Suggested in #1186 (comment)

Suggested in #1186 (comment)

mu-hun

I've reviewed and applied all of your suggestions.

mu-hun · 2026-08-28T06:23:39Z

-yarn build --download-stats --skip=12,24
+# Scoped download replaces the cache with just these filters — pair with a matching scoped build
+yarn build --download-stats --include=1,2,3 && yarn build:local --include=1,2,3
+yarn build --download-stats --skip=12,24 && yarn build:local --skip=12,24


Scoped downloads must be paired with an equally scoped build

Should I also mentioned above for the --generate-cache flag?

AS-IS

FiltersRegistry/DEVELOPMENT.md

Lines 248 to 252 in a7a1007

# Cache generation with filter selection

yarn build --generate-cache

yarn build --generate-cache --include=1,2,3

yarn build --generate-cache --skip=12,24

yarn build --generate-cache --report='report.txt'

TO-BE

# Cache generation with filter selection yarn build --generate-cache # Scoped download replaces the cache with just these filters — pair with a matching scoped build yarn build --generate-cache --include=1,2,3 && yarn build:local --include=1,2,3 ...

mu-hun · 2026-08-28T06:26:38Z


        const filtersDir = path.join(tmpDir, 'filters');
-        optimizationDir = path.join(tmpDir, 'optimization_config');
+        optimizationStatsBasePath = path.join(tmpDir, 'optimization', 'stats');


I've updated the outdated optimization_config base path name to optimization/stats:

Alex-302 and others added 5 commits March 28, 2026 20:40

Fix #1180 Add the ability to build filters completely locally

a8ac982

fix/lint errors in build.js

dce899d

fix/Add --strip-generated-meta build flag for debug purposes

395bd2f

Strip volatile metadata lines (Checksum, Diff-Path, TimeUpdated, Version) from compiled filter files after build. Reduces noise when comparing outputs between compiler versions.

fix/lint

b656361

merge parent branch into current one, resolve conflicts

a1af0ab

mu-hun requested a review from Alex-302 April 3, 2026 12:17

mu-hun self-assigned this Apr 3, 2026

mu-hun mentioned this pull request Apr 6, 2026

Add the ability to use local optimization_config.json when build filters AdguardTeam/FiltersCompiler#216

Open

Alex-302 added 3 commits April 6, 2026 22:19

- strip_generated_meta.js renamed to kebab-case

53c6a5c

- one-liner => helper function - .txt extension to the constant in strip_generated_meta.js

Reused find_files.js in strip-generated-meta.js and build.js

7af7209

use fs/promises in strip-generated-meta

576f7ef

mu-hun force-pushed the local_optimization_config branch 2 times, most recently from 6507dd5 to 3bcca2e Compare April 7, 2026 10:01

Alex-302 added 2 commits April 7, 2026 15:52

convert strip-generated-meta.js to TS

a12947d

Upd docs

54a6df8

Alex-302 reviewed Apr 7, 2026

View reviewed changes

Comment thread yarn.lock

mu-hun force-pushed the local_optimization_config branch 2 times, most recently from df6f136 to 424e89f Compare April 8, 2026 15:53

mu-hun marked this pull request as ready for review April 8, 2026 16:34

mu-hun requested a review from Alex-302 April 8, 2026 16:34

Alex-302 mentioned this pull request Apr 24, 2026

Add the ability to use local optimization_config AdguardTeam/FiltersCompiler#276

Closed

mu-hun force-pushed the local_optimization_config branch from e04f297 to 867ab88 Compare April 27, 2026 08:19

Alex-302 and others added 6 commits April 28, 2026 00:02

Fixed invoking strip-generated-meta

7a10cdb

Co-authored-by: Copilot <copilot@github.com>

Fix lint error

337d24a

Co-authored-by: Copilot <copilot@github.com>

strip meta in both platforms/ and temp/platforms/, add tests

9c694a6

Co-authored-by: Copilot <copilot@github.com>

lint errors

6875b74

Co-authored-by: Copilot <copilot@github.com>

Fix recursive walk inside platforms dirs

a6919b5

Co-authored-by: Copilot <copilot@github.com>

feat(build): add CLI argument validation, added tests

aea8ac6

Co-authored-by: Copilot <copilot@github.com>

mu-hun added 4 commits August 24, 2026 15:07

Apply suggestion to test workflow

8961190

Create directory before downloads

ec47a3a

Fix test failure

5fa70ee

Merge branch 'master' into local_optimization_config

d643eda

zloyden approved these changes Aug 26, 2026

View reviewed changes

Alex-302 approved these changes Aug 26, 2026

View reviewed changes

mu-hun requested review from maximtop and removed request for maximtop and piquark6046 August 26, 2026 11:56

maximtop approved these changes Aug 26, 2026

View reviewed changes

maximtop reviewed Aug 26, 2026

View reviewed changes

Comment thread DEVELOPMENT.md

mu-hun mentioned this pull request Aug 27, 2026

AG-28970 update compiler to v3.3.0 #1233

Open

mu-hun added 2 commits August 27, 2026 15:25

Revert "Create directory before downloads"

a6594c2

Suggested in #1186 (comment) This reverts commit ec47a3a.

mu-hun force-pushed the local_optimization_config branch from b902864 to 5cc9a45 Compare August 27, 2026 11:15

mu-hun added 3 commits August 27, 2026 20:41

Add missed end with dot

8fb54e1

#1186 (comment)

Restore line_lenght markdownlint for tables

ae66038

#1186 (comment)

Describe test suites under the scripts/build/__tests__/ path

5302738

#1186 (comment)

slvvko approved these changes Aug 27, 2026

View reviewed changes

mu-hun added 3 commits August 28, 2026 15:11

test: update the outdated names in intergration test cases

2b56604

'optimization_config' base path name to 'optimization/stats': build.js now uses optimizationStatsDir = path.join(tempDir, 'optimization', 'stats'). Mirror that. Suggested in #1186 (comment)

refactor: dedupe build-mode mutual-exclusion checks

cd98e2d

--use-cache, --generate-cache, and --download-stats are one mutually exclusive group. Replace the three near-identical pairwise blocks with a single table + count check, so the wording lives in one place. Suggested in #1186 (comment)

test: collapse three --download-stats cases into it.each

dec4b8b

Suggested in #1186 (comment)

mu-hun commented Aug 28, 2026

View reviewed changes

mu-hun requested a review from maximtop August 28, 2026 06:29

slvvko approved these changes Aug 28, 2026

View reviewed changes

	# Cache generation with filter selection
	yarn build --generate-cache
	yarn build --generate-cache --include=1,2,3
	yarn build --generate-cache --skip=12,24
	yarn build --generate-cache --report='report.txt'

Uh oh!

Conversation

mu-hun commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The workflow

Uh oh!

Uh oh!

mu-hun commented Apr 8, 2026

Please specify how to perform the test.

Uh oh!

mu-hun commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Crash fixed

Uh oh!

mu-hun commented Aug 26, 2026

Uh oh!

maximtop left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mu-hun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

mu-hun commented Apr 3, 2026 •

edited

Loading

mu-hun commented Apr 27, 2026 •

edited

Loading