Load cops lazily instead of requiring all 271 files at startup - #1122
Open
tas50 wants to merge 2 commits into
Open
Load cops lazily instead of requiring all 271 files at startup#1122tas50 wants to merge 2 commits into
tas50 wants to merge 2 commits into
Conversation
`lib/cookstyle.rb` globbed `lib/rubocop/cop/**/*.rb` and required every file, so every invocation paid to parse and define all 270 cop classes before it inspected a single line — including `cookstyle --version` and runs filtered down to one cop with `--only`. RuboCop 1.89 added `RuboCop::Cop::LazyLoader`, which rubocop-performance adopted in rubocop/rubocop-performance#530. `register_cop` sets an `autoload` for the cop's constant and registers its *name* with the global registry, so config validation, `--only` resolution, and department listings all work without reading the cop's file. The class is loaded the first time it is actually needed, which for a disabled cop is never. Registrations live in the new `lib/rubocop/cop/chef.rb` and `lib/rubocop/cop/inspec.rb`, in the order the files were previously required, so cop execution order is unchanged. Offense output on both a cookbook and this repository is byte-for-byte identical before and after. Measured on Ruby 4.0.6 / macOS arm64, median of 15 runs: require "cookstyle" (isolated) 488.4ms -> 394.9ms -19.1% cookstyle --version 755.8ms -> 632.9ms -16.3% cookstyle --only <one cop> 760.2ms -> 687.7ms -9.5% full lint, 4-file cookbook 1014.3ms -> 992.7ms -2.1% Cop files read at require time drop from 271 to 0. A full cookbook lint loads 249 of 270 (the 21 cops disabled in the default config never load), a single `--only` cop loads 1, and `--version` loads none. Adding a cop now means adding a `register_cop` line as well as the file. `spec/lazy_cop_loading_spec.rb` makes a missing, duplicated, or misnamed registration a test failure, and asserts that requiring cookstyle still loads no cop classes. A missing registration also fails `rake validate_config` and `require "cookstyle"` itself, because the config entry then names a cop RuboCop has never heard of. `rake validate_config` needed one fix: `const_source_location` on a pending autoload returns the `register_cop` call site rather than the cop's own file, so it now asks `autoload?` first to tell Cookstyle and Chefstyle cops apart. Signed-off-by: Tim Smith <tsmith84@proton.me>
The new files follow the same header convention as the rest of the tree, but with the copyright attributed to their author rather than to Progress, which is what the existing individually-authored files here do (see the Kantrowitz and Henry headers) and what Chef/Style/CopyrightCommentFormat documents as the correct form for an individual holder. Signed-off-by: Tim Smith <tsmith84@proton.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
lib/cookstyle.rbglobbedlib/rubocop/cop/**/*.rband required every file, so every invocation parsed and defined all 270 cop classes before inspecting a single line — includingcookstyle --versionand runs narrowed to one cop with--only.This applies the technique RuboCop core added in 1.89 and rubocop-performance adopted in rubocop/rubocop-performance#530.
RuboCop::Cop::LazyLoader#register_copdoes two things: it sets anautoloadfor the cop's constant, and it registers the cop's name with RuboCop's global registry as a string. The registry can then validate the config, resolve--onlyarguments, and list departments knowing only names. A cop's file is read the first time the class itself is needed — which, for a cop disabled in the config, is never.Registrations live in two new files,
lib/rubocop/cop/chef.rbandlib/rubocop/cop/inspec.rb, in the same order the files were previously required, so cop execution order is unchanged.No cop files were moved or renamed.
register_coptakes an explicit path, so the 92 cops whose class name doesn't match their filename and the directories that don't match their department (chef/deprecation/→Chef::Deprecations,chef/redundant/→Chef::RedundantCode,chefstyle/ruby/→Chef::Ruby) all work as they are.Benchmark
Ruby 4.0.6, macOS arm64, RuboCop 1.90.0. Median of 15 runs, before and after measured back to back on the same machine. Fixture is a 4-file cookbook.
require "cookstyle"(isolated, excludes Ruby/Bundler boot)cookstyle --version(end to end)cookstyle --only Chef/Modernize/FoodcriticComments(1 file)cookstylefull lint (4-file cookbook)Cop files read during
require "cookstyle":And loaded over the life of a real run:
cookstyle --versioncookstyle --only <one cop>cookstylefull cookbook lintThe full-lint case still improves because the 21 cops disabled in
cookstyle.ymlare now never loaded at all. The--versionand--onlycases are where the win is largest, and those matter for editor integrations and for the tools that embed Cookstyle.Reproduce the isolated require measurement with:
Correctness
--format jsonoutput is byte-for-byte identical before and after, both on a cookbook fixture and across this repository's 557 Ruby files (11 offenses, same cops, same lines, same messages).rake validate_config,bundle exec cookstyle,--chefstyle,--show-cops,--only <cop>,--only <department>, and-aall verified by hand.rake generate_cops_yml_documentationandrake update_readme_cop_countstill work — the docs task goes throughRegistry#cops/#sort!, which force every lazy cop to load.Keeping registrations honest
Lazy loading means adding a cop file is no longer enough to ship a cop, so
spec/lazy_cop_loading_spec.rbturns that into a test failure. It asserts that every cop file is registered exactly once, that no registration points at a missing file, that each registered constant matches the badge its class actually declares, and that requiringcookstylestill loads zero cop classes. Each of those was mutation-tested against a deliberately broken registration.A missing registration is loud in two other places anyway:
rake validate_configfails, and so doesrequire "cookstyle"itself, since the config entry then names a cop RuboCop has never heard of.rake validate_configneeded one fix. It usedconst_source_locationto tell a Cookstyle cop from a Chefstyle cop, but on a pending autoloadconst_source_locationreturns theregister_copcall site rather than the cop's own file. It now asksautoload?first, falling back toconst_source_locationfor cops that are already loaded.WRITING_RULES.mdandDEVELOPER_GUIDE.mdare updated: "the five files" is now six, with theregister_copline as step 2.Requirements
None new. Cookstyle already pins RuboCop
1.90.0, well past the 1.89.0 that introducedLazyLoader.