AirspeedVelocity.jl for benchmark in CI #162

Activity

1. m-fila commented on May 27, 2025

   

   m-fila

   on May 27, 2025

   Member

   More actions

   I don't know, those benchmarks are quite heavy and I feel were written to make more comprehensive comparisons with different variants and fastjet.
   Perhaps something like simplified instrumented-reco would be more appropriate for CI to early detect regressions. And easier to put in BenchmarkTools layout

   👍React with 👍1Reacted by Jerry Ling
2. graeme-a-stewart commented on Jun 19, 2025

   

   graeme-a-stewart

   on Jun 19, 2025

   Member

   More actions

   This looks very interesting - we should investigate. JetReconstructionBenchmarks.jl is way too much. The value here is to quickly identify regressions in the main use cases that we have. If you look at examples/benchmark.sh those are the baseline measurements to take to show that none of the algorithms regressed.
3. m-fila commented on Jun 19, 2025

   

   m-fila

   on Jun 19, 2025

   Last edited by m-fila

   Member

   More actions

   I took a look and compared Airspeed velocity and a few alternatives but forgot to post my results. IIRC with noise on GitHub runner it was impossible to see a difference (or sometimes it even reported the faster code to be 5% slower) between find_min using LoopVecotrization and #151 for the benchmarks as in examples/benchmark.sh but perhaps setting a much bigger sample could help?? I'll revisit this later
4. m-fila commented on Jun 20, 2025

   

   m-fila

   on Jun 20, 2025

   Member

   More actions

   I checked:

   • AirspeedVelocity - Bechmark in CI with AirspeedVelocity m-fila/JetReconstruction.jl#4 - I think it's quite nice although young. The action runs code from PR and upstream and make a comparison. The results can be posted either as job summary or comment, but each julia version creates a separate comment so it's quite spammy and having different architectures overwrites the comment - job summary doesn't have these problems but is less visible.
   • BenchmarkCI.jl - Bechmark in CI with BenchmarkCI m-fila/JetReconstruction.jl#5 - this seems to be abandoned, also works only on x86_64 for whatever reason. The reports produced are posted as PR comments and are very verbose
   • github-action-benchmark - Benchmark in CI with action-benchmark m-fila/JetReconstruction.jl#6 - This is by far the most mature one out of the comparison and also not Julia-specific. The difference here is that the results for commits are stored in the repository and the action compares benchmark with the last one stored from repository - I don't like this conceptually because then we are comparing results from different workers. But it's very customizable and allows to print reports either as comments or in action summary or even set a threshold and post comment or fail job if the performance drops below the threshold

   At least AirspeedVelocity is expecting the benchmarks to be run to be already present in the master so I did this in my fork. The bechmarks were cases from examples/benchmark.sh adapted to use BenchmarkTools.jl - I'll still have to check if changing the number of samples/tuning allows to see a few percent difference in the performance.
5. m-fila commented on Sep 14, 2026

   

   m-fila

   on Sep 14, 2026

   Member

   More actions

   Just to keep track, there are some more alternatives like Tachometer.jl and PerfChecker