Skip to content

Add CI Benchmark Suite#493

Open
dorephysicsresearcher wants to merge 1 commit into
qojulia:masterfrom
dorephysicsresearcher:master
Open

Add CI Benchmark Suite#493
dorephysicsresearcher wants to merge 1 commit into
qojulia:masterfrom
dorephysicsresearcher:master

Conversation

@dorephysicsresearcher

Copy link
Copy Markdown

Resolves #407.

This PR introduces an automated benchmark suite utilizing BenchmarkTools.jl. It tests the Schroedinger, Lindblad, and MCWF time evolution solvers. A GitHub Action workflow has been added to automatically trigger these benchmarks on new commits to track performance regressions.

@github-actions

Copy link
Copy Markdown
Contributor

Benchmark Results (Julia v1)

Time benchmarks
master 55154eb... master / 55154eb...
master/base array types/1//2 29.3 ± 6.8 μs 29.4 ± 6.9 μs 0.995 ± 0.33
master/base array types/100//1 24.3 s 24.9 s 0.977
master/base array types/20//1 0.0463 ± 0.00075 s 0.0466 ± 0.0008 s 0.994 ± 0.023
master/base array types/50//1 1.37 ± 0.0057 s 1.38 ± 0.009 s 0.989 ± 0.0077
master/qo types/1//2 0.0328 ± 0.0073 ms 0.0334 ± 0.0072 ms 0.981 ± 0.3
master/qo types/100//1 20.9 s 21.2 s 0.988
master/qo types/20//1 0.04 ± 0.0013 s 0.04 ± 0.0011 s 1 ± 0.043
master/qo types/50//1 1.2 ± 0.021 s 1.22 ± 0.0098 s 0.978 ± 0.019
schroedinger/base array types/1//2 5.8 ± 0.32 μs 5.77 ± 0.32 μs 1 ± 0.078
schroedinger/base array types/100//1 3.33 ± 0.04 ms 3.32 ± 0.037 ms 1 ± 0.016
schroedinger/base array types/20//1 0.137 ± 0.0041 ms 0.137 ± 0.0041 ms 1 ± 0.042
schroedinger/base array types/50//1 0.705 ± 0.0098 ms 0.704 ± 0.0099 ms 1 ± 0.02
schroedinger/qo types/1//2 6.36 ± 0.36 μs 6.32 ± 0.35 μs 1.01 ± 0.08
schroedinger/qo types/100//1 3.09 ± 0.023 ms 3.03 ± 0.023 ms 1.02 ± 0.011
schroedinger/qo types/20//1 0.214 ± 0.005 ms 0.213 ± 0.0053 ms 1 ± 0.034
schroedinger/qo types/50//1 0.758 ± 0.0081 ms 0.746 ± 0.0084 ms 1.02 ± 0.016
stochastic master/base array types/1//2 0.244 ± 0.045 ms 0.237 ± 0.047 ms 1.03 ± 0.28
stochastic master/base array types/100//1 0.483 ± 0.022 s 0.467 ± 0.012 s 1.03 ± 0.053
stochastic master/base array types/20//1 31.3 ± 2.8 ms 28.8 ± 2.1 ms 1.09 ± 0.13
stochastic master/base array types/50//1 0.134 ± 0.0038 s 0.12 ± 0.0079 s 1.11 ± 0.08
stochastic master/qo types/1//2 0.286 ± 0.037 ms 0.276 ± 0.038 ms 1.03 ± 0.19
stochastic master/qo types/100//1 0.73 ± 0.016 s 0.706 ± 0.01 s 1.03 ± 0.028
stochastic master/qo types/20//1 25.4 ± 1.9 ms 25.5 ± 1.9 ms 0.996 ± 0.1
stochastic master/qo types/50//1 0.159 ± 0.0015 s 0.157 ± 0.0042 s 1.01 ± 0.029
stochastic schroedinger/base array types/1//2 0.0476 ± 0.00077 ms 0.0449 ± 0.0011 ms 1.06 ± 0.031
stochastic schroedinger/base array types/100//1 1.37 ± 0.1 ms 1.42 ± 0.1 ms 0.967 ± 0.1
stochastic schroedinger/base array types/20//1 0.267 ± 0.0061 ms 0.267 ± 0.0065 ms 0.998 ± 0.033
stochastic schroedinger/base array types/50//1 0.643 ± 0.01 ms 0.644 ± 0.011 ms 0.999 ± 0.023
stochastic schroedinger/qo types/1//2 0.0535 ± 0.0052 ms 0.0532 ± 0.0051 ms 1 ± 0.14
stochastic schroedinger/qo types/100//1 1.49 ± 0.22 ms 1.46 ± 0.22 ms 1.02 ± 0.21
stochastic schroedinger/qo types/20//1 0.334 ± 0.0088 ms 0.335 ± 0.0086 ms 0.998 ± 0.037
stochastic schroedinger/qo types/50//1 0.774 ± 0.026 ms 0.764 ± 0.036 ms 1.01 ± 0.059
time_to_load 2.96 ± 0.011 s 2.98 ± 0.038 s 0.993 ± 0.013
Memory benchmarks
master 55154eb... master / 55154eb...
master/base array types/1//2 0.52 k allocs: 29.3 kB 0.52 k allocs: 29.3 kB 1
master/base array types/100//1 0.135 M allocs: 11.6 GB 0.135 M allocs: 11.6 GB 1
master/base array types/20//1 7.96 k allocs: 29.6 MB 7.96 k allocs: 29.6 MB 1
master/base array types/50//1 0.0375 M allocs: 0.817 GB 0.0375 M allocs: 0.817 GB 1
master/qo types/1//2 0.647 k allocs: 31.8 kB 0.647 k allocs: 31.8 kB 1
master/qo types/100//1 0.155 M allocs: 11.7 GB 0.155 M allocs: 11.7 GB 1
master/qo types/20//1 9.19 k allocs: 29.9 MB 9.19 k allocs: 29.9 MB 1
master/qo types/50//1 0.043 M allocs: 0.819 GB 0.043 M allocs: 0.819 GB 1
schroedinger/base array types/1//2 0.069 k allocs: 3.78 kB 0.069 k allocs: 3.78 kB 1
schroedinger/base array types/100//1 0.085 k allocs: 0.0531 MB 0.085 k allocs: 0.0531 MB 1
schroedinger/base array types/20//1 0.069 k allocs: 14.3 kB 0.069 k allocs: 14.3 kB 1
schroedinger/base array types/50//1 0.069 k allocs: 31 kB 0.069 k allocs: 31 kB 1
schroedinger/qo types/1//2 0.08 k allocs: 4.28 kB 0.08 k allocs: 4.28 kB 1
schroedinger/qo types/100//1 0.094 k allocs: 0.0474 MB 0.094 k allocs: 0.0474 MB 1
schroedinger/qo types/20//1 0.08 k allocs: 13.5 kB 0.08 k allocs: 13.5 kB 1
schroedinger/qo types/50//1 0.08 k allocs: 28.1 kB 0.08 k allocs: 28.1 kB 1
stochastic master/base array types/1//2 3.85 k allocs: 0.207 MB 3.85 k allocs: 0.207 MB 1
stochastic master/base array types/100//1 2.69 k allocs: 0.218 GB 2.69 k allocs: 0.218 GB 1
stochastic master/base array types/20//1 4.31 k allocs: 15.4 MB 4.31 k allocs: 15.4 MB 1
stochastic master/base array types/50//1 3.11 k allocs: 0.0643 GB 3.09 k allocs: 0.064 GB 1
stochastic master/qo types/1//2 4.81 k allocs: 0.227 MB 4.81 k allocs: 0.227 MB 1
stochastic master/qo types/100//1 5.46 k allocs: 0.389 GB 5.46 k allocs: 0.389 GB 1
stochastic master/qo types/20//1 5.46 k allocs: 16.8 MB 5.46 k allocs: 16.8 MB 1
stochastic master/qo types/50//1 5.46 k allocs: 0.0984 GB 5.46 k allocs: 0.0984 GB 1
stochastic schroedinger/base array types/1//2 0.072 k allocs: 3.94 kB 0.072 k allocs: 3.94 kB 1
stochastic schroedinger/base array types/100//1 0.265 k allocs: 0.0649 MB 0.265 k allocs: 0.0649 MB 1
stochastic schroedinger/base array types/20//1 0.072 k allocs: 14.4 kB 0.072 k allocs: 14.4 kB 1
stochastic schroedinger/base array types/50//1 0.072 k allocs: 31.2 kB 0.072 k allocs: 31.2 kB 1
stochastic schroedinger/qo types/1//2 0.403 k allocs: 19.5 kB 0.403 k allocs: 19.5 kB 1
stochastic schroedinger/qo types/100//1 0.419 k allocs: 0.0684 MB 0.419 k allocs: 0.0684 MB 1
stochastic schroedinger/qo types/20//1 0.403 k allocs: 30 kB 0.403 k allocs: 30 kB 1
stochastic schroedinger/qo types/50//1 0.403 k allocs: 0.0456 MB 0.403 k allocs: 0.0456 MB 1
time_to_load 0.149 k allocs: 11.1 kB 0.149 k allocs: 11.1 kB 1

@dorephysicsresearcher

Copy link
Copy Markdown
Author

I did notice that the quantumoptics-tests-amdgpu check failed, but since this PR only adds the benchmark script and the workflow file, I'm guessing it might be an unrelated flaky test or a timeout on the server side.

Could a maintainer take a look or re-trigger the checks when you have a minute? Let me know if you need me to make any tweaks to the benchmark script!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update the benchmark suite and bring it into the CI runner [$400]

1 participant