Significant compilation time increase with `--llzk concrete --llzk_plaintext`

[alexcere]

We have noticed significant compilation slowdowns when compiling the circuits in the circomlib with the llzk option enabled compared to the default pipeline. We are using the following commits:

• circom: https://github.com/project-llzk/circom/tree/de0cbb3d34524ba26035924b3d6c907b3749ad5c
• llzk: https://github.com/project-llzk/llzk-rs#0a320c48d96ff2352af598407e476056021e60fe

Below are the average compilation times and standard deviations after 5 executions. The table compares the default pipeline with no flags enabled (Plain* results) against the pipeline using the flags --llzk concrete --llzk_plaintext (Concrete* results), sorted by the largest average time difference (Diff (s)):

File	Concrete avg (s)	Concrete std	Plain avg (s)	Plain std	Diff (s)
poseidonex_test.circom	51.7255	1.8523	0.6362	0.0205	51.0893
smtprocessor10_test.circom	39.4369	2.5737	0.6428	0.0649	38.7941
eddsaposeidon_test.circom	38.905	2.0762	0.8399	0.0846	38.065
poseidon6_test.circom	37.3933	2.4227	0.3866	0.0561	37.0066
smtverifier10_test.circom	34.8534	0.3879	0.4944	0.0133	34.3591
poseidon3_test.circom	33.2177	1.8266	0.3176	0.0464	32.9001
pedersen_test.circom	15.1396	0.6857	1.6956	0.2112	13.444
sha256_2_test.circom	9.7156	0.6344	2.3768	0.1277	7.3388
sha256_test512.circom	10.8813	0.6264	3.7449	0.2311	7.1364
escalarmul_test.circom	8	0.3983	0.9132	0.1386	7.0868
sha256_test448.circom	10.4752	0.8304	3.4998	0.1767	6.9754
escalarmul_min_test.circom	7.6751	0.2403	0.7601	0.0418	6.915
escalarmul_test_min.circom	7.6009	0.159	0.765	0.058	6.8359
eddsa_test.circom	2.641	0.0939	0.7936	0.0859	1.8473
eddsamimc_test.circom	1.8469	0.1425	0.4823	0.0583	1.3646
escalarmulfix_test.circom	0.952	0.0533	0.194	0.0046	0.758
babypbk_test.circom	0.9166	0.0852	0.2268	0.0103	0.6898
pedersen2_test.circom	0.8485	0.0648	0.1923	0.0201	0.6562
escalarmulany_test.circom	0.7133	0.101	0.2586	0.0821	0.4546
mimc_sponge_hash_test.circom	0.4778	0.0553	0.082	0.0125	0.3957
mimc_sponge_test.circom	0.3815	0.0291	0.0506	0.0037	0.331
mux4_1.circom	0.2259	0.0145	0.045	0.0101	0.1809
pointbits_loopback.circom	0.4375	0.0571	0.2718	0.027	0.1657
mimc_test.circom	0.1759	0.0111	0.0273	0.0032	0.1486
constants_test.circom	0.1333	0.0179	0.0257	0.0027	0.1076
mux3_1.circom	0.132	0.0089	0.0355	0.004	0.0965
mux2_1.circom	0.1051	0.0044	0.0334	0.0048	0.0717
sum_test.circom	0.1142	0.0206	0.0613	0.0188	0.0529
binsub_test.circom	0.0887	0.0073	0.0396	0.006	0.0491
sign_test.circom	0.1103	0.0184	0.0621	0.0129	0.0482
mux1_1.circom	0.0766	0.0033	0.032	0.0052	0.0446
escalarmulw4table_test.circom	0.0652	0.0079	0.0232	0.0043	0.042
escalarmulw4table.circom	0.0609	0.0047	0.0194	0.0026	0.0415
aliascheck_test.circom	0.1032	0.0081	0.0633	0.0051	0.0399
escalarmulw4table_test3.circom	0.0609	0.0026	0.0211	0.0034	0.0398
greatereqthan.circom	0.0588	0.0082	0.0303	0.0039	0.0284
greaterthan.circom	0.0546	0.0072	0.027	0.0015	0.0276
lesseqthan.circom	0.0622	0.0019	0.0373	0.005	0.0248
montgomerydouble.circom	0.035	0.0017	0.0189	0.0027	0.0161
lessthan.circom	0.0622	0.015	0.0464	0.0251	0.0158
babyadd_tester.circom	0.0631	0.0093	0.0486	0.0071	0.0145
isequal.circom	0.0438	0.0066	0.0306	0.0045	0.0132
edwards2montgomery.circom	0.0315	0.0052	0.0203	0.0033	0.0112
montgomeryadd.circom	0.0465	0.0149	0.0374	0.0103	0.0091
montgomery2edwards.circom	0.028	0.0033	0.02	0.0043	0.008
babycheck_test.circom	0.0532	0.0039	0.0456	0.0042	0.0076
iszero.circom	0.043	0.0112	0.0435	0.0174	-0.0004

Conversely, removing the --llzk_plaintext flag drastically improves compilation times, though they remain slower than the default pipeline:

File	Concrete avg (s)	Concrete std	Plain avg (s)	Plain std	Diff (s)
poseidonex_test.circom	8.1916	0.3461	0.7397	0.0701	7.4519
eddsaposeidon_test.circom	7.4566	0.5984	0.9794	0.1456	6.4772
poseidon6_test.circom	6.7975	0.4075	0.528	0.0991	6.2695
smtprocessor10_test.circom	6.9465	1.1708	0.7938	0.2361	6.1527
poseidon3_test.circom	6.172	0.3808	0.3635	0.0544	5.8085
smtverifier10_test.circom	5.5381	0.1263	0.541	0.0369	4.9972
pedersen_test.circom	3.328	0.3637	1.7848	0.1611	1.5431
sha256_test512.circom	5.4597	0.9642	4.0876	0.7881	1.3721
sha256_2_test.circom	4.4663	0.4451	3.1329	0.3626	1.3334
sha256_test448.circom	4.5044	0.0783	3.5049	0.0799	0.9994
escalarmul_test_min.circom	1.5701	0.1334	0.8627	0.0315	0.7074
escalarmul_min_test.circom	1.5578	0.1636	0.8529	0.0466	0.7049
escalarmul_test.circom	1.6332	0.1034	1.0312	0.1007	0.602
eddsamimc_test.circom	0.8616	0.0897	0.552	0.0658	0.3095
eddsa_test.circom	1.1462	0.0585	0.9216	0.0852	0.2247
pedersen2_test.circom	0.368	0.027	0.2162	0.0135	0.1519
babypbk_test.circom	0.3898	0.0638	0.2451	0.0259	0.1446
escalarmulfix_test.circom	0.3567	0.0496	0.2272	0.0211	0.1295
escalarmulany_test.circom	0.3488	0.0374	0.2196	0.0218	0.1292
mimc_sponge_hash_test.circom	0.2075	0.0196	0.1004	0.0135	0.107
mimc_sponge_test.circom	0.1299	0.0091	0.056	0.0082	0.0739
pointbits_loopback.circom	0.3573	0.0505	0.298	0.0389	0.0592
mimc_test.circom	0.094	0.0208	0.0414	0.0084	0.0526
mux4_1.circom	0.0866	0.0057	0.0456	0.0064	0.041
constants_test.circom	0.0464	0.0038	0.0208	0.0027	0.0256
mux3_1.circom	0.0635	0.0049	0.0386	0.0009	0.0249
sign_test.circom	0.0803	0.0123	0.0595	0.0036	0.0208
mux1_1.circom	0.0544	0.0075	0.0351	0.0038	0.0192
mux2_1.circom	0.0606	0.009	0.0433	0.0089	0.0173
binsub_test.circom	0.0554	0.0057	0.041	0.0058	0.0144
montgomeryadd.circom	0.0412	0.0112	0.0281	0.0046	0.0131
sum_test.circom	0.0814	0.0054	0.0685	0.0079	0.0129
escalarmulw4table_test.circom	0.0382	0.0072	0.0261	0.0064	0.0121
lesseqthan.circom	0.0487	0.0029	0.0378	0.0038	0.0108
greatereqthan.circom	0.0443	0.0027	0.0336	0.0021	0.0107
escalarmulw4table_test3.circom	0.0361	0.0031	0.0254	0.0032	0.0107
edwards2montgomery.circom	0.0317	0.0043	0.0219	0.0034	0.0098
greaterthan.circom	0.0456	0.0034	0.0365	0.0021	0.0091
aliascheck_test.circom	0.0731	0.0044	0.0646	0.0057	0.0085
montgomerydouble.circom	0.0308	0.0035	0.0229	0.0034	0.0079
isequal.circom	0.0424	0.0062	0.0347	0.0036	0.0077
escalarmulw4table.circom	0.045	0.011	0.0385	0.017	0.0065
babycheck_test.circom	0.0548	0.0066	0.0497	0.0072	0.0052
lessthan.circom	0.0427	0.0073	0.0388	0.0076	0.0039
iszero.circom	0.0374	0.0013	0.0342	0.0043	0.0033
babyadd_tester.circom	0.0566	0.0088	0.054	0.0061	0.0027
montgomery2edwards.circom	0.0271	0.0011	0.0248	0.004	0.0023

The raw data can be found in the following files:

• compile_times_no_plain.csv
• compile_times_plain.csv

[tim-hoffman]

@alexcere thanks for reporting this!
I have a couple PRs in progress that speed up some aspects that were hot spots in profiling I performed.
#446
project-llzk/llzk-lib#612
The latter only helps when generating bytecode, not plaintext, but the former helps in both cases.

[tim-hoffman]

@alexcere the fixes are merged to the llzk branch. On my machine, the poseidonex_test baseline is around 0.33s, concrete llzk around 0.78s, and the concrete llzk plaintext around 2.21s. I'd be interested to see your updated numbers for all tests.

[alexcere]

@tim-hoffman thank you for addressing the issue!

I have re-run the experiments with the new branches:

• circom: https://github.com/project-llzk/circom/tree/2d4d27b06808b6741dd76f7ddb61d5f35b46f11a
• llzk-rs: https://github.com/project-llzk/llzk-rs/tree/837c14f06a32d7be5aa26d94aaa0f7ec075807c0
• llzk-lib: https://github.com/project-llzk/llzk-lib/tree/ab2a300378db7c538d1858ef976d79753c7c50af

and for the version with --llzk_plaintext, the times have basically decreased significantly, although not as much as in your findings. I imagine this difference has to do with the machine used.

File	Concrete avg (s)	Concrete std	Plain avg (s)	Plain std	Diff (s)
smtverifier10_test.circom	32.3971	1.4844	0.6011	0.091	31.7959
smtprocessor10_test.circom	32.3445	1.3032	0.5869	0.0519	31.7575
poseidonex_test.circom	31.5946	1.0068	0.8555	0.1751	30.7391
eddsaposeidon_test.circom	30.9924	0.9731	0.7896	0.1052	30.2028
poseidon3_test.circom	29.3376	1.0108	0.2839	0.0328	29.0537
poseidon6_test.circom	26.1377	0.9038	0.3276	0.0309	25.8101
pedersen_test.circom	14.1246	0.5483	1.4205	0.08	12.704
escalarmul_test.circom	7.4333	0.4779	0.8124	0.1233	6.6209
sha256_test448.circom	10.139	0.5812	3.5429	0.3986	6.5961
escalarmul_min_test.circom	7.3298	0.4013	0.7831	0.0936	6.5466
escalarmul_test_min.circom	7.1923	0.3011	0.7839	0.1134	6.4083
sha256_test512.circom	9.6768	0.4259	3.3481	0.5275	6.3287
sha256_2_test.circom	7.999	0.163	2.0466	0.0615	5.9524
eddsa_test.circom	2.6784	0.1571	0.805	0.0969	1.8734
eddsamimc_test.circom	1.8587	0.0836	0.4945	0.034	1.3642
escalarmulfix_test.circom	0.9849	0.067	0.2291	0.023	0.7558
babypbk_test.circom	0.9227	0.038	0.2114	0.0277	0.7113
pedersen2_test.circom	0.8218	0.0467	0.1977	0.0335	0.6241
escalarmulany_test.circom	0.6668	0.0577	0.2013	0.0257	0.4655
mimc_sponge_hash_test.circom	0.4395	0.02	0.0783	0.0086	0.3612
mimc_sponge_test.circom	0.3776	0.0219	0.0537	0.0094	0.3238
mux4_1.circom	0.2151	0.0035	0.042	0.0051	0.1731
mimc_test.circom	0.1802	0.0203	0.0287	0.0034	0.1515
pointbits_loopback.circom	0.4064	0.0119	0.2624	0.0451	0.144
mux3_1.circom	0.1341	0.0118	0.0333	0.0038	0.1008
constants_test.circom	0.123	0.0033	0.0229	0.0026	0.1001
mux2_1.circom	0.1008	0.0067	0.0344	0.004	0.0664
binsub_test_13.circom	0.0972	0.0096	0.0387	0.0056	0.0585
binsub_test_2.circom	0.0923	0.0023	0.035	0.0035	0.0573
binsub_test_3.circom	0.0948	0.0046	0.0385	0.003	0.0563
binsub_test.circom	0.0933	0.0071	0.0403	0.0052	0.053
sum_test.circom	0.0938	0.0083	0.0415	0.008	0.0523
mux1_1.circom	0.087	0.0054	0.0356	0.0043	0.0514
escalarmulw4table_test.circom	0.0741	0.0063	0.0227	0.0022	0.0514
aliascheck_test.circom	0.1119	0.0136	0.064	0.0075	0.0478
escalarmulw4table_test3.circom	0.0742	0.0093	0.0265	0.0084	0.0477
escalarmulw4table.circom	0.0736	0.0052	0.0276	0.0047	0.046
sign_test.circom	0.1117	0.0092	0.0738	0.014	0.0379
greatereqthan.circom	0.0679	0.0043	0.0337	0.0037	0.0341
lesseqthan.circom	0.0652	0.004	0.0401	0.0082	0.0251
lessthan.circom	0.0557	0.0033	0.0311	0.0031	0.0245
greaterthan.circom	0.0567	0.0048	0.0325	0.006	0.0242
montgomerydouble.circom	0.0369	0.0038	0.0206	0.0015	0.0162
isequal.circom	0.0526	0.0072	0.0368	0.004	0.0158
montgomeryadd.circom	0.0349	0.0014	0.0209	0.0035	0.014
edwards2montgomery.circom	0.0329	0.0043	0.0217	0.002	0.0112
babyadd_tester.circom	0.068	0.0074	0.0577	0.0048	0.0103
babycheck_test.circom	0.0603	0.0084	0.0501	0.0049	0.0103
iszero.circom	0.0385	0.0024	0.0286	0.0013	0.0099

With the flag --llzk_plaintext disabled, the obtained results are slightly improved as well:

File	Concrete avg (s)	Concrete std	Plain avg (s)	Plain std	Diff (s)
smtprocessor10_test.circom	5.2315	0.4619	0.6865	0.0735	4.5449
poseidonex_test.circom	5.3012	0.3036	0.8084	0.0767	4.4928
smtverifier10_test.circom	5.1014	0.4218	0.6253	0.0423	4.4761
eddsaposeidon_test.circom	5.2876	0.3334	0.8769	0.0765	4.4107
poseidon3_test.circom	4.5392	0.0566	0.35	0.0547	4.1891
poseidon6_test.circom	3.9289	0.2262	0.3624	0.0539	3.5664
pedersen_test.circom	2.8663	0.1982	1.5412	0.0606	1.3251
sha256_test448.circom	4.7442	0.4726	3.641	0.4122	1.1032
sha256_2_test.circom	2.9434	0.1062	2.0579	0.0372	0.8855
sha256_test512.circom	3.9206	0.0955	3.0733	0.0489	0.8473
escalarmul_test_min.circom	1.4889	0.1243	0.8023	0.0752	0.6866
escalarmul_test.circom	1.5716	0.0843	0.9028	0.086	0.6688
escalarmul_min_test.circom	1.4311	0.0773	0.8362	0.0843	0.5949
eddsa_test.circom	1.1903	0.065	0.8415	0.0763	0.3488
eddsamimc_test.circom	0.7542	0.0811	0.5078	0.0379	0.2464
escalarmulfix_test.circom	0.4086	0.0293	0.2264	0.0205	0.1823
babypbk_test.circom	0.3563	0.0335	0.2187	0.0223	0.1377
pedersen2_test.circom	0.3458	0.0216	0.2161	0.0468	0.1297
escalarmulany_test.circom	0.2958	0.0227	0.2102	0.0162	0.0856
mimc_sponge_hash_test.circom	0.1775	0.0203	0.094	0.011	0.0834
mimc_sponge_test.circom	0.1235	0.0126	0.0512	0.0092	0.0724
pointbits_loopback.circom	0.3304	0.0298	0.2772	0.0226	0.0531
mux4_1.circom	0.0864	0.0095	0.0418	0.0052	0.0446
mimc_test.circom	0.0703	0.0097	0.0332	0.0025	0.0371
mux3_1.circom	0.0614	0.0045	0.037	0.0029	0.0244
constants_test.circom	0.0448	0.006	0.0213	0.001	0.0235
mux2_1.circom	0.0571	0.004	0.0354	0.0027	0.0217
binsub_test_2.circom	0.0514	0.0043	0.0333	0.0024	0.0181
binsub_test_3.circom	0.0531	0.0064	0.0367	0.0037	0.0163
escalarmulw4table_test.circom	0.0425	0.0056	0.0269	0.004	0.0156
binsub_test_13.circom	0.0554	0.0069	0.0404	0.0073	0.015
greaterthan.circom	0.0529	0.0097	0.0393	0.0047	0.0137
aliascheck_test.circom	0.0759	0.0038	0.0623	0.0041	0.0136
sum_test.circom	0.0529	0.0039	0.0396	0.0037	0.0132
escalarmulw4table.circom	0.0387	0.0035	0.0261	0.0046	0.0126
binsub_test.circom	0.0518	0.0038	0.0398	0.0032	0.012
montgomeryadd.circom	0.0335	0.0017	0.0232	0.0022	0.0103
montgomery2edwards.circom	0.0358	0.0054	0.0255	0.0038	0.0103
escalarmulw4table_test3.circom	0.0401	0.0058	0.0302	0.0058	0.0098
greatereqthan.circom	0.0464	0.0058	0.0366	0.0057	0.0098
mux1_1.circom	0.0484	0.0023	0.0396	0.0065	0.0088
babyadd_tester.circom	0.0605	0.0142	0.0521	0.0075	0.0085
lesseqthan.circom	0.0453	0.0054	0.0368	0.0038	0.0085
isequal.circom	0.0397	0.0056	0.0318	0.0034	0.0079
montgomerydouble.circom	0.0302	0.0012	0.0228	0.0022	0.0074
sign_test.circom	0.0695	0.0025	0.0621	0.0051	0.0074
edwards2montgomery.circom	0.0262	0.0022	0.0208	0.0021	0.0054
iszero.circom	0.0385	0.005	0.0338	0.0059	0.0047
babycheck_test.circom	0.0512	0.0028	0.0468	0.0045	0.0044
lessthan.circom	0.0404	0.0034	0.0362	0.0055	0.0042

Again, the raw data can be found in the following files:

• compile_times_no_plain_new.csv
• compile_times_plain_new.csv

I will try testing in other machines as well!