benchmark

MicroTaint test artifacts

In this sub directory are all the test files run and used to run the benchmarks mentionned in the MicroTaint submission.

General comparison benchmark

You need to go into the precision_soundess directory to run these commands

The overall benchmark needs a few tools to work, the script that sets them all up is setup-envs.sh

In order to install all the different tools in their correct version, a working uv, git and docker will be needed.

Once both work you can run

./setup-envs.sh

It will build the docker images, download all the missing files and install the dependencies in the local dir in which you are in.

Once this is done you can run the benchmark we ran in the submission with the following command

BATCH_TIMEOUT=18000 uv run benchmark.py  --number 7500   --sequences 1000  --sweep  --all-suites  --quiet  --seed 12 -w taintgrind,libdft64,microtaint,triton,maat,angr,panda 2>&1 | tee benchmark_$(date +%Y%m%d_%H%M%S).log

The BATCH_TIMEOUT can be adjusted 18000 corresponds to the number of seconds in 5 hours.

The slow engines amongst these are panda,maat, libdft64, taintgrind

You can adjust the engines used through the -w flag, to change the number of bits used in the ground truth threshold then you can adjust the variable GT_BIT_BUDGET on line 218.

During the submission 25 tests were skipped as they triggered a bug of state leakage in the ground truth emulator as well as the fallback unicorn emulator of MicroTaint. We did not have enough time to debug and fix the issue before the submission deadline. It is now fixed. If you want to get the exact same result as the paper, then you need to change the variable SUBMISSION to True line 65.

If you are wondering why MicroTaint uses unicorn on these specific tests: it is because the instructions that were complex programs (such as using a CBRANCH or a for loop in them) used unicorn so as to keep our pcode simulator at minimal complexity. This explains the p99 of microtaint which is so different from the p50, it concerns these instructions that fallback to unicorn. The CBRANCH limitation will be fixed shortly.

Overhead

This set of tests is in the overhead directory The command to run is

uv run overhead_bench.py --build-bench bench.c --gen-input 256 --runs 100 --only native --only qiling-only --only microtaint-all --native-timeout 5 --qiling-timeout 120 --microtaint-timeout 1800 --json overhead_results.json

nftables bug

The nftable bug and it's associated README.md can be found in the nftables subdir. The README is AI generated as well as the reproduction script, they have been checked and validated by the authors. All the instructions for reproducing the test are in this subdir.

DNS header parsing

The file is in the dns subdirectory. The file is commented and prints many debbug information, so no extra information will be provided here. Running it is simply

uv run dns_experiment.py

Cryptography

The test files, the associated README for all these tests are present in the crypto sub directory. Once again most of the test scripts were generated by AI but checked by the authors.

Square and multiply

The file to check square and multiply is check_side_channel.py. Simply run uv run check_side_channel.py.

Siphash

To show that we always overtaint the Siphash we run it on many runs of the algorithm with the eval_siphash_avalanche.py This shows that with taint tracking we cannot detect that we are xoring a bit to itself as the zeroing idiom is only detected when it is in an instruction, we cannot detect such bit dependencies. This is future line of work for taint engines being used in cryptographic purposes.