#586: Investigate erratic and unpredictable CI error regarding synthetic case (NEW)

[cwschilly]

Fixes #586

[ppebay]

Running 100 times the corresponding case (with only 8 iterations and 1 object per transfer):

for run in {1..100}; do python LBAF_app.py -c synthetic-acceptance.yaml; (cat ../../../output/imbalance.txt; echo) >> res; done

on macOS 14.3 with Python 3.9.19 yields:

awk '{ sum += $1 } END { print sum }' res
0

In other all 100 runs correctly converged to an imbalance of 0.0.

This seems to indicate rather a problem with the CI system itself, and specifically with Python 3.8
@cwschilly

[ppebay]

In keeping with the above comment, I created a Python 3.8 environment and performed the same test (100 runs):

for run in {1..100}; do python LBAF_app.py -c synthetic-acceptance.yaml; (cat ../../../output/imbalance.txt; echo) >> res; done

and obtained as well

awk '{ sum += $1 } END { print sum }' res
0

i.e., all 100 runs passed the test.

Specifically the version of Python is:

 python --version
Python 3.8.20

@cwschilly can we verify if this is the same version that runs in CI?

[cwschilly]

@ppebay The CI uses Python 3.8.18. I have been able to recreate the problem locally by running tox with Python 3.8.19, so I don't think it's just a problem with CI.

[cwschilly]

After changes to the ordering of ranks, Python 3.8 acceptance passed several times in a row. However, Python 3.11 failed: https://github.com/DARMA-tasking/LB-analysis-framework/actions/runs/13397121375/job/37418861102?pr=590

[ppebay]

Approving and resolving conflicts

[ppebay]

@cwschilly do you mind checking the changes I made to this PR? I can't make you a reviewer of your own PR.

[cwschilly]

@ppebay All changes look good, although I never succeeded in identifying the underlying issue

[ppebay]

Thanks, and I am thinking of modifying the test slightly to ensure convergence in all cases (even probabilistically).

[ppebay]

As part of this PR the upper-bound update formulae for memory footprint, valid for both node-based (e.g. max 36 per node):

and rank-based memory constraints (e.g. max 18 per rank):

now correct.