Feature request: Coverage in DifferentiationInterfaceTest

[seabbs]

To preface, this may be impossible and I really like DIT as it stands.

Ask

I would like to be able to use DifferentiaionInterfaceTest as my unit testing interface, benchmarking tool, and code coverage generator when creating packages meant to work well with AD (both a specific AD package and for multiple as far as I am aware, none of them has good tools for this). Related package level issue: EpiAware/CensoredDistributions.jl#272

The missing part of this is having an understanding of what is being covered by my DIT scenarios which it seems code coverage is designed for.

The reason I want this is we are moving towards making a larger Julia ecosystem of packages and we really want them on a firm base/with some good practices established before hand.

What we are doing at the moment:

We currently run DIT in our CI per benchmark, in our docs (https://censoreddistributions.epiaware.org/stable/getting-started/tutorials/ad-backends) and in our benchmarking suite from a shared set of scenarios. Early days but very happy with DIT as it stands. The only additional nice to have would be a shared way to test rules but that seems like the real wild west so I imagine is out of scope.

Solution

I don't know. I don't know anything about how the AD version of the code interacts with the original version or if there is enough commonality between backends to make this a thing that is plausible to do across backends.

[gdalle]

Thanks for reaching out! I'm not sure what you mean with your code coverage request. All DIT scenarios do is run your own code through AD systems, so any lines of your own function that get executed will be covered. Perhaps the issue is that your docs and benchmark CI runs don't get added to the statistics computed by Codecov on your final README badge?

[seabbs]

Yes, what I would like to be able to understand the proportion of the code base being tested for AD compatibility and then dive into that to see line-by-line coverage. That should, I think, not be in the general coverage stats but in a second version that is AD specific. Potentially, this could be managed externally to DIT, but even if so I imagine this would be of use to anyone using it and so be something for the docs (perhaps if it is something to get by wrapping people are already doing it and I just haven't found it).

[gdalle]

I had a similar issue with the tests of DI and DIT themselves, and one solution I found is Codecov flags, to isolate and analyze coverage across CI runs. Could this be a good answer for you?

[seabbs]

It sounds like it could be indeed. So I guess this issue becomes documenting that approach for users of DIT?

[gdalle]

You're the first serious user of DIT besides myself, so I'm happy to document anything you find relevant! Want to open the PR?

[seabbs]

I'm just arguing with Claude to implement something using this in CensoredDistributions. Will see where that lands and then can circle back. At that point happy to open a PR.

first serious user

🙈

[seabbs]

It looks like this all works. I have an outstanding to do to improve benchmarking and then think I have something I am reasonably happy with. What format/thing would you like in the PR? Documentation about CI as for the tagbot action, a generator for an action or other?

[gdalle]

Just a couple sentences in the DIT docs about classifying coverage by backend using codecov tags for those user who deem it it necessary? Might belong in the README