ROAD returns 0 all the time

[krebso]

Description

Hello there, as the title says - looks like ROAD returns 0 all the time. In the one of provided tutorials, the output of ROAD is 0 for all percentages. In my own experiments, the metric acts the same.

Steps to reproduce the behavior

• Initialize road metric
• Evaluate it on arbitrary data
• I would expect to at least some of the percentage to display different result than 0
• Explanations were generated using captum's LRP, I tried for Occlusion as well, same result
• Screenshot from local development
  

Code for easier reproducibility:

import quantus
road = quantus.metrics.ROAD(
    percentages=[10, 15, 20, 25, 30],
    display_progressbar=True,
)
road(
    model=model.model,
    x_batch=x.cpu().numpy(),
    y_batch=y.cpu().numpy(),
    a_batch=lrp_attributions.sum(axis=1).cpu().numpy(),
    softmax=True,
    device="cuda",
)

Minimum acceptance criteria

• Please double check the metric whether it works as supposed. It is weird to consistently return 0 for all inputs
• Random tag because the description guideline says so: @annahedstroem

Thanks in advance!

[krebso]

I just tested implementation from pytorch-grad-cam and it expectedly reports the non-zero values.

[krebso]

@annahedstroem were you able to confirm?

[annahedstroem]

hi @krebso ! thanks so much for this. we'll look into what is causing this.

what dataset did you use for this? would it be possible for you to try ROAD with a low-dim dataset?