ValueError: 'subset_size' must be smaller than input size. [1 >= 1.0] in FaithfulnessCorrelation for TABULAR DATA

[officialarijit]

Description:
I encountered a ValueError when running the FaithfulnessCorrelation metric in Quantus. The error message indicates that subset_size must be smaller than the input size, but even after setting it to a reasonable value, the issue persists.

Error Message:

ValueError: 'subset_size' must be smaller than input size. [1 >= 1.0]

Steps to Reproduce:

1. Load a tabular dataset.
2. Train a TensorFlow model on the dataset.
3. Compute SHAP explanations for the model.
4. Pass the model, input data, labels, and explanations to FaithfulnessCorrelation.
5. Observe the error.

Expected Behavior:
The metric should execute without errors if subset_size is properly defined within the valid range.

Code Snippet:

import quantus as q
import numpy as np
import pandas as pd
import tensorflow as tf
import shap

# Load dataset
df = pd.read_csv("your_data.csv")
X = df.drop(columns=["target"]).values  # Features
y = df["target"].values  # Target

# Load model
model = tf.keras.models.load_model("your_model.h5")

# Generate SHAP explanations
explainer = shap.Explainer(model, X)
shap_values = explainer(X)
explanations = shap_values.values

# Fix subset_size issue
num_features = X.shape[1]
subset_size = max(1, min(num_features - 1, int(num_features * 0.5)))

faithfulness_metric = q.FaithfulnessCorrelation(
    nr_runs=100,
    subset_size=subset_size,
    perturb_func=q.perturb_func.feature_removal_by_indices,
    similarity_func=q.similarity_func.correlation_pearson,
    abs=False,
    return_aggregate=False
)

# Compute Faithfulness Correlation
faithfulness_scores = []
for x_batch, y_batch, a_batch in zip(x_batches, y_batches, a_batches):
    score = faithfulness_metric(model=model, x_batch=x_batch, y_batch=y_batch, a_batch=a_batch)
    faithfulness_scores.append(score)

print("Faithfulness Correlation Scores per Batch:", faithfulness_scores)

Environment:

• OS: Ubuntu 22.04
• Python: 3.11
• TensorFlow: 2.3
• Quantus: Latest
• SHAP: Latest

[craigybaeb]

Hi there @officialarijit , I am facing the same issue. Were you able to resolve this? Thanks