Monte Carlo (trades): calmar_ratio uses total return while the 'original' column uses CAGR, making the two incomparable

[ernestodeoliveira]

Summary

In the trades-based Monte Carlo, the calmar_ratio shown in the original column and the one shown in the worst_5 / median / best_5 columns are computed with two different formulas. They end up side by side in the same summary row, so the table reads as if reshuffling the trade order improves Calmar by an order of magnitude.

Version: jesse 2.5.0.

Where

The backtest metric is annualized (CAGR / MaxDD) — jesse/services/metrics.py:120:

def calmar_ratio(returns):
    """
    Calculates the calmar ratio (CAGR% / MaxDD%)
    """

The Monte Carlo scenario metric uses cumulative total return instead — jesse/research/monte_carlo/monte_carlo_trades.py:317:

calmar_ratio = total_return / abs(max_drawdown) if max_drawdown < 0 else 0

The original column does not go through that function. At monte_carlo_trades.py:398, calmar_ratio falls into the else branch and is taken straight from the backtest metrics:

original_value = original_metrics.get(metric_name, 0)

So original is CAGR-based while every percentile is total-return-based.

Reproduction

Strategy on Binance Perpetual Futures BTC-USDT 1D, 2020-06-01 → 2026-07-29, 500 trades scenarios. Reported summary:

metric	original	worst_5	median	best_5
total_return	1272.17	1272.17	1272.17	1272.17
max_drawdown	-21.07	-68.07	-21.18	-7.88
calmar_ratio	2.51	18.69	60.07	161.39

The percentile values are exactly total_return / abs(max_drawdown):

• 1272.17 / 21.18 = 60.06 → reported median 60.07
• 1272.17 / 68.07 = 18.69 → reported worst_5 18.69
• 1272.17 / 7.88 = 161.44 → reported best_5 161.39

Over this ~6-year window the inflation is roughly 24× relative to the original value.

Why it matters

1. The summary table is misleading. A reader compares original 2.51 against median 60.07 and concludes the historical ordering was extraordinarily unlucky. The two numbers are not comparable at all.

2. The Calmar p-value is always 1.0. At monte_carlo_trades.py:414:

if metric_name in ['total_return', 'sharpe_ratio', 'calmar_ratio']:
    p_value = np.sum(values_array >= original_value) / len(values_array)

Every scenario Calmar is on the inflated scale, so it always exceeds the CAGR-based original. The test can never reject, for any strategy.

3. The metric carries no information in this mode. Reshuffling trade order does not change the compounded final equity, so total_return is constant across scenarios (visible in the table above). The numerator is fixed, which makes the scenario Calmar a pure restatement of drawdown rather than an independent metric.

Suggested fix

Compute the scenario Calmar the same way metrics.py does — annualize the return over the scenario's own time span before dividing by max drawdown — so original and the percentiles share a definition. Alternatively, derive the original value with the scenario formula so both columns are at least internally consistent.

Possibly related: sharpe_ratio in the same mode

_calculate_volatility_metrics (monte_carlo_trades.py:360) annualizes with ANNUALIZATION_FACTOR = 365 (common.py:19, commented "Trading days per year"):

annualized_volatility = daily_std * np.sqrt(ANNUALIZATION_FACTOR)

But the equity curve it consumes is rebuilt from trades (_reconstruct_equity_curve_from_trades), so each consecutive pair is one trade, not one day. In the run above that is 64 points across ~6 years — average spacing ~35 days — while the factor assumes daily spacing. The resulting Sharpe is not on the same scale as the backtest's original either (median 0.76 vs original 1.71 here).

I have not verified this one as thoroughly as the Calmar issue, so I am flagging it rather than asserting it. Happy to open it separately if you prefer.

[Jaco-Ren]

I’m working on a focused fix for this issue, keeping it separate from #608. The patch will align scenario Calmar with the backtest definition by annualizing the equity-curve return over its timestamp span, with a pure regression test. @ernestodeoliveira, please let me know if you already have a patch in progress and I’ll coordinate.

[ernestodeoliveira]

I’m working on a focused fix for this issue, keeping it separate from #608. The patch will align scenario Calmar with the backtest definition by annualizing the equity-curve return over its timestamp span, with a pure regression test. @ernestodeoliveira, please let me know if you already have a patch in progress and I’ll coordinate.

Hi @Jaco-Ren, thanks for picking this up — no patch in progress on my side for #606 (I held off deliberately because it touches monte_carlo_trades.py, the same file as PR #608, so whichever lands first will force a rebase of the other — worth keeping in mind).

Two notes on scope: (1) the original value comes from metrics.py which annualizes over the full backtest window, so the scenario CAGR should use that same span rather than first-to-last trade timestamps (the reconstructed equity curve is per-trade, not per-day); (2) the sharpe_ratio annualization issue mentioned in this thread is a separate defect — fine to leave it out of your focused fix.