Issues about cup

[GoodEureka]

Hello, thank you for your excellent work and for releasing this implementation!

While using the CUP algorithm, I observed that the Lagrange multiplier has an upper bound of 0.2. During training, it very quickly reaches this upper bound. At the same time, the KL divergence constraint triggers the second-stage early stopping mechanism, which causes the second stage to usually update only once per iteration.

As a result, the algorithm cannot effectively reduce the cost, and in practice, CUP seems to degrade directly into PPO during training.

Is this expected behavior, or is it possible that there is an implementation detail or hyperparameter that I should adjust?

Thanks again for your contribution and looking forward to your insights!
Here is my config.json and result figure
{
"batch_size": 64,
"cost_limit": 25.0,
"device": "cpu",
"device_id": 0,
"exp_name": "SafetyPointGoal1-v0-cup-seed-0",
"experiment": "single_agent_exp",
"gamma": 0.99,
"headless": false,
"hidden_sizes": [
64,
64
],
"lagrangian_multiplier_init": 0.001,
"lagrangian_multiplier_lr": 0.035,
"learning_iters": 40,
"log_dir": "../runs/single_agent_exp/SafetyPointGoal1-v0/cup/seed-000-2025-09-24-13-42-40",
"max_grad_norm": 40.0,
"num_envs": 10,
"randomize": false,
"seed": 0,
"steps_per_epoch": 20000,
"target_kl": 0.02,
"task": "SafetyPointGoal1-v0",
"total_steps": 10000000,
"use_eval": false,
"write_terminal": true
}