Add criterion-aware Generation RL and multi-advantage Reasoning RL

[dlgjr]

Generation RL

• keep criterion-normalized dynamic-rubric shaping and the existing acceptance gate
• keep Generation isolated from Reasoning gold injection

Reasoning RL

1. Structured interleaved rollout

Reasoning rollouts use <think> ... <perception page="N">...</perception> ... </think><answer>...</answer>.
<perception> marks an explicit perception action; it is not treated as a hard visual-token class.

2. Hard construction-aware verification

reasoning_process_verifier.py reuses Finance World evidence facts and executable-program metadata to check:

• visual fact value/page grounding
• perception sufficiency, productivity and redundancy
• arithmetic validity and reasoning-step coverage
• first verifiable error position

Explicit process contradictions veto an otherwise correct terminal answer.

3. True multi-advantage optimization

Reasoning no longer collapses outcome / perception / reasoning into one scalar reward.
For mixed Pass@8 groups the trainer maintains three separate channels:

• strict binary outcome advantage
• perception criterion advantage
• reasoning criterion advantage

Perception and reasoning criteria are normalized independently inside each rollout group and zero-variance criteria receive no pressure.

4. Soft visual-token dependency

Perception advantage is distributed with a model-derived token weight instead of an XML span mask.
The trainer performs an additional zero-image counterfactual forward and uses the per-token log-probability change as soft visual dependency, with a mild later-token compensation inspired by VGPO.

This avoids assuming that every token inside <perception> is visually dependent or that visually dependent tokens cannot appear elsewhere in the CoT.

5. Save the good prefix

The deterministic verifier exposes the earliest hard error. For a rejected rollout:

• negative strict-outcome pressure starts only at the first verifiable error
• if the process is clean but the final answer is wrong, the cutoff is the <answer> span
• positive auxiliary advantage is allowed on the verified-good prefix
• negative auxiliary advantage is applied to the erroneous suffix

This follows the process-credit principle of Save the Good Prefix / Verifiable Prefix Policy Optimization instead of broadcasting a negative rollout reward over every earlier correct step.

6. Pass@8 interaction

• k=0: keep the existing partial-reward + batch-scale normalization path; no perception/reasoning auxiliary channel
• 1<=k<=7: use strict binary outcome advantage plus independent perception/reasoning token-level channels
• k=8: keep the mastered-group RL skip
• Gold-injected rows remain supervised-only and are excluded from multi-advantage statistics

7. Training cost

The soft visual dependency requires one additional counterfactual model forward for Reasoning RL training batches. It is not used by Generation RL.

Main references reflected in the implementation

• VPPO: sparse token-level visual dependency and intra-/inter-trajectory visual reweighting
• PAPO: counterfactual/noisy-image perception signal
• VGPO: visual-focus reweighting and temporal visual-forgetting compensation
• Learning When to Look: interleaved perception actions
• R1-VL / StepGRPO: intermediate reasoning validity
• Save the Good Prefix: first-error localization and prefix-preserving process credit