Logical issue in temperature handling for near-zero values

[genglinWang]

Issue Description

In _get_temperatures function from sarathi-serve/sarathi/model_executor/layers/sampler.py , when the temperature value is close to zero (< _SAMPLING_EPS), it's being set to 1.0. This behavior is not the same with greedy sampling and is not expected. Does sarathi handle greedy sampling somewhere else?

Current code:

_SAMPLING_EPS = 1e-5
def _get_temperatures(seq_metadata_list: List[SequenceMetadata]) -> List[float]:
    # Collect the temperatures for the logits.
    temperatures: List[float] = []
    for seq_metadata in seq_metadata_list:
        temperature = seq_metadata.seq.sampling_params.temperature
        if temperature < _SAMPLING_EPS:
            # NOTE: Zero temperature means deterministic sampling
            # (i.e., greedy sampling or beam search).
            # Set the temperature to 1 to avoid division by zero.
            temperature = 1.0
        temperatures.append(temperature)
    return temperatures