Unexpected system RAM growth during long-context inference on r9700 GPU

[AtaaEddin]

Unexpected system RAM growth during long-context inference on r9700 GPU

Problem

I am seeing continuous system RAM growth during a long-running coding/agent task when using the R9700 configuration, even though the agent itself is performing context compaction as expected.

My coding agent (PI) detects when the context becomes too large and automatically compacts/summarizes the conversation. After compaction, the logical context size does not continue growing significantly.

However, the server process continues consuming system RAM over time.

For example, the behavior is approximately:

Logical context:
20K tokens
    ↓
automatic compaction
    ↓
~10–12K tokens
    ↓
remains around this range

System RAM:
10 GB
    ↓
15 GB
    ↓
20 GB
    ↓
25 GB
    ↓
30+ GB

So the RAM growth does not appear to correlate directly with the current logical context size.

Configuration

• AMD Radeon AI PRO R9700 — 32 GB VRAM
• AMD Radeon RX 9070 XT — 16 GB VRAM
• ROCm / HIP
• Qwen3.8-27B-UD-IQ4_XS
• DFlash 2 Q8_0
• K cache: q8_0
• V cache: q8_0
• Target and draft both on hip:1

--target-device hip:1 \
--draft-device hip:1 \
--draft-block-size 16 \
--max-ctx 94072 \
--cache-type-k q8_0 \
--cache-type-v q8_0

The server reports:

Device 1: AMD Radeon AI PRO R9700
VRAM: 32624 MiB

target loaded: layers [0,64)
850 tensors on GPU 12.43 GiB

target_device = hip:1
draft_device = hip:1
draft_exec = local
peer_access = off
cache_type_k = q8_0
cache_type_v = q8_0
max_ctx = 94072

Important observation

The target model only reports approximately 12.43 GiB on the GPU, yet system RAM can grow to approximately 30–31 GB during the task.

More importantly, the agent is already compacting its context, so the logical context does not simply grow continuously toward 94K tokens.

After restarting the server, RAM usage returns to normal and then gradually grows again as the task proceeds.

This makes me wonder whether memory associated with previous contexts/requests is being retained rather than released.

Possible causes?

Could this be caused by one of the following?

• KV cache allocations not being released after context compaction
• KV cache growing to accommodate previous maximum context and not shrinking
• GPU/CPU allocator pools retaining freed memory
• DFlash speculative decoding buffers growing over time
• temporary prefill/verification buffers being retained
• CPU-side copies of context/KV data
• request/session history being retained internally
• memory fragmentation preventing reuse
• mmap/page-cache behavior
• some other host-memory allocation specific to DFlash 2

What I would like to understand

Is there a way to inspect the server's memory allocations during a request and determine what specifically is responsible for the increasing host RAM usage?

In particular, it would be useful to know:

1. Current KV-cache size in tokens and bytes.
2. GPU KV-cache allocation versus CPU/host allocation.
3. Whether KV memory is released when the context is compacted/reset.
4. Whether allocator pools intentionally retain previously allocated memory.
5. Whether DFlash 2 maintains additional host-side caches/buffers.
6. Whether there is a maximum host-memory limit or a way to prevent host-memory fallback.

The surprising part is that the logical context is being compacted, but the process's host RAM usage continues to increase substantially anyway.

Thanks!