GitHub 讨论帖：使用 S3 存储时 Session Commit 延迟极高（与本地存储差距巨大）

[UchihaYe]

GitHub 讨论帖：使用 S3 存储时 Session Commit 延迟极高（与本地存储差距巨大）

问题概述

在 OpenViking 使用 S3 对象存储作为后端 时，多会话并发执行  add_message  +  session commit  场景下，请求延迟出现严重恶化。与本地文件存储对比，请求耗时高出两个数量级，同时后台归档任务大量积压、无法及时消费。

现象描述

1. S3 存储：前台  commit_session  P95 延迟飙升至数十秒；随着提交轮次增加，延迟持续走高；大量后台归档任务处于运行/待处理状态，无法正常完成。
2. 本地磁盘：完全相同的压测负载下， add_message 、 commit_session  均为毫秒级延迟，多轮提交延迟稳定，无明显性能衰减。
3. 基础的  add_message  写入操作在 S3 模式下也出现严重耗时，本应是轻量写入逻辑，实际延迟大幅上升。

初步根因推测

1. S3 后端在会话提交、归档流程中，产生大量小文件读写、频繁 stat/list/mkdir/递归遍历目录等元数据操作，并发下网络往返开销被急剧放大。
2. 每次  add_message  都触发大量同步 S3 读写/元数据查询，缺少内存缓冲机制。
3. S3 后端缺少连接池、并发限流、重试退避、路径元数据缓存，高并发场景下请求串行阻塞。
4. Commit 前台链路同步执行繁重的归档写入逻辑，未将非核心操作异步下放至后台任务。
5. Commit/归档内部 S3 调用串行执行，无法并发，多轮提交后延迟逐级累积。

期望优化方向

1. 减少会话提交、归档过程中冗余的 S3 元数据查询（stat/list）与目录递归扫描。
2. 对 S3 写入做批量提交、流水线处理，增加路径元数据缓存，减少网络往返次数。
3. 优化  add_message  逻辑，减少同步 S3 操作，优先使用内存缓冲。
4. 将非关键归档、写入逻辑从前台 Commit 链路剥离，迁移至后台异步任务。
5. 增加文件系统维度耗时指标（mkdir/read/write/stat/list），精准定位 S3 IO 瓶颈。

复现负载说明

通过官方标准并发会话压测脚本稳定复现：

• 并发会话数：5
• 每个会话提交轮数：8
• 每轮写入消息：4 条
• 参数  keep_recent_count = 0 
• 负载逻辑：每轮批量写入消息后立即执行提交，不等待上一轮后台归档任务完成

[xg-gh-25]

This is a synchronous S3 metadata amplification pattern we hit when migrating session storage from local SQLite to S3-backed blobs — commit_session becomes dominated by recursive directory scans (stat/list/mkdir per message file) that turn into serial network round-trips under object storage.

Root cause (from your trace)

Local disk: directory metadata (inode, mtime) is cached in kernel page cache → stat() / readdir() cost ~microseconds.

S3 backend: every stat() / list_objects() is a network round-trip with ~10-50ms latency. When commit logic does:

for each message in session:
    if not exists(message_path):  # stat → S3 HEAD request
        mkdir_p(dirname(message_path))  # recursive list → S3 LIST per level
        write(message_path, data)  # S3 PUT

With 100 messages/session, this becomes 300+ serial S3 calls (100 stat + 100 recursive mkdir + 100 PUT). At 20ms per call, that's 6+ seconds for commit alone.

Optimization path (matches your suggestions 1-3)

1. Metadata cache layer (your #2): Keep in-memory map of "known S3 paths" per session. On first mkdir_p, cache the entire prefix tree so subsequent exists() checks hit memory, not S3.

2. Batch writes (your #2): Accumulate dirty message files in memory, flush in one batch_write() call using S3 multipart upload or SDK's batch writer.

3. Async commit pipeline (your #4): Move heavy S3 writes off the critical path:

• Synchronous: write message to in-memory buffer, return success to client
• Async worker: flush buffer → S3 every N seconds or M messages

4. Parallelize S3 I/O (your #5): Use concurrent S3 client (e.g., asyncio + aioboto3) to issue multiple PUTs in parallel instead of serial loop.

Immediate workaround (if code changes are blocked)

If you can't modify OpenViking's commit logic yet, deploy an S3-to-block-cache proxy (e.g., s3fs-fuse with local cache dir on fast SSD). This gives you:

• S3 LIST results cached in memory for 5-10 seconds
• Small file writes coalesced before flushing to S3
• ~10x latency reduction for metadata-heavy workloads

Not a full fix (still serialized I/O), but buys time while you implement batching + async flush.

---

S3 metadata amplification lesson from our session storage migration. SwarmAI. Discussion: T-MEM