[RFC] 关于 OpenViking 支持存储后端多写的设计方案/OpenViking supports a multi-write design for its storage backend.

[baojun-zhang]

本文档为双语版本（中文 + English）。中文版在前，英文版在后。

This document is bilingual (Chinese + English). The Chinese version comes first, and the English version follows.

中文版 / Chinese Version

概述

本文主要讨论 OpenViking 多存储后端的具体实现， 功能包括多 backend 配置支持， primary/backup 角色设定支持， 文件重定向支持，文件排除能力支持，加密配置调整，数据同/异步策略支持。

设计理念:
文件相关操作尽量內聚到 RAGFS 中， 减轻 viking_fs 压力

背景

1. Openviking 目前支持灵活替换存储后端，比如 localfs，s3fs，kvfs、memfs 等
2. Openviking 的 viking_fs 层支持多种对资源的操作
3. 不同的存储有不同的特性或者特别功能
4. 数据高可用

场景

Primary/Backup 同构

Primary/Backup 同构指 ov 配置的 primary/backup 数据源为相同类型，比如都为 localfs，Primary/Backup 同构主要可用于如下场景

数据高可用

双写/多写作为备份策略的一种， 相当于连续数据保护（CDP），副存储实时拥有主存储的完整副本，Openviking 通过自身就可以保证数据的高可用性，RPO 趋近于零。

多AZ区域路由优先

当实例部署在不同的 AZ ，不同 AZ 的网络延迟会极大的影响到整体请求响应，通过多 AZ backup 节点，可以提高请求响应能力。

Primary/Backup 异构

数据迁移

当需要把数据从旧系统迁移到新系统，仅仅使用 ovpack 会丢失部分实时数据，而通过双写+ovpack 方式，可以达到零丢失。

异构数据服务

同一份数据存储到不同形态的存储中，可提供更丰富的能力,比如:

• 存储到 tos, 提供预览的能力
• 存储到 kvfs， 提供高速读取的能力
• 存储到数据湖仓，提供数据分析能力

术语定义

• backend：由 backend plugin registry 根据 backend + params 初始化得到的底层句柄（如 localfs、s3fs、memfs、kvfs）。它不感知加密，仅负责纯字节读写；只允许作为 wrapper 的构造输入，不对外暴露，也禁止用它直接写入业务数据/元数据。

• primary_backend：ragfs 内部对 primary 的写入句柄（上层不可见）。全局加密开启时为 EncryptionWrappedFS(primary)，否则为 primary。MultiWriteWrappedFS 在写入业务文件与 .redirect.json / .sync_log.json 系统元数据时都通过它，从而保证 primary 命名空间下的加密语义一致。上层（viking_fs / Python）只感知统一的 FileSystem trait，不感知 primary/backup。

• backup_backend：ragfs 内部对 backup 的写入句柄（上层不可见）。全局加密开启且该 backup 自身 encryption.enabled=true 时为 EncryptionWrappedFS(backup)，否则为 backup。

• 元数据旁路直写（负面示例）：指保留一个未加密的 primary 句柄（即跳过 EncryptionWrappedFS），直接明文写 .redirect.json / .sync_log.json 的做法。这是错误路径——会造成同一 primary 目录下「业务文件加密 + 系统文件明文」的安全语义退化，必须禁止。

配置结构

现有配置模式:

// 单 backend 模式（现有，不变）
{
    "storage": {
        "workspace": "./data",
        "agfs": {
            "backend": "s3",
            "timeout": 10,
            "s3": {
                "bucket": "test-zbj",
                "region": "cn-beijing",
                "access_key": "xxx",
                "secret_key": "xxx",
                "endpoint": "http://tos-s3-cn-beijing.volces.com",
                "prefix": "",
                "use_ssl": false,
                "use_path_style": false,
                "directory_marker_mode": "nonempty"
            }
        }
    }
}

双写/多写模式配置模式:

增加backups 配置， 当在backups 中配置了 backend 后 ，对应 backend 将作为备份存储。

//  backups 备份
{
    "storage": {
        "workspace": "./data",
        "agfs": {
            "backend": "local",
            "backups": {
                "sync_type": "async",
                "items": [
                    {
                        "name": "memfs-cache",
                        "backend": "memfs",
                        "encryption": { "enabled": false },
                        "operations": [
                            { "operation": "read", "priority": 100 }
                        ]
                    }
                ]
            }
        }
    }
}

关键方案

多 Backend 配置

兼容模式

顶层的 backend / s3 / local 等参数始终作为 primary 配置。
backups.items 中的 backend 始终作为 backup 配置。

backups 与原有 backend-map 结构共存：

• backups 为空或不存在 → 保持现有单 backend 逻辑不变
• backups 不为空 → 顶层 backend 为 primary，backups.items 为 backup

// 精简示例：原有单 backend（不变）
{ "agfs": { "backend": "s3", "s3": { ... } } }

// 精简示例：多 backend 模式（primary = 顶层 backend，backup = backups.items）
{ "agfs": { "backend": "s3", "s3": { ... }, "backups": { "sync_type": "async", "items": [...] } } }

字段定义

primary（顶层 backend）在现有 AGFSConfig 结构基础上新增如下字段。

字段	类型	必填	说明
name	string	否	默认值为 primary
encryption	object	否	多写新增字段，控制该 primary 是否启用加密
operations	array	否	多写新增字段，控制该 primary 参与的操作及优先级
redirects	array	否	多写新增字段，控制该 primary 的重定向策略

backup 节点（backups.items）每个 item 直接复用 AGFSConfig，并额外增加多写相关字段, 新增字段如下：

字段	类型	必填	说明
name	string	是	多写新增字段，逻辑名称，全局唯一，用于 .sync_log.json / .redirect.json 中引用
encryption	object	否	多写新增字段，控制该 backup 节点是否启用加密
operations	array	否	多写新增字段，控制该 backup 节点参与的操作及优先级
excludes	array	否	多写新增字段，控制该 backup 节点的排除策略

backups 本身的容器字段包含：

字段	类型	必填	说明
sync_type	string	否	双写一致性模式：async / sync
write_ack_count	number	否	sync 模式下最少确认的 backup 节点数
write_ack_timeout_ms	number	否	sync 模式下等待 backup 节点确认的超时
write_concurrency	number	否	async 模式下后台写入并发上限
items	array	是	backup 节点列表，每一项复用 AGFSConfig 并增加多写字段

配置示例

{
  "agfs": {
    "backend": "local",
    "timeout": 10,
    "encryption": { "enabled": true },
    "redirects": [
      { "type": "FileExtensionPolicy", "extensions": ["(pdf|ppt)"], "target": ["s3-backup"] }
    ],
    "backups": {
      "sync_type": "async",
      "write_ack_count": 1,
      "write_ack_timeout_ms": 5000,
      "write_concurrency": 8,
      "items": [
        {
          "name": "s3-backup",
          "backend": "s3",
          "timeout": 10,
          "queuefs": { "backend": "sqlite" },
          "s3": {
            "bucket": "test-zbj-backup",
            "region": "cn-beijing",
            "access_key": "xxx",
            "secret_key": "xxx",
            "endpoint": "http://tos-s3-cn-beijing.volces.com",
            "prefix": "backup",
            "use_ssl": false,
            "use_path_style": false,
            "directory_marker_mode": "nonempty"
          },
          "encryption": { "enabled": false },
          "excludes": [
            { "type": "FileOverSizePolicy", "max_size_mb": 1024 }
          ],
          "operations": [{"operation": "read", "priority": 100}]
        }
      ]
    }
  }
}

Primary/Backup 区分

通过配置位置区分 primary/backup，不使用 role 字段：

位置	角色	说明
顶层 backend / s3 / local 等	primary	所有写操作的权威数据源，有且仅有一个
backups.items[]	backup	接收双写同步，可有多个

文件重定向 (redirects)

文件重定向指某些符合规则的文件可以写入到指定的 backend，而不是写到 primary，比如 PDF/PPT 等特殊格式的文件，或者超过一定 size 的文件。

适用场景

primary 可配置 redirects，将特定文件不写入 primary，而写入指定的 backup backend。

支持策略

策略	说明	配置示例
FileOverSizePolicy	文件超过指定大小时 redirect	{"type": "FileOverSizePolicy", "max_size_mb": 100, "target": ["s3-backup"]}
FileExtensionPolicy	文件后缀匹配时 redirect	`{"type": "FileExtensionPolicy", "extensions": ["(mp4

target 统一为字符串数组（可 redirect 到多个 backup，对应 .redirect.json 的 targets）。extensions 为正则字符串数组（与配置示例 (pdf|ppt) 一致）。

配置示例

{
  "agfs": {
    "backend": "local",
    "redirects": [
      { "type": "FileOverSizePolicy",  "max_size_mb": 100, "target": ["s3-backup"] },
      { "type": "FileExtensionPolicy", "extensions": ["(pdf|ppt)"], "target":  ["s3-backup"]}
    ],
    "backups": { "sync_type": "async", "items": [...] }
  }
}

.redirect.json 元数据

redirect 写入后，在 primary 对应资源所在目录下生成 .redirect.json，记录文件与目标 backend name 的映射。某个目录下如果存在 redirect 文件则生成一份。

文件结构:
/resource/<tenant>/
  ├── .redirect.json       ← 记录 redirect 映射
  └── large_pdf.pdf      ← 实际存储在 s3-backup,xxx-backup
  └── resource_AA_dir
              ├── .redirect.json
              └──  large_ppt.ppt     

.redirect.json 内容:

{
  "version": 1,
  "entries": {
    "large_pdf.pdf": {
      "targets": ["s3-backup", "xxx-backup"]
    }
  }
}

• .redirect.json 仅存在于 primary，不双写到 backup；primary 启用加密时，该文件也必须经 primary_backend 加密落盘
• 删除/移动 redirect 文件时同步更新 .redirect.json
• .redirect.json 加入 _INTERNAL_NAMES，对用户不可见
• 同一目录下的 .redirect.json 与 .sync_log.json 必须串行更新：由 MetaStateStore 的目录级互斥锁保护，在锁内同时读取/合并两份元数据、再原子写回。禁止在未持锁的情况下先读后写（裸 read_meta → write_meta），否则高频并发写会覆盖掉对方的 entry。
  • 示例：线程 A 写 large_pdf.pdf → 要在 .redirect.json 里新增一条 large_pdf.pdf → [s3-backup]；线程 B 同时写 report.doc → 要在 .sync_log.json 里 latest_seq++。如果不串行化，B 读到的 latest_seq 可能是旧值，写回后 A 刚写入的 redirect entry 就会被覆盖丢。
  • 修正：所有对 .redirect.json / .sync_log.json 的读写都必须通过 MetaStateStore::update_dir_meta(dir, ctx, |redirect_meta, sync_meta| { ... }) 完成。内部由 dir_locks[dir] 保证同一目录下的两份元数据不会被并发修改。

redirect 写入流程

sequenceDiagram
    participant Client
    participant MultiWrite
    participant Primary
    participant Backup as Backup (s3-backup,xxx-backup)

    Client->>MultiWrite: write(large_pdf.pdf)
    Note over MultiWrite: 检查 redirects 策略<br/>命中 FileOverSizePolicy
    MultiWrite->>Backup: write(large_pdf.pdf)
    Backup-->>MultiWrite: OK
    MultiWrite->>Primary: write(.redirect.json)<br/>(记录 large_pdf.pdf → s3-backup，xxx-backup)
    MultiWrite-->>Client: OK

Loading

redirect 命中后，写入目标从 primary + write_backups 切换为 targets，但同步状态机不变：

• .sync_log.json 仍然记录这次写入的 latest_seq，并为 targets 中的每个 backup 维护 acked_seq
• sync 模式下仍按 write_ack_count / write_ack_timeout_ms 计算确认；async 模式下仍由后台任务推进 acked_seq
• 任一 target 失败时，不允许只写 .redirect.json 而不落 .sync_log.json；必须保留未追平状态，由 retry_loop 继续补偿
• 配置变更后仍按当前生效配置计算应补偿的目标；已被移除或替换掉的旧 backup 目标忽略

文件排除 (excludes)

适用场景

仅 backup 节点可配置 excludes，指定哪些文件不写入当前 backup 节点。比如将 memfs/kvfs 作为 backup 节点，以提供系统响应速度。

策略类型

与 redirects 一致，支持 FileOverSizePolicy、FileExtensionPolicy 等。

配置示例

{
  "name": "memfs-cache",
  "backend": "memfs",
  "excludes": [
    { "type": "FileOverSizePolicy", "max_size_mb": 500 },
    { "type": "FileExtensionPolicy", "extensions": [".mp4"] }
  ]
}

与 redirects 冲突

如果 primary 配置 redirect 到 backend A，但 backend A 又 exclude 了该文件，属于用户自身配置冲突，本方案不兜底（该文件将不会被写入任何 backend）。

加密配置 (encryption)

encryption 已下沉到 RAGFS/Rust wrapper 实现。多写方案可以复用 EncryptionWrappedFS，无需 Python 层处理加密。backend / primary 只允许作为 EncryptionWrappedFS 的内部构造输入，禁止直接写入业务数据或元数据。所有对 primary 的写入——包括业务文件与 .redirect.json / .sync_log.json——都必须走 primary_backend 这一个入口。

决策逻辑

server.encryption.enabled
  │
  ├── false  →  全局加密关闭，所有 backend 全部明文
  │
  └── true   →  全局加密开启
          │
          ├── primary backend  →  **强制加密**（跟随 root_key；不允许配置 primary.encryption.enabled）
          └── backup  backend  →  由 item.encryption.enabled 决定；为 true 则加密(默认为 true)

双写场景下的加密

允许 primary 和 backup 使用不同的加密策略：

// primary 加密（跟随全局开关，不单独声明），backup (memfs) 不加密
{
  "agfs": {
    "backend": "local",
    // 沿用现有全局 encryption.enabled 配置；primary 不允许再配 encryption.enabled（强制跟随）
    "encryption": { "enabled": true },
    "backups": {
      "items": [
        // backup 允许自配 encryption.enabled
        { "name": "memfs-cache", "backend": "memfs", "encryption": { "enabled": false } }
      ]
    }
  }
}

各 backend 使用统一密钥管理，具体加密/解密只由 Rust EncryptionWrappedFS 完成。多写实现需把 EncryptionWrappedFS 作为可复用的 per-backend 数据 wrapper 使用：

• primary_backend：当全局 encryption.enabled=true 时为 EncryptionWrappedFS(primary)，否则为 primary；primary 不允许单独关闭加密。
• backup_backend：当全局 encryption.enabled=true 且该 backup 自身 encryption.enabled=true 时为 EncryptionWrappedFS(backup)，否则为 backup。
• .redirect.json / .sync_log.json 存在于 primary backend，属于 primary 命名空间中的系统文件，必须通过 primary_backend 读写，从而继承 primary 的加密配置。
• 禁止为内部元数据保留未加密的旁路直写入口（例如直接用未加密的 primary 去写 .redirect.json / .sync_log.json）。否则 primary 开启加密时，同一 backend 内会同时存在加密业务文件和明文系统状态文件，造成安全语义退化。
• 多写场景下，当前 encrypt迁移.md / 代码里的 Stats(Encryption(Mountable)) 全局栈需要演进为 Stats(MultiWriteWrappedFS)，由 MultiWriteWrappedFS 内部为 primary/backup 分别持有 primary_backend / backup_backend。EncryptionWrappedFS 的实现不变，只调整组装位置。

【性能与加密权衡 — 架构取舍】
若多个 backend 都配置了加密，写入同一份数据时会在每个 backend 的 EncryptionWrappedFS 内独立重新生成信封密钥并加密一次——这意味着 CPU 代价与加密 backend 数量成正比。这不是冗余 bug，而是多 backend 并行写入架构下的合理成本：每个 backend 的信封是隔离的（独立生成 file_key / data_iv / key_iv），不允许跨 backend 共享密文字节流。如果 CPU 成为瓶颈，应让某些 backup（如 memfs、cache backend）关闭 encryption.enabled，而不是通过绕过 primary_backend 去换取算力。后续优化方案可见后续可扩展项)

内部文件过滤

_INTERNAL_NAMES 扩展

原有 _INTERNAL_NAMES 列表需新增以下文件，使其对用户不可见：

_INTERNAL_NAMES = [
    ".path.ovlock",      // 原有：锁文件
    ".sync_log.json",    // 新增：同步日志
    ".redirect.json",    // 新增：重定向映射
]

过滤位置

在 VikingFS 的 ls、tree、glob 等目录遍历方法中统一过滤。这些文件对用户透明，但 MultiWriteWrappedFS 内部可直接读写。若 primary 开启加密，文件名仍按上述规则隐藏，文件内容由 primary_backend 加密。

操作路由 (operations)

操作类型

双写按 RAGFS 的真实方法分类。

tree、glob、read_file、write_file、append_file、move_file、ls 等是 Python 层 API, 可以尽量下沉到 RAGFS 中

操作类型	涵盖方法 (Rust trait)	说明
write	write, mkdir, remove, remove_all, rename, create, chmod, truncate, ensure_parent_dirs	写入类操作，需双写到所有可写 backup 节点
read	read, stat, exists ,read_dir, grep	读取类操作，按 priority 选择 backend

优先级规则

每个 backup 节点可通过 operations 字段声明支持的操作类型及其优先级：

{
  "name": "kvfs-cache",
  "backend": "kvfs",
  "operations": [
    {"operation": "read", "priority": 50}
  ]
}

规则	说明
优先级为正整数	越小越优先
顶层 primary 不配置 operations	primary 总是参与 write 且作为读兜底（忽略 primary 的 operations 配置，构建期校验拒绝）
backup 不配置 operations	默认仅参与 write（双写），不进入 read 路由；冷备语义
backup 显式声明 read	才进入读路由，按 priority 升序选择
backup 显式声明 write/缺省	参与双写；若只声明 read 不声明 write 则只读不双写（如 kvfs 纯读缓存）

【消歧】 早期"不配置 operations 默认参与所有操作"会让所有 backup 都进入读路由，异步窗口期内极易命中过期数据（语义退化）。修正为：backup 缺省只参与 write；只有显式声明 read 才进入读路由。这样冷备 backup 默认不影响读一致性，需要读加速的 backup（memfs/kvfs）必须显式声明 read 优先级，语义清晰且与「读操作路由」回退链路自洽。

位置约束

• 顶层 backend 对应的 primary 总是写入第一站，也是读路由的最终兜底，不配置 operations
• backups.items 中的 backup 节点可配置 operations 的 read/write 优先级；缺省时仅参与 write（不进入读路由），需要读加速须显式声明 read

// backup 节点只参与读，不接收双写数据（仅声明 read，不声明 write）
{ "name": "kvfs-cache", "backend": "kvfs", "operations": [{"operation": "read", "priority": 1}] }

读操作路由（回退链路）

读操作按以下顺序尝试，找到即返回：

1. 按 priority 升序尝试 read-enabled backup backend
       │
       │  文件不存在
       ▼
2. 回退到 primary
       │
       │  文件不存在
       ▼
3. 查询 primary 上 `.redirect.json` 的 `targets`（可能多个），按 targets 顺序逐个读取，命中即返回
       │
       │  文件不存在
       ▼
4. 返回 NotFound

【边界】 redirect targets 为多个时，读路由按 targets 列表顺序依次尝试（典型为异构副本，内容一致，命中第一个即可）。stat/exists 同。

sequenceDiagram
    participant Client
    participant MultiWrite
    participant Backup as Backup (read-enabled)
    participant Primary
    participant Redirect as .redirect.json

    Client->>MultiWrite: read(path)
    MultiWrite->>Backup: read(path)
    Backup-->>MultiWrite: NotFound
    MultiWrite->>Primary: read(path)
    Primary-->>MultiWrite: NotFound
    MultiWrite->>Primary: read(.redirect.json)
    Primary-->>MultiWrite: targets=["s3-backup"]
    MultiWrite->>Redirect: read(path) (s3-backup)
    Redirect-->>MultiWrite: data
    MultiWrite-->>Client: data

Loading

写操作路由

所有写操作一律先写 primary，再根据 sync_type 同步/异步写入所有可写 backup 节点：

sequenceDiagram
    participant Client
    participant MultiWrite
    participant Primary
    participant Backup as Backup (write-enabled)

    Client->>MultiWrite: write(data)
    MultiWrite->>Primary: write(data)
    Primary-->>MultiWrite: OK

    Note over MultiWrite: sync 模式: 等所有 backup 节点

    MultiWrite->>Backup: write(data)
    Backup-->>MultiWrite: OK
    MultiWrite-->>Client: OK

Loading

目录类操作路由

操作	路由策略	原因
read_dir	primary，并合并 .redirect.json 中本目录的 redirect 文件名	redirect 文件本体不在 primary，必须补回，否则用户 ls 看不到自己写入的文件（语义退化）
grep	primary 为主；命中 redirect 文件名时按需回源 target backend 读取内容匹配	否则 redirect 文件内容漏匹配
stat / exists	priority → primary → redirect	同读操作回退链路

【redirect 文件可见性，语义退化防护】 redirect 文件的数据本体写到 backup，primary 上只有 .redirect.json 元数据。若 read_dir 仅返回 primary 物理 entry，用户 ls 将看不到被 redirect 的文件。修正：read_dir 在返回 primary 真实 entry 后，读取本目录 .redirect.json 的 entries key 集合，对其中在 primary 上不存在的文件名补一条虚拟 FileInfo（size/mtime 取自 redirect 记录或回源 stat）。grep 同理——遍历到 redirect 文件名时回源 target backend 读取内容参与匹配。.redirect.json / .sync_log.json 自身始终被 _INTERNAL_NAMES 过滤，不出现在结果中。

Python 层高阶 API (ls / tree / glob) 内部组合 read_dir 等 trait 方法，自动落入 MultiWriteWrappedFS 的 primary 路由。

文件同步日志 (.sync_log.json)

定位

.sync_log.json 是记录 backup 节点同步进度的元数据文件，sync / async 模式都使用它表达“latest_seq 与各 backup acked_seq 是否追平”。

它记录的不是“某个文件是否曾经同步过”，而是“当前路径的最新版本号，以及各 backup 已追到哪个版本号”。

存储规则

规则	说明
位置	primary 的每个目录下
粒度	每个目录一个 .sync_log.json；当前不做分片
可见性	加入 _INTERNAL_NAMES，对用户不可见
读操作	不依赖该文件，按 priority 路由读取
加密	跟随 primary backend 加密配置；primary 启用加密时必须经 primary_backend 加密落盘

内容结构

{
  "entries": {
    "file.txt": {
      "latest_seq": 12,
      "last_op": "write",
      "backends": {
        "memfs-cache": {
          "acked_seq": 12
        },
        "local-az2": {
          "acked_seq": 11
        }
      }
    },
    "report.pdf": {
      "latest_seq": 3,
      "last_op": "remove",
      "backends": {
        "memfs-cache": {
          "acked_seq": 3
        }
      }
    }
  }
}

字段语义：

• entries 的 key 使用当前目录下的文件名。.sync_log.json 本身就是目录级元数据，目录路径由该文件所在位置隐含提供，不再重复存完整 uri
• latest_seq：当前路径最新一次成功写入 primary 的单调递增版本号；判断同步完成以它为准
• last_op：该 latest_seq 对应的操作类型（write / mkdir / remove / remove_all / rename / chmod / truncate）。retry_loop 据此决定如何在落后 backup 上重放：write/truncate 回源 primary（或 redirect target）读最新内容重写；remove/remove_all 重放删除；mkdir/chmod 重放对应操作；rename 额外在 rename_to 记录目标 uri。无此字段则 retry 无法重放删除/重命名等非内容类操作（语义退化）。
• rename_to（仅 last_op==rename 时存在）：重命名目标的规范化 uri。之所以保留完整路径，是因为 rename 可能跨目录
• backends.{name}.acked_seq：该 backup 已确认成功应用到的最新版本号
• backends 中的 key 使用当前配置中的 backend name 作为身份标识

account_id 不写入 .sync_log.json，避免冗余真相字段。后台 retry_loop 需要 ctx 时，必须从元数据文件所在的 primary 物理路径按 _uri_to_path 的同源规则反推 FsContext；如果路径无法反推出 account_id，说明该目录不满足加密后台恢复前提，必须报错并跳过，不能用空 ctx 或未加密的 backend 句柄兜底。

同步完成条件：

对 entries[file_name]：
  target_backends = 根据当前生效配置计算该文件当前应写入的 backup 集合
                   （配置变更需重启后生效；若 backup 被替换，则旧目标忽略）

  对每个 backup ∈ target_backends
    若 backends[backup].acked_seq == latest_seq
    则该 backup 已同步到当前最新版本

当且仅当所有 target_backends 都满足上述条件时，
该文件才视为“同步完成”。

生命周期

写入文件时 (sync / async 模式):
  1. 生成新的 latest_seq
  2. 更新 `.sync_log.json`
       ├── 写入该 file_name 的 latest_seq
       └── 保留各 backup 现有 acked_seq
  3. sync: 等待 quorum；async: tokio::spawn { backup.write(data) }
       ├── 成功/已确认 → 将该 backup 的 acked_seq 更新为 latest_seq
       └── 失败/超时/未确认 → 保持 acked_seq 不变

后台重试 (见失败重试机制):
  retry_loop 扫描 `.sync_log.json`
    ├── 根据当前生效配置计算 target_backends（配置变更需重启；旧目标忽略）
    ├── 找到缺失 acked_seq 或 acked_seq < latest_seq 的 backup → 重试写入
    └── 所有 target_backends 都满足 acked_seq == latest_seq → 跳过

手动同步接口

提供命令行工具供运维手动触发同步状态检查和修复：

ov system backend sync-status viking://resources     # 查看 viking://resources
ov system backend sync-retry  viking://resources     # 手动重试 viking://resources

CLI 通过 ragfs-python binding 新增的专用 API（如 client.system_sync_status(path, ctx) / client.system_sync_retry(path, ctx)）读写同步状态，不直接读 .sync_log.json 文件——元数据经 MultiWriteWrappedFS 的 metadata API 统一读写，并在内部走 primary_backend，从而继承 primary 加密配置、目录级元数据 guard 与并发锁，避免脱机读和锁绕过。

并发安全

.sync_log.json / .redirect.json 都是目录级 JSON 元数据，读-改-写操作存在并发竞争风险（前台写、多个 tokio::spawn ack 更新、retry_loop、手动 sync-retry 可能同时修改同一目录）。

统一 MetaStateStore

这里开始进入具体实现约束：

• MetaStateStore：MultiWriteWrappedFS 内部的目录级元数据管理器，负责 .redirect.json 与 .sync_log.json 的统一读-改-写，并持有目录锁 / 路径串行化队列。
• FsContextResolver：后台任务使用的上下文恢复 helper；当 retry_loop、backfill、system_sync_retry 没有前台请求上下文时，由它从规范化路径恢复 account_id，再进入 FS_CTX.scope(ctx, ...) 执行加密读写。

/// 统一封装 .redirect.json / .sync_log.json 的读改写。
/// 所有方法必须在 `primary_backend` 上执行，因此自动继承 primary 加密配置。
struct MetaStateStore {
    primary_backend: Arc<dyn FileSystem>,
    dir_locks: Mutex<HashMap<String, Arc<tokio::sync::Mutex<()>>>>,
    path_queues: PathSerializer,
    ctx_resolver: Arc<dyn FsContextResolver>,
}

impl MetaStateStore {
    /// 对同一目录内的元数据文件串行执行 read-modify-write，防止丢更新。
    async fn update_dir_meta<T>(&self, dir: &str, ctx: FsContext, op: T) -> Result<()>
    where
        T: FnOnce(&mut RedirectMeta, &mut SyncLogMeta) -> Result<()>;
}

每次更新 .redirect.json 或 .sync_log.json 时，必须先获取对应目录的锁，并在同一临界区内完成两个文件的 read-modify-write。redirect 写入场景下，.redirect.json 的 entries 更新与 .sync_log.json 的 latest_seq 更新必须同锁提交，避免"数据已写到 redirect target，但 redirect 映射或 sync 版本丢失"。

另外，为避免同一路径的多个写操作在 backup 侧乱序生效，必须配合 per-path 串行化队列。.sync_log.json 负责记录“每个 backup 已确认到哪个版本号”，per-path FIFO 负责保证版本不会倒退。

对于 rename，若源目录与目标目录不同，单目录锁不够，必须升级为双目录锁：

• 先按目录字符串的稳定顺序（如字典序）排序，再依次获取两把目录锁，禁止按调用顺序随意加锁，以免死锁
• 在同一临界区内同时更新 source dir / target dir 的 .redirect.json 与 .sync_log.json
• 若 source_dir == target_dir，退化为单目录锁
• rename_to 仅用于记录目标 uri；真正的元数据一致性由双目录锁保障，而不是依赖后台重试兜底

同步模式 (sync_type)

OpenViking 通过sync_type 支持设置不同一致性策略:

• 强一致性（同步双写），写主成功且写从成功，才返回客户端成功。风险：最慢的后端决定整体延迟，从写失败，则需要主进行回退。
• 最终一致性（异步双写），先写主存储返回成功，后台异步同步到副存储。风险：双写窗口期内后端不一致，读请求可能路由到旧数据。

两种模式

模式	值	默认	行为
异步双写	async	✅	先写 primary 返回成功，后台同步到 backup
同步双写	sync		写 primary 成功 + 写 backup 成功，才返回客户端成功

同步写 (sync)

数据流：

Client
  │
  │ write(data)
  ▼
MultiWriteWrappedFS
  │
  ├──► primary.write()  ──► OK
  │
  ├──► backup1.write() ──► OK
  │
  └──► backup2.write() ──► OK
  │
  ▼
返回成功给 Client

时序图：

sequenceDiagram
    participant Client
    participant MultiWrite
    participant Primary
    participant Backup1
    participant Backup2
    
    Client->>MultiWrite: write(data)
    MultiWrite->>Primary: write(data)
    Primary-->>MultiWrite: OK
    MultiWrite->>Backup1: write(data)
    Backup1-->>MultiWrite: OK
    MultiWrite->>Backup2: write(data)
    Backup2-->>MultiWrite: OK
    MultiWrite-->>Client: OK

Loading

同步模式 quorum 支持

当前设计：sync 模式必须等所有 backup 节点写完才返回。一个慢 backup 节点会拖垮整体延迟。增加最小确认数和超时：

{
  "backups": {
    "sync_type": "sync",
    "write_ack_count": 2,
    "write_ack_timeout_ms": 5000
  }
}

参数	默认值	说明
write_ack_count	全部 backup 节点	最少需要多少个 backup 节点确认写入成功
write_ack_timeout_ms	无超时	等待 backup 节点确认的超时（ms），超时的 backup 节点通过 retry 补偿

Note

风险与失败语义：

• 最慢的 backup 节点决定整体延迟（已由 write_ack_count + write_ack_timeout_ms 缓解：达到 quorum 即返回）
• primary 写失败 → 整个 write 立即返回失败，不触发任何 backup 写（数据未落地，无不一致）
• primary 成功、backup 未达 quorum → 返回客户端失败，但 primary 数据已落地且无法回滚（文件系统无事务）。此时 .sync_log.json 已记录 latest_seq，未追平的 backup 由 retry_loop 持续补偿至最终一致。即"客户端看到失败，但系统保证最终把数据同步到位"，需在 API 文档明确该语义，避免调用方误判数据丢失。

异步写 (async)

数据流：

Client
  │
  │ write(data)
  ▼
MultiWriteWrappedFS
  │
  ├──► primary.write()  ──► OK  ──► 立即返回成功给 Client
  │
  └──► tokio::spawn { backup1.write() }  ──► 完成/失败后更新 `.sync_log.json`
       tokio::spawn { backup2.write() }  ──► 完成/失败后更新 `.sync_log.json`

时序图：

sequenceDiagram
    participant Client
    participant MultiWriteWrappedFS
    participant Primary
    participant Backup1

    Client->> MultiWriteWrappedFS: write(data)
    MultiWriteWrappedFS->>Primary: write(data)
    Primary-->>MultiWriteWrappedFS: OK
    MultiWriteWrappedFS-->>Client: OK

    Note over MultiWriteWrappedFS: tokio::spawn

    MultiWriteWrappedFS->>Backup1: write(data)
    Note right of Client: Client 已拿到响应
    Backup1-->> MultiWriteWrappedFS: OK/ERR
    Note over MultiWriteWrappedFS: 更新 .sync_log.json

Loading

Note

风险：

• 双写窗口期内 backup 与 primary 数据不一致
• 读请求可能路由到旧数据（最终一致性下可接受）
• backup 节点写入失败的修复依赖后台重试（见异步写入失败重试）

异步写入并发控制

当前不限制异步写入的并发数。如果未来出现 backup 节点写入速度慢于调用方写入速度的场景（如 primary=localfs，backup=S3），可通过 write_concurrency 配置启用 tokio Semaphore 限流：

// 未来可扩展
{ "backups": { "sync_type": "async", "write_concurrency": 8 } }

• 未配置时：无限制
• 配置后：同时最多 N 个异步写入任务在飞行中

异步写入顺序保证（per-path 串行化）

异步双写下，同一文件路径的多次操作（如 write → remove → write）在 backup 节点上可能乱序执行，导致最终数据不一致。解决方案：按路径维护 FIFO 队列，同一路径的操作在 backup 节点上顺序执行。

• 每个 path 独立队列，不同 path 之间仍然并行
• 队列为空时自动回收 worker

读路径一致性子句

异步模式下，删除操作在 backup 节点上可能延迟执行。读请求可能命中尚未删除的旧数据：

• 默认行为：容忍最终一致性，不额外检查

失败重试机制

同步双写场景下，backup 节点写入可能因网络抖动等临时故障而超时失败（quorum 满足即返回客户端成功，但未达 ack 的 backup 仍需补偿）。

异步双写下，tokio::spawn 的 backup 节点写入可能因网络抖动等临时故障而失败。

两种模式都通过 .sync_log.json 落地"未追平"状态：

• async：写 primary 后即更新 latest_seq，spawn 成功才更新 acked_seq。
• sync：达到 write_ack_count 即返回；写 primary 时同步更新 latest_seq，对未确认/超时/失败的 backup 不更新其 acked_seq（保持落后），交由 retry_loop 补偿。

因此 retry_loop 在两种模式下都需运行。只要存在 write-enabled backup，new() 即启动 retry_loop（不再限定 async）。

参见 MultiWriteWrappedFS 结构 一节：new() 中 if !inner.write_backends().is_empty() { tokio::spawn(Inner::retry_loop(...)) }。

参数定义：

参数	默认值	说明
扫描间隔	30s	定期扫描 .sync_log.json
单文件重试上限	3 次/轮	避免单文件死循环

独立任务队列

.sync_log.json 本身就是持久化的记录。不需要引入 RabbitMQ/Kafka 等外部任务队列。数据流：

写操作  → 生成新的 latest_seq
                                  │
retry_loop (每 30s) ◄─────────────┘
  │
  ├── 读取 `.sync_log.json`
  ├── 根据当前配置计算 target_backends
  ├── 查找缺失 acked_seq 或 acked_seq < latest_seq 的 backup
  ├── 重试写入 → 成功后将该 backup 的 acked_seq 更新为 latest_seq
  └── 所有 target_backends 都满足 acked_seq == latest_seq → 跳过

具体实现

Encrypt 能力下沉到 ragfs 层，双写能力在 ragfs 层实现，python 层只负责配置传递。

VikingFS (Python)                  ← URI 映射、访问控制、向量索引同步、锁管理
    ↓                                    （不感知双写，不感知加密）
RAGFSBindingClient (PyO3)          ← FFI 桥接，传递 primary + backups 配置
    ↓
StatsWrappedFS                     ← 顶层端到端统计（含双写/加密耗时）
    ↓
MultiWriteWrappedFS                ← 【新增】双写路由、读回退、sync_log/redirect 管理
    ├── primary_backend        ← primary 真实入口；全局加密开启时为 EncryptionWrappedFS(primary)，否则为 primary
    └── backup[].backend       ← backup 真实入口；全局加密开启 + backup.encryption.enabled → EncryptionWrappedFS(backup)，否则 backup
            ↓
        localfs / s3fs / memfs / ...       ← 实际存储后端

每层职责：

• VikingFS：只做 URI→路径 转换、访问控制（_ensure_access）、向量索引同步、锁管理。配置传递给 Rust 侧。
• MultiWriteWrappedFS：接收 primary 配置和 backups.items 列表，管理读写路由、.sync_log.json、.redirect.json；内部元数据也通过 primary_backend 读写。
• EncryptionWrappedFS：透明 wrapper，对上层无感，每个 backend 独立决定是否包装；primary 强制跟随全局开关，不允许通过未加密的 backend 句柄绕过加密。

配置透传（FFI）

【可行性硬约束 — 现状 FFI 无法透传嵌套结构，必须先扩展】

核对现状代码：

• types.rs PluginConfig 仅 {name, mount_path, params: HashMap<String, ConfigValue>}；ConfigValue 只有 String / Int / Bool / StringList，无嵌套 dict / list。
• lib.rs::py_dict_to_config 把非标量值降级为 v.str() 的 repr 字符串。
• lib.rs::mount 只接收扁平 config dict。
• agfs_utils.py::mount_agfs_backend 单 backend 挂载到 /local，无 backups 组装。

因此 backups 嵌套结构无法经现有通道传入。最小改造（推荐，复用现有 JSON 能力）：

1. Rust 新增 ConfigValue::Json(serde_json::Value) 变体，并让 py_dict_to_config 对 dict/list 递归转为 serde_json::Value（lib.rs 已有 serde_json_to_py 反向转换可复用，只需补正向 py_any_to_json）。这样无需为每个嵌套字段定义 PyO3 提取逻辑，backups 整体以一个 JSON 值透传。
2. PluginConfig 新增字段：

// crates/ragfs/src/core/types.rs
pub struct PluginConfig {
    pub name: String,
    pub mount_path: String,
    pub params: HashMap<String, ConfigValue>,
    // 【新增】多写配置，None = 单 backend 模式（现状不变）
    pub backups: Option<BackendsConfig>,
    // 【新增】全局加密开关（server.encryption.enabled）
    pub server_encryption_enabled: bool,
    // 【新增】primary 自身的 encryption / redirects（顶层 backend 的多写字段）
    pub primary_encryption_enabled: bool,
    pub primary_redirects: Vec<RedirectPolicy>,
}

#[derive(Debug, Clone, Deserialize)]
pub struct BackendsConfig {
    pub sync_type: Option<String>,            // "sync" | "async"，默认 async
    pub write_ack_count: Option<usize>,
    pub write_ack_timeout_ms: Option<u64>,
    pub write_concurrency: Option<usize>,
    pub items: Vec<BackendItemConfig>,
}

#[derive(Debug, Clone, Deserialize)]
pub struct BackendItemConfig {
    pub name: String,                          // 必填，全局唯一（身份标识）
    pub backend: String,                       // plugin 类型（local/s3/memfs/...）
    // 该 backend 的 plugin 子参数，沿用顶层约定：以 backend 类型为 key 的子对象
    // （如 backend="s3" 时取 item["s3"]，backend="local" 时取 item["local"]）。
    // 反序列化时按 backend 名提取对应子对象为 params。
    pub params: serde_json::Value,
    pub timeout: Option<u64>,
    pub encryption: Option<EncryptionConfig>,
    pub operations: Option<Vec<OperationItemConfig>>,
    pub excludes: Option<Vec<RedirectPolicy>>,
}

#[derive(Debug, Clone, Deserialize)]
pub struct OperationItemConfig { pub operation: String, pub priority: u32 }

#[derive(Debug, Clone, Deserialize)]
pub struct EncryptionConfig { pub enabled: bool }

#[derive(Debug, Clone, Deserialize)]
#[serde(tag = "type")]
pub enum RedirectPolicy {
    FileOverSizePolicy { max_size_mb: u64, target: Option<Vec<String>> },
    FileExtensionPolicy { extensions: Vec<String>, target: Option<Vec<String>> },
}

3. BackendsConfig 由 serde_json::from_value 反序列化（mount() 识别 params["backups"] 或单独参数），避免逐字段 PyO3 提取。
4. Python 侧 agfs_utils.py 把 agfs.backups（含每个 item 的 plugin 子参数）原样塞进 mount 的 config dict；backend/s3/local 顶层参数继续作为 primary 的 params。

该改造与 encrypt迁移.md 的 root_key 注入正交：root_key 仍由 binding 构造期的 encryption 配置注入；backups 是挂载期静态配置，只决定哪些 backend 需要套 EncryptionWrappedFS。多写挂载启用后，不能再使用全局 Stats(Encryption(Mountable)) 作为唯一数据栈，而应由 build_multi_write_fs() 为每个 BackendEntry 组装 backend。

MountableFS::mount() 集成

mount() 中检测 config.backups：为空走现状单 backend 逻辑（零改动）；非空则调用新增私有方法 build_multi_write_fs() 构建 MultiWriteWrappedFS，外层仍包 StatsWrappedFS（保持 metrics 一致）。多写挂载内的 primary/backup 不再挂回同一个 MountableFS，而是由 registry 初始化 backend 后按配置包装为 BackendEntry::backend，避免全局 Encryption wrapper 与多写调度顺序相互绕过。

// 伪代码：mount() 分支
let inner_fs: Arc<dyn FileSystem> = match &config.backups {
    None => self.build_single_backend(&config).await?,        // 现状逻辑
    Some(bc) => self.build_multi_write_fs(&config, bc).await?, // 多写
};
let wrapped = StatsWrappedFS::new_arc(inner_fs);               // 复用现有 wrapper

build_multi_write_fs() 步骤：

这里开始进入 backend 组装细节：BackendEntry 是 MultiWriteWrappedFS 内部保存的 backend 描述结构，字段包括 name、role（Primary / Backup）、backend、operations（仅 backup 允许）、excludes。

1. primary：用顶层 backend/params 经 registry 初始化 primary（即未加密的底层句柄）；跟随全局 server.encryption.enabled 决定是否包装为 EncryptionWrappedFS(primary)，结果记为 primary_backend。primary 不允许关闭加密。
2. 遍历 bc.items：各自经 registry 初始化 backup（即未加密的底层句柄）；按 server.encryption.enabled && item.encryption.enabled 决定是否包装为 EncryptionWrappedFS(backup)，结果记为 backup_backend。
3. 校验：有且仅有一个 primary（顶层）；item.name 全局唯一；primary 不接受 operations（否则报错）；redirects.target / excludes 引用的 name 必须存在。
4. 构建 BackendEntry { name, backend, role, operations, excludes }，组装为 MultiWriteWrappedFS。内部元数据文件必须使用 primary_backend，不得保存未加密的 primary 句柄作为元数据旁路直写入口。

MultiWriteWrappedFS 结构（Arc 固化）

【可行性 — 异步 spawn 与 retry_loop 必须共享 Arc】 异步双写的 tokio::spawn 闭包与后台 retry_loop 都要回写 .sync_log.json，需持有自身 Arc。直接 tokio::spawn(self...) 会 move 掉 self。固化为 wrapper(壳) + Arc<Inner>(状态) 模式：

pub struct MultiWriteWrappedFS { inner: Arc<Inner> }

struct Inner {
    backends: Vec<BackendEntry>,
    primary_idx: usize,
    sync_type: SyncType,
    write_ack_count: Option<usize>,
    write_ack_timeout_ms: Option<u64>,
    write_sem: Option<Arc<tokio::sync::Semaphore>>,   // write_concurrency 限流
    redirects: Vec<RedirectPolicy>,                   // primary 的 redirect 策略
    meta_store: MetaStateStore,                        // encrypted metadata + per-dir guard
    path_queues: PathSerializer,                       // per-path 串行化队列
}

impl MultiWriteWrappedFS {
    pub fn new(...) -> Result<Self> {
        let inner = Arc::new(Inner { ... });
        if !inner.write_backends().is_empty() {
            tokio::spawn(Inner::retry_loop(Arc::clone(&inner)));
        }
        Ok(Self { inner })
    }
}

// FileSystem trait 委托给 self.inner.*；spawn 闭包用 Arc::clone(&self.inner)

【元数据与加密分层 — 致命语义防护】 .redirect.json / .sync_log.json 主存储在 primary backend，必须跟随 primary 加密配置。若 primary 开启加密，两个文件落盘也必须是 OVE1 信封；若 primary 未开启加密，则保持明文 JSON。禁止任何未加密的元数据旁路直写。

方案：MetaStateStore 持有 primary_backend，所有元数据读写都经该 wrapped 入口。前台请求使用当前 FS_CTX；异步 backup ack task 在 spawn 前 clone 当前 FsContext 并在闭包内重新进入 FS_CTX.scope(ctx, ...)；retry_loop / backfill / system_sync_retry 通过 FsContextResolver 从规范化路径恢复 account_id 后再进入 scope。若无法从路径恢复 ctx，必须返回错误并跳过该元数据文件，不能降级为直接读取文件。

代码复用抽象（写/读/策略三处去重）

用户硬要求：抽象、提取、复用。当前各写方法（write/mkdir/remove/remove_all/rename/chmod/truncate/ensure_parent_dirs）逻辑高度雷同，必须抽公共骨架，禁止 8 份复制粘贴。

1. 写 fanout 骨架 fanout_write —— 所有写操作复用：

/// 对所有 write-enabled backup 执行同一闭包 op，按 sync_type 分流。
/// sync: 按 write_ack_count / timeout 收集确认；async: spawn + 更新 sync_log。
async fn fanout_write<F, Fut>(&self, path: &str, op: F) -> Result<()>
where F: Fn(Arc<dyn FileSystem>) -> Fut + Send + Sync + 'static,
      Fut: Future<Output = Result<()>> + Send {
    let targets = self.write_backups_for(path);   // 过滤 excludes 命中的 backup
    match self.sync_type {
        SyncType::Sync  => self.fanout_sync(path, &targets, &op).await,
        SyncType::Async => { self.fanout_async(path, targets, op); Ok(()) }
    }
}
// 各方法只需：先写 primary，再 self.fanout_write(path, |fs| async move { fs.mkdir(...) }).await

2. 读 backend 解析 resolve_read_backend —— read/stat/exists 复用回退链路：

/// 返回应命中的 backend 句柄（按 priority 升序 → primary → redirect target）。
async fn resolve_read_backend(&self, path: &str) -> Option<Arc<dyn FileSystem>>;
// read/stat/exists 都先调它拿到 fs，再执行对应 trait 方法；无 read backend 时直连 primary（避免空循环）。

3. 文件策略 FilePolicy trait —— redirects 与 excludes 同源：

/// redirects 与 excludes 共用同一套策略求值，避免两份判定逻辑。
trait FilePolicy { fn matches(&self, path: &str, size: u64) -> bool; }
impl FilePolicy for RedirectPolicy { /* FileOverSize / FileExtension */ }
// redirect: 命中 → 写 target backup + 记 .redirect.json；exclude: 命中 → 跳过该 backup。

4. 元数据 json 编解码复用 —— .redirect.json / .sync_log.json 均为 JSON（符合"系统内部状态用 JSON"约束），共用 MetaStateStore::read_meta::<T>() / write_meta::<T>() / update_dir_meta() 泛型 helper（基于 primary_backend + serde_json），不重复手写解析。primary 开启加密时 helper 读写的是加密信封，反序列化前由 EncryptionWrappedFS 解密；primary 未开启加密时 helper 直接读写明文 JSON。

5. FsContext 快照复用 —— 所有后台任务必须显式携带或恢复 ctx，避免重复散落 FS_CTX.scope：

/// 从当前 task 克隆 FsContext；无 ctx 时返回错误，禁止后台任务用空 ctx 操作加密元数据。
fn current_required_ctx() -> Result<FsContext>;

/// 在指定 ctx 下执行 primary/backup 文件系统操作。
async fn with_fs_ctx<F, Fut, T>(ctx: FsContext, op: F) -> Result<T>
where
    F: FnOnce() -> Fut,
    Fut: Future<Output = Result<T>>;

写路径完整流程（含 redirect / exclude）

write(path, data, offset, flags):
  0. ctx = current_required_ctx()                      # 与 _uri_to_path 同源
  1. 评估 primary.redirects（FilePolicy）：
       命中 ⇒ targets = 策略 target backups
              1.1 在 path_queues[path] 中按 FIFO 对 target 的 `backup_backend` 执行 write
              1.2 meta_store.update_dir_meta(dir, ctx, ...)
                  ├── 通过 `primary_backend` 写 .redirect.json（记录 file_name → targets）
                  └── 通过 `primary_backend` 更新 .sync_log.json（latest_seq++，并为 targets 建立 ack 状态）
              1.3 sync 模式按 quorum 等待 target 确认；async 模式由后台推进 acked_seq
              1.4 任一 target 未追平 ⇒ 保持落后 acked_seq，交由 retry_loop 补偿
              return
  2. 写 `primary_backend`（经 primary 加密层，业务数据）
  3. self.fanout_write(path, |fs| fs.write(...))     # 自动过滤 excludes
  4. async 模式：spawn 前 clone ctx；spawn 内用 FS_CTX.scope(ctx, ...) 写 backup 并更新 acked_seq

文件清单

文件	操作	说明
crates/ragfs/src/core/multiwrite_wrapper.rs	新增	MultiWriteWrappedFS + Inner + BackendEntry::backend + fanout_write/resolve_read_backend/FilePolicy
crates/ragfs/src/core/multiwrite_meta.rs	新增	MetaStateStore + MetaLockProvider + encrypted JSON 元数据 helper + ctx 恢复
crates/ragfs/src/core/types.rs	修改	ConfigValue::Json；PluginConfig 新增 backups/encryption 字段；BackendsConfig 等
crates/ragfs/src/core/mountable.rs	修改	mount() 识别 backups；新增 build_multi_write_fs()；multiwrite 挂载需直接初始化 backend entry，不通过全局 Encryption(Mountable) 绕行
crates/ragfs/src/core/mod.rs	修改	pub mod multiwrite_wrapper; pub mod multiwrite_meta;
crates/ragfs/src/core/filesystem.rs	不改	trait 签名不变
crates/ragfs-python/src/lib.rs	修改	py_dict_to_config 支持嵌套；mount() 透传 backups；新增 system_sync_status(path, ctx) / system_sync_retry(path, ctx)
openviking/utils/agfs_utils.py	修改	组装 agfs.backups 下传 mount
openviking/storage/viking_fs.py	修改	_INTERNAL_NAMES 增 .sync_log.json/.redirect.json

后续可扩展项

多进程 / 多实例边界

本期不实现跨进程 / 多实例的并发安全协调。当前方案中的 tokio::sync::Mutex 只能保证单进程内安全；如果未来部署形态允许多个 RAGFS 进程同时写同一 primary mount，需要补齐独立的 MetaLockProvider，例如 backend-native conditional write / Redis / etcd。

约束要求：

• 未接入跨进程锁提供者前，不能声称 multi-instance 写入安全。
• 若未来要支持 multi-instance 写入模式，启动期应显式校验锁能力是否存在，不能静默退化为本地 mutex。

写放大风险

本期不实现目录级元数据写放大优化。当前设计每写一个文件就会重写所在目录的 .redirect.json / .sync_log.json，在热点目录下可能成为瓶颈。

后续可选优化方向：

• 以规范化 uri 的 hash 作为分片键，拆分为多个分片文件（仍在目录锁与 MetaStateStore 的管控下）。
• 改为 batch / append-only 日志结构，降低单文件全量重写开销。

Wrapper 分组调度

当前方案直接在 MultiWriteWrappedFS 内部持有 primary_backend / backup_backend。如果后续需要进一步优化 “per-backend wrapper 组装” 的重复开销，可以考虑引入 GroupedWrappedFS，但本次不实现。

核心思路：

• GroupedWrappedFS 负责按稳定的“分组特征”对请求分类，然后为不同分组挂接不同的后续 wrapper 链。
• 例如：
  • StatsWrappedFS -> GroupedWrappedFS -> EncryptionWrappedFS -> MultiWriteWrappedFS
  • StatsWrappedFS -> GroupedWrappedFS -> MultiWriteWrappedFS
• 加密组走 EncryptionWrappedFS -> MultiWriteWrappedFS，明文组直接走 MultiWriteWrappedFS，从而避免当前按 backend 逐个决定后续 wrapper 的心智负担。

这个抽象不只适用于多写，也可扩展到单写场景：例如按配置 / 路径 / 能力标签为不同请求加载不同的 wrapper 组合，而不是把是否挂某个 wrapper 固化在全局挂载链上。

实现前提与边界：

• GroupedWrappedFS 的分组条件必须是稳定且可重放的；写路径如何分组，读路径也必须能用同样规则命中同一条 wrapper 链，否则会出现“写时走加密链、读时走明文链”的语义错乱。
• 分组条件不能依赖只在写时可见、但读时无法恢复的信息；如果依赖这类信息，就必须把分组结果持久化为元数据，供读/重试/backfill/retry_loop 复用。
• 可观测性仍应保持分层：StatsWrappedFS 统计总耗时，GroupedWrappedFS 可额外暴露各分组命中次数，EncryptionWrappedFS 继续承担加密链路的性能统计。
• 若未来实现该方案，需要重新审视 build_multi_write_fs() 的职责边界：它不再直接组装最终 wrapper，而是产出分组规则与每组对应的下游链。

结论：GroupedWrappedFS 是一个可实现的后续优化方向，而且比“per-backend 分别决定 wrapper”更通用；但它会显著抬高挂载编排复杂度，只有在确认当前多写 + 加密的组合路径已经成为主要复杂度来源时，才值得引入。

可观测性

MultiWriteWrappedFS 应通过 StatsWrappedFS 暴露 metrics：

• backup 节点同步成功率 / 延迟
• retry 次数
• .sync_log.json 体积

健康检查

在 openviking/server/routers/system.py::checks["agfs"] 中加入 multiwrite 健康度（primary / 各 backup 节点可用性）。

测试矩阵

实现后至少覆盖：

• 维度组合：sync/async × primary(local/s3) × backup(memfs/local/s3) × encryption(on/off) × redirects(on/off) × excludes(on/off)
• 故障注入：backup 写失败 / backup 网络抖动 / primary 写失败 / 进程崩溃后 retry_loop 恢复 / 后台 task 丢 ctx 时必须失败而不是直接读取文件
• 加密回归：primary 加密时 .redirect.json / .sync_log.json 落盘为 OVE1 信封；primary 明文时为 JSON；加密 primary + 不加密 backup 下读写往返；retry_loop 通过路径恢复 ctx 后可读取加密 sync log
• 语义回归：redirect 文件在 ls/grep 可见且内容正确；exclude 文件不写入对应 backup；同名多 backup（多 AZ）下 .sync_log.json 按 name 区分；.redirect.json 与 .sync_log.json 同目录并发更新不丢 entry

English Version / 英文版

Overview

This document mainly discusses the concrete implementation of multiple storage backends in OpenViking. The features include support for multiple backend configurations, support for primary/backup role assignment, support for file redirection, support for file exclusion, encryption configuration adjustments, and support for synchronous/asynchronous data strategies.

Design philosophy:
Keep file-related operations as consolidated as possible within RAGFS to reduce pressure on viking_fs

Background

1. OpenViking currently supports flexible replacement of storage backends, such as localfs, s3fs, kvfs, memfs, etc.
2. The viking_fs layer of OpenViking supports multiple operations on resources.
3. Different storage backends have different characteristics or specialized features.
4. High data availability.

Scenarios

Primary/Backup Homogeneous

Primary/Backup homogeneous means that the primary/backup data sources configured in OV are of the same type, for example both are localfs. Primary/Backup homogeneous is mainly applicable to the following scenarios.

High Data Availability

Dual-write/multi-write, as one type of backup strategy, is equivalent to continuous data protection (CDP). The secondary storage holds a complete real-time replica of the primary storage. OpenViking itself can guarantee high data availability, with RPO approaching zero.

Multi-AZ Routing Priority

When instances are deployed in different AZs, network latency across different AZs can greatly affect the overall request response. With multi-AZ backup nodes, request responsiveness can be improved.

Primary/Backup Heterogeneous

Data Migration

When data needs to be migrated from an old system to a new one, using only ovpack will lose some real-time data. With dual-write + ovpack, zero loss can be achieved.

Heterogeneous Data Services

The same data can be stored in different forms of storage to provide richer capabilities, for example:

• Store in TOS to provide preview capability
• Store in kvfs to provide high-speed read capability
• Store in a lakehouse to provide data analytics capability

Terminology Definitions

• backend: The underlying handle initialized by the backend plugin registry based on backend + params (such as localfs, s3fs, memfs, kvfs). It is not aware of encryption and is only responsible for raw byte read/write. It may only be used as constructor input for wrappers, must not be exposed externally, and it is also forbidden to use it to directly write business data/metadata.

• primary_backend: The write handle for primary inside ragfs (not visible to upper layers). When global encryption is enabled, it is EncryptionWrappedFS(primary); otherwise it is primary. MultiWriteWrappedFS uses it when writing both business files and system metadata such as .redirect.json / .sync_log.json, thereby ensuring consistent encryption semantics under the primary namespace. Upper layers (viking_fs / Python) only perceive the unified FileSystem trait and do not perceive primary/backup.

• backup_backend: The write handle for backup inside ragfs (not visible to upper layers). When global encryption is enabled and the backup itself has encryption.enabled=true, it is EncryptionWrappedFS(backup); otherwise it is backup.

• Metadata bypass direct write (negative example): Refers to retaining an unencrypted primary handle, that is, bypassing EncryptionWrappedFS, and directly writing .redirect.json / .sync_log.json in plaintext. This is an incorrect path because it causes a degradation of security semantics under the same primary directory, where business files are encrypted while system files are plaintext, and it must be forbidden.

Configuration Structure

Existing configuration pattern:

// Single backend mode (existing, unchanged)
{
    "storage": {
        "workspace": "./data",
        "agfs": {
            "backend": "s3",
            "timeout": 10,
            "s3": {
                "bucket": "test-zbj",
                "region": "cn-beijing",
                "access_key": "xxx",
                "secret_key": "xxx",
                "endpoint": "http://tos-s3-cn-beijing.volces.com",
                "prefix": "",
                "use_ssl": false,
                "use_path_style": false,
                "directory_marker_mode": "nonempty"
            }
        }
    }
}

Dual-write/multi-write configuration pattern:

Add the backups configuration. When a backend is configured in backups, the corresponding backend will be used as backup storage.

// backups
{
    "storage": {
        "workspace": "./data",
        "agfs": {
            "backend": "local",
            "backups": {
                "sync_type": "async",
                "items": [
                    {
                        "name": "memfs-cache",
                        "backend": "memfs",
                        "encryption": { "enabled": false },
                        "operations": [
                            { "operation": "read", "priority": 100 }
                        ]
                    }
                ]
            }
        }
    }
}

Key Design

Multiple Backend Configuration

Compatibility Mode

The top-level backend / s3 / local and similar parameters are always used as the primary configuration.
The backends in backups.items are always used as backup configurations.

backups coexists with the original backend-map structure:

• backups is empty or does not exist -> keep the existing single-backend logic unchanged
• backups is not empty -> the top-level backend is primary, and backups.items are backups

// Simplified example: original single backend (unchanged)
{ "agfs": { "backend": "s3", "s3": { ... } } }

// Simplified example: multi-backend mode (primary = top-level backend, backup = backups.items)
{ "agfs": { "backend": "s3", "s3": { ... }, "backups": { "sync_type": "async", "items": [...] } } }

Field Definitions

Primary, the top-level backend, adds the following fields based on the existing AGFSConfig structure.

Field	Type	Required	Description
name	string	No	Default value is primary
encryption	object	No	New field for multi-write, controls whether this primary enables encryption
operations	array	No	New field for multi-write, controls which operations this primary participates in and their priorities
redirects	array	No	New field for multi-write, controls the redirection strategy of this primary

Each backup node in backups.items directly reuses AGFSConfig and additionally adds multi-write-related fields. The new fields are as follows:

Field	Type	Required	Description
name	string	Yes	New field for multi-write, logical name, globally unique, used for references in .sync_log.json / .redirect.json
encryption	object	No	New field for multi-write, controls whether this backup node enables encryption
operations	array	No	New field for multi-write, controls which operations this backup node participates in and their priorities
excludes	array	No	New field for multi-write, controls the exclusion strategy of this backup node

The container fields of backups itself include:

Field	Type	Required	Description
sync_type	string	No	Dual-write consistency mode: async / sync
write_ack_count	number	No	Minimum number of backup node acknowledgements required in sync mode
write_ack_timeout_ms	number	No	Timeout for waiting for backup node acknowledgements in sync mode
write_concurrency	number	No	Maximum background write concurrency in async mode
items	array	Yes	Backup node list; each item reuses AGFSConfig and adds multi-write fields

Configuration Example

{
  "agfs": {
    "backend": "local",
    "timeout": 10,
    "encryption": { "enabled": true },
    "redirects": [
      { "type": "FileExtensionPolicy", "extensions": ["(pdf|ppt)"], "target": ["s3-backup"] }
    ],
    "backups": {
      "sync_type": "async",
      "write_ack_count": 1,
      "write_ack_timeout_ms": 5000,
      "write_concurrency": 8,
      "items": [
        {
          "name": "s3-backup",
          "backend": "s3",
          "timeout": 10,
          "queuefs": { "backend": "sqlite" },
          "s3": {
            "bucket": "test-zbj-backup",
            "region": "cn-beijing",
            "access_key": "xxx",
            "secret_key": "xxx",
            "endpoint": "http://tos-s3-cn-beijing.volces.com",
            "prefix": "backup",
            "use_ssl": false,
            "use_path_style": false,
            "directory_marker_mode": "nonempty"
          },
          "encryption": { "enabled": false },
          "excludes": [
            { "type": "FileOverSizePolicy", "max_size_mb": 1024 }
          ],
          "operations": [{"operation": "read", "priority": 100}]
        }
      ]
    }
  }
}

Primary/Backup Distinction

Primary/backup is distinguished by configuration location, without using a role field:

Location	Role	Description
Top-level backend / s3 / local etc.	primary	The authoritative data source for all write operations; there is exactly one
backups.items[]	backup	Receives dual-write synchronization; there can be multiple

File Redirection (redirects)

File redirection means that certain files matching specific rules can be written to designated backends instead of being written to primary, such as files in special formats like PDF/PPT, or files exceeding a certain size.

Applicable Scenario

Primary can be configured with redirects so that specific files are not written to primary, but instead written to designated backup backends.

Supported Policies

Policy	Description	Configuration Example
FileOverSizePolicy	Redirect when the file exceeds the specified size	{"type": "FileOverSizePolicy", "max_size_mb": 100, "target": ["s3-backup"]}
FileExtensionPolicy	Redirect when the file suffix matches	`{"type": "FileExtensionPolicy", "extensions": ["(mp4

target is uniformly a string array. It can redirect to multiple backups, corresponding to the targets field in .redirect.json. extensions is an array of regex strings, consistent with the configuration example (pdf|ppt).

Configuration Example

{
  "agfs": {
    "backend": "local",
    "redirects": [
      { "type": "FileOverSizePolicy",  "max_size_mb": 100, "target": ["s3-backup"] },
      { "type": "FileExtensionPolicy", "extensions": ["(pdf|ppt)"], "target":  ["s3-backup"]}
    ],
    "backups": { "sync_type": "async", "items": [...] }
  }
}

.redirect.json Metadata

After a redirect write, a .redirect.json file is generated under the corresponding resource directory in primary to record the mapping between files and target backend names. If redirect files exist under a directory, one such file is generated for that directory.

File structure:
/resource/<tenant>/
  ├── .redirect.json       ← records redirect mappings
  └── large_pdf.pdf      ← actually stored in s3-backup,xxx-backup
  └── resource_AA_dir
              ├── .redirect.json
              └──  large_ppt.ppt

Contents of .redirect.json:

{
  "version": 1,
  "entries": {
    "large_pdf.pdf": {
      "targets": ["s3-backup", "xxx-backup"]
    }
  }
}

• .redirect.json exists only in primary and is not dual-written to backup; when primary encryption is enabled, this file must also be encrypted and persisted through primary_backend
• When deleting or moving redirected files, .redirect.json must be updated synchronously
• .redirect.json is added to _INTERNAL_NAMES and is invisible to users
• .redirect.json and .sync_log.json under the same directory must be updated serially: this is protected by the directory-level mutex of MetaStateStore; both metadata files must be read and merged, then atomically written back within the lock. Reading first and writing later without holding the lock, namely bare read_meta -> write_meta, is forbidden, otherwise high-frequency concurrent writes may overwrite the other party's entry.
  • Example: thread A writes large_pdf.pdf and needs to add an entry large_pdf.pdf -> [s3-backup] to .redirect.json; thread B simultaneously writes report.doc and needs to do latest_seq++ in .sync_log.json. Without serialization, the latest_seq read by B may be stale, and when it writes back, the redirect entry just written by A may be overwritten and lost.
  • Fix: all reads and writes to .redirect.json / .sync_log.json must go through MetaStateStore::update_dir_meta(dir, ctx, |redirect_meta, sync_meta| { ... }). Internally, dir_locks[dir] guarantees that the two metadata files in the same directory cannot be modified concurrently.

Redirect Write Flow

sequenceDiagram
    participant Client
    participant MultiWrite
    participant Primary
    participant Backup as Backup (s3-backup,xxx-backup)

    Client->>MultiWrite: write(large_pdf.pdf)
    Note over MultiWrite: Check the redirects policy<br/>Hit `FileOverSizePolicy`
    MultiWrite->>Backup: write(large_pdf.pdf)
    Backup-->>MultiWrite: OK
    MultiWrite->>Primary: write(.redirect.json)<br/>(record `large_pdf.pdf` -> `s3-backup`, `xxx-backup`)
    MultiWrite-->>Client: OK

Loading

After a redirect hit, the write targets switch from primary + write_backups to targets, but the synchronization state machine remains unchanged:

• .sync_log.json still records the latest_seq of this write, and maintains acked_seq for each backup in targets
• In sync mode, confirmation is still calculated by write_ack_count / write_ack_timeout_ms; in async mode, background tasks still advance acked_seq
• If any target fails, .redirect.json must not be written without also persisting .sync_log.json; the not-yet-caught-up state must be retained for retry_loop compensation
• After configuration changes, compensation targets are still computed from the currently effective configuration; old backup targets that have been removed or replaced are ignored

File Exclusion (excludes)

Applicable Scenarios

Only backup nodes can configure excludes, specifying which files are not written to the current backup node. For example, memfs/kvfs can be used as backup nodes to improve system responsiveness.

Policy Types

Consistent with redirects, it supports FileOverSizePolicy, FileExtensionPolicy, and so on.

Configuration Example

{
  "name": "memfs-cache",
  "backend": "memfs",
  "excludes": [
    { "type": "FileOverSizePolicy", "max_size_mb": 500 },
    { "type": "FileExtensionPolicy", "extensions": [".mp4"] }
  ]
}

Conflict with redirects

If primary is configured to redirect to backend A, but backend A excludes that file, this is considered a user configuration conflict. This design does not provide a fallback for it, and the file will not be written to any backend.

Encryption Configuration (encryption)

Encryption has already been pushed down to the RAGFS/Rust wrapper implementation. The multi-write solution can reuse EncryptionWrappedFS without handling encryption in the Python layer. backend / primary may only be used as internal constructor inputs for EncryptionWrappedFS, and must never be used to write business data or metadata directly. All writes to primary, including business files and .redirect.json / .sync_log.json, must go through the single entry point primary_backend.

Decision Logic

server.encryption.enabled
  │
  ├── false  ->  Global encryption is disabled; all backends are plaintext
  │
  └── true   ->  Global encryption is enabled
          │
          ├── primary backend  ->  **forced encryption** (follows `root_key`; configuring `primary.encryption.enabled` is not allowed)
          └── backup  backend  ->  determined by `item.encryption.enabled`; encrypt when true (default is true)

Encryption in Dual-Write Scenarios

primary and backup are allowed to use different encryption strategies:

// primary is encrypted (follows the global switch), backup (memfs) is not encrypted
{
  "agfs": {
    "backend": "local",
    // Reuse the existing global encryption.enabled configuration;
    // primary is not allowed to configure encryption.enabled again (it must follow the global setting)
    "encryption": { "enabled": true },
    "backups": {
      "items": [
        // backup may configure encryption.enabled on its own
        { "name": "memfs-cache", "backend": "memfs", "encryption": { "enabled": false } }
      ]
    }
  }
}

All backends use unified key management, and the actual encryption/decryption is handled only by Rust EncryptionWrappedFS. The multi-write implementation must reuse EncryptionWrappedFS as a per-backend data wrapper:

• primary_backend: when global encryption.enabled=true, it is EncryptionWrappedFS(primary); otherwise it is primary. Encryption cannot be disabled separately for primary.
• backup_backend: when global encryption.enabled=true and the backup's own encryption.enabled=true, it is EncryptionWrappedFS(backup); otherwise it is backup.
• .redirect.json / .sync_log.json live in the primary backend and are system files in the primary namespace. They must be read and written through primary_backend, so they inherit the primary encryption configuration.
• It is forbidden to keep an unencrypted bypass write path for internal metadata, such as writing .redirect.json / .sync_log.json directly with the unencrypted primary. Otherwise, when primary encryption is enabled, encrypted business files and plaintext system state files would coexist in the same backend, weakening the security semantics.
• In the multi-write scenario, the current global stack Stats(Encryption(Mountable)) in encrypt迁移.md / the code needs to evolve into Stats(MultiWriteWrappedFS), where MultiWriteWrappedFS internally holds primary_backend / backup_backend separately for primary/backup. The implementation of EncryptionWrappedFS remains unchanged; only the assembly location changes.

[Performance vs. Encryption Tradeoff - Architectural Choice]
If multiple backends are configured with encryption, writing the same piece of data will cause each backend's EncryptionWrappedFS to independently generate a new envelope key and encrypt once. This means the CPU cost grows proportionally with the number of encrypted backends. This is not a redundant bug, but a reasonable cost under the multi-backend parallel-write architecture: each backend's envelope is isolated, with independently generated file_key / data_iv / key_iv, and ciphertext bytes must not be shared across backends. If CPU becomes a bottleneck, some backups such as memfs or cache backends should disable encryption.enabled, rather than bypassing primary_backend to save compute. See the follow-up extensibility section below for later optimization ideas.

Internal File Filtering

_INTERNAL_NAMES Extension

The existing _INTERNAL_NAMES list needs to add the following files so that they are invisible to users:

_INTERNAL_NAMES = [
    ".path.ovlock",      // existing: lock file
    ".sync_log.json",    // new: sync log
    ".redirect.json",    // new: redirect mapping
]

Filtering Location

Apply unified filtering in VikingFS directory traversal methods such as ls, tree, and glob. These files are transparent to users, but MultiWriteWrappedFS can read and write them directly internally. If primary encryption is enabled, file names are still hidden according to the above rules, while file contents are encrypted by primary_backend.

Operation Routing (operations)

Operation Types

Dual-write is classified according to the actual methods in RAGFS.

tree, glob, read_file, write_file, append_file, move_file, ls, and so on are Python-layer APIs and can be pushed down into RAGFS as much as possible.

Operation Type	Covered Methods (Rust trait)	Description
write	write, mkdir, remove, remove_all, rename, create, chmod, truncate, ensure_parent_dirs	Write operations, must be dual-written to all writable backup nodes
read	read, stat, exists, read_dir, grep	Read operations, choose backend by priority

Priority Rules

Each backup node can declare the supported operation types and their priorities through the operations field:

{
  "name": "kvfs-cache",
  "backend": "kvfs",
  "operations": [
    {"operation": "read", "priority": 50}
  ]
}

Rule	Description
Priority must be a positive integer	Smaller value means higher priority
Top-level primary does not configure operations	primary always participates in write and serves as the final read fallback. Any operations config on primary is ignored and rejected by build-time validation
backup does not configure operations	By default, it only participates in write and does not enter read routing; this is the cold-backup semantic
backup explicitly declares read	Only then does it enter read routing, selected in ascending order of priority
backup explicitly declares write / omitted	It participates in dual-write. If it declares only read and not write, then it is read-only and not dual-written, for example a pure-read kvfs cache

[Clarification] The early behavior of "if operations is not configured, it participates in all operations by default" would make every backup enter read routing, which would easily hit stale data during the asynchronous window and degrade semantics. This is corrected to: backup participates only in write by default; it enters read routing only when read is explicitly declared. In this way, a cold backup does not affect read consistency by default, and any backup used for read acceleration, such as memfs or kvfs, must explicitly declare a read priority, making the semantics clear and consistent with the read fallback chain.

Placement Constraints

• The primary corresponding to the top-level backend is always the first stop for writes and the final fallback for read routing, and it does not configure operations
• Backup nodes under backups.items may configure read/write priorities in operations; if omitted, they participate only in write and do not enter read routing. Read acceleration requires explicitly declaring read

// This backup node only participates in reads and does not receive dual-write data
// (declares only read, without write)
{ "name": "kvfs-cache", "backend": "kvfs", "operations": [{"operation": "read", "priority": 1}] }

Read Operation Routing (Fallback Chain)

Read operations are attempted in the following order, returning immediately once found:

1. Try read-enabled backup backends in ascending order of priority
       │
       │  file does not exist
       ▼
2. Fallback to primary
       │
       │  file does not exist
       ▼
3. Query `targets` in `.redirect.json` on primary, possibly multiple targets, and read them one by one in target order, returning immediately on the first hit
       │
       │  file does not exist
       ▼
4. Return NotFound

[Boundary] When redirect targets contains multiple entries, read routing tries them one by one in the order of the targets list. This is typical for heterogeneous replicas where the content is consistent, so hitting the first available one is enough. stat / exists behave the same.

sequenceDiagram
    participant Client
    participant MultiWrite
    participant Backup as Backup (read-enabled)
    participant Primary
    participant Redirect as .redirect.json

    Client->>MultiWrite: read(path)
    MultiWrite->>Backup: read(path)
    Backup-->>MultiWrite: NotFound
    MultiWrite->>Primary: read(path)
    Primary-->>MultiWrite: NotFound
    MultiWrite->>Primary: read(.redirect.json)
    Primary-->>MultiWrite: targets=["s3-backup"]
    MultiWrite->>Redirect: read(path) (s3-backup)
    Redirect-->>MultiWrite: data
    MultiWrite-->>Client: data

Loading

Write Operation Routing

All write operations are always written to primary first, and then synchronously or asynchronously written to all writable backup nodes according to sync_type:

sequenceDiagram
    participant Client
    participant MultiWrite
    participant Primary
    participant Backup as Backup (write-enabled)

    Client->>MultiWrite: write(data)
    MultiWrite->>Primary: write(data)
    Primary-->>MultiWrite: OK

    Note over MultiWrite: sync mode: wait for all backup nodes

    MultiWrite->>Backup: write(data)
    Backup-->>MultiWrite: OK
    MultiWrite-->>Client: OK

Loading

Directory-Like Operation Routing

Operation	Routing Strategy	Reason
read_dir	primary, and merge redirect file names from .redirect.json in the current directory	The actual redirect file body is not in primary; it must be added back, otherwise users will not see files they wrote in ls, which degrades semantics
grep	primary first; when a redirect file name is hit, read the content from the target backend on demand for matching	Otherwise the content of redirect files will be missed in matching
stat / exists	priority -> primary -> redirect	Same as the read fallback chain

[Redirect File Visibility, Semantic Degradation Protection] The actual data body of a redirect file is written to the backup, while only the .redirect.json metadata exists on primary. If read_dir returns only the physical entries on primary, users will not see redirected files in ls. The fix is: after returning the real entries on primary, read_dir reads the entries key set from .redirect.json in the current directory and adds a virtual FileInfo for file names that do not exist on primary. The size/mtime comes from the redirect record or a stat call back to the target. grep behaves similarly: when iterating over a redirect file name, it reads the content from the target backend and includes it in matching. .redirect.json / .sync_log.json themselves are always filtered by _INTERNAL_NAMES and never appear in results.

Higher-level Python APIs such as ls, tree, and glob internally compose trait methods such as read_dir, and naturally fall into the primary routing of MultiWriteWrappedFS.

File Sync Log (.sync_log.json)

Role

.sync_log.json is a metadata file that records the synchronization progress of backup nodes. Both sync and async modes use it to express whether latest_seq has caught up with each backup's acked_seq.

What it records is not whether a file has ever been synchronized, but the latest version number of the current path and which version each backup has caught up to.

Storage Rules

Rule	Description
Location	Under each directory of primary
Granularity	One .sync_log.json per directory; currently no sharding
Visibility	Added to _INTERNAL_NAMES and invisible to users
Read operations	Do not depend on this file; reads are routed by priority
Encryption	Follows the encryption configuration of the primary backend; when primary encryption is enabled, it must be persisted through primary_backend with encryption

Content Structure

{
  "entries": {
    "file.txt": {
      "latest_seq": 12,
      "last_op": "write",
      "backends": {
        "memfs-cache": {
          "acked_seq": 12
        },
        "local-az2": {
          "acked_seq": 11
        }
      }
    },
    "report.pdf": {
      "latest_seq": 3,
      "last_op": "remove",
      "backends": {
        "memfs-cache": {
          "acked_seq": 3
        }
      }
    }
  }
}

Field semantics:

• The key of entries uses the file name under the current directory. Since .sync_log.json itself is directory-level metadata, the directory path is already implied by the file location, so the full uri does not need to be stored again
• latest_seq: the monotonically increasing version number of the latest successful write to primary for the current path; synchronization completion is judged against it
• last_op: the operation type corresponding to this latest_seq, namely write / mkdir / remove / remove_all / rename / chmod / truncate. retry_loop uses this to decide how to replay on lagging backups: for write / truncate, read the latest content back from primary, or the redirect target, and rewrite it; for remove / remove_all, replay deletion; for mkdir / chmod, replay the corresponding operation; for rename, additionally record the target URI in rename_to. Without this field, retry cannot replay non-content operations such as delete or rename, causing semantic degradation
• rename_to, present only when last_op == rename: the normalized target uri of the rename operation. The full path is intentionally retained here because rename may cross directories
• backends.{name}.acked_seq: the latest version number that this backup has successfully acknowledged and applied
• The keys in backends use the configured backend name as the identity

account_id is not written into .sync_log.json to avoid redundant source-of-truth fields. When the background retry_loop needs ctx, it must reconstruct FsContext from the physical path of the primary where the metadata file resides, according to the same-origin rule of _uri_to_path. If the path cannot be reversed to an account_id, it means the directory does not satisfy the prerequisite for encrypted background recovery, and the process must report an error and skip it, rather than falling back to an empty ctx or an unencrypted backend handle.

Synchronization completion condition:

For `entries[file_name]`:
  `target_backends` = compute the set of backup backends that this file should currently be written to based on the currently effective configuration
                     (configuration changes take effect only after restart; if a backup is replaced, the old target is ignored)

  For each `backup ∈ target_backends`
    if `backends[backup].acked_seq == latest_seq`
    then this `backup` has been synchronized to the current latest version

If and only if all `target_backends` satisfy the above condition,
the file is considered synchronized.

Lifecycle

When writing a file, in either `sync` or `async` mode:
  1. Generate a new `latest_seq`
  2. Update `.sync_log.json`
       ├── Write the `latest_seq` for this `file_name`
       └── Preserve the existing `acked_seq` of each backup
  3. `sync`: wait for quorum; `async`: `tokio::spawn { backup.write(data) }`
       ├── Success or acknowledged -> update this backup's `acked_seq` to `latest_seq`
       └── Failure, timeout, or not acknowledged -> keep `acked_seq` unchanged

Background retry, see the failure retry mechanism below:
  `retry_loop` scans `.sync_log.json`
    ├── Computes `target_backends` based on the currently effective configuration, requiring restart for config changes; old targets are ignored
    ├── Finds backups with missing `acked_seq` or `acked_seq < latest_seq` and retries the write
    └── Skips when all `target_backends` satisfy `acked_seq == latest_seq`

Manual Sync Interface

Provide a CLI tool for operations to manually trigger synchronization status checks and repairs:

ov system backend sync-status viking://resources     # View synchronization status under viking://resources
ov system backend sync-retry  viking://resources     # Manually retry failed synchronizations under viking://resources

The CLI reads and writes sync state through dedicated APIs newly added to the ragfs-python binding, such as client.system_sync_status(path, ctx) / client.system_sync_retry(path, ctx). It does not read the .sync_log.json file directly. Metadata is read and written uniformly through the metadata API of MultiWriteWrappedFS, and internally goes through primary_backend, thereby inheriting the primary encryption configuration, directory-level metadata guard, and concurrency locks, avoiding offline reads and lock bypass.

Concurrency Safety

.sync_log.json and .redirect.json are both directory-level JSON metadata files, and read-modify-write operations have concurrency race risks. Foreground writes, multiple tokio::spawn ack updates, retry_loop, and manual sync-retry may all modify the same directory at the same time.

Unified MetaStateStore

The following begins the concrete implementation constraints:

• MetaStateStore: the directory-level metadata manager inside MultiWriteWrappedFS, responsible for unified read-modify-write of .redirect.json and .sync_log.json, and holding directory locks and path-serialization queues
• FsContextResolver: a helper for restoring context in background tasks. When retry_loop, backfill, or system_sync_retry has no foreground request context, it restores account_id from the normalized path, then executes encrypted reads and writes inside FS_CTX.scope(ctx, ...)

/// Uniformly encapsulates read-modify-write for .redirect.json / .sync_log.json.
/// All methods must execute on `primary_backend`, thereby automatically inheriting the primary encryption configuration.
struct MetaStateStore {
    primary_backend: Arc<dyn FileSystem>,
    dir_locks: Mutex<HashMap<String, Arc<tokio::sync::Mutex<()>>>>,
    path_queues: PathSerializer,
    ctx_resolver: Arc<dyn FsContextResolver>,
}

impl MetaStateStore {
    /// Serially executes read-modify-write on metadata files within the same directory to prevent lost updates.
    async fn update_dir_meta<T>(&self, dir: &str, ctx: FsContext, op: T) -> Result<()>
    where
        T: FnOnce(&mut RedirectMeta, &mut SyncLogMeta) -> Result<()>;
}

Whenever .redirect.json or .sync_log.json is updated, the lock for the corresponding directory must be acquired first, and the read-modify-write of both files must be completed within the same critical section. In redirect write scenarios, the entries update in .redirect.json and the latest_seq update in .sync_log.json must be committed under the same lock, to avoid the situation where data has already been written to the redirect target, but the redirect mapping or sync version is lost.

In addition, to avoid out-of-order effects of multiple writes on the same path on the backup side, this must be paired with a per-path serialization queue. .sync_log.json is responsible for recording which version each backup has acknowledged, while per-path FIFO is responsible for ensuring versions never move backward.

For rename, if the source directory and target directory differ, a single directory lock is insufficient and must be upgraded to dual-directory locking:

• First sort the directory strings in a stable order, such as lexicographic order, then acquire the two locks in that order. Arbitrary locking by call order is forbidden, otherwise deadlocks may occur
• Update .redirect.json and .sync_log.json of both the source directory and target directory within the same critical section
• If source_dir == target_dir, degrade to a single directory lock
• rename_to is used only to record the target uri; actual metadata consistency is guaranteed by the dual-directory lock rather than being delegated to background retry

Synchronization Mode (sync_type)

OpenViking supports different consistency strategies through sync_type:

• Strong consistency, namely synchronous dual write: success is returned to the client only when the write to the primary succeeds and the write to the replica also succeeds. Risk: the slowest backend determines overall latency; if the replica write fails, the primary needs rollback
• Eventual consistency, namely asynchronous dual write: success is returned immediately after writing to the primary storage, and synchronization to replica storage happens asynchronously in the background. Risk: during the dual-write window, backends are inconsistent, and read requests may be routed to stale data

Two Modes

Mode	Value	Default	Behavior
Asynchronous dual write	async	✅	Return success after writing to primary first, then synchronize to backup in the background
Synchronous dual write	sync		Return success to the client only after primary write succeeds and backup write succeeds

Synchronous Write (sync)

Data Flow:

Client
  │
  │ write(data)
  ▼
MultiWriteWrappedFS
  │
  ├──► primary.write()  ──► OK
  │
  ├──► backup1.write() ──► OK
  │
  └──► backup2.write() ──► OK
  │
  ▼
Return success to Client

Sequence Diagram:

sequenceDiagram
    participant Client
    participant MultiWrite
    participant Primary
    participant Backup1
    participant Backup2

    Client->>MultiWrite: write(data)
    MultiWrite->>Primary: write(data)
    Primary-->>MultiWrite: OK
    MultiWrite->>Backup1: write(data)
    Backup1-->>MultiWrite: OK
    MultiWrite->>Backup2: write(data)
    Backup2-->>MultiWrite: OK
    MultiWrite-->>Client: OK

Loading

Quorum Support in Synchronous Mode

Current design: in sync mode, the call returns only after all backup nodes have completed writing. One slow backup node can drag down overall latency. Add a minimum acknowledgment count and timeout:

{
  "backups": {
    "sync_type": "sync",
    "write_ack_count": 2,
    "write_ack_timeout_ms": 5000
  }
}

Parameter	Default	Description
write_ack_count	All backup nodes	Minimum number of backup nodes that must acknowledge a successful write
write_ack_timeout_ms	No timeout	Timeout for waiting for backup node acknowledgment in milliseconds; timed-out backup nodes are compensated by retry

Note

Risks and Failure Semantics:

• The slowest backup node determines overall latency. This is mitigated by write_ack_count + write_ack_timeout_ms: return as soon as quorum is reached
• Primary write failure -> the entire write returns failure immediately, and no backup write is triggered at all. Data is not persisted, so there is no inconsistency
• Primary succeeds, backup quorum is not reached -> return failure to the client, but primary data has already been persisted and cannot be rolled back because the file system has no transaction support. At this point, .sync_log.json has already recorded latest_seq, and backups that are not yet caught up are continuously compensated by retry_loop until eventual consistency is reached. In other words, the client sees a failure, but the system guarantees that the data will eventually be synchronized. This semantic must be made explicit in the API documentation to prevent callers from misjudging it as data loss

Asynchronous Write (async)

Data Flow:

Client
  │
  │ write(data)
  ▼
MultiWriteWrappedFS
  │
  ├──► primary.write()  ──► OK  ──► immediately return success to Client
  │
  └──► tokio::spawn { backup1.write() }  ──► update `.sync_log.json` after completion or failure
       tokio::spawn { backup2.write() }  ──► update `.sync_log.json` after completion or failure

Sequence Diagram:

sequenceDiagram
    participant Client
    participant MultiWriteWrappedFS
    participant Primary
    participant Backup1

    Client->>MultiWriteWrappedFS: write(data)
    MultiWriteWrappedFS->>Primary: write(data)
    Primary-->>MultiWriteWrappedFS: OK
    MultiWriteWrappedFS-->>Client: OK

    Note over MultiWriteWrappedFS: tokio::spawn

    MultiWriteWrappedFS->>Backup1: write(data)
    Note right of Client: Client has already received the response
    Backup1-->>MultiWriteWrappedFS: OK/ERR
    Note over MultiWriteWrappedFS: Update `.sync_log.json`

Loading

Note

Risks:

• Backup and primary data are inconsistent during the dual-write window
• Read requests may be routed to stale data, which is acceptable under eventual consistency
• Recovery from backup node write failures depends on background retry, see asynchronous write failure retry

Asynchronous Write Concurrency Control

Currently, there is no limit on the concurrency of asynchronous writes. If a future scenario appears where backup-node write speed is slower than caller write speed, for example primary=localfs and backup=S3, tokio::Semaphore rate limiting can be enabled through the write_concurrency configuration:

// Extensible in the future
{ "backups": { "sync_type": "async", "write_concurrency": 8 } }

• When not configured: unlimited
• After configuration: at most N asynchronous write tasks may be in flight simultaneously

Asynchronous Write Ordering Guarantee, Per-Path Serialization

Under asynchronous dual write, multiple operations on the same file path, such as write -> remove -> write, may execute out of order on backup nodes, resulting in inconsistent final data. The solution is to maintain a FIFO queue per path so that operations on the same path execute in order on backup nodes.

• Independent queue for each path; different paths still run in parallel
• The worker is automatically recycled when the queue becomes empty

Read-Path Consistency Clause

In asynchronous mode, delete operations may be delayed on backup nodes. Read requests may hit stale data that has not yet been deleted:

• Default behavior: tolerate eventual consistency, with no additional checks

Failure Retry Mechanism

In synchronous dual-write scenarios, backup-node writes may time out and fail due to temporary faults such as network jitter. Even if quorum is satisfied and success is returned to the client, backups that have not reached ack still need compensation.

Under asynchronous dual write, backup-node writes in tokio::spawn may also fail due to temporary faults such as network jitter.

Both modes persist the not-yet-caught-up state through .sync_log.json:

• async: update latest_seq immediately after writing the primary; update acked_seq only after the spawned write succeeds
• sync: return as soon as write_ack_count is reached; update latest_seq synchronously when writing the primary, and do not update acked_seq for unacknowledged, timed-out, or failed backups so they remain behind and can be compensated by retry_loop

Therefore, retry_loop must run in both modes. As long as there is any write-enabled backup, new() starts retry_loop immediately, no longer limited to async.

See the MultiWriteWrappedFS structure section: in new(), if !inner.write_backends().is_empty() { tokio::spawn(Inner::retry_loop(...)) }.

Parameter definitions:

Parameter	Default	Description
Scan interval	30s	Periodically scan .sync_log.json
Per-file retry limit	3 times per round	Avoid infinite retry loops for a single file

Independent task queue

.sync_log.json itself is already a persistent record. There is no need to introduce an external task queue such as RabbitMQ or Kafka. Data flow:

Write operation  ->  generate a new `latest_seq`
                                  │
retry_loop (every 30s) ◄─────────────┘
  │
  ├── Read `.sync_log.json`
  ├── Compute `target_backends` based on the current configuration
  ├── Find backups with missing `acked_seq` or `acked_seq < latest_seq`
  ├── Retry write -> after success, update that backup's `acked_seq` to `latest_seq`
  └── Skip when all `target_backends` satisfy `acked_seq == latest_seq`

Concrete Implementation

Encryption capability is pushed down to the ragfs layer, and the dual-write capability is implemented in the ragfs layer. The Python layer is responsible only for passing configuration.

VikingFS (Python)                  ← URI mapping, access control, vector index synchronization, lock management
    ↓                                    (not aware of dual write, not aware of encryption)
RAGFSBindingClient (PyO3)          ← FFI bridge, passing primary + backups configuration
    ↓
StatsWrappedFS                     ← Top-level end-to-end statistics, including dual-write / encryption latency
    ↓
MultiWriteWrappedFS                ← [New] dual-write routing, read fallback, sync_log/redirect management
    ├── primary_backend        ← Actual primary entry; when global encryption is enabled, this is `EncryptionWrappedFS(primary)`, otherwise `primary`
    └── backup[].backend       ← Actual backup entry; when global encryption is enabled + `backup.encryption.enabled`, this is `EncryptionWrappedFS(backup)`, otherwise `backup`
            ↓
        localfs / s3fs / memfs / ...       ← Actual storage backends

Responsibilities of each layer:

• VikingFS: only performs URI to path conversion, access control via _ensure_access, vector index synchronization, and lock management. Configuration is passed to the Rust side
• MultiWriteWrappedFS: receives the primary configuration and the backups.items list, and manages read/write routing, .sync_log.json, and .redirect.json; its internal metadata is also read and written through primary_backend
• EncryptionWrappedFS: a transparent wrapper that is invisible to upper layers. Each backend independently decides whether it should be wrapped. The primary must follow the global switch and must not bypass encryption through an unencrypted backend handle

Configuration Passthrough (FFI)

[Feasibility hard constraint - the current FFI cannot pass nested structures through, so it must be extended first]

Check the current code status:

• types.rs PluginConfig is only {name, mount_path, params: HashMap<String, ConfigValue>}; ConfigValue supports only String / Int / Bool / StringList, and does not support nested dict / list
• lib.rs::py_dict_to_config degrades non-scalar values into the repr string of v.str()
• lib.rs::mount only accepts a flat config dict
• agfs_utils.py::mount_agfs_backend mounts a single backend to /local, with no assembly for a backups group

Therefore, the nested backups structure cannot be passed through the existing channel. The minimal change set is as follows, recommended because it reuses the existing JSON capability:

1. Add a new ConfigValue::Json(serde_json::Value) variant in Rust, and make py_dict_to_config recursively convert dict/list into serde_json::Value. lib.rs already has serde_json_to_py for reverse conversion, so only a forward py_any_to_json needs to be added. This avoids defining PyO3 extraction logic for every nested field and allows the entire backups structure to pass through as a single JSON value.
2. Add new fields to PluginConfig:

// crates/ragfs/src/core/types.rs
pub struct PluginConfig {
    pub name: String,
    pub mount_path: String,
    pub params: HashMap<String, ConfigValue>,
    // [New] Multi-write config; None = single-backend mode (current behavior unchanged)
    pub backups: Option<BackendsConfig>,
    // [New] Global encryption switch (server.encryption.enabled)
    pub server_encryption_enabled: bool,
    // [New] Encryption / redirects of primary itself (multi-write fields of the top-level backend)
    pub primary_encryption_enabled: bool,
    pub primary_redirects: Vec<RedirectPolicy>,
}

#[derive(Debug, Clone, Deserialize)]
pub struct BackendsConfig {
    pub sync_type: Option<String>,            // "sync" | "async", default async
    pub write_ack_count: Option<usize>,
    pub write_ack_timeout_ms: Option<u64>,
    pub write_concurrency: Option<usize>,
    pub items: Vec<BackendItemConfig>,
}

#[derive(Debug, Clone, Deserialize)]
pub struct BackendItemConfig {
    pub name: String,                          // required, globally unique identity marker
    pub backend: String,                       // plugin type, local/s3/memfs/...
    // Plugin sub-parameters of this backend, following the top-level convention:
    // use the child object keyed by backend type.
    // For example, when backend="s3", read item["s3"]; when backend="local", read item["local"].
    // During deserialization, extract the corresponding child object as params based on backend name.
    pub params: serde_json::Value,
    pub timeout: Option<u64>,
    pub encryption: Option<EncryptionConfig>,
    pub operations: Option<Vec<OperationItemConfig>>,
    pub excludes: Option<Vec<RedirectPolicy>>,
}

#[derive(Debug, Clone, Deserialize)]
pub struct OperationItemConfig { pub operation: String, pub priority: u32 }

#[derive(Debug, Clone, Deserialize)]
pub struct EncryptionConfig { pub enabled: bool }

#[derive(Debug, Clone, Deserialize)]
#[serde(tag = "type")]
pub enum RedirectPolicy {
    FileOverSizePolicy { max_size_mb: u64, target: Option<Vec<String>> },
    FileExtensionPolicy { extensions: Vec<String>, target: Option<Vec<String>> },
}

3. Deserialize BackendsConfig via serde_json::from_value, where mount() recognizes params["backups"] or a standalone parameter, avoiding field-by-field PyO3 extraction.
4. On the Python side, agfs_utils.py places agfs.backups directly into the config dict passed to mount, including the plugin sub-parameters of each item; the top-level backend / s3 / local parameters continue to serve as the primary params.

This change is orthogonal to the root_key injection described in encrypt迁移.md: root_key is still injected from the encryption config during binding construction, while backups is static configuration at mount time and only determines which backends should be wrapped by EncryptionWrappedFS. After multi-write mounting is enabled, the global Stats(Encryption(Mountable)) stack can no longer be used as the only data stack. Instead, build_multi_write_fs() should assemble backend for each BackendEntry.

MountableFS::mount() Integration

In mount(), detect config.backups: if empty, keep the current single-backend logic unchanged; if not empty, call the newly added private method build_multi_write_fs() to construct MultiWriteWrappedFS, and still wrap it with StatsWrappedFS on the outside to keep metrics consistent. The primary and backup inside a multi-write mount should no longer be mounted back into the same MountableFS; instead, after registry initialization, each backend should be wrapped into BackendEntry::backend according to configuration, preventing the global Encryption wrapper and the multi-write scheduling order from bypassing each other.

// Pseudocode: branch inside mount()
let inner_fs: Arc<dyn FileSystem> = match &config.backups {
    None => self.build_single_backend(&config).await?,        // current logic
    Some(bc) => self.build_multi_write_fs(&config, bc).await?, // multi-write
};
let wrapped = StatsWrappedFS::new_arc(inner_fs);               // reuse existing wrapper

build_multi_write_fs() steps:

The backend assembly details begin here: BackendEntry is the backend descriptor structure stored internally by MultiWriteWrappedFS. Its fields include name, role, namely Primary / Backup, backend, operations, allowed only for backup, and excludes.

1. primary: initialize the primary from the top-level backend / params through the registry, yielding the unencrypted underlying handle; follow the global server.encryption.enabled to decide whether to wrap it as EncryptionWrappedFS(primary), and record the result as primary_backend. The primary is not allowed to disable encryption.
2. Iterate over bc.items: initialize each backup through the registry, yielding an unencrypted underlying handle; decide whether to wrap it as EncryptionWrappedFS(backup) based on server.encryption.enabled && item.encryption.enabled, and record the result as backup_backend.
3. Validation: there must be exactly one primary and it must be the top-level one; item.name must be globally unique; the primary must not accept operations, otherwise report an error; every redirects.target / excludes reference must point to an existing name.
4. Build BackendEntry { name, backend, role, operations, excludes } and assemble them into MultiWriteWrappedFS. Internal metadata files must use primary_backend; it is forbidden to keep an unencrypted primary handle as a metadata side-channel direct-write entry point.

MultiWriteWrappedFS Structure, Arc<Inner> Solidified

[Feasibility - async spawn and retry_loop must share Arc] The async dual-write tokio::spawn closure and the background retry_loop both need to write back to .sync_log.json, so they must hold Arc to themselves. Calling tokio::spawn(self...) directly would move self. Solidify it into the pattern wrapper + Arc<Inner>:

pub struct MultiWriteWrappedFS { inner: Arc<Inner> }

struct Inner {
    backends: Vec<BackendEntry>,
    primary_idx: usize,
    sync_type: SyncType,
    write_ack_count: Option<usize>,
    write_ack_timeout_ms: Option<u64>,
    write_sem: Option<Arc<tokio::sync::Semaphore>>,   // write_concurrency throttling
    redirects: Vec<RedirectPolicy>,                   // redirect policy of primary
    meta_store: MetaStateStore,                        // encrypted metadata + per-dir guard
    path_queues: PathSerializer,                       // per-path serialization queue
}

impl MultiWriteWrappedFS {
    pub fn new(...) -> Result<Self> {
        let inner = Arc::new(Inner { ... });
        if !inner.write_backends().is_empty() {
            tokio::spawn(Inner::retry_loop(Arc::clone(&inner)));
        }
        Ok(Self { inner })
    }
}

// FileSystem trait delegates to self.inner.*; spawn closures use Arc::clone(&self.inner)

[Metadata and encryption layering - critical semantic safeguard] .redirect.json / .sync_log.json are primarily stored in the primary backend and must follow the primary encryption configuration. If the primary has encryption enabled, both files must also be persisted as OVE1 envelopes; if the primary does not enable encryption, they remain plain JSON. Any unencrypted metadata side-channel direct write is forbidden.

Plan: MetaStateStore holds primary_backend, and all metadata reads and writes go through that wrapped entry. Foreground requests use the current FS_CTX; async backup ack tasks clone the current FsContext before spawn and re-enter FS_CTX.scope(ctx, ...) inside the closure; retry_loop, backfill, and system_sync_retry recover account_id from the normalized path through FsContextResolver, then re-enter scope. If the context cannot be recovered from the path, an error must be returned and that metadata file must be skipped. Falling back to direct file reads is not allowed.

Reusable Abstractions in Code, Deduplicate Write / Read / Policy in Three Places

Hard requirement from the user: abstract, extract, reuse. The current write methods such as write, mkdir, remove, remove_all, rename, chmod, truncate, and ensure_parent_dirs are highly repetitive in logic. A common skeleton must be extracted; copying and pasting eight versions is forbidden.

1. Write fanout skeleton fanout_write for all write operations:

/// Apply the same closure op to all write-enabled backups, branching by sync_type.
/// sync: collect acknowledgements according to write_ack_count / timeout;
/// async: spawn + update sync_log.
async fn fanout_write<F, Fut>(&self, path: &str, op: F) -> Result<()>
where F: Fn(Arc<dyn FileSystem>) -> Fut + Send + Sync + 'static,
      Fut: Future<Output = Result<()>> + Send {
    let targets = self.write_backups_for(path);   // filter out backups hit by excludes
    match self.sync_type {
        SyncType::Sync  => self.fanout_sync(path, &targets, &op).await,
        SyncType::Async => { self.fanout_async(path, targets, op); Ok(()) }
    }
}
// Each method only needs: write primary first, then self.fanout_write(path, |fs| async move { fs.mkdir(...) }).await

2. Read backend resolution resolve_read_backend to reuse the fallback chain for read, stat, and exists:

/// Return the backend handle that should be hit, priority ascending -> primary -> redirect target.
async fn resolve_read_backend(&self, path: &str) -> Option<Arc<dyn FileSystem>>;
// read/stat/exists all call it first to get fs, then execute the corresponding trait method;
// if no read backend exists, go directly to primary to avoid an empty loop.

3. File policy FilePolicy trait for shared logic between redirects and excludes:

/// redirects and excludes share one policy evaluation implementation to avoid duplicated logic.
trait FilePolicy { fn matches(&self, path: &str, size: u64) -> bool; }
impl FilePolicy for RedirectPolicy { /* FileOverSize / FileExtension */ }
// redirect: if matched -> write target backup + record .redirect.json;
// exclude: if matched -> skip this backup.

4. Reuse metadata JSON encoding and decoding. .redirect.json and .sync_log.json are both JSON and meet the constraint that internal system state should use JSON. They share generic helpers MetaStateStore::read_meta::<T>(), write_meta::<T>(), and update_dir_meta(), all based on primary_backend + serde_json, without handwritten parsing duplication. When encryption is enabled on the primary, the helpers read and write encrypted envelopes and EncryptionWrappedFS decrypts before deserialization; when encryption is not enabled, the helpers read and write plain JSON directly.

5. Reuse FsContext snapshots. All background tasks must explicitly carry or recover ctx, avoiding repeated scattered FS_CTX.scope calls:

/// Clone FsContext from the current task; return an error if no ctx exists.
/// Background tasks must not operate on encrypted metadata with an empty ctx.
fn current_required_ctx() -> Result<FsContext>;

/// Execute primary/backup filesystem operations under the specified ctx.
async fn with_fs_ctx<F, Fut, T>(ctx: FsContext, op: F) -> Result<T>
where
    F: FnOnce() -> Fut,
    Fut: Future<Output = Result<T>>;

Complete Write-Path Flow, Including redirect / exclude

write(path, data, offset, flags):
  0. ctx = current_required_ctx()                      # same source as _uri_to_path
  1. Evaluate primary.redirects using FilePolicy:
       matched => targets = policy target backups
              1.1 Execute write on target `backup_backend` in FIFO order inside path_queues[path]
              1.2 meta_store.update_dir_meta(dir, ctx, ...)
                  ├── Write .redirect.json through `primary_backend`, record file_name -> targets
                  └── Update .sync_log.json through `primary_backend`, latest_seq++ and initialize ack state for targets
              1.3 In sync mode, wait for target confirmations by quorum; in async mode, background tasks advance `acked_seq`
              1.4 If any target is not caught up, keep the lagging `acked_seq` and let `retry_loop` compensate
              return
  2. Write `primary_backend`, namely business data through the primary encryption layer
  3. self.fanout_write(path, |fs| fs.write(...))     # automatically filters excludes
  4. Async mode: clone ctx before spawn; inside spawn, use `FS_CTX.scope(ctx, ...)` to write backup and update `acked_seq`

File List

File	Action	Notes
crates/ragfs/src/core/multiwrite_wrapper.rs	New	MultiWriteWrappedFS + Inner + BackendEntry::backend + fanout_write / resolve_read_backend / FilePolicy
crates/ragfs/src/core/multiwrite_meta.rs	New	MetaStateStore + MetaLockProvider + encrypted JSON metadata helper + ctx recovery
crates/ragfs/src/core/types.rs	Modified	ConfigValue::Json; add backups/encryption fields to PluginConfig; BackendsConfig, etc.
crates/ragfs/src/core/mountable.rs	Modified	mount() recognizes backups; add build_multi_write_fs(); multiwrite mount directly initializes backend entries instead of detouring through global Encryption(Mountable)
crates/ragfs/src/core/mod.rs	Modified	pub mod multiwrite_wrapper; pub mod multiwrite_meta;
crates/ragfs/src/core/filesystem.rs	Unchanged	Trait signatures remain unchanged
crates/ragfs-python/src/lib.rs	Modified	py_dict_to_config supports nesting; mount() passes backups through; add system_sync_status(path, ctx) / system_sync_retry(path, ctx)
openviking/utils/agfs_utils.py	Modified	Assemble agfs.backups and pass it into mount
openviking/storage/viking_fs.py	Modified	Add .sync_log.json / .redirect.json to _INTERNAL_NAMES

Follow-up Extensibility

Multi-process / Multi-instance Boundary

This phase does not implement cross-process or multi-instance concurrency coordination. The tokio::sync::Mutex in the current design can only guarantee safety within a single process. If future deployment allows multiple RAGFS processes to write to the same primary mount at the same time, an independent MetaLockProvider must be added, such as backend-native conditional write, Redis, or etcd.

Constraint requirements:

• Before a cross-process lock provider is integrated, the system must not claim multi-instance write safety
• If multi-instance write mode is to be supported in the future, startup must explicitly verify that locking capability exists; it must not silently degrade to a local mutex

Write Amplification Risk

This phase does not implement optimization for directory-level metadata write amplification. In the current design, every file write rewrites the .redirect.json and .sync_log.json of its directory, which may become a bottleneck in hot directories.

Optional follow-up optimization directions:

• Use the hash of the normalized URI as the shard key and split into multiple shard files, still managed under directory locks and MetaStateStore
• Switch to a batch or append-only log structure to reduce full-file rewrite overhead

Wrapper Group Scheduling

The current design directly holds primary_backend and backup_backend inside MultiWriteWrappedFS. If future work needs to further optimize the repeated cost of per-backend wrapper assembly, GroupedWrappedFS can be considered, but it is not implemented in this iteration.

Core idea:

• GroupedWrappedFS classifies requests by stable group characteristics and attaches different downstream wrapper chains to different groups
• For example:
  • StatsWrappedFS -> GroupedWrappedFS -> EncryptionWrappedFS -> MultiWriteWrappedFS
  • StatsWrappedFS -> GroupedWrappedFS -> MultiWriteWrappedFS
• The encrypted group goes through EncryptionWrappedFS -> MultiWriteWrappedFS, while the plaintext group goes directly through MultiWriteWrappedFS, thus avoiding the mental burden of deciding downstream wrappers one by one for each backend in the current design

This abstraction is not only applicable to multi-write, but can also be extended to single-write scenarios. For example, different wrapper combinations can be loaded for different requests based on configuration, path, or capability tags, instead of hardcoding whether a wrapper is mounted into the global mount chain.

Prerequisites and boundaries for implementation:

• The grouping condition of GroupedWrappedFS must be stable and replayable. If the write path is grouped in a certain way, the read path must also be able to hit the same wrapper chain with the same rule, otherwise semantic confusion like encrypted on write but plaintext on read will occur
• The grouping condition must not depend on information that is visible only during writes but cannot be recovered during reads. If it depends on such information, the grouping result must be persisted as metadata for reuse by reads, retries, backfill, and retry_loop
• Observability should remain layered: StatsWrappedFS measures total latency, GroupedWrappedFS can additionally expose hit counts by group, and EncryptionWrappedFS continues to measure encryption-path performance
• If this plan is implemented in the future, the responsibility boundary of build_multi_write_fs() must be revisited. It will no longer directly assemble the final wrapper, but instead produce grouping rules and the downstream chain corresponding to each group

Conclusion: GroupedWrappedFS is a feasible future optimization direction, and it is more general than deciding wrappers per backend. However, it significantly raises mount orchestration complexity, so it is only worth introducing after confirming that the current combination path of multi-write plus encryption has become the main source of complexity.

Observability

MultiWriteWrappedFS should expose metrics through StatsWrappedFS:

• Backup node synchronization success rate and latency
• Retry count
• .sync_log.json size

Health Check

Add multi-write health status, including primary availability and the availability of each backup node, into openviking/server/routers/system.py::checks["agfs"].

Test Matrix

At minimum, the implementation should cover:

• Dimension combinations: sync/async x primary(local/s3) x backup(memfs/local/s3) x encryption(on/off) x redirects(on/off) x excludes(on/off)
• Fault injection: backup write failure / backup network jitter / primary write failure / retry_loop recovery after process crash / background task missing ctx must fail instead of reading files directly
• Encryption regression: when primary encryption is enabled, .redirect.json / .sync_log.json must persist as OVE1 envelopes; when primary is plaintext, they remain JSON; round-trip read/write with encrypted primary + unencrypted backup; retry_loop can read encrypted sync logs after recovering ctx from path
• Semantic regression: redirected files remain visible and correct in ls / grep; excluded files are not written to the corresponding backup; under multiple backups with the same file name (multi-AZ), .sync_log.json distinguishes by name; concurrent updates to .redirect.json and .sync_log.json in the same directory do not lose entries