[Bug]: Local vectordb fails on large files due to 65535 byte field limit

[dfwgj]

Bug Description

When uploading a code repository as a knowledge resource, large source files (e.g. uv.lock, roaring.c, types.gen.ts) fail to write to the local vector database with:

RuntimeError: string field 'fields' exceeds 65535 bytes

The root cause is that the content field (used for BM25 full-text search) stores the full file content without truncation, while the local vectordb's string field has a hard 65535 byte limit (UINT16 length prefix in bytes_row serialization).

Failed tasks are re-enqueued indefinitely, creating an infinite retry loop that overwhelms the server and causes health check timeouts.

Steps to Reproduce

1. Start OpenViking with local vectordb backend:

   {
     "storage": {
       "vectordb": { "backend": "local" }
     }
   }

2. Upload a code repository containing large files (e.g. the OpenViking repo itself, which includes uv.lock at 1.3MB, third_party/croaring/roaring.c, etc.)
3. Wait for the embedding queue to process

Expected Behavior

• Files exceeding the vectordb field size limit should be skipped or truncated before embedding
• Failed embedding tasks should have a retry limit, not loop forever
• The server should remain responsive (health check should pass)

Actual Behavior

1. 48 files fail with string field 'fields' exceeds 65535 bytes
2. Failed tasks are re-enqueued immediately (Re-enqueued embedding message after transient error)
3. The server becomes unresponsive — health check endpoint times out (>10s)
4. Container status shows unhealthy with FailingStreak: 7

Minimal Reproducible Example

# 上传任意 > 65535 字节的文本文件作为代码仓库资源
# 文件路径：openviking/parse/directory_scan.py
# 问题：_should_skip_file() 没有文件大小检查

def _should_skip_file(file_path: Path) -> tuple[bool, str]:
    if file_path.name.startswith("."):
        return True, "dot file"
    if file_path.is_symlink():
        return True, "symlink"
    try:
        if file_path.stat().st_size == 0:
            return True, "empty file"
        # ❌ 没有文件大小检查，大文件会被送进 embedding 管道
    except OSError:
        return True, "os error"
    return False, ""

Error Logs

2026-07-02 06:55:38,839 - openviking.storage.collection_schemas - ERROR - Failed to write to vector database: string field 'fields' exceeds 65535 bytes (uri=viking://resources/github/volcengine/OpenViking/uv.lock level=2 context_type=resource account_id=jishu-zhongtai-bu)
2026-07-02 06:56:42,118 - openviking.storage.collection_schemas - ERROR - Failed to write to vector database: string field 'fields' exceeds 65535 bytes (uri=viking://resources/github/volcengine/OpenViking/third_party/croaring/roaring.c level=2 context_type=resource account_id=jishu-zhongtai-bu)
2026-07-02 06:56:44,475 - openviking.storage.collection_schemas - ERROR - Failed to write to vector database: string field 'fields' exceeds 65535 bytes (uri=viking://resources/github/volcengine/OpenViking/third_party/croaring/roaring.h level=2 context_type=resource account_id=jishu-zhongtai-bu)
2026-07-02 06:57:06,293 - openviking.storage.collection_schemas - WARNING - Failed to generate embedding: Embedding failed: 'NoneType' object is not subscriptable (uri=viking://resources/github/volcengine/OpenViking/bot/package-lock.json level=2 context_type=resource account_id=jishu-zhongtai-bu)
2026-07-02 06:59:42,340 - openviking.storage.collection_schemas - INFO - Re-enqueued embedding message after transient error (uri=viking://resources/github/volcengine/OpenViking/bot/vikingbot/cron/service.py level=2 context_type=resource account_id=jishu-zhongtai-bu)

OpenViking Version

main

Python Version

3.13

Operating System

Windows

Model Backend

None

Additional Context

Root Cause Analysis

1. content field stores full file content without truncation

The embedding pipeline has two text fields:

• Vectorize.text — used for embedding (truncated by embedder's input_guard)
• Vectorize.full_text — used for BM25 full-text search (NOT truncated)

In embedding_msg_converter.py:72-74:

full_content = context.vectorize.full_text or vectorization_text
context_data["content"] = full_content  # Full file content, no truncation

The content field is stored in the vector database as part of the fields JSON string. When the file content exceeds 65535 bytes, the bytes_row serialization fails.

2. Code repository parser does not chunk files

The CodeRepositoryParser explicitly states: "Preserves directory structure without chunking" (code.py:56). Unlike MarkdownParser which splits large content into sections via _smart_split_content(), code files are stored as-is.

This is why:

• Direct upload of 1MB Markdown → parser chunks it → no error
• Code repo with 1MB files → no chunking → content exceeds 65535 → error

3. Local vectordb has a hard 64KB string field limit

The local vectordb uses bytes_row serialization with a UINT16 (2 bytes) length prefix for string fields, capping them at 65,535 bytes. Unlike VikingDB backend (which had its limit raised to 1MB in PR #2825), the local backend's limit is a fundamental storage constraint.

# bytes_row.py:15
STRING_MAX_UINT16_LENGTH = 0xFFFF  # 65535

# bytes_row.py:150-153
if bytes_item_len > STRING_MAX_UINT16_LENGTH:
    raise ValueError(f"string field '{field_meta.name}' exceeds 65535 bytes")

4. Non-auth errors are re-enqueued indefinitely

PR #2919 fixes infinite re-enqueue for auth errors (401/403), but other errors like string field 'fields' exceeds 65535 bytes are still treated as transient and re-enqueued forever.

5. Missing entries in ignore lists

IGNORE_DIRS in openviking/parse/parsers/constants.py does not include:

• target/ (Rust build artifacts)

IGNORE_EXTENSIONS does not include:

• .lock (dependency lock files like uv.lock, package-lock.json)

Affected Files (from logs)

viking://resources/github/volcengine/OpenViking/uv.lock                    (1.3MB)
viking://resources/github/volcengine/OpenViking/third_party/croaring/roaring.c
viking://resources/github/volcengine/OpenViking/third_party/croaring/roaring.h
viking://resources/github/volcengine/OpenViking/web-studio/src/gen/ov-client/types.gen.ts
viking://resources/github/volcengine/OpenViking/examples/openclaw-plugin/tests/ut/tools.test.ts

Suggested Fixes

Solution 1: Skip oversized files (simplest)

Add a file size check to _should_skip_file():

# openviking/parse/directory_scan.py
def _should_skip_file(file_path: Path) -> tuple[bool, str]:
    if file_path.name.startswith("."):
        return True, "dot file"
    if file_path.is_symlink():
        return True, "symlink"
    try:
        size = file_path.stat().st_size
        if size == 0:
            return True, "empty file"
        if size > 100 * 1024:  # 100KB
            return True, "file too large"
    except OSError:
        return True, "os error"
    return False, ""

Also add missing entries to ignore lists:

# openviking/parse/parsers/constants.py
IGNORE_DIRS = {
    # ... existing entries ...
    "target",  # Rust build artifacts
}

IGNORE_EXTENSIONS = {
    # ... existing entries ...
    ".lock",  # Dependency lock files
}

Pros	Cons
Simplest, 1 line change	Large files completely unsearchable
No errors	Loses valuable large file content
Best performance	Threshold is arbitrary (100KB? 200KB?)

Solution 2: Truncate content before writing (recommended)

Truncate the content field in embedding_msg_converter.py before storing:

# openviking/storage/queuefs/embedding_msg_converter.py:72-74
full_content = context.vectorize.full_text or vectorization_text

# Truncate content to fit local vectordb's 65535 byte limit
CONTENT_MAX_BYTES = 60000
if full_content:
    content_bytes = full_content.encode("utf-8")
    if len(content_bytes) > CONTENT_MAX_BYTES:
        full_content = content_bytes[:CONTENT_MAX_BYTES].decode("utf-8", errors="ignore")

context_data["content"] = full_content

Pros	Cons
Small change, 1 location	Content after 60KB is lost
Large files partially searchable	Truncation point may be mid-code
Universal fix	Need to choose appropriate threshold

Solution 3: Add chunking for code repository files (most complete)

Add chunking logic to CodeRepositoryParser similar to MarkdownParser._smart_split_content():

# New file: openviking/parse/parsers/code/code_chunker.py
def chunk_code_file(content: str, file_path: str, max_chars: int = 4000) -> List[str]:
    """Split code files by function/class/paragraph."""
    # 1. Try AST parsing for supported languages (Python, JS, TS, etc.)
    # 2. Fall back to line-based splitting for unsupported languages
    # 3. Each chunk preserves file path prefix as context
    ...

Pros	Cons
Large files fully searchable	Large code change, needs new chunking logic
Preserves code semantic integrity	Needs AST parser support for various languages
Consistent with Markdown parser behavior	Higher performance overhead

Related Issues/PRs

• fix(memory): truncate memory abstracts before vector writes #2774 — Fixed memory abstract truncation (50KB cap), but only for abstract field, not content
• fix(embedding): truncate resource abstracts before vector writes (extend #2774 to indexing paths) #2779 — Extended abstract truncation to resource indexing, same limitation
• fix: VikingDB grep fallback, exclude URI filtering, and text field byte limit #2825 — Raised VikingDB text field limit to 1MB, but only for VikingDB backend, not local
• perf(content-write): anchor semantic refresh at the file's direct parent / 写入语义刷新锚点改为文件直接父目录 #2864 — Fixed content-write cascade (anchored refresh to direct parent)
• fix(embedding): stop re-enqueueing auth errors forever (#2916) #2919 — Fixed infinite re-enqueue for auth errors, but not for field-size errors

Additional Notes

Hello, I'm an agent development engineer at a listed gaming company in China. I've been following this project since early April and am a deep user. At that time, I did some internal research and development based on the project, including:

• Feishu (Lark) document parsing support
• Feishu document image recognition
• Frontend/backend admin panel refactoring
• Video multimodal understanding with highlight frame extraction
• Entity module (adapted for game business departments — users can register entities like game characters or products, and new character uploads are automatically classified under the corresponding entity without manual tagging, solving the issue where VLM recognition couldn't identify which character was stored; integrated into production pipelines for marketing departments like manga/comic content)

These were internal explorations and were not submitted to the official repo. Due to other business priorities, I only recently had time to revisit the project. I'm very grateful for the open-source team's efforts — the current version is so much better than before! 😄

I'd appreciate it if the project maintainers could review my proposed solutions. Once approved, I'll submit PRs accordingly.