Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/copilot-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Project

Zig 0.16.x code intelligence server. Tests live in `src/tests.zig`. Build and test with `zig build test`.
Zig 0.17.0-dev code intelligence server (pinned to `0.17.0-dev.813+2153f8143`). Tests live in `src/test_*.zig`. Build and test with `zig build test`.

## Rules

Expand Down
70 changes: 48 additions & 22 deletions .github/workflows/bench-regression.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ permissions:
contents: read
pull-requests: write

env:
ZIG_VERSION: 0.17.0-dev.813+2153f8143
ZIG_SHA256: b0d46ffc4587b9e8dd0b524ee5bc4da1e67f28bba55e7c534cec64af2f2d7a74
LEGACY_ZIG_VERSION: 0.16.0
LEGACY_ZIG_SHA256: 70e49664a74374b48b51e6f3fdfbf437f6395d42509050588bd49abe52ba3d00

jobs:
bench:
runs-on: ubuntu-latest
Expand All @@ -15,41 +21,61 @@ jobs:
with:
fetch-depth: 0

- name: Install Zig
- name: Install pinned Zig toolchains
run: |
curl -L https://ziglang.org/download/0.16.0/zig-x86_64-linux-0.16.0.tar.xz -o zig.tar.xz
tar -xf zig.tar.xz
echo "$PWD/zig-x86_64-linux-0.16.0" >> "$GITHUB_PATH"
set -euo pipefail

head_archive="zig-x86_64-linux-${ZIG_VERSION}.tar.xz"
head_dir="zig-x86_64-linux-${ZIG_VERSION}"
curl --fail --show-error --location --retry 3 \
"https://ziglang.org/builds/${head_archive}" -o zig-head.tar.xz
echo "${ZIG_SHA256} zig-head.tar.xz" | sha256sum --check
tar -xf zig-head.tar.xz
test "$("$PWD/${head_dir}/zig" version)" = "$ZIG_VERSION"

legacy_archive="zig-x86_64-linux-${LEGACY_ZIG_VERSION}.tar.xz"
legacy_dir="zig-x86_64-linux-${LEGACY_ZIG_VERSION}"
curl --fail --show-error --location --retry 3 \
"https://ziglang.org/download/${LEGACY_ZIG_VERSION}/${legacy_archive}" -o zig-legacy.tar.xz
echo "${LEGACY_ZIG_SHA256} zig-legacy.tar.xz" | sha256sum --check
tar -xf zig-legacy.tar.xz
test "$("$PWD/${legacy_dir}/zig" version)" = "$LEGACY_ZIG_VERSION"

echo "$PWD/${head_dir}" >> "$GITHUB_PATH"

- name: Benchmark head
run: python3 scripts/run-bench-json.py bench-head.json

- name: Benchmark base
- name: Benchmark base with its declared compiler
env:
BASE_REF: ${{ github.event.pull_request.base.ref }}
run: |
git fetch origin "${{ github.event.pull_request.base.ref }}" --depth=1
set -euo pipefail
git fetch origin "$BASE_REF" --depth=1
git worktree add ../codedb-base FETCH_HEAD
cd ../codedb-base
if python3 "$GITHUB_WORKSPACE/scripts/run-bench-json.py" "$GITHUB_WORKSPACE/bench-base.json"; then
echo "have_base_json=true" >> "$GITHUB_ENV"
else
echo "have_base_json=false" >> "$GITHUB_ENV"
fi

base_zig_version=$(sed -n 's/.*\.minimum_zig_version = "\([^"]*\)".*/\1/p' build.zig.zon)
case "$base_zig_version" in
"$ZIG_VERSION")
base_zig_dir="$GITHUB_WORKSPACE/zig-x86_64-linux-${ZIG_VERSION}"
;;
"$LEGACY_ZIG_VERSION")
base_zig_dir="$GITHUB_WORKSPACE/zig-x86_64-linux-${LEGACY_ZIG_VERSION}"
;;
*)
echo "unsupported base Zig version: ${base_zig_version:-missing}" >&2
exit 1
;;
esac
test "$("$base_zig_dir/zig" version)" = "$base_zig_version"
PATH="$base_zig_dir:$PATH" \
python3 "$GITHUB_WORKSPACE/scripts/run-bench-json.py" "$GITHUB_WORKSPACE/bench-base.json"

- name: Compare
if: env.have_base_json == 'true'
run: |
python3 scripts/compare-bench.py bench-base.json bench-head.json --threshold-pct 10 --markdown-out bench-report.md

- name: Bootstrap report
if: env.have_base_json != 'true'
run: |
cat > bench-report.md <<'EOF'
## Benchmark Regression Report

Skipped strict comparison because the base branch does not yet emit machine-readable benchmark JSON.
This PR introduces the JSON benchmark format that future PRs will compare against.
EOF

- name: Upload artifacts
uses: actions/upload-artifact@v7
with:
Expand Down
33 changes: 21 additions & 12 deletions .github/workflows/release-binaries.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ permissions:
contents: write

env:
ZIG_VERSION: 0.16.0
ZIG_VERSION: 0.17.0-dev.813+2153f8143

jobs:
build:
Expand All @@ -23,25 +23,25 @@ jobs:
matrix:
include:
- runner: macos-15-intel
zig_host: x86_64-macos
zig_sha256: 3938c46ae4bca3c13f423b09503e3ef00bb4b7ef12b8bc1e5122ede366057a5b
zig_target: x86_64-macos
asset_name: codedb-darwin-x86_64
zig_archive: zig-x86_64-macos-0.16.0.tar.xz
zig_dir: zig-x86_64-macos-0.16.0
- runner: macos-14
zig_host: aarch64-macos
zig_sha256: 36673d2513afa4a96c86780648ba504beedd7f0451389091cf9d53e38d5b4840
zig_target: aarch64-macos
asset_name: codedb-darwin-arm64
zig_archive: zig-aarch64-macos-0.16.0.tar.xz
zig_dir: zig-aarch64-macos-0.16.0
- runner: ubuntu-24.04
zig_host: x86_64-linux
zig_sha256: b0d46ffc4587b9e8dd0b524ee5bc4da1e67f28bba55e7c534cec64af2f2d7a74
zig_target: x86_64-linux
asset_name: codedb-linux-x86_64
zig_archive: zig-x86_64-linux-0.16.0.tar.xz
zig_dir: zig-x86_64-linux-0.16.0
- runner: ubuntu-24.04
zig_host: x86_64-linux
zig_sha256: b0d46ffc4587b9e8dd0b524ee5bc4da1e67f28bba55e7c534cec64af2f2d7a74
zig_target: aarch64-linux
asset_name: codedb-linux-arm64
zig_archive: zig-x86_64-linux-0.16.0.tar.xz
zig_dir: zig-x86_64-linux-0.16.0
env:
RELEASE_TAG: ${{ github.event_name == 'workflow_dispatch' && inputs.tag || github.event.release.tag_name }}
APPLE_CERTIFICATE_P12: ${{ secrets.APPLE_CERTIFICATE_P12 }}
Expand All @@ -53,13 +53,22 @@ jobs:
fetch-depth: 0
ref: ${{ env.RELEASE_TAG }}

- name: Install Zig
- name: Install pinned Zig
shell: bash
run: |
set -euo pipefail
curl -L "https://ziglang.org/download/${ZIG_VERSION}/${{ matrix.zig_archive }}" -o zig.tar.xz
grep -Fq ".minimum_zig_version = \"${ZIG_VERSION}\"" build.zig.zon || {
echo "release tag does not require the workflow's pinned Zig ${ZIG_VERSION}" >&2
exit 1
}
archive="zig-${{ matrix.zig_host }}-${ZIG_VERSION}.tar.xz"
zig_dir="zig-${{ matrix.zig_host }}-${ZIG_VERSION}"
curl --fail --show-error --location --retry 3 \
"https://ziglang.org/builds/${archive}" -o zig.tar.xz
echo "${{ matrix.zig_sha256 }} zig.tar.xz" | shasum -a 256 --check
tar -xf zig.tar.xz
echo "$PWD/${{ matrix.zig_dir }}" >> "$GITHUB_PATH"
test "$("$PWD/${zig_dir}/zig" version)" = "$ZIG_VERSION"
echo "$PWD/${zig_dir}" >> "$GITHUB_PATH"

- name: Import Apple signing certificate
if: runner.os == 'macOS' && env.APPLE_CERTIFICATE_P12 != ''
Expand Down
88 changes: 88 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,94 @@
# Changelog


## 0.2.5829 - 2026-07-11

The release toolchain moves to pinned Zig `0.17.0-dev.813+2153f8143` without
changing codedb's retrieval semantics, while a focused performance pass makes
user-facing search faster than the previous Zig 0.16.0 build. The migration,
performance work, and retrieval-parity requirements are tracked in #673.

### Zig 0.17 migration

- **Compiler and release jobs are pinned to the exact tested development
snapshot.** `build.zig.zon` and binary-release CI select and assert
`0.17.0-dev.813+2153f8143`; transition benchmark CI selects each revision's
verified 0.16/0.17 compiler instead of silently skipping the base comparison.
Release downloads are verified before use. The build graph uses Zig 0.17
passthrough/lazy-path APIs, and removed
language/library APIs were migrated without changing command behavior.
- **Published targets remain covered.** Native, Windows, and freestanding/WASM
paths use target-appropriate atomics and platform guards. Normal POSIX and
Windows argv are borrowed for process lifetime rather than copied again by
the command parser.
- **Dependencies are reproducible.** The Zig-0.17-compatible `nanoregex` source
is vendored with its license; the small protocol-neutral JSON helpers formerly
supplied by `mcp-zig` now live in `src/mcp_json.zig`, avoiding an otherwise
incompatible server-framework dependency.
- **Migration details are documented.** `docs/zig-0.17-migration.md` records the
removed APIs, dependency strategy, target-specific fixes, verification flow,
and the exact codedb file/subsystem inventory.

### Faster than Zig 0.16, with unchanged retrieval

Same-machine A/B used native macOS arm64 `ReleaseFast` binaries, clean
`428d8df` on Zig 0.16.0 as the baseline, and an immutable 631-file
`git archive` corpus. Direct tests ran 20 alternating A/B pairs with 200
iterations per query. CLI tests used 5 warmups and 40 measured Hyperfine runs,
isolated homes/caches, and daemon/telemetry disabled.

- **Cold single-token CLI search: 127.0ms → 105.7ms** — **1.20× faster**
(−16.8%). Warm snapshot-backed CLI search: 117.2ms → 102.8ms — **1.14×
faster** (−12.3%). Hyperfine reported outliers in both sets; the Zig 0.17
build also had materially lower variance.
- **Exact-word lookup geometric mean: 3.217× faster.** Representative medians:
`config` 6607.5ns → 2452.5ns, `error` 12792.5ns → 3227.5ns, `request`
6245ns → 2345ns, and `response` 10860ns → 2885ns. Pair-sorted postings now
compact adjacent duplicate `(doc_id, line)` hits linearly; malformed, legacy,
or fragmented postings retain full hash deduplication and first-occurrence
order through a detected fallback.
- **Tier-0 content retrieval avoids a per-query document map** when posting doc
IDs are nondecreasing. Any ordering break selects the previous direct-slot or
hash-map path, preserving defensive behavior for incrementally fragmented
indexes. Ranking weights, tier order, result caps, and tie-breakers are
unchanged.
- **Cold trigram construction reuses one directory handle per worker** and rolls
both raw and normalized byte windows, reducing overlapping loads and ASCII
normalization from roughly four per byte to one. Whitespace skipping,
`loc_mask`, `next_mask`, case folding, and final-trigram boundaries are
unchanged.
- **Release startup avoids unnecessary work.** Optimized builds execute directly
on the process stack instead of creating and immediately joining a worker
thread; Debug retains its 64MB-stack trampoline. Borrowed argv parsing removes
per-argument allocation/copy/free while retaining Windows bootstrap lifetime.
Completed word-index searches also retain their existing shared lock instead
of unlocking and immediately reacquiring it.

No scoring, ranking, tokenization, parser, filtering, or result-rendering
formula changed in this pass. Direct benchmark hit counts are identical, and
parity tests cover sorted and malformed word postings, grouped and fragmented
Tier-0 aggregation, sequential versus parallel indexing, persisted/mmap index
round-trips, and lazy word-index rebuilds.

The overall direct-query geometric mean is **1.405× faster**; search-only and
symbol groups are effectively flat to slightly faster (1.007× and 1.013×).
The remaining known gap is the generic full parse/serial-commit initial scan:
166ms → 169ms median (**1.8% slower**). Production cold single-token search uses
the optimized trigram scan path above. Closing the generic gap requires a
larger parser/commit ownership or pipeline redesign and is intentionally not
mixed into this behavior-preserving release.

### Verification

- uncached full Zig suite: 23/23 build steps; 892/896 tests passed, 4 platform skips
- focused suites include 181/181 index, 132/132 explore, 113/113 search,
and 152/156 MCP tests (4 platform skips)
- MCP E2E: 20/20 passed across roots handshake, explicit-root, no-roots, and
inline-argument scenarios
- immutable-corpus direct-query hit counts matched the Zig 0.16.0 baseline
- `git diff --check` and modified-file lint passed


## 0.2.5828 - 2026-07-05

Windows warm-daemon parity, a 2× faster `codedb_context`, OSTree (Fedora
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<p align="center">
<a href="https://github.com/justrach/codedb/releases/latest"><img src="https://img.shields.io/github/v/release/justrach/codedb?style=flat-square&label=version" alt="Release" /></a>
<a href="https://github.com/justrach/codedb/blob/main/LICENSE"><img src="https://img.shields.io/github/license/justrach/codedb?style=flat-square" alt="License" /></a>
<img src="https://img.shields.io/badge/zig-0.16-f7a41d?style=flat-square" alt="Zig 0.16" />
<img src="https://img.shields.io/badge/zig-0.17.0--dev-f7a41d?style=flat-square" alt="Zig 0.17.0-dev" />
<img src="https://img.shields.io/badge/status-alpha-orange?style=flat-square" alt="Alpha" />
<a href="https://deepwiki.com/justrach/codedb"><img src="https://deepwiki.com/badge.svg" alt="Ask DeepWiki" /></a>
<br />
Expand Down Expand Up @@ -121,6 +121,7 @@ This replaces the `codedb` binary with the latest GitHub Release and keeps your
- **[CLI reference](docs/cli.md)** — every command, every flag
- **[Architecture](docs/architecture.md)** — engine internals, index layout
- **[Benchmarks](docs/benchmarks.md)** — micro-benchmarks + agentic-eval results vs codegraph, FTS5, lean-ctx
- **[Zig 0.17.0-dev migration guide](docs/zig-0.17-migration.md)** — repeatable zigup workflow and API change recipes

| Platform | Binary | Signed |
|----------|--------|--------|
Expand Down Expand Up @@ -454,7 +455,7 @@ rm -f codedb.snapshot # remove snapshot from current project only

## 🔨 Building from Source

**Requirements:** Zig 0.16+
**Requirements:** Zig `0.17.0-dev.813+2153f8143` (the exact tested development snapshot). See the [migration guide](docs/zig-0.17-migration.md) for reproducible zigup setup.

```bash
git clone https://github.com/justrach/codedb.git
Expand Down
Loading