Skip to content

Avoid quadratic suffix scans when deduplicating query column names #912

Open

Description

@workstonedai-collab

sqlite_utils.utils.dedupe_keys reserves all original column names upfront, then searches for each duplicate name by restarting suffix generation at _2. With many repeated names, this repeatedly checks the same suffixes and performs quadratic work.

Add an optimization that avoids restarting the suffix search while preserving the exact existing output names and order, including collisions with preexisting suffixed names. The implementation must continue to accept generators and other iterables. Add correctness coverage for duplicate names, preexisting suffixes, adversarial collisions, and generator inputs; include a benchmark for 500, 2000, and 5000 duplicates without making timing-dependent tests flaky.

Source baseline: 9c66bafa52ae23d40499c74b0db3b635bcddb876, CPython 3.12.15 on macOS arm64. A local synthetic benchmark measured 2,000 identical names at approximately 0.115 seconds and 5,000 at 0.746 seconds (best of three); when original suffixed names are also reserved, the corresponding times were approximately 0.382 and 2.504 seconds. These are helper benchmarks, not end-to-end database timings.

I plan a small patch with correctness coverage and an independently runnable benchmark. AI assistance: MiniMax is analyzing/implementing this contribution; Codex runs the validation and submits the results.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions