Fix keyword search missing files that contain the keyword

[agustingroh]

Fixes #930

Problem

• Excluded folders. The keyword index had its own hard-coded exclusion list and never read Include all file types. vendor/, node_modules/ and dot-folders were never indexed.
• Whole-word matching. The flexsearch index only matched whole words, so copyright missed SPDX-FileCopyrightText, copyrights and copyrightText, and crypt missed encryption. The code viewer highlights by substring, so the viewer and the search results disagreed.

Changes

• Trigram index. Every token is split into 3-character pieces through a custom flexsearch encoder. Candidates are then checked against the file content, one page at a time, so results match a case-insensitive substring search.
  • A single term of 3 characters needs no check, because it is itself a trigram.
  • The flexsearch 0.7 options that 0.8 ignored (stemmer, depth, minlength) are removed.
  • Dedupe and numeric normalization are disabled, since they produced false positives.
• Index coverage. The index now honors Include all file types: with it enabled, every folder is indexed and only binaries and images are skipped.
• Indexer robustness.
  • Binaries are skipped and UTF-16 files are decoded.
  • Running out of heap stops indexing and logs how many files were left out, instead of skipping them one by one without a warning.
  • Writes are awaited, and the dictionary is written to a temporary folder and then swapped in.
  • A path containing ' no longer breaks the file query and leaves the project without an index.
• Existing projects. Dictionaries now carry a version file. Opening a project with an older dictionary rebuilds it in the background when its source code is available. Until the rebuild finishes, and for projects without source code, the old dictionary keeps working.
• Smaller fixes.
  • The in-memory index is now released, since closeIndex previously lost this.
  • Overlapping searches no longer drop responses.
  • Export skips temporary dictionary folders.

Verification

• Unit tests. New tests in src/__tests__/search/indexer-searcher.test.ts cover:
  • an extension-less license file
  • copyright in a header and in inline comments
  • SPDX-FileCopyrightText
  • crypt found in encryption
  • a UTF-16 file
  • a binary, which is not indexed
  • a path containing a quote
  • multi-term AND
  • pagination
  • loading a legacy dictionary
• Full suite. All 101 tests pass.
• Test project. A project with a nested vendor/ folder, extension-less LICENSE files and copyright headers, with Include all file types enabled:

Keyword	Files found before	Files found after	grep -ril
copyright	2	21	21
license	3	20	20
crypt	0	4	4

• Benchmark on 48.6k files. 100% of the files that contain each keyword are found, with no false positives after verification. The index uses about half the heap of the old one (20 MB vs 41 MB), and the export is 27 MB vs 13 MB.

Known limitations

• No source code. Without source code, candidates can't be checked, so multi-term and 4+ character searches may include false positives.
• Very long tokens. Only the first 256 characters of a single token are indexed, so a match past that point in minified code or a base64 blob is missed.
• Short keywords. Keywords of 1 or 2 characters match whole words only, as before.