fix(history): give each thread its own SQLite connection (fixes #39)#81
Open
calvinlow18 wants to merge 1 commit into
Open
fix(history): give each thread its own SQLite connection (fixes #39)#81calvinlow18 wants to merge 1 commit into
calvinlow18 wants to merge 1 commit into
Conversation
The daemon runs every sync through asyncio.to_thread(), so SyncHistory's single
cached connection gets handed to whichever pool thread runs the next source.
sqlite3 refuses it:
SQLite objects created in a thread can only be used in that same thread.
The first source to finish wins the connection and succeeds; every other source
raises. Because SyncHistory.log() is called after the upload completes, the
documents do reach the knowledge base -- but the sync is recorded and reported as
an error, so the daemon's /health, its dashboard and its history all show failures
for syncs that actually worked.
Hold the connection in a threading.local() instead, so each thread gets its own.
Also open with WAL and a busy timeout, so the now-concurrent writers wait on the
brief write lock rather than raising 'database is locked'.
Fixes open-webui#39
Contributor
|
Should be addressed! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #39.
The bug
SyncHistorycaches a singlesqlite3.Connectioninself._conn, created lazily by whichever thread calls_get_conn()first. The daemon runs every sync throughasyncio.to_thread(), so the next source lands on a different pool thread — andsqlite3(with its defaultcheck_same_thread=True) refuses it:The first source to finish wins the connection and succeeds; every other source raises. That matches #39's report exactly (4 of 5 sources failing), and it is connector-agnostic — #39 hit it with Google Drive, I hit it with
outline:on 4 sources.Why it's worse than it looks
SyncHistory.log()is called afterrun_sync()has already uploaded, and_run_entry_locked()catches the exception in its broadexcept Exception— which overwrites the just-set success state:So the documents do reach the knowledge base, while
/health, the dashboard and the history all reporterrorfor a sync that actually worked. Anything monitoring the daemon is permanently red, and the counts (files_added, etc.) are lost.The fix
Hold the connection in a
threading.local()so each thread gets its own. Also open with WAL and a busy timeout, so the now-genuinely-concurrent writers wait on the brief write lock instead of raisingdatabase is locked.One file, no API change.
Reproduction
Against
main, drivingSyncHistorythe way the daemon does:Before: 4 of 5 raise
ProgrammingError: SQLite objects created in a thread..., and the follow-upquery()raises too.After:
sources logged OK: 5/5,rows in history: 5.Also re-checked the single-threaded CLI path —
log,query(plain /errors_only/kb_id),last_sync,clear, and a doubleclose()— all unchanged.