Replace SQLite storage with file-based Markdown + JSON layout#51
Replace SQLite storage with file-based Markdown + JSON layout#51
Conversation
Replaces the SQLite database with a plain-file layout so notes are
self-describing on disk and editable in any Markdown tool (Obsidian,
VS Code, etc.).
Layout:
<data_dir>/
notebooks/<id>.yaml # Notebook metadata
notes/<id>.md # Markdown body + YAML frontmatter
drawings/<id>.json # Stylus strokes (one file per note)
- New FileNoteStore is the primary storage; mirrors the old NexaNoteDB
public API so the REST routes, WebDAV provider, and sync engine
required only import/type updates.
- Single-page typed notes have a clean Markdown body with no NexaNote
markers, so they round-trip cleanly through external editors.
- Atomic writes (tmp + os.replace) and per-path RLocks keep concurrent
API + WebDAV traffic safe inside the backend process.
- run_migration() detects pre-v1 nexanote.db on startup, copies every
notebook/note/page/stroke into the new layout, renames the original
to nexanote.db.legacy_backup, and writes a .nexanote_migrated marker
so it never re-runs. Soft-deleted notes are preserved.
- legacy_db.py is the old SQLite layer kept read-only for migration.
- Bumps backend + Flutter app to v1.0.0; adds CHANGELOG and updates
README/docker.md to describe the new layout. REST API contract is
unchanged so existing Flutter clients keep working.
- 36 new tests cover file-store roundtrip, multi-page splitting,
concurrency, and end-to-end SQLite -> file migration. Suite is
102 passing.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 704f571e18
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| report.ran = True | ||
| marker.write_text( | ||
| _marker_payload( |
There was a problem hiding this comment.
The migration is marked as successful and finalized even when per-record conversion failures were already captured in report.errors. In the failure scenario (e.g., one or more notes fail to write due to malformed data or filesystem errors), this still writes the migration marker and skips future runs, leaving the file store incomplete and requiring manual backup recovery. Migration finalization should be gated on zero conversion errors (or raise and keep migration retryable).
Useful? React with 👍 / 👎.
| id=meta["id"], | ||
| notebook_id=meta.get("notebook_id"), | ||
| title=meta.get("title", "Sans titre"), | ||
| note_type=NoteType(meta.get("note_type", "typed")), |
There was a problem hiding this comment.
A single note file with an unexpected note_type (or sync_status) value raises ValueError during deserialization, and callers like list_notes() do not catch it. In the file-based model where notes can be edited externally, one malformed frontmatter value can make /notes and /notes/{id} fail with 500 instead of skipping or defaulting the bad record.
Useful? React with 👍 / 👎.
2 participants
Summary
This PR replaces the legacy SQLite database backend with a file-based storage system modeled after Obsidian. Notes are now stored as plain Markdown files with YAML frontmatter, drawings as separate JSON files, and notebooks as YAML metadata files. An automatic migration tool copies all data from pre-v1.0 SQLite databases into the new layout on first startup.
Key Changes
New
FileNoteStoreclass (nexanote/storage/file_store.py): Primary storage backend implementing the same public API as the legacyNexaNoteDBso REST API, WebDAV provider, and sync engine work unchanged. Features:notes/<id>.mdwith YAML frontmatter + Markdown bodydrawings/<id>.json(only created if note has strokes)notebooks/<id>.yaml<!-- nexanote:page N -->HTML comment markers for splittingtmp + os.replace) for concurrency safetyAutomatic SQLite → file migration (
nexanote/storage/migration.py):needs_migration(data_dir)detects pre-v1.0nexanote.dbrun_migration(data_dir)copies all notebooks, notes, pages, and strokes into new layoutnexanote.db.legacy_backup(never deleted).nexanote_migratedmarker prevents re-running migrationis_deletedflag intactComprehensive test coverage (
tests/test_file_store.py,tests/test_migration.py):Updated all storage consumers:
nexanote/api/routes.py: Changed type hint fromNexaNoteDBtoFileNoteStorenexanote/sync/webdav_provider.py: Updated importsnexanote/sync/client.py: Updated importsnexanote/sync/server.py: Added migration call on startupmain.py: Added migration call before initializing storageLegacy SQLite layer preserved (
nexanote/storage/legacy_db.py): Kept read-only for migration tool; no longer used for normal operationDependencies: Added
PyYAML==6.0.3for YAML frontmatter serializationDocumentation & versioning:
README.mdto reflect file-based storagedocs/docker.mdto describe new directory structure1.0.0inapp/pubspec.yaml,routes.py, andmetadata/org.nexanote.app.ymlCHANGELOG.mddocumenting the v1.0.0 releaseNotable Implementation Details
os.fsync()+os.replace()prevents corruption if process is killed mid-writeRLockinstances for thread-safe concurrent accesshttps://claude.ai/code/session_01NgyB5ckJecLX3v25tPTV3d