Skip to content

Open audit items: privacy promises kept, Follow-through on Think, Parakeet repinned - #478

Merged
stevyhacker merged 5 commits into
masterfrom
claude/project-audit-open-items
Nov 10, 2026
Merged

stevyhacker merged 5 commits into
masterfrom
claude/project-audit-open-items

Conversation

@stevyhacker

Copy link
Copy Markdown
Owner

Summary

The second pass on the 2026-11-09 project audit. PR 474 left 157 findings open, most of them because they needed a decision. This PR makes those decisions: 76 are fixed, 22 fixed in part, 2 turned out not to be defects, and 57 are left, each with a reason (performance changes nobody could measure here, recovery formats that need a crash on real hardware, typing behaviour that needs the real apps). Over all 283 findings that is 201 fixed, 34 in part, 45 left, 3 not defects.

Five commits, one per area. What a user would notice:

  • Privacy promises kept (Mac). A paired phone's Ask reads only what is shared, and nothing when nothing is shared. Revoking a transcription server stops the rest of the meeting's uploads. On the Claude plan nothing from a meeting is on the claude command line. Revoking an origin ends a running Agent Mode turn. A variable that wrote screen-derived model output to debug.log is gone.
  • Follow-through runs on the Think model from Settings → Models, like Live Radar since PR 474. Its model picker is gone. Fast live text is the only thing left on Meta's API.
  • Windows and Linux. Parakeet v3 and v2 download again (issue 458). Recordings "waiting for models" are processed when the models arrive. A dictation no longer waits for a meeting's transcription. Accepting a suggestion with non-ASCII text over AT-SPI inserts the right text on GTK and Qt. A calendar link over http:// is refused unless it is on this computer.
  • Phones. App Lock covers sheets, the recording screen and alerts on iOS, and the recents picture on Android, where it could also be skipped after a process restart. Lists page past 1,000 meetings.
  • Releases and CI. The next release attaches LokalBot.mcpb and lists in the MCP registry. A skipped unit test fails the run unless it is on a checked-in list. The nightly model-drift check no longer reports "No drift detected" when it recorded nothing.
  • macOS 26 UI run. Three of its five failing tests have a fix; the two right-to-left Settings tests have a mitigation that nothing here could run.

Three read-only reviews of the finished diff found 16 defects before anything was committed, among them: a phone's Ask reading every meeting when the shared set was empty, a parked meeting being transcribed twice, and the new Parakeet pins making every existing install count as missing. Fifteen are fixed here; one low one, on Windows, is listed under "Not verified".

Validation

Mac, on the final tree:

xcodegen generate
xcodebuild -project LokalBot.xcodeproj -scheme LokalBot -configuration Debug \
  -destination 'platform=macOS' CODE_SIGNING_ALLOWED=NO -only-testing:LokalBotTests build-for-testing
# ** TEST BUILD SUCCEEDED **
xcodebuild test-without-building -xctestrun … -destination 'platform=macOS,arch=arm64'
# Executed 4545 tests, with 57 tests skipped and 0 failures (0 unexpected)
python3 Scripts/ci/unit-skips.py check <that run's .xcresult>
# 57 unit tests skipped, all listed; exit 0
CODE_SIGNING_ALLOWED=NO bash Scripts/ui-tests.sh --build-only
# ** TEST BUILD SUCCEEDED **   (UI tests were not run)
swiftlint lint --strict --quiet
# exit 0

Windows and Linux, whole workspace on the Ubuntu box, with the translated catalogs:

cargo fmt --all -- --check                                        # exit 0
cargo clippy --workspace --all-targets --locked -- -D warnings    # exit 0
cargo test --workspace --locked                                   # 2,383 passed, 0 failed, 28 ignored
python3 -m unittest discover -s scripts/tests                     # Ran 8 tests, OK
python3 scripts/sync-localizations.py --check                     # exit 0
python3 ../Scripts/ci/check-model-pins.py                         # all 13 pins match
  • Parakeet: each new archive was downloaded from the URL the app uses, measured (v3 487,211,443 bytes, v2 482,440,461 bytes, hashes as pinned), and run through the port's recognizer beside the previous export: the same text on every clip, and word error rate within 0.2 points on the sample meetings.
  • AT-SPI: live on the box under Xvfb with accessibility on, GTK 3, GTK 4, Qt 5 and Qt 6 text fields, text with emoji and Cyrillic: 24 of 24 accepts right, 8 of 24 before.
  • Dictation behind a meeting: 0.35 s worst case with a real model, against 0.19 s alone.

Phones:

Scripts/build-sync-android.sh                                              # on the Ubuntu box
./gradlew :app:assembleDebug :app:testDebugUnitTest :app:lintDebug         # 139 tests, 0 failures; lint clean
./gradlew :app:verifyRoborazziDebug                                        # unchanged
swift test --package-path Companion/ios/LokalBotCompanionKit               # 74 tests pass
python3 -m unittest discover -s Companion/e2e                              # Ran 23 tests, OK
  • The Android emulator trip passed on a release build, with a new cancel-then-allow step. The iOS app compiles (compile only).

Site, scripts, distribution:

python3 Scripts/render_web.py --check            # All 40 generated web files are up to date.
python3 -m unittest discover -s Scripts/tests    # Ran 222 tests, OK
(cd Distribution/npm && node --test)             # 50 pass
  • The seven edited site pages were rendered in headless Chromium at 1280 and 390 px: no console errors, every JSON-LD block parses. No site CSS or JavaScript changed.
  • For most fixes the fix was undone and its test seen to fail before it was put back. The exceptions are in the list below.

Not verified:

  • Nothing ran on macOS 26. The weekly workflow has to be started on this branch. Watch QuickRecallUITests/testSearchShowsLocalMeetingEvidenceAndInlineAsk, RedesignUITests/testReviewAndSearchAccessibilityInBothAppearances, RedesignUITests/testProjectsHighlightsTheOpenProjectAndSelectionKeepsTheColumnWidth, and the Urdu and Egyptian Arabic SettingsUITests. The right-to-left crash does not reproduce on macOS 27, so its mitigation is unproven.
  • UI tests were compiled, not run; the required suite runs when this is marked ready for review.
  • Follow-through on a real Think model, and every Mac view change (read and compiled only).
  • Claude plan: the stdin form passed the repository's opt-in live test on Claude Code 2.1.318 on the Mac. Older Claude Code versions, and the port's copy against the real program, were not run.
  • Windows: nothing was run there. The notification click code only type-checks for a Windows target. A click in Action Center after the banner timed out still does nothing; the notification crate hands out one event per notification.
  • The port's desktop-app changes in a running app, a real microphone unplug, and an Agent Mode task with the built-in model.
  • iOS App Lock has never been seen running (the cover window, and keyboard commands switched off under it). An alert already open when the app locks keeps its own keys. Android recents were not checked on a real phone.
  • Releases: no release or registry workflow was run. The registry publish job's first run is the next stable tag.
  • Translations (18 strings added or changed across the Mac and port catalogs, 23 languages) are from a model; a few are marked unsure in Hausa, Swahili, Nigerian Pidgin, Arabic and Telugu.

Changed test expectations

Mac:

  • CompanionAskResponderTests/testMeetingFromADeviceIsAStartingPointNotALimit: _lokalbot_meeting_ids was expected to be nil → every shared meeting's ID.
    Why the old expectation was wrong: nil is what let a phone's question reach the sample meeting. The test's point, that the focused meeting is not the limit, is kept by asserting the other meeting is in the list.
  • ClaudePlanEngineTests/testStreamingAndStructuredArguments: expected --json-schema <json> in the arguments → no --json-schema, and --input-format stream-json.
    Why: the old value put meeting text on the command line, which PRIVACY.md rules out.
  • CotypingCoordinatorTests/testASuggestionLeftAloneIsTakenDownSoTabReachesTheApp, DictationShortcutTests (two checks), AgentAccessManagerTests (two checks): fixed sleeps of 120 to 600 ms → a wait of up to 3 to 6 s for the condition. Same assertions. The old margins could be eaten by a busy runner.
  • OrphanedLlamaServerReclaimTests helpers: two XCTSkip → a thrown failure. The skips depended on the code under test, so a regression showed as skipped tests.
  • Removed with the unused code they tested: CotypingABBenchmarkTests, two cases in AskRoutingTests, and the PromptSectionBudget cases in ReuseSubsystemsTests.
  • UI shard plan: functional-1 and functional-2 go from 52 and 53 tests to 53 and 52, with different members.

Windows and Linux:

  • lokalbot-ai/src/plans/claude/tests.rs, arguments_carry_the_mac_isolation_flags and runs_claude_in_a_private_empty_folder_with_content_on_stdin: arguments ending --json-schema <schema> and a plain-text stdin → stream-json flags, no --json-schema, and stdin as two JSON lines carrying the same text. Same reason as on the Mac.
  • lokalbot-cli/tests/cli.rs, mcp_round_trip: 8 replies → 11, for three new list_meetings calls with until. No earlier assertion changed.
  • calendar/feed.rs, links_are_normalized_and_redacted_to_their_host: is_err() → the exact error. autocomplete/atspi.rs, grapheme_counts_become_character_counts: the same three values, asked of the function that replaced chars_in_graphemes.

Phones, site and scripts: none. Changed test files only gained tests.

Linked issues

Fixes #458.

Risk / rollout notes

  • Follow-through now uses the Think model and its approval. The setting museFollowThroughModel and its picker are removed; saved values are ignored. The Meta Muse approval label now says "meeting audio", which is all the app still sends Meta.
  • Claude plan: structured requests use Claude Code's stream-json input with an initialize request. A Claude Code that ignores the schema now gives an error where it used to be handed prose.
  • Agent Mode on the built-in model (Mac): the task holds the model only during a turn, so after ten idle minutes the next prompt waits for the model to load. On the port the model is still held, and choosing another built-in model meanwhile is refused with a message; matching the Mac needs a server token that survives a restart.
  • The port's speech model is now shared per span of audio. That is new concurrency. The review found no deadlock or lock-order problem.
  • Parakeet: new downloads get upstream's re-exported files. Installs of the previous files still count as installed.
  • Calendar links over http:// stop being fetched on Windows and Linux unless they are on this computer. A saved link is kept and shown as refused, with the reason.
  • App Lock on Android before 13 also blocks the person's own screenshots while it is on.
  • Scripts/unit-tests.sh fails on an unlisted skip. A Mac without Bun gets a named failure where the stub-server tests used to go quiet.
  • build.yml gained a step, so the unit job's compiler cache rebuilds once.
  • Releases publish to the MCP registry by default (MCP_REGISTRY_PUBLISH=false turns it off). v0.12.2 is still missing from the registry.
  • Plugins are bumped (Claude 0.3.2, OpenAI 0.2.3). The Claude plugin's repository syncs when this merges; both directories need a re-check, and the OpenAI plugin a new ZIP.
  • On macOS 26, right-to-left languages keep the window's splits left-to-right. macOS 15 and 27 keep the mirrored sidebar.
  • The nightly model-drift job fails on a scheduled run with no key, and when a case has no answer.
  • The hackathon film workflow, its scripts and the META_API_KEY secret's only user are removed. HackathonFilmUITests stays in the UI suite.

…odel

Privacy and consent:
- A paired phone's Ask reads only the shared meetings and no Timeline day
  before the shared window, never the sample meeting. An empty shared set
  now means nothing is readable; it used to mean everything.
- Revoking a transcription server, or choosing an on-device model, stops
  the remaining uploads of the meeting in progress. The meeting restarts
  on the current settings or waits.
- On the Claude plan the schema and prompt go over stdin, not the
  `claude` command line. A reply that ignores the schema is an error.
- Revoking an origin ends a running Agent Mode turn. An open Agent task
  holds the built-in model only during a turn.
- The environment variable that wrote screen-derived model output to
  debug.log is gone.

Follow-through runs on the Think model from Settings → Models, through
ThinkChat, like Live Radar. Its model picker and `museFollowThroughModel`
are removed; the Meta Muse section now covers fast live text only. The
UI-test host keeps Meta's request format for the film test.

Windows and lists:
- List columns open at their set width and no longer jump to maximum on
  selection. The system-wide Ask window opens with its field focused.
- On macOS 26, right-to-left languages keep the window's splits
  left-to-right, where the mirrored window crashed Settings.
- The floating Live Radar panel hides when its setting is turned off.

Also: a failed screen-retention pass is retried within the hour, up to
three times; `list_meetings` takes `until`; file search widens when
filters drop hits; smaller fixes; unused code removed
(CotypingABComparison, AskScopeReconciler, AskEscalationScope,
PromptSectionBudget).

Strings: 6 added, 3 changed, 5 removed, in all 24 languages.
- `Scripts/ci/unit-skips.json` names the 58 opt-in tests that may skip,
  with a reason for each group. `Scripts/ci/unit-skips.py` fails the unit
  job and `Scripts/unit-tests.sh` on any other skip, and the hosted job
  also fails when a listed test is missing from the results.
- The run-on standup replay, which had skipped for over a month, now
  runs on recordings from the last nightly run that recorded.
- A helper that turned a regression into skipped tests now fails.
- The contrast audit measures rendered text, so two labels that pass
  4.5:1 no longer fail on macOS 26.
- UI shards are planned from two green runs: 1,230 s and 1,226 s.
- Three timing-sensitive unit tests wait for their condition instead of
  sleeping a fixed time.
…r a meeting

- Parakeet v3 and v2 are pinned to upstream's re-exported archives. Old
  and new files gave the same text on every clip through the port's
  recognizer. Installs of the previous files still count as installed.
- Recordings parked "waiting for models" are processed within about 5 s
  of the models arriving, without repeating a finished transcription. A
  job that waited only for an automatic summary goes on when automatic
  summaries are turned off.
- The speech model is shared per span of audio, so a dictation no longer
  waits for a whole meeting. Dictation ends when the microphone is lost.
- Accepting a non-ASCII suggestion over AT-SPI inserts the right text on
  GTK 3/4 and Qt 5/6: offsets are counted per toolkit.
- A calendar link over http:// is refused unless it is on this computer,
  with the reason shown.
- On the Claude plan the schema goes over stdin, not the command line.
- While an Agent Mode task is open on the built-in model, choosing
  another built-in model is refused with a message; it used to make
  every Think job wait forever.
- A click on a notification's body is handled on Windows (not run).
- `list_meetings` takes `until`; file search reads every matching
  passage until it has the files asked for; smaller fixes.

Strings: 4 changed, 5 added, in all 24 languages; catalogs regenerated.
- iOS: App Lock covers sheets, the recording screen and alerts with a
  cover window per scene. While locked, keyboard commands are off and
  the answer banner shows the question only.
- Android: App Lock hides the recents picture and can no longer be
  skipped when Android restores the app after ending its process.
- Meetings page past 1,000 on both apps; Android search covers the whole
  library, and People, Projects and Timeline lose the same cap.
- A change the computer refuses shows "Not changed on your computer"
  with the computer's reason.
- Android keeps a pairing allowed after Cancel. Action changes no longer
  wait behind queued answers, and iOS Ask no longer waits behind sync.
- Releases attach LokalBot.mcpb and publish to the MCP registry. Both
  were gated on a repository variable that was never set.
- The nightly model-drift check fails and opens an issue when it
  recorded nothing; it used to report "No drift detected". A pull
  request without the key skips with a notice.
- Two crates compiled into the Mac and phone apps are added to the
  licence notices, and a CI check keeps the notices complete.
- generate_appcast.py refuses the OWNER/REPO placeholder.
- The site's privacy page names the autocomplete server, the
  transcription server and the ChatGPT and Claude plans; the licence
  line states the AGPLv3 Cotabby portion in the README's words.
- The agent skill says which Personal Context items are the user's own
  words and which were learned. Plugins: Claude 0.3.2, OpenAI 0.2.3.
- The hackathon film workflow and its scripts are removed; the film test
  stays in the UI suite.
- Two shell tests that could not fail under bash 3.2 are fixed.
- Documents corrected against the code; files nothing references removed.
@stevyhacker
stevyhacker marked this pull request as ready for review November 10, 2026 15:54
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
lokalbot 0e9ee8a Commit Preview URL

Branch Preview URL
Nov 10 2026, 03:28 PM

@stevyhacker
stevyhacker merged commit 0fbd046 into master Nov 10, 2026
85 of 92 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A pinned model no longer matches its release

1 participant