Skip to content

Bring autocomplete to other apps on Wayland as an input method - #432

Merged
stevyhacker merged 18 commits into
masterfrom
claude/wayland-autocomplete
Nov 3, 2026
Merged

stevyhacker merged 18 commits into
masterfrom
claude/wayland-autocomplete

Conversation

@stevyhacker

@stevyhacker stevyhacker commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Summary

This PR merges the whole stack. It contains #430 (Windows and Linux parity) and #431 (screen text in more scripts), so the required suite runs once on all of it. Those two have their own descriptions, including their changed test expectations. It also adds the two explainers the next release's notes link: web/explainers/windows-linux-app.pdf and web/explainers/wayland-autocomplete.pdf.

Autocomplete now works in other apps on Sway, Hyprland and GNOME, as an input method. Before this, every Wayland session got only the practice editor in Settings, because X11's approach (key grabs, an overlay placed by coordinates, XTest typing) isn't allowed there.

  • Sway, Hyprland, Omarchy: zwp_input_method_v2, plus zwp_virtual_keyboard_v1 to give keys back. The keyboard is grabbed only while a suggestion shows. Nothing to select.
  • GNOME: an IBus engine the user adds and chooses as an input source. Packages install its component file; the .tar.gz gets a command to copy in Settings.
  • KDE Plasma, other compositors: not supported yet, and Settings says so. KWin only runs an input method it starts itself (input-method-v1), usually Fcitx 5, whose engines are C++ plugins, and kwin_wayland isn't installed here. Other compositors have no way to name the focused app.
  • Other input methods: on Sway and Hyprland, if Fcitx or IBus is running or the compositor says another client holds the seat, LokalBot takes no part, says why and retakes the seat only after it has been gone for a while. Omarchy runs Fcitx 5 by default (from its repository), so there autocomplete stays out of the way until the user stops it; the README gives the command.

Reuses the existing suggestion, accept-key and learning logic; only the ghost's drawing changes. X11 and Windows code paths are unchanged.

Validation

Everything ran inside isolated with a private bus, runtime folder and WAYLAND_DISPLAY. Only LOKALBOT_FAKE_AUTOCOMPLETE supplied suggestions in app runs, so no real model produced a suggestion on these routes; the engine takes the same FieldContext it always did.

Real LokalBot app, end to end (type, ghost, Tab, full accept, typing through, Esc):

Headless Sway 1.9 Headless GNOME Shell 46
GTK 3 entry ✓ ✓
Firefox 159 ✓ engine path ✓ with the example below
Chrome 156, no flags ✓ ✓, with a stand-in for Window Calls

autocomplete_watch --ghost example: GTK 3 and GTK 4 entries and text views on both desktops, including pass-through of a turned-down key; Firefox on GNOME with the same stand-in. Chrome with --enable-wayland-ime and --wayland-text-input-version=3 on Sway behaves the same as with no flags.

Refused, with no text reported: GTK 3 and 4, Firefox and Chrome password fields on Sway; GTK 3 and 4 on GNOME, where GNOME also swaps the engine out, so keys never reach it, and it comes back afterwards. foot refused on both.

Conflicts on Sway: a process owning org.fcitx.Fcitx5 before and after LokalBot starts, and a second copy of the route holding the seat. Typing reached the field throughout. Real Fcitx 5 isn't installed here.

Others: Kate (Qt 5) never activated the input method. With the source chosen and autocomplete off or LokalBot not running, typing on GNOME is untouched. Visible text over AT-SPI reached the snapshots on Sway. The app log held no typed text in any app run with autocomplete debug logging on. The Settings row was checked in a screenshot on GNOME.

Not run:

  • Hyprland: not installed. It uses the same protocols, but nothing here proves it, and this needs a run before saying it works.
  • The real Window Calls extension: the stand-in exports its interface from GNOME Shell's unsafe mode.
  • The setup command's sudo step and the built packages: tested through IBUS_COMPONENT_PATH; a test checks the packaged file matches --ibus-component.
  • Physical keyboards and real GNOME sessions: all keys came from a virtual keyboard or Mutter's RemoteDesktop API.
  • Qt 6, older Chromium, Electron: README marks these as from documentation.
# from LinuxWindows/
cargo fmt --all -- --check                                                # exit 0
cargo clippy --workspace --all-targets --locked -- -D warnings            # exit 0
cargo test -p lokalbot-platform -p lokalbot-app -p lokalbot-i18n --locked # exit 0
cargo check --target x86_64-pc-windows-gnu --workspace --all-targets --locked  # exit 0
python3 scripts/sync-localizations.py --check                             # exit 0
python3 -m unittest discover -s scripts/tests                             # OK
crates/lokalbot-platform/scripts/live-sway.sh  # the new test: 1 passed (run through the same steps, inside isolated)

Changed test expectations

This PR's own changes: none. No existing test referred to the old Wayland message. From the stack: see the two PRs above, which include the two FileSearchServiceTests timing changes.

Linked issues

Refs #430, #431

Risk / rollout notes

  • New crate: wayland-protocols-misc 0.3.12 (MIT), from the same project as wayland-client, for the two protocols. xkbcommon 0.9 was already in the tree and is now a direct dependency. LICENSES/rust-crates.md is regenerated.
  • What's seen: the PRIVACY.md section for Linux and Windows has the wording. On GNOME the desktop asks the engine about every key in a text field while it's the chosen source; on Sway and Hyprland keys pass through only while a suggestion shows.
  • Chromium reports its text only in answer to an input method's commit, not as the user types. Fields that don't report by themselves are sent an empty commit, every 200 ms while typing and every second when idle, and only until they show they do report.
  • GNOME identifies apps through Window Calls or accessibility. Without either, a browser gets no suggestions and its typing is untouched. That's the fail-closed rule for exclusions.
  • Known gaps: GTK 3 doesn't report selections to an input method, so an accept there replaces selected text as typing would. Window text (OCR) isn't used on these routes. The ghost shows in the app's preedit style, usually underlined, not dimmed.
  • Translations of the 11 new strings are mine and need native review, as the README says of desktop strings.

🤖 Generated with Claude Code

stevyhacker and others added 8 commits November 2, 2026 21:24
Ports what the Mac has and the Rust app lacked: selection actions; the
dictation and autocomplete gaps (delivery guard, Compose context, live
transcript, accept keys, macros, statistics, learning, an autocomplete server);
spoken summaries; remembered speakers; Files; screen search by meaning; the
overnight review; Personal Context; coding-agent sessions; note and daily
exports; routines; library health; retention review; the Actions, People and
Projects workspaces; agent handoff; the command palette; Settings Overview and
search; agents changing settings, behind the Mac's approval; and hosting paired
phones and tablets.

Also fixes bugs that reached users. Saving settings dropped keys this port
didn't model, so a library from a Mac lost them. Microphone pickers were always
empty. Closing the window during a recording lost its end time and job. Mic
transcript rows were deleted on word overlap alone. Debug logs held typed text.
Think accepted plain HTTP to a server off this computer, where the Mac requires
HTTPS. Picture vectors outlived their pixels, and deleted sources left generated
journals and work memory behind.

Coding-agent sessions need an explicit choice here, unlike on the Mac: installs
already saved that key as true without asking, so an update must not start
reading them silently. Remembered speakers only suggest names, because the
Mac's thresholds aren't calibrated for this voice model on real calls.

Not ported: Gmail in meeting briefs, and Mac-only speech runtimes. All Windows
code is only type-checked, and Wayland, real phones and real calls were never
run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
589 strings into the 23 other interface languages, machine-assisted like the
rest of the desktop catalog and in need of native-speaker review. The two hints
that show the "Remember" trigger keep it in English or Serbian, the only words
the parser recognises.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Moves features out of the Mac-only lists, and says what differs where it does.
Also removes a paragraph in PRIVACY.md that an old merge had duplicated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and Windows

Follows #424 and #426 on the Mac: every pass walks the chosen folders again, so
a change the crawl would leave out no longer starts one. It leaves out tmp and
temp folders too, while a folder chosen on its own is still read whole.

Unlike FSEvents, the watchers here often don't say whether a path is a file or
a folder, so the path answers where it still exists. When nothing can tell, the
change counts, unless it's in a hidden folder.

Also restores the test of clean_other_names, lost when that function moved to
lokalbot-core.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ocrs, the only engine so far, reads unaccented Latin letters, and the Mac's
Vision reads the rest with nothing to download.

Windows now uses Windows.Media.Ocr with the language packs installed there.
Linux runs the system's Tesseract for the app and system language, over a
pipe, with no image file at any point. It uses the language data it has, or
pinned tessdata_fast files the person chooses to download. ocrs stays as the
fallback, and stays the engine for English where its models are installed.

Settings and onboarding now offer whichever download helps, right under the
choice that needs it, and say what is read and what is missing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The folder picker saves a rule under the home folder as `~\Code\client`, but only
`~/` was expanded, so on Windows that exclusion matched nothing and sessions from
an excluded folder were still read. A home path with backslashes was also left
unnormalized.

Found by the first real Windows CI run, which failed on a test that fed Unix
paths to `Path::join`; that test now uses each platform's separator. The Windows
job now runs every crate, so one run shows every failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wayland lets no ordinary app take another app's keys, show text at its caret
or type into it, so autocomplete worked only in the practice editor there. An
input method is the supported way in: it sees the text around the caret, shows
the suggestion inline as preedit and commits text without the clipboard.

Sway and Hyprland use zwp_input_method_v2, with the keyboard grabbed only while
a suggestion shows. GNOME uses an IBus engine the user chooses as an input
source. Both name the field's owner before using any of its text, refuse
password fields, terminals, excluded apps and apps they cannot name, and stay
out of the way of another input method. X11 and Windows are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
lokalbot 40b877e Commit Preview URL

Branch Preview URL
Nov 03 2026, 01:36 AM

stevyhacker and others added 8 commits November 3, 2026 00:58
Stopping a pipe server woke its accept with a connection it closed at once. A
Windows pipe listener skips a client that has already left, so if the thread
hadn't reached ConnectNamedPipe yet, accept kept waiting and join waited on it
forever. That could freeze the app on quit, or when agent settings control is
turned off. The wake-up connection now stays open until the thread has gone.
IpcServer, which already shipped, had the same bug.

Found by the first full Windows CI run, where the test hung until the job timed
out. Also fixes two tests that run failed: one fed a Unix path to Path::join, and
one compared days across midnight on the runner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both failed on and off in xcodebuild test on master and on pull requests.

Turning Files off deletes the index on the reader's and worker's queues, and
the test gave that 300 ms. It now waits up to 20 s for the folder to go.

FSEvents can report notes.txt, written just before its stream began, and that
starts a second pass during the wait that should show none. The test now
outlasts the watcher's latency before it counts, and again after the writes
that can't alter the index: with a 2 s latency, the old 1 s wait could end
before those writes had been reported at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Windows and Linux checks photographed the app's older pages only, so
Files, Devices and Personal Context had never been seen on a real desktop.
On Windows the palette is opened with Ctrl+K, as a person would.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The update's end-to-end check on a real Windows desktop failed with "could not
replace lokalbot.exe: The process cannot access the file because it is being
used by another process", and the whole update gave up on the first try.
Master passed that check on the same step, so nothing shows this had reached
users, but anything that opens the file without allowing its rename can cause
it: antivirus after a new file, or reading a running executable's version
details, as LokalBot does to name the focused app.

Each rename on Windows now keeps trying for up to 30 s on a sharing violation,
and says when it waited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stevyhacker
stevyhacker changed the base branch from claude/ocr-all-scripts to master November 3, 2026 01:31
stevyhacker and others added 2 commits November 3, 2026 01:34
Release notes for the next version link these. The screenshots are the demo
library on a Windows CI desktop, and each explainer says what hasn't been run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stevyhacker
stevyhacker marked this pull request as ready for review November 3, 2026 01:37
@stevyhacker
stevyhacker merged commit e668b3d into master Nov 3, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant