feat(gesture): native pan and pinch zoom, real trackpad gestures on any button

[akincan-kilic]

Hey! I love this project, it is the reason my MX Master is usable on macOS at all, and pan and zoom were the two things I kept missing from Options+. So I spent a few days building them. This is 22 commits squashed into one.

The short version: hold a bound button and your mouse becomes a trackpad. Not a scroll emulation, an actual macOS trackpad gesture, so apps treat it exactly like two fingers on the pad.

What I built

• Pan. Hold, move, and the view follows your hand. The cursor freezes while you do it.
• Real two-axis motion. It is one gesture on both axes at once, so a diagonal drag pans diagonally. Not four-way scroll snapped to up, down, left, right.
• Pinch zoom. Hold and drag up or down for continuous magnification, the same event a trackpad pinch produces.
• Smart zoom on a click. Click the same Zoom button without dragging and you get the native smart-zoom toggle. Both zoom gestures on one button.
• DPI normalised. Speed is derived from a live getSensorDpi read, so cycling your DPI does not change how far a gesture travels.
• Two settings. Zoom Sensitivity on the same 1 to 100 slider as the existing ones, and Invert pan direction. Both snapshot at button-down, so saving settings mid-gesture will not change the scale under your hand.
• All 23 locales filled in.

Demo

pan-zoom-demo.mp4

Panning a spreadsheet, then zooming in and back out, on the Forward and Back buttons of an MX Master 3S over Bluetooth. The recording does not show the button presses: pan is Forward held and dragged, zoom is Back held and dragged, and the jump in magnification early on is a single click firing smart zoom.

Issues

Closes #360 (Zoom gesture).

Related:

• [Bug]: First raw-XY sample after GestureButton press is not discarded — poisons swipe accumulator with spurious deltas #752 reports that the first raw-XY sample after a GestureButton press is not discarded. Same root cause I hit, and I have hardware numbers below that back it up. I did not fix it here because that is the swipe path, not hold mode, but it should be a small follow-up.
• [Feature]: Add SmartZoom browser action #268 asks for smart zoom as a bindable action, which feat(macos): add native Smart Zoom action #1119 does directly. This is the same event fired from a different place.
• fix(macos): write the scroll point delta last so pixel precision survives #1156 fixes the CoreGraphics scroll-delta field ordering on the scroll path. I hit the identical thing on the pan path, independently, before I saw that PR.

Smart zoom's event synthesis, the ZoomToggle subtype and the two private payload markers, is Kyle Foley's work from #1119. Reused with thanks and credited at the call site.

How it works

HID++ feature 0x1b04 diverts the button with the raw_xy flag, so the firmware streams sensor deltas over HID++ instead of moving the pointer. Those counts convert to screen pixels for pan, or to a magnification increment for zoom, scaled by a live 0x2201 getSensorDpi read. Then they go out as real OS gesture events.

Three crates are involved. openlogi-device tracks the hold and decides whether it cleared the click/drag deadzone. openlogi-agent-core does the counts-to-output conversion. openlogi-inject holds the OS gesture session and is the only place that knows a pan or pinch is currently open.

What broke along the way, and how I fixed it (long, but if you are ever implementing this, it will save you a week 😄)

Everything below was measured on an MX Master 3S over Bluetooth, MacBookPro18,1, macOS 26, sensor at 950 DPI.

Hold mode never armed and never said why

Binding Pan did nothing. No error, no log line.

live_sensor_dpi() in orchestrator.rs was not a sensor read. It read the configured DPI-preset cycle, which is empty for anyone who never set one, so sensor_dpi was None and the arming gate required it. Hold mode failed closed silently.

Fixed with a real 0x2201 read and a fallback chain: live read, then cycle preset, then configured DPI, then a named constant with a warning. Injection availability is the only hard gate now, because arming a raw-XY divert with no injector freezes the cursor with nothing to deliver the motion. The capture log line carries the resolved DPI and hold-button count so this is visible next time.

Zoom did nothing at all

I was posting NSEventTypeMagnify, CGEventType 30, which is what the AppKit docs point you at.

I wrote a throwaway CGEventTap probe and captured 339 events from real trackpad pinches. Every real pinch is CGEventType 29 (NSEventTypeGesture), never 30, carrying field 110 = 8 (kIOHIDEventTypeZoom), field 113 = magnitude, field 132 = phase. AppKit is what promotes 29 into the NSEventTypeMagnify that apps see. Posting 30 at the HID layer does nothing.

It also needs a Mach timestamp. Without one Safari accepts a zoom-in and then refuses the gesture that toggles back out. #1119 documents this and it saved me an afternoon.

Small pans disappeared

Anything under a few pixels produced nothing.

CoreGraphics recomputes the sibling scroll-delta fields every time you write one, and a line delta of 0, which is any pixel delta under 10, zeroes pointDelta behind you. Write the line field first, pixel field second.

Found by capturing 1304 real two-finger trackpad scroll events and diffing field by field. Of 546 scroll events, 541 carry a non-zero pointDelta, 209 also carry fixedPt and a line delta, and zero carry fixedPt without pointDelta. #1156 hit the same behaviour on the scroll path.

The pan jump

This was the hard one. Activating pan made the view jump, sometimes most of a screen, while the cursor stayed put. Intermittent, different distance every time.

Wrong guesses, in case they look tempting to you too:

• Missing event location on the posted scroll events. Disproven by measurement: CGEventGetLocation on an unposted scroll event already returns live cursor coordinates, and tapping our own posted pans showed the right location on every frame including Began.
• An accumulator banking travel between holds. Ruled out by reading the code. The stream accumulator returns the raw per-report delta, never the running total, and pan begin resets the quantizer.
• Pre-existing gesture-mode scrolling. I asserted this too early on stale log lines that predated the config change they were meant to explain. The actual cause of that symptom was Mac Mouse Fix running alongside my dev build, so I spent a while debugging someone else's app.

What found it was instrumentation plus a scripted repro. Every raw-XY report logged with a per-hold sequence number, the pixel delta it became, and the frames posted. Then: sit on a known cell, activate without moving, move a long way with the gesture off, activate again, repeat.

Every jump was sequence number 1. Never anywhere else in the stream.

observed	seq	raw counts	implied hand speed	posted
F23 to F69	1	-1694, +1619	6264 mm/s	-996, +952 px
"maybe 5px"	1	21, -105	448 mm/s	12, -61 px
"nothing happened"	1	-6, -2	negligible	0, 0
F23 to F24	1	7, +27	negligible	4, +16 px

F23 to F69 is 46 rows, and at Sheets' 21 px row height that is 966 pixels against the 952 we posted. The fastest real motion in five minutes of panning was 1306 mm/s, so 6264 mm/s is not a hand.

The evidence that settled it was an absence: zero raw-XY log lines between app start at 20:07:46 and the first press at 20:13:14. The firmware does not stream diverted raw XY until the divert engages, and the first packet after it engages flushes whatever the sensor banked beforehand. That is also why later presses were usually clean.

So: drop the first raw-XY report of every hold-mode stream. Not a threshold or a speed clamp, because the report is pre-press travel by construction. The cost is one HID report, about 8 ms, from a hand that is not moving at button-down.

I first put this in the agent's hold dispatcher. It stopped the jump but it belonged one layer up. The device layer already had a skip_first_raw_xy flag for the haptic panel's contact jump, and begin_hold just never set it for hold-mode streams. Moving it there made it smaller and kept the backlog out of the click/drag deadzone, which matters because smart zoom depends on that deadzone being clean. Downstream, the deadzone still saw 1694 counts and would have read every first press as a drag.

This is the same root cause as #752 on the swipe path.

Five more found by review

I had this working on hardware and then ran two adversarial review passes over it. Five real bugs came out, all fixed here, each with a test that fails without its fix.

• HoldEnd only carried traveled: bool, so a teardown looked identical to the user letting go. A Zoom button still held through a Bluetooth reconnect fired a smart zoom into whatever was frontmost. It carries HoldRelease::{Released { traveled }, Interrupted} now.
• The stale-hold bound measured from the press instead of the last sample, so an actively streaming pan was force-ended 10 seconds after button-down. Reproduced at 10.14 s mid-drag. It measures quiet time now, which is what actually indicates a lost button-up, since the firmware only streams while the control is down.
• openlogi-inject computed each gesture frame under the GESTURE_SESSIONS mutex and posted it after releasing the guard, so a late Began could outrace a shutdown Ended. The post happens under the guard now.
• Closing the sessions at shutdown was not terminal. zoom_continuous opens a pinch from closed by contract, which is how each new hold starts one, and the gesture watcher is not joined before exit. A thread still streaming reopened a pinch after the final flush, and process::exit skips Drop. On Linux and Windows that is a held Ctrl with nothing to release it. The exit path seals now instead of flushing.
• sensor_dpi was on CaptureSpec, which is part of CaptureTarget's identity, so the first getSensorDpi completing a second after connect changed the target and retired the whole capture session, killing any hold in progress. The armed spec carries only the fallback now. The live reading still reaches the millimetre conversion through the dispatch plan, and the deadzone reads the sensor cache at press time.

Testing

cargo fmt --check clean, cargo clippy --workspace --all-targets -D warnings clean, 37 suites, 0 failures.

Exercised by hand on the device, not just in tests: pan in Sheets with no jump across repeated activations, pan held well past 10 seconds, zoom in and out, zoom sensitivity at 1, 14 and 100, inverted pan, smart zoom toggling in and back out, and continuous pinch on the same button.

One flake turned up that is not mine: tray::tests::startup_stays_suspended_when_the_display_is_already_asleep posts a real NSWorkspace notification while sibling tests hold observers, and fails roughly one run in five. I ran the same suite eight times on a clean worktree at the base commit with none of my code in it and it failed there too, same assertion. #1168 looks like it is aimed at this.

Notes for review

• The first-sample drop is unconditional for hold-mode streams. I tried a plausibility gate on implied hand speed and dropped it: a small backlog passes the gate and still jumps, and the cost of dropping unconditionally is one HID report at the start of a gesture.
• AppSettings keeps #[serde(deny_unknown_fields)], so the two new fields downgrade the same way thumbwheel_sensitivity already does. I did not want to change that policy inside a feature PR, but say the word.
• A DPI refresh landing mid-hold still ends that one hold through the dispatch plan. Much smaller than retiring the capture session, and it happens at most once per device per process. Happy to teach the reconciler that a DPI-only delta is not worth cancelling input lifecycles if you would rather.
• Google Sheets scrolls a little while zooming. I checked with a real trackpad pinch and it does the same thing, so it is Sheets, not this.

Happy to split this up or change anything.

[greptile-apps]

Greptile Summary

This PR adds native hold-to-pan and hold-to-pinch gestures, including DPI-normalized device capture, platform-specific injection, lifecycle-safe teardown, configurable sensitivity and direction, UI integration, IPC versioning, and localization.

• Adds raw-XY hold capture and conversion into pan, continuous zoom, and click-to-smart-zoom behavior.
• Adds macOS native gesture events and Linux/Windows gesture approximations with centralized session cleanup.
• Extends bindings, settings, desktop controls, wire-format tests, and all supported locales.

Confidence Score: 4/5

The PR appears safe to merge, with only non-blocking cleanup needed around duplicate DPI-read coalescing and shutdown documentation.

Gesture retirement and process-exit sealing cover the critical injected-input lifecycle, while the remaining issues cause redundant background device traffic and misleading maintenance guidance rather than a current correctness failure.

Files Needing Attention: crates/openlogi-agent-core/src/orchestrator.rs, crates/openlogi-agent-core/src/runtime.rs

Important Files Changed

Filename	Overview
crates/openlogi-agent-core/src/orchestrator.rs	Adds host capability and live-DPI planning, but sensor refreshes can enqueue duplicate reads while an earlier request remains in flight.
crates/openlogi-device/src/session/gesture.rs	Adds raw-XY hold lifecycle, first-report suppression, deadzone tracking, interruption handling, and stale-session cleanup with substantial focused coverage.
crates/openlogi-agent-core/src/watchers/gesture/dispatch/hold.rs	Converts hold events into DPI-normalized pan and zoom commands while preventing retired sessions and interruptions from reopening or clicking.
crates/openlogi-inject/src/inject/gesture.rs	Centralizes serialized gesture sessions, terminal sealing, quantization, and platform-independent lifecycle behavior.
crates/openlogi-inject/src/inject/macos.rs	Implements native Core Graphics pan, pinch, and smart-zoom event synthesis with explicit phase and timestamp handling.
crates/openlogi-agent/src/shutdown.rs	Funnels explicit process exits and relaunches through a terminal gesture-session seal.
crates/openlogi-agent-core/src/runtime.rs	Flushes gesture state during runtime shutdown, but its documentation incorrectly describes that reopenable flush as the terminal emit.
crates/openlogi-core/src/binding/action.rs	Appends Pan and Zoom to the persisted action vocabulary without shifting existing serialized variant indexes.
crates/openlogi-ipc/src/ipc.rs	Correctly bumps the strict IPC protocol version for the newly serialized action variants.
crates/openlogi-core/src/config/settings.rs	Adds validated zoom-sensitivity and pan-direction settings used consistently by agent and desktop state.
crates/openlogi-desktop/src/windows/settings/general.rs	Exposes the new zoom sensitivity and pan inversion controls in general settings.

Sequence Diagram

sequenceDiagram
    participant M as Mouse firmware
    participant D as Device hold session
    participant A as Agent dispatcher
    participant I as Injection session
    participant O as Operating system
    M->>D: Button down + raw XY
    D->>D: Drop first sample and apply deadzone
    D->>A: HoldBegin / HoldMotion
    A->>A: Normalize counts using sensor DPI
    A->>I: Pan or pinch frame
    I->>O: Native gesture event
    M->>D: Button release
    D->>A: HoldEnd(released or interrupted)
    A->>I: End gesture or smart zoom
    I->>O: Terminal gesture event

Loading

_{Reviews (1): Last reviewed commit: "feat: add hold-mode Pan and Zoom actions..." | Re-trigger Greptile}

[greptile-apps]

DPI reads lack coalescing

If device runtime state is republished before an earlier getSensorDpi request completes, the completed-value cache remains empty and another request is detached for the same route. These redundant HID++ transactions and worker threads queue behind the receiver lease and can unnecessarily delay later receiver operations on slow devices; track an in-flight request per route or otherwise coalesce refreshes.

Knowledge Base Used: Background agent service

[greptile-apps]

Flush is not terminal

This documentation calls flush_gesture_sessions the guaranteed terminal emit, but flushing permits later watcher input to reopen a gesture; the actual terminal operation is the subsequent seal_gesture_sessions call in the process-exit funnel. Describing that distinction here avoids encouraging a future shutdown path to rely on the flush alone and strand a pinch or held Ctrl state.

Suggested change

	/// Reject new button input, emit terminal cancellation, and join the worker.
	///
	/// Also flushes hold-mode inject sessions. The agent calls `process::exit`
	/// after this, which skips `Drop`; this is the guaranteed terminal emit.
	pub fn shutdown(&mut self) {
	openlogi_inject::flush_gesture_sessions();
	/// Reject new button input, emit cancellation, and join the worker.
	///
	/// Also flushes hold-mode inject sessions. This flush is reopenable; the
	/// process-exit funnel performs the guaranteed terminal seal afterward.
	pub fn shutdown(&mut self) {
	openlogi_inject::flush_gesture_sessions();

Knowledge Base Used:

• Background agent service
• Synthetic input injection

[foleykyle01]

Thanks for the credit! This looks great! Nice to see the Smart Zoom work carry over into the broader pan/zoom implementation. Credit to @AprilNEA big time. Love working on this.

[rmacasieb]

I'm so stoked for this!