Skip to content

feat(meetings): label Zoom speakers from Accessibility - #833

Open
Rahulk644 wants to merge 1 commit into
rowboatlabs:mainfrom
Rahulk644:codex/zoom-accessibility-evidence
Open

feat(meetings): label Zoom speakers from Accessibility#833
Rahulk644 wants to merge 1 commit into
rowboatlabs:mainfrom
Rahulk644:codex/zoom-accessibility-evidence

Conversation

@Rahulk644

@Rahulk644 Rahulk644 commented Aug 5, 2026

Copy link
Copy Markdown

Problem

Rowboat captures the local microphone and system audio separately, but diarization cannot map system-audio clusters to the participant display names shown by native Zoom. Meeting end also needs a stronger signal than a quiet audio track.

What this adds

  • An optional macOS Swift Accessibility helper scoped to native Zoom (us.zoom.xos).
  • A typed, privacy-bounded event contract for permission, meeting-surface, and active-speaker evidence.
  • Conservative alignment of system-audio transcript intervals to Zoom display names.
  • A three-poll disappearance gate, enabled only after a validated meeting surface was observed, before emitting Rowboat's existing external-call-ended event.
  • Best-effort packaging plus parser and attribution tests.
  • Explicit Open Settings permission UX; starting notes does not raise a surprise macOS Accessibility prompt.

The microphone remains You. Missing, stale, self, overlapping, or conflicting evidence keeps the existing generic Speaker N label instead of guessing. Capture and transcription remain usable when Accessibility or the optional helper is unavailable.

Privacy, provenance, and bounds

The helper inspects at most two native Zoom processes, eight windows per process, 1,800 AX nodes total, 18 levels, 128 children per node, and one 600 ms budget per observation. It does not read editable/secure-field values or emit raw AX nodes, arbitrary labels, window titles, screenshots, transcript text, process paths, or participant rosters. It exits when Electron closes its stdin lifetime pipe.

Traversal and explicit-label parsing are adapted from Anarlog's MIT-licensed meeting_ax at revision 609ee772801f29292e4edee453f269089ebf0e8b. The exact revision, retained copyright/license, and Rowboat-specific changes are documented beside the helper.

Scope and composition

This PR supports native Zoom on macOS only. The documentation gives concrete extension paths for Chrome/Meet via a metadata-only extension/native host when AX is insufficient, Teams via its own bundle-scoped AX parser, and Windows via UI Automation. Unsupported platforms keep generic diarization and never block recording.

The evidence resolver is independent of the transcription vendor. #835 now exposes provider-neutral stable-audio intervals; the two PRs have been exercised together on a combined integration branch without creating a source dependency between them.

Verification

Passed on the PR head:

  • workspace TypeScript check
  • workspace ESLint
  • shared/core/preload dependency builds
  • main-process bundle, including helper compile and self-test
  • focused Zoom attribution tests: 5/5
  • standalone native helper compile and parser self-test
  • git diff --check

The broader core suite passed 585/586 tests; the unchanged ChatGPT token-revocation test timed out while attempting its network path in the restricted test environment. This branch has no core-package diff.

Remaining qualification

This PR is ready for code review, but unit/parser/build checks are not physical Zoom or macOS TCC evidence. A signed packaged build still needs the documented matrix: local-only speech, remote-only speech, overlap, mute/unmute, minimization, screen sharing, identical display names, permission revoke/re-grant, and meeting end while Zoom remains open.

Add an optional, bounded macOS helper that reads only native Zoom Accessibility evidence while Rowboat is already recording. Align active-speaker observations with system-audio transcript intervals, preserve generic diarization on stale, self, overlap, or conflicting evidence, and emit a conservative meeting-end hint after a proven surface disappears.

The helper is adapted from Anarlog meeting_ax at revision 609ee772801f29292e4edee453f269089ebf0e8b under its retained MIT notice. It is fail-open for recording, uses an explicit System Settings action instead of a surprise permission prompt, and makes no Chrome, Teams, Windows, or full physical-matrix claim.
@Rahulk644
Rahulk644 force-pushed the codex/zoom-accessibility-evidence branch from 355a7ec to cccc737 Compare August 6, 2026 07:07
@Rahulk644
Rahulk644 marked this pull request as ready for review August 6, 2026 11:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant