Native iOS companion for Vobot V05E and Apple Watch recording workflows. It connects to V05E over BLE, manages recordings, and runs ASR (Azure, local Whisper, or fully on-device). Project overview and licence: nomily-app.
| Device | Link | What works |
|---|---|---|
| Vobot V05E | BLE | Native recording transfer workflow; compatibility depends on the tested device and firmware. |
| Apple Watch | watchOS companion | Records on the watch and transfers to the paired iPhone. |
| Audio files already on iPhone | File import | Import into the library for transcription and summary workflows. This is not a device-integration claim. |
No other recorder model is publicly represented as compatible with Nomily merely because it records audio, exposes Bluetooth, or provides an export function.
Native companion app for the Vobot V05E workflow, Apple Watch recording, local library management, live and file transcription (Azure, local Whisper, or on-device), and Wi-Fi transfer where supported by the V05E workflow.
- Scan + connect with escalating-timeout retries.
- Full Device sheet: switches, mic-gain / NR sliders, idle-off picker, Bluetooth-name editor with live byte counter, sync-time, and Format / Factory-reset / Shut-down with destructive confirmations.
- Recordings tab has two segments:
- Device — lists device files, transfers them over BLE with live progress, cancel mid-transfer, swipe-to-delete. Each download is ChaCha20-decrypted (if a key is configured) and OGG-wrapped via in-app
OpusOgg, so every clip lands as a real Ogg-Opus container. A per-rowtext.bubblebutton ships the file to the configured ASR provider; transcripts are written as{name}.txt+{name}.asr.jsonnext to the audio. When the pending backlog exceeds 512 KB, a Fast Transfer button appears at the bottom — tapping it asks the device to bring up its Wi-Fi access point, joins it viaNEHotspotConfigurationManager, and pulls the queue over that link (~20× faster than BLE). - Library — lists all clips on disk, sorted newest-first, with file size, Ogg-derived duration, and a transcript-ready badge. Tap → Clip detail (filename / size / duration / mtime hero, AVAudioPlayer-backed play/pause + scrubber, transcript reader with speaker labels, Transcribe / Re-transcribe buttons, delete-locally action).
- Device — lists device files, transfers them over BLE with live progress, cancel mid-transfer, swipe-to-delete. Each download is ChaCha20-decrypted (if a key is configured) and OGG-wrapped via in-app
- Stream tab starts/stops the real-time OPUS stream and, with the Live Transcription toggle on, pipes every OPUS frame through a WebSocket to the local faster-whisper server (
ws://{host}:{port}/v1/listen). A transcript panel renders committed segments plus the current italic-grey partial. - Settings tab is fully bound to
config.json(ASR providers, Azure key/region, local server, ChaCha20 key, Wi-Fi AP creds, known devices).min_transcribe_durationis enforced before the network call. - Apple Watch companion (
Watch/, embedded in the iPhone app) — a single recorder screen:- Watch — records AAC/M4A at 24 kHz mono / 48 kbps (~21 MB/hour) straight from the wrist, standalone, with
audiobackground mode so a session survives screen-off. Clips are namedyyyyMMddHHmmss.m4aand queued to the iPhone viaWCSession.transferFile; the watch-side copy is deleted only once the phone acknowledges it stored the clip, and anything still local is re-queued on launch. On the phone they land inaudio_clips/decrypted/like any other clip, get a manifest entry withtransport: "watch"(the Library row shows anapplewatchbadge), and honour the same Auto-transcribe after download toggle. - Settings → Apple Watch on the phone shows whether a watch is paired and whether the companion app is installed, with the install steps inline — a bundled watch app otherwise arrives with no prompt at all.
- Action Button —
ToggleRecordingIntentis published as an App Shortcut, so the Ultra's Action Button can start/stop a wrist recording (Settings → Action Button → Action: Shortcut, then pick it). One action, so it toggles.
- Watch — records AAC/M4A at 24 kHz mono / 48 kbps (~21 MB/hour) straight from the wrist, standalone, with
Still missing: libopus binary target for iOS 16 .opus playback (iOS 17+ plays Opus natively today).
- macOS with Xcode 15+
- iOS 16.0+ device (BLE not available in the simulator)
- watchOS 10.0+ paired Apple Watch, for the companion app (mic, background recording and battery behaviour can't be verified in the simulator)
- XcodeGen:
brew install xcodegen
cd ios
xcodegen generate # produces nomily_ios.xcodeproj from project.yml
open nomily_ios.xcodeprojIn Xcode:
- Select the
nomily_iostarget → Signing & Capabilities → set your Team and a unique bundle ID (default:com.nomily.app). - Do the same for the
nomily_watchtarget. Its bundle ID must stay{iPhone bundle ID}.watchkitapp, andWKCompanionAppBundleIdentifierinproject.ymlmust match the iPhone ID — a mismatch installs the watch app as an orphan that never pairs. - Plug in an iPhone, select it as the run destination, ⌘R. The watch app is embedded in the iPhone app; you don't run the watch scheme separately unless you're debugging it. With Automatic App Install on (Watch app → General), it lands on the wrist by itself a few minutes later — otherwise open the Watch app on the iPhone → Available Apps → Install next to Nomily AI. The app's own Settings → Apple Watch row reports which state you're in.
- First launch will request Bluetooth permission. Grant it. The watch app asks for microphone permission the first time you record on the wrist.
Sources/
├── App/ # @main, RootView, Info.plist, entitlements
├── Components/ # Reusable views (ConnectionPill, RSSIBars, …)
├── Features/
│ ├── Recordings/ # Device + Library segments, Clip detail, playback
│ ├── Stream/ # Live transcription
│ ├── Settings/ # ASR providers, keys, known devices
│ ├── Device/ # Modal sheet for device info + settings
│ └── Scanner/ # BLE scan + connect sheet
├── Services/
│ ├── ASR/ # AsrTypes, AzureASR, LocalASR, LiveStreamASR, OnDeviceASR, TranscriptionService
│ ├── BLE/ # CoreBluetooth: protocol, codec, scanner, client
│ ├── Crypto/ # ChaCha20 raw stream cipher (RFC 8439)
│ ├── Audio/ # OpusOgg (raw frames → Ogg-Opus container)
│ ├── Storage/ # config.json, library manifest, PostDownload pipeline
│ ├── Watch/ # WatchSyncService (phone side) + the shared wire contract
│ └── Wifi/ # WifiClient + HotspotJoiner (NEHotspotConfiguration)
└── Resources/
└── Assets.xcassets/
Watch/ # watchOS target: @main, WatchLink (WCSession), recorder, Action Button intent
Three files are compiled into both targets so phone and watch can't drift: Sources/Services/Watch/WatchLinkMessage.swift (the WatchConnectivity wire contract), Sources/Services/Storage/RecordingName.swift (the yyyyMMddHHmmss clip-naming convention) and Sources/App/L10n.swift + Sources/Resources/*.lproj (the same 10 localizations). See the sources: list on the nomily_watch target in project.yml.
- After a run on a physical device, collect the log archive:
rm -rf .logs/dnote.logarchive
sudo log collect --device --last 3m --output ./.logs/dnote.logarchive
# BEGINSWITH, not ==: the app logs under com.nomily.app.ios and the
# watch app under com.nomily.app.watch
log show ./.logs/dnote.logarchive \
--predicate 'subsystem BEGINSWITH "com.nomily.app"' \
--info --debug > ./.logs/dnote.log
The run is then readable in ./.logs/dnote.log.
Three providers, picked in Settings → Speech Recognition and tried in the order
primary → fallbacks → on-device:
| Provider | Needs | Live | Notes |
|---|---|---|---|
| Azure Speech | key + region | yes (and live translation) | best quality, costs money |
| Local server | faster-whisper box on the LAN | yes | ws://host:port/v1/listen |
| On-device | nothing | yes, in ~3 s batches | Apple Speech, requiresOnDeviceRecognition — audio never leaves the phone |
The on-device path (Sources/Services/ASR/OnDeviceASR.swift) exists for two reasons:
- It closes the last gap that required an account. With no key and no server, live
transcription used to fail inside
openASRbefore the card was told to record — the symptom was "the record button did nothing", not "no text". Recording and transcription are now independent: if no service is available the session still records and streams, says so, and the saved clip is transcribed afterwards. - The decoder. Everything our recorders produce is Opus in Ogg and
SFSpeechRecognizerwants LPCM.OnDeviceASRdecodes withAVAssetReader(16 kHz mono Int16) — the same pathAudioExporter.exportAsWAValready uses.AudioConverterwithkAudioFormatOpuswas tried and rejected: the platform decoder latches end-of-stream after the first fill that drains its input, so it cannot be fed packet-by-packet.
Live on-device recognition batches ~3 s of packets, wraps them into an Ogg container in a temp file, decodes, and appends the PCM to a running recognition request — so text arrives in phrase bursts rather than word by word. A configured Azure/whisper service, when present, wins on quality and is tried first.
Summarize right after transcribing (Settings, on by default) hands each fresh transcript to the configured LLM as a summary job. It is skipped when no LLM is configured, and it never overwrites a summary that already exists on disk.
The Local server (faster-whisper) and Ollama providers need no account and no API key — but they do have to be installed and started on a computer that shares the phone's Wi-Fi. If that sounds like sysadmin work, let an AI do it: open ChatGPT or Claude Code on that computer, paste the prompt below, and let it install, write and start everything. When it finishes it prints the exact values to type into the app. The same prompt is copyable from inside the app under Settings → Transcription → Local ASR server → Copy setup prompt.
The prompt (English)
Set up this computer as the local speech backend for the Nomily AI recorder app, then print exactly what I should type into the app. Three steps:
1. Install and start Ollama, enable auto-start, expose it to the LAN (OLLAMA_HOST=0.0.0.0), and pull a chat model that fits this machine's RAM (qwen2.5:3b for 8 GB, qwen2.5:7b for 16 GB+).
2. pip-install faster-whisper and write a Python service (FastAPI + uvicorn + websockets) that runs at login (launchd on macOS, systemd/Task Scheduler elsewhere) with firewall ports 12300 and 12301 open. It must implement:
- HTTP batch transcription: listen on port 12301, POST /v1/transcribe, request body is raw audio bytes (Content-Type: application/octet-stream, usually an Ogg-Opus container that faster-whisper decodes directly), respond with JSON: {"text": "full text", "segments": [{"start": sec, "end": sec, "text": "segment"}]}.
- WebSocket live stream: listen on port 12300 at path /v1/listen. The client streams equal-length raw Opus packets (binary frames, no container header — reassemble by fixed packet length, decode to PCM, feed a streaming VAD + faster-whisper). For each recognized segment reply with a JSON text frame {"text": "...", "is_final": true or false}; on receiving the text frame {"type":"CloseStream"}, flush remaining audio, emit the final segment, and close.
3. When everything runs, print four lines: (1) this computer's LAN IP (from ifconfig/ipconfig, the 192.168.x or 10.x one); (2) the host to enter in Nomily AI under Settings → Transcription → Local ASR server (that LAN IP); (3) the port for that page (always 12300); (4) the address for Settings → AI Providers → Ollama (http://LAN-IP:11434) and the model name. Remind me the phone and this computer must be on the same Wi-Fi, and deployment is done only when both verify buttons in the app pass.
What to type into Nomily AI afterwards (the AI prints these too):
| Settings page | Field | Value |
|---|---|---|
| Settings → Transcription → Local ASR server | Host / Port | the computer's LAN IP / 12300 |
| Settings → AI Providers → Ollama | Endpoint / Model | http://<LAN-IP>:11434 / e.g. qwen2.5:3b |
Then tap both verify buttons — a green result is the whole deployment check. The port
convention is fixed: 12300 is the live WebSocket, batch transcription rides 12301
(port + 1) — the prompt bakes both into the generated server, and the app does the same
(see LocalASR / LiveStreamASR).
Notes for the curious: the phone and the computer must stay on the same Wi-Fi; iOS asks for Local Network permission on first use, grant it; a CPU-only box is fine for faster-whisper (an hour of audio takes minutes), while the summary quality is bounded by the Ollama model size. With both wired up, turn on Auto-transcribe imported recordings and Summarise right after transcribing and the whole pipeline — record → transcribe → summarize — runs without a single tap or a single key.
- Bluetooth, hotspot configuration, and an app group are declared in
Sources/App/nomily_ios.entitlements. The hotspot capability requires a paid Apple Developer account (free profiles cannot enable it). - Sources are organised by feature; the BLE service is the foundation everything else depends on.
- The watch recorder started life as the standalone
apple-watch-recorderprototype (Apple Watch Ultra long-recording experiment); the audio settings and the screen-off background-mode approach carry over from it. - Watch-target resources are declared under
sources:withbuildPhase: resources, not underresources:. XcodeGen dedupes a path already claimed by another target'sresources:, which silently ships a watch bundle with no.lprojand no asset catalog — the UI then renders raw localization keys.
- iOS 16
.opusplayback via libopus + libogg SPM binary target. iOS 17+ already plays Ogg-Opus natively throughAVAudioPlayer. - Fast Transfer polish: mid-BLE-transfer handoff (cancel in-flight downloads when the user taps the button).
- Watch: verify multi-hour screen-off recording on real hardware (the prototype's open question), and consider an Action-Button / complication launch path.