Meeting Assistant

Problem

In long meetings, questions get asked and then lost. Nobody remembers which ones were answered, and by the next meeting in a series they’re gone. Cloud meeting bots handle this by recording everyone and sending the audio to a server. That’s expensive for hours-long calls, and hard to justify for people who never agreed to be recorded. I wanted a personal tool that listens to any meeting app, shows a live transcript and keeps a running list of open questions across meetings, while meeting audio stays on the machine.

My role

Sole engineer: domain model, architecture decisions (four ADRs), Rust core, Swift audio sidecar, Tauri shell and React UI. Work ran through GitHub issues with triage labels and a shared glossary, so any issue could go to a coding agent or be picked up by hand.

Architecture

• meeting-core (Rust crate) owns the Meeting lifecycle, storage (SQLite via rusqlite), question tracking, speaker recognition and echo handling. With no dependency on Tauri or the UI, it keeps the Tauri layer and React frontend thin.
• Swift sidecar (audio-capture) streams 16 kHz mono Float32 audio over stdout. The source is the echo-cancelled microphone (macOS voice processing), system audio through a Core Audio process tap, or a decoded audio/video file for imports. It never writes audio to disk.
• Transcription runs whisper.cpp through whisper-rs with Metal, with selectable models from base up to large-v3.
• Voice analysis uses sherpa-onnx from Rust: pyannote segmentation 3.0 finds speaker turns and a WeSpeaker ResNet34 model produces voice embeddings. Each Speaker keeps an embedding, never audio.
• Question extraction sends finalised transcript text to Claude roughly every 10 seconds, either through the locally installed Claude Code CLI or the Anthropic API with a key held in the macOS Keychain.

Key decisions

• Local transcription, cloud question extraction (ADR-0001). Cost drove this. Per-minute streaming speech-to-text is what makes a long meeting expensive; sending text to an LLM is cheap. I rejected local LLMs because they were weaker at Urdu and mixed Urdu/English speech, and at judging whether a question was actually answered. Transcription keeps working offline; extraction catches up when the connection returns.
• Tauri with a Rust core over native SwiftUI (ADR-0003, superseding ADR-0002). The first plan was SwiftUI with WhisperKit. I reversed it to keep other platforms open. The newer Apple audio APIs stay in a small Swift sidecar, since Rust’s Objective-C bindings for them are less proven.
• Seams around everything platform-specific. SpeechSource, VoiceAnalyser and the question engine are traits, so the sidecar, models or LLM can be swapped without touching the core. A scripted SpeechSource behind a Cargo feature lets the whole app run without audio.
• Speaker recognition stays on-device (ADR-0004). Voiceprints are biometric data about people who never consented to the app, so cloud diarization was out. For an unclear match the app suggests a known Person (“Maybe ?”), and it never relabels an earlier meeting without asking.
• Echo dropping in pure logic. Without headphones the microphone picks up remote speakers. A “You” segment that closely matches overlapping “Others” text (similarity 0.85, 1 s timing slack) gets dropped, so remote speech appears once.

Outcomes

• Working macOS app (Apple Silicon, macOS 14.2+) with live and imported meetings, speaker naming, question tracking, search, export and crash recovery.
• 288 integration tests over the core, including contract tests for both Claude adapters.