Speech Swift

AI speech models for Apple Silicon, powered by MLX Swift and CoreML.

📖 Read in: English · 中文 · 日本語 · 한국어 · Español · Deutsch · Français · हिन्दी · Português · Русский · العربية · Tiếng Việt · Türkçe · ไทย

On-device speech recognition, synthesis, and understanding for Mac and iOS. Runs locally on Apple Silicon — no cloud, no API keys, no data leaves your device.

📚 Full Documentation → · 🤗 HuggingFace Models · 📝 Blog · 💬 Discord

Local Speech AI on a MacBook — watch the 4-minute open-source library tour on YouTube

Use cases: Voice Agents · Transcription · Speech Generation

Built with Speech Swift

15 public repositories with verifiable Speech Swift package references.

Palmier Pro · Anarlog · ClawdHome · Jabber · Ora · VoxFlow · LokalBot · Voicey · HushType · DexDictate macOS · Watchtower · Wishper App · FriSpeak · Scribe · VoicePen

OpenAI-compatible apps: AnythingLLM (merged WAV support)

Capability groups: STT / ASR · Alignment · TTS · LLMs & translation · Speech-to-speech · Enhancement/restoration · Source separation · Music/audio generation · Wake word, VAD, diarization & speaker identity

STT / ASR

• Qwen3-ASR — Speech-to-text (automatic speech recognition, 52 languages, MLX + CoreML)
• WhisperASR — Whisper Large-v3 Turbo speech-to-text via native CoreML runtime (ANE, multilingual)
• Parakeet TDT — Speech-to-text via CoreML (Neural Engine, NVIDIA FastConformer + TDT decoder, 25 languages)
• Omnilingual ASR — Speech-to-text (Meta wav2vec2 + CTC, 1,672 languages across 32 scripts, CoreML 300M + MLX 300M/1B/3B/7B) — 0.28 RTF on iPhone 16 Pro
• Streaming Dictation — Real-time dictation with partials and end-of-utterance detection (Parakeet-EOU-120M) — 0.04 RTF on iPhone 16 Pro
• Nemotron Streaming (Multilingual) — Low-latency streaming ASR with native punctuation and capitalization (NVIDIA Nemotron-3.5-ASR-Streaming-0.6B, CoreML + MLX, 40 language-locales)
• Nemotron Streaming (English) — Low-latency streaming ASR with native punctuation and capitalization (NVIDIA Nemotron-Speech-Streaming-0.6B, CoreML, English-only, smaller and faster than the multilingual variant)

Alignment

• Qwen3-ForcedAligner — Word-level timestamp alignment (audio + text → timestamps)

TTS / Speech Generation

• Qwen3-TTS — Text-to-speech (highest quality, streaming, custom speakers, 10 languages)
• CosyVoice TTS — Streaming TTS with voice cloning, multi-speaker dialogue, emotion tags (9 languages)
• VoxCPM2 — 48 kHz studio-quality TTS with voice cloning + instruction-driven voice design (2B, MLX bf16/int8, 30 languages)
• IndexTTS2 — Native MLX voice cloning from a reference voice (IndexTeam IndexTTS-2, 1.5B-class fp16 bundle, speaker/emotion/pause controls)
• F5-TTS — Zero-shot voice cloning from a short reference clip + transcript (SWivid F5-TTS v1 Base, DiT flow matching + Vocos, MLX fp16, 24 kHz, English + Mandarin; non-commercial license)
• Higgs TTS 3 — Conversational TTS with zero-shot voice cloning and inline emotion/style/SFX/prosody tags (Boson Higgs TTS 3, Qwen3-4B backbone, MLX bf16, 24 kHz, 100+ languages; research/non-commercial license)
• Kokoro TTS — On-device TTS (82M, CoreML/Neural Engine, 54 voices, iOS-ready, 10 languages) — 0.08 RTF on iPhone 16 Pro
• VibeVoice TTS — Long-form / multi-speaker TTS (Microsoft VibeVoice Realtime-0.5B + 1.5B, MLX, up to 90-min podcast/audiobook synthesis, EN/ZH)
• Magpie TTS — Multilingual TTS (NVIDIA Magpie-TTS Multilingual 357M, MLX INT8 411 MB or CoreML INT8 342 MB, 9 languages, 5 baked speakers, streaming on MLX)
• Supertonic TTS — On-device flow-matching TTS (Supertone Supertonic-3 99M, CoreML/Neural Engine, 31 languages, 10 voices, G2P-free, 44.1 kHz) — 0.15 RTF on iPhone 16 Pro
• Chatterbox TTS — Multilingual TTS with zero-shot voice cloning (Resemble AI Chatterbox Multilingual, MLX fp16 ~1.3 GB, 23 runtime languages; Hebrew requires niqqud, MIT) plus Chatterbox Flash Core ML text-to-waveform from precomputed reference conditioning.
• OmniVoice TTS — Non-autoregressive diffusion TTS with zero-shot voice cloning (k2-fsa OmniVoice, Qwen3 backbone, MLX fp16 default / int8 available, 600+ languages, Apache-2.0)
• Indic-Mio — Hindi/Indic TTS with inline emotion markers and optional reference-voice cloning (MLX, 24 kHz)

LLMs & Translation

• Qwen3Chat — On-device LLM chat (Qwen3.5-0.8B MLX/CoreML plus dense Qwen3 4B and Gemma 4 E2B/E4B MLX backends, streaming tokens)
• FunctionGemma — On-device LLM for structured function / tool calls (Gemma 3 270M, CoreML 8-bit palettized, Neural Engine, ~252 tok/s on M5 Pro · 128 tok/s on iPhone 16 Pro)
• MADLAD-400 — Many-to-many translation across 400+ languages (3B, MLX INT4 + INT8, T5 v1.1, Apache 2.0)

Speech-to-Speech & Voice Agents

• Hibiki Zero-3B — Streaming speech-to-speech translation (FR/ES/PT/DE → EN, MLX INT4 + INT8, Kyutai Moshi/Mimi stack, CC-BY-4.0)
• PersonaPlex — Full-duplex speech-to-speech (7B, audio in → audio out, 18 voice presets)
• Audio2Face-3D — Speech-driven facial animation for avatars (NVIDIA Audio2Face-3D v2.3 Mark, 301 facial coefficients, MLX)

Enhancement, Separation & Audio Generation

• DeepFilterNet3 — Real-time noise suppression (2.1M params, 48 kHz). Long-form audio above the 60 s single-shot cap is auto-chunked with crossfade — see enhanceChunked(...)
• Source Separation — Music source separation via HTDemucs (Demucs v4) + Open-Unmix (UMX-HQ / UMX-L, 4 stems: vocals/drums/bass/other, 44.1 kHz stereo)
• MAGNeT — Text-to-music generation (Meta MAGNeT Small 300M / Medium 1.5B, MLX INT8, 30 s clips at 32 kHz mono, masked parallel decoding)
• Stable Audio 3 — Text-to-audio/music generation (Stable Audio 3 Medium, MLX INT8/INT4, 44.1 kHz stereo, variable length)
• FlashSR — Audio super-resolution (FlashSR ICASSP 2025, MLX, 48 kHz mono, 1-step distilled diffusion, INT4 363 MB / INT8 720 MB)

Turn Detection, Diarization & Speaker Identity

• Wake-word — On-device keyword spotting (KWS Zipformer 3M, CoreML, 26× real-time, configurable keyword list)
• VAD — Voice activity detection (Silero streaming, Pyannote offline, FireRedVAD 100+ languages)
• Speaker Diarization — Who spoke when (Pyannote pipeline, Sortformer end-to-end on Neural Engine) — now with an incremental streaming session (stable speaker IDs, updates every 480 ms)
• Speaker Embeddings — WeSpeaker ResNet34 (256-dim), ReDimNet2-B6 named identity (192-dim), CAM++ (192-dim)

Papers: Qwen3-ASR (Alibaba) · Qwen3-TTS (Alibaba) · Omnilingual ASR (Meta) · Parakeet TDT (NVIDIA) · CosyVoice 3 (Alibaba) · Kokoro (StyleTTS 2) · PersonaPlex (NVIDIA) · Mimi (Kyutai) · Hibiki (Kyutai) · Sortformer (NVIDIA)

On-device iPhone 16 Pro CoreML benchmarks (RTF, tokens/s, peak memory): docs/benchmarks/ios-coreml.md.

News

• 19 Apr 2026 — MLX vs CoreML on Apple Silicon — A Practical Guide to Picking the Right Backend
• 20 Mar 2026 — We Beat Whisper Large v3 with a 600M Model Running Entirely on Your Mac
• 26 Feb 2026 — Speaker Diarization and Voice Activity Detection on Apple Silicon — Native Swift with MLX
• 23 Feb 2026 — NVIDIA PersonaPlex 7B on Apple Silicon — Full-Duplex Speech-to-Speech in Native Swift with MLX
• 12 Feb 2026 — Qwen3-ASR Swift: On-Device ASR + TTS for Apple Silicon — Architecture and Benchmarks

Quick start

Add the package to your Package.swift:

.package(url: "https://github.com/soniqo/speech-swift", branch: "main")

Import only the modules you need — every model is its own SPM library, so you don't pay for what you don't use:

.product(name: "ParakeetStreamingASR", package: "speech-swift"),
.product(name: "SpeechUI",             package: "speech-swift"),  // optional SwiftUI views

Transcribe an audio buffer in 3 lines:

import ParakeetStreamingASR

let model = try await ParakeetStreamingASRModel.fromPretrained()
let text = try model.transcribeAudio(audioSamples, sampleRate: 16000)

Live streaming with partials:

for await partial in model.transcribeStream(audio: samples, sampleRate: 16000) {
    print(partial.isFinal ? "FINAL: \(partial.text)" : "... \(partial.text)")
}

SwiftUI dictation view in ~10 lines:

import SwiftUI
import ParakeetStreamingASR
import SpeechUI

@MainActor
struct DictateView: View {
    @State private var store = TranscriptionStore()

    var body: some View {
        TranscriptionView(finals: store.finalLines, currentPartial: store.currentPartial)
            .task {
                let model = try? await ParakeetStreamingASRModel.fromPretrained()
                guard let model else { return }
                for await p in model.transcribeStream(audio: samples, sampleRate: 16000) {
                    store.apply(text: p.text, isFinal: p.isFinal)
                }
            }
    }
}

SpeechUI ships only TranscriptionView (finals + partials) and TranscriptionStore (streaming ASR adapter). Use AVFoundation for audio visualization and playback.

Available SPM products: Qwen3ASR, WhisperASR, Qwen3TTS, Qwen3TTSCoreML, ParakeetASR, ParakeetStreamingASR, NemotronStreamingASR, OmnilingualASR, KokoroTTS, SupertonicTTS, VibeVoiceTTS, CosyVoiceTTS, VoxCPM2TTS, IndexTTS2TTS, F5TTS, HiggsTTS, ChatterboxTTS, OmniVoiceTTS, IndicMioTTS, FishAudioTTS, MagpieTTS, MagpieTTSCoreML, MAGNeTMusicGen, StableAudio3MusicGen, FlashSR, PersonaPlex, Audio2Face3D, HibikiTranslate, MADLADTranslation, SpeechVAD, SpeechWakeWord, SpeechEnhancement, SpeechRestoration, SourceSeparation, Qwen3Chat, FunctionGemma, SpeechCore, SpeechUI, AudioCommon.

Models

Compact view below. Full model catalogue with sizes, quantisations, download URLs, and memory tables → soniqo.audio/architecture.

Model	Task	Backends	Sizes	Languages
Qwen3-ASR	Speech → Text	MLX, CoreML (hybrid)	0.6B, 1.7B	52
WhisperASR	Speech → Text	CoreML (ANE)	Large-v3 Turbo	Multi
Parakeet TDT	Speech → Text	CoreML (ANE)	0.6B	25 European
Parakeet EOU	Speech → Text (streaming)	CoreML (ANE)	120M	25 European
Nemotron Streaming (Multilingual)	Speech → Text (streaming, punctuated)	CoreML (ANE), MLX	0.6B	40
Nemotron Streaming (English)	Speech → Text (streaming, punctuated)	CoreML (ANE)	0.6B	EN
Omnilingual ASR	Speech → Text	CoreML (ANE), MLX	300M / 1B / 3B / 7B	1,672
Qwen3-ForcedAligner	Audio + Text → Timestamps	MLX, CoreML	0.6B	Multi
Qwen3-TTS	Text → Speech	MLX, CoreML	0.6B, 1.7B	10
CosyVoice3	Text → Speech	MLX	0.5B	9
VoxCPM2	Text → Speech (48 kHz, voice design + cloning)	MLX	2B (bf16/int8)	30
IndexTTS2	Text → Speech (zero-shot voice cloning)	MLX	1.5B-class (fp16)	EN/ZH
F5-TTS	Text → Speech (zero-shot voice cloning)	MLX	336M (fp16)	EN/ZH
Higgs TTS 3	Text → Speech (conversational, zero-shot voice cloning)	MLX	4B (bf16)	100+
Kokoro-82M	Text → Speech	CoreML (ANE)	82M	10
Supertonic-3	Text → Speech (44.1 kHz, flow-matching, G2P-free)	CoreML (ANE)	99M	31
VibeVoice Realtime-0.5B	Text → Speech (long-form, multi-speaker)	MLX	0.5B	EN/ZH
VibeVoice 1.5B	Text → Speech (up to 90-min podcast)	MLX	1.5B	EN/ZH
Magpie-TTS Multilingual	Text → Speech (5 baked speakers, streaming)	MLX / CoreML	357M (MLX INT8, CoreML INT8)	9 (CoreML excludes JA)
Chatterbox Multilingual	Text → Speech (zero-shot cloning)	MLX	0.8B (fp16)	23 (HE requires niqqud)
Chatterbox Flash	Text → Speech (voice cloning with external reference conditioning)	CoreML + MLX conditioning bridge	0.8B (fp16 Core ML)	EN
OmniVoice	Text → Speech (NAR diffusion, zero-shot cloning)	MLX	0.8B (fp16 default / int8)	600+
Indic-Mio	Text → Speech (Hindi/Indic, emotion tags, voice cloning)	MLX	fp16	Hindi / Indic
Fish Audio S2 Pro	Text → Speech (zero-shot cloning, explicit style markers)	MLX	0.5B-class (fp16)	Multilingual
Qwen3.5 Chat	Text → Text (LLM)	MLX, CoreML	0.8B	Multi
Qwen3 Dense Chat	Text → Text (LLM)	MLX	4B	Multi
Gemma 4 Chat	Text → Text (LLM)	MLX	E2B / E4B (4-bit)	Multi
FunctionGemma	Text → Tool calls (LLM)	CoreML	270M	EN-tuned
MADLAD-400	Text → Text (Translation)	MLX	3B	400+
Hibiki Zero-3B	Speech → Speech (Translation)	MLX	3B	FR/ES/PT/DE → EN
PersonaPlex	Speech → Speech	MLX	7B	EN
Audio2Face-3D	Speech → Facial animation	MLX	v2.3 Mark	Agnostic
Silero VAD	Voice Activity Detection	MLX, CoreML	309K	Agnostic
KWS Zipformer	Audio → Wake word	CoreML (ANE)	3M	EN/custom keywords
Pyannote	VAD + Diarization	MLX	1.5M	Agnostic
Pyannote Community-1	Diarization + speaker embeddings	CoreML (ANE) + Swift VBx	8.35M	Agnostic
Sortformer	Diarization (E2E), incremental streaming	CoreML (ANE)	117M	Agnostic
Ultra-Sortformer 8spk	Diarization (E2E, up to 8 speakers, experimental)	CoreML (ANE)	117M	Agnostic
DeepFilterNet3	Speech Enhancement	CoreML	2.1M	Agnostic
Sidon	Speech Restoration (denoise + dereverb, 48 kHz)	CoreML	w2v-BERT 2.0 + DAC (fp16/int8)	Agnostic
HTDemucs (Demucs v4)	Source Separation	MLX	168M	Agnostic
Open-Unmix	Source Separation	MLX	8.6M	Agnostic
MAGNeT	Text → Music (30s @ 32 kHz)	MLX	300M / 1.5B (int4/int8)	EN prompts
Stable Audio 3	Text → Music/audio (44.1 kHz stereo)	MLX	Medium 1.4B (int4/int8)	EN prompts
FlashSR	Audio super-resolution (48 kHz)	MLX	363 MB / 720 MB (int4/int8)	Agnostic
WeSpeaker	Speaker Embedding	MLX, CoreML	6.6M	Agnostic
ReDimNet2-B6	Named Voice Identity	CoreML	12.3M	Agnostic

Installation

Homebrew

Requires native ARM Homebrew (/opt/homebrew). Rosetta/x86_64 Homebrew is not supported.

brew install speech

Then:

speech transcribe recording.wav
speech speak "Hello world"
speech translate "Hello, how are you?" --to es
speech respond --input question.wav --transcript
speech-server --port 8080            # local HTTP / WebSocket server (OpenAI-compatible /v1/realtime + /v1/audio/transcriptions)

Full CLI reference →

Swift Package Manager

dependencies: [
    .package(url: "https://github.com/soniqo/speech-swift", branch: "main")
]

Import only what you need — every model is its own SPM target:

import Qwen3ASR             // Speech recognition (MLX)
import WhisperASR           // Whisper Large-v3 Turbo (CoreML)
import ParakeetASR          // Speech recognition (CoreML, batch)
import ParakeetStreamingASR // Streaming dictation with partials + EOU
import NemotronStreamingASR // Multilingual streaming ASR with native punctuation (0.6B, 40 langs)
import OmnilingualASR       // 1,672 languages (CoreML + MLX)
import Qwen3TTS             // Text-to-speech
import CosyVoiceTTS         // Text-to-speech with voice cloning
import VoxCPM2TTS           // 48 kHz TTS with voice cloning + voice design (2B)
import IndexTTS2TTS         // Native MLX voice cloning from reference audio
import F5TTS                // Zero-shot voice cloning (DiT flow matching + Vocos)
import HiggsTTS             // Conversational TTS + cloning (Qwen3 backbone, control tags)
import KokoroTTS            // Text-to-speech (iOS-ready)
import VibeVoiceTTS         // Long-form / multi-speaker TTS (EN/ZH)
import MagpieTTS            // Multilingual TTS (NVIDIA Magpie 357M, MLX, 9 langs)
import MagpieTTSCoreML      // Magpie CoreML backend (hybrid CoreML + MLX, 8 langs)
import FishAudioTTS         // Experimental Fish Audio S2 Pro runtime with voice cloning
import IndicMioTTS          // Hindi/Indic TTS with emotion markers
import Qwen3Chat            // On-device LLM chat
import FunctionGemma    // On-device tool-call LLM
import MADLADTranslation    // Many-to-many translation across 400+ languages
import HibikiTranslate      // Streaming speech-to-speech translation (FR/ES/PT/DE → EN)
import PersonaPlex          // Full-duplex speech-to-speech
import SpeechVAD            // VAD + speaker diarization + embeddings
import SpeechWakeWord       // Wake-word / keyword spotting
import SpeechEnhancement    // Noise suppression
import SpeechRestoration    // Speech restoration — denoise + dereverb (Sidon, CoreML, 48 kHz)
import SourceSeparation     // Music source separation (Open-Unmix, 4 stems)
import StableAudio3MusicGen // Text-to-audio/music generation (Stable Audio 3)
import SpeechUI             // SwiftUI components for streaming transcripts
import AudioCommon          // Shared protocols and utilities

Requirements

• Swift 6+, Xcode 16+ (with Metal Toolchain)
• macOS 15+ (Sequoia) or iOS 18+, Apple Silicon (M1/M2/M3/M4)

The macOS 15 / iOS 18 minimum comes from MLState — Apple's persistent ANE state API used by the CoreML pipelines (Qwen3-ASR, Qwen3-Chat, Qwen3-TTS) to keep KV caches resident on the Neural Engine across token steps.

Build from source

git clone https://github.com/soniqo/speech-swift
cd speech-swift
make build

make build compiles the Swift package and the MLX Metal shader library. The Metal library is required for GPU inference — without it you'll see Failed to load the default metallib at runtime. make debug for debug builds, make test for the test suite.

Full build and install guide →

Demo apps

• DictateDemo (docs) — macOS menu-bar streaming dictation with live partials, VAD-driven end-of-utterance detection, and one-click copy. Runs as a background agent (Parakeet-EOU-120M + Silero VAD).
• iOSEchoDemo — iOS echo demo (Parakeet ASR + Kokoro TTS). Device and simulator.
• PersonaPlexDemo — Conversational voice assistant with mic input, VAD, and multi-turn context. macOS. RTF ~0.94 on M2 Max (faster than real-time).
• SpeechDemo — Dictation and TTS synthesis in a tabbed interface. macOS.

Each demo's README has build instructions.

Code examples

The snippets below show the minimal path for each domain. Every section links to a full guide on soniqo.audio with configuration options, multiple backends, streaming patterns, and CLI recipes.

Speech-to-Text — full guide →

import Qwen3ASR

let model = try await Qwen3ASRModel.fromPretrained()
let text = model.transcribe(audio: audioSamples, sampleRate: 16000)

Alternative backends: WhisperASR (Whisper Large-v3 Turbo, native CoreML), Parakeet TDT (CoreML, 32× realtime), Omnilingual ASR (1,672 languages, CoreML or MLX), Streaming dictation (live partials).

Forced Alignment — full guide →

import Qwen3ASR

let aligner = try await Qwen3ForcedAligner.fromPretrained()
let aligned = aligner.align(
    audio: audioSamples,
    text: "Can you guarantee that the replacement part will be shipped tomorrow?",
    sampleRate: 24000
)
for word in aligned {
    print("[\(word.startTime)s - \(word.endTime)s] \(word.text)")
}

Text-to-Speech — full guide →

import Qwen3TTS
import AudioCommon

let model = try await Qwen3TTSModel.fromPretrained()
let audio = model.synthesize(text: "Hello world", language: "english")
try WAVWriter.write(samples: audio, sampleRate: 24000, to: outputURL)

Alternative TTS engines: CosyVoice3 (streaming + voice cloning + emotion tags), Kokoro-82M (iOS-ready, 54 voices), VibeVoice (long-form podcast / multi-speaker, EN/ZH), Fish Audio S2 Pro (experimental zero-shot cloning + bracket style markers), Voice cloning.

Speech-to-Speech — full guide →

import PersonaPlex

let model = try await PersonaPlexModel.fromPretrained()
let responseAudio = model.respond(userAudio: userSamples)
// 24 kHz mono Float32 output ready for playback

LLM Chat — full guide →

import Qwen3Chat
import FunctionGemma

let chat = try await Qwen35MLXChat.fromPretrained()
chat.chat(messages: [(.user, "Explain MLX in one sentence")]) { token, isFinal in
    print(token, terminator: "")
}

Translation — full guide →

import MADLADTranslation

let translator = try await MADLADTranslator.fromPretrained()
let es = try translator.translate("Hello, how are you?", to: "es")
// → "Hola, ¿cómo estás?"

Speech Translation — full guide →

import HibikiTranslate
import AudioCommon

let model = try await HibikiTranslateModel.fromPretrained()
let pcm = try AudioFileLoader.load(url: input, targetSampleRate: 24000)
let (englishAudio, textTokens) = model.translate(
    sourceAudio: pcm, sourceLanguage: .fr
)
// Hibiki Zero-3B — FR/ES/PT/DE → EN, on-device, streaming Mimi codec

Voice Activity Detection — full guide →

import SpeechVAD

let vad = try await SileroVADModel.fromPretrained()
let segments = vad.detectSpeech(audio: samples, sampleRate: 16000)
for s in segments { print("\(s.startTime)s → \(s.endTime)s") }

Speaker Diarization — full guide →

import SpeechVAD

let diarizer = try await DiarizationPipeline.fromPretrained()
let segments = diarizer.diarize(audio: samples, sampleRate: 16000)
for s in segments { print("Speaker \(s.speakerId): \(s.startTime)s - \(s.endTime)s") }

Speech Enhancement — full guide →

import SpeechEnhancement

let denoiser = try await DeepFilterNet3Model.fromPretrained()
let clean = try denoiser.enhance(audio: noisySamples, sampleRate: 48000)

Speech Restoration — full guide →

Joint denoise and dereverb with Sidon (w2v-BERT 2.0 predictor + DAC vocoder, Core ML). Unlike a generic noise suppressor, Sidon is trained to preserve speaker identity, so it is well suited to cleaning a noisy or reverberant voice-cloning reference before TTS. Input is 16 kHz; output is 48 kHz mono.

import SpeechRestoration

let restorer = try await SpeechRestorer.fromPretrained()          // .fp16 (default) or .int8
let clean = try restorer.restore(audio: noisySamples, sampleRate: 16000)  // → 48 kHz

From the CLI:

speech restore noisy.wav -o clean.wav            # denoise + dereverb, 48 kHz output
speech restore noisy.wav --variant int8          # smaller, lower peak RAM

# Clean a voice-cloning reference before TTS (opt-in; preserves speaker identity):
speech speak "Hello" --engine voxcpm2 --voice-sample ref.wav --clean-reference

Voice Pipeline (ASR → LLM → TTS) — full guide →

import SpeechCore

let pipeline = VoicePipeline(
    stt: parakeetASR,
    tts: qwen3TTS,
    vad: sileroVAD,
    config: .init(mode: .voicePipeline),
    onEvent: { event in print(event) }
)
pipeline.start()
pipeline.pushAudio(micSamples)

VoicePipeline is the real-time voice-agent state machine (powered by speech-core) with VAD-driven turn detection, interruption handling, and eager STT. It connects any SpeechRecognitionModel + SpeechGenerationModel + StreamingVADProvider.

HTTP API server

speech-server --port 8080

Exposes every model via HTTP REST + WebSocket endpoints, including OpenAI-compatible APIs: a Realtime WebSocket at /v1/realtime and a transcription REST endpoint at /v1/audio/transcriptions. See Sources/AudioServer/.

Architecture

speech-swift is split into one SPM target per model so consumers only pay for what they import. Shared infrastructure lives in AudioCommon (protocols, audio I/O, HuggingFace downloader, SentencePieceModel) and MLXCommon (weight loading, QuantizedLinear helpers, SDPA multi-head attention helper).

Full architecture diagram with backends, memory tables, and module map → soniqo.audio/architecture · API reference → soniqo.audio/api · Benchmarks → soniqo.audio/benchmarks

Local docs (repo):

• Models: Qwen3-ASR · WhisperASR · Qwen3-TTS · CosyVoice · Kokoro · VoxCPM2 · IndexTTS2 · F5-TTS · Higgs TTS 3 · VibeVoice · Supertonic · Chatterbox · Indic-Mio · Fish Audio S2 Pro · Magpie TTS · Parakeet TDT · Parakeet Streaming · Nemotron Streaming · Omnilingual ASR · PersonaPlex · Hibiki · MADLAD-400 · FunctionGemma · Qwen3.5 Chat · Gemma 4 Chat · Qwen3 Dense Chat · FireRedVAD · KWS Zipformer · Sidon · Source Separation · HTDemucs · MAGNeT · Stable Audio 3 · FlashSR · Audio2Face-3D
• Inference: Qwen3-ASR · WhisperASR · Parakeet TDT · Parakeet Streaming · Nemotron Streaming · Omnilingual ASR · TTS · VoxCPM2 · IndexTTS2 · F5-TTS · Higgs TTS 3 · VibeVoice · Fish Audio S2 Pro · Magpie TTS · Hibiki · MADLAD-400 · MAGNeT · Stable Audio 3 · FlashSR · Forced Aligner · Silero VAD · FireRedVAD · Wake-word · Speaker Diarization · Speech Enhancement · Sidon · Cache/offline
• Reference: Shared Protocols

Cache configuration

Model weights download from HuggingFace on first use and cache to ~/Library/Caches/qwen3-speech/. Override with QWEN3_CACHE_DIR (CLI) or cacheDir: (Swift API). All fromPretrained() entry points also accept offlineMode: true to skip network when weights are already cached.

Users in mainland China (or anywhere huggingface.co is slow/blocked) can fetch from a mirror by setting HF_ENDPOINT, e.g. export HF_ENDPOINT=https://hf-mirror.com.

See docs/inference/cache-and-offline.md for full details including sandboxed iOS container paths.

MLX Metal library

If you see Failed to load the default metallib at runtime, the Metal shader library is missing. Run make build or ./scripts/build_mlx_metallib.sh release after a manual swift build. If the Metal Toolchain is missing, install it first:

xcodebuild -downloadComponent MetalToolchain

Testing

make test                            # full suite (unit + E2E with model downloads)
swift test --skip E2E                # unit only (CI-safe, no downloads)
swift test --filter Qwen3ASRTests    # specific module

E2E test classes use the E2E prefix so CI can filter them out with --skip E2E. See CLAUDE.md for the full testing convention.

Contributing

PRs welcome — bug fixes, new model integrations, documentation. Fork, create a feature branch, make build && make test, open a PR against main.

License

Apache 2.0