voice-isolator-cpu

A causal, streaming primary-speaker isolator for voice agents, on the CPU: it keeps the caller and removes other voices and noise before the VAD, the turn detector and the recogniser hear the call. 16 kHz in, 16 kHz out, 10 ms frames, 10 ms of delay. onnxruntime and numpy only: no torch, no GPU.

Paper: The Voice on the Line (arXiv:2610.35272).

Speech-to-text word error rate after each system, the mean of the 9 test sets in Results, every system live except VoxLock's benchmark protocol, which takes its sample from the first 15 s of the call. Live VoxLock is ahead on two of them: Krisp VI benchmark (265 recordings); Call-centre calls, caller alone.

A test meeting streamed through a real LiveKit agent with this model on one CPU thread: first the call as LiveKit hands it to the agent, then what the agent's VAD and speech-to-text hear after the isolator. Bars: the call's level, coloured by who is talking (the corpus annotations). Words: what speech-to-text writes, the caller's in blue and other people's in orange. More below.

It is our GPU release, Scicom-intl/voice-isolator (M + causal WavLM, 555M parameters), distilled into a 22M-parameter transformer of the same design (8 layers of width 384 that see the last 10 s, a frame LSTM, band LSTMs).

Use in a LiveKit agent

How to use it in a LiveKit agent: STT-API's voice_isolator plugin (this model inside the agent as LocalIsolator, the GPU release on a server as RemoteIsolator, prewarming, and a streaming benchmark through real LiveKit agents). The same plugin is in this repository's voice_isolator/ folder.

from livekit.agents.voice import room_io
from stt_api.livekit_plugin.voice_isolator import LocalIsolator, VoiceIsolator   # or this repository's voice_isolator/

await session.start(
    agent=MyAgent(), room=ctx.room,
    room_options=room_io.RoomOptions(audio_input=room_io.AudioInputOptions(
        sample_rate=16000,
        noise_cancellation=lambda params: VoiceIsolator(backend=LocalIsolator("Scicom-intl/voice-isolator-cpu")))),
)

An agent that speaks first (it greets the caller) can pass LocalIsolator(..., voicebot=True): see the voicebot mode.

Without LiveKit: voice_isolator.OnnxStream(model_dir).push(chunk) takes 16 kHz float32 audio in any amounts and returns the isolated audio of every whole 10 ms hop. example.py isolates a file.

Examples

AMI test meeting ES2004c, 70 s: the caller stops for 38 s while the others talk, then comes back

Speech-to-text (parakeet-tdt-0.6b-v3) writes 120 words from other people without the isolator and 0 with it, and gets 38 of the caller's 38 right without it and 38 with it. before.wav · after.wav

AMI test meeting EN2002c, 60 s, a hard one: the others are faint here, and the isolator can cost the caller words

Speech-to-text (parakeet-tdt-0.6b-v3) writes 6 words from other people without the isolator and 1 with it, and gets 36 of the caller's 49 right without it and 36 with it. before.wav · after.wav

AMI test meeting TS3003b, 60 s: 33 s of the others talking on their own

Speech-to-text (parakeet-tdt-0.6b-v3) writes 64 words from other people without the isolator and 0 with it, and gets 46 of the caller's 48 right without it and 45 with it. before.wav · after.wav

AliMeeting R1019_M4035 (Mandarin), 13 s: another voice after the caller

Speech-to-text (whisper-large-v3) writes 0 characters from other people without the isolator and 0 with it, and gets 42 of the caller's 47 right without it and 44 with it. before.wav · after.wav

Hard: AMI test meeting EN2002d, 60 s: the others talk over the caller for 24 s, almost as loud as the caller in the caller's own headset

Speech-to-text (parakeet-tdt-0.6b-v3) writes 24 words from other people without the isolator and 0 with it, and gets 35 of the caller's 67 right without it and 58 with it. before.wav · after.wav

Hard: AMI test meeting EN2002a, 14 s: in the caller's own headset the others are louder than the caller, and the isolator lets them through

Speech-to-text (parakeet-tdt-0.6b-v3) writes 0 words from other people without the isolator and 0 with it, and gets 21 of the caller's 24 right without it and 21 with it. before.wav · after.wav

Hard: AliMeeting R0008_M0058 (Mandarin), 17 s: another voice as loud as the caller talks over 6 s of the caller's speech

Speech-to-text (whisper-large-v3) writes 0 characters from other people without the isolator and 0 with it, and gets 45 of the caller's 59 right without it and 46 with it. before.wav · after.wav

Hard: AliMeeting R0004_M0018 (Mandarin), 9 s: the other voice is 12 dB louder than the caller

Speech-to-text (whisper-large-v3) writes 0 characters from other people without the isolator and 0 with it, and gets 22 of the caller's 25 right without it and 23 with it. before.wav · after.wav

Hard: AMI test meeting ES2004b, 12 s: a caller nearly alone, talking from the first second, before the live enrollment exists

Speech-to-text (parakeet-tdt-0.6b-v3) writes 0 words from other people without the isolator and 0 with it, and gets 34 of the caller's 35 right without it and 34 with it. before.wav · after.wav

All recorded inside a LiveKit agent: the caller streams the recording over WebRTC in real time, and this model runs as the agent's noise_cancellation (one CPU thread, 50 ms frames). Chosen by rules on the input, not the result: the longest stretches of others talking alone; the most words from others before processing; the most speech overlapped by a loud voice, and the loudest other voices (one window per meeting not already shown); a caller nearly alone from the first second. From AMI (CC BY 4.0) and AliMeeting (CC BY-SA 4.0) test meetings.

Results

Word error rate (%) after processing; Parakeet TDT 0.6B v2 (Krisp, calls) or v3 (meetings), Whisper large-v3 for AliMeeting. Live: the enrollment comes from the call itself (from 3 s in, the loudest 3 s heard so far).

test set	no processing	VoxLock live (CPU)	this model, live (CPU)	our GPU release, live
Krisp VI benchmark (265 recordings)	23.19	5.63	5.66	5.74
AMI headset, VoxLock's windows	33.00	11.71	11.23	10.19
AMI test meetings	42.59	12.07	10.29	10.18
AMI test meetings, 60 s windows	121.79	23.66	15.55	16.13
NOTSOFAR	34.70	8.84	7.58	7.67
AliMeeting (character error rate)	29.40	16.28	15.82	15.69
Call-centre calls, caller alone	0.04	2.92	5.32	5.58
Call-centre calls + another voice	69.79	56.65	49.90	47.52
ai-coustics test calls	66.36	24.55	18.03	14.26

Chosen on development sets only (AMI and NOTSOFAR development meetings, 60 s AMI windows, clean development calls): their mean word error rate is 10.66% for this model, 13.51% for VoxLock live and 10.49% for the GPU release it was distilled from.

The agent's VAD

A noise filter leaves other voices in, and LiveKit's VAD takes them for the caller: it starts turns for them, holds the caller's turn open while they talk, and the agent answers late or not at all. Behind the isolator it hears far less of them. LiveKit's Silero VAD with LiveKit's own rule on 120 one-minute test calls with another voice and 120 with the caller alone, against the same VAD on the caller's own channel:

	speech heard in the caller's pauses	false starts / min	turn ends held	turn end p90 (s)	caller's speech missed	caller's speech missed, caller alone
no processing	47.9%	3.33	28%	2.43	0.1%	0.0%
GTCRN (noise only)	48.3%	3.58	26%	2.40	0.3%	0.6%
VoxLock live	27.5%	2.28	24%	1.51	11.0%	2.6%
this model, live	26.2%	2.27	20%	1.34	6.7%	4.9%
our GPU release, live	22.8%	1.94	18%	1.00	8.4%	5.2%

The same in a running LiveKit agent: 32 of the test calls with another voice streamed at once into rooms of a LiveKit server, each answered by an AgentSession with LiveKit's Silero VAD, the filter on its audio input in LiveKit's 50 ms frames, against the same rooms streaming each caller's own channel with no filter:

filter	speech heard in the caller's pauses	false starts / min	turn ends held	turn end p90 (s)	caller's speech missed
no filter	47.5%	2.84	23%	2.04	0.3%
GTCRN (noise only)	46.8%	2.88	22%	2.03	0.7%
this model	27.7%	2.12	18%	1.03	3.0%
this model, 8-bit	28.2%	2.25	16%	1.04	2.9%
our GPU release, through its server	24.9%	1.75	18%	1.06	6.9%

Interruptions

An agent that talks can be stopped in mid-sentence by noise: LiveKit pauses the agent's reply once its VAD has heard half a second of speech, and that half second includes the VAD's own hangover, so about a tenth of a second of noise is enough. Counted in LiveKit agents (Silero VAD at a deployed agent's settings) holding whole conversations with simulated phone callers: LibriTTS-R speakers who take eight turns and cut into about a third of the replies, over babble, street and transport, home sounds, music or another voice at 0, 10 or 20 dB under the caller, or a clean line, all through an 8 kHz G.711 telephone line. A pause is false when the caller had not talked in the 2 s before it. Every system ran on the audio that the agents of 17 live test sessions received (816 calls), scored by a replay of LiveKit's VAD and its interruption rule:

This model (the figure's CPU student), in 8 bits inside the agent at 18% of one core per call: 1.36 false interruptions a minute of agent speech, against 2.77 with no filter and 5.33 behind GTCRN (a voice cleaned of its noise sounds more like speech); the VAD missed 0.3% of the caller's turns behind it. VoxLock live made 1.18, but the VAD missed 6.8% of the turns behind it, and it took 220% of a core: three threads at full precision, which it needs to keep up with 50 ms frames. The GPU systems go lower: our GPU release 1.05, NVIDIA's Real-time RE-USE 0.95 (its licence allows research use only) and the release fine-tuned further for agents 0.86 (not released).

Agents that speak first: the voicebot mode

An agent that greets the caller is interrupted most often during its greeting. The bench's callers wait for it, so the agent's speech before the caller's first word is 11% of its speech, but it holds 48% of the false interruptions behind this model (78% behind RE-USE, 22% with no filter): at the start of a call the model lets any voice through, and at 3 s the live enrollment takes the loudest 3 s heard so far, which during a greeting is the TV, the music or the other person. The voicebot mode is for agents that speak first. Until the caller is enrolled the model is told to keep a decoy, a voice that is in no call (4 s of LibriTTS-R, in voice_isolator/resources/voicebot_decoy.npy), so it keeps no voice rather than every voice, and the first enrollment waits until the loudest 3 s heard is 6 dB over the call's first 3 s, or until the call is 10 s old:

backend = LocalIsolator("Scicom-intl/voice-isolator-cpu", voicebot=True)

This model, in 8 bits through the plugin, makes 1.13 false interruptions a minute of agent speech in the voicebot mode instead of 1.36 (-0.23, with a 90% interval from -0.34 to -0.14); the GPU release goes to 0.78 in the mode, below RE-USE's 0.95. All of it is in the greeting (5.77 to 3.61 a minute of greeting; after the caller's first word 0.82, against 0.80 by default), and the VAD missed 0.3% of the caller's turns (by default 0.3%); Whisper's errors on the caller's turns were 11.7% (by default 11.8%). The cost falls on a caller who talks before the enrollment: one who starts in the first second loses 25% of that first turn on a clean line and 31% over background noise, against 0-1% by default, and one who cuts into the greeting is not heard until their voice is 6 dB over the call's start. So it is opt-in, for agents that speak first (more).

CPU

Share of one CPU core per live call, one thread, every system through its own streaming entry point on the same calls, timed in one run (lower is better):

frames	this model	this model, int8 (quantized=True)	GTCRN (noise only)	VoxLock, its default int8	VoxLock, float32
10 ms	54%	34%	5%	227%	264%
20 ms	33%	23%	5%	132%	158%
50 ms (LiveKit's default)	17%	12%	5%	65%	88%

GTCRN, the noise suppressor STT-API ships, is the cheapest, and removes only noise: another voice is speech, and it keeps it (see the VAD tables above). Each model runs once per frame, so larger frames cost less.

The 8-bit model sets its activation scales at every step, so its output depends on how the call is cut into frames (the same call in 20 ms and in 50 ms frames agrees to 40 dB; the float model to rounding); on the development sets it makes 10.52% word errors against 10.66% in float.

In a LiveKit agent carrying 32 calls at once on one machine (STT-API's streaming benchmark: LiveKit server, Agents worker, Silero VAD, callers in real time, 50 ms frames), the isolator took 15 ms of a frame at the median and 25 ms at the 99th percentile in float, 9 and 17 ms in int8: wall clock on the agent's audio loop, with every call's agent and caller sharing the machine, so 32% of a core per call rather than the figure above. Its output stood 10 ms behind the call.

Citation

@misc{zolkepli2026voice,
  title         = {The Voice on the Line},
  author        = {Zolkepli, Husein},
  year          = {2026},
  eprint        = {2610.35272},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD},
  url           = {https://arxiv.org/abs/2610.35272}
}