NVIDIA has released Nemotron 3 Diarization, a 99.2-million-parameter open-weight speech model designed to track up to eight speakers in real time across live and recorded conversations.
The model was released on September 23, 2026, through NVIDIA’s Nemotron 3 Diarization model repository on Hugging Face. NVIDIA describes it as a roughly 100-million-parameter model, while the published model contains 99.2 million parameters.
Nemotron 3 Diarization is designed for speaker diarization, the process of determining who spoke and when during an audio recording. It does not identify people by name.
Instead, the model produces anonymous speaker channels. An application can assign the first detected speaker to one channel, the next distinct speaker to another, and continue the process for up to eight channels.
The model supports both streaming and offline inference. NVIDIA designed the streaming system to maintain speaker assignments as additional audio arrives rather than treating every incoming segment as an independent recording.
Nemotron 3 accepts 16 kHz mono audio in WAV, FLAC, OPUS and MP3 formats. Its output is a speaker-activity probability tensor with a shape of [T, 8], where the eight output channels correspond to the model’s maximum supported speaker streams.
The default temporal resolution is 10 milliseconds. NVIDIA says the model technically supports an 80-millisecond minimum buffer, but recommends a 0.32-second buffer as the minimum practical streaming configuration.
The published operating points include 0.32 seconds, 0.64 seconds and 1.04 seconds for streaming, as well as a 30.4-second offline-style configuration.
Those figures describe input-buffer latency rather than complete end-to-end response time. NVIDIA says the measurements exclude model computation, networking, automatic speech recognition and other application processing.
The model’s eight speaker channels represent an expansion over NVIDIA’s previous Streaming Sortformer model, which was designed for four speakers. The new system is intended to handle larger group conversations without increasing the number of separate diarization systems required to cover the conversation.
NVIDIA’s architecture is based on a 31-layer Transformer encoder with Rotary Positional Embeddings. Audio is converted into 10-millisecond Mel-spectrogram frames, stacked by a factor of eight, and processed as 80-millisecond encoder frames before the system produces 10-millisecond speaker activity outputs.
The streaming design includes an Arrival-Order Speaker Cache, or AOSC, together with a first-in-first-out context queue. The cache retains information about speakers encountered earlier in a conversation, while the context queue provides recent frame history to the current inference step.
NVIDIA’s approach assigns speaker channels according to arrival order. The first distinct voice encountered receives the first channel, followed by subsequent voices. This avoids having to repeatedly solve the speaker-label permutation problem when processing separate streaming chunks.
The model also uses right context, allowing the system to look slightly ahead in incoming audio before making an output decision.
Nemotron 3 is a diarization model, not an automatic speech recognition model. To produce a transcript showing which person said each sentence, developers must combine the diarization output with an ASR system.
NVIDIA’s documentation gives an example of combining diarization with offline transcription by using word timestamps and a midpoint-based method to assign words to speakers. NVIDIA notes that this is an illustrative alignment approach and does not completely solve cases where multiple people speak at the same time.
The release is aimed at applications including meetings, telephone and customer-service calls, podcasts and speech-recognition systems that require speaker-attributed transcripts.
The speaker-activity signal can also be used for voice activity detection, speaker-specific endpointing, interruption detection and turn-taking in voice-agent systems.
NVIDIA says Nemotron 3 ranked first in the initial VoiceArena Diarization-Bench evaluation, recording a 14.72 percent Diarization Error Rate. The next-ranked system recorded 19.3 percent.
NVIDIA calculates that result as approximately a 24 percent relative reduction in error compared with the next-ranked system.
The initial VoiceArena evaluation covered 139 English-language conversations representing about 22 hours of audio. The comparison included 12 systems and 17 system configurations, with overlapping speech included in the evaluation.
The benchmark used system-generated speech activity detection and a zero-second boundary collar. NVIDIA says the VoiceArena leaderboard may change as the Version 1 evaluation and paired statistical analysis are completed.
Diarization Error Rate combines missed speech, false-alarm time and speaker-confusion errors. The measure does not mean that a 14.72 percent DER is equivalent to a model correctly identifying 85.28 percent of people.
NVIDIA also reports results from a broader internal evaluation covering 901 condition-specific recordings. The evaluation included telephone conversations, meetings, near-field and far-field microphone conditions and multi-microphone environments, with overlapping speech included.
At the 1.04-second latency configuration, Nemotron 3 recorded a 13.18 percent DER on DIHARD III, compared with 19.60 percent for the previous Streaming Sortformer.
On CALLHOME-Part2, Nemotron 3 recorded 10.29 percent compared with 11.31 percent for the earlier model.
On AliMeeting Near, the new model recorded 6.59 percent compared with 12.47 percent. On AliMeeting Far, it recorded 10.80 percent compared with 15.58 percent.
On AMI MHM, Nemotron 3 recorded 9.48 percent compared with 16.36 percent, while AMI SDM recorded 12.80 percent compared with 21.73 percent.
On NOTSOFAR1 MHM, the new model recorded 7.70 percent compared with 22.12 percent. On NOTSOFAR1 SC, it recorded 12.77 percent compared with 31.81 percent.
NVIDIA calculates an unweighted average relative DER reduction of 41.0 percent across those eight conditions at the 1.04-second configuration. That figure is an average of relative reductions rather than a pooled DER across all recordings.
The results are not uniformly better in every condition. NVIDIA gives the two-speaker CALLHOME subset as an example. In that subset at the 30.4-second setting, the earlier model recorded 5.68 percent DER while Nemotron 3 recorded 5.98 percent.
The largest reported gains appear in higher-speaker-count conditions. On NOTSOFAR1 MHM at the 30.4-second setting, the previous model recorded 11.14 percent DER for recordings with three to four speakers and 29.38 percent for recordings with five to seven speakers.
Nemotron 3 recorded 5.25 percent for the three-to-four-speaker group and 7.86 percent for the five-to-seven-speaker group.
The model’s maximum of eight speakers is a hard limit. NVIDIA warns that recordings with more than eight speakers can result in missed or incorrectly assigned speech.
That limitation is relevant to the DIHARD III evaluation because that benchmark includes recordings with five through nine speakers. A model with eight output channels cannot represent more than eight simultaneous speaker identities within its channel structure.
The model was trained with about 10,000 hours of real conversational audio and 82,611 hours of simulated multi-talker mixtures created from about 28,000 hours of single-speaker recordings.
NVIDIA lists training sources including Fisher, AMI, ICSI, VoxConverse, AISHELL-4, DIHARD III, CALLHOME, AliMeeting, DiPCo, NOTSOFAR1, DISPLACE 2024, DISPLACE-M 2026, licensed David AI multispeaker recordings and a pseudo-labelled YODAS-v2 subset.
The simulated mixtures were created from material including LibriSpeech, AMI, AliMeeting, Fisher and licensed David AI datasets.
NVIDIA also reports that licensed David AI conversational data contributed measurable improvements. Adding the David AI data reduced compound DER from 11.19 percent to 10.42 percent, an absolute reduction of 0.77 percentage points at both the offline-style and ultra-low-latency operating points.
The model card identifies a David AI D12 Human Transcripts dataset covering 21 languages. That does not mean the model’s published benchmark establishes equal performance across all 21 languages. The detailed evaluation results reported by NVIDIA are centred on the datasets and language conditions described in its testing.
Training began from a Transformer-based NEST self-supervised checkpoint. NVIDIA trained the model using eight nodes containing eight NVIDIA A100-SXM4-80GB GPUs each, for a total of 64 A100 GPUs.
The training process included an offline stage based on simulated data followed by streaming fine-tuning using both real conversations and simulated multi-speaker mixtures.
For inference, NVIDIA reports measurements on an RTX PRO 5000 Blackwell GPU using BF16 precision and the PyTorch backend. The tests compared standard eager execution with torch.compile().
At the 1.04-second latency configuration and batch size one, NVIDIA reports 38 times real-time factor for eager execution and 164 times for the compiled configuration.
At batch size 32, the reported figures increase to 581 times and 865 times real-time factor respectively.
At the 0.32-second configuration, batch size one reached 12.5 times real-time factor with eager execution and 54 times with compilation. At batch size 32, the figures were 199 times and 292 times.
These are hardware- and configuration-specific measurements and should not be treated as universal performance figures for every supported NVIDIA GPU.
The model card lists compatibility across NVIDIA’s Ampere, Ada Lovelace, Hopper and Blackwell architectures. The documented hardware range includes GeForce GPUs, RTX workstation products and data-centre GPUs such as the A100, H100 and H200.
NVIDIA recommends Linux for deployment.
The current model is released under the OpenMDW License Agreement 1.1. The model repository describes Nemotron 3 Diarization as an open-weight release.
Discover more from Aree Blog
Subscribe now to keep reading and get access to the full archive.
