Nemotron 3 Diarization
Nemotron 3 DiarizationAn open-weight speaker diarization model that identifies who spoke when in real time, with accurate, low-latency tracking—even during overlapping speech.
Model details
NVIDIA Nemotron 3 Diarization answers "who spoke when" in real-world audio: up to eight speakers, labels ordered by first arrival, and a single checkpoint that runs at any of several algorithmic latencies, from an 0.32-second buffer for live agents to a 30-second buffer for recordings. It is the successor to Streaming Sortformer v2.1 and is released under the OpenMDW 1.1 license for commercial use.
Nemotron 3 Diarization offers exceptional accuracy at three, request-configurable latency profiles.Using the model
On Baseten, Nemotron 3 Diarization is available as three presets, each tuned for a different way of consuming it. Every preset runs on one NVIDIA RTX PRO 6000 and is served with a CUDA-graph engine that we built around NVIDIA's NeMo streaming loop.
Real-Time Diarization
A low-latency WebSocket endpoint for live audio, carrying speaker identity throughout the conversation without re-clustering. A GPU can sustain 500+ concurrent streams at the 1.04-second algorithmic latency profile, and 200 at 0.32 algorithmic latency.
1import asyncio, base64, json, os, wave
2import websockets # pip install websockets
3
4KEY = os.environ["BASETEN_API_KEY"]
5MODEL = os.environ["BASETEN_MODEL_ID"]
6URL = f"wss://model-{MODEL}.api.baseten.co"
7URL += "/environments/production/websocket"
8
9
10async def diarize(path, latency="low"):
11 # 16 kHz mono 16-bit WAV
12 with wave.open(path, "rb") as f:
13 pcm = f.readframes(f.getnframes())
14 auth = {"Authorization": f"Bearer {KEY}"}
15 async with websockets.connect(
16 URL, additional_headers=auth, open_timeout=600
17 ) as ws:
18 # "low", "ultralow", or "offline"
19 await ws.send(json.dumps({"latency": latency}))
20
21 async def send():
22 # 100 ms frames at real-time pace
23 for i in range(0, len(pcm), 3200):
24 b64 = base64.b64encode(pcm[i:i + 3200])
25 await ws.send(json.dumps({
26 "type": "input_audio_buffer.append",
27 "audio": b64.decode(),
28 }))
29 await asyncio.sleep(0.1)
30 await ws.send(json.dumps(
31 {"type": "input_audio_buffer.commit"}))
32
33 sender = asyncio.create_task(send())
34 async for message in ws:
35 frame = json.loads(message)
36 if frame.get("type") == "error":
37 raise RuntimeError(frame["error"])
38 # Full turn list each frame: replace, don't append.
39 turns = ", ".join(
40 f"{t['speaker']} {t['start']:.1f}-{t['end']:.1f}"
41 for t in frame["turns"]
42 )
43 print(f"{frame['processed_s']:6.1f}s {turns}")
44 if frame.get("is_final"):
45 break
46 await sender
47
48asyncio.run(diarize("meeting.wav"))Batch Diarization
An HTTP endpoint for recorded audio. Send a file URL or base64 audio, and pick a latency profile per request, get back speaker turns. One RTX-PRO-6000 can process 200 six-minute files per minute.
1import os
2import requests
3
4model_id = os.environ["BASETEN_MODEL_ID"]
5response = requests.post(
6 f"https://model-{model_id}.api.baseten.co/environments/production/predict",
7 headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"},
8 json={
9 "diarization_input": {
10 "audio": {"url": "https://github.com/ggerganov/whisper.cpp/raw/master/samples/jfk.wav"},
11 "latency": "offline",
12 }
13 },
14 timeout=300,
15)
16response.raise_for_status()
17for turn in response.json()["turns"]:
18 print(f"{turn['speaker']} {turn['start']:7.2f} {turn['end']:7.2f}")
19Real-Time Diarized Transcription
Nemotron 3 Diarization paired with NVIDIA's multitalker Parakeet 0.6B ASR (more alternative ASR options are in progress!). Send live audio, receive speaker-tagged words with timings, per-speaker partials, and overlap flags. Speaker labels are final on arrival and committed words are never revised. One GPU holds 190 hour-long streams. Please check out our documentation page for a client example.
