molto commited on
Commit
b2d4f1f
1 Parent(s): 0f59c10

Upload 5 files

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ speakerverification_en_titanet_large.nemo filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,3 +1,233 @@
1
  ---
2
- license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - en
4
+ library_name: nemo
5
+ datasets:
6
+ - VOXCELEB-1
7
+ - VOXCELEB-2
8
+ - FISHER
9
+ - switchboard
10
+ - librispeech_asr
11
+ - SRE(2004-2010)
12
+ thumbnail: null
13
+ tags:
14
+ - speaker
15
+ - speech
16
+ - audio
17
+ - speaker-verification
18
+ - speaker-recognition
19
+ - speaker-diarization
20
+ - titanet
21
+ - NeMo
22
+ - pytorch
23
+ license: cc-by-4.0
24
+ widget:
25
+ - src: https://huggingface.co/nvidia/speakerverification_en_titanet_large/resolve/main/an255-fash-b.wav
26
+ example_title: Speech sample 1
27
+ - src: https://huggingface.co/nvidia/speakerverification_en_titanet_large/resolve/main/cen7-fash-b.wav
28
+ example_title: Speech sample 2
29
+ model-index:
30
+ - name: speakerverification_en_titanet_large
31
+ results:
32
+ - task:
33
+ name: Speaker Verification
34
+ type: speaker-verification
35
+ dataset:
36
+ name: voxceleb1
37
+ type: voxceleb1-O
38
+ config: clean
39
+ split: test
40
+ args:
41
+ language: en
42
+ metrics:
43
+ - name: Test EER
44
+ type: eer
45
+ value: 0.66
46
+ - task:
47
+ type: Speaker Diarization
48
+ name: speaker-diarization
49
+ dataset:
50
+ name: ami-mixheadset
51
+ type: ami_diarization
52
+ config: oracle-vad-known-number-of-speakers
53
+ split: test
54
+ args:
55
+ language: en
56
+ metrics:
57
+ - name: Test DER
58
+ type: der
59
+ value: 1.73
60
+ - task:
61
+ type: Speaker Diarization
62
+ name: speaker-diarization
63
+ dataset:
64
+ name: ami-lapel
65
+ type: ami_diarization
66
+ config: oracle-vad-known-number-of-speakers
67
+ split: test
68
+ args:
69
+ language: en
70
+ metrics:
71
+ - name: Test DER
72
+ type: der
73
+ value: 2.03
74
+ - task:
75
+ type: Speaker Diarization
76
+ name: speaker-diarization
77
+ dataset:
78
+ name: ch109
79
+ type: callhome_diarization
80
+ config: oracle-vad-known-number-of-speakers
81
+ split: test
82
+ args:
83
+ language: en
84
+ metrics:
85
+ - name: Test DER
86
+ type: der
87
+ value: 1.19
88
+ - task:
89
+ type: Speaker Diarization
90
+ name: speaker-diarization
91
+ dataset:
92
+ name: nist-sre-2000
93
+ type: nist-sre_diarization
94
+ config: oracle-vad-known-number-of-speakers
95
+ split: test
96
+ args:
97
+ language: en
98
+ metrics:
99
+ - name: Test DER
100
+ type: der
101
+ value: 6.73
102
  ---
103
+
104
+ # NVIDIA TitaNet-Large (en-US)
105
+
106
+ <style>
107
+ img {
108
+ display: inline;
109
+ }
110
+ </style>
111
+
112
+ | [![Model architecture](https://img.shields.io/badge/Model_Arch-TitaNet--Large-lightgrey#model-badge)](#model-architecture)
113
+ | [![Model size](https://img.shields.io/badge/Params-23M-lightgrey#model-badge)](#model-architecture)
114
+ | [![Language](https://img.shields.io/badge/Language-en--US-lightgrey#model-badge)](#datasets)
115
+
116
+
117
+ This model extracts speaker embeddings from given speech, which is the backbone for speaker verification and diarization tasks.
118
+ It is a "large" version of TitaNet (around 23M parameters) models.
119
+ See the [model architecture](#model-architecture) section and [NeMo documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/speaker_recognition/models.html#titanet) for complete architecture details.
120
+
121
+ ## NVIDIA NeMo: Training
122
+
123
+ To train, fine-tune or play with the model you will need to install [NVIDIA NeMo](https://github.com/NVIDIA/NeMo). We recommend you install it after you've installed the latest Pytorch version.
124
+ ```
125
+ pip install nemo_toolkit['all']
126
+ ```
127
+
128
+ ## How to Use this Model
129
+
130
+ The model is available for use in the NeMo toolkit [3] and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.
131
+
132
+ ### Automatically instantiate the model
133
+
134
+ ```python
135
+ import nemo.collections.asr as nemo_asr
136
+ speaker_model = nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained("nvidia/speakerverification_en_titanet_large")
137
+ ```
138
+
139
+ ### Embedding Extraction
140
+
141
+ Using
142
+
143
+ ```python
144
+ emb = speaker_model.get_embedding("an255-fash-b.wav")
145
+ ```
146
+
147
+ ### Verifying two utterances (Speaker Verification)
148
+
149
+ Now to check if two audio files are from the same speaker or not, simply do:
150
+
151
+ ```python
152
+ speaker_model.verify_speakers("an255-fash-b.wav","cen7-fash-b.wav")
153
+ ```
154
+
155
+ ### Extracting Embeddings for more audio files
156
+
157
+ To extract embeddings from a bunch of audio files:
158
+
159
+ Write audio files to a `manifest.json` file with lines as in format:
160
+
161
+ ```json
162
+ {"audio_filepath": "<absolute path to dataset>/audio_file.wav", "duration": "duration of file in sec", "label": "speaker_id"}
163
+ ```
164
+
165
+ Then running following script will extract embeddings and writes to current working directory:
166
+ ```shell
167
+ python <NeMo_root>/examples/speaker_tasks/recognition/extract_speaker_embeddings.py --manifest=manifest.json
168
+ ```
169
+
170
+ ### Input
171
+
172
+ This model accepts 16000 KHz Mono-channel Audio (wav files) as input.
173
+
174
+ ### Output
175
+
176
+ This model provides speaker embeddings for an audio file.
177
+
178
+ ## Model Architecture
179
+
180
+ TitaNet model is a depth-wise separable conv1D model [1] for Speaker Verification and diarization tasks. You may find more info on the detail of this model here: [TitaNet-Model](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/speaker_recognition/models.html).
181
+
182
+ ## Training
183
+
184
+ The NeMo toolkit [3] was used for training the models for over several hundred epochs. These model are trained with this [example script](https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_tasks/recognition/speaker_reco.py) and this [base config](https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_tasks/recognition/conf/titanet-large.yaml).
185
+
186
+ ### Datasets
187
+
188
+ All the models in this collection are trained on a composite dataset comprising several thousand hours of English speech:
189
+
190
+ - Voxceleb-1
191
+ - Voxceleb-2
192
+ - Fisher
193
+ - Switchboard
194
+ - Librispeech
195
+ - SRE (2004-2010)
196
+
197
+ ## Performance
198
+
199
+ Performances of the these models are reported in terms of Equal Error Rate (EER%) on speaker verification evaluation trial files and as Diarization Error Rate (DER%) on diarization test sessions.
200
+
201
+ * Speaker Verification (EER%)
202
+ | Version | Model | Model Size | VoxCeleb1 (Cleaned trial file) |
203
+ |---------|--------------|-----|---------------|
204
+ | 1.10.0 | TitaNet-Large | 23M | 0.66 |
205
+
206
+ * Speaker Diarization (DER%)
207
+ | Version | Model | Model Size | Evaluation Condition | NIST SRE 2000 | AMI (Lapel) | AMI (MixHeadset) | CH109 |
208
+ |---------|--------------|-----|----------------------|---------------|-------------|------------------|-------|
209
+ | 1.10.0 | TitaNet-Large | 23M | Oracle VAD KNOWN # of Speakers | 6.73 | 2.03 | 1.73 | 1.19 |
210
+ | 1.10.0 | TitaNet-Large | 23M | Oracle VAD UNKNOWN # of Speakers | 5.38 | 2.03 | 1.89 | 1.63 |
211
+
212
+ ## Limitations
213
+ This model is trained on both telephonic and non-telephonic speech from voxceleb datasets, Fisher and switch board. If your domain of data differs from trained data or doesnot show relatively good performance consider finetuning for that speech domain.
214
+
215
+ ## NVIDIA Riva: Deployment
216
+
217
+ [NVIDIA Riva](https://developer.nvidia.com/riva), is an accelerated speech AI SDK deployable on-prem, in all clouds, multi-cloud, hybrid, on edge, and embedded.
218
+ Additionally, Riva provides:
219
+
220
+ * World-class out-of-the-box accuracy for the most common languages with model checkpoints trained on proprietary data with hundreds of thousands of GPU-compute hours
221
+ * Best in class accuracy with run-time word boosting (e.g., brand and product names) and customization of acoustic model, language model, and inverse text normalization
222
+ * Streaming speech recognition, Kubernetes compatible scaling, and enterprise-grade support
223
+
224
+ Although this model isn’t supported yet by Riva, the [list of supported models is here](https://huggingface.co/models?other=Riva).
225
+ Check out [Riva live demo](https://developer.nvidia.com/riva#demos).
226
+
227
+ ## References
228
+ [1] [TitaNet: Neural Model for Speaker Representation with 1D Depth-wise Separable convolutions and global context](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9746806)
229
+ [2] [NVIDIA NeMo Toolkit](https://github.com/NVIDIA/NeMo)
230
+
231
+ ## Licence
232
+
233
+ License to use this model is covered by the [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/). By downloading the public and release version of the model, you accept the terms and conditions of the [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) license.
an255-fash-b.wav ADDED
Binary file (83.2 kB). View file
 
cen7-fash-b.wav ADDED
Binary file (80 kB). View file
 
gitattributes ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ftz filter=lfs diff=lfs merge=lfs -text
6
+ *.gz filter=lfs diff=lfs merge=lfs -text
7
+ *.h5 filter=lfs diff=lfs merge=lfs -text
8
+ *.joblib filter=lfs diff=lfs merge=lfs -text
9
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
10
+ *.model filter=lfs diff=lfs merge=lfs -text
11
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
12
+ *.onnx filter=lfs diff=lfs merge=lfs -text
13
+ *.ot filter=lfs diff=lfs merge=lfs -text
14
+ *.parquet filter=lfs diff=lfs merge=lfs -text
15
+ *.pb filter=lfs diff=lfs merge=lfs -text
16
+ *.pt filter=lfs diff=lfs merge=lfs -text
17
+ *.pth filter=lfs diff=lfs merge=lfs -text
18
+ *.rar filter=lfs diff=lfs merge=lfs -text
19
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
20
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
21
+ *.tflite filter=lfs diff=lfs merge=lfs -text
22
+ *.tgz filter=lfs diff=lfs merge=lfs -text
23
+ *.wasm filter=lfs diff=lfs merge=lfs -text
24
+ *.xz filter=lfs diff=lfs merge=lfs -text
25
+ *.zip filter=lfs diff=lfs merge=lfs -text
26
+ *.zstandard filter=lfs diff=lfs merge=lfs -text
27
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
28
+ speakerverification_en_titanet_large.nemo filter=lfs diff=lfs merge=lfs -text
speakerverification_en_titanet_large.nemo ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e838520693f269e7984f55bc8eb3c2d60ccf246bf4b896d4be9bcabe3e4b0fe3
3
+ size 101621760