File size: 9,562 Bytes
45ee559
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview\n",
    "\n",
    "This notebook can be used with both a single or multi- speaker corpus and allows the interactive plotting of speaker embeddings linked to underlying audio (see instructions in the repo's speaker_embedding directory)\n",
    "\n",
    "Depending on the directory structure used for your corpus, you may need to adjust handling of **speaker_to_utter** and **locations**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import glob\n",
    "import numpy as np\n",
    "import umap\n",
    "\n",
    "from TTS.utils.audio import AudioProcessor\n",
    "from TTS.config import load_config\n",
    "\n",
    "from bokeh.io import output_notebook, show\n",
    "from bokeh.plotting import figure\n",
    "from bokeh.models import HoverTool, ColumnDataSource, BoxZoomTool, ResetTool, OpenURL, TapTool\n",
    "from bokeh.transform import factor_cmap\n",
    "from bokeh.palettes import Category10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For larger sets of speakers, you can use **Category20**, but you need to change it in the **pal** variable too\n",
    "\n",
    "List of Bokeh palettes here: http://docs.bokeh.org/en/1.4.0/docs/reference/palettes.html\n",
    "\n",
    "**NB:** if you have problems with other palettes, first see https://stackoverflow.com/questions/48333820/why-do-some-bokeh-palettes-raise-a-valueerror-when-used-in-factor-cmap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should also adjust all the path constants to point at the relevant locations for you locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_RUN_PATH = \"/media/erogol/data_ssd/Models/libri_tts/speaker_encoder/libritts_360-half-October-31-2019_04+54PM-19d2f5f/\"\n",
    "MODEL_PATH = MODEL_RUN_PATH + \"best_model.pth\"\n",
    "CONFIG_PATH = MODEL_RUN_PATH + \"config.json\"\n",
    "\n",
    "# My single speaker locations\n",
    "#EMBED_PATH = \"/home/neil/main/Projects/TTS3/embeddings/neil14/\"\n",
    "#AUDIO_PATH = \"/home/neil/data/Projects/NeilTTS/neil14/wavs/\"\n",
    "\n",
    "# My multi speaker locations\n",
    "EMBED_PATH = \"/home/erogol/Data/Libri-TTS/train-clean-360-embed_128/\"\n",
    "AUDIO_PATH = \"/home/erogol/Data/Libri-TTS/train-clean-360/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -1 $MODEL_RUN_PATH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CONFIG = load_config(CONFIG_PATH)\n",
    "ap = AudioProcessor(**CONFIG['audio'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bring in the embeddings created by **compute_embeddings.py**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embed_files = glob.glob(EMBED_PATH+\"/**/*.npy\", recursive=True)\n",
    "print(f'Embeddings found: {len(embed_files)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check that we did indeed find an embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embed_files[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Process the speakers\n",
    "\n",
    "Assumes count of **speaker_paths** corresponds to number of speakers (so a corpus in just one directory would be treated like a single speaker and the multiple directories of LibriTTS are treated as distinct speakers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "speaker_paths = list(set([os.path.dirname(os.path.dirname(embed_file)) for embed_file in embed_files]))\n",
    "speaker_to_utter = {}\n",
    "for embed_file in embed_files:\n",
    "    speaker_path = os.path.dirname(os.path.dirname(embed_file))\n",
    "    try:\n",
    "        speaker_to_utter[speaker_path].append(embed_file)\n",
    "    except:\n",
    "        speaker_to_utter[speaker_path]=[embed_file]\n",
    "print(f'Speaker count: {len(speaker_paths)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up the embeddings\n",
    "\n",
    "Adjust the number of speakers to select and the number of utterances from each speaker and they will be randomly sampled from the corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeds = []\n",
    "labels = []\n",
    "locations = []\n",
    "\n",
    "# single speaker \n",
    "#num_speakers = 1\n",
    "#num_utters = 1000\n",
    "\n",
    "# multi speaker\n",
    "num_speakers = 10\n",
    "num_utters = 20\n",
    "\n",
    "\n",
    "speaker_idxs = np.random.choice(range(len(speaker_paths)), num_speakers, replace=False )\n",
    "\n",
    "for speaker_num, speaker_idx in enumerate(speaker_idxs):\n",
    "    speaker_path = speaker_paths[speaker_idx]\n",
    "    speakers_utter = speaker_to_utter[speaker_path]\n",
    "    utter_idxs = np.random.randint(0, len(speakers_utter) , num_utters)\n",
    "    for utter_idx in utter_idxs:\n",
    "            embed_path = speaker_to_utter[speaker_path][utter_idx]\n",
    "            embed = np.load(embed_path)\n",
    "            embeds.append(embed)\n",
    "            labels.append(str(speaker_num))\n",
    "            locations.append(embed_path.replace(EMBED_PATH, '').replace('.npy','.wav'))\n",
    "embeds = np.concatenate(embeds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load embeddings with UMAP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = umap.UMAP()\n",
    "projection = model.fit_transform(embeds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactively charting the data in Bokeh\n",
    "\n",
    "Set up various details for Bokeh to plot the data\n",
    "\n",
    "You can use the regular Bokeh [tools](http://docs.bokeh.org/en/1.4.0/docs/user_guide/tools.html?highlight=tools) to explore the data, with reset setting it back to normal\n",
    "\n",
    "Once you have started the local server (see cell below) you can then click on plotted points which will open a tab to play the audio for that point, enabling easy exploration of your corpus\n",
    "\n",
    "File location in the tooltip is given relative to **AUDIO_PATH**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_wav_stems = ColumnDataSource(\n",
    "        data=dict(\n",
    "            x = projection.T[0].tolist(),\n",
    "            y = projection.T[1].tolist(),\n",
    "            desc=locations,\n",
    "            label=labels\n",
    "        )\n",
    "    )\n",
    "\n",
    "hover = HoverTool(\n",
    "        tooltips=[\n",
    "            (\"file\", \"@desc\"),\n",
    "            (\"speaker\", \"@label\"),\n",
    "        ]\n",
    "    )\n",
    "\n",
    "# optionally consider adding these to the tooltips if you want additional detail\n",
    "# for the coordinates: (\"(x,y)\", \"($x, $y)\"),\n",
    "# for the index of the embedding / wav file: (\"index\", \"$index\"),\n",
    "\n",
    "factors = list(set(labels))\n",
    "pal_size = max(len(factors), 3)\n",
    "pal = Category10[pal_size]\n",
    "\n",
    "p = figure(plot_width=600, plot_height=400, tools=[hover,BoxZoomTool(), ResetTool(), TapTool()])\n",
    "\n",
    "\n",
    "p.circle('x', 'y',  source=source_wav_stems, color=factor_cmap('label', palette=pal, factors=factors),)\n",
    "\n",
    "url = \"http://localhost:8000/@desc\"\n",
    "taptool = p.select(type=TapTool)\n",
    "taptool.callback = OpenURL(url=url)\n",
    "\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Local server to serve wav files from corpus\n",
    "\n",
    "This is required so that when you click on a data point the hyperlink associated with it will be served the file locally.\n",
    "\n",
    "There are other ways to serve this if you prefer and you can also run the commands manually on the command line\n",
    "\n",
    "The server will continue to run until stopped. To stop it simply interupt the kernel (ie square button or under Kernel menu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd $AUDIO_PATH\n",
    "%pwd\n",
    "!python -m http.server"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}