Nephele commited on
Commit
25639d3
1 Parent(s): fc6b7ec

push_231009003337

Browse files
LICENSE ADDED
@@ -0,0 +1,661 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ GNU AFFERO GENERAL PUBLIC LICENSE
2
+ Version 3, 19 November 2007
3
+
4
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
5
+ Everyone is permitted to copy and distribute verbatim copies
6
+ of this license document, but changing it is not allowed.
7
+
8
+ Preamble
9
+
10
+ The GNU Affero General Public License is a free, copyleft license for
11
+ software and other kinds of works, specifically designed to ensure
12
+ cooperation with the community in the case of network server software.
13
+
14
+ The licenses for most software and other practical works are designed
15
+ to take away your freedom to share and change the works. By contrast,
16
+ our General Public Licenses are intended to guarantee your freedom to
17
+ share and change all versions of a program--to make sure it remains free
18
+ software for all its users.
19
+
20
+ When we speak of free software, we are referring to freedom, not
21
+ price. Our General Public Licenses are designed to make sure that you
22
+ have the freedom to distribute copies of free software (and charge for
23
+ them if you wish), that you receive source code or can get it if you
24
+ want it, that you can change the software or use pieces of it in new
25
+ free programs, and that you know you can do these things.
26
+
27
+ Developers that use our General Public Licenses protect your rights
28
+ with two steps: (1) assert copyright on the software, and (2) offer
29
+ you this License which gives you legal permission to copy, distribute
30
+ and/or modify the software.
31
+
32
+ A secondary benefit of defending all users' freedom is that
33
+ improvements made in alternate versions of the program, if they
34
+ receive widespread use, become available for other developers to
35
+ incorporate. Many developers of free software are heartened and
36
+ encouraged by the resulting cooperation. However, in the case of
37
+ software used on network servers, this result may fail to come about.
38
+ The GNU General Public License permits making a modified version and
39
+ letting the public access it on a server without ever releasing its
40
+ source code to the public.
41
+
42
+ The GNU Affero General Public License is designed specifically to
43
+ ensure that, in such cases, the modified source code becomes available
44
+ to the community. It requires the operator of a network server to
45
+ provide the source code of the modified version running there to the
46
+ users of that server. Therefore, public use of a modified version, on
47
+ a publicly accessible server, gives the public access to the source
48
+ code of the modified version.
49
+
50
+ An older license, called the Affero General Public License and
51
+ published by Affero, was designed to accomplish similar goals. This is
52
+ a different license, not a version of the Affero GPL, but Affero has
53
+ released a new version of the Affero GPL which permits relicensing under
54
+ this license.
55
+
56
+ The precise terms and conditions for copying, distribution and
57
+ modification follow.
58
+
59
+ TERMS AND CONDITIONS
60
+
61
+ 0. Definitions.
62
+
63
+ "This License" refers to version 3 of the GNU Affero General Public License.
64
+
65
+ "Copyright" also means copyright-like laws that apply to other kinds of
66
+ works, such as semiconductor masks.
67
+
68
+ "The Program" refers to any copyrightable work licensed under this
69
+ License. Each licensee is addressed as "you". "Licensees" and
70
+ "recipients" may be individuals or organizations.
71
+
72
+ To "modify" a work means to copy from or adapt all or part of the work
73
+ in a fashion requiring copyright permission, other than the making of an
74
+ exact copy. The resulting work is called a "modified version" of the
75
+ earlier work or a work "based on" the earlier work.
76
+
77
+ A "covered work" means either the unmodified Program or a work based
78
+ on the Program.
79
+
80
+ To "propagate" a work means to do anything with it that, without
81
+ permission, would make you directly or secondarily liable for
82
+ infringement under applicable copyright law, except executing it on a
83
+ computer or modifying a private copy. Propagation includes copying,
84
+ distribution (with or without modification), making available to the
85
+ public, and in some countries other activities as well.
86
+
87
+ To "convey" a work means any kind of propagation that enables other
88
+ parties to make or receive copies. Mere interaction with a user through
89
+ a computer network, with no transfer of a copy, is not conveying.
90
+
91
+ An interactive user interface displays "Appropriate Legal Notices"
92
+ to the extent that it includes a convenient and prominently visible
93
+ feature that (1) displays an appropriate copyright notice, and (2)
94
+ tells the user that there is no warranty for the work (except to the
95
+ extent that warranties are provided), that licensees may convey the
96
+ work under this License, and how to view a copy of this License. If
97
+ the interface presents a list of user commands or options, such as a
98
+ menu, a prominent item in the list meets this criterion.
99
+
100
+ 1. Source Code.
101
+
102
+ The "source code" for a work means the preferred form of the work
103
+ for making modifications to it. "Object code" means any non-source
104
+ form of a work.
105
+
106
+ A "Standard Interface" means an interface that either is an official
107
+ standard defined by a recognized standards body, or, in the case of
108
+ interfaces specified for a particular programming language, one that
109
+ is widely used among developers working in that language.
110
+
111
+ The "System Libraries" of an executable work include anything, other
112
+ than the work as a whole, that (a) is included in the normal form of
113
+ packaging a Major Component, but which is not part of that Major
114
+ Component, and (b) serves only to enable use of the work with that
115
+ Major Component, or to implement a Standard Interface for which an
116
+ implementation is available to the public in source code form. A
117
+ "Major Component", in this context, means a major essential component
118
+ (kernel, window system, and so on) of the specific operating system
119
+ (if any) on which the executable work runs, or a compiler used to
120
+ produce the work, or an object code interpreter used to run it.
121
+
122
+ The "Corresponding Source" for a work in object code form means all
123
+ the source code needed to generate, install, and (for an executable
124
+ work) run the object code and to modify the work, including scripts to
125
+ control those activities. However, it does not include the work's
126
+ System Libraries, or general-purpose tools or generally available free
127
+ programs which are used unmodified in performing those activities but
128
+ which are not part of the work. For example, Corresponding Source
129
+ includes interface definition files associated with source files for
130
+ the work, and the source code for shared libraries and dynamically
131
+ linked subprograms that the work is specifically designed to require,
132
+ such as by intimate data communication or control flow between those
133
+ subprograms and other parts of the work.
134
+
135
+ The Corresponding Source need not include anything that users
136
+ can regenerate automatically from other parts of the Corresponding
137
+ Source.
138
+
139
+ The Corresponding Source for a work in source code form is that
140
+ same work.
141
+
142
+ 2. Basic Permissions.
143
+
144
+ All rights granted under this License are granted for the term of
145
+ copyright on the Program, and are irrevocable provided the stated
146
+ conditions are met. This License explicitly affirms your unlimited
147
+ permission to run the unmodified Program. The output from running a
148
+ covered work is covered by this License only if the output, given its
149
+ content, constitutes a covered work. This License acknowledges your
150
+ rights of fair use or other equivalent, as provided by copyright law.
151
+
152
+ You may make, run and propagate covered works that you do not
153
+ convey, without conditions so long as your license otherwise remains
154
+ in force. You may convey covered works to others for the sole purpose
155
+ of having them make modifications exclusively for you, or provide you
156
+ with facilities for running those works, provided that you comply with
157
+ the terms of this License in conveying all material for which you do
158
+ not control copyright. Those thus making or running the covered works
159
+ for you must do so exclusively on your behalf, under your direction
160
+ and control, on terms that prohibit them from making any copies of
161
+ your copyrighted material outside their relationship with you.
162
+
163
+ Conveying under any other circumstances is permitted solely under
164
+ the conditions stated below. Sublicensing is not allowed; section 10
165
+ makes it unnecessary.
166
+
167
+ 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
168
+
169
+ No covered work shall be deemed part of an effective technological
170
+ measure under any applicable law fulfilling obligations under article
171
+ 11 of the WIPO copyright treaty adopted on 20 December 1996, or
172
+ similar laws prohibiting or restricting circumvention of such
173
+ measures.
174
+
175
+ When you convey a covered work, you waive any legal power to forbid
176
+ circumvention of technological measures to the extent such circumvention
177
+ is effected by exercising rights under this License with respect to
178
+ the covered work, and you disclaim any intention to limit operation or
179
+ modification of the work as a means of enforcing, against the work's
180
+ users, your or third parties' legal rights to forbid circumvention of
181
+ technological measures.
182
+
183
+ 4. Conveying Verbatim Copies.
184
+
185
+ You may convey verbatim copies of the Program's source code as you
186
+ receive it, in any medium, provided that you conspicuously and
187
+ appropriately publish on each copy an appropriate copyright notice;
188
+ keep intact all notices stating that this License and any
189
+ non-permissive terms added in accord with section 7 apply to the code;
190
+ keep intact all notices of the absence of any warranty; and give all
191
+ recipients a copy of this License along with the Program.
192
+
193
+ You may charge any price or no price for each copy that you convey,
194
+ and you may offer support or warranty protection for a fee.
195
+
196
+ 5. Conveying Modified Source Versions.
197
+
198
+ You may convey a work based on the Program, or the modifications to
199
+ produce it from the Program, in the form of source code under the
200
+ terms of section 4, provided that you also meet all of these conditions:
201
+
202
+ a) The work must carry prominent notices stating that you modified
203
+ it, and giving a relevant date.
204
+
205
+ b) The work must carry prominent notices stating that it is
206
+ released under this License and any conditions added under section
207
+ 7. This requirement modifies the requirement in section 4 to
208
+ "keep intact all notices".
209
+
210
+ c) You must license the entire work, as a whole, under this
211
+ License to anyone who comes into possession of a copy. This
212
+ License will therefore apply, along with any applicable section 7
213
+ additional terms, to the whole of the work, and all its parts,
214
+ regardless of how they are packaged. This License gives no
215
+ permission to license the work in any other way, but it does not
216
+ invalidate such permission if you have separately received it.
217
+
218
+ d) If the work has interactive user interfaces, each must display
219
+ Appropriate Legal Notices; however, if the Program has interactive
220
+ interfaces that do not display Appropriate Legal Notices, your
221
+ work need not make them do so.
222
+
223
+ A compilation of a covered work with other separate and independent
224
+ works, which are not by their nature extensions of the covered work,
225
+ and which are not combined with it such as to form a larger program,
226
+ in or on a volume of a storage or distribution medium, is called an
227
+ "aggregate" if the compilation and its resulting copyright are not
228
+ used to limit the access or legal rights of the compilation's users
229
+ beyond what the individual works permit. Inclusion of a covered work
230
+ in an aggregate does not cause this License to apply to the other
231
+ parts of the aggregate.
232
+
233
+ 6. Conveying Non-Source Forms.
234
+
235
+ You may convey a covered work in object code form under the terms
236
+ of sections 4 and 5, provided that you also convey the
237
+ machine-readable Corresponding Source under the terms of this License,
238
+ in one of these ways:
239
+
240
+ a) Convey the object code in, or embodied in, a physical product
241
+ (including a physical distribution medium), accompanied by the
242
+ Corresponding Source fixed on a durable physical medium
243
+ customarily used for software interchange.
244
+
245
+ b) Convey the object code in, or embodied in, a physical product
246
+ (including a physical distribution medium), accompanied by a
247
+ written offer, valid for at least three years and valid for as
248
+ long as you offer spare parts or customer support for that product
249
+ model, to give anyone who possesses the object code either (1) a
250
+ copy of the Corresponding Source for all the software in the
251
+ product that is covered by this License, on a durable physical
252
+ medium customarily used for software interchange, for a price no
253
+ more than your reasonable cost of physically performing this
254
+ conveying of source, or (2) access to copy the
255
+ Corresponding Source from a network server at no charge.
256
+
257
+ c) Convey individual copies of the object code with a copy of the
258
+ written offer to provide the Corresponding Source. This
259
+ alternative is allowed only occasionally and noncommercially, and
260
+ only if you received the object code with such an offer, in accord
261
+ with subsection 6b.
262
+
263
+ d) Convey the object code by offering access from a designated
264
+ place (gratis or for a charge), and offer equivalent access to the
265
+ Corresponding Source in the same way through the same place at no
266
+ further charge. You need not require recipients to copy the
267
+ Corresponding Source along with the object code. If the place to
268
+ copy the object code is a network server, the Corresponding Source
269
+ may be on a different server (operated by you or a third party)
270
+ that supports equivalent copying facilities, provided you maintain
271
+ clear directions next to the object code saying where to find the
272
+ Corresponding Source. Regardless of what server hosts the
273
+ Corresponding Source, you remain obligated to ensure that it is
274
+ available for as long as needed to satisfy these requirements.
275
+
276
+ e) Convey the object code using peer-to-peer transmission, provided
277
+ you inform other peers where the object code and Corresponding
278
+ Source of the work are being offered to the general public at no
279
+ charge under subsection 6d.
280
+
281
+ A separable portion of the object code, whose source code is excluded
282
+ from the Corresponding Source as a System Library, need not be
283
+ included in conveying the object code work.
284
+
285
+ A "User Product" is either (1) a "consumer product", which means any
286
+ tangible personal property which is normally used for personal, family,
287
+ or household purposes, or (2) anything designed or sold for incorporation
288
+ into a dwelling. In determining whether a product is a consumer product,
289
+ doubtful cases shall be resolved in favor of coverage. For a particular
290
+ product received by a particular user, "normally used" refers to a
291
+ typical or common use of that class of product, regardless of the status
292
+ of the particular user or of the way in which the particular user
293
+ actually uses, or expects or is expected to use, the product. A product
294
+ is a consumer product regardless of whether the product has substantial
295
+ commercial, industrial or non-consumer uses, unless such uses represent
296
+ the only significant mode of use of the product.
297
+
298
+ "Installation Information" for a User Product means any methods,
299
+ procedures, authorization keys, or other information required to install
300
+ and execute modified versions of a covered work in that User Product from
301
+ a modified version of its Corresponding Source. The information must
302
+ suffice to ensure that the continued functioning of the modified object
303
+ code is in no case prevented or interfered with solely because
304
+ modification has been made.
305
+
306
+ If you convey an object code work under this section in, or with, or
307
+ specifically for use in, a User Product, and the conveying occurs as
308
+ part of a transaction in which the right of possession and use of the
309
+ User Product is transferred to the recipient in perpetuity or for a
310
+ fixed term (regardless of how the transaction is characterized), the
311
+ Corresponding Source conveyed under this section must be accompanied
312
+ by the Installation Information. But this requirement does not apply
313
+ if neither you nor any third party retains the ability to install
314
+ modified object code on the User Product (for example, the work has
315
+ been installed in ROM).
316
+
317
+ The requirement to provide Installation Information does not include a
318
+ requirement to continue to provide support service, warranty, or updates
319
+ for a work that has been modified or installed by the recipient, or for
320
+ the User Product in which it has been modified or installed. Access to a
321
+ network may be denied when the modification itself materially and
322
+ adversely affects the operation of the network or violates the rules and
323
+ protocols for communication across the network.
324
+
325
+ Corresponding Source conveyed, and Installation Information provided,
326
+ in accord with this section must be in a format that is publicly
327
+ documented (and with an implementation available to the public in
328
+ source code form), and must require no special password or key for
329
+ unpacking, reading or copying.
330
+
331
+ 7. Additional Terms.
332
+
333
+ "Additional permissions" are terms that supplement the terms of this
334
+ License by making exceptions from one or more of its conditions.
335
+ Additional permissions that are applicable to the entire Program shall
336
+ be treated as though they were included in this License, to the extent
337
+ that they are valid under applicable law. If additional permissions
338
+ apply only to part of the Program, that part may be used separately
339
+ under those permissions, but the entire Program remains governed by
340
+ this License without regard to the additional permissions.
341
+
342
+ When you convey a copy of a covered work, you may at your option
343
+ remove any additional permissions from that copy, or from any part of
344
+ it. (Additional permissions may be written to require their own
345
+ removal in certain cases when you modify the work.) You may place
346
+ additional permissions on material, added by you to a covered work,
347
+ for which you have or can give appropriate copyright permission.
348
+
349
+ Notwithstanding any other provision of this License, for material you
350
+ add to a covered work, you may (if authorized by the copyright holders of
351
+ that material) supplement the terms of this License with terms:
352
+
353
+ a) Disclaiming warranty or limiting liability differently from the
354
+ terms of sections 15 and 16 of this License; or
355
+
356
+ b) Requiring preservation of specified reasonable legal notices or
357
+ author attributions in that material or in the Appropriate Legal
358
+ Notices displayed by works containing it; or
359
+
360
+ c) Prohibiting misrepresentation of the origin of that material, or
361
+ requiring that modified versions of such material be marked in
362
+ reasonable ways as different from the original version; or
363
+
364
+ d) Limiting the use for publicity purposes of names of licensors or
365
+ authors of the material; or
366
+
367
+ e) Declining to grant rights under trademark law for use of some
368
+ trade names, trademarks, or service marks; or
369
+
370
+ f) Requiring indemnification of licensors and authors of that
371
+ material by anyone who conveys the material (or modified versions of
372
+ it) with contractual assumptions of liability to the recipient, for
373
+ any liability that these contractual assumptions directly impose on
374
+ those licensors and authors.
375
+
376
+ All other non-permissive additional terms are considered "further
377
+ restrictions" within the meaning of section 10. If the Program as you
378
+ received it, or any part of it, contains a notice stating that it is
379
+ governed by this License along with a term that is a further
380
+ restriction, you may remove that term. If a license document contains
381
+ a further restriction but permits relicensing or conveying under this
382
+ License, you may add to a covered work material governed by the terms
383
+ of that license document, provided that the further restriction does
384
+ not survive such relicensing or conveying.
385
+
386
+ If you add terms to a covered work in accord with this section, you
387
+ must place, in the relevant source files, a statement of the
388
+ additional terms that apply to those files, or a notice indicating
389
+ where to find the applicable terms.
390
+
391
+ Additional terms, permissive or non-permissive, may be stated in the
392
+ form of a separately written license, or stated as exceptions;
393
+ the above requirements apply either way.
394
+
395
+ 8. Termination.
396
+
397
+ You may not propagate or modify a covered work except as expressly
398
+ provided under this License. Any attempt otherwise to propagate or
399
+ modify it is void, and will automatically terminate your rights under
400
+ this License (including any patent licenses granted under the third
401
+ paragraph of section 11).
402
+
403
+ However, if you cease all violation of this License, then your
404
+ license from a particular copyright holder is reinstated (a)
405
+ provisionally, unless and until the copyright holder explicitly and
406
+ finally terminates your license, and (b) permanently, if the copyright
407
+ holder fails to notify you of the violation by some reasonable means
408
+ prior to 60 days after the cessation.
409
+
410
+ Moreover, your license from a particular copyright holder is
411
+ reinstated permanently if the copyright holder notifies you of the
412
+ violation by some reasonable means, this is the first time you have
413
+ received notice of violation of this License (for any work) from that
414
+ copyright holder, and you cure the violation prior to 30 days after
415
+ your receipt of the notice.
416
+
417
+ Termination of your rights under this section does not terminate the
418
+ licenses of parties who have received copies or rights from you under
419
+ this License. If your rights have been terminated and not permanently
420
+ reinstated, you do not qualify to receive new licenses for the same
421
+ material under section 10.
422
+
423
+ 9. Acceptance Not Required for Having Copies.
424
+
425
+ You are not required to accept this License in order to receive or
426
+ run a copy of the Program. Ancillary propagation of a covered work
427
+ occurring solely as a consequence of using peer-to-peer transmission
428
+ to receive a copy likewise does not require acceptance. However,
429
+ nothing other than this License grants you permission to propagate or
430
+ modify any covered work. These actions infringe copyright if you do
431
+ not accept this License. Therefore, by modifying or propagating a
432
+ covered work, you indicate your acceptance of this License to do so.
433
+
434
+ 10. Automatic Licensing of Downstream Recipients.
435
+
436
+ Each time you convey a covered work, the recipient automatically
437
+ receives a license from the original licensors, to run, modify and
438
+ propagate that work, subject to this License. You are not responsible
439
+ for enforcing compliance by third parties with this License.
440
+
441
+ An "entity transaction" is a transaction transferring control of an
442
+ organization, or substantially all assets of one, or subdividing an
443
+ organization, or merging organizations. If propagation of a covered
444
+ work results from an entity transaction, each party to that
445
+ transaction who receives a copy of the work also receives whatever
446
+ licenses to the work the party's predecessor in interest had or could
447
+ give under the previous paragraph, plus a right to possession of the
448
+ Corresponding Source of the work from the predecessor in interest, if
449
+ the predecessor has it or can get it with reasonable efforts.
450
+
451
+ You may not impose any further restrictions on the exercise of the
452
+ rights granted or affirmed under this License. For example, you may
453
+ not impose a license fee, royalty, or other charge for exercise of
454
+ rights granted under this License, and you may not initiate litigation
455
+ (including a cross-claim or counterclaim in a lawsuit) alleging that
456
+ any patent claim is infringed by making, using, selling, offering for
457
+ sale, or importing the Program or any portion of it.
458
+
459
+ 11. Patents.
460
+
461
+ A "contributor" is a copyright holder who authorizes use under this
462
+ License of the Program or a work on which the Program is based. The
463
+ work thus licensed is called the contributor's "contributor version".
464
+
465
+ A contributor's "essential patent claims" are all patent claims
466
+ owned or controlled by the contributor, whether already acquired or
467
+ hereafter acquired, that would be infringed by some manner, permitted
468
+ by this License, of making, using, or selling its contributor version,
469
+ but do not include claims that would be infringed only as a
470
+ consequence of further modification of the contributor version. For
471
+ purposes of this definition, "control" includes the right to grant
472
+ patent sublicenses in a manner consistent with the requirements of
473
+ this License.
474
+
475
+ Each contributor grants you a non-exclusive, worldwide, royalty-free
476
+ patent license under the contributor's essential patent claims, to
477
+ make, use, sell, offer for sale, import and otherwise run, modify and
478
+ propagate the contents of its contributor version.
479
+
480
+ In the following three paragraphs, a "patent license" is any express
481
+ agreement or commitment, however denominated, not to enforce a patent
482
+ (such as an express permission to practice a patent or covenant not to
483
+ sue for patent infringement). To "grant" such a patent license to a
484
+ party means to make such an agreement or commitment not to enforce a
485
+ patent against the party.
486
+
487
+ If you convey a covered work, knowingly relying on a patent license,
488
+ and the Corresponding Source of the work is not available for anyone
489
+ to copy, free of charge and under the terms of this License, through a
490
+ publicly available network server or other readily accessible means,
491
+ then you must either (1) cause the Corresponding Source to be so
492
+ available, or (2) arrange to deprive yourself of the benefit of the
493
+ patent license for this particular work, or (3) arrange, in a manner
494
+ consistent with the requirements of this License, to extend the patent
495
+ license to downstream recipients. "Knowingly relying" means you have
496
+ actual knowledge that, but for the patent license, your conveying the
497
+ covered work in a country, or your recipient's use of the covered work
498
+ in a country, would infringe one or more identifiable patents in that
499
+ country that you have reason to believe are valid.
500
+
501
+ If, pursuant to or in connection with a single transaction or
502
+ arrangement, you convey, or propagate by procuring conveyance of, a
503
+ covered work, and grant a patent license to some of the parties
504
+ receiving the covered work authorizing them to use, propagate, modify
505
+ or convey a specific copy of the covered work, then the patent license
506
+ you grant is automatically extended to all recipients of the covered
507
+ work and works based on it.
508
+
509
+ A patent license is "discriminatory" if it does not include within
510
+ the scope of its coverage, prohibits the exercise of, or is
511
+ conditioned on the non-exercise of one or more of the rights that are
512
+ specifically granted under this License. You may not convey a covered
513
+ work if you are a party to an arrangement with a third party that is
514
+ in the business of distributing software, under which you make payment
515
+ to the third party based on the extent of your activity of conveying
516
+ the work, and under which the third party grants, to any of the
517
+ parties who would receive the covered work from you, a discriminatory
518
+ patent license (a) in connection with copies of the covered work
519
+ conveyed by you (or copies made from those copies), or (b) primarily
520
+ for and in connection with specific products or compilations that
521
+ contain the covered work, unless you entered into that arrangement,
522
+ or that patent license was granted, prior to 28 March 2007.
523
+
524
+ Nothing in this License shall be construed as excluding or limiting
525
+ any implied license or other defenses to infringement that may
526
+ otherwise be available to you under applicable patent law.
527
+
528
+ 12. No Surrender of Others' Freedom.
529
+
530
+ If conditions are imposed on you (whether by court order, agreement or
531
+ otherwise) that contradict the conditions of this License, they do not
532
+ excuse you from the conditions of this License. If you cannot convey a
533
+ covered work so as to satisfy simultaneously your obligations under this
534
+ License and any other pertinent obligations, then as a consequence you may
535
+ not convey it at all. For example, if you agree to terms that obligate you
536
+ to collect a royalty for further conveying from those to whom you convey
537
+ the Program, the only way you could satisfy both those terms and this
538
+ License would be to refrain entirely from conveying the Program.
539
+
540
+ 13. Remote Network Interaction; Use with the GNU General Public License.
541
+
542
+ Notwithstanding any other provision of this License, if you modify the
543
+ Program, your modified version must prominently offer all users
544
+ interacting with it remotely through a computer network (if your version
545
+ supports such interaction) an opportunity to receive the Corresponding
546
+ Source of your version by providing access to the Corresponding Source
547
+ from a network server at no charge, through some standard or customary
548
+ means of facilitating copying of software. This Corresponding Source
549
+ shall include the Corresponding Source for any work covered by version 3
550
+ of the GNU General Public License that is incorporated pursuant to the
551
+ following paragraph.
552
+
553
+ Notwithstanding any other provision of this License, you have
554
+ permission to link or combine any covered work with a work licensed
555
+ under version 3 of the GNU General Public License into a single
556
+ combined work, and to convey the resulting work. The terms of this
557
+ License will continue to apply to the part which is the covered work,
558
+ but the work with which it is combined will remain governed by version
559
+ 3 of the GNU General Public License.
560
+
561
+ 14. Revised Versions of this License.
562
+
563
+ The Free Software Foundation may publish revised and/or new versions of
564
+ the GNU Affero General Public License from time to time. Such new versions
565
+ will be similar in spirit to the present version, but may differ in detail to
566
+ address new problems or concerns.
567
+
568
+ Each version is given a distinguishing version number. If the
569
+ Program specifies that a certain numbered version of the GNU Affero General
570
+ Public License "or any later version" applies to it, you have the
571
+ option of following the terms and conditions either of that numbered
572
+ version or of any later version published by the Free Software
573
+ Foundation. If the Program does not specify a version number of the
574
+ GNU Affero General Public License, you may choose any version ever published
575
+ by the Free Software Foundation.
576
+
577
+ If the Program specifies that a proxy can decide which future
578
+ versions of the GNU Affero General Public License can be used, that proxy's
579
+ public statement of acceptance of a version permanently authorizes you
580
+ to choose that version for the Program.
581
+
582
+ Later license versions may give you additional or different
583
+ permissions. However, no additional obligations are imposed on any
584
+ author or copyright holder as a result of your choosing to follow a
585
+ later version.
586
+
587
+ 15. Disclaimer of Warranty.
588
+
589
+ THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
590
+ APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
591
+ HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
592
+ OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
593
+ THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
594
+ PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
595
+ IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
596
+ ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
597
+
598
+ 16. Limitation of Liability.
599
+
600
+ IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
601
+ WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
602
+ THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
603
+ GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
604
+ USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
605
+ DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
606
+ PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
607
+ EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
608
+ SUCH DAMAGES.
609
+
610
+ 17. Interpretation of Sections 15 and 16.
611
+
612
+ If the disclaimer of warranty and limitation of liability provided
613
+ above cannot be given local legal effect according to their terms,
614
+ reviewing courts shall apply local law that most closely approximates
615
+ an absolute waiver of all civil liability in connection with the
616
+ Program, unless a warranty or assumption of liability accompanies a
617
+ copy of the Program in return for a fee.
618
+
619
+ END OF TERMS AND CONDITIONS
620
+
621
+ How to Apply These Terms to Your New Programs
622
+
623
+ If you develop a new program, and you want it to be of the greatest
624
+ possible use to the public, the best way to achieve this is to make it
625
+ free software which everyone can redistribute and change under these terms.
626
+
627
+ To do so, attach the following notices to the program. It is safest
628
+ to attach them to the start of each source file to most effectively
629
+ state the exclusion of warranty; and each file should have at least
630
+ the "copyright" line and a pointer to where the full notice is found.
631
+
632
+ <one line to give the program's name and a brief idea of what it does.>
633
+ Copyright (C) <year> <name of author>
634
+
635
+ This program is free software: you can redistribute it and/or modify
636
+ it under the terms of the GNU Affero General Public License as published
637
+ by the Free Software Foundation, either version 3 of the License, or
638
+ (at your option) any later version.
639
+
640
+ This program is distributed in the hope that it will be useful,
641
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
642
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
643
+ GNU Affero General Public License for more details.
644
+
645
+ You should have received a copy of the GNU Affero General Public License
646
+ along with this program. If not, see <https://www.gnu.org/licenses/>.
647
+
648
+ Also add information on how to contact you by electronic and paper mail.
649
+
650
+ If your software can interact with users remotely through a computer
651
+ network, you should also make sure that it provides a way for users to
652
+ get its source. For example, if your program is a web application, its
653
+ interface could display a "Source" link that leads users to an archive
654
+ of the code. There are many ways you could offer source, and different
655
+ solutions will be better for different programs; see section 13 for the
656
+ specific requirements.
657
+
658
+ You should also get your employer (if you work as a programmer) or school,
659
+ if any, to sign a "copyright disclaimer" for the program, if necessary.
660
+ For more information on this, and how to apply and follow the GNU AGPL, see
661
+ <https://www.gnu.org/licenses/>.
app.py ADDED
@@ -0,0 +1,162 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import sys, os
2
+ import torch
3
+ import argparse
4
+ import commons
5
+ import utils
6
+ from models import SynthesizerTrn
7
+ from text.symbols import symbols
8
+ from text import cleaned_text_to_sequence, get_bert
9
+ from text.cleaner import clean_text
10
+ import gradio as gr
11
+ import soundfile as sf
12
+ from datetime import datetime
13
+ import pytz
14
+
15
+
16
+ net_g = None
17
+ models = {
18
+ "V1": "v1100.pth",
19
+ "V2": "180_3000.pth",
20
+ "V3":"v3_8000.pth"
21
+
22
+ }
23
+
24
+ def get_text(text, language_str, hps):
25
+ norm_text, phone, tone, word2ph = clean_text(text, language_str)
26
+ phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
27
+
28
+ if hps.data.add_blank:
29
+ phone = commons.intersperse(phone, 0)
30
+ tone = commons.intersperse(tone, 0)
31
+ language = commons.intersperse(language, 0)
32
+ for i in range(len(word2ph)):
33
+ word2ph[i] = word2ph[i] * 2
34
+ word2ph[0] += 1
35
+ bert = get_bert(norm_text, word2ph, language_str)
36
+ del word2ph
37
+
38
+ assert bert.shape[-1] == len(phone)
39
+
40
+ phone = torch.LongTensor(phone)
41
+ tone = torch.LongTensor(tone)
42
+ language = torch.LongTensor(language)
43
+
44
+ return bert, phone, tone, language
45
+
46
+ def infer(text, sdp_ratio, noise_scale, noise_scale_w, length_scale, sid, model_dir):
47
+ global net_g
48
+ bert, phones, tones, lang_ids = get_text(text, "ZH", hps)
49
+ with torch.no_grad():
50
+ x_tst=phones.to(device).unsqueeze(0)
51
+ tones=tones.to(device).unsqueeze(0)
52
+ lang_ids=lang_ids.to(device).unsqueeze(0)
53
+ bert = bert.to(device).unsqueeze(0)
54
+ x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
55
+ del phones
56
+ speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
57
+ audio = net_g.infer(x_tst, x_tst_lengths, speakers, tones, lang_ids, bert, sdp_ratio=sdp_ratio
58
+ , noise_scale=noise_scale, noise_scale_w=noise_scale_w, length_scale=length_scale)[0][0,0].data.cpu().float().numpy()
59
+ del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers
60
+ sf.write("tmp.wav", audio, 44100)
61
+ return audio
62
+ tz = pytz.timezone('Asia/Shanghai')
63
+
64
+ def convert_wav_to_mp3(wav_file):
65
+ global tz
66
+ now = datetime.now(tz).strftime('%m%d%H%M%S')
67
+ os.makedirs('out', exist_ok=True)
68
+ output_path_mp3 = os.path.join('out', f"{now}.mp3")
69
+
70
+ renamed_input_path = os.path.join('in', f"in.wav")
71
+ os.makedirs('in', exist_ok=True)
72
+ os.rename(wav_file.name, renamed_input_path)
73
+ command = ["ffmpeg", "-i", renamed_input_path, "-acodec", "libmp3lame", "-y", output_path_mp3]
74
+ os.system(" ".join(command))
75
+ return output_path_mp3
76
+
77
+ def tts_generator(text, sdp_ratio, noise_scale, noise_scale_w, length_scale, model):
78
+ global net_g,speakers
79
+ model_path = models[model]
80
+ net_g, _, _, _ = utils.load_checkpoint(model_path, net_g, None, skip_optimizer=True)
81
+ text = text[:500]
82
+ try:
83
+ with torch.no_grad():
84
+ audio = infer(text, sdp_ratio=sdp_ratio, noise_scale=noise_scale, noise_scale_w=noise_scale_w, length_scale=length_scale, sid=speaker,model_dir=model)
85
+ with open('tmp.wav', 'rb') as wav_file:
86
+ mp3 = convert_wav_to_mp3(wav_file)
87
+ return "生成语音成功", (hps.data.sampling_rate, audio), mp3
88
+ except Exception as e:
89
+ return "生成语音失败:" + str(e), None, None
90
+
91
+
92
+ if __name__ == "__main__":
93
+ hps = utils.get_hparams_from_file("./configs/config.json")
94
+ device = "cuda:0" if torch.cuda.is_available() else "cpu"
95
+
96
+ net_g = SynthesizerTrn(
97
+ len(symbols),
98
+ hps.data.filter_length // 2 + 1,
99
+ hps.train.segment_size // hps.data.hop_length,
100
+ n_speakers=hps.data.n_speakers,
101
+ **hps.model).to(device)
102
+ _ = net_g.eval()
103
+
104
+ speaker_ids = hps.data.spk2id
105
+ speaker = list(speaker_ids.keys())[0]
106
+ theme='remilia/Ghostly'
107
+
108
+ with gr.Blocks(theme=theme) as app:
109
+ with gr.Row():
110
+ with gr.Column():
111
+
112
+ gr.Markdown("""**测试用**""")
113
+ text = gr.TextArea(label="✨输入需要生成语音的文字", placeholder="输入文字",
114
+ value="漩涡帮可不是吃素的,我是碰巧路过听人说,他们要整一个全金属和尚",
115
+ info="使用huggingface的免费CPU进行推理,因此速度不快,最多生成500字,多余的会被忽略。字数越多越耗时,请耐心等待,只会说中文",
116
+ )
117
+ model = gr.Radio(choices=list(models.keys()), value=list(models.keys())[0], label='📢音声模型')
118
+ with gr.Accordion(label="💡展开设置生成参数", open=False):
119
+ sdp_ratio = gr.Slider(minimum=0, maximum=1, value=0.2, step=0.01, label='SDP/DP混合比',info='可控制一定程度的语调变化')
120
+ noise_scale = gr.Slider(minimum=0.1, maximum=1.5, value=0.5, step=0.01, label='感情变化')
121
+ noise_scale_w = gr.Slider(minimum=0.1, maximum=1.4, value=0.9, step=0.01, label='音节长度')
122
+ length_scale = gr.Slider(minimum=0.1, maximum=2, value=1, step=0.01, label='生成语音总长度',info='数值越大,语速越慢')
123
+ btn = gr.Button("🪄生成", variant="primary")
124
+ with gr.Column():
125
+ audio_output = gr.Audio(label="🔊试听")
126
+ MP3_output = gr.File(label="💾下载")
127
+ text_output = gr.Textbox(label="❗调试信息")
128
+ gr.Markdown("""
129
+
130
+ """)
131
+ btn.click(
132
+ tts_generator,
133
+ inputs=[text, sdp_ratio, noise_scale, noise_scale_w, length_scale, model],
134
+ outputs=[text_output, audio_output,MP3_output]
135
+ )
136
+ gr.Examples(
137
+ fn=tts_generator,
138
+ examples=[
139
+ [
140
+ "我?当警察,上次我说这话的时候才六岁"
141
+ ],
142
+ [
143
+ "但对我来说,回忆中的夜之城反而笼罩在一种暖暖淡淡的,有奶油质感的颜色中。"
144
+ ],
145
+ [
146
+ "与我打卡过的北京其他几家社区图书馆一样,环境那叫一个整洁优雅,工作日那叫一个人烟稀少。书虽不多,但好书不少,而且崭新得烫手。"
147
+ ],
148
+ [
149
+ "《神笔狗良》冒险解谜涂色游戏,对小朋友来说或许有点幼稚但对我来说刚刚好!"
150
+ ],
151
+ [
152
+ "不知道有没有使用过不同读取速度内存卡的姐妹,游戏加载和运行速度会差很多吗?"
153
+ ]
154
+ ,
155
+ ],
156
+ inputs=[text],
157
+ outputs=[audio_output]
158
+ )
159
+ #gr.HTML('''<div align=center><img id="visitor-badge" alt="visitor badge" src="https://visitor-badge.laobi.icu/badge?page_id=Ailyth/DLM" /></div>''')
160
+ app.launch(show_error=True)
161
+
162
+
attentions.py ADDED
@@ -0,0 +1,344 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import copy
2
+ import math
3
+ import torch
4
+ from torch import nn
5
+ from torch.nn import functional as F
6
+
7
+ import commons
8
+ import logging
9
+
10
+ logger = logging.getLogger(__name__)
11
+
12
+ class LayerNorm(nn.Module):
13
+ def __init__(self, channels, eps=1e-5):
14
+ super().__init__()
15
+ self.channels = channels
16
+ self.eps = eps
17
+
18
+ self.gamma = nn.Parameter(torch.ones(channels))
19
+ self.beta = nn.Parameter(torch.zeros(channels))
20
+
21
+ def forward(self, x):
22
+ x = x.transpose(1, -1)
23
+ x = F.layer_norm(x, (self.channels,), self.gamma, self.beta, self.eps)
24
+ return x.transpose(1, -1)
25
+
26
+
27
+
28
+ @torch.jit.script
29
+ def fused_add_tanh_sigmoid_multiply(input_a, input_b, n_channels):
30
+ n_channels_int = n_channels[0]
31
+ in_act = input_a + input_b
32
+ t_act = torch.tanh(in_act[:, :n_channels_int, :])
33
+ s_act = torch.sigmoid(in_act[:, n_channels_int:, :])
34
+ acts = t_act * s_act
35
+ return acts
36
+
37
+ class Encoder(nn.Module):
38
+ def __init__(self, hidden_channels, filter_channels, n_heads, n_layers, kernel_size=1, p_dropout=0., window_size=4, isflow = True, **kwargs):
39
+ super().__init__()
40
+ self.hidden_channels = hidden_channels
41
+ self.filter_channels = filter_channels
42
+ self.n_heads = n_heads
43
+ self.n_layers = n_layers
44
+ self.kernel_size = kernel_size
45
+ self.p_dropout = p_dropout
46
+ self.window_size = window_size
47
+ #if isflow:
48
+ # cond_layer = torch.nn.Conv1d(256, 2*hidden_channels*n_layers, 1)
49
+ # self.cond_pre = torch.nn.Conv1d(hidden_channels, 2*hidden_channels, 1)
50
+ # self.cond_layer = weight_norm(cond_layer, name='weight')
51
+ # self.gin_channels = 256
52
+ self.cond_layer_idx = self.n_layers
53
+ if 'gin_channels' in kwargs:
54
+ self.gin_channels = kwargs['gin_channels']
55
+ if self.gin_channels != 0:
56
+ self.spk_emb_linear = nn.Linear(self.gin_channels, self.hidden_channels)
57
+ # vits2 says 3rd block, so idx is 2 by default
58
+ self.cond_layer_idx = kwargs['cond_layer_idx'] if 'cond_layer_idx' in kwargs else 2
59
+ logging.debug(self.gin_channels, self.cond_layer_idx)
60
+ assert self.cond_layer_idx < self.n_layers, 'cond_layer_idx should be less than n_layers'
61
+ self.drop = nn.Dropout(p_dropout)
62
+ self.attn_layers = nn.ModuleList()
63
+ self.norm_layers_1 = nn.ModuleList()
64
+ self.ffn_layers = nn.ModuleList()
65
+ self.norm_layers_2 = nn.ModuleList()
66
+ for i in range(self.n_layers):
67
+ self.attn_layers.append(MultiHeadAttention(hidden_channels, hidden_channels, n_heads, p_dropout=p_dropout, window_size=window_size))
68
+ self.norm_layers_1.append(LayerNorm(hidden_channels))
69
+ self.ffn_layers.append(FFN(hidden_channels, hidden_channels, filter_channels, kernel_size, p_dropout=p_dropout))
70
+ self.norm_layers_2.append(LayerNorm(hidden_channels))
71
+ def forward(self, x, x_mask, g=None):
72
+ attn_mask = x_mask.unsqueeze(2) * x_mask.unsqueeze(-1)
73
+ x = x * x_mask
74
+ for i in range(self.n_layers):
75
+ if i == self.cond_layer_idx and g is not None:
76
+ g = self.spk_emb_linear(g.transpose(1, 2))
77
+ g = g.transpose(1, 2)
78
+ x = x + g
79
+ x = x * x_mask
80
+ y = self.attn_layers[i](x, x, attn_mask)
81
+ y = self.drop(y)
82
+ x = self.norm_layers_1[i](x + y)
83
+
84
+ y = self.ffn_layers[i](x, x_mask)
85
+ y = self.drop(y)
86
+ x = self.norm_layers_2[i](x + y)
87
+ x = x * x_mask
88
+ return x
89
+
90
+
91
+ class Decoder(nn.Module):
92
+ def __init__(self, hidden_channels, filter_channels, n_heads, n_layers, kernel_size=1, p_dropout=0., proximal_bias=False, proximal_init=True, **kwargs):
93
+ super().__init__()
94
+ self.hidden_channels = hidden_channels
95
+ self.filter_channels = filter_channels
96
+ self.n_heads = n_heads
97
+ self.n_layers = n_layers
98
+ self.kernel_size = kernel_size
99
+ self.p_dropout = p_dropout
100
+ self.proximal_bias = proximal_bias
101
+ self.proximal_init = proximal_init
102
+
103
+ self.drop = nn.Dropout(p_dropout)
104
+ self.self_attn_layers = nn.ModuleList()
105
+ self.norm_layers_0 = nn.ModuleList()
106
+ self.encdec_attn_layers = nn.ModuleList()
107
+ self.norm_layers_1 = nn.ModuleList()
108
+ self.ffn_layers = nn.ModuleList()
109
+ self.norm_layers_2 = nn.ModuleList()
110
+ for i in range(self.n_layers):
111
+ self.self_attn_layers.append(MultiHeadAttention(hidden_channels, hidden_channels, n_heads, p_dropout=p_dropout, proximal_bias=proximal_bias, proximal_init=proximal_init))
112
+ self.norm_layers_0.append(LayerNorm(hidden_channels))
113
+ self.encdec_attn_layers.append(MultiHeadAttention(hidden_channels, hidden_channels, n_heads, p_dropout=p_dropout))
114
+ self.norm_layers_1.append(LayerNorm(hidden_channels))
115
+ self.ffn_layers.append(FFN(hidden_channels, hidden_channels, filter_channels, kernel_size, p_dropout=p_dropout, causal=True))
116
+ self.norm_layers_2.append(LayerNorm(hidden_channels))
117
+
118
+ def forward(self, x, x_mask, h, h_mask):
119
+ """
120
+ x: decoder input
121
+ h: encoder output
122
+ """
123
+ self_attn_mask = commons.subsequent_mask(x_mask.size(2)).to(device=x.device, dtype=x.dtype)
124
+ encdec_attn_mask = h_mask.unsqueeze(2) * x_mask.unsqueeze(-1)
125
+ x = x * x_mask
126
+ for i in range(self.n_layers):
127
+ y = self.self_attn_layers[i](x, x, self_attn_mask)
128
+ y = self.drop(y)
129
+ x = self.norm_layers_0[i](x + y)
130
+
131
+ y = self.encdec_attn_layers[i](x, h, encdec_attn_mask)
132
+ y = self.drop(y)
133
+ x = self.norm_layers_1[i](x + y)
134
+
135
+ y = self.ffn_layers[i](x, x_mask)
136
+ y = self.drop(y)
137
+ x = self.norm_layers_2[i](x + y)
138
+ x = x * x_mask
139
+ return x
140
+
141
+
142
+ class MultiHeadAttention(nn.Module):
143
+ def __init__(self, channels, out_channels, n_heads, p_dropout=0., window_size=None, heads_share=True, block_length=None, proximal_bias=False, proximal_init=False):
144
+ super().__init__()
145
+ assert channels % n_heads == 0
146
+
147
+ self.channels = channels
148
+ self.out_channels = out_channels
149
+ self.n_heads = n_heads
150
+ self.p_dropout = p_dropout
151
+ self.window_size = window_size
152
+ self.heads_share = heads_share
153
+ self.block_length = block_length
154
+ self.proximal_bias = proximal_bias
155
+ self.proximal_init = proximal_init
156
+ self.attn = None
157
+
158
+ self.k_channels = channels // n_heads
159
+ self.conv_q = nn.Conv1d(channels, channels, 1)
160
+ self.conv_k = nn.Conv1d(channels, channels, 1)
161
+ self.conv_v = nn.Conv1d(channels, channels, 1)
162
+ self.conv_o = nn.Conv1d(channels, out_channels, 1)
163
+ self.drop = nn.Dropout(p_dropout)
164
+
165
+ if window_size is not None:
166
+ n_heads_rel = 1 if heads_share else n_heads
167
+ rel_stddev = self.k_channels**-0.5
168
+ self.emb_rel_k = nn.Parameter(torch.randn(n_heads_rel, window_size * 2 + 1, self.k_channels) * rel_stddev)
169
+ self.emb_rel_v = nn.Parameter(torch.randn(n_heads_rel, window_size * 2 + 1, self.k_channels) * rel_stddev)
170
+
171
+ nn.init.xavier_uniform_(self.conv_q.weight)
172
+ nn.init.xavier_uniform_(self.conv_k.weight)
173
+ nn.init.xavier_uniform_(self.conv_v.weight)
174
+ if proximal_init:
175
+ with torch.no_grad():
176
+ self.conv_k.weight.copy_(self.conv_q.weight)
177
+ self.conv_k.bias.copy_(self.conv_q.bias)
178
+
179
+ def forward(self, x, c, attn_mask=None):
180
+ q = self.conv_q(x)
181
+ k = self.conv_k(c)
182
+ v = self.conv_v(c)
183
+
184
+ x, self.attn = self.attention(q, k, v, mask=attn_mask)
185
+
186
+ x = self.conv_o(x)
187
+ return x
188
+
189
+ def attention(self, query, key, value, mask=None):
190
+ # reshape [b, d, t] -> [b, n_h, t, d_k]
191
+ b, d, t_s, t_t = (*key.size(), query.size(2))
192
+ query = query.view(b, self.n_heads, self.k_channels, t_t).transpose(2, 3)
193
+ key = key.view(b, self.n_heads, self.k_channels, t_s).transpose(2, 3)
194
+ value = value.view(b, self.n_heads, self.k_channels, t_s).transpose(2, 3)
195
+
196
+ scores = torch.matmul(query / math.sqrt(self.k_channels), key.transpose(-2, -1))
197
+ if self.window_size is not None:
198
+ assert t_s == t_t, "Relative attention is only available for self-attention."
199
+ key_relative_embeddings = self._get_relative_embeddings(self.emb_rel_k, t_s)
200
+ rel_logits = self._matmul_with_relative_keys(query /math.sqrt(self.k_channels), key_relative_embeddings)
201
+ scores_local = self._relative_position_to_absolute_position(rel_logits)
202
+ scores = scores + scores_local
203
+ if self.proximal_bias:
204
+ assert t_s == t_t, "Proximal bias is only available for self-attention."
205
+ scores = scores + self._attention_bias_proximal(t_s).to(device=scores.device, dtype=scores.dtype)
206
+ if mask is not None:
207
+ scores = scores.masked_fill(mask == 0, -1e4)
208
+ if self.block_length is not None:
209
+ assert t_s == t_t, "Local attention is only available for self-attention."
210
+ block_mask = torch.ones_like(scores).triu(-self.block_length).tril(self.block_length)
211
+ scores = scores.masked_fill(block_mask == 0, -1e4)
212
+ p_attn = F.softmax(scores, dim=-1) # [b, n_h, t_t, t_s]
213
+ p_attn = self.drop(p_attn)
214
+ output = torch.matmul(p_attn, value)
215
+ if self.window_size is not None:
216
+ relative_weights = self._absolute_position_to_relative_position(p_attn)
217
+ value_relative_embeddings = self._get_relative_embeddings(self.emb_rel_v, t_s)
218
+ output = output + self._matmul_with_relative_values(relative_weights, value_relative_embeddings)
219
+ output = output.transpose(2, 3).contiguous().view(b, d, t_t) # [b, n_h, t_t, d_k] -> [b, d, t_t]
220
+ return output, p_attn
221
+
222
+ def _matmul_with_relative_values(self, x, y):
223
+ """
224
+ x: [b, h, l, m]
225
+ y: [h or 1, m, d]
226
+ ret: [b, h, l, d]
227
+ """
228
+ ret = torch.matmul(x, y.unsqueeze(0))
229
+ return ret
230
+
231
+ def _matmul_with_relative_keys(self, x, y):
232
+ """
233
+ x: [b, h, l, d]
234
+ y: [h or 1, m, d]
235
+ ret: [b, h, l, m]
236
+ """
237
+ ret = torch.matmul(x, y.unsqueeze(0).transpose(-2, -1))
238
+ return ret
239
+
240
+ def _get_relative_embeddings(self, relative_embeddings, length):
241
+ max_relative_position = 2 * self.window_size + 1
242
+ # Pad first before slice to avoid using cond ops.
243
+ pad_length = max(length - (self.window_size + 1), 0)
244
+ slice_start_position = max((self.window_size + 1) - length, 0)
245
+ slice_end_position = slice_start_position + 2 * length - 1
246
+ if pad_length > 0:
247
+ padded_relative_embeddings = F.pad(
248
+ relative_embeddings,
249
+ commons.convert_pad_shape([[0, 0], [pad_length, pad_length], [0, 0]]))
250
+ else:
251
+ padded_relative_embeddings = relative_embeddings
252
+ used_relative_embeddings = padded_relative_embeddings[:,slice_start_position:slice_end_position]
253
+ return used_relative_embeddings
254
+
255
+ def _relative_position_to_absolute_position(self, x):
256
+ """
257
+ x: [b, h, l, 2*l-1]
258
+ ret: [b, h, l, l]
259
+ """
260
+ batch, heads, length, _ = x.size()
261
+ # Concat columns of pad to shift from relative to absolute indexing.
262
+ x = F.pad(x, commons.convert_pad_shape([[0,0],[0,0],[0,0],[0,1]]))
263
+
264
+ # Concat extra elements so to add up to shape (len+1, 2*len-1).
265
+ x_flat = x.view([batch, heads, length * 2 * length])
266
+ x_flat = F.pad(x_flat, commons.convert_pad_shape([[0,0],[0,0],[0,length-1]]))
267
+
268
+ # Reshape and slice out the padded elements.
269
+ x_final = x_flat.view([batch, heads, length+1, 2*length-1])[:, :, :length, length-1:]
270
+ return x_final
271
+
272
+ def _absolute_position_to_relative_position(self, x):
273
+ """
274
+ x: [b, h, l, l]
275
+ ret: [b, h, l, 2*l-1]
276
+ """
277
+ batch, heads, length, _ = x.size()
278
+ # padd along column
279
+ x = F.pad(x, commons.convert_pad_shape([[0, 0], [0, 0], [0, 0], [0, length-1]]))
280
+ x_flat = x.view([batch, heads, length**2 + length*(length -1)])
281
+ # add 0's in the beginning that will skew the elements after reshape
282
+ x_flat = F.pad(x_flat, commons.convert_pad_shape([[0, 0], [0, 0], [length, 0]]))
283
+ x_final = x_flat.view([batch, heads, length, 2*length])[:,:,:,1:]
284
+ return x_final
285
+
286
+ def _attention_bias_proximal(self, length):
287
+ """Bias for self-attention to encourage attention to close positions.
288
+ Args:
289
+ length: an integer scalar.
290
+ Returns:
291
+ a Tensor with shape [1, 1, length, length]
292
+ """
293
+ r = torch.arange(length, dtype=torch.float32)
294
+ diff = torch.unsqueeze(r, 0) - torch.unsqueeze(r, 1)
295
+ return torch.unsqueeze(torch.unsqueeze(-torch.log1p(torch.abs(diff)), 0), 0)
296
+
297
+
298
+ class FFN(nn.Module):
299
+ def __init__(self, in_channels, out_channels, filter_channels, kernel_size, p_dropout=0., activation=None, causal=False):
300
+ super().__init__()
301
+ self.in_channels = in_channels
302
+ self.out_channels = out_channels
303
+ self.filter_channels = filter_channels
304
+ self.kernel_size = kernel_size
305
+ self.p_dropout = p_dropout
306
+ self.activation = activation
307
+ self.causal = causal
308
+
309
+ if causal:
310
+ self.padding = self._causal_padding
311
+ else:
312
+ self.padding = self._same_padding
313
+
314
+ self.conv_1 = nn.Conv1d(in_channels, filter_channels, kernel_size)
315
+ self.conv_2 = nn.Conv1d(filter_channels, out_channels, kernel_size)
316
+ self.drop = nn.Dropout(p_dropout)
317
+
318
+ def forward(self, x, x_mask):
319
+ x = self.conv_1(self.padding(x * x_mask))
320
+ if self.activation == "gelu":
321
+ x = x * torch.sigmoid(1.702 * x)
322
+ else:
323
+ x = torch.relu(x)
324
+ x = self.drop(x)
325
+ x = self.conv_2(self.padding(x * x_mask))
326
+ return x * x_mask
327
+
328
+ def _causal_padding(self, x):
329
+ if self.kernel_size == 1:
330
+ return x
331
+ pad_l = self.kernel_size - 1
332
+ pad_r = 0
333
+ padding = [[0, 0], [0, 0], [pad_l, pad_r]]
334
+ x = F.pad(x, commons.convert_pad_shape(padding))
335
+ return x
336
+
337
+ def _same_padding(self, x):
338
+ if self.kernel_size == 1:
339
+ return x
340
+ pad_l = (self.kernel_size - 1) // 2
341
+ pad_r = self.kernel_size // 2
342
+ padding = [[0, 0], [0, 0], [pad_l, pad_r]]
343
+ x = F.pad(x, commons.convert_pad_shape(padding))
344
+ return x
bert_gen.py ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch
2
+ from torch.utils.data import DataLoader
3
+ from multiprocessing import Pool
4
+ import commons
5
+ import utils
6
+ from data_utils import TextAudioSpeakerLoader, TextAudioSpeakerCollate
7
+ from tqdm import tqdm
8
+ import warnings
9
+
10
+ from text import cleaned_text_to_sequence, get_bert
11
+
12
+ config_path = 'configs/config.json'
13
+ hps = utils.get_hparams_from_file(config_path)
14
+
15
+ def process_line(line):
16
+ _id, spk, language_str, text, phones, tone, word2ph = line.strip().split("|")
17
+ phone = phones.split(" ")
18
+ tone = [int(i) for i in tone.split(" ")]
19
+ word2ph = [int(i) for i in word2ph.split(" ")]
20
+ w2pho = [i for i in word2ph]
21
+ word2ph = [i for i in word2ph]
22
+ phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
23
+
24
+ if hps.data.add_blank:
25
+ phone = commons.intersperse(phone, 0)
26
+ tone = commons.intersperse(tone, 0)
27
+ language = commons.intersperse(language, 0)
28
+ for i in range(len(word2ph)):
29
+ word2ph[i] = word2ph[i] * 2
30
+ word2ph[0] += 1
31
+ wav_path = f'{_id}'
32
+
33
+ bert_path = wav_path.replace(".wav", ".bert.pt")
34
+ try:
35
+ bert = torch.load(bert_path)
36
+ assert bert.shape[-1] == len(phone)
37
+ except:
38
+ bert = get_bert(text, word2ph, language_str)
39
+ assert bert.shape[-1] == len(phone)
40
+ torch.save(bert, bert_path)
41
+
42
+
43
+ if __name__ == '__main__':
44
+ lines = []
45
+ with open(hps.data.training_files, encoding='utf-8' ) as f:
46
+ lines.extend(f.readlines())
47
+
48
+ with open(hps.data.validation_files, encoding='utf-8' ) as f:
49
+ lines.extend(f.readlines())
50
+
51
+ with Pool(processes=12) as pool: #A100 40GB suitable config,if coom,please decrease the processess number.
52
+ for _ in tqdm(pool.imap_unordered(process_line, lines)):
53
+ pass
commons.py ADDED
@@ -0,0 +1,161 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import math
2
+ import numpy as np
3
+ import torch
4
+ from torch import nn
5
+ from torch.nn import functional as F
6
+
7
+
8
+ def init_weights(m, mean=0.0, std=0.01):
9
+ classname = m.__class__.__name__
10
+ if classname.find("Conv") != -1:
11
+ m.weight.data.normal_(mean, std)
12
+
13
+
14
+ def get_padding(kernel_size, dilation=1):
15
+ return int((kernel_size*dilation - dilation)/2)
16
+
17
+
18
+ def convert_pad_shape(pad_shape):
19
+ l = pad_shape[::-1]
20
+ pad_shape = [item for sublist in l for item in sublist]
21
+ return pad_shape
22
+
23
+
24
+ def intersperse(lst, item):
25
+ result = [item] * (len(lst) * 2 + 1)
26
+ result[1::2] = lst
27
+ return result
28
+
29
+
30
+ def kl_divergence(m_p, logs_p, m_q, logs_q):
31
+ """KL(P||Q)"""
32
+ kl = (logs_q - logs_p) - 0.5
33
+ kl += 0.5 * (torch.exp(2. * logs_p) + ((m_p - m_q)**2)) * torch.exp(-2. * logs_q)
34
+ return kl
35
+
36
+
37
+ def rand_gumbel(shape):
38
+ """Sample from the Gumbel distribution, protect from overflows."""
39
+ uniform_samples = torch.rand(shape) * 0.99998 + 0.00001
40
+ return -torch.log(-torch.log(uniform_samples))
41
+
42
+
43
+ def rand_gumbel_like(x):
44
+ g = rand_gumbel(x.size()).to(dtype=x.dtype, device=x.device)
45
+ return g
46
+
47
+
48
+ def slice_segments(x, ids_str, segment_size=4):
49
+ ret = torch.zeros_like(x[:, :, :segment_size])
50
+ for i in range(x.size(0)):
51
+ idx_str = ids_str[i]
52
+ idx_end = idx_str + segment_size
53
+ ret[i] = x[i, :, idx_str:idx_end]
54
+ return ret
55
+
56
+
57
+ def rand_slice_segments(x, x_lengths=None, segment_size=4):
58
+ b, d, t = x.size()
59
+ if x_lengths is None:
60
+ x_lengths = t
61
+ ids_str_max = x_lengths - segment_size + 1
62
+ ids_str = (torch.rand([b]).to(device=x.device) * ids_str_max).to(dtype=torch.long)
63
+ ret = slice_segments(x, ids_str, segment_size)
64
+ return ret, ids_str
65
+
66
+
67
+ def get_timing_signal_1d(
68
+ length, channels, min_timescale=1.0, max_timescale=1.0e4):
69
+ position = torch.arange(length, dtype=torch.float)
70
+ num_timescales = channels // 2
71
+ log_timescale_increment = (
72
+ math.log(float(max_timescale) / float(min_timescale)) /
73
+ (num_timescales - 1))
74
+ inv_timescales = min_timescale * torch.exp(
75
+ torch.arange(num_timescales, dtype=torch.float) * -log_timescale_increment)
76
+ scaled_time = position.unsqueeze(0) * inv_timescales.unsqueeze(1)
77
+ signal = torch.cat([torch.sin(scaled_time), torch.cos(scaled_time)], 0)
78
+ signal = F.pad(signal, [0, 0, 0, channels % 2])
79
+ signal = signal.view(1, channels, length)
80
+ return signal
81
+
82
+
83
+ def add_timing_signal_1d(x, min_timescale=1.0, max_timescale=1.0e4):
84
+ b, channels, length = x.size()
85
+ signal = get_timing_signal_1d(length, channels, min_timescale, max_timescale)
86
+ return x + signal.to(dtype=x.dtype, device=x.device)
87
+
88
+
89
+ def cat_timing_signal_1d(x, min_timescale=1.0, max_timescale=1.0e4, axis=1):
90
+ b, channels, length = x.size()
91
+ signal = get_timing_signal_1d(length, channels, min_timescale, max_timescale)
92
+ return torch.cat([x, signal.to(dtype=x.dtype, device=x.device)], axis)
93
+
94
+
95
+ def subsequent_mask(length):
96
+ mask = torch.tril(torch.ones(length, length)).unsqueeze(0).unsqueeze(0)
97
+ return mask
98
+
99
+
100
+ @torch.jit.script
101
+ def fused_add_tanh_sigmoid_multiply(input_a, input_b, n_channels):
102
+ n_channels_int = n_channels[0]
103
+ in_act = input_a + input_b
104
+ t_act = torch.tanh(in_act[:, :n_channels_int, :])
105
+ s_act = torch.sigmoid(in_act[:, n_channels_int:, :])
106
+ acts = t_act * s_act
107
+ return acts
108
+
109
+
110
+ def convert_pad_shape(pad_shape):
111
+ l = pad_shape[::-1]
112
+ pad_shape = [item for sublist in l for item in sublist]
113
+ return pad_shape
114
+
115
+
116
+ def shift_1d(x):
117
+ x = F.pad(x, convert_pad_shape([[0, 0], [0, 0], [1, 0]]))[:, :, :-1]
118
+ return x
119
+
120
+
121
+ def sequence_mask(length, max_length=None):
122
+ if max_length is None:
123
+ max_length = length.max()
124
+ x = torch.arange(max_length, dtype=length.dtype, device=length.device)
125
+ return x.unsqueeze(0) < length.unsqueeze(1)
126
+
127
+
128
+ def generate_path(duration, mask):
129
+ """
130
+ duration: [b, 1, t_x]
131
+ mask: [b, 1, t_y, t_x]
132
+ """
133
+ device = duration.device
134
+
135
+ b, _, t_y, t_x = mask.shape
136
+ cum_duration = torch.cumsum(duration, -1)
137
+
138
+ cum_duration_flat = cum_duration.view(b * t_x)
139
+ path = sequence_mask(cum_duration_flat, t_y).to(mask.dtype)
140
+ path = path.view(b, t_x, t_y)
141
+ path = path - F.pad(path, convert_pad_shape([[0, 0], [1, 0], [0, 0]]))[:, :-1]
142
+ path = path.unsqueeze(1).transpose(2,3) * mask
143
+ return path
144
+
145
+
146
+ def clip_grad_value_(parameters, clip_value, norm_type=2):
147
+ if isinstance(parameters, torch.Tensor):
148
+ parameters = [parameters]
149
+ parameters = list(filter(lambda p: p.grad is not None, parameters))
150
+ norm_type = float(norm_type)
151
+ if clip_value is not None:
152
+ clip_value = float(clip_value)
153
+
154
+ total_norm = 0
155
+ for p in parameters:
156
+ param_norm = p.grad.data.norm(norm_type)
157
+ total_norm += param_norm.item() ** norm_type
158
+ if clip_value is not None:
159
+ p.grad.data.clamp_(min=-clip_value, max=clip_value)
160
+ total_norm = total_norm ** (1. / norm_type)
161
+ return total_norm
configs/config.json ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "train": {
3
+ "log_interval": 10,
4
+ "eval_interval": 100,
5
+ "seed": 52,
6
+ "epochs": 10000,
7
+ "learning_rate": 0.0003,
8
+ "betas": [
9
+ 0.8,
10
+ 0.99
11
+ ],
12
+ "eps": 1e-09,
13
+ "batch_size": 18,
14
+ "fp16_run": false,
15
+ "lr_decay": 0.999875,
16
+ "segment_size": 16384,
17
+ "init_lr_ratio": 1,
18
+ "warmup_epochs": 0,
19
+ "c_mel": 45,
20
+ "c_kl": 1.0
21
+ },
22
+ "data": {
23
+ "use_mel_posterior_encoder": false,
24
+ "training_files": "filelists/train.list",
25
+ "validation_files": "filelists/val.list",
26
+ "max_wav_value": 32768.0,
27
+ "sampling_rate": 44100,
28
+ "filter_length": 2048,
29
+ "hop_length": 512,
30
+ "win_length": 2048,
31
+ "n_mel_channels": 128,
32
+ "mel_fmin": 0.0,
33
+ "mel_fmax": null,
34
+ "add_blank": true,
35
+ "n_speakers": 1,
36
+ "cleaned_text": true,
37
+ "spk2id": {
38
+ "BE_Talk": 0
39
+ }
40
+ },
41
+ "model": {
42
+ "use_spk_conditioned_encoder": true,
43
+ "use_noise_scaled_mas": true,
44
+ "use_mel_posterior_encoder": false,
45
+ "use_duration_discriminator": true,
46
+ "inter_channels": 192,
47
+ "hidden_channels": 192,
48
+ "filter_channels": 768,
49
+ "n_heads": 2,
50
+ "n_layers": 6,
51
+ "kernel_size": 3,
52
+ "p_dropout": 0.1,
53
+ "resblock": "1",
54
+ "resblock_kernel_sizes": [
55
+ 3,
56
+ 7,
57
+ 11
58
+ ],
59
+ "resblock_dilation_sizes": [
60
+ [
61
+ 1,
62
+ 3,
63
+ 5
64
+ ],
65
+ [
66
+ 1,
67
+ 3,
68
+ 5
69
+ ],
70
+ [
71
+ 1,
72
+ 3,
73
+ 5
74
+ ]
75
+ ],
76
+ "upsample_rates": [
77
+ 8,
78
+ 8,
79
+ 2,
80
+ 2,
81
+ 2
82
+ ],
83
+ "upsample_initial_channel": 512,
84
+ "upsample_kernel_sizes": [
85
+ 16,
86
+ 16,
87
+ 8,
88
+ 2,
89
+ 2
90
+ ],
91
+ "n_layers_q": 3,
92
+ "use_spectral_norm": false,
93
+ "gin_channels": 256
94
+ }
95
+ }
data_utils.py ADDED
@@ -0,0 +1,321 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import time
2
+ import os
3
+ import random
4
+ import numpy as np
5
+ import torch
6
+ import torch.utils.data
7
+ import commons
8
+ from mel_processing import spectrogram_torch, mel_spectrogram_torch, spec_to_mel_torch
9
+ from utils import load_wav_to_torch, load_filepaths_and_text
10
+ from text import cleaned_text_to_sequence, get_bert
11
+
12
+ """Multi speaker version"""
13
+
14
+
15
+ class TextAudioSpeakerLoader(torch.utils.data.Dataset):
16
+ """
17
+ 1) loads audio, speaker_id, text pairs
18
+ 2) normalizes text and converts them to sequences of integers
19
+ 3) computes spectrograms from audio files.
20
+ """
21
+
22
+ def __init__(self, audiopaths_sid_text, hparams):
23
+ self.audiopaths_sid_text = load_filepaths_and_text(audiopaths_sid_text)
24
+ self.max_wav_value = hparams.max_wav_value
25
+ self.sampling_rate = hparams.sampling_rate
26
+ self.filter_length = hparams.filter_length
27
+ self.hop_length = hparams.hop_length
28
+ self.win_length = hparams.win_length
29
+ self.sampling_rate = hparams.sampling_rate
30
+ self.spk_map = hparams.spk2id
31
+ self.hparams = hparams
32
+
33
+ self.use_mel_spec_posterior = getattr(hparams, "use_mel_posterior_encoder", False)
34
+ if self.use_mel_spec_posterior:
35
+ self.n_mel_channels = getattr(hparams, "n_mel_channels", 80)
36
+
37
+ self.cleaned_text = getattr(hparams, "cleaned_text", False)
38
+
39
+ self.add_blank = hparams.add_blank
40
+ self.min_text_len = getattr(hparams, "min_text_len", 1)
41
+ self.max_text_len = getattr(hparams, "max_text_len", 300)
42
+
43
+ random.seed(1234)
44
+ random.shuffle(self.audiopaths_sid_text)
45
+ self._filter()
46
+
47
+ def _filter(self):
48
+ """
49
+ Filter text & store spec lengths
50
+ """
51
+ # Store spectrogram lengths for Bucketing
52
+ # wav_length ~= file_size / (wav_channels * Bytes per dim) = file_size / (1 * 2)
53
+ # spec_length = wav_length // hop_length
54
+
55
+ audiopaths_sid_text_new = []
56
+ lengths = []
57
+ skipped = 0
58
+ for _id, spk, language, text, phones, tone, word2ph in self.audiopaths_sid_text:
59
+ audiopath = f'{_id}'
60
+ if self.min_text_len <= len(phones) and len(phones) <= self.max_text_len:
61
+ phones = phones.split(" ")
62
+ tone = [int(i) for i in tone.split(" ")]
63
+ word2ph = [int(i) for i in word2ph.split(" ")]
64
+ audiopaths_sid_text_new.append([audiopath, spk, language, text, phones, tone, word2ph])
65
+ lengths.append(os.path.getsize(audiopath) // (2 * self.hop_length))
66
+ else:
67
+ skipped += 1
68
+ print("skipped: ", skipped, ", total: ", len(self.audiopaths_sid_text))
69
+ self.audiopaths_sid_text = audiopaths_sid_text_new
70
+ self.lengths = lengths
71
+
72
+ def get_audio_text_speaker_pair(self, audiopath_sid_text):
73
+ # separate filename, speaker_id and text
74
+ audiopath, sid, language, text, phones, tone, word2ph = audiopath_sid_text
75
+
76
+ bert, phones, tone, language = self.get_text(text, word2ph, phones, tone, language, audiopath)
77
+
78
+ spec, wav = self.get_audio(audiopath)
79
+ sid = torch.LongTensor([int(self.spk_map[sid])])
80
+ return (phones, spec, wav, sid, tone, language, bert)
81
+
82
+ def get_audio(self, filename):
83
+ audio, sampling_rate = load_wav_to_torch(filename)
84
+ if sampling_rate != self.sampling_rate:
85
+ raise ValueError("{} {} SR doesn't match target {} SR".format(
86
+ sampling_rate, self.sampling_rate))
87
+ audio_norm = audio / self.max_wav_value
88
+ audio_norm = audio_norm.unsqueeze(0)
89
+ spec_filename = filename.replace(".wav", ".spec.pt")
90
+ if self.use_mel_spec_posterior:
91
+ spec_filename = spec_filename.replace(".spec.pt", ".mel.pt")
92
+ try:
93
+ spec = torch.load(spec_filename)
94
+ except:
95
+ if self.use_mel_spec_posterior:
96
+ spec = mel_spectrogram_torch(audio_norm, self.filter_length,
97
+ self.n_mel_channels, self.sampling_rate, self.hop_length,
98
+ self.win_length, self.hparams.mel_fmin, self.hparams.mel_fmax, center=False)
99
+ else:
100
+ spec = spectrogram_torch(audio_norm, self.filter_length,
101
+ self.sampling_rate, self.hop_length, self.win_length,
102
+ center=False)
103
+ spec = torch.squeeze(spec, 0)
104
+ torch.save(spec, spec_filename)
105
+ return spec, audio_norm
106
+
107
+ def get_text(self, text, word2ph, phone, tone, language_str, wav_path):
108
+ pold = phone
109
+ w2pho = [i for i in word2ph]
110
+ word2ph = [i for i in word2ph]
111
+ phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
112
+ pold2 = phone
113
+
114
+ if self.add_blank:
115
+ p1 = len(phone)
116
+ phone = commons.intersperse(phone, 0)
117
+ p2 = len(phone)
118
+ t1 = len(tone)
119
+ tone = commons.intersperse(tone, 0)
120
+ t2 = len(tone)
121
+ language = commons.intersperse(language, 0)
122
+ for i in range(len(word2ph)):
123
+ word2ph[i] = word2ph[i] * 2
124
+ word2ph[0] += 1
125
+ bert_path = wav_path.replace(".wav", ".bert.pt")
126
+ try:
127
+ bert = torch.load(bert_path)
128
+ assert bert.shape[-1] == len(phone)
129
+ except:
130
+ bert = get_bert(text, word2ph, language_str)
131
+ torch.save(bert, bert_path)
132
+ #print(bert.shape[-1], bert_path, text, pold)
133
+ assert bert.shape[-1] == len(phone)
134
+
135
+ assert bert.shape[-1] == len(phone), (
136
+ bert.shape, len(phone), sum(word2ph), p1, p2, t1, t2, pold, pold2, word2ph, text, w2pho)
137
+ phone = torch.LongTensor(phone)
138
+ tone = torch.LongTensor(tone)
139
+ language = torch.LongTensor(language)
140
+ return bert, phone, tone, language
141
+
142
+ def get_sid(self, sid):
143
+ sid = torch.LongTensor([int(sid)])
144
+ return sid
145
+
146
+ def __getitem__(self, index):
147
+ return self.get_audio_text_speaker_pair(self.audiopaths_sid_text[index])
148
+
149
+ def __len__(self):
150
+ return len(self.audiopaths_sid_text)
151
+
152
+
153
+ class TextAudioSpeakerCollate():
154
+ """ Zero-pads model inputs and targets
155
+ """
156
+
157
+ def __init__(self, return_ids=False):
158
+ self.return_ids = return_ids
159
+
160
+ def __call__(self, batch):
161
+ """Collate's training batch from normalized text, audio and speaker identities
162
+ PARAMS
163
+ ------
164
+ batch: [text_normalized, spec_normalized, wav_normalized, sid]
165
+ """
166
+ # Right zero-pad all one-hot text sequences to max input length
167
+ _, ids_sorted_decreasing = torch.sort(
168
+ torch.LongTensor([x[1].size(1) for x in batch]),
169
+ dim=0, descending=True)
170
+
171
+ max_text_len = max([len(x[0]) for x in batch])
172
+ max_spec_len = max([x[1].size(1) for x in batch])
173
+ max_wav_len = max([x[2].size(1) for x in batch])
174
+
175
+ text_lengths = torch.LongTensor(len(batch))
176
+ spec_lengths = torch.LongTensor(len(batch))
177
+ wav_lengths = torch.LongTensor(len(batch))
178
+ sid = torch.LongTensor(len(batch))
179
+
180
+ text_padded = torch.LongTensor(len(batch), max_text_len)
181
+ tone_padded = torch.LongTensor(len(batch), max_text_len)
182
+ language_padded = torch.LongTensor(len(batch), max_text_len)
183
+ bert_padded = torch.FloatTensor(len(batch), 1024, max_text_len)
184
+
185
+ spec_padded = torch.FloatTensor(len(batch), batch[0][1].size(0), max_spec_len)
186
+ wav_padded = torch.FloatTensor(len(batch), 1, max_wav_len)
187
+ text_padded.zero_()
188
+ tone_padded.zero_()
189
+ language_padded.zero_()
190
+ spec_padded.zero_()
191
+ wav_padded.zero_()
192
+ bert_padded.zero_()
193
+ for i in range(len(ids_sorted_decreasing)):
194
+ row = batch[ids_sorted_decreasing[i]]
195
+
196
+ text = row[0]
197
+ text_padded[i, :text.size(0)] = text
198
+ text_lengths[i] = text.size(0)
199
+
200
+ spec = row[1]
201
+ spec_padded[i, :, :spec.size(1)] = spec
202
+ spec_lengths[i] = spec.size(1)
203
+
204
+ wav = row[2]
205
+ wav_padded[i, :, :wav.size(1)] = wav
206
+ wav_lengths[i] = wav.size(1)
207
+
208
+ sid[i] = row[3]
209
+
210
+ tone = row[4]
211
+ tone_padded[i, :tone.size(0)] = tone
212
+
213
+ language = row[5]
214
+ language_padded[i, :language.size(0)] = language
215
+
216
+ bert = row[6]
217
+ bert_padded[i, :, :bert.size(1)] = bert
218
+
219
+ return text_padded, text_lengths, spec_padded, spec_lengths, wav_padded, wav_lengths, sid, tone_padded, language_padded, bert_padded
220
+
221
+
222
+ class DistributedBucketSampler(torch.utils.data.distributed.DistributedSampler):
223
+ """
224
+ Maintain similar input lengths in a batch.
225
+ Length groups are specified by boundaries.
226
+ Ex) boundaries = [b1, b2, b3] -> any batch is included either {x | b1 < length(x) <=b2} or {x | b2 < length(x) <= b3}.
227
+
228
+ It removes samples which are not included in the boundaries.
229
+ Ex) boundaries = [b1, b2, b3] -> any x s.t. length(x) <= b1 or length(x) > b3 are discarded.
230
+ """
231
+
232
+ def __init__(self, dataset, batch_size, boundaries, num_replicas=None, rank=None, shuffle=True):
233
+ super().__init__(dataset, num_replicas=num_replicas, rank=rank, shuffle=shuffle)
234
+ self.lengths = dataset.lengths
235
+ self.batch_size = batch_size
236
+ self.boundaries = boundaries
237
+
238
+ self.buckets, self.num_samples_per_bucket = self._create_buckets()
239
+ self.total_size = sum(self.num_samples_per_bucket)
240
+ self.num_samples = self.total_size // self.num_replicas
241
+
242
+ def _create_buckets(self):
243
+ buckets = [[] for _ in range(len(self.boundaries) - 1)]
244
+ for i in range(len(self.lengths)):
245
+ length = self.lengths[i]
246
+ idx_bucket = self._bisect(length)
247
+ if idx_bucket != -1:
248
+ buckets[idx_bucket].append(i)
249
+
250
+ for i in range(len(buckets) - 1, 0, -1):
251
+ if len(buckets[i]) == 0:
252
+ buckets.pop(i)
253
+ self.boundaries.pop(i + 1)
254
+
255
+ num_samples_per_bucket = []
256
+ for i in range(len(buckets)):
257
+ len_bucket = len(buckets[i])
258
+ total_batch_size = self.num_replicas * self.batch_size
259
+ rem = (total_batch_size - (len_bucket % total_batch_size)) % total_batch_size
260
+ num_samples_per_bucket.append(len_bucket + rem)
261
+ return buckets, num_samples_per_bucket
262
+
263
+ def __iter__(self):
264
+ # deterministically shuffle based on epoch
265
+ g = torch.Generator()
266
+ g.manual_seed(self.epoch)
267
+
268
+ indices = []
269
+ if self.shuffle:
270
+ for bucket in self.buckets:
271
+ indices.append(torch.randperm(len(bucket), generator=g).tolist())
272
+ else:
273
+ for bucket in self.buckets:
274
+ indices.append(list(range(len(bucket))))
275
+
276
+ batches = []
277
+ for i in range(len(self.buckets)):
278
+ bucket = self.buckets[i]
279
+ len_bucket = len(bucket)
280
+ if (len_bucket == 0):
281
+ continue
282
+ ids_bucket = indices[i]
283
+ num_samples_bucket = self.num_samples_per_bucket[i]
284
+
285
+ # add extra samples to make it evenly divisible
286
+ rem = num_samples_bucket - len_bucket
287
+ ids_bucket = ids_bucket + ids_bucket * (rem // len_bucket) + ids_bucket[:(rem % len_bucket)]
288
+
289
+ # subsample
290
+ ids_bucket = ids_bucket[self.rank::self.num_replicas]
291
+
292
+ # batching
293
+ for j in range(len(ids_bucket) // self.batch_size):
294
+ batch = [bucket[idx] for idx in ids_bucket[j * self.batch_size:(j + 1) * self.batch_size]]
295
+ batches.append(batch)
296
+
297
+ if self.shuffle:
298
+ batch_ids = torch.randperm(len(batches), generator=g).tolist()
299
+ batches = [batches[i] for i in batch_ids]
300
+ self.batches = batches
301
+
302
+ assert len(self.batches) * self.batch_size == self.num_samples
303
+ return iter(self.batches)
304
+
305
+ def _bisect(self, x, lo=0, hi=None):
306
+ if hi is None:
307
+ hi = len(self.boundaries) - 1
308
+
309
+ if hi > lo:
310
+ mid = (hi + lo) // 2
311
+ if self.boundaries[mid] < x and x <= self.boundaries[mid + 1]:
312
+ return mid
313
+ elif x <= self.boundaries[mid]:
314
+ return self._bisect(x, lo, mid)
315
+ else:
316
+ return self._bisect(x, mid + 1, hi)
317
+ else:
318
+ return -1
319
+
320
+ def __len__(self):
321
+ return self.num_samples // self.batch_size
losses.py ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch
2
+ from torch.nn import functional as F
3
+
4
+ import commons
5
+
6
+
7
+ def feature_loss(fmap_r, fmap_g):
8
+ loss = 0
9
+ for dr, dg in zip(fmap_r, fmap_g):
10
+ for rl, gl in zip(dr, dg):
11
+ rl = rl.float().detach()
12
+ gl = gl.float()
13
+ loss += torch.mean(torch.abs(rl - gl))
14
+
15
+ return loss * 2
16
+
17
+
18
+ def discriminator_loss(disc_real_outputs, disc_generated_outputs):
19
+ loss = 0
20
+ r_losses = []
21
+ g_losses = []
22
+ for dr, dg in zip(disc_real_outputs, disc_generated_outputs):
23
+ dr = dr.float()
24
+ dg = dg.float()
25
+ r_loss = torch.mean((1-dr)**2)
26
+ g_loss = torch.mean(dg**2)
27
+ loss += (r_loss + g_loss)
28
+ r_losses.append(r_loss.item())
29
+ g_losses.append(g_loss.item())
30
+
31
+ return loss, r_losses, g_losses
32
+
33
+
34
+ def generator_loss(disc_outputs):
35
+ loss = 0
36
+ gen_losses = []
37
+ for dg in disc_outputs:
38
+ dg = dg.float()
39
+ l = torch.mean((1-dg)**2)
40
+ gen_losses.append(l)
41
+ loss += l
42
+
43
+ return loss, gen_losses
44
+
45
+
46
+ def kl_loss(z_p, logs_q, m_p, logs_p, z_mask):
47
+ """
48
+ z_p, logs_q: [b, h, t_t]
49
+ m_p, logs_p: [b, h, t_t]
50
+ """
51
+ z_p = z_p.float()
52
+ logs_q = logs_q.float()
53
+ m_p = m_p.float()
54
+ logs_p = logs_p.float()
55
+ z_mask = z_mask.float()
56
+
57
+ kl = logs_p - logs_q - 0.5
58
+ kl += 0.5 * ((z_p - m_p)**2) * torch.exp(-2. * logs_p)
59
+ kl = torch.sum(kl * z_mask)
60
+ l = kl / torch.sum(z_mask)
61
+ return l
mel_processing.py ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import math
2
+ import os
3
+ import random
4
+ import torch
5
+ from torch import nn
6
+ import torch.nn.functional as F
7
+ import torch.utils.data
8
+ import numpy as np
9
+ import librosa
10
+ import librosa.util as librosa_util
11
+ from librosa.util import normalize, pad_center, tiny
12
+ from scipy.signal import get_window
13
+ from scipy.io.wavfile import read
14
+ from librosa.filters import mel as librosa_mel_fn
15
+
16
+ MAX_WAV_VALUE = 32768.0
17
+
18
+
19
+ def dynamic_range_compression_torch(x, C=1, clip_val=1e-5):
20
+ """
21
+ PARAMS
22
+ ------
23
+ C: compression factor
24
+ """
25
+ return torch.log(torch.clamp(x, min=clip_val) * C)
26
+
27
+
28
+ def dynamic_range_decompression_torch(x, C=1):
29
+ """
30
+ PARAMS
31
+ ------
32
+ C: compression factor used to compress
33
+ """
34
+ return torch.exp(x) / C
35
+
36
+
37
+ def spectral_normalize_torch(magnitudes):
38
+ output = dynamic_range_compression_torch(magnitudes)
39
+ return output
40
+
41
+
42
+ def spectral_de_normalize_torch(magnitudes):
43
+ output = dynamic_range_decompression_torch(magnitudes)
44
+ return output
45
+
46
+
47
+ mel_basis = {}
48
+ hann_window = {}
49
+
50
+
51
+ def spectrogram_torch(y, n_fft, sampling_rate, hop_size, win_size, center=False):
52
+ if torch.min(y) < -1.:
53
+ print('min value is ', torch.min(y))
54
+ if torch.max(y) > 1.:
55
+ print('max value is ', torch.max(y))
56
+
57
+ global hann_window
58
+ dtype_device = str(y.dtype) + '_' + str(y.device)
59
+ wnsize_dtype_device = str(win_size) + '_' + dtype_device
60
+ if wnsize_dtype_device not in hann_window:
61
+ hann_window[wnsize_dtype_device] = torch.hann_window(win_size).to(dtype=y.dtype, device=y.device)
62
+
63
+ y = torch.nn.functional.pad(y.unsqueeze(1), (int((n_fft-hop_size)/2), int((n_fft-hop_size)/2)), mode='reflect')
64
+ y = y.squeeze(1)
65
+
66
+ spec = torch.stft(y, n_fft, hop_length=hop_size, win_length=win_size, window=hann_window[wnsize_dtype_device],
67
+ center=center, pad_mode='reflect', normalized=False, onesided=True, return_complex=False)
68
+
69
+ spec = torch.sqrt(spec.pow(2).sum(-1) + 1e-6)
70
+ return spec
71
+
72
+
73
+ def spec_to_mel_torch(spec, n_fft, num_mels, sampling_rate, fmin, fmax):
74
+ global mel_basis
75
+ dtype_device = str(spec.dtype) + '_' + str(spec.device)
76
+ fmax_dtype_device = str(fmax) + '_' + dtype_device
77
+ if fmax_dtype_device not in mel_basis:
78
+ mel = librosa_mel_fn(sampling_rate, n_fft, num_mels, fmin, fmax)
79
+ mel_basis[fmax_dtype_device] = torch.from_numpy(mel).to(dtype=spec.dtype, device=spec.device)
80
+ spec = torch.matmul(mel_basis[fmax_dtype_device], spec)
81
+ spec = spectral_normalize_torch(spec)
82
+ return spec
83
+
84
+
85
+ def mel_spectrogram_torch(y, n_fft, num_mels, sampling_rate, hop_size, win_size, fmin, fmax, center=False):
86
+ if torch.min(y) < -1.:
87
+ print('min value is ', torch.min(y))
88
+ if torch.max(y) > 1.:
89
+ print('max value is ', torch.max(y))
90
+
91
+ global mel_basis, hann_window
92
+ dtype_device = str(y.dtype) + '_' + str(y.device)
93
+ fmax_dtype_device = str(fmax) + '_' + dtype_device
94
+ wnsize_dtype_device = str(win_size) + '_' + dtype_device
95
+ if fmax_dtype_device not in mel_basis:
96
+ mel = librosa_mel_fn(sampling_rate, n_fft, num_mels, fmin, fmax)
97
+ mel_basis[fmax_dtype_device] = torch.from_numpy(mel).to(dtype=y.dtype, device=y.device)
98
+ if wnsize_dtype_device not in hann_window:
99
+ hann_window[wnsize_dtype_device] = torch.hann_window(win_size).to(dtype=y.dtype, device=y.device)
100
+
101
+ y = torch.nn.functional.pad(y.unsqueeze(1), (int((n_fft-hop_size)/2), int((n_fft-hop_size)/2)), mode='reflect')
102
+ y = y.squeeze(1)
103
+
104
+ spec = torch.stft(y, n_fft, hop_length=hop_size, win_length=win_size, window=hann_window[wnsize_dtype_device],
105
+ center=center, pad_mode='reflect', normalized=False, onesided=True, return_complex=False)
106
+
107
+ spec = torch.sqrt(spec.pow(2).sum(-1) + 1e-6)
108
+
109
+ spec = torch.matmul(mel_basis[fmax_dtype_device], spec)
110
+ spec = spectral_normalize_torch(spec)
111
+
112
+ return spec
models.py ADDED
@@ -0,0 +1,707 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import copy
2
+ import math
3
+ import torch
4
+ from torch import nn
5
+ from torch.nn import functional as F
6
+
7
+ import commons
8
+ import modules
9
+ import attentions
10
+ import monotonic_align
11
+
12
+ from torch.nn import Conv1d, ConvTranspose1d, AvgPool1d, Conv2d
13
+ from torch.nn.utils import weight_norm, remove_weight_norm, spectral_norm
14
+
15
+ from commons import init_weights, get_padding
16
+ from text import symbols, num_tones, num_languages
17
+ class DurationDiscriminator(nn.Module): #vits2
18
+ def __init__(self, in_channels, filter_channels, kernel_size, p_dropout, gin_channels=0):
19
+ super().__init__()
20
+
21
+ self.in_channels = in_channels
22
+ self.filter_channels = filter_channels
23
+ self.kernel_size = kernel_size
24
+ self.p_dropout = p_dropout
25
+ self.gin_channels = gin_channels
26
+
27
+ self.drop = nn.Dropout(p_dropout)
28
+ self.conv_1 = nn.Conv1d(in_channels, filter_channels, kernel_size, padding=kernel_size//2)
29
+ self.norm_1 = modules.LayerNorm(filter_channels)
30
+ self.conv_2 = nn.Conv1d(filter_channels, filter_channels, kernel_size, padding=kernel_size//2)
31
+ self.norm_2 = modules.LayerNorm(filter_channels)
32
+ self.dur_proj = nn.Conv1d(1, filter_channels, 1)
33
+
34
+ self.pre_out_conv_1 = nn.Conv1d(2*filter_channels, filter_channels, kernel_size, padding=kernel_size//2)
35
+ self.pre_out_norm_1 = modules.LayerNorm(filter_channels)
36
+ self.pre_out_conv_2 = nn.Conv1d(filter_channels, filter_channels, kernel_size, padding=kernel_size//2)
37
+ self.pre_out_norm_2 = modules.LayerNorm(filter_channels)
38
+
39
+ if gin_channels != 0:
40
+ self.cond = nn.Conv1d(gin_channels, in_channels, 1)
41
+
42
+ self.output_layer = nn.Sequential(
43
+ nn.Linear(filter_channels, 1),
44
+ nn.Sigmoid()
45
+ )
46
+
47
+ def forward_probability(self, x, x_mask, dur, g=None):
48
+ dur = self.dur_proj(dur)
49
+ x = torch.cat([x, dur], dim=1)
50
+ x = self.pre_out_conv_1(x * x_mask)
51
+ x = torch.relu(x)
52
+ x = self.pre_out_norm_1(x)
53
+ x = self.drop(x)
54
+ x = self.pre_out_conv_2(x * x_mask)
55
+ x = torch.relu(x)
56
+ x = self.pre_out_norm_2(x)
57
+ x = self.drop(x)
58
+ x = x * x_mask
59
+ x = x.transpose(1, 2)
60
+ output_prob = self.output_layer(x)
61
+ return output_prob
62
+
63
+ def forward(self, x, x_mask, dur_r, dur_hat, g=None):
64
+ x = torch.detach(x)
65
+ if g is not None:
66
+ g = torch.detach(g)
67
+ x = x + self.cond(g)
68
+ x = self.conv_1(x * x_mask)
69
+ x = torch.relu(x)
70
+ x = self.norm_1(x)
71
+ x = self.drop(x)
72
+ x = self.conv_2(x * x_mask)
73
+ x = torch.relu(x)
74
+ x = self.norm_2(x)
75
+ x = self.drop(x)
76
+
77
+ output_probs = []
78
+ for dur in [dur_r, dur_hat]:
79
+ output_prob = self.forward_probability(x, x_mask, dur, g)
80
+ output_probs.append(output_prob)
81
+
82
+ return output_probs
83
+
84
+ class TransformerCouplingBlock(nn.Module):
85
+ def __init__(self,
86
+ channels,
87
+ hidden_channels,
88
+ filter_channels,
89
+ n_heads,
90
+ n_layers,
91
+ kernel_size,
92
+ p_dropout,
93
+ n_flows=4,
94
+ gin_channels=0,
95
+ share_parameter=False
96
+ ):
97
+
98
+ super().__init__()
99
+ self.channels = channels
100
+ self.hidden_channels = hidden_channels
101
+ self.kernel_size = kernel_size
102
+ self.n_layers = n_layers
103
+ self.n_flows = n_flows
104
+ self.gin_channels = gin_channels
105
+
106
+ self.flows = nn.ModuleList()
107
+
108
+ self.wn = attentions.FFT(hidden_channels, filter_channels, n_heads, n_layers, kernel_size, p_dropout, isflow = True, gin_channels = self.gin_channels) if share_parameter else None
109
+
110
+ for i in range(n_flows):
111
+ self.flows.append(
112
+ modules.TransformerCouplingLayer(channels, hidden_channels, kernel_size, n_layers, n_heads, p_dropout, filter_channels, mean_only=True, wn_sharing_parameter=self.wn, gin_channels = self.gin_channels))
113
+ self.flows.append(modules.Flip())
114
+
115
+ def forward(self, x, x_mask, g=None, reverse=False):
116
+ if not reverse:
117
+ for flow in self.flows:
118
+ x, _ = flow(x, x_mask, g=g, reverse=reverse)
119
+ else:
120
+ for flow in reversed(self.flows):
121
+ x = flow(x, x_mask, g=g, reverse=reverse)
122
+ return x
123
+
124
+ class StochasticDurationPredictor(nn.Module):
125
+ def __init__(self, in_channels, filter_channels, kernel_size, p_dropout, n_flows=4, gin_channels=0):
126
+ super().__init__()
127
+ filter_channels = in_channels # it needs to be removed from future version.
128
+ self.in_channels = in_channels
129
+ self.filter_channels = filter_channels
130
+ self.kernel_size = kernel_size
131
+ self.p_dropout = p_dropout
132
+ self.n_flows = n_flows
133
+ self.gin_channels = gin_channels
134
+
135
+ self.log_flow = modules.Log()
136
+ self.flows = nn.ModuleList()
137
+ self.flows.append(modules.ElementwiseAffine(2))
138
+ for i in range(n_flows):
139
+ self.flows.append(modules.ConvFlow(2, filter_channels, kernel_size, n_layers=3))
140
+ self.flows.append(modules.Flip())
141
+
142
+ self.post_pre = nn.Conv1d(1, filter_channels, 1)
143
+ self.post_proj = nn.Conv1d(filter_channels, filter_channels, 1)
144
+ self.post_convs = modules.DDSConv(filter_channels, kernel_size, n_layers=3, p_dropout=p_dropout)
145
+ self.post_flows = nn.ModuleList()
146
+ self.post_flows.append(modules.ElementwiseAffine(2))
147
+ for i in range(4):
148
+ self.post_flows.append(modules.ConvFlow(2, filter_channels, kernel_size, n_layers=3))
149
+ self.post_flows.append(modules.Flip())
150
+
151
+ self.pre = nn.Conv1d(in_channels, filter_channels, 1)
152
+ self.proj = nn.Conv1d(filter_channels, filter_channels, 1)
153
+ self.convs = modules.DDSConv(filter_channels, kernel_size, n_layers=3, p_dropout=p_dropout)
154
+ if gin_channels != 0:
155
+ self.cond = nn.Conv1d(gin_channels, filter_channels, 1)
156
+
157
+ def forward(self, x, x_mask, w=None, g=None, reverse=False, noise_scale=1.0):
158
+ x = torch.detach(x)
159
+ x = self.pre(x)
160
+ if g is not None:
161
+ g = torch.detach(g)
162
+ x = x + self.cond(g)
163
+ x = self.convs(x, x_mask)
164
+ x = self.proj(x) * x_mask
165
+
166
+ if not reverse:
167
+ flows = self.flows
168
+ assert w is not None
169
+
170
+ logdet_tot_q = 0
171
+ h_w = self.post_pre(w)
172
+ h_w = self.post_convs(h_w, x_mask)
173
+ h_w = self.post_proj(h_w) * x_mask
174
+ e_q = torch.randn(w.size(0), 2, w.size(2)).to(device=x.device, dtype=x.dtype) * x_mask
175
+ z_q = e_q
176
+ for flow in self.post_flows:
177
+ z_q, logdet_q = flow(z_q, x_mask, g=(x + h_w))
178
+ logdet_tot_q += logdet_q
179
+ z_u, z1 = torch.split(z_q, [1, 1], 1)
180
+ u = torch.sigmoid(z_u) * x_mask
181
+ z0 = (w - u) * x_mask
182
+ logdet_tot_q += torch.sum((F.logsigmoid(z_u) + F.logsigmoid(-z_u)) * x_mask, [1, 2])
183
+ logq = torch.sum(-0.5 * (math.log(2 * math.pi) + (e_q ** 2)) * x_mask, [1, 2]) - logdet_tot_q
184
+
185
+ logdet_tot = 0
186
+ z0, logdet = self.log_flow(z0, x_mask)
187
+ logdet_tot += logdet
188
+ z = torch.cat([z0, z1], 1)
189
+ for flow in flows:
190
+ z, logdet = flow(z, x_mask, g=x, reverse=reverse)
191
+ logdet_tot = logdet_tot + logdet
192
+ nll = torch.sum(0.5 * (math.log(2 * math.pi) + (z ** 2)) * x_mask, [1, 2]) - logdet_tot
193
+ return nll + logq # [b]
194
+ else:
195
+ flows = list(reversed(self.flows))
196
+ flows = flows[:-2] + [flows[-1]] # remove a useless vflow
197
+ z = torch.randn(x.size(0), 2, x.size(2)).to(device=x.device, dtype=x.dtype) * noise_scale
198
+ for flow in flows:
199
+ z = flow(z, x_mask, g=x, reverse=reverse)
200
+ z0, z1 = torch.split(z, [1, 1], 1)
201
+ logw = z0
202
+ return logw
203
+
204
+
205
+ class DurationPredictor(nn.Module):
206
+ def __init__(self, in_channels, filter_channels, kernel_size, p_dropout, gin_channels=0):
207
+ super().__init__()
208
+
209
+ self.in_channels = in_channels
210
+ self.filter_channels = filter_channels
211
+ self.kernel_size = kernel_size
212
+ self.p_dropout = p_dropout
213
+ self.gin_channels = gin_channels
214
+
215
+ self.drop = nn.Dropout(p_dropout)
216
+ self.conv_1 = nn.Conv1d(in_channels, filter_channels, kernel_size, padding=kernel_size // 2)
217
+ self.norm_1 = modules.LayerNorm(filter_channels)
218
+ self.conv_2 = nn.Conv1d(filter_channels, filter_channels, kernel_size, padding=kernel_size // 2)
219
+ self.norm_2 = modules.LayerNorm(filter_channels)
220
+ self.proj = nn.Conv1d(filter_channels, 1, 1)
221
+
222
+ if gin_channels != 0:
223
+ self.cond = nn.Conv1d(gin_channels, in_channels, 1)
224
+
225
+ def forward(self, x, x_mask, g=None):
226
+ x = torch.detach(x)
227
+ if g is not None:
228
+ g = torch.detach(g)
229
+ x = x + self.cond(g)
230
+ x = self.conv_1(x * x_mask)
231
+ x = torch.relu(x)
232
+ x = self.norm_1(x)
233
+ x = self.drop(x)
234
+ x = self.conv_2(x * x_mask)
235
+ x = torch.relu(x)
236
+ x = self.norm_2(x)
237
+ x = self.drop(x)
238
+ x = self.proj(x * x_mask)
239
+ return x * x_mask
240
+
241
+
242
+ class TextEncoder(nn.Module):
243
+ def __init__(self,
244
+ n_vocab,
245
+ out_channels,
246
+ hidden_channels,
247
+ filter_channels,
248
+ n_heads,
249
+ n_layers,
250
+ kernel_size,
251
+ p_dropout,
252
+ gin_channels=0):
253
+ super().__init__()
254
+ self.n_vocab = n_vocab
255
+ self.out_channels = out_channels
256
+ self.hidden_channels = hidden_channels
257
+ self.filter_channels = filter_channels
258
+ self.n_heads = n_heads
259
+ self.n_layers = n_layers
260
+ self.kernel_size = kernel_size
261
+ self.p_dropout = p_dropout
262
+ self.gin_channels = gin_channels
263
+ self.emb = nn.Embedding(len(symbols), hidden_channels)
264
+ nn.init.normal_(self.emb.weight, 0.0, hidden_channels ** -0.5)
265
+ self.tone_emb = nn.Embedding(num_tones, hidden_channels)
266
+ nn.init.normal_(self.tone_emb.weight, 0.0, hidden_channels ** -0.5)
267
+ self.language_emb = nn.Embedding(num_languages, hidden_channels)
268
+ nn.init.normal_(self.language_emb.weight, 0.0, hidden_channels ** -0.5)
269
+ self.bert_proj = nn.Conv1d(1024, hidden_channels, 1)
270
+
271
+ self.encoder = attentions.Encoder(
272
+ hidden_channels,
273
+ filter_channels,
274
+ n_heads,
275
+ n_layers,
276
+ kernel_size,
277
+ p_dropout,
278
+ gin_channels=self.gin_channels)
279
+ self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
280
+
281
+ def forward(self, x, x_lengths, tone, language, bert, g=None):
282
+ x = (self.emb(x)+ self.tone_emb(tone)+ self.language_emb(language)+self.bert_proj(bert).transpose(1,2)) * math.sqrt(self.hidden_channels) # [b, t, h]
283
+ x = torch.transpose(x, 1, -1) # [b, h, t]
284
+ x_mask = torch.unsqueeze(commons.sequence_mask(x_lengths, x.size(2)), 1).to(x.dtype)
285
+
286
+ x = self.encoder(x * x_mask, x_mask, g=g)
287
+ stats = self.proj(x) * x_mask
288
+
289
+ m, logs = torch.split(stats, self.out_channels, dim=1)
290
+ return x, m, logs, x_mask
291
+
292
+
293
+ class ResidualCouplingBlock(nn.Module):
294
+ def __init__(self,
295
+ channels,
296
+ hidden_channels,
297
+ kernel_size,
298
+ dilation_rate,
299
+ n_layers,
300
+ n_flows=4,
301
+ gin_channels=0):
302
+ super().__init__()
303
+ self.channels = channels
304
+ self.hidden_channels = hidden_channels
305
+ self.kernel_size = kernel_size
306
+ self.dilation_rate = dilation_rate
307
+ self.n_layers = n_layers
308
+ self.n_flows = n_flows
309
+ self.gin_channels = gin_channels
310
+
311
+ self.flows = nn.ModuleList()
312
+ for i in range(n_flows):
313
+ self.flows.append(
314
+ modules.ResidualCouplingLayer(channels, hidden_channels, kernel_size, dilation_rate, n_layers,
315
+ gin_channels=gin_channels, mean_only=True))
316
+ self.flows.append(modules.Flip())
317
+
318
+ def forward(self, x, x_mask, g=None, reverse=False):
319
+ if not reverse:
320
+ for flow in self.flows:
321
+ x, _ = flow(x, x_mask, g=g, reverse=reverse)
322
+ else:
323
+ for flow in reversed(self.flows):
324
+ x = flow(x, x_mask, g=g, reverse=reverse)
325
+ return x
326
+
327
+
328
+ class PosteriorEncoder(nn.Module):
329
+ def __init__(self,
330
+ in_channels,
331
+ out_channels,
332
+ hidden_channels,
333
+ kernel_size,
334
+ dilation_rate,
335
+ n_layers,
336
+ gin_channels=0):
337
+ super().__init__()
338
+ self.in_channels = in_channels
339
+ self.out_channels = out_channels
340
+ self.hidden_channels = hidden_channels
341
+ self.kernel_size = kernel_size
342
+ self.dilation_rate = dilation_rate
343
+ self.n_layers = n_layers
344
+ self.gin_channels = gin_channels
345
+
346
+ self.pre = nn.Conv1d(in_channels, hidden_channels, 1)
347
+ self.enc = modules.WN(hidden_channels, kernel_size, dilation_rate, n_layers, gin_channels=gin_channels)
348
+ self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
349
+
350
+ def forward(self, x, x_lengths, g=None):
351
+ x_mask = torch.unsqueeze(commons.sequence_mask(x_lengths, x.size(2)), 1).to(x.dtype)
352
+ x = self.pre(x) * x_mask
353
+ x = self.enc(x, x_mask, g=g)
354
+ stats = self.proj(x) * x_mask
355
+ m, logs = torch.split(stats, self.out_channels, dim=1)
356
+ z = (m + torch.randn_like(m) * torch.exp(logs)) * x_mask
357
+ return z, m, logs, x_mask
358
+
359
+
360
+ class Generator(torch.nn.Module):
361
+ def __init__(self, initial_channel, resblock, resblock_kernel_sizes, resblock_dilation_sizes, upsample_rates,
362
+ upsample_initial_channel, upsample_kernel_sizes, gin_channels=0):
363
+ super(Generator, self).__init__()
364
+ self.num_kernels = len(resblock_kernel_sizes)
365
+ self.num_upsamples = len(upsample_rates)
366
+ self.conv_pre = Conv1d(initial_channel, upsample_initial_channel, 7, 1, padding=3)
367
+ resblock = modules.ResBlock1 if resblock == '1' else modules.ResBlock2
368
+
369
+ self.ups = nn.ModuleList()
370
+ for i, (u, k) in enumerate(zip(upsample_rates, upsample_kernel_sizes)):
371
+ self.ups.append(weight_norm(
372
+ ConvTranspose1d(upsample_initial_channel // (2 ** i), upsample_initial_channel // (2 ** (i + 1)),
373
+ k, u, padding=(k - u) // 2)))
374
+
375
+ self.resblocks = nn.ModuleList()
376
+ for i in range(len(self.ups)):
377
+ ch = upsample_initial_channel // (2 ** (i + 1))
378
+ for j, (k, d) in enumerate(zip(resblock_kernel_sizes, resblock_dilation_sizes)):
379
+ self.resblocks.append(resblock(ch, k, d))
380
+
381
+ self.conv_post = Conv1d(ch, 1, 7, 1, padding=3, bias=False)
382
+ self.ups.apply(init_weights)
383
+
384
+ if gin_channels != 0:
385
+ self.cond = nn.Conv1d(gin_channels, upsample_initial_channel, 1)
386
+
387
+ def forward(self, x, g=None):
388
+ x = self.conv_pre(x)
389
+ if g is not None:
390
+ x = x + self.cond(g)
391
+
392
+ for i in range(self.num_upsamples):
393
+ x = F.leaky_relu(x, modules.LRELU_SLOPE)
394
+ x = self.ups[i](x)
395
+ xs = None
396
+ for j in range(self.num_kernels):
397
+ if xs is None:
398
+ xs = self.resblocks[i * self.num_kernels + j](x)
399
+ else:
400
+ xs += self.resblocks[i * self.num_kernels + j](x)
401
+ x = xs / self.num_kernels
402
+ x = F.leaky_relu(x)
403
+ x = self.conv_post(x)
404
+ x = torch.tanh(x)
405
+
406
+ return x
407
+
408
+ def remove_weight_norm(self):
409
+ print('Removing weight norm...')
410
+ for l in self.ups:
411
+ remove_weight_norm(l)
412
+ for l in self.resblocks:
413
+ l.remove_weight_norm()
414
+
415
+
416
+ class DiscriminatorP(torch.nn.Module):
417
+ def __init__(self, period, kernel_size=5, stride=3, use_spectral_norm=False):
418
+ super(DiscriminatorP, self).__init__()
419
+ self.period = period
420
+ self.use_spectral_norm = use_spectral_norm
421
+ norm_f = weight_norm if use_spectral_norm == False else spectral_norm
422
+ self.convs = nn.ModuleList([
423
+ norm_f(Conv2d(1, 32, (kernel_size, 1), (stride, 1), padding=(get_padding(kernel_size, 1), 0))),
424
+ norm_f(Conv2d(32, 128, (kernel_size, 1), (stride, 1), padding=(get_padding(kernel_size, 1), 0))),
425
+ norm_f(Conv2d(128, 512, (kernel_size, 1), (stride, 1), padding=(get_padding(kernel_size, 1), 0))),
426
+ norm_f(Conv2d(512, 1024, (kernel_size, 1), (stride, 1), padding=(get_padding(kernel_size, 1), 0))),
427
+ norm_f(Conv2d(1024, 1024, (kernel_size, 1), 1, padding=(get_padding(kernel_size, 1), 0))),
428
+ ])
429
+ self.conv_post = norm_f(Conv2d(1024, 1, (3, 1), 1, padding=(1, 0)))
430
+
431
+ def forward(self, x):
432
+ fmap = []
433
+
434
+ # 1d to 2d
435
+ b, c, t = x.shape
436
+ if t % self.period != 0: # pad first
437
+ n_pad = self.period - (t % self.period)
438
+ x = F.pad(x, (0, n_pad), "reflect")
439
+ t = t + n_pad
440
+ x = x.view(b, c, t // self.period, self.period)
441
+
442
+ for l in self.convs:
443
+ x = l(x)
444
+ x = F.leaky_relu(x, modules.LRELU_SLOPE)
445
+ fmap.append(x)
446
+ x = self.conv_post(x)
447
+ fmap.append(x)
448
+ x = torch.flatten(x, 1, -1)
449
+
450
+ return x, fmap
451
+
452
+
453
+ class DiscriminatorS(torch.nn.Module):
454
+ def __init__(self, use_spectral_norm=False):
455
+ super(DiscriminatorS, self).__init__()
456
+ norm_f = weight_norm if use_spectral_norm == False else spectral_norm
457
+ self.convs = nn.ModuleList([
458
+ norm_f(Conv1d(1, 16, 15, 1, padding=7)),
459
+ norm_f(Conv1d(16, 64, 41, 4, groups=4, padding=20)),
460
+ norm_f(Conv1d(64, 256, 41, 4, groups=16, padding=20)),
461
+ norm_f(Conv1d(256, 1024, 41, 4, groups=64, padding=20)),
462
+ norm_f(Conv1d(1024, 1024, 41, 4, groups=256, padding=20)),
463
+ norm_f(Conv1d(1024, 1024, 5, 1, padding=2)),
464
+ ])
465
+ self.conv_post = norm_f(Conv1d(1024, 1, 3, 1, padding=1))
466
+
467
+ def forward(self, x):
468
+ fmap = []
469
+
470
+ for l in self.convs:
471
+ x = l(x)
472
+ x = F.leaky_relu(x, modules.LRELU_SLOPE)
473
+ fmap.append(x)
474
+ x = self.conv_post(x)
475
+ fmap.append(x)
476
+ x = torch.flatten(x, 1, -1)
477
+
478
+ return x, fmap
479
+
480
+
481
+ class MultiPeriodDiscriminator(torch.nn.Module):
482
+ def __init__(self, use_spectral_norm=False):
483
+ super(MultiPeriodDiscriminator, self).__init__()
484
+ periods = [2, 3, 5, 7, 11]
485
+
486
+ discs = [DiscriminatorS(use_spectral_norm=use_spectral_norm)]
487
+ discs = discs + [DiscriminatorP(i, use_spectral_norm=use_spectral_norm) for i in periods]
488
+ self.discriminators = nn.ModuleList(discs)
489
+
490
+ def forward(self, y, y_hat):
491
+ y_d_rs = []
492
+ y_d_gs = []
493
+ fmap_rs = []
494
+ fmap_gs = []
495
+ for i, d in enumerate(self.discriminators):
496
+ y_d_r, fmap_r = d(y)
497
+ y_d_g, fmap_g = d(y_hat)
498
+ y_d_rs.append(y_d_r)
499
+ y_d_gs.append(y_d_g)
500
+ fmap_rs.append(fmap_r)
501
+ fmap_gs.append(fmap_g)
502
+
503
+ return y_d_rs, y_d_gs, fmap_rs, fmap_gs
504
+
505
+ class ReferenceEncoder(nn.Module):
506
+ '''
507
+ inputs --- [N, Ty/r, n_mels*r] mels
508
+ outputs --- [N, ref_enc_gru_size]
509
+ '''
510
+
511
+ def __init__(self, spec_channels, gin_channels=0):
512
+
513
+ super().__init__()
514
+ self.spec_channels = spec_channels
515
+ ref_enc_filters = [32, 32, 64, 64, 128, 128]
516
+ K = len(ref_enc_filters)
517
+ filters = [1] + ref_enc_filters
518
+ convs = [weight_norm(nn.Conv2d(in_channels=filters[i],
519
+ out_channels=filters[i + 1],
520
+ kernel_size=(3, 3),
521
+ stride=(2, 2),
522
+ padding=(1, 1))) for i in range(K)]
523
+ self.convs = nn.ModuleList(convs)
524
+ # self.wns = nn.ModuleList([weight_norm(num_features=ref_enc_filters[i]) for i in range(K)])
525
+
526
+ out_channels = self.calculate_channels(spec_channels, 3, 2, 1, K)
527
+ self.gru = nn.GRU(input_size=ref_enc_filters[-1] * out_channels,
528
+ hidden_size=256 // 2,
529
+ batch_first=True)
530
+ self.proj = nn.Linear(128, gin_channels)
531
+
532
+ def forward(self, inputs, mask=None):
533
+ N = inputs.size(0)
534
+ out = inputs.view(N, 1, -1, self.spec_channels) # [N, 1, Ty, n_freqs]
535
+ for conv in self.convs:
536
+ out = conv(out)
537
+ # out = wn(out)
538
+ out = F.relu(out) # [N, 128, Ty//2^K, n_mels//2^K]
539
+
540
+ out = out.transpose(1, 2) # [N, Ty//2^K, 128, n_mels//2^K]
541
+ T = out.size(1)
542
+ N = out.size(0)
543
+ out = out.contiguous().view(N, T, -1) # [N, Ty//2^K, 128*n_mels//2^K]
544
+
545
+ self.gru.flatten_parameters()
546
+ memory, out = self.gru(out) # out --- [1, N, 128]
547
+
548
+ return self.proj(out.squeeze(0))
549
+
550
+ def calculate_channels(self, L, kernel_size, stride, pad, n_convs):
551
+ for i in range(n_convs):
552
+ L = (L - kernel_size + 2 * pad) // stride + 1
553
+ return L
554
+
555
+
556
+ class SynthesizerTrn(nn.Module):
557
+ """
558
+ Synthesizer for Training
559
+ """
560
+
561
+ def __init__(self,
562
+ n_vocab,
563
+ spec_channels,
564
+ segment_size,
565
+ inter_channels,
566
+ hidden_channels,
567
+ filter_channels,
568
+ n_heads,
569
+ n_layers,
570
+ kernel_size,
571
+ p_dropout,
572
+ resblock,
573
+ resblock_kernel_sizes,
574
+ resblock_dilation_sizes,
575
+ upsample_rates,
576
+ upsample_initial_channel,
577
+ upsample_kernel_sizes,
578
+ n_speakers=256,
579
+ gin_channels=256,
580
+ use_sdp=True,
581
+ n_flow_layer = 4,
582
+ n_layers_trans_flow = 3,
583
+ flow_share_parameter = False,
584
+ use_transformer_flow = True,
585
+ **kwargs):
586
+
587
+ super().__init__()
588
+ self.n_vocab = n_vocab
589
+ self.spec_channels = spec_channels
590
+ self.inter_channels = inter_channels
591
+ self.hidden_channels = hidden_channels
592
+ self.filter_channels = filter_channels
593
+ self.n_heads = n_heads
594
+ self.n_layers = n_layers
595
+ self.kernel_size = kernel_size
596
+ self.p_dropout = p_dropout
597
+ self.resblock = resblock
598
+ self.resblock_kernel_sizes = resblock_kernel_sizes
599
+ self.resblock_dilation_sizes = resblock_dilation_sizes
600
+ self.upsample_rates = upsample_rates
601
+ self.upsample_initial_channel = upsample_initial_channel
602
+ self.upsample_kernel_sizes = upsample_kernel_sizes
603
+ self.segment_size = segment_size
604
+ self.n_speakers = n_speakers
605
+ self.gin_channels = gin_channels
606
+ self.n_layers_trans_flow = n_layers_trans_flow
607
+ self.use_spk_conditioned_encoder = kwargs.get("use_spk_conditioned_encoder", True)
608
+ self.use_sdp = use_sdp
609
+ self.use_noise_scaled_mas = kwargs.get("use_noise_scaled_mas", False)
610
+ self.mas_noise_scale_initial = kwargs.get("mas_noise_scale_initial", 0.01)
611
+ self.noise_scale_delta = kwargs.get("noise_scale_delta", 2e-6)
612
+ self.current_mas_noise_scale = self.mas_noise_scale_initial
613
+ if self.use_spk_conditioned_encoder and gin_channels > 0:
614
+ self.enc_gin_channels = gin_channels
615
+ self.enc_p = TextEncoder(n_vocab,
616
+ inter_channels,
617
+ hidden_channels,
618
+ filter_channels,
619
+ n_heads,
620
+ n_layers,
621
+ kernel_size,
622
+ p_dropout,
623
+ gin_channels=self.enc_gin_channels)
624
+ self.dec = Generator(inter_channels, resblock, resblock_kernel_sizes, resblock_dilation_sizes, upsample_rates,
625
+ upsample_initial_channel, upsample_kernel_sizes, gin_channels=gin_channels)
626
+ self.enc_q = PosteriorEncoder(spec_channels, inter_channels, hidden_channels, 5, 1, 16,
627
+ gin_channels=gin_channels)
628
+ if use_transformer_flow:
629
+ self.flow = TransformerCouplingBlock(inter_channels, hidden_channels, filter_channels, n_heads, n_layers_trans_flow, 5, p_dropout, n_flow_layer, gin_channels=gin_channels,share_parameter= flow_share_parameter)
630
+ else:
631
+ self.flow = ResidualCouplingBlock(inter_channels, hidden_channels, 5, 1, n_flow_layer, gin_channels=gin_channels)
632
+ self.sdp = StochasticDurationPredictor(hidden_channels, 192, 3, 0.5, 4, gin_channels=gin_channels)
633
+ self.dp = DurationPredictor(hidden_channels, 256, 3, 0.5, gin_channels=gin_channels)
634
+
635
+ if n_speakers >= 1:
636
+ self.emb_g = nn.Embedding(n_speakers, gin_channels)
637
+ else:
638
+ self.ref_enc = ReferenceEncoder(spec_channels, gin_channels)
639
+
640
+ def forward(self, x, x_lengths, y, y_lengths, sid, tone, language, bert):
641
+ if self.n_speakers > 0:
642
+ g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
643
+ else:
644
+ g = self.ref_enc(y.transpose(1,2)).unsqueeze(-1)
645
+ x, m_p, logs_p, x_mask = self.enc_p(x, x_lengths, tone, language, bert,g=g)
646
+ z, m_q, logs_q, y_mask = self.enc_q(y, y_lengths, g=g)
647
+ z_p = self.flow(z, y_mask, g=g)
648
+
649
+ with torch.no_grad():
650
+ # negative cross-entropy
651
+ s_p_sq_r = torch.exp(-2 * logs_p) # [b, d, t]
652
+ neg_cent1 = torch.sum(-0.5 * math.log(2 * math.pi) - logs_p, [1], keepdim=True) # [b, 1, t_s]
653
+ neg_cent2 = torch.matmul(-0.5 * (z_p ** 2).transpose(1, 2),
654
+ s_p_sq_r) # [b, t_t, d] x [b, d, t_s] = [b, t_t, t_s]
655
+ neg_cent3 = torch.matmul(z_p.transpose(1, 2), (m_p * s_p_sq_r)) # [b, t_t, d] x [b, d, t_s] = [b, t_t, t_s]
656
+ neg_cent4 = torch.sum(-0.5 * (m_p ** 2) * s_p_sq_r, [1], keepdim=True) # [b, 1, t_s]
657
+ neg_cent = neg_cent1 + neg_cent2 + neg_cent3 + neg_cent4
658
+ if self.use_noise_scaled_mas:
659
+ epsilon = torch.std(neg_cent) * torch.randn_like(neg_cent) * self.current_mas_noise_scale
660
+ neg_cent = neg_cent + epsilon
661
+
662
+ attn_mask = torch.unsqueeze(x_mask, 2) * torch.unsqueeze(y_mask, -1)
663
+ attn = monotonic_align.maximum_path(neg_cent, attn_mask.squeeze(1)).unsqueeze(1).detach()
664
+
665
+ w = attn.sum(2)
666
+
667
+ l_length_sdp = self.sdp(x, x_mask, w, g=g)
668
+ l_length_sdp = l_length_sdp / torch.sum(x_mask)
669
+
670
+ logw_ = torch.log(w + 1e-6) * x_mask
671
+ logw = self.dp(x, x_mask, g=g)
672
+ l_length_dp = torch.sum((logw - logw_) ** 2, [1, 2]) / torch.sum(x_mask) # for averaging
673
+
674
+ l_length = l_length_dp + l_length_sdp
675
+
676
+ # expand prior
677
+ m_p = torch.matmul(attn.squeeze(1), m_p.transpose(1, 2)).transpose(1, 2)
678
+ logs_p = torch.matmul(attn.squeeze(1), logs_p.transpose(1, 2)).transpose(1, 2)
679
+
680
+ z_slice, ids_slice = commons.rand_slice_segments(z, y_lengths, self.segment_size)
681
+ o = self.dec(z_slice, g=g)
682
+ return o, l_length, attn, ids_slice, x_mask, y_mask, (z, z_p, m_p, logs_p, m_q, logs_q), (x, logw, logw_)
683
+
684
+ def infer(self, x, x_lengths, sid, tone, language, bert, noise_scale=.667, length_scale=1, noise_scale_w=0.8, max_len=None, sdp_ratio=0,y=None):
685
+ #x, m_p, logs_p, x_mask = self.enc_p(x, x_lengths, tone, language, bert)
686
+ # g = self.gst(y)
687
+ if self.n_speakers > 0:
688
+ g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
689
+ else:
690
+ g = self.ref_enc(y.transpose(1,2)).unsqueeze(-1)
691
+ x, m_p, logs_p, x_mask = self.enc_p(x, x_lengths, tone, language, bert,g=g)
692
+ logw = self.sdp(x, x_mask, g=g, reverse=True, noise_scale=noise_scale_w) * (sdp_ratio) + self.dp(x, x_mask, g=g) * (1 - sdp_ratio)
693
+ w = torch.exp(logw) * x_mask * length_scale
694
+ w_ceil = torch.ceil(w)
695
+ y_lengths = torch.clamp_min(torch.sum(w_ceil, [1, 2]), 1).long()
696
+ y_mask = torch.unsqueeze(commons.sequence_mask(y_lengths, None), 1).to(x_mask.dtype)
697
+ attn_mask = torch.unsqueeze(x_mask, 2) * torch.unsqueeze(y_mask, -1)
698
+ attn = commons.generate_path(w_ceil, attn_mask)
699
+
700
+ m_p = torch.matmul(attn.squeeze(1), m_p.transpose(1, 2)).transpose(1, 2) # [b, t', t], [b, t, d] -> [b, d, t']
701
+ logs_p = torch.matmul(attn.squeeze(1), logs_p.transpose(1, 2)).transpose(1,
702
+ 2) # [b, t', t], [b, t, d] -> [b, d, t']
703
+
704
+ z_p = m_p + torch.randn_like(m_p) * torch.exp(logs_p) * noise_scale
705
+ z = self.flow(z_p, y_mask, g=g, reverse=True)
706
+ o = self.dec((z * y_mask)[:, :, :max_len], g=g)
707
+ return o, attn, y_mask, (z, z_p, m_p, logs_p)
modules.py ADDED
@@ -0,0 +1,452 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import copy
2
+ import math
3
+ import numpy as np
4
+ import scipy
5
+ import torch
6
+ from torch import nn
7
+ from torch.nn import functional as F
8
+
9
+ from torch.nn import Conv1d, ConvTranspose1d, AvgPool1d, Conv2d
10
+ from torch.nn.utils import weight_norm, remove_weight_norm
11
+
12
+ import commons
13
+ from commons import init_weights, get_padding
14
+ from transforms import piecewise_rational_quadratic_transform
15
+ from attentions import Encoder
16
+
17
+ LRELU_SLOPE = 0.1
18
+
19
+ class LayerNorm(nn.Module):
20
+ def __init__(self, channels, eps=1e-5):
21
+ super().__init__()
22
+ self.channels = channels
23
+ self.eps = eps
24
+
25
+ self.gamma = nn.Parameter(torch.ones(channels))
26
+ self.beta = nn.Parameter(torch.zeros(channels))
27
+
28
+ def forward(self, x):
29
+ x = x.transpose(1, -1)
30
+ x = F.layer_norm(x, (self.channels,), self.gamma, self.beta, self.eps)
31
+ return x.transpose(1, -1)
32
+
33
+ class ConvReluNorm(nn.Module):
34
+ def __init__(self, in_channels, hidden_channels, out_channels, kernel_size, n_layers, p_dropout):
35
+ super().__init__()
36
+ self.in_channels = in_channels
37
+ self.hidden_channels = hidden_channels
38
+ self.out_channels = out_channels
39
+ self.kernel_size = kernel_size
40
+ self.n_layers = n_layers
41
+ self.p_dropout = p_dropout
42
+ assert n_layers > 1, "Number of layers should be larger than 0."
43
+
44
+ self.conv_layers = nn.ModuleList()
45
+ self.norm_layers = nn.ModuleList()
46
+ self.conv_layers.append(nn.Conv1d(in_channels, hidden_channels, kernel_size, padding=kernel_size//2))
47
+ self.norm_layers.append(LayerNorm(hidden_channels))
48
+ self.relu_drop = nn.Sequential(
49
+ nn.ReLU(),
50
+ nn.Dropout(p_dropout))
51
+ for _ in range(n_layers-1):
52
+ self.conv_layers.append(nn.Conv1d(hidden_channels, hidden_channels, kernel_size, padding=kernel_size//2))
53
+ self.norm_layers.append(LayerNorm(hidden_channels))
54
+ self.proj = nn.Conv1d(hidden_channels, out_channels, 1)
55
+ self.proj.weight.data.zero_()
56
+ self.proj.bias.data.zero_()
57
+
58
+ def forward(self, x, x_mask):
59
+ x_org = x
60
+ for i in range(self.n_layers):
61
+ x = self.conv_layers[i](x * x_mask)
62
+ x = self.norm_layers[i](x)
63
+ x = self.relu_drop(x)
64
+ x = x_org + self.proj(x)
65
+ return x * x_mask
66
+
67
+
68
+ class DDSConv(nn.Module):
69
+ """
70
+ Dialted and Depth-Separable Convolution
71
+ """
72
+ def __init__(self, channels, kernel_size, n_layers, p_dropout=0.):
73
+ super().__init__()
74
+ self.channels = channels
75
+ self.kernel_size = kernel_size
76
+ self.n_layers = n_layers
77
+ self.p_dropout = p_dropout
78
+
79
+ self.drop = nn.Dropout(p_dropout)
80
+ self.convs_sep = nn.ModuleList()
81
+ self.convs_1x1 = nn.ModuleList()
82
+ self.norms_1 = nn.ModuleList()
83
+ self.norms_2 = nn.ModuleList()
84
+ for i in range(n_layers):
85
+ dilation = kernel_size ** i
86
+ padding = (kernel_size * dilation - dilation) // 2
87
+ self.convs_sep.append(nn.Conv1d(channels, channels, kernel_size,
88
+ groups=channels, dilation=dilation, padding=padding
89
+ ))
90
+ self.convs_1x1.append(nn.Conv1d(channels, channels, 1))
91
+ self.norms_1.append(LayerNorm(channels))
92
+ self.norms_2.append(LayerNorm(channels))
93
+
94
+ def forward(self, x, x_mask, g=None):
95
+ if g is not None:
96
+ x = x + g
97
+ for i in range(self.n_layers):
98
+ y = self.convs_sep[i](x * x_mask)
99
+ y = self.norms_1[i](y)
100
+ y = F.gelu(y)
101
+ y = self.convs_1x1[i](y)
102
+ y = self.norms_2[i](y)
103
+ y = F.gelu(y)
104
+ y = self.drop(y)
105
+ x = x + y
106
+ return x * x_mask
107
+
108
+
109
+ class WN(torch.nn.Module):
110
+ def __init__(self, hidden_channels, kernel_size, dilation_rate, n_layers, gin_channels=0, p_dropout=0):
111
+ super(WN, self).__init__()
112
+ assert(kernel_size % 2 == 1)
113
+ self.hidden_channels =hidden_channels
114
+ self.kernel_size = kernel_size,
115
+ self.dilation_rate = dilation_rate
116
+ self.n_layers = n_layers
117
+ self.gin_channels = gin_channels
118
+ self.p_dropout = p_dropout
119
+
120
+ self.in_layers = torch.nn.ModuleList()
121
+ self.res_skip_layers = torch.nn.ModuleList()
122
+ self.drop = nn.Dropout(p_dropout)
123
+
124
+ if gin_channels != 0:
125
+ cond_layer = torch.nn.Conv1d(gin_channels, 2*hidden_channels*n_layers, 1)
126
+ self.cond_layer = torch.nn.utils.weight_norm(cond_layer, name='weight')
127
+
128
+ for i in range(n_layers):
129
+ dilation = dilation_rate ** i
130
+ padding = int((kernel_size * dilation - dilation) / 2)
131
+ in_layer = torch.nn.Conv1d(hidden_channels, 2*hidden_channels, kernel_size,
132
+ dilation=dilation, padding=padding)
133
+ in_layer = torch.nn.utils.weight_norm(in_layer, name='weight')
134
+ self.in_layers.append(in_layer)
135
+
136
+ # last one is not necessary
137
+ if i < n_layers - 1:
138
+ res_skip_channels = 2 * hidden_channels
139
+ else:
140
+ res_skip_channels = hidden_channels
141
+
142
+ res_skip_layer = torch.nn.Conv1d(hidden_channels, res_skip_channels, 1)
143
+ res_skip_layer = torch.nn.utils.weight_norm(res_skip_layer, name='weight')
144
+ self.res_skip_layers.append(res_skip_layer)
145
+
146
+ def forward(self, x, x_mask, g=None, **kwargs):
147
+ output = torch.zeros_like(x)
148
+ n_channels_tensor = torch.IntTensor([self.hidden_channels])
149
+
150
+ if g is not None:
151
+ g = self.cond_layer(g)
152
+
153
+ for i in range(self.n_layers):
154
+ x_in = self.in_layers[i](x)
155
+ if g is not None:
156
+ cond_offset = i * 2 * self.hidden_channels
157
+ g_l = g[:,cond_offset:cond_offset+2*self.hidden_channels,:]
158
+ else:
159
+ g_l = torch.zeros_like(x_in)
160
+
161
+ acts = commons.fused_add_tanh_sigmoid_multiply(
162
+ x_in,
163
+ g_l,
164
+ n_channels_tensor)
165
+ acts = self.drop(acts)
166
+
167
+ res_skip_acts = self.res_skip_layers[i](acts)
168
+ if i < self.n_layers - 1:
169
+ res_acts = res_skip_acts[:,:self.hidden_channels,:]
170
+ x = (x + res_acts) * x_mask
171
+ output = output + res_skip_acts[:,self.hidden_channels:,:]
172
+ else:
173
+ output = output + res_skip_acts
174
+ return output * x_mask
175
+
176
+ def remove_weight_norm(self):
177
+ if self.gin_channels != 0:
178
+ torch.nn.utils.remove_weight_norm(self.cond_layer)
179
+ for l in self.in_layers:
180
+ torch.nn.utils.remove_weight_norm(l)
181
+ for l in self.res_skip_layers:
182
+ torch.nn.utils.remove_weight_norm(l)
183
+
184
+
185
+ class ResBlock1(torch.nn.Module):
186
+ def __init__(self, channels, kernel_size=3, dilation=(1, 3, 5)):
187
+ super(ResBlock1, self).__init__()
188
+ self.convs1 = nn.ModuleList([
189
+ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[0],
190
+ padding=get_padding(kernel_size, dilation[0]))),
191
+ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[1],
192
+ padding=get_padding(kernel_size, dilation[1]))),
193
+ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[2],
194
+ padding=get_padding(kernel_size, dilation[2])))
195
+ ])
196
+ self.convs1.apply(init_weights)
197
+
198
+ self.convs2 = nn.ModuleList([
199
+ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1,
200
+ padding=get_padding(kernel_size, 1))),
201
+ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1,
202
+ padding=get_padding(kernel_size, 1))),
203
+ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=1,
204
+ padding=get_padding(kernel_size, 1)))
205
+ ])
206
+ self.convs2.apply(init_weights)
207
+
208
+ def forward(self, x, x_mask=None):
209
+ for c1, c2 in zip(self.convs1, self.convs2):
210
+ xt = F.leaky_relu(x, LRELU_SLOPE)
211
+ if x_mask is not None:
212
+ xt = xt * x_mask
213
+ xt = c1(xt)
214
+ xt = F.leaky_relu(xt, LRELU_SLOPE)
215
+ if x_mask is not None:
216
+ xt = xt * x_mask
217
+ xt = c2(xt)
218
+ x = xt + x
219
+ if x_mask is not None:
220
+ x = x * x_mask
221
+ return x
222
+
223
+ def remove_weight_norm(self):
224
+ for l in self.convs1:
225
+ remove_weight_norm(l)
226
+ for l in self.convs2:
227
+ remove_weight_norm(l)
228
+
229
+
230
+ class ResBlock2(torch.nn.Module):
231
+ def __init__(self, channels, kernel_size=3, dilation=(1, 3)):
232
+ super(ResBlock2, self).__init__()
233
+ self.convs = nn.ModuleList([
234
+ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[0],
235
+ padding=get_padding(kernel_size, dilation[0]))),
236
+ weight_norm(Conv1d(channels, channels, kernel_size, 1, dilation=dilation[1],
237
+ padding=get_padding(kernel_size, dilation[1])))
238
+ ])
239
+ self.convs.apply(init_weights)
240
+
241
+ def forward(self, x, x_mask=None):
242
+ for c in self.convs:
243
+ xt = F.leaky_relu(x, LRELU_SLOPE)
244
+ if x_mask is not None:
245
+ xt = xt * x_mask
246
+ xt = c(xt)
247
+ x = xt + x
248
+ if x_mask is not None:
249
+ x = x * x_mask
250
+ return x
251
+
252
+ def remove_weight_norm(self):
253
+ for l in self.convs:
254
+ remove_weight_norm(l)
255
+
256
+
257
+ class Log(nn.Module):
258
+ def forward(self, x, x_mask, reverse=False, **kwargs):
259
+ if not reverse:
260
+ y = torch.log(torch.clamp_min(x, 1e-5)) * x_mask
261
+ logdet = torch.sum(-y, [1, 2])
262
+ return y, logdet
263
+ else:
264
+ x = torch.exp(x) * x_mask
265
+ return x
266
+
267
+
268
+ class Flip(nn.Module):
269
+ def forward(self, x, *args, reverse=False, **kwargs):
270
+ x = torch.flip(x, [1])
271
+ if not reverse:
272
+ logdet = torch.zeros(x.size(0)).to(dtype=x.dtype, device=x.device)
273
+ return x, logdet
274
+ else:
275
+ return x
276
+
277
+
278
+ class ElementwiseAffine(nn.Module):
279
+ def __init__(self, channels):
280
+ super().__init__()
281
+ self.channels = channels
282
+ self.m = nn.Parameter(torch.zeros(channels,1))
283
+ self.logs = nn.Parameter(torch.zeros(channels,1))
284
+
285
+ def forward(self, x, x_mask, reverse=False, **kwargs):
286
+ if not reverse:
287
+ y = self.m + torch.exp(self.logs) * x
288
+ y = y * x_mask
289
+ logdet = torch.sum(self.logs * x_mask, [1,2])
290
+ return y, logdet
291
+ else:
292
+ x = (x - self.m) * torch.exp(-self.logs) * x_mask
293
+ return x
294
+
295
+
296
+ class ResidualCouplingLayer(nn.Module):
297
+ def __init__(self,
298
+ channels,
299
+ hidden_channels,
300
+ kernel_size,
301
+ dilation_rate,
302
+ n_layers,
303
+ p_dropout=0,
304
+ gin_channels=0,
305
+ mean_only=False):
306
+ assert channels % 2 == 0, "channels should be divisible by 2"
307
+ super().__init__()
308
+ self.channels = channels
309
+ self.hidden_channels = hidden_channels
310
+ self.kernel_size = kernel_size
311
+ self.dilation_rate = dilation_rate
312
+ self.n_layers = n_layers
313
+ self.half_channels = channels // 2
314
+ self.mean_only = mean_only
315
+
316
+ self.pre = nn.Conv1d(self.half_channels, hidden_channels, 1)
317
+ self.enc = WN(hidden_channels, kernel_size, dilation_rate, n_layers, p_dropout=p_dropout, gin_channels=gin_channels)
318
+ self.post = nn.Conv1d(hidden_channels, self.half_channels * (2 - mean_only), 1)
319
+ self.post.weight.data.zero_()
320
+ self.post.bias.data.zero_()
321
+
322
+ def forward(self, x, x_mask, g=None, reverse=False):
323
+ x0, x1 = torch.split(x, [self.half_channels]*2, 1)
324
+ h = self.pre(x0) * x_mask
325
+ h = self.enc(h, x_mask, g=g)
326
+ stats = self.post(h) * x_mask
327
+ if not self.mean_only:
328
+ m, logs = torch.split(stats, [self.half_channels]*2, 1)
329
+ else:
330
+ m = stats
331
+ logs = torch.zeros_like(m)
332
+
333
+ if not reverse:
334
+ x1 = m + x1 * torch.exp(logs) * x_mask
335
+ x = torch.cat([x0, x1], 1)
336
+ logdet = torch.sum(logs, [1,2])
337
+ return x, logdet
338
+ else:
339
+ x1 = (x1 - m) * torch.exp(-logs) * x_mask
340
+ x = torch.cat([x0, x1], 1)
341
+ return x
342
+
343
+
344
+ class ConvFlow(nn.Module):
345
+ def __init__(self, in_channels, filter_channels, kernel_size, n_layers, num_bins=10, tail_bound=5.0):
346
+ super().__init__()
347
+ self.in_channels = in_channels
348
+ self.filter_channels = filter_channels
349
+ self.kernel_size = kernel_size
350
+ self.n_layers = n_layers
351
+ self.num_bins = num_bins
352
+ self.tail_bound = tail_bound
353
+ self.half_channels = in_channels // 2
354
+
355
+ self.pre = nn.Conv1d(self.half_channels, filter_channels, 1)
356
+ self.convs = DDSConv(filter_channels, kernel_size, n_layers, p_dropout=0.)
357
+ self.proj = nn.Conv1d(filter_channels, self.half_channels * (num_bins * 3 - 1), 1)
358
+ self.proj.weight.data.zero_()
359
+ self.proj.bias.data.zero_()
360
+
361
+ def forward(self, x, x_mask, g=None, reverse=False):
362
+ x0, x1 = torch.split(x, [self.half_channels]*2, 1)
363
+ h = self.pre(x0)
364
+ h = self.convs(h, x_mask, g=g)
365
+ h = self.proj(h) * x_mask
366
+
367
+ b, c, t = x0.shape
368
+ h = h.reshape(b, c, -1, t).permute(0, 1, 3, 2) # [b, cx?, t] -> [b, c, t, ?]
369
+
370
+ unnormalized_widths = h[..., :self.num_bins] / math.sqrt(self.filter_channels)
371
+ unnormalized_heights = h[..., self.num_bins:2*self.num_bins] / math.sqrt(self.filter_channels)
372
+ unnormalized_derivatives = h[..., 2 * self.num_bins:]
373
+
374
+ x1, logabsdet = piecewise_rational_quadratic_transform(x1,
375
+ unnormalized_widths,
376
+ unnormalized_heights,
377
+ unnormalized_derivatives,
378
+ inverse=reverse,
379
+ tails='linear',
380
+ tail_bound=self.tail_bound
381
+ )
382
+
383
+ x = torch.cat([x0, x1], 1) * x_mask
384
+ logdet = torch.sum(logabsdet * x_mask, [1,2])
385
+ if not reverse:
386
+ return x, logdet
387
+ else:
388
+ return x
389
+ class TransformerCouplingLayer(nn.Module):
390
+ def __init__(self,
391
+ channels,
392
+ hidden_channels,
393
+ kernel_size,
394
+ n_layers,
395
+ n_heads,
396
+ p_dropout=0,
397
+ filter_channels=0,
398
+ mean_only=False,
399
+ wn_sharing_parameter=None,
400
+ gin_channels = 0
401
+ ):
402
+ assert channels % 2 == 0, "channels should be divisible by 2"
403
+ super().__init__()
404
+ self.channels = channels
405
+ self.hidden_channels = hidden_channels
406
+ self.kernel_size = kernel_size
407
+ self.n_layers = n_layers
408
+ self.half_channels = channels // 2
409
+ self.mean_only = mean_only
410
+
411
+ self.pre = nn.Conv1d(self.half_channels, hidden_channels, 1)
412
+ self.enc = Encoder(hidden_channels, filter_channels, n_heads, n_layers, kernel_size, p_dropout, isflow = True, gin_channels = gin_channels) if wn_sharing_parameter is None else wn_sharing_parameter
413
+ self.post = nn.Conv1d(hidden_channels, self.half_channels * (2 - mean_only), 1)
414
+ self.post.weight.data.zero_()
415
+ self.post.bias.data.zero_()
416
+
417
+ def forward(self, x, x_mask, g=None, reverse=False):
418
+ x0, x1 = torch.split(x, [self.half_channels]*2, 1)
419
+ h = self.pre(x0) * x_mask
420
+ h = self.enc(h, x_mask, g=g)
421
+ stats = self.post(h) * x_mask
422
+ if not self.mean_only:
423
+ m, logs = torch.split(stats, [self.half_channels]*2, 1)
424
+ else:
425
+ m = stats
426
+ logs = torch.zeros_like(m)
427
+
428
+ if not reverse:
429
+ x1 = m + x1 * torch.exp(logs) * x_mask
430
+ x = torch.cat([x0, x1], 1)
431
+ logdet = torch.sum(logs, [1,2])
432
+ return x, logdet
433
+ else:
434
+ x1 = (x1 - m) * torch.exp(-logs) * x_mask
435
+ x = torch.cat([x0, x1], 1)
436
+ return x
437
+
438
+ x1, logabsdet = piecewise_rational_quadratic_transform(x1,
439
+ unnormalized_widths,
440
+ unnormalized_heights,
441
+ unnormalized_derivatives,
442
+ inverse=reverse,
443
+ tails='linear',
444
+ tail_bound=self.tail_bound
445
+ )
446
+
447
+ x = torch.cat([x0, x1], 1) * x_mask
448
+ logdet = torch.sum(logabsdet * x_mask, [1,2])
449
+ if not reverse:
450
+ return x, logdet
451
+ else:
452
+ return x
monotonic_align/__init__.py ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from numpy import zeros, int32, float32
2
+ from torch import from_numpy
3
+
4
+ from .core import maximum_path_jit
5
+
6
+ def maximum_path(neg_cent, mask):
7
+ device = neg_cent.device
8
+ dtype = neg_cent.dtype
9
+ neg_cent = neg_cent.data.cpu().numpy().astype(float32)
10
+ path = zeros(neg_cent.shape, dtype=int32)
11
+
12
+ t_t_max = mask.sum(1)[:, 0].data.cpu().numpy().astype(int32)
13
+ t_s_max = mask.sum(2)[:, 0].data.cpu().numpy().astype(int32)
14
+ maximum_path_jit(path, neg_cent, t_t_max, t_s_max)
15
+ return from_numpy(path).to(device=device, dtype=dtype)
monotonic_align/core.py ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import numba
2
+
3
+
4
+ @numba.jit(numba.void(numba.int32[:,:,::1], numba.float32[:,:,::1], numba.int32[::1], numba.int32[::1]), nopython=True, nogil=True)
5
+ def maximum_path_jit(paths, values, t_ys, t_xs):
6
+ b = paths.shape[0]
7
+ max_neg_val=-1e9
8
+ for i in range(int(b)):
9
+ path = paths[i]
10
+ value = values[i]
11
+ t_y = t_ys[i]
12
+ t_x = t_xs[i]
13
+
14
+ v_prev = v_cur = 0.0
15
+ index = t_x - 1
16
+
17
+ for y in range(t_y):
18
+ for x in range(max(0, t_x + y - t_y), min(t_x, y + 1)):
19
+ if x == y:
20
+ v_cur = max_neg_val
21
+ else:
22
+ v_cur = value[y-1, x]
23
+ if x == 0:
24
+ if y == 0:
25
+ v_prev = 0.
26
+ else:
27
+ v_prev = max_neg_val
28
+ else:
29
+ v_prev = value[y-1, x-1]
30
+ value[y, x] += max(v_prev, v_cur)
31
+
32
+ for y in range(t_y - 1, -1, -1):
33
+ path[y, index] = 1
34
+ if index != 0 and (index == y or value[y-1, index] < value[y-1, index-1]):
35
+ index = index - 1
preprocess_text.py ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import json
2
+ from random import shuffle
3
+
4
+ import tqdm
5
+ from text.cleaner import clean_text
6
+ from collections import defaultdict
7
+ stage = [1,2,3]
8
+
9
+ transcription_path = 'filelists/genshin.list'
10
+ train_path = 'filelists/train.list'
11
+ val_path = 'filelists/val.list'
12
+ config_path = "configs/config.json"
13
+ val_per_spk = 4
14
+ max_val_total = 8
15
+
16
+ if 1 in stage:
17
+ with open( transcription_path+'.cleaned', 'w', encoding='utf-8') as f:
18
+ for line in tqdm.tqdm(open(transcription_path, encoding='utf-8').readlines()):
19
+ try:
20
+ utt, spk, language, text = line.strip().split('|')
21
+ norm_text, phones, tones, word2ph = clean_text(text, language)
22
+ f.write('{}|{}|{}|{}|{}|{}|{}\n'.format(utt, spk, language, norm_text, ' '.join(phones),
23
+ " ".join([str(i) for i in tones]),
24
+ " ".join([str(i) for i in word2ph])))
25
+ except Exception as error :
26
+ print("err!", utt, error)
27
+
28
+ if 2 in stage:
29
+ spk_utt_map = defaultdict(list)
30
+ spk_id_map = {}
31
+ current_sid = 0
32
+
33
+ with open( transcription_path+'.cleaned', encoding='utf-8') as f:
34
+ for line in f.readlines():
35
+ utt, spk, language, text, phones, tones, word2ph = line.strip().split('|')
36
+ spk_utt_map[spk].append(line)
37
+ if spk not in spk_id_map.keys():
38
+ spk_id_map[spk] = current_sid
39
+ current_sid += 1
40
+ train_list = []
41
+ val_list = []
42
+
43
+ for spk, utts in spk_utt_map.items():
44
+ shuffle(utts)
45
+ val_list+=utts[:val_per_spk]
46
+ train_list+=utts[val_per_spk:]
47
+ if len(val_list) > max_val_total:
48
+ train_list+=val_list[max_val_total:]
49
+ val_list = val_list[:max_val_total]
50
+
51
+ with open( train_path,"w", encoding='utf-8') as f:
52
+ for line in train_list:
53
+ f.write(line)
54
+
55
+ with open(val_path, "w", encoding='utf-8') as f:
56
+ for line in val_list:
57
+ f.write(line)
58
+
59
+ if 3 in stage:
60
+ assert 2 in stage
61
+ config = json.load(open(config_path, encoding='utf-8'))
62
+ config["data"]['spk2id'] = spk_id_map
63
+ with open(config_path, 'w', encoding='utf-8') as f:
64
+ json.dump(config, f, indent=2, ensure_ascii=False)
requirements.txt ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ librosa==0.9.1
2
+ matplotlib
3
+ numpy
4
+ numba
5
+ phonemizer
6
+ scipy
7
+ tensorboard
8
+ torch
9
+ torchvision
10
+ Unidecode
11
+ amfm_decompy
12
+ jieba
13
+ transformers
14
+ pypinyin
15
+ cn2an
16
+ gradio
17
+ av
resample.py ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ import argparse
3
+ import librosa
4
+ import numpy as np
5
+ from multiprocessing import Pool, cpu_count
6
+
7
+ import soundfile
8
+ from scipy.io import wavfile
9
+ from tqdm import tqdm
10
+
11
+
12
+ def process(item):
13
+ spkdir, wav_name, args = item
14
+ speaker = spkdir.replace("\\", "/").split("/")[-1]
15
+ wav_path = os.path.join(args.in_dir, speaker, wav_name)
16
+ if os.path.exists(wav_path) and '.wav' in wav_path:
17
+ os.makedirs(os.path.join(args.out_dir, speaker), exist_ok=True)
18
+ wav, sr = librosa.load(wav_path, sr=args.sr)
19
+ soundfile.write(
20
+ os.path.join(args.out_dir, speaker, wav_name),
21
+ wav,
22
+ sr
23
+ )
24
+
25
+
26
+
27
+ if __name__ == "__main__":
28
+ parser = argparse.ArgumentParser()
29
+ parser.add_argument("--sr", type=int, default=44100, help="sampling rate")
30
+ parser.add_argument("--in_dir", type=str, default="./raw", help="path to source dir")
31
+ parser.add_argument("--out_dir", type=str, default="./dataset", help="path to target dir")
32
+ args = parser.parse_args()
33
+ # processs = 8
34
+ processs = cpu_count()-2 if cpu_count() >4 else 1
35
+ pool = Pool(processes=processs)
36
+
37
+ for speaker in os.listdir(args.in_dir):
38
+ spk_dir = os.path.join(args.in_dir, speaker)
39
+ if os.path.isdir(spk_dir):
40
+ print(spk_dir)
41
+ for _ in tqdm(pool.imap_unordered(process, [(spk_dir, i, args) for i in os.listdir(spk_dir) if i.endswith("wav")])):
42
+ pass
server.py ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from flask import Flask, request, Response
2
+ from io import BytesIO
3
+ import torch
4
+ from av import open as avopen
5
+
6
+ import commons
7
+ import utils
8
+ from models import SynthesizerTrn
9
+ from text.symbols import symbols
10
+ from text import cleaned_text_to_sequence, get_bert
11
+ from text.cleaner import clean_text
12
+ from scipy.io import wavfile
13
+
14
+ # Flask Init
15
+ app = Flask(__name__)
16
+ app.config['JSON_AS_ASCII'] = False
17
+ def get_text(text, language_str, hps):
18
+ norm_text, phone, tone, word2ph = clean_text(text, language_str)
19
+ print([f"{p}{t}" for p, t in zip(phone, tone)])
20
+ phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
21
+
22
+ if hps.data.add_blank:
23
+ phone = commons.intersperse(phone, 0)
24
+ tone = commons.intersperse(tone, 0)
25
+ language = commons.intersperse(language, 0)
26
+ for i in range(len(word2ph)):
27
+ word2ph[i] = word2ph[i] * 2
28
+ word2ph[0] += 1
29
+ bert = get_bert(norm_text, word2ph, language_str)
30
+
31
+ assert bert.shape[-1] == len(phone)
32
+
33
+ phone = torch.LongTensor(phone)
34
+ tone = torch.LongTensor(tone)
35
+ language = torch.LongTensor(language)
36
+
37
+ return bert, phone, tone, language
38
+
39
+ def infer(text, sdp_ratio, noise_scale, noise_scale_w,length_scale,sid):
40
+ bert, phones, tones, lang_ids = get_text(text,"ZH", hps,)
41
+ with torch.no_grad():
42
+ x_tst=phones.to(dev).unsqueeze(0)
43
+ tones=tones.to(dev).unsqueeze(0)
44
+ lang_ids=lang_ids.to(dev).unsqueeze(0)
45
+ bert = bert.to(dev).unsqueeze(0)
46
+ x_tst_lengths = torch.LongTensor([phones.size(0)]).to(dev)
47
+ speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(dev)
48
+ audio = net_g.infer(x_tst, x_tst_lengths, speakers, tones, lang_ids,bert, sdp_ratio=sdp_ratio
49
+ , noise_scale=noise_scale, noise_scale_w=noise_scale_w, length_scale=length_scale)[0][0,0].data.cpu().float().numpy()
50
+ return audio
51
+
52
+ def replace_punctuation(text, i=2):
53
+ punctuation = ",。?!"
54
+ for char in punctuation:
55
+ text = text.replace(char, char * i)
56
+ return text
57
+
58
+ def wav2(i, o, format):
59
+ inp = avopen(i, 'rb')
60
+ out = avopen(o, 'wb', format=format)
61
+ if format == "ogg": format = "libvorbis"
62
+
63
+ ostream = out.add_stream(format)
64
+
65
+ for frame in inp.decode(audio=0):
66
+ for p in ostream.encode(frame): out.mux(p)
67
+
68
+ for p in ostream.encode(None): out.mux(p)
69
+
70
+ out.close()
71
+ inp.close()
72
+
73
+ # Load Generator
74
+ hps = utils.get_hparams_from_file("./configs/config.json")
75
+
76
+ dev='cuda'
77
+ net_g = SynthesizerTrn(
78
+ len(symbols),
79
+ hps.data.filter_length // 2 + 1,
80
+ hps.train.segment_size // hps.data.hop_length,
81
+ n_speakers=hps.data.n_speakers,
82
+ **hps.model).to(dev)
83
+ _ = net_g.eval()
84
+
85
+ _ = utils.load_checkpoint("logs/G_649000.pth", net_g, None,skip_optimizer=True)
86
+
87
+ @app.route("/",methods=['GET','POST'])
88
+ def main():
89
+ if request.method == 'GET':
90
+ try:
91
+ speaker = request.args.get('speaker')
92
+ text = request.args.get('text').replace("/n","")
93
+ sdp_ratio = float(request.args.get("sdp_ratio", 0.2))
94
+ noise = float(request.args.get("noise", 0.5))
95
+ noisew = float(request.args.get("noisew", 0.6))
96
+ length = float(request.args.get("length", 1.2))
97
+ if length >= 2:
98
+ return "Too big length"
99
+ if len(text) >=200:
100
+ return "Too long text"
101
+ fmt = request.args.get("format", "wav")
102
+ if None in (speaker, text):
103
+ return "Missing Parameter"
104
+ if fmt not in ("mp3", "wav", "ogg"):
105
+ return "Invalid Format"
106
+ except:
107
+ return "Invalid Parameter"
108
+
109
+ with torch.no_grad():
110
+ audio = infer(text, sdp_ratio=sdp_ratio, noise_scale=noise, noise_scale_w=noisew, length_scale=length, sid=speaker)
111
+
112
+ with BytesIO() as wav:
113
+ wavfile.write(wav, hps.data.sampling_rate, audio)
114
+ torch.cuda.empty_cache()
115
+ if fmt == "wav":
116
+ return Response(wav.getvalue(), mimetype="audio/wav")
117
+ wav.seek(0, 0)
118
+ with BytesIO() as ofp:
119
+ wav2(wav, ofp, fmt)
120
+ return Response(
121
+ ofp.getvalue(),
122
+ mimetype="audio/mpeg" if fmt == "mp3" else "audio/ogg"
123
+ )
text/__init__.py ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from text.symbols import *
2
+
3
+
4
+ _symbol_to_id = {s: i for i, s in enumerate(symbols)}
5
+
6
+ def cleaned_text_to_sequence(cleaned_text, tones, language):
7
+ '''Converts a string of text to a sequence of IDs corresponding to the symbols in the text.
8
+ Args:
9
+ text: string to convert to a sequence
10
+ Returns:
11
+ List of integers corresponding to the symbols in the text
12
+ '''
13
+ phones = [_symbol_to_id[symbol] for symbol in cleaned_text]
14
+ tone_start = language_tone_start_map[language]
15
+ tones = [i + tone_start for i in tones]
16
+ lang_id = language_id_map[language]
17
+ lang_ids = [lang_id for i in phones]
18
+ return phones, tones, lang_ids
19
+
20
+ def get_bert(norm_text, word2ph, language):
21
+ from .chinese_bert import get_bert_feature as zh_bert
22
+ from .english_bert_mock import get_bert_feature as en_bert
23
+ lang_bert_func_map = {
24
+ 'ZH': zh_bert,
25
+ 'EN': en_bert
26
+ }
27
+ bert = lang_bert_func_map[language](norm_text, word2ph)
28
+ return bert
text/chinese.py ADDED
@@ -0,0 +1,193 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ import re
3
+
4
+ import cn2an
5
+ from pypinyin import lazy_pinyin, Style
6
+
7
+ from text import symbols
8
+ from text.symbols import punctuation
9
+ from text.tone_sandhi import ToneSandhi
10
+
11
+ current_file_path = os.path.dirname(__file__)
12
+ pinyin_to_symbol_map = {line.split("\t")[0]: line.strip().split("\t")[1] for line in
13
+ open(os.path.join(current_file_path, 'opencpop-strict.txt')).readlines()}
14
+
15
+ import jieba.posseg as psg
16
+
17
+
18
+ rep_map = {
19
+ ':': ',',
20
+ ';': ',',
21
+ ',': ',',
22
+ '。': '.',
23
+ '!': '!',
24
+ '?': '?',
25
+ '\n': '.',
26
+ "·": ",",
27
+ '、': ",",
28
+ '...': '…',
29
+ '$': '.',
30
+ '“': "'",
31
+ '”': "'",
32
+ '‘': "'",
33
+ '’': "'",
34
+ '(': "'",
35
+ ')': "'",
36
+ '(': "'",
37
+ ')': "'",
38
+ '《': "'",
39
+ '》': "'",
40
+ '【': "'",
41
+ '】': "'",
42
+ '[': "'",
43
+ ']': "'",
44
+ '—': "-",
45
+ '~': "-",
46
+ '~': "-",
47
+ '「': "'",
48
+ '」': "'",
49
+
50
+ }
51
+
52
+ tone_modifier = ToneSandhi()
53
+
54
+ def replace_punctuation(text):
55
+ text = text.replace("嗯", "恩").replace("呣","母")
56
+ pattern = re.compile('|'.join(re.escape(p) for p in rep_map.keys()))
57
+
58
+ replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
59
+
60
+ replaced_text = re.sub(r'[^\u4e00-\u9fa5'+"".join(punctuation)+r']+', '', replaced_text)
61
+
62
+ return replaced_text
63
+
64
+ def g2p(text):
65
+ pattern = r'(?<=[{0}])\s*'.format(''.join(punctuation))
66
+ sentences = [i for i in re.split(pattern, text) if i.strip()!='']
67
+ phones, tones, word2ph = _g2p(sentences)
68
+ assert sum(word2ph) == len(phones)
69
+ assert len(word2ph) == len(text) #Sometimes it will crash,you can add a try-catch.
70
+ phones = ['_'] + phones + ["_"]
71
+ tones = [0] + tones + [0]
72
+ word2ph = [1] + word2ph + [1]
73
+ return phones, tones, word2ph
74
+
75
+
76
+ def _get_initials_finals(word):
77
+ initials = []
78
+ finals = []
79
+ orig_initials = lazy_pinyin(
80
+ word, neutral_tone_with_five=True, style=Style.INITIALS)
81
+ orig_finals = lazy_pinyin(
82
+ word, neutral_tone_with_five=True, style=Style.FINALS_TONE3)
83
+ for c, v in zip(orig_initials, orig_finals):
84
+ initials.append(c)
85
+ finals.append(v)
86
+ return initials, finals
87
+
88
+
89
+ def _g2p(segments):
90
+ phones_list = []
91
+ tones_list = []
92
+ word2ph = []
93
+ for seg in segments:
94
+ pinyins = []
95
+ # Replace all English words in the sentence
96
+ seg = re.sub('[a-zA-Z]+', '', seg)
97
+ seg_cut = psg.lcut(seg)
98
+ initials = []
99
+ finals = []
100
+ seg_cut = tone_modifier.pre_merge_for_modify(seg_cut)
101
+ for word, pos in seg_cut:
102
+ if pos == 'eng':
103
+ continue
104
+ sub_initials, sub_finals = _get_initials_finals(word)
105
+ sub_finals = tone_modifier.modified_tone(word, pos,
106
+ sub_finals)
107
+ initials.append(sub_initials)
108
+ finals.append(sub_finals)
109
+
110
+ # assert len(sub_initials) == len(sub_finals) == len(word)
111
+ initials = sum(initials, [])
112
+ finals = sum(finals, [])
113
+ #
114
+ for c, v in zip(initials, finals):
115
+ raw_pinyin = c+v
116
+ # NOTE: post process for pypinyin outputs
117
+ # we discriminate i, ii and iii
118
+ if c == v:
119
+ assert c in punctuation
120
+ phone = [c]
121
+ tone = '0'
122
+ word2ph.append(1)
123
+ else:
124
+ v_without_tone = v[:-1]
125
+ tone = v[-1]
126
+
127
+ pinyin = c+v_without_tone
128
+ assert tone in '12345'
129
+
130
+ if c:
131
+ # 多音节
132
+ v_rep_map = {
133
+ "uei": 'ui',
134
+ 'iou': 'iu',
135
+ 'uen': 'un',
136
+ }
137
+ if v_without_tone in v_rep_map.keys():
138
+ pinyin = c+v_rep_map[v_without_tone]
139
+ else:
140
+ # 单音节
141
+ pinyin_rep_map = {
142
+ 'ing': 'ying',
143
+ 'i': 'yi',
144
+ 'in': 'yin',
145
+ 'u': 'wu',
146
+ }
147
+ if pinyin in pinyin_rep_map.keys():
148
+ pinyin = pinyin_rep_map[pinyin]
149
+ else:
150
+ single_rep_map = {
151
+ 'v': 'yu',
152
+ 'e': 'e',
153
+ 'i': 'y',
154
+ 'u': 'w',
155
+ }
156
+ if pinyin[0] in single_rep_map.keys():
157
+ pinyin = single_rep_map[pinyin[0]]+pinyin[1:]
158
+
159
+ assert pinyin in pinyin_to_symbol_map.keys(), (pinyin, seg, raw_pinyin)
160
+ phone = pinyin_to_symbol_map[pinyin].split(' ')
161
+ word2ph.append(len(phone))
162
+
163
+ phones_list += phone
164
+ tones_list += [int(tone)] * len(phone)
165
+ return phones_list, tones_list, word2ph
166
+
167
+
168
+
169
+ def text_normalize(text):
170
+ numbers = re.findall(r'\d+(?:\.?\d+)?', text)
171
+ for number in numbers:
172
+ text = text.replace(number, cn2an.an2cn(number), 1)
173
+ text = replace_punctuation(text)
174
+ return text
175
+
176
+ def get_bert_feature(text, word2ph):
177
+ from text import chinese_bert
178
+ return chinese_bert.get_bert_feature(text, word2ph)
179
+
180
+ if __name__ == '__main__':
181
+ from text.chinese_bert import get_bert_feature
182
+ text = "啊!但是《原神》是由,米哈\游自主, [研发]的一款全.新开放世界.冒险游戏"
183
+ text = text_normalize(text)
184
+ print(text)
185
+ phones, tones, word2ph = g2p(text)
186
+ bert = get_bert_feature(text, word2ph)
187
+
188
+ print(phones, tones, word2ph, bert.shape)
189
+
190
+
191
+ # # 示例用法
192
+ # text = "这是一个示例文本:,你好!这是一个测试...."
193
+ # print(g2p_paddle(text)) # 输出: 这是一个示例文本你好这是一个测试
text/chinese_bert.py ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch
2
+ from transformers import AutoTokenizer, AutoModelForMaskedLM
3
+
4
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
5
+
6
+ tokenizer = AutoTokenizer.from_pretrained("./bert/chinese-roberta-wwm-ext-large")
7
+ model = AutoModelForMaskedLM.from_pretrained("./bert/chinese-roberta-wwm-ext-large").to(device)
8
+
9
+ def get_bert_feature(text, word2ph):
10
+ with torch.no_grad():
11
+ inputs = tokenizer(text, return_tensors='pt')
12
+ for i in inputs:
13
+ inputs[i] = inputs[i].to(device)
14
+ res = model(**inputs, output_hidden_states=True)
15
+ res = torch.cat(res['hidden_states'][-3:-2], -1)[0].cpu()
16
+
17
+ assert len(word2ph) == len(text)+2
18
+ word2phone = word2ph
19
+ phone_level_feature = []
20
+ for i in range(len(word2phone)):
21
+ repeat_feature = res[i].repeat(word2phone[i], 1)
22
+ phone_level_feature.append(repeat_feature)
23
+
24
+ phone_level_feature = torch.cat(phone_level_feature, dim=0)
25
+
26
+
27
+ return phone_level_feature.T
28
+
29
+ if __name__ == '__main__':
30
+ # feature = get_bert_feature('你好,我是说的道理。')
31
+ import torch
32
+
33
+ word_level_feature = torch.rand(38, 1024) # 12个词,每个词1024维特征
34
+ word2phone = [1, 2, 1, 2, 2, 1, 2, 2, 1, 2, 2, 1, 2, 2, 2, 2, 2, 1, 1, 2, 2, 1, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 1]
35
+
36
+ # 计算总帧数
37
+ total_frames = sum(word2phone)
38
+ print(word_level_feature.shape)
39
+ print(word2phone)
40
+ phone_level_feature = []
41
+ for i in range(len(word2phone)):
42
+ print(word_level_feature[i].shape)
43
+
44
+ # 对每个词重复word2phone[i]次
45
+ repeat_feature = word_level_feature[i].repeat(word2phone[i], 1)
46
+ phone_level_feature.append(repeat_feature)
47
+
48
+ phone_level_feature = torch.cat(phone_level_feature, dim=0)
49
+ print(phone_level_feature.shape) # torch.Size([36, 1024])
50
+
text/cleaner.py ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from text import chinese, cleaned_text_to_sequence
2
+
3
+
4
+ language_module_map = {
5
+ 'ZH': chinese
6
+ }
7
+
8
+
9
+ def clean_text(text, language):
10
+ language_module = language_module_map[language]
11
+ norm_text = language_module.text_normalize(text)
12
+ phones, tones, word2ph = language_module.g2p(norm_text)
13
+ return norm_text, phones, tones, word2ph
14
+
15
+ def clean_text_bert(text, language):
16
+ language_module = language_module_map[language]
17
+ norm_text = language_module.text_normalize(text)
18
+ phones, tones, word2ph = language_module.g2p(norm_text)
19
+ bert = language_module.get_bert_feature(norm_text, word2ph)
20
+ return phones, tones, bert
21
+
22
+ def text_to_sequence(text, language):
23
+ norm_text, phones, tones, word2ph = clean_text(text, language)
24
+ return cleaned_text_to_sequence(phones, tones, language)
25
+
26
+ if __name__ == '__main__':
27
+ pass
text/cmudict.rep ADDED
The diff for this file is too large to render. See raw diff
 
text/cmudict_cache.pickle ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b9b21b20325471934ba92f2e4a5976989e7d920caa32e7a286eacb027d197949
3
+ size 6212655
text/english.py ADDED
@@ -0,0 +1,138 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import pickle
2
+ import os
3
+ import re
4
+ from g2p_en import G2p
5
+ from string import punctuation
6
+
7
+ from text import symbols
8
+
9
+ current_file_path = os.path.dirname(__file__)
10
+ CMU_DICT_PATH = os.path.join(current_file_path, 'cmudict.rep')
11
+ CACHE_PATH = os.path.join(current_file_path, 'cmudict_cache.pickle')
12
+ _g2p = G2p()
13
+
14
+ arpa = {'AH0', 'S', 'AH1', 'EY2', 'AE2', 'EH0', 'OW2', 'UH0', 'NG', 'B', 'G', 'AY0', 'M', 'AA0', 'F', 'AO0', 'ER2', 'UH1', 'IY1', 'AH2', 'DH', 'IY0', 'EY1', 'IH0', 'K', 'N', 'W', 'IY2', 'T', 'AA1', 'ER1', 'EH2', 'OY0', 'UH2', 'UW1', 'Z', 'AW2', 'AW1', 'V', 'UW2', 'AA2', 'ER', 'AW0', 'UW0', 'R', 'OW1', 'EH1', 'ZH', 'AE0', 'IH2', 'IH', 'Y', 'JH', 'P', 'AY1', 'EY0', 'OY2', 'TH', 'HH', 'D', 'ER0', 'CH', 'AO1', 'AE1', 'AO2', 'OY1', 'AY2', 'IH1', 'OW0', 'L', 'SH'}
15
+
16
+
17
+ def post_replace_ph(ph):
18
+ rep_map = {
19
+ ':': ',',
20
+ ';': ',',
21
+ ',': ',',
22
+ '。': '.',
23
+ '!': '!',
24
+ '?': '?',
25
+ '\n': '.',
26
+ "·": ",",
27
+ '、': ",",
28
+ '...': '…',
29
+ 'v': "V"
30
+ }
31
+ if ph in rep_map.keys():
32
+ ph = rep_map[ph]
33
+ if ph in symbols:
34
+ return ph
35
+ if ph not in symbols:
36
+ ph = 'UNK'
37
+ return ph
38
+
39
+ def read_dict():
40
+ g2p_dict = {}
41
+ start_line = 49
42
+ with open(CMU_DICT_PATH) as f:
43
+ line = f.readline()
44
+ line_index = 1
45
+ while line:
46
+ if line_index >= start_line:
47
+ line = line.strip()
48
+ word_split = line.split(' ')
49
+ word = word_split[0]
50
+
51
+ syllable_split = word_split[1].split(' - ')
52
+ g2p_dict[word] = []
53
+ for syllable in syllable_split:
54
+ phone_split = syllable.split(' ')
55
+ g2p_dict[word].append(phone_split)
56
+
57
+ line_index = line_index + 1
58
+ line = f.readline()
59
+
60
+ return g2p_dict
61
+
62
+
63
+ def cache_dict(g2p_dict, file_path):
64
+ with open(file_path, 'wb') as pickle_file:
65
+ pickle.dump(g2p_dict, pickle_file)
66
+
67
+
68
+ def get_dict():
69
+ if os.path.exists(CACHE_PATH):
70
+ with open(CACHE_PATH, 'rb') as pickle_file:
71
+ g2p_dict = pickle.load(pickle_file)
72
+ else:
73
+ g2p_dict = read_dict()
74
+ cache_dict(g2p_dict, CACHE_PATH)
75
+
76
+ return g2p_dict
77
+
78
+ eng_dict = get_dict()
79
+
80
+ def refine_ph(phn):
81
+ tone = 0
82
+ if re.search(r'\d$', phn):
83
+ tone = int(phn[-1]) + 1
84
+ phn = phn[:-1]
85
+ return phn.lower(), tone
86
+
87
+ def refine_syllables(syllables):
88
+ tones = []
89
+ phonemes = []
90
+ for phn_list in syllables:
91
+ for i in range(len(phn_list)):
92
+ phn = phn_list[i]
93
+ phn, tone = refine_ph(phn)
94
+ phonemes.append(phn)
95
+ tones.append(tone)
96
+ return phonemes, tones
97
+
98
+
99
+ def text_normalize(text):
100
+ # todo: eng text normalize
101
+ return text
102
+
103
+ def g2p(text):
104
+
105
+ phones = []
106
+ tones = []
107
+ words = re.split(r"([,;.\-\?\!\s+])", text)
108
+ for w in words:
109
+ if w.upper() in eng_dict:
110
+ phns, tns = refine_syllables(eng_dict[w.upper()])
111
+ phones += phns
112
+ tones += tns
113
+ else:
114
+ phone_list = list(filter(lambda p: p != " ", _g2p(w)))
115
+ for ph in phone_list:
116
+ if ph in arpa:
117
+ ph, tn = refine_ph(ph)
118
+ phones.append(ph)
119
+ tones.append(tn)
120
+ else:
121
+ phones.append(ph)
122
+ tones.append(0)
123
+ # todo: implement word2ph
124
+ word2ph = [1 for i in phones]
125
+
126
+ phones = [post_replace_ph(i) for i in phones]
127
+ return phones, tones, word2ph
128
+
129
+ if __name__ == "__main__":
130
+ # print(get_dict())
131
+ # print(eng_word_to_phoneme("hello"))
132
+ print(g2p("In this paper, we propose 1 DSPGAN, a GAN-based universal vocoder."))
133
+ # all_phones = set()
134
+ # for k, syllables in eng_dict.items():
135
+ # for group in syllables:
136
+ # for ph in group:
137
+ # all_phones.add(ph)
138
+ # print(all_phones)
text/english_bert_mock.py ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ import torch
2
+
3
+
4
+ def get_bert_feature(norm_text, word2ph):
5
+ return torch.zeros(1024, sum(word2ph))
text/japanese.py ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # modified from https://github.com/CjangCjengh/vits/blob/main/text/japanese.py
2
+ import re
3
+ import sys
4
+
5
+ import pyopenjtalk
6
+
7
+ from text import symbols
8
+
9
+ # Regular expression matching Japanese without punctuation marks:
10
+ _japanese_characters = re.compile(
11
+ r'[A-Za-z\d\u3005\u3040-\u30ff\u4e00-\u9fff\uff11-\uff19\uff21-\uff3a\uff41-\uff5a\uff66-\uff9d]')
12
+
13
+ # Regular expression matching non-Japanese characters or punctuation marks:
14
+ _japanese_marks = re.compile(
15
+ r'[^A-Za-z\d\u3005\u3040-\u30ff\u4e00-\u9fff\uff11-\uff19\uff21-\uff3a\uff41-\uff5a\uff66-\uff9d]')
16
+
17
+ # List of (symbol, Japanese) pairs for marks:
18
+ _symbols_to_japanese = [(re.compile('%s' % x[0]), x[1]) for x in [
19
+ ('%', 'パーセント')
20
+ ]]
21
+
22
+
23
+ # List of (consonant, sokuon) pairs:
24
+ _real_sokuon = [(re.compile('%s' % x[0]), x[1]) for x in [
25
+ (r'Q([↑↓]*[kg])', r'k#\1'),
26
+ (r'Q([↑↓]*[tdjʧ])', r't#\1'),
27
+ (r'Q([↑↓]*[sʃ])', r's\1'),
28
+ (r'Q([↑↓]*[pb])', r'p#\1')
29
+ ]]
30
+
31
+ # List of (consonant, hatsuon) pairs:
32
+ _real_hatsuon = [(re.compile('%s' % x[0]), x[1]) for x in [
33
+ (r'N([↑↓]*[pbm])', r'm\1'),
34
+ (r'N([↑↓]*[ʧʥj])', r'n^\1'),
35
+ (r'N([↑↓]*[tdn])', r'n\1'),
36
+ (r'N([↑↓]*[kg])', r'ŋ\1')
37
+ ]]
38
+
39
+
40
+
41
+ def post_replace_ph(ph):
42
+ rep_map = {
43
+ ':': ',',
44
+ ';': ',',
45
+ ',': ',',
46
+ '。': '.',
47
+ '!': '!',
48
+ '?': '?',
49
+ '\n': '.',
50
+ "·": ",",
51
+ '、': ",",
52
+ '...': '…',
53
+ 'v': "V"
54
+ }
55
+ if ph in rep_map.keys():
56
+ ph = rep_map[ph]
57
+ if ph in symbols:
58
+ return ph
59
+ if ph not in symbols:
60
+ ph = 'UNK'
61
+ return ph
62
+
63
+ def symbols_to_japanese(text):
64
+ for regex, replacement in _symbols_to_japanese:
65
+ text = re.sub(regex, replacement, text)
66
+ return text
67
+
68
+
69
+ def preprocess_jap(text):
70
+ '''Reference https://r9y9.github.io/ttslearn/latest/notebooks/ch10_Recipe-Tacotron.html'''
71
+ text = symbols_to_japanese(text)
72
+ sentences = re.split(_japanese_marks, text)
73
+ marks = re.findall(_japanese_marks, text)
74
+ text = []
75
+ for i, sentence in enumerate(sentences):
76
+ if re.match(_japanese_characters, sentence):
77
+ p = pyopenjtalk.g2p(sentence)
78
+ text += p.split(" ")
79
+
80
+ if i < len(marks):
81
+ text += [marks[i].replace(' ', '')]
82
+ return text
83
+
84
+ def text_normalize(text):
85
+ # todo: jap text normalize
86
+ return text
87
+
88
+ def g2p(norm_text):
89
+ phones = preprocess_jap(norm_text)
90
+ phones = [post_replace_ph(i) for i in phones]
91
+ # todo: implement tones and word2ph
92
+ tones = [0 for i in phones]
93
+ word2ph = [1 for i in phones]
94
+ return phones, tones, word2ph
95
+
96
+
97
+ if __name__ == '__main__':
98
+ for line in open("../../../Downloads/transcript_utf8.txt").readlines():
99
+ text = line.split(":")[1]
100
+ phones, tones, word2ph = g2p(text)
101
+ for p in phones:
102
+ if p == "z":
103
+ print(text, phones)
104
+ sys.exit(0)
text/opencpop-strict.txt ADDED
@@ -0,0 +1,429 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ a AA a
2
+ ai AA ai
3
+ an AA an
4
+ ang AA ang
5
+ ao AA ao
6
+ ba b a
7
+ bai b ai
8
+ ban b an
9
+ bang b ang
10
+ bao b ao
11
+ bei b ei
12
+ ben b en
13
+ beng b eng
14
+ bi b i
15
+ bian b ian
16
+ biao b iao
17
+ bie b ie
18
+ bin b in
19
+ bing b ing
20
+ bo b o
21
+ bu b u
22
+ ca c a
23
+ cai c ai
24
+ can c an
25
+ cang c ang
26
+ cao c ao
27
+ ce c e
28
+ cei c ei
29
+ cen c en
30
+ ceng c eng
31
+ cha ch a
32
+ chai ch ai
33
+ chan ch an
34
+ chang ch ang
35
+ chao ch ao
36
+ che ch e
37
+ chen ch en
38
+ cheng ch eng
39
+ chi ch ir
40
+ chong ch ong
41
+ chou ch ou
42
+ chu ch u
43
+ chua ch ua
44
+ chuai ch uai
45
+ chuan ch uan
46
+ chuang ch uang
47
+ chui ch ui
48
+ chun ch un
49
+ chuo ch uo
50
+ ci c i0
51
+ cong c ong
52
+ cou c ou
53
+ cu c u
54
+ cuan c uan
55
+ cui c ui
56
+ cun c un
57
+ cuo c uo
58
+ da d a
59
+ dai d ai
60
+ dan d an
61
+ dang d ang
62
+ dao d ao
63
+ de d e
64
+ dei d ei
65
+ den d en
66
+ deng d eng
67
+ di d i
68
+ dia d ia
69
+ dian d ian
70
+ diao d iao
71
+ die d ie
72
+ ding d ing
73
+ diu d iu
74
+ dong d ong
75
+ dou d ou
76
+ du d u
77
+ duan d uan
78
+ dui d ui
79
+ dun d un
80
+ duo d uo
81
+ e EE e
82
+ ei EE ei
83
+ en EE en
84
+ eng EE eng
85
+ er EE er
86
+ fa f a
87
+ fan f an
88
+ fang f ang
89
+ fei f ei
90
+ fen f en
91
+ feng f eng
92
+ fo f o
93
+ fou f ou
94
+ fu f u
95
+ ga g a
96
+ gai g ai
97
+ gan g an
98
+ gang g ang
99
+ gao g ao
100
+ ge g e
101
+ gei g ei
102
+ gen g en
103
+ geng g eng
104
+ gong g ong
105
+ gou g ou
106
+ gu g u
107
+ gua g ua
108
+ guai g uai
109
+ guan g uan
110
+ guang g uang
111
+ gui g ui
112
+ gun g un
113
+ guo g uo
114
+ ha h a
115
+ hai h ai
116
+ han h an
117
+ hang h ang
118
+ hao h ao
119
+ he h e
120
+ hei h ei
121
+ hen h en
122
+ heng h eng
123
+ hong h ong
124
+ hou h ou
125
+ hu h u
126
+ hua h ua
127
+ huai h uai
128
+ huan h uan
129
+ huang h uang
130
+ hui h ui
131
+ hun h un
132
+ huo h uo
133
+ ji j i
134
+ jia j ia
135
+ jian j ian
136
+ jiang j iang
137
+ jiao j iao
138
+ jie j ie
139
+ jin j in
140
+ jing j ing
141
+ jiong j iong
142
+ jiu j iu
143
+ ju j v
144
+ jv j v
145
+ juan j van
146
+ jvan j van
147
+ jue j ve
148
+ jve j ve
149
+ jun j vn
150
+ jvn j vn
151
+ ka k a
152
+ kai k ai
153
+ kan k an
154
+ kang k ang
155
+ kao k ao
156
+ ke k e
157
+ kei k ei
158
+ ken k en
159
+ keng k eng
160
+ kong k ong
161
+ kou k ou
162
+ ku k u
163
+ kua k ua
164
+ kuai k uai
165
+ kuan k uan
166
+ kuang k uang
167
+ kui k ui
168
+ kun k un
169
+ kuo k uo
170
+ la l a
171
+ lai l ai
172
+ lan l an
173
+ lang l ang
174
+ lao l ao
175
+ le l e
176
+ lei l ei
177
+ leng l eng
178
+ li l i
179
+ lia l ia
180
+ lian l ian
181
+ liang l iang
182
+ liao l iao
183
+ lie l ie
184
+ lin l in
185
+ ling l ing
186
+ liu l iu
187
+ lo l o
188
+ long l ong
189
+ lou l ou
190
+ lu l u
191
+ luan l uan
192
+ lun l un
193
+ luo l uo
194
+ lv l v
195
+ lve l ve
196
+ ma m a
197
+ mai m ai
198
+ man m an
199
+ mang m ang
200
+ mao m ao
201
+ me m e
202
+ mei m ei
203
+ men m en
204
+ meng m eng
205
+ mi m i
206
+ mian m ian
207
+ miao m iao
208
+ mie m ie
209
+ min m in
210
+ ming m ing
211
+ miu m iu
212
+ mo m o
213
+ mou m ou
214
+ mu m u
215
+ na n a
216
+ nai n ai
217
+ nan n an
218
+ nang n ang
219
+ nao n ao
220
+ ne n e
221
+ nei n ei
222
+ nen n en
223
+ neng n eng
224
+ ni n i
225
+ nian n ian
226
+ niang n iang
227
+ niao n iao
228
+ nie n ie
229
+ nin n in
230
+ ning n ing
231
+ niu n iu
232
+ nong n ong
233
+ nou n ou
234
+ nu n u
235
+ nuan n uan
236
+ nun n un
237
+ nuo n uo
238
+ nv n v
239
+ nve n ve
240
+ o OO o
241
+ ou OO ou
242
+ pa p a
243
+ pai p ai
244
+ pan p an
245
+ pang p ang
246
+ pao p ao
247
+ pei p ei
248
+ pen p en
249
+ peng p eng
250
+ pi p i
251
+ pian p ian
252
+ piao p iao
253
+ pie p ie
254
+ pin p in
255
+ ping p ing
256
+ po p o
257
+ pou p ou
258
+ pu p u
259
+ qi q i
260
+ qia q ia
261
+ qian q ian
262
+ qiang q iang
263
+ qiao q iao
264
+ qie q ie
265
+ qin q in
266
+ qing q ing
267
+ qiong q iong
268
+ qiu q iu
269
+ qu q v
270
+ qv q v
271
+ quan q van
272
+ qvan q van
273
+ que q ve
274
+ qve q ve
275
+ qun q vn
276
+ qvn q vn
277
+ ran r an
278
+ rang r ang
279
+ rao r ao
280
+ re r e
281
+ ren r en
282
+ reng r eng
283
+ ri r ir
284
+ rong r ong
285
+ rou r ou
286
+ ru r u
287
+ rua r ua
288
+ ruan r uan
289
+ rui r ui
290
+ run r un
291
+ ruo r uo
292
+ sa s a
293
+ sai s ai
294
+ san s an
295
+ sang s ang
296
+ sao s ao
297
+ se s e
298
+ sen s en
299
+ seng s eng
300
+ sha sh a
301
+ shai sh ai
302
+ shan sh an
303
+ shang sh ang
304
+ shao sh ao
305
+ she sh e
306
+ shei sh ei
307
+ shen sh en
308
+ sheng sh eng
309
+ shi sh ir
310
+ shou sh ou
311
+ shu sh u
312
+ shua sh ua
313
+ shuai sh uai
314
+ shuan sh uan
315
+ shuang sh uang
316
+ shui sh ui
317
+ shun sh un
318
+ shuo sh uo
319
+ si s i0
320
+ song s ong
321
+ sou s ou
322
+ su s u
323
+ suan s uan
324
+ sui s ui
325
+ sun s un
326
+ suo s uo
327
+ ta t a
328
+ tai t ai
329
+ tan t an
330
+ tang t ang
331
+ tao t ao
332
+ te t e
333
+ tei t ei
334
+ teng t eng
335
+ ti t i
336
+ tian t ian
337
+ tiao t iao
338
+ tie t ie
339
+ ting t ing
340
+ tong t ong
341
+ tou t ou
342
+ tu t u
343
+ tuan t uan
344
+ tui t ui
345
+ tun t un
346
+ tuo t uo
347
+ wa w a
348
+ wai w ai
349
+ wan w an
350
+ wang w ang
351
+ wei w ei
352
+ wen w en
353
+ weng w eng
354
+ wo w o
355
+ wu w u
356
+ xi x i
357
+ xia x ia
358
+ xian x ian
359
+ xiang x iang
360
+ xiao x iao
361
+ xie x ie
362
+ xin x in
363
+ xing x ing
364
+ xiong x iong
365
+ xiu x iu
366
+ xu x v
367
+ xv x v
368
+ xuan x van
369
+ xvan x van
370
+ xue x ve
371
+ xve x ve
372
+ xun x vn
373
+ xvn x vn
374
+ ya y a
375
+ yan y En
376
+ yang y ang
377
+ yao y ao
378
+ ye y E
379
+ yi y i
380
+ yin y in
381
+ ying y ing
382
+ yo y o
383
+ yong y ong
384
+ you y ou
385
+ yu y v
386
+ yv y v
387
+ yuan y van
388
+ yvan y van
389
+ yue y ve
390
+ yve y ve
391
+ yun y vn
392
+ yvn y vn
393
+ za z a
394
+ zai z ai
395
+ zan z an
396
+ zang z ang
397
+ zao z ao
398
+ ze z e
399
+ zei z ei
400
+ zen z en
401
+ zeng z eng
402
+ zha zh a
403
+ zhai zh ai
404
+ zhan zh an
405
+ zhang zh ang
406
+ zhao zh ao
407
+ zhe zh e
408
+ zhei zh ei
409
+ zhen zh en
410
+ zheng zh eng
411
+ zhi zh ir
412
+ zhong zh ong
413
+ zhou zh ou
414
+ zhu zh u
415
+ zhua zh ua
416
+ zhuai zh uai
417
+ zhuan zh uan
418
+ zhuang zh uang
419
+ zhui zh ui
420
+ zhun zh un
421
+ zhuo zh uo
422
+ zi z i0
423
+ zong z ong
424
+ zou z ou
425
+ zu z u
426
+ zuan z uan
427
+ zui z ui
428
+ zun z un
429
+ zuo z uo
text/symbols.py ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ punctuation = ['!', '?', '…', ",", ".", "'", '-']
2
+ pu_symbols = punctuation + ["SP", "UNK"]
3
+ pad = '_'
4
+
5
+ # chinese
6
+ zh_symbols = ['E', 'En', 'a', 'ai', 'an', 'ang', 'ao', 'b', 'c', 'ch', 'd', 'e', 'ei', 'en', 'eng', 'er', 'f', 'g', 'h',
7
+ 'i', 'i0', 'ia', 'ian', 'iang', 'iao', 'ie', 'in', 'ing', 'iong', 'ir', 'iu', 'j', 'k', 'l', 'm', 'n', 'o',
8
+ 'ong',
9
+ 'ou', 'p', 'q', 'r', 's', 'sh', 't', 'u', 'ua', 'uai', 'uan', 'uang', 'ui', 'un', 'uo', 'v', 'van', 've', 'vn',
10
+ 'w', 'x', 'y', 'z', 'zh',
11
+ "AA", "EE", "OO"]
12
+ num_zh_tones = 6
13
+
14
+ # japanese
15
+ ja_symbols = ['I', 'N', 'U', 'a', 'b', 'by', 'ch', 'cl', 'd', 'dy', 'e', 'f', 'g', 'gy', 'h', 'hy', 'i', 'j', 'k', 'ky',
16
+ 'm', 'my', 'n', 'ny', 'o', 'p', 'py', 'r', 'ry', 's', 'sh', 't', 'ts', 'u', 'V', 'w', 'y', 'z']
17
+ num_ja_tones = 1
18
+
19
+ # English
20
+ en_symbols = ['aa', 'ae', 'ah', 'ao', 'aw', 'ay', 'b', 'ch', 'd', 'dh', 'eh', 'er', 'ey', 'f', 'g', 'hh', 'ih', 'iy',
21
+ 'jh', 'k', 'l', 'm', 'n', 'ng', 'ow', 'oy', 'p', 'r', 's',
22
+ 'sh', 't', 'th', 'uh', 'uw', 'V', 'w', 'y', 'z', 'zh']
23
+ num_en_tones = 4
24
+
25
+ # combine all symbols
26
+ normal_symbols = sorted(set(zh_symbols + ja_symbols + en_symbols))
27
+ symbols = [pad] + normal_symbols + pu_symbols
28
+ sil_phonemes_ids = [symbols.index(i) for i in pu_symbols]
29
+
30
+ # combine all tones
31
+ num_tones = num_zh_tones + num_ja_tones + num_en_tones
32
+
33
+ # language maps
34
+ language_id_map = {
35
+ 'ZH': 0,
36
+ "JA": 1,
37
+ "EN": 2
38
+ }
39
+ num_languages = len(language_id_map.keys())
40
+
41
+ language_tone_start_map = {
42
+ 'ZH': 0,
43
+ "JA": num_zh_tones,
44
+ "EN": num_zh_tones + num_ja_tones
45
+ }
46
+
47
+ if __name__ == '__main__':
48
+ a = set(zh_symbols)
49
+ b = set(en_symbols)
50
+ print(sorted(a&b))
51
+
text/tone_sandhi.py ADDED
@@ -0,0 +1,351 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+ from typing import List
15
+ from typing import Tuple
16
+
17
+ import jieba
18
+ from pypinyin import lazy_pinyin
19
+ from pypinyin import Style
20
+
21
+
22
+ class ToneSandhi():
23
+ def __init__(self):
24
+ self.must_neural_tone_words = {
25
+ '麻烦', '麻利', '鸳鸯', '高粱', '骨头', '骆驼', '马虎', '首饰', '馒头', '馄饨', '风筝',
26
+ '难为', '队伍', '阔气', '闺女', '门道', '锄头', '铺盖', '铃铛', '铁匠', '钥匙', '里脊',
27
+ '里头', '部分', '那么', '道士', '造化', '迷糊', '连累', '这么', '这个', '运气', '过去',
28
+ '软和', '转悠', '踏实', '跳蚤', '跟头', '趔趄', '财主', '豆腐', '讲究', '记性', '记号',
29
+ '认识', '规矩', '见识', '裁缝', '补丁', '衣裳', '衣服', '衙门', '街坊', '行李', '行当',
30
+ '蛤蟆', '蘑菇', '薄荷', '葫芦', '葡萄', '萝卜', '荸荠', '苗条', '苗头', '苍蝇', '芝麻',
31
+ '舒服', '舒坦', '舌头', '自在', '膏药', '脾气', '脑袋', '脊梁', '能耐', '胳膊', '胭脂',
32
+ '胡萝', '胡琴', '胡同', '聪明', '耽误', '耽搁', '耷拉', '耳朵', '老爷', '老实', '老婆',
33
+ '老头', '老太', '翻腾', '罗嗦', '罐头', '编辑', '结实', '红火', '累赘', '糨糊', '糊涂',
34
+ '精神', '粮食', '簸箕', '篱笆', '算计', '算盘', '答应', '笤帚', '笑语', '笑话', '窟窿',
35
+ '窝囊', '窗户', '稳当', '稀罕', '称呼', '秧歌', '秀气', '秀才', '福气', '祖宗', '砚台',
36
+ '码头', '石榴', '石头', '石匠', '知识', '眼睛', '眯缝', '眨巴', '眉毛', '相声', '盘算',
37
+ '白净', '痢疾', '痛快', '疟疾', '疙瘩', '疏忽', '畜生', '生意', '甘蔗', '琵琶', '琢磨',
38
+ '琉璃', '玻璃', '玫瑰', '玄乎', '狐狸', '状元', '特务', '牲口', '牙碜', '牌楼', '爽快',
39
+ '爱人', '热闹', '烧饼', '烟筒', '烂糊', '点心', '炊帚', '灯笼', '火候', '漂亮', '滑溜',
40
+ '溜达', '温和', '清楚', '消息', '浪头', '活泼', '比方', '正经', '欺负', '模糊', '槟榔',
41
+ '棺材', '棒槌', '棉花', '核桃', '栅栏', '柴火', '架势', '枕头', '枇杷', '机灵', '本事',
42
+ '木头', '木匠', '朋友', '月饼', '月亮', '暖和', '明白', '时候', '新鲜', '故事', '收拾',
43
+ '收成', '提防', '挖苦', '挑剔', '指甲', '指头', '拾掇', '拳头', '拨弄', '招牌', '招呼',
44
+ '抬举', '护士', '折腾', '扫帚', '打量', '打算', '打点', '打扮', '打听', '打发', '扎实',
45
+ '扁担', '戒指', '懒得', '意识', '意思', '情形', '悟性', '怪物', '思量', '怎么', '念头',
46
+ '念叨', '快活', '忙活', '志气', '心思', '得罪', '张罗', '弟兄', '开通', '应酬', '庄稼',
47
+ '干事', '帮手', '帐篷', '希罕', '师父', '师傅', '巴结', '巴掌', '差事', '工夫', '岁数',
48
+ '屁股', '尾巴', '少爷', '小气', '小伙', '将就', '对头', '对付', '寡妇', '家伙', '客气',
49
+ '实在', '官司', '学问', '学生', '字号', '嫁妆', '媳妇', '媒人', '婆家', '娘家', '委屈',
50
+ '姑娘', '姐夫', '妯娌', '妥当', '妖精', '奴才', '女婿', '头发', '太阳', '大爷', '大方',
51
+ '大意', '大夫', '多少', '多么', '外甥', '壮实', '地道', '地方', '在乎', '困难', '嘴巴',
52
+ '嘱咐', '嘟囔', '嘀咕', '喜欢', '喇嘛', '喇叭', '商量', '唾沫', '哑巴', '哈欠', '哆嗦',
53
+ '咳嗽', '和尚', '告诉', '告示', '含糊', '吓唬', '后头', '名字', '名堂', '合同', '吆喝',
54
+ '叫唤', '口袋', '厚道', '厉害', '千斤', '包袱', '包涵', '匀称', '勤快', '动静', '动弹',
55
+ '功夫', '力气', '前头', '刺猬', '刺激', '别扭', '利落', '利索', '利害', '分析', '出息',
56
+ '凑合', '凉快', '冷战', '冤枉', '冒失', '养活', '关系', '先生', '兄弟', '便宜', '使唤',
57
+ '佩服', '作坊', '体面', '位置', '似的', '伙计', '休息', '什么', '人家', '亲戚', '亲家',
58
+ '交情', '云彩', '事情', '买卖', '主意', '丫头', '丧气', '两口', '东西', '东家', '世故',
59
+ '不由', '不在', '下水', '下巴', '上头', '上司', '丈夫', '丈人', '一辈', '那个', '菩萨',
60
+ '父亲', '母亲', '咕噜', '邋遢', '费用', '冤家', '甜头', '介绍', '荒唐', '大人', '泥鳅',
61
+ '幸福', '熟悉', '计划', '扑腾', '蜡烛', '姥爷', '照顾', '喉咙', '吉他', '弄堂', '蚂蚱',
62
+ '凤凰', '拖沓', '寒碜', '糟蹋', '倒腾', '报复', '逻辑', '盘缠', '喽啰', '牢骚', '咖喱',
63
+ '扫把', '惦记'
64
+ }
65
+ self.must_not_neural_tone_words = {
66
+ "男子", "女子", "分子", "原子", "量子", "莲子", "石子", "瓜子", "电子", "人人", "虎虎"
67
+ }
68
+ self.punc = ":,;。?!“”‘’':,;.?!"
69
+
70
+ # the meaning of jieba pos tag: https://blog.csdn.net/weixin_44174352/article/details/113731041
71
+ # e.g.
72
+ # word: "家里"
73
+ # pos: "s"
74
+ # finals: ['ia1', 'i3']
75
+ def _neural_sandhi(self, word: str, pos: str,
76
+ finals: List[str]) -> List[str]:
77
+
78
+ # reduplication words for n. and v. e.g. 奶奶, 试试, 旺旺
79
+ for j, item in enumerate(word):
80
+ if j - 1 >= 0 and item == word[j - 1] and pos[0] in {
81
+ "n", "v", "a"
82
+ } and word not in self.must_not_neural_tone_words:
83
+ finals[j] = finals[j][:-1] + "5"
84
+ ge_idx = word.find("个")
85
+ if len(word) >= 1 and word[-1] in "吧呢啊呐噻嘛吖嗨呐哦哒额滴哩哟喽啰耶喔诶":
86
+ finals[-1] = finals[-1][:-1] + "5"
87
+ elif len(word) >= 1 and word[-1] in "的地得":
88
+ finals[-1] = finals[-1][:-1] + "5"
89
+ # e.g. 走了, 看着, 去过
90
+ # elif len(word) == 1 and word in "了着过" and pos in {"ul", "uz", "ug"}:
91
+ # finals[-1] = finals[-1][:-1] + "5"
92
+ elif len(word) > 1 and word[-1] in "们子" and pos in {
93
+ "r", "n"
94
+ } and word not in self.must_not_neural_tone_words:
95
+ finals[-1] = finals[-1][:-1] + "5"
96
+ # e.g. 桌上, 地下, 家里
97
+ elif len(word) > 1 and word[-1] in "上下里" and pos in {"s", "l", "f"}:
98
+ finals[-1] = finals[-1][:-1] + "5"
99
+ # e.g. 上来, 下去
100
+ elif len(word) > 1 and word[-1] in "来去" and word[-2] in "上下进出回过起开":
101
+ finals[-1] = finals[-1][:-1] + "5"
102
+ # 个做量词
103
+ elif (ge_idx >= 1 and
104
+ (word[ge_idx - 1].isnumeric() or
105
+ word[ge_idx - 1] in "几有两半多各整每做是")) or word == '个':
106
+ finals[ge_idx] = finals[ge_idx][:-1] + "5"
107
+ else:
108
+ if word in self.must_neural_tone_words or word[
109
+ -2:] in self.must_neural_tone_words:
110
+ finals[-1] = finals[-1][:-1] + "5"
111
+
112
+ word_list = self._split_word(word)
113
+ finals_list = [finals[:len(word_list[0])], finals[len(word_list[0]):]]
114
+ for i, word in enumerate(word_list):
115
+ # conventional neural in Chinese
116
+ if word in self.must_neural_tone_words or word[
117
+ -2:] in self.must_neural_tone_words:
118
+ finals_list[i][-1] = finals_list[i][-1][:-1] + "5"
119
+ finals = sum(finals_list, [])
120
+ return finals
121
+
122
+ def _bu_sandhi(self, word: str, finals: List[str]) -> List[str]:
123
+ # e.g. 看不懂
124
+ if len(word) == 3 and word[1] == "不":
125
+ finals[1] = finals[1][:-1] + "5"
126
+ else:
127
+ for i, char in enumerate(word):
128
+ # "不" before tone4 should be bu2, e.g. 不怕
129
+ if char == "不" and i + 1 < len(word) and finals[i +
130
+ 1][-1] == "4":
131
+ finals[i] = finals[i][:-1] + "2"
132
+ return finals
133
+
134
+ def _yi_sandhi(self, word: str, finals: List[str]) -> List[str]:
135
+ # "一" in number sequences, e.g. 一零零, 二一零
136
+ if word.find("一") != -1 and all(
137
+ [item.isnumeric() for item in word if item != "一"]):
138
+ return finals
139
+ # "一" between reduplication words shold be yi5, e.g. 看一看
140
+ elif len(word) == 3 and word[1] == "一" and word[0] == word[-1]:
141
+ finals[1] = finals[1][:-1] + "5"
142
+ # when "一" is ordinal word, it should be yi1
143
+ elif word.startswith("第一"):
144
+ finals[1] = finals[1][:-1] + "1"
145
+ else:
146
+ for i, char in enumerate(word):
147
+ if char == "一" and i + 1 < len(word):
148
+ # "一" before tone4 should be yi2, e.g. 一段
149
+ if finals[i + 1][-1] == "4":
150
+ finals[i] = finals[i][:-1] + "2"
151
+ # "一" before non-tone4 should be yi4, e.g. 一天
152
+ else:
153
+ # "一" 后面如果是标点,还读一声
154
+ if word[i + 1] not in self.punc:
155
+ finals[i] = finals[i][:-1] + "4"
156
+ return finals
157
+
158
+ def _split_word(self, word: str) -> List[str]:
159
+ word_list = jieba.cut_for_search(word)
160
+ word_list = sorted(word_list, key=lambda i: len(i), reverse=False)
161
+ first_subword = word_list[0]
162
+ first_begin_idx = word.find(first_subword)
163
+ if first_begin_idx == 0:
164
+ second_subword = word[len(first_subword):]
165
+ new_word_list = [first_subword, second_subword]
166
+ else:
167
+ second_subword = word[:-len(first_subword)]
168
+ new_word_list = [second_subword, first_subword]
169
+ return new_word_list
170
+
171
+ def _three_sandhi(self, word: str, finals: List[str]) -> List[str]:
172
+ if len(word) == 2 and self._all_tone_three(finals):
173
+ finals[0] = finals[0][:-1] + "2"
174
+ elif len(word) == 3:
175
+ word_list = self._split_word(word)
176
+ if self._all_tone_three(finals):
177
+ # disyllabic + monosyllabic, e.g. 蒙古/包
178
+ if len(word_list[0]) == 2:
179
+ finals[0] = finals[0][:-1] + "2"
180
+ finals[1] = finals[1][:-1] + "2"
181
+ # monosyllabic + disyllabic, e.g. 纸/老虎
182
+ elif len(word_list[0]) == 1:
183
+ finals[1] = finals[1][:-1] + "2"
184
+ else:
185
+ finals_list = [
186
+ finals[:len(word_list[0])], finals[len(word_list[0]):]
187
+ ]
188
+ if len(finals_list) == 2:
189
+ for i, sub in enumerate(finals_list):
190
+ # e.g. 所有/人
191
+ if self._all_tone_three(sub) and len(sub) == 2:
192
+ finals_list[i][0] = finals_list[i][0][:-1] + "2"
193
+ # e.g. 好/喜欢
194
+ elif i == 1 and not self._all_tone_three(sub) and finals_list[i][0][-1] == "3" and \
195
+ finals_list[0][-1][-1] == "3":
196
+
197
+ finals_list[0][-1] = finals_list[0][-1][:-1] + "2"
198
+ finals = sum(finals_list, [])
199
+ # split idiom into two words who's length is 2
200
+ elif len(word) == 4:
201
+ finals_list = [finals[:2], finals[2:]]
202
+ finals = []
203
+ for sub in finals_list:
204
+ if self._all_tone_three(sub):
205
+ sub[0] = sub[0][:-1] + "2"
206
+ finals += sub
207
+
208
+ return finals
209
+
210
+ def _all_tone_three(self, finals: List[str]) -> bool:
211
+ return all(x[-1] == "3" for x in finals)
212
+
213
+ # merge "不" and the word behind it
214
+ # if don't merge, "不" sometimes appears alone according to jieba, which may occur sandhi error
215
+ def _merge_bu(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
216
+ new_seg = []
217
+ last_word = ""
218
+ for word, pos in seg:
219
+ if last_word == "不":
220
+ word = last_word + word
221
+ if word != "不":
222
+ new_seg.append((word, pos))
223
+ last_word = word[:]
224
+ if last_word == "不":
225
+ new_seg.append((last_word, 'd'))
226
+ last_word = ""
227
+ return new_seg
228
+
229
+ # function 1: merge "一" and reduplication words in it's left and right, e.g. "听","一","听" ->"听一听"
230
+ # function 2: merge single "一" and the word behind it
231
+ # if don't merge, "一" sometimes appears alone according to jieba, which may occur sandhi error
232
+ # e.g.
233
+ # input seg: [('听', 'v'), ('一', 'm'), ('听', 'v')]
234
+ # output seg: [['听一听', 'v']]
235
+ def _merge_yi(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
236
+ new_seg = []
237
+ # function 1
238
+ for i, (word, pos) in enumerate(seg):
239
+ if i - 1 >= 0 and word == "一" and i + 1 < len(seg) and seg[i - 1][
240
+ 0] == seg[i + 1][0] and seg[i - 1][1] == "v":
241
+ new_seg[i - 1][0] = new_seg[i - 1][0] + "一" + new_seg[i - 1][0]
242
+ else:
243
+ if i - 2 >= 0 and seg[i - 1][0] == "一" and seg[i - 2][
244
+ 0] == word and pos == "v":
245
+ continue
246
+ else:
247
+ new_seg.append([word, pos])
248
+ seg = new_seg
249
+ new_seg = []
250
+ # function 2
251
+ for i, (word, pos) in enumerate(seg):
252
+ if new_seg and new_seg[-1][0] == "一":
253
+ new_seg[-1][0] = new_seg[-1][0] + word
254
+ else:
255
+ new_seg.append([word, pos])
256
+ return new_seg
257
+
258
+ # the first and the second words are all_tone_three
259
+ def _merge_continuous_three_tones(
260
+ self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
261
+ new_seg = []
262
+ sub_finals_list = [
263
+ lazy_pinyin(
264
+ word, neutral_tone_with_five=True, style=Style.FINALS_TONE3)
265
+ for (word, pos) in seg
266
+ ]
267
+ assert len(sub_finals_list) == len(seg)
268
+ merge_last = [False] * len(seg)
269
+ for i, (word, pos) in enumerate(seg):
270
+ if i - 1 >= 0 and self._all_tone_three(
271
+ sub_finals_list[i - 1]) and self._all_tone_three(
272
+ sub_finals_list[i]) and not merge_last[i - 1]:
273
+ # if the last word is reduplication, not merge, because reduplication need to be _neural_sandhi
274
+ if not self._is_reduplication(seg[i - 1][0]) and len(
275
+ seg[i - 1][0]) + len(seg[i][0]) <= 3:
276
+ new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
277
+ merge_last[i] = True
278
+ else:
279
+ new_seg.append([word, pos])
280
+ else:
281
+ new_seg.append([word, pos])
282
+
283
+ return new_seg
284
+
285
+ def _is_reduplication(self, word: str) -> bool:
286
+ return len(word) == 2 and word[0] == word[1]
287
+
288
+ # the last char of first word and the first char of second word is tone_three
289
+ def _merge_continuous_three_tones_2(
290
+ self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
291
+ new_seg = []
292
+ sub_finals_list = [
293
+ lazy_pinyin(
294
+ word, neutral_tone_with_five=True, style=Style.FINALS_TONE3)
295
+ for (word, pos) in seg
296
+ ]
297
+ assert len(sub_finals_list) == len(seg)
298
+ merge_last = [False] * len(seg)
299
+ for i, (word, pos) in enumerate(seg):
300
+ if i - 1 >= 0 and sub_finals_list[i - 1][-1][-1] == "3" and sub_finals_list[i][0][-1] == "3" and not \
301
+ merge_last[i - 1]:
302
+ # if the last word is reduplication, not merge, because reduplication need to be _neural_sandhi
303
+ if not self._is_reduplication(seg[i - 1][0]) and len(
304
+ seg[i - 1][0]) + len(seg[i][0]) <= 3:
305
+ new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
306
+ merge_last[i] = True
307
+ else:
308
+ new_seg.append([word, pos])
309
+ else:
310
+ new_seg.append([word, pos])
311
+ return new_seg
312
+
313
+ def _merge_er(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
314
+ new_seg = []
315
+ for i, (word, pos) in enumerate(seg):
316
+ if i - 1 >= 0 and word == "儿" and seg[i-1][0] != "#":
317
+ new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
318
+ else:
319
+ new_seg.append([word, pos])
320
+ return new_seg
321
+
322
+ def _merge_reduplication(
323
+ self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
324
+ new_seg = []
325
+ for i, (word, pos) in enumerate(seg):
326
+ if new_seg and word == new_seg[-1][0]:
327
+ new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
328
+ else:
329
+ new_seg.append([word, pos])
330
+ return new_seg
331
+
332
+ def pre_merge_for_modify(
333
+ self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
334
+ seg = self._merge_bu(seg)
335
+ try:
336
+ seg = self._merge_yi(seg)
337
+ except:
338
+ print("_merge_yi failed")
339
+ seg = self._merge_reduplication(seg)
340
+ seg = self._merge_continuous_three_tones(seg)
341
+ seg = self._merge_continuous_three_tones_2(seg)
342
+ seg = self._merge_er(seg)
343
+ return seg
344
+
345
+ def modified_tone(self, word: str, pos: str,
346
+ finals: List[str]) -> List[str]:
347
+ finals = self._bu_sandhi(word, finals)
348
+ finals = self._yi_sandhi(word, finals)
349
+ finals = self._neural_sandhi(word, pos, finals)
350
+ finals = self._three_sandhi(word, finals)
351
+ return finals
train_ms.py ADDED
@@ -0,0 +1,402 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ import json
3
+ import argparse
4
+ import itertools
5
+ import math
6
+ import torch
7
+ import shutil
8
+ from torch import nn, optim
9
+ from torch.nn import functional as F
10
+ from torch.utils.data import DataLoader
11
+ from torch.utils.tensorboard import SummaryWriter
12
+ import torch.multiprocessing as mp
13
+ import torch.distributed as dist
14
+ from torch.nn.parallel import DistributedDataParallel as DDP
15
+ from torch.cuda.amp import autocast, GradScaler
16
+ from tqdm import tqdm
17
+ import logging
18
+ logging.getLogger('numba').setLevel(logging.WARNING)
19
+ import commons
20
+ import utils
21
+ from data_utils import (
22
+ TextAudioSpeakerLoader,
23
+ TextAudioSpeakerCollate,
24
+ DistributedBucketSampler
25
+ )
26
+ from models import (
27
+ SynthesizerTrn,
28
+ MultiPeriodDiscriminator,
29
+ DurationDiscriminator,
30
+ )
31
+ from losses import (
32
+ generator_loss,
33
+ discriminator_loss,
34
+ feature_loss,
35
+ kl_loss
36
+ )
37
+ from mel_processing import mel_spectrogram_torch, spec_to_mel_torch
38
+ from text.symbols import symbols
39
+
40
+ torch.backends.cudnn.benchmark = True
41
+ torch.backends.cuda.matmul.allow_tf32 = True
42
+ torch.backends.cudnn.allow_tf32 = True
43
+ torch.set_float32_matmul_precision('medium')
44
+ global_step = 0
45
+
46
+
47
+ def main():
48
+ """Assume Single Node Multi GPUs Training Only"""
49
+ assert torch.cuda.is_available(), "CPU training is not allowed."
50
+
51
+ n_gpus = torch.cuda.device_count()
52
+ os.environ['MASTER_ADDR'] = 'localhost'
53
+ os.environ['MASTER_PORT'] = '65280'
54
+
55
+ hps = utils.get_hparams()
56
+ if not hps.cont:
57
+ shutil.copy('./pretrained_models/D_0.pth','./logs/OUTPUT_MODEL/D_0.pth')
58
+ shutil.copy('./pretrained_models/G_0.pth','./logs/OUTPUT_MODEL/G_0.pth')
59
+ shutil.copy('./pretrained_models/DUR_0.pth','./logs/OUTPUT_MODEL/DUR_0.pth')
60
+ mp.spawn(run, nprocs=n_gpus, args=(n_gpus, hps,))
61
+
62
+
63
+ def run(rank, n_gpus, hps):
64
+ global global_step
65
+ if rank == 0:
66
+ logger = utils.get_logger(hps.model_dir)
67
+ logger.info(hps)
68
+ utils.check_git_hash(hps.model_dir)
69
+ writer = SummaryWriter(log_dir=hps.model_dir)
70
+ writer_eval = SummaryWriter(log_dir=os.path.join(hps.model_dir, "eval"))
71
+
72
+ dist.init_process_group(backend= 'gloo' if os.name == 'nt' else 'nccl', init_method='env://', world_size=n_gpus, rank=rank)
73
+ torch.manual_seed(hps.train.seed)
74
+ torch.cuda.set_device(rank)
75
+
76
+ train_dataset = TextAudioSpeakerLoader(hps.data.training_files, hps.data)
77
+ train_sampler = DistributedBucketSampler(
78
+ train_dataset,
79
+ hps.train.batch_size,
80
+ [32, 300, 400, 500, 600, 700, 800, 900, 1000],
81
+ num_replicas=n_gpus,
82
+ rank=rank,
83
+ shuffle=True)
84
+ collate_fn = TextAudioSpeakerCollate()
85
+ train_loader = DataLoader(train_dataset, num_workers=2, shuffle=False, pin_memory=True,
86
+ collate_fn=collate_fn, batch_sampler=train_sampler)
87
+ if rank == 0:
88
+ eval_dataset = TextAudioSpeakerLoader(hps.data.validation_files, hps.data)
89
+ eval_loader = DataLoader(eval_dataset, num_workers=0, shuffle=False,
90
+ batch_size=1, pin_memory=True,
91
+ drop_last=False, collate_fn=collate_fn)
92
+ if "use_noise_scaled_mas" in hps.model.keys() and hps.model.use_noise_scaled_mas == True:
93
+ print("Using noise scaled MAS for VITS2")
94
+ use_noise_scaled_mas = True
95
+ mas_noise_scale_initial = 0.01
96
+ noise_scale_delta = 2e-6
97
+ else:
98
+ print("Using normal MAS for VITS1")
99
+ use_noise_scaled_mas = False
100
+ mas_noise_scale_initial = 0.0
101
+ noise_scale_delta = 0.0
102
+ if "use_duration_discriminator" in hps.model.keys() and hps.model.use_duration_discriminator == True:
103
+ print("Using duration discriminator for VITS2")
104
+ use_duration_discriminator = True
105
+ net_dur_disc = DurationDiscriminator(
106
+ hps.model.hidden_channels,
107
+ hps.model.hidden_channels,
108
+ 3,
109
+ 0.1,
110
+ gin_channels=hps.model.gin_channels if hps.data.n_speakers != 0 else 0,
111
+ ).cuda(rank)
112
+ if "use_spk_conditioned_encoder" in hps.model.keys() and hps.model.use_spk_conditioned_encoder == True:
113
+ if hps.data.n_speakers == 0:
114
+ raise ValueError("n_speakers must be > 0 when using spk conditioned encoder to train multi-speaker model")
115
+ use_spk_conditioned_encoder = True
116
+ else:
117
+ print("Using normal encoder for VITS1")
118
+ use_spk_conditioned_encoder = False
119
+
120
+ net_g = SynthesizerTrn(
121
+ len(symbols),
122
+ hps.data.filter_length // 2 + 1,
123
+ hps.train.segment_size // hps.data.hop_length,
124
+ n_speakers=hps.data.n_speakers,
125
+ mas_noise_scale_initial = mas_noise_scale_initial,
126
+ noise_scale_delta = noise_scale_delta,
127
+ **hps.model).cuda(rank)
128
+
129
+ freeze_enc = getattr(hps.model, "freeze_enc", False)
130
+ if freeze_enc:
131
+ print("freeze encoder !!!")
132
+ for param in net_g.enc_p.parameters():
133
+ param.requires_grad = False
134
+
135
+ net_d = MultiPeriodDiscriminator(hps.model.use_spectral_norm).cuda(rank)
136
+ optim_g = torch.optim.AdamW(
137
+ filter(lambda p: p.requires_grad, net_g.parameters()),
138
+ hps.train.learning_rate,
139
+ betas=hps.train.betas,
140
+ eps=hps.train.eps)
141
+ optim_d = torch.optim.AdamW(
142
+ net_d.parameters(),
143
+ hps.train.learning_rate,
144
+ betas=hps.train.betas,
145
+ eps=hps.train.eps)
146
+ if net_dur_disc is not None:
147
+ optim_dur_disc = torch.optim.AdamW(
148
+ net_dur_disc.parameters(),
149
+ hps.train.learning_rate,
150
+ betas=hps.train.betas,
151
+ eps=hps.train.eps)
152
+ else:
153
+ optim_dur_disc = None
154
+ net_g = DDP(net_g, device_ids=[rank], find_unused_parameters=True)
155
+ net_d = DDP(net_d, device_ids=[rank], find_unused_parameters=True)
156
+ if net_dur_disc is not None:
157
+ net_dur_disc = DDP(net_dur_disc, device_ids=[rank], find_unused_parameters=True)
158
+
159
+ pretrain_dir = None
160
+ if pretrain_dir is None:
161
+ try:
162
+ if net_dur_disc is not None:
163
+ _, optim_dur_disc, _, epoch_str = utils.load_checkpoint(utils.latest_checkpoint_path(hps.model_dir, "DUR_*.pth"), net_dur_disc, optim_dur_disc, skip_optimizer=not hps.cont)
164
+ _, optim_g, _, epoch_str = utils.load_checkpoint(utils.latest_checkpoint_path(hps.model_dir, "G_*.pth"), net_g,
165
+ optim_g, skip_optimizer=not hps.cont)
166
+ _, optim_d, _, epoch_str = utils.load_checkpoint(utils.latest_checkpoint_path(hps.model_dir, "D_*.pth"), net_d,
167
+ optim_d, skip_optimizer=not hps.cont)
168
+
169
+ epoch_str = max(epoch_str, 1)
170
+ global_step = (epoch_str - 1) * len(train_loader)
171
+ except Exception as e:
172
+ print(e)
173
+ epoch_str = 1
174
+ global_step = 0
175
+ else:
176
+ _, _, _, epoch_str = utils.load_checkpoint(utils.latest_checkpoint_path(pretrain_dir, "G_*.pth"), net_g,
177
+ optim_g, True)
178
+ _, _, _, epoch_str = utils.load_checkpoint(utils.latest_checkpoint_path(pretrain_dir, "D_*.pth"), net_d,
179
+ optim_d, True)
180
+
181
+
182
+
183
+ scheduler_g = torch.optim.lr_scheduler.ExponentialLR(optim_g, gamma=hps.train.lr_decay, last_epoch=epoch_str - 2)
184
+ scheduler_d = torch.optim.lr_scheduler.ExponentialLR(optim_d, gamma=hps.train.lr_decay, last_epoch=epoch_str - 2)
185
+ if net_dur_disc is not None:
186
+ scheduler_dur_disc = torch.optim.lr_scheduler.ExponentialLR(optim_dur_disc, gamma=hps.train.lr_decay, last_epoch=epoch_str-2)
187
+ else:
188
+ scheduler_dur_disc = None
189
+ scaler = GradScaler(enabled=hps.train.fp16_run)
190
+
191
+ for epoch in range(epoch_str, hps.train.epochs + 1):
192
+ if rank == 0:
193
+ train_and_evaluate(rank, epoch, hps, [net_g, net_d, net_dur_disc], [optim_g, optim_d, optim_dur_disc], [scheduler_g, scheduler_d, scheduler_dur_disc], scaler, [train_loader, eval_loader], logger, [writer, writer_eval])
194
+ else:
195
+ train_and_evaluate(rank, epoch, hps, [net_g, net_d, net_dur_disc], [optim_g, optim_d, optim_dur_disc], [scheduler_g, scheduler_d, scheduler_dur_disc], scaler, [train_loader, None], None, None)
196
+ scheduler_g.step()
197
+ scheduler_d.step()
198
+ if net_dur_disc is not None:
199
+ scheduler_dur_disc.step()
200
+
201
+
202
+ def train_and_evaluate(rank, epoch, hps, nets, optims, schedulers, scaler, loaders, logger, writers):
203
+ net_g, net_d, net_dur_disc = nets
204
+ optim_g, optim_d, optim_dur_disc = optims
205
+ scheduler_g, scheduler_d, scheduler_dur_disc = schedulers
206
+ train_loader, eval_loader = loaders
207
+ if writers is not None:
208
+ writer, writer_eval = writers
209
+
210
+ train_loader.batch_sampler.set_epoch(epoch)
211
+ global global_step
212
+
213
+ net_g.train()
214
+ net_d.train()
215
+ if net_dur_disc is not None:
216
+ net_dur_disc.train()
217
+ for batch_idx, (x, x_lengths, spec, spec_lengths, y, y_lengths, speakers, tone, language, bert) in tqdm(enumerate(train_loader)):
218
+ if net_g.module.use_noise_scaled_mas:
219
+ current_mas_noise_scale = net_g.module.mas_noise_scale_initial - net_g.module.noise_scale_delta * global_step
220
+ net_g.module.current_mas_noise_scale = max(current_mas_noise_scale, 0.0)
221
+ x, x_lengths = x.cuda(rank, non_blocking=True), x_lengths.cuda(rank, non_blocking=True)
222
+ spec, spec_lengths = spec.cuda(rank, non_blocking=True), spec_lengths.cuda(rank, non_blocking=True)
223
+ y, y_lengths = y.cuda(rank, non_blocking=True), y_lengths.cuda(rank, non_blocking=True)
224
+ speakers = speakers.cuda(rank, non_blocking=True)
225
+ tone = tone.cuda(rank, non_blocking=True)
226
+ language = language.cuda(rank, non_blocking=True)
227
+ bert = bert.cuda(rank, non_blocking=True)
228
+
229
+ with autocast(enabled=hps.train.fp16_run):
230
+ y_hat, l_length, attn, ids_slice, x_mask, z_mask, \
231
+ (z, z_p, m_p, logs_p, m_q, logs_q), (hidden_x, logw, logw_) = net_g(x, x_lengths, spec, spec_lengths, speakers, tone, language, bert)
232
+ mel = spec_to_mel_torch(
233
+ spec,
234
+ hps.data.filter_length,
235
+ hps.data.n_mel_channels,
236
+ hps.data.sampling_rate,
237
+ hps.data.mel_fmin,
238
+ hps.data.mel_fmax)
239
+ y_mel = commons.slice_segments(mel, ids_slice, hps.train.segment_size // hps.data.hop_length)
240
+ y_hat_mel = mel_spectrogram_torch(
241
+ y_hat.squeeze(1),
242
+ hps.data.filter_length,
243
+ hps.data.n_mel_channels,
244
+ hps.data.sampling_rate,
245
+ hps.data.hop_length,
246
+ hps.data.win_length,
247
+ hps.data.mel_fmin,
248
+ hps.data.mel_fmax
249
+ )
250
+
251
+ y = commons.slice_segments(y, ids_slice * hps.data.hop_length, hps.train.segment_size) # slice
252
+
253
+ # Discriminator
254
+ y_d_hat_r, y_d_hat_g, _, _ = net_d(y, y_hat.detach())
255
+ with autocast(enabled=False):
256
+ loss_disc, losses_disc_r, losses_disc_g = discriminator_loss(y_d_hat_r, y_d_hat_g)
257
+ loss_disc_all = loss_disc
258
+ if net_dur_disc is not None:
259
+ y_dur_hat_r, y_dur_hat_g = net_dur_disc(hidden_x.detach(), x_mask.detach(), logw.detach(), logw_.detach())
260
+ with autocast(enabled=False):
261
+ # TODO: I think need to mean using the mask, but for now, just mean all
262
+ loss_dur_disc, losses_dur_disc_r, losses_dur_disc_g = discriminator_loss(y_dur_hat_r, y_dur_hat_g)
263
+ loss_dur_disc_all = loss_dur_disc
264
+ optim_dur_disc.zero_grad()
265
+ scaler.scale(loss_dur_disc_all).backward()
266
+ scaler.unscale_(optim_dur_disc)
267
+ grad_norm_dur_disc = commons.clip_grad_value_(net_dur_disc.parameters(), None)
268
+ scaler.step(optim_dur_disc)
269
+
270
+ optim_d.zero_grad()
271
+ scaler.scale(loss_disc_all).backward()
272
+ scaler.unscale_(optim_d)
273
+ grad_norm_d = commons.clip_grad_value_(net_d.parameters(), None)
274
+ scaler.step(optim_d)
275
+
276
+ with autocast(enabled=hps.train.fp16_run):
277
+ # Generator
278
+ y_d_hat_r, y_d_hat_g, fmap_r, fmap_g = net_d(y, y_hat)
279
+ if net_dur_disc is not None:
280
+ y_dur_hat_r, y_dur_hat_g = net_dur_disc(hidden_x, x_mask, logw, logw_)
281
+ with autocast(enabled=False):
282
+ loss_dur = torch.sum(l_length.float())
283
+ loss_mel = F.l1_loss(y_mel, y_hat_mel) * hps.train.c_mel
284
+ loss_kl = kl_loss(z_p, logs_q, m_p, logs_p, z_mask) * hps.train.c_kl
285
+
286
+ loss_fm = feature_loss(fmap_r, fmap_g)
287
+ loss_gen, losses_gen = generator_loss(y_d_hat_g)
288
+ loss_gen_all = loss_gen + loss_fm + loss_mel + loss_dur + loss_kl
289
+ if net_dur_disc is not None:
290
+ loss_dur_gen, losses_dur_gen = generator_loss(y_dur_hat_g)
291
+ loss_gen_all += loss_dur_gen
292
+ optim_g.zero_grad()
293
+ scaler.scale(loss_gen_all).backward()
294
+ scaler.unscale_(optim_g)
295
+ grad_norm_g = commons.clip_grad_value_(net_g.parameters(), None)
296
+ scaler.step(optim_g)
297
+ scaler.update()
298
+
299
+ if rank == 0:
300
+ if global_step % hps.train.log_interval == 0:
301
+ lr = optim_g.param_groups[0]['lr']
302
+ losses = [loss_disc, loss_gen, loss_fm, loss_mel, loss_dur, loss_kl]
303
+ logger.info('Train Epoch: {} [{:.0f}%]'.format(
304
+ epoch,
305
+ 100. * batch_idx / len(train_loader)))
306
+ logger.info([x.item() for x in losses] + [global_step, lr])
307
+
308
+ scalar_dict = {"loss/g/total": loss_gen_all, "loss/d/total": loss_disc_all, "learning_rate": lr,
309
+ "grad_norm_d": grad_norm_d, "grad_norm_g": grad_norm_g}
310
+ scalar_dict.update(
311
+ {"loss/g/fm": loss_fm, "loss/g/mel": loss_mel, "loss/g/dur": loss_dur, "loss/g/kl": loss_kl})
312
+ scalar_dict.update({"loss/g/{}".format(i): v for i, v in enumerate(losses_gen)})
313
+ scalar_dict.update({"loss/d_r/{}".format(i): v for i, v in enumerate(losses_disc_r)})
314
+ scalar_dict.update({"loss/d_g/{}".format(i): v for i, v in enumerate(losses_disc_g)})
315
+
316
+ image_dict = {
317
+ "slice/mel_org": utils.plot_spectrogram_to_numpy(y_mel[0].data.cpu().numpy()),
318
+ "slice/mel_gen": utils.plot_spectrogram_to_numpy(y_hat_mel[0].data.cpu().numpy()),
319
+ "all/mel": utils.plot_spectrogram_to_numpy(mel[0].data.cpu().numpy()),
320
+ "all/attn": utils.plot_alignment_to_numpy(attn[0, 0].data.cpu().numpy())
321
+ }
322
+ utils.summarize(
323
+ writer=writer,
324
+ global_step=global_step,
325
+ images=image_dict,
326
+ scalars=scalar_dict)
327
+
328
+ if global_step % hps.train.eval_interval == 0:
329
+ evaluate(hps, net_g, eval_loader, writer_eval)
330
+ utils.save_checkpoint(net_g, optim_g, hps.train.learning_rate, epoch,
331
+ os.path.join(hps.model_dir, "G_{}.pth".format(global_step)))
332
+ utils.save_checkpoint(net_d, optim_d, hps.train.learning_rate, epoch,
333
+ os.path.join(hps.model_dir, "D_{}.pth".format(global_step)))
334
+ if net_dur_disc is not None:
335
+ utils.save_checkpoint(net_dur_disc, optim_dur_disc, hps.train.learning_rate, epoch, os.path.join(hps.model_dir, "DUR_{}.pth".format(global_step)))
336
+ keep_ckpts = getattr(hps.train, 'keep_ckpts', 5)
337
+ if keep_ckpts > 0:
338
+ utils.clean_checkpoints(path_to_models=hps.model_dir, n_ckpts_to_keep=keep_ckpts, sort_by_time=True)
339
+
340
+
341
+ global_step += 1
342
+
343
+ if rank == 0:
344
+ logger.info('====> Epoch: {}'.format(epoch))
345
+
346
+
347
+
348
+ def evaluate(hps, generator, eval_loader, writer_eval):
349
+ generator.eval()
350
+ image_dict = {}
351
+ audio_dict = {}
352
+ print("Evaluating ...")
353
+ with torch.no_grad():
354
+ for batch_idx, (x, x_lengths, spec, spec_lengths, y, y_lengths, speakers, tone, language, bert) in enumerate(eval_loader):
355
+ x, x_lengths = x.cuda(), x_lengths.cuda()
356
+ spec, spec_lengths = spec.cuda(), spec_lengths.cuda()
357
+ y, y_lengths = y.cuda(), y_lengths.cuda()
358
+ speakers = speakers.cuda()
359
+ bert = bert.cuda()
360
+ tone = tone.cuda()
361
+ language = language.cuda()
362
+ for use_sdp in [True, False]:
363
+ y_hat, attn, mask, *_ = generator.module.infer(x, x_lengths, speakers, tone, language, bert, y=spec, max_len=1000, sdp_ratio=0.0 if not use_sdp else 1.0)
364
+ y_hat_lengths = mask.sum([1, 2]).long() * hps.data.hop_length
365
+
366
+ mel = spec_to_mel_torch(
367
+ spec,
368
+ hps.data.filter_length,
369
+ hps.data.n_mel_channels,
370
+ hps.data.sampling_rate,
371
+ hps.data.mel_fmin,
372
+ hps.data.mel_fmax)
373
+ y_hat_mel = mel_spectrogram_torch(
374
+ y_hat.squeeze(1).float(),
375
+ hps.data.filter_length,
376
+ hps.data.n_mel_channels,
377
+ hps.data.sampling_rate,
378
+ hps.data.hop_length,
379
+ hps.data.win_length,
380
+ hps.data.mel_fmin,
381
+ hps.data.mel_fmax
382
+ )
383
+ image_dict.update({
384
+ f"gen/mel_{batch_idx}": utils.plot_spectrogram_to_numpy(y_hat_mel[0].cpu().numpy())
385
+ })
386
+ audio_dict.update({
387
+ f"gen/audio_{batch_idx}_{use_sdp}": y_hat[0, :, :y_hat_lengths[0]]
388
+ })
389
+ image_dict.update({f"gt/mel_{batch_idx}": utils.plot_spectrogram_to_numpy(mel[0].cpu().numpy())})
390
+ audio_dict.update({f"gt/audio_{batch_idx}": y[0, :, :y_lengths[0]]})
391
+
392
+ utils.summarize(
393
+ writer=writer_eval,
394
+ global_step=global_step,
395
+ images=image_dict,
396
+ audios=audio_dict,
397
+ audio_sampling_rate=hps.data.sampling_rate
398
+ )
399
+ generator.train()
400
+
401
+ if __name__ == "__main__":
402
+ main()
transforms.py ADDED
@@ -0,0 +1,193 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch
2
+ from torch.nn import functional as F
3
+
4
+ import numpy as np
5
+
6
+
7
+ DEFAULT_MIN_BIN_WIDTH = 1e-3
8
+ DEFAULT_MIN_BIN_HEIGHT = 1e-3
9
+ DEFAULT_MIN_DERIVATIVE = 1e-3
10
+
11
+
12
+ def piecewise_rational_quadratic_transform(inputs,
13
+ unnormalized_widths,
14
+ unnormalized_heights,
15
+ unnormalized_derivatives,
16
+ inverse=False,
17
+ tails=None,
18
+ tail_bound=1.,
19
+ min_bin_width=DEFAULT_MIN_BIN_WIDTH,
20
+ min_bin_height=DEFAULT_MIN_BIN_HEIGHT,
21
+ min_derivative=DEFAULT_MIN_DERIVATIVE):
22
+
23
+ if tails is None:
24
+ spline_fn = rational_quadratic_spline
25
+ spline_kwargs = {}
26
+ else:
27
+ spline_fn = unconstrained_rational_quadratic_spline
28
+ spline_kwargs = {
29
+ 'tails': tails,
30
+ 'tail_bound': tail_bound
31
+ }
32
+
33
+ outputs, logabsdet = spline_fn(
34
+ inputs=inputs,
35
+ unnormalized_widths=unnormalized_widths,
36
+ unnormalized_heights=unnormalized_heights,
37
+ unnormalized_derivatives=unnormalized_derivatives,
38
+ inverse=inverse,
39
+ min_bin_width=min_bin_width,
40
+ min_bin_height=min_bin_height,
41
+ min_derivative=min_derivative,
42
+ **spline_kwargs
43
+ )
44
+ return outputs, logabsdet
45
+
46
+
47
+ def searchsorted(bin_locations, inputs, eps=1e-6):
48
+ bin_locations[..., -1] += eps
49
+ return torch.sum(
50
+ inputs[..., None] >= bin_locations,
51
+ dim=-1
52
+ ) - 1
53
+
54
+
55
+ def unconstrained_rational_quadratic_spline(inputs,
56
+ unnormalized_widths,
57
+ unnormalized_heights,
58
+ unnormalized_derivatives,
59
+ inverse=False,
60
+ tails='linear',
61
+ tail_bound=1.,
62
+ min_bin_width=DEFAULT_MIN_BIN_WIDTH,
63
+ min_bin_height=DEFAULT_MIN_BIN_HEIGHT,
64
+ min_derivative=DEFAULT_MIN_DERIVATIVE):
65
+ inside_interval_mask = (inputs >= -tail_bound) & (inputs <= tail_bound)
66
+ outside_interval_mask = ~inside_interval_mask
67
+
68
+ outputs = torch.zeros_like(inputs)
69
+ logabsdet = torch.zeros_like(inputs)
70
+
71
+ if tails == 'linear':
72
+ unnormalized_derivatives = F.pad(unnormalized_derivatives, pad=(1, 1))
73
+ constant = np.log(np.exp(1 - min_derivative) - 1)
74
+ unnormalized_derivatives[..., 0] = constant
75
+ unnormalized_derivatives[..., -1] = constant
76
+
77
+ outputs[outside_interval_mask] = inputs[outside_interval_mask]
78
+ logabsdet[outside_interval_mask] = 0
79
+ else:
80
+ raise RuntimeError('{} tails are not implemented.'.format(tails))
81
+
82
+ outputs[inside_interval_mask], logabsdet[inside_interval_mask] = rational_quadratic_spline(
83
+ inputs=inputs[inside_interval_mask],
84
+ unnormalized_widths=unnormalized_widths[inside_interval_mask, :],
85
+ unnormalized_heights=unnormalized_heights[inside_interval_mask, :],
86
+ unnormalized_derivatives=unnormalized_derivatives[inside_interval_mask, :],
87
+ inverse=inverse,
88
+ left=-tail_bound, right=tail_bound, bottom=-tail_bound, top=tail_bound,
89
+ min_bin_width=min_bin_width,
90
+ min_bin_height=min_bin_height,
91
+ min_derivative=min_derivative
92
+ )
93
+
94
+ return outputs, logabsdet
95
+
96
+ def rational_quadratic_spline(inputs,
97
+ unnormalized_widths,
98
+ unnormalized_heights,
99
+ unnormalized_derivatives,
100
+ inverse=False,
101
+ left=0., right=1., bottom=0., top=1.,
102
+ min_bin_width=DEFAULT_MIN_BIN_WIDTH,
103
+ min_bin_height=DEFAULT_MIN_BIN_HEIGHT,
104
+ min_derivative=DEFAULT_MIN_DERIVATIVE):
105
+ if torch.min(inputs) < left or torch.max(inputs) > right:
106
+ raise ValueError('Input to a transform is not within its domain')
107
+
108
+ num_bins = unnormalized_widths.shape[-1]
109
+
110
+ if min_bin_width * num_bins > 1.0:
111
+ raise ValueError('Minimal bin width too large for the number of bins')
112
+ if min_bin_height * num_bins > 1.0:
113
+ raise ValueError('Minimal bin height too large for the number of bins')
114
+
115
+ widths = F.softmax(unnormalized_widths, dim=-1)
116
+ widths = min_bin_width + (1 - min_bin_width * num_bins) * widths
117
+ cumwidths = torch.cumsum(widths, dim=-1)
118
+ cumwidths = F.pad(cumwidths, pad=(1, 0), mode='constant', value=0.0)
119
+ cumwidths = (right - left) * cumwidths + left
120
+ cumwidths[..., 0] = left
121
+ cumwidths[..., -1] = right
122
+ widths = cumwidths[..., 1:] - cumwidths[..., :-1]
123
+
124
+ derivatives = min_derivative + F.softplus(unnormalized_derivatives)
125
+
126
+ heights = F.softmax(unnormalized_heights, dim=-1)
127
+ heights = min_bin_height + (1 - min_bin_height * num_bins) * heights
128
+ cumheights = torch.cumsum(heights, dim=-1)
129
+ cumheights = F.pad(cumheights, pad=(1, 0), mode='constant', value=0.0)
130
+ cumheights = (top - bottom) * cumheights + bottom
131
+ cumheights[..., 0] = bottom
132
+ cumheights[..., -1] = top
133
+ heights = cumheights[..., 1:] - cumheights[..., :-1]
134
+
135
+ if inverse:
136
+ bin_idx = searchsorted(cumheights, inputs)[..., None]
137
+ else:
138
+ bin_idx = searchsorted(cumwidths, inputs)[..., None]
139
+
140
+ input_cumwidths = cumwidths.gather(-1, bin_idx)[..., 0]
141
+ input_bin_widths = widths.gather(-1, bin_idx)[..., 0]
142
+
143
+ input_cumheights = cumheights.gather(-1, bin_idx)[..., 0]
144
+ delta = heights / widths
145
+ input_delta = delta.gather(-1, bin_idx)[..., 0]
146
+
147
+ input_derivatives = derivatives.gather(-1, bin_idx)[..., 0]
148
+ input_derivatives_plus_one = derivatives[..., 1:].gather(-1, bin_idx)[..., 0]
149
+
150
+ input_heights = heights.gather(-1, bin_idx)[..., 0]
151
+
152
+ if inverse:
153
+ a = (((inputs - input_cumheights) * (input_derivatives
154
+ + input_derivatives_plus_one
155
+ - 2 * input_delta)
156
+ + input_heights * (input_delta - input_derivatives)))
157
+ b = (input_heights * input_derivatives
158
+ - (inputs - input_cumheights) * (input_derivatives
159
+ + input_derivatives_plus_one
160
+ - 2 * input_delta))
161
+ c = - input_delta * (inputs - input_cumheights)
162
+
163
+ discriminant = b.pow(2) - 4 * a * c
164
+ assert (discriminant >= 0).all()
165
+
166
+ root = (2 * c) / (-b - torch.sqrt(discriminant))
167
+ outputs = root * input_bin_widths + input_cumwidths
168
+
169
+ theta_one_minus_theta = root * (1 - root)
170
+ denominator = input_delta + ((input_derivatives + input_derivatives_plus_one - 2 * input_delta)
171
+ * theta_one_minus_theta)
172
+ derivative_numerator = input_delta.pow(2) * (input_derivatives_plus_one * root.pow(2)
173
+ + 2 * input_delta * theta_one_minus_theta
174
+ + input_derivatives * (1 - root).pow(2))
175
+ logabsdet = torch.log(derivative_numerator) - 2 * torch.log(denominator)
176
+
177
+ return outputs, -logabsdet
178
+ else:
179
+ theta = (inputs - input_cumwidths) / input_bin_widths
180
+ theta_one_minus_theta = theta * (1 - theta)
181
+
182
+ numerator = input_heights * (input_delta * theta.pow(2)
183
+ + input_derivatives * theta_one_minus_theta)
184
+ denominator = input_delta + ((input_derivatives + input_derivatives_plus_one - 2 * input_delta)
185
+ * theta_one_minus_theta)
186
+ outputs = input_cumheights + numerator / denominator
187
+
188
+ derivative_numerator = input_delta.pow(2) * (input_derivatives_plus_one * theta.pow(2)
189
+ + 2 * input_delta * theta_one_minus_theta
190
+ + input_derivatives * (1 - theta).pow(2))
191
+ logabsdet = torch.log(derivative_numerator) - 2 * torch.log(denominator)
192
+
193
+ return outputs, logabsdet
utils.py ADDED
@@ -0,0 +1,293 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ import glob
3
+ import sys
4
+ import argparse
5
+ import logging
6
+ import json
7
+ import subprocess
8
+ import numpy as np
9
+ from scipy.io.wavfile import read
10
+ import torch
11
+
12
+ MATPLOTLIB_FLAG = False
13
+
14
+ logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
15
+ logger = logging
16
+
17
+
18
+ def load_checkpoint(checkpoint_path, model, optimizer=None, skip_optimizer=False):
19
+ assert os.path.isfile(checkpoint_path)
20
+ checkpoint_dict = torch.load(checkpoint_path, map_location='cpu')
21
+ iteration = checkpoint_dict['iteration']
22
+ learning_rate = checkpoint_dict['learning_rate']
23
+ if optimizer is not None and not skip_optimizer and checkpoint_dict['optimizer'] is not None:
24
+ optimizer.load_state_dict(checkpoint_dict['optimizer'])
25
+ elif optimizer is None and not skip_optimizer:
26
+ #else: #Disable this line if Infer ,and enable the line upper
27
+ new_opt_dict = optimizer.state_dict()
28
+ new_opt_dict_params = new_opt_dict['param_groups'][0]['params']
29
+ new_opt_dict['param_groups'] = checkpoint_dict['optimizer']['param_groups']
30
+ new_opt_dict['param_groups'][0]['params'] = new_opt_dict_params
31
+ optimizer.load_state_dict(new_opt_dict)
32
+ saved_state_dict = checkpoint_dict['model']
33
+ if hasattr(model, 'module'):
34
+ state_dict = model.module.state_dict()
35
+ else:
36
+ state_dict = model.state_dict()
37
+ new_state_dict = {}
38
+ for k, v in state_dict.items():
39
+ try:
40
+ #assert "emb_g" not in k
41
+ # print("load", k)
42
+ new_state_dict[k] = saved_state_dict[k]
43
+ assert saved_state_dict[k].shape == v.shape, (saved_state_dict[k].shape, v.shape)
44
+ except:
45
+ print("error, %s is not in the checkpoint" % k)
46
+ new_state_dict[k] = v
47
+ if hasattr(model, 'module'):
48
+ model.module.load_state_dict(new_state_dict, strict=False)
49
+ else:
50
+ model.load_state_dict(new_state_dict, strict=False)
51
+ print("load ")
52
+ logger.info("Loaded checkpoint '{}' (iteration {})".format(
53
+ checkpoint_path, iteration))
54
+ return model, optimizer, learning_rate, iteration
55
+
56
+
57
+ def save_checkpoint(model, optimizer, learning_rate, iteration, checkpoint_path):
58
+ logger.info("Saving model and optimizer state at iteration {} to {}".format(
59
+ iteration, checkpoint_path))
60
+ if hasattr(model, 'module'):
61
+ state_dict = model.module.state_dict()
62
+ else:
63
+ state_dict = model.state_dict()
64
+ torch.save({'model': state_dict,
65
+ 'iteration': iteration,
66
+ 'optimizer': optimizer.state_dict(),
67
+ 'learning_rate': learning_rate}, checkpoint_path)
68
+
69
+
70
+ def summarize(writer, global_step, scalars={}, histograms={}, images={}, audios={}, audio_sampling_rate=22050):
71
+ for k, v in scalars.items():
72
+ writer.add_scalar(k, v, global_step)
73
+ for k, v in histograms.items():
74
+ writer.add_histogram(k, v, global_step)
75
+ for k, v in images.items():
76
+ writer.add_image(k, v, global_step, dataformats='HWC')
77
+ for k, v in audios.items():
78
+ writer.add_audio(k, v, global_step, audio_sampling_rate)
79
+
80
+
81
+ def latest_checkpoint_path(dir_path, regex="G_*.pth"):
82
+ f_list = glob.glob(os.path.join(dir_path, regex))
83
+ f_list.sort(key=lambda f: int("".join(filter(str.isdigit, f))))
84
+ x = f_list[-1]
85
+ print(x)
86
+ return x
87
+
88
+
89
+ def plot_spectrogram_to_numpy(spectrogram):
90
+ global MATPLOTLIB_FLAG
91
+ if not MATPLOTLIB_FLAG:
92
+ import matplotlib
93
+ matplotlib.use("Agg")
94
+ MATPLOTLIB_FLAG = True
95
+ mpl_logger = logging.getLogger('matplotlib')
96
+ mpl_logger.setLevel(logging.WARNING)
97
+ import matplotlib.pylab as plt
98
+ import numpy as np
99
+
100
+ fig, ax = plt.subplots(figsize=(10, 2))
101
+ im = ax.imshow(spectrogram, aspect="auto", origin="lower",
102
+ interpolation='none')
103
+ plt.colorbar(im, ax=ax)
104
+ plt.xlabel("Frames")
105
+ plt.ylabel("Channels")
106
+ plt.tight_layout()
107
+
108
+ fig.canvas.draw()
109
+ data = np.fromstring(fig.canvas.tostring_rgb(), dtype=np.uint8, sep='')
110
+ data = data.reshape(fig.canvas.get_width_height()[::-1] + (3,))
111
+ plt.close()
112
+ return data
113
+
114
+
115
+ def plot_alignment_to_numpy(alignment, info=None):
116
+ global MATPLOTLIB_FLAG
117
+ if not MATPLOTLIB_FLAG:
118
+ import matplotlib
119
+ matplotlib.use("Agg")
120
+ MATPLOTLIB_FLAG = True
121
+ mpl_logger = logging.getLogger('matplotlib')
122
+ mpl_logger.setLevel(logging.WARNING)
123
+ import matplotlib.pylab as plt
124
+ import numpy as np
125
+
126
+ fig, ax = plt.subplots(figsize=(6, 4))
127
+ im = ax.imshow(alignment.transpose(), aspect='auto', origin='lower',
128
+ interpolation='none')
129
+ fig.colorbar(im, ax=ax)
130
+ xlabel = 'Decoder timestep'
131
+ if info is not None:
132
+ xlabel += '\n\n' + info
133
+ plt.xlabel(xlabel)
134
+ plt.ylabel('Encoder timestep')
135
+ plt.tight_layout()
136
+
137
+ fig.canvas.draw()
138
+ data = np.fromstring(fig.canvas.tostring_rgb(), dtype=np.uint8, sep='')
139
+ data = data.reshape(fig.canvas.get_width_height()[::-1] + (3,))
140
+ plt.close()
141
+ return data
142
+
143
+
144
+ def load_wav_to_torch(full_path):
145
+ sampling_rate, data = read(full_path)
146
+ return torch.FloatTensor(data.astype(np.float32)), sampling_rate
147
+
148
+
149
+ def load_filepaths_and_text(filename, split="|"):
150
+ with open(filename, encoding='utf-8') as f:
151
+ filepaths_and_text = [line.strip().split(split) for line in f]
152
+ return filepaths_and_text
153
+
154
+
155
+ def get_hparams(init=True):
156
+ parser = argparse.ArgumentParser()
157
+ parser.add_argument('-c', '--config', type=str, default="./configs/base.json",
158
+ help='JSON file for configuration')
159
+ parser.add_argument('-m', '--model', type=str, default="./OUTPUT_MODEL",
160
+ help='Model name')
161
+ parser.add_argument('--cont', dest='cont', action="store_true", default=False, help="whether to continue training on the latest checkpoint")
162
+
163
+ args = parser.parse_args()
164
+ model_dir = os.path.join("./logs", args.model)
165
+
166
+ if not os.path.exists(model_dir):
167
+ os.makedirs(model_dir)
168
+
169
+ config_path = args.config
170
+ config_save_path = os.path.join(model_dir, "config.json")
171
+ if init:
172
+ with open(config_path, "r") as f:
173
+ data = f.read()
174
+ with open(config_save_path, "w") as f:
175
+ f.write(data)
176
+ else:
177
+ with open(config_save_path, "r") as f:
178
+ data = f.read()
179
+ config = json.loads(data)
180
+
181
+ hparams = HParams(**config)
182
+ hparams.model_dir = model_dir
183
+ hparams.cont = args.cont
184
+ return hparams
185
+
186
+
187
+ def clean_checkpoints(path_to_models='logs/44k/', n_ckpts_to_keep=2, sort_by_time=True):
188
+ """Freeing up space by deleting saved ckpts
189
+
190
+ Arguments:
191
+ path_to_models -- Path to the model directory
192
+ n_ckpts_to_keep -- Number of ckpts to keep, excluding G_0.pth and D_0.pth
193
+ sort_by_time -- True -> chronologically delete ckpts
194
+ False -> lexicographically delete ckpts
195
+ """
196
+ import re
197
+ ckpts_files = [f for f in os.listdir(path_to_models) if os.path.isfile(os.path.join(path_to_models, f))]
198
+ name_key = (lambda _f: int(re.compile('._(\d+)\.pth').match(_f).group(1)))
199
+ time_key = (lambda _f: os.path.getmtime(os.path.join(path_to_models, _f)))
200
+ sort_key = time_key if sort_by_time else name_key
201
+ x_sorted = lambda _x: sorted([f for f in ckpts_files if f.startswith(_x) and not f.endswith('_0.pth')],
202
+ key=sort_key)
203
+ to_del = [os.path.join(path_to_models, fn) for fn in
204
+ (x_sorted('G')[:-n_ckpts_to_keep] + x_sorted('D')[:-n_ckpts_to_keep])]
205
+ del_info = lambda fn: logger.info(f".. Free up space by deleting ckpt {fn}")
206
+ del_routine = lambda x: [os.remove(x), del_info(x)]
207
+ rs = [del_routine(fn) for fn in to_del]
208
+
209
+ def get_hparams_from_dir(model_dir):
210
+ config_save_path = os.path.join(model_dir, "config.json")
211
+ with open(config_save_path, "r") as f:
212
+ data = f.read()
213
+ config = json.loads(data)
214
+
215
+ hparams = HParams(**config)
216
+ hparams.model_dir = model_dir
217
+ return hparams
218
+
219
+
220
+ def get_hparams_from_file(config_path):
221
+ with open(config_path, "r") as f:
222
+ data = f.read()
223
+ config = json.loads(data)
224
+
225
+ hparams = HParams(**config)
226
+ return hparams
227
+
228
+
229
+ def check_git_hash(model_dir):
230
+ source_dir = os.path.dirname(os.path.realpath(__file__))
231
+ if not os.path.exists(os.path.join(source_dir, ".git")):
232
+ logger.warn("{} is not a git repository, therefore hash value comparison will be ignored.".format(
233
+ source_dir
234
+ ))
235
+ return
236
+
237
+ cur_hash = subprocess.getoutput("git rev-parse HEAD")
238
+
239
+ path = os.path.join(model_dir, "githash")
240
+ if os.path.exists(path):
241
+ saved_hash = open(path).read()
242
+ if saved_hash != cur_hash:
243
+ logger.warn("git hash values are different. {}(saved) != {}(current)".format(
244
+ saved_hash[:8], cur_hash[:8]))
245
+ else:
246
+ open(path, "w").write(cur_hash)
247
+
248
+
249
+ def get_logger(model_dir, filename="train.log"):
250
+ global logger
251
+ logger = logging.getLogger(os.path.basename(model_dir))
252
+ logger.setLevel(logging.DEBUG)
253
+
254
+ formatter = logging.Formatter("%(asctime)s\t%(name)s\t%(levelname)s\t%(message)s")
255
+ if not os.path.exists(model_dir):
256
+ os.makedirs(model_dir)
257
+ h = logging.FileHandler(os.path.join(model_dir, filename))
258
+ h.setLevel(logging.DEBUG)
259
+ h.setFormatter(formatter)
260
+ logger.addHandler(h)
261
+ return logger
262
+
263
+
264
+ class HParams():
265
+ def __init__(self, **kwargs):
266
+ for k, v in kwargs.items():
267
+ if type(v) == dict:
268
+ v = HParams(**v)
269
+ self[k] = v
270
+
271
+ def keys(self):
272
+ return self.__dict__.keys()
273
+
274
+ def items(self):
275
+ return self.__dict__.items()
276
+
277
+ def values(self):
278
+ return self.__dict__.values()
279
+
280
+ def __len__(self):
281
+ return len(self.__dict__)
282
+
283
+ def __getitem__(self, key):
284
+ return getattr(self, key)
285
+
286
+ def __setitem__(self, key, value):
287
+ return setattr(self, key, value)
288
+
289
+ def __contains__(self, key):
290
+ return key in self.__dict__
291
+
292
+ def __repr__(self):
293
+ return self.__dict__.__repr__()