Hugging Face's logo

tuandaodev
/
xtts-vi-infore1-25hours-5epochs

TensorBoard
Model card Files Metrics Community
xtts-vi-infore1-25hours-5epochs / trainer_0_log.txt
tuandaodev's picture
tuandaodev
Upload folder using huggingface_hub
fb69023 verified
raw
history blame
141 kB
> Training Environment:
| > Backend: Torch
| > Mixed precision: False
| > Precision: float32
| > Current device: 0
| > Num. of GPUs: 1
| > Num. of CPUs: 4
| > Num. of Torch Threads: 1
| > Torch seed: 54321
| > Torch CUDNN: True
| > Torch CUDNN deterministic: False
| > Torch CUDNN benchmark: False
| > Torch TF32 MatMul: False
> Start Tensorboard: tensorboard --logdir=/kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b
> Model has 522165080 parameters
 > EPOCH: 0/5
--> /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b
 > TRAINING (2024-10-19 23:36:51) 
 --> TIME: 2024-10-19 23:36:56 -- STEP: 0/3361 -- GLOBAL_STEP: 0
| > loss_text_ce: 0.2276160567998886 (0.2276160567998886)
| > loss_mel_ce: 5.073640823364258 (5.073640823364258)
| > loss: 1.3253141641616821 (1.3253141641616821)
| > current_lr: 5e-06
| > step_time: 0.9855 (0.9855413436889648)
| > loader_time: 3.9741 (3.974128007888794)
 --> TIME: 2024-10-19 23:37:28 -- STEP: 50/3361 -- GLOBAL_STEP: 50
| > loss_text_ce: 0.1961478441953659 (0.21089269667863847)
| > loss_mel_ce: 4.774014472961426 (5.021885404586793)
| > loss: 1.2425405979156494 (1.3081945204734806)
| > current_lr: 5e-06
| > step_time: 0.4324 (0.3791870260238647)
| > loader_time: 0.0095 (0.014342236518859863)
 --> TIME: 2024-10-19 23:38:00 -- STEP: 100/3361 -- GLOBAL_STEP: 100
| > loss_text_ce: 0.19649465382099152 (0.20336569979786873)
| > loss_mel_ce: 4.587543487548828 (4.869443783760071)
| > loss: 1.1960095167160034 (1.2682023656368255)
| > current_lr: 5e-06
| > step_time: 0.2369 (0.37330648183822635)
| > loader_time: 0.0147 (0.013298330307006835)
 --> TIME: 2024-10-19 23:38:32 -- STEP: 150/3361 -- GLOBAL_STEP: 150
| > loss_text_ce: 0.17126446962356567 (0.195263701180617)
| > loss_mel_ce: 4.586864471435547 (4.784979950586955)
| > loss: 1.1895322799682617 (1.2450609103838604)
| > current_lr: 5e-06
| > step_time: 0.3026 (0.37735692818959554)
| > loader_time: 0.0143 (0.012361326217651366)
 --> TIME: 2024-10-19 23:39:05 -- STEP: 200/3361 -- GLOBAL_STEP: 200
| > loss_text_ce: 0.1639341115951538 (0.18802625849843024)
| > loss_mel_ce: 4.630383491516113 (4.733785247802734)
| > loss: 1.1985794305801392 (1.2304528760910034)
| > current_lr: 5e-06
| > step_time: 0.2442 (0.37673797488212585)
| > loader_time: 0.0183 (0.011802190542221067)
 --> TIME: 2024-10-19 23:39:36 -- STEP: 250/3361 -- GLOBAL_STEP: 250
| > loss_text_ce: 0.15474987030029297 (0.18128131020069121)
| > loss_mel_ce: 4.447690963745117 (4.686841909408569)
| > loss: 1.1506102085113525 (1.2170308060646058)
| > current_lr: 5e-06
| > step_time: 0.3412 (0.37635364627838136)
| > loader_time: 0.0101 (0.011550966262817382)
 --> TIME: 2024-10-19 23:40:08 -- STEP: 300/3361 -- GLOBAL_STEP: 300
| > loss_text_ce: 0.1450122743844986 (0.17507749925057092)
| > loss_mel_ce: 4.760715007781982 (4.6537174924214675)
| > loss: 1.2264318466186523 (1.2071987493832905)
| > current_lr: 5e-06
| > step_time: 0.2737 (0.3747914385795592)
| > loader_time: 0.0089 (0.011329036553700772)
 --> TIME: 2024-10-19 23:40:39 -- STEP: 350/3361 -- GLOBAL_STEP: 350
| > loss_text_ce: 0.13236135244369507 (0.1689298430936677)
| > loss_mel_ce: 4.3863911628723145 (4.623111551829745)
| > loss: 1.1296881437301636 (1.1980103506360738)
| > current_lr: 5e-06
| > step_time: 0.4296 (0.3739049707140241)
| > loader_time: 0.0098 (0.01116123471941268)
 --> TIME: 2024-10-19 23:41:11 -- STEP: 400/3361 -- GLOBAL_STEP: 400
| > loss_text_ce: 0.11837068200111389 (0.1625033115968108)
| > loss_mel_ce: 4.637821197509766 (4.5994156956672665)
| > loss: 1.189047932624817 (1.1904797530174254)
| > current_lr: 5e-06
| > step_time: 0.2738 (0.37281567573547364)
| > loader_time: 0.0105 (0.010975514650344858)
 --> TIME: 2024-10-19 23:41:42 -- STEP: 450/3361 -- GLOBAL_STEP: 450
| > loss_text_ce: 0.09800536185503006 (0.1564699565867583)
| > loss_mel_ce: 4.699829578399658 (4.576134746339584)
| > loss: 1.1994587182998657 (1.1831511762407079)
| > current_lr: 5e-06
| > step_time: 0.4133 (0.3720897224214342)
| > loader_time: 0.0109 (0.010839088228013789)
 --> TIME: 2024-10-19 23:42:13 -- STEP: 500/3361 -- GLOBAL_STEP: 500
| > loss_text_ce: 0.11643479764461517 (0.15131162096560002)
| > loss_mel_ce: 4.275294303894043 (4.55400635910034)
| > loss: 1.09793221950531 (1.1763294947147358)
| > current_lr: 5e-06
| > step_time: 0.2165 (0.3713303875923157)
| > loader_time: 0.0096 (0.010830031394958504)
 --> TIME: 2024-10-19 23:42:46 -- STEP: 550/3361 -- GLOBAL_STEP: 550
| > loss_text_ce: 0.09418845921754837 (0.14674024343490602)
| > loss_mel_ce: 4.262049674987793 (4.536511275985024)
| > loss: 1.089059591293335 (1.1708128790421906)
| > current_lr: 5e-06
| > step_time: 0.4339 (0.3719711026278409)
| > loader_time: 0.0107 (0.010789505785161803)
 --> TIME: 2024-10-19 23:43:18 -- STEP: 600/3361 -- GLOBAL_STEP: 600
| > loss_text_ce: 0.1037636324763298 (0.14271454828480878)
| > loss_mel_ce: 4.0221052169799805 (4.519321223497389)
| > loss: 1.0314671993255615 (1.165508942604063)
| > current_lr: 5e-06
| > step_time: 0.3561 (0.37222115119298294)
| > loader_time: 0.011 (0.01072237730026246)
 --> TIME: 2024-10-19 23:43:50 -- STEP: 650/3361 -- GLOBAL_STEP: 650
| > loss_text_ce: 0.10018370300531387 (0.13922331652962247)
| > loss_mel_ce: 4.329763889312744 (4.505329917394195)
| > loss: 1.1074868440628052 (1.1611383080482474)
| > current_lr: 5e-06
| > step_time: 0.3411 (0.3732313658640932)
| > loader_time: 0.0102 (0.010698353694035474)
 --> TIME: 2024-10-19 23:44:22 -- STEP: 700/3361 -- GLOBAL_STEP: 700
| > loss_text_ce: 0.09295706450939178 (0.1359132102131844)
| > loss_mel_ce: 4.329279899597168 (4.492836109910698)
| > loss: 1.105559229850769 (1.1571873297010142)
| > current_lr: 5e-06
| > step_time: 0.2642 (0.3730241230555941)
| > loader_time: 0.0177 (0.010678539276123064)
 --> TIME: 2024-10-19 23:44:54 -- STEP: 750/3361 -- GLOBAL_STEP: 750
| > loss_text_ce: 0.08847153186798096 (0.13301689797639846)
| > loss_mel_ce: 4.319281101226807 (4.478802271842963)
| > loss: 1.1019381284713745 (1.1529547921816505)
| > current_lr: 5e-06
| > step_time: 0.4389 (0.37313773218790675)
| > loader_time: 0.0108 (0.010637306849161803)
 --> TIME: 2024-10-19 23:45:26 -- STEP: 800/3361 -- GLOBAL_STEP: 800
| > loss_text_ce: 0.10376894474029541 (0.13030017000623043)
| > loss_mel_ce: 4.416518211364746 (4.466628861725337)
| > loss: 1.130071759223938 (1.1492322574555873)
| > current_lr: 5e-06
| > step_time: 0.3356 (0.373155618906021)
| > loader_time: 0.01 (0.010589956939220446)
 --> TIME: 2024-10-19 23:45:58 -- STEP: 850/3361 -- GLOBAL_STEP: 850
| > loss_text_ce: 0.09408652782440186 (0.12781955107169998)
| > loss_mel_ce: 4.137832164764404 (4.455110899139858)
| > loss: 1.057979702949524 (1.1457326119086317)
| > current_lr: 5e-06
| > step_time: 0.2973 (0.37324726581573475)
| > loader_time: 0.0089 (0.010562418769387655)
 --> TIME: 2024-10-19 23:46:29 -- STEP: 900/3361 -- GLOBAL_STEP: 900
| > loss_text_ce: 0.08723258972167969 (0.12548734435604691)
| > loss_mel_ce: 4.102336883544922 (4.445588976277252)
| > loss: 1.0473923683166504 (1.1427690789434646)
| > current_lr: 5e-06
| > step_time: 0.3471 (0.37272283368640463)
| > loader_time: 0.0094 (0.010525485144721153)
 --> TIME: 2024-10-19 23:47:01 -- STEP: 950/3361 -- GLOBAL_STEP: 950
| > loss_text_ce: 0.08267280459403992 (0.12352421604489039)
| > loss_mel_ce: 4.262170314788818 (4.436302085926666)
| > loss: 1.0862107276916504 (1.1399565746909694)
| > current_lr: 5e-06
| > step_time: 0.4787 (0.372880862386603)
| > loader_time: 0.0108 (0.010516446766100447)
 --> TIME: 2024-10-19 23:47:33 -- STEP: 1000/3361 -- GLOBAL_STEP: 1000
| > loss_text_ce: 0.08907080441713333 (0.12161937542259706)
| > loss_mel_ce: 4.300572395324707 (4.427557614326484)
| > loss: 1.097410798072815 (1.1372942463755609)
| > current_lr: 5e-06
| > step_time: 0.2458 (0.3725577991008757)
| > loader_time: 0.0089 (0.010495249271392837)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_1000.pth
 --> TIME: 2024-10-19 23:48:15 -- STEP: 1050/3361 -- GLOBAL_STEP: 1050
| > loss_text_ce: 0.0684773176908493 (0.11973775178194057)
| > loss_mel_ce: 4.429049491882324 (4.419832839057566)
| > loss: 1.1243816614151 (1.1348926466419584)
| > current_lr: 5e-06
| > step_time: 0.4162 (0.37267442498888276)
| > loader_time: 0.0106 (0.010481156848725833)
 --> TIME: 2024-10-19 23:48:47 -- STEP: 1100/3361 -- GLOBAL_STEP: 1100
| > loss_text_ce: 0.08142425119876862 (0.11798097466880635)
| > loss_mel_ce: 4.6108479499816895 (4.41094342123379)
| > loss: 1.1730680465698242 (1.1322310978174208)
| > current_lr: 5e-06
| > step_time: 0.3115 (0.3729662624272432)
| > loader_time: 0.0095 (0.01046122507615524)
 --> TIME: 2024-10-19 23:49:19 -- STEP: 1150/3361 -- GLOBAL_STEP: 1150
| > loss_text_ce: 0.08587858825922012 (0.11642520604574172)
| > loss_mel_ce: 4.308182239532471 (4.402262189284622)
| > loss: 1.0985151529312134 (1.129671847872112)
| > current_lr: 5e-06
| > step_time: 0.3725 (0.3727190674906191)
| > loader_time: 0.0103 (0.010423968771229626)
 --> TIME: 2024-10-19 23:49:50 -- STEP: 1200/3361 -- GLOBAL_STEP: 1200
| > loss_text_ce: 0.08292829245328903 (0.11482038536419481)
| > loss_mel_ce: 4.307161808013916 (4.394019900560389)
| > loss: 1.097522497177124 (1.127210070540508)
| > current_lr: 5e-06
| > step_time: 0.3411 (0.37206052283446)
| > loader_time: 0.0117 (0.010386224389076241)
 --> TIME: 2024-10-19 23:50:22 -- STEP: 1250/3361 -- GLOBAL_STEP: 1250
| > loss_text_ce: 0.07650288939476013 (0.11328100087642679)
| > loss_mel_ce: 4.521561622619629 (4.386720856857313)
| > loss: 1.1495161056518555 (1.1250004634380348)
| > current_lr: 5e-06
| > step_time: 0.3442 (0.3723619564056397)
| > loader_time: 0.0099 (0.010367479133605965)
 --> TIME: 2024-10-19 23:50:54 -- STEP: 1300/3361 -- GLOBAL_STEP: 1300
| > loss_text_ce: 0.08411215245723724 (0.11187859488794445)
| > loss_mel_ce: 4.311476707458496 (4.378952882289896)
| > loss: 1.0988972187042236 (1.1227078686769207)
| > current_lr: 5e-06
| > step_time: 0.3467 (0.37273642576657834)
| > loader_time: 0.0101 (0.010353383284348715)
 --> TIME: 2024-10-19 23:51:26 -- STEP: 1350/3361 -- GLOBAL_STEP: 1350
| > loss_text_ce: 0.07017528265714645 (0.11054472846565432)
| > loss_mel_ce: 4.101247310638428 (4.371401038876297)
| > loss: 1.0428556203842163 (1.1204864410559354)
| > current_lr: 5e-06
| > step_time: 0.3274 (0.3727424639242669)
| > loader_time: 0.009 (0.010344070858425569)
 --> TIME: 2024-10-19 23:51:58 -- STEP: 1400/3361 -- GLOBAL_STEP: 1400
| > loss_text_ce: 0.07565019279718399 (0.10928024096148363)
| > loss_mel_ce: 3.7528905868530273 (4.364196109771742)
| > loss: 0.9571352005004883 (1.1183690869382463)
| > current_lr: 5e-06
| > step_time: 0.3347 (0.372393846511841)
| > loader_time: 0.0104 (0.01032195346696037)
 --> TIME: 2024-10-19 23:52:29 -- STEP: 1450/3361 -- GLOBAL_STEP: 1450
| > loss_text_ce: 0.07104843109846115 (0.10808574935485585)
| > loss_mel_ce: 4.024135112762451 (4.357662479630846)
| > loss: 1.0237958431243896 (1.116437056722312)
| > current_lr: 5e-06
| > step_time: 0.4162 (0.3723816107059349)
| > loader_time: 0.0112 (0.010314793751157571)
 --> TIME: 2024-10-19 23:53:02 -- STEP: 1500/3361 -- GLOBAL_STEP: 1500
| > loss_text_ce: 0.0696822851896286 (0.1068931187863152)
| > loss_mel_ce: 4.51359748840332 (4.351274299780541)
| > loss: 1.145819902420044 (1.114541854182878)
| > current_lr: 5e-06
| > step_time: 0.2727 (0.37269472678502413)
| > loader_time: 0.0105 (0.010296331087748217)
 --> TIME: 2024-10-19 23:53:33 -- STEP: 1550/3361 -- GLOBAL_STEP: 1550
| > loss_text_ce: 0.07638457417488098 (0.10576395642132536)
| > loss_mel_ce: 4.133301258087158 (4.344630276618483)
| > loss: 1.0524214506149292 (1.112598557972137)
| > current_lr: 5e-06
| > step_time: 0.421 (0.37251200337563806)
| > loader_time: 0.009 (0.010285784506028706)
 --> TIME: 2024-10-19 23:54:05 -- STEP: 1600/3361 -- GLOBAL_STEP: 1600
| > loss_text_ce: 0.07427678257226944 (0.10470495257293813)
| > loss_mel_ce: 4.1771016120910645 (4.33880649805071)
| > loss: 1.0628446340560913 (1.1108778622746458)
| > current_lr: 5e-06
| > step_time: 0.3331 (0.37268786862492576)
| > loader_time: 0.0117 (0.010273052901029596)
 --> TIME: 2024-10-19 23:54:38 -- STEP: 1650/3361 -- GLOBAL_STEP: 1650
| > loss_text_ce: 0.0688512995839119 (0.10366868965779297)
| > loss_mel_ce: 4.372562408447266 (4.333273339271568)
| > loss: 1.1103534698486328 (1.10923550670797)
| > current_lr: 5e-06
| > step_time: 0.3577 (0.3731871593359746)
| > loader_time: 0.0107 (0.010268038836392497)
 --> TIME: 2024-10-19 23:55:10 -- STEP: 1700/3361 -- GLOBAL_STEP: 1700
| > loss_text_ce: 0.06728693842887878 (0.1026264984883807)
| > loss_mel_ce: 3.8259117603302 (4.327509120913133)
| > loss: 0.9732996821403503 (1.1075339043491004)
| > current_lr: 5e-06
| > step_time: 0.2424 (0.3731728926826927)
| > loader_time: 0.0093 (0.010254677323734068)
 --> TIME: 2024-10-19 23:55:43 -- STEP: 1750/3361 -- GLOBAL_STEP: 1750
| > loss_text_ce: 0.07139215618371964 (0.1016620131390436)
| > loss_mel_ce: 4.0196332931518555 (4.321489932332737)
| > loss: 1.0227563381195068 (1.105787986108232)
| > current_lr: 5e-06
| > step_time: 0.3334 (0.37345534951346276)
| > loader_time: 0.0088 (0.010236183302743103)
 --> TIME: 2024-10-19 23:56:15 -- STEP: 1800/3361 -- GLOBAL_STEP: 1800
| > loss_text_ce: 0.06817279011011124 (0.10072386607320778)
| > loss_mel_ce: 4.305867671966553 (4.3159451058176)
| > loss: 1.0935101509094238 (1.1041672426793276)
| > current_lr: 5e-06
| > step_time: 0.2647 (0.3735462660259672)
| > loader_time: 0.0089 (0.01022007279925877)
 --> TIME: 2024-10-19 23:56:47 -- STEP: 1850/3361 -- GLOBAL_STEP: 1850
| > loss_text_ce: 0.06191863492131233 (0.0997890573478229)
| > loss_mel_ce: 4.1208672523498535 (4.310428036225824)
| > loss: 1.045696496963501 (1.102554273347594)
| > current_lr: 5e-06
| > step_time: 0.341 (0.37339078568123496)
| > loader_time: 0.0094 (0.010211075576576035)
 --> TIME: 2024-10-19 23:57:18 -- STEP: 1900/3361 -- GLOBAL_STEP: 1900
| > loss_text_ce: 0.06634141504764557 (0.09888596163376387)
| > loss_mel_ce: 4.26849365234375 (4.304874293176769)
| > loss: 1.0837087631225586 (1.1009400636585107)
| > current_lr: 5e-06
| > step_time: 0.3123 (0.3732277792378477)
| > loader_time: 0.0107 (0.010206241356699097)
 --> TIME: 2024-10-19 23:57:50 -- STEP: 1950/3361 -- GLOBAL_STEP: 1950
| > loss_text_ce: 0.07070682942867279 (0.09804161607837071)
| > loss_mel_ce: 4.126957416534424 (4.298485514934264)
| > loss: 1.0494160652160645 (1.099131782666228)
| > current_lr: 5e-06
| > step_time: 0.4269 (0.3732747751627214)
| > loader_time: 0.0108 (0.010196818449558363)
 --> TIME: 2024-10-19 23:58:22 -- STEP: 2000/3361 -- GLOBAL_STEP: 2000
| > loss_text_ce: 0.06333263963460922 (0.09720117196254438)
| > loss_mel_ce: 3.9823367595672607 (4.293063371539132)
| > loss: 1.0114173889160156 (1.0975661356747124)
| > current_lr: 5e-06
| > step_time: 0.3422 (0.3733924993276597)
| > loader_time: 0.0114 (0.010184644460678108)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_2000.pth
 --> TIME: 2024-10-19 23:59:04 -- STEP: 2050/3361 -- GLOBAL_STEP: 2050
| > loss_text_ce: 0.07292477041482925 (0.09640612774324132)
| > loss_mel_ce: 3.918966770172119 (4.286913629973823)
| > loss: 0.9979729056358337 (1.0958299392316373)
| > current_lr: 5e-06
| > step_time: 0.3183 (0.3732672268006861)
| > loader_time: 0.01 (0.01018628969425109)
 --> TIME: 2024-10-19 23:59:36 -- STEP: 2100/3361 -- GLOBAL_STEP: 2100
| > loss_text_ce: 0.05803373083472252 (0.09563219899755154)
| > loss_mel_ce: 4.1672043800354 (4.281224559375234)
| > loss: 1.0563095808029175 (1.0942141895350932)
| > current_lr: 5e-06
| > step_time: 0.317 (0.37334919940857664)
| > loader_time: 0.0096 (0.010179950850350523)
 --> TIME: 2024-10-20 00:00:09 -- STEP: 2150/3361 -- GLOBAL_STEP: 2150
| > loss_text_ce: 0.07027828693389893 (0.09487137659517834)
| > loss_mel_ce: 4.074728488922119 (4.276105512020217)
| > loss: 1.0362516641616821 (1.0927442217982068)
| > current_lr: 5e-06
| > step_time: 0.358 (0.37359854487485666)
| > loader_time: 0.0108 (0.010167785023534022)
 --> TIME: 2024-10-20 00:00:40 -- STEP: 2200/3361 -- GLOBAL_STEP: 2200
| > loss_text_ce: 0.06147142872214317 (0.09415466819127856)
| > loss_mel_ce: 4.060927391052246 (4.2708785486221466)
| > loss: 1.0305997133255005 (1.0912583039565495)
| > current_lr: 5e-06
| > step_time: 0.3124 (0.37341777844862495)
| > loader_time: 0.0088 (0.010151374773545701)
 --> TIME: 2024-10-20 00:01:12 -- STEP: 2250/3361 -- GLOBAL_STEP: 2250
| > loss_text_ce: 0.0592532679438591 (0.09346097443997865)
| > loss_mel_ce: 4.088995933532715 (4.26577778604297)
| > loss: 1.0370622873306274 (1.089809689680733)
| > current_lr: 5e-06
| > step_time: 0.3943 (0.3732477170096502)
| > loader_time: 0.0093 (0.010148488998413085)
 --> TIME: 2024-10-20 00:01:45 -- STEP: 2300/3361 -- GLOBAL_STEP: 2300
| > loss_text_ce: 0.05973433703184128 (0.0927704642372935)
| > loss_mel_ce: 4.142950057983398 (4.261306318303825)
| > loss: 1.050671100616455 (1.0885191953182198)
| > current_lr: 5e-06
| > step_time: 0.316 (0.3735742027863211)
| > loader_time: 0.0106 (0.010143468172653866)
 --> TIME: 2024-10-20 00:02:17 -- STEP: 2350/3361 -- GLOBAL_STEP: 2350
| > loss_text_ce: 0.0632677748799324 (0.09211754334258279)
| > loss_mel_ce: 3.9679794311523438 (4.256112595010319)
| > loss: 1.0078117847442627 (1.0870575342787052)
| > current_lr: 5e-06
| > step_time: 0.4684 (0.373737255867491)
| > loader_time: 0.0102 (0.01013411846566708)
 --> TIME: 2024-10-20 00:02:48 -- STEP: 2400/3361 -- GLOBAL_STEP: 2400
| > loss_text_ce: 0.06617214530706406 (0.09147963946840423)
| > loss_mel_ce: 4.05083703994751 (4.251065555413567)
| > loss: 1.029252290725708 (1.085636298432944)
| > current_lr: 5e-06
| > step_time: 0.2359 (0.37363693376382134)
| > loader_time: 0.0095 (0.010126104454199468)
 --> TIME: 2024-10-20 00:03:20 -- STEP: 2450/3361 -- GLOBAL_STEP: 2450
| > loss_text_ce: 0.05000381916761398 (0.0908709145945555)
| > loss_mel_ce: 3.871734619140625 (4.246652321036983)
| > loss: 0.9804345965385437 (1.0843808086307658)
| > current_lr: 5e-06
| > step_time: 0.3701 (0.37357356207711284)
| > loader_time: 0.0088 (0.010115676704718139)
 --> TIME: 2024-10-20 00:03:52 -- STEP: 2500/3361 -- GLOBAL_STEP: 2500
| > loss_text_ce: 0.06191703677177429 (0.09026267003268017)
| > loss_mel_ce: 4.080785274505615 (4.241838634967805)
| > loss: 1.0356755256652832 (1.0830253259420368)
| > current_lr: 5e-06
| > step_time: 0.2731 (0.37364912557601854)
| > loader_time: 0.0096 (0.010112837028503414)
 --> TIME: 2024-10-20 00:04:25 -- STEP: 2550/3361 -- GLOBAL_STEP: 2550
| > loss_text_ce: 0.06187329813838005 (0.08967712040187109)
| > loss_mel_ce: 3.9666688442230225 (4.237735574292203)
| > loss: 1.0071355104446411 (1.081853173433563)
| > current_lr: 5e-06
| > step_time: 0.3046 (0.37373014066733484)
| > loader_time: 0.009 (0.010112988154093427)
 --> TIME: 2024-10-20 00:04:57 -- STEP: 2600/3361 -- GLOBAL_STEP: 2600
| > loss_text_ce: 0.06246669963002205 (0.08912380175951591)
| > loss_mel_ce: 3.9966068267822266 (4.234155957056928)
| > loss: 1.014768362045288 (1.0808199393978473)
| > current_lr: 5e-06
| > step_time: 0.3483 (0.3738965059243708)
| > loader_time: 0.0102 (0.010113027279193589)
 --> TIME: 2024-10-20 00:05:29 -- STEP: 2650/3361 -- GLOBAL_STEP: 2650
| > loss_text_ce: 0.0602920725941658 (0.08857007859872204)
| > loss_mel_ce: 3.8786299228668213 (4.230075319668037)
| > loss: 0.9847304821014404 (1.0796613493055658)
| > current_lr: 5e-06
| > step_time: 0.3352 (0.3738692178366311)
| > loader_time: 0.0092 (0.010110449071200393)
 --> TIME: 2024-10-20 00:06:01 -- STEP: 2700/3361 -- GLOBAL_STEP: 2700
| > loss_text_ce: 0.060905251652002335 (0.08802893280845003)
| > loss_mel_ce: 4.000962257385254 (4.225500499760666)
| > loss: 1.0154669284820557 (1.0783823577783709)
| > current_lr: 5e-06
| > step_time: 0.3291 (0.37374789873758885)
| > loader_time: 0.0107 (0.010111093609421347)
 --> TIME: 2024-10-20 00:06:33 -- STEP: 2750/3361 -- GLOBAL_STEP: 2750
| > loss_text_ce: 0.06092409789562225 (0.087514858617024)
| > loss_mel_ce: 3.8547213077545166 (4.221181369434707)
| > loss: 0.9789113402366638 (1.077174056638369)
| > current_lr: 5e-06
| > step_time: 0.4815 (0.3739039562398729)
| > loader_time: 0.0093 (0.010105457912791861)
 --> TIME: 2024-10-20 00:07:05 -- STEP: 2800/3361 -- GLOBAL_STEP: 2800
| > loss_text_ce: 0.06143264099955559 (0.08701184531939885)
| > loss_mel_ce: 4.019796848297119 (4.216923976199974)
| > loss: 1.0203074216842651 (1.0759839550299288)
| > current_lr: 5e-06
| > step_time: 0.2735 (0.3738308864831916)
| > loader_time: 0.0091 (0.01010349988937378)
 --> TIME: 2024-10-20 00:07:36 -- STEP: 2850/3361 -- GLOBAL_STEP: 2850
| > loss_text_ce: 0.06067034229636192 (0.08651230146356849)
| > loss_mel_ce: 3.9777588844299316 (4.2130854128954764)
| > loss: 1.0096073150634766 (1.0748994283090547)
| > current_lr: 5e-06
| > step_time: 0.3746 (0.37365791663788833)
| > loader_time: 0.0088 (0.010102545671295695)
 --> TIME: 2024-10-20 00:08:09 -- STEP: 2900/3361 -- GLOBAL_STEP: 2900
| > loss_text_ce: 0.0590413473546505 (0.08602534580975783)
| > loss_mel_ce: 3.9361588954925537 (4.208741374755734)
| > loss: 0.9988000392913818 (1.0736916798147638)
| > current_lr: 5e-06
| > step_time: 0.245 (0.3738863233862244)
| > loader_time: 0.0129 (0.010097931253498997)
 --> TIME: 2024-10-20 00:08:42 -- STEP: 2950/3361 -- GLOBAL_STEP: 2950
| > loss_text_ce: 0.05851312354207039 (0.08556444178445886)
| > loss_mel_ce: 3.8445589542388916 (4.204549880108596)
| > loss: 0.9757680296897888 (1.0725285801240934)
| > current_lr: 5e-06
| > step_time: 0.3277 (0.3740873582484351)
| > loader_time: 0.0091 (0.010090556467993783)
 --> TIME: 2024-10-20 00:09:13 -- STEP: 3000/3361 -- GLOBAL_STEP: 3000
| > loss_text_ce: 0.052952103316783905 (0.08509168772771973)
| > loss_mel_ce: 3.7599077224731445 (4.200607080777491)
| > loss: 0.9532149434089661 (1.0714246917963006)
| > current_lr: 5e-06
| > step_time: 0.2744 (0.37406464068094814)
| > loader_time: 0.0097 (0.010087447325388594)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_3000.pth
 --> TIME: 2024-10-20 00:09:56 -- STEP: 3050/3361 -- GLOBAL_STEP: 3050
| > loss_text_ce: 0.05916506052017212 (0.08463779314375335)
| > loss_mel_ce: 3.8810315132141113 (4.196892655091211)
| > loss: 0.9850491285324097 (1.0703826116929271)
| > current_lr: 5e-06
| > step_time: 0.3444 (0.37403536546425664)
| > loader_time: 0.0089 (0.010088322827073403)
 --> TIME: 2024-10-20 00:10:29 -- STEP: 3100/3361 -- GLOBAL_STEP: 3100
| > loss_text_ce: 0.06803478300571442 (0.08422918403581285)
| > loss_mel_ce: 3.9018285274505615 (4.192189099711762)
| > loss: 0.9924658536911011 (1.0691045704772382)
| > current_lr: 5e-06
| > step_time: 0.3126 (0.3742765197446262)
| > loader_time: 0.0108 (0.010089200004454596)
 --> TIME: 2024-10-20 00:11:00 -- STEP: 3150/3361 -- GLOBAL_STEP: 3150
| > loss_text_ce: 0.055581387132406235 (0.08381401562974586)
| > loss_mel_ce: 3.712331533432007 (4.187636630345909)
| > loss: 0.9419782161712646 (1.0678626610740767)
| > current_lr: 5e-06
| > step_time: 0.2662 (0.37410105145166717)
| > loader_time: 0.0104 (0.010091308109344013)
 --> TIME: 2024-10-20 00:11:32 -- STEP: 3200/3361 -- GLOBAL_STEP: 3200
| > loss_text_ce: 0.05516338720917702 (0.08340078805340473)
| > loss_mel_ce: 3.8296306133270264 (4.183248083516961)
| > loss: 0.9711984992027283 (1.0666622174717475)
| > current_lr: 5e-06
| > step_time: 0.3456 (0.37414847671985557)
| > loader_time: 0.0096 (0.010088659971952449)
 --> TIME: 2024-10-20 00:12:04 -- STEP: 3250/3361 -- GLOBAL_STEP: 3250
| > loss_text_ce: 0.058774255216121674 (0.08300397763917099)
| > loss_mel_ce: 4.043879508972168 (4.179145359039312)
| > loss: 1.0256634950637817 (1.0655373337635616)
| > current_lr: 5e-06
| > step_time: 0.3054 (0.3741027381603527)
| > loader_time: 0.0098 (0.010083743462195774)
 --> TIME: 2024-10-20 00:12:36 -- STEP: 3300/3361 -- GLOBAL_STEP: 3300
| > loss_text_ce: 0.05571791157126427 (0.08262370134393393)
| > loss_mel_ce: 4.039697647094727 (4.175351190856015)
| > loss: 1.0238538980484009 (1.0644937225724693)
| > current_lr: 5e-06
| > step_time: 0.355 (0.37421785773652905)
| > loader_time: 0.0105 (0.010079322367003481)
 --> TIME: 2024-10-20 00:13:08 -- STEP: 3350/3361 -- GLOBAL_STEP: 3350
| > loss_text_ce: 0.0516965389251709 (0.08223508200761116)
| > loss_mel_ce: 4.055713653564453 (4.171607753767902)
| > loss: 1.0268526077270508 (1.0634607085541095)
| > current_lr: 5e-06
| > step_time: 0.4114 (0.37421482968686165)
| > loader_time: 0.0087 (0.010072118702219504)
 > EVALUATION 
--> EVAL PERFORMANCE
| > avg_loader_time: 0.3817212197088426 (+0)
| > avg_loss_text_ce: 0.05487057370364025 (+0)
| > avg_loss_mel_ce: 3.7456871719770533 (+0)
| > avg_loss: 3.8005577415548344 (+0)
 > EPOCH: 1/5
--> /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b
 > TRAINING (2024-10-20 00:15:40) 
 --> TIME: 2024-10-20 00:16:06 -- STEP: 39/3361 -- GLOBAL_STEP: 3400
| > loss_text_ce: 0.05803585425019264 (0.05740316842611019)
| > loss_mel_ce: 4.036578178405762 (3.9442618443415713)
| > loss: 1.023653507232666 (1.0004162574425723)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4891 (0.36824990541507036)
| > loader_time: 0.0111 (0.0100095883393899)
 --> TIME: 2024-10-20 00:16:38 -- STEP: 89/3361 -- GLOBAL_STEP: 3450
| > loss_text_ce: 0.05387208238244057 (0.057160088897086264)
| > loss_mel_ce: 3.8522324562072754 (3.9176201766796326)
| > loss: 0.976526141166687 (0.9936950695648622)
| > current_lr: 5e-06
| > step_time: 0.3078 (0.3701958361636386)
| > loader_time: 0.0101 (0.00997726836901033)
 --> TIME: 2024-10-20 00:17:10 -- STEP: 139/3361 -- GLOBAL_STEP: 3500
| > loss_text_ce: 0.05204734206199646 (0.05694234341704588)
| > loss_mel_ce: 4.180244445800781 (3.895403874006203)
| > loss: 1.0580729246139526 (0.9880865542151087)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3894 (0.3720308207779479)
| > loader_time: 0.0087 (0.009898750044459064)
 --> TIME: 2024-10-20 00:17:42 -- STEP: 189/3361 -- GLOBAL_STEP: 3550
| > loss_text_ce: 0.05554436519742012 (0.05678758705182681)
| > loss_mel_ce: 4.228486061096191 (3.8946746341765874)
| > loss: 1.0710076093673706 (0.9878655549079652)
| > current_lr: 5e-06
| > step_time: 0.3746 (0.37256827934709175)
| > loader_time: 0.0099 (0.00987515752277678)
 --> TIME: 2024-10-20 00:18:13 -- STEP: 239/3361 -- GLOBAL_STEP: 3600
| > loss_text_ce: 0.05677724629640579 (0.05658898376333664)
| > loss_mel_ce: 3.8501322269439697 (3.890875055201383)
| > loss: 0.9767273664474487 (0.9868660100334359)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.34 (0.3716044445915701)
| > loader_time: 0.0093 (0.009896839014157102)
 --> TIME: 2024-10-20 00:18:44 -- STEP: 289/3361 -- GLOBAL_STEP: 3650
| > loss_text_ce: 0.056557197123765945 (0.056536512623402486)
| > loss_mel_ce: 3.79600191116333 (3.8830668728244344)
| > loss: 0.9631397724151611 (0.9849008476445419)
| > current_lr: 5e-06
| > step_time: 0.3711 (0.36885856262127814)
| > loader_time: 0.0107 (0.009862431192893067)
 --> TIME: 2024-10-20 00:19:17 -- STEP: 339/3361 -- GLOBAL_STEP: 3700
| > loss_text_ce: 0.054546210914850235 (0.056364989078502035)
| > loss_mel_ce: 3.885460615158081 (3.8856594998576295)
| > loss: 0.9850016832351685 (0.9855061231461246)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.5478 (0.3709612870286699)
| > loader_time: 0.0111 (0.009907801242704593)
 --> TIME: 2024-10-20 00:19:48 -- STEP: 389/3361 -- GLOBAL_STEP: 3750
| > loss_text_ce: 0.05074138566851616 (0.056212213034035306)
| > loss_mel_ce: 3.9127519130706787 (3.887668692360807)
| > loss: 0.9908733367919922 (0.9859702276080311)
| > current_lr: 5e-06
| > step_time: 0.428 (0.3711606475565304)
| > loader_time: 0.0108 (0.009874057647195157)
 --> TIME: 2024-10-20 00:20:20 -- STEP: 439/3361 -- GLOBAL_STEP: 3800
| > loss_text_ce: 0.05833661928772926 (0.05614948455999544)
| > loss_mel_ce: 3.801664352416992 (3.884764766910353)
| > loss: 0.9650002717971802 (0.9852285640386351)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3721 (0.3714690550584727)
| > loader_time: 0.0103 (0.009873224010771672)
 --> TIME: 2024-10-20 00:20:53 -- STEP: 489/3361 -- GLOBAL_STEP: 3850
| > loss_text_ce: 0.05502821505069733 (0.05603404230379132)
| > loss_mel_ce: 3.728093385696411 (3.880742844872192)
| > loss: 0.9457803964614868 (0.9841942235004683)
| > current_lr: 5e-06
| > step_time: 0.3332 (0.3729910514106048)
| > loader_time: 0.0089 (0.009863832976920481)
 --> TIME: 2024-10-20 00:21:26 -- STEP: 539/3361 -- GLOBAL_STEP: 3900
| > loss_text_ce: 0.05435081198811531 (0.056033372885939806)
| > loss_mel_ce: 3.737413167953491 (3.8785685820570683)
| > loss: 0.94794100522995 (0.9836504911668668)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4252 (0.37414260299838303)
| > loader_time: 0.0095 (0.00982684460995592)
 --> TIME: 2024-10-20 00:21:57 -- STEP: 589/3361 -- GLOBAL_STEP: 3950
| > loss_text_ce: 0.052592385560274124 (0.056006750961566404)
| > loss_mel_ce: 3.746580123901367 (3.8754911434873054)
| > loss: 0.9497931003570557 (0.9828744762417416)
| > current_lr: 5e-06
| > step_time: 0.3749 (0.37371972825612176)
| > loader_time: 0.0095 (0.009833886384559211)
 --> TIME: 2024-10-20 00:22:29 -- STEP: 639/3361 -- GLOBAL_STEP: 4000
| > loss_text_ce: 0.05487600341439247 (0.05590341322368858)
| > loss_mel_ce: 3.9473564624786377 (3.872275245021766)
| > loss: 1.0005581378936768 (0.982044667052178)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3879 (0.3738370978962831)
| > loader_time: 0.0092 (0.009821679857042106)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_4000.pth
 --> TIME: 2024-10-20 00:23:14 -- STEP: 689/3361 -- GLOBAL_STEP: 4050
| > loss_text_ce: 0.05411239340901375 (0.055867320956616326)
| > loss_mel_ce: 3.7576539516448975 (3.8687809928927264)
| > loss: 0.9529415965080261 (0.9811620808305173)
| > current_lr: 5e-06
| > step_time: 0.4296 (0.37478768046950717)
| > loader_time: 0.0103 (0.009839309490297987)
 --> TIME: 2024-10-20 00:23:46 -- STEP: 739/3361 -- GLOBAL_STEP: 4100
| > loss_text_ce: 0.0570734404027462 (0.055781693188965066)
| > loss_mel_ce: 3.8863158226013184 (3.867509481226156)
| > loss: 0.9858472943305969 (0.9808227957827473)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3456 (0.3744364284856069)
| > loader_time: 0.0095 (0.009838796920156942)
 --> TIME: 2024-10-20 00:24:17 -- STEP: 789/3361 -- GLOBAL_STEP: 4150
| > loss_text_ce: 0.05545391887426376 (0.05572522812167652)
| > loss_mel_ce: 3.894559860229492 (3.8668422457232072)
| > loss: 0.9875034689903259 (0.9806418704442652)
| > current_lr: 5e-06
| > step_time: 0.3955 (0.3739068774064984)
| > loader_time: 0.0098 (0.009847505917567279)
 --> TIME: 2024-10-20 00:24:49 -- STEP: 839/3361 -- GLOBAL_STEP: 4200
| > loss_text_ce: 0.048767924308776855 (0.05559879375779301)
| > loss_mel_ce: 3.6436502933502197 (3.865561713478989)
| > loss: 0.9231045246124268 (0.9802901287328927)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4775 (0.37385038369034396)
| > loader_time: 0.0108 (0.009849785905913051)
 --> TIME: 2024-10-20 00:25:22 -- STEP: 889/3361 -- GLOBAL_STEP: 4250
| > loss_text_ce: 0.055653173476457596 (0.05556447896163562)
| > loss_mel_ce: 3.7202305793762207 (3.8628411008751162)
| > loss: 0.9439709186553955 (0.9796013970879254)
| > current_lr: 5e-06
| > step_time: 0.3718 (0.3747204725868386)
| > loader_time: 0.009 (0.009871045435507483)
 --> TIME: 2024-10-20 00:25:54 -- STEP: 939/3361 -- GLOBAL_STEP: 4300
| > loss_text_ce: 0.05329810455441475 (0.05552343320779906)
| > loss_mel_ce: 3.8362557888031006 (3.859880526860555)
| > loss: 0.9723884463310242 (0.978850992215962)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.461 (0.37450057426834493)
| > loader_time: 0.0095 (0.009879123669462858)
 --> TIME: 2024-10-20 00:26:26 -- STEP: 989/3361 -- GLOBAL_STEP: 4350
| > loss_text_ce: 0.05188177898526192 (0.05547151509653569)
| > loss_mel_ce: 3.7463533878326416 (3.858084853908049)
| > loss: 0.9495587944984436 (0.978389094403829)
| > current_lr: 5e-06
| > step_time: 0.3389 (0.37451345855473733)
| > loader_time: 0.01 (0.009874366291613155)
 --> TIME: 2024-10-20 00:26:57 -- STEP: 1039/3361 -- GLOBAL_STEP: 4400
| > loss_text_ce: 0.05315755307674408 (0.055404065257517385)
| > loss_mel_ce: 4.059134006500244 (3.8563873499373735)
| > loss: 1.0280728340148926 (0.9779478560512862)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.463 (0.37416290777938876)
| > loader_time: 0.0101 (0.009874617169979576)
 --> TIME: 2024-10-20 00:27:29 -- STEP: 1089/3361 -- GLOBAL_STEP: 4450
| > loss_text_ce: 0.05398701876401901 (0.05530707812136854)
| > loss_mel_ce: 3.651801586151123 (3.8551332309116892)
| > loss: 0.9264471530914307 (0.977610079491215)
| > current_lr: 5e-06
| > step_time: 0.3285 (0.37405674845062337)
| > loader_time: 0.0086 (0.009866176776868487)
 --> TIME: 2024-10-20 00:28:01 -- STEP: 1139/3361 -- GLOBAL_STEP: 4500
| > loss_text_ce: 0.05266125500202179 (0.05522818537633389)
| > loss_mel_ce: 3.816060781478882 (3.8524879304016757)
| > loss: 0.9671804904937744 (0.976929031044271)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4832 (0.3738998722046274)
| > loader_time: 0.0114 (0.00985939969505733)
 --> TIME: 2024-10-20 00:28:33 -- STEP: 1189/3361 -- GLOBAL_STEP: 4550
| > loss_text_ce: 0.050825513899326324 (0.055144575364179604)
| > loss_mel_ce: 3.6082472801208496 (3.849461859068618)
| > loss: 0.9147682189941406 (0.9761516108444701)
| > current_lr: 5e-06
| > step_time: 0.2678 (0.37407235281120543)
| > loader_time: 0.0097 (0.009851126434223515)
 --> TIME: 2024-10-20 00:29:04 -- STEP: 1239/3361 -- GLOBAL_STEP: 4600
| > loss_text_ce: 0.056511420756578445 (0.05509090997561317)
| > loss_mel_ce: 3.6945228576660156 (3.846852075293528)
| > loss: 0.9377585649490356 (0.9754857485384784)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.5009 (0.3736923908975647)
| > loader_time: 0.0092 (0.009838911099622483)
 --> TIME: 2024-10-20 00:29:37 -- STEP: 1289/3361 -- GLOBAL_STEP: 4650
| > loss_text_ce: 0.057481300085783005 (0.055016553643556204)
| > loss_mel_ce: 3.841536045074463 (3.84445180201179)
| > loss: 0.9747543334960938 (0.9748670911492998)
| > current_lr: 5e-06
| > step_time: 0.3056 (0.3741835676671183)
| > loader_time: 0.0101 (0.009838919199373113)
 --> TIME: 2024-10-20 00:30:09 -- STEP: 1339/3361 -- GLOBAL_STEP: 4700
| > loss_text_ce: 0.05459923669695854 (0.05495443729832599)
| > loss_mel_ce: 3.5833394527435303 (3.841443535282685)
| > loss: 0.9094846844673157 (0.9740994955017284)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4246 (0.3741715670166128)
| > loader_time: 0.0093 (0.009824278107145276)
 --> TIME: 2024-10-20 00:30:41 -- STEP: 1389/3361 -- GLOBAL_STEP: 4750
| > loss_text_ce: 0.05468934029340744 (0.05488722276846394)
| > loss_mel_ce: 3.722651958465576 (3.838512401258113)
| > loss: 0.9443353414535522 (0.9733499083010905)
| > current_lr: 5e-06
| > step_time: 0.3922 (0.3743942389821215)
| > loader_time: 0.0105 (0.009819968644451805)
 --> TIME: 2024-10-20 00:31:13 -- STEP: 1439/3361 -- GLOBAL_STEP: 4800
| > loss_text_ce: 0.0551159530878067 (0.05482199154095237)
| > loss_mel_ce: 3.8832504749298096 (3.8361284149612627)
| > loss: 0.9845916032791138 (0.9727376038370404)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3457 (0.3745205248288596)
| > loader_time: 0.0105 (0.009818825479842789)
 --> TIME: 2024-10-20 00:31:46 -- STEP: 1489/3361 -- GLOBAL_STEP: 4850
| > loss_text_ce: 0.05104149132966995 (0.054750571161248346)
| > loss_mel_ce: 3.526203155517578 (3.8339387963809286)
| > loss: 0.8943111896514893 (0.9721723442673282)
| > current_lr: 5e-06
| > step_time: 0.3055 (0.37465434349808124)
| > loader_time: 0.0093 (0.009820129824933629)
 --> TIME: 2024-10-20 00:32:17 -- STEP: 1539/3361 -- GLOBAL_STEP: 4900
| > loss_text_ce: 0.0571925975382328 (0.054688119172778805)
| > loss_mel_ce: 3.850430965423584 (3.830406569598014)
| > loss: 0.976905882358551 (0.9712736745636675)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.2878 (0.3744973553861402)
| > loader_time: 0.0093 (0.009813829704567234)
 --> TIME: 2024-10-20 00:32:50 -- STEP: 1589/3361 -- GLOBAL_STEP: 4950
| > loss_text_ce: 0.05071800947189331 (0.054632533395567937)
| > loss_mel_ce: 3.786651611328125 (3.8284057811643435)
| > loss: 0.9593424201011658 (0.9707595809515802)
| > current_lr: 5e-06
| > step_time: 0.4317 (0.37486315898883243)
| > loader_time: 0.0091 (0.009814189919291092)
 --> TIME: 2024-10-20 00:33:21 -- STEP: 1639/3361 -- GLOBAL_STEP: 5000
| > loss_text_ce: 0.05272924154996872 (0.05457410791603582)
| > loss_mel_ce: 3.7636656761169434 (3.8264297995354943)
| > loss: 0.9540987014770508 (0.9702509792204526)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4982 (0.3746423821684933)
| > loader_time: 0.0106 (0.009817314555253682)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_5000.pth
 --> TIME: 2024-10-20 00:34:04 -- STEP: 1689/3361 -- GLOBAL_STEP: 5050
| > loss_text_ce: 0.053886573761701584 (0.05451192842468909)
| > loss_mel_ce: 3.6620142459869385 (3.8244482417866066)
| > loss: 0.9289752244949341 (0.9697400449365465)
| > current_lr: 5e-06
| > step_time: 0.3776 (0.3748740332588208)
| > loader_time: 0.0088 (0.009828670551397162)
 --> TIME: 2024-10-20 00:34:37 -- STEP: 1739/3361 -- GLOBAL_STEP: 5100
| > loss_text_ce: 0.05141803249716759 (0.0544602110759152)
| > loss_mel_ce: 3.5928444862365723 (3.820919352608758)
| > loss: 0.9110656380653381 (0.9688448931994008)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.2895 (0.37512943811838884)
| > loader_time: 0.0088 (0.009832062482696757)
 --> TIME: 2024-10-20 00:35:08 -- STEP: 1789/3361 -- GLOBAL_STEP: 5150
| > loss_text_ce: 0.056414783000946045 (0.054400483986530426)
| > loss_mel_ce: 3.5568387508392334 (3.818622766625888)
| > loss: 0.903313398361206 (0.9682558148582932)
| > current_lr: 5e-06
| > step_time: 0.3325 (0.37502415170877446)
| > loader_time: 0.0103 (0.009836978923015046)
 --> TIME: 2024-10-20 00:35:40 -- STEP: 1839/3361 -- GLOBAL_STEP: 5200
| > loss_text_ce: 0.05202397704124451 (0.05435034733563417)
| > loss_mel_ce: 3.6310930252075195 (3.816511524054199)
| > loss: 0.9207792282104492 (0.9677154700868865)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.334 (0.374689039813234)
| > loader_time: 0.0086 (0.00983762896663796)
 --> TIME: 2024-10-20 00:36:12 -- STEP: 1889/3361 -- GLOBAL_STEP: 5250
| > loss_text_ce: 0.049998633563518524 (0.05429816679105106)
| > loss_mel_ce: 3.725067138671875 (3.814177731599298)
| > loss: 0.9437664151191711 (0.9671189767018142)
| > current_lr: 5e-06
| > step_time: 0.432 (0.3749042445859812)
| > loader_time: 0.0097 (0.00983408887024087)
 --> TIME: 2024-10-20 00:36:45 -- STEP: 1939/3361 -- GLOBAL_STEP: 5300
| > loss_text_ce: 0.05160053074359894 (0.05423524983845892)
| > loss_mel_ce: 3.95269513130188 (3.8125401895768007)
| > loss: 1.001073956489563 (0.9666938619124268)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4847 (0.3752489017419681)
| > loader_time: 0.0089 (0.009828678537361885)
 --> TIME: 2024-10-20 00:37:16 -- STEP: 1989/3361 -- GLOBAL_STEP: 5350
| > loss_text_ce: 0.05233506113290787 (0.05417726464595789)
| > loss_mel_ce: 4.050232887268066 (3.8093292049093423)
| > loss: 1.0256420373916626 (0.9658766194106707)
| > current_lr: 5e-06
| > step_time: 0.303 (0.3747676720626393)
| > loader_time: 0.0105 (0.009829002388042678)
 --> TIME: 2024-10-20 00:37:48 -- STEP: 2039/3361 -- GLOBAL_STEP: 5400
| > loss_text_ce: 0.05259113013744354 (0.054126247990183986)
| > loss_mel_ce: 3.6885979175567627 (3.8070934112309356)
| > loss: 0.9352972507476807 (0.9653049167492274)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4207 (0.3747711464609217)
| > loader_time: 0.0111 (0.009824151090104651)
 --> TIME: 2024-10-20 00:38:19 -- STEP: 2089/3361 -- GLOBAL_STEP: 5450
| > loss_text_ce: 0.04826216772198677 (0.054073872357649676)
| > loss_mel_ce: 3.61075496673584 (3.8050423009707175)
| > loss: 0.9147542715072632 (0.9647790451492629)
| > current_lr: 5e-06
| > step_time: 0.4413 (0.37473718774210263)
| > loader_time: 0.0107 (0.009820947811442618)
 --> TIME: 2024-10-20 00:38:52 -- STEP: 2139/3361 -- GLOBAL_STEP: 5500
| > loss_text_ce: 0.04554080218076706 (0.05400996907235807)
| > loss_mel_ce: 3.5199358463287354 (3.802293091301831)
| > loss: 0.8913691639900208 (0.9640757669461782)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4984 (0.37496582555570235)
| > loader_time: 0.0104 (0.009825523125896836)
 --> TIME: 2024-10-20 00:39:25 -- STEP: 2189/3361 -- GLOBAL_STEP: 5550
| > loss_text_ce: 0.05215875059366226 (0.053962629445002665)
| > loss_mel_ce: 3.4752392768859863 (3.8001412673321093)
| > loss: 0.8818495273590088 (0.963525975986723)
| > current_lr: 5e-06
| > step_time: 0.3729 (0.37516021565140617)
| > loader_time: 0.0089 (0.009817448964996825)
 --> TIME: 2024-10-20 00:39:56 -- STEP: 2239/3361 -- GLOBAL_STEP: 5600
| > loss_text_ce: 0.04801148176193237 (0.05391795849057377)
| > loss_mel_ce: 3.7264914512634277 (3.797605789895034)
| > loss: 0.9436257481575012 (0.9628809388858391)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4895 (0.3751126565588217)
| > loader_time: 0.009 (0.009810711454738186)
 --> TIME: 2024-10-20 00:40:29 -- STEP: 2289/3361 -- GLOBAL_STEP: 5650
| > loss_text_ce: 0.05482006445527077 (0.05388443434815208)
| > loss_mel_ce: 3.9797515869140625 (3.795354474939702)
| > loss: 1.0086429119110107 (0.9623097291207303)
| > current_lr: 5e-06
| > step_time: 0.414 (0.3753624505067611)
| > loader_time: 0.0091 (0.009804255996744091)
 --> TIME: 2024-10-20 00:41:00 -- STEP: 2339/3361 -- GLOBAL_STEP: 5700
| > loss_text_ce: 0.053768549114465714 (0.053837192649767884)
| > loss_mel_ce: 3.6103670597076416 (3.792830330329657)
| > loss: 0.916033923625946 (0.9616668826497075)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4228 (0.37508875817098697)
| > loader_time: 0.0099 (0.009796541029696479)
 --> TIME: 2024-10-20 00:41:31 -- STEP: 2389/3361 -- GLOBAL_STEP: 5750
| > loss_text_ce: 0.05098041519522667 (0.053785190121892486)
| > loss_mel_ce: 3.7739956378936768 (3.790576577735476)
| > loss: 0.9562439918518066 (0.9610904438967682)
| > current_lr: 5e-06
| > step_time: 0.4136 (0.37480044953740266)
| > loader_time: 0.009 (0.009783994807214997)
 --> TIME: 2024-10-20 00:42:03 -- STEP: 2439/3361 -- GLOBAL_STEP: 5800
| > loss_text_ce: 0.051108840852975845 (0.053738851268218625)
| > loss_mel_ce: 3.423971652984619 (3.787982714083313)
| > loss: 0.8687701225280762 (0.9604303932375672)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.425 (0.3746569493893182)
| > loader_time: 0.0098 (0.009782257604032028)
 --> TIME: 2024-10-20 00:42:35 -- STEP: 2489/3361 -- GLOBAL_STEP: 5850
| > loss_text_ce: 0.047307513654232025 (0.053671666948957655)
| > loss_mel_ce: 3.593897581100464 (3.786233845845242)
| > loss: 0.9103012681007385 (0.9599763800600726)
| > current_lr: 5e-06
| > step_time: 0.3052 (0.37466022804493243)
| > loader_time: 0.0105 (0.009783980261134827)
 --> TIME: 2024-10-20 00:43:06 -- STEP: 2539/3361 -- GLOBAL_STEP: 5900
| > loss_text_ce: 0.04697243124246597 (0.053605987948254344)
| > loss_mel_ce: 3.405848741531372 (3.783619592864948)
| > loss: 0.8632053136825562 (0.9593063970164268)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3768 (0.374510133768828)
| > loader_time: 0.0096 (0.009779982042857256)
 --> TIME: 2024-10-20 00:43:38 -- STEP: 2589/3361 -- GLOBAL_STEP: 5950
| > loss_text_ce: 0.04804262891411781 (0.05354277639660238)
| > loss_mel_ce: 3.705813407897949 (3.7827198630642735)
| > loss: 0.9384639859199524 (0.9590656616688695)
| > current_lr: 5e-06
| > step_time: 0.3419 (0.37453008278220223)
| > loader_time: 0.0092 (0.009780242480841231)
 --> TIME: 2024-10-20 00:44:09 -- STEP: 2639/3361 -- GLOBAL_STEP: 6000
| > loss_text_ce: 0.04612743854522705 (0.05346968814211162)
| > loss_mel_ce: 3.9188263416290283 (3.7808196491344086)
| > loss: 0.9912384748458862 (0.9585723361990677)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4981 (0.3743246874605093)
| > loader_time: 0.0112 (0.009783610018332285)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_6000.pth
 --> TIME: 2024-10-20 00:44:52 -- STEP: 2689/3361 -- GLOBAL_STEP: 6050
| > loss_text_ce: 0.047469258308410645 (0.05341444585531915)
| > loss_mel_ce: 3.861588954925537 (3.778166536329049)
| > loss: 0.9772645235061646 (0.957895247454801)
| > current_lr: 5e-06
| > step_time: 0.3318 (0.37431536185426717)
| > loader_time: 0.0103 (0.00978775129038058)
 --> TIME: 2024-10-20 00:45:23 -- STEP: 2739/3361 -- GLOBAL_STEP: 6100
| > loss_text_ce: 0.04789832606911659 (0.053361904739724214)
| > loss_mel_ce: 3.5844626426696777 (3.77603444142601)
| > loss: 0.9080902338027954 (0.9573490884278804)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3335 (0.3741204217212319)
| > loader_time: 0.0099 (0.009783599010215445)
 --> TIME: 2024-10-20 00:45:55 -- STEP: 2789/3361 -- GLOBAL_STEP: 6150
| > loss_text_ce: 0.052916835993528366 (0.05331185084222335)
| > loss_mel_ce: 3.5830535888671875 (3.7740368574654686)
| > loss: 0.90899258852005 (0.9568371789435752)
| > current_lr: 5e-06
| > step_time: 0.3046 (0.3741858853538141)
| > loader_time: 0.0084 (0.009779214517067484)
 --> TIME: 2024-10-20 00:46:27 -- STEP: 2839/3361 -- GLOBAL_STEP: 6200
| > loss_text_ce: 0.05071615055203438 (0.053262250472349594)
| > loss_mel_ce: 3.695129156112671 (3.771301807085785)
| > loss: 0.936461329460144 (0.9561410162095328)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.5436 (0.3742936511575533)
| > loader_time: 0.0091 (0.00977529728651971)
 --> TIME: 2024-10-20 00:47:00 -- STEP: 2889/3361 -- GLOBAL_STEP: 6250
| > loss_text_ce: 0.05398828536272049 (0.05322098417878768)
| > loss_mel_ce: 3.478079319000244 (3.7687003193284534)
| > loss: 0.8830168843269348 (0.9554803276111528)
| > current_lr: 5e-06
| > step_time: 0.2948 (0.37435020431592575)
| > loader_time: 0.0097 (0.009775939223757246)
 --> TIME: 2024-10-20 00:47:31 -- STEP: 2939/3361 -- GLOBAL_STEP: 6300
| > loss_text_ce: 0.05640152841806412 (0.05317959931304194)
| > loss_mel_ce: 3.542383909225464 (3.766597809131229)
| > loss: 0.8996963500976562 (0.9549443538434735)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3899 (0.3743035755176948)
| > loader_time: 0.0096 (0.009772745270678769)
 --> TIME: 2024-10-20 00:48:03 -- STEP: 2989/3361 -- GLOBAL_STEP: 6350
| > loss_text_ce: 0.05089443922042847 (0.053131582746989495)
| > loss_mel_ce: 3.53525710105896 (3.764354299725239)
| > loss: 0.8965378999710083 (0.9543714722529347)
| > current_lr: 5e-06
| > step_time: 0.3915 (0.37433643443164155)
| > loader_time: 0.0088 (0.009772129464045778)
 --> TIME: 2024-10-20 00:48:36 -- STEP: 3039/3361 -- GLOBAL_STEP: 6400
| > loss_text_ce: 0.046930961310863495 (0.05308791380418526)
| > loss_mel_ce: 3.706780433654785 (3.7627756988660956)
| > loss: 0.9384278655052185 (0.9539659047213068)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4973 (0.3744112501492748)
| > loader_time: 0.0094 (0.009775070659264634)
 --> TIME: 2024-10-20 00:49:07 -- STEP: 3089/3361 -- GLOBAL_STEP: 6450
| > loss_text_ce: 0.04575106129050255 (0.053032684769733454)
| > loss_mel_ce: 3.75093936920166 (3.7611461492293317)
| > loss: 0.9491726160049438 (0.9535447100328759)
| > current_lr: 5e-06
| > step_time: 0.4194 (0.37436858069437534)
| > loader_time: 0.0091 (0.009773730922957073)
 --> TIME: 2024-10-20 00:49:40 -- STEP: 3139/3361 -- GLOBAL_STEP: 6500
| > loss_text_ce: 0.05457288771867752 (0.05298690562673189)
| > loss_mel_ce: 3.7172043323516846 (3.759212153092354)
| > loss: 0.9429442882537842 (0.9530497661970648)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.5654 (0.3745672619240274)
| > loader_time: 0.0095 (0.009773702445247437)
 --> TIME: 2024-10-20 00:50:12 -- STEP: 3189/3361 -- GLOBAL_STEP: 6550
| > loss_text_ce: 0.04842592775821686 (0.05293102836713553)
| > loss_mel_ce: 3.452249526977539 (3.7573863935156577)
| > loss: 0.8751688599586487 (0.9525793569519364)
| > current_lr: 5e-06
| > step_time: 0.2971 (0.3745487653637209)
| > loader_time: 0.01 (0.009774107363351874)
 --> TIME: 2024-10-20 00:50:43 -- STEP: 3239/3361 -- GLOBAL_STEP: 6600
| > loss_text_ce: 0.05154899135231972 (0.05288972797383492)
| > loss_mel_ce: 3.525009870529175 (3.755932199914502)
| > loss: 0.8941397070884705 (0.9522054834879633)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.481 (0.3743514210105354)
| > loader_time: 0.0099 (0.009773110636740263)
 --> TIME: 2024-10-20 00:51:16 -- STEP: 3289/3361 -- GLOBAL_STEP: 6650
| > loss_text_ce: 0.05035821348428726 (0.052853074836564214)
| > loss_mel_ce: 3.56316876411438 (3.754109559721699)
| > loss: 0.9033817648887634 (0.951740660178839)
| > current_lr: 5e-06
| > step_time: 0.4077 (0.3745306703811657)
| > loader_time: 0.0105 (0.009773014961096529)
 --> TIME: 2024-10-20 00:51:48 -- STEP: 3339/3361 -- GLOBAL_STEP: 6700
| > loss_text_ce: 0.047619469463825226 (0.05279993967413365)
| > loss_mel_ce: 3.5058529376983643 (3.7519333840987397)
| > loss: 0.8883681297302246 (0.951183332533635)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4219 (0.37463738312082967)
| > loader_time: 0.0108 (0.009774323100992282)
 > EVALUATION 
--> EVAL PERFORMANCE
| > avg_loader_time: 0.3849575237561297 (+0.0032363040472870863)
| > avg_loss_text_ce: 0.04702420168185747 (-0.007846372021782781)
| > avg_loss_mel_ce: 3.4355908209277737 (-0.3100963510492796)
| > avg_loss: 3.482615009430916 (-0.3179427321239183)
 > EPOCH: 2/5
--> /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b
 > TRAINING (2024-10-20 00:53:55) 
 --> TIME: 2024-10-20 00:54:15 -- STEP: 28/3361 -- GLOBAL_STEP: 6750
| > loss_text_ce: 0.04403690621256828 (0.04905040069882359)
| > loss_mel_ce: 3.7761855125427246 (3.686231689793723)
| > loss: 0.9550555944442749 (0.9338205137423107)
| > current_lr: 5e-06
| > step_time: 0.2669 (0.3828866481781006)
| > loader_time: 0.0095 (0.010155720370156425)
 --> TIME: 2024-10-20 00:54:46 -- STEP: 78/3361 -- GLOBAL_STEP: 6800
| > loss_text_ce: 0.047489896416664124 (0.04935918151377103)
| > loss_mel_ce: 3.2789604663848877 (3.62562545751914)
| > loss: 0.8316125869750977 (0.9187461550419147)
| > current_lr: 5e-06
| > step_time: 0.4336 (0.3764091027088655)
| > loader_time: 0.0105 (0.010129986665187739)
 --> TIME: 2024-10-20 00:55:18 -- STEP: 128/3361 -- GLOBAL_STEP: 6850
| > loss_text_ce: 0.053600654006004333 (0.04948139251791872)
| > loss_mel_ce: 3.313396692276001 (3.610981924459338)
| > loss: 0.8417493104934692 (0.9151158249005675)
| > current_lr: 5e-06
| > step_time: 0.2737 (0.37236367911100393)
| > loader_time: 0.0109 (0.010153220966458322)
 --> TIME: 2024-10-20 00:55:50 -- STEP: 178/3361 -- GLOBAL_STEP: 6900
| > loss_text_ce: 0.04995616152882576 (0.04949212647723348)
| > loss_mel_ce: 3.5250840187072754 (3.6204694884546686)
| > loss: 0.8937600255012512 (0.9174904006250789)
| > current_lr: 5e-06
| > step_time: 0.4846 (0.3750775664040211)
| > loader_time: 0.0092 (0.010136076573575476)
 --> TIME: 2024-10-20 00:56:24 -- STEP: 228/3361 -- GLOBAL_STEP: 6950
| > loss_text_ce: 0.047617699950933456 (0.04926429808205157)
| > loss_mel_ce: 3.59977650642395 (3.6181379849450632)
| > loss: 0.9118485450744629 (0.9168505681711331)
| > current_lr: 5e-06
| > step_time: 0.3983 (0.3783655584904185)
| > loader_time: 0.01 (0.010034571614181783)
 --> TIME: 2024-10-20 00:56:55 -- STEP: 278/3361 -- GLOBAL_STEP: 7000
| > loss_text_ce: 0.04799577221274376 (0.049197411218242684)
| > loss_mel_ce: 3.7218735218048096 (3.6140262754701027)
| > loss: 0.9424673318862915 (0.9158059191360749)
| > current_lr: 5e-06
| > step_time: 0.3743 (0.3763897959276927)
| > loader_time: 0.009 (0.009955753525384033)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_7000.pth
 --> TIME: 2024-10-20 00:57:35 -- STEP: 328/3361 -- GLOBAL_STEP: 7050
| > loss_text_ce: 0.04574914276599884 (0.049158351098346285)
| > loss_mel_ce: 3.567535877227783 (3.613539861469734)
| > loss: 0.9033212661743164 (0.9156745514128266)
| > current_lr: 5e-06
| > step_time: 0.2103 (0.3746326173224102)
| > loader_time: 0.0092 (0.00990271422921157)
 --> TIME: 2024-10-20 00:58:08 -- STEP: 378/3361 -- GLOBAL_STEP: 7100
| > loss_text_ce: 0.048721931874752045 (0.049147929641462515)
| > loss_mel_ce: 3.5802829265594482 (3.6114332770544384)
| > loss: 0.907251238822937 (0.9151453005258369)
| > current_lr: 5e-06
| > step_time: 0.4294 (0.376694580234548)
| > loader_time: 0.0108 (0.009864083042851196)
 --> TIME: 2024-10-20 00:58:40 -- STEP: 428/3361 -- GLOBAL_STEP: 7150
| > loss_text_ce: 0.051922909915447235 (0.04917553177747493)
| > loss_mel_ce: 3.5437474250793457 (3.607163674920519)
| > loss: 0.898917555809021 (0.9140848006322005)
| > current_lr: 5e-06
| > step_time: 0.2716 (0.37683266401290916)
| > loader_time: 0.0094 (0.009827861719042336)
 --> TIME: 2024-10-20 00:59:12 -- STEP: 478/3361 -- GLOBAL_STEP: 7200
| > loss_text_ce: 0.045843224972486496 (0.04910156095719986)
| > loss_mel_ce: 3.5055999755859375 (3.6058153352976845)
| > loss: 0.8878607749938965 (0.9137292228732647)
| > current_lr: 5e-06
| > step_time: 0.3413 (0.3767390929505418)
| > loader_time: 0.0091 (0.00982440415785402)
 --> TIME: 2024-10-20 00:59:44 -- STEP: 528/3361 -- GLOBAL_STEP: 7250
| > loss_text_ce: 0.048054084181785583 (0.04909205055005397)
| > loss_mel_ce: 3.536403179168701 (3.60500498928807)
| > loss: 0.8961142897605896 (0.9135242587689197)
| > current_lr: 5e-06
| > step_time: 0.2992 (0.3766030235724017)
| > loader_time: 0.0095 (0.009845270351930094)
 --> TIME: 2024-10-20 01:00:17 -- STEP: 578/3361 -- GLOBAL_STEP: 7300
| > loss_text_ce: 0.05416757985949516 (0.0490665393446453)
| > loss_mel_ce: 3.5705678462982178 (3.603965314201831)
| > loss: 0.9061838388442993 (0.9132579624859107)
| > current_lr: 5e-06
| > step_time: 0.3706 (0.37739562947032373)
| > loader_time: 0.009 (0.009825011025663062)
 --> TIME: 2024-10-20 01:00:48 -- STEP: 628/3361 -- GLOBAL_STEP: 7350
| > loss_text_ce: 0.047263436019420624 (0.049029604579280496)
| > loss_mel_ce: 3.5175342559814453 (3.603023174462046)
| > loss: 0.8911994099617004 (0.9130131940173495)
| > current_lr: 5e-06
| > step_time: 0.3131 (0.3764249720391194)
| > loader_time: 0.009 (0.009792880267853939)
 --> TIME: 2024-10-20 01:01:21 -- STEP: 678/3361 -- GLOBAL_STEP: 7400
| > loss_text_ce: 0.049490731209516525 (0.0490141717777491)
| > loss_mel_ce: 3.326827049255371 (3.6022712867871856)
| > loss: 0.8440794348716736 (0.9128213640862861)
| > current_lr: 5e-06
| > step_time: 0.4932 (0.37664768055828635)
| > loader_time: 0.0095 (0.009787028869696412)
 --> TIME: 2024-10-20 01:01:53 -- STEP: 728/3361 -- GLOBAL_STEP: 7450
| > loss_text_ce: 0.04762418195605278 (0.04900444910994595)
| > loss_mel_ce: 3.558835744857788 (3.602000492614705)
| > loss: 0.9016149640083313 (0.9127512351497189)
| > current_lr: 5e-06
| > step_time: 0.314 (0.37692653346847715)
| > loader_time: 0.0098 (0.009809368586802208)
 --> TIME: 2024-10-20 01:02:25 -- STEP: 778/3361 -- GLOBAL_STEP: 7500
| > loss_text_ce: 0.04937560111284256 (0.04896448014883422)
| > loss_mel_ce: 3.9129478931427 (3.6037924369380536)
| > loss: 0.9905808568000793 (0.9131892292082769)
| > current_lr: 5e-06
| > step_time: 0.3562 (0.37699541150757476)
| > loader_time: 0.009 (0.009813950423409816)
 --> TIME: 2024-10-20 01:02:57 -- STEP: 828/3361 -- GLOBAL_STEP: 7550
| > loss_text_ce: 0.04987458512187004 (0.04891771870408799)
| > loss_mel_ce: 3.4430720806121826 (3.6009757144439627)
| > loss: 0.8732366561889648 (0.9124733581059221)
| > current_lr: 5e-06
| > step_time: 0.3872 (0.3765455469993003)
| > loader_time: 0.0099 (0.009804343255821633)
 --> TIME: 2024-10-20 01:03:28 -- STEP: 878/3361 -- GLOBAL_STEP: 7600
| > loss_text_ce: 0.04912569746375084 (0.04892470960023732)
| > loss_mel_ce: 3.4806666374206543 (3.597835117429156)
| > loss: 0.8824480772018433 (0.9116899567340121)
| > current_lr: 5e-06
| > step_time: 0.3901 (0.3757378245812072)
| > loader_time: 0.0088 (0.009797889442270058)
 --> TIME: 2024-10-20 01:04:01 -- STEP: 928/3361 -- GLOBAL_STEP: 7650
| > loss_text_ce: 0.04729423299431801 (0.04890506303917357)
| > loss_mel_ce: 3.8602240085601807 (3.5957908889856838)
| > loss: 0.9768795371055603 (0.9111739878767523)
| > current_lr: 5e-06
| > step_time: 0.3303 (0.3764181083132482)
| > loader_time: 0.0087 (0.009782458687650746)
 --> TIME: 2024-10-20 01:04:33 -- STEP: 978/3361 -- GLOBAL_STEP: 7700
| > loss_text_ce: 0.0476219542324543 (0.04887541249424887)
| > loss_mel_ce: 3.6902410984039307 (3.5933213326584834)
| > loss: 0.934465765953064 (0.9105491861853375)
| > current_lr: 5e-06
| > step_time: 0.4139 (0.3762613827465503)
| > loader_time: 0.0095 (0.009784645341900473)
 --> TIME: 2024-10-20 01:05:06 -- STEP: 1028/3361 -- GLOBAL_STEP: 7750
| > loss_text_ce: 0.050320666283369064 (0.048847640831936635)
| > loss_mel_ce: 3.489208698272705 (3.5914727294027577)
| > loss: 0.8848823308944702 (0.9100800926583287)
| > current_lr: 5e-06
| > step_time: 0.3856 (0.37657562837526504)
| > loader_time: 0.0097 (0.009803709816839909)
 --> TIME: 2024-10-20 01:05:38 -- STEP: 1078/3361 -- GLOBAL_STEP: 7800
| > loss_text_ce: 0.04912106320261955 (0.048804982256054424)
| > loss_mel_ce: 3.4618194103240967 (3.587999959823595)
| > loss: 0.8777351379394531 (0.909201235629632)
| > current_lr: 5e-06
| > step_time: 0.3768 (0.3767289274919898)
| > loader_time: 0.0094 (0.009814880111000755)
 --> TIME: 2024-10-20 01:06:10 -- STEP: 1128/3361 -- GLOBAL_STEP: 7850
| > loss_text_ce: 0.05011963099241257 (0.048788163781245335)
| > loss_mel_ce: 3.5565550327301025 (3.587737088719159)
| > loss: 0.9016686677932739 (0.909131313205188)
| > current_lr: 5e-06
| > step_time: 0.3358 (0.37648225506992233)
| > loader_time: 0.0089 (0.009822922395476214)
 --> TIME: 2024-10-20 01:06:43 -- STEP: 1178/3361 -- GLOBAL_STEP: 7900
| > loss_text_ce: 0.04669732600450516 (0.048759036461849564)
| > loss_mel_ce: 3.1603102684020996 (3.5851203856929654)
| > loss: 0.8017519116401672 (0.9084698557853699)
| > current_lr: 5e-06
| > step_time: 0.3734 (0.37704856853128915)
| > loader_time: 0.0087 (0.009813111585348294)
 --> TIME: 2024-10-20 01:07:14 -- STEP: 1228/3361 -- GLOBAL_STEP: 7950
| > loss_text_ce: 0.04521196335554123 (0.048707531449606434)
| > loss_mel_ce: 3.4588823318481445 (3.585536780885454)
| > loss: 0.8760235905647278 (0.9085610783158373)
| > current_lr: 5e-06
| > step_time: 0.3439 (0.3768606422778451)
| > loader_time: 0.0086 (0.009805944532835524)
 --> TIME: 2024-10-20 01:07:48 -- STEP: 1278/3361 -- GLOBAL_STEP: 8000
| > loss_text_ce: 0.04893951490521431 (0.048680408736792906)
| > loss_mel_ce: 3.475900173187256 (3.584775358298575)
| > loss: 0.8812099099159241 (0.9083639419694461)
| > current_lr: 5e-06
| > step_time: 0.4076 (0.37746544193960335)
| > loader_time: 0.0092 (0.00980452752449143)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_8000.pth
 --> TIME: 2024-10-20 01:08:32 -- STEP: 1328/3361 -- GLOBAL_STEP: 8050
| > loss_text_ce: 0.04482944682240486 (0.04863053328548394)
| > loss_mel_ce: 3.514838695526123 (3.5835139115890824)
| > loss: 0.8899170160293579 (0.9080361113939659)
| > current_lr: 5e-06
| > step_time: 0.3505 (0.3775392822831512)
| > loader_time: 0.0095 (0.00981970687946642)
 --> TIME: 2024-10-20 01:09:05 -- STEP: 1378/3361 -- GLOBAL_STEP: 8100
| > loss_text_ce: 0.045615728944540024 (0.048578062096099765)
| > loss_mel_ce: 3.5625221729278564 (3.583668395638985)
| > loss: 0.9020344614982605 (0.9080616145763757)
| > current_lr: 5e-06
| > step_time: 0.3422 (0.37772889061140236)
| > loader_time: 0.01 (0.00982503915213709)
 --> TIME: 2024-10-20 01:09:37 -- STEP: 1428/3361 -- GLOBAL_STEP: 8150
| > loss_text_ce: 0.047519903630018234 (0.048547342737891463)
| > loss_mel_ce: 3.6124684810638428 (3.581988831361135)
| > loss: 0.9149971008300781 (0.9076340436851945)
| > current_lr: 5e-06
| > step_time: 0.2656 (0.3779737376699262)
| > loader_time: 0.0095 (0.009825288247661443)
 --> TIME: 2024-10-20 01:10:10 -- STEP: 1478/3361 -- GLOBAL_STEP: 8200
| > loss_text_ce: 0.04936197027564049 (0.04851635845430975)
| > loss_mel_ce: 3.5139453411102295 (3.580858142998609)
| > loss: 0.8908268213272095 (0.9073436254980761)
| > current_lr: 5e-06
| > step_time: 0.4088 (0.37810928605406163)
| > loader_time: 0.0097 (0.00982494218746284)
 --> TIME: 2024-10-20 01:10:42 -- STEP: 1528/3361 -- GLOBAL_STEP: 8250
| > loss_text_ce: 0.04918792098760605 (0.04848537729146128)
| > loss_mel_ce: 3.376131772994995 (3.5808331872780284)
| > loss: 0.8563299179077148 (0.9073296412435502)
| > current_lr: 5e-06
| > step_time: 0.3339 (0.3781200748775643)
| > loader_time: 0.0095 (0.009824010402120225)
 --> TIME: 2024-10-20 01:11:14 -- STEP: 1578/3361 -- GLOBAL_STEP: 8300
| > loss_text_ce: 0.04867768660187721 (0.048471816391678246)
| > loss_mel_ce: 3.5776634216308594 (3.578830152561154)
| > loss: 0.9065852761268616 (0.906825492250602)
| > current_lr: 5e-06
| > step_time: 0.3276 (0.37784483858841017)
| > loader_time: 0.0087 (0.00981416539094295)
 --> TIME: 2024-10-20 01:11:46 -- STEP: 1628/3361 -- GLOBAL_STEP: 8350
| > loss_text_ce: 0.04532700031995773 (0.048449824335332764)
| > loss_mel_ce: 3.543966293334961 (3.5778604450038376)
| > loss: 0.8973233103752136 (0.9065775673571032)
| > current_lr: 5e-06
| > step_time: 0.3354 (0.3776157829333876)
| > loader_time: 0.0099 (0.009799206579053737)
 --> TIME: 2024-10-20 01:12:17 -- STEP: 1678/3361 -- GLOBAL_STEP: 8400
| > loss_text_ce: 0.04563120752573013 (0.04841454139953933)
| > loss_mel_ce: 3.4720375537872314 (3.5776013940110962)
| > loss: 0.8794171810150146 (0.9065039838454436)
| > current_lr: 5e-06
| > step_time: 0.3709 (0.37741188909214657)
| > loader_time: 0.0088 (0.009793770185386479)
 --> TIME: 2024-10-20 01:12:49 -- STEP: 1728/3361 -- GLOBAL_STEP: 8450
| > loss_text_ce: 0.05147872492671013 (0.048384104855358635)
| > loss_mel_ce: 3.6369025707244873 (3.576328690405245)
| > loss: 0.9220952987670898 (0.906178198720294)
| > current_lr: 5e-06
| > step_time: 0.3306 (0.37728250123284496)
| > loader_time: 0.0102 (0.009796891775396147)
 --> TIME: 2024-10-20 01:13:21 -- STEP: 1778/3361 -- GLOBAL_STEP: 8500
| > loss_text_ce: 0.04927783086895943 (0.04836768526437248)
| > loss_mel_ce: 3.509384870529175 (3.5736189018620683)
| > loss: 0.8896656632423401 (0.90549664667004)
| > current_lr: 5e-06
| > step_time: 0.3454 (0.3771043242163815)
| > loader_time: 0.0096 (0.009791791103822186)
 --> TIME: 2024-10-20 01:13:53 -- STEP: 1828/3361 -- GLOBAL_STEP: 8550
| > loss_text_ce: 0.04953066632151604 (0.048366361587049565)
| > loss_mel_ce: 3.6733510494232178 (3.5723333584401487)
| > loss: 0.9307204484939575 (0.9051749299339444)
| > current_lr: 5e-06
| > step_time: 0.2723 (0.37695589094245474)
| > loader_time: 0.0114 (0.009782656668573303)
 --> TIME: 2024-10-20 01:14:25 -- STEP: 1878/3361 -- GLOBAL_STEP: 8600
| > loss_text_ce: 0.04815003275871277 (0.04833432593451334)
| > loss_mel_ce: 3.39089298248291 (3.5706544065881713)
| > loss: 0.8597607612609863 (0.9047471829814017)
| > current_lr: 5e-06
| > step_time: 0.427 (0.37701883737525504)
| > loader_time: 0.0093 (0.009764435700080403)
 --> TIME: 2024-10-20 01:14:58 -- STEP: 1928/3361 -- GLOBAL_STEP: 8650
| > loss_text_ce: 0.04663149267435074 (0.048306525483943895)
| > loss_mel_ce: 3.3355255126953125 (3.5698014803199847)
| > loss: 0.8455392718315125 (0.9045270012374735)
| > current_lr: 5e-06
| > step_time: 0.3483 (0.37734258817933913)
| > loader_time: 0.0099 (0.009774215364851903)
 --> TIME: 2024-10-20 01:15:30 -- STEP: 1978/3361 -- GLOBAL_STEP: 8700
| > loss_text_ce: 0.04672824218869209 (0.048289880877844756)
| > loss_mel_ce: 3.442249059677124 (3.5682690238807995)
| > loss: 0.8722442984580994 (0.9041397260281627)
| > current_lr: 5e-06
| > step_time: 0.4095 (0.37710894805476214)
| > loader_time: 0.0092 (0.009775729584380529)
 --> TIME: 2024-10-20 01:16:02 -- STEP: 2028/3361 -- GLOBAL_STEP: 8750
| > loss_text_ce: 0.04540479928255081 (0.04825842313292114)
| > loss_mel_ce: 3.524287462234497 (3.567732476388679)
| > loss: 0.8924230933189392 (0.9039977247251797)
| > current_lr: 5e-06
| > step_time: 0.2754 (0.3770918593369058)
| > loader_time: 0.0109 (0.009774529369625124)
 --> TIME: 2024-10-20 01:16:33 -- STEP: 2078/3361 -- GLOBAL_STEP: 8800
| > loss_text_ce: 0.043983638286590576 (0.048219160568524856)
| > loss_mel_ce: 3.5472054481506348 (3.566295558724757)
| > loss: 0.8977972865104675 (0.9036286797686421)
| > current_lr: 5e-06
| > step_time: 0.4283 (0.37684804103601666)
| > loader_time: 0.0101 (0.009771026362583441)
 --> TIME: 2024-10-20 01:17:06 -- STEP: 2128/3361 -- GLOBAL_STEP: 8850
| > loss_text_ce: 0.04918619617819786 (0.04819233537322487)
| > loss_mel_ce: 3.391493797302246 (3.565849136701204)
| > loss: 0.8601700067520142 (0.9035103678759328)
| > current_lr: 5e-06
| > step_time: 0.2636 (0.3770903514740162)
| > loader_time: 0.0094 (0.009773158937468576)
 --> TIME: 2024-10-20 01:17:39 -- STEP: 2178/3361 -- GLOBAL_STEP: 8900
| > loss_text_ce: 0.0547124482691288 (0.04816222397475425)
| > loss_mel_ce: 3.6788527965545654 (3.5645782656358724)
| > loss: 0.9333913326263428 (0.903185122296611)
| > current_lr: 5e-06
| > step_time: 0.3439 (0.3773379673970962)
| > loader_time: 0.0105 (0.009771051595143167)
 --> TIME: 2024-10-20 01:18:11 -- STEP: 2228/3361 -- GLOBAL_STEP: 8950
| > loss_text_ce: 0.04527541995048523 (0.04813232319847527)
| > loss_mel_ce: 3.706040382385254 (3.5633151346091956)
| > loss: 0.9378289580345154 (0.9028618643110488)
| > current_lr: 5e-06
| > step_time: 0.2648 (0.3773191904037175)
| > loader_time: 0.0104 (0.009769387797350522)
 --> TIME: 2024-10-20 01:18:43 -- STEP: 2278/3361 -- GLOBAL_STEP: 9000
| > loss_text_ce: 0.042789604514837265 (0.048098762065852514)
| > loss_mel_ce: 3.569603681564331 (3.561746247511773)
| > loss: 0.9030983448028564 (0.9024612522627616)
| > current_lr: 5e-06
| > step_time: 0.4348 (0.37725567137389143)
| > loader_time: 0.0097 (0.009770785747022363)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_9000.pth
 --> TIME: 2024-10-20 01:19:25 -- STEP: 2328/3361 -- GLOBAL_STEP: 9050
| > loss_text_ce: 0.04657544568181038 (0.04807176884372568)
| > loss_mel_ce: 3.4539883136749268 (3.5602467105151043)
| > loss: 0.8751409649848938 (0.9020796196976886)
| > current_lr: 5e-06
| > step_time: 0.3703 (0.3772568883969612)
| > loader_time: 0.011 (0.009778630999765047)
 --> TIME: 2024-10-20 01:19:57 -- STEP: 2378/3361 -- GLOBAL_STEP: 9100
| > loss_text_ce: 0.04268953576683998 (0.04804769095544109)
| > loss_mel_ce: 3.6826560497283936 (3.558529170748564)
| > loss: 0.9313364028930664 (0.9016442152509374)
| > current_lr: 5e-06
| > step_time: 0.3398 (0.37726983671132214)
| > loader_time: 0.0108 (0.009784870432443842)
 --> TIME: 2024-10-20 01:20:29 -- STEP: 2428/3361 -- GLOBAL_STEP: 9150
| > loss_text_ce: 0.04784495756030083 (0.048017254220459896)
| > loss_mel_ce: 3.2817625999450684 (3.55805171125214)
| > loss: 0.8324018716812134 (0.9015172412358161)
| > current_lr: 5e-06
| > step_time: 0.3146 (0.3770558785960152)
| > loader_time: 0.0111 (0.009788657825507693)
 --> TIME: 2024-10-20 01:21:00 -- STEP: 2478/3361 -- GLOBAL_STEP: 9200
| > loss_text_ce: 0.04175335913896561 (0.047998460361476994)
| > loss_mel_ce: 3.682319402694702 (3.556396670549314)
| > loss: 0.9310181736946106 (0.901098782589014)
| > current_lr: 5e-06
| > step_time: 0.3292 (0.3768024727449577)
| > loader_time: 0.0102 (0.009790577457449549)
 --> TIME: 2024-10-20 01:21:32 -- STEP: 2528/3361 -- GLOBAL_STEP: 9250
| > loss_text_ce: 0.04678019881248474 (0.04796844289072239)
| > loss_mel_ce: 3.5981311798095703 (3.5554795194464393)
| > loss: 0.911227822303772 (0.9008619905083992)
| > current_lr: 5e-06
| > step_time: 0.272 (0.3767167896598198)
| > loader_time: 0.0098 (0.009790274727193635)
 --> TIME: 2024-10-20 01:22:03 -- STEP: 2578/3361 -- GLOBAL_STEP: 9300
| > loss_text_ce: 0.04527261480689049 (0.04794080875079894)
| > loss_mel_ce: 3.4495139122009277 (3.55347470269821)
| > loss: 0.8736966252326965 (0.9003538777654349)
| > current_lr: 5e-06
| > step_time: 0.4813 (0.37643767143054185)
| > loader_time: 0.0097 (0.009784305530337996)
 --> TIME: 2024-10-20 01:22:35 -- STEP: 2628/3361 -- GLOBAL_STEP: 9350
| > loss_text_ce: 0.04605727642774582 (0.047905134299381695)
| > loss_mel_ce: 3.8004796504974365 (3.552657171834368)
| > loss: 0.9616342186927795 (0.9001405764841778)
| > current_lr: 5e-06
| > step_time: 0.2983 (0.3764948804991794)
| > loader_time: 0.0107 (0.009790125412120867)
 --> TIME: 2024-10-20 01:23:07 -- STEP: 2678/3361 -- GLOBAL_STEP: 9400
| > loss_text_ce: 0.043804291635751724 (0.047882563374029685)
| > loss_mel_ce: 3.435140609741211 (3.551404430761188)
| > loss: 0.8697362542152405 (0.8998217484993751)
| > current_lr: 5e-06
| > step_time: 0.4414 (0.37655566075206093)
| > loader_time: 0.0095 (0.009788562505435746)
 --> TIME: 2024-10-20 01:23:39 -- STEP: 2728/3361 -- GLOBAL_STEP: 9450
| > loss_text_ce: 0.05015170946717262 (0.04786144649948595)
| > loss_mel_ce: 3.592085599899292 (3.550092155545338)
| > loss: 0.910559356212616 (0.8994884004166397)
| > current_lr: 5e-06
| > step_time: 0.2727 (0.37655943046567025)
| > loader_time: 0.009 (0.009790379805299211)
 --> TIME: 2024-10-20 01:24:11 -- STEP: 2778/3361 -- GLOBAL_STEP: 9500
| > loss_text_ce: 0.04628805071115494 (0.04783492111731691)
| > loss_mel_ce: 3.421419620513916 (3.549318913335505)
| > loss: 0.866926908493042 (0.8992884584988853)
| > current_lr: 5e-06
| > step_time: 0.3323 (0.3764905528768178)
| > loader_time: 0.0106 (0.009790914877444575)
 --> TIME: 2024-10-20 01:24:43 -- STEP: 2828/3361 -- GLOBAL_STEP: 9550
| > loss_text_ce: 0.04558764770627022 (0.04781904644721434)
| > loss_mel_ce: 3.2457244396209717 (3.5478047599549676)
| > loss: 0.8228279948234558 (0.8989059515139337)
| > current_lr: 5e-06
| > step_time: 0.315 (0.3763812427979379)
| > loader_time: 0.0111 (0.009782957642891115)
 --> TIME: 2024-10-20 01:25:14 -- STEP: 2878/3361 -- GLOBAL_STEP: 9600
| > loss_text_ce: 0.045083776116371155 (0.047794801320898854)
| > loss_mel_ce: 3.9052932262420654 (3.5468353375202257)
| > loss: 0.9875942468643188 (0.8986575346675657)
| > current_lr: 5e-06
| > step_time: 0.373 (0.3762246082854652)
| > loader_time: 0.0089 (0.009778903456841017)
 --> TIME: 2024-10-20 01:25:46 -- STEP: 2928/3361 -- GLOBAL_STEP: 9650
| > loss_text_ce: 0.04821404814720154 (0.04778221920845132)
| > loss_mel_ce: 3.5854885578155518 (3.546300680503819)
| > loss: 0.9084256291389465 (0.8985207248173774)
| > current_lr: 5e-06
| > step_time: 0.265 (0.3761550551229487)
| > loader_time: 0.0088 (0.00977481079231845)
 --> TIME: 2024-10-20 01:26:18 -- STEP: 2978/3361 -- GLOBAL_STEP: 9700
| > loss_text_ce: 0.046400487422943115 (0.047759666937591726)
| > loss_mel_ce: 3.357914447784424 (3.5448905813205633)
| > loss: 0.8510787487030029 (0.8981625620273233)
| > current_lr: 5e-06
| > step_time: 0.4432 (0.3761592442914093)
| > loader_time: 0.0088 (0.009770671900204147)
 --> TIME: 2024-10-20 01:26:51 -- STEP: 3028/3361 -- GLOBAL_STEP: 9750
| > loss_text_ce: 0.0519859679043293 (0.04774229847942948)
| > loss_mel_ce: 3.3926987648010254 (3.5437855810281307)
| > loss: 0.8611711859703064 (0.8978819698292165)
| > current_lr: 5e-06
| > step_time: 0.3159 (0.3763601943712726)
| > loader_time: 0.0095 (0.009767645337622564)
 --> TIME: 2024-10-20 01:27:23 -- STEP: 3078/3361 -- GLOBAL_STEP: 9800
| > loss_text_ce: 0.04087809845805168 (0.047722860169854234)
| > loss_mel_ce: 3.3796744346618652 (3.542765019774979)
| > loss: 0.8551381230354309 (0.8976219699435453)
| > current_lr: 5e-06
| > step_time: 0.342 (0.3762636706765555)
| > loader_time: 0.0088 (0.009768155195869497)
 --> TIME: 2024-10-20 01:27:55 -- STEP: 3128/3361 -- GLOBAL_STEP: 9850
| > loss_text_ce: 0.048090122640132904 (0.04769306221043179)
| > loss_mel_ce: 3.387538433074951 (3.541276809733237)
| > loss: 0.858907163143158 (0.8972424678980845)
| > current_lr: 5e-06
| > step_time: 0.2728 (0.37625613214109865)
| > loader_time: 0.0093 (0.0097666510840511)
 --> TIME: 2024-10-20 01:28:27 -- STEP: 3178/3361 -- GLOBAL_STEP: 9900
| > loss_text_ce: 0.0475609228014946 (0.047669609727266526)
| > loss_mel_ce: 3.4589006900787354 (3.5400787889769876)
| > loss: 0.8766154050827026 (0.8969370995661684)
| > current_lr: 5e-06
| > step_time: 0.3972 (0.37626130442862393)
| > loader_time: 0.009 (0.009765567608491803)
 --> TIME: 2024-10-20 01:28:59 -- STEP: 3228/3361 -- GLOBAL_STEP: 9950
| > loss_text_ce: 0.047529496252536774 (0.04764427237046481)
| > loss_mel_ce: 3.8835809230804443 (3.5399227575889456)
| > loss: 0.9827775955200195 (0.8968917573657915)
| > current_lr: 5e-06
| > step_time: 0.2965 (0.3762442364716323)
| > loader_time: 0.0105 (0.009769178086792505)
 --> TIME: 2024-10-20 01:29:31 -- STEP: 3278/3361 -- GLOBAL_STEP: 10000
| > loss_text_ce: 0.04803500697016716 (0.04761744798129844)
| > loss_mel_ce: 3.3600974082946777 (3.538903193767657)
| > loss: 0.8520330786705017 (0.896630160269612)
| > current_lr: 5e-06
| > step_time: 0.3416 (0.37622738066353834)
| > loader_time: 0.0094 (0.009767672941406864)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_10000.pth
 --> TIME: 2024-10-20 01:30:13 -- STEP: 3328/3361 -- GLOBAL_STEP: 10050
| > loss_text_ce: 0.04764594882726669 (0.047592870925795)
| > loss_mel_ce: 3.632420539855957 (3.537776723718987)
| > loss: 0.9200166463851929 (0.8963423984913297)
| > current_lr: 5e-06
| > step_time: 0.3302 (0.37621467653661966)
| > loader_time: 0.0095 (0.009771530373165221)
 > EVALUATION 
--> EVAL PERFORMANCE
| > avg_loader_time: 0.37726180527799874 (-0.007695718478130942)
| > avg_loss_text_ce: 0.04344167616418612 (-0.0035825255176713464)
| > avg_loss_mel_ce: 3.27118952556323 (-0.16440129536454373)
| > avg_loss: 3.3146311980421825 (-0.16798381138873353)
> BEST MODEL : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/best_model_10083.pth
 > EPOCH: 3/5
--> /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b
 > TRAINING (2024-10-20 01:32:59) 
 --> TIME: 2024-10-20 01:33:10 -- STEP: 17/3361 -- GLOBAL_STEP: 10100
| > loss_text_ce: 0.04386524111032486 (0.04697125514640527)
| > loss_mel_ce: 3.385413408279419 (3.4981692959280575)
| > loss: 0.8573196530342102 (0.8862851367277258)
| > current_lr: 5e-06
| > step_time: 0.3442 (0.3555046249838436)
| > loader_time: 0.0098 (0.010421248043284696)
 --> TIME: 2024-10-20 01:33:43 -- STEP: 67/3361 -- GLOBAL_STEP: 10150
| > loss_text_ce: 0.04214071109890938 (0.046040518014733474)
| > loss_mel_ce: 3.695300817489624 (3.470060277341017)
| > loss: 0.9343603849411011 (0.879025197740811)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4269 (0.374709196944735)
| > loader_time: 0.0102 (0.009930838399858614)
 --> TIME: 2024-10-20 01:34:14 -- STEP: 117/3361 -- GLOBAL_STEP: 10200
| > loss_text_ce: 0.04726540297269821 (0.04610895591541233)
| > loss_mel_ce: 3.4293832778930664 (3.4879054391486015)
| > loss: 0.8691621422767639 (0.8835035990446042)
| > current_lr: 5e-06
| > step_time: 0.3975 (0.37159106058952146)
| > loader_time: 0.0109 (0.009970159612150272)
 --> TIME: 2024-10-20 01:34:46 -- STEP: 167/3361 -- GLOBAL_STEP: 10250
| > loss_text_ce: 0.05034831911325455 (0.04611547081591842)
| > loss_mel_ce: 3.211472272872925 (3.478035532785747)
| > loss: 0.8154551386833191 (0.8810377520715406)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4432 (0.3729824891347372)
| > loader_time: 0.009 (0.009938856798731632)
 --> TIME: 2024-10-20 01:35:19 -- STEP: 217/3361 -- GLOBAL_STEP: 10300
| > loss_text_ce: 0.04303652048110962 (0.046182117206023054)
| > loss_mel_ce: 3.1857612133026123 (3.4779486150785526)
| > loss: 0.8071994185447693 (0.8810326841569716)
| > current_lr: 5e-06
| > step_time: 0.4433 (0.374492391463249)
| > loader_time: 0.0093 (0.009856836037701723)
 --> TIME: 2024-10-20 01:35:49 -- STEP: 267/3361 -- GLOBAL_STEP: 10350
| > loss_text_ce: 0.047204095870256424 (0.046089042898979096)
| > loss_mel_ce: 3.3309218883514404 (3.4724885845898688)
| > loss: 0.8445314764976501 (0.8796444077616774)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3401 (0.37183528178640074)
| > loader_time: 0.0087 (0.009844224551197293)
 --> TIME: 2024-10-20 01:36:21 -- STEP: 317/3361 -- GLOBAL_STEP: 10400
| > loss_text_ce: 0.045164935290813446 (0.04600713257486889)
| > loss_mel_ce: 3.6646311283111572 (3.4749113001657954)
| > loss: 0.9274489879608154 (0.880229608117594)
| > current_lr: 5e-06
| > step_time: 0.4408 (0.3714463176787464)
| > loader_time: 0.0095 (0.009783504137105367)
 --> TIME: 2024-10-20 01:36:52 -- STEP: 367/3361 -- GLOBAL_STEP: 10450
| > loss_text_ce: 0.04286389797925949 (0.045877355138190114)
| > loss_mel_ce: 3.4389150142669678 (3.4698129999540153)
| > loss: 0.8704447150230408 (0.8789225886563189)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4865 (0.3712051090167721)
| > loader_time: 0.0108 (0.009785189615608232)
 --> TIME: 2024-10-20 01:37:25 -- STEP: 417/3361 -- GLOBAL_STEP: 10500
| > loss_text_ce: 0.048703934997320175 (0.04582360561946027)
| > loss_mel_ce: 3.785539388656616 (3.4709189807196594)
| > loss: 0.9585608243942261 (0.8791856462721058)
| > current_lr: 5e-06
| > step_time: 0.4103 (0.3722521103829219)
| > loader_time: 0.0092 (0.009791807590914569)
 --> TIME: 2024-10-20 01:37:57 -- STEP: 467/3361 -- GLOBAL_STEP: 10550
| > loss_text_ce: 0.045499563217163086 (0.04583273282398248)
| > loss_mel_ce: 3.4453530311584473 (3.465878671956521)
| > loss: 0.8727131485939026 (0.8779278508600893)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.2936 (0.37276712423728947)
| > loader_time: 0.0088 (0.009811640806831176)
 --> TIME: 2024-10-20 01:38:29 -- STEP: 517/3361 -- GLOBAL_STEP: 10600
| > loss_text_ce: 0.04884975031018257 (0.04582477258644436)
| > loss_mel_ce: 3.207493543624878 (3.4628814525714917)
| > loss: 0.814085841178894 (0.8771765561814003)
| > current_lr: 5e-06
| > step_time: 0.3301 (0.3732260878358175)
| > loader_time: 0.0091 (0.009831420226991749)
 --> TIME: 2024-10-20 01:39:01 -- STEP: 567/3361 -- GLOBAL_STEP: 10650
| > loss_text_ce: 0.04280979186296463 (0.045831149414418224)
| > loss_mel_ce: 3.378239154815674 (3.461804541330488)
| > loss: 0.8552622199058533 (0.8769089227420732)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3453 (0.3732173430225839)
| > loader_time: 0.0093 (0.009825989893084055)
 --> TIME: 2024-10-20 01:39:33 -- STEP: 617/3361 -- GLOBAL_STEP: 10700
| > loss_text_ce: 0.04689433425664902 (0.045754293109648035)
| > loss_mel_ce: 3.3812503814697266 (3.4619904638689185)
| > loss: 0.8570361733436584 (0.876936189262755)
| > current_lr: 5e-06
| > step_time: 0.3925 (0.3735560111133942)
| > loader_time: 0.0095 (0.009841294883715657)
 --> TIME: 2024-10-20 01:40:05 -- STEP: 667/3361 -- GLOBAL_STEP: 10750
| > loss_text_ce: 0.04266295209527016 (0.0456879992244662)
| > loss_mel_ce: 3.3630940914154053 (3.46313495364325)
| > loss: 0.8514392375946045 (0.8772057384207868)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4245 (0.37379144442671286)
| > loader_time: 0.0111 (0.009839120118514344)
 --> TIME: 2024-10-20 01:40:38 -- STEP: 717/3361 -- GLOBAL_STEP: 10800
| > loss_text_ce: 0.05172465369105339 (0.04570071628860185)
| > loss_mel_ce: 3.2466700077056885 (3.46129972565623)
| > loss: 0.8245986700057983 (0.8767501106654584)
| > current_lr: 5e-06
| > step_time: 0.341 (0.3741034816332279)
| > loader_time: 0.0106 (0.009837797329848106)
 --> TIME: 2024-10-20 01:41:09 -- STEP: 767/3361 -- GLOBAL_STEP: 10850
| > loss_text_ce: 0.04403238371014595 (0.045711836324715725)
| > loss_mel_ce: 3.8820736408233643 (3.4588085069494734)
| > loss: 0.9815264940261841 (0.8761300860358963)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4974 (0.3738556818980751)
| > loader_time: 0.0094 (0.009833316914882928)
 --> TIME: 2024-10-20 01:41:41 -- STEP: 817/3361 -- GLOBAL_STEP: 10900
| > loss_text_ce: 0.04481789469718933 (0.045687686401160596)
| > loss_mel_ce: 3.3437418937683105 (3.4635588652938787)
| > loss: 0.8471399545669556 (0.8773116379647973)
| > current_lr: 5e-06
| > step_time: 0.3944 (0.37394031287698765)
| > loader_time: 0.0095 (0.009831293122421123)
 --> TIME: 2024-10-20 01:42:13 -- STEP: 867/3361 -- GLOBAL_STEP: 10950
| > loss_text_ce: 0.05023163557052612 (0.04567214789464126)
| > loss_mel_ce: 3.3756706714630127 (3.461011304987343)
| > loss: 0.8564755916595459 (0.876670863457892)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4452 (0.3737019953568229)
| > loader_time: 0.0094 (0.009818215408677175)
 --> TIME: 2024-10-20 01:42:44 -- STEP: 917/3361 -- GLOBAL_STEP: 11000
| > loss_text_ce: 0.04785856604576111 (0.04562381485572559)
| > loss_mel_ce: 3.364436149597168 (3.4592360974398146)
| > loss: 0.8530736565589905 (0.8762149782191186)
| > current_lr: 5e-06
| > step_time: 0.3411 (0.37366046354320887)
| > loader_time: 0.0105 (0.009806825914455725)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_11000.pth
 --> TIME: 2024-10-20 01:43:25 -- STEP: 967/3361 -- GLOBAL_STEP: 11050
| > loss_text_ce: 0.04600205644965172 (0.04561833730266039)
| > loss_mel_ce: 3.5278804302215576 (3.457496856829496)
| > loss: 0.8934706449508667 (0.8757787986649845)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3897 (0.37305284360078916)
| > loader_time: 0.0098 (0.009819003147571006)
 --> TIME: 2024-10-20 01:43:57 -- STEP: 1017/3361 -- GLOBAL_STEP: 11100
| > loss_text_ce: 0.04780234768986702 (0.045599416601622714)
| > loss_mel_ce: 3.2322351932525635 (3.454736575498928)
| > loss: 0.8200094103813171 (0.8750839981240391)
| > current_lr: 5e-06
| > step_time: 0.3965 (0.3729844527024203)
| > loader_time: 0.0094 (0.0098143270937039)
 --> TIME: 2024-10-20 01:44:28 -- STEP: 1067/3361 -- GLOBAL_STEP: 11150
| > loss_text_ce: 0.04303880035877228 (0.04557502307451655)
| > loss_mel_ce: 3.1907825469970703 (3.4526526755893356)
| > loss: 0.8084553480148315 (0.8745569246554888)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4266 (0.3729878837188608)
| > loader_time: 0.0098 (0.009801903280456593)
 --> TIME: 2024-10-20 01:45:00 -- STEP: 1117/3361 -- GLOBAL_STEP: 11200
| > loss_text_ce: 0.04823419824242592 (0.045567612428028044)
| > loss_mel_ce: 4.063570022583008 (3.4510888609378183)
| > loss: 1.0279510021209717 (0.8741641183889866)
| > current_lr: 5e-06
| > step_time: 0.3332 (0.37310730183668855)
| > loader_time: 0.0094 (0.009789423101590795)
 --> TIME: 2024-10-20 01:45:33 -- STEP: 1167/3361 -- GLOBAL_STEP: 11250
| > loss_text_ce: 0.0466812439262867 (0.04556122947972662)
| > loss_mel_ce: 3.205491542816162 (3.4504569024230656)
| > loss: 0.8130431771278381 (0.8740045330722614)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3883 (0.3738090622190134)
| > loader_time: 0.0095 (0.0098025966323944)
 --> TIME: 2024-10-20 01:46:05 -- STEP: 1217/3361 -- GLOBAL_STEP: 11300
| > loss_text_ce: 0.04266681149601936 (0.045536092204717514)
| > loss_mel_ce: 3.636619806289673 (3.44905358430491)
| > loss: 0.9198216795921326 (0.8736474192720409)
| > current_lr: 5e-06
| > step_time: 0.4277 (0.37357596540020077)
| > loader_time: 0.0091 (0.009802650602116571)
 --> TIME: 2024-10-20 01:46:37 -- STEP: 1267/3361 -- GLOBAL_STEP: 11350
| > loss_text_ce: 0.043237507343292236 (0.04552184921507772)
| > loss_mel_ce: 3.1879427433013916 (3.4461687098550535)
| > loss: 0.8077950477600098 (0.8729226399145448)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.545 (0.3738300482306601)
| > loader_time: 0.0093 (0.009796628259539879)
 --> TIME: 2024-10-20 01:47:09 -- STEP: 1317/3361 -- GLOBAL_STEP: 11400
| > loss_text_ce: 0.04297908768057823 (0.045494050124126946)
| > loss_mel_ce: 3.098487138748169 (3.445298686929064)
| > loss: 0.7853665351867676 (0.8726981843884639)
| > current_lr: 5e-06
| > step_time: 0.3311 (0.37390816854362574)
| > loader_time: 0.0103 (0.009803349444969358)
 --> TIME: 2024-10-20 01:47:41 -- STEP: 1367/3361 -- GLOBAL_STEP: 11450
| > loss_text_ce: 0.04727546498179436 (0.04548580103827029)
| > loss_mel_ce: 3.4735140800476074 (3.4428421135967286)
| > loss: 0.8801974058151245 (0.8720819787466358)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.483 (0.3738833208021321)
| > loader_time: 0.0092 (0.009806743222775898)
 --> TIME: 2024-10-20 01:48:13 -- STEP: 1417/3361 -- GLOBAL_STEP: 11500
| > loss_text_ce: 0.04696996882557869 (0.04546378995899845)
| > loss_mel_ce: 3.4015119075775146 (3.4421401881969724)
| > loss: 0.8621204495429993 (0.8719009946750432)
| > current_lr: 5e-06
| > step_time: 0.5081 (0.3737989569013189)
| > loader_time: 0.0104 (0.009797040896257537)
 --> TIME: 2024-10-20 01:48:45 -- STEP: 1467/3361 -- GLOBAL_STEP: 11550
| > loss_text_ce: 0.044216107577085495 (0.04545340134972362)
| > loss_mel_ce: 3.291405439376831 (3.440216698610791)
| > loss: 0.8339053988456726 (0.8714175251120193)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4973 (0.3738921473606478)
| > loader_time: 0.0108 (0.009800101269160946)
 --> TIME: 2024-10-20 01:49:18 -- STEP: 1517/3361 -- GLOBAL_STEP: 11600
| > loss_text_ce: 0.04237224906682968 (0.04541745957235168)
| > loss_mel_ce: 3.4833874702453613 (3.437472724883118)
| > loss: 0.8814399242401123 (0.8707225460242344)
| > current_lr: 5e-06
| > step_time: 0.3329 (0.37426536446271497)
| > loader_time: 0.0088 (0.009796220374563125)
 --> TIME: 2024-10-20 01:49:50 -- STEP: 1567/3361 -- GLOBAL_STEP: 11650
| > loss_text_ce: 0.043478481471538544 (0.04540074599316033)
| > loss_mel_ce: 3.2932302951812744 (3.4390092498569937)
| > loss: 0.8341771960258484 (0.8711024989500574)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.5027 (0.3746144473362518)
| > loader_time: 0.0094 (0.009793423054396287)
 --> TIME: 2024-10-20 01:50:22 -- STEP: 1617/3361 -- GLOBAL_STEP: 11700
| > loss_text_ce: 0.041785042732954025 (0.045383355543984996)
| > loss_mel_ce: 3.2549021244049072 (3.43981561501526)
| > loss: 0.824171781539917 (0.871299742693951)
| > current_lr: 5e-06
| > step_time: 0.4082 (0.3745398368080532)
| > loader_time: 0.0097 (0.009789079224982206)
 --> TIME: 2024-10-20 01:50:54 -- STEP: 1667/3361 -- GLOBAL_STEP: 11750
| > loss_text_ce: 0.044465482234954834 (0.045369411876227374)
| > loss_mel_ce: 3.373613119125366 (3.438965404112323)
| > loss: 0.8545196652412415 (0.8710837041323385)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3738 (0.37457816888274886)
| > loader_time: 0.0089 (0.009785240255720803)
 --> TIME: 2024-10-20 01:51:25 -- STEP: 1717/3361 -- GLOBAL_STEP: 11800
| > loss_text_ce: 0.043654706329107285 (0.04535353077216487)
| > loss_mel_ce: 3.4367294311523438 (3.4394294256100353)
| > loss: 0.8700960278511047 (0.8711957391015164)
| > current_lr: 5e-06
| > step_time: 0.3279 (0.3742698932985408)
| > loader_time: 0.0098 (0.009778505851865554)
 --> TIME: 2024-10-20 01:51:58 -- STEP: 1767/3361 -- GLOBAL_STEP: 11850
| > loss_text_ce: 0.04286897927522659 (0.045324536488429934)
| > loss_mel_ce: 3.091407537460327 (3.4378820990328745)
| > loss: 0.7835691571235657 (0.8708016589393571)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4647 (0.3746508205415979)
| > loader_time: 0.0096 (0.009779719491428595)
 --> TIME: 2024-10-20 01:52:30 -- STEP: 1817/3361 -- GLOBAL_STEP: 11900
| > loss_text_ce: 0.044489096850156784 (0.045306128924881686)
| > loss_mel_ce: 3.4446122646331787 (3.434943928584727)
| > loss: 0.8722753524780273 (0.8700625143276837)
| > current_lr: 5e-06
| > step_time: 0.3748 (0.3747132023940983)
| > loader_time: 0.0092 (0.009790742928731135)
 --> TIME: 2024-10-20 01:53:01 -- STEP: 1867/3361 -- GLOBAL_STEP: 11950
| > loss_text_ce: 0.04551025852560997 (0.04529719577676227)
| > loss_mel_ce: 3.373769521713257 (3.43550654829155)
| > loss: 0.8548199534416199 (0.870200935920803)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4817 (0.37446563321763854)
| > loader_time: 0.0094 (0.009788677918444847)
 --> TIME: 2024-10-20 01:53:34 -- STEP: 1917/3361 -- GLOBAL_STEP: 12000
| > loss_text_ce: 0.03975873813033104 (0.04528394429694257)
| > loss_mel_ce: 3.337285280227661 (3.4359212065961873)
| > loss: 0.8442609906196594 (0.8703012876542955)
| > current_lr: 5e-06
| > step_time: 0.4123 (0.3746762464739224)
| > loader_time: 0.0106 (0.009790282480780113)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_12000.pth
 --> TIME: 2024-10-20 01:54:17 -- STEP: 1967/3361 -- GLOBAL_STEP: 12050
| > loss_text_ce: 0.04682973399758339 (0.04526911055679605)
| > loss_mel_ce: 4.056046962738037 (3.436260929917344)
| > loss: 1.025719165802002 (0.8703825101673026)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3722 (0.37507091395189524)
| > loader_time: 0.0099 (0.009801731778694135)
 --> TIME: 2024-10-20 01:54:49 -- STEP: 2017/3361 -- GLOBAL_STEP: 12100
| > loss_text_ce: 0.04704705625772476 (0.04524998690417691)
| > loss_mel_ce: 3.3677549362182617 (3.4361582207809773)
| > loss: 0.8537005186080933 (0.8703520520819727)
| > current_lr: 5e-06
| > step_time: 0.3057 (0.374842264774003)
| > loader_time: 0.0088 (0.00980050267118831)
 --> TIME: 2024-10-20 01:55:20 -- STEP: 2067/3361 -- GLOBAL_STEP: 12150
| > loss_text_ce: 0.04836978018283844 (0.04522184339673207)
| > loss_mel_ce: 3.221724510192871 (3.4350216229525983)
| > loss: 0.8175235986709595 (0.8700608667626032)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4277 (0.37457450934522224)
| > loader_time: 0.0085 (0.009801711445565242)
 --> TIME: 2024-10-20 01:55:52 -- STEP: 2117/3361 -- GLOBAL_STEP: 12200
| > loss_text_ce: 0.041277926415205 (0.04519546315615956)
| > loss_mel_ce: 3.5539653301239014 (3.4345948362778422)
| > loss: 0.8988108038902283 (0.8699475751206961)
| > current_lr: 5e-06
| > step_time: 0.4406 (0.3744075272406506)
| > loader_time: 0.0094 (0.009799530104077175)
 --> TIME: 2024-10-20 01:56:24 -- STEP: 2167/3361 -- GLOBAL_STEP: 12250
| > loss_text_ce: 0.043600600212812424 (0.045186322316506745)
| > loss_mel_ce: 3.2849605083465576 (3.433843436745053)
| > loss: 0.8321402668952942 (0.8697574399394487)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4201 (0.3745438092086082)
| > loader_time: 0.0087 (0.009803145174942737)
 --> TIME: 2024-10-20 01:56:55 -- STEP: 2217/3361 -- GLOBAL_STEP: 12300
| > loss_text_ce: 0.04543566703796387 (0.04517698947661193)
| > loss_mel_ce: 3.4641506671905518 (3.4332194155166964)
| > loss: 0.8773965835571289 (0.8695991014541676)
| > current_lr: 5e-06
| > step_time: 0.3439 (0.374354049543244)
| > loader_time: 0.0094 (0.009805234961537642)
 --> TIME: 2024-10-20 01:57:28 -- STEP: 2267/3361 -- GLOBAL_STEP: 12350
| > loss_text_ce: 0.04675550386309624 (0.0451725262492882)
| > loss_mel_ce: 3.3266305923461914 (3.4326016073138597)
| > loss: 0.8433465361595154 (0.8694435337013645)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3468 (0.3745551318600675)
| > loader_time: 0.009 (0.009808486392438046)
 --> TIME: 2024-10-20 01:57:59 -- STEP: 2317/3361 -- GLOBAL_STEP: 12400
| > loss_text_ce: 0.04151281341910362 (0.04515179921782474)
| > loss_mel_ce: 3.1535322666168213 (3.4317728494537976)
| > loss: 0.798761248588562 (0.8692311624745946)
| > current_lr: 5e-06
| > step_time: 0.3418 (0.37451987005175325)
| > loader_time: 0.0086 (0.00980887569432427)
 --> TIME: 2024-10-20 01:58:32 -- STEP: 2367/3361 -- GLOBAL_STEP: 12450
| > loss_text_ce: 0.04229551553726196 (0.045148368671467175)
| > loss_mel_ce: 3.2210004329681396 (3.430814805794326)
| > loss: 0.8158239722251892 (0.8689907938808539)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4828 (0.3747464335312257)
| > loader_time: 0.0105 (0.00981024382332984)
 --> TIME: 2024-10-20 01:59:04 -- STEP: 2417/3361 -- GLOBAL_STEP: 12500
| > loss_text_ce: 0.0443447045981884 (0.04513368604687011)
| > loss_mel_ce: 3.286813735961914 (3.4296375612212353)
| > loss: 0.8327895998954773 (0.8686928119992832)
| > current_lr: 5e-06
| > step_time: 0.3053 (0.37469193814064194)
| > loader_time: 0.01 (0.00981032606684376)
 --> TIME: 2024-10-20 01:59:36 -- STEP: 2467/3361 -- GLOBAL_STEP: 12550
| > loss_text_ce: 0.045515257865190506 (0.04512597635587699)
| > loss_mel_ce: 3.5041096210479736 (3.428387396239925)
| > loss: 0.8874062299728394 (0.8683783433241158)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3342 (0.37473538224398517)
| > loader_time: 0.0099 (0.009815803654627791)
 --> TIME: 2024-10-20 02:00:09 -- STEP: 2517/3361 -- GLOBAL_STEP: 12600
| > loss_text_ce: 0.0403006412088871 (0.04510258801834989)
| > loss_mel_ce: 3.3379220962524414 (3.4280766162978407)
| > loss: 0.844555675983429 (0.8682948011560099)
| > current_lr: 5e-06
| > step_time: 0.3751 (0.37487539072381565)
| > loader_time: 0.0109 (0.009816988242918111)
 --> TIME: 2024-10-20 02:00:42 -- STEP: 2567/3361 -- GLOBAL_STEP: 12650
| > loss_text_ce: 0.04288014397025108 (0.04509773093918249)
| > loss_mel_ce: 3.288858652114868 (3.4260108755124326)
| > loss: 0.8329346776008606 (0.8677771517075957)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.389 (0.3752143952370065)
| > loader_time: 0.0097 (0.009822266084190264)
 --> TIME: 2024-10-20 02:01:13 -- STEP: 2617/3361 -- GLOBAL_STEP: 12700
| > loss_text_ce: 0.04661820828914642 (0.04508226625777894)
| > loss_mel_ce: 3.3759257793426514 (3.426038921995385)
| > loss: 0.8556360006332397 (0.8677802970949635)
| > current_lr: 5e-06
| > step_time: 0.2947 (0.37497939952329973)
| > loader_time: 0.0094 (0.0098253846122966)
 --> TIME: 2024-10-20 02:01:45 -- STEP: 2667/3361 -- GLOBAL_STEP: 12750
| > loss_text_ce: 0.044218242168426514 (0.04506286233104825)
| > loss_mel_ce: 3.6532299518585205 (3.4246745852437144)
| > loss: 0.924362063407898 (0.8674343619223249)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4868 (0.3750956985536568)
| > loader_time: 0.0092 (0.009827119740020204)
 --> TIME: 2024-10-20 02:02:16 -- STEP: 2717/3361 -- GLOBAL_STEP: 12800
| > loss_text_ce: 0.041796378791332245 (0.04505104149143652)
| > loss_mel_ce: 3.069788932800293 (3.4237373812390812)
| > loss: 0.7778963446617126 (0.8671971057371305)
| > current_lr: 5e-06
| > step_time: 0.4088 (0.3748981202398981)
| > loader_time: 0.0107 (0.00982653598904741)
 --> TIME: 2024-10-20 02:02:49 -- STEP: 2767/3361 -- GLOBAL_STEP: 12850
| > loss_text_ce: 0.04316914081573486 (0.04503791029644125)
| > loss_mel_ce: 3.131343364715576 (3.423758802320882)
| > loss: 0.7936280965805054 (0.8671991782452078)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4655 (0.37505522074146214)
| > loader_time: 0.0104 (0.009828401206416)
 --> TIME: 2024-10-20 02:03:20 -- STEP: 2817/3361 -- GLOBAL_STEP: 12900
| > loss_text_ce: 0.046656232327222824 (0.04501954110899327)
| > loss_mel_ce: 3.635425090789795 (3.4238399453818227)
| > loss: 0.9205203056335449 (0.8672148717351104)
| > current_lr: 5e-06
| > step_time: 0.3918 (0.3749253896267057)
| > loader_time: 0.0089 (0.009831450448461057)
 --> TIME: 2024-10-20 02:03:51 -- STEP: 2867/3361 -- GLOBAL_STEP: 12950
| > loss_text_ce: 0.04399809241294861 (0.044997297027279795)
| > loss_mel_ce: 3.229576587677002 (3.422730980328579)
| > loss: 0.8183936476707458 (0.8669320693779454)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.3898 (0.3745152650912297)
| > loader_time: 0.0092 (0.009833865686289541)
 --> TIME: 2024-10-20 02:04:22 -- STEP: 2917/3361 -- GLOBAL_STEP: 13000
| > loss_text_ce: 0.045591291040182114 (0.04497031228445057)
| > loss_mel_ce: 3.6106908321380615 (3.422237732004212)
| > loss: 0.9140705466270447 (0.8668020111590079)
| > current_lr: 5e-06
| > step_time: 0.3444 (0.3744682490028744)
| > loader_time: 0.0102 (0.009837184820348432)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_13000.pth
 --> TIME: 2024-10-20 02:05:05 -- STEP: 2967/3361 -- GLOBAL_STEP: 13050
| > loss_text_ce: 0.0435701459646225 (0.044948636828838484)
| > loss_mel_ce: 3.2321619987487793 (3.4209875995396843)
| > loss: 0.8189330101013184 (0.866484059140784)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4589 (0.37451388074928227)
| > loader_time: 0.0106 (0.009842868200005756)
 --> TIME: 2024-10-20 02:05:38 -- STEP: 3017/3361 -- GLOBAL_STEP: 13100
| > loss_text_ce: 0.04389602690935135 (0.04493448949231579)
| > loss_mel_ce: 3.2054622173309326 (3.4204812088587997)
| > loss: 0.8123395442962646 (0.8663539246569255)
| > current_lr: 5e-06
| > step_time: 0.3433 (0.37459014953269354)
| > loader_time: 0.0092 (0.0098414506428598)
 --> TIME: 2024-10-20 02:06:10 -- STEP: 3067/3361 -- GLOBAL_STEP: 13150
| > loss_text_ce: 0.04617796465754509 (0.044920595558946216)
| > loss_mel_ce: 3.227992296218872 (3.4195069690694395)
| > loss: 0.8185425400733948 (0.8661068912278488)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4677 (0.3747628785361285)
| > loader_time: 0.0093 (0.009840129678786514)
 --> TIME: 2024-10-20 02:06:42 -- STEP: 3117/3361 -- GLOBAL_STEP: 13200
| > loss_text_ce: 0.042707689106464386 (0.04491019368955706)
| > loss_mel_ce: 3.2197930812835693 (3.4186071596124212)
| > loss: 0.8156251907348633 (0.86587933833147)
| > current_lr: 5e-06
| > step_time: 0.409 (0.37470273769624646)
| > loader_time: 0.0107 (0.009839347895040002)
 --> TIME: 2024-10-20 02:07:14 -- STEP: 3167/3361 -- GLOBAL_STEP: 13250
| > loss_text_ce: 0.04247237741947174 (0.044894521159067204)
| > loss_mel_ce: 3.3792779445648193 (3.4170680701036273)
| > loss: 0.8554375767707825 (0.8654906478709588)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.4468 (0.37466244830067386)
| > loader_time: 0.0113 (0.009841234103565064)
 --> TIME: 2024-10-20 02:07:45 -- STEP: 3217/3361 -- GLOBAL_STEP: 13300
| > loss_text_ce: 0.04384773224592209 (0.0448808852313355)
| > loss_mel_ce: 3.4374778270721436 (3.4173526868414057)
| > loss: 0.8703314065933228 (0.8655583930934551)
| > current_lr: 5e-06
| > step_time: 0.3324 (0.37453380481849213)
| > loader_time: 0.0105 (0.009846873115892904)
 --> TIME: 2024-10-20 02:08:18 -- STEP: 3267/3361 -- GLOBAL_STEP: 13350
| > loss_text_ce: 0.0455152653157711 (0.044865900441379274)
| > loss_mel_ce: 3.3864834308624268 (3.416893625463737)
| > loss: 0.8579996824264526 (0.8654398815253108)
| > grad_norm: 0 (0.0)
| > current_lr: 5e-06
| > step_time: 0.444 (0.37472096641092145)
| > loader_time: 0.011 (0.009851177132089519)
 --> TIME: 2024-10-20 02:08:49 -- STEP: 3317/3361 -- GLOBAL_STEP: 13400
| > loss_text_ce: 0.04139714315533638 (0.04483872734713268)
| > loss_mel_ce: 3.251816987991333 (3.416392544216516)
| > loss: 0.8233035206794739 (0.8653078179852515)
| > current_lr: 5e-06
| > step_time: 0.2561 (0.3746120028370721)
| > loader_time: 0.0093 (0.009851044475726763)
 > EVALUATION 
--> EVAL PERFORMANCE
| > avg_loader_time: 0.3778952552426247 (+0.0006334499646259428)
| > avg_loss_text_ce: 0.04140995150452017 (-0.0020317246596659524)
| > avg_loss_mel_ce: 3.1711383763179986 (-0.1000511492452314)
| > avg_loss: 3.21254833539327 (-0.10208286264891253)
> BEST MODEL : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/best_model_13444.pth
 > EPOCH: 4/5
--> /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b
 > TRAINING (2024-10-20 02:11:46) 
 --> TIME: 2024-10-20 02:11:51 -- STEP: 6/3361 -- GLOBAL_STEP: 13450
| > loss_text_ce: 0.04218747466802597 (0.04258658178150654)
| > loss_mel_ce: 3.332731008529663 (3.278356671333313)
| > loss: 0.8437296152114868 (0.8302358090877533)
| > current_lr: 5e-06
| > step_time: 0.3784 (0.34697389602661133)
| > loader_time: 0.0096 (0.010201772054036459)
 --> TIME: 2024-10-20 02:12:23 -- STEP: 56/3361 -- GLOBAL_STEP: 13500
| > loss_text_ce: 0.045623574405908585 (0.04398354063076631)
| > loss_mel_ce: 3.8470444679260254 (3.3895332387515476)
| > loss: 0.9731670022010803 (0.85837919903653)
| > current_lr: 5e-06
| > step_time: 0.3448 (0.3710901652063643)
| > loader_time: 0.0103 (0.009938184704099384)
 --> TIME: 2024-10-20 02:12:55 -- STEP: 106/3361 -- GLOBAL_STEP: 13550
| > loss_text_ce: 0.04445193335413933 (0.043825697575537656)
| > loss_mel_ce: 3.2017033100128174 (3.3736256473469286)
| > loss: 0.811538815498352 (0.8543628392354498)
| > current_lr: 5e-06
| > step_time: 0.2652 (0.37404405395939677)
| > loader_time: 0.009 (0.00981641490504427)
 --> TIME: 2024-10-20 02:13:27 -- STEP: 156/3361 -- GLOBAL_STEP: 13600
| > loss_text_ce: 0.04687873274087906 (0.04368266580292046)
| > loss_mel_ce: 3.556082248687744 (3.373449661792852)
| > loss: 0.9007402658462524 (0.8542830833257773)
| > current_lr: 5e-06
| > step_time: 0.3338 (0.3716987386727945)
| > loader_time: 0.0096 (0.009778794569846915)
 --> TIME: 2024-10-20 02:13:59 -- STEP: 206/3361 -- GLOBAL_STEP: 13650
| > loss_text_ce: 0.04147164896130562 (0.04358611028026608)
| > loss_mel_ce: 3.6620306968688965 (3.3716245033208603)
| > loss: 0.9258756041526794 (0.8538026543496882)
| > current_lr: 5e-06
| > step_time: 0.4133 (0.3729133767989075)
| > loader_time: 0.0097 (0.009783061962683217)
 --> TIME: 2024-10-20 02:14:32 -- STEP: 256/3361 -- GLOBAL_STEP: 13700
| > loss_text_ce: 0.04409633204340935 (0.043609272077446797)
| > loss_mel_ce: 3.1696982383728027 (3.3641294278204437)
| > loss: 0.8034486174583435 (0.8519346755929291)
| > current_lr: 5e-06
| > step_time: 0.2645 (0.3750447416678071)
| > loader_time: 0.0107 (0.009823349304497237)
 --> TIME: 2024-10-20 02:15:04 -- STEP: 306/3361 -- GLOBAL_STEP: 13750
| > loss_text_ce: 0.04763133078813553 (0.04354089683566998)
| > loss_mel_ce: 3.3492703437805176 (3.358471124000797)
| > loss: 0.8492254018783569 (0.850503006012611)
| > current_lr: 5e-06
| > step_time: 0.3438 (0.37534330483355555)
| > loader_time: 0.0095 (0.009806627541585686)
 --> TIME: 2024-10-20 02:15:36 -- STEP: 356/3361 -- GLOBAL_STEP: 13800
| > loss_text_ce: 0.040769223123788834 (0.04356232056331437)
| > loss_mel_ce: 3.5312817096710205 (3.358752216515915)
| > loss: 0.8930127620697021 (0.8505786349264421)
| > current_lr: 5e-06
| > step_time: 0.3887 (0.3760112068626318)
| > loader_time: 0.01 (0.009826761283231584)
 --> TIME: 2024-10-20 02:16:08 -- STEP: 406/3361 -- GLOBAL_STEP: 13850
| > loss_text_ce: 0.046686802059412 (0.04353852070904719)
| > loss_mel_ce: 3.1642913818359375 (3.35460764962464)
| > loss: 0.8027445673942566 (0.8495365434092252)
| > current_lr: 5e-06
| > step_time: 0.3028 (0.37603559106441553)
| > loader_time: 0.01 (0.009861112815405927)
 --> TIME: 2024-10-20 02:16:40 -- STEP: 456/3361 -- GLOBAL_STEP: 13900
| > loss_text_ce: 0.04211386665701866 (0.043516929048979504)
| > loss_mel_ce: 3.4279661178588867 (3.3584800671067145)
| > loss: 0.8675199747085571 (0.8504992499947547)
| > current_lr: 5e-06
| > step_time: 0.3128 (0.37571073101277935)
| > loader_time: 0.0086 (0.009866837346762934)
 --> TIME: 2024-10-20 02:17:13 -- STEP: 506/3361 -- GLOBAL_STEP: 13950
| > loss_text_ce: 0.043664440512657166 (0.04350922338428942)
| > loss_mel_ce: 3.516002893447876 (3.365730038744658)
| > loss: 0.8899168372154236 (0.8523098163218366)
| > current_lr: 5e-06
| > step_time: 0.376 (0.3766041942264723)
| > loader_time: 0.0107 (0.009907983037322871)
 --> TIME: 2024-10-20 02:17:44 -- STEP: 556/3361 -- GLOBAL_STEP: 14000
| > loss_text_ce: 0.038978710770606995 (0.043512208420756496)
| > loss_mel_ce: 3.1659882068634033 (3.360013530408735)
| > loss: 0.8012417554855347 (0.850881435030656)
| > current_lr: 5e-06
| > step_time: 0.3549 (0.37569527231532024)
| > loader_time: 0.0109 (0.009921835480834078)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_14000.pth
 --> TIME: 2024-10-20 02:18:28 -- STEP: 606/3361 -- GLOBAL_STEP: 14050
| > loss_text_ce: 0.03997735306620598 (0.04353958326499454)
| > loss_mel_ce: 3.1434693336486816 (3.365101049048671)
| > loss: 0.7958616614341736 (0.8521601585271732)
| > current_lr: 5e-06
| > step_time: 0.3924 (0.37639728040978465)
| > loader_time: 0.0092 (0.009938992289426695)
 --> TIME: 2024-10-20 02:19:00 -- STEP: 656/3361 -- GLOBAL_STEP: 14100
| > loss_text_ce: 0.04379415884613991 (0.04352298771349216)
| > loss_mel_ce: 3.4719841480255127 (3.364289275029809)
| > loss: 0.8789445757865906 (0.8519530663403072)
| > current_lr: 5e-06
| > step_time: 0.2412 (0.37598983107543565)
| > loader_time: 0.0094 (0.009922782458910127)
 --> TIME: 2024-10-20 02:19:32 -- STEP: 706/3361 -- GLOBAL_STEP: 14150
| > loss_text_ce: 0.04108851030468941 (0.04352049126946724)
| > loss_mel_ce: 3.5078001022338867 (3.365766045392065)
| > loss: 0.887222170829773 (0.8523216351402383)
| > current_lr: 5e-06
| > step_time: 0.432 (0.3764305361269554)
| > loader_time: 0.0099 (0.009930196970110234)
 --> TIME: 2024-10-20 02:20:04 -- STEP: 756/3361 -- GLOBAL_STEP: 14200
| > loss_text_ce: 0.041327524930238724 (0.0434930988306564)
| > loss_mel_ce: 3.2496447563171387 (3.363260011508981)
| > loss: 0.8227430582046509 (0.8516882787305845)
| > current_lr: 5e-06
| > step_time: 0.329 (0.37616385890062526)
| > loader_time: 0.0095 (0.009909986188172033)
 --> TIME: 2024-10-20 02:20:36 -- STEP: 806/3361 -- GLOBAL_STEP: 14250
| > loss_text_ce: 0.03913938254117966 (0.043490528325473995)
| > loss_mel_ce: 3.4655745029449463 (3.364514757918364)
| > loss: 0.8761784434318542 (0.8520013225581453)
| > current_lr: 5e-06
| > step_time: 0.395 (0.37637911570575044)
| > loader_time: 0.0101 (0.009920249505906784)
 --> TIME: 2024-10-20 02:21:08 -- STEP: 856/3361 -- GLOBAL_STEP: 14300
| > loss_text_ce: 0.041906122118234634 (0.043468420245797834)
| > loss_mel_ce: 3.5537428855895996 (3.3637058712611676)
| > loss: 0.898912250995636 (0.8517935738385284)
| > current_lr: 5e-06
| > step_time: 0.3119 (0.37598651675420386)
| > loader_time: 0.0104 (0.009900728954333015)
 --> TIME: 2024-10-20 02:21:39 -- STEP: 906/3361 -- GLOBAL_STEP: 14350
| > loss_text_ce: 0.043285299092531204 (0.043472079852098396)
| > loss_mel_ce: 3.286458730697632 (3.363266183314207)
| > loss: 0.8324360251426697 (0.8516845666283279)
| > current_lr: 5e-06
| > step_time: 0.4279 (0.37519479271591893)
| > loader_time: 0.0142 (0.009895196550466107)
 --> TIME: 2024-10-20 02:22:10 -- STEP: 956/3361 -- GLOBAL_STEP: 14400
| > loss_text_ce: 0.041109610348939896 (0.043473483180183)
| > loss_mel_ce: 3.326003313064575 (3.3631330122508754)
| > loss: 0.8417782187461853 (0.8516516245309281)
| > current_lr: 5e-06
| > step_time: 0.2971 (0.374720933297688)
| > loader_time: 0.0116 (0.00989291059422194)
 --> TIME: 2024-10-20 02:22:42 -- STEP: 1006/3361 -- GLOBAL_STEP: 14450
| > loss_text_ce: 0.04267358034849167 (0.04345590186492349)
| > loss_mel_ce: 3.3953325748443604 (3.3609538547561364)
| > loss: 0.8595015406608582 (0.8511024397125777)
| > current_lr: 5e-06
| > step_time: 0.3428 (0.37477593156499844)
| > loader_time: 0.009 (0.009898023387309812)
 --> TIME: 2024-10-20 02:23:14 -- STEP: 1056/3361 -- GLOBAL_STEP: 14500
| > loss_text_ce: 0.039795197546482086 (0.04344303786020839)
| > loss_mel_ce: 3.1137518882751465 (3.35917877676812)
| > loss: 0.7883867621421814 (0.850655454295603)
| > current_lr: 5e-06
| > step_time: 0.296 (0.3749279420484196)
| > loader_time: 0.0094 (0.009898246237725925)
 --> TIME: 2024-10-20 02:23:46 -- STEP: 1106/3361 -- GLOBAL_STEP: 14550
| > loss_text_ce: 0.04497344419360161 (0.04342685892140564)
| > loss_mel_ce: 3.1941349506378174 (3.3563017004437294)
| > loss: 0.8097770810127258 (0.8499321404029407)
| > current_lr: 5e-06
| > step_time: 0.4831 (0.37497546961682615)
| > loader_time: 0.0094 (0.009898357322375557)
 --> TIME: 2024-10-20 02:24:18 -- STEP: 1156/3361 -- GLOBAL_STEP: 14600
| > loss_text_ce: 0.05118125304579735 (0.04341617007143668)
| > loss_mel_ce: 3.0756499767303467 (3.3546003092944003)
| > loss: 0.7817078232765198 (0.8495041204674436)
| > current_lr: 5e-06
| > step_time: 0.3348 (0.37483308781389535)
| > loader_time: 0.011 (0.009883925370279068)
 --> TIME: 2024-10-20 02:24:49 -- STEP: 1206/3361 -- GLOBAL_STEP: 14650
| > loss_text_ce: 0.047178130596876144 (0.04340132807148237)
| > loss_mel_ce: 3.2075843811035156 (3.353524711594653)
| > loss: 0.8136906027793884 (0.8492315104648253)
| > current_lr: 5e-06
| > step_time: 0.3394 (0.3744580769420263)
| > loader_time: 0.0108 (0.009874612735476261)
 --> TIME: 2024-10-20 02:25:21 -- STEP: 1256/3361 -- GLOBAL_STEP: 14700
| > loss_text_ce: 0.040922343730926514 (0.04340411143697754)
| > loss_mel_ce: 3.0344512462615967 (3.351216148039338)
| > loss: 0.768843412399292 (0.8486550655334623)
| > current_lr: 5e-06
| > step_time: 0.3548 (0.37435099767271907)
| > loader_time: 0.0092 (0.009869077023427202)
 --> TIME: 2024-10-20 02:25:54 -- STEP: 1306/3361 -- GLOBAL_STEP: 14750
| > loss_text_ce: 0.043831005692481995 (0.04338768764953796)
| > loss_mel_ce: 3.3428115844726562 (3.3530943813586855)
| > loss: 0.8466606736183167 (0.849120517876028)
| > current_lr: 5e-06
| > step_time: 0.4858 (0.3746798794265914)
| > loader_time: 0.0096 (0.009866920401089782)
 --> TIME: 2024-10-20 02:26:26 -- STEP: 1356/3361 -- GLOBAL_STEP: 14800
| > loss_text_ce: 0.04674549773335457 (0.04338476714021519)
| > loss_mel_ce: 3.245988368988037 (3.3537734330227944)
| > loss: 0.8231834769248962 (0.8492895506300409)
| > current_lr: 5e-06
| > step_time: 0.3845 (0.3747306175991501)
| > loader_time: 0.0092 (0.009857949078258876)
 --> TIME: 2024-10-20 02:26:57 -- STEP: 1406/3361 -- GLOBAL_STEP: 14850
| > loss_text_ce: 0.04541696980595589 (0.04338177127411242)
| > loss_mel_ce: 3.1508896350860596 (3.3522465773361336)
| > loss: 0.7990766763687134 (0.8489070875970266)
| > current_lr: 5e-06
| > step_time: 0.3409 (0.3744548187825627)
| > loader_time: 0.0093 (0.009856832146135888)
 --> TIME: 2024-10-20 02:27:28 -- STEP: 1456/3361 -- GLOBAL_STEP: 14900
| > loss_text_ce: 0.0430903360247612 (0.043365632403887584)
| > loss_mel_ce: 3.4218108654022217 (3.349643322792682)
| > loss: 0.8662253022193909 (0.8482522392993448)
| > current_lr: 5e-06
| > step_time: 0.2738 (0.3738230476667597)
| > loader_time: 0.0107 (0.009860832448844078)
 --> TIME: 2024-10-20 02:28:00 -- STEP: 1506/3361 -- GLOBAL_STEP: 14950
| > loss_text_ce: 0.041327379643917084 (0.0433498491517022)
| > loss_mel_ce: 3.213487148284912 (3.3493747807752245)
| > loss: 0.8137036561965942 (0.8481811579084336)
| > current_lr: 5e-06
| > step_time: 0.4443 (0.37404442419252276)
| > loader_time: 0.0112 (0.009870298988474004)
 --> TIME: 2024-10-20 02:28:32 -- STEP: 1556/3361 -- GLOBAL_STEP: 15000
| > loss_text_ce: 0.043370772153139114 (0.0433355381276583)
| > loss_mel_ce: 3.200894832611084 (3.348205763607221)
| > loss: 0.8110663890838623 (0.8478853257042587)
| > current_lr: 5e-06
| > step_time: 0.3503 (0.37390149574353376)
| > loader_time: 0.0099 (0.009879069493791434)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_15000.pth
 --> TIME: 2024-10-20 02:29:13 -- STEP: 1606/3361 -- GLOBAL_STEP: 15050
| > loss_text_ce: 0.040539246052503586 (0.04333394316081909)
| > loss_mel_ce: 3.2453808784484863 (3.3468510259875326)
| > loss: 0.8214800357818604 (0.8475462425393455)
| > current_lr: 5e-06
| > step_time: 0.3425 (0.3736290137467917)
| > loader_time: 0.0094 (0.009883364289071158)
 --> TIME: 2024-10-20 02:29:44 -- STEP: 1656/3361 -- GLOBAL_STEP: 15100
| > loss_text_ce: 0.041311707347631454 (0.043320907633031765)
| > loss_mel_ce: 3.5958211421966553 (3.3458771780493177)
| > loss: 0.9092832207679749 (0.8472995217079704)
| > current_lr: 5e-06
| > step_time: 0.3133 (0.37350795277650845)
| > loader_time: 0.0111 (0.00987322347751562)
 --> TIME: 2024-10-20 02:30:16 -- STEP: 1706/3361 -- GLOBAL_STEP: 15150
| > loss_text_ce: 0.04214273765683174 (0.04329393436566677)
| > loss_mel_ce: 3.3591790199279785 (3.34583445772057)
| > loss: 0.8503304123878479 (0.8472820981585828)
| > current_lr: 5e-06
| > step_time: 0.4414 (0.37350877675753025)
| > loader_time: 0.0092 (0.009861578136368064)
 --> TIME: 2024-10-20 02:30:47 -- STEP: 1756/3361 -- GLOBAL_STEP: 15200
| > loss_text_ce: 0.043406903743743896 (0.04328855383793287)
| > loss_mel_ce: 3.467005491256714 (3.3466667384927526)
| > loss: 0.8776031136512756 (0.8474888233361866)
| > current_lr: 5e-06
| > step_time: 0.333 (0.37319373011860635)
| > loader_time: 0.0093 (0.009854932037735853)
 --> TIME: 2024-10-20 02:31:18 -- STEP: 1806/3361 -- GLOBAL_STEP: 15250
| > loss_text_ce: 0.04122260957956314 (0.043277650508423186)
| > loss_mel_ce: 3.6228456497192383 (3.3469821526230636)
| > loss: 0.9160170555114746 (0.8475649511299262)
| > current_lr: 5e-06
| > step_time: 0.3422 (0.37290383480976097)
| > loader_time: 0.0092 (0.009851938466297033)
 --> TIME: 2024-10-20 02:31:50 -- STEP: 1856/3361 -- GLOBAL_STEP: 15300
| > loss_text_ce: 0.04111756384372711 (0.04326138178544954)
| > loss_mel_ce: 3.236495018005371 (3.345419908777393)
| > loss: 0.8194031715393066 (0.8471703228765523)
| > current_lr: 5e-06
| > step_time: 0.2677 (0.3728802304329547)
| > loader_time: 0.0108 (0.009853955101350256)
 --> TIME: 2024-10-20 02:32:23 -- STEP: 1906/3361 -- GLOBAL_STEP: 15350
| > loss_text_ce: 0.04561467841267586 (0.0432446542482311)
| > loss_mel_ce: 3.1759002208709717 (3.343184289877212)
| > loss: 0.8053787350654602 (0.8466072363343096)
| > current_lr: 5e-06
| > step_time: 0.3718 (0.37316976939518615)
| > loader_time: 0.0108 (0.009854661330095991)
 --> TIME: 2024-10-20 02:32:54 -- STEP: 1956/3361 -- GLOBAL_STEP: 15400
| > loss_text_ce: 0.04514142870903015 (0.043227853807706405)
| > loss_mel_ce: 3.7339606285095215 (3.343681211120512)
| > loss: 0.9447755217552185 (0.8467272666639104)
| > current_lr: 5e-06
| > step_time: 0.3132 (0.3729808320784621)
| > loader_time: 0.0094 (0.009854103280478945)
 --> TIME: 2024-10-20 02:33:27 -- STEP: 2006/3361 -- GLOBAL_STEP: 15450
| > loss_text_ce: 0.044246867299079895 (0.04321134599447547)
| > loss_mel_ce: 3.073195457458496 (3.3438472198703115)
| > loss: 0.7793605923652649 (0.8467646420061412)
| > current_lr: 5e-06
| > step_time: 0.3289 (0.3733023584542701)
| > loader_time: 0.0102 (0.009853981903281548)
 --> TIME: 2024-10-20 02:33:59 -- STEP: 2056/3361 -- GLOBAL_STEP: 15500
| > loss_text_ce: 0.04231102764606476 (0.04320369416608653)
| > loss_mel_ce: 3.146184206008911 (3.3436386884418443)
| > loss: 0.7971237897872925 (0.8467105962191113)
| > current_lr: 5e-06
| > step_time: 0.3296 (0.3735750739676483)
| > loader_time: 0.0112 (0.009853950154456646)
 --> TIME: 2024-10-20 02:34:32 -- STEP: 2106/3361 -- GLOBAL_STEP: 15550
| > loss_text_ce: 0.04851767048239708 (0.043201351097845966)
| > loss_mel_ce: 3.2179629802703857 (3.3433823642114855)
| > loss: 0.8166201710700989 (0.8466459293367745)
| > current_lr: 5e-06
| > step_time: 0.3399 (0.3736674039565378)
| > loader_time: 0.0097 (0.00985253649672665)
 --> TIME: 2024-10-20 02:35:03 -- STEP: 2156/3361 -- GLOBAL_STEP: 15600
| > loss_text_ce: 0.04181588441133499 (0.04318926972207477)
| > loss_mel_ce: 3.297445058822632 (3.3427740298971833)
| > loss: 0.834815263748169 (0.8464908254832197)
| > current_lr: 5e-06
| > step_time: 0.3499 (0.37356952944138105)
| > loader_time: 0.0099 (0.009845251497399609)
 --> TIME: 2024-10-20 02:35:35 -- STEP: 2206/3361 -- GLOBAL_STEP: 15650
| > loss_text_ce: 0.04130377620458603 (0.04318598364011223)
| > loss_mel_ce: 3.1319963932037354 (3.3422171480528133)
| > loss: 0.7933250665664673 (0.8463507834501085)
| > current_lr: 5e-06
| > step_time: 0.3916 (0.37357278088494483)
| > loader_time: 0.0098 (0.009844925872650562)
 --> TIME: 2024-10-20 02:36:07 -- STEP: 2256/3361 -- GLOBAL_STEP: 15700
| > loss_text_ce: 0.04426877945661545 (0.04318243447453417)
| > loss_mel_ce: 3.1627094745635986 (3.3417251985123806)
| > loss: 0.8017445802688599 (0.8462269087949542)
| > current_lr: 5e-06
| > step_time: 0.2747 (0.3734862113252605)
| > loader_time: 0.0095 (0.009843703597149952)
 --> TIME: 2024-10-20 02:36:39 -- STEP: 2306/3361 -- GLOBAL_STEP: 15750
| > loss_text_ce: 0.041298702359199524 (0.04315780556671066)
| > loss_mel_ce: 3.124805212020874 (3.3405950526413655)
| > loss: 0.7915259599685669 (0.8459382151792287)
| > current_lr: 5e-06
| > step_time: 0.4433 (0.37359404553564807)
| > loader_time: 0.0129 (0.009846728879890538)
 --> TIME: 2024-10-20 02:37:11 -- STEP: 2356/3361 -- GLOBAL_STEP: 15800
| > loss_text_ce: 0.04402311518788338 (0.043152859462946884)
| > loss_mel_ce: 3.4008374214172363 (3.340425018644898)
| > loss: 0.8612151145935059 (0.845894470127815)
| > current_lr: 5e-06
| > step_time: 0.3286 (0.37368188477129255)
| > loader_time: 0.0106 (0.009847130658063901)
 --> TIME: 2024-10-20 02:37:43 -- STEP: 2406/3361 -- GLOBAL_STEP: 15850
| > loss_text_ce: 0.03998379036784172 (0.04313613363818037)
| > loss_mel_ce: 3.198335886001587 (3.3398356255351334)
| > loss: 0.8095799088478088 (0.8457429404320169)
| > current_lr: 5e-06
| > step_time: 0.4964 (0.3737038702937041)
| > loader_time: 0.0094 (0.009845961259983025)
 --> TIME: 2024-10-20 02:38:16 -- STEP: 2456/3361 -- GLOBAL_STEP: 15900
| > loss_text_ce: 0.040953829884529114 (0.043121636062496436)
| > loss_mel_ce: 3.21343731880188 (3.339520607107234)
| > loss: 0.8135977983474731 (0.8456605613862657)
| > current_lr: 5e-06
| > step_time: 0.3581 (0.37380436975326947)
| > loader_time: 0.0097 (0.009839947429464382)
 --> TIME: 2024-10-20 02:38:48 -- STEP: 2506/3361 -- GLOBAL_STEP: 15950
| > loss_text_ce: 0.0415506437420845 (0.04310880687625903)
| > loss_mel_ce: 3.3829474449157715 (3.338792998697502)
| > loss: 0.8561245203018188 (0.8454754520014396)
| > current_lr: 5e-06
| > step_time: 0.4815 (0.37399479996558715)
| > loader_time: 0.009 (0.00982972097130461)
 --> TIME: 2024-10-20 02:39:20 -- STEP: 2556/3361 -- GLOBAL_STEP: 16000
| > loss_text_ce: 0.04153384268283844 (0.04309028748230475)
| > loss_mel_ce: 3.4101197719573975 (3.337869835011833)
| > loss: 0.8629134297370911 (0.8452400312364011)
| > current_lr: 5e-06
| > step_time: 0.3321 (0.3739765604523615)
| > loader_time: 0.0093 (0.009837146003108253)
> CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_16000.pth
 --> TIME: 2024-10-20 02:40:02 -- STEP: 2606/3361 -- GLOBAL_STEP: 16050
| > loss_text_ce: 0.044827427715063095 (0.0430758015159017)
| > loss_mel_ce: 3.3700437545776367 (3.3376472514131272)
| > loss: 0.8537178039550781 (0.8451807638451586)
| > current_lr: 5e-06
| > step_time: 0.3326 (0.3737125197283966)
| > loader_time: 0.0104 (0.00983854037289974)
 --> TIME: 2024-10-20 02:40:34 -- STEP: 2656/3361 -- GLOBAL_STEP: 16100
| > loss_text_ce: 0.04216808080673218 (0.04306527884138452)
| > loss_mel_ce: 3.410971164703369 (3.3377770967332685)
| > loss: 0.8632848262786865 (0.8452105945178188)
| > current_lr: 5e-06
| > step_time: 0.33 (0.37373760216925517)
| > loader_time: 0.0109 (0.009834927966795767)
 --> TIME: 2024-10-20 02:41:06 -- STEP: 2706/3361 -- GLOBAL_STEP: 16150
| > loss_text_ce: 0.044430822134017944 (0.0430597535404049)
| > loss_mel_ce: 3.2338714599609375 (3.337480453852801)
| > loss: 0.8195755481719971 (0.8451350524702167)
| > current_lr: 5e-06
| > step_time: 0.3061 (0.37378184317837615)
| > loader_time: 0.0099 (0.009834342503318588)
 --> TIME: 2024-10-20 02:41:38 -- STEP: 2756/3361 -- GLOBAL_STEP: 16200
| > loss_text_ce: 0.04327122122049332 (0.04304295761918937)
| > loss_mel_ce: 2.923125982284546 (3.3371628782399654)
| > loss: 0.7415993213653564 (0.8450514595031049)
| > current_lr: 5e-06
| > step_time: 0.2433 (0.37380930214731006)
| > loader_time: 0.0091 (0.009836543108105483)
 --> TIME: 2024-10-20 02:42:10 -- STEP: 2806/3361 -- GLOBAL_STEP: 16250
| > loss_text_ce: 0.041747577488422394 (0.043037881924693044)
| > loss_mel_ce: 3.2888636589050293 (3.3371500277298285)
| > loss: 0.8326528072357178 (0.8450469779951272)
| > current_lr: 5e-06
| > step_time: 0.4812 (0.3739183638151933)
| > loader_time: 0.0089 (0.009833243261977589)
 --> TIME: 2024-10-20 02:42:42 -- STEP: 2856/3361 -- GLOBAL_STEP: 16300
| > loss_text_ce: 0.04618898406624794 (0.043028440912115835)
| > loss_mel_ce: 3.1570663452148438 (3.3359200244858127)
| > loss: 0.8008138537406921 (0.8447371168630801)
| > current_lr: 5e-06
| > step_time: 0.2977 (0.37397973352119684)
| > loader_time: 0.0098 (0.009833188534450792)
 --> TIME: 2024-10-20 02:43:14 -- STEP: 2906/3361 -- GLOBAL_STEP: 16350
| > loss_text_ce: 0.04405193775892258 (0.043023765260188244)
| > loss_mel_ce: 3.3160107135772705 (3.3349478833854214)
| > loss: 0.8400156497955322 (0.8444929126411659)
| > current_lr: 5e-06
| > step_time: 0.3443 (0.3739515754656386)
| > loader_time: 0.0093 (0.009837625277756161)
 --> TIME: 2024-10-20 02:43:46 -- STEP: 2956/3361 -- GLOBAL_STEP: 16400
| > loss_text_ce: 0.03851513937115669 (0.0430099739036516)
| > loss_mel_ce: 3.1437385082244873 (3.3339321412478773)
| > loss: 0.7955633997917175 (0.8442355293140363)
| > current_lr: 5e-06
| > step_time: 0.2145 (0.3739763057118983)
| > loader_time: 0.01 (0.009843850410032974)
 --> TIME: 2024-10-20 02:44:18 -- STEP: 3006/3361 -- GLOBAL_STEP: 16450
| > loss_text_ce: 0.041888102889060974 (0.04299571098052135)
| > loss_mel_ce: 3.3893206119537354 (3.333208246066103)
| > loss: 0.857802152633667 (0.8440509898140682)
| > current_lr: 5e-06
| > step_time: 0.3864 (0.3740643212895192)
| > loader_time: 0.0106 (0.00984884141210072)
 --> TIME: 2024-10-20 02:44:51 -- STEP: 3056/3361 -- GLOBAL_STEP: 16500
| > loss_text_ce: 0.042515724897384644 (0.042986133012735295)
| > loss_mel_ce: 3.4355268478393555 (3.3327198423328177)
| > loss: 0.8695106506347656 (0.8439264943852478)
| > current_lr: 5e-06
| > step_time: 0.2655 (0.37418316187658845)
| > loader_time: 0.0093 (0.00985704128342773)
 --> TIME: 2024-10-20 02:45:23 -- STEP: 3106/3361 -- GLOBAL_STEP: 16550
| > loss_text_ce: 0.04620397463440895 (0.04297397403839279)
| > loss_mel_ce: 3.0770668983459473 (3.331866742408588)
| > loss: 0.7808177471160889 (0.8437101797219333)
| > current_lr: 5e-06
| > step_time: 0.3309 (0.374201875460662)
| > loader_time: 0.0104 (0.009857491148109212)
 --> TIME: 2024-10-20 02:45:54 -- STEP: 3156/3361 -- GLOBAL_STEP: 16600
| > loss_text_ce: 0.03918001055717468 (0.04296290355702653)
| > loss_mel_ce: 3.073801040649414 (3.331039714344854)
| > loss: 0.7782452702522278 (0.8435006549603136)
| > current_lr: 5e-06
| > step_time: 0.316 (0.3741178535236604)
| > loader_time: 0.0103 (0.009862533389237732)
 --> TIME: 2024-10-20 02:46:27 -- STEP: 3206/3361 -- GLOBAL_STEP: 16650
| > loss_text_ce: 0.044319815933704376 (0.04295722946189968)
| > loss_mel_ce: 3.2032668590545654 (3.3305540042450015)
| > loss: 0.8118966817855835 (0.8433778088537518)
| > current_lr: 5e-06
| > step_time: 0.4352 (0.3742924399920276)
| > loader_time: 0.0105 (0.00986770206291973)
 --> TIME: 2024-10-20 02:47:00 -- STEP: 3256/3361 -- GLOBAL_STEP: 16700
| > loss_text_ce: 0.03914668783545494 (0.042953169736461974)
| > loss_mel_ce: 3.193128824234009 (3.329916554235414)
| > loss: 0.8080688714981079 (0.8432174314737908)
| > current_lr: 5e-06
| > step_time: 0.2983 (0.3744111431611549)
| > loader_time: 0.0106 (0.009869851409949584)
 --> TIME: 2024-10-20 02:47:32 -- STEP: 3306/3361 -- GLOBAL_STEP: 16750
| > loss_text_ce: 0.04873709753155708 (0.04293744601376906)
| > loss_mel_ce: 3.3574013710021973 (3.3297421446152313)
| > loss: 0.8515346050262451 (0.84316989811925)
| > current_lr: 5e-06
| > step_time: 0.4348 (0.3745028200974552)
| > loader_time: 0.011 (0.009873052618392494)
 --> TIME: 2024-10-20 02:48:04 -- STEP: 3356/3361 -- GLOBAL_STEP: 16800
| > loss_text_ce: 0.04252580925822258 (0.042922385484290955)
| > loss_mel_ce: 3.7723355293273926 (3.328910979099294)
| > loss: 0.9537153244018555 (0.8429583415158176)
| > current_lr: 5e-06
| > step_time: 0.3307 (0.3744130439354778)
| > loader_time: 0.0092 (0.009873790451113442)
 > EVALUATION 
--> EVAL PERFORMANCE
| > avg_loader_time: 0.3775467693164783 (-0.0003484859261463913)
| > avg_loss_text_ce: 0.039982772161883694 (-0.0014271793426364754)
| > avg_loss_mel_ce: 3.101211165869108 (-0.06992721044889061)
| > avg_loss: 3.141193941075315 (-0.0713543943179551)
> BEST MODEL : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/best_model_16805.pth