> Training Environment: | > Backend: Torch | > Mixed precision: False | > Precision: float32 | > Current device: 0 | > Num. of GPUs: 1 | > Num. of CPUs: 4 | > Num. of Torch Threads: 1 | > Torch seed: 54321 | > Torch CUDNN: True | > Torch CUDNN deterministic: False | > Torch CUDNN benchmark: False | > Torch TF32 MatMul: False > Start Tensorboard: tensorboard --logdir=/kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b > Model has 522165080 parameters  > EPOCH: 0/5 --> /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b  > TRAINING (2024-10-19 23:36:51)   --> TIME: 2024-10-19 23:36:56 -- STEP: 0/3361 -- GLOBAL_STEP: 0 | > loss_text_ce: 0.2276160567998886 (0.2276160567998886) | > loss_mel_ce: 5.073640823364258 (5.073640823364258) | > loss: 1.3253141641616821 (1.3253141641616821) | > current_lr: 5e-06 | > step_time: 0.9855 (0.9855413436889648) | > loader_time: 3.9741 (3.974128007888794)  --> TIME: 2024-10-19 23:37:28 -- STEP: 50/3361 -- GLOBAL_STEP: 50 | > loss_text_ce: 0.1961478441953659 (0.21089269667863847) | > loss_mel_ce: 4.774014472961426 (5.021885404586793) | > loss: 1.2425405979156494 (1.3081945204734806) | > current_lr: 5e-06 | > step_time: 0.4324 (0.3791870260238647) | > loader_time: 0.0095 (0.014342236518859863)  --> TIME: 2024-10-19 23:38:00 -- STEP: 100/3361 -- GLOBAL_STEP: 100 | > loss_text_ce: 0.19649465382099152 (0.20336569979786873) | > loss_mel_ce: 4.587543487548828 (4.869443783760071) | > loss: 1.1960095167160034 (1.2682023656368255) | > current_lr: 5e-06 | > step_time: 0.2369 (0.37330648183822635) | > loader_time: 0.0147 (0.013298330307006835)  --> TIME: 2024-10-19 23:38:32 -- STEP: 150/3361 -- GLOBAL_STEP: 150 | > loss_text_ce: 0.17126446962356567 (0.195263701180617) | > loss_mel_ce: 4.586864471435547 (4.784979950586955) | > loss: 1.1895322799682617 (1.2450609103838604) | > current_lr: 5e-06 | > step_time: 0.3026 (0.37735692818959554) | > loader_time: 0.0143 (0.012361326217651366)  --> TIME: 2024-10-19 23:39:05 -- STEP: 200/3361 -- GLOBAL_STEP: 200 | > loss_text_ce: 0.1639341115951538 (0.18802625849843024) | > loss_mel_ce: 4.630383491516113 (4.733785247802734) | > loss: 1.1985794305801392 (1.2304528760910034) | > current_lr: 5e-06 | > step_time: 0.2442 (0.37673797488212585) | > loader_time: 0.0183 (0.011802190542221067)  --> TIME: 2024-10-19 23:39:36 -- STEP: 250/3361 -- GLOBAL_STEP: 250 | > loss_text_ce: 0.15474987030029297 (0.18128131020069121) | > loss_mel_ce: 4.447690963745117 (4.686841909408569) | > loss: 1.1506102085113525 (1.2170308060646058) | > current_lr: 5e-06 | > step_time: 0.3412 (0.37635364627838136) | > loader_time: 0.0101 (0.011550966262817382)  --> TIME: 2024-10-19 23:40:08 -- STEP: 300/3361 -- GLOBAL_STEP: 300 | > loss_text_ce: 0.1450122743844986 (0.17507749925057092) | > loss_mel_ce: 4.760715007781982 (4.6537174924214675) | > loss: 1.2264318466186523 (1.2071987493832905) | > current_lr: 5e-06 | > step_time: 0.2737 (0.3747914385795592) | > loader_time: 0.0089 (0.011329036553700772)  --> TIME: 2024-10-19 23:40:39 -- STEP: 350/3361 -- GLOBAL_STEP: 350 | > loss_text_ce: 0.13236135244369507 (0.1689298430936677) | > loss_mel_ce: 4.3863911628723145 (4.623111551829745) | > loss: 1.1296881437301636 (1.1980103506360738) | > current_lr: 5e-06 | > step_time: 0.4296 (0.3739049707140241) | > loader_time: 0.0098 (0.01116123471941268)  --> TIME: 2024-10-19 23:41:11 -- STEP: 400/3361 -- GLOBAL_STEP: 400 | > loss_text_ce: 0.11837068200111389 (0.1625033115968108) | > loss_mel_ce: 4.637821197509766 (4.5994156956672665) | > loss: 1.189047932624817 (1.1904797530174254) | > current_lr: 5e-06 | > step_time: 0.2738 (0.37281567573547364) | > loader_time: 0.0105 (0.010975514650344858)  --> TIME: 2024-10-19 23:41:42 -- STEP: 450/3361 -- GLOBAL_STEP: 450 | > loss_text_ce: 0.09800536185503006 (0.1564699565867583) | > loss_mel_ce: 4.699829578399658 (4.576134746339584) | > loss: 1.1994587182998657 (1.1831511762407079) | > current_lr: 5e-06 | > step_time: 0.4133 (0.3720897224214342) | > loader_time: 0.0109 (0.010839088228013789)  --> TIME: 2024-10-19 23:42:13 -- STEP: 500/3361 -- GLOBAL_STEP: 500 | > loss_text_ce: 0.11643479764461517 (0.15131162096560002) | > loss_mel_ce: 4.275294303894043 (4.55400635910034) | > loss: 1.09793221950531 (1.1763294947147358) | > current_lr: 5e-06 | > step_time: 0.2165 (0.3713303875923157) | > loader_time: 0.0096 (0.010830031394958504)  --> TIME: 2024-10-19 23:42:46 -- STEP: 550/3361 -- GLOBAL_STEP: 550 | > loss_text_ce: 0.09418845921754837 (0.14674024343490602) | > loss_mel_ce: 4.262049674987793 (4.536511275985024) | > loss: 1.089059591293335 (1.1708128790421906) | > current_lr: 5e-06 | > step_time: 0.4339 (0.3719711026278409) | > loader_time: 0.0107 (0.010789505785161803)  --> TIME: 2024-10-19 23:43:18 -- STEP: 600/3361 -- GLOBAL_STEP: 600 | > loss_text_ce: 0.1037636324763298 (0.14271454828480878) | > loss_mel_ce: 4.0221052169799805 (4.519321223497389) | > loss: 1.0314671993255615 (1.165508942604063) | > current_lr: 5e-06 | > step_time: 0.3561 (0.37222115119298294) | > loader_time: 0.011 (0.01072237730026246)  --> TIME: 2024-10-19 23:43:50 -- STEP: 650/3361 -- GLOBAL_STEP: 650 | > loss_text_ce: 0.10018370300531387 (0.13922331652962247) | > loss_mel_ce: 4.329763889312744 (4.505329917394195) | > loss: 1.1074868440628052 (1.1611383080482474) | > current_lr: 5e-06 | > step_time: 0.3411 (0.3732313658640932) | > loader_time: 0.0102 (0.010698353694035474)  --> TIME: 2024-10-19 23:44:22 -- STEP: 700/3361 -- GLOBAL_STEP: 700 | > loss_text_ce: 0.09295706450939178 (0.1359132102131844) | > loss_mel_ce: 4.329279899597168 (4.492836109910698) | > loss: 1.105559229850769 (1.1571873297010142) | > current_lr: 5e-06 | > step_time: 0.2642 (0.3730241230555941) | > loader_time: 0.0177 (0.010678539276123064)  --> TIME: 2024-10-19 23:44:54 -- STEP: 750/3361 -- GLOBAL_STEP: 750 | > loss_text_ce: 0.08847153186798096 (0.13301689797639846) | > loss_mel_ce: 4.319281101226807 (4.478802271842963) | > loss: 1.1019381284713745 (1.1529547921816505) | > current_lr: 5e-06 | > step_time: 0.4389 (0.37313773218790675) | > loader_time: 0.0108 (0.010637306849161803)  --> TIME: 2024-10-19 23:45:26 -- STEP: 800/3361 -- GLOBAL_STEP: 800 | > loss_text_ce: 0.10376894474029541 (0.13030017000623043) | > loss_mel_ce: 4.416518211364746 (4.466628861725337) | > loss: 1.130071759223938 (1.1492322574555873) | > current_lr: 5e-06 | > step_time: 0.3356 (0.373155618906021) | > loader_time: 0.01 (0.010589956939220446)  --> TIME: 2024-10-19 23:45:58 -- STEP: 850/3361 -- GLOBAL_STEP: 850 | > loss_text_ce: 0.09408652782440186 (0.12781955107169998) | > loss_mel_ce: 4.137832164764404 (4.455110899139858) | > loss: 1.057979702949524 (1.1457326119086317) | > current_lr: 5e-06 | > step_time: 0.2973 (0.37324726581573475) | > loader_time: 0.0089 (0.010562418769387655)  --> TIME: 2024-10-19 23:46:29 -- STEP: 900/3361 -- GLOBAL_STEP: 900 | > loss_text_ce: 0.08723258972167969 (0.12548734435604691) | > loss_mel_ce: 4.102336883544922 (4.445588976277252) | > loss: 1.0473923683166504 (1.1427690789434646) | > current_lr: 5e-06 | > step_time: 0.3471 (0.37272283368640463) | > loader_time: 0.0094 (0.010525485144721153)  --> TIME: 2024-10-19 23:47:01 -- STEP: 950/3361 -- GLOBAL_STEP: 950 | > loss_text_ce: 0.08267280459403992 (0.12352421604489039) | > loss_mel_ce: 4.262170314788818 (4.436302085926666) | > loss: 1.0862107276916504 (1.1399565746909694) | > current_lr: 5e-06 | > step_time: 0.4787 (0.372880862386603) | > loader_time: 0.0108 (0.010516446766100447)  --> TIME: 2024-10-19 23:47:33 -- STEP: 1000/3361 -- GLOBAL_STEP: 1000 | > loss_text_ce: 0.08907080441713333 (0.12161937542259706) | > loss_mel_ce: 4.300572395324707 (4.427557614326484) | > loss: 1.097410798072815 (1.1372942463755609) | > current_lr: 5e-06 | > step_time: 0.2458 (0.3725577991008757) | > loader_time: 0.0089 (0.010495249271392837) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_1000.pth  --> TIME: 2024-10-19 23:48:15 -- STEP: 1050/3361 -- GLOBAL_STEP: 1050 | > loss_text_ce: 0.0684773176908493 (0.11973775178194057) | > loss_mel_ce: 4.429049491882324 (4.419832839057566) | > loss: 1.1243816614151 (1.1348926466419584) | > current_lr: 5e-06 | > step_time: 0.4162 (0.37267442498888276) | > loader_time: 0.0106 (0.010481156848725833)  --> TIME: 2024-10-19 23:48:47 -- STEP: 1100/3361 -- GLOBAL_STEP: 1100 | > loss_text_ce: 0.08142425119876862 (0.11798097466880635) | > loss_mel_ce: 4.6108479499816895 (4.41094342123379) | > loss: 1.1730680465698242 (1.1322310978174208) | > current_lr: 5e-06 | > step_time: 0.3115 (0.3729662624272432) | > loader_time: 0.0095 (0.01046122507615524)  --> TIME: 2024-10-19 23:49:19 -- STEP: 1150/3361 -- GLOBAL_STEP: 1150 | > loss_text_ce: 0.08587858825922012 (0.11642520604574172) | > loss_mel_ce: 4.308182239532471 (4.402262189284622) | > loss: 1.0985151529312134 (1.129671847872112) | > current_lr: 5e-06 | > step_time: 0.3725 (0.3727190674906191) | > loader_time: 0.0103 (0.010423968771229626)  --> TIME: 2024-10-19 23:49:50 -- STEP: 1200/3361 -- GLOBAL_STEP: 1200 | > loss_text_ce: 0.08292829245328903 (0.11482038536419481) | > loss_mel_ce: 4.307161808013916 (4.394019900560389) | > loss: 1.097522497177124 (1.127210070540508) | > current_lr: 5e-06 | > step_time: 0.3411 (0.37206052283446) | > loader_time: 0.0117 (0.010386224389076241)  --> TIME: 2024-10-19 23:50:22 -- STEP: 1250/3361 -- GLOBAL_STEP: 1250 | > loss_text_ce: 0.07650288939476013 (0.11328100087642679) | > loss_mel_ce: 4.521561622619629 (4.386720856857313) | > loss: 1.1495161056518555 (1.1250004634380348) | > current_lr: 5e-06 | > step_time: 0.3442 (0.3723619564056397) | > loader_time: 0.0099 (0.010367479133605965)  --> TIME: 2024-10-19 23:50:54 -- STEP: 1300/3361 -- GLOBAL_STEP: 1300 | > loss_text_ce: 0.08411215245723724 (0.11187859488794445) | > loss_mel_ce: 4.311476707458496 (4.378952882289896) | > loss: 1.0988972187042236 (1.1227078686769207) | > current_lr: 5e-06 | > step_time: 0.3467 (0.37273642576657834) | > loader_time: 0.0101 (0.010353383284348715)  --> TIME: 2024-10-19 23:51:26 -- STEP: 1350/3361 -- GLOBAL_STEP: 1350 | > loss_text_ce: 0.07017528265714645 (0.11054472846565432) | > loss_mel_ce: 4.101247310638428 (4.371401038876297) | > loss: 1.0428556203842163 (1.1204864410559354) | > current_lr: 5e-06 | > step_time: 0.3274 (0.3727424639242669) | > loader_time: 0.009 (0.010344070858425569)  --> TIME: 2024-10-19 23:51:58 -- STEP: 1400/3361 -- GLOBAL_STEP: 1400 | > loss_text_ce: 0.07565019279718399 (0.10928024096148363) | > loss_mel_ce: 3.7528905868530273 (4.364196109771742) | > loss: 0.9571352005004883 (1.1183690869382463) | > current_lr: 5e-06 | > step_time: 0.3347 (0.372393846511841) | > loader_time: 0.0104 (0.01032195346696037)  --> TIME: 2024-10-19 23:52:29 -- STEP: 1450/3361 -- GLOBAL_STEP: 1450 | > loss_text_ce: 0.07104843109846115 (0.10808574935485585) | > loss_mel_ce: 4.024135112762451 (4.357662479630846) | > loss: 1.0237958431243896 (1.116437056722312) | > current_lr: 5e-06 | > step_time: 0.4162 (0.3723816107059349) | > loader_time: 0.0112 (0.010314793751157571)  --> TIME: 2024-10-19 23:53:02 -- STEP: 1500/3361 -- GLOBAL_STEP: 1500 | > loss_text_ce: 0.0696822851896286 (0.1068931187863152) | > loss_mel_ce: 4.51359748840332 (4.351274299780541) | > loss: 1.145819902420044 (1.114541854182878) | > current_lr: 5e-06 | > step_time: 0.2727 (0.37269472678502413) | > loader_time: 0.0105 (0.010296331087748217)  --> TIME: 2024-10-19 23:53:33 -- STEP: 1550/3361 -- GLOBAL_STEP: 1550 | > loss_text_ce: 0.07638457417488098 (0.10576395642132536) | > loss_mel_ce: 4.133301258087158 (4.344630276618483) | > loss: 1.0524214506149292 (1.112598557972137) | > current_lr: 5e-06 | > step_time: 0.421 (0.37251200337563806) | > loader_time: 0.009 (0.010285784506028706)  --> TIME: 2024-10-19 23:54:05 -- STEP: 1600/3361 -- GLOBAL_STEP: 1600 | > loss_text_ce: 0.07427678257226944 (0.10470495257293813) | > loss_mel_ce: 4.1771016120910645 (4.33880649805071) | > loss: 1.0628446340560913 (1.1108778622746458) | > current_lr: 5e-06 | > step_time: 0.3331 (0.37268786862492576) | > loader_time: 0.0117 (0.010273052901029596)  --> TIME: 2024-10-19 23:54:38 -- STEP: 1650/3361 -- GLOBAL_STEP: 1650 | > loss_text_ce: 0.0688512995839119 (0.10366868965779297) | > loss_mel_ce: 4.372562408447266 (4.333273339271568) | > loss: 1.1103534698486328 (1.10923550670797) | > current_lr: 5e-06 | > step_time: 0.3577 (0.3731871593359746) | > loader_time: 0.0107 (0.010268038836392497)  --> TIME: 2024-10-19 23:55:10 -- STEP: 1700/3361 -- GLOBAL_STEP: 1700 | > loss_text_ce: 0.06728693842887878 (0.1026264984883807) | > loss_mel_ce: 3.8259117603302 (4.327509120913133) | > loss: 0.9732996821403503 (1.1075339043491004) | > current_lr: 5e-06 | > step_time: 0.2424 (0.3731728926826927) | > loader_time: 0.0093 (0.010254677323734068)  --> TIME: 2024-10-19 23:55:43 -- STEP: 1750/3361 -- GLOBAL_STEP: 1750 | > loss_text_ce: 0.07139215618371964 (0.1016620131390436) | > loss_mel_ce: 4.0196332931518555 (4.321489932332737) | > loss: 1.0227563381195068 (1.105787986108232) | > current_lr: 5e-06 | > step_time: 0.3334 (0.37345534951346276) | > loader_time: 0.0088 (0.010236183302743103)  --> TIME: 2024-10-19 23:56:15 -- STEP: 1800/3361 -- GLOBAL_STEP: 1800 | > loss_text_ce: 0.06817279011011124 (0.10072386607320778) | > loss_mel_ce: 4.305867671966553 (4.3159451058176) | > loss: 1.0935101509094238 (1.1041672426793276) | > current_lr: 5e-06 | > step_time: 0.2647 (0.3735462660259672) | > loader_time: 0.0089 (0.01022007279925877)  --> TIME: 2024-10-19 23:56:47 -- STEP: 1850/3361 -- GLOBAL_STEP: 1850 | > loss_text_ce: 0.06191863492131233 (0.0997890573478229) | > loss_mel_ce: 4.1208672523498535 (4.310428036225824) | > loss: 1.045696496963501 (1.102554273347594) | > current_lr: 5e-06 | > step_time: 0.341 (0.37339078568123496) | > loader_time: 0.0094 (0.010211075576576035)  --> TIME: 2024-10-19 23:57:18 -- STEP: 1900/3361 -- GLOBAL_STEP: 1900 | > loss_text_ce: 0.06634141504764557 (0.09888596163376387) | > loss_mel_ce: 4.26849365234375 (4.304874293176769) | > loss: 1.0837087631225586 (1.1009400636585107) | > current_lr: 5e-06 | > step_time: 0.3123 (0.3732277792378477) | > loader_time: 0.0107 (0.010206241356699097)  --> TIME: 2024-10-19 23:57:50 -- STEP: 1950/3361 -- GLOBAL_STEP: 1950 | > loss_text_ce: 0.07070682942867279 (0.09804161607837071) | > loss_mel_ce: 4.126957416534424 (4.298485514934264) | > loss: 1.0494160652160645 (1.099131782666228) | > current_lr: 5e-06 | > step_time: 0.4269 (0.3732747751627214) | > loader_time: 0.0108 (0.010196818449558363)  --> TIME: 2024-10-19 23:58:22 -- STEP: 2000/3361 -- GLOBAL_STEP: 2000 | > loss_text_ce: 0.06333263963460922 (0.09720117196254438) | > loss_mel_ce: 3.9823367595672607 (4.293063371539132) | > loss: 1.0114173889160156 (1.0975661356747124) | > current_lr: 5e-06 | > step_time: 0.3422 (0.3733924993276597) | > loader_time: 0.0114 (0.010184644460678108) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_2000.pth  --> TIME: 2024-10-19 23:59:04 -- STEP: 2050/3361 -- GLOBAL_STEP: 2050 | > loss_text_ce: 0.07292477041482925 (0.09640612774324132) | > loss_mel_ce: 3.918966770172119 (4.286913629973823) | > loss: 0.9979729056358337 (1.0958299392316373) | > current_lr: 5e-06 | > step_time: 0.3183 (0.3732672268006861) | > loader_time: 0.01 (0.01018628969425109)  --> TIME: 2024-10-19 23:59:36 -- STEP: 2100/3361 -- GLOBAL_STEP: 2100 | > loss_text_ce: 0.05803373083472252 (0.09563219899755154) | > loss_mel_ce: 4.1672043800354 (4.281224559375234) | > loss: 1.0563095808029175 (1.0942141895350932) | > current_lr: 5e-06 | > step_time: 0.317 (0.37334919940857664) | > loader_time: 0.0096 (0.010179950850350523)  --> TIME: 2024-10-20 00:00:09 -- STEP: 2150/3361 -- GLOBAL_STEP: 2150 | > loss_text_ce: 0.07027828693389893 (0.09487137659517834) | > loss_mel_ce: 4.074728488922119 (4.276105512020217) | > loss: 1.0362516641616821 (1.0927442217982068) | > current_lr: 5e-06 | > step_time: 0.358 (0.37359854487485666) | > loader_time: 0.0108 (0.010167785023534022)  --> TIME: 2024-10-20 00:00:40 -- STEP: 2200/3361 -- GLOBAL_STEP: 2200 | > loss_text_ce: 0.06147142872214317 (0.09415466819127856) | > loss_mel_ce: 4.060927391052246 (4.2708785486221466) | > loss: 1.0305997133255005 (1.0912583039565495) | > current_lr: 5e-06 | > step_time: 0.3124 (0.37341777844862495) | > loader_time: 0.0088 (0.010151374773545701)  --> TIME: 2024-10-20 00:01:12 -- STEP: 2250/3361 -- GLOBAL_STEP: 2250 | > loss_text_ce: 0.0592532679438591 (0.09346097443997865) | > loss_mel_ce: 4.088995933532715 (4.26577778604297) | > loss: 1.0370622873306274 (1.089809689680733) | > current_lr: 5e-06 | > step_time: 0.3943 (0.3732477170096502) | > loader_time: 0.0093 (0.010148488998413085)  --> TIME: 2024-10-20 00:01:45 -- STEP: 2300/3361 -- GLOBAL_STEP: 2300 | > loss_text_ce: 0.05973433703184128 (0.0927704642372935) | > loss_mel_ce: 4.142950057983398 (4.261306318303825) | > loss: 1.050671100616455 (1.0885191953182198) | > current_lr: 5e-06 | > step_time: 0.316 (0.3735742027863211) | > loader_time: 0.0106 (0.010143468172653866)  --> TIME: 2024-10-20 00:02:17 -- STEP: 2350/3361 -- GLOBAL_STEP: 2350 | > loss_text_ce: 0.0632677748799324 (0.09211754334258279) | > loss_mel_ce: 3.9679794311523438 (4.256112595010319) | > loss: 1.0078117847442627 (1.0870575342787052) | > current_lr: 5e-06 | > step_time: 0.4684 (0.373737255867491) | > loader_time: 0.0102 (0.01013411846566708)  --> TIME: 2024-10-20 00:02:48 -- STEP: 2400/3361 -- GLOBAL_STEP: 2400 | > loss_text_ce: 0.06617214530706406 (0.09147963946840423) | > loss_mel_ce: 4.05083703994751 (4.251065555413567) | > loss: 1.029252290725708 (1.085636298432944) | > current_lr: 5e-06 | > step_time: 0.2359 (0.37363693376382134) | > loader_time: 0.0095 (0.010126104454199468)  --> TIME: 2024-10-20 00:03:20 -- STEP: 2450/3361 -- GLOBAL_STEP: 2450 | > loss_text_ce: 0.05000381916761398 (0.0908709145945555) | > loss_mel_ce: 3.871734619140625 (4.246652321036983) | > loss: 0.9804345965385437 (1.0843808086307658) | > current_lr: 5e-06 | > step_time: 0.3701 (0.37357356207711284) | > loader_time: 0.0088 (0.010115676704718139)  --> TIME: 2024-10-20 00:03:52 -- STEP: 2500/3361 -- GLOBAL_STEP: 2500 | > loss_text_ce: 0.06191703677177429 (0.09026267003268017) | > loss_mel_ce: 4.080785274505615 (4.241838634967805) | > loss: 1.0356755256652832 (1.0830253259420368) | > current_lr: 5e-06 | > step_time: 0.2731 (0.37364912557601854) | > loader_time: 0.0096 (0.010112837028503414)  --> TIME: 2024-10-20 00:04:25 -- STEP: 2550/3361 -- GLOBAL_STEP: 2550 | > loss_text_ce: 0.06187329813838005 (0.08967712040187109) | > loss_mel_ce: 3.9666688442230225 (4.237735574292203) | > loss: 1.0071355104446411 (1.081853173433563) | > current_lr: 5e-06 | > step_time: 0.3046 (0.37373014066733484) | > loader_time: 0.009 (0.010112988154093427)  --> TIME: 2024-10-20 00:04:57 -- STEP: 2600/3361 -- GLOBAL_STEP: 2600 | > loss_text_ce: 0.06246669963002205 (0.08912380175951591) | > loss_mel_ce: 3.9966068267822266 (4.234155957056928) | > loss: 1.014768362045288 (1.0808199393978473) | > current_lr: 5e-06 | > step_time: 0.3483 (0.3738965059243708) | > loader_time: 0.0102 (0.010113027279193589)  --> TIME: 2024-10-20 00:05:29 -- STEP: 2650/3361 -- GLOBAL_STEP: 2650 | > loss_text_ce: 0.0602920725941658 (0.08857007859872204) | > loss_mel_ce: 3.8786299228668213 (4.230075319668037) | > loss: 0.9847304821014404 (1.0796613493055658) | > current_lr: 5e-06 | > step_time: 0.3352 (0.3738692178366311) | > loader_time: 0.0092 (0.010110449071200393)  --> TIME: 2024-10-20 00:06:01 -- STEP: 2700/3361 -- GLOBAL_STEP: 2700 | > loss_text_ce: 0.060905251652002335 (0.08802893280845003) | > loss_mel_ce: 4.000962257385254 (4.225500499760666) | > loss: 1.0154669284820557 (1.0783823577783709) | > current_lr: 5e-06 | > step_time: 0.3291 (0.37374789873758885) | > loader_time: 0.0107 (0.010111093609421347)  --> TIME: 2024-10-20 00:06:33 -- STEP: 2750/3361 -- GLOBAL_STEP: 2750 | > loss_text_ce: 0.06092409789562225 (0.087514858617024) | > loss_mel_ce: 3.8547213077545166 (4.221181369434707) | > loss: 0.9789113402366638 (1.077174056638369) | > current_lr: 5e-06 | > step_time: 0.4815 (0.3739039562398729) | > loader_time: 0.0093 (0.010105457912791861)  --> TIME: 2024-10-20 00:07:05 -- STEP: 2800/3361 -- GLOBAL_STEP: 2800 | > loss_text_ce: 0.06143264099955559 (0.08701184531939885) | > loss_mel_ce: 4.019796848297119 (4.216923976199974) | > loss: 1.0203074216842651 (1.0759839550299288) | > current_lr: 5e-06 | > step_time: 0.2735 (0.3738308864831916) | > loader_time: 0.0091 (0.01010349988937378)  --> TIME: 2024-10-20 00:07:36 -- STEP: 2850/3361 -- GLOBAL_STEP: 2850 | > loss_text_ce: 0.06067034229636192 (0.08651230146356849) | > loss_mel_ce: 3.9777588844299316 (4.2130854128954764) | > loss: 1.0096073150634766 (1.0748994283090547) | > current_lr: 5e-06 | > step_time: 0.3746 (0.37365791663788833) | > loader_time: 0.0088 (0.010102545671295695)  --> TIME: 2024-10-20 00:08:09 -- STEP: 2900/3361 -- GLOBAL_STEP: 2900 | > loss_text_ce: 0.0590413473546505 (0.08602534580975783) | > loss_mel_ce: 3.9361588954925537 (4.208741374755734) | > loss: 0.9988000392913818 (1.0736916798147638) | > current_lr: 5e-06 | > step_time: 0.245 (0.3738863233862244) | > loader_time: 0.0129 (0.010097931253498997)  --> TIME: 2024-10-20 00:08:42 -- STEP: 2950/3361 -- GLOBAL_STEP: 2950 | > loss_text_ce: 0.05851312354207039 (0.08556444178445886) | > loss_mel_ce: 3.8445589542388916 (4.204549880108596) | > loss: 0.9757680296897888 (1.0725285801240934) | > current_lr: 5e-06 | > step_time: 0.3277 (0.3740873582484351) | > loader_time: 0.0091 (0.010090556467993783)  --> TIME: 2024-10-20 00:09:13 -- STEP: 3000/3361 -- GLOBAL_STEP: 3000 | > loss_text_ce: 0.052952103316783905 (0.08509168772771973) | > loss_mel_ce: 3.7599077224731445 (4.200607080777491) | > loss: 0.9532149434089661 (1.0714246917963006) | > current_lr: 5e-06 | > step_time: 0.2744 (0.37406464068094814) | > loader_time: 0.0097 (0.010087447325388594) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_3000.pth  --> TIME: 2024-10-20 00:09:56 -- STEP: 3050/3361 -- GLOBAL_STEP: 3050 | > loss_text_ce: 0.05916506052017212 (0.08463779314375335) | > loss_mel_ce: 3.8810315132141113 (4.196892655091211) | > loss: 0.9850491285324097 (1.0703826116929271) | > current_lr: 5e-06 | > step_time: 0.3444 (0.37403536546425664) | > loader_time: 0.0089 (0.010088322827073403)  --> TIME: 2024-10-20 00:10:29 -- STEP: 3100/3361 -- GLOBAL_STEP: 3100 | > loss_text_ce: 0.06803478300571442 (0.08422918403581285) | > loss_mel_ce: 3.9018285274505615 (4.192189099711762) | > loss: 0.9924658536911011 (1.0691045704772382) | > current_lr: 5e-06 | > step_time: 0.3126 (0.3742765197446262) | > loader_time: 0.0108 (0.010089200004454596)  --> TIME: 2024-10-20 00:11:00 -- STEP: 3150/3361 -- GLOBAL_STEP: 3150 | > loss_text_ce: 0.055581387132406235 (0.08381401562974586) | > loss_mel_ce: 3.712331533432007 (4.187636630345909) | > loss: 0.9419782161712646 (1.0678626610740767) | > current_lr: 5e-06 | > step_time: 0.2662 (0.37410105145166717) | > loader_time: 0.0104 (0.010091308109344013)  --> TIME: 2024-10-20 00:11:32 -- STEP: 3200/3361 -- GLOBAL_STEP: 3200 | > loss_text_ce: 0.05516338720917702 (0.08340078805340473) | > loss_mel_ce: 3.8296306133270264 (4.183248083516961) | > loss: 0.9711984992027283 (1.0666622174717475) | > current_lr: 5e-06 | > step_time: 0.3456 (0.37414847671985557) | > loader_time: 0.0096 (0.010088659971952449)  --> TIME: 2024-10-20 00:12:04 -- STEP: 3250/3361 -- GLOBAL_STEP: 3250 | > loss_text_ce: 0.058774255216121674 (0.08300397763917099) | > loss_mel_ce: 4.043879508972168 (4.179145359039312) | > loss: 1.0256634950637817 (1.0655373337635616) | > current_lr: 5e-06 | > step_time: 0.3054 (0.3741027381603527) | > loader_time: 0.0098 (0.010083743462195774)  --> TIME: 2024-10-20 00:12:36 -- STEP: 3300/3361 -- GLOBAL_STEP: 3300 | > loss_text_ce: 0.05571791157126427 (0.08262370134393393) | > loss_mel_ce: 4.039697647094727 (4.175351190856015) | > loss: 1.0238538980484009 (1.0644937225724693) | > current_lr: 5e-06 | > step_time: 0.355 (0.37421785773652905) | > loader_time: 0.0105 (0.010079322367003481)  --> TIME: 2024-10-20 00:13:08 -- STEP: 3350/3361 -- GLOBAL_STEP: 3350 | > loss_text_ce: 0.0516965389251709 (0.08223508200761116) | > loss_mel_ce: 4.055713653564453 (4.171607753767902) | > loss: 1.0268526077270508 (1.0634607085541095) | > current_lr: 5e-06 | > step_time: 0.4114 (0.37421482968686165) | > loader_time: 0.0087 (0.010072118702219504)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.3817212197088426 (+0) | > avg_loss_text_ce: 0.05487057370364025 (+0) | > avg_loss_mel_ce: 3.7456871719770533 (+0) | > avg_loss: 3.8005577415548344 (+0)  > EPOCH: 1/5 --> /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b  > TRAINING (2024-10-20 00:15:40)   --> TIME: 2024-10-20 00:16:06 -- STEP: 39/3361 -- GLOBAL_STEP: 3400 | > loss_text_ce: 0.05803585425019264 (0.05740316842611019) | > loss_mel_ce: 4.036578178405762 (3.9442618443415713) | > loss: 1.023653507232666 (1.0004162574425723) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4891 (0.36824990541507036) | > loader_time: 0.0111 (0.0100095883393899)  --> TIME: 2024-10-20 00:16:38 -- STEP: 89/3361 -- GLOBAL_STEP: 3450 | > loss_text_ce: 0.05387208238244057 (0.057160088897086264) | > loss_mel_ce: 3.8522324562072754 (3.9176201766796326) | > loss: 0.976526141166687 (0.9936950695648622) | > current_lr: 5e-06 | > step_time: 0.3078 (0.3701958361636386) | > loader_time: 0.0101 (0.00997726836901033)  --> TIME: 2024-10-20 00:17:10 -- STEP: 139/3361 -- GLOBAL_STEP: 3500 | > loss_text_ce: 0.05204734206199646 (0.05694234341704588) | > loss_mel_ce: 4.180244445800781 (3.895403874006203) | > loss: 1.0580729246139526 (0.9880865542151087) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3894 (0.3720308207779479) | > loader_time: 0.0087 (0.009898750044459064)  --> TIME: 2024-10-20 00:17:42 -- STEP: 189/3361 -- GLOBAL_STEP: 3550 | > loss_text_ce: 0.05554436519742012 (0.05678758705182681) | > loss_mel_ce: 4.228486061096191 (3.8946746341765874) | > loss: 1.0710076093673706 (0.9878655549079652) | > current_lr: 5e-06 | > step_time: 0.3746 (0.37256827934709175) | > loader_time: 0.0099 (0.00987515752277678)  --> TIME: 2024-10-20 00:18:13 -- STEP: 239/3361 -- GLOBAL_STEP: 3600 | > loss_text_ce: 0.05677724629640579 (0.05658898376333664) | > loss_mel_ce: 3.8501322269439697 (3.890875055201383) | > loss: 0.9767273664474487 (0.9868660100334359) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.34 (0.3716044445915701) | > loader_time: 0.0093 (0.009896839014157102)  --> TIME: 2024-10-20 00:18:44 -- STEP: 289/3361 -- GLOBAL_STEP: 3650 | > loss_text_ce: 0.056557197123765945 (0.056536512623402486) | > loss_mel_ce: 3.79600191116333 (3.8830668728244344) | > loss: 0.9631397724151611 (0.9849008476445419) | > current_lr: 5e-06 | > step_time: 0.3711 (0.36885856262127814) | > loader_time: 0.0107 (0.009862431192893067)  --> TIME: 2024-10-20 00:19:17 -- STEP: 339/3361 -- GLOBAL_STEP: 3700 | > loss_text_ce: 0.054546210914850235 (0.056364989078502035) | > loss_mel_ce: 3.885460615158081 (3.8856594998576295) | > loss: 0.9850016832351685 (0.9855061231461246) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.5478 (0.3709612870286699) | > loader_time: 0.0111 (0.009907801242704593)  --> TIME: 2024-10-20 00:19:48 -- STEP: 389/3361 -- GLOBAL_STEP: 3750 | > loss_text_ce: 0.05074138566851616 (0.056212213034035306) | > loss_mel_ce: 3.9127519130706787 (3.887668692360807) | > loss: 0.9908733367919922 (0.9859702276080311) | > current_lr: 5e-06 | > step_time: 0.428 (0.3711606475565304) | > loader_time: 0.0108 (0.009874057647195157)  --> TIME: 2024-10-20 00:20:20 -- STEP: 439/3361 -- GLOBAL_STEP: 3800 | > loss_text_ce: 0.05833661928772926 (0.05614948455999544) | > loss_mel_ce: 3.801664352416992 (3.884764766910353) | > loss: 0.9650002717971802 (0.9852285640386351) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3721 (0.3714690550584727) | > loader_time: 0.0103 (0.009873224010771672)  --> TIME: 2024-10-20 00:20:53 -- STEP: 489/3361 -- GLOBAL_STEP: 3850 | > loss_text_ce: 0.05502821505069733 (0.05603404230379132) | > loss_mel_ce: 3.728093385696411 (3.880742844872192) | > loss: 0.9457803964614868 (0.9841942235004683) | > current_lr: 5e-06 | > step_time: 0.3332 (0.3729910514106048) | > loader_time: 0.0089 (0.009863832976920481)  --> TIME: 2024-10-20 00:21:26 -- STEP: 539/3361 -- GLOBAL_STEP: 3900 | > loss_text_ce: 0.05435081198811531 (0.056033372885939806) | > loss_mel_ce: 3.737413167953491 (3.8785685820570683) | > loss: 0.94794100522995 (0.9836504911668668) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4252 (0.37414260299838303) | > loader_time: 0.0095 (0.00982684460995592)  --> TIME: 2024-10-20 00:21:57 -- STEP: 589/3361 -- GLOBAL_STEP: 3950 | > loss_text_ce: 0.052592385560274124 (0.056006750961566404) | > loss_mel_ce: 3.746580123901367 (3.8754911434873054) | > loss: 0.9497931003570557 (0.9828744762417416) | > current_lr: 5e-06 | > step_time: 0.3749 (0.37371972825612176) | > loader_time: 0.0095 (0.009833886384559211)  --> TIME: 2024-10-20 00:22:29 -- STEP: 639/3361 -- GLOBAL_STEP: 4000 | > loss_text_ce: 0.05487600341439247 (0.05590341322368858) | > loss_mel_ce: 3.9473564624786377 (3.872275245021766) | > loss: 1.0005581378936768 (0.982044667052178) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3879 (0.3738370978962831) | > loader_time: 0.0092 (0.009821679857042106) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_4000.pth  --> TIME: 2024-10-20 00:23:14 -- STEP: 689/3361 -- GLOBAL_STEP: 4050 | > loss_text_ce: 0.05411239340901375 (0.055867320956616326) | > loss_mel_ce: 3.7576539516448975 (3.8687809928927264) | > loss: 0.9529415965080261 (0.9811620808305173) | > current_lr: 5e-06 | > step_time: 0.4296 (0.37478768046950717) | > loader_time: 0.0103 (0.009839309490297987)  --> TIME: 2024-10-20 00:23:46 -- STEP: 739/3361 -- GLOBAL_STEP: 4100 | > loss_text_ce: 0.0570734404027462 (0.055781693188965066) | > loss_mel_ce: 3.8863158226013184 (3.867509481226156) | > loss: 0.9858472943305969 (0.9808227957827473) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3456 (0.3744364284856069) | > loader_time: 0.0095 (0.009838796920156942)  --> TIME: 2024-10-20 00:24:17 -- STEP: 789/3361 -- GLOBAL_STEP: 4150 | > loss_text_ce: 0.05545391887426376 (0.05572522812167652) | > loss_mel_ce: 3.894559860229492 (3.8668422457232072) | > loss: 0.9875034689903259 (0.9806418704442652) | > current_lr: 5e-06 | > step_time: 0.3955 (0.3739068774064984) | > loader_time: 0.0098 (0.009847505917567279)  --> TIME: 2024-10-20 00:24:49 -- STEP: 839/3361 -- GLOBAL_STEP: 4200 | > loss_text_ce: 0.048767924308776855 (0.05559879375779301) | > loss_mel_ce: 3.6436502933502197 (3.865561713478989) | > loss: 0.9231045246124268 (0.9802901287328927) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4775 (0.37385038369034396) | > loader_time: 0.0108 (0.009849785905913051)  --> TIME: 2024-10-20 00:25:22 -- STEP: 889/3361 -- GLOBAL_STEP: 4250 | > loss_text_ce: 0.055653173476457596 (0.05556447896163562) | > loss_mel_ce: 3.7202305793762207 (3.8628411008751162) | > loss: 0.9439709186553955 (0.9796013970879254) | > current_lr: 5e-06 | > step_time: 0.3718 (0.3747204725868386) | > loader_time: 0.009 (0.009871045435507483)  --> TIME: 2024-10-20 00:25:54 -- STEP: 939/3361 -- GLOBAL_STEP: 4300 | > loss_text_ce: 0.05329810455441475 (0.05552343320779906) | > loss_mel_ce: 3.8362557888031006 (3.859880526860555) | > loss: 0.9723884463310242 (0.978850992215962) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.461 (0.37450057426834493) | > loader_time: 0.0095 (0.009879123669462858)  --> TIME: 2024-10-20 00:26:26 -- STEP: 989/3361 -- GLOBAL_STEP: 4350 | > loss_text_ce: 0.05188177898526192 (0.05547151509653569) | > loss_mel_ce: 3.7463533878326416 (3.858084853908049) | > loss: 0.9495587944984436 (0.978389094403829) | > current_lr: 5e-06 | > step_time: 0.3389 (0.37451345855473733) | > loader_time: 0.01 (0.009874366291613155)  --> TIME: 2024-10-20 00:26:57 -- STEP: 1039/3361 -- GLOBAL_STEP: 4400 | > loss_text_ce: 0.05315755307674408 (0.055404065257517385) | > loss_mel_ce: 4.059134006500244 (3.8563873499373735) | > loss: 1.0280728340148926 (0.9779478560512862) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.463 (0.37416290777938876) | > loader_time: 0.0101 (0.009874617169979576)  --> TIME: 2024-10-20 00:27:29 -- STEP: 1089/3361 -- GLOBAL_STEP: 4450 | > loss_text_ce: 0.05398701876401901 (0.05530707812136854) | > loss_mel_ce: 3.651801586151123 (3.8551332309116892) | > loss: 0.9264471530914307 (0.977610079491215) | > current_lr: 5e-06 | > step_time: 0.3285 (0.37405674845062337) | > loader_time: 0.0086 (0.009866176776868487)  --> TIME: 2024-10-20 00:28:01 -- STEP: 1139/3361 -- GLOBAL_STEP: 4500 | > loss_text_ce: 0.05266125500202179 (0.05522818537633389) | > loss_mel_ce: 3.816060781478882 (3.8524879304016757) | > loss: 0.9671804904937744 (0.976929031044271) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4832 (0.3738998722046274) | > loader_time: 0.0114 (0.00985939969505733)  --> TIME: 2024-10-20 00:28:33 -- STEP: 1189/3361 -- GLOBAL_STEP: 4550 | > loss_text_ce: 0.050825513899326324 (0.055144575364179604) | > loss_mel_ce: 3.6082472801208496 (3.849461859068618) | > loss: 0.9147682189941406 (0.9761516108444701) | > current_lr: 5e-06 | > step_time: 0.2678 (0.37407235281120543) | > loader_time: 0.0097 (0.009851126434223515)  --> TIME: 2024-10-20 00:29:04 -- STEP: 1239/3361 -- GLOBAL_STEP: 4600 | > loss_text_ce: 0.056511420756578445 (0.05509090997561317) | > loss_mel_ce: 3.6945228576660156 (3.846852075293528) | > loss: 0.9377585649490356 (0.9754857485384784) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.5009 (0.3736923908975647) | > loader_time: 0.0092 (0.009838911099622483)  --> TIME: 2024-10-20 00:29:37 -- STEP: 1289/3361 -- GLOBAL_STEP: 4650 | > loss_text_ce: 0.057481300085783005 (0.055016553643556204) | > loss_mel_ce: 3.841536045074463 (3.84445180201179) | > loss: 0.9747543334960938 (0.9748670911492998) | > current_lr: 5e-06 | > step_time: 0.3056 (0.3741835676671183) | > loader_time: 0.0101 (0.009838919199373113)  --> TIME: 2024-10-20 00:30:09 -- STEP: 1339/3361 -- GLOBAL_STEP: 4700 | > loss_text_ce: 0.05459923669695854 (0.05495443729832599) | > loss_mel_ce: 3.5833394527435303 (3.841443535282685) | > loss: 0.9094846844673157 (0.9740994955017284) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4246 (0.3741715670166128) | > loader_time: 0.0093 (0.009824278107145276)  --> TIME: 2024-10-20 00:30:41 -- STEP: 1389/3361 -- GLOBAL_STEP: 4750 | > loss_text_ce: 0.05468934029340744 (0.05488722276846394) | > loss_mel_ce: 3.722651958465576 (3.838512401258113) | > loss: 0.9443353414535522 (0.9733499083010905) | > current_lr: 5e-06 | > step_time: 0.3922 (0.3743942389821215) | > loader_time: 0.0105 (0.009819968644451805)  --> TIME: 2024-10-20 00:31:13 -- STEP: 1439/3361 -- GLOBAL_STEP: 4800 | > loss_text_ce: 0.0551159530878067 (0.05482199154095237) | > loss_mel_ce: 3.8832504749298096 (3.8361284149612627) | > loss: 0.9845916032791138 (0.9727376038370404) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3457 (0.3745205248288596) | > loader_time: 0.0105 (0.009818825479842789)  --> TIME: 2024-10-20 00:31:46 -- STEP: 1489/3361 -- GLOBAL_STEP: 4850 | > loss_text_ce: 0.05104149132966995 (0.054750571161248346) | > loss_mel_ce: 3.526203155517578 (3.8339387963809286) | > loss: 0.8943111896514893 (0.9721723442673282) | > current_lr: 5e-06 | > step_time: 0.3055 (0.37465434349808124) | > loader_time: 0.0093 (0.009820129824933629)  --> TIME: 2024-10-20 00:32:17 -- STEP: 1539/3361 -- GLOBAL_STEP: 4900 | > loss_text_ce: 0.0571925975382328 (0.054688119172778805) | > loss_mel_ce: 3.850430965423584 (3.830406569598014) | > loss: 0.976905882358551 (0.9712736745636675) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.2878 (0.3744973553861402) | > loader_time: 0.0093 (0.009813829704567234)  --> TIME: 2024-10-20 00:32:50 -- STEP: 1589/3361 -- GLOBAL_STEP: 4950 | > loss_text_ce: 0.05071800947189331 (0.054632533395567937) | > loss_mel_ce: 3.786651611328125 (3.8284057811643435) | > loss: 0.9593424201011658 (0.9707595809515802) | > current_lr: 5e-06 | > step_time: 0.4317 (0.37486315898883243) | > loader_time: 0.0091 (0.009814189919291092)  --> TIME: 2024-10-20 00:33:21 -- STEP: 1639/3361 -- GLOBAL_STEP: 5000 | > loss_text_ce: 0.05272924154996872 (0.05457410791603582) | > loss_mel_ce: 3.7636656761169434 (3.8264297995354943) | > loss: 0.9540987014770508 (0.9702509792204526) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4982 (0.3746423821684933) | > loader_time: 0.0106 (0.009817314555253682) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_5000.pth  --> TIME: 2024-10-20 00:34:04 -- STEP: 1689/3361 -- GLOBAL_STEP: 5050 | > loss_text_ce: 0.053886573761701584 (0.05451192842468909) | > loss_mel_ce: 3.6620142459869385 (3.8244482417866066) | > loss: 0.9289752244949341 (0.9697400449365465) | > current_lr: 5e-06 | > step_time: 0.3776 (0.3748740332588208) | > loader_time: 0.0088 (0.009828670551397162)  --> TIME: 2024-10-20 00:34:37 -- STEP: 1739/3361 -- GLOBAL_STEP: 5100 | > loss_text_ce: 0.05141803249716759 (0.0544602110759152) | > loss_mel_ce: 3.5928444862365723 (3.820919352608758) | > loss: 0.9110656380653381 (0.9688448931994008) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.2895 (0.37512943811838884) | > loader_time: 0.0088 (0.009832062482696757)  --> TIME: 2024-10-20 00:35:08 -- STEP: 1789/3361 -- GLOBAL_STEP: 5150 | > loss_text_ce: 0.056414783000946045 (0.054400483986530426) | > loss_mel_ce: 3.5568387508392334 (3.818622766625888) | > loss: 0.903313398361206 (0.9682558148582932) | > current_lr: 5e-06 | > step_time: 0.3325 (0.37502415170877446) | > loader_time: 0.0103 (0.009836978923015046)  --> TIME: 2024-10-20 00:35:40 -- STEP: 1839/3361 -- GLOBAL_STEP: 5200 | > loss_text_ce: 0.05202397704124451 (0.05435034733563417) | > loss_mel_ce: 3.6310930252075195 (3.816511524054199) | > loss: 0.9207792282104492 (0.9677154700868865) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.334 (0.374689039813234) | > loader_time: 0.0086 (0.00983762896663796)  --> TIME: 2024-10-20 00:36:12 -- STEP: 1889/3361 -- GLOBAL_STEP: 5250 | > loss_text_ce: 0.049998633563518524 (0.05429816679105106) | > loss_mel_ce: 3.725067138671875 (3.814177731599298) | > loss: 0.9437664151191711 (0.9671189767018142) | > current_lr: 5e-06 | > step_time: 0.432 (0.3749042445859812) | > loader_time: 0.0097 (0.00983408887024087)  --> TIME: 2024-10-20 00:36:45 -- STEP: 1939/3361 -- GLOBAL_STEP: 5300 | > loss_text_ce: 0.05160053074359894 (0.05423524983845892) | > loss_mel_ce: 3.95269513130188 (3.8125401895768007) | > loss: 1.001073956489563 (0.9666938619124268) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4847 (0.3752489017419681) | > loader_time: 0.0089 (0.009828678537361885)  --> TIME: 2024-10-20 00:37:16 -- STEP: 1989/3361 -- GLOBAL_STEP: 5350 | > loss_text_ce: 0.05233506113290787 (0.05417726464595789) | > loss_mel_ce: 4.050232887268066 (3.8093292049093423) | > loss: 1.0256420373916626 (0.9658766194106707) | > current_lr: 5e-06 | > step_time: 0.303 (0.3747676720626393) | > loader_time: 0.0105 (0.009829002388042678)  --> TIME: 2024-10-20 00:37:48 -- STEP: 2039/3361 -- GLOBAL_STEP: 5400 | > loss_text_ce: 0.05259113013744354 (0.054126247990183986) | > loss_mel_ce: 3.6885979175567627 (3.8070934112309356) | > loss: 0.9352972507476807 (0.9653049167492274) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4207 (0.3747711464609217) | > loader_time: 0.0111 (0.009824151090104651)  --> TIME: 2024-10-20 00:38:19 -- STEP: 2089/3361 -- GLOBAL_STEP: 5450 | > loss_text_ce: 0.04826216772198677 (0.054073872357649676) | > loss_mel_ce: 3.61075496673584 (3.8050423009707175) | > loss: 0.9147542715072632 (0.9647790451492629) | > current_lr: 5e-06 | > step_time: 0.4413 (0.37473718774210263) | > loader_time: 0.0107 (0.009820947811442618)  --> TIME: 2024-10-20 00:38:52 -- STEP: 2139/3361 -- GLOBAL_STEP: 5500 | > loss_text_ce: 0.04554080218076706 (0.05400996907235807) | > loss_mel_ce: 3.5199358463287354 (3.802293091301831) | > loss: 0.8913691639900208 (0.9640757669461782) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4984 (0.37496582555570235) | > loader_time: 0.0104 (0.009825523125896836)  --> TIME: 2024-10-20 00:39:25 -- STEP: 2189/3361 -- GLOBAL_STEP: 5550 | > loss_text_ce: 0.05215875059366226 (0.053962629445002665) | > loss_mel_ce: 3.4752392768859863 (3.8001412673321093) | > loss: 0.8818495273590088 (0.963525975986723) | > current_lr: 5e-06 | > step_time: 0.3729 (0.37516021565140617) | > loader_time: 0.0089 (0.009817448964996825)  --> TIME: 2024-10-20 00:39:56 -- STEP: 2239/3361 -- GLOBAL_STEP: 5600 | > loss_text_ce: 0.04801148176193237 (0.05391795849057377) | > loss_mel_ce: 3.7264914512634277 (3.797605789895034) | > loss: 0.9436257481575012 (0.9628809388858391) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4895 (0.3751126565588217) | > loader_time: 0.009 (0.009810711454738186)  --> TIME: 2024-10-20 00:40:29 -- STEP: 2289/3361 -- GLOBAL_STEP: 5650 | > loss_text_ce: 0.05482006445527077 (0.05388443434815208) | > loss_mel_ce: 3.9797515869140625 (3.795354474939702) | > loss: 1.0086429119110107 (0.9623097291207303) | > current_lr: 5e-06 | > step_time: 0.414 (0.3753624505067611) | > loader_time: 0.0091 (0.009804255996744091)  --> TIME: 2024-10-20 00:41:00 -- STEP: 2339/3361 -- GLOBAL_STEP: 5700 | > loss_text_ce: 0.053768549114465714 (0.053837192649767884) | > loss_mel_ce: 3.6103670597076416 (3.792830330329657) | > loss: 0.916033923625946 (0.9616668826497075) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4228 (0.37508875817098697) | > loader_time: 0.0099 (0.009796541029696479)  --> TIME: 2024-10-20 00:41:31 -- STEP: 2389/3361 -- GLOBAL_STEP: 5750 | > loss_text_ce: 0.05098041519522667 (0.053785190121892486) | > loss_mel_ce: 3.7739956378936768 (3.790576577735476) | > loss: 0.9562439918518066 (0.9610904438967682) | > current_lr: 5e-06 | > step_time: 0.4136 (0.37480044953740266) | > loader_time: 0.009 (0.009783994807214997)  --> TIME: 2024-10-20 00:42:03 -- STEP: 2439/3361 -- GLOBAL_STEP: 5800 | > loss_text_ce: 0.051108840852975845 (0.053738851268218625) | > loss_mel_ce: 3.423971652984619 (3.787982714083313) | > loss: 0.8687701225280762 (0.9604303932375672) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.425 (0.3746569493893182) | > loader_time: 0.0098 (0.009782257604032028)  --> TIME: 2024-10-20 00:42:35 -- STEP: 2489/3361 -- GLOBAL_STEP: 5850 | > loss_text_ce: 0.047307513654232025 (0.053671666948957655) | > loss_mel_ce: 3.593897581100464 (3.786233845845242) | > loss: 0.9103012681007385 (0.9599763800600726) | > current_lr: 5e-06 | > step_time: 0.3052 (0.37466022804493243) | > loader_time: 0.0105 (0.009783980261134827)  --> TIME: 2024-10-20 00:43:06 -- STEP: 2539/3361 -- GLOBAL_STEP: 5900 | > loss_text_ce: 0.04697243124246597 (0.053605987948254344) | > loss_mel_ce: 3.405848741531372 (3.783619592864948) | > loss: 0.8632053136825562 (0.9593063970164268) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3768 (0.374510133768828) | > loader_time: 0.0096 (0.009779982042857256)  --> TIME: 2024-10-20 00:43:38 -- STEP: 2589/3361 -- GLOBAL_STEP: 5950 | > loss_text_ce: 0.04804262891411781 (0.05354277639660238) | > loss_mel_ce: 3.705813407897949 (3.7827198630642735) | > loss: 0.9384639859199524 (0.9590656616688695) | > current_lr: 5e-06 | > step_time: 0.3419 (0.37453008278220223) | > loader_time: 0.0092 (0.009780242480841231)  --> TIME: 2024-10-20 00:44:09 -- STEP: 2639/3361 -- GLOBAL_STEP: 6000 | > loss_text_ce: 0.04612743854522705 (0.05346968814211162) | > loss_mel_ce: 3.9188263416290283 (3.7808196491344086) | > loss: 0.9912384748458862 (0.9585723361990677) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4981 (0.3743246874605093) | > loader_time: 0.0112 (0.009783610018332285) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_6000.pth  --> TIME: 2024-10-20 00:44:52 -- STEP: 2689/3361 -- GLOBAL_STEP: 6050 | > loss_text_ce: 0.047469258308410645 (0.05341444585531915) | > loss_mel_ce: 3.861588954925537 (3.778166536329049) | > loss: 0.9772645235061646 (0.957895247454801) | > current_lr: 5e-06 | > step_time: 0.3318 (0.37431536185426717) | > loader_time: 0.0103 (0.00978775129038058)  --> TIME: 2024-10-20 00:45:23 -- STEP: 2739/3361 -- GLOBAL_STEP: 6100 | > loss_text_ce: 0.04789832606911659 (0.053361904739724214) | > loss_mel_ce: 3.5844626426696777 (3.77603444142601) | > loss: 0.9080902338027954 (0.9573490884278804) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3335 (0.3741204217212319) | > loader_time: 0.0099 (0.009783599010215445)  --> TIME: 2024-10-20 00:45:55 -- STEP: 2789/3361 -- GLOBAL_STEP: 6150 | > loss_text_ce: 0.052916835993528366 (0.05331185084222335) | > loss_mel_ce: 3.5830535888671875 (3.7740368574654686) | > loss: 0.90899258852005 (0.9568371789435752) | > current_lr: 5e-06 | > step_time: 0.3046 (0.3741858853538141) | > loader_time: 0.0084 (0.009779214517067484)  --> TIME: 2024-10-20 00:46:27 -- STEP: 2839/3361 -- GLOBAL_STEP: 6200 | > loss_text_ce: 0.05071615055203438 (0.053262250472349594) | > loss_mel_ce: 3.695129156112671 (3.771301807085785) | > loss: 0.936461329460144 (0.9561410162095328) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.5436 (0.3742936511575533) | > loader_time: 0.0091 (0.00977529728651971)  --> TIME: 2024-10-20 00:47:00 -- STEP: 2889/3361 -- GLOBAL_STEP: 6250 | > loss_text_ce: 0.05398828536272049 (0.05322098417878768) | > loss_mel_ce: 3.478079319000244 (3.7687003193284534) | > loss: 0.8830168843269348 (0.9554803276111528) | > current_lr: 5e-06 | > step_time: 0.2948 (0.37435020431592575) | > loader_time: 0.0097 (0.009775939223757246)  --> TIME: 2024-10-20 00:47:31 -- STEP: 2939/3361 -- GLOBAL_STEP: 6300 | > loss_text_ce: 0.05640152841806412 (0.05317959931304194) | > loss_mel_ce: 3.542383909225464 (3.766597809131229) | > loss: 0.8996963500976562 (0.9549443538434735) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3899 (0.3743035755176948) | > loader_time: 0.0096 (0.009772745270678769)  --> TIME: 2024-10-20 00:48:03 -- STEP: 2989/3361 -- GLOBAL_STEP: 6350 | > loss_text_ce: 0.05089443922042847 (0.053131582746989495) | > loss_mel_ce: 3.53525710105896 (3.764354299725239) | > loss: 0.8965378999710083 (0.9543714722529347) | > current_lr: 5e-06 | > step_time: 0.3915 (0.37433643443164155) | > loader_time: 0.0088 (0.009772129464045778)  --> TIME: 2024-10-20 00:48:36 -- STEP: 3039/3361 -- GLOBAL_STEP: 6400 | > loss_text_ce: 0.046930961310863495 (0.05308791380418526) | > loss_mel_ce: 3.706780433654785 (3.7627756988660956) | > loss: 0.9384278655052185 (0.9539659047213068) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4973 (0.3744112501492748) | > loader_time: 0.0094 (0.009775070659264634)  --> TIME: 2024-10-20 00:49:07 -- STEP: 3089/3361 -- GLOBAL_STEP: 6450 | > loss_text_ce: 0.04575106129050255 (0.053032684769733454) | > loss_mel_ce: 3.75093936920166 (3.7611461492293317) | > loss: 0.9491726160049438 (0.9535447100328759) | > current_lr: 5e-06 | > step_time: 0.4194 (0.37436858069437534) | > loader_time: 0.0091 (0.009773730922957073)  --> TIME: 2024-10-20 00:49:40 -- STEP: 3139/3361 -- GLOBAL_STEP: 6500 | > loss_text_ce: 0.05457288771867752 (0.05298690562673189) | > loss_mel_ce: 3.7172043323516846 (3.759212153092354) | > loss: 0.9429442882537842 (0.9530497661970648) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.5654 (0.3745672619240274) | > loader_time: 0.0095 (0.009773702445247437)  --> TIME: 2024-10-20 00:50:12 -- STEP: 3189/3361 -- GLOBAL_STEP: 6550 | > loss_text_ce: 0.04842592775821686 (0.05293102836713553) | > loss_mel_ce: 3.452249526977539 (3.7573863935156577) | > loss: 0.8751688599586487 (0.9525793569519364) | > current_lr: 5e-06 | > step_time: 0.2971 (0.3745487653637209) | > loader_time: 0.01 (0.009774107363351874)  --> TIME: 2024-10-20 00:50:43 -- STEP: 3239/3361 -- GLOBAL_STEP: 6600 | > loss_text_ce: 0.05154899135231972 (0.05288972797383492) | > loss_mel_ce: 3.525009870529175 (3.755932199914502) | > loss: 0.8941397070884705 (0.9522054834879633) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.481 (0.3743514210105354) | > loader_time: 0.0099 (0.009773110636740263)  --> TIME: 2024-10-20 00:51:16 -- STEP: 3289/3361 -- GLOBAL_STEP: 6650 | > loss_text_ce: 0.05035821348428726 (0.052853074836564214) | > loss_mel_ce: 3.56316876411438 (3.754109559721699) | > loss: 0.9033817648887634 (0.951740660178839) | > current_lr: 5e-06 | > step_time: 0.4077 (0.3745306703811657) | > loader_time: 0.0105 (0.009773014961096529)  --> TIME: 2024-10-20 00:51:48 -- STEP: 3339/3361 -- GLOBAL_STEP: 6700 | > loss_text_ce: 0.047619469463825226 (0.05279993967413365) | > loss_mel_ce: 3.5058529376983643 (3.7519333840987397) | > loss: 0.8883681297302246 (0.951183332533635) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4219 (0.37463738312082967) | > loader_time: 0.0108 (0.009774323100992282)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.3849575237561297 (+0.0032363040472870863) | > avg_loss_text_ce: 0.04702420168185747 (-0.007846372021782781) | > avg_loss_mel_ce: 3.4355908209277737 (-0.3100963510492796) | > avg_loss: 3.482615009430916 (-0.3179427321239183)  > EPOCH: 2/5 --> /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b  > TRAINING (2024-10-20 00:53:55)   --> TIME: 2024-10-20 00:54:15 -- STEP: 28/3361 -- GLOBAL_STEP: 6750 | > loss_text_ce: 0.04403690621256828 (0.04905040069882359) | > loss_mel_ce: 3.7761855125427246 (3.686231689793723) | > loss: 0.9550555944442749 (0.9338205137423107) | > current_lr: 5e-06 | > step_time: 0.2669 (0.3828866481781006) | > loader_time: 0.0095 (0.010155720370156425)  --> TIME: 2024-10-20 00:54:46 -- STEP: 78/3361 -- GLOBAL_STEP: 6800 | > loss_text_ce: 0.047489896416664124 (0.04935918151377103) | > loss_mel_ce: 3.2789604663848877 (3.62562545751914) | > loss: 0.8316125869750977 (0.9187461550419147) | > current_lr: 5e-06 | > step_time: 0.4336 (0.3764091027088655) | > loader_time: 0.0105 (0.010129986665187739)  --> TIME: 2024-10-20 00:55:18 -- STEP: 128/3361 -- GLOBAL_STEP: 6850 | > loss_text_ce: 0.053600654006004333 (0.04948139251791872) | > loss_mel_ce: 3.313396692276001 (3.610981924459338) | > loss: 0.8417493104934692 (0.9151158249005675) | > current_lr: 5e-06 | > step_time: 0.2737 (0.37236367911100393) | > loader_time: 0.0109 (0.010153220966458322)  --> TIME: 2024-10-20 00:55:50 -- STEP: 178/3361 -- GLOBAL_STEP: 6900 | > loss_text_ce: 0.04995616152882576 (0.04949212647723348) | > loss_mel_ce: 3.5250840187072754 (3.6204694884546686) | > loss: 0.8937600255012512 (0.9174904006250789) | > current_lr: 5e-06 | > step_time: 0.4846 (0.3750775664040211) | > loader_time: 0.0092 (0.010136076573575476)  --> TIME: 2024-10-20 00:56:24 -- STEP: 228/3361 -- GLOBAL_STEP: 6950 | > loss_text_ce: 0.047617699950933456 (0.04926429808205157) | > loss_mel_ce: 3.59977650642395 (3.6181379849450632) | > loss: 0.9118485450744629 (0.9168505681711331) | > current_lr: 5e-06 | > step_time: 0.3983 (0.3783655584904185) | > loader_time: 0.01 (0.010034571614181783)  --> TIME: 2024-10-20 00:56:55 -- STEP: 278/3361 -- GLOBAL_STEP: 7000 | > loss_text_ce: 0.04799577221274376 (0.049197411218242684) | > loss_mel_ce: 3.7218735218048096 (3.6140262754701027) | > loss: 0.9424673318862915 (0.9158059191360749) | > current_lr: 5e-06 | > step_time: 0.3743 (0.3763897959276927) | > loader_time: 0.009 (0.009955753525384033) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_7000.pth  --> TIME: 2024-10-20 00:57:35 -- STEP: 328/3361 -- GLOBAL_STEP: 7050 | > loss_text_ce: 0.04574914276599884 (0.049158351098346285) | > loss_mel_ce: 3.567535877227783 (3.613539861469734) | > loss: 0.9033212661743164 (0.9156745514128266) | > current_lr: 5e-06 | > step_time: 0.2103 (0.3746326173224102) | > loader_time: 0.0092 (0.00990271422921157)  --> TIME: 2024-10-20 00:58:08 -- STEP: 378/3361 -- GLOBAL_STEP: 7100 | > loss_text_ce: 0.048721931874752045 (0.049147929641462515) | > loss_mel_ce: 3.5802829265594482 (3.6114332770544384) | > loss: 0.907251238822937 (0.9151453005258369) | > current_lr: 5e-06 | > step_time: 0.4294 (0.376694580234548) | > loader_time: 0.0108 (0.009864083042851196)  --> TIME: 2024-10-20 00:58:40 -- STEP: 428/3361 -- GLOBAL_STEP: 7150 | > loss_text_ce: 0.051922909915447235 (0.04917553177747493) | > loss_mel_ce: 3.5437474250793457 (3.607163674920519) | > loss: 0.898917555809021 (0.9140848006322005) | > current_lr: 5e-06 | > step_time: 0.2716 (0.37683266401290916) | > loader_time: 0.0094 (0.009827861719042336)  --> TIME: 2024-10-20 00:59:12 -- STEP: 478/3361 -- GLOBAL_STEP: 7200 | > loss_text_ce: 0.045843224972486496 (0.04910156095719986) | > loss_mel_ce: 3.5055999755859375 (3.6058153352976845) | > loss: 0.8878607749938965 (0.9137292228732647) | > current_lr: 5e-06 | > step_time: 0.3413 (0.3767390929505418) | > loader_time: 0.0091 (0.00982440415785402)  --> TIME: 2024-10-20 00:59:44 -- STEP: 528/3361 -- GLOBAL_STEP: 7250 | > loss_text_ce: 0.048054084181785583 (0.04909205055005397) | > loss_mel_ce: 3.536403179168701 (3.60500498928807) | > loss: 0.8961142897605896 (0.9135242587689197) | > current_lr: 5e-06 | > step_time: 0.2992 (0.3766030235724017) | > loader_time: 0.0095 (0.009845270351930094)  --> TIME: 2024-10-20 01:00:17 -- STEP: 578/3361 -- GLOBAL_STEP: 7300 | > loss_text_ce: 0.05416757985949516 (0.0490665393446453) | > loss_mel_ce: 3.5705678462982178 (3.603965314201831) | > loss: 0.9061838388442993 (0.9132579624859107) | > current_lr: 5e-06 | > step_time: 0.3706 (0.37739562947032373) | > loader_time: 0.009 (0.009825011025663062)  --> TIME: 2024-10-20 01:00:48 -- STEP: 628/3361 -- GLOBAL_STEP: 7350 | > loss_text_ce: 0.047263436019420624 (0.049029604579280496) | > loss_mel_ce: 3.5175342559814453 (3.603023174462046) | > loss: 0.8911994099617004 (0.9130131940173495) | > current_lr: 5e-06 | > step_time: 0.3131 (0.3764249720391194) | > loader_time: 0.009 (0.009792880267853939)  --> TIME: 2024-10-20 01:01:21 -- STEP: 678/3361 -- GLOBAL_STEP: 7400 | > loss_text_ce: 0.049490731209516525 (0.0490141717777491) | > loss_mel_ce: 3.326827049255371 (3.6022712867871856) | > loss: 0.8440794348716736 (0.9128213640862861) | > current_lr: 5e-06 | > step_time: 0.4932 (0.37664768055828635) | > loader_time: 0.0095 (0.009787028869696412)  --> TIME: 2024-10-20 01:01:53 -- STEP: 728/3361 -- GLOBAL_STEP: 7450 | > loss_text_ce: 0.04762418195605278 (0.04900444910994595) | > loss_mel_ce: 3.558835744857788 (3.602000492614705) | > loss: 0.9016149640083313 (0.9127512351497189) | > current_lr: 5e-06 | > step_time: 0.314 (0.37692653346847715) | > loader_time: 0.0098 (0.009809368586802208)  --> TIME: 2024-10-20 01:02:25 -- STEP: 778/3361 -- GLOBAL_STEP: 7500 | > loss_text_ce: 0.04937560111284256 (0.04896448014883422) | > loss_mel_ce: 3.9129478931427 (3.6037924369380536) | > loss: 0.9905808568000793 (0.9131892292082769) | > current_lr: 5e-06 | > step_time: 0.3562 (0.37699541150757476) | > loader_time: 0.009 (0.009813950423409816)  --> TIME: 2024-10-20 01:02:57 -- STEP: 828/3361 -- GLOBAL_STEP: 7550 | > loss_text_ce: 0.04987458512187004 (0.04891771870408799) | > loss_mel_ce: 3.4430720806121826 (3.6009757144439627) | > loss: 0.8732366561889648 (0.9124733581059221) | > current_lr: 5e-06 | > step_time: 0.3872 (0.3765455469993003) | > loader_time: 0.0099 (0.009804343255821633)  --> TIME: 2024-10-20 01:03:28 -- STEP: 878/3361 -- GLOBAL_STEP: 7600 | > loss_text_ce: 0.04912569746375084 (0.04892470960023732) | > loss_mel_ce: 3.4806666374206543 (3.597835117429156) | > loss: 0.8824480772018433 (0.9116899567340121) | > current_lr: 5e-06 | > step_time: 0.3901 (0.3757378245812072) | > loader_time: 0.0088 (0.009797889442270058)  --> TIME: 2024-10-20 01:04:01 -- STEP: 928/3361 -- GLOBAL_STEP: 7650 | > loss_text_ce: 0.04729423299431801 (0.04890506303917357) | > loss_mel_ce: 3.8602240085601807 (3.5957908889856838) | > loss: 0.9768795371055603 (0.9111739878767523) | > current_lr: 5e-06 | > step_time: 0.3303 (0.3764181083132482) | > loader_time: 0.0087 (0.009782458687650746)  --> TIME: 2024-10-20 01:04:33 -- STEP: 978/3361 -- GLOBAL_STEP: 7700 | > loss_text_ce: 0.0476219542324543 (0.04887541249424887) | > loss_mel_ce: 3.6902410984039307 (3.5933213326584834) | > loss: 0.934465765953064 (0.9105491861853375) | > current_lr: 5e-06 | > step_time: 0.4139 (0.3762613827465503) | > loader_time: 0.0095 (0.009784645341900473)  --> TIME: 2024-10-20 01:05:06 -- STEP: 1028/3361 -- GLOBAL_STEP: 7750 | > loss_text_ce: 0.050320666283369064 (0.048847640831936635) | > loss_mel_ce: 3.489208698272705 (3.5914727294027577) | > loss: 0.8848823308944702 (0.9100800926583287) | > current_lr: 5e-06 | > step_time: 0.3856 (0.37657562837526504) | > loader_time: 0.0097 (0.009803709816839909)  --> TIME: 2024-10-20 01:05:38 -- STEP: 1078/3361 -- GLOBAL_STEP: 7800 | > loss_text_ce: 0.04912106320261955 (0.048804982256054424) | > loss_mel_ce: 3.4618194103240967 (3.587999959823595) | > loss: 0.8777351379394531 (0.909201235629632) | > current_lr: 5e-06 | > step_time: 0.3768 (0.3767289274919898) | > loader_time: 0.0094 (0.009814880111000755)  --> TIME: 2024-10-20 01:06:10 -- STEP: 1128/3361 -- GLOBAL_STEP: 7850 | > loss_text_ce: 0.05011963099241257 (0.048788163781245335) | > loss_mel_ce: 3.5565550327301025 (3.587737088719159) | > loss: 0.9016686677932739 (0.909131313205188) | > current_lr: 5e-06 | > step_time: 0.3358 (0.37648225506992233) | > loader_time: 0.0089 (0.009822922395476214)  --> TIME: 2024-10-20 01:06:43 -- STEP: 1178/3361 -- GLOBAL_STEP: 7900 | > loss_text_ce: 0.04669732600450516 (0.048759036461849564) | > loss_mel_ce: 3.1603102684020996 (3.5851203856929654) | > loss: 0.8017519116401672 (0.9084698557853699) | > current_lr: 5e-06 | > step_time: 0.3734 (0.37704856853128915) | > loader_time: 0.0087 (0.009813111585348294)  --> TIME: 2024-10-20 01:07:14 -- STEP: 1228/3361 -- GLOBAL_STEP: 7950 | > loss_text_ce: 0.04521196335554123 (0.048707531449606434) | > loss_mel_ce: 3.4588823318481445 (3.585536780885454) | > loss: 0.8760235905647278 (0.9085610783158373) | > current_lr: 5e-06 | > step_time: 0.3439 (0.3768606422778451) | > loader_time: 0.0086 (0.009805944532835524)  --> TIME: 2024-10-20 01:07:48 -- STEP: 1278/3361 -- GLOBAL_STEP: 8000 | > loss_text_ce: 0.04893951490521431 (0.048680408736792906) | > loss_mel_ce: 3.475900173187256 (3.584775358298575) | > loss: 0.8812099099159241 (0.9083639419694461) | > current_lr: 5e-06 | > step_time: 0.4076 (0.37746544193960335) | > loader_time: 0.0092 (0.00980452752449143) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_8000.pth  --> TIME: 2024-10-20 01:08:32 -- STEP: 1328/3361 -- GLOBAL_STEP: 8050 | > loss_text_ce: 0.04482944682240486 (0.04863053328548394) | > loss_mel_ce: 3.514838695526123 (3.5835139115890824) | > loss: 0.8899170160293579 (0.9080361113939659) | > current_lr: 5e-06 | > step_time: 0.3505 (0.3775392822831512) | > loader_time: 0.0095 (0.00981970687946642)  --> TIME: 2024-10-20 01:09:05 -- STEP: 1378/3361 -- GLOBAL_STEP: 8100 | > loss_text_ce: 0.045615728944540024 (0.048578062096099765) | > loss_mel_ce: 3.5625221729278564 (3.583668395638985) | > loss: 0.9020344614982605 (0.9080616145763757) | > current_lr: 5e-06 | > step_time: 0.3422 (0.37772889061140236) | > loader_time: 0.01 (0.00982503915213709)  --> TIME: 2024-10-20 01:09:37 -- STEP: 1428/3361 -- GLOBAL_STEP: 8150 | > loss_text_ce: 0.047519903630018234 (0.048547342737891463) | > loss_mel_ce: 3.6124684810638428 (3.581988831361135) | > loss: 0.9149971008300781 (0.9076340436851945) | > current_lr: 5e-06 | > step_time: 0.2656 (0.3779737376699262) | > loader_time: 0.0095 (0.009825288247661443)  --> TIME: 2024-10-20 01:10:10 -- STEP: 1478/3361 -- GLOBAL_STEP: 8200 | > loss_text_ce: 0.04936197027564049 (0.04851635845430975) | > loss_mel_ce: 3.5139453411102295 (3.580858142998609) | > loss: 0.8908268213272095 (0.9073436254980761) | > current_lr: 5e-06 | > step_time: 0.4088 (0.37810928605406163) | > loader_time: 0.0097 (0.00982494218746284)  --> TIME: 2024-10-20 01:10:42 -- STEP: 1528/3361 -- GLOBAL_STEP: 8250 | > loss_text_ce: 0.04918792098760605 (0.04848537729146128) | > loss_mel_ce: 3.376131772994995 (3.5808331872780284) | > loss: 0.8563299179077148 (0.9073296412435502) | > current_lr: 5e-06 | > step_time: 0.3339 (0.3781200748775643) | > loader_time: 0.0095 (0.009824010402120225)  --> TIME: 2024-10-20 01:11:14 -- STEP: 1578/3361 -- GLOBAL_STEP: 8300 | > loss_text_ce: 0.04867768660187721 (0.048471816391678246) | > loss_mel_ce: 3.5776634216308594 (3.578830152561154) | > loss: 0.9065852761268616 (0.906825492250602) | > current_lr: 5e-06 | > step_time: 0.3276 (0.37784483858841017) | > loader_time: 0.0087 (0.00981416539094295)  --> TIME: 2024-10-20 01:11:46 -- STEP: 1628/3361 -- GLOBAL_STEP: 8350 | > loss_text_ce: 0.04532700031995773 (0.048449824335332764) | > loss_mel_ce: 3.543966293334961 (3.5778604450038376) | > loss: 0.8973233103752136 (0.9065775673571032) | > current_lr: 5e-06 | > step_time: 0.3354 (0.3776157829333876) | > loader_time: 0.0099 (0.009799206579053737)  --> TIME: 2024-10-20 01:12:17 -- STEP: 1678/3361 -- GLOBAL_STEP: 8400 | > loss_text_ce: 0.04563120752573013 (0.04841454139953933) | > loss_mel_ce: 3.4720375537872314 (3.5776013940110962) | > loss: 0.8794171810150146 (0.9065039838454436) | > current_lr: 5e-06 | > step_time: 0.3709 (0.37741188909214657) | > loader_time: 0.0088 (0.009793770185386479)  --> TIME: 2024-10-20 01:12:49 -- STEP: 1728/3361 -- GLOBAL_STEP: 8450 | > loss_text_ce: 0.05147872492671013 (0.048384104855358635) | > loss_mel_ce: 3.6369025707244873 (3.576328690405245) | > loss: 0.9220952987670898 (0.906178198720294) | > current_lr: 5e-06 | > step_time: 0.3306 (0.37728250123284496) | > loader_time: 0.0102 (0.009796891775396147)  --> TIME: 2024-10-20 01:13:21 -- STEP: 1778/3361 -- GLOBAL_STEP: 8500 | > loss_text_ce: 0.04927783086895943 (0.04836768526437248) | > loss_mel_ce: 3.509384870529175 (3.5736189018620683) | > loss: 0.8896656632423401 (0.90549664667004) | > current_lr: 5e-06 | > step_time: 0.3454 (0.3771043242163815) | > loader_time: 0.0096 (0.009791791103822186)  --> TIME: 2024-10-20 01:13:53 -- STEP: 1828/3361 -- GLOBAL_STEP: 8550 | > loss_text_ce: 0.04953066632151604 (0.048366361587049565) | > loss_mel_ce: 3.6733510494232178 (3.5723333584401487) | > loss: 0.9307204484939575 (0.9051749299339444) | > current_lr: 5e-06 | > step_time: 0.2723 (0.37695589094245474) | > loader_time: 0.0114 (0.009782656668573303)  --> TIME: 2024-10-20 01:14:25 -- STEP: 1878/3361 -- GLOBAL_STEP: 8600 | > loss_text_ce: 0.04815003275871277 (0.04833432593451334) | > loss_mel_ce: 3.39089298248291 (3.5706544065881713) | > loss: 0.8597607612609863 (0.9047471829814017) | > current_lr: 5e-06 | > step_time: 0.427 (0.37701883737525504) | > loader_time: 0.0093 (0.009764435700080403)  --> TIME: 2024-10-20 01:14:58 -- STEP: 1928/3361 -- GLOBAL_STEP: 8650 | > loss_text_ce: 0.04663149267435074 (0.048306525483943895) | > loss_mel_ce: 3.3355255126953125 (3.5698014803199847) | > loss: 0.8455392718315125 (0.9045270012374735) | > current_lr: 5e-06 | > step_time: 0.3483 (0.37734258817933913) | > loader_time: 0.0099 (0.009774215364851903)  --> TIME: 2024-10-20 01:15:30 -- STEP: 1978/3361 -- GLOBAL_STEP: 8700 | > loss_text_ce: 0.04672824218869209 (0.048289880877844756) | > loss_mel_ce: 3.442249059677124 (3.5682690238807995) | > loss: 0.8722442984580994 (0.9041397260281627) | > current_lr: 5e-06 | > step_time: 0.4095 (0.37710894805476214) | > loader_time: 0.0092 (0.009775729584380529)  --> TIME: 2024-10-20 01:16:02 -- STEP: 2028/3361 -- GLOBAL_STEP: 8750 | > loss_text_ce: 0.04540479928255081 (0.04825842313292114) | > loss_mel_ce: 3.524287462234497 (3.567732476388679) | > loss: 0.8924230933189392 (0.9039977247251797) | > current_lr: 5e-06 | > step_time: 0.2754 (0.3770918593369058) | > loader_time: 0.0109 (0.009774529369625124)  --> TIME: 2024-10-20 01:16:33 -- STEP: 2078/3361 -- GLOBAL_STEP: 8800 | > loss_text_ce: 0.043983638286590576 (0.048219160568524856) | > loss_mel_ce: 3.5472054481506348 (3.566295558724757) | > loss: 0.8977972865104675 (0.9036286797686421) | > current_lr: 5e-06 | > step_time: 0.4283 (0.37684804103601666) | > loader_time: 0.0101 (0.009771026362583441)  --> TIME: 2024-10-20 01:17:06 -- STEP: 2128/3361 -- GLOBAL_STEP: 8850 | > loss_text_ce: 0.04918619617819786 (0.04819233537322487) | > loss_mel_ce: 3.391493797302246 (3.565849136701204) | > loss: 0.8601700067520142 (0.9035103678759328) | > current_lr: 5e-06 | > step_time: 0.2636 (0.3770903514740162) | > loader_time: 0.0094 (0.009773158937468576)  --> TIME: 2024-10-20 01:17:39 -- STEP: 2178/3361 -- GLOBAL_STEP: 8900 | > loss_text_ce: 0.0547124482691288 (0.04816222397475425) | > loss_mel_ce: 3.6788527965545654 (3.5645782656358724) | > loss: 0.9333913326263428 (0.903185122296611) | > current_lr: 5e-06 | > step_time: 0.3439 (0.3773379673970962) | > loader_time: 0.0105 (0.009771051595143167)  --> TIME: 2024-10-20 01:18:11 -- STEP: 2228/3361 -- GLOBAL_STEP: 8950 | > loss_text_ce: 0.04527541995048523 (0.04813232319847527) | > loss_mel_ce: 3.706040382385254 (3.5633151346091956) | > loss: 0.9378289580345154 (0.9028618643110488) | > current_lr: 5e-06 | > step_time: 0.2648 (0.3773191904037175) | > loader_time: 0.0104 (0.009769387797350522)  --> TIME: 2024-10-20 01:18:43 -- STEP: 2278/3361 -- GLOBAL_STEP: 9000 | > loss_text_ce: 0.042789604514837265 (0.048098762065852514) | > loss_mel_ce: 3.569603681564331 (3.561746247511773) | > loss: 0.9030983448028564 (0.9024612522627616) | > current_lr: 5e-06 | > step_time: 0.4348 (0.37725567137389143) | > loader_time: 0.0097 (0.009770785747022363) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_9000.pth  --> TIME: 2024-10-20 01:19:25 -- STEP: 2328/3361 -- GLOBAL_STEP: 9050 | > loss_text_ce: 0.04657544568181038 (0.04807176884372568) | > loss_mel_ce: 3.4539883136749268 (3.5602467105151043) | > loss: 0.8751409649848938 (0.9020796196976886) | > current_lr: 5e-06 | > step_time: 0.3703 (0.3772568883969612) | > loader_time: 0.011 (0.009778630999765047)  --> TIME: 2024-10-20 01:19:57 -- STEP: 2378/3361 -- GLOBAL_STEP: 9100 | > loss_text_ce: 0.04268953576683998 (0.04804769095544109) | > loss_mel_ce: 3.6826560497283936 (3.558529170748564) | > loss: 0.9313364028930664 (0.9016442152509374) | > current_lr: 5e-06 | > step_time: 0.3398 (0.37726983671132214) | > loader_time: 0.0108 (0.009784870432443842)  --> TIME: 2024-10-20 01:20:29 -- STEP: 2428/3361 -- GLOBAL_STEP: 9150 | > loss_text_ce: 0.04784495756030083 (0.048017254220459896) | > loss_mel_ce: 3.2817625999450684 (3.55805171125214) | > loss: 0.8324018716812134 (0.9015172412358161) | > current_lr: 5e-06 | > step_time: 0.3146 (0.3770558785960152) | > loader_time: 0.0111 (0.009788657825507693)  --> TIME: 2024-10-20 01:21:00 -- STEP: 2478/3361 -- GLOBAL_STEP: 9200 | > loss_text_ce: 0.04175335913896561 (0.047998460361476994) | > loss_mel_ce: 3.682319402694702 (3.556396670549314) | > loss: 0.9310181736946106 (0.901098782589014) | > current_lr: 5e-06 | > step_time: 0.3292 (0.3768024727449577) | > loader_time: 0.0102 (0.009790577457449549)  --> TIME: 2024-10-20 01:21:32 -- STEP: 2528/3361 -- GLOBAL_STEP: 9250 | > loss_text_ce: 0.04678019881248474 (0.04796844289072239) | > loss_mel_ce: 3.5981311798095703 (3.5554795194464393) | > loss: 0.911227822303772 (0.9008619905083992) | > current_lr: 5e-06 | > step_time: 0.272 (0.3767167896598198) | > loader_time: 0.0098 (0.009790274727193635)  --> TIME: 2024-10-20 01:22:03 -- STEP: 2578/3361 -- GLOBAL_STEP: 9300 | > loss_text_ce: 0.04527261480689049 (0.04794080875079894) | > loss_mel_ce: 3.4495139122009277 (3.55347470269821) | > loss: 0.8736966252326965 (0.9003538777654349) | > current_lr: 5e-06 | > step_time: 0.4813 (0.37643767143054185) | > loader_time: 0.0097 (0.009784305530337996)  --> TIME: 2024-10-20 01:22:35 -- STEP: 2628/3361 -- GLOBAL_STEP: 9350 | > loss_text_ce: 0.04605727642774582 (0.047905134299381695) | > loss_mel_ce: 3.8004796504974365 (3.552657171834368) | > loss: 0.9616342186927795 (0.9001405764841778) | > current_lr: 5e-06 | > step_time: 0.2983 (0.3764948804991794) | > loader_time: 0.0107 (0.009790125412120867)  --> TIME: 2024-10-20 01:23:07 -- STEP: 2678/3361 -- GLOBAL_STEP: 9400 | > loss_text_ce: 0.043804291635751724 (0.047882563374029685) | > loss_mel_ce: 3.435140609741211 (3.551404430761188) | > loss: 0.8697362542152405 (0.8998217484993751) | > current_lr: 5e-06 | > step_time: 0.4414 (0.37655566075206093) | > loader_time: 0.0095 (0.009788562505435746)  --> TIME: 2024-10-20 01:23:39 -- STEP: 2728/3361 -- GLOBAL_STEP: 9450 | > loss_text_ce: 0.05015170946717262 (0.04786144649948595) | > loss_mel_ce: 3.592085599899292 (3.550092155545338) | > loss: 0.910559356212616 (0.8994884004166397) | > current_lr: 5e-06 | > step_time: 0.2727 (0.37655943046567025) | > loader_time: 0.009 (0.009790379805299211)  --> TIME: 2024-10-20 01:24:11 -- STEP: 2778/3361 -- GLOBAL_STEP: 9500 | > loss_text_ce: 0.04628805071115494 (0.04783492111731691) | > loss_mel_ce: 3.421419620513916 (3.549318913335505) | > loss: 0.866926908493042 (0.8992884584988853) | > current_lr: 5e-06 | > step_time: 0.3323 (0.3764905528768178) | > loader_time: 0.0106 (0.009790914877444575)  --> TIME: 2024-10-20 01:24:43 -- STEP: 2828/3361 -- GLOBAL_STEP: 9550 | > loss_text_ce: 0.04558764770627022 (0.04781904644721434) | > loss_mel_ce: 3.2457244396209717 (3.5478047599549676) | > loss: 0.8228279948234558 (0.8989059515139337) | > current_lr: 5e-06 | > step_time: 0.315 (0.3763812427979379) | > loader_time: 0.0111 (0.009782957642891115)  --> TIME: 2024-10-20 01:25:14 -- STEP: 2878/3361 -- GLOBAL_STEP: 9600 | > loss_text_ce: 0.045083776116371155 (0.047794801320898854) | > loss_mel_ce: 3.9052932262420654 (3.5468353375202257) | > loss: 0.9875942468643188 (0.8986575346675657) | > current_lr: 5e-06 | > step_time: 0.373 (0.3762246082854652) | > loader_time: 0.0089 (0.009778903456841017)  --> TIME: 2024-10-20 01:25:46 -- STEP: 2928/3361 -- GLOBAL_STEP: 9650 | > loss_text_ce: 0.04821404814720154 (0.04778221920845132) | > loss_mel_ce: 3.5854885578155518 (3.546300680503819) | > loss: 0.9084256291389465 (0.8985207248173774) | > current_lr: 5e-06 | > step_time: 0.265 (0.3761550551229487) | > loader_time: 0.0088 (0.00977481079231845)  --> TIME: 2024-10-20 01:26:18 -- STEP: 2978/3361 -- GLOBAL_STEP: 9700 | > loss_text_ce: 0.046400487422943115 (0.047759666937591726) | > loss_mel_ce: 3.357914447784424 (3.5448905813205633) | > loss: 0.8510787487030029 (0.8981625620273233) | > current_lr: 5e-06 | > step_time: 0.4432 (0.3761592442914093) | > loader_time: 0.0088 (0.009770671900204147)  --> TIME: 2024-10-20 01:26:51 -- STEP: 3028/3361 -- GLOBAL_STEP: 9750 | > loss_text_ce: 0.0519859679043293 (0.04774229847942948) | > loss_mel_ce: 3.3926987648010254 (3.5437855810281307) | > loss: 0.8611711859703064 (0.8978819698292165) | > current_lr: 5e-06 | > step_time: 0.3159 (0.3763601943712726) | > loader_time: 0.0095 (0.009767645337622564)  --> TIME: 2024-10-20 01:27:23 -- STEP: 3078/3361 -- GLOBAL_STEP: 9800 | > loss_text_ce: 0.04087809845805168 (0.047722860169854234) | > loss_mel_ce: 3.3796744346618652 (3.542765019774979) | > loss: 0.8551381230354309 (0.8976219699435453) | > current_lr: 5e-06 | > step_time: 0.342 (0.3762636706765555) | > loader_time: 0.0088 (0.009768155195869497)  --> TIME: 2024-10-20 01:27:55 -- STEP: 3128/3361 -- GLOBAL_STEP: 9850 | > loss_text_ce: 0.048090122640132904 (0.04769306221043179) | > loss_mel_ce: 3.387538433074951 (3.541276809733237) | > loss: 0.858907163143158 (0.8972424678980845) | > current_lr: 5e-06 | > step_time: 0.2728 (0.37625613214109865) | > loader_time: 0.0093 (0.0097666510840511)  --> TIME: 2024-10-20 01:28:27 -- STEP: 3178/3361 -- GLOBAL_STEP: 9900 | > loss_text_ce: 0.0475609228014946 (0.047669609727266526) | > loss_mel_ce: 3.4589006900787354 (3.5400787889769876) | > loss: 0.8766154050827026 (0.8969370995661684) | > current_lr: 5e-06 | > step_time: 0.3972 (0.37626130442862393) | > loader_time: 0.009 (0.009765567608491803)  --> TIME: 2024-10-20 01:28:59 -- STEP: 3228/3361 -- GLOBAL_STEP: 9950 | > loss_text_ce: 0.047529496252536774 (0.04764427237046481) | > loss_mel_ce: 3.8835809230804443 (3.5399227575889456) | > loss: 0.9827775955200195 (0.8968917573657915) | > current_lr: 5e-06 | > step_time: 0.2965 (0.3762442364716323) | > loader_time: 0.0105 (0.009769178086792505)  --> TIME: 2024-10-20 01:29:31 -- STEP: 3278/3361 -- GLOBAL_STEP: 10000 | > loss_text_ce: 0.04803500697016716 (0.04761744798129844) | > loss_mel_ce: 3.3600974082946777 (3.538903193767657) | > loss: 0.8520330786705017 (0.896630160269612) | > current_lr: 5e-06 | > step_time: 0.3416 (0.37622738066353834) | > loader_time: 0.0094 (0.009767672941406864) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_10000.pth  --> TIME: 2024-10-20 01:30:13 -- STEP: 3328/3361 -- GLOBAL_STEP: 10050 | > loss_text_ce: 0.04764594882726669 (0.047592870925795) | > loss_mel_ce: 3.632420539855957 (3.537776723718987) | > loss: 0.9200166463851929 (0.8963423984913297) | > current_lr: 5e-06 | > step_time: 0.3302 (0.37621467653661966) | > loader_time: 0.0095 (0.009771530373165221)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.37726180527799874 (-0.007695718478130942) | > avg_loss_text_ce: 0.04344167616418612 (-0.0035825255176713464) | > avg_loss_mel_ce: 3.27118952556323 (-0.16440129536454373) | > avg_loss: 3.3146311980421825 (-0.16798381138873353) > BEST MODEL : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/best_model_10083.pth  > EPOCH: 3/5 --> /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b  > TRAINING (2024-10-20 01:32:59)   --> TIME: 2024-10-20 01:33:10 -- STEP: 17/3361 -- GLOBAL_STEP: 10100 | > loss_text_ce: 0.04386524111032486 (0.04697125514640527) | > loss_mel_ce: 3.385413408279419 (3.4981692959280575) | > loss: 0.8573196530342102 (0.8862851367277258) | > current_lr: 5e-06 | > step_time: 0.3442 (0.3555046249838436) | > loader_time: 0.0098 (0.010421248043284696)  --> TIME: 2024-10-20 01:33:43 -- STEP: 67/3361 -- GLOBAL_STEP: 10150 | > loss_text_ce: 0.04214071109890938 (0.046040518014733474) | > loss_mel_ce: 3.695300817489624 (3.470060277341017) | > loss: 0.9343603849411011 (0.879025197740811) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4269 (0.374709196944735) | > loader_time: 0.0102 (0.009930838399858614)  --> TIME: 2024-10-20 01:34:14 -- STEP: 117/3361 -- GLOBAL_STEP: 10200 | > loss_text_ce: 0.04726540297269821 (0.04610895591541233) | > loss_mel_ce: 3.4293832778930664 (3.4879054391486015) | > loss: 0.8691621422767639 (0.8835035990446042) | > current_lr: 5e-06 | > step_time: 0.3975 (0.37159106058952146) | > loader_time: 0.0109 (0.009970159612150272)  --> TIME: 2024-10-20 01:34:46 -- STEP: 167/3361 -- GLOBAL_STEP: 10250 | > loss_text_ce: 0.05034831911325455 (0.04611547081591842) | > loss_mel_ce: 3.211472272872925 (3.478035532785747) | > loss: 0.8154551386833191 (0.8810377520715406) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4432 (0.3729824891347372) | > loader_time: 0.009 (0.009938856798731632)  --> TIME: 2024-10-20 01:35:19 -- STEP: 217/3361 -- GLOBAL_STEP: 10300 | > loss_text_ce: 0.04303652048110962 (0.046182117206023054) | > loss_mel_ce: 3.1857612133026123 (3.4779486150785526) | > loss: 0.8071994185447693 (0.8810326841569716) | > current_lr: 5e-06 | > step_time: 0.4433 (0.374492391463249) | > loader_time: 0.0093 (0.009856836037701723)  --> TIME: 2024-10-20 01:35:49 -- STEP: 267/3361 -- GLOBAL_STEP: 10350 | > loss_text_ce: 0.047204095870256424 (0.046089042898979096) | > loss_mel_ce: 3.3309218883514404 (3.4724885845898688) | > loss: 0.8445314764976501 (0.8796444077616774) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3401 (0.37183528178640074) | > loader_time: 0.0087 (0.009844224551197293)  --> TIME: 2024-10-20 01:36:21 -- STEP: 317/3361 -- GLOBAL_STEP: 10400 | > loss_text_ce: 0.045164935290813446 (0.04600713257486889) | > loss_mel_ce: 3.6646311283111572 (3.4749113001657954) | > loss: 0.9274489879608154 (0.880229608117594) | > current_lr: 5e-06 | > step_time: 0.4408 (0.3714463176787464) | > loader_time: 0.0095 (0.009783504137105367)  --> TIME: 2024-10-20 01:36:52 -- STEP: 367/3361 -- GLOBAL_STEP: 10450 | > loss_text_ce: 0.04286389797925949 (0.045877355138190114) | > loss_mel_ce: 3.4389150142669678 (3.4698129999540153) | > loss: 0.8704447150230408 (0.8789225886563189) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4865 (0.3712051090167721) | > loader_time: 0.0108 (0.009785189615608232)  --> TIME: 2024-10-20 01:37:25 -- STEP: 417/3361 -- GLOBAL_STEP: 10500 | > loss_text_ce: 0.048703934997320175 (0.04582360561946027) | > loss_mel_ce: 3.785539388656616 (3.4709189807196594) | > loss: 0.9585608243942261 (0.8791856462721058) | > current_lr: 5e-06 | > step_time: 0.4103 (0.3722521103829219) | > loader_time: 0.0092 (0.009791807590914569)  --> TIME: 2024-10-20 01:37:57 -- STEP: 467/3361 -- GLOBAL_STEP: 10550 | > loss_text_ce: 0.045499563217163086 (0.04583273282398248) | > loss_mel_ce: 3.4453530311584473 (3.465878671956521) | > loss: 0.8727131485939026 (0.8779278508600893) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.2936 (0.37276712423728947) | > loader_time: 0.0088 (0.009811640806831176)  --> TIME: 2024-10-20 01:38:29 -- STEP: 517/3361 -- GLOBAL_STEP: 10600 | > loss_text_ce: 0.04884975031018257 (0.04582477258644436) | > loss_mel_ce: 3.207493543624878 (3.4628814525714917) | > loss: 0.814085841178894 (0.8771765561814003) | > current_lr: 5e-06 | > step_time: 0.3301 (0.3732260878358175) | > loader_time: 0.0091 (0.009831420226991749)  --> TIME: 2024-10-20 01:39:01 -- STEP: 567/3361 -- GLOBAL_STEP: 10650 | > loss_text_ce: 0.04280979186296463 (0.045831149414418224) | > loss_mel_ce: 3.378239154815674 (3.461804541330488) | > loss: 0.8552622199058533 (0.8769089227420732) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3453 (0.3732173430225839) | > loader_time: 0.0093 (0.009825989893084055)  --> TIME: 2024-10-20 01:39:33 -- STEP: 617/3361 -- GLOBAL_STEP: 10700 | > loss_text_ce: 0.04689433425664902 (0.045754293109648035) | > loss_mel_ce: 3.3812503814697266 (3.4619904638689185) | > loss: 0.8570361733436584 (0.876936189262755) | > current_lr: 5e-06 | > step_time: 0.3925 (0.3735560111133942) | > loader_time: 0.0095 (0.009841294883715657)  --> TIME: 2024-10-20 01:40:05 -- STEP: 667/3361 -- GLOBAL_STEP: 10750 | > loss_text_ce: 0.04266295209527016 (0.0456879992244662) | > loss_mel_ce: 3.3630940914154053 (3.46313495364325) | > loss: 0.8514392375946045 (0.8772057384207868) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4245 (0.37379144442671286) | > loader_time: 0.0111 (0.009839120118514344)  --> TIME: 2024-10-20 01:40:38 -- STEP: 717/3361 -- GLOBAL_STEP: 10800 | > loss_text_ce: 0.05172465369105339 (0.04570071628860185) | > loss_mel_ce: 3.2466700077056885 (3.46129972565623) | > loss: 0.8245986700057983 (0.8767501106654584) | > current_lr: 5e-06 | > step_time: 0.341 (0.3741034816332279) | > loader_time: 0.0106 (0.009837797329848106)  --> TIME: 2024-10-20 01:41:09 -- STEP: 767/3361 -- GLOBAL_STEP: 10850 | > loss_text_ce: 0.04403238371014595 (0.045711836324715725) | > loss_mel_ce: 3.8820736408233643 (3.4588085069494734) | > loss: 0.9815264940261841 (0.8761300860358963) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4974 (0.3738556818980751) | > loader_time: 0.0094 (0.009833316914882928)  --> TIME: 2024-10-20 01:41:41 -- STEP: 817/3361 -- GLOBAL_STEP: 10900 | > loss_text_ce: 0.04481789469718933 (0.045687686401160596) | > loss_mel_ce: 3.3437418937683105 (3.4635588652938787) | > loss: 0.8471399545669556 (0.8773116379647973) | > current_lr: 5e-06 | > step_time: 0.3944 (0.37394031287698765) | > loader_time: 0.0095 (0.009831293122421123)  --> TIME: 2024-10-20 01:42:13 -- STEP: 867/3361 -- GLOBAL_STEP: 10950 | > loss_text_ce: 0.05023163557052612 (0.04567214789464126) | > loss_mel_ce: 3.3756706714630127 (3.461011304987343) | > loss: 0.8564755916595459 (0.876670863457892) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4452 (0.3737019953568229) | > loader_time: 0.0094 (0.009818215408677175)  --> TIME: 2024-10-20 01:42:44 -- STEP: 917/3361 -- GLOBAL_STEP: 11000 | > loss_text_ce: 0.04785856604576111 (0.04562381485572559) | > loss_mel_ce: 3.364436149597168 (3.4592360974398146) | > loss: 0.8530736565589905 (0.8762149782191186) | > current_lr: 5e-06 | > step_time: 0.3411 (0.37366046354320887) | > loader_time: 0.0105 (0.009806825914455725) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_11000.pth  --> TIME: 2024-10-20 01:43:25 -- STEP: 967/3361 -- GLOBAL_STEP: 11050 | > loss_text_ce: 0.04600205644965172 (0.04561833730266039) | > loss_mel_ce: 3.5278804302215576 (3.457496856829496) | > loss: 0.8934706449508667 (0.8757787986649845) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3897 (0.37305284360078916) | > loader_time: 0.0098 (0.009819003147571006)  --> TIME: 2024-10-20 01:43:57 -- STEP: 1017/3361 -- GLOBAL_STEP: 11100 | > loss_text_ce: 0.04780234768986702 (0.045599416601622714) | > loss_mel_ce: 3.2322351932525635 (3.454736575498928) | > loss: 0.8200094103813171 (0.8750839981240391) | > current_lr: 5e-06 | > step_time: 0.3965 (0.3729844527024203) | > loader_time: 0.0094 (0.0098143270937039)  --> TIME: 2024-10-20 01:44:28 -- STEP: 1067/3361 -- GLOBAL_STEP: 11150 | > loss_text_ce: 0.04303880035877228 (0.04557502307451655) | > loss_mel_ce: 3.1907825469970703 (3.4526526755893356) | > loss: 0.8084553480148315 (0.8745569246554888) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4266 (0.3729878837188608) | > loader_time: 0.0098 (0.009801903280456593)  --> TIME: 2024-10-20 01:45:00 -- STEP: 1117/3361 -- GLOBAL_STEP: 11200 | > loss_text_ce: 0.04823419824242592 (0.045567612428028044) | > loss_mel_ce: 4.063570022583008 (3.4510888609378183) | > loss: 1.0279510021209717 (0.8741641183889866) | > current_lr: 5e-06 | > step_time: 0.3332 (0.37310730183668855) | > loader_time: 0.0094 (0.009789423101590795)  --> TIME: 2024-10-20 01:45:33 -- STEP: 1167/3361 -- GLOBAL_STEP: 11250 | > loss_text_ce: 0.0466812439262867 (0.04556122947972662) | > loss_mel_ce: 3.205491542816162 (3.4504569024230656) | > loss: 0.8130431771278381 (0.8740045330722614) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3883 (0.3738090622190134) | > loader_time: 0.0095 (0.0098025966323944)  --> TIME: 2024-10-20 01:46:05 -- STEP: 1217/3361 -- GLOBAL_STEP: 11300 | > loss_text_ce: 0.04266681149601936 (0.045536092204717514) | > loss_mel_ce: 3.636619806289673 (3.44905358430491) | > loss: 0.9198216795921326 (0.8736474192720409) | > current_lr: 5e-06 | > step_time: 0.4277 (0.37357596540020077) | > loader_time: 0.0091 (0.009802650602116571)  --> TIME: 2024-10-20 01:46:37 -- STEP: 1267/3361 -- GLOBAL_STEP: 11350 | > loss_text_ce: 0.043237507343292236 (0.04552184921507772) | > loss_mel_ce: 3.1879427433013916 (3.4461687098550535) | > loss: 0.8077950477600098 (0.8729226399145448) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.545 (0.3738300482306601) | > loader_time: 0.0093 (0.009796628259539879)  --> TIME: 2024-10-20 01:47:09 -- STEP: 1317/3361 -- GLOBAL_STEP: 11400 | > loss_text_ce: 0.04297908768057823 (0.045494050124126946) | > loss_mel_ce: 3.098487138748169 (3.445298686929064) | > loss: 0.7853665351867676 (0.8726981843884639) | > current_lr: 5e-06 | > step_time: 0.3311 (0.37390816854362574) | > loader_time: 0.0103 (0.009803349444969358)  --> TIME: 2024-10-20 01:47:41 -- STEP: 1367/3361 -- GLOBAL_STEP: 11450 | > loss_text_ce: 0.04727546498179436 (0.04548580103827029) | > loss_mel_ce: 3.4735140800476074 (3.4428421135967286) | > loss: 0.8801974058151245 (0.8720819787466358) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.483 (0.3738833208021321) | > loader_time: 0.0092 (0.009806743222775898)  --> TIME: 2024-10-20 01:48:13 -- STEP: 1417/3361 -- GLOBAL_STEP: 11500 | > loss_text_ce: 0.04696996882557869 (0.04546378995899845) | > loss_mel_ce: 3.4015119075775146 (3.4421401881969724) | > loss: 0.8621204495429993 (0.8719009946750432) | > current_lr: 5e-06 | > step_time: 0.5081 (0.3737989569013189) | > loader_time: 0.0104 (0.009797040896257537)  --> TIME: 2024-10-20 01:48:45 -- STEP: 1467/3361 -- GLOBAL_STEP: 11550 | > loss_text_ce: 0.044216107577085495 (0.04545340134972362) | > loss_mel_ce: 3.291405439376831 (3.440216698610791) | > loss: 0.8339053988456726 (0.8714175251120193) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4973 (0.3738921473606478) | > loader_time: 0.0108 (0.009800101269160946)  --> TIME: 2024-10-20 01:49:18 -- STEP: 1517/3361 -- GLOBAL_STEP: 11600 | > loss_text_ce: 0.04237224906682968 (0.04541745957235168) | > loss_mel_ce: 3.4833874702453613 (3.437472724883118) | > loss: 0.8814399242401123 (0.8707225460242344) | > current_lr: 5e-06 | > step_time: 0.3329 (0.37426536446271497) | > loader_time: 0.0088 (0.009796220374563125)  --> TIME: 2024-10-20 01:49:50 -- STEP: 1567/3361 -- GLOBAL_STEP: 11650 | > loss_text_ce: 0.043478481471538544 (0.04540074599316033) | > loss_mel_ce: 3.2932302951812744 (3.4390092498569937) | > loss: 0.8341771960258484 (0.8711024989500574) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.5027 (0.3746144473362518) | > loader_time: 0.0094 (0.009793423054396287)  --> TIME: 2024-10-20 01:50:22 -- STEP: 1617/3361 -- GLOBAL_STEP: 11700 | > loss_text_ce: 0.041785042732954025 (0.045383355543984996) | > loss_mel_ce: 3.2549021244049072 (3.43981561501526) | > loss: 0.824171781539917 (0.871299742693951) | > current_lr: 5e-06 | > step_time: 0.4082 (0.3745398368080532) | > loader_time: 0.0097 (0.009789079224982206)  --> TIME: 2024-10-20 01:50:54 -- STEP: 1667/3361 -- GLOBAL_STEP: 11750 | > loss_text_ce: 0.044465482234954834 (0.045369411876227374) | > loss_mel_ce: 3.373613119125366 (3.438965404112323) | > loss: 0.8545196652412415 (0.8710837041323385) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3738 (0.37457816888274886) | > loader_time: 0.0089 (0.009785240255720803)  --> TIME: 2024-10-20 01:51:25 -- STEP: 1717/3361 -- GLOBAL_STEP: 11800 | > loss_text_ce: 0.043654706329107285 (0.04535353077216487) | > loss_mel_ce: 3.4367294311523438 (3.4394294256100353) | > loss: 0.8700960278511047 (0.8711957391015164) | > current_lr: 5e-06 | > step_time: 0.3279 (0.3742698932985408) | > loader_time: 0.0098 (0.009778505851865554)  --> TIME: 2024-10-20 01:51:58 -- STEP: 1767/3361 -- GLOBAL_STEP: 11850 | > loss_text_ce: 0.04286897927522659 (0.045324536488429934) | > loss_mel_ce: 3.091407537460327 (3.4378820990328745) | > loss: 0.7835691571235657 (0.8708016589393571) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4647 (0.3746508205415979) | > loader_time: 0.0096 (0.009779719491428595)  --> TIME: 2024-10-20 01:52:30 -- STEP: 1817/3361 -- GLOBAL_STEP: 11900 | > loss_text_ce: 0.044489096850156784 (0.045306128924881686) | > loss_mel_ce: 3.4446122646331787 (3.434943928584727) | > loss: 0.8722753524780273 (0.8700625143276837) | > current_lr: 5e-06 | > step_time: 0.3748 (0.3747132023940983) | > loader_time: 0.0092 (0.009790742928731135)  --> TIME: 2024-10-20 01:53:01 -- STEP: 1867/3361 -- GLOBAL_STEP: 11950 | > loss_text_ce: 0.04551025852560997 (0.04529719577676227) | > loss_mel_ce: 3.373769521713257 (3.43550654829155) | > loss: 0.8548199534416199 (0.870200935920803) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4817 (0.37446563321763854) | > loader_time: 0.0094 (0.009788677918444847)  --> TIME: 2024-10-20 01:53:34 -- STEP: 1917/3361 -- GLOBAL_STEP: 12000 | > loss_text_ce: 0.03975873813033104 (0.04528394429694257) | > loss_mel_ce: 3.337285280227661 (3.4359212065961873) | > loss: 0.8442609906196594 (0.8703012876542955) | > current_lr: 5e-06 | > step_time: 0.4123 (0.3746762464739224) | > loader_time: 0.0106 (0.009790282480780113) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_12000.pth  --> TIME: 2024-10-20 01:54:17 -- STEP: 1967/3361 -- GLOBAL_STEP: 12050 | > loss_text_ce: 0.04682973399758339 (0.04526911055679605) | > loss_mel_ce: 4.056046962738037 (3.436260929917344) | > loss: 1.025719165802002 (0.8703825101673026) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3722 (0.37507091395189524) | > loader_time: 0.0099 (0.009801731778694135)  --> TIME: 2024-10-20 01:54:49 -- STEP: 2017/3361 -- GLOBAL_STEP: 12100 | > loss_text_ce: 0.04704705625772476 (0.04524998690417691) | > loss_mel_ce: 3.3677549362182617 (3.4361582207809773) | > loss: 0.8537005186080933 (0.8703520520819727) | > current_lr: 5e-06 | > step_time: 0.3057 (0.374842264774003) | > loader_time: 0.0088 (0.00980050267118831)  --> TIME: 2024-10-20 01:55:20 -- STEP: 2067/3361 -- GLOBAL_STEP: 12150 | > loss_text_ce: 0.04836978018283844 (0.04522184339673207) | > loss_mel_ce: 3.221724510192871 (3.4350216229525983) | > loss: 0.8175235986709595 (0.8700608667626032) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4277 (0.37457450934522224) | > loader_time: 0.0085 (0.009801711445565242)  --> TIME: 2024-10-20 01:55:52 -- STEP: 2117/3361 -- GLOBAL_STEP: 12200 | > loss_text_ce: 0.041277926415205 (0.04519546315615956) | > loss_mel_ce: 3.5539653301239014 (3.4345948362778422) | > loss: 0.8988108038902283 (0.8699475751206961) | > current_lr: 5e-06 | > step_time: 0.4406 (0.3744075272406506) | > loader_time: 0.0094 (0.009799530104077175)  --> TIME: 2024-10-20 01:56:24 -- STEP: 2167/3361 -- GLOBAL_STEP: 12250 | > loss_text_ce: 0.043600600212812424 (0.045186322316506745) | > loss_mel_ce: 3.2849605083465576 (3.433843436745053) | > loss: 0.8321402668952942 (0.8697574399394487) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4201 (0.3745438092086082) | > loader_time: 0.0087 (0.009803145174942737)  --> TIME: 2024-10-20 01:56:55 -- STEP: 2217/3361 -- GLOBAL_STEP: 12300 | > loss_text_ce: 0.04543566703796387 (0.04517698947661193) | > loss_mel_ce: 3.4641506671905518 (3.4332194155166964) | > loss: 0.8773965835571289 (0.8695991014541676) | > current_lr: 5e-06 | > step_time: 0.3439 (0.374354049543244) | > loader_time: 0.0094 (0.009805234961537642)  --> TIME: 2024-10-20 01:57:28 -- STEP: 2267/3361 -- GLOBAL_STEP: 12350 | > loss_text_ce: 0.04675550386309624 (0.0451725262492882) | > loss_mel_ce: 3.3266305923461914 (3.4326016073138597) | > loss: 0.8433465361595154 (0.8694435337013645) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3468 (0.3745551318600675) | > loader_time: 0.009 (0.009808486392438046)  --> TIME: 2024-10-20 01:57:59 -- STEP: 2317/3361 -- GLOBAL_STEP: 12400 | > loss_text_ce: 0.04151281341910362 (0.04515179921782474) | > loss_mel_ce: 3.1535322666168213 (3.4317728494537976) | > loss: 0.798761248588562 (0.8692311624745946) | > current_lr: 5e-06 | > step_time: 0.3418 (0.37451987005175325) | > loader_time: 0.0086 (0.00980887569432427)  --> TIME: 2024-10-20 01:58:32 -- STEP: 2367/3361 -- GLOBAL_STEP: 12450 | > loss_text_ce: 0.04229551553726196 (0.045148368671467175) | > loss_mel_ce: 3.2210004329681396 (3.430814805794326) | > loss: 0.8158239722251892 (0.8689907938808539) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4828 (0.3747464335312257) | > loader_time: 0.0105 (0.00981024382332984)  --> TIME: 2024-10-20 01:59:04 -- STEP: 2417/3361 -- GLOBAL_STEP: 12500 | > loss_text_ce: 0.0443447045981884 (0.04513368604687011) | > loss_mel_ce: 3.286813735961914 (3.4296375612212353) | > loss: 0.8327895998954773 (0.8686928119992832) | > current_lr: 5e-06 | > step_time: 0.3053 (0.37469193814064194) | > loader_time: 0.01 (0.00981032606684376)  --> TIME: 2024-10-20 01:59:36 -- STEP: 2467/3361 -- GLOBAL_STEP: 12550 | > loss_text_ce: 0.045515257865190506 (0.04512597635587699) | > loss_mel_ce: 3.5041096210479736 (3.428387396239925) | > loss: 0.8874062299728394 (0.8683783433241158) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3342 (0.37473538224398517) | > loader_time: 0.0099 (0.009815803654627791)  --> TIME: 2024-10-20 02:00:09 -- STEP: 2517/3361 -- GLOBAL_STEP: 12600 | > loss_text_ce: 0.0403006412088871 (0.04510258801834989) | > loss_mel_ce: 3.3379220962524414 (3.4280766162978407) | > loss: 0.844555675983429 (0.8682948011560099) | > current_lr: 5e-06 | > step_time: 0.3751 (0.37487539072381565) | > loader_time: 0.0109 (0.009816988242918111)  --> TIME: 2024-10-20 02:00:42 -- STEP: 2567/3361 -- GLOBAL_STEP: 12650 | > loss_text_ce: 0.04288014397025108 (0.04509773093918249) | > loss_mel_ce: 3.288858652114868 (3.4260108755124326) | > loss: 0.8329346776008606 (0.8677771517075957) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.389 (0.3752143952370065) | > loader_time: 0.0097 (0.009822266084190264)  --> TIME: 2024-10-20 02:01:13 -- STEP: 2617/3361 -- GLOBAL_STEP: 12700 | > loss_text_ce: 0.04661820828914642 (0.04508226625777894) | > loss_mel_ce: 3.3759257793426514 (3.426038921995385) | > loss: 0.8556360006332397 (0.8677802970949635) | > current_lr: 5e-06 | > step_time: 0.2947 (0.37497939952329973) | > loader_time: 0.0094 (0.0098253846122966)  --> TIME: 2024-10-20 02:01:45 -- STEP: 2667/3361 -- GLOBAL_STEP: 12750 | > loss_text_ce: 0.044218242168426514 (0.04506286233104825) | > loss_mel_ce: 3.6532299518585205 (3.4246745852437144) | > loss: 0.924362063407898 (0.8674343619223249) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4868 (0.3750956985536568) | > loader_time: 0.0092 (0.009827119740020204)  --> TIME: 2024-10-20 02:02:16 -- STEP: 2717/3361 -- GLOBAL_STEP: 12800 | > loss_text_ce: 0.041796378791332245 (0.04505104149143652) | > loss_mel_ce: 3.069788932800293 (3.4237373812390812) | > loss: 0.7778963446617126 (0.8671971057371305) | > current_lr: 5e-06 | > step_time: 0.4088 (0.3748981202398981) | > loader_time: 0.0107 (0.00982653598904741)  --> TIME: 2024-10-20 02:02:49 -- STEP: 2767/3361 -- GLOBAL_STEP: 12850 | > loss_text_ce: 0.04316914081573486 (0.04503791029644125) | > loss_mel_ce: 3.131343364715576 (3.423758802320882) | > loss: 0.7936280965805054 (0.8671991782452078) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4655 (0.37505522074146214) | > loader_time: 0.0104 (0.009828401206416)  --> TIME: 2024-10-20 02:03:20 -- STEP: 2817/3361 -- GLOBAL_STEP: 12900 | > loss_text_ce: 0.046656232327222824 (0.04501954110899327) | > loss_mel_ce: 3.635425090789795 (3.4238399453818227) | > loss: 0.9205203056335449 (0.8672148717351104) | > current_lr: 5e-06 | > step_time: 0.3918 (0.3749253896267057) | > loader_time: 0.0089 (0.009831450448461057)  --> TIME: 2024-10-20 02:03:51 -- STEP: 2867/3361 -- GLOBAL_STEP: 12950 | > loss_text_ce: 0.04399809241294861 (0.044997297027279795) | > loss_mel_ce: 3.229576587677002 (3.422730980328579) | > loss: 0.8183936476707458 (0.8669320693779454) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.3898 (0.3745152650912297) | > loader_time: 0.0092 (0.009833865686289541)  --> TIME: 2024-10-20 02:04:22 -- STEP: 2917/3361 -- GLOBAL_STEP: 13000 | > loss_text_ce: 0.045591291040182114 (0.04497031228445057) | > loss_mel_ce: 3.6106908321380615 (3.422237732004212) | > loss: 0.9140705466270447 (0.8668020111590079) | > current_lr: 5e-06 | > step_time: 0.3444 (0.3744682490028744) | > loader_time: 0.0102 (0.009837184820348432) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_13000.pth  --> TIME: 2024-10-20 02:05:05 -- STEP: 2967/3361 -- GLOBAL_STEP: 13050 | > loss_text_ce: 0.0435701459646225 (0.044948636828838484) | > loss_mel_ce: 3.2321619987487793 (3.4209875995396843) | > loss: 0.8189330101013184 (0.866484059140784) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4589 (0.37451388074928227) | > loader_time: 0.0106 (0.009842868200005756)  --> TIME: 2024-10-20 02:05:38 -- STEP: 3017/3361 -- GLOBAL_STEP: 13100 | > loss_text_ce: 0.04389602690935135 (0.04493448949231579) | > loss_mel_ce: 3.2054622173309326 (3.4204812088587997) | > loss: 0.8123395442962646 (0.8663539246569255) | > current_lr: 5e-06 | > step_time: 0.3433 (0.37459014953269354) | > loader_time: 0.0092 (0.0098414506428598)  --> TIME: 2024-10-20 02:06:10 -- STEP: 3067/3361 -- GLOBAL_STEP: 13150 | > loss_text_ce: 0.04617796465754509 (0.044920595558946216) | > loss_mel_ce: 3.227992296218872 (3.4195069690694395) | > loss: 0.8185425400733948 (0.8661068912278488) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4677 (0.3747628785361285) | > loader_time: 0.0093 (0.009840129678786514)  --> TIME: 2024-10-20 02:06:42 -- STEP: 3117/3361 -- GLOBAL_STEP: 13200 | > loss_text_ce: 0.042707689106464386 (0.04491019368955706) | > loss_mel_ce: 3.2197930812835693 (3.4186071596124212) | > loss: 0.8156251907348633 (0.86587933833147) | > current_lr: 5e-06 | > step_time: 0.409 (0.37470273769624646) | > loader_time: 0.0107 (0.009839347895040002)  --> TIME: 2024-10-20 02:07:14 -- STEP: 3167/3361 -- GLOBAL_STEP: 13250 | > loss_text_ce: 0.04247237741947174 (0.044894521159067204) | > loss_mel_ce: 3.3792779445648193 (3.4170680701036273) | > loss: 0.8554375767707825 (0.8654906478709588) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.4468 (0.37466244830067386) | > loader_time: 0.0113 (0.009841234103565064)  --> TIME: 2024-10-20 02:07:45 -- STEP: 3217/3361 -- GLOBAL_STEP: 13300 | > loss_text_ce: 0.04384773224592209 (0.0448808852313355) | > loss_mel_ce: 3.4374778270721436 (3.4173526868414057) | > loss: 0.8703314065933228 (0.8655583930934551) | > current_lr: 5e-06 | > step_time: 0.3324 (0.37453380481849213) | > loader_time: 0.0105 (0.009846873115892904)  --> TIME: 2024-10-20 02:08:18 -- STEP: 3267/3361 -- GLOBAL_STEP: 13350 | > loss_text_ce: 0.0455152653157711 (0.044865900441379274) | > loss_mel_ce: 3.3864834308624268 (3.416893625463737) | > loss: 0.8579996824264526 (0.8654398815253108) | > grad_norm: 0 (0.0) | > current_lr: 5e-06 | > step_time: 0.444 (0.37472096641092145) | > loader_time: 0.011 (0.009851177132089519)  --> TIME: 2024-10-20 02:08:49 -- STEP: 3317/3361 -- GLOBAL_STEP: 13400 | > loss_text_ce: 0.04139714315533638 (0.04483872734713268) | > loss_mel_ce: 3.251816987991333 (3.416392544216516) | > loss: 0.8233035206794739 (0.8653078179852515) | > current_lr: 5e-06 | > step_time: 0.2561 (0.3746120028370721) | > loader_time: 0.0093 (0.009851044475726763)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.3778952552426247 (+0.0006334499646259428) | > avg_loss_text_ce: 0.04140995150452017 (-0.0020317246596659524) | > avg_loss_mel_ce: 3.1711383763179986 (-0.1000511492452314) | > avg_loss: 3.21254833539327 (-0.10208286264891253) > BEST MODEL : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/best_model_13444.pth  > EPOCH: 4/5 --> /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b  > TRAINING (2024-10-20 02:11:46)   --> TIME: 2024-10-20 02:11:51 -- STEP: 6/3361 -- GLOBAL_STEP: 13450 | > loss_text_ce: 0.04218747466802597 (0.04258658178150654) | > loss_mel_ce: 3.332731008529663 (3.278356671333313) | > loss: 0.8437296152114868 (0.8302358090877533) | > current_lr: 5e-06 | > step_time: 0.3784 (0.34697389602661133) | > loader_time: 0.0096 (0.010201772054036459)  --> TIME: 2024-10-20 02:12:23 -- STEP: 56/3361 -- GLOBAL_STEP: 13500 | > loss_text_ce: 0.045623574405908585 (0.04398354063076631) | > loss_mel_ce: 3.8470444679260254 (3.3895332387515476) | > loss: 0.9731670022010803 (0.85837919903653) | > current_lr: 5e-06 | > step_time: 0.3448 (0.3710901652063643) | > loader_time: 0.0103 (0.009938184704099384)  --> TIME: 2024-10-20 02:12:55 -- STEP: 106/3361 -- GLOBAL_STEP: 13550 | > loss_text_ce: 0.04445193335413933 (0.043825697575537656) | > loss_mel_ce: 3.2017033100128174 (3.3736256473469286) | > loss: 0.811538815498352 (0.8543628392354498) | > current_lr: 5e-06 | > step_time: 0.2652 (0.37404405395939677) | > loader_time: 0.009 (0.00981641490504427)  --> TIME: 2024-10-20 02:13:27 -- STEP: 156/3361 -- GLOBAL_STEP: 13600 | > loss_text_ce: 0.04687873274087906 (0.04368266580292046) | > loss_mel_ce: 3.556082248687744 (3.373449661792852) | > loss: 0.9007402658462524 (0.8542830833257773) | > current_lr: 5e-06 | > step_time: 0.3338 (0.3716987386727945) | > loader_time: 0.0096 (0.009778794569846915)  --> TIME: 2024-10-20 02:13:59 -- STEP: 206/3361 -- GLOBAL_STEP: 13650 | > loss_text_ce: 0.04147164896130562 (0.04358611028026608) | > loss_mel_ce: 3.6620306968688965 (3.3716245033208603) | > loss: 0.9258756041526794 (0.8538026543496882) | > current_lr: 5e-06 | > step_time: 0.4133 (0.3729133767989075) | > loader_time: 0.0097 (0.009783061962683217)  --> TIME: 2024-10-20 02:14:32 -- STEP: 256/3361 -- GLOBAL_STEP: 13700 | > loss_text_ce: 0.04409633204340935 (0.043609272077446797) | > loss_mel_ce: 3.1696982383728027 (3.3641294278204437) | > loss: 0.8034486174583435 (0.8519346755929291) | > current_lr: 5e-06 | > step_time: 0.2645 (0.3750447416678071) | > loader_time: 0.0107 (0.009823349304497237)  --> TIME: 2024-10-20 02:15:04 -- STEP: 306/3361 -- GLOBAL_STEP: 13750 | > loss_text_ce: 0.04763133078813553 (0.04354089683566998) | > loss_mel_ce: 3.3492703437805176 (3.358471124000797) | > loss: 0.8492254018783569 (0.850503006012611) | > current_lr: 5e-06 | > step_time: 0.3438 (0.37534330483355555) | > loader_time: 0.0095 (0.009806627541585686)  --> TIME: 2024-10-20 02:15:36 -- STEP: 356/3361 -- GLOBAL_STEP: 13800 | > loss_text_ce: 0.040769223123788834 (0.04356232056331437) | > loss_mel_ce: 3.5312817096710205 (3.358752216515915) | > loss: 0.8930127620697021 (0.8505786349264421) | > current_lr: 5e-06 | > step_time: 0.3887 (0.3760112068626318) | > loader_time: 0.01 (0.009826761283231584)  --> TIME: 2024-10-20 02:16:08 -- STEP: 406/3361 -- GLOBAL_STEP: 13850 | > loss_text_ce: 0.046686802059412 (0.04353852070904719) | > loss_mel_ce: 3.1642913818359375 (3.35460764962464) | > loss: 0.8027445673942566 (0.8495365434092252) | > current_lr: 5e-06 | > step_time: 0.3028 (0.37603559106441553) | > loader_time: 0.01 (0.009861112815405927)  --> TIME: 2024-10-20 02:16:40 -- STEP: 456/3361 -- GLOBAL_STEP: 13900 | > loss_text_ce: 0.04211386665701866 (0.043516929048979504) | > loss_mel_ce: 3.4279661178588867 (3.3584800671067145) | > loss: 0.8675199747085571 (0.8504992499947547) | > current_lr: 5e-06 | > step_time: 0.3128 (0.37571073101277935) | > loader_time: 0.0086 (0.009866837346762934)  --> TIME: 2024-10-20 02:17:13 -- STEP: 506/3361 -- GLOBAL_STEP: 13950 | > loss_text_ce: 0.043664440512657166 (0.04350922338428942) | > loss_mel_ce: 3.516002893447876 (3.365730038744658) | > loss: 0.8899168372154236 (0.8523098163218366) | > current_lr: 5e-06 | > step_time: 0.376 (0.3766041942264723) | > loader_time: 0.0107 (0.009907983037322871)  --> TIME: 2024-10-20 02:17:44 -- STEP: 556/3361 -- GLOBAL_STEP: 14000 | > loss_text_ce: 0.038978710770606995 (0.043512208420756496) | > loss_mel_ce: 3.1659882068634033 (3.360013530408735) | > loss: 0.8012417554855347 (0.850881435030656) | > current_lr: 5e-06 | > step_time: 0.3549 (0.37569527231532024) | > loader_time: 0.0109 (0.009921835480834078) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_14000.pth  --> TIME: 2024-10-20 02:18:28 -- STEP: 606/3361 -- GLOBAL_STEP: 14050 | > loss_text_ce: 0.03997735306620598 (0.04353958326499454) | > loss_mel_ce: 3.1434693336486816 (3.365101049048671) | > loss: 0.7958616614341736 (0.8521601585271732) | > current_lr: 5e-06 | > step_time: 0.3924 (0.37639728040978465) | > loader_time: 0.0092 (0.009938992289426695)  --> TIME: 2024-10-20 02:19:00 -- STEP: 656/3361 -- GLOBAL_STEP: 14100 | > loss_text_ce: 0.04379415884613991 (0.04352298771349216) | > loss_mel_ce: 3.4719841480255127 (3.364289275029809) | > loss: 0.8789445757865906 (0.8519530663403072) | > current_lr: 5e-06 | > step_time: 0.2412 (0.37598983107543565) | > loader_time: 0.0094 (0.009922782458910127)  --> TIME: 2024-10-20 02:19:32 -- STEP: 706/3361 -- GLOBAL_STEP: 14150 | > loss_text_ce: 0.04108851030468941 (0.04352049126946724) | > loss_mel_ce: 3.5078001022338867 (3.365766045392065) | > loss: 0.887222170829773 (0.8523216351402383) | > current_lr: 5e-06 | > step_time: 0.432 (0.3764305361269554) | > loader_time: 0.0099 (0.009930196970110234)  --> TIME: 2024-10-20 02:20:04 -- STEP: 756/3361 -- GLOBAL_STEP: 14200 | > loss_text_ce: 0.041327524930238724 (0.0434930988306564) | > loss_mel_ce: 3.2496447563171387 (3.363260011508981) | > loss: 0.8227430582046509 (0.8516882787305845) | > current_lr: 5e-06 | > step_time: 0.329 (0.37616385890062526) | > loader_time: 0.0095 (0.009909986188172033)  --> TIME: 2024-10-20 02:20:36 -- STEP: 806/3361 -- GLOBAL_STEP: 14250 | > loss_text_ce: 0.03913938254117966 (0.043490528325473995) | > loss_mel_ce: 3.4655745029449463 (3.364514757918364) | > loss: 0.8761784434318542 (0.8520013225581453) | > current_lr: 5e-06 | > step_time: 0.395 (0.37637911570575044) | > loader_time: 0.0101 (0.009920249505906784)  --> TIME: 2024-10-20 02:21:08 -- STEP: 856/3361 -- GLOBAL_STEP: 14300 | > loss_text_ce: 0.041906122118234634 (0.043468420245797834) | > loss_mel_ce: 3.5537428855895996 (3.3637058712611676) | > loss: 0.898912250995636 (0.8517935738385284) | > current_lr: 5e-06 | > step_time: 0.3119 (0.37598651675420386) | > loader_time: 0.0104 (0.009900728954333015)  --> TIME: 2024-10-20 02:21:39 -- STEP: 906/3361 -- GLOBAL_STEP: 14350 | > loss_text_ce: 0.043285299092531204 (0.043472079852098396) | > loss_mel_ce: 3.286458730697632 (3.363266183314207) | > loss: 0.8324360251426697 (0.8516845666283279) | > current_lr: 5e-06 | > step_time: 0.4279 (0.37519479271591893) | > loader_time: 0.0142 (0.009895196550466107)  --> TIME: 2024-10-20 02:22:10 -- STEP: 956/3361 -- GLOBAL_STEP: 14400 | > loss_text_ce: 0.041109610348939896 (0.043473483180183) | > loss_mel_ce: 3.326003313064575 (3.3631330122508754) | > loss: 0.8417782187461853 (0.8516516245309281) | > current_lr: 5e-06 | > step_time: 0.2971 (0.374720933297688) | > loader_time: 0.0116 (0.00989291059422194)  --> TIME: 2024-10-20 02:22:42 -- STEP: 1006/3361 -- GLOBAL_STEP: 14450 | > loss_text_ce: 0.04267358034849167 (0.04345590186492349) | > loss_mel_ce: 3.3953325748443604 (3.3609538547561364) | > loss: 0.8595015406608582 (0.8511024397125777) | > current_lr: 5e-06 | > step_time: 0.3428 (0.37477593156499844) | > loader_time: 0.009 (0.009898023387309812)  --> TIME: 2024-10-20 02:23:14 -- STEP: 1056/3361 -- GLOBAL_STEP: 14500 | > loss_text_ce: 0.039795197546482086 (0.04344303786020839) | > loss_mel_ce: 3.1137518882751465 (3.35917877676812) | > loss: 0.7883867621421814 (0.850655454295603) | > current_lr: 5e-06 | > step_time: 0.296 (0.3749279420484196) | > loader_time: 0.0094 (0.009898246237725925)  --> TIME: 2024-10-20 02:23:46 -- STEP: 1106/3361 -- GLOBAL_STEP: 14550 | > loss_text_ce: 0.04497344419360161 (0.04342685892140564) | > loss_mel_ce: 3.1941349506378174 (3.3563017004437294) | > loss: 0.8097770810127258 (0.8499321404029407) | > current_lr: 5e-06 | > step_time: 0.4831 (0.37497546961682615) | > loader_time: 0.0094 (0.009898357322375557)  --> TIME: 2024-10-20 02:24:18 -- STEP: 1156/3361 -- GLOBAL_STEP: 14600 | > loss_text_ce: 0.05118125304579735 (0.04341617007143668) | > loss_mel_ce: 3.0756499767303467 (3.3546003092944003) | > loss: 0.7817078232765198 (0.8495041204674436) | > current_lr: 5e-06 | > step_time: 0.3348 (0.37483308781389535) | > loader_time: 0.011 (0.009883925370279068)  --> TIME: 2024-10-20 02:24:49 -- STEP: 1206/3361 -- GLOBAL_STEP: 14650 | > loss_text_ce: 0.047178130596876144 (0.04340132807148237) | > loss_mel_ce: 3.2075843811035156 (3.353524711594653) | > loss: 0.8136906027793884 (0.8492315104648253) | > current_lr: 5e-06 | > step_time: 0.3394 (0.3744580769420263) | > loader_time: 0.0108 (0.009874612735476261)  --> TIME: 2024-10-20 02:25:21 -- STEP: 1256/3361 -- GLOBAL_STEP: 14700 | > loss_text_ce: 0.040922343730926514 (0.04340411143697754) | > loss_mel_ce: 3.0344512462615967 (3.351216148039338) | > loss: 0.768843412399292 (0.8486550655334623) | > current_lr: 5e-06 | > step_time: 0.3548 (0.37435099767271907) | > loader_time: 0.0092 (0.009869077023427202)  --> TIME: 2024-10-20 02:25:54 -- STEP: 1306/3361 -- GLOBAL_STEP: 14750 | > loss_text_ce: 0.043831005692481995 (0.04338768764953796) | > loss_mel_ce: 3.3428115844726562 (3.3530943813586855) | > loss: 0.8466606736183167 (0.849120517876028) | > current_lr: 5e-06 | > step_time: 0.4858 (0.3746798794265914) | > loader_time: 0.0096 (0.009866920401089782)  --> TIME: 2024-10-20 02:26:26 -- STEP: 1356/3361 -- GLOBAL_STEP: 14800 | > loss_text_ce: 0.04674549773335457 (0.04338476714021519) | > loss_mel_ce: 3.245988368988037 (3.3537734330227944) | > loss: 0.8231834769248962 (0.8492895506300409) | > current_lr: 5e-06 | > step_time: 0.3845 (0.3747306175991501) | > loader_time: 0.0092 (0.009857949078258876)  --> TIME: 2024-10-20 02:26:57 -- STEP: 1406/3361 -- GLOBAL_STEP: 14850 | > loss_text_ce: 0.04541696980595589 (0.04338177127411242) | > loss_mel_ce: 3.1508896350860596 (3.3522465773361336) | > loss: 0.7990766763687134 (0.8489070875970266) | > current_lr: 5e-06 | > step_time: 0.3409 (0.3744548187825627) | > loader_time: 0.0093 (0.009856832146135888)  --> TIME: 2024-10-20 02:27:28 -- STEP: 1456/3361 -- GLOBAL_STEP: 14900 | > loss_text_ce: 0.0430903360247612 (0.043365632403887584) | > loss_mel_ce: 3.4218108654022217 (3.349643322792682) | > loss: 0.8662253022193909 (0.8482522392993448) | > current_lr: 5e-06 | > step_time: 0.2738 (0.3738230476667597) | > loader_time: 0.0107 (0.009860832448844078)  --> TIME: 2024-10-20 02:28:00 -- STEP: 1506/3361 -- GLOBAL_STEP: 14950 | > loss_text_ce: 0.041327379643917084 (0.0433498491517022) | > loss_mel_ce: 3.213487148284912 (3.3493747807752245) | > loss: 0.8137036561965942 (0.8481811579084336) | > current_lr: 5e-06 | > step_time: 0.4443 (0.37404442419252276) | > loader_time: 0.0112 (0.009870298988474004)  --> TIME: 2024-10-20 02:28:32 -- STEP: 1556/3361 -- GLOBAL_STEP: 15000 | > loss_text_ce: 0.043370772153139114 (0.0433355381276583) | > loss_mel_ce: 3.200894832611084 (3.348205763607221) | > loss: 0.8110663890838623 (0.8478853257042587) | > current_lr: 5e-06 | > step_time: 0.3503 (0.37390149574353376) | > loader_time: 0.0099 (0.009879069493791434) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_15000.pth  --> TIME: 2024-10-20 02:29:13 -- STEP: 1606/3361 -- GLOBAL_STEP: 15050 | > loss_text_ce: 0.040539246052503586 (0.04333394316081909) | > loss_mel_ce: 3.2453808784484863 (3.3468510259875326) | > loss: 0.8214800357818604 (0.8475462425393455) | > current_lr: 5e-06 | > step_time: 0.3425 (0.3736290137467917) | > loader_time: 0.0094 (0.009883364289071158)  --> TIME: 2024-10-20 02:29:44 -- STEP: 1656/3361 -- GLOBAL_STEP: 15100 | > loss_text_ce: 0.041311707347631454 (0.043320907633031765) | > loss_mel_ce: 3.5958211421966553 (3.3458771780493177) | > loss: 0.9092832207679749 (0.8472995217079704) | > current_lr: 5e-06 | > step_time: 0.3133 (0.37350795277650845) | > loader_time: 0.0111 (0.00987322347751562)  --> TIME: 2024-10-20 02:30:16 -- STEP: 1706/3361 -- GLOBAL_STEP: 15150 | > loss_text_ce: 0.04214273765683174 (0.04329393436566677) | > loss_mel_ce: 3.3591790199279785 (3.34583445772057) | > loss: 0.8503304123878479 (0.8472820981585828) | > current_lr: 5e-06 | > step_time: 0.4414 (0.37350877675753025) | > loader_time: 0.0092 (0.009861578136368064)  --> TIME: 2024-10-20 02:30:47 -- STEP: 1756/3361 -- GLOBAL_STEP: 15200 | > loss_text_ce: 0.043406903743743896 (0.04328855383793287) | > loss_mel_ce: 3.467005491256714 (3.3466667384927526) | > loss: 0.8776031136512756 (0.8474888233361866) | > current_lr: 5e-06 | > step_time: 0.333 (0.37319373011860635) | > loader_time: 0.0093 (0.009854932037735853)  --> TIME: 2024-10-20 02:31:18 -- STEP: 1806/3361 -- GLOBAL_STEP: 15250 | > loss_text_ce: 0.04122260957956314 (0.043277650508423186) | > loss_mel_ce: 3.6228456497192383 (3.3469821526230636) | > loss: 0.9160170555114746 (0.8475649511299262) | > current_lr: 5e-06 | > step_time: 0.3422 (0.37290383480976097) | > loader_time: 0.0092 (0.009851938466297033)  --> TIME: 2024-10-20 02:31:50 -- STEP: 1856/3361 -- GLOBAL_STEP: 15300 | > loss_text_ce: 0.04111756384372711 (0.04326138178544954) | > loss_mel_ce: 3.236495018005371 (3.345419908777393) | > loss: 0.8194031715393066 (0.8471703228765523) | > current_lr: 5e-06 | > step_time: 0.2677 (0.3728802304329547) | > loader_time: 0.0108 (0.009853955101350256)  --> TIME: 2024-10-20 02:32:23 -- STEP: 1906/3361 -- GLOBAL_STEP: 15350 | > loss_text_ce: 0.04561467841267586 (0.0432446542482311) | > loss_mel_ce: 3.1759002208709717 (3.343184289877212) | > loss: 0.8053787350654602 (0.8466072363343096) | > current_lr: 5e-06 | > step_time: 0.3718 (0.37316976939518615) | > loader_time: 0.0108 (0.009854661330095991)  --> TIME: 2024-10-20 02:32:54 -- STEP: 1956/3361 -- GLOBAL_STEP: 15400 | > loss_text_ce: 0.04514142870903015 (0.043227853807706405) | > loss_mel_ce: 3.7339606285095215 (3.343681211120512) | > loss: 0.9447755217552185 (0.8467272666639104) | > current_lr: 5e-06 | > step_time: 0.3132 (0.3729808320784621) | > loader_time: 0.0094 (0.009854103280478945)  --> TIME: 2024-10-20 02:33:27 -- STEP: 2006/3361 -- GLOBAL_STEP: 15450 | > loss_text_ce: 0.044246867299079895 (0.04321134599447547) | > loss_mel_ce: 3.073195457458496 (3.3438472198703115) | > loss: 0.7793605923652649 (0.8467646420061412) | > current_lr: 5e-06 | > step_time: 0.3289 (0.3733023584542701) | > loader_time: 0.0102 (0.009853981903281548)  --> TIME: 2024-10-20 02:33:59 -- STEP: 2056/3361 -- GLOBAL_STEP: 15500 | > loss_text_ce: 0.04231102764606476 (0.04320369416608653) | > loss_mel_ce: 3.146184206008911 (3.3436386884418443) | > loss: 0.7971237897872925 (0.8467105962191113) | > current_lr: 5e-06 | > step_time: 0.3296 (0.3735750739676483) | > loader_time: 0.0112 (0.009853950154456646)  --> TIME: 2024-10-20 02:34:32 -- STEP: 2106/3361 -- GLOBAL_STEP: 15550 | > loss_text_ce: 0.04851767048239708 (0.043201351097845966) | > loss_mel_ce: 3.2179629802703857 (3.3433823642114855) | > loss: 0.8166201710700989 (0.8466459293367745) | > current_lr: 5e-06 | > step_time: 0.3399 (0.3736674039565378) | > loader_time: 0.0097 (0.00985253649672665)  --> TIME: 2024-10-20 02:35:03 -- STEP: 2156/3361 -- GLOBAL_STEP: 15600 | > loss_text_ce: 0.04181588441133499 (0.04318926972207477) | > loss_mel_ce: 3.297445058822632 (3.3427740298971833) | > loss: 0.834815263748169 (0.8464908254832197) | > current_lr: 5e-06 | > step_time: 0.3499 (0.37356952944138105) | > loader_time: 0.0099 (0.009845251497399609)  --> TIME: 2024-10-20 02:35:35 -- STEP: 2206/3361 -- GLOBAL_STEP: 15650 | > loss_text_ce: 0.04130377620458603 (0.04318598364011223) | > loss_mel_ce: 3.1319963932037354 (3.3422171480528133) | > loss: 0.7933250665664673 (0.8463507834501085) | > current_lr: 5e-06 | > step_time: 0.3916 (0.37357278088494483) | > loader_time: 0.0098 (0.009844925872650562)  --> TIME: 2024-10-20 02:36:07 -- STEP: 2256/3361 -- GLOBAL_STEP: 15700 | > loss_text_ce: 0.04426877945661545 (0.04318243447453417) | > loss_mel_ce: 3.1627094745635986 (3.3417251985123806) | > loss: 0.8017445802688599 (0.8462269087949542) | > current_lr: 5e-06 | > step_time: 0.2747 (0.3734862113252605) | > loader_time: 0.0095 (0.009843703597149952)  --> TIME: 2024-10-20 02:36:39 -- STEP: 2306/3361 -- GLOBAL_STEP: 15750 | > loss_text_ce: 0.041298702359199524 (0.04315780556671066) | > loss_mel_ce: 3.124805212020874 (3.3405950526413655) | > loss: 0.7915259599685669 (0.8459382151792287) | > current_lr: 5e-06 | > step_time: 0.4433 (0.37359404553564807) | > loader_time: 0.0129 (0.009846728879890538)  --> TIME: 2024-10-20 02:37:11 -- STEP: 2356/3361 -- GLOBAL_STEP: 15800 | > loss_text_ce: 0.04402311518788338 (0.043152859462946884) | > loss_mel_ce: 3.4008374214172363 (3.340425018644898) | > loss: 0.8612151145935059 (0.845894470127815) | > current_lr: 5e-06 | > step_time: 0.3286 (0.37368188477129255) | > loader_time: 0.0106 (0.009847130658063901)  --> TIME: 2024-10-20 02:37:43 -- STEP: 2406/3361 -- GLOBAL_STEP: 15850 | > loss_text_ce: 0.03998379036784172 (0.04313613363818037) | > loss_mel_ce: 3.198335886001587 (3.3398356255351334) | > loss: 0.8095799088478088 (0.8457429404320169) | > current_lr: 5e-06 | > step_time: 0.4964 (0.3737038702937041) | > loader_time: 0.0094 (0.009845961259983025)  --> TIME: 2024-10-20 02:38:16 -- STEP: 2456/3361 -- GLOBAL_STEP: 15900 | > loss_text_ce: 0.040953829884529114 (0.043121636062496436) | > loss_mel_ce: 3.21343731880188 (3.339520607107234) | > loss: 0.8135977983474731 (0.8456605613862657) | > current_lr: 5e-06 | > step_time: 0.3581 (0.37380436975326947) | > loader_time: 0.0097 (0.009839947429464382)  --> TIME: 2024-10-20 02:38:48 -- STEP: 2506/3361 -- GLOBAL_STEP: 15950 | > loss_text_ce: 0.0415506437420845 (0.04310880687625903) | > loss_mel_ce: 3.3829474449157715 (3.338792998697502) | > loss: 0.8561245203018188 (0.8454754520014396) | > current_lr: 5e-06 | > step_time: 0.4815 (0.37399479996558715) | > loader_time: 0.009 (0.00982972097130461)  --> TIME: 2024-10-20 02:39:20 -- STEP: 2556/3361 -- GLOBAL_STEP: 16000 | > loss_text_ce: 0.04153384268283844 (0.04309028748230475) | > loss_mel_ce: 3.4101197719573975 (3.337869835011833) | > loss: 0.8629134297370911 (0.8452400312364011) | > current_lr: 5e-06 | > step_time: 0.3321 (0.3739765604523615) | > loader_time: 0.0093 (0.009837146003108253) > CHECKPOINT : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/checkpoint_16000.pth  --> TIME: 2024-10-20 02:40:02 -- STEP: 2606/3361 -- GLOBAL_STEP: 16050 | > loss_text_ce: 0.044827427715063095 (0.0430758015159017) | > loss_mel_ce: 3.3700437545776367 (3.3376472514131272) | > loss: 0.8537178039550781 (0.8451807638451586) | > current_lr: 5e-06 | > step_time: 0.3326 (0.3737125197283966) | > loader_time: 0.0104 (0.00983854037289974)  --> TIME: 2024-10-20 02:40:34 -- STEP: 2656/3361 -- GLOBAL_STEP: 16100 | > loss_text_ce: 0.04216808080673218 (0.04306527884138452) | > loss_mel_ce: 3.410971164703369 (3.3377770967332685) | > loss: 0.8632848262786865 (0.8452105945178188) | > current_lr: 5e-06 | > step_time: 0.33 (0.37373760216925517) | > loader_time: 0.0109 (0.009834927966795767)  --> TIME: 2024-10-20 02:41:06 -- STEP: 2706/3361 -- GLOBAL_STEP: 16150 | > loss_text_ce: 0.044430822134017944 (0.0430597535404049) | > loss_mel_ce: 3.2338714599609375 (3.337480453852801) | > loss: 0.8195755481719971 (0.8451350524702167) | > current_lr: 5e-06 | > step_time: 0.3061 (0.37378184317837615) | > loader_time: 0.0099 (0.009834342503318588)  --> TIME: 2024-10-20 02:41:38 -- STEP: 2756/3361 -- GLOBAL_STEP: 16200 | > loss_text_ce: 0.04327122122049332 (0.04304295761918937) | > loss_mel_ce: 2.923125982284546 (3.3371628782399654) | > loss: 0.7415993213653564 (0.8450514595031049) | > current_lr: 5e-06 | > step_time: 0.2433 (0.37380930214731006) | > loader_time: 0.0091 (0.009836543108105483)  --> TIME: 2024-10-20 02:42:10 -- STEP: 2806/3361 -- GLOBAL_STEP: 16250 | > loss_text_ce: 0.041747577488422394 (0.043037881924693044) | > loss_mel_ce: 3.2888636589050293 (3.3371500277298285) | > loss: 0.8326528072357178 (0.8450469779951272) | > current_lr: 5e-06 | > step_time: 0.4812 (0.3739183638151933) | > loader_time: 0.0089 (0.009833243261977589)  --> TIME: 2024-10-20 02:42:42 -- STEP: 2856/3361 -- GLOBAL_STEP: 16300 | > loss_text_ce: 0.04618898406624794 (0.043028440912115835) | > loss_mel_ce: 3.1570663452148438 (3.3359200244858127) | > loss: 0.8008138537406921 (0.8447371168630801) | > current_lr: 5e-06 | > step_time: 0.2977 (0.37397973352119684) | > loader_time: 0.0098 (0.009833188534450792)  --> TIME: 2024-10-20 02:43:14 -- STEP: 2906/3361 -- GLOBAL_STEP: 16350 | > loss_text_ce: 0.04405193775892258 (0.043023765260188244) | > loss_mel_ce: 3.3160107135772705 (3.3349478833854214) | > loss: 0.8400156497955322 (0.8444929126411659) | > current_lr: 5e-06 | > step_time: 0.3443 (0.3739515754656386) | > loader_time: 0.0093 (0.009837625277756161)  --> TIME: 2024-10-20 02:43:46 -- STEP: 2956/3361 -- GLOBAL_STEP: 16400 | > loss_text_ce: 0.03851513937115669 (0.0430099739036516) | > loss_mel_ce: 3.1437385082244873 (3.3339321412478773) | > loss: 0.7955633997917175 (0.8442355293140363) | > current_lr: 5e-06 | > step_time: 0.2145 (0.3739763057118983) | > loader_time: 0.01 (0.009843850410032974)  --> TIME: 2024-10-20 02:44:18 -- STEP: 3006/3361 -- GLOBAL_STEP: 16450 | > loss_text_ce: 0.041888102889060974 (0.04299571098052135) | > loss_mel_ce: 3.3893206119537354 (3.333208246066103) | > loss: 0.857802152633667 (0.8440509898140682) | > current_lr: 5e-06 | > step_time: 0.3864 (0.3740643212895192) | > loader_time: 0.0106 (0.00984884141210072)  --> TIME: 2024-10-20 02:44:51 -- STEP: 3056/3361 -- GLOBAL_STEP: 16500 | > loss_text_ce: 0.042515724897384644 (0.042986133012735295) | > loss_mel_ce: 3.4355268478393555 (3.3327198423328177) | > loss: 0.8695106506347656 (0.8439264943852478) | > current_lr: 5e-06 | > step_time: 0.2655 (0.37418316187658845) | > loader_time: 0.0093 (0.00985704128342773)  --> TIME: 2024-10-20 02:45:23 -- STEP: 3106/3361 -- GLOBAL_STEP: 16550 | > loss_text_ce: 0.04620397463440895 (0.04297397403839279) | > loss_mel_ce: 3.0770668983459473 (3.331866742408588) | > loss: 0.7808177471160889 (0.8437101797219333) | > current_lr: 5e-06 | > step_time: 0.3309 (0.374201875460662) | > loader_time: 0.0104 (0.009857491148109212)  --> TIME: 2024-10-20 02:45:54 -- STEP: 3156/3361 -- GLOBAL_STEP: 16600 | > loss_text_ce: 0.03918001055717468 (0.04296290355702653) | > loss_mel_ce: 3.073801040649414 (3.331039714344854) | > loss: 0.7782452702522278 (0.8435006549603136) | > current_lr: 5e-06 | > step_time: 0.316 (0.3741178535236604) | > loader_time: 0.0103 (0.009862533389237732)  --> TIME: 2024-10-20 02:46:27 -- STEP: 3206/3361 -- GLOBAL_STEP: 16650 | > loss_text_ce: 0.044319815933704376 (0.04295722946189968) | > loss_mel_ce: 3.2032668590545654 (3.3305540042450015) | > loss: 0.8118966817855835 (0.8433778088537518) | > current_lr: 5e-06 | > step_time: 0.4352 (0.3742924399920276) | > loader_time: 0.0105 (0.00986770206291973)  --> TIME: 2024-10-20 02:47:00 -- STEP: 3256/3361 -- GLOBAL_STEP: 16700 | > loss_text_ce: 0.03914668783545494 (0.042953169736461974) | > loss_mel_ce: 3.193128824234009 (3.329916554235414) | > loss: 0.8080688714981079 (0.8432174314737908) | > current_lr: 5e-06 | > step_time: 0.2983 (0.3744111431611549) | > loader_time: 0.0106 (0.009869851409949584)  --> TIME: 2024-10-20 02:47:32 -- STEP: 3306/3361 -- GLOBAL_STEP: 16750 | > loss_text_ce: 0.04873709753155708 (0.04293744601376906) | > loss_mel_ce: 3.3574013710021973 (3.3297421446152313) | > loss: 0.8515346050262451 (0.84316989811925) | > current_lr: 5e-06 | > step_time: 0.4348 (0.3745028200974552) | > loader_time: 0.011 (0.009873052618392494)  --> TIME: 2024-10-20 02:48:04 -- STEP: 3356/3361 -- GLOBAL_STEP: 16800 | > loss_text_ce: 0.04252580925822258 (0.042922385484290955) | > loss_mel_ce: 3.7723355293273926 (3.328910979099294) | > loss: 0.9537153244018555 (0.8429583415158176) | > current_lr: 5e-06 | > step_time: 0.3307 (0.3744130439354778) | > loader_time: 0.0092 (0.009873790451113442)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.3775467693164783 (-0.0003484859261463913) | > avg_loss_text_ce: 0.039982772161883694 (-0.0014271793426364754) | > avg_loss_mel_ce: 3.101211165869108 (-0.06992721044889061) | > avg_loss: 3.141193941075315 (-0.0713543943179551) > BEST MODEL : /kaggle/temp/checkpoints/GPT_XTTS_FT-October-19-2024_11+36PM-1a53d9b/best_model_16805.pth