| /home/dat/pino/lib/python3.8/site-packages/jax/lib/xla_bridge.py:382: UserWarning: jax.host_count has been renamed to jax.process_count. This alias will eventually be removed; please update your code. | |
| warnings.warn( | |
| /home/dat/pino/lib/python3.8/site-packages/jax/lib/xla_bridge.py:369: UserWarning: jax.host_id has been renamed to jax.process_index. This alias will eventually be removed; please update your code. | |
| warnings.warn( | |
| Epoch ... (1/5): 0%| | 0/5 [00:00<?, ?it/s] | |
| Training...: 1%| | 500/92767 [06:08<14:38:48, 1.75it/s] | |
| Training...: 1%|▏ | 1000/92767 [10:54<14:33:13, 1.75it/s] | |
| Training...: 2%|▎ | 1500/92767 [15:40<14:28:03, 1.75it/s] | |
| Training...: 2%|▍ | 2005/92767 [20:46<43:13:25, 1.71s/it] | |
| Training...: 3%|▌ | 2500/92767 [25:13<14:19:55, 1.75it/s] | |
| Training...: 3%|▋ | 3000/92767 [29:59<14:13:31, 1.75it/s] | |
| Training...: 4%|▊ | 3501/92767 [35:04<153:51:07, 6.20s/it] | |
| Training...: 4%|▉ | 4007/92767 [39:51<26:26:24, 1.07s/it] | |
| Training...: 5%|█ | 4513/92767 [44:38<5:17:18, 4.64it/s] | |
| Training...: 5%|█▏ | 5000/92767 [49:03<13:56:15, 1.75it/s] | |
| Training...: 6%|█▎ | 5500/92767 [53:49<13:50:21, 1.75it/s] | |
| Training...: 6%|█▍ | 6000/92767 [58:36<13:45:23, 1.75it/s] | |
| Training...: 7%|█▍ | 6500/92767 [1:03:22<13:40:53, 1.75it/s] | |
| Training...: 8%|█▌ | 7003/92767 [1:08:27<73:43:46, 3.09s/it] | |
| Training...: 8%|█▌ | 7500/92767 [1:12:54<13:31:06, 1.75it/s] | |
| Training...: 9%|█▋ | 8000/92767 [1:17:40<13:26:37, 1.75it/s] | |
| Training...: 9%|█▊ | 8503/92767 [1:22:46<72:29:48, 3.10s/it] | |
| Training...: 10%|█▉ | 9006/92767 [1:27:32<26:20:08, 1.13s/it] | |
| Training...: 10%|██▏ | 9512/92767 [1:32:19<7:27:35, 3.10it/s] | |
| Training...: 11%|██ | 10000/92767 [1:36:45<13:06:55, 1.75it/s] | |
| Training...: 11%|██▏ | 10500/92767 [1:41:31<13:02:26, 1.75it/s] | |
| Training...: 12%|██▎ | 11000/92767 [1:46:17<12:58:46, 1.75it/s] | |
| Training...: 12%|██▎ | 11500/92767 [1:51:03<12:53:11, 1.75it/s] | |
| Training...: 13%|██▍ | 12002/92767 [1:56:09<98:23:42, 4.39s/it] | |
| Training...: 13%|██▌ | 12508/92767 [2:00:56<14:34:49, 1.53it/s] | |
| Training...: 14%|██▊ | 13014/92767 [2:05:42<4:42:51, 4.70it/s] | |
| Training...: 15%|██▊ | 13500/92767 [2:10:08<12:31:38, 1.76it/s] | |
| Training...: 15%|██▊ | 14003/92767 [2:15:13<67:44:48, 3.10s/it] | |
| Training...: 16%|███▏ | 14511/92767 [2:20:00<5:56:33, 3.66it/s] | |
| Training...: 16%|███ | 15000/92767 [2:24:26<12:20:02, 1.75it/s] | |
| Training...: 17%|███ | 15501/92767 [2:29:32<133:12:40, 6.21s/it] | |
| Training...: 17%|███▎ | 16006/92767 [2:34:18<29:26:21, 1.38s/it] | |
| Training...: 18%|███▌ | 16513/92767 [2:39:05<4:37:44, 4.58it/s] | |
| Training...: 18%|███▋ | 17018/92767 [2:43:52<3:24:31, 6.17it/s] | |
| Training...: 19%|███▌ | 17503/92767 [2:48:36<64:45:38, 3.10s/it] | |
| Training...: 19%|███▉ | 18011/92767 [2:53:23<8:05:33, 2.57it/s] | |
| Training...: 20%|███▉ | 18515/92767 [2:58:10<3:45:16, 5.49it/s] | |
| Training...: 20%|███▉ | 19000/92767 [3:02:35<11:41:57, 1.75it/s] | |
| Training...: 21%|███▉ | 19500/92767 [3:07:21<11:38:06, 1.75it/s] | |
| Training...: 22%|████ | 20000/92767 [3:12:07<11:32:26, 1.75it/s] | |
| git-lfs/2.9.2 (GitHub; linux amd64; go 1.13.5)2767 [3:12:27<11:32:26, 1.75it/s] | |
| [04:20:50] - DEBUG - huggingface_hub.repository - [Repository] is a valid git repo | |
| [04:21:05] - INFO - huggingface_hub.repository - Uploading LFS objects: 100% (2/2), 260 MB | 30 MB/s, done. | |
| [04:21:06] - INFO - absl - Saving checkpoint at step: 20000 | |
| tcmalloc: large alloc 1363968000 bytes == 0x2ed6e2000 @ 0x7f170bb8c680 0x7f170bbacbdd 0x7f143fe0e20d 0x7f143fe1c340 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe17bd3 0x7f143fe181fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56fb87 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a 0x68cdc7 | |
| [04:21:11] - INFO - absl - Saved checkpoint at checkpoint_20000 | |
| Training...: 22%|████▏ | 20502/92767 [3:17:36<87:57:44, 4.38s/it] | |
| Training...: 23%|████▎ | 21000/92767 [3:22:02<11:22:51, 1.75it/s] | |
| Training...: 23%|████▍ | 21500/92767 [3:26:49<11:18:04, 1.75it/s] | |
| Training...: 24%|████▌ | 22000/92767 [3:31:35<11:14:17, 1.75it/s] | |
| Training...: 24%|████▌ | 22508/92767 [3:36:41<13:33:34, 1.44it/s] | |
| Training...: 25%|████▉ | 23011/92767 [3:41:27<8:59:08, 2.16it/s] | |
| Training...: 25%|████▊ | 23500/92767 [3:45:53<10:59:56, 1.75it/s] | |
| Training...: 26%|████▉ | 24008/92767 [3:50:59<11:35:36, 1.65it/s] | |
| Training...: 26%|█████ | 24500/92767 [3:55:25<10:49:26, 1.75it/s] | |
| Training...: 27%|█████ | 25000/92767 [4:00:12<10:45:34, 1.75it/s] | |
| Training...: 27%|█████▏ | 25502/92767 [4:05:17<81:55:08, 4.38s/it] | |
| Training...: 28%|█████▎ | 26000/92767 [4:09:44<10:35:12, 1.75it/s] | |
| Training...: 29%|█████▍ | 26500/92767 [4:14:30<10:31:21, 1.75it/s] | |
| Training...: 29%|█████▌ | 27002/92767 [4:19:36<80:03:02, 4.38s/it] | |
| Training...: 30%|█████▋ | 27506/92767 [4:24:22<23:51:18, 1.32s/it] | |
| Training...: 30%|█████▋ | 28000/92767 [4:28:48<10:17:08, 1.75it/s] | |
| Training...: 31%|█████▊ | 28502/92767 [4:33:54<78:09:02, 4.38s/it] | |
| Training...: 31%|█████▉ | 29006/92767 [4:38:41<19:59:49, 1.13s/it] | |
| Training...: 32%|██████ | 29500/92767 [4:43:07<10:03:04, 1.75it/s] | |
| Training...: 32%|█████▊ | 30001/92767 [4:48:12<108:10:35, 6.20s/it] | |
| Training...: 33%|██████▌ | 30500/92767 [4:52:39<9:51:44, 1.75it/s] | |
| Training...: 33%|██████▋ | 31000/92767 [4:57:25<9:48:31, 1.75it/s] | |
| Training...: 34%|██████▊ | 31500/92767 [5:02:11<9:42:31, 1.75it/s] | |
| Training...: 34%|██████▉ | 32000/92767 [5:06:58<9:38:05, 1.75it/s] | |
| Training...: 35%|███████ | 32500/92767 [5:11:44<9:34:14, 1.75it/s] | |
| Training...: 36%|███████ | 33000/92767 [5:16:30<9:28:40, 1.75it/s] | |
| Training...: 36%|███████▏ | 33500/92767 [5:21:16<9:23:42, 1.75it/s] | |
| Training...: 37%|██████▉ | 34003/92767 [5:26:22<50:31:48, 3.10s/it] | |
| Training...: 37%|███████ | 34507/92767 [5:31:08<16:42:23, 1.03s/it] | |
| Training...: 38%|███████▌ | 35000/92767 [5:35:34<9:09:39, 1.75it/s] | |
| Training...: 38%|███████▎ | 35503/92767 [5:40:40<49:14:50, 3.10s/it] | |
| Training...: 39%|███████▊ | 36008/92767 [5:45:27<9:33:07, 1.65it/s] | |
| Training...: 39%|███████▊ | 36500/92767 [5:49:53<8:55:17, 1.75it/s] | |
| Training...: 40%|███████▉ | 37000/92767 [5:54:39<8:51:16, 1.75it/s] | |
| Training...: 40%|████████ | 37500/92767 [5:59:25<8:45:54, 1.75it/s] | |
| Training...: 41%|████████▏ | 38000/92767 [6:04:11<8:40:19, 1.75it/s] | |
| Training...: 42%|████████▎ | 38500/92767 [6:08:57<8:37:09, 1.75it/s] | |
| Training...: 42%|███████▉ | 39005/92767 [6:14:03<25:34:35, 1.71s/it] | |
| Training...: 43%|████████▌ | 39512/92767 [6:18:50<3:58:29, 3.72it/s] | |
| Training...: 43%|████████▌ | 40000/92767 [6:23:16<8:22:42, 1.75it/s] | |
| git-lfs/2.9.2 (GitHub; linux amd64; go 1.13.5)92767 [6:23:35<8:22:42, 1.75it/s] | |
| [07:31:58] - DEBUG - huggingface_hub.repository - [Repository] is a valid git repo | |
| [07:32:37] - INFO - huggingface_hub.repository - Uploading LFS objects: 100% (3/3), 1.0 GB | 38 MB/s, done. | |
| [07:32:38] - INFO - absl - Saving checkpoint at step: 40000 | |
| tcmalloc: large alloc 1363968000 bytes == 0x2ed6e2000 @ 0x7f170bb8c680 0x7f170bbacbdd 0x7f143fe0e20d 0x7f143fe1c340 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe17bd3 0x7f143fe181fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56fb87 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a 0x68cdc7 | |
| [07:32:43] - INFO - absl - Saved checkpoint at checkpoint_40000 | |
| Training...: 44%|████████▋ | 40510/92767 [6:29:08<5:04:12, 2.86it/s] | |
| Training...: 44%|████████▊ | 41010/92767 [6:33:55<6:35:41, 2.18it/s] | |
| Training...: 45%|████████▌ | 41503/92767 [6:38:40<44:05:30, 3.10s/it] | |
| Training...: 45%|█████████ | 42000/92767 [6:43:07<8:03:02, 1.75it/s] | |
| Training...: 46%|█████████▏ | 42500/92767 [6:47:53<7:58:54, 1.75it/s] | |
| Training...: 46%|████████▊ | 43001/92767 [6:52:58<85:49:55, 6.21s/it] | |
| Training...: 47%|████████▉ | 43507/92767 [6:57:45<11:14:32, 1.22it/s] | |
| Training...: 47%|█████████▍ | 44009/92767 [7:02:31<6:09:57, 2.20it/s] | |
| Training...: 48%|█████████▌ | 44500/92767 [7:06:57<7:39:27, 1.75it/s] | |
| Training...: 49%|█████████▏ | 45005/92767 [7:12:03<20:50:46, 1.57s/it] | |
| Training...: 49%|█████████▊ | 45510/92767 [7:16:50<6:01:25, 2.18it/s] | |
| Training...: 50%|█████████▉ | 46000/92767 [7:21:16<7:24:18, 1.75it/s] | |
| Training...: 50%|██████████ | 46500/92767 [7:26:02<7:20:11, 1.75it/s] | |
| Training...: 51%|█████████▋ | 47004/92767 [7:31:08<27:58:05, 2.20s/it] | |
| Training...: 51%|██████████▏ | 47511/92767 [7:35:55<3:26:53, 3.65it/s] | |
| Training...: 52%|██████████▎ | 48020/92767 [7:40:42<1:21:42, 9.13it/s] | |
| Training...: 52%|██████████▍ | 48500/92767 [7:45:06<7:01:43, 1.75it/s] | |
| Training...: 53%|██████████▌ | 49000/92767 [7:49:53<6:55:29, 1.76it/s] | |
| Training...: 53%|██████████▋ | 49500/92767 [7:54:39<6:51:41, 1.75it/s] | |
| Training...: 54%|██████████▏ | 50001/92767 [7:59:44<73:42:07, 6.20s/it] | |
| Training...: 54%|██████████▎ | 50507/92767 [8:04:31<12:35:49, 1.07s/it] | |
| Training...: 55%|██████████▉ | 51000/92767 [8:08:57<6:37:26, 1.75it/s] | |
| Training...: 56%|██████████▌ | 51501/92767 [8:14:03<71:09:39, 6.21s/it] | |
| Training...: 56%|██████████▋ | 52007/92767 [8:18:49<11:18:40, 1.00it/s] | |
| Training...: 57%|███████████▎ | 52500/92767 [8:23:16<6:23:12, 1.75it/s] | |
| Training...: 57%|███████████▍ | 53000/92767 [8:28:02<6:18:56, 1.75it/s] | |
| Training...: 58%|███████████▌ | 53500/92767 [8:32:48<6:13:32, 1.75it/s] | |
| Training...: 58%|███████████▋ | 54000/92767 [8:37:34<6:08:49, 1.75it/s] | |
| Training...: 59%|███████████▋ | 54500/92767 [8:42:20<6:03:59, 1.75it/s] | |
| Training...: 59%|███████████▎ | 55003/92767 [8:47:26<32:27:24, 3.09s/it] | |
| Training...: 60%|███████████▉ | 55500/92767 [8:51:52<5:54:31, 1.75it/s] | |
| Training...: 60%|████████████ | 56000/92767 [8:56:39<5:49:57, 1.75it/s] | |
| Training...: 61%|███████████▌ | 56502/92767 [9:01:44<44:06:39, 4.38s/it] | |
| Training...: 61%|████████████▎ | 57010/92767 [9:06:31<3:27:55, 2.87it/s] | |
| Training...: 62%|████████████▍ | 57511/92767 [9:11:18<3:26:12, 2.85it/s] | |
| Training...: 63%|███████████▉ | 58001/92767 [9:16:03<59:58:15, 6.21s/it] | |
| Training...: 63%|████████████▌ | 58500/92767 [9:20:29<5:26:06, 1.75it/s] | |
| Training...: 64%|████████████▋ | 59000/92767 [9:25:15<5:21:19, 1.75it/s] | |
| Training...: 64%|████████████▊ | 59500/92767 [9:30:02<5:16:37, 1.75it/s] | |
| Training...: 65%|████████████▉ | 60000/92767 [9:35:07<5:11:39, 1.75it/s] | |
| git-lfs/2.9.2 (GitHub; linux amd64; go 1.13.5)92767 [9:35:07<5:11:39, 1.75it/s] | |
| [10:43:30] - DEBUG - huggingface_hub.repository - [Repository] is a valid git repo | |
| [10:44:08] - INFO - huggingface_hub.repository - Uploading LFS objects: 100% (3/3), 1.0 GB | 43 MB/s, done. | |
| [10:44:09] - INFO - absl - Saving checkpoint at step: 60000 | |
| tcmalloc: large alloc 1363968000 bytes == 0x2ed6e2000 @ 0x7f170bb8c680 0x7f170bbacbdd 0x7f143fe0e20d 0x7f143fe1c340 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe1be87 0x7f143fe17bd3 0x7f143fe181fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56fb87 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a 0x68cdc7 | |
| [10:44:13] - INFO - absl - Saved checkpoint at checkpoint_60000 | |