|
[21:39:59] - INFO - absl - A polynomial schedule was set with a non-positive `transition_steps` value |
|
/home/dat/pino/lib/python3.8/site-packages/jax/_src/numpy/lax_numpy.py:3132: UserWarning: Explicitly requested dtype <class 'jax._src.numpy.lax_numpy.int64'> requested in zeros is not available, and will be truncated to dtype int32. To enable more dtypes, set the jax_enable_x64 configuration option or the JAX_ENABLE_X64 shell environment variable. See https://github.com/google/jax#current-gotchas for more. |
|
lax._check_user_dtype_supported(dtype, "zeros") |
|
/home/dat/pino/lib/python3.8/site-packages/jax/lib/xla_bridge.py:386: UserWarning: jax.host_count has been renamed to jax.process_count. This alias will eventually be removed |
|
warnings.warn( |
|
/home/dat/pino/lib/python3.8/site-packages/jax/lib/xla_bridge.py:373: UserWarning: jax.host_id has been renamed to jax.process_index. This alias will eventually be removed |
|
warnings.warn( |
|
Epoch ... (1/5): 0%| | 0/5 [00:00<?, ?it/s][21:40:00] - INFO - __main__ - Skipping to epoch 0 step 0 |
|
|
|
|
|
[21:41:37] - INFO - huggingface_hub.repository - git version 2.25.1 | 5/1250 [01:27<2:38:49, 7.65s/it] |
|
git-lfs/2.9.2 (GitHub |
|
[21:41:38] - DEBUG - huggingface_hub.repository - [Repository] is a valid git repo |
|
[21:42:44] - INFO - huggingface_hub.repository - Uploading LFS objects: 100% (4/4), 2.0 GB | 35 MB/s, done. |
|
tcmalloc: large alloc 1354776576 bytes == 0x310a78000 @ 0x7fa277001680 0x7fa277021bdd 0x7f9fab23220d 0x7f9fab240340 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23bbd3 0x7f9fab23c1fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56aadf 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a 0x68cdc7 0x67e161 |
|
tcmalloc: large alloc 2715181056 bytes == 0x36167c000 @ 0x7fa277001680 0x7fa277021bdd 0x7f9fab23220d 0x7f9fab240340 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23fe87 0x7f9fab23bbd3 0x7f9fab23c1fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56aadf 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a 0x68cdc7 0x67e161 0x67e1df |
|
tcmalloc: large alloc 1530273792 bytes == 0x4033e2000 @ 0x7fa277001680 0x7fa277022824 0x5f7b11 0x7f9fab23bc6f 0x7f9fab23c1fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56aadf 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a 0x68cdc7 0x67e161 0x67e1df 0x67e281 0x67e627 0x6b6e62 0x6b71ed 0x7fa276e160b3 0x5f96de |
|
[21:43:04] - INFO - __main__ - checkpoint saved |
|
[21:43:05] - INFO - __main__ - SAVING CHECKPOINT IN ./... |
|
[21:43:09] - INFO - huggingface_hub.repository - git version 2.25.1 | 10/1250 [02:59<2:44:46, 7.97s/it] |
|
git-lfs/2.9.2 (GitHub |
|
[21:43:09] - DEBUG - huggingface_hub.repository - [Repository] is a valid git repo |
|
Training...: 1%|█ | 10/1250 [03:10<6:34:28, 19.09s/it] |
|
Epoch ... (1/5): 0%| | 0/5 [03:15<?, ?it/s] |
|
Traceback (most recent call last): |
|
File "./run_mlm_flax.py", line 861, in <module> |
|
save_model_checkpoint(model, training_args.output_dir, state, with_opt=model_args.save_optimizer, |
|
File "./run_mlm_flax.py", line 464, in save_model_checkpoint |
|
model.save_pretrained( |
|
File "/home/dat/transformers/src/transformers/modeling_flax_utils.py", line 484, in save_pretrained |
|
f.write(model_bytes) |
|
KeyboardInterrupt |
