icefall-asr-gigaspeech-conformer-ctc / log /log-decode-2022-04-09-01-40-41
wgb14's picture
upload model, lexicon and log
fe00c53
2022-04-09 01:40:41,909 INFO [decode_test.py:583] Decoding started
2022-04-09 01:40:41,910 INFO [decode_test.py:584] {'subsampling_factor': 4, 'vgg_frontend': False, 'use_feat_batchnorm': True, 'feature_dim': 80, 'nhead': 8, 'attention_dim': 512, 'num_decoder_layers': 6, 'search_beam': 20, 'output_beam': 8, 'min_active_states': 30, 'max_active_states': 10000, 'use_double_scores': True, 'env_info': {'k2-version': '1.14', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '6833270cb228aba7bf9681fccd41e2b52f7d984c', 'k2-git-date': 'Wed Mar 16 11:16:05 2022', 'lhotse-version': '1.0.0.dev+git.d917411.clean', 'torch-cuda-available': True, 'torch-cuda-version': '11.1', 'python-version': '3.7', 'icefall-git-branch': 'gigaspeech_recipe', 'icefall-git-sha1': 'c3993a5-dirty', 'icefall-git-date': 'Mon Mar 21 13:49:39 2022', 'icefall-path': '/userhome/user/guanbo/icefall_decode', 'k2-path': '/opt/conda/lib/python3.7/site-packages/k2-1.14.dev20220408+cuda11.1.torch1.10.0-py3.7-linux-x86_64.egg/k2/__init__.py', 'lhotse-path': '/userhome/user/guanbo/lhotse/lhotse/__init__.py', 'hostname': 'c8861f400b70d011ec0a3ee069db84328338-chenx8564-0', 'IP address': '10.9.150.55'}, 'epoch': 18, 'avg': 6, 'method': 'attention-decoder', 'num_paths': 1000, 'nbest_scale': 0.5, 'exp_dir': PosixPath('conformer_ctc/exp_500_8_2'), 'lang_dir': PosixPath('data/lang_bpe_500'), 'lm_dir': PosixPath('data/lm'), 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 20, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 1, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'subset': 'XL', 'lazy_load': True, 'small_dev': False}
2022-04-09 01:40:42,371 INFO [lexicon.py:176] Loading pre-compiled data/lang_bpe_500/Linv.pt
2022-04-09 01:40:42,473 INFO [decode_test.py:594] device: cuda:0
2022-04-09 01:40:46,249 INFO [decode_test.py:656] Loading pre-compiled G_4_gram.pt
2022-04-09 01:40:47,406 INFO [decode_test.py:692] averaging ['conformer_ctc/exp_500_8_2/epoch-13.pt', 'conformer_ctc/exp_500_8_2/epoch-14.pt', 'conformer_ctc/exp_500_8_2/epoch-15.pt', 'conformer_ctc/exp_500_8_2/epoch-16.pt', 'conformer_ctc/exp_500_8_2/epoch-17.pt', 'conformer_ctc/exp_500_8_2/epoch-18.pt']
2022-04-09 01:40:53,065 INFO [decode_test.py:699] Number of model parameters: 109226120
2022-04-09 01:40:53,065 INFO [asr_datamodule.py:381] About to get test cuts
2022-04-09 01:40:56,361 INFO [decode_test.py:497] batch 0/?, cuts processed until now is 3
2022-04-09 01:41:24,462 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 5.93 GiB (GPU 0; 31.75 GiB total capacity; 27.23 GiB already allocated; 1.90 GiB free; 28.49 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 01:41:24,462 INFO [decode.py:743] num_arcs before pruning: 324363
2022-04-09 01:41:24,462 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:41:24,473 INFO [decode.py:757] num_arcs after pruning: 7174
2022-04-09 01:41:40,284 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 4.67 GiB (GPU 0; 31.75 GiB total capacity; 25.69 GiB already allocated; 2.92 GiB free; 27.47 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 01:41:40,285 INFO [decode.py:743] num_arcs before pruning: 368362
2022-04-09 01:41:40,285 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:41:40,305 INFO [decode.py:757] num_arcs after pruning: 8521
2022-04-09 01:42:38,727 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 2.18 GiB (GPU 0; 31.75 GiB total capacity; 26.05 GiB already allocated; 1.42 GiB free; 28.98 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 01:42:38,727 INFO [decode.py:743] num_arcs before pruning: 432616
2022-04-09 01:42:38,728 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:42:38,736 INFO [decode.py:757] num_arcs after pruning: 9233
2022-04-09 01:43:13,573 INFO [decode_test.py:497] batch 100/?, cuts processed until now is 297
2022-04-09 01:43:48,362 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 25.34 GiB already allocated; 2.20 GiB free; 28.20 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 01:43:48,363 INFO [decode.py:743] num_arcs before pruning: 319907
2022-04-09 01:43:48,363 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:43:48,372 INFO [decode.py:757] num_arcs after pruning: 6358
2022-04-09 01:43:59,713 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 2.74 GiB (GPU 0; 31.75 GiB total capacity; 27.51 GiB already allocated; 2.19 GiB free; 28.20 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 01:43:59,713 INFO [decode.py:743] num_arcs before pruning: 313596
2022-04-09 01:43:59,713 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:43:59,724 INFO [decode.py:757] num_arcs after pruning: 8252
2022-04-09 01:44:54,463 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 25.25 GiB already allocated; 3.07 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 01:44:54,463 INFO [decode.py:743] num_arcs before pruning: 353355
2022-04-09 01:44:54,463 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:44:54,485 INFO [decode.py:757] num_arcs after pruning: 7520
2022-04-09 01:45:20,716 INFO [decode_test.py:497] batch 200/?, cuts processed until now is 570
2022-04-09 01:47:19,457 INFO [decode_test.py:497] batch 300/?, cuts processed until now is 806
2022-04-09 01:47:38,292 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 2.28 GiB (GPU 0; 31.75 GiB total capacity; 26.28 GiB already allocated; 1.48 GiB free; 28.92 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 01:47:38,293 INFO [decode.py:743] num_arcs before pruning: 596002
2022-04-09 01:47:38,293 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:47:38,312 INFO [decode.py:757] num_arcs after pruning: 10745
2022-04-09 01:49:18,493 INFO [decode.py:736] Caught exception:
Some bad things happened. Please read the above error messages and stack
trace. If you are using Python, the following command may be helpful:
gdb --args python /path/to/your/code.py
(You can use `gdb` to debug the code. Please consider compiling
a debug version of k2.).
If you are unable to fix it, please open an issue at:
https://github.com/k2-fsa/k2/issues/new
2022-04-09 01:49:18,494 INFO [decode.py:743] num_arcs before pruning: 398202
2022-04-09 01:49:18,494 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:49:18,541 INFO [decode.py:757] num_arcs after pruning: 14003
2022-04-09 01:49:21,800 INFO [decode_test.py:497] batch 400/?, cuts processed until now is 1082
2022-04-09 01:50:58,700 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 4.85 GiB (GPU 0; 31.75 GiB total capacity; 25.89 GiB already allocated; 1.48 GiB free; 28.92 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 01:50:58,701 INFO [decode.py:743] num_arcs before pruning: 398349
2022-04-09 01:50:58,701 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:50:58,709 INFO [decode.py:757] num_arcs after pruning: 10321
2022-04-09 01:51:31,627 INFO [decode_test.py:497] batch 500/?, cuts processed until now is 1334
2022-04-09 01:52:05,232 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.62 GiB already allocated; 1.47 GiB free; 28.93 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 01:52:05,232 INFO [decode.py:743] num_arcs before pruning: 212665
2022-04-09 01:52:05,232 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:52:05,241 INFO [decode.py:757] num_arcs after pruning: 6301
2022-04-09 01:53:29,890 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 1.91 GiB (GPU 0; 31.75 GiB total capacity; 25.66 GiB already allocated; 1.48 GiB free; 28.92 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 01:53:29,891 INFO [decode.py:743] num_arcs before pruning: 883555
2022-04-09 01:53:29,891 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:53:29,905 INFO [decode.py:757] num_arcs after pruning: 14819
2022-04-09 01:53:38,676 INFO [decode_test.py:497] batch 600/?, cuts processed until now is 1651
2022-04-09 01:54:57,438 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 25.34 GiB already allocated; 1.48 GiB free; 28.92 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 01:54:57,438 INFO [decode.py:743] num_arcs before pruning: 515795
2022-04-09 01:54:57,438 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:54:57,447 INFO [decode.py:757] num_arcs after pruning: 10132
2022-04-09 01:55:28,356 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.46 GiB already allocated; 1.48 GiB free; 28.92 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 01:55:28,356 INFO [decode.py:743] num_arcs before pruning: 670748
2022-04-09 01:55:28,356 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:55:28,365 INFO [decode.py:757] num_arcs after pruning: 10497
2022-04-09 01:55:42,238 INFO [decode_test.py:497] batch 700/?, cuts processed until now is 1956
2022-04-09 01:57:57,456 INFO [decode_test.py:497] batch 800/?, cuts processed until now is 2238
2022-04-09 01:58:04,281 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.45 GiB already allocated; 3.07 GiB free; 27.33 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 01:58:04,282 INFO [decode.py:743] num_arcs before pruning: 175423
2022-04-09 01:58:04,282 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:58:04,296 INFO [decode.py:757] num_arcs after pruning: 7926
2022-04-09 01:59:07,916 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 4.68 GiB (GPU 0; 31.75 GiB total capacity; 24.40 GiB already allocated; 3.06 GiB free; 27.33 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 01:59:07,917 INFO [decode.py:743] num_arcs before pruning: 259758
2022-04-09 01:59:07,917 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 01:59:07,928 INFO [decode.py:757] num_arcs after pruning: 6026
2022-04-09 02:00:00,623 INFO [decode_test.py:497] batch 900/?, cuts processed until now is 2536
2022-04-09 02:01:22,959 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.44 GiB already allocated; 3.08 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:01:22,959 INFO [decode.py:743] num_arcs before pruning: 749228
2022-04-09 02:01:22,959 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:01:22,968 INFO [decode.py:757] num_arcs after pruning: 23868
2022-04-09 02:01:59,449 INFO [decode_test.py:497] batch 1000/?, cuts processed until now is 2824
2022-04-09 02:03:05,494 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.38 GiB already allocated; 3.06 GiB free; 27.33 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:03:05,494 INFO [decode.py:743] num_arcs before pruning: 255135
2022-04-09 02:03:05,494 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:03:05,504 INFO [decode.py:757] num_arcs after pruning: 5955
2022-04-09 02:03:48,017 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.61 GiB already allocated; 3.08 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:03:48,017 INFO [decode.py:743] num_arcs before pruning: 517077
2022-04-09 02:03:48,017 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:03:48,026 INFO [decode.py:757] num_arcs after pruning: 7695
2022-04-09 02:04:09,806 INFO [decode_test.py:497] batch 1100/?, cuts processed until now is 3105
2022-04-09 02:04:31,410 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.34 GiB already allocated; 3.08 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:04:31,411 INFO [decode.py:743] num_arcs before pruning: 859561
2022-04-09 02:04:31,411 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:04:31,422 INFO [decode.py:757] num_arcs after pruning: 13014
2022-04-09 02:06:11,496 INFO [decode_test.py:497] batch 1200/?, cuts processed until now is 3401
2022-04-09 02:08:10,727 INFO [decode_test.py:497] batch 1300/?, cuts processed until now is 3730
2022-04-09 02:10:17,677 INFO [decode_test.py:497] batch 1400/?, cuts processed until now is 4067
2022-04-09 02:12:13,175 INFO [decode_test.py:497] batch 1500/?, cuts processed until now is 4329
2022-04-09 02:13:02,842 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.55 GiB already allocated; 3.08 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:13:02,843 INFO [decode.py:743] num_arcs before pruning: 475511
2022-04-09 02:13:02,843 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:13:02,849 INFO [decode.py:757] num_arcs after pruning: 8439
2022-04-09 02:13:46,588 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 2.37 GiB (GPU 0; 31.75 GiB total capacity; 26.83 GiB already allocated; 1.45 GiB free; 28.94 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:13:46,588 INFO [decode.py:743] num_arcs before pruning: 595488
2022-04-09 02:13:46,588 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:13:46,598 INFO [decode.py:757] num_arcs after pruning: 13475
2022-04-09 02:14:21,206 INFO [decode_test.py:497] batch 1600/?, cuts processed until now is 4598
2022-04-09 02:16:42,740 INFO [decode_test.py:497] batch 1700/?, cuts processed until now is 4969
2022-04-09 02:17:13,672 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 25.39 GiB already allocated; 1.45 GiB free; 28.94 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:17:13,673 INFO [decode.py:743] num_arcs before pruning: 615734
2022-04-09 02:17:13,673 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:17:13,685 INFO [decode.py:757] num_arcs after pruning: 8684
2022-04-09 02:18:54,514 INFO [decode_test.py:497] batch 1800/?, cuts processed until now is 5260
2022-04-09 02:18:59,938 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.36 GiB already allocated; 3.08 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:18:59,938 INFO [decode.py:743] num_arcs before pruning: 360099
2022-04-09 02:18:59,938 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:18:59,949 INFO [decode.py:757] num_arcs after pruning: 6898
2022-04-09 02:19:48,186 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 6.00 GiB (GPU 0; 31.75 GiB total capacity; 27.15 GiB already allocated; 967.75 MiB free; 29.45 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:19:48,186 INFO [decode.py:743] num_arcs before pruning: 168720
2022-04-09 02:19:48,186 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:19:48,201 INFO [decode.py:757] num_arcs after pruning: 5346
2022-04-09 02:20:52,049 INFO [decode_test.py:497] batch 1900/?, cuts processed until now is 5585
2022-04-09 02:22:12,107 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.45 GiB already allocated; 973.75 MiB free; 29.44 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:22:12,107 INFO [decode.py:743] num_arcs before pruning: 1151735
2022-04-09 02:22:12,107 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:22:12,120 INFO [decode.py:757] num_arcs after pruning: 8335
2022-04-09 02:23:01,497 INFO [decode_test.py:497] batch 2000/?, cuts processed until now is 5902
2022-04-09 02:25:26,356 INFO [decode_test.py:497] batch 2100/?, cuts processed until now is 6219
2022-04-09 02:25:56,466 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.34 GiB already allocated; 973.75 MiB free; 29.44 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:25:56,467 INFO [decode.py:743] num_arcs before pruning: 612804
2022-04-09 02:25:56,467 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:25:56,477 INFO [decode.py:757] num_arcs after pruning: 10853
2022-04-09 02:27:26,441 INFO [decode_test.py:497] batch 2200/?, cuts processed until now is 6480
2022-04-09 02:29:28,073 INFO [decode_test.py:497] batch 2300/?, cuts processed until now is 6768
2022-04-09 02:31:41,553 INFO [decode_test.py:497] batch 2400/?, cuts processed until now is 7120
2022-04-09 02:31:55,632 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.42 GiB already allocated; 3.07 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:31:55,632 INFO [decode.py:743] num_arcs before pruning: 411490
2022-04-09 02:31:55,632 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:31:55,638 INFO [decode.py:757] num_arcs after pruning: 8626
2022-04-09 02:33:22,034 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.42 GiB already allocated; 3.07 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:33:22,034 INFO [decode.py:743] num_arcs before pruning: 625728
2022-04-09 02:33:22,035 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:33:22,043 INFO [decode.py:757] num_arcs after pruning: 9502
2022-04-09 02:33:37,663 INFO [decode_test.py:497] batch 2500/?, cuts processed until now is 7387
2022-04-09 02:34:18,300 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.51 GiB already allocated; 3.07 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:34:18,301 INFO [decode.py:743] num_arcs before pruning: 1015956
2022-04-09 02:34:18,301 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:34:18,314 INFO [decode.py:757] num_arcs after pruning: 14404
2022-04-09 02:34:20,220 INFO [decode.py:841] Caught exception:
CUDA out of memory. Tried to allocate 5.58 GiB (GPU 0; 31.75 GiB total capacity; 24.87 GiB already allocated; 3.07 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:34:20,221 INFO [decode.py:843] num_paths before decreasing: 1000
2022-04-09 02:34:20,221 INFO [decode.py:852] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:34:20,221 INFO [decode.py:858] num_paths after decreasing: 500
2022-04-09 02:34:40,089 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.38 GiB already allocated; 3.07 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:34:40,089 INFO [decode.py:743] num_arcs before pruning: 570686
2022-04-09 02:34:40,089 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:34:40,098 INFO [decode.py:757] num_arcs after pruning: 9182
2022-04-09 02:35:50,624 INFO [decode_test.py:497] batch 2600/?, cuts processed until now is 7764
2022-04-09 02:36:44,519 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.61 GiB already allocated; 3.08 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:36:44,519 INFO [decode.py:743] num_arcs before pruning: 1066267
2022-04-09 02:36:44,519 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:36:44,530 INFO [decode.py:757] num_arcs after pruning: 6963
2022-04-09 02:38:18,717 INFO [decode_test.py:497] batch 2700/?, cuts processed until now is 8078
2022-04-09 02:40:07,021 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.42 GiB already allocated; 3.07 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:40:07,022 INFO [decode.py:743] num_arcs before pruning: 1023667
2022-04-09 02:40:07,022 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:40:07,034 INFO [decode.py:757] num_arcs after pruning: 13090
2022-04-09 02:40:25,184 INFO [decode_test.py:497] batch 2800/?, cuts processed until now is 8444
2022-04-09 02:41:27,080 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.32 GiB already allocated; 3.08 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:41:27,080 INFO [decode.py:743] num_arcs before pruning: 739744
2022-04-09 02:41:27,080 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:41:27,093 INFO [decode.py:757] num_arcs after pruning: 9791
2022-04-09 02:42:44,319 INFO [decode_test.py:497] batch 2900/?, cuts processed until now is 8765
2022-04-09 02:42:44,656 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.73 GiB already allocated; 3.08 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:42:44,656 INFO [decode.py:743] num_arcs before pruning: 666168
2022-04-09 02:42:44,656 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:42:44,665 INFO [decode.py:757] num_arcs after pruning: 17223
2022-04-09 02:43:05,748 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 5.60 GiB (GPU 0; 31.75 GiB total capacity; 26.18 GiB already allocated; 1.14 GiB free; 29.26 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:43:05,748 INFO [decode.py:743] num_arcs before pruning: 188729
2022-04-09 02:43:05,748 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:43:05,762 INFO [decode.py:757] num_arcs after pruning: 8688
2022-04-09 02:44:54,469 INFO [decode_test.py:497] batch 3000/?, cuts processed until now is 9050
2022-04-09 02:46:55,167 INFO [decode_test.py:497] batch 3100/?, cuts processed until now is 9296
2022-04-09 02:47:28,418 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 20.00 GiB already allocated; 3.07 GiB free; 27.33 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:47:28,419 INFO [decode.py:743] num_arcs before pruning: 160153
2022-04-09 02:47:28,419 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:47:28,448 INFO [decode.py:757] num_arcs after pruning: 7778
2022-04-09 02:49:21,448 INFO [decode_test.py:497] batch 3200/?, cuts processed until now is 9652
2022-04-09 02:50:17,558 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 6.13 GiB (GPU 0; 31.75 GiB total capacity; 27.60 GiB already allocated; 895.75 MiB free; 29.52 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:50:17,558 INFO [decode.py:743] num_arcs before pruning: 388116
2022-04-09 02:50:17,559 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:50:17,565 INFO [decode.py:757] num_arcs after pruning: 10555
2022-04-09 02:51:30,675 INFO [decode_test.py:497] batch 3300/?, cuts processed until now is 10071
2022-04-09 02:53:49,565 INFO [decode_test.py:497] batch 3400/?, cuts processed until now is 10342
2022-04-09 02:55:49,392 INFO [decode_test.py:497] batch 3500/?, cuts processed until now is 10642
2022-04-09 02:58:07,518 INFO [decode_test.py:497] batch 3600/?, cuts processed until now is 10951
2022-04-09 02:58:16,360 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.29 GiB already allocated; 3.07 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 02:58:16,361 INFO [decode.py:743] num_arcs before pruning: 396714
2022-04-09 02:58:16,361 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 02:58:16,374 INFO [decode.py:757] num_arcs after pruning: 9543
2022-04-09 03:00:00,485 INFO [decode_test.py:497] batch 3700/?, cuts processed until now is 11231
2022-04-09 03:00:17,600 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.45 GiB already allocated; 3.07 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:00:17,601 INFO [decode.py:743] num_arcs before pruning: 854366
2022-04-09 03:00:17,601 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:00:17,612 INFO [decode.py:757] num_arcs after pruning: 10487
2022-04-09 03:00:20,098 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.68 GiB already allocated; 3.07 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:00:20,098 INFO [decode.py:743] num_arcs before pruning: 442824
2022-04-09 03:00:20,098 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:00:20,108 INFO [decode.py:757] num_arcs after pruning: 5265
2022-04-09 03:02:00,114 INFO [decode_test.py:497] batch 3800/?, cuts processed until now is 11509
2022-04-09 03:02:11,570 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.19 GiB already allocated; 3.07 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:02:11,571 INFO [decode.py:743] num_arcs before pruning: 285638
2022-04-09 03:02:11,571 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:02:11,579 INFO [decode.py:757] num_arcs after pruning: 5903
2022-04-09 03:04:02,757 INFO [decode_test.py:497] batch 3900/?, cuts processed until now is 11774
2022-04-09 03:05:19,989 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.73 GiB already allocated; 3.08 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:05:19,990 INFO [decode.py:743] num_arcs before pruning: 637327
2022-04-09 03:05:19,990 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:05:19,999 INFO [decode.py:757] num_arcs after pruning: 6357
2022-04-09 03:06:01,953 INFO [decode_test.py:497] batch 4000/?, cuts processed until now is 12045
2022-04-09 03:07:49,854 INFO [decode_test.py:497] batch 4100/?, cuts processed until now is 12300
2022-04-09 03:09:15,137 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.45 GiB already allocated; 3.08 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:09:15,138 INFO [decode.py:743] num_arcs before pruning: 507733
2022-04-09 03:09:15,138 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:09:15,148 INFO [decode.py:757] num_arcs after pruning: 4196
2022-04-09 03:09:47,397 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 5.86 GiB (GPU 0; 31.75 GiB total capacity; 27.78 GiB already allocated; 925.75 MiB free; 29.49 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:09:47,397 INFO [decode.py:743] num_arcs before pruning: 514118
2022-04-09 03:09:47,397 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:09:47,407 INFO [decode.py:757] num_arcs after pruning: 7168
2022-04-09 03:10:00,013 INFO [decode_test.py:497] batch 4200/?, cuts processed until now is 12580
2022-04-09 03:10:33,411 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 2.80 GiB (GPU 0; 31.75 GiB total capacity; 27.70 GiB already allocated; 925.75 MiB free; 29.49 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:10:33,411 INFO [decode.py:743] num_arcs before pruning: 374935
2022-04-09 03:10:33,411 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:10:33,418 INFO [decode.py:757] num_arcs after pruning: 10023
2022-04-09 03:12:04,333 INFO [decode_test.py:497] batch 4300/?, cuts processed until now is 12807
2022-04-09 03:14:06,889 INFO [decode_test.py:497] batch 4400/?, cuts processed until now is 13050
2022-04-09 03:14:34,787 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.47 GiB already allocated; 925.75 MiB free; 29.49 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:14:34,788 INFO [decode.py:743] num_arcs before pruning: 767465
2022-04-09 03:14:34,788 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:14:34,797 INFO [decode.py:757] num_arcs after pruning: 19151
2022-04-09 03:15:08,864 INFO [decode.py:736] Caught exception:
Some bad things happened. Please read the above error messages and stack
trace. If you are using Python, the following command may be helpful:
gdb --args python /path/to/your/code.py
(You can use `gdb` to debug the code. Please consider compiling
a debug version of k2.).
If you are unable to fix it, please open an issue at:
https://github.com/k2-fsa/k2/issues/new
2022-04-09 03:15:08,864 INFO [decode.py:743] num_arcs before pruning: 123833
2022-04-09 03:15:08,864 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:15:08,913 INFO [decode.py:757] num_arcs after pruning: 4150
2022-04-09 03:15:34,899 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 25.64 GiB already allocated; 3.07 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:15:34,899 INFO [decode.py:743] num_arcs before pruning: 444800
2022-04-09 03:15:34,899 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:15:34,908 INFO [decode.py:757] num_arcs after pruning: 11839
2022-04-09 03:16:08,462 INFO [decode_test.py:497] batch 4500/?, cuts processed until now is 13295
2022-04-09 03:17:56,946 INFO [decode_test.py:497] batch 4600/?, cuts processed until now is 13593
2022-04-09 03:18:16,099 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 5.53 GiB (GPU 0; 31.75 GiB total capacity; 26.53 GiB already allocated; 1.12 GiB free; 29.28 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:18:16,099 INFO [decode.py:743] num_arcs before pruning: 350609
2022-04-09 03:18:16,100 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:18:16,105 INFO [decode.py:757] num_arcs after pruning: 9262
2022-04-09 03:19:57,230 INFO [decode_test.py:497] batch 4700/?, cuts processed until now is 13858
2022-04-09 03:20:19,775 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 4.87 GiB (GPU 0; 31.75 GiB total capacity; 25.78 GiB already allocated; 1.12 GiB free; 29.28 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:20:19,775 INFO [decode.py:743] num_arcs before pruning: 375071
2022-04-09 03:20:19,775 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:20:19,785 INFO [decode.py:757] num_arcs after pruning: 6365
2022-04-09 03:21:29,481 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.42 GiB already allocated; 1.12 GiB free; 29.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:21:29,481 INFO [decode.py:743] num_arcs before pruning: 872088
2022-04-09 03:21:29,481 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:21:29,492 INFO [decode.py:757] num_arcs after pruning: 10043
2022-04-09 03:22:01,760 INFO [decode_test.py:497] batch 4800/?, cuts processed until now is 14079
2022-04-09 03:24:10,370 INFO [decode_test.py:497] batch 4900/?, cuts processed until now is 14298
2022-04-09 03:26:10,811 INFO [decode_test.py:497] batch 5000/?, cuts processed until now is 14515
2022-04-09 03:27:46,191 INFO [decode.py:736] Caught exception:
Some bad things happened. Please read the above error messages and stack
trace. If you are using Python, the following command may be helpful:
gdb --args python /path/to/your/code.py
(You can use `gdb` to debug the code. Please consider compiling
a debug version of k2.).
If you are unable to fix it, please open an issue at:
https://github.com/k2-fsa/k2/issues/new
2022-04-09 03:27:46,192 INFO [decode.py:743] num_arcs before pruning: 246382
2022-04-09 03:27:46,192 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:27:46,253 INFO [decode.py:757] num_arcs after pruning: 6775
2022-04-09 03:28:15,199 INFO [decode_test.py:497] batch 5100/?, cuts processed until now is 14718
2022-04-09 03:29:19,807 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 6.15 GiB (GPU 0; 31.75 GiB total capacity; 26.67 GiB already allocated; 1.11 GiB free; 29.29 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:29:19,808 INFO [decode.py:743] num_arcs before pruning: 220820
2022-04-09 03:29:19,808 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:29:19,815 INFO [decode.py:757] num_arcs after pruning: 13482
2022-04-09 03:30:16,045 INFO [decode_test.py:497] batch 5200/?, cuts processed until now is 14930
2022-04-09 03:32:12,235 INFO [decode_test.py:497] batch 5300/?, cuts processed until now is 15128
2022-04-09 03:33:06,358 INFO [decode.py:736] Caught exception:
Some bad things happened. Please read the above error messages and stack
trace. If you are using Python, the following command may be helpful:
gdb --args python /path/to/your/code.py
(You can use `gdb` to debug the code. Please consider compiling
a debug version of k2.).
If you are unable to fix it, please open an issue at:
https://github.com/k2-fsa/k2/issues/new
2022-04-09 03:33:06,359 INFO [decode.py:743] num_arcs before pruning: 190203
2022-04-09 03:33:06,359 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:33:06,413 INFO [decode.py:757] num_arcs after pruning: 6202
2022-04-09 03:34:14,862 INFO [decode_test.py:497] batch 5400/?, cuts processed until now is 15327
2022-04-09 03:36:18,973 INFO [decode_test.py:497] batch 5500/?, cuts processed until now is 15531
2022-04-09 03:38:18,633 INFO [decode_test.py:497] batch 5600/?, cuts processed until now is 15724
2022-04-09 03:38:48,490 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.52 GiB already allocated; 3.07 GiB free; 27.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:38:48,491 INFO [decode.py:743] num_arcs before pruning: 554330
2022-04-09 03:38:48,491 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:38:48,500 INFO [decode.py:757] num_arcs after pruning: 10730
2022-04-09 03:39:51,281 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 4.83 GiB (GPU 0; 31.75 GiB total capacity; 25.96 GiB already allocated; 1.31 GiB free; 29.08 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:39:51,281 INFO [decode.py:743] num_arcs before pruning: 160031
2022-04-09 03:39:51,281 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:39:51,288 INFO [decode.py:757] num_arcs after pruning: 4270
2022-04-09 03:40:28,016 INFO [decode_test.py:497] batch 5700/?, cuts processed until now is 15908
2022-04-09 03:40:46,608 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 2.58 GiB (GPU 0; 31.75 GiB total capacity; 27.28 GiB already allocated; 1.32 GiB free; 29.07 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:40:46,608 INFO [decode.py:743] num_arcs before pruning: 406026
2022-04-09 03:40:46,608 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:40:46,616 INFO [decode.py:757] num_arcs after pruning: 11179
2022-04-09 03:42:16,464 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 2.29 GiB (GPU 0; 31.75 GiB total capacity; 26.71 GiB already allocated; 1.32 GiB free; 29.07 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:42:16,464 INFO [decode.py:743] num_arcs before pruning: 639824
2022-04-09 03:42:16,464 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:42:16,476 INFO [decode.py:757] num_arcs after pruning: 5520
2022-04-09 03:42:52,683 INFO [decode_test.py:497] batch 5800/?, cuts processed until now is 16094
2022-04-09 03:44:51,754 INFO [decode_test.py:497] batch 5900/?, cuts processed until now is 16289
2022-04-09 03:46:52,121 INFO [decode_test.py:497] batch 6000/?, cuts processed until now is 16488
2022-04-09 03:48:54,739 INFO [decode_test.py:497] batch 6100/?, cuts processed until now is 16661
2022-04-09 03:49:24,829 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 1.84 GiB (GPU 0; 31.75 GiB total capacity; 28.87 GiB already allocated; 409.75 MiB free; 29.99 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:49:24,830 INFO [decode.py:743] num_arcs before pruning: 443401
2022-04-09 03:49:24,830 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:49:24,837 INFO [decode.py:757] num_arcs after pruning: 5211
2022-04-09 03:50:27,492 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.35 GiB already allocated; 2.15 GiB free; 28.24 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:50:27,493 INFO [decode.py:743] num_arcs before pruning: 361598
2022-04-09 03:50:27,493 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:50:27,507 INFO [decode.py:757] num_arcs after pruning: 8660
2022-04-09 03:51:02,856 INFO [decode_test.py:497] batch 6200/?, cuts processed until now is 16828
2022-04-09 03:53:03,912 INFO [decode_test.py:497] batch 6300/?, cuts processed until now is 17002
2022-04-09 03:55:04,964 INFO [decode_test.py:497] batch 6400/?, cuts processed until now is 17181
2022-04-09 03:55:08,345 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 4.89 GiB (GPU 0; 31.75 GiB total capacity; 26.28 GiB already allocated; 2.16 GiB free; 28.24 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:55:08,345 INFO [decode.py:743] num_arcs before pruning: 867262
2022-04-09 03:55:08,345 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:55:08,356 INFO [decode.py:757] num_arcs after pruning: 6494
2022-04-09 03:56:03,884 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 1.90 GiB (GPU 0; 31.75 GiB total capacity; 28.97 GiB already allocated; 1.16 GiB free; 29.23 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:56:03,885 INFO [decode.py:743] num_arcs before pruning: 233755
2022-04-09 03:56:03,885 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:56:03,910 INFO [decode.py:757] num_arcs after pruning: 5823
2022-04-09 03:57:08,774 INFO [decode_test.py:497] batch 6500/?, cuts processed until now is 17347
2022-04-09 03:59:01,245 INFO [decode_test.py:497] batch 6600/?, cuts processed until now is 17502
2022-04-09 03:59:13,147 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 5.80 GiB (GPU 0; 31.75 GiB total capacity; 26.73 GiB already allocated; 1.17 GiB free; 29.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 03:59:13,147 INFO [decode.py:743] num_arcs before pruning: 174004
2022-04-09 03:59:13,147 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 03:59:13,155 INFO [decode.py:757] num_arcs after pruning: 6857
2022-04-09 04:00:59,687 INFO [decode_test.py:497] batch 6700/?, cuts processed until now is 17661
2022-04-09 04:03:01,660 INFO [decode_test.py:497] batch 6800/?, cuts processed until now is 17823
2022-04-09 04:04:55,219 INFO [decode_test.py:497] batch 6900/?, cuts processed until now is 17997
2022-04-09 04:07:05,841 INFO [decode_test.py:497] batch 7000/?, cuts processed until now is 18159
2022-04-09 04:09:04,994 INFO [decode_test.py:497] batch 7100/?, cuts processed until now is 18299
2022-04-09 04:11:07,439 INFO [decode_test.py:497] batch 7200/?, cuts processed until now is 18432
2022-04-09 04:13:18,126 INFO [decode_test.py:497] batch 7300/?, cuts processed until now is 18552
2022-04-09 04:15:23,102 INFO [decode_test.py:497] batch 7400/?, cuts processed until now is 18656
2022-04-09 04:17:49,550 INFO [decode_test.py:497] batch 7500/?, cuts processed until now is 18798
2022-04-09 04:19:16,128 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.34 GiB already allocated; 2.12 GiB free; 28.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 04:19:16,129 INFO [decode.py:743] num_arcs before pruning: 1155990
2022-04-09 04:19:16,129 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 04:19:16,143 INFO [decode.py:757] num_arcs after pruning: 9141
2022-04-09 04:20:19,961 INFO [decode_test.py:497] batch 7600/?, cuts processed until now is 18945
2022-04-09 04:22:44,642 INFO [decode_test.py:497] batch 7700/?, cuts processed until now is 19084
2022-04-09 04:23:18,184 INFO [decode.py:841] Caught exception:
CUDA out of memory. Tried to allocate 1.26 GiB (GPU 0; 31.75 GiB total capacity; 27.36 GiB already allocated; 881.75 MiB free; 29.53 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 04:23:18,184 INFO [decode.py:843] num_paths before decreasing: 1000
2022-04-09 04:23:18,184 INFO [decode.py:852] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 04:23:18,184 INFO [decode.py:858] num_paths after decreasing: 500
2022-04-09 04:24:52,959 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.53 GiB already allocated; 2.12 GiB free; 28.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 04:24:52,960 INFO [decode.py:743] num_arcs before pruning: 624026
2022-04-09 04:24:52,960 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 04:24:52,972 INFO [decode.py:757] num_arcs after pruning: 10008
2022-04-09 04:25:07,718 INFO [decode_test.py:497] batch 7800/?, cuts processed until now is 19232
2022-04-09 04:25:31,876 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.51 GiB already allocated; 2.12 GiB free; 28.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 04:25:31,876 INFO [decode.py:743] num_arcs before pruning: 688909
2022-04-09 04:25:31,877 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 04:25:31,887 INFO [decode.py:757] num_arcs after pruning: 8886
2022-04-09 04:25:57,970 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 5.04 GiB (GPU 0; 31.75 GiB total capacity; 25.95 GiB already allocated; 2.12 GiB free; 28.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 04:25:57,971 INFO [decode.py:743] num_arcs before pruning: 891176
2022-04-09 04:25:57,971 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 04:25:57,982 INFO [decode.py:757] num_arcs after pruning: 10106
2022-04-09 04:26:19,609 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 2.63 GiB (GPU 0; 31.75 GiB total capacity; 27.60 GiB already allocated; 327.75 MiB free; 30.07 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 04:26:19,609 INFO [decode.py:743] num_arcs before pruning: 415376
2022-04-09 04:26:19,609 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 04:26:19,620 INFO [decode.py:757] num_arcs after pruning: 7771
2022-04-09 04:27:33,059 INFO [decode_test.py:497] batch 7900/?, cuts processed until now is 19375
2022-04-09 04:29:43,649 INFO [decode_test.py:497] batch 8000/?, cuts processed until now is 19510
2022-04-09 04:30:20,590 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.65 GiB already allocated; 2.12 GiB free; 28.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 04:30:20,591 INFO [decode.py:743] num_arcs before pruning: 330767
2022-04-09 04:30:20,591 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 04:30:20,606 INFO [decode.py:757] num_arcs after pruning: 5820
2022-04-09 04:31:55,818 INFO [decode_test.py:497] batch 8100/?, cuts processed until now is 19643
2022-04-09 04:34:11,720 INFO [decode_test.py:497] batch 8200/?, cuts processed until now is 19776
2022-04-09 04:35:04,147 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 4.49 GiB (GPU 0; 31.75 GiB total capacity; 24.38 GiB already allocated; 2.12 GiB free; 28.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 04:35:04,147 INFO [decode.py:743] num_arcs before pruning: 533967
2022-04-09 04:35:04,147 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 04:35:04,157 INFO [decode.py:757] num_arcs after pruning: 3449
2022-04-09 04:36:15,595 INFO [decode.py:736] Caught exception:
CUDA out of memory. Tried to allocate 8.00 GiB (GPU 0; 31.75 GiB total capacity; 19.67 GiB already allocated; 2.12 GiB free; 28.27 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2022-04-09 04:36:15,595 INFO [decode.py:743] num_arcs before pruning: 397138
2022-04-09 04:36:15,596 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 04:36:15,605 INFO [decode.py:757] num_arcs after pruning: 6775
2022-04-09 04:36:31,844 INFO [decode_test.py:497] batch 8300/?, cuts processed until now is 19882
2022-04-09 04:37:04,130 INFO [decode.py:736] Caught exception:
Some bad things happened. Please read the above error messages and stack
trace. If you are using Python, the following command may be helpful:
gdb --args python /path/to/your/code.py
(You can use `gdb` to debug the code. Please consider compiling
a debug version of k2.).
If you are unable to fix it, please open an issue at:
https://github.com/k2-fsa/k2/issues/new
2022-04-09 04:37:04,130 INFO [decode.py:743] num_arcs before pruning: 456591
2022-04-09 04:37:04,130 INFO [decode.py:746] This OOM is not an error. You can ignore it. If your model does not converge well, or --max-duration is too large, or the input sound file is difficult to decode, you will meet this exception.
2022-04-09 04:37:04,180 INFO [decode.py:757] num_arcs after pruning: 5275
2022-04-09 04:57:33,432 INFO [decode_test.py:567]
For test, WER of different settings are:
ngram_lm_scale_0.3_attention_scale_0.7 10.58 best for test
ngram_lm_scale_0.5_attention_scale_1.3 10.58
ngram_lm_scale_0.3_attention_scale_0.5 10.59
ngram_lm_scale_0.3_attention_scale_0.6 10.59
ngram_lm_scale_0.3_attention_scale_0.9 10.59
ngram_lm_scale_0.3_attention_scale_1.0 10.59
ngram_lm_scale_0.3_attention_scale_1.1 10.59
ngram_lm_scale_0.3_attention_scale_1.2 10.59
ngram_lm_scale_0.3_attention_scale_1.3 10.59
ngram_lm_scale_0.5_attention_scale_1.0 10.59
ngram_lm_scale_0.5_attention_scale_1.1 10.59
ngram_lm_scale_0.5_attention_scale_1.2 10.59
ngram_lm_scale_0.5_attention_scale_1.5 10.59
ngram_lm_scale_0.5_attention_scale_1.7 10.59
ngram_lm_scale_0.5_attention_scale_1.9 10.59
ngram_lm_scale_0.5_attention_scale_2.0 10.59
ngram_lm_scale_0.5_attention_scale_2.1 10.59
ngram_lm_scale_0.5_attention_scale_2.2 10.59
ngram_lm_scale_0.5_attention_scale_2.3 10.59
ngram_lm_scale_0.6_attention_scale_1.9 10.59
ngram_lm_scale_0.6_attention_scale_2.0 10.59
ngram_lm_scale_0.6_attention_scale_2.1 10.59
ngram_lm_scale_0.6_attention_scale_2.2 10.59
ngram_lm_scale_0.6_attention_scale_2.3 10.59
ngram_lm_scale_0.6_attention_scale_2.5 10.59
ngram_lm_scale_0.3_attention_scale_1.5 10.6
ngram_lm_scale_0.3_attention_scale_1.7 10.6
ngram_lm_scale_0.3_attention_scale_1.9 10.6
ngram_lm_scale_0.3_attention_scale_2.0 10.6
ngram_lm_scale_0.3_attention_scale_2.1 10.6
ngram_lm_scale_0.3_attention_scale_2.2 10.6
ngram_lm_scale_0.3_attention_scale_2.3 10.6
ngram_lm_scale_0.3_attention_scale_2.5 10.6
ngram_lm_scale_0.5_attention_scale_0.9 10.6
ngram_lm_scale_0.5_attention_scale_2.5 10.6
ngram_lm_scale_0.5_attention_scale_3.0 10.6
ngram_lm_scale_0.6_attention_scale_1.3 10.6
ngram_lm_scale_0.6_attention_scale_1.5 10.6
ngram_lm_scale_0.6_attention_scale_1.7 10.6
ngram_lm_scale_0.6_attention_scale_3.0 10.6
ngram_lm_scale_0.3_attention_scale_0.3 10.61
ngram_lm_scale_0.3_attention_scale_3.0 10.61
ngram_lm_scale_0.5_attention_scale_4.0 10.61
ngram_lm_scale_0.5_attention_scale_5.0 10.61
ngram_lm_scale_0.6_attention_scale_1.2 10.61
ngram_lm_scale_0.6_attention_scale_4.0 10.61
ngram_lm_scale_0.6_attention_scale_5.0 10.61
ngram_lm_scale_0.7_attention_scale_1.7 10.61
ngram_lm_scale_0.7_attention_scale_1.9 10.61
ngram_lm_scale_0.7_attention_scale_2.0 10.61
ngram_lm_scale_0.7_attention_scale_2.1 10.61
ngram_lm_scale_0.7_attention_scale_2.2 10.61
ngram_lm_scale_0.7_attention_scale_2.3 10.61
ngram_lm_scale_0.7_attention_scale_2.5 10.61
ngram_lm_scale_0.7_attention_scale_3.0 10.61
ngram_lm_scale_0.7_attention_scale_4.0 10.61
ngram_lm_scale_0.7_attention_scale_5.0 10.61
ngram_lm_scale_0.1_attention_scale_1.1 10.62
ngram_lm_scale_0.3_attention_scale_4.0 10.62
ngram_lm_scale_0.3_attention_scale_5.0 10.62
ngram_lm_scale_0.5_attention_scale_0.7 10.62
ngram_lm_scale_0.6_attention_scale_1.0 10.62
ngram_lm_scale_0.6_attention_scale_1.1 10.62
ngram_lm_scale_0.7_attention_scale_1.5 10.62
ngram_lm_scale_0.9_attention_scale_3.0 10.62
ngram_lm_scale_0.9_attention_scale_4.0 10.62
ngram_lm_scale_0.9_attention_scale_5.0 10.62
ngram_lm_scale_1.0_attention_scale_4.0 10.62
ngram_lm_scale_1.1_attention_scale_5.0 10.62
ngram_lm_scale_0.05_attention_scale_1.1 10.63
ngram_lm_scale_0.05_attention_scale_1.2 10.63
ngram_lm_scale_0.08_attention_scale_0.9 10.63
ngram_lm_scale_0.08_attention_scale_1.0 10.63
ngram_lm_scale_0.08_attention_scale_1.1 10.63
ngram_lm_scale_0.08_attention_scale_1.2 10.63
ngram_lm_scale_0.08_attention_scale_1.3 10.63
ngram_lm_scale_0.08_attention_scale_1.9 10.63
ngram_lm_scale_0.08_attention_scale_2.0 10.63
ngram_lm_scale_0.08_attention_scale_2.1 10.63
ngram_lm_scale_0.08_attention_scale_2.2 10.63
ngram_lm_scale_0.08_attention_scale_2.3 10.63
ngram_lm_scale_0.08_attention_scale_3.0 10.63
ngram_lm_scale_0.1_attention_scale_0.5 10.63
ngram_lm_scale_0.1_attention_scale_0.6 10.63
ngram_lm_scale_0.1_attention_scale_0.7 10.63
ngram_lm_scale_0.1_attention_scale_0.9 10.63
ngram_lm_scale_0.1_attention_scale_1.0 10.63
ngram_lm_scale_0.1_attention_scale_1.2 10.63
ngram_lm_scale_0.1_attention_scale_1.3 10.63
ngram_lm_scale_0.1_attention_scale_1.7 10.63
ngram_lm_scale_0.1_attention_scale_1.9 10.63
ngram_lm_scale_0.1_attention_scale_2.0 10.63
ngram_lm_scale_0.1_attention_scale_2.1 10.63
ngram_lm_scale_0.1_attention_scale_2.2 10.63
ngram_lm_scale_0.1_attention_scale_2.3 10.63
ngram_lm_scale_0.1_attention_scale_2.5 10.63
ngram_lm_scale_0.1_attention_scale_3.0 10.63
ngram_lm_scale_0.1_attention_scale_5.0 10.63
ngram_lm_scale_0.5_attention_scale_0.6 10.63
ngram_lm_scale_0.6_attention_scale_0.9 10.63
ngram_lm_scale_0.9_attention_scale_2.3 10.63
ngram_lm_scale_0.9_attention_scale_2.5 10.63
ngram_lm_scale_1.0_attention_scale_5.0 10.63
ngram_lm_scale_1.2_attention_scale_5.0 10.63
ngram_lm_scale_0.01_attention_scale_0.9 10.64
ngram_lm_scale_0.01_attention_scale_1.0 10.64
ngram_lm_scale_0.01_attention_scale_1.1 10.64
ngram_lm_scale_0.01_attention_scale_1.2 10.64
ngram_lm_scale_0.01_attention_scale_4.0 10.64
ngram_lm_scale_0.01_attention_scale_5.0 10.64
ngram_lm_scale_0.05_attention_scale_0.5 10.64
ngram_lm_scale_0.05_attention_scale_0.6 10.64
ngram_lm_scale_0.05_attention_scale_0.7 10.64
ngram_lm_scale_0.05_attention_scale_0.9 10.64
ngram_lm_scale_0.05_attention_scale_1.0 10.64
ngram_lm_scale_0.05_attention_scale_1.3 10.64
ngram_lm_scale_0.05_attention_scale_1.5 10.64
ngram_lm_scale_0.05_attention_scale_1.7 10.64
ngram_lm_scale_0.05_attention_scale_1.9 10.64
ngram_lm_scale_0.05_attention_scale_2.0 10.64
ngram_lm_scale_0.05_attention_scale_2.1 10.64
ngram_lm_scale_0.05_attention_scale_2.2 10.64
ngram_lm_scale_0.05_attention_scale_2.3 10.64
ngram_lm_scale_0.05_attention_scale_2.5 10.64
ngram_lm_scale_0.05_attention_scale_3.0 10.64
ngram_lm_scale_0.05_attention_scale_4.0 10.64
ngram_lm_scale_0.05_attention_scale_5.0 10.64
ngram_lm_scale_0.08_attention_scale_0.5 10.64
ngram_lm_scale_0.08_attention_scale_0.6 10.64
ngram_lm_scale_0.08_attention_scale_0.7 10.64
ngram_lm_scale_0.08_attention_scale_1.5 10.64
ngram_lm_scale_0.08_attention_scale_1.7 10.64
ngram_lm_scale_0.08_attention_scale_2.5 10.64
ngram_lm_scale_0.08_attention_scale_4.0 10.64
ngram_lm_scale_0.08_attention_scale_5.0 10.64
ngram_lm_scale_0.1_attention_scale_0.3 10.64
ngram_lm_scale_0.1_attention_scale_1.5 10.64
ngram_lm_scale_0.1_attention_scale_4.0 10.64
ngram_lm_scale_0.7_attention_scale_1.3 10.64
ngram_lm_scale_0.9_attention_scale_2.2 10.64
ngram_lm_scale_1.0_attention_scale_3.0 10.64
ngram_lm_scale_1.1_attention_scale_4.0 10.64
ngram_lm_scale_1.3_attention_scale_5.0 10.64
ngram_lm_scale_0.01_attention_scale_0.6 10.65
ngram_lm_scale_0.01_attention_scale_0.7 10.65
ngram_lm_scale_0.01_attention_scale_1.3 10.65
ngram_lm_scale_0.01_attention_scale_1.5 10.65
ngram_lm_scale_0.01_attention_scale_1.7 10.65
ngram_lm_scale_0.01_attention_scale_1.9 10.65
ngram_lm_scale_0.01_attention_scale_2.0 10.65
ngram_lm_scale_0.01_attention_scale_2.1 10.65
ngram_lm_scale_0.01_attention_scale_2.2 10.65
ngram_lm_scale_0.01_attention_scale_2.3 10.65
ngram_lm_scale_0.01_attention_scale_2.5 10.65
ngram_lm_scale_0.01_attention_scale_3.0 10.65
ngram_lm_scale_0.08_attention_scale_0.3 10.65
ngram_lm_scale_0.5_attention_scale_0.5 10.65
ngram_lm_scale_0.6_attention_scale_0.7 10.65
ngram_lm_scale_0.7_attention_scale_1.1 10.65
ngram_lm_scale_0.7_attention_scale_1.2 10.65
ngram_lm_scale_0.9_attention_scale_2.1 10.65
ngram_lm_scale_1.2_attention_scale_4.0 10.65
ngram_lm_scale_0.05_attention_scale_0.3 10.66
ngram_lm_scale_0.7_attention_scale_1.0 10.66
ngram_lm_scale_0.9_attention_scale_1.9 10.66
ngram_lm_scale_0.9_attention_scale_2.0 10.66
ngram_lm_scale_1.0_attention_scale_2.5 10.66
ngram_lm_scale_1.1_attention_scale_3.0 10.66
ngram_lm_scale_0.01_attention_scale_0.5 10.67
ngram_lm_scale_0.1_attention_scale_0.08 10.67
ngram_lm_scale_0.1_attention_scale_0.1 10.67
ngram_lm_scale_0.6_attention_scale_0.6 10.67
ngram_lm_scale_0.9_attention_scale_1.7 10.67
ngram_lm_scale_1.0_attention_scale_2.2 10.67
ngram_lm_scale_1.0_attention_scale_2.3 10.67
ngram_lm_scale_1.3_attention_scale_4.0 10.67
ngram_lm_scale_1.5_attention_scale_5.0 10.67
ngram_lm_scale_0.01_attention_scale_0.3 10.68
ngram_lm_scale_0.08_attention_scale_0.08 10.68
ngram_lm_scale_0.08_attention_scale_0.1 10.68
ngram_lm_scale_0.3_attention_scale_0.08 10.68
ngram_lm_scale_0.3_attention_scale_0.1 10.68
ngram_lm_scale_0.7_attention_scale_0.9 10.68
ngram_lm_scale_1.0_attention_scale_2.0 10.68
ngram_lm_scale_1.0_attention_scale_2.1 10.68
ngram_lm_scale_1.1_attention_scale_2.5 10.68
ngram_lm_scale_1.2_attention_scale_3.0 10.68
ngram_lm_scale_0.1_attention_scale_0.05 10.69
ngram_lm_scale_0.5_attention_scale_0.3 10.69
ngram_lm_scale_0.9_attention_scale_1.5 10.69
ngram_lm_scale_1.0_attention_scale_1.9 10.69
ngram_lm_scale_1.1_attention_scale_2.3 10.69
ngram_lm_scale_0.05_attention_scale_0.1 10.7
ngram_lm_scale_0.08_attention_scale_0.05 10.7
ngram_lm_scale_0.3_attention_scale_0.05 10.7
ngram_lm_scale_0.6_attention_scale_0.5 10.7
ngram_lm_scale_1.1_attention_scale_2.2 10.7
ngram_lm_scale_1.5_attention_scale_4.0 10.7
ngram_lm_scale_1.7_attention_scale_5.0 10.7
ngram_lm_scale_0.05_attention_scale_0.08 10.71
ngram_lm_scale_1.1_attention_scale_2.1 10.71
ngram_lm_scale_1.2_attention_scale_2.5 10.71
ngram_lm_scale_1.3_attention_scale_3.0 10.71
ngram_lm_scale_0.01_attention_scale_0.1 10.72
ngram_lm_scale_0.05_attention_scale_0.05 10.72
ngram_lm_scale_0.08_attention_scale_0.01 10.72
ngram_lm_scale_0.1_attention_scale_0.01 10.72
ngram_lm_scale_0.3_attention_scale_0.01 10.72
ngram_lm_scale_0.7_attention_scale_0.7 10.72
ngram_lm_scale_0.9_attention_scale_1.3 10.72
ngram_lm_scale_1.0_attention_scale_1.7 10.72
ngram_lm_scale_1.1_attention_scale_2.0 10.72
ngram_lm_scale_0.01_attention_scale_0.08 10.73
ngram_lm_scale_0.9_attention_scale_1.2 10.73
ngram_lm_scale_1.1_attention_scale_1.9 10.73
ngram_lm_scale_1.2_attention_scale_2.3 10.73
ngram_lm_scale_1.0_attention_scale_1.5 10.74
ngram_lm_scale_1.2_attention_scale_2.2 10.74
ngram_lm_scale_1.3_attention_scale_2.5 10.74
ngram_lm_scale_1.9_attention_scale_5.0 10.74
ngram_lm_scale_0.01_attention_scale_0.05 10.75
ngram_lm_scale_0.05_attention_scale_0.01 10.75
ngram_lm_scale_0.7_attention_scale_0.6 10.75
ngram_lm_scale_0.9_attention_scale_1.1 10.75
ngram_lm_scale_1.1_attention_scale_1.7 10.75
ngram_lm_scale_1.2_attention_scale_2.1 10.75
ngram_lm_scale_1.7_attention_scale_4.0 10.75
ngram_lm_scale_1.2_attention_scale_2.0 10.76
ngram_lm_scale_1.3_attention_scale_2.3 10.76
ngram_lm_scale_2.0_attention_scale_5.0 10.76
ngram_lm_scale_1.0_attention_scale_1.3 10.77
ngram_lm_scale_1.2_attention_scale_1.9 10.77
ngram_lm_scale_1.5_attention_scale_3.0 10.77
ngram_lm_scale_0.01_attention_scale_0.01 10.78
ngram_lm_scale_0.6_attention_scale_0.3 10.78
ngram_lm_scale_0.7_attention_scale_0.5 10.78
ngram_lm_scale_0.9_attention_scale_1.0 10.78
ngram_lm_scale_2.1_attention_scale_5.0 10.78
ngram_lm_scale_1.1_attention_scale_1.5 10.79
ngram_lm_scale_1.3_attention_scale_2.2 10.79
ngram_lm_scale_0.5_attention_scale_0.1 10.8
ngram_lm_scale_1.0_attention_scale_1.2 10.8
ngram_lm_scale_1.3_attention_scale_2.1 10.8
ngram_lm_scale_1.9_attention_scale_4.0 10.8
ngram_lm_scale_2.2_attention_scale_5.0 10.8
ngram_lm_scale_0.5_attention_scale_0.08 10.81
ngram_lm_scale_0.9_attention_scale_0.9 10.81
ngram_lm_scale_1.2_attention_scale_1.7 10.81
ngram_lm_scale_1.3_attention_scale_2.0 10.81
ngram_lm_scale_1.0_attention_scale_1.1 10.82
ngram_lm_scale_0.5_attention_scale_0.05 10.83
ngram_lm_scale_1.1_attention_scale_1.3 10.83
ngram_lm_scale_1.3_attention_scale_1.9 10.83
ngram_lm_scale_1.5_attention_scale_2.5 10.84
ngram_lm_scale_2.3_attention_scale_5.0 10.84
ngram_lm_scale_1.0_attention_scale_1.0 10.85
ngram_lm_scale_1.2_attention_scale_1.5 10.85
ngram_lm_scale_2.0_attention_scale_4.0 10.85
ngram_lm_scale_1.1_attention_scale_1.2 10.86
ngram_lm_scale_1.7_attention_scale_3.0 10.86
ngram_lm_scale_0.5_attention_scale_0.01 10.87
ngram_lm_scale_1.5_attention_scale_2.3 10.87
ngram_lm_scale_0.7_attention_scale_0.3 10.88
ngram_lm_scale_0.9_attention_scale_0.7 10.88
ngram_lm_scale_1.3_attention_scale_1.7 10.88
ngram_lm_scale_1.0_attention_scale_0.9 10.89
ngram_lm_scale_1.5_attention_scale_2.2 10.89
ngram_lm_scale_2.1_attention_scale_4.0 10.89
ngram_lm_scale_1.1_attention_scale_1.1 10.91
ngram_lm_scale_0.6_attention_scale_0.1 10.92
ngram_lm_scale_0.9_attention_scale_0.6 10.92
ngram_lm_scale_1.5_attention_scale_2.1 10.92
ngram_lm_scale_1.2_attention_scale_1.3 10.93
ngram_lm_scale_2.5_attention_scale_5.0 10.93
ngram_lm_scale_0.6_attention_scale_0.08 10.94
ngram_lm_scale_2.2_attention_scale_4.0 10.94
ngram_lm_scale_1.1_attention_scale_1.0 10.95
ngram_lm_scale_1.3_attention_scale_1.5 10.95
ngram_lm_scale_1.5_attention_scale_2.0 10.96
ngram_lm_scale_1.2_attention_scale_1.2 10.97
ngram_lm_scale_1.7_attention_scale_2.5 10.97
ngram_lm_scale_0.6_attention_scale_0.05 10.98
ngram_lm_scale_1.9_attention_scale_3.0 10.98
ngram_lm_scale_1.0_attention_scale_0.7 10.99
ngram_lm_scale_1.5_attention_scale_1.9 10.99
ngram_lm_scale_2.3_attention_scale_4.0 10.99
ngram_lm_scale_0.9_attention_scale_0.5 11.0
ngram_lm_scale_1.1_attention_scale_0.9 11.0
ngram_lm_scale_0.6_attention_scale_0.01 11.02
ngram_lm_scale_1.2_attention_scale_1.1 11.02
ngram_lm_scale_1.7_attention_scale_2.3 11.03
ngram_lm_scale_1.3_attention_scale_1.3 11.05
ngram_lm_scale_2.0_attention_scale_3.0 11.05
ngram_lm_scale_1.7_attention_scale_2.2 11.07
ngram_lm_scale_1.0_attention_scale_0.6 11.08
ngram_lm_scale_1.5_attention_scale_1.7 11.08
ngram_lm_scale_1.2_attention_scale_1.0 11.09
ngram_lm_scale_0.7_attention_scale_0.1 11.1
ngram_lm_scale_1.3_attention_scale_1.2 11.1
ngram_lm_scale_1.7_attention_scale_2.1 11.11
ngram_lm_scale_2.1_attention_scale_3.0 11.12
ngram_lm_scale_2.5_attention_scale_4.0 11.12
ngram_lm_scale_0.7_attention_scale_0.08 11.13
ngram_lm_scale_1.9_attention_scale_2.5 11.13
ngram_lm_scale_1.7_attention_scale_2.0 11.14
ngram_lm_scale_1.2_attention_scale_0.9 11.16
ngram_lm_scale_1.1_attention_scale_0.7 11.17
ngram_lm_scale_1.3_attention_scale_1.1 11.17
ngram_lm_scale_3.0_attention_scale_5.0 11.17
ngram_lm_scale_0.7_attention_scale_0.05 11.18
ngram_lm_scale_1.5_attention_scale_1.5 11.18
ngram_lm_scale_1.0_attention_scale_0.5 11.19
ngram_lm_scale_1.7_attention_scale_1.9 11.2
ngram_lm_scale_2.2_attention_scale_3.0 11.21
ngram_lm_scale_1.9_attention_scale_2.3 11.22
ngram_lm_scale_2.0_attention_scale_2.5 11.23
ngram_lm_scale_0.9_attention_scale_0.3 11.25
ngram_lm_scale_1.3_attention_scale_1.0 11.26
ngram_lm_scale_0.7_attention_scale_0.01 11.27
ngram_lm_scale_1.9_attention_scale_2.2 11.27
ngram_lm_scale_1.1_attention_scale_0.6 11.29
ngram_lm_scale_2.3_attention_scale_3.0 11.31
ngram_lm_scale_1.7_attention_scale_1.7 11.33
ngram_lm_scale_1.5_attention_scale_1.3 11.34
ngram_lm_scale_1.9_attention_scale_2.1 11.34
ngram_lm_scale_2.0_attention_scale_2.3 11.34
ngram_lm_scale_2.1_attention_scale_2.5 11.35
ngram_lm_scale_1.3_attention_scale_0.9 11.36
ngram_lm_scale_1.2_attention_scale_0.7 11.39
ngram_lm_scale_1.9_attention_scale_2.0 11.4
ngram_lm_scale_2.0_attention_scale_2.2 11.4
ngram_lm_scale_1.5_attention_scale_1.2 11.43
ngram_lm_scale_1.1_attention_scale_0.5 11.44
ngram_lm_scale_2.0_attention_scale_2.1 11.47
ngram_lm_scale_2.1_attention_scale_2.3 11.47
ngram_lm_scale_2.2_attention_scale_2.5 11.47
ngram_lm_scale_1.9_attention_scale_1.9 11.48
ngram_lm_scale_1.7_attention_scale_1.5 11.5
ngram_lm_scale_2.5_attention_scale_3.0 11.51
ngram_lm_scale_3.0_attention_scale_4.0 11.51
ngram_lm_scale_1.0_attention_scale_0.3 11.53
ngram_lm_scale_1.2_attention_scale_0.6 11.53
ngram_lm_scale_1.5_attention_scale_1.1 11.54
ngram_lm_scale_2.1_attention_scale_2.2 11.54
ngram_lm_scale_2.0_attention_scale_2.0 11.55
ngram_lm_scale_2.3_attention_scale_2.5 11.59
ngram_lm_scale_2.2_attention_scale_2.3 11.61
ngram_lm_scale_2.1_attention_scale_2.1 11.62
ngram_lm_scale_1.3_attention_scale_0.7 11.63
ngram_lm_scale_2.0_attention_scale_1.9 11.63
ngram_lm_scale_1.9_attention_scale_1.7 11.66
ngram_lm_scale_1.5_attention_scale_1.0 11.67
ngram_lm_scale_2.2_attention_scale_2.2 11.69
ngram_lm_scale_0.9_attention_scale_0.1 11.7
ngram_lm_scale_2.1_attention_scale_2.0 11.71
ngram_lm_scale_1.2_attention_scale_0.5 11.72
ngram_lm_scale_1.7_attention_scale_1.3 11.72
ngram_lm_scale_2.3_attention_scale_2.3 11.75
ngram_lm_scale_0.9_attention_scale_0.08 11.77
ngram_lm_scale_2.2_attention_scale_2.1 11.78
ngram_lm_scale_2.1_attention_scale_1.9 11.82
ngram_lm_scale_1.3_attention_scale_0.6 11.83
ngram_lm_scale_1.5_attention_scale_0.9 11.85
ngram_lm_scale_2.0_attention_scale_1.7 11.85
ngram_lm_scale_2.3_attention_scale_2.2 11.86
ngram_lm_scale_0.9_attention_scale_0.05 11.87
ngram_lm_scale_1.1_attention_scale_0.3 11.87
ngram_lm_scale_1.7_attention_scale_1.2 11.88
ngram_lm_scale_1.9_attention_scale_1.5 11.9
ngram_lm_scale_2.2_attention_scale_2.0 11.9
ngram_lm_scale_2.5_attention_scale_2.5 11.9
ngram_lm_scale_4.0_attention_scale_5.0 11.93
ngram_lm_scale_2.3_attention_scale_2.1 11.97
ngram_lm_scale_0.9_attention_scale_0.01 12.0
ngram_lm_scale_2.2_attention_scale_1.9 12.02
ngram_lm_scale_1.7_attention_scale_1.1 12.05
ngram_lm_scale_1.3_attention_scale_0.5 12.07
ngram_lm_scale_2.1_attention_scale_1.7 12.07
ngram_lm_scale_2.3_attention_scale_2.0 12.09
ngram_lm_scale_1.0_attention_scale_0.1 12.11
ngram_lm_scale_2.5_attention_scale_2.3 12.11
ngram_lm_scale_2.0_attention_scale_1.5 12.14
ngram_lm_scale_1.0_attention_scale_0.08 12.19
ngram_lm_scale_3.0_attention_scale_3.0 12.19
ngram_lm_scale_1.9_attention_scale_1.3 12.22
ngram_lm_scale_1.7_attention_scale_1.0 12.23
ngram_lm_scale_2.3_attention_scale_1.9 12.23
ngram_lm_scale_2.5_attention_scale_2.2 12.23
ngram_lm_scale_1.5_attention_scale_0.7 12.27
ngram_lm_scale_1.2_attention_scale_0.3 12.28
ngram_lm_scale_2.2_attention_scale_1.7 12.3
ngram_lm_scale_1.0_attention_scale_0.05 12.32
ngram_lm_scale_2.5_attention_scale_2.1 12.37
ngram_lm_scale_2.1_attention_scale_1.5 12.39
ngram_lm_scale_1.9_attention_scale_1.2 12.41
ngram_lm_scale_1.7_attention_scale_0.9 12.46
ngram_lm_scale_1.0_attention_scale_0.01 12.49
ngram_lm_scale_2.0_attention_scale_1.3 12.5
ngram_lm_scale_2.5_attention_scale_2.0 12.51
ngram_lm_scale_2.3_attention_scale_1.7 12.54
ngram_lm_scale_1.5_attention_scale_0.6 12.55
ngram_lm_scale_1.1_attention_scale_0.1 12.58
ngram_lm_scale_1.9_attention_scale_1.1 12.62
ngram_lm_scale_2.2_attention_scale_1.5 12.64
ngram_lm_scale_1.1_attention_scale_0.08 12.67
ngram_lm_scale_2.5_attention_scale_1.9 12.67
ngram_lm_scale_4.0_attention_scale_4.0 12.67
ngram_lm_scale_1.3_attention_scale_0.3 12.71
ngram_lm_scale_2.0_attention_scale_1.2 12.71
ngram_lm_scale_2.1_attention_scale_1.3 12.78
ngram_lm_scale_3.0_attention_scale_2.5 12.8
ngram_lm_scale_1.1_attention_scale_0.05 12.81
ngram_lm_scale_1.9_attention_scale_1.0 12.85
ngram_lm_scale_1.5_attention_scale_0.5 12.86
ngram_lm_scale_2.3_attention_scale_1.5 12.91
ngram_lm_scale_2.0_attention_scale_1.1 12.92
ngram_lm_scale_1.7_attention_scale_0.7 12.99
ngram_lm_scale_2.1_attention_scale_1.2 12.99
ngram_lm_scale_5.0_attention_scale_5.0 13.01
ngram_lm_scale_1.1_attention_scale_0.01 13.02
ngram_lm_scale_2.5_attention_scale_1.7 13.02
ngram_lm_scale_2.2_attention_scale_1.3 13.05
ngram_lm_scale_3.0_attention_scale_2.3 13.09
ngram_lm_scale_1.2_attention_scale_0.1 13.1
ngram_lm_scale_1.9_attention_scale_0.9 13.11
ngram_lm_scale_2.0_attention_scale_1.0 13.17
ngram_lm_scale_1.2_attention_scale_0.08 13.2
ngram_lm_scale_2.1_attention_scale_1.1 13.22
ngram_lm_scale_3.0_attention_scale_2.2 13.24
ngram_lm_scale_2.2_attention_scale_1.2 13.28
ngram_lm_scale_1.7_attention_scale_0.6 13.33
ngram_lm_scale_2.3_attention_scale_1.3 13.34
ngram_lm_scale_1.2_attention_scale_0.05 13.36
ngram_lm_scale_3.0_attention_scale_2.1 13.42
ngram_lm_scale_2.5_attention_scale_1.5 13.43
ngram_lm_scale_2.0_attention_scale_0.9 13.48
ngram_lm_scale_2.1_attention_scale_1.0 13.51
ngram_lm_scale_2.2_attention_scale_1.1 13.56
ngram_lm_scale_1.2_attention_scale_0.01 13.6
ngram_lm_scale_2.3_attention_scale_1.2 13.6
ngram_lm_scale_3.0_attention_scale_2.0 13.62
ngram_lm_scale_1.3_attention_scale_0.1 13.65
ngram_lm_scale_1.5_attention_scale_0.3 13.68
ngram_lm_scale_1.7_attention_scale_0.5 13.72
ngram_lm_scale_1.3_attention_scale_0.08 13.76
ngram_lm_scale_1.9_attention_scale_0.7 13.78
ngram_lm_scale_3.0_attention_scale_1.9 13.81
ngram_lm_scale_2.1_attention_scale_0.9 13.82
ngram_lm_scale_2.2_attention_scale_1.0 13.85
ngram_lm_scale_4.0_attention_scale_3.0 13.85
ngram_lm_scale_2.3_attention_scale_1.1 13.89
ngram_lm_scale_1.3_attention_scale_0.05 13.94
ngram_lm_scale_2.5_attention_scale_1.3 13.94
ngram_lm_scale_5.0_attention_scale_4.0 13.97
ngram_lm_scale_1.9_attention_scale_0.6 14.15
ngram_lm_scale_2.0_attention_scale_0.7 14.16
ngram_lm_scale_2.2_attention_scale_0.9 14.17
ngram_lm_scale_2.3_attention_scale_1.0 14.19
ngram_lm_scale_1.3_attention_scale_0.01 14.2
ngram_lm_scale_2.5_attention_scale_1.2 14.2
ngram_lm_scale_3.0_attention_scale_1.7 14.26
ngram_lm_scale_2.5_attention_scale_1.1 14.48
ngram_lm_scale_2.3_attention_scale_0.9 14.5
ngram_lm_scale_2.1_attention_scale_0.7 14.53
ngram_lm_scale_2.0_attention_scale_0.6 14.54
ngram_lm_scale_1.9_attention_scale_0.5 14.57
ngram_lm_scale_4.0_attention_scale_2.5 14.63
ngram_lm_scale_1.7_attention_scale_0.3 14.64
ngram_lm_scale_3.0_attention_scale_1.5 14.71
ngram_lm_scale_1.5_attention_scale_0.1 14.75
ngram_lm_scale_2.5_attention_scale_1.0 14.79
ngram_lm_scale_2.2_attention_scale_0.7 14.86
ngram_lm_scale_1.5_attention_scale_0.08 14.87
ngram_lm_scale_2.1_attention_scale_0.6 14.91
ngram_lm_scale_2.0_attention_scale_0.5 14.95
ngram_lm_scale_4.0_attention_scale_2.3 14.98
ngram_lm_scale_1.5_attention_scale_0.05 15.05
ngram_lm_scale_2.5_attention_scale_0.9 15.12
ngram_lm_scale_4.0_attention_scale_2.2 15.17
ngram_lm_scale_2.3_attention_scale_0.7 15.21
ngram_lm_scale_3.0_attention_scale_1.3 15.22
ngram_lm_scale_2.2_attention_scale_0.6 15.27
ngram_lm_scale_1.5_attention_scale_0.01 15.3
ngram_lm_scale_5.0_attention_scale_3.0 15.32
ngram_lm_scale_2.1_attention_scale_0.5 15.33
ngram_lm_scale_4.0_attention_scale_2.1 15.37
ngram_lm_scale_1.9_attention_scale_0.3 15.5
ngram_lm_scale_3.0_attention_scale_1.2 15.51
ngram_lm_scale_4.0_attention_scale_2.0 15.57
ngram_lm_scale_2.3_attention_scale_0.6 15.61
ngram_lm_scale_2.2_attention_scale_0.5 15.68
ngram_lm_scale_1.7_attention_scale_0.1 15.72
ngram_lm_scale_4.0_attention_scale_1.9 15.79
ngram_lm_scale_3.0_attention_scale_1.1 15.82
ngram_lm_scale_1.7_attention_scale_0.08 15.83
ngram_lm_scale_2.5_attention_scale_0.7 15.85
ngram_lm_scale_2.0_attention_scale_0.3 15.87
ngram_lm_scale_2.3_attention_scale_0.5 16.0
ngram_lm_scale_1.7_attention_scale_0.05 16.01
ngram_lm_scale_3.0_attention_scale_1.0 16.11
ngram_lm_scale_5.0_attention_scale_2.5 16.12
ngram_lm_scale_2.5_attention_scale_0.6 16.19
ngram_lm_scale_2.1_attention_scale_0.3 16.2
ngram_lm_scale_4.0_attention_scale_1.7 16.22
ngram_lm_scale_1.7_attention_scale_0.01 16.23
ngram_lm_scale_3.0_attention_scale_0.9 16.4
ngram_lm_scale_5.0_attention_scale_2.3 16.44
ngram_lm_scale_1.9_attention_scale_0.1 16.5
ngram_lm_scale_2.2_attention_scale_0.3 16.53
ngram_lm_scale_2.5_attention_scale_0.5 16.54
ngram_lm_scale_1.9_attention_scale_0.08 16.6
ngram_lm_scale_5.0_attention_scale_2.2 16.6
ngram_lm_scale_4.0_attention_scale_1.5 16.63
ngram_lm_scale_1.9_attention_scale_0.05 16.74
ngram_lm_scale_5.0_attention_scale_2.1 16.77
ngram_lm_scale_2.3_attention_scale_0.3 16.81
ngram_lm_scale_2.0_attention_scale_0.1 16.83
ngram_lm_scale_2.0_attention_scale_0.08 16.92
ngram_lm_scale_5.0_attention_scale_2.0 16.94
ngram_lm_scale_1.9_attention_scale_0.01 16.95
ngram_lm_scale_3.0_attention_scale_0.7 16.96
ngram_lm_scale_2.0_attention_scale_0.05 17.05
ngram_lm_scale_4.0_attention_scale_1.3 17.05
ngram_lm_scale_2.1_attention_scale_0.1 17.11
ngram_lm_scale_5.0_attention_scale_1.9 17.11
ngram_lm_scale_2.1_attention_scale_0.08 17.21
ngram_lm_scale_2.0_attention_scale_0.01 17.24
ngram_lm_scale_3.0_attention_scale_0.6 17.26
ngram_lm_scale_4.0_attention_scale_1.2 17.27
ngram_lm_scale_2.5_attention_scale_0.3 17.28
ngram_lm_scale_2.1_attention_scale_0.05 17.34
ngram_lm_scale_2.2_attention_scale_0.1 17.38
ngram_lm_scale_5.0_attention_scale_1.7 17.44
ngram_lm_scale_2.2_attention_scale_0.08 17.46
ngram_lm_scale_4.0_attention_scale_1.1 17.5
ngram_lm_scale_2.1_attention_scale_0.01 17.52
ngram_lm_scale_3.0_attention_scale_0.5 17.57
ngram_lm_scale_2.2_attention_scale_0.05 17.59
ngram_lm_scale_2.3_attention_scale_0.1 17.62
ngram_lm_scale_2.3_attention_scale_0.08 17.7
ngram_lm_scale_4.0_attention_scale_1.0 17.72
ngram_lm_scale_2.2_attention_scale_0.01 17.76
ngram_lm_scale_5.0_attention_scale_1.5 17.8
ngram_lm_scale_2.3_attention_scale_0.05 17.82
ngram_lm_scale_4.0_attention_scale_0.9 17.94
ngram_lm_scale_2.3_attention_scale_0.01 17.98
ngram_lm_scale_2.5_attention_scale_0.1 18.03
ngram_lm_scale_2.5_attention_scale_0.08 18.1
ngram_lm_scale_5.0_attention_scale_1.3 18.12
ngram_lm_scale_3.0_attention_scale_0.3 18.17
ngram_lm_scale_2.5_attention_scale_0.05 18.2
ngram_lm_scale_5.0_attention_scale_1.2 18.29
ngram_lm_scale_2.5_attention_scale_0.01 18.33
ngram_lm_scale_4.0_attention_scale_0.7 18.36
ngram_lm_scale_5.0_attention_scale_1.1 18.48
ngram_lm_scale_4.0_attention_scale_0.6 18.58
ngram_lm_scale_5.0_attention_scale_1.0 18.65
ngram_lm_scale_3.0_attention_scale_0.1 18.75
ngram_lm_scale_4.0_attention_scale_0.5 18.79
ngram_lm_scale_3.0_attention_scale_0.08 18.81
ngram_lm_scale_5.0_attention_scale_0.9 18.81
ngram_lm_scale_3.0_attention_scale_0.05 18.89
ngram_lm_scale_3.0_attention_scale_0.01 18.99
ngram_lm_scale_5.0_attention_scale_0.7 19.11
ngram_lm_scale_4.0_attention_scale_0.3 19.18
ngram_lm_scale_5.0_attention_scale_0.6 19.25
ngram_lm_scale_5.0_attention_scale_0.5 19.41
ngram_lm_scale_4.0_attention_scale_0.1 19.57
ngram_lm_scale_4.0_attention_scale_0.08 19.61
ngram_lm_scale_4.0_attention_scale_0.05 19.67
ngram_lm_scale_5.0_attention_scale_0.3 19.71
ngram_lm_scale_4.0_attention_scale_0.01 19.73
ngram_lm_scale_5.0_attention_scale_0.1 19.99
ngram_lm_scale_5.0_attention_scale_0.08 20.01
ngram_lm_scale_5.0_attention_scale_0.05 20.05
ngram_lm_scale_5.0_attention_scale_0.01 20.11
2022-04-09 04:57:33,455 INFO [decode_test.py:730] Done!