logs/model_worker_1c4fbd.log

2024-07-11 22:32:31 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=40008, worker_address='http://10.140.60.182:40008', controller_address='http://10.140.60.209:10075', model_path='share_internvl/InternVL2-Pro/', model_name=None, device='auto', limit_model_concurrency=5, stream_interval=1, load_8bit=False)
2024-07-11 22:32:31 | INFO | model_worker | Loading the model InternVL2-Pro on worker 1c4fbd ...
2024-07-11 22:32:31 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-07-11 22:32:31 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-07-11 22:32:35 | ERROR | stderr | 
Loading checkpoint shards:   0%|          | 0/33 [00:00<?, ?it/s]
2024-07-11 22:32:37 | ERROR | stderr | 
Loading checkpoint shards:   3%|▎         | 1/33 [00:02<01:11,  2.24s/it]
2024-07-11 22:32:40 | ERROR | stderr | 
Loading checkpoint shards:   6%|▌         | 2/33 [00:05<01:23,  2.71s/it]
2024-07-11 22:32:43 | ERROR | stderr | 
Loading checkpoint shards:   9%|▉         | 3/33 [00:07<01:14,  2.48s/it]
2024-07-11 22:32:45 | ERROR | stderr | 
Loading checkpoint shards:  12%|█▏        | 4/33 [00:10<01:15,  2.59s/it]
2024-07-11 22:32:48 | ERROR | stderr | 
Loading checkpoint shards:  15%|█▌        | 5/33 [00:13<01:16,  2.74s/it]
2024-07-11 22:32:51 | ERROR | stderr | 
Loading checkpoint shards:  18%|█▊        | 6/33 [00:15<01:10,  2.60s/it]
2024-07-11 22:32:53 | ERROR | stderr | 
Loading checkpoint shards:  21%|██        | 7/33 [00:17<01:03,  2.44s/it]
2024-07-11 22:32:55 | ERROR | stderr | 
Loading checkpoint shards:  24%|██▍       | 8/33 [00:19<00:58,  2.33s/it]
2024-07-11 22:32:57 | ERROR | stderr | 
Loading checkpoint shards:  27%|██▋       | 9/33 [00:21<00:53,  2.24s/it]
2024-07-11 22:32:59 | ERROR | stderr | 
Loading checkpoint shards:  30%|███       | 10/33 [00:23<00:50,  2.20s/it]
2024-07-11 22:33:01 | ERROR | stderr | 
Loading checkpoint shards:  33%|███▎      | 11/33 [00:25<00:47,  2.15s/it]
2024-07-11 22:33:03 | ERROR | stderr | 
Loading checkpoint shards:  36%|███▋      | 12/33 [00:27<00:44,  2.10s/it]
2024-07-11 22:33:05 | ERROR | stderr | 
Loading checkpoint shards:  39%|███▉      | 13/33 [00:29<00:41,  2.07s/it]
2024-07-11 22:33:07 | ERROR | stderr | 
Loading checkpoint shards:  42%|████▏     | 14/33 [00:32<00:39,  2.07s/it]
2024-07-11 22:33:09 | ERROR | stderr | 
Loading checkpoint shards:  45%|████▌     | 15/33 [00:34<00:37,  2.06s/it]
2024-07-11 22:33:11 | ERROR | stderr | 
Loading checkpoint shards:  48%|████▊     | 16/33 [00:36<00:34,  2.05s/it]
2024-07-11 22:33:13 | ERROR | stderr | 
Loading checkpoint shards:  52%|█████▏    | 17/33 [00:38<00:32,  2.03s/it]
2024-07-11 22:33:15 | ERROR | stderr | 
Loading checkpoint shards:  55%|█████▍    | 18/33 [00:40<00:30,  2.01s/it]
2024-07-11 22:33:17 | ERROR | stderr | 
Loading checkpoint shards:  58%|█████▊    | 19/33 [00:41<00:27,  1.97s/it]
2024-07-11 22:33:19 | ERROR | stderr | 
Loading checkpoint shards:  61%|██████    | 20/33 [00:43<00:25,  1.95s/it]
2024-07-11 22:33:21 | ERROR | stderr | 
Loading checkpoint shards:  64%|██████▎   | 21/33 [00:45<00:23,  1.93s/it]
2024-07-11 22:33:23 | ERROR | stderr | 
Loading checkpoint shards:  67%|██████▋   | 22/33 [00:47<00:21,  1.94s/it]
2024-07-11 22:33:25 | ERROR | stderr | 
Loading checkpoint shards:  70%|██████▉   | 23/33 [00:49<00:19,  1.91s/it]
2024-07-11 22:33:27 | ERROR | stderr | 
Loading checkpoint shards:  73%|███████▎  | 24/33 [00:51<00:17,  1.89s/it]
2024-07-11 22:33:28 | ERROR | stderr | 
Loading checkpoint shards:  76%|███████▌  | 25/33 [00:53<00:14,  1.87s/it]
2024-07-11 22:33:30 | ERROR | stderr | 
Loading checkpoint shards:  79%|███████▉  | 26/33 [00:55<00:13,  1.87s/it]
2024-07-11 22:33:32 | ERROR | stderr | 
Loading checkpoint shards:  82%|████████▏ | 27/33 [00:56<00:11,  1.85s/it]
2024-07-11 22:33:34 | ERROR | stderr | 
Loading checkpoint shards:  85%|████████▍ | 28/33 [00:58<00:09,  1.84s/it]
2024-07-11 22:33:36 | ERROR | stderr | 
Loading checkpoint shards:  88%|████████▊ | 29/33 [01:00<00:07,  1.82s/it]
2024-07-11 22:33:37 | ERROR | stderr | 
Loading checkpoint shards:  91%|█████████ | 30/33 [01:02<00:05,  1.83s/it]
2024-07-11 22:33:39 | ERROR | stderr | 
Loading checkpoint shards:  94%|█████████▍| 31/33 [01:04<00:03,  1.83s/it]
2024-07-11 22:33:41 | ERROR | stderr | 
Loading checkpoint shards:  97%|█████████▋| 32/33 [01:05<00:01,  1.74s/it]
2024-07-11 22:33:42 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████| 33/33 [01:06<00:00,  1.55s/it]
2024-07-11 22:33:42 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████| 33/33 [01:06<00:00,  2.02s/it]
2024-07-11 22:33:42 | ERROR | stderr | 
2024-07-11 22:33:43 | INFO | model_worker | Register to controller
2024-07-11 22:33:43 | ERROR | stderr | INFO:     Started server process [13199]
2024-07-11 22:33:43 | ERROR | stderr | INFO:     Waiting for application startup.
2024-07-11 22:33:43 | ERROR | stderr | INFO:     Application startup complete.
2024-07-11 22:33:43 | ERROR | stderr | INFO:     Uvicorn running on http://0.0.0.0:40008 (Press CTRL+C to quit)
2024-07-11 22:33:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:34:02 | INFO | stdout | INFO:     10.140.60.209:45980 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:34:04 | INFO | stdout | INFO:     10.140.60.209:46001 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:34:05 | INFO | stdout | INFO:     10.140.60.209:46020 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:34:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:34:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:34:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:34:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:35:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:35:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0