2024-07-11 22:16:03 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=40008, worker_address='http://10.140.60.182:40008', controller_address='http://10.140.60.209:10075', model_path='share_internvl/InternVL2-Pro/', model_name=None, device='auto', limit_model_concurrency=5, stream_interval=1, load_8bit=False)
2024-07-11 22:16:03 | INFO | model_worker | Loading the model InternVL2-Pro on worker 3a45cb ...
2024-07-11 22:16:03 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-07-11 22:16:03 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-07-11 22:16:13 | ERROR | stderr | 
Loading checkpoint shards:   0%|          | 0/33 [00:00<?, ?it/s]
2024-07-11 22:16:16 | ERROR | stderr | 
Loading checkpoint shards:   3%|▎         | 1/33 [00:02<01:14,  2.32s/it]
2024-07-11 22:16:18 | ERROR | stderr | 
Loading checkpoint shards:   6%|▌         | 2/33 [00:04<01:13,  2.36s/it]
2024-07-11 22:16:20 | ERROR | stderr | 
Loading checkpoint shards:   9%|▉         | 3/33 [00:07<01:13,  2.44s/it]
2024-07-11 22:16:23 | ERROR | stderr | 
Loading checkpoint shards:  12%|█▏        | 4/33 [00:09<01:08,  2.36s/it]
2024-07-11 22:16:25 | ERROR | stderr | 
Loading checkpoint shards:  15%|█▌        | 5/33 [00:11<01:05,  2.32s/it]
2024-07-11 22:16:27 | ERROR | stderr | 
Loading checkpoint shards:  18%|█▊        | 6/33 [00:13<01:01,  2.29s/it]
2024-07-11 22:16:29 | ERROR | stderr | 
Loading checkpoint shards:  21%|██        | 7/33 [00:16<00:57,  2.23s/it]
2024-07-11 22:16:31 | ERROR | stderr | 
Loading checkpoint shards:  24%|██▍       | 8/33 [00:18<00:54,  2.18s/it]
2024-07-11 22:16:33 | ERROR | stderr | 
Loading checkpoint shards:  27%|██▋       | 9/33 [00:20<00:51,  2.13s/it]
2024-07-11 22:16:36 | ERROR | stderr | 
Loading checkpoint shards:  30%|███       | 10/33 [00:22<00:49,  2.14s/it]
2024-07-11 22:16:38 | ERROR | stderr | 
Loading checkpoint shards:  33%|███▎      | 11/33 [00:24<00:46,  2.12s/it]
2024-07-11 22:16:40 | ERROR | stderr | 
Loading checkpoint shards:  36%|███▋      | 12/33 [00:26<00:44,  2.11s/it]
2024-07-11 22:16:42 | ERROR | stderr | 
Loading checkpoint shards:  39%|███▉      | 13/33 [00:29<00:46,  2.31s/it]
2024-07-11 22:16:45 | ERROR | stderr | 
Loading checkpoint shards:  42%|████▏     | 14/33 [00:31<00:42,  2.26s/it]
2024-07-11 22:16:47 | ERROR | stderr | 
Loading checkpoint shards:  45%|████▌     | 15/33 [00:33<00:39,  2.21s/it]
2024-07-11 22:16:49 | ERROR | stderr | 
Loading checkpoint shards:  48%|████▊     | 16/33 [00:35<00:36,  2.16s/it]
2024-07-11 22:16:51 | ERROR | stderr | 
Loading checkpoint shards:  52%|█████▏    | 17/33 [00:37<00:33,  2.12s/it]
2024-07-11 22:16:53 | ERROR | stderr | 
Loading checkpoint shards:  55%|█████▍    | 18/33 [00:39<00:31,  2.12s/it]
2024-07-11 22:16:55 | ERROR | stderr | 
Loading checkpoint shards:  58%|█████▊    | 19/33 [00:41<00:29,  2.10s/it]
2024-07-11 22:16:57 | ERROR | stderr | 
Loading checkpoint shards:  61%|██████    | 20/33 [00:43<00:27,  2.08s/it]
2024-07-11 22:16:59 | ERROR | stderr | 
Loading checkpoint shards:  64%|██████▎   | 21/33 [00:45<00:24,  2.07s/it]
2024-07-11 22:17:01 | ERROR | stderr | 
Loading checkpoint shards:  67%|██████▋   | 22/33 [00:47<00:22,  2.08s/it]
2024-07-11 22:17:03 | ERROR | stderr | 
Loading checkpoint shards:  70%|██████▉   | 23/33 [00:49<00:20,  2.05s/it]
2024-07-11 22:17:05 | ERROR | stderr | 
Loading checkpoint shards:  73%|███████▎  | 24/33 [00:51<00:18,  2.03s/it]
2024-07-11 22:17:07 | ERROR | stderr | 
Loading checkpoint shards:  76%|███████▌  | 25/33 [00:53<00:16,  2.05s/it]
2024-07-11 22:17:09 | ERROR | stderr | 
Loading checkpoint shards:  79%|███████▉  | 26/33 [00:55<00:14,  2.04s/it]
2024-07-11 22:17:11 | ERROR | stderr | 
Loading checkpoint shards:  82%|████████▏ | 27/33 [00:57<00:12,  2.02s/it]
2024-07-11 22:17:13 | ERROR | stderr | 
Loading checkpoint shards:  85%|████████▍ | 28/33 [00:59<00:09,  2.00s/it]
2024-07-11 22:17:15 | ERROR | stderr | 
Loading checkpoint shards:  88%|████████▊ | 29/33 [01:01<00:07,  1.98s/it]
2024-07-11 22:17:17 | ERROR | stderr | 
Loading checkpoint shards:  91%|█████████ | 30/33 [01:03<00:05,  2.00s/it]
2024-07-11 22:17:19 | ERROR | stderr | 
Loading checkpoint shards:  94%|█████████▍| 31/33 [01:05<00:03,  1.98s/it]
2024-07-11 22:17:21 | ERROR | stderr | 
Loading checkpoint shards:  97%|█████████▋| 32/33 [01:07<00:01,  1.88s/it]
2024-07-11 22:17:22 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████| 33/33 [01:08<00:00,  1.68s/it]
2024-07-11 22:17:22 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████| 33/33 [01:08<00:00,  2.08s/it]
2024-07-11 22:17:22 | ERROR | stderr | 
2024-07-11 22:17:23 | INFO | model_worker | Register to controller
2024-07-11 22:17:23 | ERROR | stderr | INFO:     Started server process [106820]
2024-07-11 22:17:23 | ERROR | stderr | INFO:     Waiting for application startup.
2024-07-11 22:17:23 | ERROR | stderr | INFO:     Application startup complete.
2024-07-11 22:17:23 | ERROR | stderr | INFO:     Uvicorn running on http://0.0.0.0:40008 (Press CTRL+C to quit)
2024-07-11 22:17:38 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:17:49 | INFO | stdout | INFO:     10.140.60.209:39404 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:17:53 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:18:08 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:18:08 | ERROR | model_worker | heart beat error: HTTPConnectionPool(host='10.140.60.209', port=10075): Max retries exceeded with url: /receive_heart_beat (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f552f1f8700>: Failed to establish a new connection: [Errno 111] Connection refused'))
2024-07-11 22:18:13 | INFO | model_worker | Register to controller
2024-07-11 22:18:17 | INFO | stdout | INFO:     10.140.60.209:39596 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:18:21 | INFO | stdout | INFO:     10.140.60.209:39722 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:18:22 | INFO | stdout | INFO:     10.140.60.209:39742 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:18:22 | INFO | stdout | INFO:     10.140.60.209:39762 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:18:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:18:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:18:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:19:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:19:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:19:32 | INFO | stdout | INFO:     10.140.60.209:40206 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:19:32 | INFO | stdout | INFO:     10.140.60.209:40222 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:19:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:19:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:19:59 | INFO | stdout | INFO:     10.140.60.209:40318 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:19:59 | INFO | stdout | INFO:     10.140.60.209:40336 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:20:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:20:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:20:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:20:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:21:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:21:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:21:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:21:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:22:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:22:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:22:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:22:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:23:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:23:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:23:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:23:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:24:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:24:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:24:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:24:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:25:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:25:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:25:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:25:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:26:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0
2024-07-11 22:26:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0