|
2024-07-11 22:16:03 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=40008, worker_address='http://10.140.60.182:40008', controller_address='http://10.140.60.209:10075', model_path='share_internvl/InternVL2-Pro/', model_name=None, device='auto', limit_model_concurrency=5, stream_interval=1, load_8bit=False) |
|
2024-07-11 22:16:03 | INFO | model_worker | Loading the model InternVL2-Pro on worker 3a45cb ... |
|
2024-07-11 22:16:03 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. |
|
2024-07-11 22:16:03 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. |
|
2024-07-11 22:16:13 | ERROR | stderr |
Loading checkpoint shards: 0%| | 0/33 [00:00<?, ?it/s] |
|
2024-07-11 22:16:16 | ERROR | stderr |
Loading checkpoint shards: 3%|▎ | 1/33 [00:02<01:14, 2.32s/it] |
|
2024-07-11 22:16:18 | ERROR | stderr |
Loading checkpoint shards: 6%|▌ | 2/33 [00:04<01:13, 2.36s/it] |
|
2024-07-11 22:16:20 | ERROR | stderr |
Loading checkpoint shards: 9%|▉ | 3/33 [00:07<01:13, 2.44s/it] |
|
2024-07-11 22:16:23 | ERROR | stderr |
Loading checkpoint shards: 12%|█▏ | 4/33 [00:09<01:08, 2.36s/it] |
|
2024-07-11 22:16:25 | ERROR | stderr |
Loading checkpoint shards: 15%|█▌ | 5/33 [00:11<01:05, 2.32s/it] |
|
2024-07-11 22:16:27 | ERROR | stderr |
Loading checkpoint shards: 18%|█▊ | 6/33 [00:13<01:01, 2.29s/it] |
|
2024-07-11 22:16:29 | ERROR | stderr |
Loading checkpoint shards: 21%|██ | 7/33 [00:16<00:57, 2.23s/it] |
|
2024-07-11 22:16:31 | ERROR | stderr |
Loading checkpoint shards: 24%|██▍ | 8/33 [00:18<00:54, 2.18s/it] |
|
2024-07-11 22:16:33 | ERROR | stderr |
Loading checkpoint shards: 27%|██▋ | 9/33 [00:20<00:51, 2.13s/it] |
|
2024-07-11 22:16:36 | ERROR | stderr |
Loading checkpoint shards: 30%|███ | 10/33 [00:22<00:49, 2.14s/it] |
|
2024-07-11 22:16:38 | ERROR | stderr |
Loading checkpoint shards: 33%|███▎ | 11/33 [00:24<00:46, 2.12s/it] |
|
2024-07-11 22:16:40 | ERROR | stderr |
Loading checkpoint shards: 36%|███▋ | 12/33 [00:26<00:44, 2.11s/it] |
|
2024-07-11 22:16:42 | ERROR | stderr |
Loading checkpoint shards: 39%|███▉ | 13/33 [00:29<00:46, 2.31s/it] |
|
2024-07-11 22:16:45 | ERROR | stderr |
Loading checkpoint shards: 42%|████▏ | 14/33 [00:31<00:42, 2.26s/it] |
|
2024-07-11 22:16:47 | ERROR | stderr |
Loading checkpoint shards: 45%|████▌ | 15/33 [00:33<00:39, 2.21s/it] |
|
2024-07-11 22:16:49 | ERROR | stderr |
Loading checkpoint shards: 48%|████▊ | 16/33 [00:35<00:36, 2.16s/it] |
|
2024-07-11 22:16:51 | ERROR | stderr |
Loading checkpoint shards: 52%|█████▏ | 17/33 [00:37<00:33, 2.12s/it] |
|
2024-07-11 22:16:53 | ERROR | stderr |
Loading checkpoint shards: 55%|█████▍ | 18/33 [00:39<00:31, 2.12s/it] |
|
2024-07-11 22:16:55 | ERROR | stderr |
Loading checkpoint shards: 58%|█████▊ | 19/33 [00:41<00:29, 2.10s/it] |
|
2024-07-11 22:16:57 | ERROR | stderr |
Loading checkpoint shards: 61%|██████ | 20/33 [00:43<00:27, 2.08s/it] |
|
2024-07-11 22:16:59 | ERROR | stderr |
Loading checkpoint shards: 64%|██████▎ | 21/33 [00:45<00:24, 2.07s/it] |
|
2024-07-11 22:17:01 | ERROR | stderr |
Loading checkpoint shards: 67%|██████▋ | 22/33 [00:47<00:22, 2.08s/it] |
|
2024-07-11 22:17:03 | ERROR | stderr |
Loading checkpoint shards: 70%|██████▉ | 23/33 [00:49<00:20, 2.05s/it] |
|
2024-07-11 22:17:05 | ERROR | stderr |
Loading checkpoint shards: 73%|███████▎ | 24/33 [00:51<00:18, 2.03s/it] |
|
2024-07-11 22:17:07 | ERROR | stderr |
Loading checkpoint shards: 76%|███████▌ | 25/33 [00:53<00:16, 2.05s/it] |
|
2024-07-11 22:17:09 | ERROR | stderr |
Loading checkpoint shards: 79%|███████▉ | 26/33 [00:55<00:14, 2.04s/it] |
|
2024-07-11 22:17:11 | ERROR | stderr |
Loading checkpoint shards: 82%|████████▏ | 27/33 [00:57<00:12, 2.02s/it] |
|
2024-07-11 22:17:13 | ERROR | stderr |
Loading checkpoint shards: 85%|████████▍ | 28/33 [00:59<00:09, 2.00s/it] |
|
2024-07-11 22:17:15 | ERROR | stderr |
Loading checkpoint shards: 88%|████████▊ | 29/33 [01:01<00:07, 1.98s/it] |
|
2024-07-11 22:17:17 | ERROR | stderr |
Loading checkpoint shards: 91%|█████████ | 30/33 [01:03<00:05, 2.00s/it] |
|
2024-07-11 22:17:19 | ERROR | stderr |
Loading checkpoint shards: 94%|█████████▍| 31/33 [01:05<00:03, 1.98s/it] |
|
2024-07-11 22:17:21 | ERROR | stderr |
Loading checkpoint shards: 97%|█████████▋| 32/33 [01:07<00:01, 1.88s/it] |
|
2024-07-11 22:17:22 | ERROR | stderr |
Loading checkpoint shards: 100%|██████████| 33/33 [01:08<00:00, 1.68s/it] |
|
2024-07-11 22:17:22 | ERROR | stderr |
Loading checkpoint shards: 100%|██████████| 33/33 [01:08<00:00, 2.08s/it] |
|
2024-07-11 22:17:22 | ERROR | stderr | |
|
2024-07-11 22:17:23 | INFO | model_worker | Register to controller |
|
2024-07-11 22:17:23 | ERROR | stderr | INFO: Started server process [106820] |
|
2024-07-11 22:17:23 | ERROR | stderr | INFO: Waiting for application startup. |
|
2024-07-11 22:17:23 | ERROR | stderr | INFO: Application startup complete. |
|
2024-07-11 22:17:23 | ERROR | stderr | INFO: Uvicorn running on http://0.0.0.0:40008 (Press CTRL+C to quit) |
|
2024-07-11 22:17:38 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:17:49 | INFO | stdout | INFO: 10.140.60.209:39404 - "POST /worker_get_status HTTP/1.1" 200 OK |
|
2024-07-11 22:17:53 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:18:08 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:18:08 | ERROR | model_worker | heart beat error: HTTPConnectionPool(host='10.140.60.209', port=10075): Max retries exceeded with url: /receive_heart_beat (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f552f1f8700>: Failed to establish a new connection: [Errno 111] Connection refused')) |
|
2024-07-11 22:18:13 | INFO | model_worker | Register to controller |
|
2024-07-11 22:18:17 | INFO | stdout | INFO: 10.140.60.209:39596 - "POST /worker_get_status HTTP/1.1" 200 OK |
|
2024-07-11 22:18:21 | INFO | stdout | INFO: 10.140.60.209:39722 - "POST /worker_get_status HTTP/1.1" 200 OK |
|
2024-07-11 22:18:22 | INFO | stdout | INFO: 10.140.60.209:39742 - "POST /worker_get_status HTTP/1.1" 200 OK |
|
2024-07-11 22:18:22 | INFO | stdout | INFO: 10.140.60.209:39762 - "POST /worker_get_status HTTP/1.1" 200 OK |
|
2024-07-11 22:18:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:18:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:18:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:19:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:19:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:19:32 | INFO | stdout | INFO: 10.140.60.209:40206 - "POST /worker_get_status HTTP/1.1" 200 OK |
|
2024-07-11 22:19:32 | INFO | stdout | INFO: 10.140.60.209:40222 - "POST /worker_get_status HTTP/1.1" 200 OK |
|
2024-07-11 22:19:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:19:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:19:59 | INFO | stdout | INFO: 10.140.60.209:40318 - "POST /worker_get_status HTTP/1.1" 200 OK |
|
2024-07-11 22:19:59 | INFO | stdout | INFO: 10.140.60.209:40336 - "POST /worker_get_status HTTP/1.1" 200 OK |
|
2024-07-11 22:20:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:20:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:20:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:20:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:21:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:21:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:21:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:21:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:22:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:22:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:22:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:22:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:23:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:23:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:23:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:23:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:24:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:24:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:24:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:24:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:25:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:25:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:25:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:25:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:26:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
2024-07-11 22:26:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-Pro']. Semaphore: None. global_counter: 0 |
|
|
