logs/model_worker_1abcda.log

2024-07-10 17:34:51 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=40004, worker_address='http://10.140.60.25:40004', controller_address='http://10.140.60.209:10075', model_path='share_internvl/InternVL2-8B/', model_name=None, device='cuda', limit_model_concurrency=5, stream_interval=1, load_8bit=False)
2024-07-10 17:34:51 | INFO | model_worker | Loading the model InternVL2-8B on worker 1abcda ...
2024-07-10 17:34:52 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-07-10 17:34:52 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-07-10 17:34:53 | ERROR | stderr | /mnt/petrelfs/wangweiyun/miniconda3/envs/internvl-apex/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:397: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `None` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
2024-07-10 17:34:53 | ERROR | stderr |   warnings.warn(
2024-07-10 17:34:53 | ERROR | stderr | 
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
2024-07-10 17:35:05 | ERROR | stderr | 
Loading checkpoint shards:  25%|██▌       | 1/4 [00:11<00:35, 11.69s/it]
2024-07-10 17:35:16 | ERROR | stderr | 
Loading checkpoint shards:  50%|█████     | 2/4 [00:22<00:22, 11.18s/it]
2024-07-10 17:35:26 | ERROR | stderr | 
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:33<00:10, 10.88s/it]
2024-07-10 17:35:29 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████| 4/4 [00:35<00:00,  7.71s/it]
2024-07-10 17:35:29 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████| 4/4 [00:35<00:00,  8.97s/it]
2024-07-10 17:35:29 | ERROR | stderr | 
2024-07-10 17:35:46 | INFO | model_worker | Register to controller
2024-07-10 17:35:46 | ERROR | stderr | INFO:     Started server process [4949]
2024-07-10 17:35:46 | ERROR | stderr | INFO:     Waiting for application startup.
2024-07-10 17:35:46 | ERROR | stderr | INFO:     Application startup complete.
2024-07-10 17:35:46 | ERROR | stderr | INFO:     Uvicorn running on http://0.0.0.0:40004 (Press CTRL+C to quit)
2024-07-10 17:36:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:36:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:36:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:36:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:37:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:37:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:37:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:37:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:38:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:38:15 | INFO | stdout | INFO:     10.140.60.209:33782 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:38:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:38:17 | INFO | stdout | INFO:     10.140.60.209:33852 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:38:26 | INFO | stdout | INFO:     10.140.60.209:33896 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:38:29 | INFO | stdout | INFO:     10.140.60.209:33912 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:38:30 | INFO | stdout | INFO:     10.140.60.209:33927 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:38:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:38:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:38:50 | INFO | stdout | INFO:     10.140.60.209:33950 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:39:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:39:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:39:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:39:42 | INFO | stdout | INFO:     10.140.60.209:34154 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:39:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:40:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:40:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:40:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:40:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:41:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:41:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:41:31 | INFO | stdout | INFO:     10.140.60.209:34568 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:41:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:41:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:42:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:42:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:42:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:42:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:43:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:43:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:43:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:43:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:44:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:44:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:44:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:44:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:45:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:45:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:45:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:45:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:46:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:46:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:46:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:46:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:47:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:47:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:47:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:47:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:48:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:48:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:48:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:48:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:49:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:49:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:49:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:49:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:50:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:50:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:50:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:50:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:51:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:51:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:51:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:51:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:52:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:52:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:52:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:52:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:53:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:53:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:53:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:53:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:54:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:54:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:54:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:54:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:55:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:55:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:55:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:55:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:56:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:56:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:56:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:56:33 | INFO | stdout | INFO:     10.140.60.209:38264 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:56:36 | INFO | stdout | INFO:     10.140.60.209:38286 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:56:37 | INFO | stdout | INFO:     10.140.60.209:38304 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:56:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:57:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:57:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:57:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:57:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:58:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:58:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:58:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:58:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:59:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:59:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:59:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 17:59:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:00:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:00:15 | INFO | stdout | INFO:     10.140.60.209:39098 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 18:00:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:00:18 | INFO | stdout | INFO:     10.140.60.209:39178 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 18:00:19 | INFO | stdout | INFO:     10.140.60.209:39196 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 18:00:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:00:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:01:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:01:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:01:30 | INFO | stdout | INFO:     10.140.60.209:39414 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 18:01:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:01:33 | INFO | stdout | INFO:     10.140.60.209:39428 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 18:01:36 | INFO | stdout | INFO:     10.140.60.209:39444 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 18:01:38 | INFO | stdout | INFO:     10.140.60.209:39460 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 18:01:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:02:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:02:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:02:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:02:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:03:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:03:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:03:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:03:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:04:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:04:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:04:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:04:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:05:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0
2024-07-10 18:05:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-8B']. Semaphore: None. global_counter: 0