InternVL

Running

App Files Files Community

InternVL / logs /model_worker_2370fb.log

czczup

Upload folder using huggingface_hub

3f1b7f0 verified 5 months ago

raw

history blame

8.96 kB

	2024-07-11 22:26:52 \| INFO \| model_worker \| args: Namespace(host='0.0.0.0', port=40007, worker_address='http://10.140.66.196:40007', controller_address='http://10.140.60.209:10075', model_path='share_internvl/InternVL2-78B/', model_name=None, device='auto', limit_model_concurrency=5, stream_interval=1, load_8bit=False)
	2024-07-11 22:26:52 \| INFO \| model_worker \| Loading the model InternVL2-78B on worker 2370fb ...
	2024-07-11 22:26:53 \| WARNING \| transformers.tokenization_utils_base \| Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
	2024-07-11 22:26:53 \| WARNING \| transformers.tokenization_utils_base \| Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
	2024-07-11 22:26:59 \| ERROR \| stderr \| Loading checkpoint shards: 0%\| \| 0/33 [00:00<?, ?it/s]
	2024-07-11 22:27:01 \| ERROR \| stderr \| Loading checkpoint shards: 3%\|▎ \| 1/33 [00:01<01:01, 1.93s/it]
	2024-07-11 22:27:03 \| ERROR \| stderr \| Loading checkpoint shards: 6%\|▌ \| 2/33 [00:03<00:58, 1.89s/it]
	2024-07-11 22:27:05 \| ERROR \| stderr \| Loading checkpoint shards: 9%\|▉ \| 3/33 [00:05<00:56, 1.87s/it]
	2024-07-11 22:27:06 \| ERROR \| stderr \| Loading checkpoint shards: 12%\|█▏ \| 4/33 [00:07<00:55, 1.90s/it]
	2024-07-11 22:27:08 \| ERROR \| stderr \| Loading checkpoint shards: 15%\|█▌ \| 5/33 [00:09<00:53, 1.90s/it]
	2024-07-11 22:27:10 \| ERROR \| stderr \| Loading checkpoint shards: 18%\|█▊ \| 6/33 [00:11<00:52, 1.93s/it]
	2024-07-11 22:27:12 \| ERROR \| stderr \| Loading checkpoint shards: 21%\|██ \| 7/33 [00:13<00:50, 1.93s/it]
	2024-07-11 22:27:14 \| ERROR \| stderr \| Loading checkpoint shards: 24%\|██▍ \| 8/33 [00:15<00:48, 1.92s/it]
	2024-07-11 22:27:16 \| ERROR \| stderr \| Loading checkpoint shards: 27%\|██▋ \| 9/33 [00:17<00:45, 1.91s/it]
	2024-07-11 22:27:18 \| ERROR \| stderr \| Loading checkpoint shards: 30%\|███ \| 10/33 [00:19<00:44, 1.92s/it]
	2024-07-11 22:27:20 \| ERROR \| stderr \| Loading checkpoint shards: 33%\|███▎ \| 11/33 [00:21<00:42, 1.92s/it]
	2024-07-11 22:27:22 \| ERROR \| stderr \| Loading checkpoint shards: 36%\|███▋ \| 12/33 [00:22<00:39, 1.90s/it]
	2024-07-11 22:27:24 \| ERROR \| stderr \| Loading checkpoint shards: 39%\|███▉ \| 13/33 [00:24<00:37, 1.89s/it]
	2024-07-11 22:27:26 \| ERROR \| stderr \| Loading checkpoint shards: 42%\|████▏ \| 14/33 [00:26<00:36, 1.90s/it]
	2024-07-11 22:27:27 \| ERROR \| stderr \| Loading checkpoint shards: 45%\|████▌ \| 15/33 [00:28<00:33, 1.88s/it]
	2024-07-11 22:27:29 \| ERROR \| stderr \| Loading checkpoint shards: 48%\|████▊ \| 16/33 [00:30<00:31, 1.88s/it]
	2024-07-11 22:27:31 \| ERROR \| stderr \| Loading checkpoint shards: 52%\|█████▏ \| 17/33 [00:32<00:30, 1.88s/it]
	2024-07-11 22:27:33 \| ERROR \| stderr \| Loading checkpoint shards: 55%\|█████▍ \| 18/33 [00:34<00:28, 1.91s/it]
	2024-07-11 22:27:35 \| ERROR \| stderr \| Loading checkpoint shards: 58%\|█████▊ \| 19/33 [00:36<00:26, 1.90s/it]
	2024-07-11 22:27:38 \| ERROR \| stderr \| Loading checkpoint shards: 61%\|██████ \| 20/33 [00:38<00:27, 2.10s/it]
	2024-07-11 22:27:40 \| ERROR \| stderr \| Loading checkpoint shards: 64%\|██████▎ \| 21/33 [00:40<00:24, 2.05s/it]
	2024-07-11 22:27:42 \| ERROR \| stderr \| Loading checkpoint shards: 67%\|██████▋ \| 22/33 [00:42<00:22, 2.04s/it]
	2024-07-11 22:27:44 \| ERROR \| stderr \| Loading checkpoint shards: 70%\|██████▉ \| 23/33 [00:44<00:20, 2.03s/it]
	2024-07-11 22:27:46 \| ERROR \| stderr \| Loading checkpoint shards: 73%\|███████▎ \| 24/33 [00:46<00:18, 2.07s/it]
	2024-07-11 22:27:49 \| ERROR \| stderr \| Loading checkpoint shards: 76%\|███████▌ \| 25/33 [00:49<00:18, 2.32s/it]
	2024-07-11 22:27:51 \| ERROR \| stderr \| Loading checkpoint shards: 79%\|███████▉ \| 26/33 [00:51<00:15, 2.21s/it]
	2024-07-11 22:27:53 \| ERROR \| stderr \| Loading checkpoint shards: 82%\|████████▏ \| 27/33 [00:54<00:13, 2.29s/it]
	2024-07-11 22:27:55 \| ERROR \| stderr \| Loading checkpoint shards: 85%\|████████▍ \| 28/33 [00:56<00:10, 2.18s/it]
	2024-07-11 22:27:57 \| ERROR \| stderr \| Loading checkpoint shards: 88%\|████████▊ \| 29/33 [00:57<00:08, 2.10s/it]
	2024-07-11 22:27:59 \| ERROR \| stderr \| Loading checkpoint shards: 91%\|█████████ \| 30/33 [01:00<00:06, 2.09s/it]
	2024-07-11 22:28:01 \| ERROR \| stderr \| Loading checkpoint shards: 94%\|█████████▍\| 31/33 [01:02<00:04, 2.07s/it]
	2024-07-11 22:28:03 \| ERROR \| stderr \| Loading checkpoint shards: 97%\|█████████▋\| 32/33 [01:03<00:01, 1.96s/it]
	2024-07-11 22:28:04 \| ERROR \| stderr \| Loading checkpoint shards: 100%\|██████████\| 33/33 [01:04<00:00, 1.73s/it]
	2024-07-11 22:28:04 \| ERROR \| stderr \| Loading checkpoint shards: 100%\|██████████\| 33/33 [01:04<00:00, 1.97s/it]
	2024-07-11 22:28:04 \| ERROR \| stderr \|
	2024-07-11 22:28:05 \| INFO \| model_worker \| Register to controller
	2024-07-11 22:28:05 \| ERROR \| stderr \| INFO: Started server process [28208]
	2024-07-11 22:28:05 \| ERROR \| stderr \| INFO: Waiting for application startup.
	2024-07-11 22:28:05 \| ERROR \| stderr \| INFO: Application startup complete.
	2024-07-11 22:28:05 \| ERROR \| stderr \| INFO: Uvicorn running on http://0.0.0.0:40007 (Press CTRL+C to quit)
	2024-07-11 22:28:07 \| INFO \| stdout \| INFO: 10.140.60.209:42234 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:10 \| INFO \| stdout \| INFO: 10.140.60.209:42262 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:12 \| INFO \| stdout \| INFO: 10.140.60.209:42306 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:14 \| INFO \| stdout \| INFO: 10.140.60.209:42350 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:14 \| INFO \| stdout \| INFO: 10.140.60.209:42375 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:20 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:28:22 \| INFO \| stdout \| INFO: 10.140.60.209:42566 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:25 \| INFO \| stdout \| INFO: 10.140.60.209:42586 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:26 \| INFO \| stdout \| INFO: 10.140.60.209:42610 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:26 \| INFO \| stdout \| INFO: 10.140.60.209:42630 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:35 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:28:36 \| INFO \| stdout \| INFO: 10.140.60.209:42690 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:37 \| INFO \| stdout \| INFO: 10.140.60.209:42714 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:38 \| INFO \| stdout \| INFO: 10.140.60.209:42738 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:50 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:29:05 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:29:20 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:29:35 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:29:50 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:30:05 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:30:20 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:30:35 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:30:50 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:31:05 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:31:20 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:31:35 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:31:50 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:32:05 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0

	2024-07-11 22:26:52 \| INFO \| model_worker \| args: Namespace(host='0.0.0.0', port=40007, worker_address='http://10.140.66.196:40007', controller_address='http://10.140.60.209:10075', model_path='share_internvl/InternVL2-78B/', model_name=None, device='auto', limit_model_concurrency=5, stream_interval=1, load_8bit=False)
	2024-07-11 22:26:52 \| INFO \| model_worker \| Loading the model InternVL2-78B on worker 2370fb ...
	2024-07-11 22:26:53 \| WARNING \| transformers.tokenization_utils_base \| Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
	2024-07-11 22:26:53 \| WARNING \| transformers.tokenization_utils_base \| Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
	2024-07-11 22:26:59 \| ERROR \| stderr \| Loading checkpoint shards: 0%\| \| 0/33 [00:00<?, ?it/s]
	2024-07-11 22:27:01 \| ERROR \| stderr \| Loading checkpoint shards: 3%\|▎ \| 1/33 [00:01<01:01, 1.93s/it]
	2024-07-11 22:27:03 \| ERROR \| stderr \| Loading checkpoint shards: 6%\|▌ \| 2/33 [00:03<00:58, 1.89s/it]
	2024-07-11 22:27:05 \| ERROR \| stderr \| Loading checkpoint shards: 9%\|▉ \| 3/33 [00:05<00:56, 1.87s/it]
	2024-07-11 22:27:06 \| ERROR \| stderr \| Loading checkpoint shards: 12%\|█▏ \| 4/33 [00:07<00:55, 1.90s/it]
	2024-07-11 22:27:08 \| ERROR \| stderr \| Loading checkpoint shards: 15%\|█▌ \| 5/33 [00:09<00:53, 1.90s/it]
	2024-07-11 22:27:10 \| ERROR \| stderr \| Loading checkpoint shards: 18%\|█▊ \| 6/33 [00:11<00:52, 1.93s/it]
	2024-07-11 22:27:12 \| ERROR \| stderr \| Loading checkpoint shards: 21%\|██ \| 7/33 [00:13<00:50, 1.93s/it]
	2024-07-11 22:27:14 \| ERROR \| stderr \| Loading checkpoint shards: 24%\|██▍ \| 8/33 [00:15<00:48, 1.92s/it]
	2024-07-11 22:27:16 \| ERROR \| stderr \| Loading checkpoint shards: 27%\|██▋ \| 9/33 [00:17<00:45, 1.91s/it]
	2024-07-11 22:27:18 \| ERROR \| stderr \| Loading checkpoint shards: 30%\|███ \| 10/33 [00:19<00:44, 1.92s/it]
	2024-07-11 22:27:20 \| ERROR \| stderr \| Loading checkpoint shards: 33%\|███▎ \| 11/33 [00:21<00:42, 1.92s/it]
	2024-07-11 22:27:22 \| ERROR \| stderr \| Loading checkpoint shards: 36%\|███▋ \| 12/33 [00:22<00:39, 1.90s/it]
	2024-07-11 22:27:24 \| ERROR \| stderr \| Loading checkpoint shards: 39%\|███▉ \| 13/33 [00:24<00:37, 1.89s/it]
	2024-07-11 22:27:26 \| ERROR \| stderr \| Loading checkpoint shards: 42%\|████▏ \| 14/33 [00:26<00:36, 1.90s/it]
	2024-07-11 22:27:27 \| ERROR \| stderr \| Loading checkpoint shards: 45%\|████▌ \| 15/33 [00:28<00:33, 1.88s/it]
	2024-07-11 22:27:29 \| ERROR \| stderr \| Loading checkpoint shards: 48%\|████▊ \| 16/33 [00:30<00:31, 1.88s/it]
	2024-07-11 22:27:31 \| ERROR \| stderr \| Loading checkpoint shards: 52%\|█████▏ \| 17/33 [00:32<00:30, 1.88s/it]
	2024-07-11 22:27:33 \| ERROR \| stderr \| Loading checkpoint shards: 55%\|█████▍ \| 18/33 [00:34<00:28, 1.91s/it]
	2024-07-11 22:27:35 \| ERROR \| stderr \| Loading checkpoint shards: 58%\|█████▊ \| 19/33 [00:36<00:26, 1.90s/it]
	2024-07-11 22:27:38 \| ERROR \| stderr \| Loading checkpoint shards: 61%\|██████ \| 20/33 [00:38<00:27, 2.10s/it]
	2024-07-11 22:27:40 \| ERROR \| stderr \| Loading checkpoint shards: 64%\|██████▎ \| 21/33 [00:40<00:24, 2.05s/it]
	2024-07-11 22:27:42 \| ERROR \| stderr \| Loading checkpoint shards: 67%\|██████▋ \| 22/33 [00:42<00:22, 2.04s/it]
	2024-07-11 22:27:44 \| ERROR \| stderr \| Loading checkpoint shards: 70%\|██████▉ \| 23/33 [00:44<00:20, 2.03s/it]
	2024-07-11 22:27:46 \| ERROR \| stderr \| Loading checkpoint shards: 73%\|███████▎ \| 24/33 [00:46<00:18, 2.07s/it]
	2024-07-11 22:27:49 \| ERROR \| stderr \| Loading checkpoint shards: 76%\|███████▌ \| 25/33 [00:49<00:18, 2.32s/it]
	2024-07-11 22:27:51 \| ERROR \| stderr \| Loading checkpoint shards: 79%\|███████▉ \| 26/33 [00:51<00:15, 2.21s/it]
	2024-07-11 22:27:53 \| ERROR \| stderr \| Loading checkpoint shards: 82%\|████████▏ \| 27/33 [00:54<00:13, 2.29s/it]
	2024-07-11 22:27:55 \| ERROR \| stderr \| Loading checkpoint shards: 85%\|████████▍ \| 28/33 [00:56<00:10, 2.18s/it]
	2024-07-11 22:27:57 \| ERROR \| stderr \| Loading checkpoint shards: 88%\|████████▊ \| 29/33 [00:57<00:08, 2.10s/it]
	2024-07-11 22:27:59 \| ERROR \| stderr \| Loading checkpoint shards: 91%\|█████████ \| 30/33 [01:00<00:06, 2.09s/it]
	2024-07-11 22:28:01 \| ERROR \| stderr \| Loading checkpoint shards: 94%\|█████████▍\| 31/33 [01:02<00:04, 2.07s/it]
	2024-07-11 22:28:03 \| ERROR \| stderr \| Loading checkpoint shards: 97%\|█████████▋\| 32/33 [01:03<00:01, 1.96s/it]
	2024-07-11 22:28:04 \| ERROR \| stderr \| Loading checkpoint shards: 100%\|██████████\| 33/33 [01:04<00:00, 1.73s/it]
	2024-07-11 22:28:04 \| ERROR \| stderr \| Loading checkpoint shards: 100%\|██████████\| 33/33 [01:04<00:00, 1.97s/it]
	2024-07-11 22:28:04 \| ERROR \| stderr \|
	2024-07-11 22:28:05 \| INFO \| model_worker \| Register to controller
	2024-07-11 22:28:05 \| ERROR \| stderr \| INFO: Started server process [28208]
	2024-07-11 22:28:05 \| ERROR \| stderr \| INFO: Waiting for application startup.
	2024-07-11 22:28:05 \| ERROR \| stderr \| INFO: Application startup complete.
	2024-07-11 22:28:05 \| ERROR \| stderr \| INFO: Uvicorn running on http://0.0.0.0:40007 (Press CTRL+C to quit)
	2024-07-11 22:28:07 \| INFO \| stdout \| INFO: 10.140.60.209:42234 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:10 \| INFO \| stdout \| INFO: 10.140.60.209:42262 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:12 \| INFO \| stdout \| INFO: 10.140.60.209:42306 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:14 \| INFO \| stdout \| INFO: 10.140.60.209:42350 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:14 \| INFO \| stdout \| INFO: 10.140.60.209:42375 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:20 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:28:22 \| INFO \| stdout \| INFO: 10.140.60.209:42566 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:25 \| INFO \| stdout \| INFO: 10.140.60.209:42586 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:26 \| INFO \| stdout \| INFO: 10.140.60.209:42610 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:26 \| INFO \| stdout \| INFO: 10.140.60.209:42630 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:35 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:28:36 \| INFO \| stdout \| INFO: 10.140.60.209:42690 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:37 \| INFO \| stdout \| INFO: 10.140.60.209:42714 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:38 \| INFO \| stdout \| INFO: 10.140.60.209:42738 - "POST /worker_get_status HTTP/1.1" 200 OK
	2024-07-11 22:28:50 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:29:05 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:29:20 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:29:35 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:29:50 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:30:05 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:30:20 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:30:35 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:30:50 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:31:05 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:31:20 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:31:35 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:31:50 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
	2024-07-11 22:32:05 \| INFO \| model_worker \| Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0