2024-07-04 06:38:20 | INFO | model_worker | args: Namespace(awq_ckpt=None, awq_groupsize=-1, awq_wbits=16, controller_address='http://127.0.0.1:21002', conv_template=None, cpu_offloading=False, debug=False, device='cuda', dtype=None, embed_in_truncate=False, enable_exllama=False, enable_xft=False, exllama_cache_8bit=False, exllama_gpu_split=None, exllama_max_seq_len=4096, gptq_act_order=False, gptq_ckpt=None, gptq_groupsize=-1, gptq_wbits=16, gpus=None, host='127.0.0.1', limit_worker_concurrency=5, load_8bit=False, max_gpu_memory=None, model_names=None, model_path='lmsys/vicuna-7b-v1.5', no_register=False, num_gpus=1, port=21003, revision='main', seed=None, ssl=False, stream_interval=2, worker_address='http://127.0.0.1:21003', xft_dtype=None, xft_max_seq_len=4096)
2024-07-04 06:38:20 | INFO | model_worker | Loading the model ['vicuna-7b-v1.5'] on worker be046cb3 ...
2024-07-04 06:38:21 | ERROR | stderr | /usr/local/lib/python3.8/dist-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2024-07-04 06:38:21 | ERROR | stderr |   warnings.warn(message, UserWarning)
2024-07-04 06:38:21 | ERROR | stderr | 
Loading checkpoint shards:   0%|                     | 0/2 [00:00<?, ?it/s]
2024-07-04 06:38:25 | ERROR | stderr | 
Loading checkpoint shards:  50%|██████▌      | 1/2 [00:04<00:04,  4.61s/it]
2024-07-04 06:38:27 | ERROR | stderr | 
Loading checkpoint shards: 100%|█████████████| 2/2 [00:06<00:00,  2.96s/it]
2024-07-04 06:38:27 | ERROR | stderr | 
Loading checkpoint shards: 100%|█████████████| 2/2 [00:06<00:00,  3.21s/it]
2024-07-04 06:38:27 | ERROR | stderr | 
2024-07-04 06:38:27 | ERROR | stderr | /usr/local/lib/python3.8/dist-packages/transformers/generation/configuration_utils.py:540: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
2024-07-04 06:38:27 | ERROR | stderr |   warnings.warn(
2024-07-04 06:38:27 | ERROR | stderr | /usr/local/lib/python3.8/dist-packages/transformers/generation/configuration_utils.py:545: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
2024-07-04 06:38:27 | ERROR | stderr |   warnings.warn(
2024-07-04 06:38:27 | ERROR | stderr | /usr/local/lib/python3.8/dist-packages/transformers/generation/configuration_utils.py:540: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
2024-07-04 06:38:27 | ERROR | stderr |   warnings.warn(
2024-07-04 06:38:27 | ERROR | stderr | /usr/local/lib/python3.8/dist-packages/transformers/generation/configuration_utils.py:545: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
2024-07-04 06:38:27 | ERROR | stderr |   warnings.warn(
2024-07-04 06:38:47 | INFO | model_worker | Register to controller
2024-07-04 06:38:48 | ERROR | stderr | INFO:     Started server process [24766]
2024-07-04 06:38:48 | ERROR | stderr | INFO:     Waiting for application startup.
2024-07-04 06:38:48 | ERROR | stderr | INFO:     Application startup complete.
2024-07-04 06:38:48 | ERROR | stderr | INFO:     Uvicorn running on http://127.0.0.1:21003 (Press CTRL+C to quit)
2024-07-04 06:39:15 | INFO | stdout | INFO:     127.0.0.1:35014 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-04 06:39:32 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: None. call_ct: 0. worker_id: be046cb3. 
2024-07-04 06:40:17 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: None. call_ct: 0. worker_id: be046cb3. 
2024-07-04 06:41:02 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: None. call_ct: 0. worker_id: be046cb3. 
2024-07-04 06:41:04 | INFO | stdout | INFO:     127.0.0.1:34306 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-04 06:41:47 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: None. call_ct: 0. worker_id: be046cb3. 
2024-07-04 06:41:51 | INFO | stdout | INFO:     127.0.0.1:34704 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-04 06:42:24 | INFO | stdout | INFO:     127.0.0.1:38580 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-04 06:42:32 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: None. call_ct: 0. worker_id: be046cb3. 
2024-07-04 06:43:05 | INFO | stdout | INFO:     127.0.0.1:39830 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-04 06:43:17 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: None. call_ct: 0. worker_id: be046cb3. 
2024-07-04 06:43:29 | INFO | stdout | INFO:     127.0.0.1:40290 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-04 06:44:03 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: None. call_ct: 0. worker_id: be046cb3. 
2024-07-04 06:44:48 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: None. call_ct: 0. worker_id: be046cb3. 
2024-07-04 06:45:33 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: None. call_ct: 0. worker_id: be046cb3. 
2024-07-04 06:45:48 | INFO | stdout | INFO:     127.0.0.1:39152 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-04 06:46:18 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: None. call_ct: 0. worker_id: be046cb3. 
2024-07-04 06:46:53 | INFO | stdout | INFO:     127.0.0.1:32816 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-04 06:47:03 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: None. call_ct: 0. worker_id: be046cb3. 
2024-07-04 06:47:16 | INFO | stdout | INFO:     127.0.0.1:42598 - "POST /worker_generate_stream HTTP/1.1" 200 OK
2024-07-04 06:47:48 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 1. worker_id: be046cb3. 
2024-07-04 06:48:33 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 1. worker_id: be046cb3. 
2024-07-04 06:49:18 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 1. worker_id: be046cb3. 
2024-07-04 06:50:03 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 1. worker_id: be046cb3. 
2024-07-04 06:50:48 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 1. worker_id: be046cb3. 
2024-07-04 06:51:33 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 1. worker_id: be046cb3. 
2024-07-04 06:52:18 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 1. worker_id: be046cb3. 
2024-07-04 06:52:57 | INFO | stdout | INFO:     127.0.0.1:35628 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-04 06:53:03 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 1. worker_id: be046cb3. 
2024-07-04 06:53:12 | INFO | stdout | INFO:     127.0.0.1:58546 - "POST /worker_generate_stream HTTP/1.1" 200 OK
2024-07-04 06:53:48 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 2. worker_id: be046cb3. 
2024-07-04 06:54:33 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 2. worker_id: be046cb3. 
2024-07-04 06:55:18 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 2. worker_id: be046cb3. 
2024-07-04 06:56:03 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 2. worker_id: be046cb3. 
2024-07-04 06:56:48 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 2. worker_id: be046cb3. 
2024-07-04 06:56:52 | INFO | stdout | INFO:     127.0.0.1:38170 - "POST /worker_generate_stream HTTP/1.1" 200 OK
2024-07-04 06:57:33 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 06:58:18 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 06:59:03 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 06:59:48 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:00:33 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:01:18 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:02:03 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:02:49 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:03:34 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:04:19 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:05:04 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:05:49 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:06:34 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:07:19 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:08:04 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:08:49 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:09:34 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:10:19 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:11:04 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:11:49 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:12:34 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:13:19 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:14:04 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:14:49 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:15:34 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:16:19 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:17:04 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:17:49 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:18:34 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:19:19 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:20:04 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:20:49 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:21:35 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:22:20 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:23:05 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:23:50 | INFO | model_worker | Send heart beat. Models: ['vicuna-7b-v1.5']. Semaphore: Semaphore(value=5, locked=False). call_ct: 3. worker_id: be046cb3. 
2024-07-04 07:24:08 | ERROR | stderr | INFO:     Shutting down
2024-07-04 07:24:08 | ERROR | stderr | INFO:     Waiting for application shutdown.
2024-07-04 07:24:08 | ERROR | stderr | INFO:     Application shutdown complete.
2024-07-04 07:24:08 | ERROR | stderr | INFO:     Finished server process [24766]
2024-07-04 07:24:08 | ERROR | stderr | Traceback (most recent call last):
2024-07-04 07:24:08 | ERROR | stderr |   File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2024-07-04 07:24:08 | ERROR | stderr |     return _run_code(code, main_globals, None,
2024-07-04 07:24:08 | ERROR | stderr |   File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
2024-07-04 07:24:08 | ERROR | stderr |     exec(code, run_globals)
2024-07-04 07:24:08 | ERROR | stderr |   File "/LLM_32T/evelyn/FastChat/fastchat/serve/model_worker.py", line 425, in <module>
2024-07-04 07:24:08 | ERROR | stderr |     uvicorn.run(app, host=args.host, port=args.port, log_level="info")
2024-07-04 07:24:08 | ERROR | stderr |   File "/usr/local/lib/python3.8/dist-packages/uvicorn/main.py", line 577, in run
2024-07-04 07:24:08 | ERROR | stderr |     server.run()
2024-07-04 07:24:08 | ERROR | stderr |   File "/usr/local/lib/python3.8/dist-packages/uvicorn/server.py", line 65, in run
2024-07-04 07:24:08 | ERROR | stderr |     return asyncio.run(self.serve(sockets=sockets))
2024-07-04 07:24:08 | ERROR | stderr |   File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
2024-07-04 07:24:08 | ERROR | stderr |     return loop.run_until_complete(main)
2024-07-04 07:24:08 | ERROR | stderr |   File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete
2024-07-04 07:24:08 | ERROR | stderr |   File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete
2024-07-04 07:24:08 | ERROR | stderr |   File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever
2024-07-04 07:24:08 | ERROR | stderr |   File "uvloop/loop.pyx", line 555, in uvloop.loop.Loop._run
2024-07-04 07:24:08 | ERROR | stderr |   File "uvloop/loop.pyx", line 474, in uvloop.loop.Loop._on_idle
2024-07-04 07:24:08 | ERROR | stderr |   File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run
2024-07-04 07:24:08 | ERROR | stderr |   File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
2024-07-04 07:24:08 | ERROR | stderr |   File "/usr/local/lib/python3.8/dist-packages/uvicorn/server.py", line 69, in serve
2024-07-04 07:24:08 | ERROR | stderr |     await self._serve(sockets)
2024-07-04 07:24:08 | ERROR | stderr |   File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
2024-07-04 07:24:08 | ERROR | stderr |     next(self.gen)
2024-07-04 07:24:08 | ERROR | stderr |   File "/usr/local/lib/python3.8/dist-packages/uvicorn/server.py", line 328, in capture_signals
2024-07-04 07:24:08 | ERROR | stderr |     signal.raise_signal(captured_signal)
2024-07-04 07:24:08 | ERROR | stderr | KeyboardInterrupt