[2023-06-20 23:12:18,875][33484] Saving configuration to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json... [2023-06-20 23:12:18,897][33484] Rollout worker 0 uses device cpu [2023-06-20 23:12:18,898][33484] Rollout worker 1 uses device cpu [2023-06-20 23:12:18,898][33484] Rollout worker 2 uses device cpu [2023-06-20 23:12:18,899][33484] Rollout worker 3 uses device cpu [2023-06-20 23:12:18,899][33484] Rollout worker 4 uses device cpu [2023-06-20 23:12:18,899][33484] Rollout worker 5 uses device cpu [2023-06-20 23:12:18,900][33484] Rollout worker 6 uses device cpu [2023-06-20 23:12:18,900][33484] Rollout worker 7 uses device cpu [2023-06-20 23:13:38,093][33484] Environment doom_basic already registered, overwriting... [2023-06-20 23:13:38,095][33484] Environment doom_two_colors_easy already registered, overwriting... [2023-06-20 23:13:38,096][33484] Environment doom_two_colors_hard already registered, overwriting... [2023-06-20 23:13:38,097][33484] Environment doom_dm already registered, overwriting... [2023-06-20 23:13:38,098][33484] Environment doom_dwango5 already registered, overwriting... [2023-06-20 23:13:38,098][33484] Environment doom_my_way_home_flat_actions already registered, overwriting... [2023-06-20 23:13:38,099][33484] Environment doom_defend_the_center_flat_actions already registered, overwriting... [2023-06-20 23:13:38,100][33484] Environment doom_my_way_home already registered, overwriting... [2023-06-20 23:13:38,100][33484] Environment doom_deadly_corridor already registered, overwriting... [2023-06-20 23:13:38,101][33484] Environment doom_defend_the_center already registered, overwriting... [2023-06-20 23:13:38,102][33484] Environment doom_defend_the_line already registered, overwriting... [2023-06-20 23:13:38,103][33484] Environment doom_health_gathering already registered, overwriting... [2023-06-20 23:13:38,103][33484] Environment doom_health_gathering_supreme already registered, overwriting... [2023-06-20 23:13:38,104][33484] Environment doom_battle already registered, overwriting... [2023-06-20 23:13:38,104][33484] Environment doom_battle2 already registered, overwriting... [2023-06-20 23:13:38,105][33484] Environment doom_duel_bots already registered, overwriting... [2023-06-20 23:13:38,108][33484] Environment doom_deathmatch_bots already registered, overwriting... [2023-06-20 23:13:38,109][33484] Environment doom_duel already registered, overwriting... [2023-06-20 23:13:38,109][33484] Environment doom_deathmatch_full already registered, overwriting... [2023-06-20 23:13:38,110][33484] Environment doom_benchmark already registered, overwriting... [2023-06-20 23:13:38,110][33484] register_encoder_factory: [2023-06-20 23:13:38,146][33484] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-20 23:13:38,151][33484] Experiment dir /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment already exists! [2023-06-20 23:13:38,152][33484] Resuming existing experiment from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment... [2023-06-20 23:13:38,152][33484] Weights and Biases integration disabled [2023-06-20 23:13:38,155][33484] Environment var CUDA_VISIBLE_DEVICES is [2023-06-20 23:13:38,890][33484] Starting experiment with the following configuration: help=False algo=APPO env=doom_health_gathering_supreme experiment=default_experiment train_dir=/Users/md/Code/python/jubilant-memory/RL/train_dir restart_behavior=resume device=gpu seed=None num_policies=1 async_rl=True serial_mode=False batched_sampling=False num_batches_to_accumulate=2 worker_num_splits=2 policy_workers_per_policy=1 max_policy_lag=1000 num_workers=8 num_envs_per_worker=4 batch_size=1024 num_batches_per_epoch=1 num_epochs=1 rollout=32 recurrence=32 shuffle_minibatches=False gamma=0.99 reward_scale=1.0 reward_clip=1000.0 value_bootstrap=False normalize_returns=True exploration_loss_coeff=0.001 value_loss_coeff=0.5 kl_loss_coeff=0.0 exploration_loss=symmetric_kl gae_lambda=0.95 ppo_clip_ratio=0.1 ppo_clip_value=0.2 with_vtrace=False vtrace_rho=1.0 vtrace_c=1.0 optimizer=adam adam_eps=1e-06 adam_beta1=0.9 adam_beta2=0.999 max_grad_norm=4.0 learning_rate=0.0001 lr_schedule=constant lr_schedule_kl_threshold=0.008 lr_adaptive_min=1e-06 lr_adaptive_max=0.01 obs_subtract_mean=0.0 obs_scale=255.0 normalize_input=True normalize_input_keys=None decorrelate_experience_max_seconds=0 decorrelate_envs_on_one_worker=True actor_worker_gpus=[] set_workers_cpu_affinity=True force_envs_single_thread=False default_niceness=0 log_to_file=True experiment_summaries_interval=10 flush_summaries_interval=30 stats_avg=100 summaries_use_frameskip=True heartbeat_interval=20 heartbeat_reporting_interval=600 train_for_env_steps=4000000 train_for_seconds=10000000000 save_every_sec=120 keep_checkpoints=2 load_checkpoint_kind=latest save_milestones_sec=-1 save_best_every_sec=5 save_best_metric=reward save_best_after=100000 benchmark=False encoder_mlp_layers=[512, 512] encoder_conv_architecture=convnet_simple encoder_conv_mlp_layers=[512] use_rnn=True rnn_size=512 rnn_type=gru rnn_num_layers=1 decoder_mlp_layers=[] nonlinearity=elu policy_initialization=orthogonal policy_init_gain=1.0 actor_critic_share_weights=True adaptive_stddev=True continuous_tanh_scale=0.0 initial_stddev=1.0 use_env_info_cache=False env_gpu_actions=False env_gpu_observations=True env_frameskip=4 env_framestack=1 pixel_format=CHW use_record_episode_statistics=False with_wandb=False wandb_user=None wandb_project=sample_factory wandb_group=None wandb_job_type=SF wandb_tags=[] with_pbt=False pbt_mix_policies_in_one_env=True pbt_period_env_steps=5000000 pbt_start_mutation=20000000 pbt_replace_fraction=0.3 pbt_mutation_rate=0.15 pbt_replace_reward_gap=0.1 pbt_replace_reward_gap_absolute=1e-06 pbt_optimize_gamma=False pbt_target_objective=true_objective pbt_perturb_min=1.1 pbt_perturb_max=1.5 num_agents=-1 num_humans=0 num_bots=-1 start_bot_difficulty=None timelimit=None res_w=128 res_h=72 wide_aspect_ratio=False eval_env_frameskip=1 fps=35 command_line=--env=doom_health_gathering_supreme --num_workers=8 --num_envs_per_worker=4 --train_for_env_steps=4000000 cli_args={'env': 'doom_health_gathering_supreme', 'num_workers': 8, 'num_envs_per_worker': 4, 'train_for_env_steps': 4000000} git_hash=02ef46648112e6fd3adc4475dfd889e784c0ef87 git_repo_name=https://github.com/mihirdeo16/jubilant-memory.git [2023-06-20 23:13:38,891][33484] Saving configuration to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json... [2023-06-20 23:13:38,904][33484] Rollout worker 0 uses device cpu [2023-06-20 23:13:38,905][33484] Rollout worker 1 uses device cpu [2023-06-20 23:13:38,906][33484] Rollout worker 2 uses device cpu [2023-06-20 23:13:38,906][33484] Rollout worker 3 uses device cpu [2023-06-20 23:13:38,906][33484] Rollout worker 4 uses device cpu [2023-06-20 23:13:38,906][33484] Rollout worker 5 uses device cpu [2023-06-20 23:13:38,907][33484] Rollout worker 6 uses device cpu [2023-06-20 23:13:38,907][33484] Rollout worker 7 uses device cpu [2023-06-20 23:17:03,197][33484] Environment doom_basic already registered, overwriting... [2023-06-20 23:17:03,199][33484] Environment doom_two_colors_easy already registered, overwriting... [2023-06-20 23:17:03,200][33484] Environment doom_two_colors_hard already registered, overwriting... [2023-06-20 23:17:03,201][33484] Environment doom_dm already registered, overwriting... [2023-06-20 23:17:03,202][33484] Environment doom_dwango5 already registered, overwriting... [2023-06-20 23:17:03,203][33484] Environment doom_my_way_home_flat_actions already registered, overwriting... [2023-06-20 23:17:03,203][33484] Environment doom_defend_the_center_flat_actions already registered, overwriting... [2023-06-20 23:17:03,204][33484] Environment doom_my_way_home already registered, overwriting... [2023-06-20 23:17:03,205][33484] Environment doom_deadly_corridor already registered, overwriting... [2023-06-20 23:17:03,206][33484] Environment doom_defend_the_center already registered, overwriting... [2023-06-20 23:17:03,206][33484] Environment doom_defend_the_line already registered, overwriting... [2023-06-20 23:17:03,207][33484] Environment doom_health_gathering already registered, overwriting... [2023-06-20 23:17:03,208][33484] Environment doom_health_gathering_supreme already registered, overwriting... [2023-06-20 23:17:03,208][33484] Environment doom_battle already registered, overwriting... [2023-06-20 23:17:03,209][33484] Environment doom_battle2 already registered, overwriting... [2023-06-20 23:17:03,210][33484] Environment doom_duel_bots already registered, overwriting... [2023-06-20 23:17:03,210][33484] Environment doom_deathmatch_bots already registered, overwriting... [2023-06-20 23:17:03,210][33484] Environment doom_duel already registered, overwriting... [2023-06-20 23:17:03,211][33484] Environment doom_deathmatch_full already registered, overwriting... [2023-06-20 23:17:03,211][33484] Environment doom_benchmark already registered, overwriting... [2023-06-20 23:17:03,212][33484] register_encoder_factory: [2023-06-20 23:17:03,251][33484] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-20 23:17:03,257][33484] Experiment dir /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment already exists! [2023-06-20 23:17:03,259][33484] Resuming existing experiment from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment... [2023-06-20 23:17:03,260][33484] Weights and Biases integration disabled [2023-06-20 23:17:03,264][33484] Environment var CUDA_VISIBLE_DEVICES is [2023-06-20 23:17:04,047][33484] Starting experiment with the following configuration: help=False algo=APPO env=doom_health_gathering_supreme experiment=default_experiment train_dir=/Users/md/Code/python/jubilant-memory/RL/train_dir restart_behavior=resume device=cpu seed=None num_policies=1 async_rl=True serial_mode=False batched_sampling=False num_batches_to_accumulate=2 worker_num_splits=2 policy_workers_per_policy=1 max_policy_lag=1000 num_workers=8 num_envs_per_worker=4 batch_size=1024 num_batches_per_epoch=1 num_epochs=1 rollout=32 recurrence=32 shuffle_minibatches=False gamma=0.99 reward_scale=1.0 reward_clip=1000.0 value_bootstrap=False normalize_returns=True exploration_loss_coeff=0.001 value_loss_coeff=0.5 kl_loss_coeff=0.0 exploration_loss=symmetric_kl gae_lambda=0.95 ppo_clip_ratio=0.1 ppo_clip_value=0.2 with_vtrace=False vtrace_rho=1.0 vtrace_c=1.0 optimizer=adam adam_eps=1e-06 adam_beta1=0.9 adam_beta2=0.999 max_grad_norm=4.0 learning_rate=0.0001 lr_schedule=constant lr_schedule_kl_threshold=0.008 lr_adaptive_min=1e-06 lr_adaptive_max=0.01 obs_subtract_mean=0.0 obs_scale=255.0 normalize_input=True normalize_input_keys=None decorrelate_experience_max_seconds=0 decorrelate_envs_on_one_worker=True actor_worker_gpus=[] set_workers_cpu_affinity=True force_envs_single_thread=False default_niceness=0 log_to_file=True experiment_summaries_interval=10 flush_summaries_interval=30 stats_avg=100 summaries_use_frameskip=True heartbeat_interval=20 heartbeat_reporting_interval=600 train_for_env_steps=4000000 train_for_seconds=10000000000 save_every_sec=120 keep_checkpoints=2 load_checkpoint_kind=latest save_milestones_sec=-1 save_best_every_sec=5 save_best_metric=reward save_best_after=100000 benchmark=False encoder_mlp_layers=[512, 512] encoder_conv_architecture=convnet_simple encoder_conv_mlp_layers=[512] use_rnn=True rnn_size=512 rnn_type=gru rnn_num_layers=1 decoder_mlp_layers=[] nonlinearity=elu policy_initialization=orthogonal policy_init_gain=1.0 actor_critic_share_weights=True adaptive_stddev=True continuous_tanh_scale=0.0 initial_stddev=1.0 use_env_info_cache=False env_gpu_actions=False env_gpu_observations=True env_frameskip=4 env_framestack=1 pixel_format=CHW use_record_episode_statistics=False with_wandb=False wandb_user=None wandb_project=sample_factory wandb_group=None wandb_job_type=SF wandb_tags=[] with_pbt=False pbt_mix_policies_in_one_env=True pbt_period_env_steps=5000000 pbt_start_mutation=20000000 pbt_replace_fraction=0.3 pbt_mutation_rate=0.15 pbt_replace_reward_gap=0.1 pbt_replace_reward_gap_absolute=1e-06 pbt_optimize_gamma=False pbt_target_objective=true_objective pbt_perturb_min=1.1 pbt_perturb_max=1.5 num_agents=-1 num_humans=0 num_bots=-1 start_bot_difficulty=None timelimit=None res_w=128 res_h=72 wide_aspect_ratio=False eval_env_frameskip=1 fps=35 command_line=--env=doom_health_gathering_supreme --num_workers=8 --num_envs_per_worker=4 --train_for_env_steps=4000000 cli_args={'env': 'doom_health_gathering_supreme', 'num_workers': 8, 'num_envs_per_worker': 4, 'train_for_env_steps': 4000000} git_hash=02ef46648112e6fd3adc4475dfd889e784c0ef87 git_repo_name=https://github.com/mihirdeo16/jubilant-memory.git [2023-06-20 23:17:04,048][33484] Saving configuration to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json... [2023-06-20 23:17:04,060][33484] Rollout worker 0 uses device cpu [2023-06-20 23:17:04,061][33484] Rollout worker 1 uses device cpu [2023-06-20 23:17:04,062][33484] Rollout worker 2 uses device cpu [2023-06-20 23:17:04,063][33484] Rollout worker 3 uses device cpu [2023-06-20 23:17:04,063][33484] Rollout worker 4 uses device cpu [2023-06-20 23:17:04,063][33484] Rollout worker 5 uses device cpu [2023-06-20 23:17:04,064][33484] Rollout worker 6 uses device cpu [2023-06-20 23:17:04,064][33484] Rollout worker 7 uses device cpu [2023-06-20 23:17:04,103][33484] InferenceWorker_p0-w0: min num requests: 2 [2023-06-20 23:17:04,139][33484] Starting all processes... [2023-06-20 23:17:04,139][33484] Starting process learner_proc0 [2023-06-20 23:17:04,193][33484] Starting all processes... [2023-06-20 23:17:04,196][33484] Starting process inference_proc0-0 [2023-06-20 23:17:04,197][33484] Starting process rollout_proc0 [2023-06-20 23:17:04,197][33484] Starting process rollout_proc1 [2023-06-20 23:17:04,197][33484] Starting process rollout_proc2 [2023-06-20 23:17:04,197][33484] Starting process rollout_proc3 [2023-06-20 23:17:04,197][33484] Starting process rollout_proc4 [2023-06-20 23:17:04,197][33484] Starting process rollout_proc5 [2023-06-20 23:17:04,197][33484] Starting process rollout_proc6 [2023-06-20 23:17:04,197][33484] Starting process rollout_proc7 [2023-06-20 23:17:06,218][33882] On MacOS, not setting affinity [2023-06-20 23:17:06,227][33878] Starting seed is not provided [2023-06-20 23:17:06,228][33878] Initializing actor-critic model on device cpu [2023-06-20 23:17:06,228][33878] RunningMeanStd input shape: (3, 72, 128) [2023-06-20 23:17:06,229][33878] RunningMeanStd input shape: (1,) [2023-06-20 23:17:06,245][33878] ConvEncoder: input_channels=3 [2023-06-20 23:17:06,257][33883] On MacOS, not setting affinity [2023-06-20 23:17:06,257][33884] On MacOS, not setting affinity [2023-06-20 23:17:06,261][33888] On MacOS, not setting affinity [2023-06-20 23:17:06,276][33885] On MacOS, not setting affinity [2023-06-20 23:17:06,276][33880] On MacOS, not setting affinity [2023-06-20 23:17:06,283][33881] On MacOS, not setting affinity [2023-06-20 23:17:06,310][33887] On MacOS, not setting affinity [2023-06-20 23:17:06,329][33878] Conv encoder output size: 512 [2023-06-20 23:17:06,329][33878] Policy head output size: 512 [2023-06-20 23:17:06,345][33878] Created Actor Critic model with architecture: [2023-06-20 23:17:06,345][33878] ActorCriticSharedWeights( (obs_normalizer): ObservationNormalizer( (running_mean_std): RunningMeanStdDictInPlace( (running_mean_std): ModuleDict( (obs): RunningMeanStdInPlace() ) ) ) (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace) (encoder): VizdoomEncoder( (basic_encoder): ConvEncoder( (enc): RecursiveScriptModule( original_name=ConvEncoderImpl (conv_head): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Conv2d) (1): RecursiveScriptModule(original_name=ELU) (2): RecursiveScriptModule(original_name=Conv2d) (3): RecursiveScriptModule(original_name=ELU) (4): RecursiveScriptModule(original_name=Conv2d) (5): RecursiveScriptModule(original_name=ELU) ) (mlp_layers): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Linear) (1): RecursiveScriptModule(original_name=ELU) ) ) ) ) (core): ModelCoreRNN( (core): GRU(512, 512) ) (decoder): MlpDecoder( (mlp): Identity() ) (critic_linear): Linear(in_features=512, out_features=1, bias=True) (action_parameterization): ActionParameterizationDefault( (distribution_linear): Linear(in_features=512, out_features=5, bias=True) ) ) [2023-06-20 23:17:06,348][33878] Using optimizer [2023-06-20 23:17:06,348][33878] No checkpoints found [2023-06-20 23:17:06,348][33878] Did not load from checkpoint, starting from scratch! [2023-06-20 23:17:06,349][33878] Initialized policy 0 weights for model version 0 [2023-06-20 23:17:06,350][33878] LearnerWorker_p0 finished initialization! [2023-06-20 23:17:06,351][33879] RunningMeanStd input shape: (3, 72, 128) [2023-06-20 23:17:06,351][33879] RunningMeanStd input shape: (1,) [2023-06-20 23:17:06,358][33879] ConvEncoder: input_channels=3 [2023-06-20 23:17:06,423][33879] Conv encoder output size: 512 [2023-06-20 23:17:06,424][33879] Policy head output size: 512 [2023-06-20 23:17:06,431][33484] Inference worker 0-0 is ready! [2023-06-20 23:17:06,433][33484] All inference workers are ready! Signal rollout workers to start! [2023-06-20 23:17:06,468][33883] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-20 23:17:06,470][33888] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-20 23:17:06,471][33880] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-20 23:17:06,475][33885] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-20 23:17:06,475][33884] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-20 23:17:06,476][33887] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-20 23:17:06,479][33882] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-20 23:17:06,481][33881] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-20 23:17:08,268][33484] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 0. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-20 23:17:08,618][33881] Decorrelating experience for 0 frames... [2023-06-20 23:17:08,618][33882] Decorrelating experience for 0 frames... [2023-06-20 23:17:08,627][33885] Decorrelating experience for 0 frames... [2023-06-20 23:17:08,632][33880] Decorrelating experience for 0 frames... [2023-06-20 23:17:08,634][33888] Decorrelating experience for 0 frames... [2023-06-20 23:17:09,679][33882] Decorrelating experience for 32 frames... [2023-06-20 23:17:09,679][33880] Decorrelating experience for 32 frames... [2023-06-20 23:17:09,679][33885] Decorrelating experience for 32 frames... [2023-06-20 23:17:09,679][33881] Decorrelating experience for 32 frames... [2023-06-20 23:17:09,699][33887] Decorrelating experience for 0 frames... [2023-06-20 23:17:09,700][33884] Decorrelating experience for 0 frames... [2023-06-20 23:17:10,538][33884] Decorrelating experience for 32 frames... [2023-06-20 23:17:10,538][33887] Decorrelating experience for 32 frames... [2023-06-20 23:17:10,541][33888] Decorrelating experience for 32 frames... [2023-06-20 23:17:11,454][33880] Decorrelating experience for 64 frames... [2023-06-20 23:17:11,455][33881] Decorrelating experience for 64 frames... [2023-06-20 23:17:11,455][33885] Decorrelating experience for 64 frames... [2023-06-20 23:17:11,455][33882] Decorrelating experience for 64 frames... [2023-06-20 23:17:12,306][33883] Decorrelating experience for 0 frames... [2023-06-20 23:17:12,309][33884] Decorrelating experience for 64 frames... [2023-06-20 23:17:12,309][33887] Decorrelating experience for 64 frames... [2023-06-20 23:17:13,079][33883] Decorrelating experience for 32 frames... [2023-06-20 23:17:13,079][33888] Decorrelating experience for 64 frames... [2023-06-20 23:17:13,269][33484] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-20 23:17:13,916][33880] Decorrelating experience for 96 frames... [2023-06-20 23:17:13,917][33885] Decorrelating experience for 96 frames... [2023-06-20 23:17:13,922][33882] Decorrelating experience for 96 frames... [2023-06-20 23:17:14,809][33887] Decorrelating experience for 96 frames... [2023-06-20 23:17:14,815][33883] Decorrelating experience for 64 frames... [2023-06-20 23:17:14,816][33884] Decorrelating experience for 96 frames... [2023-06-20 23:17:14,836][33881] Decorrelating experience for 96 frames... [2023-06-20 23:17:15,458][33888] Decorrelating experience for 96 frames... [2023-06-20 23:17:17,109][33883] Decorrelating experience for 96 frames... [2023-06-20 23:17:18,268][33484] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-20 23:17:18,269][33484] Avg episode reward: [(0, '0.320')] [2023-06-20 23:17:23,269][33484] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 89.6. Samples: 1344. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-20 23:17:23,271][33484] Avg episode reward: [(0, '1.377')] [2023-06-20 23:17:24,087][33484] Heartbeat connected on Batcher_0 [2023-06-20 23:17:24,121][33484] Heartbeat connected on InferenceWorker_p0-w0 [2023-06-20 23:17:24,171][33484] Heartbeat connected on RolloutWorker_w2 [2023-06-20 23:17:24,181][33484] Heartbeat connected on RolloutWorker_w0 [2023-06-20 23:17:24,198][33484] Heartbeat connected on RolloutWorker_w4 [2023-06-20 23:17:24,199][33484] Heartbeat connected on RolloutWorker_w1 [2023-06-20 23:17:24,203][33484] Heartbeat connected on RolloutWorker_w5 [2023-06-20 23:17:24,213][33484] Heartbeat connected on RolloutWorker_w3 [2023-06-20 23:17:24,222][33484] Heartbeat connected on RolloutWorker_w6 [2023-06-20 23:17:24,267][33484] Heartbeat connected on RolloutWorker_w7 [2023-06-20 23:17:25,027][33484] Heartbeat connected on LearnerWorker_p0 [2023-06-20 23:17:28,268][33484] Fps is (10 sec: 819.2, 60 sec: 409.6, 300 sec: 409.6). Total num frames: 8192. Throughput: 0: 162.3. Samples: 3246. Policy #0 lag: (min: 0.0, avg: 0.0, max: 0.0) [2023-06-20 23:17:28,270][33484] Avg episode reward: [(0, '2.748')] [2023-06-20 23:17:33,268][33484] Fps is (10 sec: 1228.9, 60 sec: 491.5, 300 sec: 491.5). Total num frames: 12288. Throughput: 0: 167.7. Samples: 4192. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-20 23:17:33,271][33484] Avg episode reward: [(0, '3.151')] [2023-06-20 23:17:38,269][33484] Fps is (10 sec: 1228.7, 60 sec: 682.6, 300 sec: 682.6). Total num frames: 20480. Throughput: 0: 203.3. Samples: 6098. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:17:38,270][33484] Avg episode reward: [(0, '3.580')] [2023-06-20 23:17:43,269][33484] Fps is (10 sec: 1228.8, 60 sec: 702.2, 300 sec: 702.2). Total num frames: 24576. Throughput: 0: 228.2. Samples: 7986. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:17:43,271][33484] Avg episode reward: [(0, '3.854')] [2023-06-20 23:17:48,269][33484] Fps is (10 sec: 1228.8, 60 sec: 819.2, 300 sec: 819.2). Total num frames: 32768. Throughput: 0: 223.5. Samples: 8940. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:17:48,271][33484] Avg episode reward: [(0, '3.929')] [2023-06-20 23:17:53,268][33484] Fps is (10 sec: 1228.8, 60 sec: 819.2, 300 sec: 819.2). Total num frames: 36864. Throughput: 0: 239.8. Samples: 10790. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-20 23:17:53,271][33484] Avg episode reward: [(0, '4.252')] [2023-06-20 23:17:53,993][33879] Updated weights for policy 0, policy_version 10 (0.0018) [2023-06-20 23:17:58,268][33484] Fps is (10 sec: 1228.8, 60 sec: 901.1, 300 sec: 901.1). Total num frames: 45056. Throughput: 0: 281.9. Samples: 12684. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-20 23:17:58,269][33484] Avg episode reward: [(0, '4.445')] [2023-06-20 23:18:03,269][33484] Fps is (10 sec: 1228.7, 60 sec: 893.6, 300 sec: 893.6). Total num frames: 49152. Throughput: 0: 302.9. Samples: 13630. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-20 23:18:03,272][33484] Avg episode reward: [(0, '4.487')] [2023-06-20 23:18:08,269][33484] Fps is (10 sec: 1228.7, 60 sec: 955.7, 300 sec: 955.7). Total num frames: 57344. Throughput: 0: 314.9. Samples: 15514. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:18:08,270][33484] Avg episode reward: [(0, '4.483')] [2023-06-20 23:18:13,270][33484] Fps is (10 sec: 1228.7, 60 sec: 1024.0, 300 sec: 945.2). Total num frames: 61440. Throughput: 0: 314.6. Samples: 17404. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:18:13,275][33484] Avg episode reward: [(0, '4.386')] [2023-06-20 23:18:18,265][33484] Fps is (10 sec: 1229.3, 60 sec: 1160.6, 300 sec: 994.8). Total num frames: 69632. Throughput: 0: 314.5. Samples: 18344. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:18:18,266][33484] Avg episode reward: [(0, '4.367')] [2023-06-20 23:18:23,270][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 983.0). Total num frames: 73728. Throughput: 0: 314.3. Samples: 20242. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:18:23,273][33484] Avg episode reward: [(0, '4.376')] [2023-06-20 23:18:26,473][33879] Updated weights for policy 0, policy_version 20 (0.0012) [2023-06-20 23:18:28,267][33484] Fps is (10 sec: 1228.5, 60 sec: 1228.8, 300 sec: 1024.0). Total num frames: 81920. Throughput: 0: 314.6. Samples: 22142. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:18:28,268][33484] Avg episode reward: [(0, '4.513')] [2023-06-20 23:18:33,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1011.9). Total num frames: 86016. Throughput: 0: 314.1. Samples: 23074. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:18:33,273][33484] Avg episode reward: [(0, '4.401')] [2023-06-20 23:18:38,269][33484] Fps is (10 sec: 1228.6, 60 sec: 1228.8, 300 sec: 1046.7). Total num frames: 94208. Throughput: 0: 314.8. Samples: 24956. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:18:38,269][33484] Avg episode reward: [(0, '4.401')] [2023-06-20 23:18:43,269][33484] Fps is (10 sec: 1638.4, 60 sec: 1297.1, 300 sec: 1077.9). Total num frames: 102400. Throughput: 0: 314.8. Samples: 26852. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:18:43,271][33484] Avg episode reward: [(0, '4.443')] [2023-06-20 23:18:43,273][33878] Saving new best policy, reward=4.443! [2023-06-20 23:18:48,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1065.0). Total num frames: 106496. Throughput: 0: 314.7. Samples: 27792. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:18:48,269][33484] Avg episode reward: [(0, '4.453')] [2023-06-20 23:18:49,238][33878] Saving new best policy, reward=4.453! [2023-06-20 23:18:53,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1092.3). Total num frames: 114688. Throughput: 0: 315.3. Samples: 29704. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:18:53,270][33484] Avg episode reward: [(0, '4.491')] [2023-06-20 23:18:53,273][33878] Saving new best policy, reward=4.491! [2023-06-20 23:18:58,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1079.8). Total num frames: 118784. Throughput: 0: 315.4. Samples: 31596. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:18:58,271][33484] Avg episode reward: [(0, '4.466')] [2023-06-20 23:18:58,923][33879] Updated weights for policy 0, policy_version 30 (0.0009) [2023-06-20 23:19:03,268][33484] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1104.1). Total num frames: 126976. Throughput: 0: 315.7. Samples: 32550. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:19:03,269][33484] Avg episode reward: [(0, '4.545')] [2023-06-20 23:19:03,272][33878] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000031_126976.pth... [2023-06-20 23:19:03,345][33878] Saving new best policy, reward=4.545! [2023-06-20 23:19:08,267][33484] Fps is (10 sec: 1229.0, 60 sec: 1228.8, 300 sec: 1092.3). Total num frames: 131072. Throughput: 0: 315.5. Samples: 34438. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:19:08,269][33484] Avg episode reward: [(0, '4.547')] [2023-06-20 23:19:08,626][33878] Saving new best policy, reward=4.547! [2023-06-20 23:19:13,269][33484] Fps is (10 sec: 1228.6, 60 sec: 1297.1, 300 sec: 1114.1). Total num frames: 139264. Throughput: 0: 315.5. Samples: 36342. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:19:13,272][33484] Avg episode reward: [(0, '4.547')] [2023-06-20 23:19:18,268][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.7, 300 sec: 1102.8). Total num frames: 143360. Throughput: 0: 315.6. Samples: 37276. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:19:18,269][33484] Avg episode reward: [(0, '4.501')] [2023-06-20 23:19:23,264][33484] Fps is (10 sec: 1229.4, 60 sec: 1297.2, 300 sec: 1122.6). Total num frames: 151552. Throughput: 0: 316.2. Samples: 39182. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:19:23,265][33484] Avg episode reward: [(0, '4.473')] [2023-06-20 23:19:28,269][33484] Fps is (10 sec: 1638.2, 60 sec: 1297.0, 300 sec: 1141.0). Total num frames: 159744. Throughput: 0: 316.1. Samples: 41078. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:19:28,271][33484] Avg episode reward: [(0, '4.395')] [2023-06-20 23:19:31,238][33879] Updated weights for policy 0, policy_version 40 (0.0007) [2023-06-20 23:19:33,267][33484] Fps is (10 sec: 1228.4, 60 sec: 1297.1, 300 sec: 1129.9). Total num frames: 163840. Throughput: 0: 316.4. Samples: 42028. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:19:33,268][33484] Avg episode reward: [(0, '4.513')] [2023-06-20 23:19:38,268][33484] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1146.9). Total num frames: 172032. Throughput: 0: 315.6. Samples: 43906. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:19:38,271][33484] Avg episode reward: [(0, '4.432')] [2023-06-20 23:19:43,267][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1136.3). Total num frames: 176128. Throughput: 0: 315.8. Samples: 45806. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:19:43,268][33484] Avg episode reward: [(0, '4.455')] [2023-06-20 23:19:48,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1152.0). Total num frames: 184320. Throughput: 0: 315.5. Samples: 46748. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:19:48,270][33484] Avg episode reward: [(0, '4.602')] [2023-06-20 23:19:48,271][33878] Saving new best policy, reward=4.602! [2023-06-20 23:19:53,268][33484] Fps is (10 sec: 1228.6, 60 sec: 1228.8, 300 sec: 1141.9). Total num frames: 188416. Throughput: 0: 315.8. Samples: 48650. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:19:53,269][33484] Avg episode reward: [(0, '4.524')] [2023-06-20 23:19:58,269][33484] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1156.5). Total num frames: 196608. Throughput: 0: 315.5. Samples: 50540. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:19:58,272][33484] Avg episode reward: [(0, '4.580')] [2023-06-20 23:20:03,268][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1146.9). Total num frames: 200704. Throughput: 0: 315.7. Samples: 51482. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:20:03,269][33484] Avg episode reward: [(0, '4.525')] [2023-06-20 23:20:03,695][33879] Updated weights for policy 0, policy_version 50 (0.0007) [2023-06-20 23:20:08,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1297.0, 300 sec: 1160.5). Total num frames: 208896. Throughput: 0: 315.3. Samples: 53372. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:20:08,272][33484] Avg episode reward: [(0, '4.513')] [2023-06-20 23:20:13,268][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1151.3). Total num frames: 212992. Throughput: 0: 315.2. Samples: 55262. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:20:13,269][33484] Avg episode reward: [(0, '4.492')] [2023-06-20 23:20:18,267][33484] Fps is (10 sec: 1229.0, 60 sec: 1297.1, 300 sec: 1164.1). Total num frames: 221184. Throughput: 0: 315.1. Samples: 56206. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:20:18,269][33484] Avg episode reward: [(0, '4.351')] [2023-06-20 23:20:23,265][33484] Fps is (10 sec: 1638.9, 60 sec: 1297.1, 300 sec: 1176.3). Total num frames: 229376. Throughput: 0: 315.3. Samples: 58092. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:20:23,266][33484] Avg episode reward: [(0, '4.348')] [2023-06-20 23:20:28,267][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1167.4). Total num frames: 233472. Throughput: 0: 315.1. Samples: 59984. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:20:28,268][33484] Avg episode reward: [(0, '4.215')] [2023-06-20 23:20:33,269][33484] Fps is (10 sec: 1228.3, 60 sec: 1297.0, 300 sec: 1178.8). Total num frames: 241664. Throughput: 0: 315.5. Samples: 60944. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:20:33,271][33484] Avg episode reward: [(0, '4.299')] [2023-06-20 23:20:36,073][33879] Updated weights for policy 0, policy_version 60 (0.0008) [2023-06-20 23:20:38,268][33484] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1170.3). Total num frames: 245760. Throughput: 0: 314.9. Samples: 62822. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:20:38,269][33484] Avg episode reward: [(0, '4.272')] [2023-06-20 23:20:43,269][33484] Fps is (10 sec: 1228.9, 60 sec: 1297.0, 300 sec: 1181.2). Total num frames: 253952. Throughput: 0: 315.2. Samples: 64726. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:20:43,270][33484] Avg episode reward: [(0, '4.276')] [2023-06-20 23:20:48,269][33484] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1172.9). Total num frames: 258048. Throughput: 0: 315.2. Samples: 65668. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:20:48,270][33484] Avg episode reward: [(0, '4.377')] [2023-06-20 23:20:53,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1183.3). Total num frames: 266240. Throughput: 0: 315.3. Samples: 67560. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:20:53,270][33484] Avg episode reward: [(0, '4.438')] [2023-06-20 23:20:58,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1175.4). Total num frames: 270336. Throughput: 0: 315.3. Samples: 69452. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:20:58,271][33484] Avg episode reward: [(0, '4.599')] [2023-06-20 23:21:03,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1297.0, 300 sec: 1185.2). Total num frames: 278528. Throughput: 0: 315.3. Samples: 70396. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:21:03,270][33484] Avg episode reward: [(0, '4.480')] [2023-06-20 23:21:03,273][33878] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000068_278528.pth... [2023-06-20 23:21:08,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1177.6). Total num frames: 282624. Throughput: 0: 315.4. Samples: 72286. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:21:08,271][33484] Avg episode reward: [(0, '4.403')] [2023-06-20 23:21:08,421][33879] Updated weights for policy 0, policy_version 70 (0.0007) [2023-06-20 23:21:13,266][33484] Fps is (10 sec: 1229.2, 60 sec: 1297.1, 300 sec: 1187.0). Total num frames: 290816. Throughput: 0: 315.7. Samples: 74188. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:21:13,267][33484] Avg episode reward: [(0, '4.389')] [2023-06-20 23:21:18,268][33484] Fps is (10 sec: 1638.5, 60 sec: 1297.0, 300 sec: 1196.0). Total num frames: 299008. Throughput: 0: 315.3. Samples: 75132. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:21:18,270][33484] Avg episode reward: [(0, '4.296')] [2023-06-20 23:21:23,269][33484] Fps is (10 sec: 1228.4, 60 sec: 1228.7, 300 sec: 1188.6). Total num frames: 303104. Throughput: 0: 315.5. Samples: 77020. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:21:23,270][33484] Avg episode reward: [(0, '4.310')] [2023-06-20 23:21:28,269][33484] Fps is (10 sec: 1228.7, 60 sec: 1297.0, 300 sec: 1197.3). Total num frames: 311296. Throughput: 0: 315.1. Samples: 78904. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:21:28,271][33484] Avg episode reward: [(0, '4.228')] [2023-06-20 23:21:33,266][33484] Fps is (10 sec: 1229.2, 60 sec: 1228.9, 300 sec: 1190.2). Total num frames: 315392. Throughput: 0: 315.4. Samples: 79860. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:21:33,267][33484] Avg episode reward: [(0, '4.228')] [2023-06-20 23:21:38,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1297.0, 300 sec: 1198.5). Total num frames: 323584. Throughput: 0: 315.4. Samples: 81752. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:21:38,271][33484] Avg episode reward: [(0, '4.380')] [2023-06-20 23:21:40,859][33879] Updated weights for policy 0, policy_version 80 (0.0008) [2023-06-20 23:21:43,269][33484] Fps is (10 sec: 1228.4, 60 sec: 1228.8, 300 sec: 1191.6). Total num frames: 327680. Throughput: 0: 315.5. Samples: 83650. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:21:43,270][33484] Avg episode reward: [(0, '4.441')] [2023-06-20 23:21:48,268][33484] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1199.5). Total num frames: 335872. Throughput: 0: 315.5. Samples: 84592. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:21:48,270][33484] Avg episode reward: [(0, '4.444')] [2023-06-20 23:21:53,269][33484] Fps is (10 sec: 1228.9, 60 sec: 1228.8, 300 sec: 1192.9). Total num frames: 339968. Throughput: 0: 315.6. Samples: 86488. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:21:53,270][33484] Avg episode reward: [(0, '4.346')] [2023-06-20 23:21:58,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1200.5). Total num frames: 348160. Throughput: 0: 315.4. Samples: 88384. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:21:58,270][33484] Avg episode reward: [(0, '4.327')] [2023-06-20 23:22:03,269][33484] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1194.1). Total num frames: 352256. Throughput: 0: 315.5. Samples: 89332. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:22:03,272][33484] Avg episode reward: [(0, '4.296')] [2023-06-20 23:22:08,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1221.9). Total num frames: 360448. Throughput: 0: 315.4. Samples: 91214. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:22:08,270][33484] Avg episode reward: [(0, '4.320')] [2023-06-20 23:22:13,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.7, 300 sec: 1235.7). Total num frames: 364544. Throughput: 0: 315.6. Samples: 93108. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:22:13,271][33484] Avg episode reward: [(0, '4.229')] [2023-06-20 23:22:13,490][33879] Updated weights for policy 0, policy_version 90 (0.0008) [2023-06-20 23:22:18,267][33484] Fps is (10 sec: 1229.0, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 372736. Throughput: 0: 315.4. Samples: 94054. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:22:18,269][33484] Avg episode reward: [(0, '4.350')] [2023-06-20 23:22:23,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1249.6). Total num frames: 376832. Throughput: 0: 315.1. Samples: 95930. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:22:23,272][33484] Avg episode reward: [(0, '4.455')] [2023-06-20 23:22:28,268][33484] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 385024. Throughput: 0: 315.1. Samples: 97830. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:22:28,269][33484] Avg episode reward: [(0, '4.508')] [2023-06-20 23:22:33,267][33484] Fps is (10 sec: 1638.7, 60 sec: 1297.0, 300 sec: 1263.5). Total num frames: 393216. Throughput: 0: 315.3. Samples: 98778. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:22:33,269][33484] Avg episode reward: [(0, '4.591')] [2023-06-20 23:22:38,268][33484] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 397312. Throughput: 0: 315.2. Samples: 100672. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:22:38,270][33484] Avg episode reward: [(0, '4.663')] [2023-06-20 23:22:39,427][33878] Saving new best policy, reward=4.663! [2023-06-20 23:22:43,268][33484] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 405504. Throughput: 0: 315.4. Samples: 102576. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:22:43,270][33484] Avg episode reward: [(0, '4.731')] [2023-06-20 23:22:43,272][33878] Saving new best policy, reward=4.731! [2023-06-20 23:22:45,801][33879] Updated weights for policy 0, policy_version 100 (0.0009) [2023-06-20 23:22:48,269][33484] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 409600. Throughput: 0: 315.3. Samples: 103520. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:22:48,271][33484] Avg episode reward: [(0, '4.652')] [2023-06-20 23:22:53,269][33484] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 417792. Throughput: 0: 315.7. Samples: 105420. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:22:53,271][33484] Avg episode reward: [(0, '4.531')] [2023-06-20 23:22:58,268][33484] Fps is (10 sec: 1228.9, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 421888. Throughput: 0: 315.7. Samples: 107316. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:22:58,271][33484] Avg episode reward: [(0, '4.567')] [2023-06-20 23:23:03,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 430080. Throughput: 0: 315.6. Samples: 108258. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:23:03,270][33484] Avg episode reward: [(0, '4.580')] [2023-06-20 23:23:03,273][33878] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000105_430080.pth... [2023-06-20 23:23:03,348][33878] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000031_126976.pth [2023-06-20 23:23:08,269][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 434176. Throughput: 0: 316.1. Samples: 110154. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:23:08,270][33484] Avg episode reward: [(0, '4.538')] [2023-06-20 23:23:13,266][33484] Fps is (10 sec: 1229.1, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 442368. Throughput: 0: 315.9. Samples: 112044. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:23:13,268][33484] Avg episode reward: [(0, '4.424')] [2023-06-20 23:23:18,248][33879] Updated weights for policy 0, policy_version 110 (0.0010) [2023-06-20 23:23:18,269][33484] Fps is (10 sec: 1638.3, 60 sec: 1297.0, 300 sec: 1277.4). Total num frames: 450560. Throughput: 0: 315.8. Samples: 112990. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:23:18,270][33484] Avg episode reward: [(0, '4.473')] [2023-06-20 23:23:23,266][33484] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 454656. Throughput: 0: 315.9. Samples: 114888. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:23:23,267][33484] Avg episode reward: [(0, '4.485')] [2023-06-20 23:23:28,268][33484] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1277.4). Total num frames: 462848. Throughput: 0: 315.8. Samples: 116786. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:23:28,271][33484] Avg episode reward: [(0, '4.600')] [2023-06-20 23:23:33,267][33484] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 466944. Throughput: 0: 315.8. Samples: 117732. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:23:33,268][33484] Avg episode reward: [(0, '4.509')] [2023-06-20 23:23:38,269][33484] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 475136. Throughput: 0: 315.7. Samples: 119628. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:23:38,271][33484] Avg episode reward: [(0, '4.518')] [2023-06-20 23:23:43,268][33484] Fps is (10 sec: 1228.6, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 479232. Throughput: 0: 315.5. Samples: 121512. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:23:43,270][33484] Avg episode reward: [(0, '4.554')] [2023-06-20 23:23:48,272][33484] Fps is (10 sec: 1228.5, 60 sec: 1297.0, 300 sec: 1263.5). Total num frames: 487424. Throughput: 0: 314.4. Samples: 122406. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:23:48,274][33484] Avg episode reward: [(0, '4.537')] [2023-06-20 23:23:51,399][33879] Updated weights for policy 0, policy_version 120 (0.0008) [2023-06-20 23:23:53,271][33484] Fps is (10 sec: 1228.5, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 491520. Throughput: 0: 309.6. Samples: 124088. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:23:53,273][33484] Avg episode reward: [(0, '4.572')] [2023-06-20 23:23:58,272][33484] Fps is (10 sec: 819.2, 60 sec: 1228.7, 300 sec: 1249.6). Total num frames: 495616. Throughput: 0: 304.7. Samples: 125758. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:23:58,277][33484] Avg episode reward: [(0, '4.463')] [2023-06-20 23:24:03,267][33484] Fps is (10 sec: 1229.2, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 503808. Throughput: 0: 302.4. Samples: 126596. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:24:03,268][33484] Avg episode reward: [(0, '4.453')] [2023-06-20 23:24:08,270][33484] Fps is (10 sec: 1229.1, 60 sec: 1228.8, 300 sec: 1249.6). Total num frames: 507904. Throughput: 0: 297.6. Samples: 128282. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:24:08,274][33484] Avg episode reward: [(0, '4.564')] [2023-06-20 23:24:13,268][33484] Fps is (10 sec: 819.2, 60 sec: 1160.5, 300 sec: 1249.6). Total num frames: 512000. Throughput: 0: 292.6. Samples: 129954. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:24:13,269][33484] Avg episode reward: [(0, '4.550')] [2023-06-20 23:24:18,272][33484] Fps is (10 sec: 1228.6, 60 sec: 1160.5, 300 sec: 1249.6). Total num frames: 520192. Throughput: 0: 290.4. Samples: 130802. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:24:18,273][33484] Avg episode reward: [(0, '4.604')] [2023-06-20 23:24:23,273][33484] Fps is (10 sec: 1228.2, 60 sec: 1160.4, 300 sec: 1235.7). Total num frames: 524288. Throughput: 0: 285.7. Samples: 132486. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:24:23,274][33484] Avg episode reward: [(0, '4.614')] [2023-06-20 23:24:28,001][33879] Updated weights for policy 0, policy_version 130 (0.0008) [2023-06-20 23:24:28,270][33484] Fps is (10 sec: 1229.0, 60 sec: 1160.5, 300 sec: 1249.6). Total num frames: 532480. Throughput: 0: 281.1. Samples: 134160. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:24:28,271][33484] Avg episode reward: [(0, '4.603')] [2023-06-20 23:24:33,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.4, 300 sec: 1235.7). Total num frames: 536576. Throughput: 0: 280.0. Samples: 135008. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:24:33,274][33484] Avg episode reward: [(0, '4.556')] [2023-06-20 23:24:38,273][33484] Fps is (10 sec: 819.0, 60 sec: 1092.2, 300 sec: 1235.7). Total num frames: 540672. Throughput: 0: 279.7. Samples: 136676. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:24:38,275][33484] Avg episode reward: [(0, '4.688')] [2023-06-20 23:24:43,270][33484] Fps is (10 sec: 1229.0, 60 sec: 1160.5, 300 sec: 1235.7). Total num frames: 548864. Throughput: 0: 279.8. Samples: 138350. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:24:43,272][33484] Avg episode reward: [(0, '4.570')] [2023-06-20 23:24:48,270][33484] Fps is (10 sec: 1229.1, 60 sec: 1092.3, 300 sec: 1235.7). Total num frames: 552960. Throughput: 0: 279.8. Samples: 139188. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:24:48,272][33484] Avg episode reward: [(0, '4.573')] [2023-06-20 23:24:53,272][33484] Fps is (10 sec: 819.0, 60 sec: 1092.2, 300 sec: 1221.8). Total num frames: 557056. Throughput: 0: 279.7. Samples: 140868. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:24:53,274][33484] Avg episode reward: [(0, '4.513')] [2023-06-20 23:24:58,273][33484] Fps is (10 sec: 1228.5, 60 sec: 1160.5, 300 sec: 1235.7). Total num frames: 565248. Throughput: 0: 279.7. Samples: 142542. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:24:58,274][33484] Avg episode reward: [(0, '4.528')] [2023-06-20 23:25:03,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1092.2, 300 sec: 1221.8). Total num frames: 569344. Throughput: 0: 279.7. Samples: 143390. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:25:03,272][33484] Avg episode reward: [(0, '4.528')] [2023-06-20 23:25:04,692][33878] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000140_573440.pth... [2023-06-20 23:25:04,702][33879] Updated weights for policy 0, policy_version 140 (0.0008) [2023-06-20 23:25:04,755][33878] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000068_278528.pth [2023-06-20 23:25:08,272][33484] Fps is (10 sec: 819.3, 60 sec: 1092.2, 300 sec: 1221.8). Total num frames: 573440. Throughput: 0: 279.9. Samples: 145080. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:25:08,273][33484] Avg episode reward: [(0, '4.583')] [2023-06-20 23:25:13,274][33484] Fps is (10 sec: 1228.5, 60 sec: 1160.4, 300 sec: 1221.8). Total num frames: 581632. Throughput: 0: 279.8. Samples: 146754. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:25:13,275][33484] Avg episode reward: [(0, '4.574')] [2023-06-20 23:25:18,271][33484] Fps is (10 sec: 1228.9, 60 sec: 1092.3, 300 sec: 1207.9). Total num frames: 585728. Throughput: 0: 279.8. Samples: 147598. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:25:18,271][33484] Avg episode reward: [(0, '4.532')] [2023-06-20 23:25:23,271][33484] Fps is (10 sec: 1229.2, 60 sec: 1160.6, 300 sec: 1221.8). Total num frames: 593920. Throughput: 0: 280.2. Samples: 149284. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:25:23,273][33484] Avg episode reward: [(0, '4.496')] [2023-06-20 23:25:28,274][33484] Fps is (10 sec: 1228.4, 60 sec: 1092.2, 300 sec: 1208.0). Total num frames: 598016. Throughput: 0: 280.3. Samples: 150964. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:25:28,275][33484] Avg episode reward: [(0, '4.486')] [2023-06-20 23:25:33,273][33484] Fps is (10 sec: 819.0, 60 sec: 1092.3, 300 sec: 1208.0). Total num frames: 602112. Throughput: 0: 280.2. Samples: 151796. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:25:33,277][33484] Avg episode reward: [(0, '4.430')] [2023-06-20 23:25:38,271][33484] Fps is (10 sec: 1229.1, 60 sec: 1160.6, 300 sec: 1208.0). Total num frames: 610304. Throughput: 0: 280.2. Samples: 153476. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:25:38,272][33484] Avg episode reward: [(0, '4.415')] [2023-06-20 23:25:41,285][33879] Updated weights for policy 0, policy_version 150 (0.0009) [2023-06-20 23:25:43,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.2, 300 sec: 1208.0). Total num frames: 614400. Throughput: 0: 280.3. Samples: 155154. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:25:43,273][33484] Avg episode reward: [(0, '4.470')] [2023-06-20 23:25:48,266][33484] Fps is (10 sec: 819.6, 60 sec: 1092.3, 300 sec: 1194.1). Total num frames: 618496. Throughput: 0: 280.2. Samples: 155996. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:25:48,270][33484] Avg episode reward: [(0, '4.459')] [2023-06-20 23:25:53,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1208.0). Total num frames: 626688. Throughput: 0: 280.0. Samples: 157678. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:25:53,273][33484] Avg episode reward: [(0, '4.481')] [2023-06-20 23:25:58,272][33484] Fps is (10 sec: 1228.1, 60 sec: 1092.3, 300 sec: 1194.1). Total num frames: 630784. Throughput: 0: 280.0. Samples: 159352. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:25:58,274][33484] Avg episode reward: [(0, '4.504')] [2023-06-20 23:26:03,272][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1208.0). Total num frames: 638976. Throughput: 0: 280.0. Samples: 160198. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:26:03,275][33484] Avg episode reward: [(0, '4.549')] [2023-06-20 23:26:08,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1194.1). Total num frames: 643072. Throughput: 0: 279.8. Samples: 161876. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:26:08,273][33484] Avg episode reward: [(0, '4.681')] [2023-06-20 23:26:13,269][33484] Fps is (10 sec: 819.4, 60 sec: 1092.3, 300 sec: 1180.2). Total num frames: 647168. Throughput: 0: 279.9. Samples: 163558. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:26:13,272][33484] Avg episode reward: [(0, '4.631')] [2023-06-20 23:26:17,979][33879] Updated weights for policy 0, policy_version 160 (0.0009) [2023-06-20 23:26:18,271][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1194.1). Total num frames: 655360. Throughput: 0: 280.1. Samples: 164400. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:26:18,273][33484] Avg episode reward: [(0, '4.616')] [2023-06-20 23:26:23,272][33484] Fps is (10 sec: 1228.5, 60 sec: 1092.2, 300 sec: 1180.2). Total num frames: 659456. Throughput: 0: 280.2. Samples: 166084. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:26:23,273][33484] Avg episode reward: [(0, '4.504')] [2023-06-20 23:26:28,270][33484] Fps is (10 sec: 819.3, 60 sec: 1092.3, 300 sec: 1180.2). Total num frames: 663552. Throughput: 0: 280.5. Samples: 167774. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:26:28,273][33484] Avg episode reward: [(0, '4.496')] [2023-06-20 23:26:33,271][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.6, 300 sec: 1180.2). Total num frames: 671744. Throughput: 0: 280.3. Samples: 168612. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:26:33,272][33484] Avg episode reward: [(0, '4.587')] [2023-06-20 23:26:38,272][33484] Fps is (10 sec: 1228.6, 60 sec: 1092.3, 300 sec: 1180.2). Total num frames: 675840. Throughput: 0: 280.4. Samples: 170294. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:26:38,274][33484] Avg episode reward: [(0, '4.564')] [2023-06-20 23:26:43,272][33484] Fps is (10 sec: 819.2, 60 sec: 1092.3, 300 sec: 1166.3). Total num frames: 679936. Throughput: 0: 280.5. Samples: 171974. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:26:43,273][33484] Avg episode reward: [(0, '4.528')] [2023-06-20 23:26:48,266][33484] Fps is (10 sec: 1229.5, 60 sec: 1160.5, 300 sec: 1180.2). Total num frames: 688128. Throughput: 0: 280.4. Samples: 172814. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:26:48,267][33484] Avg episode reward: [(0, '4.590')] [2023-06-20 23:26:53,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1092.2, 300 sec: 1166.3). Total num frames: 692224. Throughput: 0: 280.6. Samples: 174502. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:26:53,276][33484] Avg episode reward: [(0, '4.544')] [2023-06-20 23:26:54,342][33879] Updated weights for policy 0, policy_version 170 (0.0011) [2023-06-20 23:26:58,271][33484] Fps is (10 sec: 1228.2, 60 sec: 1160.5, 300 sec: 1180.2). Total num frames: 700416. Throughput: 0: 280.4. Samples: 176176. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:26:58,272][33484] Avg episode reward: [(0, '4.609')] [2023-06-20 23:27:03,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1092.3, 300 sec: 1166.3). Total num frames: 704512. Throughput: 0: 280.4. Samples: 177016. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:27:03,273][33484] Avg episode reward: [(0, '4.640')] [2023-06-20 23:27:03,276][33878] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000172_704512.pth... [2023-06-20 23:27:03,362][33878] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000105_430080.pth [2023-06-20 23:27:08,272][33484] Fps is (10 sec: 819.2, 60 sec: 1092.3, 300 sec: 1166.3). Total num frames: 708608. Throughput: 0: 280.4. Samples: 178704. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:27:08,274][33484] Avg episode reward: [(0, '4.530')] [2023-06-20 23:27:13,272][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1166.3). Total num frames: 716800. Throughput: 0: 280.2. Samples: 180384. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:27:13,273][33484] Avg episode reward: [(0, '4.574')] [2023-06-20 23:27:18,274][33484] Fps is (10 sec: 1228.6, 60 sec: 1092.2, 300 sec: 1166.3). Total num frames: 720896. Throughput: 0: 280.2. Samples: 181220. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:27:18,279][33484] Avg episode reward: [(0, '4.525')] [2023-06-20 23:27:23,271][33484] Fps is (10 sec: 819.3, 60 sec: 1092.3, 300 sec: 1152.4). Total num frames: 724992. Throughput: 0: 280.4. Samples: 182910. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:27:23,274][33484] Avg episode reward: [(0, '4.490')] [2023-06-20 23:27:28,272][33484] Fps is (10 sec: 1229.0, 60 sec: 1160.5, 300 sec: 1152.4). Total num frames: 733184. Throughput: 0: 280.3. Samples: 184588. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:27:28,272][33484] Avg episode reward: [(0, '4.516')] [2023-06-20 23:27:30,871][33879] Updated weights for policy 0, policy_version 180 (0.0009) [2023-06-20 23:27:33,264][33484] Fps is (10 sec: 1229.7, 60 sec: 1092.4, 300 sec: 1152.5). Total num frames: 737280. Throughput: 0: 280.2. Samples: 185424. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:27:33,265][33484] Avg episode reward: [(0, '4.461')] [2023-06-20 23:27:38,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1152.4). Total num frames: 745472. Throughput: 0: 280.0. Samples: 187100. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:27:38,275][33484] Avg episode reward: [(0, '4.452')] [2023-06-20 23:27:43,270][33484] Fps is (10 sec: 1228.1, 60 sec: 1160.6, 300 sec: 1152.4). Total num frames: 749568. Throughput: 0: 279.7. Samples: 188764. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:27:43,272][33484] Avg episode reward: [(0, '4.611')] [2023-06-20 23:27:48,271][33484] Fps is (10 sec: 819.4, 60 sec: 1092.2, 300 sec: 1138.5). Total num frames: 753664. Throughput: 0: 279.9. Samples: 189610. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:27:48,274][33484] Avg episode reward: [(0, '4.650')] [2023-06-20 23:27:53,272][33484] Fps is (10 sec: 1228.6, 60 sec: 1160.6, 300 sec: 1152.4). Total num frames: 761856. Throughput: 0: 279.7. Samples: 191292. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:27:53,274][33484] Avg episode reward: [(0, '4.705')] [2023-06-20 23:27:58,268][33484] Fps is (10 sec: 1229.1, 60 sec: 1092.3, 300 sec: 1138.6). Total num frames: 765952. Throughput: 0: 280.0. Samples: 192984. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:27:58,270][33484] Avg episode reward: [(0, '4.754')] [2023-06-20 23:28:00,090][33878] Saving new best policy, reward=4.754! [2023-06-20 23:28:03,268][33484] Fps is (10 sec: 819.5, 60 sec: 1092.3, 300 sec: 1138.6). Total num frames: 770048. Throughput: 0: 279.9. Samples: 193814. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:28:03,269][33484] Avg episode reward: [(0, '4.821')] [2023-06-20 23:28:03,763][33878] Saving new best policy, reward=4.821! [2023-06-20 23:28:07,397][33879] Updated weights for policy 0, policy_version 190 (0.0011) [2023-06-20 23:28:08,270][33484] Fps is (10 sec: 1228.6, 60 sec: 1160.6, 300 sec: 1138.5). Total num frames: 778240. Throughput: 0: 279.6. Samples: 195490. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:28:08,271][33484] Avg episode reward: [(0, '4.841')] [2023-06-20 23:28:08,272][33878] Saving new best policy, reward=4.841! [2023-06-20 23:28:13,269][33484] Fps is (10 sec: 1228.7, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 782336. Throughput: 0: 279.6. Samples: 197170. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:28:13,270][33484] Avg episode reward: [(0, '4.813')] [2023-06-20 23:28:18,272][33484] Fps is (10 sec: 819.0, 60 sec: 1092.3, 300 sec: 1124.6). Total num frames: 786432. Throughput: 0: 279.8. Samples: 198016. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:28:18,275][33484] Avg episode reward: [(0, '4.881')] [2023-06-20 23:28:18,395][33878] Saving new best policy, reward=4.881! [2023-06-20 23:28:23,273][33484] Fps is (10 sec: 1228.4, 60 sec: 1160.5, 300 sec: 1124.6). Total num frames: 794624. Throughput: 0: 279.9. Samples: 199696. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:28:23,274][33484] Avg episode reward: [(0, '4.895')] [2023-06-20 23:28:23,277][33878] Saving new best policy, reward=4.895! [2023-06-20 23:28:28,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1092.2, 300 sec: 1124.6). Total num frames: 798720. Throughput: 0: 280.1. Samples: 201368. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:28:28,274][33484] Avg episode reward: [(0, '4.836')] [2023-06-20 23:28:33,271][33484] Fps is (10 sec: 1229.0, 60 sec: 1160.4, 300 sec: 1124.7). Total num frames: 806912. Throughput: 0: 279.7. Samples: 202196. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:28:33,273][33484] Avg episode reward: [(0, '4.918')] [2023-06-20 23:28:33,275][33878] Saving new best policy, reward=4.918! [2023-06-20 23:28:38,270][33484] Fps is (10 sec: 1229.1, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 811008. Throughput: 0: 279.7. Samples: 203880. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:28:38,272][33484] Avg episode reward: [(0, '4.904')] [2023-06-20 23:28:43,274][33484] Fps is (10 sec: 819.0, 60 sec: 1092.2, 300 sec: 1110.8). Total num frames: 815104. Throughput: 0: 279.6. Samples: 205568. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:28:43,275][33484] Avg episode reward: [(0, '5.109')] [2023-06-20 23:28:44,179][33878] Saving new best policy, reward=5.109! [2023-06-20 23:28:44,184][33879] Updated weights for policy 0, policy_version 200 (0.0008) [2023-06-20 23:28:48,273][33484] Fps is (10 sec: 1228.4, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 823296. Throughput: 0: 280.0. Samples: 206416. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:28:48,276][33484] Avg episode reward: [(0, '5.079')] [2023-06-20 23:28:53,271][33484] Fps is (10 sec: 1229.2, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 827392. Throughput: 0: 280.0. Samples: 208092. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:28:53,272][33484] Avg episode reward: [(0, '5.114')] [2023-06-20 23:28:55,238][33878] Saving new best policy, reward=5.114! [2023-06-20 23:28:58,272][33484] Fps is (10 sec: 819.4, 60 sec: 1092.2, 300 sec: 1110.8). Total num frames: 831488. Throughput: 0: 280.1. Samples: 209774. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:28:58,275][33484] Avg episode reward: [(0, '5.105')] [2023-06-20 23:29:03,271][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 839680. Throughput: 0: 280.0. Samples: 210616. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:29:03,274][33484] Avg episode reward: [(0, '4.977')] [2023-06-20 23:29:03,276][33878] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000205_839680.pth... [2023-06-20 23:29:03,358][33878] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000140_573440.pth [2023-06-20 23:29:08,272][33484] Fps is (10 sec: 1228.7, 60 sec: 1092.2, 300 sec: 1124.6). Total num frames: 843776. Throughput: 0: 280.1. Samples: 212300. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:29:08,274][33484] Avg episode reward: [(0, '4.964')] [2023-06-20 23:29:13,271][33484] Fps is (10 sec: 819.2, 60 sec: 1092.2, 300 sec: 1110.8). Total num frames: 847872. Throughput: 0: 280.5. Samples: 213990. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:29:13,273][33484] Avg episode reward: [(0, '4.997')] [2023-06-20 23:29:18,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 856064. Throughput: 0: 280.9. Samples: 214838. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:29:18,274][33484] Avg episode reward: [(0, '5.142')] [2023-06-20 23:29:18,275][33878] Saving new best policy, reward=5.142! [2023-06-20 23:29:20,647][33879] Updated weights for policy 0, policy_version 210 (0.0009) [2023-06-20 23:29:23,272][33484] Fps is (10 sec: 1228.7, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 860160. Throughput: 0: 280.9. Samples: 216520. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:29:23,273][33484] Avg episode reward: [(0, '4.928')] [2023-06-20 23:29:28,271][33484] Fps is (10 sec: 1229.1, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 868352. Throughput: 0: 280.7. Samples: 218198. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:29:28,273][33484] Avg episode reward: [(0, '4.980')] [2023-06-20 23:29:33,268][33484] Fps is (10 sec: 1229.3, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 872448. Throughput: 0: 280.6. Samples: 219042. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:29:33,268][33484] Avg episode reward: [(0, '4.984')] [2023-06-20 23:29:38,270][33484] Fps is (10 sec: 819.3, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 876544. Throughput: 0: 280.8. Samples: 220730. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:29:38,273][33484] Avg episode reward: [(0, '5.105')] [2023-06-20 23:29:43,267][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.7, 300 sec: 1124.7). Total num frames: 884736. Throughput: 0: 280.8. Samples: 222408. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:29:43,268][33484] Avg episode reward: [(0, '5.089')] [2023-06-20 23:29:48,269][33484] Fps is (10 sec: 1228.9, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 888832. Throughput: 0: 280.6. Samples: 223242. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:29:48,270][33484] Avg episode reward: [(0, '4.983')] [2023-06-20 23:29:53,272][33484] Fps is (10 sec: 818.8, 60 sec: 1092.2, 300 sec: 1110.8). Total num frames: 892928. Throughput: 0: 280.6. Samples: 224926. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:29:53,274][33484] Avg episode reward: [(0, '4.934')] [2023-06-20 23:29:57,217][33879] Updated weights for policy 0, policy_version 220 (0.0010) [2023-06-20 23:29:58,272][33484] Fps is (10 sec: 1228.5, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 901120. Throughput: 0: 280.3. Samples: 226604. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:29:58,272][33484] Avg episode reward: [(0, '4.804')] [2023-06-20 23:30:03,273][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 905216. Throughput: 0: 280.0. Samples: 227436. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:30:03,277][33484] Avg episode reward: [(0, '4.820')] [2023-06-20 23:30:08,270][33484] Fps is (10 sec: 1229.0, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 913408. Throughput: 0: 280.1. Samples: 229124. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:30:08,271][33484] Avg episode reward: [(0, '4.781')] [2023-06-20 23:30:13,270][33484] Fps is (10 sec: 1229.0, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 917504. Throughput: 0: 280.2. Samples: 230806. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:30:13,272][33484] Avg episode reward: [(0, '4.872')] [2023-06-20 23:30:18,272][33484] Fps is (10 sec: 819.1, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 921600. Throughput: 0: 279.9. Samples: 231638. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:30:18,274][33484] Avg episode reward: [(0, '4.931')] [2023-06-20 23:30:23,264][33484] Fps is (10 sec: 1229.6, 60 sec: 1160.7, 300 sec: 1124.7). Total num frames: 929792. Throughput: 0: 280.1. Samples: 233334. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:30:23,267][33484] Avg episode reward: [(0, '4.913')] [2023-06-20 23:30:28,272][33484] Fps is (10 sec: 1228.7, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 933888. Throughput: 0: 280.0. Samples: 235008. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:30:28,274][33484] Avg episode reward: [(0, '4.873')] [2023-06-20 23:30:33,273][33484] Fps is (10 sec: 818.5, 60 sec: 1092.2, 300 sec: 1110.8). Total num frames: 937984. Throughput: 0: 280.1. Samples: 235848. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:30:33,276][33484] Avg episode reward: [(0, '4.935')] [2023-06-20 23:30:33,790][33879] Updated weights for policy 0, policy_version 230 (0.0008) [2023-06-20 23:30:38,273][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 946176. Throughput: 0: 280.2. Samples: 237534. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:30:38,274][33484] Avg episode reward: [(0, '4.994')] [2023-06-20 23:30:43,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.2, 300 sec: 1124.6). Total num frames: 950272. Throughput: 0: 280.1. Samples: 239210. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:30:43,274][33484] Avg episode reward: [(0, '5.068')] [2023-06-20 23:30:48,272][33484] Fps is (10 sec: 819.3, 60 sec: 1092.2, 300 sec: 1110.8). Total num frames: 954368. Throughput: 0: 280.5. Samples: 240058. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:30:48,274][33484] Avg episode reward: [(0, '5.042')] [2023-06-20 23:30:53,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 962560. Throughput: 0: 280.3. Samples: 241738. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:30:53,275][33484] Avg episode reward: [(0, '4.960')] [2023-06-20 23:30:58,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 966656. Throughput: 0: 280.3. Samples: 243418. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:30:58,273][33484] Avg episode reward: [(0, '4.969')] [2023-06-20 23:31:03,271][33484] Fps is (10 sec: 1229.1, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 974848. Throughput: 0: 280.6. Samples: 244266. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:31:03,273][33484] Avg episode reward: [(0, '5.064')] [2023-06-20 23:31:03,276][33878] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000238_974848.pth... [2023-06-20 23:31:03,350][33878] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000172_704512.pth [2023-06-20 23:31:08,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 978944. Throughput: 0: 280.3. Samples: 245948. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-20 23:31:08,273][33484] Avg episode reward: [(0, '5.141')] [2023-06-20 23:31:10,243][33879] Updated weights for policy 0, policy_version 240 (0.0010) [2023-06-20 23:31:13,272][33484] Fps is (10 sec: 819.1, 60 sec: 1092.2, 300 sec: 1110.8). Total num frames: 983040. Throughput: 0: 280.4. Samples: 247628. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:31:13,274][33484] Avg episode reward: [(0, '5.015')] [2023-06-20 23:31:18,271][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 991232. Throughput: 0: 280.5. Samples: 248470. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:31:18,273][33484] Avg episode reward: [(0, '4.962')] [2023-06-20 23:31:23,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.1, 300 sec: 1124.7). Total num frames: 995328. Throughput: 0: 280.5. Samples: 250154. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:31:23,273][33484] Avg episode reward: [(0, '4.883')] [2023-06-20 23:31:28,272][33484] Fps is (10 sec: 819.1, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 999424. Throughput: 0: 280.7. Samples: 251842. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:31:28,274][33484] Avg episode reward: [(0, '5.030')] [2023-06-20 23:31:33,271][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1007616. Throughput: 0: 280.4. Samples: 252676. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:31:33,271][33484] Avg episode reward: [(0, '5.073')] [2023-06-20 23:31:38,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1011712. Throughput: 0: 280.6. Samples: 254366. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:31:38,274][33484] Avg episode reward: [(0, '5.085')] [2023-06-20 23:31:43,270][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.6, 300 sec: 1124.6). Total num frames: 1019904. Throughput: 0: 280.4. Samples: 256034. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:31:43,272][33484] Avg episode reward: [(0, '5.163')] [2023-06-20 23:31:43,274][33878] Saving new best policy, reward=5.163! [2023-06-20 23:31:46,702][33879] Updated weights for policy 0, policy_version 250 (0.0009) [2023-06-20 23:31:48,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1024000. Throughput: 0: 280.3. Samples: 256880. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:31:48,274][33484] Avg episode reward: [(0, '5.079')] [2023-06-20 23:31:53,272][33484] Fps is (10 sec: 819.0, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 1028096. Throughput: 0: 280.2. Samples: 258556. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:31:53,274][33484] Avg episode reward: [(0, '5.111')] [2023-06-20 23:31:58,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1036288. Throughput: 0: 280.5. Samples: 260250. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:31:58,274][33484] Avg episode reward: [(0, '5.255')] [2023-06-20 23:31:58,275][33878] Saving new best policy, reward=5.255! [2023-06-20 23:32:03,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 1040384. Throughput: 0: 280.4. Samples: 261088. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:32:03,274][33484] Avg episode reward: [(0, '5.293')] [2023-06-20 23:32:05,202][33878] Saving new best policy, reward=5.293! [2023-06-20 23:32:08,271][33484] Fps is (10 sec: 819.3, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 1044480. Throughput: 0: 280.2. Samples: 262762. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:32:08,272][33484] Avg episode reward: [(0, '5.113')] [2023-06-20 23:32:13,271][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1052672. Throughput: 0: 279.9. Samples: 264436. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:32:13,274][33484] Avg episode reward: [(0, '4.982')] [2023-06-20 23:32:18,273][33484] Fps is (10 sec: 1228.6, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 1056768. Throughput: 0: 280.1. Samples: 265282. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:32:18,274][33484] Avg episode reward: [(0, '4.984')] [2023-06-20 23:32:23,264][33879] Updated weights for policy 0, policy_version 260 (0.0010) [2023-06-20 23:32:23,264][33484] Fps is (10 sec: 1229.7, 60 sec: 1160.7, 300 sec: 1124.7). Total num frames: 1064960. Throughput: 0: 279.9. Samples: 266960. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:32:23,267][33484] Avg episode reward: [(0, '4.921')] [2023-06-20 23:32:28,271][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1124.6). Total num frames: 1069056. Throughput: 0: 280.3. Samples: 268648. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:32:28,272][33484] Avg episode reward: [(0, '5.072')] [2023-06-20 23:32:33,272][33484] Fps is (10 sec: 818.6, 60 sec: 1092.2, 300 sec: 1110.8). Total num frames: 1073152. Throughput: 0: 280.1. Samples: 269486. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:32:33,274][33484] Avg episode reward: [(0, '5.101')] [2023-06-20 23:32:38,272][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1081344. Throughput: 0: 280.2. Samples: 271166. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:32:38,273][33484] Avg episode reward: [(0, '5.126')] [2023-06-20 23:32:43,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 1085440. Throughput: 0: 279.8. Samples: 272840. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:32:43,275][33484] Avg episode reward: [(0, '5.115')] [2023-06-20 23:32:48,274][33484] Fps is (10 sec: 819.1, 60 sec: 1092.2, 300 sec: 1110.8). Total num frames: 1089536. Throughput: 0: 279.8. Samples: 273680. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:32:48,277][33484] Avg episode reward: [(0, '5.222')] [2023-06-20 23:32:53,265][33484] Fps is (10 sec: 1229.7, 60 sec: 1160.7, 300 sec: 1124.7). Total num frames: 1097728. Throughput: 0: 280.2. Samples: 275368. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:32:53,267][33484] Avg episode reward: [(0, '5.425')] [2023-06-20 23:32:53,269][33878] Saving new best policy, reward=5.425! [2023-06-20 23:32:58,272][33484] Fps is (10 sec: 1229.0, 60 sec: 1092.3, 300 sec: 1124.6). Total num frames: 1101824. Throughput: 0: 280.4. Samples: 277054. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:32:58,273][33484] Avg episode reward: [(0, '5.421')] [2023-06-20 23:32:59,923][33879] Updated weights for policy 0, policy_version 270 (0.0011) [2023-06-20 23:33:03,272][33484] Fps is (10 sec: 818.7, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 1105920. Throughput: 0: 280.2. Samples: 277892. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:33:03,275][33484] Avg episode reward: [(0, '5.536')] [2023-06-20 23:33:03,524][33878] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000271_1110016.pth... [2023-06-20 23:33:03,589][33878] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000205_839680.pth [2023-06-20 23:33:03,598][33878] Saving new best policy, reward=5.536! [2023-06-20 23:33:08,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1114112. Throughput: 0: 280.0. Samples: 279562. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:33:08,274][33484] Avg episode reward: [(0, '5.408')] [2023-06-20 23:33:13,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 1118208. Throughput: 0: 279.7. Samples: 281236. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:33:13,273][33484] Avg episode reward: [(0, '5.367')] [2023-06-20 23:33:18,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1126400. Throughput: 0: 280.0. Samples: 282088. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:33:18,275][33484] Avg episode reward: [(0, '5.392')] [2023-06-20 23:33:23,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1092.1, 300 sec: 1124.7). Total num frames: 1130496. Throughput: 0: 280.1. Samples: 283772. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:33:23,272][33484] Avg episode reward: [(0, '5.306')] [2023-06-20 23:33:28,267][33484] Fps is (10 sec: 819.7, 60 sec: 1092.4, 300 sec: 1110.8). Total num frames: 1134592. Throughput: 0: 280.3. Samples: 285450. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:33:28,271][33484] Avg episode reward: [(0, '5.201')] [2023-06-20 23:33:33,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1142784. Throughput: 0: 280.4. Samples: 286296. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:33:33,274][33484] Avg episode reward: [(0, '5.220')] [2023-06-20 23:33:36,266][33879] Updated weights for policy 0, policy_version 280 (0.0012) [2023-06-20 23:33:38,270][33484] Fps is (10 sec: 1228.4, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1146880. Throughput: 0: 279.9. Samples: 287966. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:33:38,271][33484] Avg episode reward: [(0, '5.245')] [2023-06-20 23:33:43,273][33484] Fps is (10 sec: 819.1, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 1150976. Throughput: 0: 279.7. Samples: 289642. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:33:43,274][33484] Avg episode reward: [(0, '5.216')] [2023-06-20 23:33:48,272][33484] Fps is (10 sec: 1228.5, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1159168. Throughput: 0: 279.8. Samples: 290482. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:33:48,274][33484] Avg episode reward: [(0, '5.225')] [2023-06-20 23:33:53,273][33484] Fps is (10 sec: 1228.9, 60 sec: 1092.1, 300 sec: 1124.7). Total num frames: 1163264. Throughput: 0: 280.1. Samples: 292166. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:33:53,276][33484] Avg episode reward: [(0, '5.280')] [2023-06-20 23:33:58,265][33484] Fps is (10 sec: 1229.7, 60 sec: 1160.7, 300 sec: 1124.7). Total num frames: 1171456. Throughput: 0: 280.1. Samples: 293840. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:33:58,267][33484] Avg episode reward: [(0, '5.372')] [2023-06-20 23:34:03,271][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1175552. Throughput: 0: 279.7. Samples: 294672. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:34:03,272][33484] Avg episode reward: [(0, '5.440')] [2023-06-20 23:34:08,272][33484] Fps is (10 sec: 818.6, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1179648. Throughput: 0: 279.8. Samples: 296364. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:34:08,274][33484] Avg episode reward: [(0, '5.553')] [2023-06-20 23:34:09,205][33878] Saving new best policy, reward=5.553! [2023-06-20 23:34:12,837][33879] Updated weights for policy 0, policy_version 290 (0.0010) [2023-06-20 23:34:13,267][33484] Fps is (10 sec: 1229.3, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1187840. Throughput: 0: 279.8. Samples: 298040. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:34:13,269][33484] Avg episode reward: [(0, '5.672')] [2023-06-20 23:34:13,272][33878] Saving new best policy, reward=5.672! [2023-06-20 23:34:18,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1191936. Throughput: 0: 279.6. Samples: 298880. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:34:18,274][33484] Avg episode reward: [(0, '5.712')] [2023-06-20 23:34:20,196][33878] Saving new best policy, reward=5.712! [2023-06-20 23:34:23,271][33484] Fps is (10 sec: 818.9, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 1196032. Throughput: 0: 280.0. Samples: 300566. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:34:23,273][33484] Avg episode reward: [(0, '5.767')] [2023-06-20 23:34:23,769][33878] Saving new best policy, reward=5.767! [2023-06-20 23:34:28,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.4, 300 sec: 1124.6). Total num frames: 1204224. Throughput: 0: 280.3. Samples: 302254. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:34:28,274][33484] Avg episode reward: [(0, '5.870')] [2023-06-20 23:34:28,275][33878] Saving new best policy, reward=5.870! [2023-06-20 23:34:33,272][33484] Fps is (10 sec: 1228.7, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1208320. Throughput: 0: 280.5. Samples: 303104. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:34:33,273][33484] Avg episode reward: [(0, '5.877')] [2023-06-20 23:34:34,685][33878] Saving new best policy, reward=5.877! [2023-06-20 23:34:38,271][33484] Fps is (10 sec: 819.3, 60 sec: 1092.2, 300 sec: 1110.8). Total num frames: 1212416. Throughput: 0: 280.5. Samples: 304788. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:34:38,272][33484] Avg episode reward: [(0, '5.916')] [2023-06-20 23:34:38,341][33878] Saving new best policy, reward=5.916! [2023-06-20 23:34:43,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1220608. Throughput: 0: 280.4. Samples: 306462. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:34:43,274][33484] Avg episode reward: [(0, '5.999')] [2023-06-20 23:34:43,276][33878] Saving new best policy, reward=5.999! [2023-06-20 23:34:48,272][33484] Fps is (10 sec: 1228.7, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1224704. Throughput: 0: 280.7. Samples: 307304. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:34:48,274][33484] Avg episode reward: [(0, '5.743')] [2023-06-20 23:34:49,332][33879] Updated weights for policy 0, policy_version 300 (0.0015) [2023-06-20 23:34:53,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1232896. Throughput: 0: 280.4. Samples: 308984. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:34:53,274][33484] Avg episode reward: [(0, '5.787')] [2023-06-20 23:34:58,271][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.1, 300 sec: 1124.7). Total num frames: 1236992. Throughput: 0: 280.6. Samples: 310670. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:34:58,273][33484] Avg episode reward: [(0, '5.759')] [2023-06-20 23:35:03,272][33484] Fps is (10 sec: 819.2, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 1241088. Throughput: 0: 280.4. Samples: 311500. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:35:03,275][33484] Avg episode reward: [(0, '5.611')] [2023-06-20 23:35:03,916][33878] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000304_1245184.pth... [2023-06-20 23:35:03,981][33878] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000238_974848.pth [2023-06-20 23:35:08,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1249280. Throughput: 0: 280.4. Samples: 313184. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:35:08,273][33484] Avg episode reward: [(0, '5.604')] [2023-06-20 23:35:13,268][33484] Fps is (10 sec: 1229.3, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 1253376. Throughput: 0: 280.2. Samples: 314862. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:35:13,270][33484] Avg episode reward: [(0, '5.426')] [2023-06-20 23:35:18,271][33484] Fps is (10 sec: 819.2, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 1257472. Throughput: 0: 280.1. Samples: 315710. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:35:18,273][33484] Avg episode reward: [(0, '5.566')] [2023-06-20 23:35:23,273][33484] Fps is (10 sec: 1228.3, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1265664. Throughput: 0: 280.1. Samples: 317392. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:35:23,274][33484] Avg episode reward: [(0, '5.583')] [2023-06-20 23:35:25,768][33879] Updated weights for policy 0, policy_version 310 (0.0010) [2023-06-20 23:35:28,273][33484] Fps is (10 sec: 1228.6, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1269760. Throughput: 0: 280.3. Samples: 319074. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:35:28,274][33484] Avg episode reward: [(0, '5.808')] [2023-06-20 23:35:33,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1277952. Throughput: 0: 280.3. Samples: 319916. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:35:33,274][33484] Avg episode reward: [(0, '5.861')] [2023-06-20 23:35:38,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1282048. Throughput: 0: 280.5. Samples: 321606. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:35:38,273][33484] Avg episode reward: [(0, '5.862')] [2023-06-20 23:35:43,264][33484] Fps is (10 sec: 819.8, 60 sec: 1092.4, 300 sec: 1124.7). Total num frames: 1286144. Throughput: 0: 280.3. Samples: 323282. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:35:43,268][33484] Avg episode reward: [(0, '6.192')] [2023-06-20 23:35:44,071][33878] Saving new best policy, reward=6.192! [2023-06-20 23:35:48,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1294336. Throughput: 0: 280.6. Samples: 324126. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:35:48,274][33484] Avg episode reward: [(0, '6.267')] [2023-06-20 23:35:48,277][33878] Saving new best policy, reward=6.267! [2023-06-20 23:35:53,270][33484] Fps is (10 sec: 1228.1, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1298432. Throughput: 0: 280.5. Samples: 325808. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:35:53,271][33484] Avg episode reward: [(0, '6.544')] [2023-06-20 23:35:55,042][33878] Saving new best policy, reward=6.544! [2023-06-20 23:35:58,271][33484] Fps is (10 sec: 819.4, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 1302528. Throughput: 0: 280.5. Samples: 327486. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:35:58,272][33484] Avg episode reward: [(0, '6.672')] [2023-06-20 23:35:58,691][33878] Saving new best policy, reward=6.672! [2023-06-20 23:36:02,268][33879] Updated weights for policy 0, policy_version 320 (0.0009) [2023-06-20 23:36:03,272][33484] Fps is (10 sec: 1228.5, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1310720. Throughput: 0: 280.3. Samples: 328322. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:36:03,273][33484] Avg episode reward: [(0, '6.585')] [2023-06-20 23:36:08,272][33484] Fps is (10 sec: 1228.7, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1314816. Throughput: 0: 280.2. Samples: 330002. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:36:08,275][33484] Avg episode reward: [(0, '6.521')] [2023-06-20 23:36:13,268][33484] Fps is (10 sec: 819.5, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 1318912. Throughput: 0: 280.2. Samples: 331680. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:36:13,270][33484] Avg episode reward: [(0, '6.406')] [2023-06-20 23:36:18,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1327104. Throughput: 0: 280.1. Samples: 332522. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:36:18,274][33484] Avg episode reward: [(0, '6.371')] [2023-06-20 23:36:23,273][33484] Fps is (10 sec: 1228.2, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1331200. Throughput: 0: 279.9. Samples: 334204. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:36:23,274][33484] Avg episode reward: [(0, '6.284')] [2023-06-20 23:36:28,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1339392. Throughput: 0: 280.2. Samples: 335894. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:36:28,275][33484] Avg episode reward: [(0, '6.216')] [2023-06-20 23:36:33,271][33484] Fps is (10 sec: 1229.0, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1343488. Throughput: 0: 280.1. Samples: 336732. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:36:33,272][33484] Avg episode reward: [(0, '5.875')] [2023-06-20 23:36:38,266][33484] Fps is (10 sec: 819.8, 60 sec: 1092.4, 300 sec: 1110.8). Total num frames: 1347584. Throughput: 0: 280.2. Samples: 338416. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:36:38,268][33484] Avg episode reward: [(0, '5.756')] [2023-06-20 23:36:38,815][33879] Updated weights for policy 0, policy_version 330 (0.0010) [2023-06-20 23:36:43,272][33484] Fps is (10 sec: 1228.6, 60 sec: 1160.4, 300 sec: 1124.7). Total num frames: 1355776. Throughput: 0: 280.2. Samples: 340096. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:36:43,275][33484] Avg episode reward: [(0, '5.760')] [2023-06-20 23:36:48,272][33484] Fps is (10 sec: 1228.0, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1359872. Throughput: 0: 280.4. Samples: 340940. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:36:48,274][33484] Avg episode reward: [(0, '5.719')] [2023-06-20 23:36:53,270][33484] Fps is (10 sec: 819.4, 60 sec: 1092.3, 300 sec: 1110.8). Total num frames: 1363968. Throughput: 0: 280.9. Samples: 342640. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:36:53,273][33484] Avg episode reward: [(0, '5.782')] [2023-06-20 23:36:58,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1372160. Throughput: 0: 280.7. Samples: 344314. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:36:58,274][33484] Avg episode reward: [(0, '5.820')] [2023-06-20 23:37:03,273][33484] Fps is (10 sec: 1228.4, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1376256. Throughput: 0: 280.6. Samples: 345150. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:37:03,275][33484] Avg episode reward: [(0, '5.908')] [2023-06-20 23:37:04,279][33878] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000337_1380352.pth... [2023-06-20 23:37:04,345][33878] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000271_1110016.pth [2023-06-20 23:37:08,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1384448. Throughput: 0: 280.8. Samples: 346840. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:37:08,274][33484] Avg episode reward: [(0, '5.773')] [2023-06-20 23:37:13,270][33484] Fps is (10 sec: 1229.1, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1388544. Throughput: 0: 280.7. Samples: 348524. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:37:13,271][33484] Avg episode reward: [(0, '5.844')] [2023-06-20 23:37:15,221][33879] Updated weights for policy 0, policy_version 340 (0.0009) [2023-06-20 23:37:18,273][33484] Fps is (10 sec: 819.1, 60 sec: 1092.2, 300 sec: 1110.7). Total num frames: 1392640. Throughput: 0: 280.8. Samples: 349368. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:37:18,275][33484] Avg episode reward: [(0, '5.962')] [2023-06-20 23:37:23,271][33484] Fps is (10 sec: 1228.6, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1400832. Throughput: 0: 280.9. Samples: 351056. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:37:23,274][33484] Avg episode reward: [(0, '6.198')] [2023-06-20 23:37:28,273][33484] Fps is (10 sec: 1228.9, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1404928. Throughput: 0: 281.2. Samples: 352748. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:37:28,274][33484] Avg episode reward: [(0, '6.376')] [2023-06-20 23:37:33,272][33484] Fps is (10 sec: 819.1, 60 sec: 1092.2, 300 sec: 1110.8). Total num frames: 1409024. Throughput: 0: 281.2. Samples: 353592. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:37:33,275][33484] Avg episode reward: [(0, '6.413')] [2023-06-20 23:37:38,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.4, 300 sec: 1124.7). Total num frames: 1417216. Throughput: 0: 280.7. Samples: 355274. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:37:38,273][33484] Avg episode reward: [(0, '6.470')] [2023-06-20 23:37:43,271][33484] Fps is (10 sec: 1228.9, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1421312. Throughput: 0: 280.8. Samples: 356950. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:37:43,275][33484] Avg episode reward: [(0, '6.445')] [2023-06-20 23:37:48,271][33484] Fps is (10 sec: 1229.0, 60 sec: 1160.6, 300 sec: 1124.6). Total num frames: 1429504. Throughput: 0: 280.4. Samples: 357768. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:37:48,273][33484] Avg episode reward: [(0, '6.396')] [2023-06-20 23:37:51,801][33879] Updated weights for policy 0, policy_version 350 (0.0008) [2023-06-20 23:37:53,273][33484] Fps is (10 sec: 1228.6, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1433600. Throughput: 0: 280.4. Samples: 359460. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:37:53,274][33484] Avg episode reward: [(0, '6.435')] [2023-06-20 23:37:58,269][33484] Fps is (10 sec: 819.3, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1437696. Throughput: 0: 280.3. Samples: 361138. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:37:58,273][33484] Avg episode reward: [(0, '6.545')] [2023-06-20 23:38:03,273][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1445888. Throughput: 0: 280.4. Samples: 361986. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:38:03,276][33484] Avg episode reward: [(0, '6.314')] [2023-06-20 23:38:08,273][33484] Fps is (10 sec: 1228.4, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1449984. Throughput: 0: 280.2. Samples: 363664. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:38:08,274][33484] Avg episode reward: [(0, '6.186')] [2023-06-20 23:38:13,273][33484] Fps is (10 sec: 819.2, 60 sec: 1092.2, 300 sec: 1110.8). Total num frames: 1454080. Throughput: 0: 280.2. Samples: 365356. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:38:13,276][33484] Avg episode reward: [(0, '6.264')] [2023-06-20 23:38:18,273][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1462272. Throughput: 0: 280.2. Samples: 366200. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:38:18,274][33484] Avg episode reward: [(0, '5.814')] [2023-06-20 23:38:23,271][33484] Fps is (10 sec: 1229.0, 60 sec: 1092.3, 300 sec: 1124.6). Total num frames: 1466368. Throughput: 0: 280.2. Samples: 367884. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:38:23,273][33484] Avg episode reward: [(0, '6.019')] [2023-06-20 23:38:28,157][33879] Updated weights for policy 0, policy_version 360 (0.0008) [2023-06-20 23:38:28,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1474560. Throughput: 0: 280.5. Samples: 369574. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:38:28,274][33484] Avg episode reward: [(0, '6.075')] [2023-06-20 23:38:33,273][33484] Fps is (10 sec: 1228.6, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1478656. Throughput: 0: 281.0. Samples: 370412. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:38:33,273][33484] Avg episode reward: [(0, '6.112')] [2023-06-20 23:38:38,273][33484] Fps is (10 sec: 819.2, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1482752. Throughput: 0: 281.0. Samples: 372106. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:38:38,275][33484] Avg episode reward: [(0, '6.124')] [2023-06-20 23:38:43,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1490944. Throughput: 0: 281.0. Samples: 373784. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:38:43,273][33484] Avg episode reward: [(0, '6.079')] [2023-06-20 23:38:48,273][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 1495040. Throughput: 0: 280.8. Samples: 374620. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:38:48,276][33484] Avg episode reward: [(0, '6.133')] [2023-06-20 23:38:53,274][33484] Fps is (10 sec: 819.1, 60 sec: 1092.3, 300 sec: 1110.7). Total num frames: 1499136. Throughput: 0: 280.8. Samples: 376302. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:38:53,276][33484] Avg episode reward: [(0, '6.295')] [2023-06-20 23:38:58,264][33484] Fps is (10 sec: 1229.9, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1507328. Throughput: 0: 280.7. Samples: 377984. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:38:58,265][33484] Avg episode reward: [(0, '6.317')] [2023-06-20 23:39:03,271][33484] Fps is (10 sec: 1229.1, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1511424. Throughput: 0: 280.5. Samples: 378820. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:39:03,272][33484] Avg episode reward: [(0, '6.408')] [2023-06-20 23:39:04,427][33878] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000370_1515520.pth... [2023-06-20 23:39:04,438][33879] Updated weights for policy 0, policy_version 370 (0.0008) [2023-06-20 23:39:04,486][33878] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000304_1245184.pth [2023-06-20 23:39:08,271][33484] Fps is (10 sec: 1227.9, 60 sec: 1160.6, 300 sec: 1124.6). Total num frames: 1519616. Throughput: 0: 280.5. Samples: 380506. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:39:08,272][33484] Avg episode reward: [(0, '6.324')] [2023-06-20 23:39:13,272][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1523712. Throughput: 0: 280.2. Samples: 382182. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:39:13,275][33484] Avg episode reward: [(0, '6.268')] [2023-06-20 23:39:18,272][33484] Fps is (10 sec: 819.1, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1527808. Throughput: 0: 280.3. Samples: 383026. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:39:18,273][33484] Avg episode reward: [(0, '6.306')] [2023-06-20 23:39:23,266][33484] Fps is (10 sec: 1229.5, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1536000. Throughput: 0: 280.2. Samples: 384714. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:39:23,268][33484] Avg episode reward: [(0, '6.398')] [2023-06-20 23:39:28,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1540096. Throughput: 0: 280.1. Samples: 386388. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:39:28,273][33484] Avg episode reward: [(0, '6.488')] [2023-06-20 23:39:33,272][33484] Fps is (10 sec: 818.7, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1544192. Throughput: 0: 280.1. Samples: 387226. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:39:33,275][33484] Avg episode reward: [(0, '6.411')] [2023-06-20 23:39:38,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1552384. Throughput: 0: 279.9. Samples: 388898. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:39:38,273][33484] Avg episode reward: [(0, '6.310')] [2023-06-20 23:39:41,036][33879] Updated weights for policy 0, policy_version 380 (0.0009) [2023-06-20 23:39:43,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1556480. Throughput: 0: 280.0. Samples: 390584. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:39:43,273][33484] Avg episode reward: [(0, '6.511')] [2023-06-20 23:39:48,274][33484] Fps is (10 sec: 819.1, 60 sec: 1092.2, 300 sec: 1110.8). Total num frames: 1560576. Throughput: 0: 280.1. Samples: 391426. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:39:48,277][33484] Avg episode reward: [(0, '6.571')] [2023-06-20 23:39:53,272][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1568768. Throughput: 0: 280.0. Samples: 393106. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:39:53,273][33484] Avg episode reward: [(0, '6.490')] [2023-06-20 23:39:58,272][33484] Fps is (10 sec: 1229.1, 60 sec: 1092.1, 300 sec: 1124.7). Total num frames: 1572864. Throughput: 0: 280.2. Samples: 394792. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:39:58,273][33484] Avg episode reward: [(0, '6.532')] [2023-06-20 23:40:03,271][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1581056. Throughput: 0: 280.3. Samples: 395640. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:40:03,274][33484] Avg episode reward: [(0, '6.731')] [2023-06-20 23:40:03,277][33878] Saving new best policy, reward=6.731! [2023-06-20 23:40:08,271][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1585152. Throughput: 0: 280.3. Samples: 397328. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:40:08,273][33484] Avg episode reward: [(0, '6.548')] [2023-06-20 23:40:13,273][33484] Fps is (10 sec: 819.1, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1589248. Throughput: 0: 280.1. Samples: 398994. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:40:13,274][33484] Avg episode reward: [(0, '6.595')] [2023-06-20 23:40:17,615][33879] Updated weights for policy 0, policy_version 390 (0.0009) [2023-06-20 23:40:18,273][33484] Fps is (10 sec: 1228.6, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1597440. Throughput: 0: 280.2. Samples: 399834. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:40:18,274][33484] Avg episode reward: [(0, '6.642')] [2023-06-20 23:40:23,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1092.1, 300 sec: 1124.7). Total num frames: 1601536. Throughput: 0: 280.5. Samples: 401520. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:40:23,275][33484] Avg episode reward: [(0, '6.800')] [2023-06-20 23:40:25,173][33878] Saving new best policy, reward=6.800! [2023-06-20 23:40:28,274][33484] Fps is (10 sec: 819.1, 60 sec: 1092.2, 300 sec: 1110.8). Total num frames: 1605632. Throughput: 0: 280.5. Samples: 403206. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:40:28,275][33484] Avg episode reward: [(0, '6.837')] [2023-06-20 23:40:28,613][33878] Saving new best policy, reward=6.837! [2023-06-20 23:40:33,271][33484] Fps is (10 sec: 1229.2, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1613824. Throughput: 0: 280.8. Samples: 404060. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:40:33,271][33484] Avg episode reward: [(0, '6.694')] [2023-06-20 23:40:38,274][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.2, 300 sec: 1124.6). Total num frames: 1617920. Throughput: 0: 280.9. Samples: 405748. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:40:38,277][33484] Avg episode reward: [(0, '6.752')] [2023-06-20 23:40:43,271][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1626112. Throughput: 0: 280.8. Samples: 407430. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:40:43,272][33484] Avg episode reward: [(0, '7.012')] [2023-06-20 23:40:43,275][33878] Saving new best policy, reward=7.012! [2023-06-20 23:40:48,273][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1630208. Throughput: 0: 280.7. Samples: 408274. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:40:48,275][33484] Avg episode reward: [(0, '6.848')] [2023-06-20 23:40:53,271][33484] Fps is (10 sec: 819.2, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1634304. Throughput: 0: 280.5. Samples: 409952. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:40:53,273][33484] Avg episode reward: [(0, '7.043')] [2023-06-20 23:40:54,096][33878] Saving new best policy, reward=7.043! [2023-06-20 23:40:54,098][33879] Updated weights for policy 0, policy_version 400 (0.0009) [2023-06-20 23:40:58,272][33484] Fps is (10 sec: 1229.0, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1642496. Throughput: 0: 280.9. Samples: 411636. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:40:58,273][33484] Avg episode reward: [(0, '7.271')] [2023-06-20 23:40:58,274][33878] Saving new best policy, reward=7.271! [2023-06-20 23:41:03,269][33484] Fps is (10 sec: 1229.0, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1646592. Throughput: 0: 281.2. Samples: 412486. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:41:03,271][33484] Avg episode reward: [(0, '7.420')] [2023-06-20 23:41:04,894][33878] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000403_1650688.pth... [2023-06-20 23:41:04,954][33878] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000337_1380352.pth [2023-06-20 23:41:04,959][33878] Saving new best policy, reward=7.420! [2023-06-20 23:41:08,271][33484] Fps is (10 sec: 819.3, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1650688. Throughput: 0: 281.3. Samples: 414176. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:41:08,272][33484] Avg episode reward: [(0, '7.554')] [2023-06-20 23:41:08,632][33878] Saving new best policy, reward=7.554! [2023-06-20 23:41:13,272][33484] Fps is (10 sec: 1228.4, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1658880. Throughput: 0: 281.4. Samples: 415868. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:41:13,274][33484] Avg episode reward: [(0, '7.491')] [2023-06-20 23:41:18,272][33484] Fps is (10 sec: 1228.6, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1662976. Throughput: 0: 281.2. Samples: 416714. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:41:18,275][33484] Avg episode reward: [(0, '7.590')] [2023-06-20 23:41:19,472][33878] Saving new best policy, reward=7.590! [2023-06-20 23:41:23,265][33484] Fps is (10 sec: 1229.7, 60 sec: 1160.7, 300 sec: 1124.7). Total num frames: 1671168. Throughput: 0: 281.3. Samples: 418404. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:41:23,266][33484] Avg episode reward: [(0, '7.641')] [2023-06-20 23:41:23,268][33878] Saving new best policy, reward=7.641! [2023-06-20 23:41:28,271][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1675264. Throughput: 0: 281.1. Samples: 420078. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:41:28,272][33484] Avg episode reward: [(0, '7.815')] [2023-06-20 23:41:28,273][33878] Saving new best policy, reward=7.815! [2023-06-20 23:41:30,445][33879] Updated weights for policy 0, policy_version 410 (0.0008) [2023-06-20 23:41:33,274][33484] Fps is (10 sec: 818.5, 60 sec: 1092.2, 300 sec: 1124.6). Total num frames: 1679360. Throughput: 0: 281.0. Samples: 420918. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:41:33,275][33484] Avg episode reward: [(0, '7.776')] [2023-06-20 23:41:38,272][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1687552. Throughput: 0: 281.2. Samples: 422606. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:41:38,275][33484] Avg episode reward: [(0, '7.522')] [2023-06-20 23:41:43,265][33484] Fps is (10 sec: 1229.9, 60 sec: 1092.4, 300 sec: 1124.7). Total num frames: 1691648. Throughput: 0: 281.2. Samples: 424288. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:41:43,266][33484] Avg episode reward: [(0, '7.659')] [2023-06-20 23:41:48,274][33484] Fps is (10 sec: 819.1, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1695744. Throughput: 0: 280.9. Samples: 425126. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:41:48,281][33484] Avg episode reward: [(0, '7.368')] [2023-06-20 23:41:53,273][33484] Fps is (10 sec: 1227.9, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1703936. Throughput: 0: 280.7. Samples: 426806. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:41:53,274][33484] Avg episode reward: [(0, '7.342')] [2023-06-20 23:41:58,273][33484] Fps is (10 sec: 1228.9, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1708032. Throughput: 0: 280.7. Samples: 428498. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:41:58,274][33484] Avg episode reward: [(0, '7.368')] [2023-06-20 23:42:03,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1716224. Throughput: 0: 280.4. Samples: 429332. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:42:03,274][33484] Avg episode reward: [(0, '7.192')] [2023-06-20 23:42:06,716][33879] Updated weights for policy 0, policy_version 420 (0.0009) [2023-06-20 23:42:08,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1720320. Throughput: 0: 280.4. Samples: 431022. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:42:08,272][33484] Avg episode reward: [(0, '7.587')] [2023-06-20 23:42:13,273][33484] Fps is (10 sec: 819.2, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1724416. Throughput: 0: 281.1. Samples: 432728. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:42:13,277][33484] Avg episode reward: [(0, '7.427')] [2023-06-20 23:42:18,272][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1732608. Throughput: 0: 280.9. Samples: 433560. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:42:18,274][33484] Avg episode reward: [(0, '7.248')] [2023-06-20 23:42:23,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1092.1, 300 sec: 1124.7). Total num frames: 1736704. Throughput: 0: 280.7. Samples: 435238. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:42:23,274][33484] Avg episode reward: [(0, '7.518')] [2023-06-20 23:42:28,270][33484] Fps is (10 sec: 819.4, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1740800. Throughput: 0: 280.6. Samples: 436914. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:42:28,274][33484] Avg episode reward: [(0, '7.358')] [2023-06-20 23:42:33,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1748992. Throughput: 0: 280.8. Samples: 437762. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:42:33,274][33484] Avg episode reward: [(0, '7.270')] [2023-06-20 23:42:38,272][33484] Fps is (10 sec: 1228.5, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1753088. Throughput: 0: 280.8. Samples: 439444. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:42:38,275][33484] Avg episode reward: [(0, '7.519')] [2023-06-20 23:42:43,202][33879] Updated weights for policy 0, policy_version 430 (0.0010) [2023-06-20 23:42:43,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.4, 300 sec: 1124.7). Total num frames: 1761280. Throughput: 0: 280.8. Samples: 441136. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:42:43,274][33484] Avg episode reward: [(0, '7.642')] [2023-06-20 23:42:48,273][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1765376. Throughput: 0: 281.3. Samples: 441992. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:42:48,273][33484] Avg episode reward: [(0, '7.837')] [2023-06-20 23:42:48,274][33878] Saving new best policy, reward=7.837! [2023-06-20 23:42:53,271][33484] Fps is (10 sec: 819.4, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1769472. Throughput: 0: 281.3. Samples: 443680. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:42:53,274][33484] Avg episode reward: [(0, '7.752')] [2023-06-20 23:42:58,269][33484] Fps is (10 sec: 1229.3, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1777664. Throughput: 0: 281.0. Samples: 445372. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:42:58,270][33484] Avg episode reward: [(0, '7.669')] [2023-06-20 23:43:03,272][33484] Fps is (10 sec: 1228.7, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1781760. Throughput: 0: 281.0. Samples: 446206. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:43:03,275][33484] Avg episode reward: [(0, '7.729')] [2023-06-20 23:43:04,986][33878] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000436_1785856.pth... [2023-06-20 23:43:05,043][33878] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000370_1515520.pth [2023-06-20 23:43:08,273][33484] Fps is (10 sec: 818.9, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 1785856. Throughput: 0: 281.3. Samples: 447896. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:43:08,281][33484] Avg episode reward: [(0, '7.774')] [2023-06-20 23:43:13,269][33484] Fps is (10 sec: 1229.2, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1794048. Throughput: 0: 281.5. Samples: 449582. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:43:13,271][33484] Avg episode reward: [(0, '7.758')] [2023-06-20 23:43:18,273][33484] Fps is (10 sec: 1228.8, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1798144. Throughput: 0: 281.6. Samples: 450432. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:43:18,274][33484] Avg episode reward: [(0, '8.138')] [2023-06-20 23:43:19,561][33878] Saving new best policy, reward=8.138! [2023-06-20 23:43:19,562][33879] Updated weights for policy 0, policy_version 440 (0.0010) [2023-06-20 23:43:23,273][33484] Fps is (10 sec: 1228.3, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1806336. Throughput: 0: 282.0. Samples: 452132. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:43:23,274][33484] Avg episode reward: [(0, '8.336')] [2023-06-20 23:43:23,276][33878] Saving new best policy, reward=8.336! [2023-06-20 23:43:28,273][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1810432. Throughput: 0: 282.0. Samples: 453824. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:43:28,275][33484] Avg episode reward: [(0, '8.494')] [2023-06-20 23:43:28,276][33878] Saving new best policy, reward=8.494! [2023-06-20 23:43:33,273][33484] Fps is (10 sec: 819.2, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1814528. Throughput: 0: 281.6. Samples: 454664. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:43:33,274][33484] Avg episode reward: [(0, '8.631')] [2023-06-20 23:43:34,145][33878] Saving new best policy, reward=8.631! [2023-06-20 23:43:38,273][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 1822720. Throughput: 0: 281.7. Samples: 456358. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:43:38,274][33484] Avg episode reward: [(0, '8.459')] [2023-06-20 23:43:43,272][33484] Fps is (10 sec: 1229.0, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1826816. Throughput: 0: 281.6. Samples: 458044. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:43:43,274][33484] Avg episode reward: [(0, '9.078')] [2023-06-20 23:43:45,168][33878] Saving new best policy, reward=9.078! [2023-06-20 23:43:48,273][33484] Fps is (10 sec: 819.2, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1830912. Throughput: 0: 281.9. Samples: 458892. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:43:48,276][33484] Avg episode reward: [(0, '9.160')] [2023-06-20 23:43:48,797][33878] Saving new best policy, reward=9.160! [2023-06-20 23:43:53,272][33484] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.6). Total num frames: 1839104. Throughput: 0: 281.8. Samples: 460578. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:43:53,274][33484] Avg episode reward: [(0, '9.153')] [2023-06-20 23:43:56,076][33879] Updated weights for policy 0, policy_version 450 (0.0010) [2023-06-20 23:43:58,272][33484] Fps is (10 sec: 1228.9, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 1843200. Throughput: 0: 282.0. Samples: 462272. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:43:58,273][33484] Avg episode reward: [(0, '9.398')] [2023-06-20 23:43:59,699][33878] Saving new best policy, reward=9.398! [2023-06-20 23:44:03,267][33484] Fps is (10 sec: 819.6, 60 sec: 1092.4, 300 sec: 1110.8). Total num frames: 1847296. Throughput: 0: 281.7. Samples: 463106. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-20 23:44:03,268][33484] Avg episode reward: [(0, '10.152')] [2023-06-20 23:44:03,417][33878] Saving new best policy, reward=10.152! [2023-06-20 23:44:08,270][33484] Fps is (10 sec: 1229.0, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1855488. Throughput: 0: 281.4. Samples: 464796. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:44:08,272][33484] Avg episode reward: [(0, '10.534')] [2023-06-20 23:44:08,273][33878] Saving new best policy, reward=10.534! [2023-06-20 23:44:13,268][33484] Fps is (10 sec: 1228.6, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1859584. Throughput: 0: 281.3. Samples: 466480. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:44:13,269][33484] Avg episode reward: [(0, '10.468')] [2023-06-20 23:44:18,269][33484] Fps is (10 sec: 1229.0, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1867776. Throughput: 0: 282.4. Samples: 467372. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-20 23:44:18,272][33484] Avg episode reward: [(0, '10.098')] [2023-06-20 23:44:23,269][33484] Fps is (10 sec: 1228.7, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 1871872. Throughput: 0: 287.1. Samples: 469278. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:44:23,272][33484] Avg episode reward: [(0, '10.173')] [2023-06-20 23:44:28,268][33484] Fps is (10 sec: 1228.9, 60 sec: 1160.6, 300 sec: 1138.6). Total num frames: 1880064. Throughput: 0: 292.0. Samples: 471182. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:44:28,271][33484] Avg episode reward: [(0, '10.363')] [2023-06-20 23:44:30,783][33879] Updated weights for policy 0, policy_version 460 (0.0013) [2023-06-20 23:44:33,270][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 1884160. Throughput: 0: 294.2. Samples: 472130. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:44:33,272][33484] Avg episode reward: [(0, '10.514')] [2023-06-20 23:44:38,270][33484] Fps is (10 sec: 1228.7, 60 sec: 1160.6, 300 sec: 1138.6). Total num frames: 1892352. Throughput: 0: 299.1. Samples: 474036. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-20 23:44:38,279][33484] Avg episode reward: [(0, '10.696')] [2023-06-20 23:44:40,378][33878] Saving new best policy, reward=10.696! [2023-06-20 23:44:41,341][33484] Keyboard interrupt detected in the event loop EvtLoop [Runner_EvtLoop, process=main process 33484], exiting... [2023-06-20 23:44:41,343][33878] Stopping Batcher_0... [2023-06-20 23:44:41,343][33484] Runner profile tree view: main_loop: 1657.2042 [2023-06-20 23:44:41,343][33484] Collected {0: 1896448}, FPS: 1144.4 [2023-06-20 23:44:41,343][33878] Loop batcher_evt_loop terminating... [2023-06-20 23:44:41,344][33878] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000463_1896448.pth... [2023-06-20 23:44:41,451][33878] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000403_1650688.pth [2023-06-20 23:44:41,471][33878] Stopping LearnerWorker_p0... [2023-06-20 23:44:41,473][33878] Loop learner_proc0_evt_loop terminating... [2023-06-20 23:44:41,492][33879] Weights refcount: 2 0 [2023-06-20 23:44:41,515][33879] Stopping InferenceWorker_p0-w0... [2023-06-20 23:44:41,516][33879] Loop inference_proc0-0_evt_loop terminating... [2023-06-20 23:44:41,599][33881] Stopping RolloutWorker_w1... [2023-06-20 23:44:41,600][33881] Loop rollout_proc1_evt_loop terminating... [2023-06-20 23:44:41,626][33887] Stopping RolloutWorker_w6... [2023-06-20 23:44:41,631][33887] Loop rollout_proc6_evt_loop terminating... [2023-06-20 23:44:41,631][33888] Stopping RolloutWorker_w7... [2023-06-20 23:44:41,632][33888] Loop rollout_proc7_evt_loop terminating... [2023-06-20 23:44:41,635][33883] Stopping RolloutWorker_w3... [2023-06-20 23:44:41,635][33883] Loop rollout_proc3_evt_loop terminating... [2023-06-20 23:44:41,635][33882] Stopping RolloutWorker_w2... [2023-06-20 23:44:41,636][33882] Loop rollout_proc2_evt_loop terminating... [2023-06-20 23:44:41,652][33885] Stopping RolloutWorker_w5... [2023-06-20 23:44:41,655][33885] Loop rollout_proc5_evt_loop terminating... [2023-06-20 23:44:41,662][33880] Stopping RolloutWorker_w0... [2023-06-20 23:44:41,663][33880] Loop rollout_proc0_evt_loop terminating... [2023-06-20 23:44:41,673][33884] Stopping RolloutWorker_w4... [2023-06-20 23:44:41,674][33884] Loop rollout_proc4_evt_loop terminating... [2023-06-20 23:44:47,471][33484] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-20 23:44:47,472][33484] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-20 23:44:47,473][33484] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-20 23:44:47,473][33484] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-20 23:44:47,473][33484] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-20 23:44:47,473][33484] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-20 23:44:47,474][33484] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! [2023-06-20 23:44:47,474][33484] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-20 23:44:47,474][33484] Adding new argument 'push_to_hub'=False that is not in the saved config file! [2023-06-20 23:44:47,475][33484] Adding new argument 'hf_repository'=None that is not in the saved config file! [2023-06-20 23:44:47,475][33484] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-20 23:44:47,475][33484] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-20 23:44:47,475][33484] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-20 23:44:47,475][33484] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-20 23:44:47,476][33484] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-20 23:44:47,481][33484] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-20 23:44:47,482][33484] RunningMeanStd input shape: (3, 72, 128) [2023-06-20 23:44:47,484][33484] RunningMeanStd input shape: (1,) [2023-06-20 23:44:47,511][33484] ConvEncoder: input_channels=3 [2023-06-20 23:44:47,574][33484] Conv encoder output size: 512 [2023-06-20 23:44:47,575][33484] Policy head output size: 512 [2023-06-20 23:44:47,591][33484] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000463_1896448.pth... [2023-06-20 23:44:49,664][33484] Num frames 100... [2023-06-20 23:44:50,448][33484] Num frames 200... [2023-06-20 23:44:51,234][33484] Num frames 300... [2023-06-20 23:44:51,955][33484] Num frames 400... [2023-06-20 23:44:52,664][33484] Avg episode rewards: #0: 7.800, true rewards: #0: 4.800 [2023-06-20 23:44:52,667][33484] Avg episode reward: 7.800, avg true_objective: 4.800 [2023-06-20 23:44:52,819][33484] Num frames 500... [2023-06-20 23:44:53,621][33484] Num frames 600... [2023-06-20 23:44:54,427][33484] Num frames 700... [2023-06-20 23:44:55,186][33484] Num frames 800... [2023-06-20 23:44:55,921][33484] Num frames 900... [2023-06-20 23:44:56,222][33484] Avg episode rewards: #0: 6.640, true rewards: #0: 4.640 [2023-06-20 23:44:56,225][33484] Avg episode reward: 6.640, avg true_objective: 4.640 [2023-06-20 23:44:56,779][33484] Num frames 1000... [2023-06-20 23:44:57,576][33484] Num frames 1100... [2023-06-20 23:44:58,334][33484] Num frames 1200... [2023-06-20 23:44:59,149][33484] Num frames 1300... [2023-06-20 23:45:00,016][33484] Num frames 1400... [2023-06-20 23:45:00,857][33484] Num frames 1500... [2023-06-20 23:45:01,769][33484] Num frames 1600... [2023-06-20 23:45:02,716][33484] Num frames 1700... [2023-06-20 23:45:03,653][33484] Num frames 1800... [2023-06-20 23:45:04,425][33484] Num frames 1900... [2023-06-20 23:45:04,924][33484] Avg episode rewards: #0: 12.507, true rewards: #0: 6.507 [2023-06-20 23:45:04,926][33484] Avg episode reward: 12.507, avg true_objective: 6.507 [2023-06-20 23:45:05,279][33484] Num frames 2000... [2023-06-20 23:45:06,004][33484] Num frames 2100... [2023-06-20 23:45:06,806][33484] Num frames 2200... [2023-06-20 23:45:07,630][33484] Num frames 2300... [2023-06-20 23:45:08,433][33484] Num frames 2400... [2023-06-20 23:45:09,250][33484] Num frames 2500... [2023-06-20 23:45:10,056][33484] Num frames 2600... [2023-06-20 23:45:10,861][33484] Num frames 2700... [2023-06-20 23:45:11,629][33484] Num frames 2800... [2023-06-20 23:45:12,371][33484] Num frames 2900... [2023-06-20 23:45:12,545][33484] Avg episode rewards: #0: 14.030, true rewards: #0: 7.280 [2023-06-20 23:45:12,547][33484] Avg episode reward: 14.030, avg true_objective: 7.280 [2023-06-20 23:45:13,285][33484] Num frames 3000... [2023-06-20 23:45:14,036][33484] Num frames 3100... [2023-06-20 23:45:14,779][33484] Num frames 3200... [2023-06-20 23:45:15,525][33484] Num frames 3300... [2023-06-20 23:45:16,325][33484] Avg episode rewards: #0: 12.784, true rewards: #0: 6.784 [2023-06-20 23:45:16,327][33484] Avg episode reward: 12.784, avg true_objective: 6.784 [2023-06-20 23:45:16,384][33484] Num frames 3400... [2023-06-20 23:45:17,111][33484] Num frames 3500... [2023-06-20 23:45:17,877][33484] Num frames 3600... [2023-06-20 23:45:18,629][33484] Num frames 3700... [2023-06-20 23:45:19,408][33484] Num frames 3800... [2023-06-20 23:45:20,213][33484] Num frames 3900... [2023-06-20 23:45:20,845][33484] Avg episode rewards: #0: 12.447, true rewards: #0: 6.613 [2023-06-20 23:45:20,847][33484] Avg episode reward: 12.447, avg true_objective: 6.613 [2023-06-20 23:45:21,080][33484] Num frames 4000... [2023-06-20 23:45:21,808][33484] Num frames 4100... [2023-06-20 23:45:22,535][33484] Num frames 4200... [2023-06-20 23:45:23,270][33484] Num frames 4300... [2023-06-20 23:45:23,977][33484] Num frames 4400... [2023-06-20 23:45:24,687][33484] Num frames 4500... [2023-06-20 23:45:25,411][33484] Num frames 4600... [2023-06-20 23:45:26,168][33484] Num frames 4700... [2023-06-20 23:45:26,509][33484] Avg episode rewards: #0: 12.480, true rewards: #0: 6.766 [2023-06-20 23:45:26,510][33484] Avg episode reward: 12.480, avg true_objective: 6.766 [2023-06-20 23:45:26,982][33484] Num frames 4800... [2023-06-20 23:45:27,749][33484] Num frames 4900... [2023-06-20 23:45:28,513][33484] Num frames 5000... [2023-06-20 23:45:29,311][33484] Num frames 5100... [2023-06-20 23:45:30,107][33484] Num frames 5200... [2023-06-20 23:45:30,914][33484] Num frames 5300... [2023-06-20 23:45:31,679][33484] Num frames 5400... [2023-06-20 23:45:32,515][33484] Num frames 5500... [2023-06-20 23:45:33,445][33484] Num frames 5600... [2023-06-20 23:45:34,211][33484] Num frames 5700... [2023-06-20 23:45:35,071][33484] Num frames 5800... [2023-06-20 23:45:35,915][33484] Avg episode rewards: #0: 13.735, true rewards: #0: 7.360 [2023-06-20 23:45:35,918][33484] Avg episode reward: 13.735, avg true_objective: 7.360 [2023-06-20 23:45:36,019][33484] Num frames 5900... [2023-06-20 23:45:36,912][33484] Num frames 6000... [2023-06-20 23:45:37,805][33484] Num frames 6100... [2023-06-20 23:45:38,704][33484] Num frames 6200... [2023-06-20 23:45:39,621][33484] Num frames 6300... [2023-06-20 23:45:40,578][33484] Num frames 6400... [2023-06-20 23:45:41,536][33484] Num frames 6500... [2023-06-20 23:45:42,449][33484] Num frames 6600... [2023-06-20 23:45:43,406][33484] Num frames 6700... [2023-06-20 23:45:44,418][33484] Num frames 6800... [2023-06-20 23:45:45,331][33484] Num frames 6900... [2023-06-20 23:45:46,033][33484] Avg episode rewards: #0: 14.751, true rewards: #0: 7.751 [2023-06-20 23:45:46,033][33484] Avg episode reward: 14.751, avg true_objective: 7.751 [2023-06-20 23:45:46,198][33484] Num frames 7000... [2023-06-20 23:45:46,950][33484] Num frames 7100... [2023-06-20 23:45:47,679][33484] Num frames 7200... [2023-06-20 23:45:48,463][33484] Num frames 7300... [2023-06-20 23:45:49,250][33484] Num frames 7400... [2023-06-20 23:45:49,994][33484] Num frames 7500... [2023-06-20 23:45:50,680][33484] Avg episode rewards: #0: 14.384, true rewards: #0: 7.584 [2023-06-20 23:45:50,683][33484] Avg episode reward: 14.384, avg true_objective: 7.584 [2023-06-20 23:45:54,782][33484] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4! [2023-06-21 00:02:44,504][62782] Saving configuration to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json... [2023-06-21 00:02:44,528][62782] Rollout worker 0 uses device cpu [2023-06-21 00:02:44,529][62782] Rollout worker 1 uses device cpu [2023-06-21 00:02:44,529][62782] Rollout worker 2 uses device cpu [2023-06-21 00:02:44,530][62782] Rollout worker 3 uses device cpu [2023-06-21 00:02:44,530][62782] Rollout worker 4 uses device cpu [2023-06-21 00:02:44,530][62782] Rollout worker 5 uses device cpu [2023-06-21 00:02:44,531][62782] Rollout worker 6 uses device cpu [2023-06-21 00:02:44,531][62782] Rollout worker 7 uses device cpu [2023-06-21 00:02:44,719][62782] InferenceWorker_p0-w0: min num requests: 2 [2023-06-21 00:02:44,752][62782] Starting all processes... [2023-06-21 00:02:44,753][62782] Starting process learner_proc0 [2023-06-21 00:02:44,803][62782] Starting all processes... [2023-06-21 00:02:44,856][62782] Starting process inference_proc0-0 [2023-06-21 00:02:44,856][62782] Starting process rollout_proc0 [2023-06-21 00:02:44,856][62782] Starting process rollout_proc1 [2023-06-21 00:02:44,856][62782] Starting process rollout_proc2 [2023-06-21 00:02:44,856][62782] Starting process rollout_proc3 [2023-06-21 00:02:44,856][62782] Starting process rollout_proc4 [2023-06-21 00:02:44,856][62782] Starting process rollout_proc5 [2023-06-21 00:02:44,856][62782] Starting process rollout_proc6 [2023-06-21 00:02:44,858][62782] Starting process rollout_proc7 [2023-06-21 00:02:46,946][62893] Starting seed is not provided [2023-06-21 00:02:46,946][62893] Initializing actor-critic model on device cpu [2023-06-21 00:02:46,946][62893] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 00:02:46,947][62896] On MacOS, not setting affinity [2023-06-21 00:02:46,947][62893] RunningMeanStd input shape: (1,) [2023-06-21 00:02:46,966][62893] ConvEncoder: input_channels=3 [2023-06-21 00:02:47,013][62903] On MacOS, not setting affinity [2023-06-21 00:02:47,013][62898] On MacOS, not setting affinity [2023-06-21 00:02:47,013][62897] On MacOS, not setting affinity [2023-06-21 00:02:47,024][62901] On MacOS, not setting affinity [2023-06-21 00:02:47,049][62899] On MacOS, not setting affinity [2023-06-21 00:02:47,066][62902] On MacOS, not setting affinity [2023-06-21 00:02:47,069][62893] Conv encoder output size: 512 [2023-06-21 00:02:47,070][62893] Policy head output size: 512 [2023-06-21 00:02:47,091][62893] Created Actor Critic model with architecture: [2023-06-21 00:02:47,091][62893] ActorCriticSharedWeights( (obs_normalizer): ObservationNormalizer( (running_mean_std): RunningMeanStdDictInPlace( (running_mean_std): ModuleDict( (obs): RunningMeanStdInPlace() ) ) ) (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace) (encoder): VizdoomEncoder( (basic_encoder): ConvEncoder( (enc): RecursiveScriptModule( original_name=ConvEncoderImpl (conv_head): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Conv2d) (1): RecursiveScriptModule(original_name=ELU) (2): RecursiveScriptModule(original_name=Conv2d) (3): RecursiveScriptModule(original_name=ELU) (4): RecursiveScriptModule(original_name=Conv2d) (5): RecursiveScriptModule(original_name=ELU) ) (mlp_layers): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Linear) (1): RecursiveScriptModule(original_name=ELU) ) ) ) ) (core): ModelCoreRNN( (core): GRU(512, 512) ) (decoder): MlpDecoder( (mlp): Identity() ) (critic_linear): Linear(in_features=512, out_features=1, bias=True) (action_parameterization): ActionParameterizationDefault( (distribution_linear): Linear(in_features=512, out_features=5, bias=True) ) ) [2023-06-21 00:02:47,095][62893] Using optimizer [2023-06-21 00:02:47,095][62893] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000463_1896448.pth... [2023-06-21 00:02:47,102][62900] On MacOS, not setting affinity [2023-06-21 00:02:47,125][62893] Loading model from checkpoint [2023-06-21 00:02:47,135][62893] Loaded experiment state at self.train_step=463, self.env_steps=1896448 [2023-06-21 00:02:47,136][62893] Initialized policy 0 weights for model version 463 [2023-06-21 00:02:47,137][62893] LearnerWorker_p0 finished initialization! [2023-06-21 00:02:47,139][62895] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 00:02:47,140][62895] RunningMeanStd input shape: (1,) [2023-06-21 00:02:47,155][62895] ConvEncoder: input_channels=3 [2023-06-21 00:02:47,207][62895] Conv encoder output size: 512 [2023-06-21 00:02:47,207][62895] Policy head output size: 512 [2023-06-21 00:02:47,219][62782] Inference worker 0-0 is ready! [2023-06-21 00:02:47,221][62782] All inference workers are ready! Signal rollout workers to start! [2023-06-21 00:02:47,263][62899] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 00:02:47,270][62901] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 00:02:47,270][62903] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 00:02:47,272][62900] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 00:02:47,273][62902] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 00:02:47,280][62897] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 00:02:47,282][62898] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 00:02:47,285][62896] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 00:02:48,705][62782] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 1896448. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 00:02:49,591][62898] Decorrelating experience for 0 frames... [2023-06-21 00:02:49,591][62899] Decorrelating experience for 0 frames... [2023-06-21 00:02:49,592][62901] Decorrelating experience for 0 frames... [2023-06-21 00:02:49,592][62902] Decorrelating experience for 0 frames... [2023-06-21 00:02:49,593][62900] Decorrelating experience for 0 frames... [2023-06-21 00:02:49,595][62903] Decorrelating experience for 0 frames... [2023-06-21 00:02:50,546][62900] Decorrelating experience for 32 frames... [2023-06-21 00:02:50,573][62899] Decorrelating experience for 32 frames... [2023-06-21 00:02:50,574][62898] Decorrelating experience for 32 frames... [2023-06-21 00:02:50,574][62902] Decorrelating experience for 32 frames... [2023-06-21 00:02:50,575][62901] Decorrelating experience for 32 frames... [2023-06-21 00:02:50,583][62897] Decorrelating experience for 0 frames... [2023-06-21 00:02:51,245][62897] Decorrelating experience for 32 frames... [2023-06-21 00:02:51,246][62896] Decorrelating experience for 0 frames... [2023-06-21 00:02:51,955][62896] Decorrelating experience for 32 frames... [2023-06-21 00:02:51,957][62903] Decorrelating experience for 32 frames... [2023-06-21 00:02:52,185][62898] Decorrelating experience for 64 frames... [2023-06-21 00:02:52,187][62901] Decorrelating experience for 64 frames... [2023-06-21 00:02:52,187][62900] Decorrelating experience for 64 frames... [2023-06-21 00:02:52,191][62902] Decorrelating experience for 64 frames... [2023-06-21 00:02:52,640][62899] Decorrelating experience for 64 frames... [2023-06-21 00:02:52,707][62897] Decorrelating experience for 64 frames... [2023-06-21 00:02:53,472][62903] Decorrelating experience for 64 frames... [2023-06-21 00:02:53,474][62896] Decorrelating experience for 64 frames... [2023-06-21 00:02:53,706][62782] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 1896448. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 00:02:54,695][62900] Decorrelating experience for 96 frames... [2023-06-21 00:02:54,696][62902] Decorrelating experience for 96 frames... [2023-06-21 00:02:54,700][62901] Decorrelating experience for 96 frames... [2023-06-21 00:02:54,702][62898] Decorrelating experience for 96 frames... [2023-06-21 00:02:55,417][62899] Decorrelating experience for 96 frames... [2023-06-21 00:02:55,419][62897] Decorrelating experience for 96 frames... [2023-06-21 00:02:55,751][62896] Decorrelating experience for 96 frames... [2023-06-21 00:02:56,156][62903] Decorrelating experience for 96 frames... [2023-06-21 00:02:58,705][62782] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 1896448. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 00:02:58,708][62782] Avg episode reward: [(0, '0.320')] [2023-06-21 00:03:03,705][62782] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 1896448. Throughput: 0: 100.1. Samples: 1502. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 00:03:03,708][62782] Avg episode reward: [(0, '1.444')] [2023-06-21 00:03:04,710][62782] Heartbeat connected on Batcher_0 [2023-06-21 00:03:04,714][62782] Heartbeat connected on LearnerWorker_p0 [2023-06-21 00:03:04,740][62782] Heartbeat connected on InferenceWorker_p0-w0 [2023-06-21 00:03:04,783][62782] Heartbeat connected on RolloutWorker_w0 [2023-06-21 00:03:04,789][62782] Heartbeat connected on RolloutWorker_w3 [2023-06-21 00:03:04,789][62782] Heartbeat connected on RolloutWorker_w4 [2023-06-21 00:03:04,810][62782] Heartbeat connected on RolloutWorker_w1 [2023-06-21 00:03:04,812][62782] Heartbeat connected on RolloutWorker_w2 [2023-06-21 00:03:04,817][62782] Heartbeat connected on RolloutWorker_w5 [2023-06-21 00:03:04,823][62782] Heartbeat connected on RolloutWorker_w6 [2023-06-21 00:03:04,824][62782] Heartbeat connected on RolloutWorker_w7 [2023-06-21 00:03:08,704][62782] Fps is (10 sec: 819.3, 60 sec: 409.6, 300 sec: 409.6). Total num frames: 1904640. Throughput: 0: 170.6. Samples: 3412. Policy #0 lag: (min: 0.0, avg: 0.0, max: 0.0) [2023-06-21 00:03:08,705][62782] Avg episode reward: [(0, '2.944')] [2023-06-21 00:03:13,703][62782] Fps is (10 sec: 1638.7, 60 sec: 655.4, 300 sec: 655.4). Total num frames: 1912832. Throughput: 0: 174.4. Samples: 4360. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 00:03:13,706][62782] Avg episode reward: [(0, '4.354')] [2023-06-21 00:03:18,705][62782] Fps is (10 sec: 1228.7, 60 sec: 682.7, 300 sec: 682.7). Total num frames: 1916928. Throughput: 0: 208.9. Samples: 6268. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 00:03:18,705][62782] Avg episode reward: [(0, '6.182')] [2023-06-21 00:03:23,700][62782] Fps is (10 sec: 1229.1, 60 sec: 819.3, 300 sec: 819.3). Total num frames: 1925120. Throughput: 0: 233.3. Samples: 8166. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 00:03:23,703][62782] Avg episode reward: [(0, '7.207')] [2023-06-21 00:03:28,709][62782] Fps is (10 sec: 1228.4, 60 sec: 819.1, 300 sec: 819.1). Total num frames: 1929216. Throughput: 0: 228.2. Samples: 9130. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 00:03:28,713][62782] Avg episode reward: [(0, '7.606')] [2023-06-21 00:03:32,792][62895] Updated weights for policy 0, policy_version 473 (0.0011) [2023-06-21 00:03:33,704][62782] Fps is (10 sec: 1228.4, 60 sec: 910.2, 300 sec: 910.2). Total num frames: 1937408. Throughput: 0: 245.0. Samples: 11026. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 00:03:33,705][62782] Avg episode reward: [(0, '8.380')] [2023-06-21 00:03:38,704][62782] Fps is (10 sec: 1229.3, 60 sec: 901.1, 300 sec: 901.1). Total num frames: 1941504. Throughput: 0: 287.3. Samples: 12928. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 00:03:38,707][62782] Avg episode reward: [(0, '8.388')] [2023-06-21 00:03:43,704][62782] Fps is (10 sec: 1228.8, 60 sec: 968.2, 300 sec: 968.2). Total num frames: 1949696. Throughput: 0: 308.5. Samples: 13884. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 00:03:43,705][62782] Avg episode reward: [(0, '8.642')] [2023-06-21 00:03:48,705][62782] Fps is (10 sec: 1228.7, 60 sec: 955.7, 300 sec: 955.7). Total num frames: 1953792. Throughput: 0: 317.3. Samples: 15780. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 00:03:48,709][62782] Avg episode reward: [(0, '8.993')] [2023-06-21 00:03:53,705][62782] Fps is (10 sec: 1228.7, 60 sec: 1092.3, 300 sec: 1008.2). Total num frames: 1961984. Throughput: 0: 316.7. Samples: 17664. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 00:03:53,708][62782] Avg episode reward: [(0, '9.600')] [2023-06-21 00:03:58,702][62782] Fps is (10 sec: 1229.2, 60 sec: 1160.6, 300 sec: 994.8). Total num frames: 1966080. Throughput: 0: 316.6. Samples: 18606. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 00:03:58,705][62782] Avg episode reward: [(0, '10.099')] [2023-06-21 00:04:03,704][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1037.7). Total num frames: 1974272. Throughput: 0: 316.5. Samples: 20512. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 00:04:03,706][62782] Avg episode reward: [(0, '9.877')] [2023-06-21 00:04:05,292][62895] Updated weights for policy 0, policy_version 483 (0.0016) [2023-06-21 00:04:08,705][62782] Fps is (10 sec: 1637.9, 60 sec: 1297.0, 300 sec: 1075.2). Total num frames: 1982464. Throughput: 0: 316.5. Samples: 22410. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 00:04:08,707][62782] Avg episode reward: [(0, '9.789')] [2023-06-21 00:04:13,704][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1060.2). Total num frames: 1986560. Throughput: 0: 316.4. Samples: 23366. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 00:04:13,706][62782] Avg episode reward: [(0, '9.602')] [2023-06-21 00:04:18,704][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1092.3). Total num frames: 1994752. Throughput: 0: 316.5. Samples: 25270. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 00:04:18,707][62782] Avg episode reward: [(0, '9.301')] [2023-06-21 00:04:23,705][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.7, 300 sec: 1077.9). Total num frames: 1998848. Throughput: 0: 316.6. Samples: 27176. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 00:04:23,709][62782] Avg episode reward: [(0, '9.217')] [2023-06-21 00:04:27,884][62893] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000490_2007040.pth... [2023-06-21 00:04:27,890][62782] Component Batcher_0 stopped! [2023-06-21 00:04:27,884][62893] Stopping Batcher_0... [2023-06-21 00:04:27,910][62893] Loop batcher_evt_loop terminating... [2023-06-21 00:04:28,051][62895] Weights refcount: 2 0 [2023-06-21 00:04:28,052][62895] Stopping InferenceWorker_p0-w0... [2023-06-21 00:04:28,052][62895] Loop inference_proc0-0_evt_loop terminating... [2023-06-21 00:04:28,052][62782] Component InferenceWorker_p0-w0 stopped! [2023-06-21 00:04:28,059][62893] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000436_1785856.pth [2023-06-21 00:04:28,093][62893] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000490_2007040.pth... [2023-06-21 00:04:28,146][62898] Stopping RolloutWorker_w2... [2023-06-21 00:04:28,147][62898] Loop rollout_proc2_evt_loop terminating... [2023-06-21 00:04:28,149][62897] Stopping RolloutWorker_w1... [2023-06-21 00:04:28,146][62782] Component RolloutWorker_w2 stopped! [2023-06-21 00:04:28,159][62897] Loop rollout_proc1_evt_loop terminating... [2023-06-21 00:04:28,159][62782] Component RolloutWorker_w1 stopped! [2023-06-21 00:04:28,172][62902] Stopping RolloutWorker_w6... [2023-06-21 00:04:28,173][62902] Loop rollout_proc6_evt_loop terminating... [2023-06-21 00:04:28,172][62903] Stopping RolloutWorker_w7... [2023-06-21 00:04:28,173][62903] Loop rollout_proc7_evt_loop terminating... [2023-06-21 00:04:28,172][62782] Component RolloutWorker_w6 stopped! [2023-06-21 00:04:28,174][62782] Component RolloutWorker_w7 stopped! [2023-06-21 00:04:28,178][62900] Stopping RolloutWorker_w4... [2023-06-21 00:04:28,179][62782] Component RolloutWorker_w4 stopped! [2023-06-21 00:04:28,179][62900] Loop rollout_proc4_evt_loop terminating... [2023-06-21 00:04:28,186][62896] Stopping RolloutWorker_w0... [2023-06-21 00:04:28,187][62896] Loop rollout_proc0_evt_loop terminating... [2023-06-21 00:04:28,189][62782] Component RolloutWorker_w0 stopped! [2023-06-21 00:04:28,238][62899] Stopping RolloutWorker_w3... [2023-06-21 00:04:28,238][62899] Loop rollout_proc3_evt_loop terminating... [2023-06-21 00:04:28,238][62782] Component RolloutWorker_w3 stopped! [2023-06-21 00:04:28,260][62901] Stopping RolloutWorker_w5... [2023-06-21 00:04:28,262][62901] Loop rollout_proc5_evt_loop terminating... [2023-06-21 00:04:28,260][62782] Component RolloutWorker_w5 stopped! [2023-06-21 00:04:28,323][62893] Stopping LearnerWorker_p0... [2023-06-21 00:04:28,323][62893] Loop learner_proc0_evt_loop terminating... [2023-06-21 00:04:28,323][62782] Component LearnerWorker_p0 stopped! [2023-06-21 00:04:28,325][62782] Waiting for process learner_proc0 to stop... [2023-06-21 00:04:28,717][62782] Waiting for process inference_proc0-0 to join... [2023-06-21 00:04:28,724][62782] Waiting for process rollout_proc0 to join... [2023-06-21 00:04:28,725][62782] Waiting for process rollout_proc1 to join... [2023-06-21 00:04:28,725][62782] Waiting for process rollout_proc2 to join... [2023-06-21 00:04:28,725][62782] Waiting for process rollout_proc3 to join... [2023-06-21 00:04:28,726][62782] Waiting for process rollout_proc4 to join... [2023-06-21 00:04:28,726][62782] Waiting for process rollout_proc5 to join... [2023-06-21 00:04:28,727][62782] Waiting for process rollout_proc6 to join... [2023-06-21 00:04:28,727][62782] Waiting for process rollout_proc7 to join... [2023-06-21 00:04:28,727][62782] Batcher 0 profile tree view: batching: 0.3430, releasing_batches: 0.0005 [2023-06-21 00:04:28,728][62782] InferenceWorker_p0-w0 profile tree view: wait_policy: 0.0015 wait_policy_total: 70.9955 update_model: 0.1313 weight_update: 0.0013 one_step: 0.0044 handle_policy_step: 28.8837 deserialize: 0.2579, stack: 0.0465, obs_to_device_normalize: 1.9937, forward: 25.4962, send_messages: 0.2765 prepare_outputs: 0.3301 to_cpu: 0.0368 [2023-06-21 00:04:28,728][62782] Learner 0 profile tree view: misc: 0.0005, prepare_batch: 12.2746 train: 41.6131 epoch_init: 0.0001, minibatch_init: 0.0003, losses_postprocess: 0.0009, kl_divergence: 0.0039, after_optimizer: 0.0241 calculate_losses: 23.6560 losses_init: 0.0000, forward_head: 22.8739, bptt_initial: 0.0424, tail: 0.0327, advantages_returns: 0.0036, losses: 0.0138 bptt: 0.6834 bptt_forward_core: 0.6739 update: 17.9093 clip: 0.0352 [2023-06-21 00:04:28,728][62782] RolloutWorker_w0 profile tree view: wait_for_trajectories: 0.0024, enqueue_policy_requests: 0.2131, env_step: 89.0550, overhead: 0.1938, complete_rollouts: 0.0033 save_policy_outputs: 0.0962 split_output_tensors: 0.0471 [2023-06-21 00:04:28,729][62782] RolloutWorker_w7 profile tree view: wait_for_trajectories: 0.0026, enqueue_policy_requests: 0.1817, env_step: 88.7206, overhead: 0.1951, complete_rollouts: 0.0031 save_policy_outputs: 0.0956 split_output_tensors: 0.0468 [2023-06-21 00:04:28,730][62782] Loop Runner_EvtLoop terminating... [2023-06-21 00:04:28,730][62782] Runner profile tree view: main_loop: 103.9782 [2023-06-21 00:04:28,730][62782] Collected {0: 2007040}, FPS: 1063.6 [2023-06-21 01:10:24,615][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 01:10:24,616][62782] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-21 01:10:24,617][62782] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-21 01:10:24,617][62782] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-21 01:10:24,617][62782] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 01:10:24,618][62782] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-21 01:10:24,618][62782] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 01:10:24,618][62782] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-21 01:10:24,619][62782] Adding new argument 'push_to_hub'=False that is not in the saved config file! [2023-06-21 01:10:24,619][62782] Adding new argument 'hf_repository'=None that is not in the saved config file! [2023-06-21 01:10:24,620][62782] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-21 01:10:24,620][62782] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-21 01:10:24,621][62782] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-21 01:10:24,621][62782] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-21 01:10:24,621][62782] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-21 01:10:24,637][62782] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 01:10:24,642][62782] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 01:10:24,657][62782] RunningMeanStd input shape: (1,) [2023-06-21 01:10:24,714][62782] ConvEncoder: input_channels=3 [2023-06-21 01:10:24,809][62782] Conv encoder output size: 512 [2023-06-21 01:10:24,809][62782] Policy head output size: 512 [2023-06-21 01:10:24,831][62782] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000490_2007040.pth... [2023-06-21 01:10:26,601][62782] Num frames 100... [2023-06-21 01:10:27,376][62782] Num frames 200... [2023-06-21 01:10:28,115][62782] Num frames 300... [2023-06-21 01:10:28,566][62782] Avg episode rewards: #0: 3.450, true rewards: #0: 3.450 [2023-06-21 01:10:28,569][62782] Avg episode reward: 3.450, avg true_objective: 3.450 [2023-06-21 01:10:28,988][62782] Num frames 400... [2023-06-21 01:10:29,799][62782] Num frames 500... [2023-06-21 01:10:30,594][62782] Num frames 600... [2023-06-21 01:10:31,385][62782] Num frames 700... [2023-06-21 01:10:32,175][62782] Num frames 800... [2023-06-21 01:10:32,923][62782] Num frames 900... [2023-06-21 01:10:33,649][62782] Avg episode rewards: #0: 6.925, true rewards: #0: 4.925 [2023-06-21 01:10:33,649][62782] Avg episode reward: 6.925, avg true_objective: 4.925 [2023-06-21 01:10:33,754][62782] Num frames 1000... [2023-06-21 01:10:34,512][62782] Num frames 1100... [2023-06-21 01:10:35,298][62782] Num frames 1200... [2023-06-21 01:10:36,117][62782] Num frames 1300... [2023-06-21 01:10:36,907][62782] Num frames 1400... [2023-06-21 01:10:37,689][62782] Num frames 1500... [2023-06-21 01:10:38,511][62782] Num frames 1600... [2023-06-21 01:10:39,326][62782] Num frames 1700... [2023-06-21 01:10:40,127][62782] Num frames 1800... [2023-06-21 01:10:40,908][62782] Num frames 1900... [2023-06-21 01:10:41,701][62782] Num frames 2000... [2023-06-21 01:10:42,373][62782] Avg episode rewards: #0: 11.243, true rewards: #0: 6.910 [2023-06-21 01:10:42,375][62782] Avg episode reward: 11.243, avg true_objective: 6.910 [2023-06-21 01:10:42,576][62782] Num frames 2100... [2023-06-21 01:10:43,384][62782] Num frames 2200... [2023-06-21 01:10:44,175][62782] Num frames 2300... [2023-06-21 01:10:44,944][62782] Num frames 2400... [2023-06-21 01:10:45,713][62782] Avg episode rewards: #0: 9.973, true rewards: #0: 6.222 [2023-06-21 01:10:45,715][62782] Avg episode reward: 9.973, avg true_objective: 6.222 [2023-06-21 01:10:45,808][62782] Num frames 2500... [2023-06-21 01:10:46,590][62782] Num frames 2600... [2023-06-21 01:10:47,418][62782] Num frames 2700... [2023-06-21 01:10:48,230][62782] Num frames 2800... [2023-06-21 01:10:49,030][62782] Num frames 2900... [2023-06-21 01:10:49,785][62782] Num frames 3000... [2023-06-21 01:10:49,889][62782] Avg episode rewards: #0: 9.802, true rewards: #0: 6.002 [2023-06-21 01:10:49,890][62782] Avg episode reward: 9.802, avg true_objective: 6.002 [2023-06-21 01:10:50,647][62782] Num frames 3100... [2023-06-21 01:10:51,401][62782] Num frames 3200... [2023-06-21 01:10:52,199][62782] Num frames 3300... [2023-06-21 01:10:52,936][62782] Num frames 3400... [2023-06-21 01:10:53,687][62782] Num frames 3500... [2023-06-21 01:10:54,480][62782] Num frames 3600... [2023-06-21 01:10:55,284][62782] Num frames 3700... [2023-06-21 01:10:56,092][62782] Num frames 3800... [2023-06-21 01:10:56,898][62782] Num frames 3900... [2023-06-21 01:10:57,610][62782] Num frames 4000... [2023-06-21 01:10:58,336][62782] Num frames 4100... [2023-06-21 01:10:59,123][62782] Num frames 4200... [2023-06-21 01:10:59,531][62782] Avg episode rewards: #0: 12.067, true rewards: #0: 7.067 [2023-06-21 01:10:59,531][62782] Avg episode reward: 12.067, avg true_objective: 7.067 [2023-06-21 01:10:59,947][62782] Num frames 4300... [2023-06-21 01:11:00,664][62782] Num frames 4400... [2023-06-21 01:11:01,403][62782] Num frames 4500... [2023-06-21 01:11:02,173][62782] Avg episode rewards: #0: 10.989, true rewards: #0: 6.560 [2023-06-21 01:11:02,175][62782] Avg episode reward: 10.989, avg true_objective: 6.560 [2023-06-21 01:11:02,239][62782] Num frames 4600... [2023-06-21 01:11:02,962][62782] Num frames 4700... [2023-06-21 01:11:03,733][62782] Num frames 4800... [2023-06-21 01:11:04,529][62782] Num frames 4900... [2023-06-21 01:11:05,337][62782] Num frames 5000... [2023-06-21 01:11:06,045][62782] Num frames 5100... [2023-06-21 01:11:06,749][62782] Num frames 5200... [2023-06-21 01:11:07,570][62782] Avg episode rewards: #0: 10.995, true rewards: #0: 6.620 [2023-06-21 01:11:07,573][62782] Avg episode reward: 10.995, avg true_objective: 6.620 [2023-06-21 01:11:07,606][62782] Num frames 5300... [2023-06-21 01:11:08,368][62782] Num frames 5400... [2023-06-21 01:11:09,164][62782] Num frames 5500... [2023-06-21 01:11:09,968][62782] Num frames 5600... [2023-06-21 01:11:10,739][62782] Num frames 5700... [2023-06-21 01:11:11,134][62782] Avg episode rewards: #0: 10.382, true rewards: #0: 6.382 [2023-06-21 01:11:11,135][62782] Avg episode reward: 10.382, avg true_objective: 6.382 [2023-06-21 01:11:11,541][62782] Num frames 5800... [2023-06-21 01:11:12,283][62782] Num frames 5900... [2023-06-21 01:11:13,055][62782] Num frames 6000... [2023-06-21 01:11:13,797][62782] Num frames 6100... [2023-06-21 01:11:14,421][62782] Avg episode rewards: #0: 10.167, true rewards: #0: 6.167 [2023-06-21 01:11:14,422][62782] Avg episode reward: 10.167, avg true_objective: 6.167 [2023-06-21 01:11:27,119][62782] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4! [2023-06-21 01:11:57,079][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 01:11:57,080][62782] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-21 01:11:57,081][62782] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-21 01:11:57,082][62782] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-21 01:11:57,082][62782] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 01:11:57,083][62782] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-21 01:11:57,083][62782] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! [2023-06-21 01:11:57,084][62782] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-21 01:11:57,085][62782] Adding new argument 'push_to_hub'=True that is not in the saved config file! [2023-06-21 01:11:57,086][62782] Adding new argument 'hf_repository'='mihirdeo16/vizdoom_health_gathering_supreme' that is not in the saved config file! [2023-06-21 01:11:57,087][62782] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-21 01:11:57,088][62782] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-21 01:11:57,089][62782] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-21 01:11:57,089][62782] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-21 01:11:57,090][62782] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-21 01:11:57,099][62782] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 01:11:57,101][62782] RunningMeanStd input shape: (1,) [2023-06-21 01:11:57,110][62782] ConvEncoder: input_channels=3 [2023-06-21 01:11:57,130][62782] Conv encoder output size: 512 [2023-06-21 01:11:57,131][62782] Policy head output size: 512 [2023-06-21 01:11:57,137][62782] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000490_2007040.pth... [2023-06-21 01:11:58,662][62782] Num frames 100... [2023-06-21 01:11:59,389][62782] Num frames 200... [2023-06-21 01:11:59,866][62782] Avg episode rewards: #0: 2.560, true rewards: #0: 2.560 [2023-06-21 01:11:59,868][62782] Avg episode reward: 2.560, avg true_objective: 2.560 [2023-06-21 01:12:00,215][62782] Num frames 300... [2023-06-21 01:12:01,006][62782] Num frames 400... [2023-06-21 01:12:01,793][62782] Num frames 500... [2023-06-21 01:12:02,570][62782] Num frames 600... [2023-06-21 01:12:03,365][62782] Num frames 700... [2023-06-21 01:12:03,485][62782] Avg episode rewards: #0: 4.020, true rewards: #0: 3.520 [2023-06-21 01:12:03,487][62782] Avg episode reward: 4.020, avg true_objective: 3.520 [2023-06-21 01:12:04,273][62782] Num frames 800... [2023-06-21 01:12:05,089][62782] Num frames 900... [2023-06-21 01:12:05,928][62782] Num frames 1000... [2023-06-21 01:12:06,695][62782] Num frames 1100... [2023-06-21 01:12:07,467][62782] Num frames 1200... [2023-06-21 01:12:07,901][62782] Avg episode rewards: #0: 6.160, true rewards: #0: 4.160 [2023-06-21 01:12:07,902][62782] Avg episode reward: 6.160, avg true_objective: 4.160 [2023-06-21 01:12:08,312][62782] Num frames 1300... [2023-06-21 01:12:09,102][62782] Num frames 1400... [2023-06-21 01:12:09,907][62782] Num frames 1500... [2023-06-21 01:12:10,724][62782] Num frames 1600... [2023-06-21 01:12:11,468][62782] Num frames 1700... [2023-06-21 01:12:12,185][62782] Num frames 1800... [2023-06-21 01:12:12,912][62782] Num frames 1900... [2023-06-21 01:12:13,717][62782] Num frames 2000... [2023-06-21 01:12:14,532][62782] Num frames 2100... [2023-06-21 01:12:15,315][62782] Num frames 2200... [2023-06-21 01:12:16,118][62782] Num frames 2300... [2023-06-21 01:12:16,243][62782] Avg episode rewards: #0: 10.760, true rewards: #0: 5.760 [2023-06-21 01:12:16,245][62782] Avg episode reward: 10.760, avg true_objective: 5.760 [2023-06-21 01:12:17,006][62782] Num frames 2400... [2023-06-21 01:12:17,747][62782] Num frames 2500... [2023-06-21 01:12:18,490][62782] Num frames 2600... [2023-06-21 01:12:19,233][62782] Avg episode rewards: #0: 9.376, true rewards: #0: 5.376 [2023-06-21 01:12:19,234][62782] Avg episode reward: 9.376, avg true_objective: 5.376 [2023-06-21 01:12:19,323][62782] Num frames 2700... [2023-06-21 01:12:20,069][62782] Num frames 2800... [2023-06-21 01:12:20,861][62782] Num frames 2900... [2023-06-21 01:12:21,650][62782] Num frames 3000... [2023-06-21 01:12:22,504][62782] Num frames 3100... [2023-06-21 01:12:22,942][62782] Avg episode rewards: #0: 8.727, true rewards: #0: 5.227 [2023-06-21 01:12:22,945][62782] Avg episode reward: 8.727, avg true_objective: 5.227 [2023-06-21 01:12:23,529][62782] Num frames 3200... [2023-06-21 01:12:24,445][62782] Num frames 3300... [2023-06-21 01:12:25,394][62782] Num frames 3400... [2023-06-21 01:12:26,338][62782] Num frames 3500... [2023-06-21 01:12:26,614][62782] Avg episode rewards: #0: 8.029, true rewards: #0: 5.029 [2023-06-21 01:12:26,617][62782] Avg episode reward: 8.029, avg true_objective: 5.029 [2023-06-21 01:12:27,245][62782] Num frames 3600... [2023-06-21 01:12:28,032][62782] Num frames 3700... [2023-06-21 01:12:28,928][62782] Num frames 3800... [2023-06-21 01:12:30,201][62782] Num frames 3900... [2023-06-21 01:12:31,224][62782] Num frames 4000... [2023-06-21 01:12:32,163][62782] Num frames 4100... [2023-06-21 01:12:32,910][62782] Num frames 4200... [2023-06-21 01:12:33,657][62782] Num frames 4300... [2023-06-21 01:12:34,468][62782] Num frames 4400... [2023-06-21 01:12:35,280][62782] Num frames 4500... [2023-06-21 01:12:36,099][62782] Num frames 4600... [2023-06-21 01:12:36,229][62782] Avg episode rewards: #0: 9.629, true rewards: #0: 5.754 [2023-06-21 01:12:36,230][62782] Avg episode reward: 9.629, avg true_objective: 5.754 [2023-06-21 01:12:37,020][62782] Num frames 4700... [2023-06-21 01:12:37,828][62782] Num frames 4800... [2023-06-21 01:12:38,618][62782] Num frames 4900... [2023-06-21 01:12:39,376][62782] Num frames 5000... [2023-06-21 01:12:40,129][62782] Num frames 5100... [2023-06-21 01:12:40,628][62782] Avg episode rewards: #0: 9.608, true rewards: #0: 5.719 [2023-06-21 01:12:40,629][62782] Avg episode reward: 9.608, avg true_objective: 5.719 [2023-06-21 01:12:41,107][62782] Num frames 5200... [2023-06-21 01:12:42,009][62782] Num frames 5300... [2023-06-21 01:12:42,929][62782] Num frames 5400... [2023-06-21 01:12:43,865][62782] Num frames 5500... [2023-06-21 01:12:44,767][62782] Num frames 5600... [2023-06-21 01:12:45,659][62782] Num frames 5700... [2023-06-21 01:12:46,594][62782] Num frames 5800... [2023-06-21 01:12:47,369][62782] Avg episode rewards: #0: 10.075, true rewards: #0: 5.875 [2023-06-21 01:12:47,371][62782] Avg episode reward: 10.075, avg true_objective: 5.875 [2023-06-21 01:12:55,232][62782] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4! [2023-06-21 01:15:12,180][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 01:15:12,181][62782] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-21 01:15:12,181][62782] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-21 01:15:12,181][62782] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-21 01:15:12,182][62782] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 01:15:12,182][62782] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-21 01:15:12,182][62782] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! [2023-06-21 01:15:12,182][62782] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-21 01:15:12,183][62782] Adding new argument 'push_to_hub'=True that is not in the saved config file! [2023-06-21 01:15:12,183][62782] Adding new argument 'hf_repository'='mihirdeo16/vizdoom_health_gathering_supreme' that is not in the saved config file! [2023-06-21 01:15:12,183][62782] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-21 01:15:12,184][62782] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-21 01:15:12,184][62782] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-21 01:15:12,184][62782] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-21 01:15:12,185][62782] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-21 01:15:12,192][62782] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 01:15:12,196][62782] RunningMeanStd input shape: (1,) [2023-06-21 01:15:12,232][62782] ConvEncoder: input_channels=3 [2023-06-21 01:15:12,256][62782] Conv encoder output size: 512 [2023-06-21 01:15:12,256][62782] Policy head output size: 512 [2023-06-21 01:15:12,274][62782] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000490_2007040.pth... [2023-06-21 01:15:14,407][62782] Num frames 100... [2023-06-21 01:15:15,193][62782] Num frames 200... [2023-06-21 01:15:15,999][62782] Num frames 300... [2023-06-21 01:15:16,784][62782] Num frames 400... [2023-06-21 01:15:16,996][62782] Avg episode rewards: #0: 6.160, true rewards: #0: 4.160 [2023-06-21 01:15:16,998][62782] Avg episode reward: 6.160, avg true_objective: 4.160 [2023-06-21 01:15:17,655][62782] Num frames 500... [2023-06-21 01:15:18,452][62782] Num frames 600... [2023-06-21 01:15:19,239][62782] Num frames 700... [2023-06-21 01:15:19,989][62782] Num frames 800... [2023-06-21 01:15:20,736][62782] Num frames 900... [2023-06-21 01:15:21,546][62782] Num frames 1000... [2023-06-21 01:15:22,311][62782] Num frames 1100... [2023-06-21 01:15:23,120][62782] Num frames 1200... [2023-06-21 01:15:23,934][62782] Num frames 1300... [2023-06-21 01:15:24,122][62782] Avg episode rewards: #0: 12.060, true rewards: #0: 6.560 [2023-06-21 01:15:24,123][62782] Avg episode reward: 12.060, avg true_objective: 6.560 [2023-06-21 01:15:24,863][62782] Num frames 1400... [2023-06-21 01:15:25,757][62782] Num frames 1500... [2023-06-21 01:15:26,617][62782] Num frames 1600... [2023-06-21 01:15:27,495][62782] Num frames 1700... [2023-06-21 01:15:28,454][62782] Num frames 1800... [2023-06-21 01:15:28,770][62782] Avg episode rewards: #0: 10.747, true rewards: #0: 6.080 [2023-06-21 01:15:28,773][62782] Avg episode reward: 10.747, avg true_objective: 6.080 [2023-06-21 01:15:29,474][62782] Num frames 1900... [2023-06-21 01:15:30,427][62782] Num frames 2000... [2023-06-21 01:15:31,384][62782] Num frames 2100... [2023-06-21 01:15:32,333][62782] Num frames 2200... [2023-06-21 01:15:33,250][62782] Num frames 2300... [2023-06-21 01:15:34,172][62782] Num frames 2400... [2023-06-21 01:15:35,121][62782] Num frames 2500... [2023-06-21 01:15:36,071][62782] Num frames 2600... [2023-06-21 01:15:37,006][62782] Num frames 2700... [2023-06-21 01:15:37,910][62782] Num frames 2800... [2023-06-21 01:15:38,836][62782] Num frames 2900... [2023-06-21 01:15:39,785][62782] Num frames 3000... [2023-06-21 01:15:40,202][62782] Avg episode rewards: #0: 14.100, true rewards: #0: 7.600 [2023-06-21 01:15:40,204][62782] Avg episode reward: 14.100, avg true_objective: 7.600 [2023-06-21 01:15:40,708][62782] Num frames 3100... [2023-06-21 01:15:41,612][62782] Num frames 3200... [2023-06-21 01:15:42,567][62782] Num frames 3300... [2023-06-21 01:15:43,508][62782] Num frames 3400... [2023-06-21 01:15:44,338][62782] Num frames 3500... [2023-06-21 01:15:45,255][62782] Num frames 3600... [2023-06-21 01:15:46,151][62782] Num frames 3700... [2023-06-21 01:15:46,679][62782] Avg episode rewards: #0: 13.888, true rewards: #0: 7.488 [2023-06-21 01:15:46,682][62782] Avg episode reward: 13.888, avg true_objective: 7.488 [2023-06-21 01:15:47,190][62782] Num frames 3800... [2023-06-21 01:15:48,133][62782] Num frames 3900... [2023-06-21 01:15:49,082][62782] Num frames 4000... [2023-06-21 01:15:49,955][62782] Num frames 4100... [2023-06-21 01:15:50,871][62782] Num frames 4200... [2023-06-21 01:15:51,719][62782] Num frames 4300... [2023-06-21 01:15:52,254][62782] Avg episode rewards: #0: 13.087, true rewards: #0: 7.253 [2023-06-21 01:15:52,256][62782] Avg episode reward: 13.087, avg true_objective: 7.253 [2023-06-21 01:15:52,649][62782] Num frames 4400... [2023-06-21 01:15:53,515][62782] Num frames 4500... [2023-06-21 01:15:54,394][62782] Num frames 4600... [2023-06-21 01:15:55,283][62782] Num frames 4700... [2023-06-21 01:15:56,214][62782] Num frames 4800... [2023-06-21 01:15:57,103][62782] Num frames 4900... [2023-06-21 01:15:57,989][62782] Num frames 5000... [2023-06-21 01:15:58,885][62782] Num frames 5100... [2023-06-21 01:15:59,762][62782] Num frames 5200... [2023-06-21 01:16:00,556][62782] Avg episode rewards: #0: 14.114, true rewards: #0: 7.543 [2023-06-21 01:16:00,558][62782] Avg episode reward: 14.114, avg true_objective: 7.543 [2023-06-21 01:16:00,741][62782] Num frames 5300... [2023-06-21 01:16:01,649][62782] Num frames 5400... [2023-06-21 01:16:02,582][62782] Num frames 5500... [2023-06-21 01:16:03,490][62782] Num frames 5600... [2023-06-21 01:16:04,383][62782] Num frames 5700... [2023-06-21 01:16:05,260][62782] Num frames 5800... [2023-06-21 01:16:05,579][62782] Avg episode rewards: #0: 13.280, true rewards: #0: 7.280 [2023-06-21 01:16:05,581][62782] Avg episode reward: 13.280, avg true_objective: 7.280 [2023-06-21 01:16:06,211][62782] Num frames 5900... [2023-06-21 01:16:06,936][62782] Num frames 6000... [2023-06-21 01:16:07,685][62782] Num frames 6100... [2023-06-21 01:16:08,455][62782] Num frames 6200... [2023-06-21 01:16:09,272][62782] Num frames 6300... [2023-06-21 01:16:09,637][62782] Avg episode rewards: #0: 12.707, true rewards: #0: 7.040 [2023-06-21 01:16:09,640][62782] Avg episode reward: 12.707, avg true_objective: 7.040 [2023-06-21 01:16:10,134][62782] Num frames 6400... [2023-06-21 01:16:10,955][62782] Num frames 6500... [2023-06-21 01:16:11,757][62782] Num frames 6600... [2023-06-21 01:16:12,562][62782] Num frames 6700... [2023-06-21 01:16:13,391][62782] Num frames 6800... [2023-06-21 01:16:14,116][62782] Num frames 6900... [2023-06-21 01:16:14,304][62782] Avg episode rewards: #0: 12.412, true rewards: #0: 6.912 [2023-06-21 01:16:14,307][62782] Avg episode reward: 12.412, avg true_objective: 6.912 [2023-06-21 01:16:23,345][62782] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4! [2023-06-21 01:16:44,521][62782] The model has been pushed to https://huggingface.co/mihirdeo16/vizdoom_health_gathering_supreme [2023-06-21 10:43:01,450][62782] Environment doom_basic already registered, overwriting... [2023-06-21 10:43:01,453][62782] Environment doom_two_colors_easy already registered, overwriting... [2023-06-21 10:43:01,454][62782] Environment doom_two_colors_hard already registered, overwriting... [2023-06-21 10:43:01,454][62782] Environment doom_dm already registered, overwriting... [2023-06-21 10:43:01,454][62782] Environment doom_dwango5 already registered, overwriting... [2023-06-21 10:43:01,455][62782] Environment doom_my_way_home_flat_actions already registered, overwriting... [2023-06-21 10:43:01,455][62782] Environment doom_defend_the_center_flat_actions already registered, overwriting... [2023-06-21 10:43:01,455][62782] Environment doom_my_way_home already registered, overwriting... [2023-06-21 10:43:01,456][62782] Environment doom_deadly_corridor already registered, overwriting... [2023-06-21 10:43:01,456][62782] Environment doom_defend_the_center already registered, overwriting... [2023-06-21 10:43:01,456][62782] Environment doom_defend_the_line already registered, overwriting... [2023-06-21 10:43:01,457][62782] Environment doom_health_gathering already registered, overwriting... [2023-06-21 10:43:01,457][62782] Environment doom_health_gathering_supreme already registered, overwriting... [2023-06-21 10:43:01,457][62782] Environment doom_battle already registered, overwriting... [2023-06-21 10:43:01,458][62782] Environment doom_battle2 already registered, overwriting... [2023-06-21 10:43:01,458][62782] Environment doom_duel_bots already registered, overwriting... [2023-06-21 10:43:01,459][62782] Environment doom_deathmatch_bots already registered, overwriting... [2023-06-21 10:43:01,459][62782] Environment doom_duel already registered, overwriting... [2023-06-21 10:43:01,459][62782] Environment doom_deathmatch_full already registered, overwriting... [2023-06-21 10:43:01,460][62782] Environment doom_benchmark already registered, overwriting... [2023-06-21 10:43:01,460][62782] register_encoder_factory: [2023-06-21 10:43:01,522][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 10:43:01,523][62782] Overriding arg 'train_for_env_steps' with value 4000000 passed from command line [2023-06-21 10:43:01,535][62782] Experiment dir /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment already exists! [2023-06-21 10:43:01,536][62782] Resuming existing experiment from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment... [2023-06-21 10:43:01,536][62782] Weights and Biases integration disabled [2023-06-21 10:43:01,542][62782] Environment var CUDA_VISIBLE_DEVICES is [2023-06-21 10:43:02,923][62782] Starting experiment with the following configuration: help=False algo=APPO env=doom_health_gathering_supreme experiment=default_experiment train_dir=/Users/md/Code/python/jubilant-memory/RL/train_dir restart_behavior=resume device=cpu seed=None num_policies=1 async_rl=True serial_mode=False batched_sampling=False num_batches_to_accumulate=2 worker_num_splits=2 policy_workers_per_policy=1 max_policy_lag=1000 num_workers=8 num_envs_per_worker=4 batch_size=1024 num_batches_per_epoch=1 num_epochs=1 rollout=32 recurrence=32 shuffle_minibatches=False gamma=0.99 reward_scale=1.0 reward_clip=1000.0 value_bootstrap=False normalize_returns=True exploration_loss_coeff=0.001 value_loss_coeff=0.5 kl_loss_coeff=0.0 exploration_loss=symmetric_kl gae_lambda=0.95 ppo_clip_ratio=0.1 ppo_clip_value=0.2 with_vtrace=False vtrace_rho=1.0 vtrace_c=1.0 optimizer=adam adam_eps=1e-06 adam_beta1=0.9 adam_beta2=0.999 max_grad_norm=4.0 learning_rate=0.0001 lr_schedule=constant lr_schedule_kl_threshold=0.008 lr_adaptive_min=1e-06 lr_adaptive_max=0.01 obs_subtract_mean=0.0 obs_scale=255.0 normalize_input=True normalize_input_keys=None decorrelate_experience_max_seconds=0 decorrelate_envs_on_one_worker=True actor_worker_gpus=[] set_workers_cpu_affinity=True force_envs_single_thread=False default_niceness=0 log_to_file=True experiment_summaries_interval=10 flush_summaries_interval=30 stats_avg=100 summaries_use_frameskip=True heartbeat_interval=20 heartbeat_reporting_interval=600 train_for_env_steps=4000000 train_for_seconds=10000000000 save_every_sec=120 keep_checkpoints=2 load_checkpoint_kind=latest save_milestones_sec=-1 save_best_every_sec=5 save_best_metric=reward save_best_after=100000 benchmark=False encoder_mlp_layers=[512, 512] encoder_conv_architecture=convnet_simple encoder_conv_mlp_layers=[512] use_rnn=True rnn_size=512 rnn_type=gru rnn_num_layers=1 decoder_mlp_layers=[] nonlinearity=elu policy_initialization=orthogonal policy_init_gain=1.0 actor_critic_share_weights=True adaptive_stddev=True continuous_tanh_scale=0.0 initial_stddev=1.0 use_env_info_cache=False env_gpu_actions=False env_gpu_observations=True env_frameskip=4 env_framestack=1 pixel_format=CHW use_record_episode_statistics=False with_wandb=False wandb_user=None wandb_project=sample_factory wandb_group=None wandb_job_type=SF wandb_tags=[] with_pbt=False pbt_mix_policies_in_one_env=True pbt_period_env_steps=5000000 pbt_start_mutation=20000000 pbt_replace_fraction=0.3 pbt_mutation_rate=0.15 pbt_replace_reward_gap=0.1 pbt_replace_reward_gap_absolute=1e-06 pbt_optimize_gamma=False pbt_target_objective=true_objective pbt_perturb_min=1.1 pbt_perturb_max=1.5 num_agents=-1 num_humans=0 num_bots=-1 start_bot_difficulty=None timelimit=None res_w=128 res_h=72 wide_aspect_ratio=False eval_env_frameskip=1 fps=35 command_line=--env=doom_health_gathering_supreme --num_workers=8 --num_envs_per_worker=4 --train_for_env_steps=4000000 cli_args={'env': 'doom_health_gathering_supreme', 'num_workers': 8, 'num_envs_per_worker': 4, 'train_for_env_steps': 4000000} git_hash=02ef46648112e6fd3adc4475dfd889e784c0ef87 git_repo_name=https://github.com/mihirdeo16/jubilant-memory.git [2023-06-21 10:43:02,925][62782] Saving configuration to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json... [2023-06-21 10:43:02,950][62782] Rollout worker 0 uses device cpu [2023-06-21 10:43:02,951][62782] Rollout worker 1 uses device cpu [2023-06-21 10:43:02,951][62782] Rollout worker 2 uses device cpu [2023-06-21 10:43:02,951][62782] Rollout worker 3 uses device cpu [2023-06-21 10:43:02,952][62782] Rollout worker 4 uses device cpu [2023-06-21 10:43:02,952][62782] Rollout worker 5 uses device cpu [2023-06-21 10:43:02,952][62782] Rollout worker 6 uses device cpu [2023-06-21 10:43:02,953][62782] Rollout worker 7 uses device cpu [2023-06-21 10:43:03,010][62782] InferenceWorker_p0-w0: min num requests: 2 [2023-06-21 10:43:03,047][62782] Starting all processes... [2023-06-21 10:43:03,048][62782] Starting process learner_proc0 [2023-06-21 10:43:03,103][62782] Starting all processes... [2023-06-21 10:43:03,116][62782] Starting process inference_proc0-0 [2023-06-21 10:43:03,119][62782] Starting process rollout_proc0 [2023-06-21 10:43:03,119][62782] Starting process rollout_proc1 [2023-06-21 10:43:03,120][62782] Starting process rollout_proc2 [2023-06-21 10:43:03,120][62782] Starting process rollout_proc3 [2023-06-21 10:43:03,120][62782] Starting process rollout_proc4 [2023-06-21 10:43:03,121][62782] Starting process rollout_proc5 [2023-06-21 10:43:03,121][62782] Starting process rollout_proc6 [2023-06-21 10:43:03,121][62782] Starting process rollout_proc7 [2023-06-21 10:43:05,405][69877] On MacOS, not setting affinity [2023-06-21 10:43:05,406][69876] Starting seed is not provided [2023-06-21 10:43:05,407][69876] Initializing actor-critic model on device cpu [2023-06-21 10:43:05,407][69876] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 10:43:05,410][69876] RunningMeanStd input shape: (1,) [2023-06-21 10:43:05,421][69876] ConvEncoder: input_channels=3 [2023-06-21 10:43:05,428][69883] On MacOS, not setting affinity [2023-06-21 10:43:05,463][69880] On MacOS, not setting affinity [2023-06-21 10:43:05,479][69879] On MacOS, not setting affinity [2023-06-21 10:43:05,550][69876] Conv encoder output size: 512 [2023-06-21 10:43:05,553][69876] Policy head output size: 512 [2023-06-21 10:43:05,559][69881] On MacOS, not setting affinity [2023-06-21 10:43:05,559][69885] On MacOS, not setting affinity [2023-06-21 10:43:05,559][69882] On MacOS, not setting affinity [2023-06-21 10:43:05,566][69884] On MacOS, not setting affinity [2023-06-21 10:43:05,571][69876] Created Actor Critic model with architecture: [2023-06-21 10:43:05,572][69876] ActorCriticSharedWeights( (obs_normalizer): ObservationNormalizer( (running_mean_std): RunningMeanStdDictInPlace( (running_mean_std): ModuleDict( (obs): RunningMeanStdInPlace() ) ) ) (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace) (encoder): VizdoomEncoder( (basic_encoder): ConvEncoder( (enc): RecursiveScriptModule( original_name=ConvEncoderImpl (conv_head): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Conv2d) (1): RecursiveScriptModule(original_name=ELU) (2): RecursiveScriptModule(original_name=Conv2d) (3): RecursiveScriptModule(original_name=ELU) (4): RecursiveScriptModule(original_name=Conv2d) (5): RecursiveScriptModule(original_name=ELU) ) (mlp_layers): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Linear) (1): RecursiveScriptModule(original_name=ELU) ) ) ) ) (core): ModelCoreRNN( (core): GRU(512, 512) ) (decoder): MlpDecoder( (mlp): Identity() ) (critic_linear): Linear(in_features=512, out_features=1, bias=True) (action_parameterization): ActionParameterizationDefault( (distribution_linear): Linear(in_features=512, out_features=5, bias=True) ) ) [2023-06-21 10:43:05,575][69876] Using optimizer [2023-06-21 10:43:05,576][69876] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000490_2007040.pth... [2023-06-21 10:43:05,621][69876] Loading model from checkpoint [2023-06-21 10:43:05,631][69876] Loaded experiment state at self.train_step=490, self.env_steps=2007040 [2023-06-21 10:43:05,638][69876] Initialized policy 0 weights for model version 490 [2023-06-21 10:43:05,645][69876] LearnerWorker_p0 finished initialization! [2023-06-21 10:43:05,651][69878] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 10:43:05,653][69878] RunningMeanStd input shape: (1,) [2023-06-21 10:43:05,668][69878] ConvEncoder: input_channels=3 [2023-06-21 10:43:05,714][69878] Conv encoder output size: 512 [2023-06-21 10:43:05,714][69878] Policy head output size: 512 [2023-06-21 10:43:05,721][62782] Inference worker 0-0 is ready! [2023-06-21 10:43:05,722][62782] All inference workers are ready! Signal rollout workers to start! [2023-06-21 10:43:05,758][69879] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 10:43:05,761][69882] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 10:43:05,761][69881] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 10:43:05,762][69885] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 10:43:05,763][69877] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 10:43:05,764][69884] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 10:43:05,764][69880] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 10:43:05,765][69883] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 10:43:06,547][62782] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 2007040. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 10:43:07,413][69885] Decorrelating experience for 0 frames... [2023-06-21 10:43:07,413][69877] Decorrelating experience for 0 frames... [2023-06-21 10:43:07,413][69880] Decorrelating experience for 0 frames... [2023-06-21 10:43:07,438][69884] Decorrelating experience for 0 frames... [2023-06-21 10:43:07,438][69881] Decorrelating experience for 0 frames... [2023-06-21 10:43:07,439][69879] Decorrelating experience for 0 frames... [2023-06-21 10:43:08,448][69880] Decorrelating experience for 32 frames... [2023-06-21 10:43:08,448][69877] Decorrelating experience for 32 frames... [2023-06-21 10:43:08,449][69884] Decorrelating experience for 32 frames... [2023-06-21 10:43:08,449][69885] Decorrelating experience for 32 frames... [2023-06-21 10:43:08,455][69883] Decorrelating experience for 0 frames... [2023-06-21 10:43:08,456][69881] Decorrelating experience for 32 frames... [2023-06-21 10:43:08,460][69879] Decorrelating experience for 32 frames... [2023-06-21 10:43:08,461][69882] Decorrelating experience for 0 frames... [2023-06-21 10:43:09,150][69883] Decorrelating experience for 32 frames... [2023-06-21 10:43:09,151][69882] Decorrelating experience for 32 frames... [2023-06-21 10:43:10,176][69879] Decorrelating experience for 64 frames... [2023-06-21 10:43:10,182][69881] Decorrelating experience for 64 frames... [2023-06-21 10:43:10,184][69880] Decorrelating experience for 64 frames... [2023-06-21 10:43:10,198][69884] Decorrelating experience for 64 frames... [2023-06-21 10:43:10,634][69883] Decorrelating experience for 64 frames... [2023-06-21 10:43:10,640][69882] Decorrelating experience for 64 frames... [2023-06-21 10:43:10,844][69877] Decorrelating experience for 64 frames... [2023-06-21 10:43:11,545][62782] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 2007040. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 10:43:11,552][69885] Decorrelating experience for 64 frames... [2023-06-21 10:43:12,482][69881] Decorrelating experience for 96 frames... [2023-06-21 10:43:12,482][69884] Decorrelating experience for 96 frames... [2023-06-21 10:43:12,483][69880] Decorrelating experience for 96 frames... [2023-06-21 10:43:12,922][69882] Decorrelating experience for 96 frames... [2023-06-21 10:43:12,923][69883] Decorrelating experience for 96 frames... [2023-06-21 10:43:13,083][69877] Decorrelating experience for 96 frames... [2023-06-21 10:43:13,165][69879] Decorrelating experience for 96 frames... [2023-06-21 10:43:13,776][69885] Decorrelating experience for 96 frames... [2023-06-21 10:43:16,547][62782] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 2007040. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 10:43:16,550][62782] Avg episode reward: [(0, '0.320')] [2023-06-21 10:43:21,548][62782] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 2007040. Throughput: 0: 113.1. Samples: 1696. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 10:43:21,553][62782] Avg episode reward: [(0, '1.473')] [2023-06-21 10:43:22,987][62782] Heartbeat connected on Batcher_0 [2023-06-21 10:43:23,024][62782] Heartbeat connected on InferenceWorker_p0-w0 [2023-06-21 10:43:23,092][62782] Heartbeat connected on RolloutWorker_w3 [2023-06-21 10:43:23,097][62782] Heartbeat connected on RolloutWorker_w0 [2023-06-21 10:43:23,102][62782] Heartbeat connected on RolloutWorker_w5 [2023-06-21 10:43:23,103][62782] Heartbeat connected on RolloutWorker_w4 [2023-06-21 10:43:23,110][62782] Heartbeat connected on RolloutWorker_w2 [2023-06-21 10:43:23,114][62782] Heartbeat connected on RolloutWorker_w1 [2023-06-21 10:43:23,117][62782] Heartbeat connected on RolloutWorker_w6 [2023-06-21 10:43:23,126][62782] Heartbeat connected on RolloutWorker_w7 [2023-06-21 10:43:25,135][62782] Heartbeat connected on LearnerWorker_p0 [2023-06-21 10:43:26,547][62782] Fps is (10 sec: 819.1, 60 sec: 409.6, 300 sec: 409.6). Total num frames: 2015232. Throughput: 0: 132.1. Samples: 2642. Policy #0 lag: (min: 0.0, avg: 0.0, max: 0.0) [2023-06-21 10:43:26,549][62782] Avg episode reward: [(0, '3.037')] [2023-06-21 10:43:31,548][62782] Fps is (10 sec: 1638.4, 60 sec: 655.3, 300 sec: 655.3). Total num frames: 2023424. Throughput: 0: 181.2. Samples: 4530. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:43:31,551][62782] Avg episode reward: [(0, '5.303')] [2023-06-21 10:43:36,546][62782] Fps is (10 sec: 1228.9, 60 sec: 682.7, 300 sec: 682.7). Total num frames: 2027520. Throughput: 0: 213.6. Samples: 6408. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:43:36,554][62782] Avg episode reward: [(0, '6.612')] [2023-06-21 10:43:41,549][62782] Fps is (10 sec: 819.2, 60 sec: 702.2, 300 sec: 702.2). Total num frames: 2031616. Throughput: 0: 210.2. Samples: 7356. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:43:41,575][62782] Avg episode reward: [(0, '7.762')] [2023-06-21 10:43:46,547][62782] Fps is (10 sec: 1228.7, 60 sec: 819.2, 300 sec: 819.2). Total num frames: 2039808. Throughput: 0: 231.1. Samples: 9246. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:43:46,551][62782] Avg episode reward: [(0, '7.826')] [2023-06-21 10:43:50,776][69878] Updated weights for policy 0, policy_version 500 (0.0021) [2023-06-21 10:43:51,547][62782] Fps is (10 sec: 1638.5, 60 sec: 910.2, 300 sec: 910.2). Total num frames: 2048000. Throughput: 0: 247.7. Samples: 11148. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:43:51,548][62782] Avg episode reward: [(0, '8.799')] [2023-06-21 10:43:56,547][62782] Fps is (10 sec: 1228.9, 60 sec: 901.1, 300 sec: 901.1). Total num frames: 2052096. Throughput: 0: 268.4. Samples: 12080. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:43:56,554][62782] Avg episode reward: [(0, '9.228')] [2023-06-21 10:44:01,545][62782] Fps is (10 sec: 1229.0, 60 sec: 968.2, 300 sec: 968.2). Total num frames: 2060288. Throughput: 0: 310.5. Samples: 13972. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:44:01,548][62782] Avg episode reward: [(0, '9.521')] [2023-06-21 10:44:06,548][62782] Fps is (10 sec: 1228.7, 60 sec: 955.7, 300 sec: 955.7). Total num frames: 2064384. Throughput: 0: 313.7. Samples: 15814. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:44:06,551][62782] Avg episode reward: [(0, '9.525')] [2023-06-21 10:44:11,547][62782] Fps is (10 sec: 1228.6, 60 sec: 1092.2, 300 sec: 1008.2). Total num frames: 2072576. Throughput: 0: 314.0. Samples: 16772. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:44:11,550][62782] Avg episode reward: [(0, '9.922')] [2023-06-21 10:44:16,545][62782] Fps is (10 sec: 1229.2, 60 sec: 1160.6, 300 sec: 994.8). Total num frames: 2076672. Throughput: 0: 314.1. Samples: 18662. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:44:16,547][62782] Avg episode reward: [(0, '11.170')] [2023-06-21 10:44:16,866][69876] Saving new best policy, reward=11.170! [2023-06-21 10:44:21,545][62782] Fps is (10 sec: 1229.0, 60 sec: 1297.1, 300 sec: 1037.7). Total num frames: 2084864. Throughput: 0: 314.7. Samples: 20568. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:44:21,546][62782] Avg episode reward: [(0, '12.319')] [2023-06-21 10:44:23,312][69876] Saving new best policy, reward=12.319! [2023-06-21 10:44:23,326][69878] Updated weights for policy 0, policy_version 510 (0.0018) [2023-06-21 10:44:26,552][62782] Fps is (10 sec: 1227.9, 60 sec: 1228.7, 300 sec: 1023.9). Total num frames: 2088960. Throughput: 0: 314.7. Samples: 21518. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:44:26,553][62782] Avg episode reward: [(0, '12.784')] [2023-06-21 10:44:26,650][69876] Saving new best policy, reward=12.784! [2023-06-21 10:44:31,546][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1060.2). Total num frames: 2097152. Throughput: 0: 315.0. Samples: 23422. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:44:31,549][62782] Avg episode reward: [(0, '12.690')] [2023-06-21 10:44:36,547][62782] Fps is (10 sec: 1229.4, 60 sec: 1228.8, 300 sec: 1046.8). Total num frames: 2101248. Throughput: 0: 314.5. Samples: 25302. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:44:36,552][62782] Avg episode reward: [(0, '12.506')] [2023-06-21 10:44:41,545][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1077.9). Total num frames: 2109440. Throughput: 0: 314.8. Samples: 26246. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:44:41,547][62782] Avg episode reward: [(0, '12.316')] [2023-06-21 10:44:46,547][62782] Fps is (10 sec: 1638.4, 60 sec: 1297.1, 300 sec: 1105.9). Total num frames: 2117632. Throughput: 0: 313.8. Samples: 28092. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:44:46,549][62782] Avg episode reward: [(0, '12.021')] [2023-06-21 10:44:51,545][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1092.3). Total num frames: 2121728. Throughput: 0: 315.0. Samples: 29990. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:44:51,546][62782] Avg episode reward: [(0, '11.554')] [2023-06-21 10:44:56,147][69878] Updated weights for policy 0, policy_version 520 (0.0014) [2023-06-21 10:44:56,546][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1117.1). Total num frames: 2129920. Throughput: 0: 314.8. Samples: 30938. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:44:56,547][62782] Avg episode reward: [(0, '11.596')] [2023-06-21 10:45:01,546][62782] Fps is (10 sec: 1228.6, 60 sec: 1228.8, 300 sec: 1104.1). Total num frames: 2134016. Throughput: 0: 314.6. Samples: 32820. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:45:01,549][62782] Avg episode reward: [(0, '11.180')] [2023-06-21 10:45:02,719][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000522_2138112.pth... [2023-06-21 10:45:02,815][69876] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000463_1896448.pth [2023-06-21 10:45:06,546][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1126.4). Total num frames: 2142208. Throughput: 0: 314.4. Samples: 34718. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:45:06,548][62782] Avg episode reward: [(0, '10.977')] [2023-06-21 10:45:11,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1114.1). Total num frames: 2146304. Throughput: 0: 314.5. Samples: 35668. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:45:11,552][62782] Avg episode reward: [(0, '10.181')] [2023-06-21 10:45:16,546][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.0, 300 sec: 1134.3). Total num frames: 2154496. Throughput: 0: 314.4. Samples: 37570. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:45:16,547][62782] Avg episode reward: [(0, '10.501')] [2023-06-21 10:45:21,548][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1122.6). Total num frames: 2158592. Throughput: 0: 314.6. Samples: 39458. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:45:21,554][62782] Avg episode reward: [(0, '10.572')] [2023-06-21 10:45:26,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.2, 300 sec: 1141.0). Total num frames: 2166784. Throughput: 0: 314.2. Samples: 40386. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:45:26,549][62782] Avg episode reward: [(0, '10.544')] [2023-06-21 10:45:28,980][69878] Updated weights for policy 0, policy_version 530 (0.0015) [2023-06-21 10:45:31,547][62782] Fps is (10 sec: 1228.9, 60 sec: 1228.8, 300 sec: 1129.9). Total num frames: 2170880. Throughput: 0: 315.2. Samples: 42278. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:45:31,556][62782] Avg episode reward: [(0, '10.948')] [2023-06-21 10:45:36,548][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1146.9). Total num frames: 2179072. Throughput: 0: 314.8. Samples: 44156. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:45:36,551][62782] Avg episode reward: [(0, '11.408')] [2023-06-21 10:45:41,552][62782] Fps is (10 sec: 1228.3, 60 sec: 1228.7, 300 sec: 1136.3). Total num frames: 2183168. Throughput: 0: 313.5. Samples: 45046. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:45:41,564][62782] Avg episode reward: [(0, '11.296')] [2023-06-21 10:45:46,545][62782] Fps is (10 sec: 1229.2, 60 sec: 1228.8, 300 sec: 1152.0). Total num frames: 2191360. Throughput: 0: 313.1. Samples: 46910. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:45:46,547][62782] Avg episode reward: [(0, '11.594')] [2023-06-21 10:45:51,547][62782] Fps is (10 sec: 1229.2, 60 sec: 1228.8, 300 sec: 1141.9). Total num frames: 2195456. Throughput: 0: 313.4. Samples: 48820. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:45:51,550][62782] Avg episode reward: [(0, '11.565')] [2023-06-21 10:45:56,547][62782] Fps is (10 sec: 1228.6, 60 sec: 1228.8, 300 sec: 1156.5). Total num frames: 2203648. Throughput: 0: 313.5. Samples: 49776. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:45:56,550][62782] Avg episode reward: [(0, '11.959')] [2023-06-21 10:46:01,463][69878] Updated weights for policy 0, policy_version 540 (0.0017) [2023-06-21 10:46:01,550][62782] Fps is (10 sec: 1638.1, 60 sec: 1297.0, 300 sec: 1170.3). Total num frames: 2211840. Throughput: 0: 313.5. Samples: 51680. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:46:01,565][62782] Avg episode reward: [(0, '12.728')] [2023-06-21 10:46:06,546][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1160.5). Total num frames: 2215936. Throughput: 0: 314.0. Samples: 53588. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:46:06,548][62782] Avg episode reward: [(0, '12.787')] [2023-06-21 10:46:07,992][69876] Saving new best policy, reward=12.787! [2023-06-21 10:46:11,546][62782] Fps is (10 sec: 1229.1, 60 sec: 1297.1, 300 sec: 1173.5). Total num frames: 2224128. Throughput: 0: 314.4. Samples: 54534. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:46:11,549][62782] Avg episode reward: [(0, '12.692')] [2023-06-21 10:46:16,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1164.1). Total num frames: 2228224. Throughput: 0: 314.7. Samples: 56440. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:46:16,551][62782] Avg episode reward: [(0, '12.702')] [2023-06-21 10:46:21,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1176.3). Total num frames: 2236416. Throughput: 0: 315.2. Samples: 58338. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:46:21,551][62782] Avg episode reward: [(0, '13.728')] [2023-06-21 10:46:21,555][69876] Saving new best policy, reward=13.728! [2023-06-21 10:46:26,548][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1167.4). Total num frames: 2240512. Throughput: 0: 316.6. Samples: 59292. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:46:26,561][62782] Avg episode reward: [(0, '13.988')] [2023-06-21 10:46:27,227][69876] Saving new best policy, reward=13.988! [2023-06-21 10:46:31,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1178.8). Total num frames: 2248704. Throughput: 0: 317.5. Samples: 61198. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:46:31,550][62782] Avg episode reward: [(0, '13.951')] [2023-06-21 10:46:33,717][69878] Updated weights for policy 0, policy_version 550 (0.0012) [2023-06-21 10:46:36,544][62782] Fps is (10 sec: 1229.2, 60 sec: 1228.9, 300 sec: 1170.3). Total num frames: 2252800. Throughput: 0: 317.2. Samples: 63092. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:46:36,547][62782] Avg episode reward: [(0, '13.768')] [2023-06-21 10:46:41,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.2, 300 sec: 1181.2). Total num frames: 2260992. Throughput: 0: 317.0. Samples: 64040. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:46:41,551][62782] Avg episode reward: [(0, '13.769')] [2023-06-21 10:46:46,547][62782] Fps is (10 sec: 1228.4, 60 sec: 1228.8, 300 sec: 1172.9). Total num frames: 2265088. Throughput: 0: 317.0. Samples: 65946. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:46:46,550][62782] Avg episode reward: [(0, '14.008')] [2023-06-21 10:46:46,551][69876] Saving new best policy, reward=14.008! [2023-06-21 10:46:51,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1183.3). Total num frames: 2273280. Throughput: 0: 316.9. Samples: 67848. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:46:51,551][62782] Avg episode reward: [(0, '14.787')] [2023-06-21 10:46:53,094][69876] Saving new best policy, reward=14.787! [2023-06-21 10:46:56,542][62782] Fps is (10 sec: 1639.2, 60 sec: 1297.2, 300 sec: 1193.2). Total num frames: 2281472. Throughput: 0: 316.9. Samples: 68792. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:46:56,545][62782] Avg episode reward: [(0, '15.735')] [2023-06-21 10:46:56,546][69876] Saving new best policy, reward=15.735! [2023-06-21 10:47:01,548][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1185.2). Total num frames: 2285568. Throughput: 0: 316.8. Samples: 70694. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:47:01,551][62782] Avg episode reward: [(0, '16.099')] [2023-06-21 10:47:02,884][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000559_2289664.pth... [2023-06-21 10:47:02,978][69876] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000490_2007040.pth [2023-06-21 10:47:03,009][69876] Saving new best policy, reward=16.099! [2023-06-21 10:47:06,106][69878] Updated weights for policy 0, policy_version 560 (0.0024) [2023-06-21 10:47:06,542][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.2, 300 sec: 1194.7). Total num frames: 2293760. Throughput: 0: 316.7. Samples: 72588. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:47:06,544][62782] Avg episode reward: [(0, '15.229')] [2023-06-21 10:47:11,547][62782] Fps is (10 sec: 1228.9, 60 sec: 1228.8, 300 sec: 1187.0). Total num frames: 2297856. Throughput: 0: 316.7. Samples: 73544. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:47:11,550][62782] Avg episode reward: [(0, '14.760')] [2023-06-21 10:47:16,547][62782] Fps is (10 sec: 1228.2, 60 sec: 1297.1, 300 sec: 1196.0). Total num frames: 2306048. Throughput: 0: 316.5. Samples: 75442. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:47:16,549][62782] Avg episode reward: [(0, '14.537')] [2023-06-21 10:47:21,548][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1188.6). Total num frames: 2310144. Throughput: 0: 317.0. Samples: 77356. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:47:21,552][62782] Avg episode reward: [(0, '14.045')] [2023-06-21 10:47:26,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1197.3). Total num frames: 2318336. Throughput: 0: 317.0. Samples: 78304. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:47:26,551][62782] Avg episode reward: [(0, '14.430')] [2023-06-21 10:47:31,548][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1190.2). Total num frames: 2322432. Throughput: 0: 316.9. Samples: 80206. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:47:31,551][62782] Avg episode reward: [(0, '14.034')] [2023-06-21 10:47:36,546][62782] Fps is (10 sec: 1229.0, 60 sec: 1297.0, 300 sec: 1198.5). Total num frames: 2330624. Throughput: 0: 316.9. Samples: 82108. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:47:36,548][62782] Avg episode reward: [(0, '14.555')] [2023-06-21 10:47:38,429][69878] Updated weights for policy 0, policy_version 570 (0.0017) [2023-06-21 10:47:41,549][62782] Fps is (10 sec: 1638.3, 60 sec: 1297.0, 300 sec: 1206.5). Total num frames: 2338816. Throughput: 0: 317.0. Samples: 83060. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:47:41,554][62782] Avg episode reward: [(0, '14.423')] [2023-06-21 10:47:46,543][62782] Fps is (10 sec: 1229.2, 60 sec: 1297.2, 300 sec: 1199.6). Total num frames: 2342912. Throughput: 0: 317.3. Samples: 84970. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:47:46,547][62782] Avg episode reward: [(0, '13.978')] [2023-06-21 10:47:51,548][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1207.2). Total num frames: 2351104. Throughput: 0: 317.7. Samples: 86884. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:47:51,550][62782] Avg episode reward: [(0, '13.981')] [2023-06-21 10:47:56,547][62782] Fps is (10 sec: 1228.3, 60 sec: 1228.7, 300 sec: 1200.6). Total num frames: 2355200. Throughput: 0: 317.4. Samples: 87828. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:47:56,551][62782] Avg episode reward: [(0, '13.250')] [2023-06-21 10:48:01,546][62782] Fps is (10 sec: 1229.0, 60 sec: 1297.1, 300 sec: 1208.0). Total num frames: 2363392. Throughput: 0: 317.6. Samples: 89736. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:48:01,549][62782] Avg episode reward: [(0, '13.169')] [2023-06-21 10:48:06,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.7, 300 sec: 1221.9). Total num frames: 2367488. Throughput: 0: 317.4. Samples: 91640. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:48:06,551][62782] Avg episode reward: [(0, '13.466')] [2023-06-21 10:48:10,617][69878] Updated weights for policy 0, policy_version 580 (0.0010) [2023-06-21 10:48:11,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1249.6). Total num frames: 2375680. Throughput: 0: 317.5. Samples: 92590. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:48:11,549][62782] Avg episode reward: [(0, '14.318')] [2023-06-21 10:48:16,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 2379776. Throughput: 0: 317.5. Samples: 94492. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:48:16,552][62782] Avg episode reward: [(0, '14.602')] [2023-06-21 10:48:21,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 2387968. Throughput: 0: 317.2. Samples: 96384. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:48:21,550][62782] Avg episode reward: [(0, '15.054')] [2023-06-21 10:48:26,547][62782] Fps is (10 sec: 1228.9, 60 sec: 1228.8, 300 sec: 1249.6). Total num frames: 2392064. Throughput: 0: 317.2. Samples: 97332. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:48:26,552][62782] Avg episode reward: [(0, '15.107')] [2023-06-21 10:48:31,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 2400256. Throughput: 0: 316.8. Samples: 99226. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:48:31,551][62782] Avg episode reward: [(0, '14.284')] [2023-06-21 10:48:36,547][62782] Fps is (10 sec: 1638.4, 60 sec: 1297.0, 300 sec: 1277.4). Total num frames: 2408448. Throughput: 0: 316.5. Samples: 101128. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:48:36,549][62782] Avg episode reward: [(0, '13.725')] [2023-06-21 10:48:41,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 2412544. Throughput: 0: 316.8. Samples: 102084. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:48:41,550][62782] Avg episode reward: [(0, '14.637')] [2023-06-21 10:48:42,956][69878] Updated weights for policy 0, policy_version 590 (0.0014) [2023-06-21 10:48:46,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.0, 300 sec: 1263.5). Total num frames: 2420736. Throughput: 0: 316.8. Samples: 103990. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:48:46,550][62782] Avg episode reward: [(0, '14.829')] [2023-06-21 10:48:51,546][62782] Fps is (10 sec: 1228.9, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 2424832. Throughput: 0: 316.4. Samples: 105878. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:48:51,549][62782] Avg episode reward: [(0, '15.085')] [2023-06-21 10:48:56,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 2433024. Throughput: 0: 316.4. Samples: 106828. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:48:56,549][62782] Avg episode reward: [(0, '15.159')] [2023-06-21 10:49:01,547][62782] Fps is (10 sec: 1228.6, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 2437120. Throughput: 0: 316.4. Samples: 108728. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:49:01,551][62782] Avg episode reward: [(0, '14.737')] [2023-06-21 10:49:02,461][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000596_2441216.pth... [2023-06-21 10:49:02,560][69876] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000522_2138112.pth [2023-06-21 10:49:06,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 2445312. Throughput: 0: 316.5. Samples: 110626. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:49:06,550][62782] Avg episode reward: [(0, '14.679')] [2023-06-21 10:49:11,549][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 2449408. Throughput: 0: 316.6. Samples: 111580. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:49:11,555][62782] Avg episode reward: [(0, '14.313')] [2023-06-21 10:49:15,282][69878] Updated weights for policy 0, policy_version 600 (0.0019) [2023-06-21 10:49:16,543][62782] Fps is (10 sec: 1229.3, 60 sec: 1297.2, 300 sec: 1263.5). Total num frames: 2457600. Throughput: 0: 316.9. Samples: 113486. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:49:16,545][62782] Avg episode reward: [(0, '13.993')] [2023-06-21 10:49:21,544][62782] Fps is (10 sec: 1229.2, 60 sec: 1228.9, 300 sec: 1263.5). Total num frames: 2461696. Throughput: 0: 317.1. Samples: 115398. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:49:21,548][62782] Avg episode reward: [(0, '14.569')] [2023-06-21 10:49:26,546][62782] Fps is (10 sec: 1228.4, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 2469888. Throughput: 0: 317.1. Samples: 116352. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:49:26,549][62782] Avg episode reward: [(0, '14.951')] [2023-06-21 10:49:31,547][62782] Fps is (10 sec: 1637.9, 60 sec: 1297.1, 300 sec: 1277.4). Total num frames: 2478080. Throughput: 0: 316.9. Samples: 118252. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:49:31,550][62782] Avg episode reward: [(0, '16.339')] [2023-06-21 10:49:31,552][69876] Saving new best policy, reward=16.339! [2023-06-21 10:49:36,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 2482176. Throughput: 0: 317.4. Samples: 120160. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:49:36,551][62782] Avg episode reward: [(0, '16.978')] [2023-06-21 10:49:37,935][69876] Saving new best policy, reward=16.978! [2023-06-21 10:49:41,546][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 2490368. Throughput: 0: 317.4. Samples: 121112. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:49:41,548][62782] Avg episode reward: [(0, '17.549')] [2023-06-21 10:49:41,551][69876] Saving new best policy, reward=17.549! [2023-06-21 10:49:46,545][62782] Fps is (10 sec: 1229.0, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 2494464. Throughput: 0: 317.8. Samples: 123028. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:49:46,548][62782] Avg episode reward: [(0, '17.416')] [2023-06-21 10:49:47,492][69878] Updated weights for policy 0, policy_version 610 (0.0014) [2023-06-21 10:49:51,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.0, 300 sec: 1263.5). Total num frames: 2502656. Throughput: 0: 317.9. Samples: 124932. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:49:51,551][62782] Avg episode reward: [(0, '17.085')] [2023-06-21 10:49:56,551][62782] Fps is (10 sec: 1228.2, 60 sec: 1228.7, 300 sec: 1263.5). Total num frames: 2506752. Throughput: 0: 317.9. Samples: 125888. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:49:56,568][62782] Avg episode reward: [(0, '16.973')] [2023-06-21 10:50:01,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 2514944. Throughput: 0: 317.7. Samples: 127784. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:50:01,551][62782] Avg episode reward: [(0, '17.665')] [2023-06-21 10:50:01,555][69876] Saving new best policy, reward=17.665! [2023-06-21 10:50:06,547][62782] Fps is (10 sec: 1229.2, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 2519040. Throughput: 0: 317.5. Samples: 129686. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:50:06,551][62782] Avg episode reward: [(0, '17.936')] [2023-06-21 10:50:06,935][69876] Saving new best policy, reward=17.936! [2023-06-21 10:50:11,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 2527232. Throughput: 0: 317.6. Samples: 130644. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:50:11,549][62782] Avg episode reward: [(0, '18.680')] [2023-06-21 10:50:13,312][69876] Saving new best policy, reward=18.680! [2023-06-21 10:50:16,548][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.7, 300 sec: 1263.5). Total num frames: 2531328. Throughput: 0: 317.8. Samples: 132554. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:50:16,554][62782] Avg episode reward: [(0, '19.383')] [2023-06-21 10:50:16,567][69876] Saving new best policy, reward=19.383! [2023-06-21 10:50:19,683][69878] Updated weights for policy 0, policy_version 620 (0.0019) [2023-06-21 10:50:21,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.0, 300 sec: 1263.5). Total num frames: 2539520. Throughput: 0: 317.8. Samples: 134460. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:50:21,549][62782] Avg episode reward: [(0, '19.876')] [2023-06-21 10:50:23,101][69876] Saving new best policy, reward=19.876! [2023-06-21 10:50:26,547][62782] Fps is (10 sec: 1638.5, 60 sec: 1297.1, 300 sec: 1277.4). Total num frames: 2547712. Throughput: 0: 317.6. Samples: 135404. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:50:26,549][62782] Avg episode reward: [(0, '20.443')] [2023-06-21 10:50:26,551][69876] Saving new best policy, reward=20.443! [2023-06-21 10:50:31,545][62782] Fps is (10 sec: 1229.0, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 2551808. Throughput: 0: 317.3. Samples: 137308. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:50:31,548][62782] Avg episode reward: [(0, '20.653')] [2023-06-21 10:50:32,632][69876] Saving new best policy, reward=20.653! [2023-06-21 10:50:36,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1277.4). Total num frames: 2560000. Throughput: 0: 317.3. Samples: 139212. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:50:36,549][62782] Avg episode reward: [(0, '20.839')] [2023-06-21 10:50:36,551][69876] Saving new best policy, reward=20.839! [2023-06-21 10:50:41,552][62782] Fps is (10 sec: 1228.0, 60 sec: 1228.7, 300 sec: 1263.5). Total num frames: 2564096. Throughput: 0: 317.3. Samples: 140168. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:50:41,561][62782] Avg episode reward: [(0, '21.224')] [2023-06-21 10:50:42,245][69876] Saving new best policy, reward=21.224! [2023-06-21 10:50:46,551][62782] Fps is (10 sec: 1228.3, 60 sec: 1296.9, 300 sec: 1277.4). Total num frames: 2572288. Throughput: 0: 313.5. Samples: 141894. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:50:46,554][62782] Avg episode reward: [(0, '20.846')] [2023-06-21 10:50:51,550][62782] Fps is (10 sec: 1229.0, 60 sec: 1228.7, 300 sec: 1263.5). Total num frames: 2576384. Throughput: 0: 308.8. Samples: 143584. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:50:51,553][62782] Avg episode reward: [(0, '21.901')] [2023-06-21 10:50:53,407][69876] Saving new best policy, reward=21.901! [2023-06-21 10:50:53,424][69878] Updated weights for policy 0, policy_version 630 (0.0020) [2023-06-21 10:50:56,552][62782] Fps is (10 sec: 819.2, 60 sec: 1228.8, 300 sec: 1249.6). Total num frames: 2580480. Throughput: 0: 306.5. Samples: 144436. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:50:56,556][62782] Avg episode reward: [(0, '21.199')] [2023-06-21 10:51:01,552][62782] Fps is (10 sec: 1228.6, 60 sec: 1228.7, 300 sec: 1263.5). Total num frames: 2588672. Throughput: 0: 301.7. Samples: 146134. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:51:01,556][62782] Avg episode reward: [(0, '21.296')] [2023-06-21 10:51:01,560][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000632_2588672.pth... [2023-06-21 10:51:01,641][69876] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000559_2289664.pth [2023-06-21 10:51:06,550][62782] Fps is (10 sec: 1229.0, 60 sec: 1228.7, 300 sec: 1249.6). Total num frames: 2592768. Throughput: 0: 297.2. Samples: 147836. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:51:06,553][62782] Avg episode reward: [(0, '21.226')] [2023-06-21 10:51:11,549][62782] Fps is (10 sec: 1229.1, 60 sec: 1228.7, 300 sec: 1263.5). Total num frames: 2600960. Throughput: 0: 295.3. Samples: 148692. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:51:11,551][62782] Avg episode reward: [(0, '21.433')] [2023-06-21 10:51:16,552][62782] Fps is (10 sec: 1228.6, 60 sec: 1228.7, 300 sec: 1249.6). Total num frames: 2605056. Throughput: 0: 290.8. Samples: 150394. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:51:16,555][62782] Avg episode reward: [(0, '21.801')] [2023-06-21 10:51:21,551][62782] Fps is (10 sec: 819.0, 60 sec: 1160.5, 300 sec: 1249.6). Total num frames: 2609152. Throughput: 0: 286.4. Samples: 152100. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:51:21,555][62782] Avg episode reward: [(0, '22.042')] [2023-06-21 10:51:22,181][69876] Saving new best policy, reward=22.042! [2023-06-21 10:51:26,552][62782] Fps is (10 sec: 1228.8, 60 sec: 1160.4, 300 sec: 1249.6). Total num frames: 2617344. Throughput: 0: 284.0. Samples: 152948. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:51:26,555][62782] Avg episode reward: [(0, '21.661')] [2023-06-21 10:51:29,369][69878] Updated weights for policy 0, policy_version 640 (0.0020) [2023-06-21 10:51:31,560][62782] Fps is (10 sec: 1227.8, 60 sec: 1160.3, 300 sec: 1249.6). Total num frames: 2621440. Throughput: 0: 283.2. Samples: 154640. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:51:31,562][62782] Avg episode reward: [(0, '21.471')] [2023-06-21 10:51:36,552][62782] Fps is (10 sec: 1228.8, 60 sec: 1160.4, 300 sec: 1249.6). Total num frames: 2629632. Throughput: 0: 283.3. Samples: 156332. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:51:36,555][62782] Avg episode reward: [(0, '20.507')] [2023-06-21 10:51:41,551][62782] Fps is (10 sec: 1229.8, 60 sec: 1160.6, 300 sec: 1249.6). Total num frames: 2633728. Throughput: 0: 283.3. Samples: 157186. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:51:41,554][62782] Avg episode reward: [(0, '20.398')] [2023-06-21 10:51:46,553][62782] Fps is (10 sec: 819.1, 60 sec: 1092.2, 300 sec: 1235.7). Total num frames: 2637824. Throughput: 0: 283.2. Samples: 158880. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:51:46,558][62782] Avg episode reward: [(0, '19.976')] [2023-06-21 10:51:51,552][62782] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1235.7). Total num frames: 2646016. Throughput: 0: 282.9. Samples: 160568. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:51:51,555][62782] Avg episode reward: [(0, '19.555')] [2023-06-21 10:51:56,550][62782] Fps is (10 sec: 1229.1, 60 sec: 1160.6, 300 sec: 1235.7). Total num frames: 2650112. Throughput: 0: 282.7. Samples: 161414. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:51:56,553][62782] Avg episode reward: [(0, '19.896')] [2023-06-21 10:52:01,553][62782] Fps is (10 sec: 819.2, 60 sec: 1092.2, 300 sec: 1221.8). Total num frames: 2654208. Throughput: 0: 282.7. Samples: 163116. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:52:01,557][62782] Avg episode reward: [(0, '19.914')] [2023-06-21 10:52:05,375][69878] Updated weights for policy 0, policy_version 650 (0.0022) [2023-06-21 10:52:06,552][62782] Fps is (10 sec: 1228.5, 60 sec: 1160.5, 300 sec: 1235.7). Total num frames: 2662400. Throughput: 0: 282.5. Samples: 164814. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:52:06,553][62782] Avg episode reward: [(0, '19.427')] [2023-06-21 10:52:11,551][62782] Fps is (10 sec: 1229.0, 60 sec: 1092.2, 300 sec: 1221.8). Total num frames: 2666496. Throughput: 0: 282.4. Samples: 165656. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:52:11,554][62782] Avg episode reward: [(0, '19.080')] [2023-06-21 10:52:16,550][62782] Fps is (10 sec: 1229.1, 60 sec: 1160.6, 300 sec: 1235.7). Total num frames: 2674688. Throughput: 0: 282.6. Samples: 167354. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:52:16,552][62782] Avg episode reward: [(0, '19.476')] [2023-06-21 10:52:21,550][62782] Fps is (10 sec: 1228.9, 60 sec: 1160.6, 300 sec: 1221.8). Total num frames: 2678784. Throughput: 0: 282.7. Samples: 169052. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:52:21,553][62782] Avg episode reward: [(0, '19.372')] [2023-06-21 10:52:26,552][62782] Fps is (10 sec: 819.0, 60 sec: 1092.3, 300 sec: 1221.8). Total num frames: 2682880. Throughput: 0: 282.6. Samples: 169902. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:52:26,561][62782] Avg episode reward: [(0, '19.504')] [2023-06-21 10:52:31,552][62782] Fps is (10 sec: 1228.5, 60 sec: 1160.7, 300 sec: 1221.8). Total num frames: 2691072. Throughput: 0: 282.7. Samples: 171602. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:52:31,555][62782] Avg episode reward: [(0, '19.435')] [2023-06-21 10:52:36,553][62782] Fps is (10 sec: 1228.8, 60 sec: 1092.3, 300 sec: 1208.0). Total num frames: 2695168. Throughput: 0: 282.9. Samples: 173300. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:52:36,556][62782] Avg episode reward: [(0, '18.693')] [2023-06-21 10:52:41,550][62782] Fps is (10 sec: 819.3, 60 sec: 1092.3, 300 sec: 1207.9). Total num frames: 2699264. Throughput: 0: 283.2. Samples: 174156. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:52:41,554][62782] Avg episode reward: [(0, '18.573')] [2023-06-21 10:52:41,641][69878] Updated weights for policy 0, policy_version 660 (0.0025) [2023-06-21 10:52:46,550][62782] Fps is (10 sec: 1229.1, 60 sec: 1160.6, 300 sec: 1208.0). Total num frames: 2707456. Throughput: 0: 282.9. Samples: 175846. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:52:46,553][62782] Avg episode reward: [(0, '18.520')] [2023-06-21 10:52:51,551][62782] Fps is (10 sec: 1228.7, 60 sec: 1092.3, 300 sec: 1208.0). Total num frames: 2711552. Throughput: 0: 282.8. Samples: 177540. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:52:51,555][62782] Avg episode reward: [(0, '18.981')] [2023-06-21 10:52:56,552][62782] Fps is (10 sec: 1228.6, 60 sec: 1160.5, 300 sec: 1207.9). Total num frames: 2719744. Throughput: 0: 282.7. Samples: 178380. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:52:56,555][62782] Avg episode reward: [(0, '18.666')] [2023-06-21 10:53:01,551][62782] Fps is (10 sec: 1228.9, 60 sec: 1160.6, 300 sec: 1208.0). Total num frames: 2723840. Throughput: 0: 282.8. Samples: 180082. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:53:01,554][62782] Avg episode reward: [(0, '18.352')] [2023-06-21 10:53:03,411][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000666_2727936.pth... [2023-06-21 10:53:03,497][69876] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000596_2441216.pth [2023-06-21 10:53:06,553][62782] Fps is (10 sec: 819.1, 60 sec: 1092.3, 300 sec: 1194.1). Total num frames: 2727936. Throughput: 0: 283.0. Samples: 181790. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:53:06,557][62782] Avg episode reward: [(0, '18.326')] [2023-06-21 10:53:11,552][62782] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1208.0). Total num frames: 2736128. Throughput: 0: 283.1. Samples: 182640. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:53:11,556][62782] Avg episode reward: [(0, '18.556')] [2023-06-21 10:53:16,544][62782] Fps is (10 sec: 1229.9, 60 sec: 1092.4, 300 sec: 1194.1). Total num frames: 2740224. Throughput: 0: 283.2. Samples: 184342. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:53:16,547][62782] Avg episode reward: [(0, '18.499')] [2023-06-21 10:53:17,672][69878] Updated weights for policy 0, policy_version 670 (0.0020) [2023-06-21 10:53:21,550][62782] Fps is (10 sec: 1229.0, 60 sec: 1160.5, 300 sec: 1208.0). Total num frames: 2748416. Throughput: 0: 283.1. Samples: 186040. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:53:21,552][62782] Avg episode reward: [(0, '18.956')] [2023-06-21 10:53:26,552][62782] Fps is (10 sec: 1227.7, 60 sec: 1160.5, 300 sec: 1194.1). Total num frames: 2752512. Throughput: 0: 282.8. Samples: 186882. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:53:26,557][62782] Avg episode reward: [(0, '19.680')] [2023-06-21 10:53:31,552][62782] Fps is (10 sec: 819.1, 60 sec: 1092.3, 300 sec: 1180.2). Total num frames: 2756608. Throughput: 0: 283.0. Samples: 188580. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:53:31,556][62782] Avg episode reward: [(0, '19.620')] [2023-06-21 10:53:36,552][62782] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1194.1). Total num frames: 2764800. Throughput: 0: 282.8. Samples: 190266. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:53:36,556][62782] Avg episode reward: [(0, '19.543')] [2023-06-21 10:53:41,552][62782] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1180.2). Total num frames: 2768896. Throughput: 0: 283.1. Samples: 191118. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:53:41,563][62782] Avg episode reward: [(0, '19.523')] [2023-06-21 10:53:46,545][62782] Fps is (10 sec: 819.7, 60 sec: 1092.4, 300 sec: 1180.2). Total num frames: 2772992. Throughput: 0: 283.0. Samples: 192816. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:53:46,550][62782] Avg episode reward: [(0, '19.102')] [2023-06-21 10:53:51,551][62782] Fps is (10 sec: 1229.0, 60 sec: 1160.5, 300 sec: 1180.2). Total num frames: 2781184. Throughput: 0: 282.8. Samples: 194516. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:53:51,553][62782] Avg episode reward: [(0, '19.450')] [2023-06-21 10:53:53,900][69878] Updated weights for policy 0, policy_version 680 (0.0019) [2023-06-21 10:53:56,553][62782] Fps is (10 sec: 1227.9, 60 sec: 1092.3, 300 sec: 1180.2). Total num frames: 2785280. Throughput: 0: 282.8. Samples: 195364. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:53:56,554][62782] Avg episode reward: [(0, '20.058')] [2023-06-21 10:54:01,552][62782] Fps is (10 sec: 1228.6, 60 sec: 1160.5, 300 sec: 1180.2). Total num frames: 2793472. Throughput: 0: 282.5. Samples: 197058. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:54:01,555][62782] Avg episode reward: [(0, '20.568')] [2023-06-21 10:54:06,552][62782] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1180.2). Total num frames: 2797568. Throughput: 0: 282.5. Samples: 198754. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:54:06,555][62782] Avg episode reward: [(0, '20.499')] [2023-06-21 10:54:11,550][62782] Fps is (10 sec: 819.4, 60 sec: 1092.3, 300 sec: 1166.3). Total num frames: 2801664. Throughput: 0: 282.6. Samples: 199600. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:54:11,554][62782] Avg episode reward: [(0, '20.412')] [2023-06-21 10:54:16,552][62782] Fps is (10 sec: 1228.8, 60 sec: 1160.4, 300 sec: 1180.2). Total num frames: 2809856. Throughput: 0: 282.8. Samples: 201304. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:54:16,556][62782] Avg episode reward: [(0, '20.391')] [2023-06-21 10:54:21,550][62782] Fps is (10 sec: 1228.7, 60 sec: 1092.3, 300 sec: 1166.3). Total num frames: 2813952. Throughput: 0: 283.3. Samples: 203012. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:54:21,554][62782] Avg episode reward: [(0, '20.677')] [2023-06-21 10:54:26,552][62782] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1166.3). Total num frames: 2822144. Throughput: 0: 283.2. Samples: 203864. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:54:26,556][62782] Avg episode reward: [(0, '20.792')] [2023-06-21 10:54:30,185][69878] Updated weights for policy 0, policy_version 690 (0.0018) [2023-06-21 10:54:31,551][62782] Fps is (10 sec: 1228.8, 60 sec: 1160.6, 300 sec: 1166.3). Total num frames: 2826240. Throughput: 0: 283.2. Samples: 205560. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:54:31,552][62782] Avg episode reward: [(0, '22.092')] [2023-06-21 10:54:31,558][69876] Saving new best policy, reward=22.092! [2023-06-21 10:54:36,551][62782] Fps is (10 sec: 819.3, 60 sec: 1092.3, 300 sec: 1152.4). Total num frames: 2830336. Throughput: 0: 283.1. Samples: 207254. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2023-06-21 10:54:36,556][62782] Avg episode reward: [(0, '21.812')] [2023-06-21 10:54:41,552][62782] Fps is (10 sec: 1228.6, 60 sec: 1160.5, 300 sec: 1166.3). Total num frames: 2838528. Throughput: 0: 283.0. Samples: 208100. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:54:41,555][62782] Avg episode reward: [(0, '21.712')] [2023-06-21 10:54:46,552][62782] Fps is (10 sec: 1228.6, 60 sec: 1160.4, 300 sec: 1152.4). Total num frames: 2842624. Throughput: 0: 282.9. Samples: 209788. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:54:46,557][62782] Avg episode reward: [(0, '21.706')] [2023-06-21 10:54:51,550][62782] Fps is (10 sec: 819.4, 60 sec: 1092.3, 300 sec: 1152.4). Total num frames: 2846720. Throughput: 0: 283.2. Samples: 211498. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:54:51,554][62782] Avg episode reward: [(0, '21.992')] [2023-06-21 10:54:56,550][62782] Fps is (10 sec: 1229.1, 60 sec: 1160.6, 300 sec: 1152.4). Total num frames: 2854912. Throughput: 0: 283.5. Samples: 212358. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:54:56,553][62782] Avg episode reward: [(0, '22.056')] [2023-06-21 10:55:01,552][62782] Fps is (10 sec: 1228.5, 60 sec: 1092.3, 300 sec: 1152.4). Total num frames: 2859008. Throughput: 0: 283.4. Samples: 214056. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:55:01,555][62782] Avg episode reward: [(0, '20.985')] [2023-06-21 10:55:02,963][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000699_2863104.pth... [2023-06-21 10:55:03,059][69876] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000632_2588672.pth [2023-06-21 10:55:06,552][62782] Fps is (10 sec: 819.0, 60 sec: 1092.3, 300 sec: 1138.5). Total num frames: 2863104. Throughput: 0: 283.1. Samples: 215754. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:55:06,554][62782] Avg episode reward: [(0, '21.373')] [2023-06-21 10:55:06,586][69878] Updated weights for policy 0, policy_version 700 (0.0029) [2023-06-21 10:55:11,551][62782] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1152.4). Total num frames: 2871296. Throughput: 0: 283.2. Samples: 216606. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:55:11,554][62782] Avg episode reward: [(0, '21.717')] [2023-06-21 10:55:16,551][62782] Fps is (10 sec: 1228.8, 60 sec: 1092.3, 300 sec: 1138.5). Total num frames: 2875392. Throughput: 0: 283.4. Samples: 218312. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:55:16,555][62782] Avg episode reward: [(0, '21.310')] [2023-06-21 10:55:21,551][62782] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1138.5). Total num frames: 2883584. Throughput: 0: 283.6. Samples: 220014. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:55:21,554][62782] Avg episode reward: [(0, '21.572')] [2023-06-21 10:55:26,553][62782] Fps is (10 sec: 1228.7, 60 sec: 1092.3, 300 sec: 1138.5). Total num frames: 2887680. Throughput: 0: 283.4. Samples: 220852. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:55:26,560][62782] Avg episode reward: [(0, '21.109')] [2023-06-21 10:55:31,552][62782] Fps is (10 sec: 819.1, 60 sec: 1092.2, 300 sec: 1124.6). Total num frames: 2891776. Throughput: 0: 283.6. Samples: 222552. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:55:31,557][62782] Avg episode reward: [(0, '21.725')] [2023-06-21 10:55:36,550][62782] Fps is (10 sec: 1229.1, 60 sec: 1160.6, 300 sec: 1138.6). Total num frames: 2899968. Throughput: 0: 283.6. Samples: 224258. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:55:36,553][62782] Avg episode reward: [(0, '22.063')] [2023-06-21 10:55:41,552][62782] Fps is (10 sec: 1228.8, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 2904064. Throughput: 0: 283.4. Samples: 225110. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:55:41,554][62782] Avg episode reward: [(0, '22.322')] [2023-06-21 10:55:42,724][69876] Saving new best policy, reward=22.322! [2023-06-21 10:55:42,727][69878] Updated weights for policy 0, policy_version 710 (0.0017) [2023-06-21 10:55:46,550][62782] Fps is (10 sec: 1228.7, 60 sec: 1160.6, 300 sec: 1138.5). Total num frames: 2912256. Throughput: 0: 283.4. Samples: 226808. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:55:46,552][62782] Avg episode reward: [(0, '21.857')] [2023-06-21 10:55:51,550][62782] Fps is (10 sec: 1229.0, 60 sec: 1160.5, 300 sec: 1138.6). Total num frames: 2916352. Throughput: 0: 283.7. Samples: 228520. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:55:51,553][62782] Avg episode reward: [(0, '22.129')] [2023-06-21 10:55:56,549][62782] Fps is (10 sec: 819.3, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 2920448. Throughput: 0: 283.6. Samples: 229366. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:55:56,552][62782] Avg episode reward: [(0, '22.348')] [2023-06-21 10:55:57,027][69876] Saving new best policy, reward=22.348! [2023-06-21 10:56:01,551][62782] Fps is (10 sec: 1228.6, 60 sec: 1160.5, 300 sec: 1138.5). Total num frames: 2928640. Throughput: 0: 283.6. Samples: 231074. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:56:01,555][62782] Avg episode reward: [(0, '22.195')] [2023-06-21 10:56:06,550][62782] Fps is (10 sec: 1228.7, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 2932736. Throughput: 0: 283.6. Samples: 232778. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:56:06,551][62782] Avg episode reward: [(0, '23.129')] [2023-06-21 10:56:07,939][69876] Saving new best policy, reward=23.129! [2023-06-21 10:56:11,552][62782] Fps is (10 sec: 819.1, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 2936832. Throughput: 0: 283.7. Samples: 233618. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:56:11,555][62782] Avg episode reward: [(0, '22.786')] [2023-06-21 10:56:16,552][62782] Fps is (10 sec: 1228.6, 60 sec: 1160.5, 300 sec: 1138.5). Total num frames: 2945024. Throughput: 0: 283.6. Samples: 235314. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:56:16,554][62782] Avg episode reward: [(0, '22.305')] [2023-06-21 10:56:18,769][69878] Updated weights for policy 0, policy_version 720 (0.0014) [2023-06-21 10:56:21,551][62782] Fps is (10 sec: 1228.9, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 2949120. Throughput: 0: 283.7. Samples: 237026. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:56:21,552][62782] Avg episode reward: [(0, '21.782')] [2023-06-21 10:56:26,546][62782] Fps is (10 sec: 1229.5, 60 sec: 1160.7, 300 sec: 1138.6). Total num frames: 2957312. Throughput: 0: 283.6. Samples: 237872. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:56:26,549][62782] Avg episode reward: [(0, '22.127')] [2023-06-21 10:56:31,551][62782] Fps is (10 sec: 1228.8, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 2961408. Throughput: 0: 283.6. Samples: 239572. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:56:31,553][62782] Avg episode reward: [(0, '21.791')] [2023-06-21 10:56:36,552][62782] Fps is (10 sec: 818.7, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 2965504. Throughput: 0: 283.5. Samples: 241276. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:56:36,555][62782] Avg episode reward: [(0, '21.149')] [2023-06-21 10:56:41,552][62782] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1138.6). Total num frames: 2973696. Throughput: 0: 283.5. Samples: 242126. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:56:41,553][62782] Avg episode reward: [(0, '20.864')] [2023-06-21 10:56:46,550][62782] Fps is (10 sec: 1229.1, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 2977792. Throughput: 0: 283.4. Samples: 243826. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:56:46,553][62782] Avg episode reward: [(0, '20.781')] [2023-06-21 10:56:51,551][62782] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1138.5). Total num frames: 2985984. Throughput: 0: 283.2. Samples: 245522. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:56:51,553][62782] Avg episode reward: [(0, '20.836')] [2023-06-21 10:56:54,865][69878] Updated weights for policy 0, policy_version 730 (0.0009) [2023-06-21 10:56:56,551][62782] Fps is (10 sec: 1228.6, 60 sec: 1160.5, 300 sec: 1138.6). Total num frames: 2990080. Throughput: 0: 283.7. Samples: 246384. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:56:56,553][62782] Avg episode reward: [(0, '21.214')] [2023-06-21 10:57:01,572][62782] Fps is (10 sec: 817.6, 60 sec: 1091.9, 300 sec: 1124.6). Total num frames: 2994176. Throughput: 0: 283.2. Samples: 248062. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:57:01,579][62782] Avg episode reward: [(0, '20.883')] [2023-06-21 10:57:02,281][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000732_2998272.pth... [2023-06-21 10:57:02,372][69876] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000666_2727936.pth [2023-06-21 10:57:06,552][62782] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1138.5). Total num frames: 3002368. Throughput: 0: 283.2. Samples: 249768. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:57:06,554][62782] Avg episode reward: [(0, '21.140')] [2023-06-21 10:57:11,551][62782] Fps is (10 sec: 1231.1, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 3006464. Throughput: 0: 283.3. Samples: 250624. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:57:11,553][62782] Avg episode reward: [(0, '20.747')] [2023-06-21 10:57:16,551][62782] Fps is (10 sec: 819.3, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 3010560. Throughput: 0: 283.1. Samples: 252312. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:57:16,553][62782] Avg episode reward: [(0, '20.341')] [2023-06-21 10:57:21,552][62782] Fps is (10 sec: 1228.7, 60 sec: 1160.5, 300 sec: 1138.6). Total num frames: 3018752. Throughput: 0: 283.1. Samples: 254016. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:57:21,553][62782] Avg episode reward: [(0, '19.390')] [2023-06-21 10:57:26,552][62782] Fps is (10 sec: 1228.6, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 3022848. Throughput: 0: 283.0. Samples: 254860. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 10:57:26,561][62782] Avg episode reward: [(0, '19.129')] [2023-06-21 10:57:31,257][69878] Updated weights for policy 0, policy_version 740 (0.0010) [2023-06-21 10:57:31,551][62782] Fps is (10 sec: 1229.0, 60 sec: 1160.5, 300 sec: 1138.6). Total num frames: 3031040. Throughput: 0: 283.0. Samples: 256562. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:57:31,552][62782] Avg episode reward: [(0, '19.170')] [2023-06-21 10:57:36,552][62782] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1138.5). Total num frames: 3035136. Throughput: 0: 283.1. Samples: 258264. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:57:36,554][62782] Avg episode reward: [(0, '20.397')] [2023-06-21 10:57:41,552][62782] Fps is (10 sec: 819.1, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 3039232. Throughput: 0: 282.6. Samples: 259100. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:57:41,556][62782] Avg episode reward: [(0, '20.716')] [2023-06-21 10:57:46,551][62782] Fps is (10 sec: 1228.9, 60 sec: 1160.5, 300 sec: 1138.6). Total num frames: 3047424. Throughput: 0: 283.4. Samples: 260808. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:57:46,554][62782] Avg episode reward: [(0, '20.711')] [2023-06-21 10:57:51,551][62782] Fps is (10 sec: 1228.9, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 3051520. Throughput: 0: 283.1. Samples: 262508. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:57:51,561][62782] Avg episode reward: [(0, '20.364')] [2023-06-21 10:57:56,550][62782] Fps is (10 sec: 819.3, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 3055616. Throughput: 0: 283.0. Samples: 263358. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:57:56,551][62782] Avg episode reward: [(0, '21.127')] [2023-06-21 10:58:01,552][62782] Fps is (10 sec: 1228.8, 60 sec: 1160.9, 300 sec: 1138.6). Total num frames: 3063808. Throughput: 0: 283.2. Samples: 265056. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:58:01,555][62782] Avg episode reward: [(0, '21.273')] [2023-06-21 10:58:06,552][62782] Fps is (10 sec: 1228.6, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 3067904. Throughput: 0: 283.4. Samples: 266768. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:58:06,555][62782] Avg episode reward: [(0, '21.392')] [2023-06-21 10:58:07,439][69878] Updated weights for policy 0, policy_version 750 (0.0010) [2023-06-21 10:58:11,552][62782] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1138.5). Total num frames: 3076096. Throughput: 0: 283.6. Samples: 267624. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:58:11,554][62782] Avg episode reward: [(0, '21.163')] [2023-06-21 10:58:16,551][62782] Fps is (10 sec: 1228.8, 60 sec: 1160.5, 300 sec: 1124.7). Total num frames: 3080192. Throughput: 0: 283.6. Samples: 269322. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:58:16,554][62782] Avg episode reward: [(0, '21.027')] [2023-06-21 10:58:21,551][62782] Fps is (10 sec: 819.3, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 3084288. Throughput: 0: 283.7. Samples: 271028. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:58:21,552][62782] Avg episode reward: [(0, '21.069')] [2023-06-21 10:58:26,551][62782] Fps is (10 sec: 1228.8, 60 sec: 1160.6, 300 sec: 1138.6). Total num frames: 3092480. Throughput: 0: 284.0. Samples: 271880. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:58:26,553][62782] Avg episode reward: [(0, '20.873')] [2023-06-21 10:58:31,552][62782] Fps is (10 sec: 1228.6, 60 sec: 1092.2, 300 sec: 1124.7). Total num frames: 3096576. Throughput: 0: 283.6. Samples: 273572. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:58:31,553][62782] Avg episode reward: [(0, '20.527')] [2023-06-21 10:58:36,547][62782] Fps is (10 sec: 1229.3, 60 sec: 1160.6, 300 sec: 1138.6). Total num frames: 3104768. Throughput: 0: 286.3. Samples: 275392. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 10:58:36,551][62782] Avg episode reward: [(0, '20.999')] [2023-06-21 10:58:41,547][62782] Fps is (10 sec: 1229.5, 60 sec: 1160.6, 300 sec: 1138.5). Total num frames: 3108864. Throughput: 0: 288.6. Samples: 276342. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:58:41,549][62782] Avg episode reward: [(0, '22.291')] [2023-06-21 10:58:42,862][69878] Updated weights for policy 0, policy_version 760 (0.0013) [2023-06-21 10:58:46,553][62782] Fps is (10 sec: 818.8, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 3112960. Throughput: 0: 293.2. Samples: 278248. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:58:46,566][62782] Avg episode reward: [(0, '22.520')] [2023-06-21 10:58:48,381][69876] Signal inference workers to stop experience collection... [2023-06-21 10:58:48,413][69878] InferenceWorker_p0-w0: stopping experience collection [2023-06-21 10:58:49,289][69876] Signal inference workers to resume experience collection... [2023-06-21 10:58:49,289][69878] InferenceWorker_p0-w0: resuming experience collection [2023-06-21 10:58:51,548][62782] Fps is (10 sec: 819.1, 60 sec: 1092.3, 300 sec: 1124.7). Total num frames: 3117056. Throughput: 0: 277.9. Samples: 279274. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:58:51,552][62782] Avg episode reward: [(0, '22.372')] [2023-06-21 10:58:56,545][62782] Fps is (10 sec: 1229.6, 60 sec: 1160.6, 300 sec: 1124.7). Total num frames: 3125248. Throughput: 0: 279.7. Samples: 280208. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 10:58:56,548][62782] Avg episode reward: [(0, '22.662')] [2023-06-21 10:59:01,546][62782] Fps is (10 sec: 1229.0, 60 sec: 1092.4, 300 sec: 1124.7). Total num frames: 3129344. Throughput: 0: 283.9. Samples: 282098. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 10:59:01,552][62782] Avg episode reward: [(0, '22.225')] [2023-06-21 10:59:01,752][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000765_3133440.pth... [2023-06-21 10:59:01,850][69876] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000699_2863104.pth [2023-06-21 10:59:06,547][62782] Fps is (10 sec: 1228.5, 60 sec: 1160.6, 300 sec: 1138.6). Total num frames: 3137536. Throughput: 0: 287.9. Samples: 283984. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 10:59:06,551][62782] Avg episode reward: [(0, '21.365')] [2023-06-21 10:59:11,548][62782] Fps is (10 sec: 1638.2, 60 sec: 1160.6, 300 sec: 1138.6). Total num frames: 3145728. Throughput: 0: 290.1. Samples: 284932. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:59:11,551][62782] Avg episode reward: [(0, '21.488')] [2023-06-21 10:59:16,547][62782] Fps is (10 sec: 1228.9, 60 sec: 1160.6, 300 sec: 1138.6). Total num frames: 3149824. Throughput: 0: 294.7. Samples: 286834. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:59:16,549][62782] Avg episode reward: [(0, '22.251')] [2023-06-21 10:59:18,006][69878] Updated weights for policy 0, policy_version 770 (0.0528) [2023-06-21 10:59:21,547][62782] Fps is (10 sec: 1228.9, 60 sec: 1228.9, 300 sec: 1138.6). Total num frames: 3158016. Throughput: 0: 296.5. Samples: 288734. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:59:21,548][62782] Avg episode reward: [(0, '22.933')] [2023-06-21 10:59:26,546][62782] Fps is (10 sec: 1228.8, 60 sec: 1160.6, 300 sec: 1138.6). Total num frames: 3162112. Throughput: 0: 296.6. Samples: 289690. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:59:26,548][62782] Avg episode reward: [(0, '23.016')] [2023-06-21 10:59:31,546][62782] Fps is (10 sec: 1229.0, 60 sec: 1228.9, 300 sec: 1152.5). Total num frames: 3170304. Throughput: 0: 296.6. Samples: 291592. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:59:31,549][62782] Avg episode reward: [(0, '23.764')] [2023-06-21 10:59:31,553][69876] Saving new best policy, reward=23.764! [2023-06-21 10:59:36,545][62782] Fps is (10 sec: 1229.0, 60 sec: 1160.6, 300 sec: 1138.6). Total num frames: 3174400. Throughput: 0: 315.9. Samples: 293488. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:59:36,548][62782] Avg episode reward: [(0, '24.120')] [2023-06-21 10:59:37,426][69876] Saving new best policy, reward=24.120! [2023-06-21 10:59:41,546][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1152.5). Total num frames: 3182592. Throughput: 0: 316.2. Samples: 294438. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:59:41,550][62782] Avg episode reward: [(0, '24.346')] [2023-06-21 10:59:41,553][69876] Saving new best policy, reward=24.346! [2023-06-21 10:59:46,554][62782] Fps is (10 sec: 1227.7, 60 sec: 1228.8, 300 sec: 1152.4). Total num frames: 3186688. Throughput: 0: 316.3. Samples: 296336. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 10:59:46,565][62782] Avg episode reward: [(0, '25.125')] [2023-06-21 10:59:47,318][69876] Saving new best policy, reward=25.125! [2023-06-21 10:59:50,487][69878] Updated weights for policy 0, policy_version 780 (0.0010) [2023-06-21 10:59:51,545][62782] Fps is (10 sec: 1229.0, 60 sec: 1297.1, 300 sec: 1152.5). Total num frames: 3194880. Throughput: 0: 316.7. Samples: 298234. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 10:59:51,547][62782] Avg episode reward: [(0, '25.195')] [2023-06-21 10:59:53,654][69876] Saving new best policy, reward=25.195! [2023-06-21 10:59:56,548][62782] Fps is (10 sec: 1229.5, 60 sec: 1228.7, 300 sec: 1152.4). Total num frames: 3198976. Throughput: 0: 316.7. Samples: 299184. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 10:59:56,550][62782] Avg episode reward: [(0, '25.239')] [2023-06-21 10:59:57,166][69876] Saving new best policy, reward=25.239! [2023-06-21 11:00:01,546][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1166.3). Total num frames: 3207168. Throughput: 0: 316.5. Samples: 301076. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:00:01,549][62782] Avg episode reward: [(0, '25.425')] [2023-06-21 11:00:03,708][69876] Saving new best policy, reward=25.425! [2023-06-21 11:00:06,549][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1152.4). Total num frames: 3211264. Throughput: 0: 316.3. Samples: 302968. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:00:06,556][62782] Avg episode reward: [(0, '24.274')] [2023-06-21 11:00:11,546][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1166.3). Total num frames: 3219456. Throughput: 0: 316.1. Samples: 303914. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:00:11,548][62782] Avg episode reward: [(0, '24.430')] [2023-06-21 11:00:16,547][62782] Fps is (10 sec: 1638.6, 60 sec: 1297.1, 300 sec: 1166.3). Total num frames: 3227648. Throughput: 0: 316.0. Samples: 305814. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:00:16,550][62782] Avg episode reward: [(0, '24.674')] [2023-06-21 11:00:21,546][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1166.3). Total num frames: 3231744. Throughput: 0: 316.2. Samples: 307716. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:00:21,548][62782] Avg episode reward: [(0, '24.861')] [2023-06-21 11:00:23,166][69878] Updated weights for policy 0, policy_version 790 (0.0015) [2023-06-21 11:00:26,547][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1180.2). Total num frames: 3239936. Throughput: 0: 316.2. Samples: 308666. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:00:26,565][62782] Avg episode reward: [(0, '24.870')] [2023-06-21 11:00:31,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1166.3). Total num frames: 3244032. Throughput: 0: 316.3. Samples: 310566. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:00:31,551][62782] Avg episode reward: [(0, '25.250')] [2023-06-21 11:00:36,546][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.0, 300 sec: 1180.2). Total num frames: 3252224. Throughput: 0: 316.5. Samples: 312478. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:00:36,549][62782] Avg episode reward: [(0, '25.525')] [2023-06-21 11:00:36,551][69876] Saving new best policy, reward=25.525! [2023-06-21 11:00:41,550][62782] Fps is (10 sec: 1229.0, 60 sec: 1228.8, 300 sec: 1166.3). Total num frames: 3256320. Throughput: 0: 316.7. Samples: 313436. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:00:41,565][62782] Avg episode reward: [(0, '25.453')] [2023-06-21 11:00:46,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.2, 300 sec: 1180.2). Total num frames: 3264512. Throughput: 0: 316.8. Samples: 315334. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:00:46,550][62782] Avg episode reward: [(0, '26.068')] [2023-06-21 11:00:46,551][69876] Saving new best policy, reward=26.068! [2023-06-21 11:00:51,547][62782] Fps is (10 sec: 1228.6, 60 sec: 1228.8, 300 sec: 1180.2). Total num frames: 3268608. Throughput: 0: 317.2. Samples: 317240. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:00:51,551][62782] Avg episode reward: [(0, '26.259')] [2023-06-21 11:00:52,005][69876] Saving new best policy, reward=26.259! [2023-06-21 11:00:55,180][69878] Updated weights for policy 0, policy_version 800 (0.0013) [2023-06-21 11:00:56,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1180.2). Total num frames: 3276800. Throughput: 0: 317.3. Samples: 318194. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:00:56,549][62782] Avg episode reward: [(0, '26.044')] [2023-06-21 11:01:01,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1180.2). Total num frames: 3280896. Throughput: 0: 317.6. Samples: 320104. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:01:01,550][62782] Avg episode reward: [(0, '25.882')] [2023-06-21 11:01:01,704][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000802_3284992.pth... [2023-06-21 11:01:01,774][69876] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000732_2998272.pth [2023-06-21 11:01:06,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1194.1). Total num frames: 3289088. Throughput: 0: 317.5. Samples: 322002. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:01:06,549][62782] Avg episode reward: [(0, '25.407')] [2023-06-21 11:01:11,547][62782] Fps is (10 sec: 1638.4, 60 sec: 1297.0, 300 sec: 1194.1). Total num frames: 3297280. Throughput: 0: 317.2. Samples: 322938. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:01:11,550][62782] Avg episode reward: [(0, '26.564')] [2023-06-21 11:01:11,553][69876] Saving new best policy, reward=26.564! [2023-06-21 11:01:16,548][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1194.1). Total num frames: 3301376. Throughput: 0: 317.2. Samples: 324838. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:01:16,551][62782] Avg episode reward: [(0, '25.871')] [2023-06-21 11:01:21,548][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.0, 300 sec: 1194.1). Total num frames: 3309568. Throughput: 0: 317.1. Samples: 326750. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:01:21,551][62782] Avg episode reward: [(0, '26.101')] [2023-06-21 11:01:26,547][62782] Fps is (10 sec: 1228.9, 60 sec: 1228.8, 300 sec: 1194.1). Total num frames: 3313664. Throughput: 0: 317.0. Samples: 327700. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:01:26,548][62782] Avg episode reward: [(0, '25.732')] [2023-06-21 11:01:27,531][69878] Updated weights for policy 0, policy_version 810 (0.0012) [2023-06-21 11:01:31,546][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1208.0). Total num frames: 3321856. Throughput: 0: 317.1. Samples: 329604. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:01:31,549][62782] Avg episode reward: [(0, '25.513')] [2023-06-21 11:01:36,547][62782] Fps is (10 sec: 1228.9, 60 sec: 1228.8, 300 sec: 1194.1). Total num frames: 3325952. Throughput: 0: 317.0. Samples: 331504. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:01:36,557][62782] Avg episode reward: [(0, '25.715')] [2023-06-21 11:01:41,546][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1208.0). Total num frames: 3334144. Throughput: 0: 316.8. Samples: 332448. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:01:41,549][62782] Avg episode reward: [(0, '24.516')] [2023-06-21 11:01:46,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1194.1). Total num frames: 3338240. Throughput: 0: 316.8. Samples: 334358. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:01:46,551][62782] Avg episode reward: [(0, '24.636')] [2023-06-21 11:01:51,547][62782] Fps is (10 sec: 1228.6, 60 sec: 1297.1, 300 sec: 1208.0). Total num frames: 3346432. Throughput: 0: 316.9. Samples: 336262. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:01:51,548][62782] Avg episode reward: [(0, '24.311')] [2023-06-21 11:01:56,566][62782] Fps is (10 sec: 1637.1, 60 sec: 1296.9, 300 sec: 1221.9). Total num frames: 3354624. Throughput: 0: 317.2. Samples: 337214. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:01:56,650][62782] Avg episode reward: [(0, '24.010')] [2023-06-21 11:02:00,281][69878] Updated weights for policy 0, policy_version 820 (0.0011) [2023-06-21 11:02:01,546][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1208.0). Total num frames: 3358720. Throughput: 0: 316.5. Samples: 339080. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:02:01,551][62782] Avg episode reward: [(0, '23.840')] [2023-06-21 11:02:06,547][62782] Fps is (10 sec: 1229.7, 60 sec: 1297.1, 300 sec: 1221.9). Total num frames: 3366912. Throughput: 0: 315.2. Samples: 340932. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:02:06,550][62782] Avg episode reward: [(0, '23.069')] [2023-06-21 11:02:11,546][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1221.9). Total num frames: 3371008. Throughput: 0: 315.2. Samples: 341882. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:02:11,547][62782] Avg episode reward: [(0, '22.050')] [2023-06-21 11:02:16,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1221.9). Total num frames: 3379200. Throughput: 0: 315.2. Samples: 343786. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:02:16,550][62782] Avg episode reward: [(0, '22.839')] [2023-06-21 11:02:21,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1221.9). Total num frames: 3383296. Throughput: 0: 315.0. Samples: 345678. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:02:21,550][62782] Avg episode reward: [(0, '22.486')] [2023-06-21 11:02:26,546][62782] Fps is (10 sec: 819.3, 60 sec: 1228.8, 300 sec: 1208.0). Total num frames: 3387392. Throughput: 0: 315.2. Samples: 346632. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:02:26,552][62782] Avg episode reward: [(0, '22.660')] [2023-06-21 11:02:31,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1221.9). Total num frames: 3395584. Throughput: 0: 313.3. Samples: 348458. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:02:31,548][62782] Avg episode reward: [(0, '21.298')] [2023-06-21 11:02:32,138][69878] Updated weights for policy 0, policy_version 830 (0.0011) [2023-06-21 11:02:36,546][62782] Fps is (10 sec: 1638.3, 60 sec: 1297.1, 300 sec: 1235.8). Total num frames: 3403776. Throughput: 0: 313.5. Samples: 350370. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:02:36,547][62782] Avg episode reward: [(0, '21.316')] [2023-06-21 11:02:41,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1221.9). Total num frames: 3407872. Throughput: 0: 313.7. Samples: 351328. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:02:41,550][62782] Avg episode reward: [(0, '21.418')] [2023-06-21 11:02:46,545][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1235.8). Total num frames: 3416064. Throughput: 0: 314.9. Samples: 353252. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:02:46,546][62782] Avg episode reward: [(0, '21.606')] [2023-06-21 11:02:51,546][62782] Fps is (10 sec: 1638.5, 60 sec: 1297.1, 300 sec: 1249.6). Total num frames: 3424256. Throughput: 0: 316.0. Samples: 355150. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:02:51,548][62782] Avg episode reward: [(0, '22.330')] [2023-06-21 11:02:56,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1229.0, 300 sec: 1235.8). Total num frames: 3428352. Throughput: 0: 316.2. Samples: 356112. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:02:56,547][62782] Avg episode reward: [(0, '22.256')] [2023-06-21 11:03:01,545][62782] Fps is (10 sec: 1229.0, 60 sec: 1297.1, 300 sec: 1249.7). Total num frames: 3436544. Throughput: 0: 316.5. Samples: 358028. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:03:01,546][62782] Avg episode reward: [(0, '22.612')] [2023-06-21 11:03:01,549][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000839_3436544.pth... [2023-06-21 11:03:01,637][69876] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000765_3133440.pth [2023-06-21 11:03:04,026][69878] Updated weights for policy 0, policy_version 840 (0.0007) [2023-06-21 11:03:06,546][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1235.8). Total num frames: 3440640. Throughput: 0: 317.0. Samples: 359944. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:03:06,547][62782] Avg episode reward: [(0, '22.676')] [2023-06-21 11:03:11,546][62782] Fps is (10 sec: 1228.6, 60 sec: 1297.1, 300 sec: 1249.7). Total num frames: 3448832. Throughput: 0: 317.0. Samples: 360898. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:03:11,548][62782] Avg episode reward: [(0, '23.135')] [2023-06-21 11:03:16,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1249.6). Total num frames: 3452928. Throughput: 0: 319.1. Samples: 362818. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:03:16,548][62782] Avg episode reward: [(0, '23.234')] [2023-06-21 11:03:21,545][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1249.7). Total num frames: 3461120. Throughput: 0: 319.0. Samples: 364724. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:03:21,546][62782] Avg episode reward: [(0, '22.683')] [2023-06-21 11:03:26,545][62782] Fps is (10 sec: 1638.7, 60 sec: 1365.3, 300 sec: 1263.5). Total num frames: 3469312. Throughput: 0: 319.1. Samples: 365686. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:03:26,546][62782] Avg episode reward: [(0, '23.499')] [2023-06-21 11:03:31,547][62782] Fps is (10 sec: 1228.6, 60 sec: 1297.1, 300 sec: 1249.6). Total num frames: 3473408. Throughput: 0: 319.0. Samples: 367606. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:03:31,549][62782] Avg episode reward: [(0, '23.500')] [2023-06-21 11:03:36,049][69878] Updated weights for policy 0, policy_version 850 (0.0007) [2023-06-21 11:03:36,543][62782] Fps is (10 sec: 1229.1, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3481600. Throughput: 0: 319.4. Samples: 369522. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 11:03:36,544][62782] Avg episode reward: [(0, '23.393')] [2023-06-21 11:03:41,546][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3485696. Throughput: 0: 319.2. Samples: 370478. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 11:03:41,549][62782] Avg episode reward: [(0, '23.537')] [2023-06-21 11:03:46,546][62782] Fps is (10 sec: 1228.4, 60 sec: 1297.1, 300 sec: 1277.4). Total num frames: 3493888. Throughput: 0: 319.2. Samples: 372392. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 11:03:46,548][62782] Avg episode reward: [(0, '23.846')] [2023-06-21 11:03:51,546][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3497984. Throughput: 0: 319.2. Samples: 374306. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 11:03:51,548][62782] Avg episode reward: [(0, '23.548')] [2023-06-21 11:03:56,546][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1277.4). Total num frames: 3506176. Throughput: 0: 319.2. Samples: 375262. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 11:03:56,548][62782] Avg episode reward: [(0, '24.741')] [2023-06-21 11:04:01,545][62782] Fps is (10 sec: 1229.0, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3510272. Throughput: 0: 317.7. Samples: 377114. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 11:04:01,550][62782] Avg episode reward: [(0, '25.102')] [2023-06-21 11:04:06,546][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3518464. Throughput: 0: 316.0. Samples: 378944. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:04:06,549][62782] Avg episode reward: [(0, '25.436')] [2023-06-21 11:04:08,927][69878] Updated weights for policy 0, policy_version 860 (0.0019) [2023-06-21 11:04:11,547][62782] Fps is (10 sec: 1228.5, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3522560. Throughput: 0: 315.8. Samples: 379898. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:04:11,551][62782] Avg episode reward: [(0, '24.798')] [2023-06-21 11:04:16,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3530752. Throughput: 0: 315.1. Samples: 381786. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:04:16,552][62782] Avg episode reward: [(0, '24.485')] [2023-06-21 11:04:21,550][62782] Fps is (10 sec: 1228.5, 60 sec: 1228.7, 300 sec: 1263.5). Total num frames: 3534848. Throughput: 0: 313.9. Samples: 383650. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:04:21,556][62782] Avg episode reward: [(0, '25.309')] [2023-06-21 11:04:26,546][62782] Fps is (10 sec: 1229.0, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3543040. Throughput: 0: 313.4. Samples: 384582. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:04:26,552][62782] Avg episode reward: [(0, '24.992')] [2023-06-21 11:04:31,548][62782] Fps is (10 sec: 1638.7, 60 sec: 1297.1, 300 sec: 1277.4). Total num frames: 3551232. Throughput: 0: 312.9. Samples: 386474. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:04:31,550][62782] Avg episode reward: [(0, '25.768')] [2023-06-21 11:04:36,545][62782] Fps is (10 sec: 1229.0, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3555328. Throughput: 0: 312.5. Samples: 388366. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:04:36,547][62782] Avg episode reward: [(0, '25.919')] [2023-06-21 11:04:41,135][69878] Updated weights for policy 0, policy_version 870 (0.0018) [2023-06-21 11:04:41,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.0, 300 sec: 1277.4). Total num frames: 3563520. Throughput: 0: 312.5. Samples: 389324. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:04:41,548][62782] Avg episode reward: [(0, '25.577')] [2023-06-21 11:04:46,546][62782] Fps is (10 sec: 1228.6, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3567616. Throughput: 0: 313.4. Samples: 391216. Policy #0 lag: (min: 0.0, avg: 0.9, max: 2.0) [2023-06-21 11:04:46,550][62782] Avg episode reward: [(0, '25.228')] [2023-06-21 11:04:51,545][62782] Fps is (10 sec: 1229.1, 60 sec: 1297.1, 300 sec: 1277.4). Total num frames: 3575808. Throughput: 0: 313.9. Samples: 393070. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:04:51,548][62782] Avg episode reward: [(0, '25.509')] [2023-06-21 11:04:56,546][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3579904. Throughput: 0: 313.7. Samples: 394012. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:04:56,549][62782] Avg episode reward: [(0, '26.024')] [2023-06-21 11:05:01,546][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.0, 300 sec: 1277.4). Total num frames: 3588096. Throughput: 0: 313.2. Samples: 395882. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:05:01,549][62782] Avg episode reward: [(0, '26.299')] [2023-06-21 11:05:01,554][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000876_3588096.pth... [2023-06-21 11:05:01,660][69876] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000802_3284992.pth [2023-06-21 11:05:06,547][62782] Fps is (10 sec: 1228.6, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3592192. Throughput: 0: 313.9. Samples: 397776. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:05:06,550][62782] Avg episode reward: [(0, '26.177')] [2023-06-21 11:05:11,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3600384. Throughput: 0: 314.2. Samples: 398722. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:05:11,550][62782] Avg episode reward: [(0, '25.293')] [2023-06-21 11:05:14,135][69878] Updated weights for policy 0, policy_version 880 (0.0018) [2023-06-21 11:05:16,548][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3604480. Throughput: 0: 314.2. Samples: 400614. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:05:16,551][62782] Avg episode reward: [(0, '25.162')] [2023-06-21 11:05:21,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3612672. Throughput: 0: 314.2. Samples: 402506. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:05:21,548][62782] Avg episode reward: [(0, '25.204')] [2023-06-21 11:05:26,542][62782] Fps is (10 sec: 1229.4, 60 sec: 1228.9, 300 sec: 1263.5). Total num frames: 3616768. Throughput: 0: 313.9. Samples: 403448. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:05:26,546][62782] Avg episode reward: [(0, '24.798')] [2023-06-21 11:05:31,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3624960. Throughput: 0: 313.8. Samples: 405338. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:05:31,550][62782] Avg episode reward: [(0, '24.380')] [2023-06-21 11:05:36,547][62782] Fps is (10 sec: 1228.3, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3629056. Throughput: 0: 315.0. Samples: 407244. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:05:36,550][62782] Avg episode reward: [(0, '23.755')] [2023-06-21 11:05:41,546][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3637248. Throughput: 0: 315.0. Samples: 408188. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:05:41,550][62782] Avg episode reward: [(0, '23.390')] [2023-06-21 11:05:46,350][69878] Updated weights for policy 0, policy_version 890 (0.0013) [2023-06-21 11:05:46,547][62782] Fps is (10 sec: 1638.3, 60 sec: 1297.0, 300 sec: 1277.4). Total num frames: 3645440. Throughput: 0: 315.6. Samples: 410086. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:05:46,550][62782] Avg episode reward: [(0, '23.469')] [2023-06-21 11:05:51,546][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3649536. Throughput: 0: 315.7. Samples: 411984. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:05:51,548][62782] Avg episode reward: [(0, '23.962')] [2023-06-21 11:05:56,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.0, 300 sec: 1277.4). Total num frames: 3657728. Throughput: 0: 315.8. Samples: 412932. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:05:56,549][62782] Avg episode reward: [(0, '23.272')] [2023-06-21 11:06:01,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3661824. Throughput: 0: 315.7. Samples: 414822. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:06:01,551][62782] Avg episode reward: [(0, '23.466')] [2023-06-21 11:06:06,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3670016. Throughput: 0: 315.8. Samples: 416718. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:06:06,549][62782] Avg episode reward: [(0, '23.595')] [2023-06-21 11:06:11,546][62782] Fps is (10 sec: 1228.9, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3674112. Throughput: 0: 315.9. Samples: 417664. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:06:11,550][62782] Avg episode reward: [(0, '23.447')] [2023-06-21 11:06:16,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3682304. Throughput: 0: 316.0. Samples: 419560. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:06:16,549][62782] Avg episode reward: [(0, '24.782')] [2023-06-21 11:06:18,870][69878] Updated weights for policy 0, policy_version 900 (0.0014) [2023-06-21 11:06:21,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3686400. Throughput: 0: 315.9. Samples: 421458. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:06:21,552][62782] Avg episode reward: [(0, '25.187')] [2023-06-21 11:06:26,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.0, 300 sec: 1263.5). Total num frames: 3694592. Throughput: 0: 315.9. Samples: 422404. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:06:26,550][62782] Avg episode reward: [(0, '23.964')] [2023-06-21 11:06:31,548][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3698688. Throughput: 0: 315.6. Samples: 424286. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:06:31,551][62782] Avg episode reward: [(0, '23.800')] [2023-06-21 11:06:36,546][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3706880. Throughput: 0: 315.1. Samples: 426162. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:06:36,549][62782] Avg episode reward: [(0, '24.069')] [2023-06-21 11:06:41,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3710976. Throughput: 0: 315.1. Samples: 427110. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:06:41,550][62782] Avg episode reward: [(0, '24.788')] [2023-06-21 11:06:46,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3719168. Throughput: 0: 315.1. Samples: 429002. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:06:46,551][62782] Avg episode reward: [(0, '25.008')] [2023-06-21 11:06:51,493][69878] Updated weights for policy 0, policy_version 910 (0.0012) [2023-06-21 11:06:51,544][62782] Fps is (10 sec: 1638.9, 60 sec: 1297.1, 300 sec: 1263.6). Total num frames: 3727360. Throughput: 0: 315.1. Samples: 430896. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:06:51,546][62782] Avg episode reward: [(0, '25.131')] [2023-06-21 11:06:56,546][62782] Fps is (10 sec: 1229.0, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3731456. Throughput: 0: 315.2. Samples: 431846. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:06:56,548][62782] Avg episode reward: [(0, '25.242')] [2023-06-21 11:07:01,545][62782] Fps is (10 sec: 1228.6, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3739648. Throughput: 0: 315.1. Samples: 433738. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:07:01,547][62782] Avg episode reward: [(0, '25.968')] [2023-06-21 11:07:01,550][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000913_3739648.pth... [2023-06-21 11:07:01,662][69876] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000839_3436544.pth [2023-06-21 11:07:06,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3743744. Throughput: 0: 314.8. Samples: 435626. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:07:06,550][62782] Avg episode reward: [(0, '25.110')] [2023-06-21 11:07:11,545][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3751936. Throughput: 0: 315.0. Samples: 436578. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:07:11,547][62782] Avg episode reward: [(0, '24.866')] [2023-06-21 11:07:16,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3756032. Throughput: 0: 315.3. Samples: 438476. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:07:16,550][62782] Avg episode reward: [(0, '25.067')] [2023-06-21 11:07:21,545][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1277.4). Total num frames: 3764224. Throughput: 0: 315.7. Samples: 440370. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:07:21,547][62782] Avg episode reward: [(0, '24.682')] [2023-06-21 11:07:23,952][69878] Updated weights for policy 0, policy_version 920 (0.0016) [2023-06-21 11:07:26,550][62782] Fps is (10 sec: 1228.6, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3768320. Throughput: 0: 315.6. Samples: 441312. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:07:26,569][62782] Avg episode reward: [(0, '23.626')] [2023-06-21 11:07:31,547][62782] Fps is (10 sec: 1228.6, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3776512. Throughput: 0: 315.5. Samples: 443198. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:07:31,550][62782] Avg episode reward: [(0, '23.502')] [2023-06-21 11:07:36,550][62782] Fps is (10 sec: 1228.9, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3780608. Throughput: 0: 315.7. Samples: 445102. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:07:36,564][62782] Avg episode reward: [(0, '22.587')] [2023-06-21 11:07:41,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3788800. Throughput: 0: 315.5. Samples: 446042. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:07:41,549][62782] Avg episode reward: [(0, '22.696')] [2023-06-21 11:07:46,547][62782] Fps is (10 sec: 1228.9, 60 sec: 1228.8, 300 sec: 1249.6). Total num frames: 3792896. Throughput: 0: 315.6. Samples: 447940. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:07:46,550][62782] Avg episode reward: [(0, '22.409')] [2023-06-21 11:07:51,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.7, 300 sec: 1263.5). Total num frames: 3801088. Throughput: 0: 315.6. Samples: 449828. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:07:51,551][62782] Avg episode reward: [(0, '22.894')] [2023-06-21 11:07:56,385][69878] Updated weights for policy 0, policy_version 930 (0.0019) [2023-06-21 11:07:56,547][62782] Fps is (10 sec: 1638.4, 60 sec: 1297.0, 300 sec: 1263.5). Total num frames: 3809280. Throughput: 0: 315.5. Samples: 450778. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:07:56,549][62782] Avg episode reward: [(0, '23.598')] [2023-06-21 11:08:01,547][62782] Fps is (10 sec: 1228.9, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3813376. Throughput: 0: 315.6. Samples: 452678. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:08:01,550][62782] Avg episode reward: [(0, '23.715')] [2023-06-21 11:08:06,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3821568. Throughput: 0: 315.6. Samples: 454572. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:08:06,549][62782] Avg episode reward: [(0, '23.309')] [2023-06-21 11:08:11,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.7, 300 sec: 1263.5). Total num frames: 3825664. Throughput: 0: 315.7. Samples: 455518. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:08:11,551][62782] Avg episode reward: [(0, '22.948')] [2023-06-21 11:08:16,545][62782] Fps is (10 sec: 1229.1, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3833856. Throughput: 0: 315.9. Samples: 457414. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:08:16,547][62782] Avg episode reward: [(0, '22.730')] [2023-06-21 11:08:21,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1249.6). Total num frames: 3837952. Throughput: 0: 315.7. Samples: 459310. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:08:21,550][62782] Avg episode reward: [(0, '22.709')] [2023-06-21 11:08:26,546][62782] Fps is (10 sec: 1228.6, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3846144. Throughput: 0: 315.7. Samples: 460248. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:08:26,549][62782] Avg episode reward: [(0, '22.977')] [2023-06-21 11:08:28,913][69878] Updated weights for policy 0, policy_version 940 (0.0015) [2023-06-21 11:08:31,548][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1249.6). Total num frames: 3850240. Throughput: 0: 315.7. Samples: 462148. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:08:31,549][62782] Avg episode reward: [(0, '23.765')] [2023-06-21 11:08:36,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3858432. Throughput: 0: 316.1. Samples: 464052. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:08:36,550][62782] Avg episode reward: [(0, '23.574')] [2023-06-21 11:08:41,545][62782] Fps is (10 sec: 1229.1, 60 sec: 1228.8, 300 sec: 1249.6). Total num frames: 3862528. Throughput: 0: 315.9. Samples: 464994. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:08:41,548][62782] Avg episode reward: [(0, '23.452')] [2023-06-21 11:08:46,547][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3870720. Throughput: 0: 315.5. Samples: 466876. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:08:46,550][62782] Avg episode reward: [(0, '23.792')] [2023-06-21 11:08:51,547][62782] Fps is (10 sec: 1228.6, 60 sec: 1228.8, 300 sec: 1249.6). Total num frames: 3874816. Throughput: 0: 315.3. Samples: 468762. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:08:51,550][62782] Avg episode reward: [(0, '25.539')] [2023-06-21 11:08:56,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3883008. Throughput: 0: 315.5. Samples: 469716. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:08:56,550][62782] Avg episode reward: [(0, '25.785')] [2023-06-21 11:09:01,429][69878] Updated weights for policy 0, policy_version 950 (0.0018) [2023-06-21 11:09:01,547][62782] Fps is (10 sec: 1638.4, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3891200. Throughput: 0: 315.5. Samples: 471614. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:09:01,548][62782] Avg episode reward: [(0, '26.302')] [2023-06-21 11:09:01,553][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000950_3891200.pth... [2023-06-21 11:09:01,654][69876] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000876_3588096.pth [2023-06-21 11:09:06,546][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1263.5). Total num frames: 3895296. Throughput: 0: 315.2. Samples: 473496. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:09:06,550][62782] Avg episode reward: [(0, '26.181')] [2023-06-21 11:09:11,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3903488. Throughput: 0: 315.2. Samples: 474434. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:09:11,549][62782] Avg episode reward: [(0, '26.355')] [2023-06-21 11:09:16,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.7, 300 sec: 1263.5). Total num frames: 3907584. Throughput: 0: 315.0. Samples: 476324. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:09:16,550][62782] Avg episode reward: [(0, '26.452')] [2023-06-21 11:09:21,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3915776. Throughput: 0: 314.6. Samples: 478210. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:09:21,550][62782] Avg episode reward: [(0, '26.087')] [2023-06-21 11:09:26,548][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.8, 300 sec: 1249.6). Total num frames: 3919872. Throughput: 0: 314.8. Samples: 479160. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:09:26,564][62782] Avg episode reward: [(0, '26.313')] [2023-06-21 11:09:31,547][62782] Fps is (10 sec: 1228.9, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3928064. Throughput: 0: 315.2. Samples: 481062. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:09:31,550][62782] Avg episode reward: [(0, '26.155')] [2023-06-21 11:09:33,951][69878] Updated weights for policy 0, policy_version 960 (0.0020) [2023-06-21 11:09:36,548][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1249.6). Total num frames: 3932160. Throughput: 0: 315.4. Samples: 482954. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:09:36,551][62782] Avg episode reward: [(0, '26.243')] [2023-06-21 11:09:41,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.0, 300 sec: 1263.5). Total num frames: 3940352. Throughput: 0: 315.2. Samples: 483900. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:09:41,551][62782] Avg episode reward: [(0, '26.313')] [2023-06-21 11:09:46,546][62782] Fps is (10 sec: 1229.0, 60 sec: 1228.8, 300 sec: 1249.6). Total num frames: 3944448. Throughput: 0: 315.1. Samples: 485794. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:09:46,549][62782] Avg episode reward: [(0, '26.556')] [2023-06-21 11:09:51,547][62782] Fps is (10 sec: 1228.8, 60 sec: 1297.1, 300 sec: 1263.5). Total num frames: 3952640. Throughput: 0: 315.5. Samples: 487694. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:09:51,556][62782] Avg episode reward: [(0, '27.144')] [2023-06-21 11:09:53,394][69876] Saving new best policy, reward=27.144! [2023-06-21 11:09:56,547][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.8, 300 sec: 1249.6). Total num frames: 3956736. Throughput: 0: 315.6. Samples: 488638. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:09:56,550][62782] Avg episode reward: [(0, '27.272')] [2023-06-21 11:09:56,664][69876] Saving new best policy, reward=27.272! [2023-06-21 11:10:01,551][62782] Fps is (10 sec: 1228.3, 60 sec: 1228.7, 300 sec: 1263.5). Total num frames: 3964928. Throughput: 0: 313.2. Samples: 490418. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:10:01,555][62782] Avg episode reward: [(0, '26.251')] [2023-06-21 11:10:06,550][62782] Fps is (10 sec: 1228.5, 60 sec: 1228.7, 300 sec: 1249.6). Total num frames: 3969024. Throughput: 0: 309.0. Samples: 492116. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:10:06,554][62782] Avg episode reward: [(0, '26.864')] [2023-06-21 11:10:07,274][69878] Updated weights for policy 0, policy_version 970 (0.0022) [2023-06-21 11:10:11,551][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.7, 300 sec: 1263.5). Total num frames: 3977216. Throughput: 0: 306.5. Samples: 492956. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:10:11,553][62782] Avg episode reward: [(0, '26.681')] [2023-06-21 11:10:16,551][62782] Fps is (10 sec: 1228.7, 60 sec: 1228.7, 300 sec: 1249.6). Total num frames: 3981312. Throughput: 0: 301.7. Samples: 494640. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:10:16,556][62782] Avg episode reward: [(0, '26.049')] [2023-06-21 11:10:21,552][62782] Fps is (10 sec: 819.1, 60 sec: 1160.4, 300 sec: 1249.6). Total num frames: 3985408. Throughput: 0: 297.3. Samples: 496332. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:10:21,556][62782] Avg episode reward: [(0, '26.103')] [2023-06-21 11:10:26,551][62782] Fps is (10 sec: 1228.8, 60 sec: 1228.7, 300 sec: 1249.6). Total num frames: 3993600. Throughput: 0: 294.9. Samples: 497172. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:10:26,553][62782] Avg episode reward: [(0, '25.261')] [2023-06-21 11:10:31,552][62782] Fps is (10 sec: 1228.8, 60 sec: 1160.4, 300 sec: 1249.6). Total num frames: 3997696. Throughput: 0: 290.1. Samples: 498850. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:10:31,558][62782] Avg episode reward: [(0, '25.540')] [2023-06-21 11:10:36,268][69876] Stopping Batcher_0... [2023-06-21 11:10:36,276][69876] Loop batcher_evt_loop terminating... [2023-06-21 11:10:36,280][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... [2023-06-21 11:10:36,274][62782] Component Batcher_0 stopped! [2023-06-21 11:10:36,372][69876] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000913_3739648.pth [2023-06-21 11:10:36,412][69876] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... [2023-06-21 11:10:36,504][69876] Stopping LearnerWorker_p0... [2023-06-21 11:10:36,504][69876] Loop learner_proc0_evt_loop terminating... [2023-06-21 11:10:36,504][62782] Component LearnerWorker_p0 stopped! [2023-06-21 11:10:36,799][62782] Component RolloutWorker_w3 stopped! [2023-06-21 11:10:36,803][62782] Component RolloutWorker_w6 stopped! [2023-06-21 11:10:36,805][62782] Component RolloutWorker_w2 stopped! [2023-06-21 11:10:36,806][62782] Component RolloutWorker_w4 stopped! [2023-06-21 11:10:36,806][62782] Component RolloutWorker_w5 stopped! [2023-06-21 11:10:36,805][69884] Stopping RolloutWorker_w6... [2023-06-21 11:10:36,807][62782] Component RolloutWorker_w1 stopped! [2023-06-21 11:10:36,813][69884] Loop rollout_proc6_evt_loop terminating... [2023-06-21 11:10:36,805][69881] Stopping RolloutWorker_w3... [2023-06-21 11:10:36,812][69882] Stopping RolloutWorker_w4... [2023-06-21 11:10:36,817][69881] Loop rollout_proc3_evt_loop terminating... [2023-06-21 11:10:36,815][69883] Stopping RolloutWorker_w5... [2023-06-21 11:10:36,822][69882] Loop rollout_proc4_evt_loop terminating... [2023-06-21 11:10:36,817][69880] Stopping RolloutWorker_w2... [2023-06-21 11:10:36,823][69883] Loop rollout_proc5_evt_loop terminating... [2023-06-21 11:10:36,823][69885] Stopping RolloutWorker_w7... [2023-06-21 11:10:36,827][69880] Loop rollout_proc2_evt_loop terminating... [2023-06-21 11:10:36,818][62782] Component RolloutWorker_w7 stopped! [2023-06-21 11:10:36,829][69885] Loop rollout_proc7_evt_loop terminating... [2023-06-21 11:10:36,828][62782] Component RolloutWorker_w0 stopped! [2023-06-21 11:10:36,828][69877] Stopping RolloutWorker_w0... [2023-06-21 11:10:36,823][69879] Stopping RolloutWorker_w1... [2023-06-21 11:10:36,832][69879] Loop rollout_proc1_evt_loop terminating... [2023-06-21 11:10:36,833][69877] Loop rollout_proc0_evt_loop terminating... [2023-06-21 11:10:37,037][69878] Weights refcount: 2 0 [2023-06-21 11:10:37,043][69878] Stopping InferenceWorker_p0-w0... [2023-06-21 11:10:37,045][69878] Loop inference_proc0-0_evt_loop terminating... [2023-06-21 11:10:37,044][62782] Component InferenceWorker_p0-w0 stopped! [2023-06-21 11:10:37,049][62782] Waiting for process learner_proc0 to stop... [2023-06-21 11:10:37,863][62782] Waiting for process inference_proc0-0 to join... [2023-06-21 11:10:37,866][62782] Waiting for process rollout_proc0 to join... [2023-06-21 11:10:37,866][62782] Waiting for process rollout_proc1 to join... [2023-06-21 11:10:37,867][62782] Waiting for process rollout_proc2 to join... [2023-06-21 11:10:37,867][62782] Waiting for process rollout_proc3 to join... [2023-06-21 11:10:37,867][62782] Waiting for process rollout_proc4 to join... [2023-06-21 11:10:37,868][62782] Waiting for process rollout_proc5 to join... [2023-06-21 11:10:37,872][62782] Waiting for process rollout_proc6 to join... [2023-06-21 11:10:37,872][62782] Waiting for process rollout_proc7 to join... [2023-06-21 11:10:37,873][62782] Batcher 0 profile tree view: batching: 6.4846, releasing_batches: 0.0073 [2023-06-21 11:10:37,874][62782] InferenceWorker_p0-w0 profile tree view: wait_policy: 0.0000 wait_policy_total: 1040.7824 update_model: 3.6621 weight_update: 0.0037 one_step: 0.0059 handle_policy_step: 591.4375 deserialize: 5.7777, stack: 1.0636, obs_to_device_normalize: 41.5510, forward: 517.7502, send_messages: 6.4697 prepare_outputs: 7.6375 to_cpu: 0.8247 [2023-06-21 11:10:37,874][62782] Learner 0 profile tree view: misc: 0.0015, prepare_batch: 240.7255 train: 843.3665 epoch_init: 0.0017, minibatch_init: 0.0076, losses_postprocess: 0.0174, kl_divergence: 0.0864, after_optimizer: 0.5407 calculate_losses: 479.9649 losses_init: 0.0010, forward_head: 462.8699, bptt_initial: 0.8487, tail: 0.8752, advantages_returns: 0.0856, losses: 0.3270 bptt: 14.7727 bptt_forward_core: 14.4951 update: 362.3651 clip: 0.8043 [2023-06-21 11:10:37,875][62782] RolloutWorker_w0 profile tree view: wait_for_trajectories: 0.0607, enqueue_policy_requests: 4.2417, env_step: 1623.2834, overhead: 4.3230, complete_rollouts: 0.0809 save_policy_outputs: 2.0853 split_output_tensors: 1.0207 [2023-06-21 11:10:37,875][62782] RolloutWorker_w7 profile tree view: wait_for_trajectories: 0.0567, enqueue_policy_requests: 4.4229, env_step: 1622.7306, overhead: 4.2237, complete_rollouts: 0.0818 save_policy_outputs: 2.0102 split_output_tensors: 0.9789 [2023-06-21 11:10:37,876][62782] Loop Runner_EvtLoop terminating... [2023-06-21 11:10:37,877][62782] Runner profile tree view: main_loop: 1654.8298 [2023-06-21 11:10:37,877][62782] Collected {0: 4005888}, FPS: 1207.9 [2023-06-21 11:16:21,383][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 11:16:21,386][62782] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-21 11:16:21,387][62782] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-21 11:16:21,388][62782] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-21 11:16:21,389][62782] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 11:16:21,390][62782] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-21 11:16:21,391][62782] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 11:16:21,391][62782] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-21 11:16:21,392][62782] Adding new argument 'push_to_hub'=False that is not in the saved config file! [2023-06-21 11:16:21,393][62782] Adding new argument 'hf_repository'=None that is not in the saved config file! [2023-06-21 11:16:21,394][62782] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-21 11:16:21,395][62782] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-21 11:16:21,395][62782] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-21 11:16:21,396][62782] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-21 11:16:21,397][62782] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-21 11:16:21,415][62782] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 11:16:21,419][62782] RunningMeanStd input shape: (1,) [2023-06-21 11:16:21,451][62782] ConvEncoder: input_channels=3 [2023-06-21 11:16:21,480][62782] Conv encoder output size: 512 [2023-06-21 11:16:21,480][62782] Policy head output size: 512 [2023-06-21 11:16:21,492][62782] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... [2023-06-21 11:16:23,138][62782] Num frames 100... [2023-06-21 11:16:23,858][62782] Num frames 200... [2023-06-21 11:16:24,581][62782] Num frames 300... [2023-06-21 11:16:25,374][62782] Num frames 400... [2023-06-21 11:16:26,147][62782] Num frames 500... [2023-06-21 11:16:26,855][62782] Num frames 600... [2023-06-21 11:16:27,557][62782] Num frames 700... [2023-06-21 11:16:28,344][62782] Num frames 800... [2023-06-21 11:16:29,079][62782] Num frames 900... [2023-06-21 11:16:29,865][62782] Avg episode rewards: #0: 18.920, true rewards: #0: 9.920 [2023-06-21 11:16:29,867][62782] Avg episode reward: 18.920, avg true_objective: 9.920 [2023-06-21 11:16:29,926][62782] Num frames 1000... [2023-06-21 11:16:30,751][62782] Num frames 1100... [2023-06-21 11:16:31,510][62782] Num frames 1200... [2023-06-21 11:16:32,203][62782] Num frames 1300... [2023-06-21 11:16:32,965][62782] Num frames 1400... [2023-06-21 11:16:33,763][62782] Num frames 1500... [2023-06-21 11:16:34,573][62782] Num frames 1600... [2023-06-21 11:16:35,275][62782] Num frames 1700... [2023-06-21 11:16:36,115][62782] Num frames 1800... [2023-06-21 11:16:36,874][62782] Num frames 1900... [2023-06-21 11:16:37,734][62782] Num frames 2000... [2023-06-21 11:16:38,601][62782] Num frames 2100... [2023-06-21 11:16:39,493][62782] Num frames 2200... [2023-06-21 11:16:40,265][62782] Num frames 2300... [2023-06-21 11:16:40,901][62782] Avg episode rewards: #0: 24.340, true rewards: #0: 11.840 [2023-06-21 11:16:40,902][62782] Avg episode reward: 24.340, avg true_objective: 11.840 [2023-06-21 11:16:41,131][62782] Num frames 2400... [2023-06-21 11:16:41,884][62782] Num frames 2500... [2023-06-21 11:16:42,662][62782] Num frames 2600... [2023-06-21 11:16:43,439][62782] Num frames 2700... [2023-06-21 11:16:44,197][62782] Num frames 2800... [2023-06-21 11:16:44,924][62782] Num frames 2900... [2023-06-21 11:16:45,328][62782] Avg episode rewards: #0: 19.147, true rewards: #0: 9.813 [2023-06-21 11:16:45,330][62782] Avg episode reward: 19.147, avg true_objective: 9.813 [2023-06-21 11:16:45,765][62782] Num frames 3000... [2023-06-21 11:16:46,676][62782] Num frames 3100... [2023-06-21 11:16:47,565][62782] Num frames 3200... [2023-06-21 11:16:48,404][62782] Num frames 3300... [2023-06-21 11:16:49,223][62782] Num frames 3400... [2023-06-21 11:16:49,974][62782] Num frames 3500... [2023-06-21 11:16:50,730][62782] Num frames 3600... [2023-06-21 11:16:51,466][62782] Num frames 3700... [2023-06-21 11:16:52,209][62782] Num frames 3800... [2023-06-21 11:16:52,991][62782] Num frames 3900... [2023-06-21 11:16:53,767][62782] Num frames 4000... [2023-06-21 11:16:54,514][62782] Num frames 4100... [2023-06-21 11:16:55,229][62782] Num frames 4200... [2023-06-21 11:16:55,982][62782] Num frames 4300... [2023-06-21 11:16:56,730][62782] Num frames 4400... [2023-06-21 11:16:57,481][62782] Num frames 4500... [2023-06-21 11:16:58,347][62782] Num frames 4600... [2023-06-21 11:16:58,979][62782] Avg episode rewards: #0: 24.900, true rewards: #0: 11.650 [2023-06-21 11:16:58,981][62782] Avg episode reward: 24.900, avg true_objective: 11.650 [2023-06-21 11:16:59,339][62782] Num frames 4700... [2023-06-21 11:17:00,240][62782] Num frames 4800... [2023-06-21 11:17:01,079][62782] Num frames 4900... [2023-06-21 11:17:01,975][62782] Num frames 5000... [2023-06-21 11:17:02,901][62782] Num frames 5100... [2023-06-21 11:17:03,697][62782] Num frames 5200... [2023-06-21 11:17:04,462][62782] Num frames 5300... [2023-06-21 11:17:05,336][62782] Num frames 5400... [2023-06-21 11:17:06,193][62782] Num frames 5500... [2023-06-21 11:17:07,090][62782] Num frames 5600... [2023-06-21 11:17:07,969][62782] Num frames 5700... [2023-06-21 11:17:08,827][62782] Num frames 5800... [2023-06-21 11:17:09,737][62782] Num frames 5900... [2023-06-21 11:17:10,594][62782] Num frames 6000... [2023-06-21 11:17:11,469][62782] Num frames 6100... [2023-06-21 11:17:12,328][62782] Num frames 6200... [2023-06-21 11:17:13,224][62782] Num frames 6300... [2023-06-21 11:17:13,825][62782] Avg episode rewards: #0: 28.912, true rewards: #0: 12.712 [2023-06-21 11:17:13,826][62782] Avg episode reward: 28.912, avg true_objective: 12.712 [2023-06-21 11:17:14,215][62782] Num frames 6400... [2023-06-21 11:17:15,081][62782] Num frames 6500... [2023-06-21 11:17:15,949][62782] Num frames 6600... [2023-06-21 11:17:16,786][62782] Num frames 6700... [2023-06-21 11:17:17,649][62782] Num frames 6800... [2023-06-21 11:17:18,524][62782] Num frames 6900... [2023-06-21 11:17:19,414][62782] Num frames 7000... [2023-06-21 11:17:20,294][62782] Num frames 7100... [2023-06-21 11:17:21,163][62782] Num frames 7200... [2023-06-21 11:17:22,050][62782] Num frames 7300... [2023-06-21 11:17:22,825][62782] Num frames 7400... [2023-06-21 11:17:23,291][62782] Avg episode rewards: #0: 28.407, true rewards: #0: 12.407 [2023-06-21 11:17:23,294][62782] Avg episode reward: 28.407, avg true_objective: 12.407 [2023-06-21 11:17:23,778][62782] Num frames 7500... [2023-06-21 11:17:24,622][62782] Num frames 7600... [2023-06-21 11:17:25,563][62782] Num frames 7700... [2023-06-21 11:17:26,412][62782] Num frames 7800... [2023-06-21 11:17:27,300][62782] Num frames 7900... [2023-06-21 11:17:28,218][62782] Num frames 8000... [2023-06-21 11:17:28,491][62782] Avg episode rewards: #0: 25.743, true rewards: #0: 11.457 [2023-06-21 11:17:28,493][62782] Avg episode reward: 25.743, avg true_objective: 11.457 [2023-06-21 11:17:29,162][62782] Num frames 8100... [2023-06-21 11:17:30,053][62782] Num frames 8200... [2023-06-21 11:17:30,929][62782] Num frames 8300... [2023-06-21 11:17:31,823][62782] Num frames 8400... [2023-06-21 11:17:32,668][62782] Num frames 8500... [2023-06-21 11:17:33,522][62782] Num frames 8600... [2023-06-21 11:17:34,391][62782] Num frames 8700... [2023-06-21 11:17:35,307][62782] Num frames 8800... [2023-06-21 11:17:36,172][62782] Num frames 8900... [2023-06-21 11:17:37,060][62782] Num frames 9000... [2023-06-21 11:17:37,904][62782] Num frames 9100... [2023-06-21 11:17:38,724][62782] Num frames 9200... [2023-06-21 11:17:39,605][62782] Num frames 9300... [2023-06-21 11:17:40,491][62782] Num frames 9400... [2023-06-21 11:17:41,440][62782] Num frames 9500... [2023-06-21 11:17:42,310][62782] Num frames 9600... [2023-06-21 11:17:43,191][62782] Num frames 9700... [2023-06-21 11:17:44,114][62782] Num frames 9800... [2023-06-21 11:17:44,945][62782] Num frames 9900... [2023-06-21 11:17:45,799][62782] Num frames 10000... [2023-06-21 11:17:46,711][62782] Num frames 10100... [2023-06-21 11:17:46,990][62782] Avg episode rewards: #0: 29.650, true rewards: #0: 12.650 [2023-06-21 11:17:46,992][62782] Avg episode reward: 29.650, avg true_objective: 12.650 [2023-06-21 11:17:47,741][62782] Num frames 10200... [2023-06-21 11:17:48,549][62782] Num frames 10300... [2023-06-21 11:17:49,448][62782] Num frames 10400... [2023-06-21 11:17:50,380][62782] Num frames 10500... [2023-06-21 11:17:51,278][62782] Num frames 10600... [2023-06-21 11:17:52,210][62782] Num frames 10700... [2023-06-21 11:17:53,150][62782] Num frames 10800... [2023-06-21 11:17:54,110][62782] Num frames 10900... [2023-06-21 11:17:55,057][62782] Num frames 11000... [2023-06-21 11:17:55,972][62782] Num frames 11100... [2023-06-21 11:17:56,884][62782] Num frames 11200... [2023-06-21 11:17:57,846][62782] Num frames 11300... [2023-06-21 11:17:58,805][62782] Num frames 11400... [2023-06-21 11:17:59,510][62782] Avg episode rewards: #0: 30.404, true rewards: #0: 12.738 [2023-06-21 11:17:59,513][62782] Avg episode reward: 30.404, avg true_objective: 12.738 [2023-06-21 11:17:59,834][62782] Num frames 11500... [2023-06-21 11:18:00,779][62782] Num frames 11600... [2023-06-21 11:18:01,729][62782] Num frames 11700... [2023-06-21 11:18:02,663][62782] Num frames 11800... [2023-06-21 11:18:03,603][62782] Num frames 11900... [2023-06-21 11:18:04,573][62782] Num frames 12000... [2023-06-21 11:18:05,559][62782] Num frames 12100... [2023-06-21 11:18:06,517][62782] Num frames 12200... [2023-06-21 11:18:07,439][62782] Num frames 12300... [2023-06-21 11:18:07,802][62782] Avg episode rewards: #0: 29.228, true rewards: #0: 12.328 [2023-06-21 11:18:07,804][62782] Avg episode reward: 29.228, avg true_objective: 12.328 [2023-06-21 11:18:24,353][62782] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4! [2023-06-21 11:20:01,788][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 11:20:01,790][62782] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-21 11:20:01,790][62782] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-21 11:20:01,790][62782] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-21 11:20:01,791][62782] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 11:20:01,791][62782] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-21 11:20:01,791][62782] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! [2023-06-21 11:20:01,792][62782] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-21 11:20:01,792][62782] Adding new argument 'push_to_hub'=True that is not in the saved config file! [2023-06-21 11:20:01,792][62782] Adding new argument 'hf_repository'='mihirdeo16/vizdoom_health_gathering_supreme' that is not in the saved config file! [2023-06-21 11:20:01,792][62782] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-21 11:20:01,793][62782] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-21 11:20:01,793][62782] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-21 11:20:01,793][62782] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-21 11:20:01,794][62782] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-21 11:20:01,801][62782] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 11:20:01,804][62782] RunningMeanStd input shape: (1,) [2023-06-21 11:20:01,820][62782] ConvEncoder: input_channels=3 [2023-06-21 11:20:01,845][62782] Conv encoder output size: 512 [2023-06-21 11:20:01,845][62782] Policy head output size: 512 [2023-06-21 11:20:01,864][62782] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... [2023-06-21 11:20:04,020][62782] Num frames 100... [2023-06-21 11:20:04,746][62782] Num frames 200... [2023-06-21 11:20:05,556][62782] Num frames 300... [2023-06-21 11:20:06,401][62782] Avg episode rewards: #0: 3.840, true rewards: #0: 3.840 [2023-06-21 11:20:06,403][62782] Avg episode reward: 3.840, avg true_objective: 3.840 [2023-06-21 11:20:06,544][62782] Num frames 400... [2023-06-21 11:20:07,449][62782] Num frames 500... [2023-06-21 11:20:08,351][62782] Num frames 600... [2023-06-21 11:20:09,244][62782] Num frames 700... [2023-06-21 11:20:10,161][62782] Num frames 800... [2023-06-21 11:20:11,082][62782] Num frames 900... [2023-06-21 11:20:11,912][62782] Num frames 1000... [2023-06-21 11:20:12,681][62782] Num frames 1100... [2023-06-21 11:20:13,505][62782] Num frames 1200... [2023-06-21 11:20:14,384][62782] Num frames 1300... [2023-06-21 11:20:15,260][62782] Num frames 1400... [2023-06-21 11:20:16,187][62782] Num frames 1500... [2023-06-21 11:20:17,107][62782] Num frames 1600... [2023-06-21 11:20:17,998][62782] Num frames 1700... [2023-06-21 11:20:18,874][62782] Num frames 1800... [2023-06-21 11:20:19,781][62782] Num frames 1900... [2023-06-21 11:20:20,686][62782] Num frames 2000... [2023-06-21 11:20:21,590][62782] Num frames 2100... [2023-06-21 11:20:22,482][62782] Num frames 2200... [2023-06-21 11:20:22,966][62782] Avg episode rewards: #0: 23.200, true rewards: #0: 11.200 [2023-06-21 11:20:22,967][62782] Avg episode reward: 23.200, avg true_objective: 11.200 [2023-06-21 11:20:23,520][62782] Num frames 2300... [2023-06-21 11:20:24,411][62782] Num frames 2400... [2023-06-21 11:20:25,313][62782] Num frames 2500... [2023-06-21 11:20:26,198][62782] Num frames 2600... [2023-06-21 11:20:27,127][62782] Num frames 2700... [2023-06-21 11:20:28,015][62782] Num frames 2800... [2023-06-21 11:20:28,859][62782] Num frames 2900... [2023-06-21 11:20:29,781][62782] Num frames 3000... [2023-06-21 11:20:30,646][62782] Num frames 3100... [2023-06-21 11:20:31,556][62782] Num frames 3200... [2023-06-21 11:20:32,413][62782] Num frames 3300... [2023-06-21 11:20:33,280][62782] Num frames 3400... [2023-06-21 11:20:33,726][62782] Avg episode rewards: #0: 24.460, true rewards: #0: 11.460 [2023-06-21 11:20:33,726][62782] Avg episode reward: 24.460, avg true_objective: 11.460 [2023-06-21 11:20:34,281][62782] Num frames 3500... [2023-06-21 11:20:35,123][62782] Num frames 3600... [2023-06-21 11:20:36,028][62782] Num frames 3700... [2023-06-21 11:20:36,910][62782] Num frames 3800... [2023-06-21 11:20:37,497][62782] Avg episode rewards: #0: 19.885, true rewards: #0: 9.635 [2023-06-21 11:20:37,499][62782] Avg episode reward: 19.885, avg true_objective: 9.635 [2023-06-21 11:20:37,902][62782] Num frames 3900... [2023-06-21 11:20:38,810][62782] Num frames 4000... [2023-06-21 11:20:39,699][62782] Num frames 4100... [2023-06-21 11:20:40,594][62782] Num frames 4200... [2023-06-21 11:20:41,481][62782] Num frames 4300... [2023-06-21 11:20:42,363][62782] Num frames 4400... [2023-06-21 11:20:43,269][62782] Num frames 4500... [2023-06-21 11:20:43,596][62782] Avg episode rewards: #0: 18.852, true rewards: #0: 9.052 [2023-06-21 11:20:43,598][62782] Avg episode reward: 18.852, avg true_objective: 9.052 [2023-06-21 11:20:44,264][62782] Num frames 4600... [2023-06-21 11:20:45,114][62782] Num frames 4700... [2023-06-21 11:20:45,970][62782] Num frames 4800... [2023-06-21 11:20:46,884][62782] Num frames 4900... [2023-06-21 11:20:47,764][62782] Num frames 5000... [2023-06-21 11:20:48,626][62782] Num frames 5100... [2023-06-21 11:20:49,547][62782] Num frames 5200... [2023-06-21 11:20:50,453][62782] Num frames 5300... [2023-06-21 11:20:51,334][62782] Num frames 5400... [2023-06-21 11:20:51,976][62782] Avg episode rewards: #0: 20.600, true rewards: #0: 9.100 [2023-06-21 11:20:51,977][62782] Avg episode reward: 20.600, avg true_objective: 9.100 [2023-06-21 11:20:52,330][62782] Num frames 5500... [2023-06-21 11:20:53,183][62782] Num frames 5600... [2023-06-21 11:20:54,063][62782] Num frames 5700... [2023-06-21 11:20:54,947][62782] Num frames 5800... [2023-06-21 11:20:55,819][62782] Num frames 5900... [2023-06-21 11:20:56,278][62782] Avg episode rewards: #0: 18.771, true rewards: #0: 8.486 [2023-06-21 11:20:56,280][62782] Avg episode reward: 18.771, avg true_objective: 8.486 [2023-06-21 11:20:56,814][62782] Num frames 6000... [2023-06-21 11:20:57,672][62782] Num frames 6100... [2023-06-21 11:20:58,528][62782] Num frames 6200... [2023-06-21 11:20:59,432][62782] Num frames 6300... [2023-06-21 11:21:00,338][62782] Num frames 6400... [2023-06-21 11:21:01,188][62782] Num frames 6500... [2023-06-21 11:21:02,100][62782] Num frames 6600... [2023-06-21 11:21:02,969][62782] Num frames 6700... [2023-06-21 11:21:03,877][62782] Num frames 6800... [2023-06-21 11:21:04,795][62782] Num frames 6900... [2023-06-21 11:21:05,687][62782] Num frames 7000... [2023-06-21 11:21:06,589][62782] Num frames 7100... [2023-06-21 11:21:07,487][62782] Num frames 7200... [2023-06-21 11:21:08,374][62782] Num frames 7300... [2023-06-21 11:21:08,804][62782] Avg episode rewards: #0: 21.171, true rewards: #0: 9.171 [2023-06-21 11:21:08,807][62782] Avg episode reward: 21.171, avg true_objective: 9.171 [2023-06-21 11:21:09,373][62782] Num frames 7400... [2023-06-21 11:21:10,275][62782] Num frames 7500... [2023-06-21 11:21:11,165][62782] Num frames 7600... [2023-06-21 11:21:12,049][62782] Num frames 7700... [2023-06-21 11:21:12,943][62782] Num frames 7800... [2023-06-21 11:21:13,831][62782] Num frames 7900... [2023-06-21 11:21:14,636][62782] Avg episode rewards: #0: 20.530, true rewards: #0: 8.863 [2023-06-21 11:21:14,637][62782] Avg episode reward: 20.530, avg true_objective: 8.863 [2023-06-21 11:21:14,835][62782] Num frames 8000... [2023-06-21 11:21:15,696][62782] Num frames 8100... [2023-06-21 11:21:16,576][62782] Num frames 8200... [2023-06-21 11:21:17,499][62782] Num frames 8300... [2023-06-21 11:21:18,384][62782] Num frames 8400... [2023-06-21 11:21:19,276][62782] Num frames 8500... [2023-06-21 11:21:20,150][62782] Num frames 8600... [2023-06-21 11:21:20,939][62782] Num frames 8700... [2023-06-21 11:21:21,716][62782] Num frames 8800... [2023-06-21 11:21:22,488][62782] Num frames 8900... [2023-06-21 11:21:22,998][62782] Avg episode rewards: #0: 21.054, true rewards: #0: 8.954 [2023-06-21 11:21:22,998][62782] Avg episode reward: 21.054, avg true_objective: 8.954 [2023-06-21 11:21:35,304][62782] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4! [2023-06-21 11:22:21,897][62782] The model has been pushed to https://huggingface.co/mihirdeo16/vizdoom_health_gathering_supreme [2023-06-21 11:25:44,789][62782] Environment doom_basic already registered, overwriting... [2023-06-21 11:25:44,791][62782] Environment doom_two_colors_easy already registered, overwriting... [2023-06-21 11:25:44,791][62782] Environment doom_two_colors_hard already registered, overwriting... [2023-06-21 11:25:44,792][62782] Environment doom_dm already registered, overwriting... [2023-06-21 11:25:44,792][62782] Environment doom_dwango5 already registered, overwriting... [2023-06-21 11:25:44,793][62782] Environment doom_my_way_home_flat_actions already registered, overwriting... [2023-06-21 11:25:44,794][62782] Environment doom_defend_the_center_flat_actions already registered, overwriting... [2023-06-21 11:25:44,795][62782] Environment doom_my_way_home already registered, overwriting... [2023-06-21 11:25:44,796][62782] Environment doom_deadly_corridor already registered, overwriting... [2023-06-21 11:25:44,796][62782] Environment doom_defend_the_center already registered, overwriting... [2023-06-21 11:25:44,797][62782] Environment doom_defend_the_line already registered, overwriting... [2023-06-21 11:25:44,797][62782] Environment doom_health_gathering already registered, overwriting... [2023-06-21 11:25:44,798][62782] Environment doom_health_gathering_supreme already registered, overwriting... [2023-06-21 11:25:44,798][62782] Environment doom_battle already registered, overwriting... [2023-06-21 11:25:44,799][62782] Environment doom_battle2 already registered, overwriting... [2023-06-21 11:25:44,799][62782] Environment doom_duel_bots already registered, overwriting... [2023-06-21 11:25:44,800][62782] Environment doom_deathmatch_bots already registered, overwriting... [2023-06-21 11:25:44,800][62782] Environment doom_duel already registered, overwriting... [2023-06-21 11:25:44,800][62782] Environment doom_deathmatch_full already registered, overwriting... [2023-06-21 11:25:44,801][62782] Environment doom_benchmark already registered, overwriting... [2023-06-21 11:25:44,802][62782] register_encoder_factory: [2023-06-21 11:25:44,827][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 11:25:44,828][62782] Overriding arg 'train_for_env_steps' with value 5000000 passed from command line [2023-06-21 11:25:44,842][62782] Experiment dir /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment already exists! [2023-06-21 11:25:44,842][62782] Resuming existing experiment from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment... [2023-06-21 11:25:44,842][62782] Weights and Biases integration disabled [2023-06-21 11:25:44,849][62782] Environment var CUDA_VISIBLE_DEVICES is [2023-06-21 11:25:46,092][62782] Starting experiment with the following configuration: help=False algo=APPO env=doom_health_gathering_supreme experiment=default_experiment train_dir=/Users/md/Code/python/jubilant-memory/RL/train_dir restart_behavior=resume device=cpu seed=None num_policies=1 async_rl=True serial_mode=False batched_sampling=False num_batches_to_accumulate=2 worker_num_splits=2 policy_workers_per_policy=1 max_policy_lag=1000 num_workers=8 num_envs_per_worker=4 batch_size=800 num_batches_per_epoch=1 num_epochs=1 rollout=32 recurrence=32 shuffle_minibatches=False gamma=0.99 reward_scale=1.0 reward_clip=1000.0 value_bootstrap=False normalize_returns=True exploration_loss_coeff=0.001 value_loss_coeff=0.5 kl_loss_coeff=0.0 exploration_loss=symmetric_kl gae_lambda=0.95 ppo_clip_ratio=0.1 ppo_clip_value=0.2 with_vtrace=False vtrace_rho=1.0 vtrace_c=1.0 optimizer=adam adam_eps=1e-06 adam_beta1=0.9 adam_beta2=0.999 max_grad_norm=4.0 learning_rate=0.0001 lr_schedule=constant lr_schedule_kl_threshold=0.008 lr_adaptive_min=1e-06 lr_adaptive_max=0.01 obs_subtract_mean=0.0 obs_scale=255.0 normalize_input=True normalize_input_keys=None decorrelate_experience_max_seconds=0 decorrelate_envs_on_one_worker=True actor_worker_gpus=[] set_workers_cpu_affinity=True force_envs_single_thread=False default_niceness=0 log_to_file=True experiment_summaries_interval=10 flush_summaries_interval=30 stats_avg=100 summaries_use_frameskip=True heartbeat_interval=20 heartbeat_reporting_interval=600 train_for_env_steps=5000000 train_for_seconds=10000000000 save_every_sec=120 keep_checkpoints=2 load_checkpoint_kind=latest save_milestones_sec=-1 save_best_every_sec=5 save_best_metric=reward save_best_after=100000 benchmark=False encoder_mlp_layers=[512, 512] encoder_conv_architecture=convnet_simple encoder_conv_mlp_layers=[512] use_rnn=True rnn_size=512 rnn_type=gru rnn_num_layers=1 decoder_mlp_layers=[] nonlinearity=elu policy_initialization=orthogonal policy_init_gain=1.0 actor_critic_share_weights=True adaptive_stddev=True continuous_tanh_scale=0.0 initial_stddev=1.0 use_env_info_cache=False env_gpu_actions=False env_gpu_observations=True env_frameskip=4 env_framestack=1 pixel_format=CHW use_record_episode_statistics=False with_wandb=False wandb_user=None wandb_project=sample_factory wandb_group=None wandb_job_type=SF wandb_tags=[] with_pbt=False pbt_mix_policies_in_one_env=True pbt_period_env_steps=5000000 pbt_start_mutation=20000000 pbt_replace_fraction=0.3 pbt_mutation_rate=0.15 pbt_replace_reward_gap=0.1 pbt_replace_reward_gap_absolute=1e-06 pbt_optimize_gamma=False pbt_target_objective=true_objective pbt_perturb_min=1.1 pbt_perturb_max=1.5 num_agents=-1 num_humans=0 num_bots=-1 start_bot_difficulty=None timelimit=None res_w=128 res_h=72 wide_aspect_ratio=False eval_env_frameskip=1 fps=35 command_line=--env=doom_health_gathering_supreme --num_workers=8 --num_envs_per_worker=4 --train_for_env_steps=4000000 cli_args={'env': 'doom_health_gathering_supreme', 'num_workers': 8, 'num_envs_per_worker': 4, 'train_for_env_steps': 4000000} git_hash=02ef46648112e6fd3adc4475dfd889e784c0ef87 git_repo_name=https://github.com/mihirdeo16/jubilant-memory.git [2023-06-21 11:25:46,094][62782] Saving configuration to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json... [2023-06-21 11:25:46,115][62782] Rollout worker 0 uses device cpu [2023-06-21 11:25:46,116][62782] Rollout worker 1 uses device cpu [2023-06-21 11:25:46,116][62782] Rollout worker 2 uses device cpu [2023-06-21 11:25:46,117][62782] Rollout worker 3 uses device cpu [2023-06-21 11:25:46,117][62782] Rollout worker 4 uses device cpu [2023-06-21 11:25:46,117][62782] Rollout worker 5 uses device cpu [2023-06-21 11:25:46,117][62782] Rollout worker 6 uses device cpu [2023-06-21 11:25:46,118][62782] Rollout worker 7 uses device cpu [2023-06-21 11:25:46,150][62782] InferenceWorker_p0-w0: min num requests: 2 [2023-06-21 11:25:46,178][62782] Starting all processes... [2023-06-21 11:25:46,178][62782] Starting process learner_proc0 [2023-06-21 11:25:46,233][62782] Starting all processes... [2023-06-21 11:25:46,241][62782] Starting process inference_proc0-0 [2023-06-21 11:25:46,241][62782] Starting process rollout_proc0 [2023-06-21 11:25:46,242][62782] Starting process rollout_proc1 [2023-06-21 11:25:46,242][62782] Starting process rollout_proc2 [2023-06-21 11:25:46,242][62782] Starting process rollout_proc3 [2023-06-21 11:25:46,242][62782] Starting process rollout_proc4 [2023-06-21 11:25:46,242][62782] Starting process rollout_proc5 [2023-06-21 11:25:46,242][62782] Starting process rollout_proc6 [2023-06-21 11:25:46,242][62782] Starting process rollout_proc7 [2023-06-21 11:25:48,943][73133] On MacOS, not setting affinity [2023-06-21 11:25:48,956][73126] On MacOS, not setting affinity [2023-06-21 11:25:48,966][73127] On MacOS, not setting affinity [2023-06-21 11:25:48,970][73128] On MacOS, not setting affinity [2023-06-21 11:25:48,972][73129] On MacOS, not setting affinity [2023-06-21 11:25:48,983][73121] Starting seed is not provided [2023-06-21 11:25:48,983][73121] Initializing actor-critic model on device cpu [2023-06-21 11:25:48,984][73121] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 11:25:48,985][73132] On MacOS, not setting affinity [2023-06-21 11:25:48,986][73130] On MacOS, not setting affinity [2023-06-21 11:25:48,986][73131] On MacOS, not setting affinity [2023-06-21 11:25:48,986][73121] RunningMeanStd input shape: (1,) [2023-06-21 11:25:48,997][73121] ConvEncoder: input_channels=3 [2023-06-21 11:25:49,062][73121] Conv encoder output size: 512 [2023-06-21 11:25:49,062][73121] Policy head output size: 512 [2023-06-21 11:25:49,083][73121] Created Actor Critic model with architecture: [2023-06-21 11:25:49,083][73121] ActorCriticSharedWeights( (obs_normalizer): ObservationNormalizer( (running_mean_std): RunningMeanStdDictInPlace( (running_mean_std): ModuleDict( (obs): RunningMeanStdInPlace() ) ) ) (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace) (encoder): VizdoomEncoder( (basic_encoder): ConvEncoder( (enc): RecursiveScriptModule( original_name=ConvEncoderImpl (conv_head): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Conv2d) (1): RecursiveScriptModule(original_name=ELU) (2): RecursiveScriptModule(original_name=Conv2d) (3): RecursiveScriptModule(original_name=ELU) (4): RecursiveScriptModule(original_name=Conv2d) (5): RecursiveScriptModule(original_name=ELU) ) (mlp_layers): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Linear) (1): RecursiveScriptModule(original_name=ELU) ) ) ) ) (core): ModelCoreRNN( (core): GRU(512, 512) ) (decoder): MlpDecoder( (mlp): Identity() ) (critic_linear): Linear(in_features=512, out_features=1, bias=True) (action_parameterization): ActionParameterizationDefault( (distribution_linear): Linear(in_features=512, out_features=5, bias=True) ) ) [2023-06-21 11:25:49,087][73121] Using optimizer [2023-06-21 11:25:49,087][73121] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... [2023-06-21 11:25:49,123][73121] Loading model from checkpoint [2023-06-21 11:25:49,131][73121] Loaded experiment state at self.train_step=978, self.env_steps=4005888 [2023-06-21 11:25:49,138][73121] Initialized policy 0 weights for model version 978 [2023-06-21 11:25:49,145][73125] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 11:25:49,145][73125] RunningMeanStd input shape: (1,) [2023-06-21 11:25:49,147][73121] LearnerWorker_p0 finished initialization! [2023-06-21 11:25:49,154][73125] ConvEncoder: input_channels=3 [2023-06-21 11:25:49,202][73125] Conv encoder output size: 512 [2023-06-21 11:25:49,202][73125] Policy head output size: 512 [2023-06-21 11:25:49,209][62782] Inference worker 0-0 is ready! [2023-06-21 11:25:49,210][62782] All inference workers are ready! Signal rollout workers to start! [2023-06-21 11:25:49,239][73131] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 11:25:49,242][73126] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 11:25:49,244][73133] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 11:25:49,248][73132] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 11:25:49,249][73129] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 11:25:49,249][73128] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 11:25:49,250][73130] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 11:25:49,250][73127] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 11:25:49,851][62782] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 4005888. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 11:25:51,883][73132] Decorrelating experience for 0 frames... [2023-06-21 11:25:51,908][73127] Decorrelating experience for 0 frames... [2023-06-21 11:25:51,908][73129] Decorrelating experience for 0 frames... [2023-06-21 11:25:51,933][73128] Decorrelating experience for 0 frames... [2023-06-21 11:25:51,934][73133] Decorrelating experience for 0 frames... [2023-06-21 11:25:53,013][73133] Decorrelating experience for 32 frames... [2023-06-21 11:25:53,014][73128] Decorrelating experience for 32 frames... [2023-06-21 11:25:53,014][73127] Decorrelating experience for 32 frames... [2023-06-21 11:25:53,016][73132] Decorrelating experience for 32 frames... [2023-06-21 11:25:53,017][73129] Decorrelating experience for 32 frames... [2023-06-21 11:25:53,019][73130] Decorrelating experience for 0 frames... [2023-06-21 11:25:53,020][73131] Decorrelating experience for 0 frames... [2023-06-21 11:25:53,791][73126] Decorrelating experience for 0 frames... [2023-06-21 11:25:53,794][73130] Decorrelating experience for 32 frames... [2023-06-21 11:25:54,591][73126] Decorrelating experience for 32 frames... [2023-06-21 11:25:54,770][73128] Decorrelating experience for 64 frames... [2023-06-21 11:25:54,767][73132] Decorrelating experience for 64 frames... [2023-06-21 11:25:54,776][73131] Decorrelating experience for 32 frames... [2023-06-21 11:25:54,854][62782] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 4005888. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 11:25:55,539][73127] Decorrelating experience for 64 frames... [2023-06-21 11:25:55,541][73129] Decorrelating experience for 64 frames... [2023-06-21 11:25:56,052][73126] Decorrelating experience for 64 frames... [2023-06-21 11:25:56,239][73130] Decorrelating experience for 64 frames... [2023-06-21 11:25:57,352][73131] Decorrelating experience for 64 frames... [2023-06-21 11:25:57,618][73133] Decorrelating experience for 64 frames... [2023-06-21 11:25:57,619][73128] Decorrelating experience for 96 frames... [2023-06-21 11:25:57,619][73132] Decorrelating experience for 96 frames... [2023-06-21 11:25:58,140][73127] Decorrelating experience for 96 frames... [2023-06-21 11:25:58,300][73129] Decorrelating experience for 96 frames... [2023-06-21 11:25:58,575][73126] Decorrelating experience for 96 frames... [2023-06-21 11:25:58,735][73130] Decorrelating experience for 96 frames... [2023-06-21 11:25:59,635][73131] Decorrelating experience for 96 frames... [2023-06-21 11:25:59,825][73133] Decorrelating experience for 96 frames... [2023-06-21 11:25:59,853][62782] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 4005888. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 11:26:04,853][62782] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 4005888. Throughput: 0: 66.7. Samples: 1000. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 11:26:04,856][62782] Avg episode reward: [(0, '1.246')] [2023-06-21 11:26:06,141][62782] Heartbeat connected on Batcher_0 [2023-06-21 11:26:06,189][62782] Heartbeat connected on InferenceWorker_p0-w0 [2023-06-21 11:26:06,236][62782] Heartbeat connected on RolloutWorker_w6 [2023-06-21 11:26:06,247][62782] Heartbeat connected on RolloutWorker_w0 [2023-06-21 11:26:06,282][62782] Heartbeat connected on RolloutWorker_w5 [2023-06-21 11:26:06,320][62782] Heartbeat connected on RolloutWorker_w1 [2023-06-21 11:26:06,342][62782] Heartbeat connected on RolloutWorker_w7 [2023-06-21 11:26:06,347][62782] Heartbeat connected on RolloutWorker_w3 [2023-06-21 11:26:06,359][62782] Heartbeat connected on RolloutWorker_w4 [2023-06-21 11:26:06,365][62782] Heartbeat connected on RolloutWorker_w2 [2023-06-21 11:26:07,667][62782] Heartbeat connected on LearnerWorker_p0 [2023-06-21 11:26:09,852][62782] Fps is (10 sec: 640.0, 60 sec: 320.0, 300 sec: 320.0). Total num frames: 4012288. Throughput: 0: 95.6. Samples: 1912. Policy #0 lag: (min: 0.0, avg: 0.0, max: 0.0) [2023-06-21 11:26:09,855][62782] Avg episode reward: [(0, '1.697')] [2023-06-21 11:26:14,852][62782] Fps is (10 sec: 960.1, 60 sec: 384.0, 300 sec: 384.0). Total num frames: 4015488. Throughput: 0: 152.1. Samples: 3802. Policy #0 lag: (min: 0.0, avg: 1.5, max: 2.0) [2023-06-21 11:26:14,856][62782] Avg episode reward: [(0, '4.654')] [2023-06-21 11:26:19,853][62782] Fps is (10 sec: 1279.9, 60 sec: 639.9, 300 sec: 639.9). Total num frames: 4025088. Throughput: 0: 190.2. Samples: 5708. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:26:19,855][62782] Avg episode reward: [(0, '6.751')] [2023-06-21 11:26:24,853][62782] Fps is (10 sec: 1599.9, 60 sec: 731.4, 300 sec: 731.4). Total num frames: 4031488. Throughput: 0: 190.0. Samples: 6652. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:26:24,856][62782] Avg episode reward: [(0, '7.130')] [2023-06-21 11:26:29,854][62782] Fps is (10 sec: 960.0, 60 sec: 719.9, 300 sec: 719.9). Total num frames: 4034688. Throughput: 0: 213.9. Samples: 8556. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:26:29,871][62782] Avg episode reward: [(0, '8.932')] [2023-06-21 11:26:30,147][73125] Updated weights for policy 0, policy_version 988 (0.0016) [2023-06-21 11:26:34,854][62782] Fps is (10 sec: 959.9, 60 sec: 782.2, 300 sec: 782.2). Total num frames: 4041088. Throughput: 0: 231.8. Samples: 10430. Policy #0 lag: (min: 0.0, avg: 1.5, max: 2.0) [2023-06-21 11:26:34,857][62782] Avg episode reward: [(0, '12.341')] [2023-06-21 11:26:39,853][62782] Fps is (10 sec: 1600.0, 60 sec: 895.9, 300 sec: 895.9). Total num frames: 4050688. Throughput: 0: 252.8. Samples: 11376. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:26:39,856][62782] Avg episode reward: [(0, '14.480')] [2023-06-21 11:26:44,851][62782] Fps is (10 sec: 1600.4, 60 sec: 930.9, 300 sec: 930.9). Total num frames: 4057088. Throughput: 0: 294.4. Samples: 13248. Policy #0 lag: (min: 0.0, avg: 1.4, max: 3.0) [2023-06-21 11:26:44,854][62782] Avg episode reward: [(0, '16.752')] [2023-06-21 11:26:49,853][62782] Fps is (10 sec: 960.1, 60 sec: 906.6, 300 sec: 906.6). Total num frames: 4060288. Throughput: 0: 314.3. Samples: 15144. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:26:49,856][62782] Avg episode reward: [(0, '18.114')] [2023-06-21 11:26:54,853][62782] Fps is (10 sec: 959.9, 60 sec: 1013.3, 300 sec: 935.3). Total num frames: 4066688. Throughput: 0: 315.0. Samples: 16088. Policy #0 lag: (min: 0.0, avg: 1.5, max: 2.0) [2023-06-21 11:26:54,856][62782] Avg episode reward: [(0, '18.673')] [2023-06-21 11:26:54,902][73125] Updated weights for policy 0, policy_version 998 (0.0011) [2023-06-21 11:26:59,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1120.0, 300 sec: 960.0). Total num frames: 4073088. Throughput: 0: 313.1. Samples: 17892. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:26:59,857][62782] Avg episode reward: [(0, '20.549')] [2023-06-21 11:27:04,849][62782] Fps is (10 sec: 1280.5, 60 sec: 1226.7, 300 sec: 981.3). Total num frames: 4079488. Throughput: 0: 312.7. Samples: 19778. Policy #0 lag: (min: 0.0, avg: 1.5, max: 2.0) [2023-06-21 11:27:04,853][62782] Avg episode reward: [(0, '21.343')] [2023-06-21 11:27:09,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1226.7, 300 sec: 1000.0). Total num frames: 4085888. Throughput: 0: 312.7. Samples: 20722. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:27:09,855][62782] Avg episode reward: [(0, '21.920')] [2023-06-21 11:27:14,040][73121] Signal inference workers to stop experience collection... [2023-06-21 11:27:14,068][73125] InferenceWorker_p0-w0: stopping experience collection [2023-06-21 11:27:14,107][73121] Signal inference workers to resume experience collection... [2023-06-21 11:27:14,108][73125] InferenceWorker_p0-w0: resuming experience collection [2023-06-21 11:27:14,854][62782] Fps is (10 sec: 1279.4, 60 sec: 1280.0, 300 sec: 1016.4). Total num frames: 4092288. Throughput: 0: 312.0. Samples: 22594. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:27:14,855][62782] Avg episode reward: [(0, '22.513')] [2023-06-21 11:27:19,854][62782] Fps is (10 sec: 1279.9, 60 sec: 1226.7, 300 sec: 1031.1). Total num frames: 4098688. Throughput: 0: 310.4. Samples: 24396. Policy #0 lag: (min: 0.0, avg: 1.4, max: 3.0) [2023-06-21 11:27:19,859][62782] Avg episode reward: [(0, '22.296')] [2023-06-21 11:27:22,527][73125] Updated weights for policy 0, policy_version 1008 (0.0300) [2023-06-21 11:27:25,008][62782] Fps is (10 sec: 945.7, 60 sec: 1170.4, 300 sec: 1008.9). Total num frames: 4101888. Throughput: 0: 306.1. Samples: 25198. Policy #0 lag: (min: 0.0, avg: 1.4, max: 3.0) [2023-06-21 11:27:25,065][62782] Avg episode reward: [(0, '22.305')] [2023-06-21 11:27:29,853][62782] Fps is (10 sec: 960.1, 60 sec: 1226.7, 300 sec: 1024.0). Total num frames: 4108288. Throughput: 0: 292.8. Samples: 26426. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:27:29,856][62782] Avg episode reward: [(0, '22.224')] [2023-06-21 11:27:34,853][62782] Fps is (10 sec: 1299.8, 60 sec: 1226.7, 300 sec: 1036.2). Total num frames: 4114688. Throughput: 0: 292.7. Samples: 28316. Policy #0 lag: (min: 0.0, avg: 1.4, max: 3.0) [2023-06-21 11:27:34,854][62782] Avg episode reward: [(0, '22.756')] [2023-06-21 11:27:39,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1173.3, 300 sec: 1047.2). Total num frames: 4121088. Throughput: 0: 292.8. Samples: 29262. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:27:39,854][62782] Avg episode reward: [(0, '23.467')] [2023-06-21 11:27:44,853][62782] Fps is (10 sec: 1279.9, 60 sec: 1173.3, 300 sec: 1057.4). Total num frames: 4127488. Throughput: 0: 294.8. Samples: 31156. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:27:44,856][62782] Avg episode reward: [(0, '24.568')] [2023-06-21 11:27:46,580][73121] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001017_4130688.pth... [2023-06-21 11:27:46,718][73121] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000950_3891200.pth [2023-06-21 11:27:49,240][73125] Updated weights for policy 0, policy_version 1018 (0.0344) [2023-06-21 11:27:49,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1226.7, 300 sec: 1066.6). Total num frames: 4133888. Throughput: 0: 294.9. Samples: 33050. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:27:49,854][62782] Avg episode reward: [(0, '25.124')] [2023-06-21 11:27:54,852][62782] Fps is (10 sec: 1280.1, 60 sec: 1226.7, 300 sec: 1075.2). Total num frames: 4140288. Throughput: 0: 295.1. Samples: 34002. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:27:54,854][62782] Avg episode reward: [(0, '25.672')] [2023-06-21 11:27:59,850][62782] Fps is (10 sec: 1280.4, 60 sec: 1226.7, 300 sec: 1083.1). Total num frames: 4146688. Throughput: 0: 295.6. Samples: 35896. Policy #0 lag: (min: 0.0, avg: 1.5, max: 2.0) [2023-06-21 11:27:59,854][62782] Avg episode reward: [(0, '27.215')] [2023-06-21 11:28:04,852][62782] Fps is (10 sec: 1280.0, 60 sec: 1226.6, 300 sec: 1090.4). Total num frames: 4153088. Throughput: 0: 297.2. Samples: 37770. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:28:04,853][62782] Avg episode reward: [(0, '28.171')] [2023-06-21 11:28:04,856][73121] Saving new best policy, reward=28.171! [2023-06-21 11:28:09,853][62782] Fps is (10 sec: 1279.6, 60 sec: 1226.7, 300 sec: 1097.1). Total num frames: 4159488. Throughput: 0: 301.6. Samples: 38722. Policy #0 lag: (min: 0.0, avg: 1.3, max: 3.0) [2023-06-21 11:28:09,854][62782] Avg episode reward: [(0, '28.705')] [2023-06-21 11:28:09,856][73121] Saving new best policy, reward=28.705! [2023-06-21 11:28:14,852][62782] Fps is (10 sec: 960.1, 60 sec: 1173.4, 300 sec: 1081.4). Total num frames: 4162688. Throughput: 0: 314.9. Samples: 40594. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:28:14,853][62782] Avg episode reward: [(0, '28.717')] [2023-06-21 11:28:14,999][73121] Saving new best policy, reward=28.717! [2023-06-21 11:28:15,001][73125] Updated weights for policy 0, policy_version 1028 (0.0008) [2023-06-21 11:28:19,852][62782] Fps is (10 sec: 1280.1, 60 sec: 1226.7, 300 sec: 1109.3). Total num frames: 4172288. Throughput: 0: 314.4. Samples: 42466. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:28:19,853][62782] Avg episode reward: [(0, '28.249')] [2023-06-21 11:28:24,850][62782] Fps is (10 sec: 1600.2, 60 sec: 1283.3, 300 sec: 1114.8). Total num frames: 4178688. Throughput: 0: 314.7. Samples: 43424. Policy #0 lag: (min: 0.0, avg: 1.1, max: 3.0) [2023-06-21 11:28:24,852][62782] Avg episode reward: [(0, '28.304')] [2023-06-21 11:28:29,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1280.0, 300 sec: 1120.0). Total num frames: 4185088. Throughput: 0: 314.8. Samples: 45322. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:28:29,855][62782] Avg episode reward: [(0, '28.288')] [2023-06-21 11:28:34,852][62782] Fps is (10 sec: 959.8, 60 sec: 1226.7, 300 sec: 1105.4). Total num frames: 4188288. Throughput: 0: 314.8. Samples: 47214. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:28:34,854][62782] Avg episode reward: [(0, '28.100')] [2023-06-21 11:28:39,742][73125] Updated weights for policy 0, policy_version 1038 (0.0009) [2023-06-21 11:28:39,853][62782] Fps is (10 sec: 1279.9, 60 sec: 1280.0, 300 sec: 1129.4). Total num frames: 4197888. Throughput: 0: 314.7. Samples: 48164. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:28:39,854][62782] Avg episode reward: [(0, '27.431')] [2023-06-21 11:28:44,849][62782] Fps is (10 sec: 1280.3, 60 sec: 1226.8, 300 sec: 1115.4). Total num frames: 4201088. Throughput: 0: 314.9. Samples: 50066. Policy #0 lag: (min: 0.0, avg: 1.4, max: 3.0) [2023-06-21 11:28:44,850][62782] Avg episode reward: [(0, '27.054')] [2023-06-21 11:28:49,853][62782] Fps is (10 sec: 1280.1, 60 sec: 1280.0, 300 sec: 1137.8). Total num frames: 4210688. Throughput: 0: 315.4. Samples: 51962. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:28:49,854][62782] Avg episode reward: [(0, '26.578')] [2023-06-21 11:28:54,853][62782] Fps is (10 sec: 1279.5, 60 sec: 1226.7, 300 sec: 1124.3). Total num frames: 4213888. Throughput: 0: 315.4. Samples: 52914. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:28:54,855][62782] Avg episode reward: [(0, '27.018')] [2023-06-21 11:28:59,860][62782] Fps is (10 sec: 959.4, 60 sec: 1226.5, 300 sec: 1128.4). Total num frames: 4220288. Throughput: 0: 315.7. Samples: 54802. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:28:59,863][62782] Avg episode reward: [(0, '27.642')] [2023-06-21 11:29:04,853][62782] Fps is (10 sec: 1280.1, 60 sec: 1226.7, 300 sec: 1132.3). Total num frames: 4226688. Throughput: 0: 315.8. Samples: 56676. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:29:04,856][62782] Avg episode reward: [(0, '27.219')] [2023-06-21 11:29:04,992][73125] Updated weights for policy 0, policy_version 1048 (0.0012) [2023-06-21 11:29:09,852][62782] Fps is (10 sec: 1280.9, 60 sec: 1226.7, 300 sec: 1136.0). Total num frames: 4233088. Throughput: 0: 315.5. Samples: 57622. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:29:09,855][62782] Avg episode reward: [(0, '26.596')] [2023-06-21 11:29:14,855][62782] Fps is (10 sec: 1279.8, 60 sec: 1279.9, 300 sec: 1139.5). Total num frames: 4239488. Throughput: 0: 313.4. Samples: 59424. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:29:14,858][62782] Avg episode reward: [(0, '26.615')] [2023-06-21 11:29:19,853][62782] Fps is (10 sec: 1279.9, 60 sec: 1226.7, 300 sec: 1142.8). Total num frames: 4245888. Throughput: 0: 313.6. Samples: 61328. Policy #0 lag: (min: 0.0, avg: 1.4, max: 3.0) [2023-06-21 11:29:19,857][62782] Avg episode reward: [(0, '26.615')] [2023-06-21 11:29:24,853][62782] Fps is (10 sec: 1280.2, 60 sec: 1226.6, 300 sec: 1146.0). Total num frames: 4252288. Throughput: 0: 313.4. Samples: 62268. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:29:24,856][62782] Avg episode reward: [(0, '25.680')] [2023-06-21 11:29:29,850][62782] Fps is (10 sec: 1280.4, 60 sec: 1226.7, 300 sec: 1149.1). Total num frames: 4258688. Throughput: 0: 313.0. Samples: 64152. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:29:29,852][62782] Avg episode reward: [(0, '25.087')] [2023-06-21 11:29:30,892][73125] Updated weights for policy 0, policy_version 1058 (0.0014) [2023-06-21 11:29:34,854][62782] Fps is (10 sec: 1279.9, 60 sec: 1280.0, 300 sec: 1152.0). Total num frames: 4265088. Throughput: 0: 312.4. Samples: 66020. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:29:34,866][62782] Avg episode reward: [(0, '24.036')] [2023-06-21 11:29:39,853][62782] Fps is (10 sec: 1279.5, 60 sec: 1226.7, 300 sec: 1154.8). Total num frames: 4271488. Throughput: 0: 312.2. Samples: 66964. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:29:39,857][62782] Avg episode reward: [(0, '24.540')] [2023-06-21 11:29:44,853][62782] Fps is (10 sec: 1280.1, 60 sec: 1279.9, 300 sec: 1157.4). Total num frames: 4277888. Throughput: 0: 312.4. Samples: 68858. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:29:44,855][62782] Avg episode reward: [(0, '24.310')] [2023-06-21 11:29:44,860][73121] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001063_4277888.pth... [2023-06-21 11:29:44,927][73121] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth [2023-06-21 11:29:49,854][62782] Fps is (10 sec: 1280.0, 60 sec: 1226.6, 300 sec: 1160.0). Total num frames: 4284288. Throughput: 0: 311.1. Samples: 70678. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:29:49,857][62782] Avg episode reward: [(0, '24.024')] [2023-06-21 11:29:54,853][62782] Fps is (10 sec: 1280.1, 60 sec: 1280.0, 300 sec: 1162.4). Total num frames: 4290688. Throughput: 0: 311.3. Samples: 71630. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:29:54,856][62782] Avg episode reward: [(0, '23.759')] [2023-06-21 11:29:56,867][73125] Updated weights for policy 0, policy_version 1068 (0.0012) [2023-06-21 11:29:59,865][62782] Fps is (10 sec: 1279.0, 60 sec: 1279.9, 300 sec: 1164.7). Total num frames: 4297088. Throughput: 0: 312.9. Samples: 73506. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:29:59,893][62782] Avg episode reward: [(0, '23.539')] [2023-06-21 11:30:04,852][62782] Fps is (10 sec: 1280.0, 60 sec: 1280.0, 300 sec: 1167.0). Total num frames: 4303488. Throughput: 0: 312.3. Samples: 75380. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:30:04,855][62782] Avg episode reward: [(0, '24.390')] [2023-06-21 11:30:09,852][62782] Fps is (10 sec: 1281.2, 60 sec: 1280.0, 300 sec: 1169.2). Total num frames: 4309888. Throughput: 0: 312.4. Samples: 76326. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:30:09,858][62782] Avg episode reward: [(0, '25.527')] [2023-06-21 11:30:14,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1280.0, 300 sec: 1171.3). Total num frames: 4316288. Throughput: 0: 311.4. Samples: 78168. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:30:14,856][62782] Avg episode reward: [(0, '25.674')] [2023-06-21 11:30:19,852][62782] Fps is (10 sec: 960.1, 60 sec: 1226.7, 300 sec: 1161.5). Total num frames: 4319488. Throughput: 0: 309.5. Samples: 79946. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:30:19,857][62782] Avg episode reward: [(0, '25.271')] [2023-06-21 11:30:23,218][73125] Updated weights for policy 0, policy_version 1078 (0.0014) [2023-06-21 11:30:24,856][62782] Fps is (10 sec: 959.8, 60 sec: 1226.6, 300 sec: 1163.6). Total num frames: 4325888. Throughput: 0: 309.3. Samples: 80884. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:30:24,858][62782] Avg episode reward: [(0, '25.714')] [2023-06-21 11:30:29,855][62782] Fps is (10 sec: 1280.0, 60 sec: 1226.6, 300 sec: 1165.7). Total num frames: 4332288. Throughput: 0: 308.4. Samples: 82736. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:30:29,858][62782] Avg episode reward: [(0, '25.219')] [2023-06-21 11:30:34,853][62782] Fps is (10 sec: 1280.3, 60 sec: 1226.7, 300 sec: 1167.7). Total num frames: 4338688. Throughput: 0: 309.3. Samples: 84596. Policy #0 lag: (min: 0.0, avg: 1.4, max: 3.0) [2023-06-21 11:30:34,858][62782] Avg episode reward: [(0, '26.149')] [2023-06-21 11:30:39,854][62782] Fps is (10 sec: 1279.7, 60 sec: 1226.7, 300 sec: 1169.6). Total num frames: 4345088. Throughput: 0: 309.1. Samples: 85538. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:30:39,862][62782] Avg episode reward: [(0, '26.457')] [2023-06-21 11:30:44,853][62782] Fps is (10 sec: 1279.9, 60 sec: 1226.7, 300 sec: 1171.5). Total num frames: 4351488. Throughput: 0: 309.6. Samples: 87434. Policy #0 lag: (min: 0.0, avg: 1.4, max: 3.0) [2023-06-21 11:30:44,857][62782] Avg episode reward: [(0, '26.567')] [2023-06-21 11:30:48,434][73125] Updated weights for policy 0, policy_version 1088 (0.0015) [2023-06-21 11:30:49,855][62782] Fps is (10 sec: 1279.8, 60 sec: 1226.6, 300 sec: 1193.2). Total num frames: 4357888. Throughput: 0: 309.9. Samples: 89328. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:30:49,857][62782] Avg episode reward: [(0, '26.150')] [2023-06-21 11:30:54,853][62782] Fps is (10 sec: 1280.1, 60 sec: 1226.7, 300 sec: 1214.9). Total num frames: 4364288. Throughput: 0: 308.9. Samples: 90226. Policy #0 lag: (min: 0.0, avg: 1.5, max: 2.0) [2023-06-21 11:30:54,857][62782] Avg episode reward: [(0, '26.924')] [2023-06-21 11:30:59,853][62782] Fps is (10 sec: 1280.3, 60 sec: 1226.8, 300 sec: 1236.6). Total num frames: 4370688. Throughput: 0: 308.5. Samples: 92052. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:30:59,856][62782] Avg episode reward: [(0, '28.312')] [2023-06-21 11:31:04,857][62782] Fps is (10 sec: 1279.5, 60 sec: 1226.6, 300 sec: 1236.6). Total num frames: 4377088. Throughput: 0: 311.1. Samples: 93948. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:31:04,866][62782] Avg episode reward: [(0, '28.224')] [2023-06-21 11:31:09,851][62782] Fps is (10 sec: 1280.2, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4383488. Throughput: 0: 311.4. Samples: 94896. Policy #0 lag: (min: 0.0, avg: 1.1, max: 3.0) [2023-06-21 11:31:09,854][62782] Avg episode reward: [(0, '28.281')] [2023-06-21 11:31:13,698][73125] Updated weights for policy 0, policy_version 1098 (0.0017) [2023-06-21 11:31:14,853][62782] Fps is (10 sec: 1280.4, 60 sec: 1226.7, 300 sec: 1236.6). Total num frames: 4389888. Throughput: 0: 312.0. Samples: 96776. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:31:14,854][62782] Avg episode reward: [(0, '27.821')] [2023-06-21 11:31:19,853][62782] Fps is (10 sec: 1279.8, 60 sec: 1280.0, 300 sec: 1236.6). Total num frames: 4396288. Throughput: 0: 312.9. Samples: 98676. Policy #0 lag: (min: 0.0, avg: 1.3, max: 3.0) [2023-06-21 11:31:19,856][62782] Avg episode reward: [(0, '27.942')] [2023-06-21 11:31:24,854][62782] Fps is (10 sec: 1279.9, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4402688. Throughput: 0: 312.8. Samples: 99616. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:31:24,856][62782] Avg episode reward: [(0, '27.980')] [2023-06-21 11:31:29,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4409088. Throughput: 0: 312.9. Samples: 101516. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:31:29,856][62782] Avg episode reward: [(0, '27.616')] [2023-06-21 11:31:34,853][62782] Fps is (10 sec: 1280.1, 60 sec: 1280.0, 300 sec: 1236.6). Total num frames: 4415488. Throughput: 0: 312.8. Samples: 103404. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:31:34,855][62782] Avg episode reward: [(0, '27.714')] [2023-06-21 11:31:39,259][73125] Updated weights for policy 0, policy_version 1108 (0.0009) [2023-06-21 11:31:39,852][62782] Fps is (10 sec: 1280.1, 60 sec: 1280.0, 300 sec: 1236.6). Total num frames: 4421888. Throughput: 0: 313.9. Samples: 104352. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:31:39,853][62782] Avg episode reward: [(0, '27.360')] [2023-06-21 11:31:44,854][62782] Fps is (10 sec: 1279.8, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4428288. Throughput: 0: 315.4. Samples: 106246. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:31:44,856][62782] Avg episode reward: [(0, '27.226')] [2023-06-21 11:31:44,860][73121] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001110_4428288.pth... [2023-06-21 11:31:44,945][73121] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001017_4130688.pth [2023-06-21 11:31:49,853][62782] Fps is (10 sec: 1279.9, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4434688. Throughput: 0: 314.8. Samples: 108114. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:31:49,855][62782] Avg episode reward: [(0, '26.854')] [2023-06-21 11:31:54,853][62782] Fps is (10 sec: 1280.1, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4441088. Throughput: 0: 314.7. Samples: 109056. Policy #0 lag: (min: 0.0, avg: 1.4, max: 3.0) [2023-06-21 11:31:54,856][62782] Avg episode reward: [(0, '27.200')] [2023-06-21 11:31:59,851][62782] Fps is (10 sec: 1280.2, 60 sec: 1280.0, 300 sec: 1247.4). Total num frames: 4447488. Throughput: 0: 313.5. Samples: 110882. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:31:59,854][62782] Avg episode reward: [(0, '26.208')] [2023-06-21 11:32:04,820][73125] Updated weights for policy 0, policy_version 1118 (0.0290) [2023-06-21 11:32:04,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1280.1, 300 sec: 1247.5). Total num frames: 4453888. Throughput: 0: 313.2. Samples: 112772. Policy #0 lag: (min: 0.0, avg: 1.3, max: 3.0) [2023-06-21 11:32:04,855][62782] Avg episode reward: [(0, '24.985')] [2023-06-21 11:32:09,853][62782] Fps is (10 sec: 959.9, 60 sec: 1226.6, 300 sec: 1236.6). Total num frames: 4457088. Throughput: 0: 313.5. Samples: 113722. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:32:09,857][62782] Avg episode reward: [(0, '25.259')] [2023-06-21 11:32:14,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4466688. Throughput: 0: 313.1. Samples: 115604. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:32:14,855][62782] Avg episode reward: [(0, '23.944')] [2023-06-21 11:32:19,851][62782] Fps is (10 sec: 1280.2, 60 sec: 1226.7, 300 sec: 1248.1). Total num frames: 4469888. Throughput: 0: 313.2. Samples: 117498. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:32:19,857][62782] Avg episode reward: [(0, '24.147')] [2023-06-21 11:32:24,856][62782] Fps is (10 sec: 959.7, 60 sec: 1226.6, 300 sec: 1247.4). Total num frames: 4476288. Throughput: 0: 312.2. Samples: 118400. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:32:24,859][62782] Avg episode reward: [(0, '23.884')] [2023-06-21 11:32:29,852][62782] Fps is (10 sec: 1279.9, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4482688. Throughput: 0: 307.3. Samples: 120072. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:32:29,855][62782] Avg episode reward: [(0, '23.444')] [2023-06-21 11:32:31,964][73125] Updated weights for policy 0, policy_version 1128 (0.0009) [2023-06-21 11:32:34,856][62782] Fps is (10 sec: 960.0, 60 sec: 1173.3, 300 sec: 1236.6). Total num frames: 4485888. Throughput: 0: 303.1. Samples: 121756. Policy #0 lag: (min: 0.0, avg: 1.0, max: 2.0) [2023-06-21 11:32:34,864][62782] Avg episode reward: [(0, '23.765')] [2023-06-21 11:32:39,874][62782] Fps is (10 sec: 958.1, 60 sec: 1172.9, 300 sec: 1236.5). Total num frames: 4492288. Throughput: 0: 300.5. Samples: 122586. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:32:39,885][62782] Avg episode reward: [(0, '23.606')] [2023-06-21 11:32:44,849][62782] Fps is (10 sec: 1280.8, 60 sec: 1173.4, 300 sec: 1236.6). Total num frames: 4498688. Throughput: 0: 300.2. Samples: 124390. Policy #0 lag: (min: 0.0, avg: 1.3, max: 3.0) [2023-06-21 11:32:44,853][62782] Avg episode reward: [(0, '23.750')] [2023-06-21 11:32:49,854][62782] Fps is (10 sec: 1282.4, 60 sec: 1173.3, 300 sec: 1236.6). Total num frames: 4505088. Throughput: 0: 300.4. Samples: 126292. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:32:49,858][62782] Avg episode reward: [(0, '24.213')] [2023-06-21 11:32:54,853][62782] Fps is (10 sec: 1279.6, 60 sec: 1173.3, 300 sec: 1236.6). Total num frames: 4511488. Throughput: 0: 300.4. Samples: 127238. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:32:54,856][62782] Avg episode reward: [(0, '24.518')] [2023-06-21 11:32:58,174][73125] Updated weights for policy 0, policy_version 1138 (0.0014) [2023-06-21 11:32:59,852][62782] Fps is (10 sec: 1280.3, 60 sec: 1173.3, 300 sec: 1236.6). Total num frames: 4517888. Throughput: 0: 300.6. Samples: 129132. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:32:59,855][62782] Avg episode reward: [(0, '25.757')] [2023-06-21 11:33:04,860][62782] Fps is (10 sec: 1279.2, 60 sec: 1173.2, 300 sec: 1236.6). Total num frames: 4524288. Throughput: 0: 300.3. Samples: 131014. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:33:04,867][62782] Avg episode reward: [(0, '25.564')] [2023-06-21 11:33:09,852][62782] Fps is (10 sec: 1279.9, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4530688. Throughput: 0: 301.4. Samples: 131962. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:33:09,853][62782] Avg episode reward: [(0, '26.508')] [2023-06-21 11:33:14,852][62782] Fps is (10 sec: 1280.9, 60 sec: 1173.3, 300 sec: 1236.6). Total num frames: 4537088. Throughput: 0: 306.7. Samples: 133872. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:33:14,856][62782] Avg episode reward: [(0, '25.781')] [2023-06-21 11:33:19,854][62782] Fps is (10 sec: 1279.8, 60 sec: 1226.6, 300 sec: 1236.6). Total num frames: 4543488. Throughput: 0: 311.3. Samples: 135766. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:33:19,858][62782] Avg episode reward: [(0, '24.979')] [2023-06-21 11:33:22,867][73125] Updated weights for policy 0, policy_version 1148 (0.0010) [2023-06-21 11:33:24,853][62782] Fps is (10 sec: 1279.8, 60 sec: 1226.7, 300 sec: 1236.6). Total num frames: 4549888. Throughput: 0: 314.0. Samples: 136712. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:33:24,855][62782] Avg episode reward: [(0, '24.684')] [2023-06-21 11:33:29,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1226.6, 300 sec: 1247.5). Total num frames: 4556288. Throughput: 0: 315.6. Samples: 138592. Policy #0 lag: (min: 0.0, avg: 1.4, max: 3.0) [2023-06-21 11:33:29,856][62782] Avg episode reward: [(0, '25.488')] [2023-06-21 11:33:34,854][62782] Fps is (10 sec: 1279.9, 60 sec: 1280.0, 300 sec: 1236.6). Total num frames: 4562688. Throughput: 0: 315.6. Samples: 140492. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:33:34,857][62782] Avg episode reward: [(0, '25.911')] [2023-06-21 11:33:39,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1280.4, 300 sec: 1247.4). Total num frames: 4569088. Throughput: 0: 315.7. Samples: 141444. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:33:39,857][62782] Avg episode reward: [(0, '26.487')] [2023-06-21 11:33:44,850][62782] Fps is (10 sec: 1280.4, 60 sec: 1280.0, 300 sec: 1236.6). Total num frames: 4575488. Throughput: 0: 315.6. Samples: 143334. Policy #0 lag: (min: 0.0, avg: 1.1, max: 3.0) [2023-06-21 11:33:44,853][62782] Avg episode reward: [(0, '27.055')] [2023-06-21 11:33:45,675][73121] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001157_4578688.pth... [2023-06-21 11:33:45,758][73121] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001063_4277888.pth [2023-06-21 11:33:48,372][73125] Updated weights for policy 0, policy_version 1158 (0.0010) [2023-06-21 11:33:49,853][62782] Fps is (10 sec: 1280.1, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4581888. Throughput: 0: 316.0. Samples: 145230. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:33:49,854][62782] Avg episode reward: [(0, '26.801')] [2023-06-21 11:33:54,854][62782] Fps is (10 sec: 1279.6, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4588288. Throughput: 0: 315.5. Samples: 146162. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:33:54,857][62782] Avg episode reward: [(0, '27.123')] [2023-06-21 11:33:59,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4594688. Throughput: 0: 312.6. Samples: 147940. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:33:59,855][62782] Avg episode reward: [(0, '27.349')] [2023-06-21 11:34:04,852][62782] Fps is (10 sec: 1280.1, 60 sec: 1280.1, 300 sec: 1247.5). Total num frames: 4601088. Throughput: 0: 312.6. Samples: 149832. Policy #0 lag: (min: 0.0, avg: 1.4, max: 3.0) [2023-06-21 11:34:04,855][62782] Avg episode reward: [(0, '26.673')] [2023-06-21 11:34:09,852][62782] Fps is (10 sec: 1280.1, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4607488. Throughput: 0: 312.5. Samples: 150776. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:34:09,854][62782] Avg episode reward: [(0, '27.364')] [2023-06-21 11:34:13,973][73125] Updated weights for policy 0, policy_version 1168 (0.0289) [2023-06-21 11:34:14,853][62782] Fps is (10 sec: 1279.9, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4613888. Throughput: 0: 312.7. Samples: 152662. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:34:14,854][62782] Avg episode reward: [(0, '26.547')] [2023-06-21 11:34:19,854][62782] Fps is (10 sec: 959.9, 60 sec: 1226.7, 300 sec: 1236.6). Total num frames: 4617088. Throughput: 0: 311.9. Samples: 154528. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:34:19,856][62782] Avg episode reward: [(0, '25.299')] [2023-06-21 11:34:24,852][62782] Fps is (10 sec: 1280.2, 60 sec: 1280.0, 300 sec: 1247.4). Total num frames: 4626688. Throughput: 0: 312.1. Samples: 155490. Policy #0 lag: (min: 0.0, avg: 1.5, max: 2.0) [2023-06-21 11:34:24,855][62782] Avg episode reward: [(0, '24.573')] [2023-06-21 11:34:29,851][62782] Fps is (10 sec: 1600.4, 60 sec: 1280.1, 300 sec: 1247.5). Total num frames: 4633088. Throughput: 0: 311.5. Samples: 157352. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:34:29,854][62782] Avg episode reward: [(0, '23.848')] [2023-06-21 11:34:34,853][62782] Fps is (10 sec: 1279.8, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4639488. Throughput: 0: 311.4. Samples: 159242. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:34:34,855][62782] Avg episode reward: [(0, '24.365')] [2023-06-21 11:34:39,714][73125] Updated weights for policy 0, policy_version 1178 (0.0013) [2023-06-21 11:34:39,853][62782] Fps is (10 sec: 1279.8, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4645888. Throughput: 0: 311.7. Samples: 160190. Policy #0 lag: (min: 0.0, avg: 1.1, max: 3.0) [2023-06-21 11:34:39,854][62782] Avg episode reward: [(0, '24.340')] [2023-06-21 11:34:44,856][62782] Fps is (10 sec: 1279.6, 60 sec: 1279.9, 300 sec: 1247.4). Total num frames: 4652288. Throughput: 0: 314.3. Samples: 162084. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:34:44,859][62782] Avg episode reward: [(0, '25.169')] [2023-06-21 11:34:49,852][62782] Fps is (10 sec: 1280.1, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4658688. Throughput: 0: 314.3. Samples: 163976. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:34:49,855][62782] Avg episode reward: [(0, '25.903')] [2023-06-21 11:34:54,852][62782] Fps is (10 sec: 960.4, 60 sec: 1226.7, 300 sec: 1236.6). Total num frames: 4661888. Throughput: 0: 314.2. Samples: 164916. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:34:54,856][62782] Avg episode reward: [(0, '24.723')] [2023-06-21 11:34:59,855][62782] Fps is (10 sec: 959.8, 60 sec: 1226.6, 300 sec: 1236.6). Total num frames: 4668288. Throughput: 0: 314.1. Samples: 166796. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:34:59,872][62782] Avg episode reward: [(0, '24.584')] [2023-06-21 11:35:04,853][62782] Fps is (10 sec: 1279.9, 60 sec: 1226.7, 300 sec: 1236.6). Total num frames: 4674688. Throughput: 0: 312.3. Samples: 168582. Policy #0 lag: (min: 0.0, avg: 1.3, max: 3.0) [2023-06-21 11:35:04,856][62782] Avg episode reward: [(0, '24.034')] [2023-06-21 11:35:06,182][73125] Updated weights for policy 0, policy_version 1188 (0.0017) [2023-06-21 11:35:09,853][62782] Fps is (10 sec: 1280.2, 60 sec: 1226.6, 300 sec: 1236.6). Total num frames: 4681088. Throughput: 0: 312.0. Samples: 169530. Policy #0 lag: (min: 0.0, avg: 1.7, max: 2.0) [2023-06-21 11:35:09,857][62782] Avg episode reward: [(0, '24.414')] [2023-06-21 11:35:14,853][62782] Fps is (10 sec: 1279.9, 60 sec: 1226.7, 300 sec: 1247.4). Total num frames: 4687488. Throughput: 0: 312.6. Samples: 171422. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:35:14,857][62782] Avg episode reward: [(0, '25.098')] [2023-06-21 11:35:19,852][62782] Fps is (10 sec: 1280.1, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4693888. Throughput: 0: 311.2. Samples: 173246. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:35:19,855][62782] Avg episode reward: [(0, '23.948')] [2023-06-21 11:35:24,853][62782] Fps is (10 sec: 1280.1, 60 sec: 1226.6, 300 sec: 1247.5). Total num frames: 4700288. Throughput: 0: 310.6. Samples: 174168. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:35:24,856][62782] Avg episode reward: [(0, '24.014')] [2023-06-21 11:35:29,852][62782] Fps is (10 sec: 1280.0, 60 sec: 1226.6, 300 sec: 1247.5). Total num frames: 4706688. Throughput: 0: 310.7. Samples: 176062. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:35:29,854][62782] Avg episode reward: [(0, '24.444')] [2023-06-21 11:35:31,360][73125] Updated weights for policy 0, policy_version 1198 (0.0308) [2023-06-21 11:35:34,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4713088. Throughput: 0: 309.9. Samples: 177920. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:35:34,856][62782] Avg episode reward: [(0, '24.798')] [2023-06-21 11:35:39,854][62782] Fps is (10 sec: 1279.8, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4719488. Throughput: 0: 310.1. Samples: 178870. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:35:39,856][62782] Avg episode reward: [(0, '24.671')] [2023-06-21 11:35:44,859][62782] Fps is (10 sec: 1279.8, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4725888. Throughput: 0: 310.5. Samples: 180770. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:35:44,866][62782] Avg episode reward: [(0, '25.314')] [2023-06-21 11:35:46,936][73121] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001204_4729088.pth... [2023-06-21 11:35:47,088][73121] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001110_4428288.pth [2023-06-21 11:35:49,852][62782] Fps is (10 sec: 1280.2, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4732288. Throughput: 0: 312.2. Samples: 182632. Policy #0 lag: (min: 0.0, avg: 1.3, max: 3.0) [2023-06-21 11:35:49,856][62782] Avg episode reward: [(0, '25.966')] [2023-06-21 11:35:54,853][62782] Fps is (10 sec: 960.1, 60 sec: 1226.6, 300 sec: 1236.6). Total num frames: 4735488. Throughput: 0: 312.2. Samples: 183578. Policy #0 lag: (min: 0.0, avg: 1.3, max: 3.0) [2023-06-21 11:35:54,861][62782] Avg episode reward: [(0, '26.503')] [2023-06-21 11:35:56,827][73125] Updated weights for policy 0, policy_version 1208 (0.0340) [2023-06-21 11:35:59,854][62782] Fps is (10 sec: 959.9, 60 sec: 1226.7, 300 sec: 1236.6). Total num frames: 4741888. Throughput: 0: 309.6. Samples: 185354. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:35:59,863][62782] Avg episode reward: [(0, '25.831')] [2023-06-21 11:36:04,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1226.7, 300 sec: 1236.6). Total num frames: 4748288. Throughput: 0: 309.9. Samples: 187194. Policy #0 lag: (min: 0.0, avg: 1.5, max: 3.0) [2023-06-21 11:36:04,854][62782] Avg episode reward: [(0, '26.257')] [2023-06-21 11:36:09,851][62782] Fps is (10 sec: 1600.3, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4757888. Throughput: 0: 310.5. Samples: 188138. Policy #0 lag: (min: 0.0, avg: 1.5, max: 2.0) [2023-06-21 11:36:09,854][62782] Avg episode reward: [(0, '26.341')] [2023-06-21 11:36:14,853][62782] Fps is (10 sec: 1280.1, 60 sec: 1226.7, 300 sec: 1236.6). Total num frames: 4761088. Throughput: 0: 310.7. Samples: 190044. Policy #0 lag: (min: 0.0, avg: 1.5, max: 2.0) [2023-06-21 11:36:14,854][62782] Avg episode reward: [(0, '27.595')] [2023-06-21 11:36:19,852][62782] Fps is (10 sec: 959.9, 60 sec: 1226.7, 300 sec: 1236.6). Total num frames: 4767488. Throughput: 0: 309.5. Samples: 191848. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:36:19,854][62782] Avg episode reward: [(0, '26.390')] [2023-06-21 11:36:22,351][73125] Updated weights for policy 0, policy_version 1218 (0.0319) [2023-06-21 11:36:24,854][62782] Fps is (10 sec: 1279.9, 60 sec: 1226.6, 300 sec: 1236.6). Total num frames: 4773888. Throughput: 0: 309.4. Samples: 192794. Policy #0 lag: (min: 1.0, avg: 1.4, max: 2.0) [2023-06-21 11:36:24,854][62782] Avg episode reward: [(0, '26.212')] [2023-06-21 11:36:29,850][62782] Fps is (10 sec: 1280.2, 60 sec: 1226.7, 300 sec: 1236.6). Total num frames: 4780288. Throughput: 0: 309.0. Samples: 194672. Policy #0 lag: (min: 0.0, avg: 1.0, max: 3.0) [2023-06-21 11:36:29,853][62782] Avg episode reward: [(0, '27.096')] [2023-06-21 11:36:34,850][62782] Fps is (10 sec: 1280.5, 60 sec: 1226.7, 300 sec: 1236.6). Total num frames: 4786688. Throughput: 0: 309.6. Samples: 196562. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:36:34,851][62782] Avg episode reward: [(0, '27.697')] [2023-06-21 11:36:39,853][62782] Fps is (10 sec: 1279.7, 60 sec: 1226.7, 300 sec: 1236.6). Total num frames: 4793088. Throughput: 0: 309.6. Samples: 197508. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:36:39,855][62782] Avg episode reward: [(0, '27.702')] [2023-06-21 11:36:44,852][62782] Fps is (10 sec: 1279.7, 60 sec: 1226.7, 300 sec: 1236.6). Total num frames: 4799488. Throughput: 0: 312.3. Samples: 199406. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:36:44,853][62782] Avg episode reward: [(0, '26.893')] [2023-06-21 11:36:48,082][73125] Updated weights for policy 0, policy_version 1228 (0.0007) [2023-06-21 11:36:49,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1226.6, 300 sec: 1236.6). Total num frames: 4805888. Throughput: 0: 313.7. Samples: 201312. Policy #0 lag: (min: 0.0, avg: 1.5, max: 2.0) [2023-06-21 11:36:49,854][62782] Avg episode reward: [(0, '27.118')] [2023-06-21 11:36:54,854][62782] Fps is (10 sec: 1279.9, 60 sec: 1280.0, 300 sec: 1236.6). Total num frames: 4812288. Throughput: 0: 313.4. Samples: 202242. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:36:54,856][62782] Avg episode reward: [(0, '27.142')] [2023-06-21 11:36:59,852][62782] Fps is (10 sec: 1280.1, 60 sec: 1280.0, 300 sec: 1236.6). Total num frames: 4818688. Throughput: 0: 313.4. Samples: 204146. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:36:59,853][62782] Avg episode reward: [(0, '27.241')] [2023-06-21 11:37:04,853][62782] Fps is (10 sec: 1280.1, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4825088. Throughput: 0: 315.5. Samples: 206044. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:37:04,854][62782] Avg episode reward: [(0, '26.391')] [2023-06-21 11:37:09,849][62782] Fps is (10 sec: 1280.4, 60 sec: 1226.7, 300 sec: 1236.6). Total num frames: 4831488. Throughput: 0: 315.5. Samples: 206990. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:37:09,850][62782] Avg episode reward: [(0, '26.677')] [2023-06-21 11:37:13,908][73125] Updated weights for policy 0, policy_version 1238 (0.0007) [2023-06-21 11:37:14,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4837888. Throughput: 0: 315.9. Samples: 208890. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:37:14,854][62782] Avg episode reward: [(0, '26.646')] [2023-06-21 11:37:19,853][62782] Fps is (10 sec: 1279.6, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4844288. Throughput: 0: 316.1. Samples: 210788. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:37:19,853][62782] Avg episode reward: [(0, '26.006')] [2023-06-21 11:37:24,852][62782] Fps is (10 sec: 1280.2, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 4850688. Throughput: 0: 316.2. Samples: 211738. Policy #0 lag: (min: 0.0, avg: 1.1, max: 3.0) [2023-06-21 11:37:24,853][62782] Avg episode reward: [(0, '27.005')] [2023-06-21 11:37:29,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1279.9, 300 sec: 1258.3). Total num frames: 4857088. Throughput: 0: 316.0. Samples: 213628. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:37:29,855][62782] Avg episode reward: [(0, '27.175')] [2023-06-21 11:37:34,853][62782] Fps is (10 sec: 1279.9, 60 sec: 1279.9, 300 sec: 1258.4). Total num frames: 4863488. Throughput: 0: 316.0. Samples: 215534. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:37:34,854][62782] Avg episode reward: [(0, '27.501')] [2023-06-21 11:37:39,541][73125] Updated weights for policy 0, policy_version 1248 (0.0007) [2023-06-21 11:37:39,851][62782] Fps is (10 sec: 1280.3, 60 sec: 1280.0, 300 sec: 1258.3). Total num frames: 4869888. Throughput: 0: 316.5. Samples: 216484. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:37:39,852][62782] Avg episode reward: [(0, '27.327')] [2023-06-21 11:37:44,852][62782] Fps is (10 sec: 1280.1, 60 sec: 1280.0, 300 sec: 1258.3). Total num frames: 4876288. Throughput: 0: 316.3. Samples: 218378. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:37:44,855][62782] Avg episode reward: [(0, '27.264')] [2023-06-21 11:37:46,600][73121] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001251_4879488.pth... [2023-06-21 11:37:46,707][73121] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001157_4578688.pth [2023-06-21 11:37:49,854][62782] Fps is (10 sec: 1279.6, 60 sec: 1280.0, 300 sec: 1258.3). Total num frames: 4882688. Throughput: 0: 316.2. Samples: 220272. Policy #0 lag: (min: 0.0, avg: 1.1, max: 3.0) [2023-06-21 11:37:49,856][62782] Avg episode reward: [(0, '27.242')] [2023-06-21 11:37:54,853][62782] Fps is (10 sec: 1279.8, 60 sec: 1280.0, 300 sec: 1258.3). Total num frames: 4889088. Throughput: 0: 316.2. Samples: 221222. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:37:54,854][62782] Avg episode reward: [(0, '27.878')] [2023-06-21 11:37:59,853][62782] Fps is (10 sec: 1280.1, 60 sec: 1280.0, 300 sec: 1258.3). Total num frames: 4895488. Throughput: 0: 316.2. Samples: 223120. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:37:59,855][62782] Avg episode reward: [(0, '28.284')] [2023-06-21 11:38:04,184][73125] Updated weights for policy 0, policy_version 1258 (0.0010) [2023-06-21 11:38:04,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1280.0, 300 sec: 1258.3). Total num frames: 4901888. Throughput: 0: 314.9. Samples: 224960. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:38:04,854][62782] Avg episode reward: [(0, '28.075')] [2023-06-21 11:38:09,852][62782] Fps is (10 sec: 1280.0, 60 sec: 1279.9, 300 sec: 1258.3). Total num frames: 4908288. Throughput: 0: 315.1. Samples: 225916. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:38:09,854][62782] Avg episode reward: [(0, '27.714')] [2023-06-21 11:38:14,852][62782] Fps is (10 sec: 1280.2, 60 sec: 1280.0, 300 sec: 1258.3). Total num frames: 4914688. Throughput: 0: 315.5. Samples: 227824. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:38:14,854][62782] Avg episode reward: [(0, '28.460')] [2023-06-21 11:38:19,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1280.0, 300 sec: 1258.3). Total num frames: 4921088. Throughput: 0: 315.2. Samples: 229716. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:38:19,854][62782] Avg episode reward: [(0, '28.247')] [2023-06-21 11:38:24,853][62782] Fps is (10 sec: 959.9, 60 sec: 1226.6, 300 sec: 1247.5). Total num frames: 4924288. Throughput: 0: 315.2. Samples: 230668. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:38:24,854][62782] Avg episode reward: [(0, '28.745')] [2023-06-21 11:38:25,181][73121] Saving new best policy, reward=28.745! [2023-06-21 11:38:29,602][73125] Updated weights for policy 0, policy_version 1268 (0.0008) [2023-06-21 11:38:29,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1280.0, 300 sec: 1258.3). Total num frames: 4933888. Throughput: 0: 315.0. Samples: 232552. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:38:29,854][62782] Avg episode reward: [(0, '28.447')] [2023-06-21 11:38:34,854][62782] Fps is (10 sec: 1280.2, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4937088. Throughput: 0: 314.8. Samples: 234438. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) [2023-06-21 11:38:34,866][62782] Avg episode reward: [(0, '28.242')] [2023-06-21 11:38:39,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1280.0, 300 sec: 1258.3). Total num frames: 4946688. Throughput: 0: 314.7. Samples: 235384. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:38:39,854][62782] Avg episode reward: [(0, '28.286')] [2023-06-21 11:38:44,852][62782] Fps is (10 sec: 1280.0, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4949888. Throughput: 0: 312.1. Samples: 237166. Policy #0 lag: (min: 0.0, avg: 1.3, max: 3.0) [2023-06-21 11:38:44,855][62782] Avg episode reward: [(0, '27.654')] [2023-06-21 11:38:49,853][62782] Fps is (10 sec: 960.0, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4956288. Throughput: 0: 313.6. Samples: 239074. Policy #0 lag: (min: 0.0, avg: 1.2, max: 2.0) [2023-06-21 11:38:49,855][62782] Avg episode reward: [(0, '27.246')] [2023-06-21 11:38:54,854][62782] Fps is (10 sec: 1279.7, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4962688. Throughput: 0: 313.5. Samples: 240022. Policy #0 lag: (min: 0.0, avg: 1.5, max: 2.0) [2023-06-21 11:38:54,856][62782] Avg episode reward: [(0, '27.106')] [2023-06-21 11:38:55,103][73125] Updated weights for policy 0, policy_version 1278 (0.0296) [2023-06-21 11:38:59,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4969088. Throughput: 0: 313.1. Samples: 241916. Policy #0 lag: (min: 0.0, avg: 1.1, max: 2.0) [2023-06-21 11:38:59,855][62782] Avg episode reward: [(0, '26.498')] [2023-06-21 11:39:04,853][62782] Fps is (10 sec: 1280.1, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4975488. Throughput: 0: 313.1. Samples: 243806. Policy #0 lag: (min: 1.0, avg: 1.5, max: 2.0) [2023-06-21 11:39:04,856][62782] Avg episode reward: [(0, '26.437')] [2023-06-21 11:39:09,853][62782] Fps is (10 sec: 1280.1, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4981888. Throughput: 0: 312.8. Samples: 244744. Policy #0 lag: (min: 0.0, avg: 1.1, max: 3.0) [2023-06-21 11:39:09,854][62782] Avg episode reward: [(0, '26.118')] [2023-06-21 11:39:14,853][62782] Fps is (10 sec: 1280.0, 60 sec: 1226.6, 300 sec: 1258.3). Total num frames: 4988288. Throughput: 0: 313.0. Samples: 246638. Policy #0 lag: (min: 0.0, avg: 1.4, max: 2.0) [2023-06-21 11:39:14,854][62782] Avg episode reward: [(0, '25.882')] [2023-06-21 11:39:19,853][62782] Fps is (10 sec: 1279.9, 60 sec: 1226.7, 300 sec: 1247.5). Total num frames: 4994688. Throughput: 0: 313.5. Samples: 248544. Policy #0 lag: (min: 0.0, avg: 1.1, max: 3.0) [2023-06-21 11:39:19,855][62782] Avg episode reward: [(0, '25.809')] [2023-06-21 11:39:20,932][73125] Updated weights for policy 0, policy_version 1288 (0.0011) [2023-06-21 11:39:24,852][62782] Fps is (10 sec: 1280.2, 60 sec: 1280.0, 300 sec: 1247.5). Total num frames: 5001088. Throughput: 0: 313.6. Samples: 249494. Policy #0 lag: (min: 0.0, avg: 1.3, max: 2.0) [2023-06-21 11:39:24,854][62782] Avg episode reward: [(0, '25.532')] [2023-06-21 11:39:25,866][73121] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001290_5004288.pth... [2023-06-21 11:39:25,872][62782] Component Batcher_0 stopped! [2023-06-21 11:39:25,867][73121] Stopping Batcher_0... [2023-06-21 11:39:25,892][73121] Loop batcher_evt_loop terminating... [2023-06-21 11:39:26,327][73130] Stopping RolloutWorker_w4... [2023-06-21 11:39:26,335][73130] Loop rollout_proc4_evt_loop terminating... [2023-06-21 11:39:26,319][73127] Stopping RolloutWorker_w1... [2023-06-21 11:39:26,339][73121] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001204_4729088.pth [2023-06-21 11:39:26,331][62782] Component RolloutWorker_w5 stopped! [2023-06-21 11:39:26,342][73127] Loop rollout_proc1_evt_loop terminating... [2023-06-21 11:39:26,342][73126] Stopping RolloutWorker_w0... [2023-06-21 11:39:26,343][73129] Stopping RolloutWorker_w3... [2023-06-21 11:39:26,342][62782] Component RolloutWorker_w1 stopped! [2023-06-21 11:39:26,347][73126] Loop rollout_proc0_evt_loop terminating... [2023-06-21 11:39:26,347][62782] Component RolloutWorker_w4 stopped! [2023-06-21 11:39:26,349][62782] Component RolloutWorker_w0 stopped! [2023-06-21 11:39:26,350][73129] Loop rollout_proc3_evt_loop terminating... [2023-06-21 11:39:26,348][73132] Stopping RolloutWorker_w6... [2023-06-21 11:39:26,335][73131] Stopping RolloutWorker_w5... [2023-06-21 11:39:26,350][73133] Stopping RolloutWorker_w7... [2023-06-21 11:39:26,358][73133] Loop rollout_proc7_evt_loop terminating... [2023-06-21 11:39:26,357][62782] Component RolloutWorker_w3 stopped! [2023-06-21 11:39:26,358][73121] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001290_5004288.pth... [2023-06-21 11:39:26,358][62782] Component RolloutWorker_w6 stopped! [2023-06-21 11:39:26,359][62782] Component RolloutWorker_w7 stopped! [2023-06-21 11:39:26,361][73132] Loop rollout_proc6_evt_loop terminating... [2023-06-21 11:39:26,362][73131] Loop rollout_proc5_evt_loop terminating... [2023-06-21 11:39:26,360][62782] Component RolloutWorker_w2 stopped! [2023-06-21 11:39:26,364][73128] Stopping RolloutWorker_w2... [2023-06-21 11:39:26,368][73128] Loop rollout_proc2_evt_loop terminating... [2023-06-21 11:39:26,501][73121] Stopping LearnerWorker_p0... [2023-06-21 11:39:26,504][73121] Loop learner_proc0_evt_loop terminating... [2023-06-21 11:39:26,501][62782] Component LearnerWorker_p0 stopped! [2023-06-21 11:39:26,669][73125] Weights refcount: 2 0 [2023-06-21 11:39:26,673][73125] Stopping InferenceWorker_p0-w0... [2023-06-21 11:39:26,674][73125] Loop inference_proc0-0_evt_loop terminating... [2023-06-21 11:39:26,674][62782] Component InferenceWorker_p0-w0 stopped! [2023-06-21 11:39:26,676][62782] Waiting for process learner_proc0 to stop... [2023-06-21 11:39:27,347][62782] Waiting for process inference_proc0-0 to join... [2023-06-21 11:39:27,349][62782] Waiting for process rollout_proc0 to join... [2023-06-21 11:39:27,350][62782] Waiting for process rollout_proc1 to join... [2023-06-21 11:39:27,350][62782] Waiting for process rollout_proc2 to join... [2023-06-21 11:39:27,350][62782] Waiting for process rollout_proc3 to join... [2023-06-21 11:39:27,351][62782] Waiting for process rollout_proc4 to join... [2023-06-21 11:39:27,351][62782] Waiting for process rollout_proc5 to join... [2023-06-21 11:39:27,351][62782] Waiting for process rollout_proc6 to join... [2023-06-21 11:39:27,352][62782] Waiting for process rollout_proc7 to join... [2023-06-21 11:39:27,352][62782] Batcher 0 profile tree view: batching: 3.1957, releasing_batches: 0.0056 [2023-06-21 11:39:27,353][62782] InferenceWorker_p0-w0 profile tree view: wait_policy: 0.0006 wait_policy_total: 526.3775 update_model: 4.0355 weight_update: 0.0372 one_step: 0.0048 handle_policy_step: 278.7788 deserialize: 2.5920, stack: 0.5194, obs_to_device_normalize: 18.4224, forward: 246.0035, send_messages: 2.7946 prepare_outputs: 3.2735 to_cpu: 0.3546 [2023-06-21 11:39:27,354][62782] Learner 0 profile tree view: misc: 0.0010, prepare_batch: 114.9491 train: 410.3481 epoch_init: 0.0008, minibatch_init: 0.0029, losses_postprocess: 0.0080, kl_divergence: 0.0415, after_optimizer: 0.2949 calculate_losses: 232.6958 losses_init: 0.0009, forward_head: 223.7004, bptt_initial: 0.4386, tail: 0.3708, advantages_returns: 0.0255, losses: 0.1252 bptt: 7.9652 bptt_forward_core: 7.8650 update: 177.1688 clip: 0.4995 [2023-06-21 11:39:27,354][62782] RolloutWorker_w0 profile tree view: wait_for_trajectories: 0.0289, enqueue_policy_requests: 2.1631, env_step: 792.9321, overhead: 2.0880, complete_rollouts: 0.0414 save_policy_outputs: 1.0060 split_output_tensors: 0.5117 [2023-06-21 11:39:27,354][62782] RolloutWorker_w7 profile tree view: wait_for_trajectories: 0.0372, enqueue_policy_requests: 2.1720, env_step: 791.6971, overhead: 1.9774, complete_rollouts: 0.0501 save_policy_outputs: 1.0087 split_output_tensors: 0.5009 [2023-06-21 11:39:27,355][62782] Loop Runner_EvtLoop terminating... [2023-06-21 11:39:27,356][62782] Runner profile tree view: main_loop: 821.1787 [2023-06-21 11:39:27,356][62782] Collected {0: 5004288}, FPS: 1215.8 [2023-06-21 12:00:13,874][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 12:00:13,876][62782] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-21 12:00:13,876][62782] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-21 12:00:13,877][62782] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-21 12:00:13,878][62782] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 12:00:13,879][62782] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-21 12:00:13,879][62782] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! [2023-06-21 12:00:13,879][62782] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-21 12:00:13,879][62782] Adding new argument 'push_to_hub'=True that is not in the saved config file! [2023-06-21 12:00:13,880][62782] Adding new argument 'hf_repository'='mihirdeo16/vizdoom_health_gathering_supreme' that is not in the saved config file! [2023-06-21 12:00:13,880][62782] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-21 12:00:13,880][62782] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-21 12:00:13,880][62782] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-21 12:00:13,881][62782] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-21 12:00:13,881][62782] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-21 12:00:13,928][62782] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 12:00:13,942][62782] RunningMeanStd input shape: (1,) [2023-06-21 12:00:13,981][62782] ConvEncoder: input_channels=3 [2023-06-21 12:00:14,040][62782] Conv encoder output size: 512 [2023-06-21 12:00:14,040][62782] Policy head output size: 512 [2023-06-21 12:00:14,065][62782] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001290_5004288.pth... [2023-06-21 12:00:16,405][62782] Num frames 100... [2023-06-21 12:00:17,176][62782] Num frames 200... [2023-06-21 12:00:17,979][62782] Num frames 300... [2023-06-21 12:00:18,769][62782] Num frames 400... [2023-06-21 12:00:19,504][62782] Num frames 500... [2023-06-21 12:00:20,254][62782] Num frames 600... [2023-06-21 12:00:21,039][62782] Num frames 700... [2023-06-21 12:00:21,858][62782] Num frames 800... [2023-06-21 12:00:22,453][62782] Avg episode rewards: #0: 16.640, true rewards: #0: 8.640 [2023-06-21 12:00:22,454][62782] Avg episode reward: 16.640, avg true_objective: 8.640 [2023-06-21 12:00:22,732][62782] Num frames 900... [2023-06-21 12:00:23,496][62782] Num frames 1000... [2023-06-21 12:00:24,243][62782] Num frames 1100... [2023-06-21 12:00:25,001][62782] Num frames 1200... [2023-06-21 12:00:25,783][62782] Num frames 1300... [2023-06-21 12:00:26,577][62782] Num frames 1400... [2023-06-21 12:00:27,375][62782] Num frames 1500... [2023-06-21 12:00:28,168][62782] Num frames 1600... [2023-06-21 12:00:28,969][62782] Num frames 1700... [2023-06-21 12:00:29,753][62782] Num frames 1800... [2023-06-21 12:00:30,547][62782] Num frames 1900... [2023-06-21 12:00:31,365][62782] Num frames 2000... [2023-06-21 12:00:32,161][62782] Num frames 2100... [2023-06-21 12:00:32,962][62782] Num frames 2200... [2023-06-21 12:00:33,793][62782] Num frames 2300... [2023-06-21 12:00:34,619][62782] Num frames 2400... [2023-06-21 12:00:35,436][62782] Num frames 2500... [2023-06-21 12:00:36,246][62782] Num frames 2600... [2023-06-21 12:00:36,579][62782] Avg episode rewards: #0: 30.650, true rewards: #0: 13.150 [2023-06-21 12:00:36,582][62782] Avg episode reward: 30.650, avg true_objective: 13.150 [2023-06-21 12:00:37,137][62782] Num frames 2700... [2023-06-21 12:00:37,940][62782] Num frames 2800... [2023-06-21 12:00:38,754][62782] Num frames 2900... [2023-06-21 12:00:39,548][62782] Num frames 3000... [2023-06-21 12:00:40,342][62782] Num frames 3100... [2023-06-21 12:00:41,012][62782] Avg episode rewards: #0: 25.247, true rewards: #0: 10.580 [2023-06-21 12:00:41,015][62782] Avg episode reward: 25.247, avg true_objective: 10.580 [2023-06-21 12:00:41,205][62782] Num frames 3200... [2023-06-21 12:00:41,995][62782] Num frames 3300... [2023-06-21 12:00:42,808][62782] Num frames 3400... [2023-06-21 12:00:43,632][62782] Num frames 3500... [2023-06-21 12:00:44,451][62782] Avg episode rewards: #0: 20.975, true rewards: #0: 8.975 [2023-06-21 12:00:44,453][62782] Avg episode reward: 20.975, avg true_objective: 8.975 [2023-06-21 12:00:44,531][62782] Num frames 3600... [2023-06-21 12:00:45,335][62782] Num frames 3700... [2023-06-21 12:00:46,136][62782] Num frames 3800... [2023-06-21 12:00:46,944][62782] Num frames 3900... [2023-06-21 12:00:47,732][62782] Num frames 4000... [2023-06-21 12:00:48,516][62782] Num frames 4100... [2023-06-21 12:00:49,304][62782] Num frames 4200... [2023-06-21 12:00:50,093][62782] Num frames 4300... [2023-06-21 12:00:50,863][62782] Num frames 4400... [2023-06-21 12:00:51,671][62782] Num frames 4500... [2023-06-21 12:00:52,454][62782] Num frames 4600... [2023-06-21 12:00:53,245][62782] Num frames 4700... [2023-06-21 12:00:53,654][62782] Avg episode rewards: #0: 22.284, true rewards: #0: 9.484 [2023-06-21 12:00:53,657][62782] Avg episode reward: 22.284, avg true_objective: 9.484 [2023-06-21 12:00:54,119][62782] Num frames 4800... [2023-06-21 12:00:54,917][62782] Num frames 4900... [2023-06-21 12:00:55,711][62782] Num frames 5000... [2023-06-21 12:00:56,504][62782] Num frames 5100... [2023-06-21 12:00:57,289][62782] Num frames 5200... [2023-06-21 12:00:58,080][62782] Num frames 5300... [2023-06-21 12:00:58,874][62782] Num frames 5400... [2023-06-21 12:00:59,698][62782] Num frames 5500... [2023-06-21 12:01:00,493][62782] Num frames 5600... [2023-06-21 12:01:01,319][62782] Num frames 5700... [2023-06-21 12:01:02,131][62782] Num frames 5800... [2023-06-21 12:01:02,952][62782] Num frames 5900... [2023-06-21 12:01:03,761][62782] Num frames 6000... [2023-06-21 12:01:04,545][62782] Avg episode rewards: #0: 23.643, true rewards: #0: 10.143 [2023-06-21 12:01:04,549][62782] Avg episode reward: 23.643, avg true_objective: 10.143 [2023-06-21 12:01:04,652][62782] Num frames 6100... [2023-06-21 12:01:05,459][62782] Num frames 6200... [2023-06-21 12:01:06,247][62782] Num frames 6300... [2023-06-21 12:01:07,059][62782] Num frames 6400... [2023-06-21 12:01:07,841][62782] Num frames 6500... [2023-06-21 12:01:08,628][62782] Num frames 6600... [2023-06-21 12:01:09,431][62782] Num frames 6700... [2023-06-21 12:01:10,213][62782] Num frames 6800... [2023-06-21 12:01:10,475][62782] Avg episode rewards: #0: 22.746, true rewards: #0: 9.746 [2023-06-21 12:01:10,477][62782] Avg episode reward: 22.746, avg true_objective: 9.746 [2023-06-21 12:01:11,088][62782] Num frames 6900... [2023-06-21 12:01:11,884][62782] Num frames 7000... [2023-06-21 12:01:12,708][62782] Num frames 7100... [2023-06-21 12:01:13,515][62782] Num frames 7200... [2023-06-21 12:01:14,309][62782] Num frames 7300... [2023-06-21 12:01:15,138][62782] Num frames 7400... [2023-06-21 12:01:15,949][62782] Num frames 7500... [2023-06-21 12:01:16,776][62782] Num frames 7600... [2023-06-21 12:01:17,569][62782] Num frames 7700... [2023-06-21 12:01:18,400][62782] Num frames 7800... [2023-06-21 12:01:18,553][62782] Avg episode rewards: #0: 22.759, true rewards: #0: 9.759 [2023-06-21 12:01:18,555][62782] Avg episode reward: 22.759, avg true_objective: 9.759 [2023-06-21 12:01:19,316][62782] Num frames 7900... [2023-06-21 12:01:20,102][62782] Num frames 8000... [2023-06-21 12:01:20,884][62782] Num frames 8100... [2023-06-21 12:01:21,679][62782] Num frames 8200... [2023-06-21 12:01:22,491][62782] Num frames 8300... [2023-06-21 12:01:23,284][62782] Num frames 8400... [2023-06-21 12:01:24,068][62782] Num frames 8500... [2023-06-21 12:01:24,847][62782] Num frames 8600... [2023-06-21 12:01:25,649][62782] Num frames 8700... [2023-06-21 12:01:26,428][62782] Num frames 8800... [2023-06-21 12:01:27,022][62782] Avg episode rewards: #0: 22.515, true rewards: #0: 9.848 [2023-06-21 12:01:27,024][62782] Avg episode reward: 22.515, avg true_objective: 9.848 [2023-06-21 12:01:27,305][62782] Num frames 8900... [2023-06-21 12:01:28,115][62782] Num frames 9000... [2023-06-21 12:01:28,916][62782] Num frames 9100... [2023-06-21 12:01:29,704][62782] Num frames 9200... [2023-06-21 12:01:30,479][62782] Num frames 9300... [2023-06-21 12:01:31,270][62782] Num frames 9400... [2023-06-21 12:01:32,068][62782] Num frames 9500... [2023-06-21 12:01:32,184][62782] Avg episode rewards: #0: 21.403, true rewards: #0: 9.503 [2023-06-21 12:01:32,186][62782] Avg episode reward: 21.403, avg true_objective: 9.503 [2023-06-21 12:01:45,122][62782] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4! [2023-06-21 12:07:38,291][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 12:07:38,292][62782] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-21 12:07:38,293][62782] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-21 12:07:38,293][62782] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-21 12:07:38,294][62782] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 12:07:38,294][62782] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-21 12:07:38,294][62782] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! [2023-06-21 12:07:38,294][62782] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-21 12:07:38,295][62782] Adding new argument 'push_to_hub'=True that is not in the saved config file! [2023-06-21 12:07:38,295][62782] Adding new argument 'hf_repository'='mihirdeo16/vizdoom_health_gathering_supreme' that is not in the saved config file! [2023-06-21 12:07:38,295][62782] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-21 12:07:38,296][62782] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-21 12:07:38,296][62782] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-21 12:07:38,296][62782] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-21 12:07:38,296][62782] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-21 12:07:38,314][62782] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 12:07:38,315][62782] RunningMeanStd input shape: (1,) [2023-06-21 12:07:38,326][62782] ConvEncoder: input_channels=3 [2023-06-21 12:07:38,366][62782] Conv encoder output size: 512 [2023-06-21 12:07:38,366][62782] Policy head output size: 512 [2023-06-21 12:07:38,396][62782] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001290_5004288.pth... [2023-06-21 12:07:40,093][62782] Num frames 100... [2023-06-21 12:07:40,876][62782] Num frames 200... [2023-06-21 12:07:41,660][62782] Num frames 300... [2023-06-21 12:07:42,491][62782] Num frames 400... [2023-06-21 12:07:43,267][62782] Num frames 500... [2023-06-21 12:07:44,044][62782] Num frames 600... [2023-06-21 12:07:44,844][62782] Num frames 700... [2023-06-21 12:07:45,619][62782] Num frames 800... [2023-06-21 12:07:46,439][62782] Num frames 900... [2023-06-21 12:07:47,244][62782] Num frames 1000... [2023-06-21 12:07:48,002][62782] Num frames 1100... [2023-06-21 12:07:48,744][62782] Num frames 1200... [2023-06-21 12:07:49,506][62782] Num frames 1300... [2023-06-21 12:07:50,309][62782] Num frames 1400... [2023-06-21 12:07:50,917][62782] Avg episode rewards: #0: 34.720, true rewards: #0: 14.720 [2023-06-21 12:07:50,920][62782] Avg episode reward: 34.720, avg true_objective: 14.720 [2023-06-21 12:07:51,138][62782] Num frames 1500... [2023-06-21 12:07:51,926][62782] Num frames 1600... [2023-06-21 12:07:52,762][62782] Num frames 1700... [2023-06-21 12:07:53,541][62782] Num frames 1800... [2023-06-21 12:07:54,335][62782] Num frames 1900... [2023-06-21 12:07:55,149][62782] Num frames 2000... [2023-06-21 12:07:55,363][62782] Avg episode rewards: #0: 21.580, true rewards: #0: 10.080 [2023-06-21 12:07:55,366][62782] Avg episode reward: 21.580, avg true_objective: 10.080 [2023-06-21 12:07:56,042][62782] Num frames 2100... [2023-06-21 12:07:56,849][62782] Num frames 2200... [2023-06-21 12:07:57,671][62782] Num frames 2300... [2023-06-21 12:07:58,489][62782] Num frames 2400... [2023-06-21 12:07:59,334][62782] Num frames 2500... [2023-06-21 12:08:00,129][62782] Num frames 2600... [2023-06-21 12:08:00,916][62782] Num frames 2700... [2023-06-21 12:08:01,720][62782] Num frames 2800... [2023-06-21 12:08:02,484][62782] Num frames 2900... [2023-06-21 12:08:03,379][62782] Num frames 3000... [2023-06-21 12:08:04,275][62782] Num frames 3100... [2023-06-21 12:08:05,221][62782] Num frames 3200... [2023-06-21 12:08:06,182][62782] Num frames 3300... [2023-06-21 12:08:07,164][62782] Num frames 3400... [2023-06-21 12:08:08,111][62782] Num frames 3500... [2023-06-21 12:08:09,083][62782] Num frames 3600... [2023-06-21 12:08:10,062][62782] Avg episode rewards: #0: 29.300, true rewards: #0: 12.300 [2023-06-21 12:08:10,065][62782] Avg episode reward: 29.300, avg true_objective: 12.300 [2023-06-21 12:08:10,147][62782] Num frames 3700... [2023-06-21 12:08:11,111][62782] Num frames 3800... [2023-06-21 12:08:12,046][62782] Num frames 3900... [2023-06-21 12:08:13,011][62782] Num frames 4000... [2023-06-21 12:08:13,999][62782] Num frames 4100... [2023-06-21 12:08:15,018][62782] Num frames 4200... [2023-06-21 12:08:15,753][62782] Num frames 4300... [2023-06-21 12:08:16,699][62782] Num frames 4400... [2023-06-21 12:08:17,657][62782] Num frames 4500... [2023-06-21 12:08:18,616][62782] Num frames 4600... [2023-06-21 12:08:19,588][62782] Num frames 4700... [2023-06-21 12:08:20,463][62782] Avg episode rewards: #0: 28.195, true rewards: #0: 11.945 [2023-06-21 12:08:20,466][62782] Avg episode reward: 28.195, avg true_objective: 11.945 [2023-06-21 12:08:20,654][62782] Num frames 4800... [2023-06-21 12:08:21,619][62782] Num frames 4900... [2023-06-21 12:08:22,561][62782] Num frames 5000... [2023-06-21 12:08:23,504][62782] Num frames 5100... [2023-06-21 12:08:24,306][62782] Num frames 5200... [2023-06-21 12:08:25,207][62782] Num frames 5300... [2023-06-21 12:08:26,161][62782] Num frames 5400... [2023-06-21 12:08:27,125][62782] Num frames 5500... [2023-06-21 12:08:28,042][62782] Num frames 5600... [2023-06-21 12:08:29,010][62782] Num frames 5700... [2023-06-21 12:08:29,886][62782] Num frames 5800... [2023-06-21 12:08:30,652][62782] Num frames 5900... [2023-06-21 12:08:31,401][62782] Num frames 6000... [2023-06-21 12:08:32,167][62782] Num frames 6100... [2023-06-21 12:08:32,997][62782] Num frames 6200... [2023-06-21 12:08:33,716][62782] Num frames 6300... [2023-06-21 12:08:34,500][62782] Num frames 6400... [2023-06-21 12:08:35,309][62782] Num frames 6500... [2023-06-21 12:08:36,076][62782] Num frames 6600... [2023-06-21 12:08:36,843][62782] Num frames 6700... [2023-06-21 12:08:37,656][62782] Num frames 6800... [2023-06-21 12:08:38,358][62782] Avg episode rewards: #0: 33.556, true rewards: #0: 13.756 [2023-06-21 12:08:38,361][62782] Avg episode reward: 33.556, avg true_objective: 13.756 [2023-06-21 12:08:38,522][62782] Num frames 6900... [2023-06-21 12:08:39,312][62782] Num frames 7000... [2023-06-21 12:08:40,099][62782] Num frames 7100... [2023-06-21 12:08:40,926][62782] Num frames 7200... [2023-06-21 12:08:41,717][62782] Num frames 7300... [2023-06-21 12:08:42,541][62782] Num frames 7400... [2023-06-21 12:08:43,334][62782] Num frames 7500... [2023-06-21 12:08:44,121][62782] Num frames 7600... [2023-06-21 12:08:44,948][62782] Num frames 7700... [2023-06-21 12:08:45,864][62782] Num frames 7800... [2023-06-21 12:08:46,775][62782] Num frames 7900... [2023-06-21 12:08:47,731][62782] Num frames 8000... [2023-06-21 12:08:48,630][62782] Num frames 8100... [2023-06-21 12:08:49,632][62782] Num frames 8200... [2023-06-21 12:08:50,566][62782] Num frames 8300... [2023-06-21 12:08:51,530][62782] Num frames 8400... [2023-06-21 12:08:52,493][62782] Num frames 8500... [2023-06-21 12:08:53,378][62782] Num frames 8600... [2023-06-21 12:08:54,302][62782] Num frames 8700... [2023-06-21 12:08:55,236][62782] Num frames 8800... [2023-06-21 12:08:56,129][62782] Num frames 8900... [2023-06-21 12:08:56,988][62782] Avg episode rewards: #0: 36.796, true rewards: #0: 14.963 [2023-06-21 12:08:56,990][62782] Avg episode reward: 36.796, avg true_objective: 14.963 [2023-06-21 12:08:57,173][62782] Num frames 9000... [2023-06-21 12:08:58,088][62782] Num frames 9100... [2023-06-21 12:08:58,948][62782] Num frames 9200... [2023-06-21 12:08:59,864][62782] Num frames 9300... [2023-06-21 12:09:00,713][62782] Num frames 9400... [2023-06-21 12:09:01,652][62782] Num frames 9500... [2023-06-21 12:09:02,593][62782] Num frames 9600... [2023-06-21 12:09:03,458][62782] Avg episode rewards: #0: 33.402, true rewards: #0: 13.831 [2023-06-21 12:09:03,460][62782] Avg episode reward: 33.402, avg true_objective: 13.831 [2023-06-21 12:09:03,604][62782] Num frames 9700... [2023-06-21 12:09:04,564][62782] Num frames 9800... [2023-06-21 12:09:05,411][62782] Num frames 9900... [2023-06-21 12:09:06,353][62782] Num frames 10000... [2023-06-21 12:09:07,215][62782] Num frames 10100... [2023-06-21 12:09:08,078][62782] Num frames 10200... [2023-06-21 12:09:08,946][62782] Num frames 10300... [2023-06-21 12:09:09,740][62782] Num frames 10400... [2023-06-21 12:09:10,503][62782] Num frames 10500... [2023-06-21 12:09:11,287][62782] Num frames 10600... [2023-06-21 12:09:12,080][62782] Num frames 10700... [2023-06-21 12:09:12,886][62782] Num frames 10800... [2023-06-21 12:09:13,658][62782] Num frames 10900... [2023-06-21 12:09:14,450][62782] Num frames 11000... [2023-06-21 12:09:15,222][62782] Num frames 11100... [2023-06-21 12:09:16,003][62782] Num frames 11200... [2023-06-21 12:09:16,795][62782] Num frames 11300... [2023-06-21 12:09:17,582][62782] Num frames 11400... [2023-06-21 12:09:18,351][62782] Num frames 11500... [2023-06-21 12:09:19,119][62782] Num frames 11600... [2023-06-21 12:09:19,911][62782] Num frames 11700... [2023-06-21 12:09:20,659][62782] Avg episode rewards: #0: 36.477, true rewards: #0: 14.728 [2023-06-21 12:09:20,661][62782] Avg episode reward: 36.477, avg true_objective: 14.728 [2023-06-21 12:09:20,795][62782] Num frames 11800... [2023-06-21 12:09:21,594][62782] Num frames 11900... [2023-06-21 12:09:22,383][62782] Num frames 12000... [2023-06-21 12:09:23,152][62782] Num frames 12100... [2023-06-21 12:09:23,933][62782] Num frames 12200... [2023-06-21 12:09:24,708][62782] Num frames 12300... [2023-06-21 12:09:25,491][62782] Num frames 12400... [2023-06-21 12:09:26,285][62782] Num frames 12500... [2023-06-21 12:09:27,071][62782] Num frames 12600... [2023-06-21 12:09:27,882][62782] Num frames 12700... [2023-06-21 12:09:28,564][62782] Avg episode rewards: #0: 34.862, true rewards: #0: 14.196 [2023-06-21 12:09:28,567][62782] Avg episode reward: 34.862, avg true_objective: 14.196 [2023-06-21 12:09:28,749][62782] Num frames 12800... [2023-06-21 12:09:29,539][62782] Num frames 12900... [2023-06-21 12:09:30,337][62782] Num frames 13000... [2023-06-21 12:09:31,121][62782] Num frames 13100... [2023-06-21 12:09:31,921][62782] Num frames 13200... [2023-06-21 12:09:32,706][62782] Num frames 13300... [2023-06-21 12:09:33,487][62782] Num frames 13400... [2023-06-21 12:09:34,287][62782] Num frames 13500... [2023-06-21 12:09:35,051][62782] Num frames 13600... [2023-06-21 12:09:35,878][62782] Num frames 13700... [2023-06-21 12:09:36,667][62782] Num frames 13800... [2023-06-21 12:09:37,560][62782] Num frames 13900... [2023-06-21 12:09:38,333][62782] Num frames 14000... [2023-06-21 12:09:39,130][62782] Num frames 14100... [2023-06-21 12:09:39,992][62782] Num frames 14200... [2023-06-21 12:09:40,780][62782] Num frames 14300... [2023-06-21 12:09:41,565][62782] Num frames 14400... [2023-06-21 12:09:42,361][62782] Num frames 14500... [2023-06-21 12:09:43,148][62782] Num frames 14600... [2023-06-21 12:09:43,929][62782] Num frames 14700... [2023-06-21 12:09:44,716][62782] Num frames 14800... [2023-06-21 12:09:45,405][62782] Avg episode rewards: #0: 36.875, true rewards: #0: 14.876 [2023-06-21 12:09:45,408][62782] Avg episode reward: 36.875, avg true_objective: 14.876 [2023-06-21 12:10:05,499][62782] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4! [2023-06-21 12:20:07,528][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 12:20:07,529][62782] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-21 12:20:07,530][62782] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-21 12:20:07,530][62782] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-21 12:20:07,530][62782] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 12:20:07,530][62782] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-21 12:20:07,531][62782] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! [2023-06-21 12:20:07,531][62782] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-21 12:20:07,531][62782] Adding new argument 'push_to_hub'=True that is not in the saved config file! [2023-06-21 12:20:07,531][62782] Adding new argument 'hf_repository'='mihirdeo16/vizdoom_health_gathering_supreme' that is not in the saved config file! [2023-06-21 12:20:07,532][62782] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-21 12:20:07,532][62782] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-21 12:20:07,532][62782] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-21 12:20:07,532][62782] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-21 12:20:07,533][62782] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-21 12:20:07,544][62782] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 12:20:07,555][62782] RunningMeanStd input shape: (1,) [2023-06-21 12:20:07,578][62782] ConvEncoder: input_channels=3 [2023-06-21 12:20:07,641][62782] Conv encoder output size: 512 [2023-06-21 12:20:07,641][62782] Policy head output size: 512 [2023-06-21 12:20:07,663][62782] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001290_5004288.pth... [2023-06-21 12:20:09,972][62782] Num frames 100... [2023-06-21 12:20:10,763][62782] Num frames 200... [2023-06-21 12:20:11,582][62782] Num frames 300... [2023-06-21 12:20:12,372][62782] Num frames 400... [2023-06-21 12:20:13,163][62782] Num frames 500... [2023-06-21 12:20:13,952][62782] Num frames 600... [2023-06-21 12:20:14,733][62782] Num frames 700... [2023-06-21 12:20:15,491][62782] Num frames 800... [2023-06-21 12:20:16,251][62782] Num frames 900... [2023-06-21 12:20:17,068][62782] Num frames 1000... [2023-06-21 12:20:17,836][62782] Num frames 1100... [2023-06-21 12:20:18,604][62782] Num frames 1200... [2023-06-21 12:20:19,410][62782] Num frames 1300... [2023-06-21 12:20:20,194][62782] Num frames 1400... [2023-06-21 12:20:20,987][62782] Num frames 1500... [2023-06-21 12:20:21,807][62782] Num frames 1600... [2023-06-21 12:20:22,615][62782] Num frames 1700... [2023-06-21 12:20:23,425][62782] Num frames 1800... [2023-06-21 12:20:24,213][62782] Num frames 1900... [2023-06-21 12:20:25,018][62782] Num frames 2000... [2023-06-21 12:20:25,874][62782] Num frames 2100... [2023-06-21 12:20:25,936][62782] Avg episode rewards: #0: 60.999, true rewards: #0: 21.000 [2023-06-21 12:20:25,937][62782] Avg episode reward: 60.999, avg true_objective: 21.000 [2023-06-21 12:20:26,713][62782] Num frames 2200... [2023-06-21 12:20:27,513][62782] Num frames 2300... [2023-06-21 12:20:28,330][62782] Num frames 2400... [2023-06-21 12:20:29,157][62782] Num frames 2500... [2023-06-21 12:20:29,950][62782] Num frames 2600... [2023-06-21 12:20:30,742][62782] Num frames 2700... [2023-06-21 12:20:31,044][62782] Avg episode rewards: #0: 36.639, true rewards: #0: 13.640 [2023-06-21 12:20:31,047][62782] Avg episode reward: 36.639, avg true_objective: 13.640 [2023-06-21 12:20:31,623][62782] Num frames 2800... [2023-06-21 12:20:32,404][62782] Num frames 2900... [2023-06-21 12:20:33,194][62782] Num frames 3000... [2023-06-21 12:20:34,011][62782] Num frames 3100... [2023-06-21 12:20:34,816][62782] Num frames 3200... [2023-06-21 12:20:35,634][62782] Num frames 3300... [2023-06-21 12:20:36,422][62782] Num frames 3400... [2023-06-21 12:20:37,247][62782] Num frames 3500... [2023-06-21 12:20:38,030][62782] Num frames 3600... [2023-06-21 12:20:38,833][62782] Num frames 3700... [2023-06-21 12:20:39,661][62782] Num frames 3800... [2023-06-21 12:20:40,441][62782] Num frames 3900... [2023-06-21 12:20:41,249][62782] Num frames 4000... [2023-06-21 12:20:42,074][62782] Avg episode rewards: #0: 35.633, true rewards: #0: 13.633 [2023-06-21 12:20:42,076][62782] Avg episode reward: 35.633, avg true_objective: 13.633 [2023-06-21 12:20:42,155][62782] Num frames 4100... [2023-06-21 12:20:42,951][62782] Num frames 4200... [2023-06-21 12:20:43,739][62782] Num frames 4300... [2023-06-21 12:20:44,536][62782] Num frames 4400... [2023-06-21 12:20:45,356][62782] Num frames 4500... [2023-06-21 12:20:46,133][62782] Num frames 4600... [2023-06-21 12:20:46,950][62782] Num frames 4700... [2023-06-21 12:20:47,751][62782] Num frames 4800... [2023-06-21 12:20:48,558][62782] Num frames 4900... [2023-06-21 12:20:49,380][62782] Num frames 5000... [2023-06-21 12:20:50,212][62782] Num frames 5100... [2023-06-21 12:20:50,911][62782] Avg episode rewards: #0: 31.945, true rewards: #0: 12.945 [2023-06-21 12:20:50,914][62782] Avg episode reward: 31.945, avg true_objective: 12.945 [2023-06-21 12:20:51,077][62782] Num frames 5200... [2023-06-21 12:20:51,888][62782] Num frames 5300... [2023-06-21 12:20:52,656][62782] Num frames 5400... [2023-06-21 12:20:53,493][62782] Num frames 5500... [2023-06-21 12:20:54,287][62782] Num frames 5600... [2023-06-21 12:20:54,445][62782] Avg episode rewards: #0: 27.416, true rewards: #0: 11.216 [2023-06-21 12:20:54,447][62782] Avg episode reward: 27.416, avg true_objective: 11.216 [2023-06-21 12:20:55,214][62782] Num frames 5700... [2023-06-21 12:20:56,052][62782] Num frames 5800... [2023-06-21 12:20:56,845][62782] Num frames 5900... [2023-06-21 12:20:57,657][62782] Num frames 6000... [2023-06-21 12:20:58,470][62782] Num frames 6100... [2023-06-21 12:20:59,299][62782] Num frames 6200... [2023-06-21 12:20:59,441][62782] Avg episode rewards: #0: 24.508, true rewards: #0: 10.342 [2023-06-21 12:20:59,442][62782] Avg episode reward: 24.508, avg true_objective: 10.342 [2023-06-21 12:21:00,215][62782] Num frames 6300... [2023-06-21 12:21:01,017][62782] Num frames 6400... [2023-06-21 12:21:01,820][62782] Num frames 6500... [2023-06-21 12:21:02,618][62782] Num frames 6600... [2023-06-21 12:21:03,129][62782] Avg episode rewards: #0: 22.218, true rewards: #0: 9.504 [2023-06-21 12:21:03,132][62782] Avg episode reward: 22.218, avg true_objective: 9.504 [2023-06-21 12:21:03,494][62782] Num frames 6700... [2023-06-21 12:21:04,295][62782] Num frames 6800... [2023-06-21 12:21:05,090][62782] Num frames 6900... [2023-06-21 12:21:05,895][62782] Num frames 7000... [2023-06-21 12:21:06,717][62782] Num frames 7100... [2023-06-21 12:21:07,318][62782] Avg episode rewards: #0: 20.581, true rewards: #0: 8.956 [2023-06-21 12:21:07,321][62782] Avg episode reward: 20.581, avg true_objective: 8.956 [2023-06-21 12:21:07,594][62782] Num frames 7200... [2023-06-21 12:21:08,403][62782] Num frames 7300... [2023-06-21 12:21:09,200][62782] Num frames 7400... [2023-06-21 12:21:10,022][62782] Num frames 7500... [2023-06-21 12:21:10,837][62782] Num frames 7600... [2023-06-21 12:21:11,643][62782] Num frames 7700... [2023-06-21 12:21:12,467][62782] Num frames 7800... [2023-06-21 12:21:13,257][62782] Num frames 7900... [2023-06-21 12:21:14,088][62782] Num frames 8000... [2023-06-21 12:21:14,889][62782] Num frames 8100... [2023-06-21 12:21:15,700][62782] Num frames 8200... [2023-06-21 12:21:16,466][62782] Num frames 8300... [2023-06-21 12:21:17,274][62782] Num frames 8400... [2023-06-21 12:21:18,111][62782] Num frames 8500... [2023-06-21 12:21:18,927][62782] Num frames 8600... [2023-06-21 12:21:19,701][62782] Num frames 8700... [2023-06-21 12:21:19,806][62782] Avg episode rewards: #0: 22.112, true rewards: #0: 9.668 [2023-06-21 12:21:19,809][62782] Avg episode reward: 22.112, avg true_objective: 9.668 [2023-06-21 12:21:20,613][62782] Num frames 8800... [2023-06-21 12:21:21,391][62782] Num frames 8900... [2023-06-21 12:21:22,173][62782] Num frames 9000... [2023-06-21 12:21:22,975][62782] Num frames 9100... [2023-06-21 12:21:23,767][62782] Num frames 9200... [2023-06-21 12:21:24,551][62782] Num frames 9300... [2023-06-21 12:21:25,358][62782] Num frames 9400... [2023-06-21 12:21:26,143][62782] Num frames 9500... [2023-06-21 12:21:26,926][62782] Num frames 9600... [2023-06-21 12:21:27,766][62782] Num frames 9700... [2023-06-21 12:21:28,576][62782] Num frames 9800... [2023-06-21 12:21:29,227][62782] Avg episode rewards: #0: 22.669, true rewards: #0: 9.869 [2023-06-21 12:21:29,230][62782] Avg episode reward: 22.669, avg true_objective: 9.869 [2023-06-21 12:21:42,665][62782] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4! [2023-06-21 12:22:28,105][62782] The model has been pushed to https://huggingface.co/mihirdeo16/vizdoom_health_gathering_supreme [2023-06-21 12:37:01,363][62782] Environment doom_basic already registered, overwriting... [2023-06-21 12:37:01,366][62782] Environment doom_two_colors_easy already registered, overwriting... [2023-06-21 12:37:01,367][62782] Environment doom_two_colors_hard already registered, overwriting... [2023-06-21 12:37:01,367][62782] Environment doom_dm already registered, overwriting... [2023-06-21 12:37:01,368][62782] Environment doom_dwango5 already registered, overwriting... [2023-06-21 12:37:01,369][62782] Environment doom_my_way_home_flat_actions already registered, overwriting... [2023-06-21 12:37:01,369][62782] Environment doom_defend_the_center_flat_actions already registered, overwriting... [2023-06-21 12:37:01,369][62782] Environment doom_my_way_home already registered, overwriting... [2023-06-21 12:37:01,369][62782] Environment doom_deadly_corridor already registered, overwriting... [2023-06-21 12:37:01,370][62782] Environment doom_defend_the_center already registered, overwriting... [2023-06-21 12:37:01,370][62782] Environment doom_defend_the_line already registered, overwriting... [2023-06-21 12:37:01,370][62782] Environment doom_health_gathering already registered, overwriting... [2023-06-21 12:37:01,371][62782] Environment doom_health_gathering_supreme already registered, overwriting... [2023-06-21 12:37:01,371][62782] Environment doom_battle already registered, overwriting... [2023-06-21 12:37:01,371][62782] Environment doom_battle2 already registered, overwriting... [2023-06-21 12:37:01,372][62782] Environment doom_duel_bots already registered, overwriting... [2023-06-21 12:37:01,372][62782] Environment doom_deathmatch_bots already registered, overwriting... [2023-06-21 12:37:01,372][62782] Environment doom_duel already registered, overwriting... [2023-06-21 12:37:01,372][62782] Environment doom_deathmatch_full already registered, overwriting... [2023-06-21 12:37:01,373][62782] Environment doom_benchmark already registered, overwriting... [2023-06-21 12:37:01,373][62782] register_encoder_factory: [2023-06-21 12:37:01,440][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 12:37:01,441][62782] Overriding arg 'train_for_env_steps' with value 8000000 passed from command line [2023-06-21 12:37:01,464][62782] Experiment dir /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment already exists! [2023-06-21 12:37:01,465][62782] Resuming existing experiment from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment... [2023-06-21 12:37:01,465][62782] Weights and Biases integration disabled [2023-06-21 12:37:01,475][62782] Environment var CUDA_VISIBLE_DEVICES is [2023-06-21 12:37:02,753][62782] Starting experiment with the following configuration: help=False algo=APPO env=doom_health_gathering_supreme experiment=default_experiment train_dir=/Users/md/Code/python/jubilant-memory/RL/train_dir restart_behavior=resume device=cpu seed=None num_policies=1 async_rl=True serial_mode=False batched_sampling=False num_batches_to_accumulate=2 worker_num_splits=2 policy_workers_per_policy=1 max_policy_lag=1000 num_workers=8 num_envs_per_worker=4 batch_size=800 num_batches_per_epoch=1 num_epochs=1 rollout=32 recurrence=32 shuffle_minibatches=False gamma=0.99 reward_scale=1.0 reward_clip=1000.0 value_bootstrap=False normalize_returns=True exploration_loss_coeff=0.001 value_loss_coeff=0.5 kl_loss_coeff=0.0 exploration_loss=symmetric_kl gae_lambda=0.95 ppo_clip_ratio=0.1 ppo_clip_value=0.2 with_vtrace=False vtrace_rho=1.0 vtrace_c=1.0 optimizer=adam adam_eps=1e-06 adam_beta1=0.9 adam_beta2=0.999 max_grad_norm=4.0 learning_rate=0.0001 lr_schedule=constant lr_schedule_kl_threshold=0.008 lr_adaptive_min=1e-06 lr_adaptive_max=0.01 obs_subtract_mean=0.0 obs_scale=255.0 normalize_input=True normalize_input_keys=None decorrelate_experience_max_seconds=0 decorrelate_envs_on_one_worker=True actor_worker_gpus=[] set_workers_cpu_affinity=True force_envs_single_thread=False default_niceness=0 log_to_file=True experiment_summaries_interval=10 flush_summaries_interval=30 stats_avg=100 summaries_use_frameskip=True heartbeat_interval=20 heartbeat_reporting_interval=600 train_for_env_steps=8000000 train_for_seconds=10000000000 save_every_sec=120 keep_checkpoints=2 load_checkpoint_kind=latest save_milestones_sec=-1 save_best_every_sec=5 save_best_metric=reward save_best_after=100000 benchmark=False encoder_mlp_layers=[512, 512] encoder_conv_architecture=convnet_simple encoder_conv_mlp_layers=[512] use_rnn=True rnn_size=512 rnn_type=gru rnn_num_layers=1 decoder_mlp_layers=[] nonlinearity=elu policy_initialization=orthogonal policy_init_gain=1.0 actor_critic_share_weights=True adaptive_stddev=True continuous_tanh_scale=0.0 initial_stddev=1.0 use_env_info_cache=False env_gpu_actions=False env_gpu_observations=True env_frameskip=4 env_framestack=1 pixel_format=CHW use_record_episode_statistics=False with_wandb=False wandb_user=None wandb_project=sample_factory wandb_group=None wandb_job_type=SF wandb_tags=[] with_pbt=False pbt_mix_policies_in_one_env=True pbt_period_env_steps=5000000 pbt_start_mutation=20000000 pbt_replace_fraction=0.3 pbt_mutation_rate=0.15 pbt_replace_reward_gap=0.1 pbt_replace_reward_gap_absolute=1e-06 pbt_optimize_gamma=False pbt_target_objective=true_objective pbt_perturb_min=1.1 pbt_perturb_max=1.5 num_agents=-1 num_humans=0 num_bots=-1 start_bot_difficulty=None timelimit=None res_w=128 res_h=72 wide_aspect_ratio=False eval_env_frameskip=1 fps=35 command_line=--env=doom_health_gathering_supreme --num_workers=8 --num_envs_per_worker=4 --train_for_env_steps=4000000 cli_args={'env': 'doom_health_gathering_supreme', 'num_workers': 8, 'num_envs_per_worker': 4, 'train_for_env_steps': 4000000} git_hash=02ef46648112e6fd3adc4475dfd889e784c0ef87 git_repo_name=https://github.com/mihirdeo16/jubilant-memory.git [2023-06-21 12:37:02,755][62782] Saving configuration to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json... [2023-06-21 12:37:02,777][62782] Rollout worker 0 uses device cpu [2023-06-21 12:37:02,778][62782] Rollout worker 1 uses device cpu [2023-06-21 12:37:02,779][62782] Rollout worker 2 uses device cpu [2023-06-21 12:37:02,780][62782] Rollout worker 3 uses device cpu [2023-06-21 12:37:02,781][62782] Rollout worker 4 uses device cpu [2023-06-21 12:37:02,781][62782] Rollout worker 5 uses device cpu [2023-06-21 12:37:02,781][62782] Rollout worker 6 uses device cpu [2023-06-21 12:37:02,782][62782] Rollout worker 7 uses device cpu [2023-06-21 12:37:02,831][62782] InferenceWorker_p0-w0: min num requests: 2 [2023-06-21 12:37:02,866][62782] Starting all processes... [2023-06-21 12:37:02,867][62782] Starting process learner_proc0 [2023-06-21 12:37:02,921][62782] Starting all processes... [2023-06-21 12:37:02,924][62782] Starting process inference_proc0-0 [2023-06-21 12:37:02,925][62782] Starting process rollout_proc0 [2023-06-21 12:37:02,925][62782] Starting process rollout_proc1 [2023-06-21 12:37:02,926][62782] Starting process rollout_proc2 [2023-06-21 12:37:02,926][62782] Starting process rollout_proc3 [2023-06-21 12:37:02,926][62782] Starting process rollout_proc4 [2023-06-21 12:37:02,926][62782] Starting process rollout_proc5 [2023-06-21 12:37:02,926][62782] Starting process rollout_proc6 [2023-06-21 12:37:02,926][62782] Starting process rollout_proc7 [2023-06-21 12:37:05,147][78402] On MacOS, not setting affinity [2023-06-21 12:37:05,155][78406] On MacOS, not setting affinity [2023-06-21 12:37:05,163][78403] On MacOS, not setting affinity [2023-06-21 12:37:05,163][78409] On MacOS, not setting affinity [2023-06-21 12:37:05,170][78400] Starting seed is not provided [2023-06-21 12:37:05,170][78400] Initializing actor-critic model on device cpu [2023-06-21 12:37:05,172][78400] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 12:37:05,177][78405] On MacOS, not setting affinity [2023-06-21 12:37:05,177][78404] On MacOS, not setting affinity [2023-06-21 12:37:05,179][78400] RunningMeanStd input shape: (1,) [2023-06-21 12:37:05,193][78408] On MacOS, not setting affinity [2023-06-21 12:37:05,197][78400] ConvEncoder: input_channels=3 [2023-06-21 12:37:05,227][78407] On MacOS, not setting affinity [2023-06-21 12:37:05,273][78400] Conv encoder output size: 512 [2023-06-21 12:37:05,273][78400] Policy head output size: 512 [2023-06-21 12:37:05,291][78400] Created Actor Critic model with architecture: [2023-06-21 12:37:05,291][78400] ActorCriticSharedWeights( (obs_normalizer): ObservationNormalizer( (running_mean_std): RunningMeanStdDictInPlace( (running_mean_std): ModuleDict( (obs): RunningMeanStdInPlace() ) ) ) (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace) (encoder): VizdoomEncoder( (basic_encoder): ConvEncoder( (enc): RecursiveScriptModule( original_name=ConvEncoderImpl (conv_head): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Conv2d) (1): RecursiveScriptModule(original_name=ELU) (2): RecursiveScriptModule(original_name=Conv2d) (3): RecursiveScriptModule(original_name=ELU) (4): RecursiveScriptModule(original_name=Conv2d) (5): RecursiveScriptModule(original_name=ELU) ) (mlp_layers): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Linear) (1): RecursiveScriptModule(original_name=ELU) ) ) ) ) (core): ModelCoreRNN( (core): GRU(512, 512) ) (decoder): MlpDecoder( (mlp): Identity() ) (critic_linear): Linear(in_features=512, out_features=1, bias=True) (action_parameterization): ActionParameterizationDefault( (distribution_linear): Linear(in_features=512, out_features=5, bias=True) ) ) [2023-06-21 12:37:05,296][78400] Using optimizer [2023-06-21 12:37:05,297][78400] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001290_5004288.pth... [2023-06-21 12:37:05,321][78400] Loading model from checkpoint [2023-06-21 12:37:05,332][78400] Loaded experiment state at self.train_step=1290, self.env_steps=5004288 [2023-06-21 12:37:05,335][78400] Initialized policy 0 weights for model version 1290 [2023-06-21 12:37:05,335][78400] LearnerWorker_p0 finished initialization! [2023-06-21 12:37:05,340][78401] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 12:37:05,342][78401] RunningMeanStd input shape: (1,) [2023-06-21 12:37:05,355][78401] ConvEncoder: input_channels=3 [2023-06-21 12:37:05,404][78401] Conv encoder output size: 512 [2023-06-21 12:37:05,404][78401] Policy head output size: 512 [2023-06-21 12:37:05,420][62782] Inference worker 0-0 is ready! [2023-06-21 12:37:05,424][62782] All inference workers are ready! Signal rollout workers to start! [2023-06-21 12:37:05,487][78406] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 12:37:05,487][78408] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 12:37:05,487][78404] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 12:37:05,488][78405] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 12:37:05,485][78407] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 12:37:05,486][78409] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 12:37:05,491][78402] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 12:37:05,491][78403] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 12:37:06,479][62782] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 5004288. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 12:37:08,078][78409] Decorrelating experience for 0 frames... [2023-06-21 12:37:08,078][78406] Decorrelating experience for 0 frames... [2023-06-21 12:37:08,078][78408] Decorrelating experience for 0 frames... [2023-06-21 12:37:08,078][78407] Decorrelating experience for 0 frames... [2023-06-21 12:37:08,078][78402] Decorrelating experience for 0 frames... [2023-06-21 12:37:08,864][78402] Decorrelating experience for 32 frames... [2023-06-21 12:37:08,864][78406] Decorrelating experience for 32 frames... [2023-06-21 12:37:08,868][78403] Decorrelating experience for 0 frames... [2023-06-21 12:37:08,952][78405] Decorrelating experience for 0 frames... [2023-06-21 12:37:09,628][78409] Decorrelating experience for 32 frames... [2023-06-21 12:37:09,633][78407] Decorrelating experience for 32 frames... [2023-06-21 12:37:09,633][78403] Decorrelating experience for 32 frames... [2023-06-21 12:37:10,492][78406] Decorrelating experience for 64 frames... [2023-06-21 12:37:10,492][78402] Decorrelating experience for 64 frames... [2023-06-21 12:37:11,124][78403] Decorrelating experience for 64 frames... [2023-06-21 12:37:11,124][78407] Decorrelating experience for 64 frames... [2023-06-21 12:37:11,478][62782] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 5004288. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 12:37:12,774][78406] Decorrelating experience for 96 frames... [2023-06-21 12:37:12,779][78402] Decorrelating experience for 96 frames... [2023-06-21 12:37:13,462][78407] Decorrelating experience for 96 frames... [2023-06-21 12:37:13,462][78403] Decorrelating experience for 96 frames... [2023-06-21 12:37:16,475][62782] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 5004288. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 12:37:16,479][62782] Avg episode reward: [(0, '0.320')] [2023-06-21 12:37:21,477][62782] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 5004288. Throughput: 0: 51.6. Samples: 774. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 12:37:21,480][62782] Avg episode reward: [(0, '0.747')] [2023-06-21 12:37:22,819][62782] Heartbeat connected on Batcher_0 [2023-06-21 12:37:22,870][62782] Heartbeat connected on InferenceWorker_p0-w0 [2023-06-21 12:37:22,902][62782] Heartbeat connected on RolloutWorker_w1 [2023-06-21 12:37:22,918][62782] Heartbeat connected on RolloutWorker_w0 [2023-06-21 12:37:22,933][62782] Heartbeat connected on RolloutWorker_w4 [2023-06-21 12:37:22,942][62782] Heartbeat connected on RolloutWorker_w5 [2023-06-21 12:37:23,859][62782] Heartbeat connected on LearnerWorker_p0 [2023-06-21 12:37:26,477][62782] Fps is (10 sec: 320.0, 60 sec: 160.0, 300 sec: 160.0). Total num frames: 5007488. Throughput: 0: 86.2. Samples: 1724. Policy #0 lag: (min: 0.0, avg: 0.0, max: 0.0) [2023-06-21 12:37:26,480][62782] Avg episode reward: [(0, '2.728')] [2023-06-21 12:37:28,117][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1 [2023-06-21 12:37:28,983][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1 [2023-06-21 12:37:30,501][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1 [2023-06-21 12:37:31,476][62782] Fps is (10 sec: 640.0, 60 sec: 256.0, 300 sec: 256.0). Total num frames: 5010688. Throughput: 0: 87.5. Samples: 2188. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:37:31,477][62782] Avg episode reward: [(0, '5.719')] [2023-06-21 12:37:36,476][62782] Fps is (10 sec: 640.0, 60 sec: 320.0, 300 sec: 320.0). Total num frames: 5013888. Throughput: 0: 104.3. Samples: 3130. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:37:36,479][62782] Avg episode reward: [(0, '7.789')] [2023-06-21 12:37:41,475][62782] Fps is (10 sec: 640.1, 60 sec: 365.7, 300 sec: 365.7). Total num frames: 5017088. Throughput: 0: 116.5. Samples: 4078. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:37:41,476][62782] Avg episode reward: [(0, '9.517')] [2023-06-21 12:37:46,476][62782] Fps is (10 sec: 640.0, 60 sec: 400.0, 300 sec: 400.0). Total num frames: 5020288. Throughput: 0: 113.8. Samples: 4552. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:37:46,479][62782] Avg episode reward: [(0, '11.648')] [2023-06-21 12:37:48,128][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 2 [2023-06-21 12:37:49,010][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 2 [2023-06-21 12:37:50,526][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 2 [2023-06-21 12:37:51,475][62782] Fps is (10 sec: 640.0, 60 sec: 426.7, 300 sec: 426.7). Total num frames: 5023488. Throughput: 0: 122.3. Samples: 5502. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:37:51,476][62782] Avg episode reward: [(0, '13.169')] [2023-06-21 12:37:56,476][62782] Fps is (10 sec: 640.0, 60 sec: 448.0, 300 sec: 448.0). Total num frames: 5026688. Throughput: 0: 143.3. Samples: 6448. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:37:56,480][62782] Avg episode reward: [(0, '13.486')] [2023-06-21 12:38:01,476][62782] Fps is (10 sec: 639.9, 60 sec: 465.5, 300 sec: 465.5). Total num frames: 5029888. Throughput: 0: 153.6. Samples: 6912. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:38:01,479][62782] Avg episode reward: [(0, '14.005')] [2023-06-21 12:38:06,476][62782] Fps is (10 sec: 640.0, 60 sec: 480.0, 300 sec: 480.0). Total num frames: 5033088. Throughput: 0: 157.4. Samples: 7858. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:38:06,477][62782] Avg episode reward: [(0, '14.676')] [2023-06-21 12:38:08,153][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 3 [2023-06-21 12:38:09,021][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 3 [2023-06-21 12:38:09,477][78401] Updated weights for policy 0, policy_version 1300 (0.0014) [2023-06-21 12:38:10,572][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 3 [2023-06-21 12:38:11,475][62782] Fps is (10 sec: 640.1, 60 sec: 533.4, 300 sec: 492.3). Total num frames: 5036288. Throughput: 0: 146.8. Samples: 8330. Policy #0 lag: (min: 0.0, avg: 0.8, max: 1.0) [2023-06-21 12:38:11,477][62782] Avg episode reward: [(0, '18.926')] [2023-06-21 12:38:16,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 502.9). Total num frames: 5039488. Throughput: 0: 157.5. Samples: 9274. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2023-06-21 12:38:16,477][62782] Avg episode reward: [(0, '18.926')] [2023-06-21 12:38:21,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 512.0). Total num frames: 5042688. Throughput: 0: 157.6. Samples: 10220. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2023-06-21 12:38:21,478][62782] Avg episode reward: [(0, '19.751')] [2023-06-21 12:38:26,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 520.0). Total num frames: 5045888. Throughput: 0: 147.0. Samples: 10692. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:38:26,477][62782] Avg episode reward: [(0, '20.313')] [2023-06-21 12:38:28,207][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 4 [2023-06-21 12:38:29,073][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 4 [2023-06-21 12:38:30,602][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 4 [2023-06-21 12:38:31,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 527.1). Total num frames: 5049088. Throughput: 0: 157.5. Samples: 11640. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:38:31,477][62782] Avg episode reward: [(0, '20.337')] [2023-06-21 12:38:36,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 533.3). Total num frames: 5052288. Throughput: 0: 157.5. Samples: 12588. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:38:36,478][62782] Avg episode reward: [(0, '20.756')] [2023-06-21 12:38:41,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 539.0). Total num frames: 5055488. Throughput: 0: 157.6. Samples: 13538. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:38:41,476][62782] Avg episode reward: [(0, '21.784')] [2023-06-21 12:38:46,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 544.0). Total num frames: 5058688. Throughput: 0: 157.7. Samples: 14008. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 12:38:46,479][62782] Avg episode reward: [(0, '23.238')] [2023-06-21 12:38:48,262][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 5 [2023-06-21 12:38:49,125][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 5 [2023-06-21 12:38:50,639][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 5 [2023-06-21 12:38:51,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 548.6). Total num frames: 5061888. Throughput: 0: 157.6. Samples: 14948. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:38:51,477][62782] Avg episode reward: [(0, '23.395')] [2023-06-21 12:38:56,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 552.7). Total num frames: 5065088. Throughput: 0: 157.6. Samples: 15422. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:38:56,477][62782] Avg episode reward: [(0, '23.117')] [2023-06-21 12:39:00,794][78401] Updated weights for policy 0, policy_version 1310 (0.0014) [2023-06-21 12:39:01,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 556.5). Total num frames: 5068288. Throughput: 0: 157.7. Samples: 16370. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:39:01,477][62782] Avg episode reward: [(0, '22.797')] [2023-06-21 12:39:01,479][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001310_5068288.pth... [2023-06-21 12:39:01,585][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001251_4879488.pth [2023-06-21 12:39:06,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 560.0). Total num frames: 5071488. Throughput: 0: 157.4. Samples: 17304. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:39:06,478][62782] Avg episode reward: [(0, '23.202')] [2023-06-21 12:39:08,293][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 6 [2023-06-21 12:39:09,165][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 6 [2023-06-21 12:39:10,685][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 6 [2023-06-21 12:39:11,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 563.2). Total num frames: 5074688. Throughput: 0: 167.8. Samples: 18242. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:39:11,477][62782] Avg episode reward: [(0, '24.044')] [2023-06-21 12:39:16,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 566.2). Total num frames: 5077888. Throughput: 0: 157.4. Samples: 18722. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:39:16,477][62782] Avg episode reward: [(0, '24.201')] [2023-06-21 12:39:21,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 568.9). Total num frames: 5081088. Throughput: 0: 157.1. Samples: 19656. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:39:21,478][62782] Avg episode reward: [(0, '23.941')] [2023-06-21 12:39:26,476][62782] Fps is (10 sec: 320.0, 60 sec: 586.7, 300 sec: 548.6). Total num frames: 5081088. Throughput: 0: 156.9. Samples: 20600. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:39:26,480][62782] Avg episode reward: [(0, '23.998')] [2023-06-21 12:39:28,314][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 7 [2023-06-21 12:39:29,176][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 7 [2023-06-21 12:39:30,735][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 7 [2023-06-21 12:39:31,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 573.8). Total num frames: 5087488. Throughput: 0: 157.0. Samples: 21074. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:39:31,478][62782] Avg episode reward: [(0, '23.676')] [2023-06-21 12:39:36,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 554.7). Total num frames: 5087488. Throughput: 0: 157.2. Samples: 22022. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:39:36,481][62782] Avg episode reward: [(0, '24.139')] [2023-06-21 12:39:41,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 578.1). Total num frames: 5093888. Throughput: 0: 157.2. Samples: 22496. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:39:41,476][62782] Avg episode reward: [(0, '24.021')] [2023-06-21 12:39:46,477][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 560.0). Total num frames: 5093888. Throughput: 0: 156.9. Samples: 23432. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:39:46,481][62782] Avg episode reward: [(0, '24.189')] [2023-06-21 12:39:48,316][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 8 [2023-06-21 12:39:49,184][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 8 [2023-06-21 12:39:50,432][78401] Updated weights for policy 0, policy_version 1320 (0.0011) [2023-06-21 12:39:50,745][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 8 [2023-06-21 12:39:51,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 581.8). Total num frames: 5100288. Throughput: 0: 157.0. Samples: 24370. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:39:51,477][62782] Avg episode reward: [(0, '24.485')] [2023-06-21 12:39:56,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 564.7). Total num frames: 5100288. Throughput: 0: 146.8. Samples: 24846. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:39:56,480][62782] Avg episode reward: [(0, '24.802')] [2023-06-21 12:40:01,477][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 585.1). Total num frames: 5106688. Throughput: 0: 157.2. Samples: 25794. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:40:01,480][62782] Avg episode reward: [(0, '24.419')] [2023-06-21 12:40:06,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 568.9). Total num frames: 5106688. Throughput: 0: 157.2. Samples: 26732. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:40:06,479][62782] Avg episode reward: [(0, '24.041')] [2023-06-21 12:40:08,323][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 9 [2023-06-21 12:40:09,187][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 9 [2023-06-21 12:40:10,758][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 9 [2023-06-21 12:40:11,476][62782] Fps is (10 sec: 320.0, 60 sec: 586.7, 300 sec: 570.8). Total num frames: 5109888. Throughput: 0: 146.7. Samples: 27200. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 12:40:11,477][62782] Avg episode reward: [(0, '24.381')] [2023-06-21 12:40:16,476][62782] Fps is (10 sec: 960.0, 60 sec: 640.0, 300 sec: 589.5). Total num frames: 5116288. Throughput: 0: 157.3. Samples: 28152. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:40:16,477][62782] Avg episode reward: [(0, '24.272')] [2023-06-21 12:40:21,475][62782] Fps is (10 sec: 640.1, 60 sec: 586.7, 300 sec: 574.4). Total num frames: 5116288. Throughput: 0: 157.3. Samples: 29102. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:40:21,476][62782] Avg episode reward: [(0, '24.112')] [2023-06-21 12:40:26,475][62782] Fps is (10 sec: 640.1, 60 sec: 693.3, 300 sec: 592.0). Total num frames: 5122688. Throughput: 0: 168.0. Samples: 30056. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:40:26,476][62782] Avg episode reward: [(0, '24.837')] [2023-06-21 12:40:28,343][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 10 [2023-06-21 12:40:29,213][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 10 [2023-06-21 12:40:30,807][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 10 [2023-06-21 12:40:31,476][62782] Fps is (10 sec: 639.9, 60 sec: 586.7, 300 sec: 577.6). Total num frames: 5122688. Throughput: 0: 157.8. Samples: 30532. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:40:31,478][62782] Avg episode reward: [(0, '25.034')] [2023-06-21 12:40:36,476][62782] Fps is (10 sec: 640.0, 60 sec: 693.3, 300 sec: 594.3). Total num frames: 5129088. Throughput: 0: 158.0. Samples: 31480. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:40:36,478][62782] Avg episode reward: [(0, '25.346')] [2023-06-21 12:40:41,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 580.5). Total num frames: 5129088. Throughput: 0: 168.4. Samples: 32424. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:40:41,477][62782] Avg episode reward: [(0, '25.854')] [2023-06-21 12:40:41,776][78401] Updated weights for policy 0, policy_version 1330 (0.0012) [2023-06-21 12:40:46,476][62782] Fps is (10 sec: 320.0, 60 sec: 640.0, 300 sec: 581.8). Total num frames: 5132288. Throughput: 0: 158.0. Samples: 32902. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:40:46,478][62782] Avg episode reward: [(0, '26.413')] [2023-06-21 12:40:48,384][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 11 [2023-06-21 12:40:49,225][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 11 [2023-06-21 12:40:50,850][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 11 [2023-06-21 12:40:51,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 583.1). Total num frames: 5135488. Throughput: 0: 158.0. Samples: 33842. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:40:51,476][62782] Avg episode reward: [(0, '27.100')] [2023-06-21 12:40:56,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 584.4). Total num frames: 5138688. Throughput: 0: 168.7. Samples: 34790. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:40:56,476][62782] Avg episode reward: [(0, '27.013')] [2023-06-21 12:41:01,476][62782] Fps is (10 sec: 959.9, 60 sec: 640.0, 300 sec: 599.1). Total num frames: 5145088. Throughput: 0: 158.1. Samples: 35266. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:41:01,479][62782] Avg episode reward: [(0, '27.052')] [2023-06-21 12:41:01,484][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001334_5145088.pth... [2023-06-21 12:41:01,595][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001290_5004288.pth [2023-06-21 12:41:06,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 586.7). Total num frames: 5145088. Throughput: 0: 157.9. Samples: 36206. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:41:06,475][62782] Avg episode reward: [(0, '26.490')] [2023-06-21 12:41:08,392][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 12 [2023-06-21 12:41:09,259][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 12 [2023-06-21 12:41:10,865][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 12 [2023-06-21 12:41:11,475][62782] Fps is (10 sec: 320.0, 60 sec: 640.0, 300 sec: 587.8). Total num frames: 5148288. Throughput: 0: 157.8. Samples: 37158. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:41:11,476][62782] Avg episode reward: [(0, '26.002')] [2023-06-21 12:41:16,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 588.8). Total num frames: 5151488. Throughput: 0: 157.7. Samples: 37630. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:41:16,477][62782] Avg episode reward: [(0, '25.482')] [2023-06-21 12:41:21,476][62782] Fps is (10 sec: 959.9, 60 sec: 693.3, 300 sec: 602.4). Total num frames: 5157888. Throughput: 0: 157.8. Samples: 38580. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:41:21,477][62782] Avg episode reward: [(0, '25.698')] [2023-06-21 12:41:26,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 590.8). Total num frames: 5157888. Throughput: 0: 147.4. Samples: 39058. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:41:26,477][62782] Avg episode reward: [(0, '25.360')] [2023-06-21 12:41:28,404][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 13 [2023-06-21 12:41:29,264][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 13 [2023-06-21 12:41:30,884][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 13 [2023-06-21 12:41:31,476][62782] Fps is (10 sec: 320.0, 60 sec: 640.0, 300 sec: 591.7). Total num frames: 5161088. Throughput: 0: 158.0. Samples: 40010. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 12:41:31,477][62782] Avg episode reward: [(0, '25.325')] [2023-06-21 12:41:33,386][78401] Updated weights for policy 0, policy_version 1340 (0.0007) [2023-06-21 12:41:36,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 592.6). Total num frames: 5164288. Throughput: 0: 158.0. Samples: 40950. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:41:36,477][62782] Avg episode reward: [(0, '25.009')] [2023-06-21 12:41:41,477][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 593.5). Total num frames: 5167488. Throughput: 0: 158.1. Samples: 41904. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:41:41,478][62782] Avg episode reward: [(0, '25.559')] [2023-06-21 12:41:46,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 594.3). Total num frames: 5170688. Throughput: 0: 158.0. Samples: 42374. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:41:46,477][62782] Avg episode reward: [(0, '25.439')] [2023-06-21 12:41:48,406][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 14 [2023-06-21 12:41:49,297][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 14 [2023-06-21 12:41:50,908][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 14 [2023-06-21 12:41:51,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 595.1). Total num frames: 5173888. Throughput: 0: 158.1. Samples: 43322. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:41:51,477][62782] Avg episode reward: [(0, '25.694')] [2023-06-21 12:41:56,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 595.9). Total num frames: 5177088. Throughput: 0: 147.6. Samples: 43802. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:41:56,478][62782] Avg episode reward: [(0, '26.092')] [2023-06-21 12:42:01,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 596.6). Total num frames: 5180288. Throughput: 0: 158.2. Samples: 44748. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:42:01,479][62782] Avg episode reward: [(0, '26.107')] [2023-06-21 12:42:06,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 607.5). Total num frames: 5183488. Throughput: 0: 158.2. Samples: 45700. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:42:06,478][62782] Avg episode reward: [(0, '25.923')] [2023-06-21 12:42:08,421][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 15 [2023-06-21 12:42:09,339][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 15 [2023-06-21 12:42:10,963][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 15 [2023-06-21 12:42:11,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 618.3). Total num frames: 5186688. Throughput: 0: 158.1. Samples: 46174. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:42:11,477][62782] Avg episode reward: [(0, '25.999')] [2023-06-21 12:42:16,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5189888. Throughput: 0: 158.0. Samples: 47122. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:42:16,477][62782] Avg episode reward: [(0, '26.061')] [2023-06-21 12:42:21,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5193088. Throughput: 0: 158.0. Samples: 48062. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:42:21,478][62782] Avg episode reward: [(0, '26.733')] [2023-06-21 12:42:22,379][78401] Updated weights for policy 0, policy_version 1350 (0.0010) [2023-06-21 12:42:26,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5196288. Throughput: 0: 157.9. Samples: 49010. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:42:26,475][62782] Avg episode reward: [(0, '27.223')] [2023-06-21 12:42:28,437][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 16 [2023-06-21 12:42:29,354][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 16 [2023-06-21 12:42:31,005][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 16 [2023-06-21 12:42:31,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5199488. Throughput: 0: 158.2. Samples: 49492. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:42:31,476][62782] Avg episode reward: [(0, '27.055')] [2023-06-21 12:42:36,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5202688. Throughput: 0: 158.2. Samples: 50440. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:42:36,477][62782] Avg episode reward: [(0, '27.045')] [2023-06-21 12:42:41,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5205888. Throughput: 0: 158.1. Samples: 50916. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:42:41,478][62782] Avg episode reward: [(0, '26.841')] [2023-06-21 12:42:46,477][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.1). Total num frames: 5209088. Throughput: 0: 158.0. Samples: 51860. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:42:46,478][62782] Avg episode reward: [(0, '26.430')] [2023-06-21 12:42:48,456][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 17 [2023-06-21 12:42:49,389][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 17 [2023-06-21 12:42:51,010][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 17 [2023-06-21 12:42:51,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5212288. Throughput: 0: 158.0. Samples: 52812. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 12:42:51,476][62782] Avg episode reward: [(0, '26.557')] [2023-06-21 12:42:56,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5215488. Throughput: 0: 168.7. Samples: 53764. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:42:56,477][62782] Avg episode reward: [(0, '26.557')] [2023-06-21 12:43:01,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5218688. Throughput: 0: 158.1. Samples: 54238. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:43:01,479][62782] Avg episode reward: [(0, '26.460')] [2023-06-21 12:43:01,483][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001357_5218688.pth... [2023-06-21 12:43:01,583][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001310_5068288.pth [2023-06-21 12:43:06,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5221888. Throughput: 0: 158.3. Samples: 55186. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:43:06,477][62782] Avg episode reward: [(0, '27.388')] [2023-06-21 12:43:08,493][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 18 [2023-06-21 12:43:09,421][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 18 [2023-06-21 12:43:11,049][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 18 [2023-06-21 12:43:11,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5225088. Throughput: 0: 158.2. Samples: 56130. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:43:11,476][62782] Avg episode reward: [(0, '27.604')] [2023-06-21 12:43:13,869][78401] Updated weights for policy 0, policy_version 1360 (0.0011) [2023-06-21 12:43:16,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5228288. Throughput: 0: 158.0. Samples: 56604. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:43:16,477][62782] Avg episode reward: [(0, '27.589')] [2023-06-21 12:43:21,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5231488. Throughput: 0: 158.1. Samples: 57556. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:43:21,477][62782] Avg episode reward: [(0, '27.461')] [2023-06-21 12:43:26,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5234688. Throughput: 0: 158.0. Samples: 58026. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:43:26,477][62782] Avg episode reward: [(0, '27.835')] [2023-06-21 12:43:28,508][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 19 [2023-06-21 12:43:29,446][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 19 [2023-06-21 12:43:31,070][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 19 [2023-06-21 12:43:31,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5237888. Throughput: 0: 158.2. Samples: 58978. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:43:31,476][62782] Avg episode reward: [(0, '28.170')] [2023-06-21 12:43:36,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5241088. Throughput: 0: 158.0. Samples: 59920. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:43:36,479][62782] Avg episode reward: [(0, '27.846')] [2023-06-21 12:43:41,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5244288. Throughput: 0: 147.4. Samples: 60398. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:43:41,478][62782] Avg episode reward: [(0, '27.690')] [2023-06-21 12:43:46,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5247488. Throughput: 0: 157.9. Samples: 61344. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:43:46,478][62782] Avg episode reward: [(0, '27.635')] [2023-06-21 12:43:48,560][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 20 [2023-06-21 12:43:49,475][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 20 [2023-06-21 12:43:51,091][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 20 [2023-06-21 12:43:51,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5250688. Throughput: 0: 158.0. Samples: 62298. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:43:51,476][62782] Avg episode reward: [(0, '27.487')] [2023-06-21 12:43:56,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.1). Total num frames: 5253888. Throughput: 0: 158.1. Samples: 63244. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:43:56,478][62782] Avg episode reward: [(0, '27.093')] [2023-06-21 12:44:01,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5257088. Throughput: 0: 158.1. Samples: 63718. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:44:01,478][62782] Avg episode reward: [(0, '27.445')] [2023-06-21 12:44:03,024][78401] Updated weights for policy 0, policy_version 1370 (0.0008) [2023-06-21 12:44:06,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5260288. Throughput: 0: 158.0. Samples: 64668. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:44:06,478][62782] Avg episode reward: [(0, '27.409')] [2023-06-21 12:44:08,604][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 21 [2023-06-21 12:44:09,511][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 21 [2023-06-21 12:44:11,106][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 21 [2023-06-21 12:44:11,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5263488. Throughput: 0: 168.8. Samples: 65620. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2023-06-21 12:44:11,476][62782] Avg episode reward: [(0, '27.382')] [2023-06-21 12:44:16,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5266688. Throughput: 0: 158.4. Samples: 66104. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:44:16,477][62782] Avg episode reward: [(0, '27.382')] [2023-06-21 12:44:21,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5269888. Throughput: 0: 158.4. Samples: 67050. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:44:21,478][62782] Avg episode reward: [(0, '27.954')] [2023-06-21 12:44:26,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5273088. Throughput: 0: 158.4. Samples: 67526. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:44:26,477][62782] Avg episode reward: [(0, '27.847')] [2023-06-21 12:44:28,630][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 22 [2023-06-21 12:44:29,527][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 22 [2023-06-21 12:44:31,125][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 22 [2023-06-21 12:44:31,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5276288. Throughput: 0: 158.3. Samples: 68466. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:44:31,477][62782] Avg episode reward: [(0, '28.799')] [2023-06-21 12:44:31,480][78400] Saving new best policy, reward=28.799! [2023-06-21 12:44:36,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5279488. Throughput: 0: 158.3. Samples: 69420. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:44:36,478][62782] Avg episode reward: [(0, '29.095')] [2023-06-21 12:44:36,479][78400] Saving new best policy, reward=29.095! [2023-06-21 12:44:41,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5282688. Throughput: 0: 158.4. Samples: 70370. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:44:41,478][62782] Avg episode reward: [(0, '28.570')] [2023-06-21 12:44:46,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5285888. Throughput: 0: 158.3. Samples: 70840. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:44:46,476][62782] Avg episode reward: [(0, '28.025')] [2023-06-21 12:44:48,632][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 23 [2023-06-21 12:44:49,572][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 23 [2023-06-21 12:44:51,142][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 23 [2023-06-21 12:44:51,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5289088. Throughput: 0: 158.4. Samples: 71794. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:44:51,476][62782] Avg episode reward: [(0, '28.117')] [2023-06-21 12:44:54,684][78401] Updated weights for policy 0, policy_version 1380 (0.0009) [2023-06-21 12:44:56,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5292288. Throughput: 0: 147.6. Samples: 72264. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:44:56,476][62782] Avg episode reward: [(0, '28.091')] [2023-06-21 12:45:01,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5295488. Throughput: 0: 158.0. Samples: 73214. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:45:01,481][62782] Avg episode reward: [(0, '27.789')] [2023-06-21 12:45:01,485][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001381_5295488.pth... [2023-06-21 12:45:01,576][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001334_5145088.pth [2023-06-21 12:45:06,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5298688. Throughput: 0: 157.9. Samples: 74154. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:45:06,477][62782] Avg episode reward: [(0, '27.136')] [2023-06-21 12:45:08,650][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 24 [2023-06-21 12:45:09,604][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 24 [2023-06-21 12:45:11,185][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 24 [2023-06-21 12:45:11,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5301888. Throughput: 0: 157.8. Samples: 74626. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:45:11,476][62782] Avg episode reward: [(0, '26.848')] [2023-06-21 12:45:16,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5305088. Throughput: 0: 155.4. Samples: 75460. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:45:16,478][62782] Avg episode reward: [(0, '26.848')] [2023-06-21 12:45:21,475][62782] Fps is (10 sec: 320.0, 60 sec: 586.7, 300 sec: 618.3). Total num frames: 5305088. Throughput: 0: 152.8. Samples: 76296. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:45:21,476][62782] Avg episode reward: [(0, '26.184')] [2023-06-21 12:45:26,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5311488. Throughput: 0: 150.2. Samples: 77128. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 12:45:26,479][62782] Avg episode reward: [(0, '26.184')] [2023-06-21 12:45:28,666][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 25 [2023-06-21 12:45:29,663][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 25 [2023-06-21 12:45:31,216][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 25 [2023-06-21 12:45:31,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 618.3). Total num frames: 5311488. Throughput: 0: 149.0. Samples: 77544. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 12:45:31,476][62782] Avg episode reward: [(0, '26.046')] [2023-06-21 12:45:36,475][62782] Fps is (10 sec: 320.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5314688. Throughput: 0: 146.3. Samples: 78376. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 12:45:36,477][62782] Avg episode reward: [(0, '25.696')] [2023-06-21 12:45:41,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5317888. Throughput: 0: 144.8. Samples: 78782. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:45:41,478][62782] Avg episode reward: [(0, '26.818')] [2023-06-21 12:45:46,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5321088. Throughput: 0: 142.5. Samples: 79628. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:45:46,476][62782] Avg episode reward: [(0, '26.430')] [2023-06-21 12:45:48,670][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 26 [2023-06-21 12:45:49,719][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 26 [2023-06-21 12:45:51,058][78401] Updated weights for policy 0, policy_version 1390 (0.0010) [2023-06-21 12:45:51,249][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 26 [2023-06-21 12:45:51,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5324288. Throughput: 0: 140.2. Samples: 80462. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:45:51,477][62782] Avg episode reward: [(0, '26.045')] [2023-06-21 12:45:56,477][62782] Fps is (10 sec: 639.9, 60 sec: 586.7, 300 sec: 618.3). Total num frames: 5327488. Throughput: 0: 148.0. Samples: 81288. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:45:56,478][62782] Avg episode reward: [(0, '26.002')] [2023-06-21 12:46:01,475][62782] Fps is (10 sec: 320.0, 60 sec: 533.3, 300 sec: 618.3). Total num frames: 5327488. Throughput: 0: 138.8. Samples: 81708. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:46:01,478][62782] Avg episode reward: [(0, '25.581')] [2023-06-21 12:46:06,475][62782] Fps is (10 sec: 320.0, 60 sec: 533.3, 300 sec: 618.3). Total num frames: 5330688. Throughput: 0: 138.8. Samples: 82542. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:46:06,476][62782] Avg episode reward: [(0, '24.860')] [2023-06-21 12:46:08,728][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 27 [2023-06-21 12:46:09,739][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 27 [2023-06-21 12:46:11,258][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 27 [2023-06-21 12:46:11,476][62782] Fps is (10 sec: 640.0, 60 sec: 533.3, 300 sec: 618.3). Total num frames: 5333888. Throughput: 0: 129.3. Samples: 82948. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:46:11,477][62782] Avg episode reward: [(0, '24.716')] [2023-06-21 12:46:16,475][62782] Fps is (10 sec: 640.0, 60 sec: 533.3, 300 sec: 607.5). Total num frames: 5337088. Throughput: 0: 138.6. Samples: 83782. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:46:16,477][62782] Avg episode reward: [(0, '25.003')] [2023-06-21 12:46:21,479][62782] Fps is (10 sec: 639.8, 60 sec: 586.6, 300 sec: 618.3). Total num frames: 5340288. Throughput: 0: 138.7. Samples: 84620. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 12:46:21,480][62782] Avg episode reward: [(0, '25.281')] [2023-06-21 12:46:26,475][62782] Fps is (10 sec: 640.0, 60 sec: 533.3, 300 sec: 618.3). Total num frames: 5343488. Throughput: 0: 138.8. Samples: 85028. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:46:26,477][62782] Avg episode reward: [(0, '24.906')] [2023-06-21 12:46:28,757][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 28 [2023-06-21 12:46:29,753][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 28 [2023-06-21 12:46:31,292][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 28 [2023-06-21 12:46:31,477][62782] Fps is (10 sec: 320.1, 60 sec: 533.3, 300 sec: 607.5). Total num frames: 5343488. Throughput: 0: 138.7. Samples: 85868. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:46:31,478][62782] Avg episode reward: [(0, '25.172')] [2023-06-21 12:46:36,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 618.3). Total num frames: 5349888. Throughput: 0: 138.7. Samples: 86704. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:46:36,478][62782] Avg episode reward: [(0, '25.493')] [2023-06-21 12:46:41,478][62782] Fps is (10 sec: 640.0, 60 sec: 533.3, 300 sec: 607.5). Total num frames: 5349888. Throughput: 0: 138.8. Samples: 87536. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:46:41,478][62782] Avg episode reward: [(0, '25.657')] [2023-06-21 12:46:46,475][62782] Fps is (10 sec: 320.0, 60 sec: 533.3, 300 sec: 607.5). Total num frames: 5353088. Throughput: 0: 138.7. Samples: 87950. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:46:46,477][62782] Avg episode reward: [(0, '25.714')] [2023-06-21 12:46:47,311][78401] Updated weights for policy 0, policy_version 1400 (0.0010) [2023-06-21 12:46:48,800][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 29 [2023-06-21 12:46:49,795][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 29 [2023-06-21 12:46:51,340][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 29 [2023-06-21 12:46:51,475][62782] Fps is (10 sec: 640.2, 60 sec: 533.3, 300 sec: 607.5). Total num frames: 5356288. Throughput: 0: 138.7. Samples: 88782. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:46:51,475][62782] Avg episode reward: [(0, '25.859')] [2023-06-21 12:46:56,479][62782] Fps is (10 sec: 639.8, 60 sec: 533.3, 300 sec: 607.5). Total num frames: 5359488. Throughput: 0: 148.1. Samples: 89612. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:46:56,480][62782] Avg episode reward: [(0, '25.671')] [2023-06-21 12:47:01,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 607.5). Total num frames: 5362688. Throughput: 0: 138.8. Samples: 90026. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:47:01,478][62782] Avg episode reward: [(0, '26.369')] [2023-06-21 12:47:01,481][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001402_5362688.pth... [2023-06-21 12:47:01,480][62782] Components not started: RolloutWorker_w2, RolloutWorker_w3, RolloutWorker_w6, RolloutWorker_w7, wait_time=600.0 seconds [2023-06-21 12:47:01,585][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001357_5218688.pth [2023-06-21 12:47:06,475][62782] Fps is (10 sec: 640.2, 60 sec: 586.7, 300 sec: 607.5). Total num frames: 5365888. Throughput: 0: 138.5. Samples: 90854. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 12:47:06,476][62782] Avg episode reward: [(0, '26.369')] [2023-06-21 12:47:08,855][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 30 [2023-06-21 12:47:09,814][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 30 [2023-06-21 12:47:11,347][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 30 [2023-06-21 12:47:11,475][62782] Fps is (10 sec: 320.0, 60 sec: 533.3, 300 sec: 596.6). Total num frames: 5365888. Throughput: 0: 138.7. Samples: 91268. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 12:47:11,476][62782] Avg episode reward: [(0, '25.953')] [2023-06-21 12:47:16,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 607.5). Total num frames: 5372288. Throughput: 0: 138.4. Samples: 92096. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:47:16,476][62782] Avg episode reward: [(0, '25.780')] [2023-06-21 12:47:21,475][62782] Fps is (10 sec: 640.0, 60 sec: 533.4, 300 sec: 596.6). Total num frames: 5372288. Throughput: 0: 138.3. Samples: 92926. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:47:21,476][62782] Avg episode reward: [(0, '25.945')] [2023-06-21 12:47:26,476][62782] Fps is (10 sec: 320.0, 60 sec: 533.3, 300 sec: 596.6). Total num frames: 5375488. Throughput: 0: 128.9. Samples: 93336. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:47:26,478][62782] Avg episode reward: [(0, '26.414')] [2023-06-21 12:47:28,913][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 31 [2023-06-21 12:47:29,848][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 31 [2023-06-21 12:47:31,382][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 31 [2023-06-21 12:47:31,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 596.6). Total num frames: 5378688. Throughput: 0: 138.2. Samples: 94170. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:47:31,476][62782] Avg episode reward: [(0, '26.211')] [2023-06-21 12:47:36,475][62782] Fps is (10 sec: 640.0, 60 sec: 533.3, 300 sec: 596.6). Total num frames: 5381888. Throughput: 0: 138.1. Samples: 94998. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:47:36,478][62782] Avg episode reward: [(0, '26.627')] [2023-06-21 12:47:41,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 596.6). Total num frames: 5385088. Throughput: 0: 128.9. Samples: 95414. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:47:41,476][62782] Avg episode reward: [(0, '26.832')] [2023-06-21 12:47:45,853][78401] Updated weights for policy 0, policy_version 1410 (0.0008) [2023-06-21 12:47:46,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 596.6). Total num frames: 5388288. Throughput: 0: 138.3. Samples: 96250. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:47:46,476][62782] Avg episode reward: [(0, '26.593')] [2023-06-21 12:47:48,914][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 32 [2023-06-21 12:47:49,860][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 32 [2023-06-21 12:47:51,383][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 32 [2023-06-21 12:47:51,476][62782] Fps is (10 sec: 320.0, 60 sec: 533.3, 300 sec: 585.8). Total num frames: 5388288. Throughput: 0: 138.4. Samples: 97084. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:47:51,476][62782] Avg episode reward: [(0, '26.121')] [2023-06-21 12:47:56,475][62782] Fps is (10 sec: 320.0, 60 sec: 533.4, 300 sec: 585.8). Total num frames: 5391488. Throughput: 0: 147.6. Samples: 97910. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 12:47:56,477][62782] Avg episode reward: [(0, '26.353')] [2023-06-21 12:48:01,476][62782] Fps is (10 sec: 640.0, 60 sec: 533.3, 300 sec: 585.8). Total num frames: 5394688. Throughput: 0: 138.4. Samples: 98326. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:48:01,477][62782] Avg episode reward: [(0, '26.669')] [2023-06-21 12:48:06,475][62782] Fps is (10 sec: 640.0, 60 sec: 533.3, 300 sec: 585.8). Total num frames: 5397888. Throughput: 0: 138.4. Samples: 99156. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:48:06,477][62782] Avg episode reward: [(0, '26.541')] [2023-06-21 12:48:08,961][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 33 [2023-06-21 12:48:09,894][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 33 [2023-06-21 12:48:11,395][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 33 [2023-06-21 12:48:11,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 585.8). Total num frames: 5401088. Throughput: 0: 138.7. Samples: 99578. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:48:11,476][62782] Avg episode reward: [(0, '26.745')] [2023-06-21 12:48:16,475][62782] Fps is (10 sec: 640.0, 60 sec: 533.3, 300 sec: 585.8). Total num frames: 5404288. Throughput: 0: 138.4. Samples: 100400. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:48:16,476][62782] Avg episode reward: [(0, '26.753')] [2023-06-21 12:48:21,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 585.8). Total num frames: 5407488. Throughput: 0: 138.4. Samples: 101228. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:48:21,476][62782] Avg episode reward: [(0, '27.183')] [2023-06-21 12:48:26,476][62782] Fps is (10 sec: 320.0, 60 sec: 533.3, 300 sec: 574.9). Total num frames: 5407488. Throughput: 0: 147.6. Samples: 102056. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:48:26,478][62782] Avg episode reward: [(0, '27.144')] [2023-06-21 12:48:29,011][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 34 [2023-06-21 12:48:29,937][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 34 [2023-06-21 12:48:31,421][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 34 [2023-06-21 12:48:31,477][62782] Fps is (10 sec: 639.9, 60 sec: 586.7, 300 sec: 585.8). Total num frames: 5413888. Throughput: 0: 138.4. Samples: 102480. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:48:31,477][62782] Avg episode reward: [(0, '27.355')] [2023-06-21 12:48:36,476][62782] Fps is (10 sec: 640.0, 60 sec: 533.3, 300 sec: 574.9). Total num frames: 5413888. Throughput: 0: 138.3. Samples: 103308. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:48:36,477][62782] Avg episode reward: [(0, '27.036')] [2023-06-21 12:48:41,477][62782] Fps is (10 sec: 320.0, 60 sec: 533.3, 300 sec: 574.9). Total num frames: 5417088. Throughput: 0: 138.5. Samples: 104142. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2023-06-21 12:48:41,478][62782] Avg episode reward: [(0, '27.168')] [2023-06-21 12:48:44,917][78401] Updated weights for policy 0, policy_version 1420 (0.0009) [2023-06-21 12:48:46,475][62782] Fps is (10 sec: 640.1, 60 sec: 533.3, 300 sec: 574.9). Total num frames: 5420288. Throughput: 0: 138.5. Samples: 104558. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 12:48:46,476][62782] Avg episode reward: [(0, '27.215')] [2023-06-21 12:48:49,051][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 35 [2023-06-21 12:48:49,962][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 35 [2023-06-21 12:48:51,475][62782] Fps is (10 sec: 640.1, 60 sec: 586.7, 300 sec: 574.9). Total num frames: 5423488. Throughput: 0: 138.7. Samples: 105396. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:48:51,475][62782] Avg episode reward: [(0, '27.032')] [2023-06-21 12:48:51,476][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 35 [2023-06-21 12:48:56,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 574.9). Total num frames: 5426688. Throughput: 0: 147.6. Samples: 106222. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:48:56,476][62782] Avg episode reward: [(0, '27.212')] [2023-06-21 12:49:01,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 574.9). Total num frames: 5429888. Throughput: 0: 138.6. Samples: 106636. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:49:01,477][62782] Avg episode reward: [(0, '27.159')] [2023-06-21 12:49:01,480][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001423_5429888.pth... [2023-06-21 12:49:01,578][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001381_5295488.pth [2023-06-21 12:49:06,477][62782] Fps is (10 sec: 320.0, 60 sec: 533.3, 300 sec: 564.1). Total num frames: 5429888. Throughput: 0: 139.7. Samples: 107514. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:49:06,480][62782] Avg episode reward: [(0, '26.945')] [2023-06-21 12:49:09,102][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 36 [2023-06-21 12:49:10,004][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 36 [2023-06-21 12:49:11,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 574.9). Total num frames: 5436288. Throughput: 0: 131.9. Samples: 107990. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:49:11,477][62782] Avg episode reward: [(0, '26.986')] [2023-06-21 12:49:11,526][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 36 [2023-06-21 12:49:16,476][62782] Fps is (10 sec: 640.0, 60 sec: 533.3, 300 sec: 564.1). Total num frames: 5436288. Throughput: 0: 143.5. Samples: 108938. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:49:16,477][62782] Avg episode reward: [(0, '26.941')] [2023-06-21 12:49:21,476][62782] Fps is (10 sec: 320.0, 60 sec: 533.3, 300 sec: 564.1). Total num frames: 5439488. Throughput: 0: 146.0. Samples: 109880. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:49:21,476][62782] Avg episode reward: [(0, '27.347')] [2023-06-21 12:49:26,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 564.1). Total num frames: 5442688. Throughput: 0: 138.1. Samples: 110358. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:49:26,480][62782] Avg episode reward: [(0, '27.501')] [2023-06-21 12:49:29,138][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 37 [2023-06-21 12:49:30,059][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 37 [2023-06-21 12:49:31,476][62782] Fps is (10 sec: 640.0, 60 sec: 533.3, 300 sec: 564.1). Total num frames: 5445888. Throughput: 0: 149.9. Samples: 111302. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:49:31,477][62782] Avg episode reward: [(0, '27.665')] [2023-06-21 12:49:31,528][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 37 [2023-06-21 12:49:36,240][78401] Updated weights for policy 0, policy_version 1430 (0.0009) [2023-06-21 12:49:36,476][62782] Fps is (10 sec: 960.1, 60 sec: 640.0, 300 sec: 574.9). Total num frames: 5452288. Throughput: 0: 152.4. Samples: 112254. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:49:36,476][62782] Avg episode reward: [(0, '27.810')] [2023-06-21 12:49:41,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 564.1). Total num frames: 5452288. Throughput: 0: 155.0. Samples: 113198. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:49:41,476][62782] Avg episode reward: [(0, '28.561')] [2023-06-21 12:49:46,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 574.9). Total num frames: 5458688. Throughput: 0: 156.3. Samples: 113670. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:49:46,476][62782] Avg episode reward: [(0, '28.031')] [2023-06-21 12:49:49,164][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 38 [2023-06-21 12:49:50,096][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 38 [2023-06-21 12:49:51,476][62782] Fps is (10 sec: 639.9, 60 sec: 586.7, 300 sec: 564.1). Total num frames: 5458688. Throughput: 0: 157.8. Samples: 114614. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:49:51,477][62782] Avg episode reward: [(0, '27.714')] [2023-06-21 12:49:51,543][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 38 [2023-06-21 12:49:56,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 574.9). Total num frames: 5465088. Throughput: 0: 168.2. Samples: 115560. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:49:56,476][62782] Avg episode reward: [(0, '27.248')] [2023-06-21 12:50:01,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 564.1). Total num frames: 5465088. Throughput: 0: 157.8. Samples: 116038. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:50:01,477][62782] Avg episode reward: [(0, '26.836')] [2023-06-21 12:50:06,476][62782] Fps is (10 sec: 320.0, 60 sec: 640.0, 300 sec: 564.1). Total num frames: 5468288. Throughput: 0: 158.0. Samples: 116988. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 12:50:06,477][62782] Avg episode reward: [(0, '27.393')] [2023-06-21 12:50:09,175][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 39 [2023-06-21 12:50:10,100][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 39 [2023-06-21 12:50:11,475][62782] Fps is (10 sec: 640.1, 60 sec: 586.7, 300 sec: 564.1). Total num frames: 5471488. Throughput: 0: 157.9. Samples: 117464. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:50:11,476][62782] Avg episode reward: [(0, '27.713')] [2023-06-21 12:50:11,574][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 39 [2023-06-21 12:50:16,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 574.9). Total num frames: 5474688. Throughput: 0: 158.1. Samples: 118416. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:50:16,477][62782] Avg episode reward: [(0, '27.417')] [2023-06-21 12:50:21,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 564.1). Total num frames: 5477888. Throughput: 0: 157.9. Samples: 119360. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:50:21,477][62782] Avg episode reward: [(0, '27.526')] [2023-06-21 12:50:26,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 574.9). Total num frames: 5481088. Throughput: 0: 158.2. Samples: 120318. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:50:26,477][62782] Avg episode reward: [(0, '27.122')] [2023-06-21 12:50:28,008][78401] Updated weights for policy 0, policy_version 1440 (0.0006) [2023-06-21 12:50:29,216][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 40 [2023-06-21 12:50:30,105][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 40 [2023-06-21 12:50:31,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 574.9). Total num frames: 5484288. Throughput: 0: 158.3. Samples: 120792. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:50:31,477][62782] Avg episode reward: [(0, '27.016')] [2023-06-21 12:50:31,617][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 40 [2023-06-21 12:50:36,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 574.9). Total num frames: 5487488. Throughput: 0: 158.4. Samples: 121742. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:50:36,477][62782] Avg episode reward: [(0, '26.797')] [2023-06-21 12:50:41,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 574.9). Total num frames: 5490688. Throughput: 0: 147.9. Samples: 122216. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:50:41,477][62782] Avg episode reward: [(0, '26.309')] [2023-06-21 12:50:46,477][62782] Fps is (10 sec: 639.9, 60 sec: 586.7, 300 sec: 574.9). Total num frames: 5493888. Throughput: 0: 158.5. Samples: 123170. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:50:46,500][62782] Avg episode reward: [(0, '26.309')] [2023-06-21 12:50:49,258][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 41 [2023-06-21 12:50:50,122][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 41 [2023-06-21 12:50:51,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 574.9). Total num frames: 5497088. Throughput: 0: 158.5. Samples: 124122. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:50:51,476][62782] Avg episode reward: [(0, '25.875')] [2023-06-21 12:50:51,642][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 41 [2023-06-21 12:50:56,475][62782] Fps is (10 sec: 640.1, 60 sec: 586.7, 300 sec: 585.8). Total num frames: 5500288. Throughput: 0: 158.4. Samples: 124592. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:50:56,477][62782] Avg episode reward: [(0, '26.697')] [2023-06-21 12:51:01,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 585.8). Total num frames: 5503488. Throughput: 0: 158.5. Samples: 125548. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:51:01,476][62782] Avg episode reward: [(0, '26.818')] [2023-06-21 12:51:01,479][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001446_5503488.pth... [2023-06-21 12:51:01,572][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001402_5362688.pth [2023-06-21 12:51:06,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 585.8). Total num frames: 5506688. Throughput: 0: 158.7. Samples: 126502. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:51:06,477][62782] Avg episode reward: [(0, '26.818')] [2023-06-21 12:51:09,268][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 42 [2023-06-21 12:51:10,121][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 42 [2023-06-21 12:51:11,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 585.8). Total num frames: 5509888. Throughput: 0: 158.3. Samples: 127442. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:51:11,476][62782] Avg episode reward: [(0, '27.367')] [2023-06-21 12:51:11,638][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 42 [2023-06-21 12:51:16,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 585.8). Total num frames: 5513088. Throughput: 0: 158.3. Samples: 127916. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:51:16,476][62782] Avg episode reward: [(0, '27.356')] [2023-06-21 12:51:16,840][78401] Updated weights for policy 0, policy_version 1450 (0.0006) [2023-06-21 12:51:21,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 585.8). Total num frames: 5516288. Throughput: 0: 158.2. Samples: 128860. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:51:21,477][62782] Avg episode reward: [(0, '28.330')] [2023-06-21 12:51:26,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 596.6). Total num frames: 5519488. Throughput: 0: 158.1. Samples: 129332. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 12:51:26,476][62782] Avg episode reward: [(0, '28.709')] [2023-06-21 12:51:29,265][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 43 [2023-06-21 12:51:30,134][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 43 [2023-06-21 12:51:31,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 585.8). Total num frames: 5522688. Throughput: 0: 157.8. Samples: 130272. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:51:31,476][62782] Avg episode reward: [(0, '28.333')] [2023-06-21 12:51:31,646][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 43 [2023-06-21 12:51:36,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 596.6). Total num frames: 5525888. Throughput: 0: 157.6. Samples: 131216. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:51:36,476][62782] Avg episode reward: [(0, '27.860')] [2023-06-21 12:51:41,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 596.6). Total num frames: 5529088. Throughput: 0: 157.7. Samples: 131690. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:51:41,476][62782] Avg episode reward: [(0, '28.141')] [2023-06-21 12:51:46,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 596.6). Total num frames: 5532288. Throughput: 0: 157.4. Samples: 132632. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:51:46,477][62782] Avg episode reward: [(0, '27.756')] [2023-06-21 12:51:49,265][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 44 [2023-06-21 12:51:50,169][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 44 [2023-06-21 12:51:51,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 596.6). Total num frames: 5535488. Throughput: 0: 157.3. Samples: 133582. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:51:51,476][62782] Avg episode reward: [(0, '27.756')] [2023-06-21 12:51:51,670][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 44 [2023-06-21 12:51:56,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 596.6). Total num frames: 5538688. Throughput: 0: 147.2. Samples: 134064. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:51:56,477][62782] Avg episode reward: [(0, '27.704')] [2023-06-21 12:52:01,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 596.6). Total num frames: 5541888. Throughput: 0: 157.6. Samples: 135006. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:52:01,476][62782] Avg episode reward: [(0, '27.805')] [2023-06-21 12:52:06,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 607.5). Total num frames: 5545088. Throughput: 0: 157.6. Samples: 135954. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:52:06,477][62782] Avg episode reward: [(0, '27.816')] [2023-06-21 12:52:08,618][78401] Updated weights for policy 0, policy_version 1460 (0.0006) [2023-06-21 12:52:09,318][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 45 [2023-06-21 12:52:10,174][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 45 [2023-06-21 12:52:11,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 596.6). Total num frames: 5548288. Throughput: 0: 157.6. Samples: 136424. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:52:11,476][62782] Avg episode reward: [(0, '27.929')] [2023-06-21 12:52:11,700][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 45 [2023-06-21 12:52:16,477][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 607.5). Total num frames: 5551488. Throughput: 0: 157.7. Samples: 137368. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:52:16,477][62782] Avg episode reward: [(0, '27.683')] [2023-06-21 12:52:21,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 607.5). Total num frames: 5554688. Throughput: 0: 157.9. Samples: 138320. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:52:21,477][62782] Avg episode reward: [(0, '27.635')] [2023-06-21 12:52:26,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 607.5). Total num frames: 5557888. Throughput: 0: 168.4. Samples: 139268. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:52:26,476][62782] Avg episode reward: [(0, '27.766')] [2023-06-21 12:52:29,342][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 46 [2023-06-21 12:52:30,195][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 46 [2023-06-21 12:52:31,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 607.5). Total num frames: 5561088. Throughput: 0: 158.0. Samples: 139742. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:52:31,477][62782] Avg episode reward: [(0, '27.766')] [2023-06-21 12:52:31,725][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 46 [2023-06-21 12:52:36,477][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 607.5). Total num frames: 5564288. Throughput: 0: 157.9. Samples: 140686. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:52:36,479][62782] Avg episode reward: [(0, '27.655')] [2023-06-21 12:52:41,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 607.5). Total num frames: 5567488. Throughput: 0: 168.2. Samples: 141634. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:52:41,476][62782] Avg episode reward: [(0, '28.014')] [2023-06-21 12:52:46,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 618.3). Total num frames: 5570688. Throughput: 0: 157.8. Samples: 142108. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2023-06-21 12:52:46,480][62782] Avg episode reward: [(0, '27.537')] [2023-06-21 12:52:49,384][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 47 [2023-06-21 12:52:50,215][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 47 [2023-06-21 12:52:51,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 618.3). Total num frames: 5573888. Throughput: 0: 157.6. Samples: 143048. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:52:51,478][62782] Avg episode reward: [(0, '27.852')] [2023-06-21 12:52:51,758][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 47 [2023-06-21 12:52:56,477][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 618.3). Total num frames: 5577088. Throughput: 0: 157.7. Samples: 143522. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:52:56,479][62782] Avg episode reward: [(0, '28.074')] [2023-06-21 12:52:59,900][78401] Updated weights for policy 0, policy_version 1470 (0.0015) [2023-06-21 12:53:01,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 618.3). Total num frames: 5580288. Throughput: 0: 157.9. Samples: 144472. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2023-06-21 12:53:01,477][62782] Avg episode reward: [(0, '28.277')] [2023-06-21 12:53:01,481][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001470_5580288.pth... [2023-06-21 12:53:01,566][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001423_5429888.pth [2023-06-21 12:53:06,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 618.3). Total num frames: 5583488. Throughput: 0: 157.7. Samples: 145418. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:53:06,479][62782] Avg episode reward: [(0, '28.453')] [2023-06-21 12:53:09,419][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 48 [2023-06-21 12:53:10,236][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 48 [2023-06-21 12:53:11,477][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 618.3). Total num frames: 5586688. Throughput: 0: 157.9. Samples: 146374. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:53:11,480][62782] Avg episode reward: [(0, '28.476')] [2023-06-21 12:53:11,797][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 48 [2023-06-21 12:53:16,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 618.3). Total num frames: 5589888. Throughput: 0: 157.9. Samples: 146846. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:53:16,479][62782] Avg episode reward: [(0, '28.425')] [2023-06-21 12:53:21,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5593088. Throughput: 0: 157.9. Samples: 147792. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:53:21,476][62782] Avg episode reward: [(0, '27.990')] [2023-06-21 12:53:26,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 618.3). Total num frames: 5596288. Throughput: 0: 147.6. Samples: 148276. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:53:26,479][62782] Avg episode reward: [(0, '28.550')] [2023-06-21 12:53:29,457][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 49 [2023-06-21 12:53:30,258][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 49 [2023-06-21 12:53:31,478][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5599488. Throughput: 0: 158.1. Samples: 149224. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:53:31,481][62782] Avg episode reward: [(0, '28.550')] [2023-06-21 12:53:31,832][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 49 [2023-06-21 12:53:36,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5602688. Throughput: 0: 158.3. Samples: 150170. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:53:36,479][62782] Avg episode reward: [(0, '28.980')] [2023-06-21 12:53:41,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.1). Total num frames: 5605888. Throughput: 0: 168.8. Samples: 151120. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:53:41,477][62782] Avg episode reward: [(0, '29.611')] [2023-06-21 12:53:41,479][78400] Saving new best policy, reward=29.611! [2023-06-21 12:53:46,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.1). Total num frames: 5609088. Throughput: 0: 158.2. Samples: 151592. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:53:46,478][62782] Avg episode reward: [(0, '29.607')] [2023-06-21 12:53:49,375][78401] Updated weights for policy 0, policy_version 1480 (0.0019) [2023-06-21 12:53:49,492][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 50 [2023-06-21 12:53:50,278][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 50 [2023-06-21 12:53:51,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5612288. Throughput: 0: 158.4. Samples: 152546. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:53:51,476][62782] Avg episode reward: [(0, '29.812')] [2023-06-21 12:53:51,479][78400] Saving new best policy, reward=29.812! [2023-06-21 12:53:51,851][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 50 [2023-06-21 12:53:56,477][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5615488. Throughput: 0: 158.3. Samples: 153498. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:53:56,479][62782] Avg episode reward: [(0, '28.902')] [2023-06-21 12:54:01,477][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5618688. Throughput: 0: 158.3. Samples: 153970. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:54:01,480][62782] Avg episode reward: [(0, '27.135')] [2023-06-21 12:54:06,479][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5621888. Throughput: 0: 158.3. Samples: 154914. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 12:54:06,485][62782] Avg episode reward: [(0, '27.135')] [2023-06-21 12:54:09,506][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 51 [2023-06-21 12:54:10,304][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 51 [2023-06-21 12:54:11,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5625088. Throughput: 0: 158.0. Samples: 155388. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:54:11,477][62782] Avg episode reward: [(0, '26.957')] [2023-06-21 12:54:11,871][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 51 [2023-06-21 12:54:16,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5628288. Throughput: 0: 158.1. Samples: 156340. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:54:16,480][62782] Avg episode reward: [(0, '26.607')] [2023-06-21 12:54:21,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5631488. Throughput: 0: 158.1. Samples: 157286. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:54:21,478][62782] Avg episode reward: [(0, '26.769')] [2023-06-21 12:54:26,476][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5634688. Throughput: 0: 147.6. Samples: 157764. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:54:26,478][62782] Avg episode reward: [(0, '26.562')] [2023-06-21 12:54:29,522][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 52 [2023-06-21 12:54:30,329][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 52 [2023-06-21 12:54:31,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5637888. Throughput: 0: 158.3. Samples: 158714. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:54:31,477][62782] Avg episode reward: [(0, '26.237')] [2023-06-21 12:54:31,880][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 52 [2023-06-21 12:54:36,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5641088. Throughput: 0: 158.3. Samples: 159670. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:54:36,479][62782] Avg episode reward: [(0, '26.643')] [2023-06-21 12:54:41,029][78401] Updated weights for policy 0, policy_version 1490 (0.0018) [2023-06-21 12:54:41,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.1). Total num frames: 5644288. Throughput: 0: 147.7. Samples: 160146. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:54:41,477][62782] Avg episode reward: [(0, '26.633')] [2023-06-21 12:54:46,477][62782] Fps is (10 sec: 319.9, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5644288. Throughput: 0: 157.9. Samples: 161076. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:54:46,481][62782] Avg episode reward: [(0, '26.815')] [2023-06-21 12:54:49,547][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 53 [2023-06-21 12:54:50,360][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 53 [2023-06-21 12:54:51,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5650688. Throughput: 0: 158.1. Samples: 162030. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:54:51,477][62782] Avg episode reward: [(0, '28.429')] [2023-06-21 12:54:51,928][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 53 [2023-06-21 12:54:56,475][62782] Fps is (10 sec: 960.2, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5653888. Throughput: 0: 168.7. Samples: 162978. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:54:56,477][62782] Avg episode reward: [(0, '28.777')] [2023-06-21 12:55:01,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5657088. Throughput: 0: 158.0. Samples: 163448. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:55:01,477][62782] Avg episode reward: [(0, '28.916')] [2023-06-21 12:55:01,479][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001494_5657088.pth... [2023-06-21 12:55:01,565][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001446_5503488.pth [2023-06-21 12:55:06,476][62782] Fps is (10 sec: 320.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5657088. Throughput: 0: 158.2. Samples: 164404. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:55:06,478][62782] Avg episode reward: [(0, '28.848')] [2023-06-21 12:55:09,589][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 54 [2023-06-21 12:55:10,370][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 54 [2023-06-21 12:55:11,477][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5663488. Throughput: 0: 168.4. Samples: 165342. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:55:11,477][62782] Avg episode reward: [(0, '28.855')] [2023-06-21 12:55:11,937][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 54 [2023-06-21 12:55:16,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5663488. Throughput: 0: 157.9. Samples: 165820. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:55:16,477][62782] Avg episode reward: [(0, '29.015')] [2023-06-21 12:55:21,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5669888. Throughput: 0: 157.6. Samples: 166762. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:55:21,477][62782] Avg episode reward: [(0, '28.965')] [2023-06-21 12:55:26,476][62782] Fps is (10 sec: 960.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5673088. Throughput: 0: 168.0. Samples: 167706. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 12:55:26,478][62782] Avg episode reward: [(0, '28.968')] [2023-06-21 12:55:29,620][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 55 [2023-06-21 12:55:30,379][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 55 [2023-06-21 12:55:31,477][62782] Fps is (10 sec: 320.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5673088. Throughput: 0: 158.0. Samples: 168184. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 12:55:31,479][62782] Avg episode reward: [(0, '28.347')] [2023-06-21 12:55:31,936][78401] Updated weights for policy 0, policy_version 1500 (0.0009) [2023-06-21 12:55:31,959][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 55 [2023-06-21 12:55:36,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5679488. Throughput: 0: 157.7. Samples: 169128. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:55:36,476][62782] Avg episode reward: [(0, '28.307')] [2023-06-21 12:55:41,477][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5679488. Throughput: 0: 157.5. Samples: 170066. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:55:41,480][62782] Avg episode reward: [(0, '28.692')] [2023-06-21 12:55:46,475][62782] Fps is (10 sec: 640.0, 60 sec: 693.4, 300 sec: 640.0). Total num frames: 5685888. Throughput: 0: 157.6. Samples: 170542. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:55:46,476][62782] Avg episode reward: [(0, '28.718')] [2023-06-21 12:55:49,630][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 56 [2023-06-21 12:55:50,395][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 56 [2023-06-21 12:55:51,476][62782] Fps is (10 sec: 640.1, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5685888. Throughput: 0: 157.5. Samples: 171492. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:55:51,477][62782] Avg episode reward: [(0, '28.457')] [2023-06-21 12:55:51,981][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 56 [2023-06-21 12:55:56,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5692288. Throughput: 0: 157.7. Samples: 172438. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:55:56,476][62782] Avg episode reward: [(0, '28.268')] [2023-06-21 12:56:01,475][62782] Fps is (10 sec: 640.1, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5692288. Throughput: 0: 157.6. Samples: 172912. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:56:01,476][62782] Avg episode reward: [(0, '27.994')] [2023-06-21 12:56:06,476][62782] Fps is (10 sec: 320.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5695488. Throughput: 0: 157.8. Samples: 173864. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:56:06,477][62782] Avg episode reward: [(0, '27.585')] [2023-06-21 12:56:09,652][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 57 [2023-06-21 12:56:10,419][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 57 [2023-06-21 12:56:11,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5698688. Throughput: 0: 157.9. Samples: 174810. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:56:11,477][62782] Avg episode reward: [(0, '27.585')] [2023-06-21 12:56:11,991][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 57 [2023-06-21 12:56:16,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5701888. Throughput: 0: 157.9. Samples: 175290. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:56:16,476][62782] Avg episode reward: [(0, '27.441')] [2023-06-21 12:56:21,476][62782] Fps is (10 sec: 960.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5708288. Throughput: 0: 158.1. Samples: 176244. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:56:21,479][62782] Avg episode reward: [(0, '27.207')] [2023-06-21 12:56:21,481][78401] Updated weights for policy 0, policy_version 1510 (0.0008) [2023-06-21 12:56:26,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5708288. Throughput: 0: 147.8. Samples: 176716. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:56:26,477][62782] Avg episode reward: [(0, '27.207')] [2023-06-21 12:56:29,676][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 58 [2023-06-21 12:56:30,450][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 58 [2023-06-21 12:56:31,477][62782] Fps is (10 sec: 640.0, 60 sec: 693.3, 300 sec: 640.0). Total num frames: 5714688. Throughput: 0: 158.4. Samples: 177672. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:56:31,478][62782] Avg episode reward: [(0, '26.626')] [2023-06-21 12:56:32,018][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 58 [2023-06-21 12:56:36,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5714688. Throughput: 0: 158.4. Samples: 178622. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:56:36,476][62782] Avg episode reward: [(0, '27.098')] [2023-06-21 12:56:41,476][62782] Fps is (10 sec: 640.0, 60 sec: 693.3, 300 sec: 640.0). Total num frames: 5721088. Throughput: 0: 148.0. Samples: 179096. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:56:41,478][62782] Avg episode reward: [(0, '27.371')] [2023-06-21 12:56:46,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5721088. Throughput: 0: 158.5. Samples: 180046. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:56:46,477][62782] Avg episode reward: [(0, '26.830')] [2023-06-21 12:56:49,695][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 59 [2023-06-21 12:56:50,452][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 59 [2023-06-21 12:56:51,476][62782] Fps is (10 sec: 320.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5724288. Throughput: 0: 158.4. Samples: 180994. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 12:56:51,476][62782] Avg episode reward: [(0, '27.335')] [2023-06-21 12:56:52,073][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 59 [2023-06-21 12:56:56,478][62782] Fps is (10 sec: 639.9, 60 sec: 586.7, 300 sec: 629.1). Total num frames: 5727488. Throughput: 0: 158.4. Samples: 181940. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:56:56,480][62782] Avg episode reward: [(0, '27.422')] [2023-06-21 12:57:01,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5730688. Throughput: 0: 158.4. Samples: 182418. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:57:01,478][62782] Avg episode reward: [(0, '27.683')] [2023-06-21 12:57:01,481][62782] Components not started: RolloutWorker_w2, RolloutWorker_w3, RolloutWorker_w6, RolloutWorker_w7, wait_time=1200.0 seconds [2023-06-21 12:57:02,711][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001518_5733888.pth... [2023-06-21 12:57:02,806][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001470_5580288.pth [2023-06-21 12:57:06,478][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.1). Total num frames: 5733888. Throughput: 0: 158.3. Samples: 183368. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:57:06,485][62782] Avg episode reward: [(0, '27.425')] [2023-06-21 12:57:09,711][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 60 [2023-06-21 12:57:10,473][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 60 [2023-06-21 12:57:11,477][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5737088. Throughput: 0: 168.9. Samples: 184318. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:57:11,477][62782] Avg episode reward: [(0, '27.688')] [2023-06-21 12:57:12,111][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 60 [2023-06-21 12:57:13,017][78401] Updated weights for policy 0, policy_version 1520 (0.0009) [2023-06-21 12:57:16,480][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5740288. Throughput: 0: 158.2. Samples: 184790. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:57:16,483][62782] Avg episode reward: [(0, '27.520')] [2023-06-21 12:57:21,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.1). Total num frames: 5743488. Throughput: 0: 158.2. Samples: 185740. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:57:21,478][62782] Avg episode reward: [(0, '27.541')] [2023-06-21 12:57:26,476][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5746688. Throughput: 0: 168.7. Samples: 186686. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:57:26,476][62782] Avg episode reward: [(0, '27.042')] [2023-06-21 12:57:29,764][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 61 [2023-06-21 12:57:30,501][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 61 [2023-06-21 12:57:31,477][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5749888. Throughput: 0: 158.2. Samples: 187164. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:57:31,477][62782] Avg episode reward: [(0, '27.047')] [2023-06-21 12:57:32,118][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 61 [2023-06-21 12:57:36,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5753088. Throughput: 0: 157.8. Samples: 188096. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:57:36,477][62782] Avg episode reward: [(0, '26.459')] [2023-06-21 12:57:41,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5756288. Throughput: 0: 147.4. Samples: 188574. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:57:41,478][62782] Avg episode reward: [(0, '26.390')] [2023-06-21 12:57:46,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5759488. Throughput: 0: 157.8. Samples: 189518. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:57:46,477][62782] Avg episode reward: [(0, '25.930')] [2023-06-21 12:57:49,801][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 62 [2023-06-21 12:57:50,554][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 62 [2023-06-21 12:57:51,477][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5762688. Throughput: 0: 157.8. Samples: 190470. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:57:51,478][62782] Avg episode reward: [(0, '25.807')] [2023-06-21 12:57:52,138][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 62 [2023-06-21 12:57:56,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5765888. Throughput: 0: 157.8. Samples: 191418. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:57:56,478][62782] Avg episode reward: [(0, '26.056')] [2023-06-21 12:58:01,477][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5769088. Throughput: 0: 157.7. Samples: 191886. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:58:01,479][62782] Avg episode reward: [(0, '26.078')] [2023-06-21 12:58:02,257][78401] Updated weights for policy 0, policy_version 1530 (0.0010) [2023-06-21 12:58:06,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5772288. Throughput: 0: 157.7. Samples: 192836. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:58:06,477][62782] Avg episode reward: [(0, '25.927')] [2023-06-21 12:58:09,819][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 63 [2023-06-21 12:58:10,577][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 63 [2023-06-21 12:58:11,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5775488. Throughput: 0: 147.2. Samples: 193310. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 12:58:11,477][62782] Avg episode reward: [(0, '25.920')] [2023-06-21 12:58:12,146][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 63 [2023-06-21 12:58:16,476][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5778688. Throughput: 0: 157.6. Samples: 194254. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:58:16,478][62782] Avg episode reward: [(0, '25.255')] [2023-06-21 12:58:21,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5781888. Throughput: 0: 157.8. Samples: 195198. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:58:21,476][62782] Avg episode reward: [(0, '25.762')] [2023-06-21 12:58:26,477][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5785088. Throughput: 0: 168.2. Samples: 196142. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:58:26,478][62782] Avg episode reward: [(0, '26.033')] [2023-06-21 12:58:29,839][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 64 [2023-06-21 12:58:30,619][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 64 [2023-06-21 12:58:31,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5788288. Throughput: 0: 157.8. Samples: 196618. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:58:31,477][62782] Avg episode reward: [(0, '25.542')] [2023-06-21 12:58:32,167][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 64 [2023-06-21 12:58:36,476][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5791488. Throughput: 0: 157.5. Samples: 197558. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:58:36,477][62782] Avg episode reward: [(0, '25.116')] [2023-06-21 12:58:41,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5794688. Throughput: 0: 146.9. Samples: 198030. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:58:41,477][62782] Avg episode reward: [(0, '24.627')] [2023-06-21 12:58:46,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5797888. Throughput: 0: 157.7. Samples: 198984. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 12:58:46,479][62782] Avg episode reward: [(0, '25.377')] [2023-06-21 12:58:49,840][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 65 [2023-06-21 12:58:50,673][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 65 [2023-06-21 12:58:51,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5801088. Throughput: 0: 157.7. Samples: 199932. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:58:51,475][62782] Avg episode reward: [(0, '25.419')] [2023-06-21 12:58:52,198][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 65 [2023-06-21 12:58:53,776][78401] Updated weights for policy 0, policy_version 1540 (0.0008) [2023-06-21 12:58:56,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5804288. Throughput: 0: 168.3. Samples: 200882. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:58:56,477][62782] Avg episode reward: [(0, '25.636')] [2023-06-21 12:59:01,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5807488. Throughput: 0: 157.6. Samples: 201346. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:59:01,479][62782] Avg episode reward: [(0, '25.590')] [2023-06-21 12:59:01,482][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001541_5807488.pth... [2023-06-21 12:59:01,570][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001494_5657088.pth [2023-06-21 12:59:06,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5810688. Throughput: 0: 157.6. Samples: 202292. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:59:06,477][62782] Avg episode reward: [(0, '25.707')] [2023-06-21 12:59:09,859][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 66 [2023-06-21 12:59:10,708][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 66 [2023-06-21 12:59:11,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5813888. Throughput: 0: 147.2. Samples: 202766. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:59:11,476][62782] Avg episode reward: [(0, '25.889')] [2023-06-21 12:59:12,229][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 66 [2023-06-21 12:59:16,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5817088. Throughput: 0: 157.8. Samples: 203718. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:59:16,476][62782] Avg episode reward: [(0, '26.079')] [2023-06-21 12:59:21,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5820288. Throughput: 0: 157.9. Samples: 204662. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:59:21,478][62782] Avg episode reward: [(0, '25.984')] [2023-06-21 12:59:26,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.1). Total num frames: 5823488. Throughput: 0: 157.9. Samples: 205136. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:59:26,478][62782] Avg episode reward: [(0, '25.629')] [2023-06-21 12:59:29,897][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 67 [2023-06-21 12:59:30,746][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 67 [2023-06-21 12:59:31,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5826688. Throughput: 0: 157.9. Samples: 206088. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 12:59:31,477][62782] Avg episode reward: [(0, '25.541')] [2023-06-21 12:59:32,243][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 67 [2023-06-21 12:59:36,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5829888. Throughput: 0: 157.8. Samples: 207034. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 12:59:36,477][62782] Avg episode reward: [(0, '25.584')] [2023-06-21 12:59:41,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5833088. Throughput: 0: 147.2. Samples: 207508. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:59:41,476][62782] Avg episode reward: [(0, '24.770')] [2023-06-21 12:59:45,051][78401] Updated weights for policy 0, policy_version 1550 (0.0009) [2023-06-21 12:59:46,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5836288. Throughput: 0: 157.9. Samples: 208450. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 12:59:46,476][62782] Avg episode reward: [(0, '24.821')] [2023-06-21 12:59:49,905][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 68 [2023-06-21 12:59:50,766][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 68 [2023-06-21 12:59:51,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5839488. Throughput: 0: 157.9. Samples: 209398. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 12:59:51,477][62782] Avg episode reward: [(0, '24.821')] [2023-06-21 12:59:52,268][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 68 [2023-06-21 12:59:56,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5842688. Throughput: 0: 168.5. Samples: 210348. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 12:59:56,478][62782] Avg episode reward: [(0, '24.987')] [2023-06-21 13:00:01,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5845888. Throughput: 0: 158.0. Samples: 210828. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:00:01,478][62782] Avg episode reward: [(0, '25.243')] [2023-06-21 13:00:06,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5849088. Throughput: 0: 157.9. Samples: 211768. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 13:00:06,477][62782] Avg episode reward: [(0, '25.008')] [2023-06-21 13:00:09,932][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 69 [2023-06-21 13:00:10,793][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 69 [2023-06-21 13:00:11,477][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5852288. Throughput: 0: 158.0. Samples: 212246. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 13:00:11,478][62782] Avg episode reward: [(0, '25.712')] [2023-06-21 13:00:12,292][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 69 [2023-06-21 13:00:16,477][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5855488. Throughput: 0: 157.9. Samples: 213194. Policy #0 lag: (min: 0.0, avg: 0.8, max: 1.0) [2023-06-21 13:00:16,481][62782] Avg episode reward: [(0, '25.593')] [2023-06-21 13:00:21,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5858688. Throughput: 0: 157.8. Samples: 214134. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 13:00:21,479][62782] Avg episode reward: [(0, '26.344')] [2023-06-21 13:00:26,477][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5861888. Throughput: 0: 168.4. Samples: 215086. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:00:26,479][62782] Avg episode reward: [(0, '26.867')] [2023-06-21 13:00:29,940][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 70 [2023-06-21 13:00:30,842][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 70 [2023-06-21 13:00:31,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5865088. Throughput: 0: 158.0. Samples: 215560. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 13:00:31,477][62782] Avg episode reward: [(0, '26.842')] [2023-06-21 13:00:32,318][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 70 [2023-06-21 13:00:34,630][78401] Updated weights for policy 0, policy_version 1560 (0.0011) [2023-06-21 13:00:36,476][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5868288. Throughput: 0: 158.0. Samples: 216506. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:00:36,476][62782] Avg episode reward: [(0, '27.022')] [2023-06-21 13:00:41,477][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.1). Total num frames: 5871488. Throughput: 0: 147.4. Samples: 216982. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 13:00:41,480][62782] Avg episode reward: [(0, '25.851')] [2023-06-21 13:00:46,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5874688. Throughput: 0: 157.7. Samples: 217924. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:00:46,477][62782] Avg episode reward: [(0, '25.231')] [2023-06-21 13:00:49,949][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 71 [2023-06-21 13:00:50,868][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 71 [2023-06-21 13:00:51,477][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5877888. Throughput: 0: 157.9. Samples: 218872. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 13:00:51,477][62782] Avg episode reward: [(0, '25.812')] [2023-06-21 13:00:52,330][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 71 [2023-06-21 13:00:56,477][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5881088. Throughput: 0: 168.4. Samples: 219826. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:00:56,478][62782] Avg episode reward: [(0, '25.688')] [2023-06-21 13:01:01,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5884288. Throughput: 0: 158.0. Samples: 220302. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:01:01,476][62782] Avg episode reward: [(0, '25.334')] [2023-06-21 13:01:01,480][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001565_5884288.pth... [2023-06-21 13:01:01,560][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001518_5733888.pth [2023-06-21 13:01:06,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5887488. Throughput: 0: 158.2. Samples: 221252. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 13:01:06,477][62782] Avg episode reward: [(0, '25.852')] [2023-06-21 13:01:09,953][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 72 [2023-06-21 13:01:10,915][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 72 [2023-06-21 13:01:11,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5890688. Throughput: 0: 158.0. Samples: 222194. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:01:11,476][62782] Avg episode reward: [(0, '26.438')] [2023-06-21 13:01:12,360][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 72 [2023-06-21 13:01:16,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5893888. Throughput: 0: 157.9. Samples: 222666. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 13:01:16,476][62782] Avg episode reward: [(0, '26.041')] [2023-06-21 13:01:21,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5897088. Throughput: 0: 157.9. Samples: 223610. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:01:21,476][62782] Avg episode reward: [(0, '25.341')] [2023-06-21 13:01:25,679][78401] Updated weights for policy 0, policy_version 1570 (0.0010) [2023-06-21 13:01:26,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5900288. Throughput: 0: 158.0. Samples: 224090. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 13:01:26,476][62782] Avg episode reward: [(0, '26.146')] [2023-06-21 13:01:30,001][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 73 [2023-06-21 13:01:30,942][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 73 [2023-06-21 13:01:31,476][62782] Fps is (10 sec: 320.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5900288. Throughput: 0: 158.2. Samples: 225042. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2023-06-21 13:01:31,476][62782] Avg episode reward: [(0, '26.061')] [2023-06-21 13:01:32,411][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 73 [2023-06-21 13:01:36,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5906688. Throughput: 0: 158.0. Samples: 225984. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 13:01:36,476][62782] Avg episode reward: [(0, '26.136')] [2023-06-21 13:01:41,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5906688. Throughput: 0: 158.0. Samples: 226936. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 13:01:41,477][62782] Avg episode reward: [(0, '25.603')] [2023-06-21 13:01:46,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5913088. Throughput: 0: 157.9. Samples: 227406. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:01:46,477][62782] Avg episode reward: [(0, '25.002')] [2023-06-21 13:01:50,047][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 74 [2023-06-21 13:01:50,965][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 74 [2023-06-21 13:01:51,476][62782] Fps is (10 sec: 959.9, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5916288. Throughput: 0: 157.9. Samples: 228358. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 13:01:51,477][62782] Avg episode reward: [(0, '24.535')] [2023-06-21 13:01:52,457][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 74 [2023-06-21 13:01:56,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5919488. Throughput: 0: 158.1. Samples: 229310. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:01:56,476][62782] Avg episode reward: [(0, '24.535')] [2023-06-21 13:02:01,476][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5922688. Throughput: 0: 158.0. Samples: 229778. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 13:02:01,478][62782] Avg episode reward: [(0, '24.632')] [2023-06-21 13:02:06,477][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5925888. Throughput: 0: 158.1. Samples: 230724. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:02:06,478][62782] Avg episode reward: [(0, '24.180')] [2023-06-21 13:02:10,069][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 75 [2023-06-21 13:02:10,999][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 75 [2023-06-21 13:02:11,475][62782] Fps is (10 sec: 320.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5925888. Throughput: 0: 168.5. Samples: 231674. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:02:11,475][62782] Avg episode reward: [(0, '24.287')] [2023-06-21 13:02:12,479][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 75 [2023-06-21 13:02:16,476][62782] Fps is (10 sec: 320.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5929088. Throughput: 0: 157.8. Samples: 232144. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2023-06-21 13:02:16,476][62782] Avg episode reward: [(0, '24.127')] [2023-06-21 13:02:17,330][78401] Updated weights for policy 0, policy_version 1580 (0.0007) [2023-06-21 13:02:21,476][62782] Fps is (10 sec: 959.9, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 5935488. Throughput: 0: 158.0. Samples: 233096. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:02:21,477][62782] Avg episode reward: [(0, '24.074')] [2023-06-21 13:02:26,475][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5935488. Throughput: 0: 157.9. Samples: 234042. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:02:26,477][62782] Avg episode reward: [(0, '24.408')] [2023-06-21 13:02:30,091][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 76 [2023-06-21 13:02:31,013][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 76 [2023-06-21 13:02:31,475][62782] Fps is (10 sec: 320.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5938688. Throughput: 0: 158.0. Samples: 234518. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 13:02:31,476][62782] Avg episode reward: [(0, '24.048')] [2023-06-21 13:02:32,515][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 76 [2023-06-21 13:02:36,476][62782] Fps is (10 sec: 639.9, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5941888. Throughput: 0: 158.0. Samples: 235468. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:02:36,478][62782] Avg episode reward: [(0, '24.652')] [2023-06-21 13:02:41,476][62782] Fps is (10 sec: 959.9, 60 sec: 693.3, 300 sec: 640.0). Total num frames: 5948288. Throughput: 0: 157.9. Samples: 236414. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:02:41,477][62782] Avg episode reward: [(0, '24.385')] [2023-06-21 13:02:46,475][62782] Fps is (10 sec: 640.1, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5948288. Throughput: 0: 158.2. Samples: 236896. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:02:46,476][62782] Avg episode reward: [(0, '24.552')] [2023-06-21 13:02:50,124][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 77 [2023-06-21 13:02:51,049][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 77 [2023-06-21 13:02:51,476][62782] Fps is (10 sec: 320.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5951488. Throughput: 0: 158.2. Samples: 237842. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 13:02:51,476][62782] Avg episode reward: [(0, '24.989')] [2023-06-21 13:02:52,558][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 77 [2023-06-21 13:02:56,476][62782] Fps is (10 sec: 639.9, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5954688. Throughput: 0: 147.6. Samples: 238316. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 13:02:56,478][62782] Avg episode reward: [(0, '24.310')] [2023-06-21 13:03:01,476][62782] Fps is (10 sec: 639.9, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5957888. Throughput: 0: 158.1. Samples: 239260. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 13:03:01,477][62782] Avg episode reward: [(0, '23.860')] [2023-06-21 13:03:02,708][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001589_5961088.pth... [2023-06-21 13:03:02,833][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001541_5807488.pth [2023-06-21 13:03:06,476][62782] Fps is (10 sec: 640.0, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5961088. Throughput: 0: 158.0. Samples: 240208. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 13:03:06,477][62782] Avg episode reward: [(0, '23.319')] [2023-06-21 13:03:06,600][78401] Updated weights for policy 0, policy_version 1590 (0.0006) [2023-06-21 13:03:10,142][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 78 [2023-06-21 13:03:11,070][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 78 [2023-06-21 13:03:11,477][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.1). Total num frames: 5964288. Throughput: 0: 158.1. Samples: 241156. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 13:03:11,478][62782] Avg episode reward: [(0, '23.378')] [2023-06-21 13:03:12,579][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 78 [2023-06-21 13:03:16,477][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5967488. Throughput: 0: 158.1. Samples: 241632. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 13:03:16,477][62782] Avg episode reward: [(0, '23.001')] [2023-06-21 13:03:21,476][62782] Fps is (10 sec: 640.1, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5970688. Throughput: 0: 158.0. Samples: 242576. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:03:21,477][62782] Avg episode reward: [(0, '23.758')] [2023-06-21 13:03:26,477][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.1). Total num frames: 5973888. Throughput: 0: 158.0. Samples: 243522. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 13:03:26,478][62782] Avg episode reward: [(0, '23.978')] [2023-06-21 13:03:30,174][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 79 [2023-06-21 13:03:31,114][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 79 [2023-06-21 13:03:31,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5977088. Throughput: 0: 157.8. Samples: 243996. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:03:31,477][62782] Avg episode reward: [(0, '24.245')] [2023-06-21 13:03:32,613][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 79 [2023-06-21 13:03:36,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5980288. Throughput: 0: 158.0. Samples: 244950. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 13:03:36,476][62782] Avg episode reward: [(0, '23.710')] [2023-06-21 13:03:41,475][62782] Fps is (10 sec: 640.1, 60 sec: 586.7, 300 sec: 629.2). Total num frames: 5983488. Throughput: 0: 168.4. Samples: 245892. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:03:41,476][62782] Avg episode reward: [(0, '23.893')] [2023-06-21 13:03:46,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5986688. Throughput: 0: 158.0. Samples: 246372. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:03:46,476][62782] Avg episode reward: [(0, '23.893')] [2023-06-21 13:03:50,176][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 80 [2023-06-21 13:03:51,149][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 80 [2023-06-21 13:03:51,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5989888. Throughput: 0: 158.0. Samples: 247320. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 13:03:51,477][62782] Avg episode reward: [(0, '23.483')] [2023-06-21 13:03:52,637][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 80 [2023-06-21 13:03:56,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5993088. Throughput: 0: 158.1. Samples: 248268. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:03:56,476][62782] Avg episode reward: [(0, '23.647')] [2023-06-21 13:03:57,978][78401] Updated weights for policy 0, policy_version 1600 (0.0006) [2023-06-21 13:04:01,477][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5996288. Throughput: 0: 158.0. Samples: 248742. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 13:04:01,478][62782] Avg episode reward: [(0, '23.833')] [2023-06-21 13:04:06,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 5999488. Throughput: 0: 158.0. Samples: 249686. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:04:06,477][62782] Avg episode reward: [(0, '24.169')] [2023-06-21 13:04:10,197][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 81 [2023-06-21 13:04:11,161][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 81 [2023-06-21 13:04:11,476][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6002688. Throughput: 0: 158.0. Samples: 250632. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 13:04:11,476][62782] Avg episode reward: [(0, '24.446')] [2023-06-21 13:04:12,665][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 81 [2023-06-21 13:04:16,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6005888. Throughput: 0: 158.0. Samples: 251108. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:04:16,477][62782] Avg episode reward: [(0, '24.516')] [2023-06-21 13:04:21,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6009088. Throughput: 0: 157.9. Samples: 252056. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:04:21,477][62782] Avg episode reward: [(0, '24.345')] [2023-06-21 13:04:26,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6012288. Throughput: 0: 157.9. Samples: 252996. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 13:04:26,479][62782] Avg episode reward: [(0, '24.396')] [2023-06-21 13:04:30,210][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 82 [2023-06-21 13:04:31,187][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 82 [2023-06-21 13:04:31,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6015488. Throughput: 0: 157.8. Samples: 253472. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:04:31,476][62782] Avg episode reward: [(0, '24.396')] [2023-06-21 13:04:32,701][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 82 [2023-06-21 13:04:36,476][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6018688. Throughput: 0: 157.9. Samples: 254424. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 13:04:36,476][62782] Avg episode reward: [(0, '24.406')] [2023-06-21 13:04:41,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6021888. Throughput: 0: 157.9. Samples: 255372. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:04:41,477][62782] Avg episode reward: [(0, '25.064')] [2023-06-21 13:04:46,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6025088. Throughput: 0: 157.9. Samples: 255846. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 13:04:46,476][62782] Avg episode reward: [(0, '25.064')] [2023-06-21 13:04:47,804][78401] Updated weights for policy 0, policy_version 1610 (0.0006) [2023-06-21 13:04:50,243][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 83 [2023-06-21 13:04:51,206][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 83 [2023-06-21 13:04:51,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6028288. Throughput: 0: 158.0. Samples: 256798. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:04:51,476][62782] Avg episode reward: [(0, '25.492')] [2023-06-21 13:04:52,721][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 83 [2023-06-21 13:04:56,477][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6031488. Throughput: 0: 157.9. Samples: 257738. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 13:04:56,478][62782] Avg episode reward: [(0, '26.565')] [2023-06-21 13:05:01,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6034688. Throughput: 0: 157.8. Samples: 258210. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:05:01,477][62782] Avg episode reward: [(0, '26.327')] [2023-06-21 13:05:01,479][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001612_6034688.pth... [2023-06-21 13:05:01,581][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001565_5884288.pth [2023-06-21 13:05:06,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6037888. Throughput: 0: 157.7. Samples: 259152. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:05:06,477][62782] Avg episode reward: [(0, '26.795')] [2023-06-21 13:05:10,253][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 84 [2023-06-21 13:05:11,245][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 84 [2023-06-21 13:05:11,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6041088. Throughput: 0: 158.1. Samples: 260110. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 13:05:11,477][62782] Avg episode reward: [(0, '26.662')] [2023-06-21 13:05:12,744][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 84 [2023-06-21 13:05:16,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6044288. Throughput: 0: 158.0. Samples: 260580. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:05:16,477][62782] Avg episode reward: [(0, '26.485')] [2023-06-21 13:05:21,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6047488. Throughput: 0: 158.0. Samples: 261536. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:05:21,476][62782] Avg episode reward: [(0, '26.443')] [2023-06-21 13:05:26,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6050688. Throughput: 0: 147.5. Samples: 262010. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:05:26,478][62782] Avg episode reward: [(0, '26.426')] [2023-06-21 13:05:30,258][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 85 [2023-06-21 13:05:31,265][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 85 [2023-06-21 13:05:31,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6053888. Throughput: 0: 158.0. Samples: 262954. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 13:05:31,476][62782] Avg episode reward: [(0, '26.342')] [2023-06-21 13:05:32,776][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 85 [2023-06-21 13:05:36,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6057088. Throughput: 0: 157.8. Samples: 263898. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 13:05:36,477][62782] Avg episode reward: [(0, '26.946')] [2023-06-21 13:05:39,264][78401] Updated weights for policy 0, policy_version 1620 (0.0007) [2023-06-21 13:05:41,477][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6060288. Throughput: 0: 158.1. Samples: 264852. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:05:41,477][62782] Avg episode reward: [(0, '26.638')] [2023-06-21 13:05:46,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6063488. Throughput: 0: 158.2. Samples: 265330. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:05:46,478][62782] Avg episode reward: [(0, '26.628')] [2023-06-21 13:05:50,299][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 86 [2023-06-21 13:05:51,287][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 86 [2023-06-21 13:05:51,476][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6066688. Throughput: 0: 158.3. Samples: 266276. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:05:51,477][62782] Avg episode reward: [(0, '26.761')] [2023-06-21 13:05:52,793][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 86 [2023-06-21 13:05:56,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6069888. Throughput: 0: 147.6. Samples: 266752. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 13:05:56,477][62782] Avg episode reward: [(0, '27.160')] [2023-06-21 13:06:01,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6073088. Throughput: 0: 158.2. Samples: 267700. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:06:01,478][62782] Avg episode reward: [(0, '27.049')] [2023-06-21 13:06:06,475][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6076288. Throughput: 0: 158.0. Samples: 268646. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2023-06-21 13:06:06,477][62782] Avg episode reward: [(0, '27.357')] [2023-06-21 13:06:10,330][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 87 [2023-06-21 13:06:11,322][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 87 [2023-06-21 13:06:11,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6079488. Throughput: 0: 168.6. Samples: 269596. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:06:11,476][62782] Avg episode reward: [(0, '27.641')] [2023-06-21 13:06:12,818][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 87 [2023-06-21 13:06:16,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6082688. Throughput: 0: 158.0. Samples: 270064. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2023-06-21 13:06:16,477][62782] Avg episode reward: [(0, '26.949')] [2023-06-21 13:06:21,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6085888. Throughput: 0: 158.3. Samples: 271020. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:06:21,477][62782] Avg episode reward: [(0, '26.780')] [2023-06-21 13:06:26,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 6089088. Throughput: 0: 147.6. Samples: 271494. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:06:26,477][62782] Avg episode reward: [(0, '26.279')] [2023-06-21 13:06:29,638][78401] Updated weights for policy 0, policy_version 1630 (0.0007) [2023-06-21 13:06:30,384][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 88 [2023-06-21 13:06:31,350][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 88 [2023-06-21 13:06:31,477][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6092288. Throughput: 0: 157.8. Samples: 272432. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:06:31,478][62782] Avg episode reward: [(0, '26.237')] [2023-06-21 13:06:32,856][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 88 [2023-06-21 13:06:36,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 640.0). Total num frames: 6095488. Throughput: 0: 158.0. Samples: 273384. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:06:36,477][62782] Avg episode reward: [(0, '26.028')] [2023-06-21 13:06:41,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6098688. Throughput: 0: 168.4. Samples: 274328. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:06:41,476][62782] Avg episode reward: [(0, '25.860')] [2023-06-21 13:06:46,475][62782] Fps is (10 sec: 640.1, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6101888. Throughput: 0: 157.8. Samples: 274800. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:06:46,477][62782] Avg episode reward: [(0, '25.481')] [2023-06-21 13:06:50,422][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 89 [2023-06-21 13:06:51,400][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 89 [2023-06-21 13:06:51,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6105088. Throughput: 0: 157.8. Samples: 275748. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2023-06-21 13:06:51,476][62782] Avg episode reward: [(0, '25.115')] [2023-06-21 13:06:52,864][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 89 [2023-06-21 13:06:56,476][62782] Fps is (10 sec: 639.9, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6108288. Throughput: 0: 158.0. Samples: 276704. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2023-06-21 13:06:56,476][62782] Avg episode reward: [(0, '25.029')] [2023-06-21 13:07:01,476][62782] Fps is (10 sec: 640.0, 60 sec: 640.0, 300 sec: 629.2). Total num frames: 6111488. Throughput: 0: 157.6. Samples: 277156. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2023-06-21 13:07:01,478][62782] Avg episode reward: [(0, '24.534')] [2023-06-21 13:07:01,481][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001636_6111488.pth... [2023-06-21 13:07:01,480][62782] Components not started: RolloutWorker_w2, RolloutWorker_w3, RolloutWorker_w6, RolloutWorker_w7, wait_time=1800.0 seconds [2023-06-21 13:07:01,482][62782] Components take too long to start: RolloutWorker_w2, RolloutWorker_w3, RolloutWorker_w6, RolloutWorker_w7. Aborting the experiment! [2023-06-21 13:07:01,491][62782] Component Batcher_0 stopped! [2023-06-21 13:07:01,491][62782] Waiting for ['LearnerWorker_p0', 'InferenceWorker_p0-w0', 'RolloutWorker_w0', 'RolloutWorker_w1', 'RolloutWorker_w2', 'RolloutWorker_w3', 'RolloutWorker_w4', 'RolloutWorker_w5', 'RolloutWorker_w6', 'RolloutWorker_w7'] to stop... [2023-06-21 13:07:01,496][78400] Stopping Batcher_0... [2023-06-21 13:07:01,516][78400] Loop batcher_evt_loop terminating... [2023-06-21 13:07:01,544][78400] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001589_5961088.pth [2023-06-21 13:07:01,586][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001636_6111488.pth... [2023-06-21 13:07:01,623][78401] Weights refcount: 2 0 [2023-06-21 13:07:01,631][78401] Stopping InferenceWorker_p0-w0... [2023-06-21 13:07:01,631][78401] Loop inference_proc0-0_evt_loop terminating... [2023-06-21 13:07:01,631][62782] Component InferenceWorker_p0-w0 stopped! [2023-06-21 13:07:01,632][62782] Waiting for ['LearnerWorker_p0', 'RolloutWorker_w0', 'RolloutWorker_w1', 'RolloutWorker_w2', 'RolloutWorker_w3', 'RolloutWorker_w4', 'RolloutWorker_w5', 'RolloutWorker_w6', 'RolloutWorker_w7'] to stop... [2023-06-21 13:07:01,835][78400] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001636_6111488.pth... [2023-06-21 13:07:01,861][78406] Stopping RolloutWorker_w4... [2023-06-21 13:07:01,862][78406] Loop rollout_proc4_evt_loop terminating... [2023-06-21 13:07:01,861][62782] Component RolloutWorker_w4 stopped! [2023-06-21 13:07:01,871][62782] Waiting for ['LearnerWorker_p0', 'RolloutWorker_w0', 'RolloutWorker_w1', 'RolloutWorker_w2', 'RolloutWorker_w3', 'RolloutWorker_w5', 'RolloutWorker_w6', 'RolloutWorker_w7'] to stop... [2023-06-21 13:07:01,879][62782] Component RolloutWorker_w5 stopped! [2023-06-21 13:07:01,879][78407] Stopping RolloutWorker_w5... [2023-06-21 13:07:01,882][78407] Loop rollout_proc5_evt_loop terminating... [2023-06-21 13:07:01,883][78402] Stopping RolloutWorker_w0... [2023-06-21 13:07:01,884][78402] Loop rollout_proc0_evt_loop terminating... [2023-06-21 13:07:01,880][62782] Waiting for ['LearnerWorker_p0', 'RolloutWorker_w0', 'RolloutWorker_w1', 'RolloutWorker_w2', 'RolloutWorker_w3', 'RolloutWorker_w6', 'RolloutWorker_w7'] to stop... [2023-06-21 13:07:01,886][62782] Component RolloutWorker_w0 stopped! [2023-06-21 13:07:01,893][62782] Waiting for ['LearnerWorker_p0', 'RolloutWorker_w1', 'RolloutWorker_w2', 'RolloutWorker_w3', 'RolloutWorker_w6', 'RolloutWorker_w7'] to stop... [2023-06-21 13:07:01,897][62782] Component RolloutWorker_w1 stopped! [2023-06-21 13:07:01,898][78403] Stopping RolloutWorker_w1... [2023-06-21 13:07:01,898][62782] Waiting for ['LearnerWorker_p0', 'RolloutWorker_w2', 'RolloutWorker_w3', 'RolloutWorker_w6', 'RolloutWorker_w7'] to stop... [2023-06-21 13:07:01,904][78403] Loop rollout_proc1_evt_loop terminating... [2023-06-21 13:07:01,984][78400] Stopping LearnerWorker_p0... [2023-06-21 13:07:01,984][78400] Loop learner_proc0_evt_loop terminating... [2023-06-21 13:07:01,984][62782] Component LearnerWorker_p0 stopped! [2023-06-21 13:07:01,985][62782] Waiting for ['RolloutWorker_w2', 'RolloutWorker_w3', 'RolloutWorker_w6', 'RolloutWorker_w7'] to stop... [2023-06-21 13:07:10,477][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 90 [2023-06-21 13:07:11,425][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 90 [2023-06-21 13:07:12,872][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 90 [2023-06-21 13:07:30,533][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 91 [2023-06-21 13:07:31,438][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 91 [2023-06-21 13:07:32,900][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 91 [2023-06-21 13:07:50,543][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 92 [2023-06-21 13:07:51,463][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 92 [2023-06-21 13:07:52,935][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 92 [2023-06-21 13:08:10,548][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 93 [2023-06-21 13:08:11,518][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 93 [2023-06-21 13:08:12,990][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 93 [2023-06-21 13:08:30,556][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 94 [2023-06-21 13:08:31,556][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 94 [2023-06-21 13:08:33,034][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 94 [2023-06-21 13:08:50,564][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 95 [2023-06-21 13:08:51,599][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 95 [2023-06-21 13:08:53,039][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 95 [2023-06-21 13:09:10,610][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 96 [2023-06-21 13:09:11,607][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 96 [2023-06-21 13:09:13,058][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 96 [2023-06-21 13:09:30,666][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 97 [2023-06-21 13:09:31,641][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 97 [2023-06-21 13:09:33,113][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 97 [2023-06-21 13:09:50,689][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 98 [2023-06-21 13:09:51,697][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 98 [2023-06-21 13:09:53,154][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 98 [2023-06-21 13:10:10,700][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 99 [2023-06-21 13:10:11,714][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 99 [2023-06-21 13:10:13,168][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 99 [2023-06-21 13:10:30,727][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 100 [2023-06-21 13:10:31,760][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 100 [2023-06-21 13:10:33,199][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 100 [2023-06-21 13:10:50,777][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 101 [2023-06-21 13:10:51,804][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 101 [2023-06-21 13:10:53,209][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 101 [2023-06-21 13:11:10,805][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 102 [2023-06-21 13:11:11,843][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 102 [2023-06-21 13:11:13,263][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 102 [2023-06-21 13:11:30,834][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 103 [2023-06-21 13:11:31,868][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 103 [2023-06-21 13:11:33,282][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 103 [2023-06-21 13:11:50,846][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 104 [2023-06-21 13:11:51,882][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 104 [2023-06-21 13:11:53,306][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 104 [2023-06-21 13:12:10,898][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 105 [2023-06-21 13:12:11,928][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 105 [2023-06-21 13:12:13,314][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 105 [2023-06-21 13:12:30,931][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 106 [2023-06-21 13:12:31,980][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 106 [2023-06-21 13:12:33,336][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 106 [2023-06-21 13:12:50,965][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 107 [2023-06-21 13:12:52,035][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 107 [2023-06-21 13:12:53,378][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 107 [2023-06-21 13:13:10,974][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 108 [2023-06-21 13:13:12,050][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 108 [2023-06-21 13:13:13,400][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 108 [2023-06-21 13:13:31,022][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 109 [2023-06-21 13:13:32,057][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 109 [2023-06-21 13:13:33,425][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 109 [2023-06-21 13:13:51,032][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 110 [2023-06-21 13:13:52,106][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 110 [2023-06-21 13:13:53,469][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 110 [2023-06-21 13:14:11,079][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 111 [2023-06-21 13:14:12,150][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 111 [2023-06-21 13:14:13,500][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 111 [2023-06-21 13:14:31,100][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 112 [2023-06-21 13:14:32,199][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 112 [2023-06-21 13:14:33,507][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 112 [2023-06-21 13:14:51,139][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 113 [2023-06-21 13:14:52,243][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 113 [2023-06-21 13:14:53,546][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 113 [2023-06-21 13:15:11,165][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 114 [2023-06-21 13:15:12,258][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 114 [2023-06-21 13:15:13,575][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 114 [2023-06-21 13:15:31,170][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 115 [2023-06-21 13:15:32,287][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 115 [2023-06-21 13:15:33,616][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 115 [2023-06-21 13:15:51,179][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 116 [2023-06-21 13:15:52,317][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 116 [2023-06-21 13:15:53,665][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 116 [2023-06-21 13:16:11,225][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 117 [2023-06-21 13:16:12,342][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 117 [2023-06-21 13:16:13,673][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 117 [2023-06-21 13:16:31,275][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 118 [2023-06-21 13:16:32,367][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 118 [2023-06-21 13:16:33,707][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 118 [2023-06-21 13:16:51,315][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 119 [2023-06-21 13:16:52,387][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 119 [2023-06-21 13:16:53,727][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 119 [2023-06-21 13:17:11,319][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 120 [2023-06-21 13:17:12,390][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 120 [2023-06-21 13:17:13,736][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 120 [2023-06-21 13:17:31,331][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 121 [2023-06-21 13:17:32,411][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 121 [2023-06-21 13:17:33,746][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 121 [2023-06-21 13:17:51,375][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 122 [2023-06-21 13:17:52,437][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 122 [2023-06-21 13:17:53,753][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 122 [2023-06-21 13:18:11,381][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 123 [2023-06-21 13:18:12,470][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 123 [2023-06-21 13:18:13,801][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 123 [2023-06-21 13:18:31,404][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 124 [2023-06-21 13:18:32,518][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 124 [2023-06-21 13:18:33,836][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 124 [2023-06-21 13:18:51,434][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 125 [2023-06-21 13:18:52,554][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 125 [2023-06-21 13:18:53,886][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 125 [2023-06-21 13:19:11,439][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 126 [2023-06-21 13:19:12,583][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 126 [2023-06-21 13:19:13,883][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 126 [2023-06-21 13:19:31,443][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 127 [2023-06-21 13:19:32,593][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 127 [2023-06-21 13:19:33,916][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 127 [2023-06-21 13:19:51,493][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 128 [2023-06-21 13:19:52,627][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 128 [2023-06-21 13:19:53,925][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 128 [2023-06-21 13:20:11,546][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 129 [2023-06-21 13:20:12,638][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 129 [2023-06-21 13:20:13,978][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 129 [2023-06-21 13:20:31,548][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 130 [2023-06-21 13:20:32,639][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 130 [2023-06-21 13:20:33,979][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 130 [2023-06-21 13:20:51,565][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 131 [2023-06-21 13:20:52,675][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 131 [2023-06-21 13:20:54,028][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 131 [2023-06-21 13:21:11,572][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 132 [2023-06-21 13:21:12,703][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 132 [2023-06-21 13:21:14,079][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 132 [2023-06-21 13:21:31,595][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 133 [2023-06-21 13:21:32,721][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 133 [2023-06-21 13:21:34,122][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 133 [2023-06-21 13:21:51,610][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 134 [2023-06-21 13:21:52,728][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 134 [2023-06-21 13:21:54,165][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 134 [2023-06-21 13:22:11,631][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 135 [2023-06-21 13:22:12,779][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 135 [2023-06-21 13:22:14,209][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 135 [2023-06-21 13:22:31,658][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 136 [2023-06-21 13:22:32,823][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 136 [2023-06-21 13:22:34,233][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 136 [2023-06-21 13:22:51,706][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 137 [2023-06-21 13:22:52,877][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 137 [2023-06-21 13:22:54,290][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 137 [2023-06-21 13:23:11,747][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 138 [2023-06-21 13:23:12,930][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 138 [2023-06-21 13:23:14,343][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 138 [2023-06-21 13:23:31,750][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 139 [2023-06-21 13:23:32,965][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 139 [2023-06-21 13:23:34,348][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 139 [2023-06-21 13:23:51,758][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 140 [2023-06-21 13:23:53,017][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 140 [2023-06-21 13:23:54,359][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 140 [2023-06-21 13:24:11,785][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 141 [2023-06-21 13:24:13,064][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 141 [2023-06-21 13:24:14,364][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 141 [2023-06-21 13:24:31,812][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 142 [2023-06-21 13:24:33,098][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 142 [2023-06-21 13:24:34,404][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 142 [2023-06-21 13:24:51,839][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 143 [2023-06-21 13:24:53,116][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 143 [2023-06-21 13:24:54,453][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 143 [2023-06-21 13:25:11,876][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 144 [2023-06-21 13:25:13,159][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 144 [2023-06-21 13:25:14,457][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 144 [2023-06-21 13:25:31,913][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 145 [2023-06-21 13:25:33,161][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 145 [2023-06-21 13:25:34,484][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 145 [2023-06-21 13:25:51,952][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 146 [2023-06-21 13:25:53,172][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 146 [2023-06-21 13:25:54,508][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 146 [2023-06-21 13:26:11,984][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 147 [2023-06-21 13:26:13,225][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 147 [2023-06-21 13:26:14,541][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 147 [2023-06-21 13:26:32,033][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 148 [2023-06-21 13:26:33,250][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 148 [2023-06-21 13:26:34,586][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 148 [2023-06-21 13:26:52,077][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 149 [2023-06-21 13:26:53,294][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 149 [2023-06-21 13:26:54,626][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 149 [2023-06-21 13:27:12,116][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 150 [2023-06-21 13:27:13,300][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 150 [2023-06-21 13:27:14,678][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 150 [2023-06-21 13:27:32,173][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 151 [2023-06-21 13:27:33,329][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 151 [2023-06-21 13:27:34,700][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 151 [2023-06-21 13:27:52,192][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 152 [2023-06-21 13:27:53,359][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 152 [2023-06-21 13:27:54,717][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 152 [2023-06-21 13:28:12,242][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 153 [2023-06-21 13:28:13,404][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 153 [2023-06-21 13:28:14,728][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 153 [2023-06-21 13:28:32,283][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 154 [2023-06-21 13:28:33,438][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 154 [2023-06-21 13:28:34,756][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 154 [2023-06-21 13:28:52,306][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 155 [2023-06-21 13:28:53,457][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 155 [2023-06-21 13:28:54,795][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 155 [2023-06-21 13:29:12,351][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 156 [2023-06-21 13:29:13,481][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 156 [2023-06-21 13:29:14,798][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 156 [2023-06-21 13:29:32,367][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 157 [2023-06-21 13:29:33,492][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 157 [2023-06-21 13:29:34,815][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 157 [2023-06-21 13:29:52,372][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 158 [2023-06-21 13:29:53,539][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 158 [2023-06-21 13:29:54,830][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 158 [2023-06-21 13:30:12,430][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 159 [2023-06-21 13:30:13,565][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 159 [2023-06-21 13:30:14,832][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 159 [2023-06-21 13:30:32,475][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 160 [2023-06-21 13:30:33,597][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 160 [2023-06-21 13:30:34,867][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 160 [2023-06-21 13:30:52,490][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 161 [2023-06-21 13:30:53,652][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 161 [2023-06-21 13:30:54,883][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 161 [2023-06-21 13:31:12,505][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 162 [2023-06-21 13:31:13,679][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 162 [2023-06-21 13:31:14,927][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 162 [2023-06-21 13:31:32,521][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 163 [2023-06-21 13:31:33,713][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 163 [2023-06-21 13:31:34,928][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 163 [2023-06-21 13:31:52,554][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 164 [2023-06-21 13:31:53,743][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 164 [2023-06-21 13:31:54,954][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 164 [2023-06-21 13:32:12,605][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 165 [2023-06-21 13:32:13,753][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 165 [2023-06-21 13:32:15,003][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 165 [2023-06-21 13:32:32,632][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 166 [2023-06-21 13:32:33,774][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 166 [2023-06-21 13:32:35,046][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 166 [2023-06-21 13:32:52,677][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 167 [2023-06-21 13:32:53,822][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 167 [2023-06-21 13:32:55,098][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 167 [2023-06-21 13:33:12,718][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 168 [2023-06-21 13:33:13,878][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 168 [2023-06-21 13:33:15,118][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 168 [2023-06-21 13:33:32,755][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 169 [2023-06-21 13:33:33,897][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 169 [2023-06-21 13:33:35,133][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 169 [2023-06-21 13:33:52,767][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 170 [2023-06-21 13:33:53,933][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 170 [2023-06-21 13:33:55,184][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 170 [2023-06-21 13:34:12,811][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 171 [2023-06-21 13:34:13,977][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 171 [2023-06-21 13:34:15,233][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 171 [2023-06-21 13:34:32,833][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 172 [2023-06-21 13:34:34,027][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 172 [2023-06-21 13:34:35,257][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 172 [2023-06-21 13:34:52,869][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 173 [2023-06-21 13:34:54,054][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 173 [2023-06-21 13:34:55,281][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 173 [2023-06-21 13:35:12,882][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 174 [2023-06-21 13:35:14,078][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 174 [2023-06-21 13:35:15,314][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 174 [2023-06-21 13:35:32,897][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 175 [2023-06-21 13:35:34,087][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 175 [2023-06-21 13:35:35,323][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 175 [2023-06-21 13:35:52,922][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 176 [2023-06-21 13:35:54,101][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 176 [2023-06-21 13:35:55,362][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 176 [2023-06-21 13:36:12,974][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 177 [2023-06-21 13:36:14,153][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 177 [2023-06-21 13:36:15,373][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 177 [2023-06-21 13:36:33,010][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 178 [2023-06-21 13:36:34,185][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 178 [2023-06-21 13:36:35,406][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 178 [2023-06-21 13:36:53,034][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 179 [2023-06-21 13:36:54,207][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 179 [2023-06-21 13:36:55,419][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 179 [2023-06-21 13:37:13,086][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 180 [2023-06-21 13:37:14,242][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 180 [2023-06-21 13:37:15,475][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 180 [2023-06-21 13:37:33,117][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 181 [2023-06-21 13:37:34,267][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 181 [2023-06-21 13:37:35,480][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 181 [2023-06-21 13:37:53,156][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 182 [2023-06-21 13:37:54,270][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 182 [2023-06-21 13:37:55,537][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 182 [2023-06-21 13:38:13,171][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 183 [2023-06-21 13:38:14,283][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 183 [2023-06-21 13:38:15,595][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 183 [2023-06-21 13:38:33,203][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 184 [2023-06-21 13:38:34,308][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 184 [2023-06-21 13:38:35,624][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 184 [2023-06-21 13:38:53,226][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 185 [2023-06-21 13:38:54,310][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 185 [2023-06-21 13:38:55,633][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 185 [2023-06-21 13:39:13,280][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 186 [2023-06-21 13:39:14,316][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 186 [2023-06-21 13:39:15,664][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 186 [2023-06-21 13:39:33,312][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 187 [2023-06-21 13:39:34,352][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 187 [2023-06-21 13:39:35,667][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 187 [2023-06-21 13:39:53,353][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 188 [2023-06-21 13:39:54,369][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 188 [2023-06-21 13:39:55,688][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 188 [2023-06-21 13:40:13,368][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 189 [2023-06-21 13:40:14,376][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 189 [2023-06-21 13:40:15,746][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 189 [2023-06-21 13:40:33,406][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 190 [2023-06-21 13:40:34,409][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 190 [2023-06-21 13:40:35,789][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 190 [2023-06-21 13:40:53,408][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 191 [2023-06-21 13:40:54,449][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 191 [2023-06-21 13:40:55,823][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 191 [2023-06-21 13:41:13,459][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 192 [2023-06-21 13:41:14,485][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 192 [2023-06-21 13:41:15,828][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 192 [2023-06-21 13:41:33,490][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 193 [2023-06-21 13:41:34,516][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 193 [2023-06-21 13:41:35,834][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 193 [2023-06-21 13:41:53,517][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 194 [2023-06-21 13:41:54,533][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 194 [2023-06-21 13:41:55,872][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 194 [2023-06-21 13:42:13,565][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 195 [2023-06-21 13:42:14,584][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 195 [2023-06-21 13:42:15,874][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 195 [2023-06-21 13:42:33,610][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 196 [2023-06-21 13:42:34,602][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 196 [2023-06-21 13:42:35,886][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 196 [2023-06-21 13:42:53,661][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 197 [2023-06-21 13:42:54,604][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 197 [2023-06-21 13:42:55,940][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 197 [2023-06-21 13:43:13,682][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 198 [2023-06-21 13:43:14,608][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 198 [2023-06-21 13:43:15,976][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 198 [2023-06-21 13:43:33,720][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 199 [2023-06-21 13:43:34,619][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 199 [2023-06-21 13:43:36,019][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 199 [2023-06-21 13:43:53,771][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 200 [2023-06-21 13:43:54,638][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 200 [2023-06-21 13:43:56,023][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 200 [2023-06-21 13:44:13,790][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 201 [2023-06-21 13:44:14,688][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 201 [2023-06-21 13:44:16,039][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 201 [2023-06-21 13:44:33,840][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 202 [2023-06-21 13:44:34,741][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 202 [2023-06-21 13:44:36,084][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 202 [2023-06-21 13:44:53,879][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 203 [2023-06-21 13:44:54,793][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 203 [2023-06-21 13:44:56,114][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 203 [2023-06-21 13:45:13,933][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 204 [2023-06-21 13:45:14,828][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 204 [2023-06-21 13:45:16,173][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 204 [2023-06-21 13:45:33,955][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 205 [2023-06-21 13:45:34,858][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 205 [2023-06-21 13:45:36,191][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 205 [2023-06-21 13:45:53,988][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 206 [2023-06-21 13:45:54,871][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 206 [2023-06-21 13:45:56,227][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 206 [2023-06-21 13:46:14,040][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 207 [2023-06-21 13:46:14,929][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 207 [2023-06-21 13:46:16,263][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 207 [2023-06-21 13:46:34,064][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 208 [2023-06-21 13:46:34,957][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 208 [2023-06-21 13:46:36,309][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 208 [2023-06-21 13:46:54,087][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 209 [2023-06-21 13:46:54,963][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 209 [2023-06-21 13:46:56,366][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 209 [2023-06-21 13:47:14,132][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 210 [2023-06-21 13:47:14,997][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 210 [2023-06-21 13:47:16,383][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 210 [2023-06-21 13:47:34,171][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 211 [2023-06-21 13:47:35,053][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 211 [2023-06-21 13:47:36,388][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 211 [2023-06-21 13:47:54,201][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 212 [2023-06-21 13:47:55,080][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 212 [2023-06-21 13:47:56,437][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 212 [2023-06-21 13:48:14,242][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 213 [2023-06-21 13:48:15,122][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 213 [2023-06-21 13:48:16,482][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 213 [2023-06-21 13:48:34,300][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 214 [2023-06-21 13:48:35,138][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 214 [2023-06-21 13:48:36,530][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 214 [2023-06-21 13:48:54,361][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 215 [2023-06-21 13:48:55,186][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 215 [2023-06-21 13:48:56,545][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 215 [2023-06-21 13:49:14,399][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 216 [2023-06-21 13:49:15,192][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 216 [2023-06-21 13:49:16,573][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 216 [2023-06-21 13:49:34,437][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 217 [2023-06-21 13:49:35,202][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 217 [2023-06-21 13:49:36,618][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 217 [2023-06-21 13:49:54,474][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 218 [2023-06-21 13:49:55,256][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 218 [2023-06-21 13:49:56,656][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 218 [2023-06-21 13:50:14,480][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 219 [2023-06-21 13:50:15,267][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 219 [2023-06-21 13:50:16,689][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 219 [2023-06-21 13:50:34,505][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 220 [2023-06-21 13:50:35,274][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 220 [2023-06-21 13:50:36,724][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 220 [2023-06-21 13:50:54,560][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 221 [2023-06-21 13:50:55,311][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 221 [2023-06-21 13:50:56,763][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 221 [2023-06-21 13:51:14,565][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 222 [2023-06-21 13:51:15,360][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 222 [2023-06-21 13:51:16,770][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 222 [2023-06-21 13:51:34,608][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 223 [2023-06-21 13:51:35,366][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 223 [2023-06-21 13:51:36,779][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 223 [2023-06-21 13:51:54,615][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 224 [2023-06-21 13:51:55,414][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 224 [2023-06-21 13:51:56,802][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 224 [2023-06-21 13:52:14,643][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 225 [2023-06-21 13:52:15,445][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 225 [2023-06-21 13:52:16,820][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 225 [2023-06-21 13:52:34,660][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 226 [2023-06-21 13:52:35,470][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 226 [2023-06-21 13:52:36,868][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 226 [2023-06-21 13:52:54,675][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 227 [2023-06-21 13:52:55,474][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 227 [2023-06-21 13:52:56,875][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 227 [2023-06-21 13:53:14,712][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 228 [2023-06-21 13:53:15,521][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 228 [2023-06-21 13:53:16,937][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 228 [2023-06-21 13:53:34,725][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 229 [2023-06-21 13:53:35,525][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 229 [2023-06-21 13:53:36,976][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 229 [2023-06-21 13:53:54,773][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 230 [2023-06-21 13:53:55,561][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 230 [2023-06-21 13:53:56,981][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 230 [2023-06-21 13:54:14,827][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 231 [2023-06-21 13:54:15,597][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 231 [2023-06-21 13:54:17,035][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 231 [2023-06-21 13:54:34,842][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 232 [2023-06-21 13:54:35,615][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 232 [2023-06-21 13:54:37,089][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 232 [2023-06-21 13:54:54,845][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 233 [2023-06-21 13:54:55,663][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 233 [2023-06-21 13:54:57,097][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 233 [2023-06-21 13:55:14,883][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 234 [2023-06-21 13:55:15,701][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 234 [2023-06-21 13:55:17,136][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 234 [2023-06-21 13:55:34,927][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 235 [2023-06-21 13:55:35,747][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 235 [2023-06-21 13:55:37,155][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 235 [2023-06-21 13:55:54,982][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 236 [2023-06-21 13:55:55,773][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 236 [2023-06-21 13:55:57,179][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 236 [2023-06-21 13:56:15,006][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 237 [2023-06-21 13:56:15,806][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 237 [2023-06-21 13:56:17,216][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 237 [2023-06-21 13:56:35,013][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 238 [2023-06-21 13:56:35,838][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 238 [2023-06-21 13:56:37,231][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 238 [2023-06-21 13:56:55,053][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 239 [2023-06-21 13:56:55,889][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 239 [2023-06-21 13:56:57,240][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 239 [2023-06-21 13:57:15,095][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 240 [2023-06-21 13:57:15,911][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 240 [2023-06-21 13:57:17,266][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 240 [2023-06-21 13:57:35,108][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 241 [2023-06-21 13:57:35,923][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 241 [2023-06-21 13:57:37,292][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 241 [2023-06-21 13:57:55,122][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 242 [2023-06-21 13:57:55,932][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 242 [2023-06-21 13:57:57,313][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 242 [2023-06-21 13:58:15,165][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 243 [2023-06-21 13:58:15,965][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 243 [2023-06-21 13:58:17,316][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 243 [2023-06-21 13:58:35,187][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 244 [2023-06-21 13:58:36,004][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 244 [2023-06-21 13:58:37,344][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 244 [2023-06-21 13:58:55,191][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 245 [2023-06-21 13:58:56,054][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 245 [2023-06-21 13:58:57,394][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 245 [2023-06-21 13:59:15,232][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 246 [2023-06-21 13:59:16,061][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 246 [2023-06-21 13:59:17,416][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 246 [2023-06-21 13:59:35,242][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 247 [2023-06-21 13:59:36,115][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 247 [2023-06-21 13:59:37,443][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 247 [2023-06-21 13:59:55,263][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 248 [2023-06-21 13:59:56,169][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 248 [2023-06-21 13:59:57,474][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 248 [2023-06-21 14:00:15,285][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 249 [2023-06-21 14:00:16,213][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 249 [2023-06-21 14:00:17,512][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 249 [2023-06-21 14:00:35,335][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 250 [2023-06-21 14:00:36,242][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 250 [2023-06-21 14:00:37,533][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 250 [2023-06-21 14:00:55,387][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 251 [2023-06-21 14:00:56,266][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 251 [2023-06-21 14:00:57,553][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 251 [2023-06-21 14:01:15,420][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 252 [2023-06-21 14:01:16,315][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 252 [2023-06-21 14:01:17,564][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 252 [2023-06-21 14:01:35,444][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 253 [2023-06-21 14:01:36,323][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 253 [2023-06-21 14:01:37,605][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 253 [2023-06-21 14:01:55,498][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 254 [2023-06-21 14:01:56,330][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 254 [2023-06-21 14:01:57,619][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 254 [2023-06-21 14:02:15,529][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 255 [2023-06-21 14:02:16,384][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 255 [2023-06-21 14:02:17,632][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 255 [2023-06-21 14:02:35,561][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 256 [2023-06-21 14:02:36,438][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 256 [2023-06-21 14:02:37,648][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 256 [2023-06-21 14:02:55,576][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 257 [2023-06-21 14:02:56,487][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 257 [2023-06-21 14:02:57,676][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 257 [2023-06-21 14:03:15,608][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 258 [2023-06-21 14:03:16,496][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 258 [2023-06-21 14:03:17,710][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 258 [2023-06-21 14:03:35,633][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 259 [2023-06-21 14:03:36,536][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 259 [2023-06-21 14:03:37,761][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 259 [2023-06-21 14:03:55,679][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 260 [2023-06-21 14:03:56,558][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 260 [2023-06-21 14:03:57,802][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 260 [2023-06-21 14:04:15,730][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 261 [2023-06-21 14:04:16,564][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 261 [2023-06-21 14:04:17,841][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 261 [2023-06-21 14:04:35,744][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 262 [2023-06-21 14:04:36,596][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 262 [2023-06-21 14:04:37,864][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 262 [2023-06-21 14:04:55,782][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 263 [2023-06-21 14:04:56,619][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 263 [2023-06-21 14:04:57,916][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 263 [2023-06-21 14:05:15,831][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 264 [2023-06-21 14:05:16,637][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 264 [2023-06-21 14:05:17,946][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 264 [2023-06-21 14:05:35,840][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 265 [2023-06-21 14:05:36,666][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 265 [2023-06-21 14:05:37,970][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 265 [2023-06-21 14:05:55,872][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 266 [2023-06-21 14:05:56,675][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 266 [2023-06-21 14:05:58,009][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 266 [2023-06-21 14:06:15,902][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 267 [2023-06-21 14:06:16,714][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 267 [2023-06-21 14:06:18,065][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 267 [2023-06-21 14:06:35,961][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 268 [2023-06-21 14:06:36,723][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 268 [2023-06-21 14:06:38,120][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 268 [2023-06-21 14:06:55,979][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 269 [2023-06-21 14:06:56,743][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 269 [2023-06-21 14:06:58,135][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 269 [2023-06-21 14:07:16,003][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 270 [2023-06-21 14:07:16,772][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 270 [2023-06-21 14:07:18,139][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 270 [2023-06-21 14:07:36,014][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 271 [2023-06-21 14:07:36,803][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 271 [2023-06-21 14:07:38,167][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 271 [2023-06-21 14:07:56,053][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 272 [2023-06-21 14:07:56,851][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 272 [2023-06-21 14:07:58,216][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 272 [2023-06-21 14:08:16,072][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 273 [2023-06-21 14:08:16,899][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 273 [2023-06-21 14:08:18,249][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 273 [2023-06-21 14:08:36,099][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 274 [2023-06-21 14:08:36,912][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 274 [2023-06-21 14:08:38,265][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 274 [2023-06-21 14:08:56,147][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 275 [2023-06-21 14:08:56,916][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 275 [2023-06-21 14:08:58,320][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 275 [2023-06-21 14:09:16,182][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 276 [2023-06-21 14:09:16,919][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 276 [2023-06-21 14:09:18,353][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 276 [2023-06-21 14:09:36,218][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 277 [2023-06-21 14:09:36,964][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 277 [2023-06-21 14:09:38,358][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 277 [2023-06-21 14:09:56,243][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 278 [2023-06-21 14:09:56,981][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 278 [2023-06-21 14:09:58,386][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 278 [2023-06-21 14:10:16,268][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 279 [2023-06-21 14:10:16,999][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 279 [2023-06-21 14:10:18,409][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 279 [2023-06-21 14:10:36,291][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 280 [2023-06-21 14:10:37,043][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 280 [2023-06-21 14:10:38,424][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 280 [2023-06-21 14:10:56,341][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 281 [2023-06-21 14:10:57,090][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 281 [2023-06-21 14:10:58,456][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 281 [2023-06-21 14:11:16,389][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 282 [2023-06-21 14:11:17,092][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 282 [2023-06-21 14:11:18,512][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 282 [2023-06-21 14:11:36,420][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 283 [2023-06-21 14:11:37,151][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 283 [2023-06-21 14:11:38,519][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 283 [2023-06-21 14:11:56,451][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 284 [2023-06-21 14:11:57,197][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 284 [2023-06-21 14:11:58,571][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 284 [2023-06-21 14:12:16,499][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 285 [2023-06-21 14:12:17,211][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 285 [2023-06-21 14:12:18,591][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 285 [2023-06-21 14:12:36,540][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 286 [2023-06-21 14:12:37,259][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 286 [2023-06-21 14:12:38,622][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 286 [2023-06-21 14:12:56,564][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 287 [2023-06-21 14:12:57,318][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 287 [2023-06-21 14:12:58,665][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 287 [2023-06-21 14:13:16,608][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 288 [2023-06-21 14:13:17,320][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 288 [2023-06-21 14:13:18,710][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 288 [2023-06-21 14:13:36,626][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 289 [2023-06-21 14:13:37,337][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 289 [2023-06-21 14:13:38,720][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 289 [2023-06-21 14:13:56,674][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 290 [2023-06-21 14:13:57,398][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 290 [2023-06-21 14:13:58,781][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 290 [2023-06-21 14:14:16,686][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 291 [2023-06-21 14:14:17,407][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 291 [2023-06-21 14:14:18,834][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 291 [2023-06-21 14:14:36,709][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 292 [2023-06-21 14:14:37,426][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 292 [2023-06-21 14:14:38,849][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 292 [2023-06-21 14:14:56,756][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 293 [2023-06-21 14:14:57,483][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 293 [2023-06-21 14:14:58,898][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 293 [2023-06-21 14:15:16,772][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 294 [2023-06-21 14:15:17,514][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 294 [2023-06-21 14:15:18,927][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 294 [2023-06-21 14:15:36,823][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 295 [2023-06-21 14:15:37,538][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 295 [2023-06-21 14:15:38,947][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 295 [2023-06-21 14:15:56,875][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 296 [2023-06-21 14:15:57,560][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 296 [2023-06-21 14:15:58,987][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 296 [2023-06-21 14:16:16,902][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 297 [2023-06-21 14:16:17,584][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 297 [2023-06-21 14:16:18,994][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 297 [2023-06-21 14:16:36,916][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 298 [2023-06-21 14:16:37,603][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 298 [2023-06-21 14:16:39,050][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 298 [2023-06-21 14:16:56,949][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 299 [2023-06-21 14:16:57,634][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 299 [2023-06-21 14:16:59,054][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 299 [2023-06-21 14:17:17,004][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 300 [2023-06-21 14:17:17,660][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 300 [2023-06-21 14:17:19,066][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 300 [2023-06-21 14:17:37,013][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 301 [2023-06-21 14:17:37,690][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 301 [2023-06-21 14:17:39,116][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 301 [2023-06-21 14:17:57,019][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 302 [2023-06-21 14:17:57,734][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 302 [2023-06-21 14:17:59,138][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 302 [2023-06-21 14:18:17,034][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 303 [2023-06-21 14:18:17,749][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 303 [2023-06-21 14:18:19,164][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 303 [2023-06-21 14:18:37,060][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 304 [2023-06-21 14:18:37,765][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 304 [2023-06-21 14:18:39,179][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 304 [2023-06-21 14:18:57,063][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 305 [2023-06-21 14:18:57,798][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 305 [2023-06-21 14:18:59,237][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 305 [2023-06-21 14:19:17,065][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 306 [2023-06-21 14:19:17,823][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 306 [2023-06-21 14:19:19,281][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 306 [2023-06-21 14:19:37,109][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 307 [2023-06-21 14:19:37,861][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 307 [2023-06-21 14:19:39,317][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 307 [2023-06-21 14:19:57,164][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 308 [2023-06-21 14:19:57,910][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 308 [2023-06-21 14:19:59,342][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 308 [2023-06-21 14:20:17,214][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 309 [2023-06-21 14:20:17,919][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 309 [2023-06-21 14:20:19,376][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 309 [2023-06-21 14:20:37,252][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 310 [2023-06-21 14:20:37,925][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 310 [2023-06-21 14:20:39,390][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 310 [2023-06-21 14:20:57,279][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 311 [2023-06-21 14:20:57,945][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 311 [2023-06-21 14:20:59,443][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 311 [2023-06-21 14:21:17,306][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 312 [2023-06-21 14:21:17,973][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 312 [2023-06-21 14:21:19,451][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 312 [2023-06-21 14:21:37,332][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 313 [2023-06-21 14:21:37,981][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 313 [2023-06-21 14:21:39,481][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 313 [2023-06-21 14:21:57,346][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 314 [2023-06-21 14:21:58,019][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 314 [2023-06-21 14:21:59,486][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 314 [2023-06-21 14:22:17,377][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 315 [2023-06-21 14:22:18,057][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 315 [2023-06-21 14:22:19,494][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 315 [2023-06-21 14:22:37,415][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 316 [2023-06-21 14:22:38,105][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 316 [2023-06-21 14:22:39,532][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 316 [2023-06-21 14:22:57,437][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 317 [2023-06-21 14:22:58,166][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 317 [2023-06-21 14:22:59,542][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 317 [2023-06-21 14:23:17,486][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 318 [2023-06-21 14:23:18,186][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 318 [2023-06-21 14:23:19,551][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 318 [2023-06-21 14:23:37,527][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 319 [2023-06-21 14:23:38,235][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 319 [2023-06-21 14:23:39,571][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 319 [2023-06-21 14:23:57,530][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 320 [2023-06-21 14:23:58,295][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 320 [2023-06-21 14:23:59,606][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 320 [2023-06-21 14:24:17,560][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 321 [2023-06-21 14:24:18,333][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 321 [2023-06-21 14:24:19,623][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 321 [2023-06-21 14:24:37,604][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 322 [2023-06-21 14:24:38,345][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 322 [2023-06-21 14:24:39,670][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 322 [2023-06-21 14:24:57,608][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 323 [2023-06-21 14:24:58,400][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 323 [2023-06-21 14:24:59,722][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 323 [2023-06-21 14:25:17,631][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 324 [2023-06-21 14:25:18,422][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 324 [2023-06-21 14:25:19,761][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 324 [2023-06-21 14:25:37,637][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 325 [2023-06-21 14:25:38,440][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 325 [2023-06-21 14:25:39,791][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 325 [2023-06-21 14:25:57,641][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 326 [2023-06-21 14:25:58,468][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 326 [2023-06-21 14:25:59,805][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 326 [2023-06-21 14:26:17,652][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 327 [2023-06-21 14:26:18,471][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 327 [2023-06-21 14:26:19,819][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 327 [2023-06-21 14:26:37,673][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 328 [2023-06-21 14:26:38,476][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 328 [2023-06-21 14:26:39,833][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 328 [2023-06-21 14:26:57,691][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 329 [2023-06-21 14:26:58,533][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 329 [2023-06-21 14:26:59,888][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 329 [2023-06-21 14:27:17,730][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 330 [2023-06-21 14:27:18,584][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 330 [2023-06-21 14:27:19,919][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 330 [2023-06-21 14:27:37,734][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 331 [2023-06-21 14:27:38,634][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 331 [2023-06-21 14:27:39,961][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 331 [2023-06-21 14:27:57,754][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 332 [2023-06-21 14:27:58,662][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 332 [2023-06-21 14:27:59,986][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 332 [2023-06-21 14:28:17,797][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 333 [2023-06-21 14:28:18,718][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 333 [2023-06-21 14:28:20,033][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 333 [2023-06-21 14:28:37,821][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 334 [2023-06-21 14:28:38,763][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 334 [2023-06-21 14:28:40,084][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 334 [2023-06-21 14:28:57,840][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 335 [2023-06-21 14:28:58,773][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 335 [2023-06-21 14:29:00,116][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 335 [2023-06-21 14:29:17,847][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 336 [2023-06-21 14:29:18,818][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 336 [2023-06-21 14:29:20,171][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 336 [2023-06-21 14:29:37,850][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 337 [2023-06-21 14:29:38,852][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 337 [2023-06-21 14:29:40,183][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 337 [2023-06-21 14:29:57,854][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 338 [2023-06-21 14:29:58,868][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 338 [2023-06-21 14:30:00,192][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 338 [2023-06-21 14:30:17,885][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 339 [2023-06-21 14:30:18,927][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 339 [2023-06-21 14:30:20,206][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 339 [2023-06-21 14:30:37,920][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 340 [2023-06-21 14:30:38,938][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 340 [2023-06-21 14:30:40,240][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 340 [2023-06-21 14:30:57,976][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 341 [2023-06-21 14:30:58,969][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 341 [2023-06-21 14:31:00,248][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 341 [2023-06-21 14:31:17,994][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 342 [2023-06-21 14:31:18,975][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 342 [2023-06-21 14:31:20,300][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 342 [2023-06-21 14:31:38,009][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 343 [2023-06-21 14:31:39,034][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 343 [2023-06-21 14:31:40,326][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 343 [2023-06-21 14:31:58,027][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 344 [2023-06-21 14:31:59,055][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 344 [2023-06-21 14:32:00,369][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 344 [2023-06-21 14:32:18,064][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 345 [2023-06-21 14:32:19,112][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 345 [2023-06-21 14:32:20,414][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 345 [2023-06-21 14:32:38,091][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 346 [2023-06-21 14:32:39,122][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 346 [2023-06-21 14:32:40,448][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 346 [2023-06-21 14:32:58,148][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 347 [2023-06-21 14:32:59,170][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 347 [2023-06-21 14:33:00,451][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 347 [2023-06-21 14:33:18,197][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 348 [2023-06-21 14:33:19,204][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 348 [2023-06-21 14:33:20,505][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 348 [2023-06-21 14:33:38,239][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 349 [2023-06-21 14:33:39,250][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 349 [2023-06-21 14:33:40,519][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 349 [2023-06-21 14:33:58,291][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 350 [2023-06-21 14:33:59,269][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 350 [2023-06-21 14:34:00,541][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 350 [2023-06-21 14:34:18,333][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 351 [2023-06-21 14:34:19,274][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 351 [2023-06-21 14:34:20,589][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 351 [2023-06-21 14:34:38,357][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 352 [2023-06-21 14:34:39,299][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 352 [2023-06-21 14:34:40,623][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 352 [2023-06-21 14:34:58,384][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 353 [2023-06-21 14:34:59,333][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 353 [2023-06-21 14:35:00,656][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 353 [2023-06-21 14:35:18,390][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 354 [2023-06-21 14:35:19,347][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 354 [2023-06-21 14:35:20,671][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 354 [2023-06-21 14:35:38,409][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 355 [2023-06-21 14:35:39,406][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 355 [2023-06-21 14:35:40,721][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 355 [2023-06-21 14:35:58,432][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 356 [2023-06-21 14:35:59,443][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 356 [2023-06-21 14:36:00,769][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 356 [2023-06-21 14:36:18,443][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 357 [2023-06-21 14:36:19,473][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 357 [2023-06-21 14:36:20,793][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 357 [2023-06-21 14:36:38,446][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 358 [2023-06-21 14:36:39,530][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 358 [2023-06-21 14:36:40,808][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 358 [2023-06-21 14:36:58,455][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 359 [2023-06-21 14:36:59,586][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 359 [2023-06-21 14:37:00,838][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 359 [2023-06-21 14:37:18,491][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 360 [2023-06-21 14:37:19,636][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 360 [2023-06-21 14:37:20,868][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 360 [2023-06-21 14:37:38,508][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 361 [2023-06-21 14:37:39,640][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 361 [2023-06-21 14:37:40,879][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 361 [2023-06-21 14:37:58,514][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 362 [2023-06-21 14:37:59,648][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 362 [2023-06-21 14:38:00,900][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 362 [2023-06-21 14:38:18,564][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 363 [2023-06-21 14:38:19,691][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 363 [2023-06-21 14:38:20,938][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 363 [2023-06-21 14:38:38,621][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 364 [2023-06-21 14:38:39,723][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 364 [2023-06-21 14:38:40,964][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 364 [2023-06-21 14:38:58,630][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 365 [2023-06-21 14:38:59,735][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 365 [2023-06-21 14:39:00,968][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 365 [2023-06-21 14:39:18,632][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 366 [2023-06-21 14:39:19,783][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 366 [2023-06-21 14:39:20,984][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 366 [2023-06-21 14:39:38,687][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 367 [2023-06-21 14:39:39,830][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 367 [2023-06-21 14:39:41,004][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 367 [2023-06-21 14:39:58,728][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 368 [2023-06-21 14:39:59,860][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 368 [2023-06-21 14:40:01,051][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 368 [2023-06-21 14:40:18,784][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 369 [2023-06-21 14:40:19,910][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 369 [2023-06-21 14:40:21,106][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 369 [2023-06-21 14:40:38,829][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 370 [2023-06-21 14:40:39,965][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 370 [2023-06-21 14:40:41,136][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 370 [2023-06-21 14:40:58,846][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 371 [2023-06-21 14:40:59,984][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 371 [2023-06-21 14:41:01,166][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 371 [2023-06-21 14:41:18,852][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 372 [2023-06-21 14:41:20,000][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 372 [2023-06-21 14:41:21,194][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 372 [2023-06-21 14:41:38,879][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 373 [2023-06-21 14:41:40,037][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 373 [2023-06-21 14:41:41,242][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 373 [2023-06-21 14:41:58,882][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 374 [2023-06-21 14:42:00,077][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 374 [2023-06-21 14:42:01,292][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 374 [2023-06-21 14:42:18,934][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 375 [2023-06-21 14:42:20,127][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 375 [2023-06-21 14:42:21,333][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 375 [2023-06-21 14:42:38,957][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 376 [2023-06-21 14:42:40,180][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 376 [2023-06-21 14:42:41,355][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 376 [2023-06-21 14:42:58,964][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 377 [2023-06-21 14:43:00,216][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 377 [2023-06-21 14:43:01,389][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 377 [2023-06-21 14:43:18,976][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 378 [2023-06-21 14:43:20,274][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 378 [2023-06-21 14:43:21,415][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 378 [2023-06-21 14:43:39,016][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 379 [2023-06-21 14:43:40,306][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 379 [2023-06-21 14:43:41,427][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 379 [2023-06-21 14:43:59,018][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 380 [2023-06-21 14:44:00,360][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 380 [2023-06-21 14:44:01,474][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 380 [2023-06-21 14:44:19,065][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 381 [2023-06-21 14:44:20,413][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 381 [2023-06-21 14:44:21,486][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 381 [2023-06-21 14:44:39,088][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 382 [2023-06-21 14:44:40,470][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 382 [2023-06-21 14:44:41,521][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 382 [2023-06-21 14:44:59,090][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 383 [2023-06-21 14:45:00,523][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 383 [2023-06-21 14:45:01,572][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 383 [2023-06-21 14:45:19,135][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 384 [2023-06-21 14:45:20,541][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 384 [2023-06-21 14:45:21,619][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 384 [2023-06-21 14:45:39,139][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 385 [2023-06-21 14:45:40,554][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 385 [2023-06-21 14:45:41,661][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 385 [2023-06-21 14:45:59,151][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 386 [2023-06-21 14:46:00,561][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 386 [2023-06-21 14:46:01,690][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 386 [2023-06-21 14:46:19,182][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 387 [2023-06-21 14:46:20,604][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 387 [2023-06-21 14:46:21,692][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 387 [2023-06-21 14:46:39,189][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 388 [2023-06-21 14:46:40,623][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 388 [2023-06-21 14:46:41,696][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 388 [2023-06-21 14:46:59,223][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 389 [2023-06-21 14:47:00,685][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 389 [2023-06-21 14:47:01,719][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 389 [2023-06-21 14:47:19,235][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 390 [2023-06-21 14:47:20,708][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 390 [2023-06-21 14:47:21,725][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 390 [2023-06-21 14:47:39,239][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 391 [2023-06-21 14:47:40,727][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 391 [2023-06-21 14:47:41,734][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 391 [2023-06-21 14:47:59,261][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 392 [2023-06-21 14:48:00,770][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 392 [2023-06-21 14:48:01,756][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 392 [2023-06-21 14:48:19,272][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 393 [2023-06-21 14:48:20,797][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 393 [2023-06-21 14:48:21,798][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 393 [2023-06-21 14:48:39,285][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 394 [2023-06-21 14:48:40,822][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 394 [2023-06-21 14:48:41,842][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 394 [2023-06-21 14:48:59,297][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 395 [2023-06-21 14:49:00,866][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 395 [2023-06-21 14:49:01,860][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 395 [2023-06-21 14:49:19,302][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 396 [2023-06-21 14:49:20,911][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 396 [2023-06-21 14:49:21,914][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 396 [2023-06-21 14:49:39,340][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 397 [2023-06-21 14:49:40,950][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 397 [2023-06-21 14:49:41,968][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 397 [2023-06-21 14:49:59,367][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 398 [2023-06-21 14:50:00,998][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 398 [2023-06-21 14:50:02,018][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 398 [2023-06-21 14:50:19,416][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 399 [2023-06-21 14:50:21,048][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 399 [2023-06-21 14:50:22,031][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 399 [2023-06-21 14:50:39,430][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 400 [2023-06-21 14:50:41,087][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 400 [2023-06-21 14:50:42,040][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 400 [2023-06-21 14:50:59,442][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 401 [2023-06-21 14:51:01,136][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 401 [2023-06-21 14:51:02,073][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 401 [2023-06-21 14:51:19,482][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 402 [2023-06-21 14:51:21,178][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 402 [2023-06-21 14:51:22,090][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 402 [2023-06-21 14:51:39,512][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 403 [2023-06-21 14:51:41,219][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 403 [2023-06-21 14:51:42,117][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 403 [2023-06-21 14:51:59,564][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 404 [2023-06-21 14:52:01,230][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 404 [2023-06-21 14:52:02,174][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 404 [2023-06-21 14:52:19,614][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 405 [2023-06-21 14:52:21,268][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 405 [2023-06-21 14:52:22,225][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 405 [2023-06-21 14:52:39,640][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 406 [2023-06-21 14:52:41,288][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 406 [2023-06-21 14:52:42,246][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 406 [2023-06-21 14:52:59,662][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 407 [2023-06-21 14:53:01,290][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 407 [2023-06-21 14:53:02,247][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 407 [2023-06-21 14:53:19,687][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 408 [2023-06-21 14:53:21,295][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 408 [2023-06-21 14:53:22,289][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 408 [2023-06-21 14:53:39,714][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 409 [2023-06-21 14:53:41,341][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 409 [2023-06-21 14:53:42,323][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 409 [2023-06-21 14:53:59,739][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 410 [2023-06-21 14:54:01,365][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 410 [2023-06-21 14:54:02,350][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 410 [2023-06-21 14:54:19,774][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 411 [2023-06-21 14:54:21,408][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 411 [2023-06-21 14:54:22,408][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 411 [2023-06-21 14:54:39,826][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 412 [2023-06-21 14:54:41,423][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 412 [2023-06-21 14:54:42,452][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 412 [2023-06-21 14:54:59,875][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 413 [2023-06-21 14:55:01,441][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 413 [2023-06-21 14:55:02,498][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 413 [2023-06-21 14:55:19,888][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 414 [2023-06-21 14:55:21,462][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 414 [2023-06-21 14:55:22,541][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 414 [2023-06-21 14:55:39,945][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 415 [2023-06-21 14:55:41,473][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 415 [2023-06-21 14:55:42,569][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 415 [2023-06-21 14:55:59,964][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 416 [2023-06-21 14:56:01,490][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 416 [2023-06-21 14:56:02,595][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 416 [2023-06-21 14:56:20,011][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 417 [2023-06-21 14:56:21,508][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 417 [2023-06-21 14:56:22,649][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 417 [2023-06-21 14:56:40,041][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 418 [2023-06-21 14:56:41,526][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 418 [2023-06-21 14:56:42,667][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 418 [2023-06-21 14:57:00,052][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 419 [2023-06-21 14:57:01,568][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 419 [2023-06-21 14:57:02,689][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 419 [2023-06-21 14:57:20,069][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 420 [2023-06-21 14:57:21,601][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 420 [2023-06-21 14:57:22,714][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 420 [2023-06-21 14:57:40,112][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 421 [2023-06-21 14:57:41,647][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 421 [2023-06-21 14:57:42,737][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 421 [2023-06-21 14:58:00,137][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 422 [2023-06-21 14:58:01,675][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 422 [2023-06-21 14:58:02,741][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 422 [2023-06-21 14:58:20,164][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 423 [2023-06-21 14:58:21,687][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 423 [2023-06-21 14:58:22,760][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 423 [2023-06-21 14:58:40,201][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 424 [2023-06-21 14:58:41,722][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 424 [2023-06-21 14:58:42,793][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 424 [2023-06-21 14:59:00,236][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 425 [2023-06-21 14:59:01,756][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 425 [2023-06-21 14:59:02,837][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 425 [2023-06-21 14:59:20,256][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 426 [2023-06-21 14:59:21,779][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 426 [2023-06-21 14:59:22,856][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 426 [2023-06-21 14:59:40,261][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 427 [2023-06-21 14:59:41,834][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 427 [2023-06-21 14:59:42,911][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 427 [2023-06-21 15:00:00,299][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 428 [2023-06-21 15:00:01,890][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 428 [2023-06-21 15:00:02,948][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 428 [2023-06-21 15:00:20,332][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 429 [2023-06-21 15:00:21,917][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 429 [2023-06-21 15:00:22,978][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 429 [2023-06-21 15:00:40,379][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 430 [2023-06-21 15:00:41,924][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 430 [2023-06-21 15:00:42,994][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 430 [2023-06-21 15:01:00,387][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 431 [2023-06-21 15:01:01,977][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 431 [2023-06-21 15:01:03,038][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 431 [2023-06-21 15:01:20,418][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 432 [2023-06-21 15:01:21,990][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 432 [2023-06-21 15:01:23,075][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 432 [2023-06-21 15:01:40,433][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 433 [2023-06-21 15:01:42,025][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 433 [2023-06-21 15:01:43,119][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 433 [2023-06-21 15:02:00,442][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 434 [2023-06-21 15:02:02,068][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 434 [2023-06-21 15:02:03,131][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 434 [2023-06-21 15:02:20,499][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 435 [2023-06-21 15:02:22,095][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 435 [2023-06-21 15:02:23,172][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 435 [2023-06-21 15:02:40,524][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 436 [2023-06-21 15:02:42,130][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 436 [2023-06-21 15:02:43,200][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 436 [2023-06-21 15:03:00,542][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 437 [2023-06-21 15:03:02,138][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 437 [2023-06-21 15:03:03,211][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 437 [2023-06-21 15:03:20,554][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 438 [2023-06-21 15:03:22,156][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 438 [2023-06-21 15:03:23,267][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 438 [2023-06-21 15:03:40,591][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 439 [2023-06-21 15:03:42,200][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 439 [2023-06-21 15:03:43,268][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 439 [2023-06-21 15:04:00,647][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 440 [2023-06-21 15:04:02,204][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 440 [2023-06-21 15:04:03,300][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 440 [2023-06-21 15:04:20,683][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 441 [2023-06-21 15:04:22,235][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 441 [2023-06-21 15:04:23,322][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 441 [2023-06-21 15:04:40,737][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 442 [2023-06-21 15:04:42,263][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 442 [2023-06-21 15:04:43,380][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 442 [2023-06-21 15:05:00,775][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 443 [2023-06-21 15:05:02,296][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 443 [2023-06-21 15:05:03,388][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 443 [2023-06-21 15:05:20,815][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 444 [2023-06-21 15:05:22,350][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 444 [2023-06-21 15:05:23,390][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 444 [2023-06-21 15:05:40,821][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 445 [2023-06-21 15:05:42,403][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 445 [2023-06-21 15:05:43,435][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 445 [2023-06-21 15:06:00,849][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 446 [2023-06-21 15:06:02,419][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 446 [2023-06-21 15:06:03,469][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 446 [2023-06-21 15:06:20,901][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 447 [2023-06-21 15:06:22,473][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 447 [2023-06-21 15:06:23,516][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 447 [2023-06-21 15:06:40,918][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 448 [2023-06-21 15:06:42,514][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 448 [2023-06-21 15:06:43,537][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 448 [2023-06-21 15:07:00,977][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 449 [2023-06-21 15:07:02,567][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 449 [2023-06-21 15:07:03,587][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 449 [2023-06-21 15:07:20,984][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 450 [2023-06-21 15:07:22,597][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 450 [2023-06-21 15:07:23,594][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 450 [2023-06-21 15:07:41,014][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 451 [2023-06-21 15:07:42,618][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 451 [2023-06-21 15:07:43,610][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 451 [2023-06-21 15:08:01,068][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 452 [2023-06-21 15:08:02,644][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 452 [2023-06-21 15:08:03,611][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 452 [2023-06-21 15:08:21,086][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 453 [2023-06-21 15:08:22,690][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 453 [2023-06-21 15:08:23,664][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 453 [2023-06-21 15:08:41,105][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 454 [2023-06-21 15:08:42,692][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 454 [2023-06-21 15:08:43,693][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 454 [2023-06-21 15:09:01,132][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 455 [2023-06-21 15:09:02,713][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 455 [2023-06-21 15:09:03,737][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 455 [2023-06-21 15:09:21,156][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 456 [2023-06-21 15:09:22,724][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 456 [2023-06-21 15:09:23,755][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 456 [2023-06-21 15:09:41,179][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 457 [2023-06-21 15:09:42,762][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 457 [2023-06-21 15:09:43,775][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 457 [2023-06-21 15:10:01,199][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 458 [2023-06-21 15:10:02,804][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 458 [2023-06-21 15:10:03,787][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 458 [2023-06-21 15:10:21,212][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 459 [2023-06-21 15:10:22,849][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 459 [2023-06-21 15:10:23,858][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 459 [2023-06-21 15:10:41,277][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 460 [2023-06-21 15:10:42,908][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 460 [2023-06-21 15:10:43,889][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 460 [2023-06-21 15:11:01,326][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 461 [2023-06-21 15:11:02,964][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 461 [2023-06-21 15:11:03,906][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 461 [2023-06-21 15:11:21,374][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 462 [2023-06-21 15:11:22,975][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 462 [2023-06-21 15:11:23,950][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 462 [2023-06-21 15:11:41,417][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 463 [2023-06-21 15:11:42,987][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 463 [2023-06-21 15:11:43,969][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 463 [2023-06-21 15:12:01,425][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 464 [2023-06-21 15:12:03,032][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 464 [2023-06-21 15:12:03,978][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 464 [2023-06-21 15:12:21,457][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 465 [2023-06-21 15:12:23,073][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 465 [2023-06-21 15:12:24,015][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 465 [2023-06-21 15:12:41,486][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 466 [2023-06-21 15:12:43,091][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 466 [2023-06-21 15:12:44,024][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 466 [2023-06-21 15:13:01,504][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 467 [2023-06-21 15:13:03,116][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 467 [2023-06-21 15:13:04,047][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 467 [2023-06-21 15:13:21,510][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 468 [2023-06-21 15:13:23,152][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 468 [2023-06-21 15:13:24,094][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 468 [2023-06-21 15:13:41,536][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 469 [2023-06-21 15:13:43,187][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 469 [2023-06-21 15:13:44,106][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 469 [2023-06-21 15:14:01,547][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 470 [2023-06-21 15:14:03,216][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 470 [2023-06-21 15:14:04,143][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 470 [2023-06-21 15:14:21,575][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 471 [2023-06-21 15:14:23,219][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 471 [2023-06-21 15:14:24,172][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 471 [2023-06-21 15:14:41,597][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 472 [2023-06-21 15:14:43,254][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 472 [2023-06-21 15:14:44,228][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 472 [2023-06-21 15:15:01,621][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 473 [2023-06-21 15:15:03,262][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 473 [2023-06-21 15:15:04,261][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 473 [2023-06-21 15:15:21,665][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 474 [2023-06-21 15:15:23,268][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 474 [2023-06-21 15:15:24,277][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 474 [2023-06-21 15:15:41,703][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 475 [2023-06-21 15:15:43,275][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 475 [2023-06-21 15:15:44,291][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 475 [2023-06-21 15:16:01,730][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 476 [2023-06-21 15:16:03,304][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 476 [2023-06-21 15:16:04,310][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 476 [2023-06-21 15:16:21,779][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 477 [2023-06-21 15:16:23,314][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 477 [2023-06-21 15:16:24,344][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 477 [2023-06-21 15:16:41,821][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 478 [2023-06-21 15:16:43,352][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 478 [2023-06-21 15:16:44,378][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 478 [2023-06-21 15:17:01,828][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 479 [2023-06-21 15:17:03,355][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 479 [2023-06-21 15:17:04,378][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 479 [2023-06-21 15:17:21,859][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 480 [2023-06-21 15:17:23,390][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 480 [2023-06-21 15:17:24,387][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 480 [2023-06-21 15:17:41,912][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 481 [2023-06-21 15:17:43,436][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 481 [2023-06-21 15:17:44,401][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 481 [2023-06-21 15:18:01,962][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 482 [2023-06-21 15:18:03,480][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 482 [2023-06-21 15:18:04,452][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 482 [2023-06-21 15:18:21,971][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 483 [2023-06-21 15:18:23,529][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 483 [2023-06-21 15:18:24,466][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 483 [2023-06-21 15:18:42,013][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 484 [2023-06-21 15:18:43,533][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 484 [2023-06-21 15:18:44,518][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 484 [2023-06-21 15:19:02,065][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 485 [2023-06-21 15:19:03,591][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 485 [2023-06-21 15:19:04,558][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 485 [2023-06-21 15:19:22,086][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 486 [2023-06-21 15:19:23,603][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 486 [2023-06-21 15:19:24,585][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 486 [2023-06-21 15:19:42,103][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 487 [2023-06-21 15:19:43,637][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 487 [2023-06-21 15:19:44,605][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 487 [2023-06-21 15:20:02,119][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 488 [2023-06-21 15:20:03,667][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 488 [2023-06-21 15:20:04,659][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 488 [2023-06-21 15:20:22,158][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 489 [2023-06-21 15:20:23,712][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 489 [2023-06-21 15:20:24,677][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 489 [2023-06-21 15:20:42,186][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 490 [2023-06-21 15:20:43,735][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 490 [2023-06-21 15:20:44,705][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 490 [2023-06-21 15:21:02,199][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 491 [2023-06-21 15:21:03,746][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 491 [2023-06-21 15:21:04,761][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 491 [2023-06-21 15:21:22,219][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 492 [2023-06-21 15:21:23,761][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 492 [2023-06-21 15:21:24,822][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 492 [2023-06-21 15:21:42,264][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 493 [2023-06-21 15:21:43,793][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 493 [2023-06-21 15:21:44,867][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 493 [2023-06-21 15:22:02,303][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 494 [2023-06-21 15:22:03,830][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 494 [2023-06-21 15:22:04,903][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 494 [2023-06-21 15:22:22,307][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 495 [2023-06-21 15:22:23,880][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 495 [2023-06-21 15:22:24,940][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 495 [2023-06-21 15:22:42,351][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 496 [2023-06-21 15:22:43,885][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 496 [2023-06-21 15:22:44,982][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 496 [2023-06-21 15:23:02,368][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 497 [2023-06-21 15:23:03,931][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 497 [2023-06-21 15:23:04,989][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 497 [2023-06-21 15:23:22,396][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 498 [2023-06-21 15:23:23,960][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 498 [2023-06-21 15:23:25,014][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 498 [2023-06-21 15:23:42,430][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 499 [2023-06-21 15:23:43,975][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 499 [2023-06-21 15:23:45,025][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 499 [2023-06-21 15:24:02,451][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 500 [2023-06-21 15:24:03,976][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 500 [2023-06-21 15:24:05,029][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 500 [2023-06-21 15:24:22,454][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 501 [2023-06-21 15:24:24,012][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 501 [2023-06-21 15:24:25,042][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 501 [2023-06-21 15:24:42,493][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 502 [2023-06-21 15:24:44,060][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 502 [2023-06-21 15:24:45,047][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 502 [2023-06-21 15:25:02,518][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 503 [2023-06-21 15:25:04,117][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 503 [2023-06-21 15:25:05,078][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 503 [2023-06-21 15:25:22,548][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 504 [2023-06-21 15:25:24,131][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 504 [2023-06-21 15:25:25,091][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 504 [2023-06-21 15:25:42,607][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 505 [2023-06-21 15:25:44,168][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 505 [2023-06-21 15:25:45,113][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 505 [2023-06-21 15:26:02,621][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 506 [2023-06-21 15:26:04,201][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 506 [2023-06-21 15:26:05,168][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 506 [2023-06-21 15:26:22,650][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 507 [2023-06-21 15:26:24,230][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 507 [2023-06-21 15:26:25,206][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 507 [2023-06-21 15:26:42,683][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 508 [2023-06-21 15:26:44,283][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 508 [2023-06-21 15:26:45,246][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 508 [2023-06-21 15:27:02,716][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 509 [2023-06-21 15:27:04,298][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 509 [2023-06-21 15:27:05,268][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 509 [2023-06-21 15:27:22,776][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 510 [2023-06-21 15:27:24,320][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 510 [2023-06-21 15:27:25,285][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 510 [2023-06-21 15:27:42,830][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 511 [2023-06-21 15:27:44,336][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 511 [2023-06-21 15:27:45,301][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 511 [2023-06-21 15:28:02,865][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 512 [2023-06-21 15:28:04,391][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 512 [2023-06-21 15:28:05,306][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 512 [2023-06-21 15:28:22,880][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 513 [2023-06-21 15:28:24,402][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 513 [2023-06-21 15:28:25,332][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 513 [2023-06-21 15:28:42,902][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 514 [2023-06-21 15:28:44,420][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 514 [2023-06-21 15:28:45,379][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 514 [2023-06-21 15:29:02,953][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 515 [2023-06-21 15:29:04,426][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 515 [2023-06-21 15:29:05,396][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 515 [2023-06-21 15:29:22,996][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 516 [2023-06-21 15:29:24,429][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 516 [2023-06-21 15:29:25,425][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 516 [2023-06-21 15:29:43,004][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 517 [2023-06-21 15:29:44,436][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 517 [2023-06-21 15:29:45,446][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 517 [2023-06-21 15:30:03,021][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 518 [2023-06-21 15:30:04,444][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 518 [2023-06-21 15:30:05,477][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 518 [2023-06-21 15:30:23,047][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 519 [2023-06-21 15:30:24,446][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 519 [2023-06-21 15:30:25,514][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 519 [2023-06-21 15:30:43,066][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 520 [2023-06-21 15:30:44,486][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 520 [2023-06-21 15:30:45,523][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 520 [2023-06-21 15:31:03,090][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 521 [2023-06-21 15:31:04,525][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 521 [2023-06-21 15:31:05,554][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 521 [2023-06-21 15:31:23,092][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 522 [2023-06-21 15:31:24,532][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 522 [2023-06-21 15:31:25,613][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 522 [2023-06-21 15:31:43,117][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 523 [2023-06-21 15:31:44,553][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 523 [2023-06-21 15:31:45,674][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 523 [2023-06-21 15:32:03,120][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 524 [2023-06-21 15:32:04,562][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 524 [2023-06-21 15:32:05,701][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 524 [2023-06-21 15:32:23,157][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 525 [2023-06-21 15:32:24,586][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 525 [2023-06-21 15:32:25,741][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 525 [2023-06-21 15:32:43,176][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 526 [2023-06-21 15:32:44,631][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 526 [2023-06-21 15:32:45,746][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 526 [2023-06-21 15:33:03,225][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 527 [2023-06-21 15:33:04,670][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 527 [2023-06-21 15:33:05,766][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 527 [2023-06-21 15:33:23,284][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 528 [2023-06-21 15:33:24,690][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 528 [2023-06-21 15:33:25,821][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 528 [2023-06-21 15:33:43,315][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 529 [2023-06-21 15:33:44,723][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 529 [2023-06-21 15:33:45,842][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 529 [2023-06-21 15:34:03,322][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 530 [2023-06-21 15:34:04,772][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 530 [2023-06-21 15:34:05,851][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 530 [2023-06-21 15:34:23,328][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 531 [2023-06-21 15:34:24,813][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 531 [2023-06-21 15:34:25,898][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 531 [2023-06-21 15:34:43,336][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 532 [2023-06-21 15:34:44,831][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 532 [2023-06-21 15:34:45,950][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 532 [2023-06-21 15:35:03,386][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 533 [2023-06-21 15:35:04,864][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 533 [2023-06-21 15:35:05,964][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 533 [2023-06-21 15:35:23,413][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 534 [2023-06-21 15:35:24,914][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 534 [2023-06-21 15:35:25,997][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 534 [2023-06-21 15:35:43,458][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 535 [2023-06-21 15:35:44,950][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 535 [2023-06-21 15:35:46,057][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 535 [2023-06-21 15:36:03,489][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 536 [2023-06-21 15:36:04,990][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 536 [2023-06-21 15:36:06,072][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 536 [2023-06-21 15:36:23,541][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 537 [2023-06-21 15:36:25,035][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 537 [2023-06-21 15:36:26,127][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 537 [2023-06-21 15:36:43,570][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 538 [2023-06-21 15:36:45,044][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 538 [2023-06-21 15:36:46,166][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 538 [2023-06-21 15:37:03,595][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 539 [2023-06-21 15:37:05,077][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 539 [2023-06-21 15:37:06,183][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 539 [2023-06-21 15:37:23,618][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 540 [2023-06-21 15:37:25,105][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 540 [2023-06-21 15:37:26,212][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 540 [2023-06-21 15:37:43,634][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 541 [2023-06-21 15:37:45,148][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 541 [2023-06-21 15:37:46,236][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 541 [2023-06-21 15:38:03,660][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 542 [2023-06-21 15:38:05,189][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 542 [2023-06-21 15:38:06,283][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 542 [2023-06-21 15:38:23,702][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 543 [2023-06-21 15:38:25,240][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 543 [2023-06-21 15:38:26,321][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 543 [2023-06-21 15:38:43,755][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 544 [2023-06-21 15:38:45,289][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 544 [2023-06-21 15:38:46,349][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 544 [2023-06-21 15:39:03,768][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 545 [2023-06-21 15:39:05,349][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 545 [2023-06-21 15:39:06,367][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 545 [2023-06-21 15:39:23,777][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 546 [2023-06-21 15:39:25,399][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 546 [2023-06-21 15:39:26,398][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 546 [2023-06-21 15:39:43,819][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 547 [2023-06-21 15:39:45,430][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 547 [2023-06-21 15:39:46,401][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 547 [2023-06-21 15:40:03,855][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 548 [2023-06-21 15:40:05,440][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 548 [2023-06-21 15:40:06,403][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 548 [2023-06-21 15:40:23,910][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 549 [2023-06-21 15:40:25,468][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 549 [2023-06-21 15:40:26,453][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 549 [2023-06-21 15:40:43,961][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 550 [2023-06-21 15:40:45,478][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 550 [2023-06-21 15:40:46,504][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 550 [2023-06-21 15:41:03,990][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 551 [2023-06-21 15:41:05,497][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 551 [2023-06-21 15:41:06,513][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 551 [2023-06-21 15:41:24,043][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 552 [2023-06-21 15:41:25,546][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 552 [2023-06-21 15:41:26,521][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 552 [2023-06-21 15:41:44,071][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 553 [2023-06-21 15:41:45,549][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 553 [2023-06-21 15:41:46,523][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 553 [2023-06-21 15:42:04,105][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 554 [2023-06-21 15:42:05,605][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 554 [2023-06-21 15:42:06,562][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 554 [2023-06-21 15:42:24,139][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 555 [2023-06-21 15:42:25,654][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 555 [2023-06-21 15:42:26,569][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 555 [2023-06-21 15:42:44,181][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 556 [2023-06-21 15:42:45,703][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 556 [2023-06-21 15:42:46,602][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 556 [2023-06-21 15:43:04,237][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 557 [2023-06-21 15:43:05,747][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 557 [2023-06-21 15:43:06,631][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 557 [2023-06-21 15:43:24,262][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 558 [2023-06-21 15:43:25,767][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 558 [2023-06-21 15:43:26,683][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 558 [2023-06-21 15:43:44,284][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 559 [2023-06-21 15:43:45,771][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 559 [2023-06-21 15:43:46,691][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 559 [2023-06-21 15:44:04,333][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 560 [2023-06-21 15:44:05,794][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 560 [2023-06-21 15:44:06,732][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 560 [2023-06-21 15:44:24,376][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 561 [2023-06-21 15:44:25,805][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 561 [2023-06-21 15:44:26,773][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 561 [2023-06-21 15:44:44,396][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 562 [2023-06-21 15:44:45,846][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 562 [2023-06-21 15:44:46,824][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 562 [2023-06-21 15:45:04,412][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 563 [2023-06-21 15:45:05,860][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 563 [2023-06-21 15:45:06,876][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 563 [2023-06-21 15:45:24,461][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 564 [2023-06-21 15:45:25,877][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 564 [2023-06-21 15:45:26,881][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 564 [2023-06-21 15:45:44,486][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 565 [2023-06-21 15:45:45,879][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 565 [2023-06-21 15:45:46,931][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 565 [2023-06-21 15:46:04,489][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 566 [2023-06-21 15:46:05,895][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 566 [2023-06-21 15:46:06,939][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 566 [2023-06-21 15:46:24,512][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 567 [2023-06-21 15:46:25,932][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 567 [2023-06-21 15:46:26,994][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 567 [2023-06-21 15:46:44,515][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 568 [2023-06-21 15:46:45,969][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 568 [2023-06-21 15:46:47,052][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 568 [2023-06-21 15:47:04,529][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 569 [2023-06-21 15:47:06,016][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 569 [2023-06-21 15:47:07,067][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 569 [2023-06-21 15:47:24,574][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 570 [2023-06-21 15:47:26,062][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 570 [2023-06-21 15:47:27,113][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 570 [2023-06-21 15:47:44,589][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 571 [2023-06-21 15:47:46,121][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 571 [2023-06-21 15:47:47,122][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 571 [2023-06-21 15:48:04,639][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 572 [2023-06-21 15:48:06,165][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 572 [2023-06-21 15:48:07,171][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 572 [2023-06-21 15:48:24,656][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 573 [2023-06-21 15:48:26,174][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 573 [2023-06-21 15:48:27,178][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 573 [2023-06-21 15:48:44,696][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 574 [2023-06-21 15:48:46,229][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 574 [2023-06-21 15:48:47,189][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 574 [2023-06-21 15:49:04,707][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 575 [2023-06-21 15:49:06,243][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 575 [2023-06-21 15:49:07,238][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 575 [2023-06-21 15:49:24,719][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 576 [2023-06-21 15:49:26,274][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 576 [2023-06-21 15:49:27,271][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 576 [2023-06-21 15:49:44,779][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 577 [2023-06-21 15:49:46,310][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 577 [2023-06-21 15:49:47,319][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 577 [2023-06-21 15:50:04,825][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 578 [2023-06-21 15:50:06,372][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 578 [2023-06-21 15:50:07,333][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 578 [2023-06-21 15:50:24,859][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 579 [2023-06-21 15:50:26,379][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 579 [2023-06-21 15:50:27,356][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 579 [2023-06-21 15:50:44,896][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 580 [2023-06-21 15:50:46,384][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 580 [2023-06-21 15:50:47,401][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 580 [2023-06-21 15:51:04,916][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 581 [2023-06-21 15:51:06,405][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 581 [2023-06-21 15:51:07,439][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 581 [2023-06-21 15:51:24,924][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 582 [2023-06-21 15:51:26,465][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 582 [2023-06-21 15:51:27,464][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 582 [2023-06-21 15:51:44,974][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 583 [2023-06-21 15:51:46,500][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 583 [2023-06-21 15:51:47,510][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 583 [2023-06-21 15:52:04,987][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 584 [2023-06-21 15:52:06,539][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 584 [2023-06-21 15:52:07,560][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 584 [2023-06-21 15:52:25,005][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 585 [2023-06-21 15:52:26,558][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 585 [2023-06-21 15:52:27,590][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 585 [2023-06-21 15:52:45,061][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 586 [2023-06-21 15:52:46,561][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 586 [2023-06-21 15:52:47,595][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 586 [2023-06-21 15:53:05,108][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 587 [2023-06-21 15:53:06,612][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 587 [2023-06-21 15:53:07,601][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 587 [2023-06-21 15:53:25,119][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 588 [2023-06-21 15:53:26,659][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 588 [2023-06-21 15:53:27,617][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 588 [2023-06-21 15:53:45,143][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 589 [2023-06-21 15:53:46,676][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 589 [2023-06-21 15:53:47,653][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 589 [2023-06-21 15:54:05,157][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 590 [2023-06-21 15:54:06,687][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 590 [2023-06-21 15:54:07,657][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 590 [2023-06-21 15:54:25,180][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 591 [2023-06-21 15:54:26,698][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 591 [2023-06-21 15:54:27,678][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 591 [2023-06-21 15:54:45,223][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 592 [2023-06-21 15:54:46,702][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 592 [2023-06-21 15:54:47,698][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 592 [2023-06-21 15:55:05,273][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 593 [2023-06-21 15:55:06,720][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 593 [2023-06-21 15:55:07,757][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 593 [2023-06-21 15:55:25,311][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 594 [2023-06-21 15:55:26,753][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 594 [2023-06-21 15:55:27,792][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 594 [2023-06-21 15:55:45,335][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 595 [2023-06-21 15:55:46,807][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 595 [2023-06-21 15:55:47,803][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 595 [2023-06-21 15:56:05,396][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 596 [2023-06-21 15:56:06,830][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 596 [2023-06-21 15:56:07,821][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 596 [2023-06-21 15:56:25,414][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 597 [2023-06-21 15:56:26,838][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 597 [2023-06-21 15:56:27,866][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 597 [2023-06-21 15:56:45,424][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 598 [2023-06-21 15:56:46,863][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 598 [2023-06-21 15:56:47,905][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 598 [2023-06-21 15:57:05,473][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 599 [2023-06-21 15:57:06,900][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 599 [2023-06-21 15:57:07,914][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 599 [2023-06-21 15:57:25,500][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 600 [2023-06-21 15:57:26,932][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 600 [2023-06-21 15:57:27,961][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 600 [2023-06-21 15:57:45,544][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 601 [2023-06-21 15:57:46,943][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 601 [2023-06-21 15:57:47,985][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 601 [2023-06-21 15:58:05,597][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 602 [2023-06-21 15:58:07,004][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 602 [2023-06-21 15:58:08,015][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 602 [2023-06-21 15:58:25,634][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 603 [2023-06-21 15:58:27,011][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 603 [2023-06-21 15:58:28,016][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 603 [2023-06-21 15:58:45,671][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 604 [2023-06-21 15:58:47,015][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 604 [2023-06-21 15:58:48,066][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 604 [2023-06-21 15:59:05,700][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 605 [2023-06-21 15:59:07,070][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 605 [2023-06-21 15:59:08,091][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 605 [2023-06-21 15:59:25,728][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 606 [2023-06-21 15:59:27,104][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 606 [2023-06-21 15:59:28,126][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 606 [2023-06-21 15:59:45,775][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 607 [2023-06-21 15:59:47,121][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 607 [2023-06-21 15:59:48,131][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 607 [2023-06-21 16:00:05,825][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 608 [2023-06-21 16:00:07,152][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 608 [2023-06-21 16:00:08,186][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 608 [2023-06-21 16:00:25,861][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 609 [2023-06-21 16:00:27,204][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 609 [2023-06-21 16:00:28,207][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 609 [2023-06-21 16:00:45,882][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 610 [2023-06-21 16:00:47,227][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 610 [2023-06-21 16:00:48,229][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 610 [2023-06-21 16:01:05,908][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 611 [2023-06-21 16:01:07,235][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 611 [2023-06-21 16:01:08,276][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 611 [2023-06-21 16:01:25,919][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 612 [2023-06-21 16:01:27,290][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 612 [2023-06-21 16:01:28,315][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 612 [2023-06-21 16:01:45,945][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 613 [2023-06-21 16:01:47,334][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 613 [2023-06-21 16:01:48,369][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 613 [2023-06-21 16:02:06,004][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 614 [2023-06-21 16:02:07,380][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 614 [2023-06-21 16:02:08,386][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 614 [2023-06-21 16:02:26,050][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 615 [2023-06-21 16:02:27,390][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 615 [2023-06-21 16:02:28,396][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 615 [2023-06-21 16:02:46,060][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 616 [2023-06-21 16:02:47,422][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 616 [2023-06-21 16:02:48,425][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 616 [2023-06-21 16:03:06,076][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 617 [2023-06-21 16:03:07,474][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 617 [2023-06-21 16:03:08,466][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 617 [2023-06-21 16:03:26,080][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 618 [2023-06-21 16:03:27,490][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 618 [2023-06-21 16:03:28,499][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 618 [2023-06-21 16:03:46,094][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 619 [2023-06-21 16:03:47,507][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 619 [2023-06-21 16:03:48,521][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 619 [2023-06-21 16:04:06,126][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 620 [2023-06-21 16:04:07,512][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 620 [2023-06-21 16:04:08,563][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 620 [2023-06-21 16:04:26,141][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 621 [2023-06-21 16:04:27,564][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 621 [2023-06-21 16:04:28,583][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 621 [2023-06-21 16:04:46,162][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 622 [2023-06-21 16:04:47,572][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 622 [2023-06-21 16:04:48,621][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 622 [2023-06-21 16:05:06,182][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 623 [2023-06-21 16:05:07,603][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 623 [2023-06-21 16:05:08,625][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 623 [2023-06-21 16:05:26,200][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 624 [2023-06-21 16:05:27,664][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 624 [2023-06-21 16:05:28,643][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 624 [2023-06-21 16:05:46,234][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 625 [2023-06-21 16:05:47,671][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 625 [2023-06-21 16:05:48,675][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 625 [2023-06-21 16:06:06,253][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 626 [2023-06-21 16:06:07,684][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 626 [2023-06-21 16:06:08,723][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 626 [2023-06-21 16:06:26,286][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 627 [2023-06-21 16:06:27,686][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 627 [2023-06-21 16:06:28,747][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 627 [2023-06-21 16:06:46,322][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 628 [2023-06-21 16:06:47,698][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 628 [2023-06-21 16:06:48,766][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 628 [2023-06-21 16:07:06,326][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 629 [2023-06-21 16:07:07,717][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 629 [2023-06-21 16:07:08,790][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 629 [2023-06-21 16:07:26,331][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 630 [2023-06-21 16:07:27,743][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 630 [2023-06-21 16:07:28,803][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 630 [2023-06-21 16:07:46,384][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 631 [2023-06-21 16:07:47,754][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 631 [2023-06-21 16:07:48,808][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 631 [2023-06-21 16:08:06,392][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 632 [2023-06-21 16:08:07,774][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 632 [2023-06-21 16:08:08,825][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 632 [2023-06-21 16:08:26,427][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 633 [2023-06-21 16:08:27,820][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 633 [2023-06-21 16:08:28,849][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 633 [2023-06-21 16:08:46,479][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 634 [2023-06-21 16:08:47,853][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 634 [2023-06-21 16:08:48,904][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 634 [2023-06-21 16:09:06,519][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 635 [2023-06-21 16:09:07,887][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 635 [2023-06-21 16:09:08,919][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 635 [2023-06-21 16:09:26,557][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 636 [2023-06-21 16:09:27,947][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 636 [2023-06-21 16:09:28,934][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 636 [2023-06-21 16:09:46,615][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 637 [2023-06-21 16:09:47,996][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 637 [2023-06-21 16:09:48,959][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 637 [2023-06-21 16:10:06,638][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 638 [2023-06-21 16:10:08,042][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 638 [2023-06-21 16:10:09,003][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 638 [2023-06-21 16:10:26,643][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 639 [2023-06-21 16:10:28,056][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 639 [2023-06-21 16:10:29,025][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 639 [2023-06-21 16:10:46,648][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 640 [2023-06-21 16:10:48,059][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 640 [2023-06-21 16:10:49,053][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 640 [2023-06-21 16:11:06,692][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 641 [2023-06-21 16:11:08,106][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 641 [2023-06-21 16:11:09,080][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 641 [2023-06-21 16:11:26,740][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 642 [2023-06-21 16:11:28,156][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 642 [2023-06-21 16:11:29,138][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 642 [2023-06-21 16:11:46,763][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 643 [2023-06-21 16:11:48,158][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 643 [2023-06-21 16:11:49,173][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 643 [2023-06-21 16:12:06,803][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 644 [2023-06-21 16:12:08,197][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 644 [2023-06-21 16:12:09,217][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 644 [2023-06-21 16:12:26,859][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 645 [2023-06-21 16:12:28,228][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 645 [2023-06-21 16:12:29,224][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 645 [2023-06-21 16:12:46,894][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 646 [2023-06-21 16:12:48,250][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 646 [2023-06-21 16:12:49,257][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 646 [2023-06-21 16:13:06,911][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 647 [2023-06-21 16:13:08,268][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 647 [2023-06-21 16:13:09,283][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 647 [2023-06-21 16:13:26,943][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 648 [2023-06-21 16:13:28,302][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 648 [2023-06-21 16:13:29,327][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 648 [2023-06-21 16:13:46,953][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 649 [2023-06-21 16:13:48,319][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 649 [2023-06-21 16:13:49,381][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 649 [2023-06-21 16:14:06,968][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 650 [2023-06-21 16:14:08,365][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 650 [2023-06-21 16:14:09,424][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 650 [2023-06-21 16:14:26,981][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 651 [2023-06-21 16:14:28,412][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 651 [2023-06-21 16:14:29,436][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 651 [2023-06-21 16:14:47,029][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 652 [2023-06-21 16:14:48,421][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 652 [2023-06-21 16:14:49,440][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 652 [2023-06-21 16:15:07,035][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 653 [2023-06-21 16:15:08,446][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 653 [2023-06-21 16:15:09,463][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 653 [2023-06-21 16:15:27,055][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 654 [2023-06-21 16:15:28,451][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 654 [2023-06-21 16:15:29,502][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 654 [2023-06-21 16:15:47,078][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 655 [2023-06-21 16:15:48,472][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 655 [2023-06-21 16:15:49,544][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 655 [2023-06-21 16:16:07,118][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 656 [2023-06-21 16:16:08,521][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 656 [2023-06-21 16:16:09,545][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 656 [2023-06-21 16:16:27,125][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 657 [2023-06-21 16:16:28,536][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 657 [2023-06-21 16:16:29,557][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 657 [2023-06-21 16:16:47,143][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 658 [2023-06-21 16:16:48,570][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 658 [2023-06-21 16:16:49,570][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 658 [2023-06-21 16:17:07,149][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 659 [2023-06-21 16:17:08,590][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 659 [2023-06-21 16:17:09,602][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 659 [2023-06-21 16:17:27,165][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 660 [2023-06-21 16:17:28,632][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 660 [2023-06-21 16:17:29,604][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 660 [2023-06-21 16:17:47,169][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 661 [2023-06-21 16:17:48,659][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 661 [2023-06-21 16:17:49,628][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 661 [2023-06-21 16:18:07,172][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 662 [2023-06-21 16:18:08,688][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 662 [2023-06-21 16:18:09,652][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 662 [2023-06-21 16:18:27,205][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 663 [2023-06-21 16:18:28,723][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 663 [2023-06-21 16:18:29,675][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 663 [2023-06-21 16:18:47,247][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 664 [2023-06-21 16:18:48,743][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 664 [2023-06-21 16:18:49,700][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 664 [2023-06-21 16:19:07,273][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 665 [2023-06-21 16:19:08,800][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 665 [2023-06-21 16:19:09,704][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 665 [2023-06-21 16:19:27,287][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 666 [2023-06-21 16:19:28,814][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 666 [2023-06-21 16:19:29,705][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 666 [2023-06-21 16:19:47,337][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 667 [2023-06-21 16:19:48,863][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 667 [2023-06-21 16:19:49,759][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 667 [2023-06-21 16:20:07,380][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 668 [2023-06-21 16:20:08,905][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 668 [2023-06-21 16:20:09,794][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 668 [2023-06-21 16:20:27,440][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 669 [2023-06-21 16:20:28,941][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 669 [2023-06-21 16:20:29,818][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 669 [2023-06-21 16:20:47,481][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 670 [2023-06-21 16:20:48,973][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 670 [2023-06-21 16:20:49,857][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 670 [2023-06-21 16:21:07,510][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 671 [2023-06-21 16:21:08,986][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 671 [2023-06-21 16:21:09,911][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 671 [2023-06-21 16:21:27,537][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 672 [2023-06-21 16:21:29,018][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 672 [2023-06-21 16:21:29,948][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 672 [2023-06-21 16:21:47,552][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 673 [2023-06-21 16:21:49,040][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 673 [2023-06-21 16:21:49,995][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 673 [2023-06-21 16:22:07,608][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 674 [2023-06-21 16:22:09,048][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 674 [2023-06-21 16:22:10,008][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 674 [2023-06-21 16:22:27,662][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 675 [2023-06-21 16:22:29,104][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 675 [2023-06-21 16:22:30,029][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 675 [2023-06-21 16:22:47,685][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 676 [2023-06-21 16:22:49,148][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 676 [2023-06-21 16:22:50,045][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 676 [2023-06-21 16:23:07,700][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 677 [2023-06-21 16:23:09,195][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 677 [2023-06-21 16:23:10,052][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 677 [2023-06-21 16:23:27,755][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 678 [2023-06-21 16:23:29,222][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 678 [2023-06-21 16:23:30,058][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 678 [2023-06-21 16:23:47,758][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 679 [2023-06-21 16:23:49,262][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 679 [2023-06-21 16:23:50,070][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 679 [2023-06-21 16:24:07,765][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 680 [2023-06-21 16:24:09,305][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 680 [2023-06-21 16:24:10,115][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 680 [2023-06-21 16:24:27,795][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 681 [2023-06-21 16:24:29,346][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 681 [2023-06-21 16:24:30,140][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 681 [2023-06-21 16:24:47,830][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 682 [2023-06-21 16:24:49,405][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 682 [2023-06-21 16:24:50,176][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 682 [2023-06-21 16:25:07,898][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 683 [2023-06-21 16:25:09,420][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 683 [2023-06-21 16:25:10,199][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 683 [2023-06-21 16:25:27,912][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 684 [2023-06-21 16:25:29,471][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 684 [2023-06-21 16:25:30,255][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 684 [2023-06-21 16:25:47,965][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 685 [2023-06-21 16:25:49,522][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 685 [2023-06-21 16:25:50,309][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 685 [2023-06-21 16:26:08,017][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 686 [2023-06-21 16:26:09,560][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 686 [2023-06-21 16:26:10,354][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 686 [2023-06-21 16:26:28,040][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 687 [2023-06-21 16:26:29,607][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 687 [2023-06-21 16:26:30,372][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 687 [2023-06-21 16:26:48,083][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 688 [2023-06-21 16:26:49,633][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 688 [2023-06-21 16:26:50,407][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 688 [2023-06-21 16:27:08,115][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 689 [2023-06-21 16:27:09,692][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 689 [2023-06-21 16:27:10,425][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 689 [2023-06-21 16:27:28,169][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 690 [2023-06-21 16:27:29,709][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 690 [2023-06-21 16:27:30,433][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 690 [2023-06-21 16:27:48,203][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 691 [2023-06-21 16:27:49,736][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 691 [2023-06-21 16:27:50,453][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 691 [2023-06-21 16:28:08,222][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 692 [2023-06-21 16:28:09,738][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 692 [2023-06-21 16:28:10,474][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 692 [2023-06-21 16:28:28,244][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 693 [2023-06-21 16:28:29,773][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 693 [2023-06-21 16:28:30,480][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 693 [2023-06-21 16:28:48,255][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 694 [2023-06-21 16:28:49,780][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 694 [2023-06-21 16:28:50,505][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 694 [2023-06-21 16:29:08,262][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 695 [2023-06-21 16:29:09,837][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 695 [2023-06-21 16:29:10,563][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 695 [2023-06-21 16:29:28,311][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 696 [2023-06-21 16:29:29,888][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 696 [2023-06-21 16:29:30,585][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 696 [2023-06-21 16:29:48,361][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 697 [2023-06-21 16:29:49,891][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 697 [2023-06-21 16:29:50,602][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 697 [2023-06-21 16:30:08,379][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 698 [2023-06-21 16:30:09,943][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 698 [2023-06-21 16:30:10,652][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 698 [2023-06-21 16:30:28,429][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 699 [2023-06-21 16:30:29,978][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 699 [2023-06-21 16:30:30,672][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 699 [2023-06-21 16:30:48,441][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 700 [2023-06-21 16:30:50,028][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 700 [2023-06-21 16:30:50,708][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 700 [2023-06-21 16:31:08,447][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 701 [2023-06-21 16:31:10,044][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 701 [2023-06-21 16:31:10,719][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 701 [2023-06-21 16:31:28,493][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 702 [2023-06-21 16:31:30,052][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 702 [2023-06-21 16:31:30,750][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 702 [2023-06-21 16:31:48,531][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 703 [2023-06-21 16:31:50,061][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 703 [2023-06-21 16:31:50,763][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 703 [2023-06-21 16:32:08,576][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 704 [2023-06-21 16:32:10,091][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 704 [2023-06-21 16:32:10,814][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 704 [2023-06-21 16:32:28,580][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 705 [2023-06-21 16:32:30,145][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 705 [2023-06-21 16:32:30,840][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 705 [2023-06-21 16:32:48,614][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 706 [2023-06-21 16:32:50,152][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 706 [2023-06-21 16:32:50,886][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 706 [2023-06-21 16:33:08,667][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 707 [2023-06-21 16:33:10,154][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 707 [2023-06-21 16:33:10,903][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 707 [2023-06-21 16:33:28,678][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 708 [2023-06-21 16:33:30,208][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 708 [2023-06-21 16:33:30,941][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 708 [2023-06-21 16:33:48,693][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 709 [2023-06-21 16:33:50,213][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 709 [2023-06-21 16:33:50,960][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 709 [2023-06-21 16:34:08,687][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 710 [2023-06-21 16:34:10,202][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 710 [2023-06-21 16:34:10,946][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 710 [2023-06-21 16:34:28,680][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 711 [2023-06-21 16:34:30,248][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 711 [2023-06-21 16:34:30,942][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 711 [2023-06-21 16:34:48,724][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 712 [2023-06-21 16:34:50,280][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 712 [2023-06-21 16:34:50,980][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 712 [2023-06-21 16:35:08,772][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 713 [2023-06-21 16:35:10,299][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 713 [2023-06-21 16:35:11,029][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 713 [2023-06-21 16:35:28,824][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 714 [2023-06-21 16:35:30,343][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 714 [2023-06-21 16:35:31,062][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 714 [2023-06-21 16:35:48,884][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 715 [2023-06-21 16:35:50,388][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 715 [2023-06-21 16:35:51,063][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 715 [2023-06-21 16:36:08,901][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 716 [2023-06-21 16:36:10,440][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 716 [2023-06-21 16:36:11,113][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 716 [2023-06-21 16:36:28,919][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 717 [2023-06-21 16:36:30,447][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 717 [2023-06-21 16:36:31,164][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 717 [2023-06-21 16:36:48,928][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 718 [2023-06-21 16:36:50,460][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 718 [2023-06-21 16:36:51,221][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 718 [2023-06-21 16:37:08,950][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 719 [2023-06-21 16:37:10,490][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 719 [2023-06-21 16:37:11,243][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 719 [2023-06-21 16:37:29,001][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 720 [2023-06-21 16:37:30,505][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 720 [2023-06-21 16:37:31,287][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 720 [2023-06-21 16:37:49,015][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 721 [2023-06-21 16:37:50,541][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 721 [2023-06-21 16:37:51,327][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 721 [2023-06-21 16:38:09,050][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 722 [2023-06-21 16:38:10,592][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 722 [2023-06-21 16:38:11,375][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 722 [2023-06-21 16:38:29,069][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 723 [2023-06-21 16:38:30,642][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 723 [2023-06-21 16:38:31,407][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 723 [2023-06-21 16:38:49,096][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 724 [2023-06-21 16:38:50,666][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 724 [2023-06-21 16:38:51,455][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 724 [2023-06-21 16:39:09,106][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 725 [2023-06-21 16:39:10,685][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 725 [2023-06-21 16:39:11,471][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 725 [2023-06-21 16:39:29,137][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 726 [2023-06-21 16:39:30,691][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 726 [2023-06-21 16:39:31,520][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 726 [2023-06-21 16:39:49,156][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 727 [2023-06-21 16:39:50,737][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 727 [2023-06-21 16:39:51,522][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 727 [2023-06-21 16:40:09,164][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 728 [2023-06-21 16:40:10,772][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 728 [2023-06-21 16:40:11,559][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 728 [2023-06-21 16:40:29,172][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 729 [2023-06-21 16:40:30,776][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 729 [2023-06-21 16:40:31,602][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 729 [2023-06-21 16:40:49,226][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 730 [2023-06-21 16:40:50,792][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 730 [2023-06-21 16:40:51,642][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 730 [2023-06-21 16:41:09,271][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 731 [2023-06-21 16:41:10,816][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 731 [2023-06-21 16:41:11,657][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 731 [2023-06-21 16:41:29,295][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 732 [2023-06-21 16:41:30,864][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 732 [2023-06-21 16:41:31,667][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 732 [2023-06-21 16:41:49,297][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 733 [2023-06-21 16:41:50,873][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 733 [2023-06-21 16:41:51,687][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 733 [2023-06-21 16:42:09,348][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 734 [2023-06-21 16:42:10,919][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 734 [2023-06-21 16:42:11,700][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 734 [2023-06-21 16:42:29,407][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 735 [2023-06-21 16:42:30,957][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 735 [2023-06-21 16:42:31,747][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 735 [2023-06-21 16:42:49,410][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 736 [2023-06-21 16:42:50,966][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 736 [2023-06-21 16:42:51,791][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 736 [2023-06-21 16:43:09,431][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 737 [2023-06-21 16:43:10,979][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 737 [2023-06-21 16:43:11,836][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 737 [2023-06-21 16:43:29,454][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 738 [2023-06-21 16:43:30,985][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 738 [2023-06-21 16:43:31,888][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 738 [2023-06-21 16:43:49,462][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 739 [2023-06-21 16:43:51,018][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 739 [2023-06-21 16:43:51,910][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 739 [2023-06-21 16:44:09,511][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 740 [2023-06-21 16:44:11,066][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 740 [2023-06-21 16:44:11,923][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 740 [2023-06-21 16:44:29,535][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 741 [2023-06-21 16:44:31,096][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 741 [2023-06-21 16:44:31,928][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 741 [2023-06-21 16:44:49,542][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 742 [2023-06-21 16:44:51,104][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 742 [2023-06-21 16:44:51,959][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 742 [2023-06-21 16:45:09,569][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 743 [2023-06-21 16:45:11,107][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 743 [2023-06-21 16:45:11,976][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 743 [2023-06-21 16:45:29,593][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 744 [2023-06-21 16:45:31,161][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 744 [2023-06-21 16:45:32,009][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 744 [2023-06-21 16:45:49,630][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 745 [2023-06-21 16:45:51,204][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 745 [2023-06-21 16:45:52,012][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 745 [2023-06-21 16:46:09,650][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 746 [2023-06-21 16:46:11,217][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 746 [2023-06-21 16:46:12,051][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 746 [2023-06-21 16:46:29,678][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 747 [2023-06-21 16:46:31,261][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 747 [2023-06-21 16:46:32,078][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 747 [2023-06-21 16:46:49,683][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 748 [2023-06-21 16:46:51,271][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 748 [2023-06-21 16:46:52,080][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 748 [2023-06-21 16:47:09,693][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 749 [2023-06-21 16:47:11,320][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 749 [2023-06-21 16:47:12,082][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 749 [2023-06-21 16:47:29,701][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 750 [2023-06-21 16:47:31,355][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 750 [2023-06-21 16:47:32,133][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 750 [2023-06-21 16:47:49,709][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 751 [2023-06-21 16:47:51,401][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 751 [2023-06-21 16:47:52,169][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 751 [2023-06-21 16:48:09,751][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 752 [2023-06-21 16:48:11,452][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 752 [2023-06-21 16:48:12,226][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 752 [2023-06-21 16:48:29,802][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 753 [2023-06-21 16:48:31,503][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 753 [2023-06-21 16:48:32,238][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 753 [2023-06-21 16:48:49,854][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 754 [2023-06-21 16:48:51,517][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 754 [2023-06-21 16:48:52,271][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 754 [2023-06-21 16:49:09,901][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 755 [2023-06-21 16:49:11,553][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 755 [2023-06-21 16:49:12,291][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 755 [2023-06-21 16:49:29,960][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 756 [2023-06-21 16:49:31,588][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 756 [2023-06-21 16:49:32,294][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 756 [2023-06-21 16:49:50,018][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 757 [2023-06-21 16:49:51,594][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 757 [2023-06-21 16:49:52,310][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 757 [2023-06-21 16:50:10,028][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 758 [2023-06-21 16:50:11,632][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 758 [2023-06-21 16:50:12,322][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 758 [2023-06-21 16:50:30,072][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 759 [2023-06-21 16:50:31,662][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 759 [2023-06-21 16:50:32,382][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 759 [2023-06-21 16:50:50,084][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 760 [2023-06-21 16:50:51,681][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 760 [2023-06-21 16:50:52,434][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 760 [2023-06-21 16:51:10,119][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 761 [2023-06-21 16:51:11,682][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 761 [2023-06-21 16:51:12,490][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 761 [2023-06-21 16:51:30,161][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 762 [2023-06-21 16:51:31,698][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 762 [2023-06-21 16:51:32,518][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 762 [2023-06-21 16:51:50,196][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 763 [2023-06-21 16:51:51,729][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 763 [2023-06-21 16:51:52,532][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 763 [2023-06-21 16:52:10,234][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 764 [2023-06-21 16:52:11,774][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 764 [2023-06-21 16:52:12,583][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 764 [2023-06-21 16:52:30,263][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 765 [2023-06-21 16:52:31,782][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 765 [2023-06-21 16:52:32,622][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 765 [2023-06-21 16:52:50,312][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 766 [2023-06-21 16:52:51,798][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 766 [2023-06-21 16:52:52,648][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 766 [2023-06-21 16:53:10,363][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 767 [2023-06-21 16:53:11,842][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 767 [2023-06-21 16:53:12,668][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 767 [2023-06-21 16:53:30,379][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 768 [2023-06-21 16:53:31,897][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 768 [2023-06-21 16:53:32,727][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 768 [2023-06-21 16:53:50,412][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 769 [2023-06-21 16:53:51,928][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 769 [2023-06-21 16:53:52,751][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 769 [2023-06-21 16:54:10,429][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 770 [2023-06-21 16:54:11,945][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 770 [2023-06-21 16:54:12,791][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 770 [2023-06-21 16:54:30,431][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 771 [2023-06-21 16:54:31,997][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 771 [2023-06-21 16:54:32,839][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 771 [2023-06-21 16:54:50,468][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 772 [2023-06-21 16:54:52,017][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 772 [2023-06-21 16:54:52,844][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 772 [2023-06-21 16:55:10,488][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 773 [2023-06-21 16:55:12,034][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 773 [2023-06-21 16:55:12,882][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 773 [2023-06-21 16:55:30,520][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 774 [2023-06-21 16:55:32,054][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 774 [2023-06-21 16:55:32,900][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 774 [2023-06-21 16:55:50,549][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 775 [2023-06-21 16:55:52,088][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 775 [2023-06-21 16:55:52,929][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 775 [2023-06-21 16:56:10,574][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 776 [2023-06-21 16:56:12,093][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 776 [2023-06-21 16:56:12,941][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 776 [2023-06-21 16:56:30,620][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 777 [2023-06-21 16:56:32,113][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 777 [2023-06-21 16:56:32,988][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 777 [2023-06-21 16:56:50,649][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 778 [2023-06-21 16:56:52,135][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 778 [2023-06-21 16:56:53,002][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 778 [2023-06-21 16:57:10,688][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 779 [2023-06-21 16:57:12,173][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 779 [2023-06-21 16:57:13,039][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 779 [2023-06-21 16:57:30,732][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 780 [2023-06-21 16:57:32,192][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 780 [2023-06-21 16:57:33,042][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 780 [2023-06-21 16:57:50,781][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 781 [2023-06-21 16:57:52,215][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 781 [2023-06-21 16:57:53,061][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 781 [2023-06-21 16:58:10,806][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 782 [2023-06-21 16:58:12,218][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 782 [2023-06-21 16:58:13,111][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 782 [2023-06-21 16:58:30,861][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 783 [2023-06-21 16:58:32,220][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 783 [2023-06-21 16:58:33,124][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 783 [2023-06-21 16:58:50,881][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 784 [2023-06-21 16:58:52,220][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 784 [2023-06-21 16:58:53,130][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 784 [2023-06-21 16:59:10,935][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 785 [2023-06-21 16:59:12,236][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 785 [2023-06-21 16:59:13,142][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 785 [2023-06-21 16:59:30,994][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 786 [2023-06-21 16:59:32,273][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 786 [2023-06-21 16:59:33,178][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 786 [2023-06-21 16:59:51,016][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 787 [2023-06-21 16:59:52,299][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 787 [2023-06-21 16:59:53,227][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 787 [2023-06-21 17:00:11,048][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 788 [2023-06-21 17:00:12,311][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 788 [2023-06-21 17:00:13,258][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 788 [2023-06-21 17:00:31,095][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 789 [2023-06-21 17:00:32,369][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 789 [2023-06-21 17:00:33,299][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 789 [2023-06-21 17:00:51,109][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 790 [2023-06-21 17:00:52,417][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 790 [2023-06-21 17:00:53,303][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 790 [2023-06-21 17:01:11,135][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 791 [2023-06-21 17:01:12,443][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 791 [2023-06-21 17:01:13,334][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 791 [2023-06-21 17:01:31,180][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 792 [2023-06-21 17:01:32,471][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 792 [2023-06-21 17:01:33,339][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 792 [2023-06-21 17:01:51,225][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 793 [2023-06-21 17:01:52,487][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 793 [2023-06-21 17:01:53,352][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 793 [2023-06-21 17:02:11,263][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 794 [2023-06-21 17:02:12,525][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 794 [2023-06-21 17:02:13,374][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 794 [2023-06-21 17:02:31,301][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 795 [2023-06-21 17:02:32,556][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 795 [2023-06-21 17:02:33,375][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 795 [2023-06-21 17:02:51,303][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 796 [2023-06-21 17:02:52,583][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 796 [2023-06-21 17:02:53,385][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 796 [2023-06-21 17:03:11,328][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 797 [2023-06-21 17:03:12,616][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 797 [2023-06-21 17:03:13,391][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 797 [2023-06-21 17:03:31,382][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 798 [2023-06-21 17:03:32,638][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 798 [2023-06-21 17:03:33,448][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 798 [2023-06-21 17:03:51,420][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 799 [2023-06-21 17:03:52,689][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 799 [2023-06-21 17:03:53,468][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 799 [2023-06-21 17:04:11,474][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 800 [2023-06-21 17:04:12,730][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 800 [2023-06-21 17:04:13,504][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 800 [2023-06-21 17:04:31,477][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 801 [2023-06-21 17:04:32,768][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 801 [2023-06-21 17:04:33,518][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 801 [2023-06-21 17:04:51,486][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 802 [2023-06-21 17:04:52,779][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 802 [2023-06-21 17:04:53,578][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 802 [2023-06-21 17:05:11,544][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 803 [2023-06-21 17:05:12,823][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 803 [2023-06-21 17:05:13,621][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 803 [2023-06-21 17:05:31,593][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 804 [2023-06-21 17:05:32,853][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 804 [2023-06-21 17:05:33,644][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 804 [2023-06-21 17:05:51,616][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 805 [2023-06-21 17:05:52,903][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 805 [2023-06-21 17:05:53,696][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 805 [2023-06-21 17:06:11,661][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 806 [2023-06-21 17:06:12,906][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 806 [2023-06-21 17:06:13,705][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 806 [2023-06-21 17:06:31,708][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 807 [2023-06-21 17:06:32,946][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 807 [2023-06-21 17:06:33,734][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 807 [2023-06-21 17:06:51,733][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 808 [2023-06-21 17:06:52,961][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 808 [2023-06-21 17:06:53,757][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 808 [2023-06-21 17:07:11,793][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 809 [2023-06-21 17:07:12,969][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 809 [2023-06-21 17:07:13,767][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 809 [2023-06-21 17:07:31,810][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 810 [2023-06-21 17:07:33,020][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 810 [2023-06-21 17:07:33,799][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 810 [2023-06-21 17:07:51,851][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 811 [2023-06-21 17:07:53,036][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 811 [2023-06-21 17:07:53,836][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 811 [2023-06-21 17:08:11,868][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 812 [2023-06-21 17:08:13,051][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 812 [2023-06-21 17:08:13,845][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 812 [2023-06-21 17:08:31,901][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 813 [2023-06-21 17:08:33,091][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 813 [2023-06-21 17:08:33,855][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 813 [2023-06-21 17:08:51,929][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 814 [2023-06-21 17:08:53,109][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 814 [2023-06-21 17:08:53,872][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 814 [2023-06-21 17:09:11,975][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 815 [2023-06-21 17:09:13,116][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 815 [2023-06-21 17:09:13,879][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 815 [2023-06-21 17:09:31,999][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 816 [2023-06-21 17:09:33,131][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 816 [2023-06-21 17:09:33,894][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 816 [2023-06-21 17:09:52,022][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 817 [2023-06-21 17:09:53,178][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 817 [2023-06-21 17:09:53,914][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 817 [2023-06-21 17:10:12,047][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 818 [2023-06-21 17:10:13,210][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 818 [2023-06-21 17:10:13,941][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 818 [2023-06-21 17:10:32,099][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 819 [2023-06-21 17:10:33,263][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 819 [2023-06-21 17:10:33,947][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 819 [2023-06-21 17:10:52,148][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 820 [2023-06-21 17:10:53,284][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 820 [2023-06-21 17:10:53,955][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 820 [2023-06-21 17:11:12,200][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 821 [2023-06-21 17:11:13,310][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 821 [2023-06-21 17:11:13,962][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 821 [2023-06-21 17:11:32,213][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 822 [2023-06-21 17:11:33,326][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 822 [2023-06-21 17:11:34,010][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 822 [2023-06-21 17:11:52,252][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 823 [2023-06-21 17:11:53,375][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 823 [2023-06-21 17:11:54,024][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 823 [2023-06-21 17:12:12,308][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 824 [2023-06-21 17:12:13,406][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 824 [2023-06-21 17:12:14,038][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 824 [2023-06-21 17:12:32,366][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 825 [2023-06-21 17:12:33,453][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 825 [2023-06-21 17:12:34,043][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 825 [2023-06-21 17:12:52,419][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 826 [2023-06-21 17:12:53,491][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 826 [2023-06-21 17:12:54,073][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 826 [2023-06-21 17:13:12,446][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 827 [2023-06-21 17:13:13,530][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 827 [2023-06-21 17:13:14,117][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 827 [2023-06-21 17:13:32,459][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 828 [2023-06-21 17:13:33,570][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 828 [2023-06-21 17:13:34,119][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 828 [2023-06-21 17:13:52,495][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 829 [2023-06-21 17:13:53,628][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 829 [2023-06-21 17:13:54,169][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 829 [2023-06-21 17:14:12,517][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 830 [2023-06-21 17:14:13,653][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 830 [2023-06-21 17:14:14,210][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 830 [2023-06-21 17:14:32,559][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 831 [2023-06-21 17:14:33,665][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 831 [2023-06-21 17:14:34,224][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 831 [2023-06-21 17:14:52,598][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 832 [2023-06-21 17:14:53,667][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 832 [2023-06-21 17:14:54,225][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 832 [2023-06-21 17:15:12,614][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 833 [2023-06-21 17:15:13,725][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 833 [2023-06-21 17:15:14,273][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 833 [2023-06-21 17:15:32,654][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 834 [2023-06-21 17:15:33,768][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 834 [2023-06-21 17:15:34,293][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 834 [2023-06-21 17:15:52,684][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 835 [2023-06-21 17:15:53,823][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 835 [2023-06-21 17:15:54,336][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 835 [2023-06-21 17:16:12,734][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 836 [2023-06-21 17:16:13,857][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 836 [2023-06-21 17:16:14,360][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 836 [2023-06-21 17:16:32,788][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 837 [2023-06-21 17:16:33,867][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 837 [2023-06-21 17:16:34,360][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 837 [2023-06-21 17:16:52,796][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 838 [2023-06-21 17:16:53,921][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 838 [2023-06-21 17:16:54,393][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 838 [2023-06-21 17:17:12,835][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 839 [2023-06-21 17:17:13,971][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 839 [2023-06-21 17:17:14,406][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 839 [2023-06-21 17:17:32,859][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 840 [2023-06-21 17:17:34,012][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 840 [2023-06-21 17:17:34,462][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 840 [2023-06-21 17:17:52,866][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 841 [2023-06-21 17:17:54,055][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 841 [2023-06-21 17:17:54,505][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 841 [2023-06-21 17:18:12,890][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 842 [2023-06-21 17:18:14,077][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 842 [2023-06-21 17:18:14,541][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 842 [2023-06-21 17:18:32,944][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 843 [2023-06-21 17:18:34,130][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 843 [2023-06-21 17:18:34,567][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 843 [2023-06-21 17:18:52,977][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 844 [2023-06-21 17:18:54,131][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 844 [2023-06-21 17:18:54,621][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 844 [2023-06-21 17:19:12,983][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 845 [2023-06-21 17:19:14,186][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 845 [2023-06-21 17:19:14,650][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 845 [2023-06-21 17:19:33,004][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 846 [2023-06-21 17:19:34,206][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 846 [2023-06-21 17:19:34,697][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 846 [2023-06-21 17:19:53,049][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 847 [2023-06-21 17:19:54,265][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 847 [2023-06-21 17:19:54,748][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 847 [2023-06-21 17:20:13,058][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 848 [2023-06-21 17:20:14,269][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 848 [2023-06-21 17:20:14,750][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 848 [2023-06-21 17:20:33,067][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 849 [2023-06-21 17:20:34,287][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 849 [2023-06-21 17:20:34,786][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 849 [2023-06-21 17:20:53,083][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 850 [2023-06-21 17:20:54,338][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 850 [2023-06-21 17:20:54,835][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 850 [2023-06-21 17:21:13,126][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 851 [2023-06-21 17:21:14,361][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 851 [2023-06-21 17:21:14,884][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 851 [2023-06-21 17:21:33,185][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 852 [2023-06-21 17:21:34,363][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 852 [2023-06-21 17:21:34,891][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 852 [2023-06-21 17:21:53,213][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 853 [2023-06-21 17:21:54,384][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 853 [2023-06-21 17:21:54,905][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 853 [2023-06-21 17:22:13,239][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 854 [2023-06-21 17:22:14,422][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 854 [2023-06-21 17:22:14,920][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 854 [2023-06-21 17:22:33,267][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 855 [2023-06-21 17:22:34,453][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 855 [2023-06-21 17:22:34,951][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 855 [2023-06-21 17:22:53,280][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 856 [2023-06-21 17:22:54,491][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 856 [2023-06-21 17:22:54,971][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 856 [2023-06-21 17:23:13,281][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 857 [2023-06-21 17:23:14,539][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 857 [2023-06-21 17:23:14,978][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 857 [2023-06-21 17:23:33,333][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 858 [2023-06-21 17:23:34,550][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 858 [2023-06-21 17:23:35,033][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 858 [2023-06-21 17:23:53,384][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 859 [2023-06-21 17:23:54,579][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 859 [2023-06-21 17:23:55,060][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 859 [2023-06-21 17:24:13,406][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 860 [2023-06-21 17:24:14,591][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 860 [2023-06-21 17:24:15,093][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 860 [2023-06-21 17:24:33,429][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 861 [2023-06-21 17:24:34,610][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 861 [2023-06-21 17:24:35,144][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 861 [2023-06-21 17:24:53,462][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 862 [2023-06-21 17:24:54,640][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 862 [2023-06-21 17:24:55,162][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 862 [2023-06-21 17:25:13,493][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 863 [2023-06-21 17:25:14,660][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 863 [2023-06-21 17:25:15,176][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 863 [2023-06-21 17:25:33,499][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 864 [2023-06-21 17:25:34,705][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 864 [2023-06-21 17:25:35,190][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 864 [2023-06-21 17:25:53,502][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 865 [2023-06-21 17:25:54,728][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 865 [2023-06-21 17:25:55,238][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 865 [2023-06-21 17:26:13,541][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 866 [2023-06-21 17:26:14,740][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 866 [2023-06-21 17:26:15,253][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 866 [2023-06-21 17:26:33,575][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 867 [2023-06-21 17:26:34,768][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 867 [2023-06-21 17:26:35,290][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 867 [2023-06-21 17:26:53,589][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 868 [2023-06-21 17:26:54,779][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 868 [2023-06-21 17:26:55,313][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 868 [2023-06-21 17:27:13,607][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 869 [2023-06-21 17:27:14,823][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 869 [2023-06-21 17:27:15,341][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 869 [2023-06-21 17:27:33,615][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 870 [2023-06-21 17:27:34,837][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 870 [2023-06-21 17:27:35,369][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 870 [2023-06-21 17:27:53,616][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 871 [2023-06-21 17:27:54,868][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 871 [2023-06-21 17:27:55,377][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 871 [2023-06-21 17:28:13,642][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 872 [2023-06-21 17:28:14,882][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 872 [2023-06-21 17:28:15,432][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 872 [2023-06-21 17:28:33,683][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 873 [2023-06-21 17:28:34,884][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 873 [2023-06-21 17:28:35,445][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 873 [2023-06-21 17:28:53,692][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 874 [2023-06-21 17:28:54,904][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 874 [2023-06-21 17:28:55,468][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 874 [2023-06-21 17:29:13,723][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 875 [2023-06-21 17:29:14,943][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 875 [2023-06-21 17:29:15,517][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 875 [2023-06-21 17:29:33,744][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 876 [2023-06-21 17:29:34,995][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 876 [2023-06-21 17:29:35,534][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 876 [2023-06-21 17:29:53,770][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 877 [2023-06-21 17:29:55,022][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 877 [2023-06-21 17:29:55,593][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 877 [2023-06-21 17:30:13,798][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 878 [2023-06-21 17:30:15,030][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 878 [2023-06-21 17:30:15,612][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 878 [2023-06-21 17:30:33,809][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 879 [2023-06-21 17:30:35,043][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 879 [2023-06-21 17:30:35,621][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 879 [2023-06-21 17:30:53,829][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 880 [2023-06-21 17:30:55,052][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 880 [2023-06-21 17:30:55,646][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 880 [2023-06-21 17:31:13,869][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 881 [2023-06-21 17:31:15,091][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 881 [2023-06-21 17:31:15,675][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 881 [2023-06-21 17:31:33,925][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 882 [2023-06-21 17:31:35,150][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 882 [2023-06-21 17:31:35,711][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 882 [2023-06-21 17:31:53,951][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 883 [2023-06-21 17:31:55,200][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 883 [2023-06-21 17:31:55,754][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 883 [2023-06-21 17:32:13,963][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 884 [2023-06-21 17:32:15,248][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 884 [2023-06-21 17:32:15,780][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 884 [2023-06-21 17:32:34,001][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 885 [2023-06-21 17:32:35,295][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 885 [2023-06-21 17:32:35,794][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 885 [2023-06-21 17:32:54,058][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 886 [2023-06-21 17:32:55,330][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 886 [2023-06-21 17:32:55,829][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 886 [2023-06-21 17:33:14,088][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 887 [2023-06-21 17:33:15,348][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 887 [2023-06-21 17:33:15,860][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 887 [2023-06-21 17:33:34,144][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 888 [2023-06-21 17:33:35,370][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 888 [2023-06-21 17:33:35,880][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 888 [2023-06-21 17:33:54,164][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 889 [2023-06-21 17:33:55,381][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 889 [2023-06-21 17:33:55,915][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 889 [2023-06-21 17:34:14,178][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 890 [2023-06-21 17:34:15,396][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 890 [2023-06-21 17:34:15,920][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 890 [2023-06-21 17:34:34,204][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 891 [2023-06-21 17:34:35,409][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 891 [2023-06-21 17:34:35,944][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 891 [2023-06-21 17:34:54,242][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 892 [2023-06-21 17:34:55,442][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 892 [2023-06-21 17:34:55,987][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 892 [2023-06-21 17:35:14,274][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 893 [2023-06-21 17:35:15,466][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 893 [2023-06-21 17:35:16,018][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 893 [2023-06-21 17:35:34,314][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 894 [2023-06-21 17:35:35,488][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 894 [2023-06-21 17:35:36,059][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 894 [2023-06-21 17:35:54,321][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 895 [2023-06-21 17:35:55,512][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 895 [2023-06-21 17:35:56,065][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 895 [2023-06-21 17:36:14,336][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 896 [2023-06-21 17:36:15,547][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 896 [2023-06-21 17:36:16,093][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 896 [2023-06-21 17:36:34,376][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 897 [2023-06-21 17:36:35,568][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 897 [2023-06-21 17:36:36,121][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 897 [2023-06-21 17:36:54,380][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 898 [2023-06-21 17:36:55,618][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 898 [2023-06-21 17:36:56,140][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 898 [2023-06-21 17:37:14,427][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 899 [2023-06-21 17:37:15,651][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 899 [2023-06-21 17:37:16,155][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 899 [2023-06-21 17:37:34,438][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 900 [2023-06-21 17:37:35,676][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 900 [2023-06-21 17:37:36,185][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 900 [2023-06-21 17:37:54,449][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 901 [2023-06-21 17:37:55,712][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 901 [2023-06-21 17:37:56,225][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 901 [2023-06-21 17:38:14,470][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 902 [2023-06-21 17:38:15,749][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 902 [2023-06-21 17:38:16,284][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 902 [2023-06-21 17:38:34,517][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 903 [2023-06-21 17:38:35,777][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 903 [2023-06-21 17:38:36,334][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 903 [2023-06-21 17:38:54,550][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 904 [2023-06-21 17:38:55,791][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 904 [2023-06-21 17:38:56,387][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 904 [2023-06-21 17:39:14,556][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 905 [2023-06-21 17:39:15,798][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 905 [2023-06-21 17:39:16,405][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 905 [2023-06-21 17:39:34,606][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 906 [2023-06-21 17:39:35,826][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 906 [2023-06-21 17:39:36,431][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 906 [2023-06-21 17:39:54,620][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 907 [2023-06-21 17:39:55,877][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 907 [2023-06-21 17:39:56,437][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 907 [2023-06-21 17:40:14,626][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 908 [2023-06-21 17:40:15,912][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 908 [2023-06-21 17:40:16,441][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 908 [2023-06-21 17:40:34,632][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 909 [2023-06-21 17:40:35,919][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 909 [2023-06-21 17:40:36,499][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 909 [2023-06-21 17:40:54,645][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 910 [2023-06-21 17:40:55,920][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 910 [2023-06-21 17:40:56,528][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 910 [2023-06-21 17:41:14,649][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 911 [2023-06-21 17:41:15,967][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 911 [2023-06-21 17:41:16,578][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 911 [2023-06-21 17:41:34,706][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 912 [2023-06-21 17:41:36,007][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 912 [2023-06-21 17:41:36,621][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 912 [2023-06-21 17:41:54,718][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 913 [2023-06-21 17:41:56,020][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 913 [2023-06-21 17:41:56,646][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 913 [2023-06-21 17:42:14,750][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 914 [2023-06-21 17:42:16,039][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 914 [2023-06-21 17:42:16,655][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 914 [2023-06-21 17:42:34,785][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 915 [2023-06-21 17:42:36,065][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 915 [2023-06-21 17:42:36,692][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 915 [2023-06-21 17:42:54,825][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 916 [2023-06-21 17:42:56,107][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 916 [2023-06-21 17:42:56,744][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 916 [2023-06-21 17:43:14,835][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 917 [2023-06-21 17:43:16,132][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 917 [2023-06-21 17:43:16,791][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 917 [2023-06-21 17:43:34,889][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 918 [2023-06-21 17:43:36,144][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 918 [2023-06-21 17:43:36,840][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 918 [2023-06-21 17:43:54,921][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 919 [2023-06-21 17:43:56,193][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 919 [2023-06-21 17:43:56,869][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 919 [2023-06-21 17:44:14,941][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 920 [2023-06-21 17:44:16,194][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 920 [2023-06-21 17:44:16,889][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 920 [2023-06-21 17:44:34,946][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 921 [2023-06-21 17:44:36,244][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 921 [2023-06-21 17:44:36,936][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 921 [2023-06-21 17:44:55,002][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 922 [2023-06-21 17:44:56,252][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 922 [2023-06-21 17:44:56,944][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 922 [2023-06-21 17:45:15,039][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 923 [2023-06-21 17:45:16,299][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 923 [2023-06-21 17:45:16,972][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 923 [2023-06-21 17:45:35,068][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 924 [2023-06-21 17:45:36,341][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 924 [2023-06-21 17:45:37,001][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 924 [2023-06-21 17:45:55,101][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 925 [2023-06-21 17:45:56,386][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 925 [2023-06-21 17:45:57,016][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 925 [2023-06-21 17:46:15,145][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 926 [2023-06-21 17:46:16,391][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 926 [2023-06-21 17:46:17,028][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 926 [2023-06-21 17:46:35,179][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 927 [2023-06-21 17:46:36,440][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 927 [2023-06-21 17:46:37,057][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 927 [2023-06-21 17:46:55,186][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 928 [2023-06-21 17:46:56,492][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 928 [2023-06-21 17:46:57,064][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 928 [2023-06-21 17:47:15,223][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 929 [2023-06-21 17:47:16,520][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 929 [2023-06-21 17:47:17,100][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 929 [2023-06-21 17:47:35,228][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 930 [2023-06-21 17:47:36,525][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 930 [2023-06-21 17:47:37,102][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 930 [2023-06-21 17:47:55,282][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 931 [2023-06-21 17:47:56,564][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 931 [2023-06-21 17:47:57,109][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 931 [2023-06-21 17:48:15,305][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 932 [2023-06-21 17:48:16,604][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 932 [2023-06-21 17:48:17,135][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 932 [2023-06-21 17:48:35,365][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 933 [2023-06-21 17:48:36,609][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 933 [2023-06-21 17:48:37,139][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 933 [2023-06-21 17:48:55,370][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 934 [2023-06-21 17:48:56,622][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 934 [2023-06-21 17:48:57,148][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 934 [2023-06-21 17:49:15,401][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 935 [2023-06-21 17:49:16,672][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 935 [2023-06-21 17:49:17,182][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 935 [2023-06-21 17:49:35,416][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 936 [2023-06-21 17:49:36,714][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 936 [2023-06-21 17:49:37,186][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 936 [2023-06-21 17:49:55,463][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 937 [2023-06-21 17:49:56,737][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 937 [2023-06-21 17:49:57,210][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 937 [2023-06-21 17:50:15,470][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 938 [2023-06-21 17:50:16,756][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 938 [2023-06-21 17:50:17,266][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 938 [2023-06-21 17:50:35,521][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 939 [2023-06-21 17:50:36,789][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 939 [2023-06-21 17:50:37,283][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 939 [2023-06-21 17:50:55,572][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 940 [2023-06-21 17:50:56,806][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 940 [2023-06-21 17:50:57,307][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 940 [2023-06-21 17:51:15,603][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 941 [2023-06-21 17:51:16,808][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 941 [2023-06-21 17:51:17,314][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 941 [2023-06-21 17:51:35,654][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 942 [2023-06-21 17:51:36,858][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 942 [2023-06-21 17:51:37,350][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 942 [2023-06-21 17:51:55,657][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 943 [2023-06-21 17:51:56,914][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 943 [2023-06-21 17:51:57,363][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 943 [2023-06-21 17:52:15,695][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 944 [2023-06-21 17:52:16,961][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 944 [2023-06-21 17:52:17,389][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 944 [2023-06-21 17:52:35,736][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 945 [2023-06-21 17:52:37,014][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 945 [2023-06-21 17:52:37,412][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 945 [2023-06-21 17:52:55,764][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 946 [2023-06-21 17:52:57,025][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 946 [2023-06-21 17:52:57,426][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 946 [2023-06-21 17:53:15,768][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 947 [2023-06-21 17:53:17,071][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 947 [2023-06-21 17:53:17,453][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 947 [2023-06-21 17:53:35,821][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 948 [2023-06-21 17:53:37,113][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 948 [2023-06-21 17:53:37,469][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 948 [2023-06-21 17:53:55,879][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 949 [2023-06-21 17:53:57,157][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 949 [2023-06-21 17:53:57,529][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 949 [2023-06-21 17:54:15,930][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 950 [2023-06-21 17:54:17,208][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 950 [2023-06-21 17:54:17,555][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 950 [2023-06-21 17:54:35,966][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 951 [2023-06-21 17:54:37,221][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 951 [2023-06-21 17:54:37,601][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 951 [2023-06-21 17:54:55,990][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 952 [2023-06-21 17:54:57,222][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 952 [2023-06-21 17:54:57,650][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 952 [2023-06-21 17:55:16,001][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 953 [2023-06-21 17:55:17,233][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 953 [2023-06-21 17:55:17,659][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 953 [2023-06-21 17:55:36,020][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 954 [2023-06-21 17:55:37,250][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 954 [2023-06-21 17:55:37,684][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 954 [2023-06-21 17:55:56,075][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 955 [2023-06-21 17:55:57,260][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 955 [2023-06-21 17:55:57,692][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 955 [2023-06-21 17:56:16,137][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 956 [2023-06-21 17:56:17,304][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 956 [2023-06-21 17:56:17,734][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 956 [2023-06-21 17:56:36,171][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 957 [2023-06-21 17:56:37,345][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 957 [2023-06-21 17:56:37,765][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 957 [2023-06-21 17:56:56,220][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 958 [2023-06-21 17:56:57,348][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 958 [2023-06-21 17:56:57,804][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 958 [2023-06-21 17:57:16,266][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 959 [2023-06-21 17:57:17,355][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 959 [2023-06-21 17:57:17,825][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 959 [2023-06-21 17:57:36,277][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 960 [2023-06-21 17:57:37,371][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 960 [2023-06-21 17:57:37,867][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 960 [2023-06-21 17:57:56,322][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 961 [2023-06-21 17:57:57,386][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 961 [2023-06-21 17:57:57,886][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 961 [2023-06-21 17:58:16,383][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 962 [2023-06-21 17:58:17,391][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 962 [2023-06-21 17:58:17,938][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 962 [2023-06-21 17:58:36,426][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 963 [2023-06-21 17:58:37,414][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 963 [2023-06-21 17:58:37,985][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 963 [2023-06-21 17:58:56,446][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 964 [2023-06-21 17:58:57,460][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 964 [2023-06-21 17:58:58,010][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 964 [2023-06-21 17:59:16,458][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 965 [2023-06-21 17:59:17,511][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 965 [2023-06-21 17:59:18,061][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 965 [2023-06-21 17:59:36,497][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 966 [2023-06-21 17:59:37,536][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 966 [2023-06-21 17:59:38,113][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 966 [2023-06-21 17:59:56,520][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 967 [2023-06-21 17:59:57,577][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 967 [2023-06-21 17:59:58,152][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 967 [2023-06-21 18:00:16,576][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 968 [2023-06-21 18:00:17,582][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 968 [2023-06-21 18:00:18,179][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 968 [2023-06-21 18:00:36,584][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 969 [2023-06-21 18:00:37,625][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 969 [2023-06-21 18:00:38,187][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 969 [2023-06-21 18:00:56,631][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 970 [2023-06-21 18:00:57,682][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 970 [2023-06-21 18:00:58,235][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 970 [2023-06-21 18:01:16,633][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 971 [2023-06-21 18:01:17,712][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 971 [2023-06-21 18:01:18,243][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 971 [2023-06-21 18:01:36,643][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 972 [2023-06-21 18:01:37,732][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 972 [2023-06-21 18:01:38,267][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 972 [2023-06-21 18:01:56,644][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 973 [2023-06-21 18:01:57,737][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 973 [2023-06-21 18:01:58,307][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 973 [2023-06-21 18:02:16,698][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 974 [2023-06-21 18:02:17,792][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 974 [2023-06-21 18:02:18,342][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 974 [2023-06-21 18:02:36,725][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 975 [2023-06-21 18:02:37,804][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 975 [2023-06-21 18:02:38,355][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 975 [2023-06-21 18:02:56,752][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 976 [2023-06-21 18:02:57,853][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 976 [2023-06-21 18:02:58,382][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 976 [2023-06-21 18:03:16,786][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 977 [2023-06-21 18:03:17,866][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 977 [2023-06-21 18:03:18,404][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 977 [2023-06-21 18:03:36,801][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 978 [2023-06-21 18:03:37,919][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 978 [2023-06-21 18:03:38,430][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 978 [2023-06-21 18:03:56,858][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 979 [2023-06-21 18:03:57,948][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 979 [2023-06-21 18:03:58,468][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 979 [2023-06-21 18:04:16,907][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 980 [2023-06-21 18:04:17,993][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 980 [2023-06-21 18:04:18,520][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 980 [2023-06-21 18:04:36,928][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 981 [2023-06-21 18:04:38,002][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 981 [2023-06-21 18:04:38,529][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 981 [2023-06-21 18:04:56,974][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 982 [2023-06-21 18:04:58,048][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 982 [2023-06-21 18:04:58,547][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 982 [2023-06-21 18:05:17,027][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 983 [2023-06-21 18:05:18,075][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 983 [2023-06-21 18:05:18,589][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 983 [2023-06-21 18:05:37,066][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 984 [2023-06-21 18:05:38,087][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 984 [2023-06-21 18:05:38,614][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 984 [2023-06-21 18:05:57,103][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 985 [2023-06-21 18:05:58,101][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 985 [2023-06-21 18:05:58,644][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 985 [2023-06-21 18:06:17,109][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 986 [2023-06-21 18:06:18,145][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 986 [2023-06-21 18:06:18,700][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 986 [2023-06-21 18:06:37,154][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 987 [2023-06-21 18:06:38,152][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 987 [2023-06-21 18:06:38,732][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 987 [2023-06-21 18:06:57,206][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 988 [2023-06-21 18:06:58,205][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 988 [2023-06-21 18:06:58,753][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 988 [2023-06-21 18:07:17,257][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 989 [2023-06-21 18:07:18,257][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 989 [2023-06-21 18:07:18,807][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 989 [2023-06-21 18:07:37,292][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 990 [2023-06-21 18:07:38,274][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 990 [2023-06-21 18:07:38,847][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 990 [2023-06-21 18:07:57,333][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 991 [2023-06-21 18:07:58,330][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 991 [2023-06-21 18:07:58,885][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 991 [2023-06-21 18:08:17,340][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 992 [2023-06-21 18:08:18,376][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 992 [2023-06-21 18:08:18,924][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 992 [2023-06-21 18:08:37,346][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 993 [2023-06-21 18:08:38,388][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 993 [2023-06-21 18:08:38,935][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 993 [2023-06-21 18:08:57,369][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 994 [2023-06-21 18:08:58,421][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 994 [2023-06-21 18:08:58,974][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 994 [2023-06-21 18:09:17,387][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 995 [2023-06-21 18:09:18,445][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 995 [2023-06-21 18:09:19,016][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 995 [2023-06-21 18:09:37,406][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 996 [2023-06-21 18:09:38,480][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 996 [2023-06-21 18:09:39,057][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 996 [2023-06-21 18:09:57,437][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 997 [2023-06-21 18:09:58,509][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 997 [2023-06-21 18:09:59,081][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 997 [2023-06-21 18:10:17,447][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 998 [2023-06-21 18:10:18,530][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 998 [2023-06-21 18:10:19,108][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 998 [2023-06-21 18:10:37,503][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 999 [2023-06-21 18:10:38,573][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 999 [2023-06-21 18:10:39,161][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 999 [2023-06-21 18:10:57,553][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1000 [2023-06-21 18:10:58,597][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1000 [2023-06-21 18:10:59,167][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1000 [2023-06-21 18:11:17,588][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1001 [2023-06-21 18:11:18,602][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1001 [2023-06-21 18:11:19,225][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1001 [2023-06-21 18:11:37,598][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1002 [2023-06-21 18:11:38,606][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1002 [2023-06-21 18:11:39,239][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1002 [2023-06-21 18:11:57,652][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1003 [2023-06-21 18:11:58,615][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1003 [2023-06-21 18:11:59,282][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1003 [2023-06-21 18:12:17,701][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1004 [2023-06-21 18:12:18,658][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1004 [2023-06-21 18:12:19,290][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1004 [2023-06-21 18:12:37,730][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1005 [2023-06-21 18:12:38,670][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1005 [2023-06-21 18:12:39,305][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1005 [2023-06-21 18:12:57,767][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1006 [2023-06-21 18:12:58,706][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1006 [2023-06-21 18:12:59,358][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1006 [2023-06-21 18:13:17,791][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1007 [2023-06-21 18:13:18,721][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1007 [2023-06-21 18:13:19,390][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1007 [2023-06-21 18:13:37,814][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1008 [2023-06-21 18:13:38,735][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1008 [2023-06-21 18:13:39,415][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1008 [2023-06-21 18:13:57,864][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1009 [2023-06-21 18:13:58,763][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1009 [2023-06-21 18:13:59,471][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1009 [2023-06-21 18:14:17,881][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1010 [2023-06-21 18:14:18,798][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1010 [2023-06-21 18:14:19,523][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1010 [2023-06-21 18:14:37,929][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1011 [2023-06-21 18:14:38,832][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1011 [2023-06-21 18:14:39,556][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1011 [2023-06-21 18:14:57,934][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1012 [2023-06-21 18:14:58,835][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1012 [2023-06-21 18:14:59,607][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1012 [2023-06-21 18:15:17,977][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1013 [2023-06-21 18:15:18,881][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1013 [2023-06-21 18:15:19,629][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1013 [2023-06-21 18:15:37,995][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1014 [2023-06-21 18:15:38,922][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1014 [2023-06-21 18:15:39,671][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1014 [2023-06-21 18:15:57,999][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1015 [2023-06-21 18:15:58,940][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1015 [2023-06-21 18:15:59,686][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1015 [2023-06-21 18:16:18,039][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1016 [2023-06-21 18:16:18,974][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1016 [2023-06-21 18:16:19,715][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1016 [2023-06-21 18:16:38,056][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1017 [2023-06-21 18:16:39,014][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1017 [2023-06-21 18:16:39,756][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1017 [2023-06-21 18:16:58,074][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1018 [2023-06-21 18:16:59,067][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1018 [2023-06-21 18:16:59,787][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1018 [2023-06-21 18:17:18,100][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1019 [2023-06-21 18:17:19,111][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1019 [2023-06-21 18:17:19,808][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1019 [2023-06-21 18:17:38,156][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1020 [2023-06-21 18:17:39,146][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1020 [2023-06-21 18:17:39,861][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1020 [2023-06-21 18:17:58,205][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1021 [2023-06-21 18:17:59,188][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1021 [2023-06-21 18:17:59,904][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1021 [2023-06-21 18:18:18,248][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1022 [2023-06-21 18:18:19,222][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1022 [2023-06-21 18:18:19,907][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1022 [2023-06-21 18:18:38,262][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1023 [2023-06-21 18:18:39,267][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1023 [2023-06-21 18:18:39,914][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1023 [2023-06-21 18:18:58,302][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1024 [2023-06-21 18:18:59,274][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1024 [2023-06-21 18:18:59,958][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1024 [2023-06-21 18:19:18,337][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1025 [2023-06-21 18:19:19,287][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1025 [2023-06-21 18:19:20,007][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1025 [2023-06-21 18:19:38,385][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1026 [2023-06-21 18:19:39,338][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1026 [2023-06-21 18:19:40,031][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1026 [2023-06-21 18:19:58,405][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1027 [2023-06-21 18:19:59,387][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1027 [2023-06-21 18:20:00,042][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1027 [2023-06-21 18:20:18,442][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1028 [2023-06-21 18:20:19,417][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1028 [2023-06-21 18:20:20,045][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1028 [2023-06-21 18:20:38,473][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1029 [2023-06-21 18:20:39,419][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1029 [2023-06-21 18:20:40,087][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1029 [2023-06-21 18:20:58,474][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1030 [2023-06-21 18:20:59,436][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1030 [2023-06-21 18:21:00,128][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1030 [2023-06-21 18:21:18,517][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1031 [2023-06-21 18:21:19,440][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1031 [2023-06-21 18:21:20,176][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1031 [2023-06-21 18:21:38,565][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1032 [2023-06-21 18:21:39,453][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1032 [2023-06-21 18:21:40,210][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1032 [2023-06-21 18:21:58,591][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1033 [2023-06-21 18:21:59,456][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1033 [2023-06-21 18:22:00,268][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1033 [2023-06-21 18:22:18,644][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1034 [2023-06-21 18:22:19,467][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1034 [2023-06-21 18:22:20,280][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1034 [2023-06-21 18:22:38,647][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1035 [2023-06-21 18:22:39,520][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1035 [2023-06-21 18:22:40,298][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1035 [2023-06-21 18:22:58,679][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1036 [2023-06-21 18:22:59,574][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1036 [2023-06-21 18:23:00,331][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1036 [2023-06-21 18:23:18,695][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1037 [2023-06-21 18:23:19,617][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1037 [2023-06-21 18:23:20,355][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1037 [2023-06-21 18:23:38,753][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1038 [2023-06-21 18:23:39,644][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1038 [2023-06-21 18:23:40,400][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1038 [2023-06-21 18:23:58,770][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1039 [2023-06-21 18:23:59,686][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1039 [2023-06-21 18:24:00,411][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1039 [2023-06-21 18:24:18,798][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1040 [2023-06-21 18:24:19,722][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1040 [2023-06-21 18:24:20,465][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1040 [2023-06-21 18:24:38,829][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1041 [2023-06-21 18:24:39,752][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1041 [2023-06-21 18:24:40,520][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1041 [2023-06-21 18:24:58,857][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1042 [2023-06-21 18:24:59,754][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1042 [2023-06-21 18:25:00,544][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1042 [2023-06-21 18:25:18,886][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1043 [2023-06-21 18:25:19,798][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1043 [2023-06-21 18:25:20,598][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1043 [2023-06-21 18:25:38,929][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1044 [2023-06-21 18:25:39,856][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1044 [2023-06-21 18:25:40,645][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1044 [2023-06-21 18:25:58,977][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1045 [2023-06-21 18:25:59,864][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1045 [2023-06-21 18:26:00,686][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1045 [2023-06-21 18:26:19,000][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1046 [2023-06-21 18:26:19,912][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1046 [2023-06-21 18:26:20,722][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1046 [2023-06-21 18:26:39,062][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1047 [2023-06-21 18:26:39,955][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1047 [2023-06-21 18:26:40,772][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1047 [2023-06-21 18:26:59,074][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1048 [2023-06-21 18:26:59,967][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1048 [2023-06-21 18:27:00,804][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1048 [2023-06-21 18:27:19,085][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1049 [2023-06-21 18:27:19,995][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1049 [2023-06-21 18:27:20,862][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1049 [2023-06-21 18:27:39,094][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1050 [2023-06-21 18:27:40,026][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1050 [2023-06-21 18:27:40,887][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1050 [2023-06-21 18:27:59,131][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1051 [2023-06-21 18:28:00,053][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1051 [2023-06-21 18:28:00,905][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1051 [2023-06-21 18:28:19,158][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1052 [2023-06-21 18:28:20,060][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1052 [2023-06-21 18:28:20,944][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1052 [2023-06-21 18:28:39,188][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1053 [2023-06-21 18:28:40,068][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1053 [2023-06-21 18:28:40,957][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1053 [2023-06-21 18:28:59,202][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1054 [2023-06-21 18:29:00,116][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1054 [2023-06-21 18:29:01,008][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1054 [2023-06-21 18:29:19,206][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1055 [2023-06-21 18:29:20,131][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1055 [2023-06-21 18:29:21,026][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1055 [2023-06-21 18:29:39,246][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1056 [2023-06-21 18:29:40,134][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1056 [2023-06-21 18:29:41,049][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1056 [2023-06-21 18:29:59,261][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1057 [2023-06-21 18:30:00,177][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1057 [2023-06-21 18:30:01,095][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1057 [2023-06-21 18:30:19,285][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1058 [2023-06-21 18:30:20,233][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1058 [2023-06-21 18:30:21,150][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1058 [2023-06-21 18:30:39,338][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1059 [2023-06-21 18:30:40,240][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1059 [2023-06-21 18:30:41,152][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1059 [2023-06-21 18:30:59,345][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1060 [2023-06-21 18:31:00,286][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1060 [2023-06-21 18:31:01,186][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1060 [2023-06-21 18:31:19,374][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1061 [2023-06-21 18:31:20,328][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1061 [2023-06-21 18:31:21,200][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1061 [2023-06-21 18:31:39,393][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1062 [2023-06-21 18:31:40,333][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1062 [2023-06-21 18:31:41,213][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1062 [2023-06-21 18:31:59,417][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1063 [2023-06-21 18:32:00,370][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1063 [2023-06-21 18:32:01,239][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1063 [2023-06-21 18:32:19,459][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1064 [2023-06-21 18:32:20,396][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1064 [2023-06-21 18:32:21,290][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1064 [2023-06-21 18:32:39,469][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1065 [2023-06-21 18:32:40,446][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1065 [2023-06-21 18:32:41,305][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1065 [2023-06-21 18:32:59,476][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1066 [2023-06-21 18:33:00,497][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1066 [2023-06-21 18:33:01,356][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1066 [2023-06-21 18:33:19,522][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1067 [2023-06-21 18:33:20,522][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1067 [2023-06-21 18:33:21,406][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1067 [2023-06-21 18:33:39,537][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1068 [2023-06-21 18:33:40,552][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1068 [2023-06-21 18:33:41,426][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1068 [2023-06-21 18:33:59,543][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1069 [2023-06-21 18:34:00,586][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1069 [2023-06-21 18:34:01,434][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1069 [2023-06-21 18:34:19,570][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1070 [2023-06-21 18:34:20,604][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1070 [2023-06-21 18:34:21,486][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1070 [2023-06-21 18:34:39,579][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1071 [2023-06-21 18:34:40,618][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1071 [2023-06-21 18:34:41,507][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1071 [2023-06-21 18:34:59,629][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1072 [2023-06-21 18:35:00,669][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1072 [2023-06-21 18:35:01,532][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1072 [2023-06-21 18:35:19,684][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1073 [2023-06-21 18:35:20,726][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1073 [2023-06-21 18:35:21,543][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1073 [2023-06-21 18:35:39,717][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1074 [2023-06-21 18:35:40,744][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1074 [2023-06-21 18:35:41,553][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1074 [2023-06-21 18:35:59,765][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1075 [2023-06-21 18:36:00,791][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1075 [2023-06-21 18:36:01,559][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1075 [2023-06-21 18:36:19,799][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1076 [2023-06-21 18:36:20,819][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1076 [2023-06-21 18:36:21,563][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1076 [2023-06-21 18:36:39,801][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1077 [2023-06-21 18:36:40,868][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1077 [2023-06-21 18:36:41,588][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1077 [2023-06-21 18:36:59,802][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1078 [2023-06-21 18:37:00,899][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1078 [2023-06-21 18:37:01,610][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1078 [2023-06-21 18:37:19,853][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1079 [2023-06-21 18:37:20,929][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1079 [2023-06-21 18:37:21,622][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1079 [2023-06-21 18:37:39,859][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1080 [2023-06-21 18:37:40,979][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1080 [2023-06-21 18:37:41,683][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1080 [2023-06-21 18:37:59,914][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1081 [2023-06-21 18:38:00,989][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1081 [2023-06-21 18:38:01,696][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1081 [2023-06-21 18:38:19,918][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1082 [2023-06-21 18:38:21,045][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1082 [2023-06-21 18:38:21,737][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1082 [2023-06-21 18:38:39,977][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1083 [2023-06-21 18:38:41,068][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1083 [2023-06-21 18:38:41,782][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1083 [2023-06-21 18:39:00,023][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1084 [2023-06-21 18:39:01,101][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1084 [2023-06-21 18:39:01,814][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1084 [2023-06-21 18:39:20,054][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1085 [2023-06-21 18:39:21,121][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1085 [2023-06-21 18:39:21,860][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1085 [2023-06-21 18:39:40,105][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1086 [2023-06-21 18:39:41,173][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1086 [2023-06-21 18:39:41,912][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1086 [2023-06-21 18:40:00,154][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1087 [2023-06-21 18:40:01,222][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1087 [2023-06-21 18:40:01,938][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1087 [2023-06-21 18:40:20,213][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1088 [2023-06-21 18:40:21,254][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1088 [2023-06-21 18:40:21,951][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1088 [2023-06-21 18:40:40,257][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1089 [2023-06-21 18:40:41,263][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1089 [2023-06-21 18:40:41,987][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1089 [2023-06-21 18:41:00,259][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1090 [2023-06-21 18:41:01,286][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1090 [2023-06-21 18:41:02,033][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1090 [2023-06-21 18:41:20,304][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1091 [2023-06-21 18:41:21,310][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1091 [2023-06-21 18:41:22,043][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1091 [2023-06-21 18:41:40,325][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1092 [2023-06-21 18:41:41,321][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1092 [2023-06-21 18:41:42,086][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1092 [2023-06-21 18:42:00,365][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1093 [2023-06-21 18:42:01,373][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1093 [2023-06-21 18:42:02,096][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1093 [2023-06-21 18:42:20,401][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1094 [2023-06-21 18:42:21,403][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1094 [2023-06-21 18:42:22,144][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1094 [2023-06-21 18:42:40,433][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1095 [2023-06-21 18:42:41,408][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1095 [2023-06-21 18:42:42,192][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1095 [2023-06-21 18:43:00,462][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1096 [2023-06-21 18:43:01,458][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1096 [2023-06-21 18:43:02,245][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1096 [2023-06-21 18:43:20,483][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1097 [2023-06-21 18:43:21,509][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1097 [2023-06-21 18:43:22,301][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1097 [2023-06-21 18:43:40,497][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1098 [2023-06-21 18:43:41,537][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1098 [2023-06-21 18:43:42,351][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1098 [2023-06-21 18:44:00,548][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1099 [2023-06-21 18:44:01,556][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1099 [2023-06-21 18:44:02,403][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1099 [2023-06-21 18:44:20,607][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1100 [2023-06-21 18:44:21,593][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1100 [2023-06-21 18:44:22,409][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1100 [2023-06-21 18:44:40,665][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1101 [2023-06-21 18:44:41,599][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1101 [2023-06-21 18:44:42,447][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1101 [2023-06-21 18:45:00,724][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1102 [2023-06-21 18:45:01,654][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1102 [2023-06-21 18:45:02,451][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1102 [2023-06-21 18:45:20,750][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1103 [2023-06-21 18:45:21,661][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1103 [2023-06-21 18:45:22,498][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1103 [2023-06-21 18:45:40,770][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1104 [2023-06-21 18:45:41,716][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1104 [2023-06-21 18:45:42,510][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1104 [2023-06-21 18:46:00,807][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1105 [2023-06-21 18:46:01,738][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1105 [2023-06-21 18:46:02,549][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1105 [2023-06-21 18:46:20,819][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1106 [2023-06-21 18:46:21,785][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1106 [2023-06-21 18:46:22,607][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1106 [2023-06-21 18:46:40,859][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1107 [2023-06-21 18:46:41,809][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1107 [2023-06-21 18:46:42,617][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1107 [2023-06-21 18:47:00,866][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1108 [2023-06-21 18:47:01,860][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1108 [2023-06-21 18:47:02,662][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1108 [2023-06-21 18:47:20,921][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1109 [2023-06-21 18:47:21,880][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1109 [2023-06-21 18:47:22,683][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1109 [2023-06-21 18:47:40,947][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1110 [2023-06-21 18:47:41,900][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1110 [2023-06-21 18:47:42,685][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1110 [2023-06-21 18:48:00,977][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1111 [2023-06-21 18:48:01,931][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1111 [2023-06-21 18:48:02,728][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1111 [2023-06-21 18:48:21,013][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1112 [2023-06-21 18:48:21,986][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1112 [2023-06-21 18:48:22,748][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1112 [2023-06-21 18:48:41,066][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1113 [2023-06-21 18:48:42,015][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1113 [2023-06-21 18:48:42,768][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1113 [2023-06-21 18:49:01,076][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1114 [2023-06-21 18:49:02,055][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1114 [2023-06-21 18:49:02,820][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1114 [2023-06-21 18:49:21,129][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1115 [2023-06-21 18:49:22,099][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1115 [2023-06-21 18:49:22,822][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1115 [2023-06-21 18:49:41,160][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1116 [2023-06-21 18:49:42,102][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1116 [2023-06-21 18:49:42,879][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1116 [2023-06-21 18:50:01,196][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1117 [2023-06-21 18:50:02,113][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1117 [2023-06-21 18:50:02,898][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1117 [2023-06-21 18:50:21,210][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1118 [2023-06-21 18:50:22,121][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1118 [2023-06-21 18:50:22,945][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1118 [2023-06-21 18:50:41,237][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1119 [2023-06-21 18:50:42,134][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1119 [2023-06-21 18:50:42,952][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1119 [2023-06-21 18:51:01,284][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1120 [2023-06-21 18:51:02,192][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1120 [2023-06-21 18:51:02,957][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1120 [2023-06-21 18:51:21,297][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1121 [2023-06-21 18:51:22,194][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1121 [2023-06-21 18:51:22,987][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1121 [2023-06-21 18:51:41,322][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1122 [2023-06-21 18:51:42,219][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1122 [2023-06-21 18:51:43,038][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1122 [2023-06-21 18:52:01,354][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1123 [2023-06-21 18:52:02,239][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1123 [2023-06-21 18:52:03,062][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1123 [2023-06-21 18:52:21,376][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1124 [2023-06-21 18:52:22,287][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1124 [2023-06-21 18:52:23,109][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1124 [2023-06-21 18:52:41,378][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1125 [2023-06-21 18:52:42,303][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1125 [2023-06-21 18:52:43,124][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1125 [2023-06-21 18:53:01,414][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1126 [2023-06-21 18:53:02,350][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1126 [2023-06-21 18:53:03,164][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1126 [2023-06-21 18:53:21,418][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1127 [2023-06-21 18:53:22,367][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1127 [2023-06-21 18:53:23,167][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1127 [2023-06-21 18:53:41,456][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1128 [2023-06-21 18:53:42,419][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1128 [2023-06-21 18:53:43,169][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1128 [2023-06-21 18:54:01,504][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1129 [2023-06-21 18:54:02,463][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1129 [2023-06-21 18:54:03,188][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1129 [2023-06-21 18:54:21,521][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1130 [2023-06-21 18:54:22,495][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1130 [2023-06-21 18:54:23,213][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1130 [2023-06-21 18:54:41,576][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1131 [2023-06-21 18:54:42,553][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1131 [2023-06-21 18:54:43,238][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1131 [2023-06-21 18:55:01,584][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1132 [2023-06-21 18:55:02,576][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1132 [2023-06-21 18:55:03,276][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1132 [2023-06-21 18:55:21,612][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1133 [2023-06-21 18:55:22,601][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1133 [2023-06-21 18:55:23,324][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1133 [2023-06-21 18:55:41,640][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1134 [2023-06-21 18:55:42,651][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1134 [2023-06-21 18:55:43,362][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1134 [2023-06-21 18:56:01,674][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1135 [2023-06-21 18:56:02,691][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1135 [2023-06-21 18:56:03,419][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1135 [2023-06-21 18:56:21,685][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1136 [2023-06-21 18:56:22,736][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1136 [2023-06-21 18:56:23,470][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1136 [2023-06-21 18:56:41,740][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1137 [2023-06-21 18:56:42,796][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1137 [2023-06-21 18:56:43,484][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1137 [2023-06-21 18:57:01,775][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1138 [2023-06-21 18:57:02,805][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1138 [2023-06-21 18:57:03,515][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1138 [2023-06-21 18:57:21,790][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1139 [2023-06-21 18:57:22,833][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1139 [2023-06-21 18:57:23,558][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1139 [2023-06-21 18:57:41,845][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1140 [2023-06-21 18:57:42,855][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1140 [2023-06-21 18:57:43,566][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1140 [2023-06-21 18:58:01,867][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1141 [2023-06-21 18:58:02,877][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1141 [2023-06-21 18:58:03,601][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1141 [2023-06-21 18:58:21,886][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1142 [2023-06-21 18:58:22,906][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1142 [2023-06-21 18:58:23,634][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1142 [2023-06-21 18:58:41,931][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1143 [2023-06-21 18:58:42,959][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1143 [2023-06-21 18:58:43,683][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1143 [2023-06-21 18:59:01,972][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1144 [2023-06-21 18:59:03,011][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1144 [2023-06-21 18:59:03,721][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1144 [2023-06-21 18:59:21,993][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1145 [2023-06-21 18:59:23,043][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1145 [2023-06-21 18:59:23,776][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1145 [2023-06-21 18:59:42,024][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1146 [2023-06-21 18:59:43,101][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1146 [2023-06-21 18:59:43,826][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1146 [2023-06-21 19:00:02,051][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1147 [2023-06-21 19:00:03,118][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1147 [2023-06-21 19:00:03,845][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1147 [2023-06-21 19:00:22,057][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1148 [2023-06-21 19:00:23,126][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1148 [2023-06-21 19:00:23,859][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1148 [2023-06-21 19:00:42,098][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1149 [2023-06-21 19:00:43,139][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1149 [2023-06-21 19:00:43,902][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1149 [2023-06-21 19:01:02,135][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1150 [2023-06-21 19:01:03,192][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1150 [2023-06-21 19:01:03,937][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1150 [2023-06-21 19:01:22,178][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1151 [2023-06-21 19:01:23,211][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1151 [2023-06-21 19:01:23,950][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1151 [2023-06-21 19:01:42,220][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1152 [2023-06-21 19:01:43,260][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1152 [2023-06-21 19:01:43,971][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1152 [2023-06-21 19:02:02,260][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1153 [2023-06-21 19:02:03,309][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1153 [2023-06-21 19:02:03,983][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1153 [2023-06-21 19:02:22,276][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1154 [2023-06-21 19:02:23,341][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1154 [2023-06-21 19:02:24,016][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1154 [2023-06-21 19:02:42,298][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1155 [2023-06-21 19:02:43,391][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1155 [2023-06-21 19:02:44,077][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1155 [2023-06-21 19:03:02,305][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1156 [2023-06-21 19:03:03,444][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1156 [2023-06-21 19:03:04,108][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1156 [2023-06-21 19:03:22,311][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1157 [2023-06-21 19:03:23,470][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1157 [2023-06-21 19:03:24,138][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1157 [2023-06-21 19:03:42,341][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1158 [2023-06-21 19:03:43,509][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1158 [2023-06-21 19:03:44,154][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1158 [2023-06-21 19:04:02,344][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1159 [2023-06-21 19:04:03,537][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1159 [2023-06-21 19:04:04,189][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1159 [2023-06-21 19:04:22,395][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1160 [2023-06-21 19:04:23,562][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1160 [2023-06-21 19:04:24,239][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1160 [2023-06-21 19:04:42,422][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1161 [2023-06-21 19:04:43,618][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1161 [2023-06-21 19:04:44,256][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1161 [2023-06-21 19:05:02,437][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1162 [2023-06-21 19:05:03,661][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1162 [2023-06-21 19:05:04,292][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1162 [2023-06-21 19:05:22,475][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1163 [2023-06-21 19:05:23,714][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1163 [2023-06-21 19:05:24,305][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1163 [2023-06-21 19:05:42,532][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1164 [2023-06-21 19:05:43,765][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1164 [2023-06-21 19:05:44,348][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1164 [2023-06-21 19:06:02,540][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1165 [2023-06-21 19:06:03,807][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1165 [2023-06-21 19:06:04,366][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1165 [2023-06-21 19:06:22,566][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1166 [2023-06-21 19:06:23,826][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1166 [2023-06-21 19:06:24,371][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1166 [2023-06-21 19:06:42,610][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1167 [2023-06-21 19:06:43,748][62782] Keyboard interrupt detected in the event loop EvtLoop [Runner_EvtLoop, process=main process 62782], exiting... [2023-06-21 19:06:43,757][62782] Runner profile tree view: main_loop: 23380.8911 [2023-06-21 19:06:43,758][62782] Collected {0: 6111488}, FPS: 47.4 [2023-06-21 19:06:43,852][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1167 [2023-06-21 19:06:44,392][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1167 [2023-06-21 19:06:48,680][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 19:06:48,681][62782] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-21 19:06:48,682][62782] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-21 19:06:48,682][62782] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-21 19:06:48,682][62782] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 19:06:48,682][62782] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-21 19:06:48,683][62782] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 19:06:48,683][62782] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-21 19:06:48,683][62782] Adding new argument 'push_to_hub'=False that is not in the saved config file! [2023-06-21 19:06:48,683][62782] Adding new argument 'hf_repository'=None that is not in the saved config file! [2023-06-21 19:06:48,684][62782] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-21 19:06:48,684][62782] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-21 19:06:48,684][62782] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-21 19:06:48,684][62782] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-21 19:06:48,685][62782] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-21 19:06:48,706][62782] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 19:06:48,720][62782] RunningMeanStd input shape: (1,) [2023-06-21 19:06:48,758][62782] ConvEncoder: input_channels=3 [2023-06-21 19:06:48,814][62782] Conv encoder output size: 512 [2023-06-21 19:06:48,815][62782] Policy head output size: 512 [2023-06-21 19:06:48,840][62782] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001636_6111488.pth... [2023-06-21 19:06:51,297][62782] Num frames 100... [2023-06-21 19:06:52,066][62782] Num frames 200... [2023-06-21 19:06:52,875][62782] Num frames 300... [2023-06-21 19:06:53,658][62782] Num frames 400... [2023-06-21 19:06:54,112][62782] Avg episode rewards: #0: 5.480, true rewards: #0: 4.480 [2023-06-21 19:06:54,114][62782] Avg episode reward: 5.480, avg true_objective: 4.480 [2023-06-21 19:06:54,502][62782] Num frames 500... [2023-06-21 19:06:55,296][62782] Num frames 600... [2023-06-21 19:06:56,093][62782] Num frames 700... [2023-06-21 19:06:56,874][62782] Num frames 800... [2023-06-21 19:06:57,216][62782] Avg episode rewards: #0: 5.660, true rewards: #0: 4.160 [2023-06-21 19:06:57,219][62782] Avg episode reward: 5.660, avg true_objective: 4.160 [2023-06-21 19:06:57,745][62782] Num frames 900... [2023-06-21 19:06:58,543][62782] Num frames 1000... [2023-06-21 19:06:59,286][62782] Num frames 1100... [2023-06-21 19:07:00,018][62782] Num frames 1200... [2023-06-21 19:07:00,781][62782] Num frames 1300... [2023-06-21 19:07:01,563][62782] Num frames 1400... [2023-06-21 19:07:02,346][62782] Num frames 1500... [2023-06-21 19:07:02,642][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1168 [2023-06-21 19:07:03,118][62782] Num frames 1600... [2023-06-21 19:07:03,905][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1168 [2023-06-21 19:07:03,926][62782] Num frames 1700... [2023-06-21 19:07:04,448][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1168 [2023-06-21 19:07:04,691][62782] Num frames 1800... [2023-06-21 19:07:04,949][62782] Avg episode rewards: #0: 11.080, true rewards: #0: 6.080 [2023-06-21 19:07:04,952][62782] Avg episode reward: 11.080, avg true_objective: 6.080 [2023-06-21 19:07:05,570][62782] Num frames 1900... [2023-06-21 19:07:06,354][62782] Num frames 2000... [2023-06-21 19:07:07,122][62782] Num frames 2100... [2023-06-21 19:07:07,884][62782] Num frames 2200... [2023-06-21 19:07:08,672][62782] Num frames 2300... [2023-06-21 19:07:09,446][62782] Num frames 2400... [2023-06-21 19:07:10,154][62782] Avg episode rewards: #0: 11.700, true rewards: #0: 6.200 [2023-06-21 19:07:10,156][62782] Avg episode reward: 11.700, avg true_objective: 6.200 [2023-06-21 19:07:10,316][62782] Num frames 2500... [2023-06-21 19:07:11,088][62782] Num frames 2600... [2023-06-21 19:07:11,866][62782] Num frames 2700... [2023-06-21 19:07:12,651][62782] Num frames 2800... [2023-06-21 19:07:13,424][62782] Num frames 2900... [2023-06-21 19:07:14,191][62782] Num frames 3000... [2023-06-21 19:07:14,960][62782] Num frames 3100... [2023-06-21 19:07:15,727][62782] Num frames 3200... [2023-06-21 19:07:16,512][62782] Num frames 3300... [2023-06-21 19:07:17,297][62782] Num frames 3400... [2023-06-21 19:07:18,091][62782] Num frames 3500... [2023-06-21 19:07:18,874][62782] Num frames 3600... [2023-06-21 19:07:19,672][62782] Num frames 3700... [2023-06-21 19:07:20,458][62782] Num frames 3800... [2023-06-21 19:07:21,236][62782] Num frames 3900... [2023-06-21 19:07:22,010][62782] Num frames 4000... [2023-06-21 19:07:22,657][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1169 [2023-06-21 19:07:22,796][62782] Num frames 4100... [2023-06-21 19:07:23,555][62782] Num frames 4200... [2023-06-21 19:07:23,949][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1169 [2023-06-21 19:07:23,952][62782] Avg episode rewards: #0: 18.080, true rewards: #0: 8.480 [2023-06-21 19:07:23,954][62782] Avg episode reward: 18.080, avg true_objective: 8.480 [2023-06-21 19:07:24,415][62782] Num frames 4300... [2023-06-21 19:07:24,482][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1169 [2023-06-21 19:07:25,193][62782] Num frames 4400... [2023-06-21 19:07:25,995][62782] Num frames 4500... [2023-06-21 19:07:26,762][62782] Num frames 4600... [2023-06-21 19:07:27,544][62782] Num frames 4700... [2023-06-21 19:07:28,325][62782] Num frames 4800... [2023-06-21 19:07:29,093][62782] Num frames 4900... [2023-06-21 19:07:29,863][62782] Num frames 5000... [2023-06-21 19:07:30,643][62782] Num frames 5100... [2023-06-21 19:07:31,408][62782] Num frames 5200... [2023-06-21 19:07:31,986][62782] Avg episode rewards: #0: 18.770, true rewards: #0: 8.770 [2023-06-21 19:07:31,988][62782] Avg episode reward: 18.770, avg true_objective: 8.770 [2023-06-21 19:07:32,261][62782] Num frames 5300... [2023-06-21 19:07:33,025][62782] Num frames 5400... [2023-06-21 19:07:33,891][62782] Num frames 5500... [2023-06-21 19:07:34,885][62782] Num frames 5600... [2023-06-21 19:07:35,826][62782] Num frames 5700... [2023-06-21 19:07:36,732][62782] Num frames 5800... [2023-06-21 19:07:37,700][62782] Num frames 5900... [2023-06-21 19:07:38,635][62782] Num frames 6000... [2023-06-21 19:07:39,558][62782] Num frames 6100... [2023-06-21 19:07:40,313][62782] Num frames 6200... [2023-06-21 19:07:41,045][62782] Num frames 6300... [2023-06-21 19:07:41,781][62782] Num frames 6400... [2023-06-21 19:07:42,546][62782] Num frames 6500... [2023-06-21 19:07:42,698][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1170 [2023-06-21 19:07:43,290][62782] Num frames 6600... [2023-06-21 19:07:43,994][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1170 [2023-06-21 19:07:44,045][62782] Num frames 6700... [2023-06-21 19:07:44,483][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1170 [2023-06-21 19:07:44,774][62782] Num frames 6800... [2023-06-21 19:07:45,021][62782] Avg episode rewards: #0: 21.887, true rewards: #0: 9.744 [2023-06-21 19:07:45,023][62782] Avg episode reward: 21.887, avg true_objective: 9.744 [2023-06-21 19:07:45,618][62782] Num frames 6900... [2023-06-21 19:07:46,343][62782] Num frames 7000... [2023-06-21 19:07:47,054][62782] Num frames 7100... [2023-06-21 19:07:47,779][62782] Num frames 7200... [2023-06-21 19:07:48,452][62782] Num frames 7300... [2023-06-21 19:07:49,157][62782] Num frames 7400... [2023-06-21 19:07:49,923][62782] Num frames 7500... [2023-06-21 19:07:50,703][62782] Num frames 7600... [2023-06-21 19:07:51,472][62782] Num frames 7700... [2023-06-21 19:07:51,693][62782] Avg episode rewards: #0: 21.271, true rewards: #0: 9.646 [2023-06-21 19:07:51,695][62782] Avg episode reward: 21.271, avg true_objective: 9.646 [2023-06-21 19:07:52,332][62782] Num frames 7800... [2023-06-21 19:07:53,104][62782] Num frames 7900... [2023-06-21 19:07:53,869][62782] Num frames 8000... [2023-06-21 19:07:54,640][62782] Num frames 8100... [2023-06-21 19:07:55,419][62782] Num frames 8200... [2023-06-21 19:07:55,563][62782] Avg episode rewards: #0: 20.008, true rewards: #0: 9.119 [2023-06-21 19:07:55,565][62782] Avg episode reward: 20.008, avg true_objective: 9.119 [2023-06-21 19:07:56,269][62782] Num frames 8300... [2023-06-21 19:07:56,977][62782] Num frames 8400... [2023-06-21 19:07:57,692][62782] Num frames 8500... [2023-06-21 19:07:58,463][62782] Num frames 8600... [2023-06-21 19:07:59,267][62782] Num frames 8700... [2023-06-21 19:08:00,028][62782] Num frames 8800... [2023-06-21 19:08:00,784][62782] Num frames 8900... [2023-06-21 19:08:01,564][62782] Num frames 9000... [2023-06-21 19:08:02,309][62782] Num frames 9100... [2023-06-21 19:08:02,724][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1171 [2023-06-21 19:08:03,072][62782] Num frames 9200... [2023-06-21 19:08:03,835][62782] Num frames 9300... [2023-06-21 19:08:04,009][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1171 [2023-06-21 19:08:04,536][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1171 [2023-06-21 19:08:04,581][62782] Num frames 9400... [2023-06-21 19:08:05,358][62782] Num frames 9500... [2023-06-21 19:08:06,088][62782] Num frames 9600... [2023-06-21 19:08:06,800][62782] Num frames 9700... [2023-06-21 19:08:06,978][62782] Avg episode rewards: #0: 21.711, true rewards: #0: 9.711 [2023-06-21 19:08:06,981][62782] Avg episode reward: 21.711, avg true_objective: 9.711 [2023-06-21 19:08:20,414][62782] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4! [2023-06-21 19:08:22,764][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1172 [2023-06-21 19:08:24,043][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1172 [2023-06-21 19:08:24,580][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1172 [2023-06-21 19:08:42,806][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1173 [2023-06-21 19:08:44,046][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1173 [2023-06-21 19:08:44,595][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1173 [2023-06-21 19:09:02,827][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1174 [2023-06-21 19:09:03,575][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 19:09:03,577][62782] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-21 19:09:03,578][62782] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-21 19:09:03,578][62782] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-21 19:09:03,579][62782] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 19:09:03,580][62782] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-21 19:09:03,580][62782] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! [2023-06-21 19:09:03,580][62782] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-21 19:09:03,581][62782] Adding new argument 'push_to_hub'=True that is not in the saved config file! [2023-06-21 19:09:03,581][62782] Adding new argument 'hf_repository'='mihirdeo16/vizdoom_health_gathering_supreme' that is not in the saved config file! [2023-06-21 19:09:03,581][62782] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-21 19:09:03,582][62782] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-21 19:09:03,582][62782] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-21 19:09:03,582][62782] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-21 19:09:03,583][62782] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-21 19:09:03,589][62782] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 19:09:03,590][62782] RunningMeanStd input shape: (1,) [2023-06-21 19:09:03,597][62782] ConvEncoder: input_channels=3 [2023-06-21 19:09:03,614][62782] Conv encoder output size: 512 [2023-06-21 19:09:03,615][62782] Policy head output size: 512 [2023-06-21 19:09:03,622][62782] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001636_6111488.pth... [2023-06-21 19:09:04,059][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1174 [2023-06-21 19:09:04,628][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1174 [2023-06-21 19:09:05,138][62782] Num frames 100... [2023-06-21 19:09:05,892][62782] Num frames 200... [2023-06-21 19:09:06,610][62782] Num frames 300... [2023-06-21 19:09:07,343][62782] Num frames 400... [2023-06-21 19:09:08,111][62782] Num frames 500... [2023-06-21 19:09:08,840][62782] Num frames 600... [2023-06-21 19:09:09,599][62782] Num frames 700... [2023-06-21 19:09:10,358][62782] Num frames 800... [2023-06-21 19:09:11,138][62782] Num frames 900... [2023-06-21 19:09:11,895][62782] Num frames 1000... [2023-06-21 19:09:12,686][62782] Num frames 1100... [2023-06-21 19:09:13,463][62782] Num frames 1200... [2023-06-21 19:09:14,238][62782] Num frames 1300... [2023-06-21 19:09:14,996][62782] Num frames 1400... [2023-06-21 19:09:15,751][62782] Num frames 1500... [2023-06-21 19:09:16,220][62782] Avg episode rewards: #0: 43.489, true rewards: #0: 15.490 [2023-06-21 19:09:16,223][62782] Avg episode reward: 43.489, avg true_objective: 15.490 [2023-06-21 19:09:16,610][62782] Num frames 1600... [2023-06-21 19:09:17,388][62782] Num frames 1700... [2023-06-21 19:09:18,178][62782] Num frames 1800... [2023-06-21 19:09:18,940][62782] Num frames 1900... [2023-06-21 19:09:19,709][62782] Num frames 2000... [2023-06-21 19:09:20,488][62782] Num frames 2100... [2023-06-21 19:09:21,261][62782] Num frames 2200... [2023-06-21 19:09:22,033][62782] Num frames 2300... [2023-06-21 19:09:22,260][62782] Avg episode rewards: #0: 29.585, true rewards: #0: 11.585 [2023-06-21 19:09:22,262][62782] Avg episode reward: 29.585, avg true_objective: 11.585 [2023-06-21 19:09:22,842][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1175 [2023-06-21 19:09:22,891][62782] Num frames 2400... [2023-06-21 19:09:23,669][62782] Num frames 2500... [2023-06-21 19:09:24,115][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1175 [2023-06-21 19:09:24,434][62782] Num frames 2600... [2023-06-21 19:09:24,666][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1175 [2023-06-21 19:09:25,188][62782] Num frames 2700... [2023-06-21 19:09:25,967][62782] Num frames 2800... [2023-06-21 19:09:26,744][62782] Num frames 2900... [2023-06-21 19:09:27,521][62782] Num frames 3000... [2023-06-21 19:09:28,283][62782] Num frames 3100... [2023-06-21 19:09:29,069][62782] Num frames 3200... [2023-06-21 19:09:29,850][62782] Num frames 3300... [2023-06-21 19:09:30,628][62782] Num frames 3400... [2023-06-21 19:09:31,411][62782] Num frames 3500... [2023-06-21 19:09:32,185][62782] Num frames 3600... [2023-06-21 19:09:32,965][62782] Num frames 3700... [2023-06-21 19:09:33,644][62782] Avg episode rewards: #0: 32.250, true rewards: #0: 12.583 [2023-06-21 19:09:33,646][62782] Avg episode reward: 32.250, avg true_objective: 12.583 [2023-06-21 19:09:33,832][62782] Num frames 3800... [2023-06-21 19:09:34,602][62782] Num frames 3900... [2023-06-21 19:09:35,370][62782] Num frames 4000... [2023-06-21 19:09:36,150][62782] Num frames 4100... [2023-06-21 19:09:36,921][62782] Num frames 4200... [2023-06-21 19:09:37,693][62782] Num frames 4300... [2023-06-21 19:09:38,456][62782] Num frames 4400... [2023-06-21 19:09:39,233][62782] Num frames 4500... [2023-06-21 19:09:40,004][62782] Num frames 4600... [2023-06-21 19:09:40,783][62782] Num frames 4700... [2023-06-21 19:09:41,548][62782] Num frames 4800... [2023-06-21 19:09:42,320][62782] Num frames 4900... [2023-06-21 19:09:42,880][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1176 [2023-06-21 19:09:43,117][62782] Num frames 5000... [2023-06-21 19:09:43,875][62782] Num frames 5100... [2023-06-21 19:09:44,159][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1176 [2023-06-21 19:09:44,628][62782] Num frames 5200... [2023-06-21 19:09:44,682][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1176 [2023-06-21 19:09:45,386][62782] Num frames 5300... [2023-06-21 19:09:46,150][62782] Num frames 5400... [2023-06-21 19:09:46,930][62782] Num frames 5500... [2023-06-21 19:09:47,693][62782] Num frames 5600... [2023-06-21 19:09:48,482][62782] Num frames 5700... [2023-06-21 19:09:49,260][62782] Num frames 5800... [2023-06-21 19:09:49,961][62782] Avg episode rewards: #0: 39.937, true rewards: #0: 14.687 [2023-06-21 19:09:49,963][62782] Avg episode reward: 39.937, avg true_objective: 14.687 [2023-06-21 19:09:50,148][62782] Num frames 5900... [2023-06-21 19:09:50,909][62782] Num frames 6000... [2023-06-21 19:09:51,687][62782] Num frames 6100... [2023-06-21 19:09:52,450][62782] Num frames 6200... [2023-06-21 19:09:53,236][62782] Num frames 6300... [2023-06-21 19:09:53,747][62782] Avg episode rewards: #0: 33.110, true rewards: #0: 12.710 [2023-06-21 19:09:53,749][62782] Avg episode reward: 33.110, avg true_objective: 12.710 [2023-06-21 19:09:54,096][62782] Num frames 6400... [2023-06-21 19:09:54,859][62782] Num frames 6500... [2023-06-21 19:09:55,635][62782] Num frames 6600... [2023-06-21 19:09:56,430][62782] Num frames 6700... [2023-06-21 19:09:57,286][62782] Num frames 6800... [2023-06-21 19:09:58,103][62782] Num frames 6900... [2023-06-21 19:09:58,896][62782] Num frames 7000... [2023-06-21 19:09:59,689][62782] Num frames 7100... [2023-06-21 19:10:00,470][62782] Num frames 7200... [2023-06-21 19:10:01,248][62782] Num frames 7300... [2023-06-21 19:10:01,978][62782] Num frames 7400... [2023-06-21 19:10:02,803][62782] Num frames 7500... [2023-06-21 19:10:02,893][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1177 [2023-06-21 19:10:03,566][62782] Num frames 7600... [2023-06-21 19:10:04,177][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1177 [2023-06-21 19:10:04,330][62782] Num frames 7700... [2023-06-21 19:10:04,712][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1177 [2023-06-21 19:10:05,081][62782] Num frames 7800... [2023-06-21 19:10:05,896][62782] Num frames 7900... [2023-06-21 19:10:06,676][62782] Num frames 8000... [2023-06-21 19:10:07,453][62782] Num frames 8100... [2023-06-21 19:10:08,242][62782] Num frames 8200... [2023-06-21 19:10:09,027][62782] Num frames 8300... [2023-06-21 19:10:09,822][62782] Avg episode rewards: #0: 36.148, true rewards: #0: 13.982 [2023-06-21 19:10:09,825][62782] Avg episode reward: 36.148, avg true_objective: 13.982 [2023-06-21 19:10:09,904][62782] Num frames 8400... [2023-06-21 19:10:10,679][62782] Num frames 8500... [2023-06-21 19:10:11,472][62782] Num frames 8600... [2023-06-21 19:10:12,236][62782] Num frames 8700... [2023-06-21 19:10:13,006][62782] Num frames 8800... [2023-06-21 19:10:13,778][62782] Num frames 8900... [2023-06-21 19:10:14,548][62782] Num frames 9000... [2023-06-21 19:10:15,330][62782] Num frames 9100... [2023-06-21 19:10:16,109][62782] Num frames 9200... [2023-06-21 19:10:16,894][62782] Num frames 9300... [2023-06-21 19:10:17,725][62782] Num frames 9400... [2023-06-21 19:10:17,916][62782] Avg episode rewards: #0: 34.590, true rewards: #0: 13.447 [2023-06-21 19:10:17,919][62782] Avg episode reward: 34.590, avg true_objective: 13.447 [2023-06-21 19:10:18,582][62782] Num frames 9500... [2023-06-21 19:10:19,364][62782] Num frames 9600... [2023-06-21 19:10:20,137][62782] Num frames 9700... [2023-06-21 19:10:20,906][62782] Num frames 9800... [2023-06-21 19:10:21,705][62782] Num frames 9900... [2023-06-21 19:10:22,260][62782] Avg episode rewards: #0: 31.696, true rewards: #0: 12.446 [2023-06-21 19:10:22,262][62782] Avg episode reward: 31.696, avg true_objective: 12.446 [2023-06-21 19:10:22,605][62782] Num frames 10000... [2023-06-21 19:10:22,943][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1178 [2023-06-21 19:10:23,361][62782] Num frames 10100... [2023-06-21 19:10:24,153][62782] Num frames 10200... [2023-06-21 19:10:24,227][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1178 [2023-06-21 19:10:24,716][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1178 [2023-06-21 19:10:24,908][62782] Num frames 10300... [2023-06-21 19:10:25,709][62782] Num frames 10400... [2023-06-21 19:10:26,475][62782] Num frames 10500... [2023-06-21 19:10:27,286][62782] Num frames 10600... [2023-06-21 19:10:28,063][62782] Num frames 10700... [2023-06-21 19:10:28,832][62782] Num frames 10800... [2023-06-21 19:10:29,618][62782] Num frames 10900... [2023-06-21 19:10:30,404][62782] Num frames 11000... [2023-06-21 19:10:31,038][62782] Avg episode rewards: #0: 31.630, true rewards: #0: 12.297 [2023-06-21 19:10:31,040][62782] Avg episode reward: 31.630, avg true_objective: 12.297 [2023-06-21 19:10:31,286][62782] Num frames 11100... [2023-06-21 19:10:32,067][62782] Num frames 11200... [2023-06-21 19:10:32,847][62782] Num frames 11300... [2023-06-21 19:10:33,674][62782] Num frames 11400... [2023-06-21 19:10:34,428][62782] Num frames 11500... [2023-06-21 19:10:35,245][62782] Num frames 11600... [2023-06-21 19:10:36,036][62782] Num frames 11700... [2023-06-21 19:10:36,184][62782] Avg episode rewards: #0: 29.507, true rewards: #0: 11.707 [2023-06-21 19:10:36,186][62782] Avg episode reward: 29.507, avg true_objective: 11.707 [2023-06-21 19:10:42,992][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1179 [2023-06-21 19:10:44,243][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1179 [2023-06-21 19:10:44,767][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1179 [2023-06-21 19:10:52,019][62782] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4! [2023-06-21 19:11:03,017][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1180 [2023-06-21 19:11:04,285][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1180 [2023-06-21 19:11:04,783][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1180 [2023-06-21 19:11:23,050][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1181 [2023-06-21 19:11:24,304][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1181 [2023-06-21 19:11:24,823][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1181 [2023-06-21 19:11:43,076][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1182 [2023-06-21 19:11:44,324][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1182 [2023-06-21 19:11:44,832][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1182 [2023-06-21 19:12:03,089][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1183 [2023-06-21 19:12:04,363][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1183 [2023-06-21 19:12:04,868][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1183 [2023-06-21 19:12:14,835][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 19:12:14,837][62782] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-21 19:12:14,838][62782] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-21 19:12:14,838][62782] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-21 19:12:14,839][62782] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 19:12:14,840][62782] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-21 19:12:14,840][62782] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! [2023-06-21 19:12:14,841][62782] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-21 19:12:14,841][62782] Adding new argument 'push_to_hub'=True that is not in the saved config file! [2023-06-21 19:12:14,842][62782] Adding new argument 'hf_repository'='mihirdeo16/vizdoom_health_gathering_supreme' that is not in the saved config file! [2023-06-21 19:12:14,842][62782] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-21 19:12:14,843][62782] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-21 19:12:14,843][62782] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-21 19:12:14,843][62782] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-21 19:12:14,844][62782] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-21 19:12:14,852][62782] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 19:12:14,855][62782] RunningMeanStd input shape: (1,) [2023-06-21 19:12:14,873][62782] ConvEncoder: input_channels=3 [2023-06-21 19:12:14,902][62782] Conv encoder output size: 512 [2023-06-21 19:12:14,904][62782] Policy head output size: 512 [2023-06-21 19:12:14,912][62782] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001636_6111488.pth... [2023-06-21 19:12:16,431][62782] Num frames 100... [2023-06-21 19:12:17,167][62782] Num frames 200... [2023-06-21 19:12:17,936][62782] Num frames 300... [2023-06-21 19:12:18,713][62782] Num frames 400... [2023-06-21 19:12:19,470][62782] Num frames 500... [2023-06-21 19:12:20,230][62782] Num frames 600... [2023-06-21 19:12:21,018][62782] Num frames 700... [2023-06-21 19:12:21,780][62782] Num frames 800... [2023-06-21 19:12:22,561][62782] Num frames 900... [2023-06-21 19:12:23,109][62782] Avg episode rewards: #0: 18.600, true rewards: #0: 9.600 [2023-06-21 19:12:23,110][62782] Avg episode reward: 18.600, avg true_objective: 9.600 [2023-06-21 19:12:23,121][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1184 [2023-06-21 19:12:23,388][62782] Num frames 1000... [2023-06-21 19:12:24,124][62782] Num frames 1100... [2023-06-21 19:12:24,413][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1184 [2023-06-21 19:12:24,868][62782] Num frames 1200... [2023-06-21 19:12:24,906][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1184 [2023-06-21 19:12:25,618][62782] Num frames 1300... [2023-06-21 19:12:26,387][62782] Num frames 1400... [2023-06-21 19:12:26,547][62782] Avg episode rewards: #0: 12.040, true rewards: #0: 7.040 [2023-06-21 19:12:26,549][62782] Avg episode reward: 12.040, avg true_objective: 7.040 [2023-06-21 19:12:27,270][62782] Num frames 1500... [2023-06-21 19:12:28,051][62782] Num frames 1600... [2023-06-21 19:12:28,837][62782] Num frames 1700... [2023-06-21 19:12:29,593][62782] Num frames 1800... [2023-06-21 19:12:30,372][62782] Num frames 1900... [2023-06-21 19:12:31,156][62782] Num frames 2000... [2023-06-21 19:12:31,918][62782] Num frames 2100... [2023-06-21 19:12:32,730][62782] Avg episode rewards: #0: 13.313, true rewards: #0: 7.313 [2023-06-21 19:12:32,733][62782] Avg episode reward: 13.313, avg true_objective: 7.313 [2023-06-21 19:12:32,778][62782] Num frames 2200... [2023-06-21 19:12:33,532][62782] Num frames 2300... [2023-06-21 19:12:34,305][62782] Num frames 2400... [2023-06-21 19:12:35,073][62782] Num frames 2500... [2023-06-21 19:12:35,836][62782] Num frames 2600... [2023-06-21 19:12:36,513][62782] Avg episode rewards: #0: 11.935, true rewards: #0: 6.685 [2023-06-21 19:12:36,516][62782] Avg episode reward: 11.935, avg true_objective: 6.685 [2023-06-21 19:12:36,713][62782] Num frames 2700... [2023-06-21 19:12:37,482][62782] Num frames 2800... [2023-06-21 19:12:38,264][62782] Num frames 2900... [2023-06-21 19:12:39,070][62782] Num frames 3000... [2023-06-21 19:12:39,852][62782] Num frames 3100... [2023-06-21 19:12:40,633][62782] Num frames 3200... [2023-06-21 19:12:41,386][62782] Num frames 3300... [2023-06-21 19:12:42,158][62782] Num frames 3400... [2023-06-21 19:12:42,925][62782] Num frames 3500... [2023-06-21 19:12:43,147][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1185 [2023-06-21 19:12:43,697][62782] Num frames 3600... [2023-06-21 19:12:44,451][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1185 [2023-06-21 19:12:44,487][62782] Num frames 3700... [2023-06-21 19:12:44,934][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1185 [2023-06-21 19:12:45,262][62782] Num frames 3800... [2023-06-21 19:12:46,031][62782] Num frames 3900... [2023-06-21 19:12:46,804][62782] Num frames 4000... [2023-06-21 19:12:47,555][62782] Num frames 4100... [2023-06-21 19:12:48,244][62782] Avg episode rewards: #0: 17.356, true rewards: #0: 8.356 [2023-06-21 19:12:48,246][62782] Avg episode reward: 17.356, avg true_objective: 8.356 [2023-06-21 19:12:48,411][62782] Num frames 4200... [2023-06-21 19:12:49,194][62782] Num frames 4300... [2023-06-21 19:12:49,956][62782] Num frames 4400... [2023-06-21 19:12:50,780][62782] Num frames 4500... [2023-06-21 19:12:51,681][62782] Num frames 4600... [2023-06-21 19:12:52,595][62782] Num frames 4700... [2023-06-21 19:12:53,520][62782] Num frames 4800... [2023-06-21 19:12:54,015][62782] Avg episode rewards: #0: 17.078, true rewards: #0: 8.078 [2023-06-21 19:12:54,018][62782] Avg episode reward: 17.078, avg true_objective: 8.078 [2023-06-21 19:12:54,454][62782] Num frames 4900... [2023-06-21 19:12:55,322][62782] Num frames 5000... [2023-06-21 19:12:56,101][62782] Num frames 5100... [2023-06-21 19:12:56,864][62782] Num frames 5200... [2023-06-21 19:12:57,649][62782] Num frames 5300... [2023-06-21 19:12:58,387][62782] Num frames 5400... [2023-06-21 19:12:59,288][62782] Num frames 5500... [2023-06-21 19:13:00,207][62782] Num frames 5600... [2023-06-21 19:13:01,082][62782] Num frames 5700... [2023-06-21 19:13:01,929][62782] Num frames 5800... [2023-06-21 19:13:02,766][62782] Num frames 5900... [2023-06-21 19:13:03,168][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1186 [2023-06-21 19:13:03,594][62782] Num frames 6000... [2023-06-21 19:13:04,502][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1186 [2023-06-21 19:13:04,517][62782] Num frames 6100... [2023-06-21 19:13:04,961][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1186 [2023-06-21 19:13:05,434][62782] Num frames 6200... [2023-06-21 19:13:06,301][62782] Num frames 6300... [2023-06-21 19:13:07,246][62782] Num frames 6400... [2023-06-21 19:13:08,061][62782] Num frames 6500... [2023-06-21 19:13:09,022][62782] Num frames 6600... [2023-06-21 19:13:09,964][62782] Num frames 6700... [2023-06-21 19:13:10,783][62782] Num frames 6800... [2023-06-21 19:13:11,655][62782] Num frames 6900... [2023-06-21 19:13:12,145][62782] Avg episode rewards: #0: 22.210, true rewards: #0: 9.924 [2023-06-21 19:13:12,146][62782] Avg episode reward: 22.210, avg true_objective: 9.924 [2023-06-21 19:13:12,615][62782] Num frames 7000... [2023-06-21 19:13:13,478][62782] Num frames 7100... [2023-06-21 19:13:14,291][62782] Num frames 7200... [2023-06-21 19:13:15,177][62782] Num frames 7300... [2023-06-21 19:13:16,023][62782] Num frames 7400... [2023-06-21 19:13:16,910][62782] Num frames 7500... [2023-06-21 19:13:17,829][62782] Num frames 7600... [2023-06-21 19:13:18,727][62782] Num frames 7700... [2023-06-21 19:13:19,544][62782] Num frames 7800... [2023-06-21 19:13:20,406][62782] Num frames 7900... [2023-06-21 19:13:21,274][62782] Num frames 8000... [2023-06-21 19:13:22,161][62782] Num frames 8100... [2023-06-21 19:13:22,269][62782] Avg episode rewards: #0: 22.376, true rewards: #0: 10.126 [2023-06-21 19:13:22,271][62782] Avg episode reward: 22.376, avg true_objective: 10.126 [2023-06-21 19:13:23,152][62782] Num frames 8200... [2023-06-21 19:13:23,201][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1187 [2023-06-21 19:13:23,950][62782] Num frames 8300... [2023-06-21 19:13:24,525][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1187 [2023-06-21 19:13:24,750][62782] Num frames 8400... [2023-06-21 19:13:24,971][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1187 [2023-06-21 19:13:25,543][62782] Num frames 8500... [2023-06-21 19:13:26,343][62782] Num frames 8600... [2023-06-21 19:13:27,113][62782] Num frames 8700... [2023-06-21 19:13:27,898][62782] Num frames 8800... [2023-06-21 19:13:28,693][62782] Num frames 8900... [2023-06-21 19:13:29,494][62782] Num frames 9000... [2023-06-21 19:13:30,323][62782] Num frames 9100... [2023-06-21 19:13:31,118][62782] Num frames 9200... [2023-06-21 19:13:31,982][62782] Num frames 9300... [2023-06-21 19:13:32,805][62782] Num frames 9400... [2023-06-21 19:13:33,639][62782] Num frames 9500... [2023-06-21 19:13:34,041][62782] Avg episode rewards: #0: 23.379, true rewards: #0: 10.601 [2023-06-21 19:13:34,041][62782] Avg episode reward: 23.379, avg true_objective: 10.601 [2023-06-21 19:13:34,504][62782] Num frames 9600... [2023-06-21 19:13:35,311][62782] Num frames 9700... [2023-06-21 19:13:36,101][62782] Num frames 9800... [2023-06-21 19:13:36,905][62782] Num frames 9900... [2023-06-21 19:13:37,763][62782] Num frames 10000... [2023-06-21 19:13:38,573][62782] Num frames 10100... [2023-06-21 19:13:39,403][62782] Num frames 10200... [2023-06-21 19:13:40,229][62782] Num frames 10300... [2023-06-21 19:13:41,028][62782] Num frames 10400... [2023-06-21 19:13:41,833][62782] Num frames 10500... [2023-06-21 19:13:42,644][62782] Num frames 10600... [2023-06-21 19:13:43,238][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1188 [2023-06-21 19:13:43,458][62782] Num frames 10700... [2023-06-21 19:13:44,326][62782] Num frames 10800... [2023-06-21 19:13:44,555][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1188 [2023-06-21 19:13:45,008][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1188 [2023-06-21 19:13:45,192][62782] Avg episode rewards: #0: 24.497, true rewards: #0: 10.897 [2023-06-21 19:13:45,194][62782] Avg episode reward: 24.497, avg true_objective: 10.897 [2023-06-21 19:13:45,216][62782] Num frames 10900... [2023-06-21 19:14:00,775][62782] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4! [2023-06-21 19:14:03,251][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1189 [2023-06-21 19:14:04,585][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1189 [2023-06-21 19:14:05,039][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1189 [2023-06-21 19:14:23,287][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1190 [2023-06-21 19:14:24,622][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1190 [2023-06-21 19:14:25,080][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1190 [2023-06-21 19:14:43,305][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1191 [2023-06-21 19:14:44,634][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1191 [2023-06-21 19:14:45,116][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1191 [2023-06-21 19:15:03,311][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1192 [2023-06-21 19:15:04,682][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1192 [2023-06-21 19:15:05,119][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1192 [2023-06-21 19:15:23,341][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1193 [2023-06-21 19:15:24,685][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1193 [2023-06-21 19:15:25,164][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1193 [2023-06-21 19:15:28,251][62782] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 19:15:28,253][62782] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-21 19:15:28,253][62782] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-21 19:15:28,254][62782] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-21 19:15:28,255][62782] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 19:15:28,255][62782] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-21 19:15:28,256][62782] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! [2023-06-21 19:15:28,256][62782] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-21 19:15:28,257][62782] Adding new argument 'push_to_hub'=True that is not in the saved config file! [2023-06-21 19:15:28,257][62782] Adding new argument 'hf_repository'='mihirdeo16/vizdoom_health_gathering_supreme' that is not in the saved config file! [2023-06-21 19:15:28,258][62782] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-21 19:15:28,258][62782] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-21 19:15:28,259][62782] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-21 19:15:28,260][62782] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-21 19:15:28,260][62782] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-21 19:15:28,281][62782] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 19:15:28,284][62782] RunningMeanStd input shape: (1,) [2023-06-21 19:15:28,302][62782] ConvEncoder: input_channels=3 [2023-06-21 19:15:28,320][62782] Conv encoder output size: 512 [2023-06-21 19:15:28,321][62782] Policy head output size: 512 [2023-06-21 19:15:28,330][62782] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001636_6111488.pth... [2023-06-21 19:15:29,962][62782] Num frames 100... [2023-06-21 19:15:30,869][62782] Num frames 200... [2023-06-21 19:15:31,756][62782] Num frames 300... [2023-06-21 19:15:32,687][62782] Num frames 400... [2023-06-21 19:15:32,933][62782] Avg episode rewards: #0: 8.160, true rewards: #0: 4.160 [2023-06-21 19:15:32,936][62782] Avg episode reward: 8.160, avg true_objective: 4.160 [2023-06-21 19:15:33,670][62782] Num frames 500... [2023-06-21 19:15:34,533][62782] Num frames 600... [2023-06-21 19:15:35,346][62782] Num frames 700... [2023-06-21 19:15:36,151][62782] Num frames 800... [2023-06-21 19:15:36,205][62782] Avg episode rewards: #0: 7.000, true rewards: #0: 4.000 [2023-06-21 19:15:36,206][62782] Avg episode reward: 7.000, avg true_objective: 4.000 [2023-06-21 19:15:36,977][62782] Num frames 900... [2023-06-21 19:15:37,743][62782] Num frames 1000... [2023-06-21 19:15:38,534][62782] Num frames 1100... [2023-06-21 19:15:39,311][62782] Num frames 1200... [2023-06-21 19:15:40,107][62782] Num frames 1300... [2023-06-21 19:15:40,877][62782] Num frames 1400... [2023-06-21 19:15:41,252][62782] Avg episode rewards: #0: 8.467, true rewards: #0: 4.800 [2023-06-21 19:15:41,253][62782] Avg episode reward: 8.467, avg true_objective: 4.800 [2023-06-21 19:15:41,694][62782] Num frames 1500... [2023-06-21 19:15:42,475][62782] Num frames 1600... [2023-06-21 19:15:43,261][62782] Num frames 1700... [2023-06-21 19:15:43,399][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1194 [2023-06-21 19:15:44,051][62782] Num frames 1800... [2023-06-21 19:15:44,340][62782] Avg episode rewards: #0: 7.810, true rewards: #0: 4.560 [2023-06-21 19:15:44,340][62782] Avg episode reward: 7.810, avg true_objective: 4.560 [2023-06-21 19:15:44,742][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1194 [2023-06-21 19:15:44,955][62782] Num frames 1900... [2023-06-21 19:15:45,175][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1194 [2023-06-21 19:15:45,721][62782] Num frames 2000... [2023-06-21 19:15:46,488][62782] Num frames 2100... [2023-06-21 19:15:47,277][62782] Num frames 2200... [2023-06-21 19:15:48,034][62782] Num frames 2300... [2023-06-21 19:15:48,825][62782] Num frames 2400... [2023-06-21 19:15:49,592][62782] Num frames 2500... [2023-06-21 19:15:50,357][62782] Num frames 2600... [2023-06-21 19:15:51,136][62782] Num frames 2700... [2023-06-21 19:15:51,888][62782] Num frames 2800... [2023-06-21 19:15:52,676][62782] Num frames 2900... [2023-06-21 19:15:53,403][62782] Num frames 3000... [2023-06-21 19:15:54,188][62782] Num frames 3100... [2023-06-21 19:15:54,946][62782] Num frames 3200... [2023-06-21 19:15:55,662][62782] Num frames 3300... [2023-06-21 19:15:56,429][62782] Num frames 3400... [2023-06-21 19:15:57,155][62782] Num frames 3500... [2023-06-21 19:15:58,089][62782] Num frames 3600... [2023-06-21 19:15:58,990][62782] Num frames 3700... [2023-06-21 19:15:59,911][62782] Num frames 3800... [2023-06-21 19:16:00,765][62782] Num frames 3900... [2023-06-21 19:16:01,043][62782] Avg episode rewards: #0: 19.048, true rewards: #0: 7.848 [2023-06-21 19:16:01,046][62782] Avg episode reward: 19.048, avg true_objective: 7.848 [2023-06-21 19:16:01,667][62782] Num frames 4000... [2023-06-21 19:16:02,451][62782] Num frames 4100... [2023-06-21 19:16:03,296][62782] Num frames 4200... [2023-06-21 19:16:03,410][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1195 [2023-06-21 19:16:04,110][62782] Num frames 4300... [2023-06-21 19:16:04,788][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1195 [2023-06-21 19:16:04,895][62782] Num frames 4400... [2023-06-21 19:16:05,191][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1195 [2023-06-21 19:16:05,750][62782] Num frames 4500... [2023-06-21 19:16:06,601][62782] Num frames 4600... [2023-06-21 19:16:07,491][62782] Num frames 4700... [2023-06-21 19:16:08,332][62782] Num frames 4800... [2023-06-21 19:16:09,223][62782] Num frames 4900... [2023-06-21 19:16:10,086][62782] Num frames 5000... [2023-06-21 19:16:10,942][62782] Num frames 5100... [2023-06-21 19:16:11,317][62782] Avg episode rewards: #0: 20.225, true rewards: #0: 8.558 [2023-06-21 19:16:11,319][62782] Avg episode reward: 20.225, avg true_objective: 8.558 [2023-06-21 19:16:11,873][62782] Num frames 5200... [2023-06-21 19:16:12,730][62782] Num frames 5300... [2023-06-21 19:16:13,588][62782] Num frames 5400... [2023-06-21 19:16:14,459][62782] Num frames 5500... [2023-06-21 19:16:15,310][62782] Avg episode rewards: #0: 19.118, true rewards: #0: 7.976 [2023-06-21 19:16:15,312][62782] Avg episode reward: 19.118, avg true_objective: 7.976 [2023-06-21 19:16:15,458][62782] Num frames 5600... [2023-06-21 19:16:16,321][62782] Num frames 5700... [2023-06-21 19:16:17,177][62782] Num frames 5800... [2023-06-21 19:16:18,083][62782] Num frames 5900... [2023-06-21 19:16:19,006][62782] Num frames 6000... [2023-06-21 19:16:19,822][62782] Num frames 6100... [2023-06-21 19:16:20,137][62782] Avg episode rewards: #0: 17.659, true rewards: #0: 7.659 [2023-06-21 19:16:20,138][62782] Avg episode reward: 17.659, avg true_objective: 7.659 [2023-06-21 19:16:20,784][62782] Num frames 6200... [2023-06-21 19:16:21,702][62782] Num frames 6300... [2023-06-21 19:16:22,568][62782] Num frames 6400... [2023-06-21 19:16:23,422][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1196 [2023-06-21 19:16:23,455][62782] Num frames 6500... [2023-06-21 19:16:24,307][62782] Num frames 6600... [2023-06-21 19:16:24,811][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1196 [2023-06-21 19:16:25,097][62782] Avg episode rewards: #0: 16.868, true rewards: #0: 7.423 [2023-06-21 19:16:25,099][62782] Avg episode reward: 16.868, avg true_objective: 7.423 [2023-06-21 19:16:25,208][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1196 [2023-06-21 19:16:25,252][62782] Num frames 6700... [2023-06-21 19:16:26,126][62782] Num frames 6800... [2023-06-21 19:16:26,991][62782] Num frames 6900... [2023-06-21 19:16:27,916][62782] Num frames 7000... [2023-06-21 19:16:28,803][62782] Num frames 7100... [2023-06-21 19:16:29,625][62782] Num frames 7200... [2023-06-21 19:16:30,596][62782] Num frames 7300... [2023-06-21 19:16:31,527][62782] Num frames 7400... [2023-06-21 19:16:32,443][62782] Num frames 7500... [2023-06-21 19:16:33,334][62782] Num frames 7600... [2023-06-21 19:16:34,157][62782] Num frames 7700... [2023-06-21 19:16:35,038][62782] Num frames 7800... [2023-06-21 19:16:35,980][62782] Num frames 7900... [2023-06-21 19:16:36,818][62782] Num frames 8000... [2023-06-21 19:16:37,541][62782] Num frames 8100... [2023-06-21 19:16:38,306][62782] Num frames 8200... [2023-06-21 19:16:39,012][62782] Num frames 8300... [2023-06-21 19:16:39,356][62782] Avg episode rewards: #0: 19.035, true rewards: #0: 8.335 [2023-06-21 19:16:39,357][62782] Avg episode reward: 19.035, avg true_objective: 8.335 [2023-06-21 19:16:43,444][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1197 [2023-06-21 19:16:44,853][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1197 [2023-06-21 19:16:45,213][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1197 [2023-06-21 19:16:50,729][62782] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4! [2023-06-21 19:17:03,450][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1198 [2023-06-21 19:17:04,899][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1198 [2023-06-21 19:17:05,230][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1198 [2023-06-21 19:17:13,689][62782] The model has been pushed to https://huggingface.co/mihirdeo16/vizdoom_health_gathering_supreme [2023-06-21 19:17:23,503][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1199 [2023-06-21 19:17:24,937][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1199 [2023-06-21 19:17:25,270][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1199 [2023-06-21 19:17:43,528][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1200 [2023-06-21 19:17:44,988][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1200 [2023-06-21 19:17:45,287][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1200 [2023-06-21 19:18:03,538][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1201 [2023-06-21 19:18:05,035][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1201 [2023-06-21 19:18:05,319][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1201 [2023-06-21 19:18:23,543][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1202 [2023-06-21 19:18:25,067][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1202 [2023-06-21 19:18:25,331][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1202 [2023-06-21 19:18:43,577][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1203 [2023-06-21 19:18:45,091][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1203 [2023-06-21 19:18:45,368][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1203 [2023-06-21 19:19:03,619][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1204 [2023-06-21 19:19:05,106][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1204 [2023-06-21 19:19:05,385][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1204 [2023-06-21 19:19:23,676][78408] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1205 [2023-06-21 19:19:25,133][78405] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1205 [2023-06-21 19:19:25,439][78409] Another process currently holds the lock /var/folders/t0/cgrmhypx1kg2h122jm20kbnm0000gn/T/sf2_md/doom_002.lockfile, attempt: 1205 [2023-06-21 19:19:30,406][78404] VizDoom game.init() threw an exception ViZDoomErrorException('Unexpected ViZDoom instance crash.'). Terminate process... [2023-06-21 19:19:30,426][78404] EvtLoop [rollout_proc2_evt_loop, process=rollout_proc2] unhandled exception in slot='init' connected to emitter=Emitter(object_id='Sampler', signal_name='_inference_workers_initialized'), args=() Traceback (most recent call last): File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sf_examples/vizdoom/doom/doom_gym.py", line 228, in _game_init self.game.init() vizdoom.vizdoom.ViZDoomErrorException: Unexpected ViZDoom instance crash. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/signal_slot/signal_slot.py", line 355, in _process_signal slot_callable(*args) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sample_factory/algo/sampling/rollout_worker.py", line 150, in init env_runner.init(self.timing) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sample_factory/algo/sampling/non_batched_sampling.py", line 418, in init self._reset() File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sample_factory/algo/sampling/non_batched_sampling.py", line 430, in _reset observations, info = e.reset(seed=seed) # new way of doing seeding since Gym 0.26.0 File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/gymnasium/core.py", line 414, in reset return self.env.reset(seed=seed, options=options) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sample_factory/algo/utils/make_env.py", line 125, in reset obs, info = self.env.reset(**kwargs) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sample_factory/algo/utils/make_env.py", line 110, in reset obs, info = self.env.reset(**kwargs) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sf_examples/vizdoom/doom/wrappers/scenario_wrappers/gathering_reward_shaping.py", line 30, in reset return self.env.reset(**kwargs) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/gymnasium/core.py", line 462, in reset obs, info = self.env.reset(seed=seed, options=options) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sample_factory/envs/env_wrappers.py", line 82, in reset obs, info = self.env.reset(**kwargs) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/gymnasium/core.py", line 414, in reset return self.env.reset(seed=seed, options=options) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sf_examples/vizdoom/doom/wrappers/multiplayer_stats.py", line 51, in reset return self.env.reset(**kwargs) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sf_examples/vizdoom/doom/doom_gym.py", line 323, in reset self._ensure_initialized() File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sf_examples/vizdoom/doom/doom_gym.py", line 274, in _ensure_initialized self.initialize() File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sf_examples/vizdoom/doom/doom_gym.py", line 269, in initialize self._game_init() File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sf_examples/vizdoom/doom/doom_gym.py", line 244, in _game_init raise EnvCriticalError() sample_factory.envs.env_utils.EnvCriticalError [2023-06-21 19:19:30,439][78404] Unhandled exception in evt loop rollout_proc2_evt_loop [2023-06-21 19:19:31,063][78408] Decorrelating experience for 32 frames... [2023-06-21 19:19:31,698][78405] Decorrelating experience for 32 frames... [2023-06-21 19:19:32,297][78409] Decorrelating experience for 64 frames... [2023-06-21 19:19:32,548][78408] Decorrelating experience for 64 frames... [2023-06-21 19:19:33,144][78405] Decorrelating experience for 64 frames... [2023-06-21 19:19:34,607][78409] Decorrelating experience for 96 frames... [2023-06-21 19:19:34,789][78408] Decorrelating experience for 96 frames... [2023-06-21 19:19:35,458][78405] Decorrelating experience for 96 frames... [2023-06-21 19:19:37,212][78409] Stopping RolloutWorker_w6... [2023-06-21 19:19:37,212][78409] Loop rollout_proc6_evt_loop terminating... [2023-06-21 19:19:38,966][78405] EvtLoop [rollout_proc3_evt_loop, process=rollout_proc3] unhandled exception in slot='init' connected to emitter=Emitter(object_id='Sampler', signal_name='_inference_workers_initialized'), args=() Traceback (most recent call last): File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/signal_slot/signal_slot.py", line 355, in _process_signal slot_callable(*args) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sample_factory/algo/sampling/rollout_worker.py", line 150, in init env_runner.init(self.timing) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sample_factory/algo/sampling/non_batched_sampling.py", line 418, in init self._reset() File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sample_factory/algo/sampling/non_batched_sampling.py", line 439, in _reset observations, rew, terminated, truncated, info = e.step(actions) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/gymnasium/core.py", line 408, in step return self.env.step(action) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sample_factory/algo/utils/make_env.py", line 129, in step obs, rew, terminated, truncated, info = self.env.step(action) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sample_factory/algo/utils/make_env.py", line 115, in step obs, rew, terminated, truncated, info = self.env.step(action) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sf_examples/vizdoom/doom/wrappers/scenario_wrappers/gathering_reward_shaping.py", line 33, in step observation, reward, terminated, truncated, info = self.env.step(action) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/gymnasium/core.py", line 469, in step observation, reward, terminated, truncated, info = self.env.step(action) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sample_factory/envs/env_wrappers.py", line 86, in step obs, reward, terminated, truncated, info = self.env.step(action) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/gymnasium/core.py", line 408, in step return self.env.step(action) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sf_examples/vizdoom/doom/wrappers/multiplayer_stats.py", line 54, in step obs, reward, terminated, truncated, info = self.env.step(action) File "/Users/md/opt/miniconda3/envs/hf/lib/python3.10/site-packages/sf_examples/vizdoom/doom/doom_gym.py", line 452, in step reward = self.game.make_action(actions_flattened, self.skip_frames) vizdoom.vizdoom.ViZDoomUnexpectedExitException: Controlled ViZDoom instance exited unexpectedly. [2023-06-21 19:19:38,971][78405] Unhandled exception Controlled ViZDoom instance exited unexpectedly. in evt loop rollout_proc3_evt_loop [2023-06-21 20:05:30,124][03300] Saving configuration to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json... [2023-06-21 20:05:30,146][03300] Rollout worker 0 uses device cpu [2023-06-21 20:05:30,147][03300] Rollout worker 1 uses device cpu [2023-06-21 20:05:30,148][03300] Rollout worker 2 uses device cpu [2023-06-21 20:05:30,148][03300] Rollout worker 3 uses device cpu [2023-06-21 20:05:30,148][03300] Rollout worker 4 uses device cpu [2023-06-21 20:05:30,148][03300] Rollout worker 5 uses device cpu [2023-06-21 20:05:30,149][03300] Rollout worker 6 uses device cpu [2023-06-21 20:05:30,149][03300] Rollout worker 7 uses device cpu [2023-06-21 20:05:30,359][03300] InferenceWorker_p0-w0: min num requests: 2 [2023-06-21 20:05:30,393][03300] Starting all processes... [2023-06-21 20:05:30,394][03300] Starting process learner_proc0 [2023-06-21 20:05:30,448][03300] Starting all processes... [2023-06-21 20:05:30,452][03300] Starting process inference_proc0-0 [2023-06-21 20:05:30,452][03300] Starting process rollout_proc0 [2023-06-21 20:05:30,452][03300] Starting process rollout_proc1 [2023-06-21 20:05:30,452][03300] Starting process rollout_proc2 [2023-06-21 20:05:30,452][03300] Starting process rollout_proc3 [2023-06-21 20:05:30,452][03300] Starting process rollout_proc4 [2023-06-21 20:05:30,452][03300] Starting process rollout_proc5 [2023-06-21 20:05:30,453][03300] Starting process rollout_proc6 [2023-06-21 20:05:30,453][03300] Starting process rollout_proc7 [2023-06-21 20:05:32,460][03737] On MacOS, not setting affinity [2023-06-21 20:05:32,464][03739] On MacOS, not setting affinity [2023-06-21 20:05:32,478][03736] Starting seed is not provided [2023-06-21 20:05:32,478][03736] Initializing actor-critic model on device cpu [2023-06-21 20:05:32,478][03736] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 20:05:32,483][03736] RunningMeanStd input shape: (1,) [2023-06-21 20:05:32,494][03740] On MacOS, not setting affinity [2023-06-21 20:05:32,494][03742] On MacOS, not setting affinity [2023-06-21 20:05:32,499][03736] ConvEncoder: input_channels=3 [2023-06-21 20:05:32,560][03743] On MacOS, not setting affinity [2023-06-21 20:05:32,579][03744] On MacOS, not setting affinity [2023-06-21 20:05:32,584][03745] On MacOS, not setting affinity [2023-06-21 20:05:32,591][03741] On MacOS, not setting affinity [2023-06-21 20:05:32,609][03736] Conv encoder output size: 512 [2023-06-21 20:05:32,609][03736] Policy head output size: 512 [2023-06-21 20:05:32,628][03736] Created Actor Critic model with architecture: [2023-06-21 20:05:32,629][03736] ActorCriticSharedWeights( (obs_normalizer): ObservationNormalizer( (running_mean_std): RunningMeanStdDictInPlace( (running_mean_std): ModuleDict( (obs): RunningMeanStdInPlace() ) ) ) (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace) (encoder): VizdoomEncoder( (basic_encoder): ConvEncoder( (enc): RecursiveScriptModule( original_name=ConvEncoderImpl (conv_head): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Conv2d) (1): RecursiveScriptModule(original_name=ELU) (2): RecursiveScriptModule(original_name=Conv2d) (3): RecursiveScriptModule(original_name=ELU) (4): RecursiveScriptModule(original_name=Conv2d) (5): RecursiveScriptModule(original_name=ELU) ) (mlp_layers): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Linear) (1): RecursiveScriptModule(original_name=ELU) ) ) ) ) (core): ModelCoreRNN( (core): GRU(512, 512) ) (decoder): MlpDecoder( (mlp): Identity() ) (critic_linear): Linear(in_features=512, out_features=1, bias=True) (action_parameterization): ActionParameterizationDefault( (distribution_linear): Linear(in_features=512, out_features=5, bias=True) ) ) [2023-06-21 20:05:32,635][03736] Using optimizer [2023-06-21 20:05:32,636][03736] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001636_6111488.pth... [2023-06-21 20:05:32,674][03736] Loading model from checkpoint [2023-06-21 20:05:32,680][03736] Loaded experiment state at self.train_step=1636, self.env_steps=6111488 [2023-06-21 20:05:32,681][03736] Initialized policy 0 weights for model version 1636 [2023-06-21 20:05:32,682][03736] LearnerWorker_p0 finished initialization! [2023-06-21 20:05:32,685][03738] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 20:05:32,685][03738] RunningMeanStd input shape: (1,) [2023-06-21 20:05:32,699][03738] ConvEncoder: input_channels=3 [2023-06-21 20:05:32,751][03738] Conv encoder output size: 512 [2023-06-21 20:05:32,751][03738] Policy head output size: 512 [2023-06-21 20:05:32,760][03300] Inference worker 0-0 is ready! [2023-06-21 20:05:32,761][03300] All inference workers are ready! Signal rollout workers to start! [2023-06-21 20:05:32,797][03744] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 20:05:32,809][03743] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 20:05:32,809][03740] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 20:05:32,810][03745] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 20:05:32,815][03742] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 20:05:32,816][03741] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 20:05:32,820][03739] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 20:05:32,819][03737] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 20:05:34,186][03741] Decorrelating experience for 0 frames... [2023-06-21 20:05:34,188][03737] Decorrelating experience for 0 frames... [2023-06-21 20:05:34,188][03743] Decorrelating experience for 0 frames... [2023-06-21 20:05:34,192][03740] Decorrelating experience for 0 frames... [2023-06-21 20:05:34,195][03742] Decorrelating experience for 0 frames... [2023-06-21 20:05:34,203][03739] Decorrelating experience for 0 frames... [2023-06-21 20:05:34,307][03300] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 6111488. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 20:05:35,168][03740] Decorrelating experience for 32 frames... [2023-06-21 20:05:35,177][03741] Decorrelating experience for 32 frames... [2023-06-21 20:05:35,180][03737] Decorrelating experience for 32 frames... [2023-06-21 20:05:35,180][03744] Decorrelating experience for 0 frames... [2023-06-21 20:05:35,183][03743] Decorrelating experience for 32 frames... [2023-06-21 20:05:35,183][03739] Decorrelating experience for 32 frames... [2023-06-21 20:05:35,943][03744] Decorrelating experience for 32 frames... [2023-06-21 20:05:35,945][03742] Decorrelating experience for 32 frames... [2023-06-21 20:05:36,706][03745] Decorrelating experience for 0 frames... [2023-06-21 20:05:36,815][03737] Decorrelating experience for 64 frames... [2023-06-21 20:05:36,815][03740] Decorrelating experience for 64 frames... [2023-06-21 20:05:36,819][03739] Decorrelating experience for 64 frames... [2023-06-21 20:05:37,483][03745] Decorrelating experience for 32 frames... [2023-06-21 20:05:37,486][03741] Decorrelating experience for 64 frames... [2023-06-21 20:05:37,550][03743] Decorrelating experience for 64 frames... [2023-06-21 20:05:38,238][03742] Decorrelating experience for 64 frames... [2023-06-21 20:05:38,241][03744] Decorrelating experience for 64 frames... [2023-06-21 20:05:39,013][03745] Decorrelating experience for 64 frames... [2023-06-21 20:05:39,063][03739] Decorrelating experience for 96 frames... [2023-06-21 20:05:39,137][03737] Decorrelating experience for 96 frames... [2023-06-21 20:05:39,169][03740] Decorrelating experience for 96 frames... [2023-06-21 20:05:39,307][03300] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 6111488. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 20:05:39,810][03741] Decorrelating experience for 96 frames... [2023-06-21 20:05:39,850][03743] Decorrelating experience for 96 frames... [2023-06-21 20:05:40,542][03742] Decorrelating experience for 96 frames... [2023-06-21 20:05:40,549][03744] Decorrelating experience for 96 frames... [2023-06-21 20:05:41,291][03745] Decorrelating experience for 96 frames... [2023-06-21 20:05:44,306][03300] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 6111488. Throughput: 0: 3.8. Samples: 38. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2023-06-21 20:05:44,308][03300] Avg episode reward: [(0, '0.570')] [2023-06-21 20:05:47,764][03736] Stopping Batcher_0... [2023-06-21 20:05:47,764][03736] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001637_6114688.pth... [2023-06-21 20:05:47,767][03300] Component Batcher_0 stopped! [2023-06-21 20:05:47,764][03736] Loop batcher_evt_loop terminating... [2023-06-21 20:05:47,910][03738] Weights refcount: 2 0 [2023-06-21 20:05:47,912][03738] Stopping InferenceWorker_p0-w0... [2023-06-21 20:05:47,912][03738] Loop inference_proc0-0_evt_loop terminating... [2023-06-21 20:05:47,914][03300] Component InferenceWorker_p0-w0 stopped! [2023-06-21 20:05:47,956][03741] Stopping RolloutWorker_w3... [2023-06-21 20:05:47,958][03741] Loop rollout_proc3_evt_loop terminating... [2023-06-21 20:05:47,974][03742] Stopping RolloutWorker_w4... [2023-06-21 20:05:47,975][03742] Loop rollout_proc4_evt_loop terminating... [2023-06-21 20:05:47,980][03736] Removing /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001612_6034688.pth [2023-06-21 20:05:47,985][03737] Stopping RolloutWorker_w0... [2023-06-21 20:05:47,985][03737] Loop rollout_proc0_evt_loop terminating... [2023-06-21 20:05:47,989][03745] Stopping RolloutWorker_w7... [2023-06-21 20:05:47,996][03745] Loop rollout_proc7_evt_loop terminating... [2023-06-21 20:05:48,006][03743] Stopping RolloutWorker_w5... [2023-06-21 20:05:48,009][03743] Loop rollout_proc5_evt_loop terminating... [2023-06-21 20:05:48,016][03736] Saving /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001637_6114688.pth... [2023-06-21 20:05:48,001][03300] Component RolloutWorker_w3 stopped! [2023-06-21 20:05:48,016][03740] Stopping RolloutWorker_w2... [2023-06-21 20:05:48,018][03740] Loop rollout_proc2_evt_loop terminating... [2023-06-21 20:05:48,018][03744] Stopping RolloutWorker_w6... [2023-06-21 20:05:48,019][03744] Loop rollout_proc6_evt_loop terminating... [2023-06-21 20:05:48,017][03300] Component RolloutWorker_w4 stopped! [2023-06-21 20:05:48,065][03739] Stopping RolloutWorker_w1... [2023-06-21 20:05:48,065][03739] Loop rollout_proc1_evt_loop terminating... [2023-06-21 20:05:48,025][03300] Component RolloutWorker_w0 stopped! [2023-06-21 20:05:48,076][03300] Component RolloutWorker_w7 stopped! [2023-06-21 20:05:48,078][03300] Component RolloutWorker_w5 stopped! [2023-06-21 20:05:48,079][03300] Component RolloutWorker_w2 stopped! [2023-06-21 20:05:48,080][03300] Component RolloutWorker_w6 stopped! [2023-06-21 20:05:48,080][03300] Component RolloutWorker_w1 stopped! [2023-06-21 20:05:48,159][03736] Stopping LearnerWorker_p0... [2023-06-21 20:05:48,159][03736] Loop learner_proc0_evt_loop terminating... [2023-06-21 20:05:48,159][03300] Component LearnerWorker_p0 stopped! [2023-06-21 20:05:48,161][03300] Waiting for process learner_proc0 to stop... [2023-06-21 20:05:48,512][03300] Waiting for process inference_proc0-0 to join... [2023-06-21 20:05:48,513][03300] Waiting for process rollout_proc0 to join... [2023-06-21 20:05:48,513][03300] Waiting for process rollout_proc1 to join... [2023-06-21 20:05:48,514][03300] Waiting for process rollout_proc2 to join... [2023-06-21 20:05:48,514][03300] Waiting for process rollout_proc3 to join... [2023-06-21 20:05:48,515][03300] Waiting for process rollout_proc4 to join... [2023-06-21 20:05:48,515][03300] Waiting for process rollout_proc5 to join... [2023-06-21 20:05:48,515][03300] Waiting for process rollout_proc6 to join... [2023-06-21 20:05:48,515][03300] Waiting for process rollout_proc7 to join... [2023-06-21 20:05:48,516][03300] Batcher 0 profile tree view: batching: 0.0072, releasing_batches: 0.0000 [2023-06-21 20:05:48,516][03300] InferenceWorker_p0-w0 profile tree view: wait_policy: 0.0040 wait_policy_total: 13.0879 update_model: 0.0109 weight_update: 0.0029 one_step: 0.0054 handle_policy_step: 1.7860 deserialize: 0.0178, stack: 0.0034, obs_to_device_normalize: 0.1252, forward: 1.5687, send_messages: 0.0158 prepare_outputs: 0.0213 to_cpu: 0.0024 [2023-06-21 20:05:48,516][03300] Learner 0 profile tree view: misc: 0.0000, prepare_batch: 0.3879 train: 1.2408 epoch_init: 0.0000, minibatch_init: 0.0000, losses_postprocess: 0.0003, kl_divergence: 0.0007, after_optimizer: 0.0031 calculate_losses: 0.7032 losses_init: 0.0000, forward_head: 0.6612, bptt_initial: 0.0082, tail: 0.0032, advantages_returns: 0.0018, losses: 0.0048 bptt: 0.0231 bptt_forward_core: 0.0229 update: 0.5322 clip: 0.0019 [2023-06-21 20:05:48,517][03300] RolloutWorker_w0 profile tree view: wait_for_trajectories: 0.0006, enqueue_policy_requests: 0.0150, env_step: 6.1927, overhead: 0.0136, complete_rollouts: 0.0002 save_policy_outputs: 0.0078 split_output_tensors: 0.0041 [2023-06-21 20:05:48,517][03300] RolloutWorker_w7 profile tree view: wait_for_trajectories: 0.0003, enqueue_policy_requests: 0.0084, env_step: 4.0969, overhead: 0.0086, complete_rollouts: 0.0002 save_policy_outputs: 0.0043 split_output_tensors: 0.0022 [2023-06-21 20:05:48,518][03300] Loop Runner_EvtLoop terminating... [2023-06-21 20:05:48,518][03300] Runner profile tree view: main_loop: 18.1250 [2023-06-21 20:05:48,518][03300] Collected {0: 6114688}, FPS: 176.6 [2023-06-21 20:07:06,012][03300] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 20:07:06,013][03300] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-21 20:07:06,014][03300] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-21 20:07:06,014][03300] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-21 20:07:06,014][03300] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 20:07:06,015][03300] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-21 20:07:06,015][03300] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 20:07:06,015][03300] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-21 20:07:06,016][03300] Adding new argument 'push_to_hub'=False that is not in the saved config file! [2023-06-21 20:07:06,016][03300] Adding new argument 'hf_repository'=None that is not in the saved config file! [2023-06-21 20:07:06,017][03300] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-21 20:07:06,017][03300] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-21 20:07:06,017][03300] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-21 20:07:06,017][03300] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-21 20:07:06,018][03300] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-21 20:07:06,023][03300] Doom resolution: 160x120, resize resolution: (128, 72) [2023-06-21 20:07:06,023][03300] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 20:07:06,024][03300] RunningMeanStd input shape: (1,) [2023-06-21 20:07:06,034][03300] ConvEncoder: input_channels=3 [2023-06-21 20:07:06,086][03300] Conv encoder output size: 512 [2023-06-21 20:07:06,086][03300] Policy head output size: 512 [2023-06-21 20:07:06,092][03300] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001637_6114688.pth... [2023-06-21 20:07:07,663][03300] Num frames 100... [2023-06-21 20:07:08,511][03300] Num frames 200... [2023-06-21 20:07:09,351][03300] Num frames 300... [2023-06-21 20:07:10,204][03300] Num frames 400... [2023-06-21 20:07:10,878][03300] Avg episode rewards: #0: 8.680, true rewards: #0: 4.680 [2023-06-21 20:07:10,881][03300] Avg episode reward: 8.680, avg true_objective: 4.680 [2023-06-21 20:07:11,150][03300] Num frames 500... [2023-06-21 20:07:12,013][03300] Num frames 600... [2023-06-21 20:07:12,877][03300] Num frames 700... [2023-06-21 20:07:13,741][03300] Num frames 800... [2023-06-21 20:07:14,605][03300] Num frames 900... [2023-06-21 20:07:15,463][03300] Num frames 1000... [2023-06-21 20:07:16,336][03300] Num frames 1100... [2023-06-21 20:07:17,178][03300] Num frames 1200... [2023-06-21 20:07:18,030][03300] Num frames 1300... [2023-06-21 20:07:18,911][03300] Num frames 1400... [2023-06-21 20:07:19,783][03300] Num frames 1500... [2023-06-21 20:07:20,668][03300] Num frames 1600... [2023-06-21 20:07:21,522][03300] Num frames 1700... [2023-06-21 20:07:22,368][03300] Num frames 1800... [2023-06-21 20:07:23,205][03300] Num frames 1900... [2023-06-21 20:07:24,079][03300] Num frames 2000... [2023-06-21 20:07:24,201][03300] Avg episode rewards: #0: 23.020, true rewards: #0: 10.020 [2023-06-21 20:07:24,203][03300] Avg episode reward: 23.020, avg true_objective: 10.020 [2023-06-21 20:07:25,021][03300] Num frames 2100... [2023-06-21 20:07:25,888][03300] Num frames 2200... [2023-06-21 20:07:26,726][03300] Num frames 2300... [2023-06-21 20:07:27,611][03300] Num frames 2400... [2023-06-21 20:07:28,457][03300] Num frames 2500... [2023-06-21 20:07:29,319][03300] Num frames 2600... [2023-06-21 20:07:30,194][03300] Num frames 2700... [2023-06-21 20:07:30,359][03300] Avg episode rewards: #0: 19.360, true rewards: #0: 9.027 [2023-06-21 20:07:30,362][03300] Avg episode reward: 19.360, avg true_objective: 9.027 [2023-06-21 20:07:31,142][03300] Num frames 2800... [2023-06-21 20:07:31,994][03300] Num frames 2900... [2023-06-21 20:07:32,874][03300] Num frames 3000... [2023-06-21 20:07:33,716][03300] Num frames 3100... [2023-06-21 20:07:34,575][03300] Num frames 3200... [2023-06-21 20:07:35,442][03300] Num frames 3300... [2023-06-21 20:07:36,286][03300] Num frames 3400... [2023-06-21 20:07:37,156][03300] Num frames 3500... [2023-06-21 20:07:37,893][03300] Avg episode rewards: #0: 18.180, true rewards: #0: 8.930 [2023-06-21 20:07:37,896][03300] Avg episode reward: 18.180, avg true_objective: 8.930 [2023-06-21 20:07:38,148][03300] Num frames 3600... [2023-06-21 20:07:39,035][03300] Num frames 3700... [2023-06-21 20:07:39,919][03300] Num frames 3800... [2023-06-21 20:07:40,790][03300] Num frames 3900... [2023-06-21 20:07:41,643][03300] Num frames 4000... [2023-06-21 20:07:42,508][03300] Num frames 4100... [2023-06-21 20:07:43,327][03300] Num frames 4200... [2023-06-21 20:07:44,046][03300] Num frames 4300... [2023-06-21 20:07:44,708][03300] Avg episode rewards: #0: 17.944, true rewards: #0: 8.744 [2023-06-21 20:07:44,710][03300] Avg episode reward: 17.944, avg true_objective: 8.744 [2023-06-21 20:07:44,942][03300] Num frames 4400... [2023-06-21 20:07:45,796][03300] Num frames 4500... [2023-06-21 20:07:46,646][03300] Num frames 4600... [2023-06-21 20:07:47,504][03300] Num frames 4700... [2023-06-21 20:07:48,368][03300] Num frames 4800... [2023-06-21 20:07:49,240][03300] Num frames 4900... [2023-06-21 20:07:50,103][03300] Num frames 5000... [2023-06-21 20:07:50,990][03300] Num frames 5100... [2023-06-21 20:07:51,873][03300] Num frames 5200... [2023-06-21 20:07:52,702][03300] Num frames 5300... [2023-06-21 20:07:53,523][03300] Num frames 5400... [2023-06-21 20:07:54,380][03300] Num frames 5500... [2023-06-21 20:07:55,259][03300] Num frames 5600... [2023-06-21 20:07:56,102][03300] Num frames 5700... [2023-06-21 20:07:56,972][03300] Num frames 5800... [2023-06-21 20:07:57,826][03300] Num frames 5900... [2023-06-21 20:07:58,679][03300] Num frames 6000... [2023-06-21 20:07:59,568][03300] Num frames 6100... [2023-06-21 20:08:00,466][03300] Num frames 6200... [2023-06-21 20:08:01,328][03300] Num frames 6300... [2023-06-21 20:08:02,202][03300] Num frames 6400... [2023-06-21 20:08:02,552][03300] Avg episode rewards: #0: 24.715, true rewards: #0: 10.715 [2023-06-21 20:08:02,554][03300] Avg episode reward: 24.715, avg true_objective: 10.715 [2023-06-21 20:08:03,175][03300] Num frames 6500... [2023-06-21 20:08:04,026][03300] Num frames 6600... [2023-06-21 20:08:04,901][03300] Num frames 6700... [2023-06-21 20:08:05,783][03300] Num frames 6800... [2023-06-21 20:08:06,639][03300] Num frames 6900... [2023-06-21 20:08:07,523][03300] Num frames 7000... [2023-06-21 20:08:08,398][03300] Num frames 7100... [2023-06-21 20:08:09,287][03300] Num frames 7200... [2023-06-21 20:08:10,069][03300] Avg episode rewards: #0: 23.967, true rewards: #0: 10.396 [2023-06-21 20:08:10,072][03300] Avg episode reward: 23.967, avg true_objective: 10.396 [2023-06-21 20:08:10,268][03300] Num frames 7300... [2023-06-21 20:08:11,157][03300] Num frames 7400... [2023-06-21 20:08:12,034][03300] Num frames 7500... [2023-06-21 20:08:12,871][03300] Num frames 7600... [2023-06-21 20:08:13,723][03300] Num frames 7700... [2023-06-21 20:08:14,579][03300] Num frames 7800... [2023-06-21 20:08:15,440][03300] Num frames 7900... [2023-06-21 20:08:16,319][03300] Num frames 8000... [2023-06-21 20:08:17,093][03300] Avg episode rewards: #0: 23.096, true rewards: #0: 10.096 [2023-06-21 20:08:17,096][03300] Avg episode reward: 23.096, avg true_objective: 10.096 [2023-06-21 20:08:17,283][03300] Num frames 8100... [2023-06-21 20:08:18,140][03300] Num frames 8200... [2023-06-21 20:08:18,980][03300] Num frames 8300... [2023-06-21 20:08:19,820][03300] Num frames 8400... [2023-06-21 20:08:20,672][03300] Num frames 8500... [2023-06-21 20:08:21,544][03300] Num frames 8600... [2023-06-21 20:08:21,820][03300] Avg episode rewards: #0: 21.468, true rewards: #0: 9.579 [2023-06-21 20:08:21,823][03300] Avg episode reward: 21.468, avg true_objective: 9.579 [2023-06-21 20:08:22,518][03300] Num frames 8700... [2023-06-21 20:08:23,385][03300] Num frames 8800... [2023-06-21 20:08:24,248][03300] Num frames 8900... [2023-06-21 20:08:24,651][03300] Avg episode rewards: #0: 19.835, true rewards: #0: 8.935 [2023-06-21 20:08:24,654][03300] Avg episode reward: 19.835, avg true_objective: 8.935 [2023-06-21 20:08:37,583][03300] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4! [2023-06-21 20:08:48,389][03300] Loading existing experiment configuration from /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/config.json [2023-06-21 20:08:48,390][03300] Overriding arg 'num_workers' with value 1 passed from command line [2023-06-21 20:08:48,391][03300] Adding new argument 'no_render'=True that is not in the saved config file! [2023-06-21 20:08:48,391][03300] Adding new argument 'save_video'=True that is not in the saved config file! [2023-06-21 20:08:48,391][03300] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2023-06-21 20:08:48,392][03300] Adding new argument 'video_name'=None that is not in the saved config file! [2023-06-21 20:08:48,392][03300] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! [2023-06-21 20:08:48,393][03300] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2023-06-21 20:08:48,394][03300] Adding new argument 'push_to_hub'=True that is not in the saved config file! [2023-06-21 20:08:48,394][03300] Adding new argument 'hf_repository'='mihirdeo16/vizdoom_health_gathering_supreme' that is not in the saved config file! [2023-06-21 20:08:48,395][03300] Adding new argument 'policy_index'=0 that is not in the saved config file! [2023-06-21 20:08:48,395][03300] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2023-06-21 20:08:48,396][03300] Adding new argument 'train_script'=None that is not in the saved config file! [2023-06-21 20:08:48,396][03300] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2023-06-21 20:08:48,396][03300] Using frameskip 1 and render_action_repeat=4 for evaluation [2023-06-21 20:08:48,402][03300] RunningMeanStd input shape: (3, 72, 128) [2023-06-21 20:08:48,403][03300] RunningMeanStd input shape: (1,) [2023-06-21 20:08:48,410][03300] ConvEncoder: input_channels=3 [2023-06-21 20:08:48,425][03300] Conv encoder output size: 512 [2023-06-21 20:08:48,425][03300] Policy head output size: 512 [2023-06-21 20:08:48,431][03300] Loading state from checkpoint /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/checkpoint_p0/checkpoint_000001637_6114688.pth... [2023-06-21 20:08:50,025][03300] Num frames 100... [2023-06-21 20:08:50,876][03300] Num frames 200... [2023-06-21 20:08:51,777][03300] Num frames 300... [2023-06-21 20:08:52,671][03300] Num frames 400... [2023-06-21 20:08:53,543][03300] Num frames 500... [2023-06-21 20:08:54,430][03300] Num frames 600... [2023-06-21 20:08:55,320][03300] Num frames 700... [2023-06-21 20:08:56,026][03300] Avg episode rewards: #0: 15.680, true rewards: #0: 7.680 [2023-06-21 20:08:56,028][03300] Avg episode reward: 15.680, avg true_objective: 7.680 [2023-06-21 20:08:56,308][03300] Num frames 800... [2023-06-21 20:08:57,182][03300] Num frames 900... [2023-06-21 20:08:58,074][03300] Num frames 1000... [2023-06-21 20:08:58,962][03300] Num frames 1100... [2023-06-21 20:08:59,853][03300] Num frames 1200... [2023-06-21 20:09:00,780][03300] Num frames 1300... [2023-06-21 20:09:01,662][03300] Num frames 1400... [2023-06-21 20:09:02,547][03300] Num frames 1500... [2023-06-21 20:09:03,396][03300] Avg episode rewards: #0: 17.430, true rewards: #0: 7.930 [2023-06-21 20:09:03,399][03300] Avg episode reward: 17.430, avg true_objective: 7.930 [2023-06-21 20:09:03,516][03300] Num frames 1600... [2023-06-21 20:09:04,394][03300] Num frames 1700... [2023-06-21 20:09:05,309][03300] Num frames 1800... [2023-06-21 20:09:06,167][03300] Num frames 1900... [2023-06-21 20:09:07,043][03300] Num frames 2000... [2023-06-21 20:09:07,961][03300] Num frames 2100... [2023-06-21 20:09:08,610][03300] Avg episode rewards: #0: 14.873, true rewards: #0: 7.207 [2023-06-21 20:09:08,612][03300] Avg episode reward: 14.873, avg true_objective: 7.207 [2023-06-21 20:09:08,949][03300] Num frames 2200... [2023-06-21 20:09:09,827][03300] Num frames 2300... [2023-06-21 20:09:10,698][03300] Num frames 2400... [2023-06-21 20:09:11,591][03300] Num frames 2500... [2023-06-21 20:09:12,482][03300] Num frames 2600... [2023-06-21 20:09:13,425][03300] Num frames 2700... [2023-06-21 20:09:14,327][03300] Num frames 2800... [2023-06-21 20:09:15,210][03300] Num frames 2900... [2023-06-21 20:09:16,100][03300] Num frames 3000... [2023-06-21 20:09:16,989][03300] Num frames 3100... [2023-06-21 20:09:17,855][03300] Num frames 3200... [2023-06-21 20:09:18,714][03300] Num frames 3300... [2023-06-21 20:09:19,612][03300] Num frames 3400... [2023-06-21 20:09:20,489][03300] Num frames 3500... [2023-06-21 20:09:21,398][03300] Num frames 3600... [2023-06-21 20:09:22,288][03300] Num frames 3700... [2023-06-21 20:09:23,143][03300] Num frames 3800... [2023-06-21 20:09:24,033][03300] Num frames 3900... [2023-06-21 20:09:24,648][03300] Avg episode rewards: #0: 22.897, true rewards: #0: 9.897 [2023-06-21 20:09:24,651][03300] Avg episode reward: 22.897, avg true_objective: 9.897 [2023-06-21 20:09:25,017][03300] Num frames 4000... [2023-06-21 20:09:25,887][03300] Num frames 4100... [2023-06-21 20:09:26,806][03300] Num frames 4200... [2023-06-21 20:09:27,693][03300] Num frames 4300... [2023-06-21 20:09:28,556][03300] Num frames 4400... [2023-06-21 20:09:29,456][03300] Num frames 4500... [2023-06-21 20:09:30,389][03300] Num frames 4600... [2023-06-21 20:09:31,287][03300] Num frames 4700... [2023-06-21 20:09:32,165][03300] Num frames 4800... [2023-06-21 20:09:33,049][03300] Num frames 4900... [2023-06-21 20:09:33,922][03300] Num frames 5000... [2023-06-21 20:09:34,709][03300] Avg episode rewards: #0: 22.758, true rewards: #0: 10.158 [2023-06-21 20:09:34,712][03300] Avg episode reward: 22.758, avg true_objective: 10.158 [2023-06-21 20:09:34,890][03300] Num frames 5100... [2023-06-21 20:09:35,771][03300] Num frames 5200... [2023-06-21 20:09:36,673][03300] Num frames 5300... [2023-06-21 20:09:37,552][03300] Num frames 5400... [2023-06-21 20:09:38,434][03300] Num frames 5500... [2023-06-21 20:09:39,311][03300] Num frames 5600... [2023-06-21 20:09:40,213][03300] Num frames 5700... [2023-06-21 20:09:41,098][03300] Num frames 5800... [2023-06-21 20:09:41,978][03300] Num frames 5900... [2023-06-21 20:09:42,893][03300] Num frames 6000... [2023-06-21 20:09:43,720][03300] Num frames 6100... [2023-06-21 20:09:44,422][03300] Num frames 6200... [2023-06-21 20:09:45,214][03300] Num frames 6300... [2023-06-21 20:09:46,139][03300] Num frames 6400... [2023-06-21 20:09:46,992][03300] Num frames 6500... [2023-06-21 20:09:47,677][03300] Avg episode rewards: #0: 25.777, true rewards: #0: 10.943 [2023-06-21 20:09:47,679][03300] Avg episode reward: 25.777, avg true_objective: 10.943 [2023-06-21 20:09:47,982][03300] Num frames 6600... [2023-06-21 20:09:48,892][03300] Num frames 6700... [2023-06-21 20:09:49,800][03300] Num frames 6800... [2023-06-21 20:09:50,701][03300] Num frames 6900... [2023-06-21 20:09:51,572][03300] Num frames 7000... [2023-06-21 20:09:52,408][03300] Num frames 7100... [2023-06-21 20:09:53,196][03300] Num frames 7200... [2023-06-21 20:09:53,984][03300] Num frames 7300... [2023-06-21 20:09:54,880][03300] Num frames 7400... [2023-06-21 20:09:55,782][03300] Num frames 7500... [2023-06-21 20:09:56,675][03300] Num frames 7600... [2023-06-21 20:09:57,545][03300] Num frames 7700... [2023-06-21 20:09:58,416][03300] Num frames 7800... [2023-06-21 20:09:59,321][03300] Num frames 7900... [2023-06-21 20:10:00,190][03300] Num frames 8000... [2023-06-21 20:10:01,075][03300] Num frames 8100... [2023-06-21 20:10:01,964][03300] Num frames 8200... [2023-06-21 20:10:02,616][03300] Avg episode rewards: #0: 27.517, true rewards: #0: 11.803 [2023-06-21 20:10:02,619][03300] Avg episode reward: 27.517, avg true_objective: 11.803 [2023-06-21 20:10:02,949][03300] Num frames 8300... [2023-06-21 20:10:03,830][03300] Num frames 8400... [2023-06-21 20:10:04,681][03300] Num frames 8500... [2023-06-21 20:10:05,555][03300] Num frames 8600... [2023-06-21 20:10:06,435][03300] Num frames 8700... [2023-06-21 20:10:07,302][03300] Num frames 8800... [2023-06-21 20:10:08,156][03300] Num frames 8900... [2023-06-21 20:10:09,030][03300] Num frames 9000... [2023-06-21 20:10:09,896][03300] Num frames 9100... [2023-06-21 20:10:10,752][03300] Num frames 9200... [2023-06-21 20:10:11,621][03300] Num frames 9300... [2023-06-21 20:10:12,495][03300] Num frames 9400... [2023-06-21 20:10:13,355][03300] Num frames 9500... [2023-06-21 20:10:14,201][03300] Num frames 9600... [2023-06-21 20:10:14,883][03300] Avg episode rewards: #0: 28.331, true rewards: #0: 12.081 [2023-06-21 20:10:14,885][03300] Avg episode reward: 28.331, avg true_objective: 12.081 [2023-06-21 20:10:15,193][03300] Num frames 9700... [2023-06-21 20:10:16,066][03300] Num frames 9800... [2023-06-21 20:10:16,924][03300] Num frames 9900... [2023-06-21 20:10:17,800][03300] Avg episode rewards: #0: 25.878, true rewards: #0: 11.100 [2023-06-21 20:10:17,803][03300] Avg episode reward: 25.878, avg true_objective: 11.100 [2023-06-21 20:10:17,887][03300] Num frames 10000... [2023-06-21 20:10:18,743][03300] Num frames 10100... [2023-06-21 20:10:19,626][03300] Num frames 10200... [2023-06-21 20:10:20,496][03300] Num frames 10300... [2023-06-21 20:10:21,379][03300] Num frames 10400... [2023-06-21 20:10:22,247][03300] Num frames 10500... [2023-06-21 20:10:23,101][03300] Num frames 10600... [2023-06-21 20:10:23,993][03300] Num frames 10700... [2023-06-21 20:10:24,325][03300] Avg episode rewards: #0: 24.626, true rewards: #0: 10.726 [2023-06-21 20:10:24,328][03300] Avg episode reward: 24.626, avg true_objective: 10.726 [2023-06-21 20:10:38,574][03300] Replay video saved to /Users/md/Code/python/jubilant-memory/RL/train_dir/default_experiment/replay.mp4!