jjae/summarization-diary

summary

This model is a fine-tuned version of gogamza/kobart-summarization on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.4011

Model description

This model summarizes the diary.

Training and evaluation data

This model was trained by the self-instruction process. All data used for fine-tuning this model were generated by chatGPT 3.5.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5.6e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 300
• num_epochs: 50

Training results

Training Loss	Epoch	Step	Validation Loss
1.4804	1.47	500	0.4035
0.2475	2.93	1000	0.4011
0.1249	4.4	1500	0.4591
0.072	5.87	2000	0.4671
0.039	7.33	2500	0.5022

Framework versions

• Transformers 4.37.2
• Pytorch 2.1.2+cu118
• Datasets 2.16.1
• Tokenizers 0.15.0

How to Get Started with the Model

Use the code below to get started with the model. You can adjust hyperparameters to fit on your data.

def diary_summary(text):
  input_ids = tokenizer.encode(text, return_tensors = 'pt').to(device)
  summary_text_ids = model.generate(input_ids = input_ids, bos_token_id = model.config.bos_token_id, eos_token_id = model.config.eos_token_id,
                                    length_penalty = 2.0, max_length = 150, num_beams = 2)
  return tokenizer.decode(summary_text_ids[0], skip_special_tokens = True)