Update README.md

README.md

@@ -4,6 +4,8 @@ license: mit
 # Overview
 This model has been fine tuned for text summary creation, and was created using LoRA to fine tunethe flan-t5-large model using the [SAMsum training dataset](https://huggingface.co/datasets/samsum).
 
+This document will explain how the model was fine tuned, saved to disk, added to Hugging Face, and then demonstrate how it is used.
+
 ## SAMsum
 SAMsum is a corpus comprised of 16k dialogues and corresponding summaries.
 
@@ -163,4 +165,62 @@ from google.colab import files
 files.download("/content/flan-t5-large-samsum.zip")
 ```
 
-Upload the contents of that zip file to huggingface 
+Upload the contents of that zip file to huggingface 
+
+# Code to utilize the fine tuned model
+
+## Notebook Source
+[Notebook using the Hugging Face hosted moded](https://colab.research.google.com/drive/1kqADOA9vaTsdecx4u-7XWJJia62WV0cY?pli=1#scrollTo=KMs70mdIxaam)
+
+## Load the model, tokenizer, and LoRA adapter (PEFT)
+
+```
+# Load the jasonmcaffee/flan-t5-large-samsum model and tokenizer
+import torch
+from peft import PeftModel, PeftConfig
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+
+peft_model_id = "jasonmcaffee/flan-t5-large-samsum"
+config = PeftConfig.from_pretrained(peft_model_id)
+
+model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path, load_in_8bit=True,  device_map="auto") 
+tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
+
+# Load the LoRA adapter
+model = PeftModel.from_pretrained(model, peft_model_id, device_map="auto")
+model.eval()
+```
+
+## Have the model summarize text!
+Finally, we now have a model that is capable of summarizing text for us.
+
+Summarization takes ~30 seconds.
+```
+dialogue = """The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.
+"""
+
+input_ids = tokenizer(dialogue, return_tensors="pt", truncation=True).input_ids.cuda()
+# with torch.inference_mode():
+outputs = model.generate(
+    input_ids=input_ids, 
+    min_length=20,
+    max_new_tokens=100, 
+    length_penalty=1.9, #Exponential penalty to the length that is used with beam-based generation. It is applied as an exponent to the sequence length, which in turn is used to divide the score of the sequence. Since the score is the log likelihood of the sequence (i.e. negative), length_penalty > 0.0 promotes longer sequences, while length_penalty < 0.0 encourages shorter sequences.
+    num_beams=4,
+    temperature=0.9,
+    top_k=150, # default 50
+    repetition_penalty=2.1,
+    # do_sample=True, 
+    top_p=0.9,
+    )
+print(f"input sentence: {dialogue}\n{'---'* 20}")
+
+summarization = tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0]
+
+print(f"summary:\n{summarization}")
+```
+Prints:
+> The Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.
+
+The notebook also loads the flan-t5 with no SAMsum training, which produces a summary of:
+> The Eiffel Tower is the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930.