Update Summarization with BART Large and Longformer Encoder Decoder

Model description

This model is a Transformer-based model that supports long document generative sequence-to-sequence.

Based on BART Large with Longformer Encode Decoder to allow for longer inputs.

The output is one edit operation which includes action (deletion or insertion), index of where edit should happen (represented by words), and the actual text to delete or insert.

Intended uses & limitations

How to use

Format your data so that each new article or evidence to add have <EV> token in front with each title prefixed by <t> and each abstract prefixed by <abs>. Please have the original summary also in the same format. You can have the list of articles and original summary concatenated in any order as long as they have the correct separator tokens.

from transformers import LEDTokenizer, LEDForConditionalGeneration

tokenizer = LEDTokenizer.from_pretrained("hyesunyun/update-summarization-led-edit-at-a-time")
model = LEDForConditionalGeneration.from_pretrained("hyesunyun/update-summarization-led-edit-at-a-time")

input = "<EV> <t> Hypoglycemic effect of bitter melon compared with metformin in newly diagnosed type 2 diabetes patients. <abs> ETHNOPHARMACOLOGICAL RELEVANCE: Bitter melon (Momordica charantia L.) has been widely used as an traditional medicine treatment for diabetic patients in Asia. In vitro and animal studies suggested its hypoglycemic activity, but limited human studies are available to support its use. AIM OF STUDY: This study was conducted to assess the efficacy and safety of three doses of bitter melon compared with metformin. MATERIALS AND METHODS: This is a 4-week, multicenter, randomized, double-blind, active-control trial. Patients were randomized into 4 groups to receive bitter melon 500 mg/day, 1,000 mg/day, and 2,000 mg/day or metformin 1,000 mg/day. All patients were followed for 4 weeks. RESULTS: There was a significant decline in fructosamine at week 4 of the metformin group (-16.8; 95% CI, -31.2, -2.4 mumol/L) and the bitter melon 2,000 mg/day group (-10.2; 95% CI, -19.1, -1.3 mumol/L). Bitter melon 500 and 1,000 mg/day did not significantly decrease fructosamine levels (-3.5; 95% CI -11.7, 4.6 and -10.3; 95% CI -22.7, 2.2 mumol/L, respectively). CONCLUSIONS: Bitter melon had a modest hypoglycemic effect and significantly reduced fructosamine levels from baseline among patients with type 2 diabetes who received 2,000 mg/day. However, the hypoglycemic effect of bitter melon was less than metformin 1,000 mg/day. <EV> <t> Momordica charantia for type 2 diabetes mellitus. <abs> There is insufficient evidence to recommend momordica charantia for type 2 diabetes mellitus. Further studies are therefore required to address the issues of standardization and the quality control of preparations. For medical nutritional therapy, further observational trials evaluating the effects of momordica charantia are needed before RCTs are established to guide any recommendations in clinical practice."
inputs_dict = tokenizer(input, padding="max_length", max_length=10240, return_tensors="pt", truncation=True)
input_ids = inputs_dict.input_ids
attention_mask = inputs_dict.attention_mask
global_attention_mask = torch.zeros_like(attention_mask)
# put global attention on <s> token
global_attention_mask[:, 0] = 1

predicted_summary_ids = model.generate(input_ids, attention_mask=attention_mask, global_attention_mask=global_attention_mask)
print(tokenizer.batch_decode(predicted_summary_ids, skip_special_tokens=False))

The expected output should be something like <s> insertion <edit_pad> zero <edit_pad> bla bla bla some text </s>

Limitations and bias

Provide examples of latent issues and potential remediations.

Training data

Used pre-trained LED model and fine-tuned using the dataset found in this github repo.

Training procedure

Preprocessing, hardware used, hyperparameters...

Eval results

BibTeX entry and citation info

@inproceedings{...,
  year={2022}
}