Experimental roberta-base model pretrained from scratch on factual information

Thanks to LAION contributors and Stability.ai

for help building datasets and compute resources.

Using the model:

This is a BERT style model pretrained using the masked language modeling task (MLM)

The model was trained like a standard roberta-base model, except it was only trained on sequences of 128 tokens.

I believe there is a bug in the training script that doesn't properly assign the <BOS> and <EOS> tokens. So make sure to manually remove those from your outputs.

from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

checkpoint="Rallio67/roberta-base-128-factchecker"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForMaskedLM.from_pretrained(checkpoint, torch_dtype=torch.float16).cuda(0)

print(checkpoint)

text="""Neurospora crassa is a species of filamentous fungi that is widely used in genetic and molecular biology research. It is known for its fast growth rate, relatively simple life cycle, and ability to undergo sexual and asexual reproduction. Here are some specific areas of research where Neurospora crassa is commonly used: Genetics and Genomics: Neurospora crassa was one of the first organisms to have its entire genome sequenced, making it a model organism for studying gene expression, gene regulation, and functional genomics."""

input = tokenizer.encode(text)
original_tokens=input[1:-1]

print("Text length:",len(input))

sequences=[]

for i in range(1,len(input)-2):
    newentry=input.copy()
    newentry[i]=50264
    sequences.append(tokenizer.decode(newentry[1:-1]))

for sequence in sequences:
  input = tokenizer(sequence, truncation=True, padding=True, return_tensors="pt")
  output = model(input["input_ids"].cuda(0))
  predicted_token_id = output[0][0].argmax(axis=-1)
  predicted_fixed=predicted_token_id[1:-1]
  replaced=(tokenizer.decode(predicted_token_id[1:-1],skip_special_tokens=True))
  for z,i in enumerate(input['input_ids'][0]):
      if i.item() == 50264:
        original=tokenizer.decode(original_tokens[z-5:z+5])
        replace=tokenizer.decode(predicted_fixed[z-5:z+5])
        if original != replace:
            print("Original: "+original,"\n","Replace: "+replace,"\n"+"-"*8)

Outputs:

Rallio67/roberta-base-128-factchecker
Text length: 112
Original:  species of filamentous fungi that is widely used in
 Replace:  species of filamentous fungus that is widely used in
--------
Original: ous fungi that is widely used in genetic and molecular
 Replace: ous fungi that is commonly used in genetic and molecular
--------
Original:  is widely used in genetic and molecular biology research.
 Replace:  is widely used in evolutionary and molecular biology research.
--------