text: 
  
  TITLE
  [Bug/Question] Write With Transformers Implementation vs. Custom Implementation
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  Hi, 
  
  Not sure if this is a bug or perhaps a misimplementation, but i'm comparing the results of using gpt2-large on the 'Write With Transformers' Text Generation example - https://transformer.huggingface.co/doc/gpt2-large, vs. my own implementation of the text generation tool. My use-case is to generate distractor options in an MCQ environment, given a few (15-20) prior examples for the style of distractor questions to generate. 
  
  The general format I am implementing is:
  `Question: _______ . Answer 1: ___<the correct answer>___ . Answer 2: _____<distractor 1>_____ . Answer 3: ____<distractor 2>_____`
  
  ## Write With Transformers
  full doc available here - https://transformer.huggingface.co/share/CZqVXdngic
  
  ### Model Config
  Model size - gpt2/large
  Top-p - 0.9
  Temperature - 1
  Max time - 2.3 
  ### Output
  On the 'Write With Transformers' page, I write in the examples: 
  eg. `Question: How is calorie related to the S.I unit of that quantity?Answer 1: 1 cal = 4.2 J.Answer 2: 1 cal = 3.2 J.Answer 3: 1 cal = 10 J.`
  
  and when I try to generate predictions for the following Question-Answer pair: 
  `Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W.`
  
  am able to generate a few solid distractors: 
  `Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2:  1 H. P. = 1. 3 W.  Answer 3 : 1 H . P. =  1. 5 W.`
  
  ## Custom Case
  
  ### Output
  Here's how I create a dataset from the set of questions I've initially written myself. 
  ```
  def make_dataset(dataset, epochs):
      total_text = '<|endoftext|>'
      qa = [t for t in dataset]
      for _ in range(epochs):
          random.shuffle(qa)
          total_text += '<|endoftext|>'.join(qa) + '<|endoftext|>'
      return total_text
  ```
  
  This is the training model params:
  
  ```
  !python run_language_modeling.py \
      --output_dir=output/$handle \
      --overwrite_output_dir \
      --overwrite_cache \
      --model_type=gpt2 \
      --model_name_or_path=gpt2 \
      --do_train --train_data_file=$handle\_train.txt \
      --logging_steps 20 \
      --per_gpu_train_batch_size 1 \
      --num_train_epochs 15
  ```
  
  num_return_sequences = 5
  
  ```
  for start in SENTENCES:
      val = !python run_generation.py \
          --model_type gpt2 \
          --model_name_or_path output/$handle \
          --length 40 \
          --num_return_sequences $num_return_sequences \
          --temperature 0.23 \
          --p 0.95 \
          --seed $seed \
          --prompt {'"<|endoftext|>' + start + '"'}
      generated = [val[-1-2*k] for k in range(num_return_sequences)[::-1]]
      print(f'\nStart of sentence: {start}')
      for i, g in enumerate(generated):
          g = g.replace('<|endoftext|>', '')
          print(f'* Generated #{i+1}: {g}')
  ```
  
  These are my generated mcq pairs:
  
  **Generated #1: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.**
  **Generated #2: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.**
  **Generated #3: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: ______________ = 905 J.Answer 3: 50 J.**
  **Generated #4: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.**
  **Generated #5: Question: How is the unit horse power related to the S.I. unit of power? Answer 1: 1 H.P. = 750 W. Answer 2: กราท.Answer 3: 50 J.**