ViPE-S-CTX7

ViPE: Visualize Pretty-much Everything, is the first automated model for translating any arbitrary piece of text into a visualizable prompt. It helps any text-to-image model in figurative or non-lexical language visualizations. It has been shown to be more robust than GPT3.5 Turbo (ChatGPT) in generating depictable and semantically meaningful prompts.

Model Description

• Developed by: Computer Graphics Group, University of Tuebingen
• Model type: Auto-Regressive
• Language: English
• License: MIT
• Based on: GPT2-Small
• Versions: ViPE-M-CTX7 (355M parameters) and ViPE-S-CTX7 (117M parameters)

Model Sources

• Repository: Github
• Paper: ViPE: Visualise Pretty-much Everything (Outstanding Paper Award at EMNLP 2023)

Down Stream Applications

ViPE provides a robust backbone for many practical applications such as music video generation and creative writing.

• Music Video Genrations

  • Repository: Github
  • Example Videos: ViPE Videos

• Creative Writing

  • Demo: Hugging Face Playground

• Summagery: Document Summarization through Images

  • Demo: Hugging Face Playground
  • Repository: Github
  • Examples: Summagery Videos

Direct Use

You can directly use the model to generate detailed prompts for any arbitrary text.

from transformers import GPT2LMHeadModel, GPT2Tokenizer


def generate(text, model, tokenizer,device,do_sample,top_k=100, epsilon_cutoff=.00005, temperature=1):
    #mark the text with special tokens
    text=[tokenizer.eos_token +  i + tokenizer.eos_token for i in text]
    batch=tokenizer(text, padding=True, return_tensors="pt")

    input_ids = batch["input_ids"].to(device)
    attention_mask = batch["attention_mask"].to(device)

    #how many new tokens to generate at max
    max_prompt_length=50

    generated_ids = model.generate(input_ids=input_ids,attention_mask=attention_mask, max_new_tokens=max_prompt_length, do_sample=do_sample,top_k=top_k, epsilon_cutoff=epsilon_cutoff, temperature=temperature)
    #return only the generated prompts
    pred_caps = tokenizer.batch_decode(generated_ids[:, -(generated_ids.shape[1] - input_ids.shape[1]):], skip_special_tokens=True)

    return pred_caps

device='cpu'
model = GPT2LMHeadModel.from_pretrained('fittar/ViPE-S-CTX7')
model.to(device)

#ViPE-M's tokenizer is identical to that of GPT2-Small
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.pad_token = tokenizer.eos_token

# A list of abstract/figurative or any arbitrary combinations of keywords
texts=['lalala', 'I wanna start learning', 'free your mind; you will see the other side of life', 'brave; fantasy']

prompts=generate(texts,model,tokenizer,do_sample=True,device=device)
for t,p in zip(texts,prompts):
    print('{} --> {}'.format(t,p))

lalala -->  A group of dancers performing an extravagant traditional dance, lalala in Spanish
I wanna start learning -->  A student intently sitting at a desk, surrounded by books and notes
free your mind; you will see the other side of life -->  A view of the night sky, stars and planets shining bright, while a woman in a field of flowers looks up in awe
brave; fantasy -->  A knight in shining armor riding a gallant horse through a sunlit valley

Recommendations

You can use either a comma or a semicolon to combine multiple keywords. for example ['dark, fantasy, brave'] or ['This is gonna be the best day of my life; do you agree?']. However, a semicolon draws a stronger boundary between the keywords and encourages the model to transfer the last keyword in a given context (previous keywords).

Training Details

Training Data

• LyricCanvas dataset: a synthetically generated dataset based on lyrics and synthetically generated prompts

Training Procedure

ViPE has been trained in the standard auto-regressive procedure: given a line (or lines) of lyrics as a prefix, the objective is to generate a plausible prompt that is both despicable and semantically related to the given lyric(c). The loss function does not include the tokens corresponding to the lyrics. So ViPE never generates any original lyrics and only learns to generate visually related prompts.

Evaluation

In all of the following evaluations, ViPE consistently demonstrates its robustness compared to ChatGPT and achieves performance that is competitive with that of human experts.

• Intrinsic evaluations
  • General understanding of figurative language using Fig-QA dataset
• Extrinsic evaluations
  • Image-text Retrieval on the HAIVMet dataset
  • Emotion visualizations: How well does ViPE transfer emotionally charged tweets into a depictable description of a scene in comparison with ChatGPT. The Emotion dataset is utilized.
• Human evaluations
  • we conducted a user study involving 30 native English-speaking participants aged between 20 and 40. Participants were presented with 3 images and a metaphor from the HAIVMet dataset. They were asked to select the images that matches the metaphor the best. The images were generated using prompts from ViPE, ChatGPT, and human experts (HAIVMet).

Citation

If you find ViPE useful, please cite our paper.

@inproceedings{shahmohammadi-etal-2023-vipe,
    title = "{V}i{PE}: Visualise Pretty-much Everything",
    author = "Shahmohammadi, Hassan  and
      Ghosh, Adhiraj  and
      Lensch, Hendrik",
    editor = "Bouamor, Houda  and
      Pino, Juan  and
      Bali, Kalika",
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.emnlp-main.333",
    pages = "5477--5494"
}

Model Card Contact

Hassan Shahmohammadi