---
title: README
emoji: 👁
colorFrom: pink
colorTo: indigo
sdk: static
pinned: false
---

<div class="lg:col-span-3">
  <p>
    This organization is a part of the NeurIPS 2021 demonstration <a href="https://training-transformers-together.github.io/">"Training Transformers Together"</a>.
  </p>
  <p>
    In this demo, we train a model similar to <a target="_blank" href="https://openai.com/blog/dall-e/">OpenAI DALL-E</a> —
    a Transformer "language model" that generates images from text descriptions.
    It is trained on <a target="_blank" href="https://laion.ai/laion-400-open-dataset/">LAION-400M</a>,
    the world's largest openly available image-text-pair dataset with 400 million samples. Our model is based on
    the <a target="_blank" href="https://github.com/lucidrains/DALLE-pytorch">dalle‑pytorch</a> implementation
    by <a target="_blank" href="https://github.com/lucidrains">Phil Wang</a> with a few tweaks to make it communication-efficient.
  </p>
  <p>
    See details about how to join and how it works on <a target="_blank" href="https://training-transformers-together.github.io/">our website</a>.
  </p>
  <p>
    This organization gathers people participating in the collaborative training and provides links to the necessary resources:
  </p>
  <ul>
    <li>👉 Starter kits for <b>Google Colab</b> and <b>Kaggle</b> (easy way to join the training)</li>
    <li>👉 <a target="_blank" href="https://huggingface.co/spaces/training-transformers-together/Dashboard">Dashboard</a> (the current training state: loss, number of peers, etc.)</li>
    <li>👉 <a target="_blank" href="https://huggingface.co/training-transformers-together/dalle-demo">Model</a> (the latest checkpoint)</li>
    <li>👉 <a target="_blank" href="https://huggingface.co/datasets/laion/laion_100m_vqgan_f8">Dataset</a></li>
  </ul>
  <p>
    Feel free to reach us on <a target="_blank" href="https://discord.gg/uGugx9zYvN">Discord</a> if you have any questions 🙂
  </p>
</div>