Leyo commited on
Commit
5edd82e
1 Parent(s): 7ec80bb

detail fixes

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -139,7 +139,7 @@ for i, t in enumerate(generated_text):
139
 
140
  # Training Details
141
 
142
- ## IDEFICS base
143
 
144
  We closely follow the training procedure layed out in [Flamingo](https://huggingface.co/papers/2204.14198). We combine two open-source pre-trained models ([laion/CLIP-ViT-H-14-laion2B-s32B-b79K](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K) and [huggyllama/llama-65b](https://huggingface.co/huggyllama/llama-65b)) by initializing new Transformer blocks. The pre-trained backbones are frozen while we train the newly initialized parameters.
145
 
@@ -273,7 +273,7 @@ Similarly to the base IDEFICS models, we performed checkpoint selection to stop
273
 
274
  | Model | Shots | <nobr>VQAv2 <br>OE VQA acc.</nobr> | <nobr>OKVQA <br>OE VQA acc.</nobr> | <nobr>TextVQA <br>OE VQA acc.</nobr> | <nobr>VizWiz<br>OE VQA acc.</nobr> | <nobr>TextCaps <br>CIDEr</nobr> | <nobr>Coco <br>CIDEr</nobr> | <nobr>NoCaps<br>CIDEr</nobr> | <nobr>Flickr<br>CIDEr</nobr> | <nobr>VisDial <br>NDCG</nobr> | <nobr>HatefulMemes<br>ROC AUC</nobr> | <nobr>ScienceQA <br>acc.</nobr> | <nobr>RenderedSST2<br>acc.</nobr> | <nobr>Winoground<br>group (text/image)</nobr> |
275
  | :--------------------- | --------: | ---------------------: | ---------------------: | -----------------------: | ----------------------: | -------------------: | ---------------: | -----------------: | -----------------: | -----------------: | -------------------------: | -----------------------: | --------------------------: | ----------------------------------: |
276
- | Finetuning data does not contain dataset | - | &#10060; | &#10060; | &#10060; | &#10004; | &#10060; | &#10060; | &#10004; | &#10004; | &#10060; | &#10004; | &#10060; | &#10004; | &#10060; |
277
  | <nobr>IDEFICS 80B Instruct<br> | 0 | 37.4 (-22.7) | 36.9 (-8.2) | 32.9 (1.9) | 26.2 (-9.8) | 76.5 (19.7) | 117.2 (25.4) | 104.5 (39.5) | 65.3 (11.7) | 49.3 (0.4) | 58.9 (-1.7) | 69.5 (0.5) | 67.3 (6.8) | 9.2/20.0/25.0 (1.2/1.2/2.5) |
278
  | | 4 | 67.5 (4.0) | 54.0 (1.7) | 37.8 (3.5) | 39.8 (-0.7) | 71.7 (-1.0) | 116.9 (6.6) | 104.0 (4.4) | 67.1 (-6.6) | 48.9 (0.5) | 57.5 (-0.3) | 60.5 (1.6) | 65.5 (-1.1) | - |
279
  | | 8 | 68.1 (3.4) | 56.9 (1.8) | 38.2 (2.5) | 44.8 (-1.3) | 72.7 (-4.9) | 116.8 (2.5) | 104.8 (-0.9) | 70.7 (-5.9) | 48.2 (0.3) | 58.0 (-0.2) | - | 68.6 (0.8) | - |
 
139
 
140
  # Training Details
141
 
142
+ ## IDEFICS
143
 
144
  We closely follow the training procedure layed out in [Flamingo](https://huggingface.co/papers/2204.14198). We combine two open-source pre-trained models ([laion/CLIP-ViT-H-14-laion2B-s32B-b79K](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K) and [huggyllama/llama-65b](https://huggingface.co/huggyllama/llama-65b)) by initializing new Transformer blocks. The pre-trained backbones are frozen while we train the newly initialized parameters.
145
 
 
273
 
274
  | Model | Shots | <nobr>VQAv2 <br>OE VQA acc.</nobr> | <nobr>OKVQA <br>OE VQA acc.</nobr> | <nobr>TextVQA <br>OE VQA acc.</nobr> | <nobr>VizWiz<br>OE VQA acc.</nobr> | <nobr>TextCaps <br>CIDEr</nobr> | <nobr>Coco <br>CIDEr</nobr> | <nobr>NoCaps<br>CIDEr</nobr> | <nobr>Flickr<br>CIDEr</nobr> | <nobr>VisDial <br>NDCG</nobr> | <nobr>HatefulMemes<br>ROC AUC</nobr> | <nobr>ScienceQA <br>acc.</nobr> | <nobr>RenderedSST2<br>acc.</nobr> | <nobr>Winoground<br>group (text/image)</nobr> |
275
  | :--------------------- | --------: | ---------------------: | ---------------------: | -----------------------: | ----------------------: | -------------------: | ---------------: | -----------------: | -----------------: | -----------------: | -------------------------: | -----------------------: | --------------------------: | ----------------------------------: |
276
+ | Finetuning data **does not** contain the evaluation dataset | - | &#10006; | &#10006; | &#10006; | &#10004; | &#10006; | &#10006; | &#10006; | &#10004; | &#10006; | &#10004; | &#10006; | &#10004; | &#10006; |
277
  | <nobr>IDEFICS 80B Instruct<br> | 0 | 37.4 (-22.7) | 36.9 (-8.2) | 32.9 (1.9) | 26.2 (-9.8) | 76.5 (19.7) | 117.2 (25.4) | 104.5 (39.5) | 65.3 (11.7) | 49.3 (0.4) | 58.9 (-1.7) | 69.5 (0.5) | 67.3 (6.8) | 9.2/20.0/25.0 (1.2/1.2/2.5) |
278
  | | 4 | 67.5 (4.0) | 54.0 (1.7) | 37.8 (3.5) | 39.8 (-0.7) | 71.7 (-1.0) | 116.9 (6.6) | 104.0 (4.4) | 67.1 (-6.6) | 48.9 (0.5) | 57.5 (-0.3) | 60.5 (1.6) | 65.5 (-1.1) | - |
279
  | | 8 | 68.1 (3.4) | 56.9 (1.8) | 38.2 (2.5) | 44.8 (-1.3) | 72.7 (-4.9) | 116.8 (2.5) | 104.8 (-0.9) | 70.7 (-5.9) | 48.2 (0.3) | 58.0 (-0.2) | - | 68.6 (0.8) | - |