Update README.md

README.md

@@ -1,201 +1,145 @@
----
-# For reference on model card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1
-# Doc / guide: https://huggingface.co/docs/hub/model-cards
-{}
----
+# Model Details  
 
-# Model Card for Model ID
+BLOOM is an autoregressive Large Language Model (LLM), trained to continue text from a prompt on vast amounts of text data using industrial-scale computational resources. As such, it is able to output coherent text in 46 languages and 13 programming languages that is hardly distinguishable from text written by humans. BLOOM can also be instructed to perform text tasks it hasn't been explicitly trained for, by casting them as text generation tasks.
 
-<!-- Provide a quick summary of what the model is/does. -->
+## Basics
+*This section provides information about the model type, version, license, funders, release date, developers, and contact information.*
+*It is useful for anyone who wants to reference the model.*
+  
+**Developed by:** BigScience ([website](https://bigscience.huggingface.co))
 
-This modelcard aims to be a base template for new models. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1).
+*All collaborators are either volunteers or have an agreement with their employer. (Further breakdown of participants forthcoming.)*
+    
+**Model Type:** Transformer-based Language Model
 
-## Model Details
+**Checkpoints format:** `transformers` (Megatron-DeepSpeed format available [here](https://huggingface.co/bigscience/bloom-optimizer-states))
 
-### Model Description
+**Version:** 1.0.0
 
-<!-- Provide a longer summary of what this model is. -->
+**Languages:** Multiple; see [training data](#training-data)
 
+**License:** RAIL License v1.0 ([link](https://huggingface.co/spaces/bigscience/license) / [article and FAQ](https://bigscience.huggingface.co/blog/the-bigscience-rail-license))
 
+**Release Date Estimate:** Monday, 11.July.2022
 
-- **Developed by:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+**Send Questions to:** bigscience-contact@googlegroups.com
 
-### Model Sources [optional]
+**Cite as:** BigScience, _BigScience Language Open-science Open-access Multilingual (BLOOM) Language Model_. International, May 2021-May 2022
 
-<!-- Provide the basic links for the model. -->
+**Funded by:** 
+    
+* The French government.
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+* Hugging Face ([website](https://huggingface.co)).
 
-## Uses
+* Organizations of contributors.  *(Further breakdown of organizations forthcoming.)*
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+## Intended Use
+
+This model is being created in order to enable public research on large language models (LLMs). LLMs are intended to be used for language generation or as a pretrained base model that can be further fine-tuned for specific tasks. Use cases below are not exhaustive.
 
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+-   Text generation
 
-[More Information Needed]
+-   Exploring characteristics of language generated by a language model
 
-### Downstream Use [optional]
+    -   Examples: Cloze tests, counterfactuals, generations with reframings
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+### Downstream Use
 
-[More Information Needed]
+-   Tasks that leverage language models include: Information Extraction, Question Answering, Summarization
 
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-### How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure 
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
+Using the model in [high-stakes](#high-stakes) settings is out of scope for this model.  The model is not designed for [critical decisions](#critical-decisions) nor uses with any material consequences on an individual's livelihood or wellbeing. The model outputs content that appears factual but may not be correct.  
 
+Out-of-scope Uses Include:
 
-#### Training Hyperparameters
+-   Usage in biomedical domains, political and legal domains, or finance domains
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+-   Usage for evaluating or scoring individuals, such as for employment, education, or credit
 
-#### Speeds, Sizes, Times [optional]
+-   Applying the model for critical automatic decisions, generating factual content, creating reliable summaries, or generating predictions that must be correct
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+#### Misuse
 
-[More Information Needed]
+Intentionally using the model for harm, violating [human rights](#human-rights), or other kinds of malicious activities, is a misuse of this model. This includes:
 
-## Evaluation
+-   Spam generation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+-   Disinformation and influence operations
 
-### Testing Data, Factors & Metrics
+-   Disparagement and defamation
 
-#### Testing Data
+-   Harassment and abuse
+  
+-   [Deception](#deception)
 
-<!-- This should link to a Data Card if possible. -->
+-   Unconsented impersonation and imitation
 
-[More Information Needed]
+-   Unconsented surveillance 
 
-#### Factors
+-   Generating content without attribution to the model, as specified in the [RAIL License, Use Restrictions](https://huggingface.co/spaces/bigscience/license)
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
+## Bias, Risks, and Limitations
+*This section identifies foreseeable harms and misunderstandings.*
+    
+Model may:
 
-<!-- Relevant interpretability work for the model goes here -->
+-   Overrepresent some viewpoints and underrepresent others
 
-[More Information Needed]
+-   Contain stereotypes
+  
+-   Contain [personal information](#personal-data-and-information)
 
-## Environmental Impact
+-   Generate:
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+    -   Hateful, abusive, or violent language
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+    -   Discriminatory or prejudicial language
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+    -   Content that may not be appropriate for all settings, including sexual content
 
-## Technical Specifications [optional]
+-   Make errors, including producing incorrect information as if it were factual
 
-### Model Architecture and Objective
+-   Generate irrelevant or repetitive outputs
 
-[More Information Needed]
+-   Induce users into attributing human traits to it, such as sentience or consciousness
 
-### Compute Infrastructure
+## Technical Specifications
+*This section includes details about the model objective and architecture, and the compute infrastructure.*
+*It is useful for people interested in model development.*
 
-[More Information Needed]
+### Compute infrastructure
+Jean Zay Public Supercomputer, provided by the French government (see [announcement](https://www.enseignementsup-recherche.gouv.fr/fr/signature-du-marche-d-acquisition-de-l-un-des-supercalculateurs-les-plus-puissants-d-europe-46733)).
 
 #### Hardware
 
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
+* 384 A100 80GB GPUs (48 nodes)
+    
+* Additional 32 A100 80GB GPUs (4 nodes) in reserve
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+* 8 GPUs per node Using NVLink 4 inter-gpu connects, 4 OmniPath links
 
-**BibTeX:**
+* CPU: AMD
 
-[More Information Needed]
+* CPU memory: 512GB per node
 
-**APA:**
+* GPU memory: 640GB per node
 
-[More Information Needed]
+* Inter-node connect: Omni-Path Architecture (OPA)
 
-## Glossary [optional]
+* NCCL-communications network: a fully dedicated subnet
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+* Disc IO network: shared network with other types of nodes
 
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
+#### Software
 
-## Model Card Contact
+* Megatron-DeepSpeed ([Github link](https://github.com/bigscience-workshop/Megatron-DeepSpeed))
 
-[More Information Needed]
+* DeepSpeed ([Github link](https://github.com/microsoft/DeepSpeed))
 
+* PyTorch (pytorch-1.11 w/ CUDA-11.5; see [Github link](https://github.com/pytorch/pytorch))
 
+* apex ([Github link](https://github.com/NVIDIA/apex))