Update README.md

README.md

@@ -47,6 +47,79 @@ base_model: utter-project/EuroLLM-1.7B
 This model was converted to GGUF format from [`utter-project/EuroLLM-1.7B`](https://huggingface.co/utter-project/EuroLLM-1.7B) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
 Refer to the [original model card](https://huggingface.co/utter-project/EuroLLM-1.7B) for more details on the model.
 
+---
+Model details:
+-
+This is the model card for the first pre-trained model of the EuroLLM
+ series: EuroLLM-1.7B. You can also check the instruction tuned version:
+ EuroLLM-1.7B-Instruct.
+
+
+Developed by: Unbabel, Instituto Superior Técnico, 
+Instituto de Telecomunicações, University of Edinburgh, Aveni, 
+University of Paris-Saclay, University of Amsterdam, Naver Labs, 
+Sorbonne Université.
+Funded by: European Union.
+Model type: A 1.7B parameter multilingual transfomer LLM.
+Language(s) (NLP): Bulgarian, Croatian, Czech, 
+Danish, Dutch, English, Estonian, Finnish, French, German, Greek, 
+Hungarian, Irish, Italian, Latvian, Lithuanian, Maltese, Polish, 
+Portuguese, Romanian, Slovak, Slovenian, Spanish, Swedish, Arabic, 
+Catalan, Chinese, Galician, Hindi, Japanese, Korean, Norwegian, Russian,
+ Turkish, and Ukrainian. 
+License: Apache License 2.0.
+
+
+
+	
+		
+	
+
+		Model Details
+	
+
+
+
+The EuroLLM project has the goal of creating a suite of LLMs capable 
+of understanding and generating text in all European Union languages as 
+well as some additional relevant languages.
+EuroLLM-1.7B is a 1.7B parameter model trained on 4 trillion tokens 
+divided across the considered languages and several data sources: Web 
+data, parallel data (en-xx and xx-en), and high-quality datasets.
+EuroLLM-1.7B-Instruct was further instruction tuned on EuroBlocks, an 
+instruction tuning dataset with focus on general instruction-following 
+and machine translation.
+
+
+
+	
+		
+	
+
+		Model Description
+	
+
+
+
+EuroLLM uses a standard, dense Transformer architecture:
+
+
+We use grouped query attention (GQA) with 8 key-value heads, since 
+it has been shown to increase speed at inference time while maintaining 
+downstream performance.
+We perform pre-layer normalization, since it improves the training stability, and use the RMSNorm, which is faster.
+We use the SwiGLU activation function, since it has been shown to lead to good results on downstream tasks.
+We use rotary positional embeddings (RoPE) in every layer, since 
+these have been shown to lead to good performances while allowing the 
+extension of the context length.
+
+
+For pre-training, we use 256 Nvidia H100 GPUs of the Marenostrum 5 
+supercomputer, training the model with a constant batch size of 3,072 
+sequences, which corresponds to approximately 12 million tokens, using 
+the Adam optimizer, and BF16 precision.
+
+---
 ## Use with llama.cpp
 Install llama.cpp through brew (works on Mac and Linux)