README.md

metadata

datasets:
  - bigcode/starcoderdata
  - cerebras/SlimPajama-627B
pipeline_tag: text-generation

Ultron_storm_sft_20231210

Ultron is a series of LLMs ranging from 160M to 1.1B parameters.

Details of Ultron_storm_sft_20231210

Parameters: 1.1B parameters

Attention: Grouped Query Attention

Sequence Length: 2048 tokens

Learning rate: 4e-4

Dataset Size: 950B tokens

Note: This model is just a placeholder and doesn't represent the final Ultron lineup.