metadata

language:
  - en
license: apache-2.0
base_model:
  - HuggingFaceTB/SmolLM2-1.7B-Instruct
datasets:
  - allenai/tulu-3-sft-mixture
  - allenai/llama-3.1-tulu-3-8b-preference-mixture
model-index:
  - name: SmolLM2-MedIT-Upscale-2B
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: IFEval (0-Shot)
          type: HuggingFaceH4/ifeval
          args:
            num_few_shot: 0
        metrics:
          - type: inst_level_strict_acc and prompt_level_strict_acc
            value: 64.29
            name: strict accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=meditsolutions/SmolLM2-MedIT-Upscale-2B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: BBH (3-Shot)
          type: BBH
          args:
            num_few_shot: 3
        metrics:
          - type: acc_norm
            value: 10.51
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=meditsolutions/SmolLM2-MedIT-Upscale-2B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MATH Lvl 5 (4-Shot)
          type: hendrycks/competition_math
          args:
            num_few_shot: 4
        metrics:
          - type: exact_match
            value: 1.06
            name: exact match
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=meditsolutions/SmolLM2-MedIT-Upscale-2B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GPQA (0-shot)
          type: Idavidrein/gpqa
          args:
            num_few_shot: 0
        metrics:
          - type: acc_norm
            value: 1.9
            name: acc_norm
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=meditsolutions/SmolLM2-MedIT-Upscale-2B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MuSR (0-shot)
          type: TAUR-Lab/MuSR
          args:
            num_few_shot: 0
        metrics:
          - type: acc_norm
            value: 2.45
            name: acc_norm
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=meditsolutions/SmolLM2-MedIT-Upscale-2B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU-PRO (5-shot)
          type: TIGER-Lab/MMLU-Pro
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 10.78
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=meditsolutions/SmolLM2-MedIT-Upscale-2B
          name: Open LLM Leaderboard

SmolLM2-MedIT-Upscale-2B

Model Summary

SmolLM2-MedIT-Upscale-2B is an expanded version of the SmolLM2-1.7B-Instruct model, increasing its parameter count to 2 billion. This expansion was achieved by doubling the number of heads in the q_proj, k_proj, v_proj, and o_proj layers, resulting in vectors of length 4096, compared to 2048 in the original model.

Purpose of Expansion

This model was developed to test the hypothesis that self-attention layers do not extend the "memory" of the model. By broadening the attention layers, we aim to observe the impact on the model's performance and memory capabilities.

Training Status

This model underwent instruction fine-tuning for 8,800 steps using a batch size of 4, gradient accumulation for 32 steps, a maximum sequence length of 1,280, and a learning rate of 1e-5. Additionally, it was fine-tuned with 1,600 steps of DPO under the same configuration.

Note: The model has undergone preliminary training focused on assessing the effects of the expanded attention layers. It is not fully trained to its maximum potential. We encourage the community to contribute to its further training; pull requests are welcome.

Analysis of Expanded Layers

During fine-tuning, we analyzed the changes in the new parameters of the expanded layers:

Minimum percentage of new parameters that changed:
- q_proj: 62.17%
- k_proj: 37.85%
- v_proj: 99.99%
- o_proj: 99.98%
Maximum percentage of new parameters that changed:
- q_proj: 98.86%
- k_proj: 97.99%
- v_proj: 99.99%
- o_proj: 99.99%
Average change in new parameters after fine-tuning:
- q_proj: 1.838e-07
- k_proj: 2.277e-07
- v_proj: 6.490e-07
- o_proj: 3.924e-07

These results are illustrated in the following charts:

Usage

To utilize this model, follow the instructions provided for the original SmolLM2-1.7B-Instruct model, adjusting for the increased parameter size.

Contributing

We welcome contributions to further train and evaluate this model. Please submit pull requests with your improvements.

License

This model is licensed under the Apache 2.0 License.

Citation

If you use this model in your research, please cite it as follows:

@misc{SmolLM2-MedIT-Upscale-2B,
  author = {Mariusz Kurman, MedIT Solutions},
  title = {SmolLM2-MedIT-Upscale-2B: An Expanded Version of SmolLM2-1.7B-Instruct},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/meditsolutions/SmolLM2-MedIT-Upscale-2B},
}

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	15.17
IFEval (0-Shot)	64.29
BBH (3-Shot)	10.51
MATH Lvl 5 (4-Shot)	1.06
GPQA (0-shot)	1.90
MuSR (0-shot)	2.45
MMLU-PRO (5-shot)	10.78