Instructions to use Sharathhebbar24/math_gpt2_sft with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Sharathhebbar24/math_gpt2_sft with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Sharathhebbar24/math_gpt2_sft")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Sharathhebbar24/math_gpt2_sft")
model = AutoModelForCausalLM.from_pretrained("Sharathhebbar24/math_gpt2_sft")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Sharathhebbar24/math_gpt2_sft with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Sharathhebbar24/math_gpt2_sft"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Sharathhebbar24/math_gpt2_sft",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Sharathhebbar24/math_gpt2_sft

SGLang

How to use Sharathhebbar24/math_gpt2_sft with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Sharathhebbar24/math_gpt2_sft" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Sharathhebbar24/math_gpt2_sft",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Sharathhebbar24/math_gpt2_sft" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Sharathhebbar24/math_gpt2_sft",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Sharathhebbar24/math_gpt2_sft with Docker Model Runner:
```
docker model run hf.co/Sharathhebbar24/math_gpt2_sft
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

This model is a finetuned version of Sharathhebbar24/math_gpt2 using meta-math/MetaMathQA

Model description

GPT-2 is a transformers model pre-trained on a very large corpus of English data in a self-supervised fashion. This means it was pre-trained on the raw texts only, with no humans labeling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it was trained to guess the next word in sentences.

More precisely, inputs are sequences of continuous text of a certain length and the targets are the same sequence, shifting one token (word or piece of word) to the right. The model uses a masking mechanism to make sure the predictions for the token i only use the inputs from 1 to i but not the future tokens.

This way, the model learns an inner representation of the English language that can then be used to extract features useful for downstream tasks. The model is best at what it was trained for, however, which is generating texts from a prompt.

To use this model

>>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> model_name = "Sharathhebbar24/math_gpt2_sft"
>>> model = AutoModelForCausalLM.from_pretrained(model_name)
>>> tokenizer = AutoTokenizer.from_pretrained(model_name)
>>> def generate_text(prompt):
>>>  inputs = tokenizer.encode(prompt, return_tensors='pt')
>>>  outputs = model.generate(inputs, max_length=64, pad_token_id=tokenizer.eos_token_id)
>>>  generated = tokenizer.decode(outputs[0], skip_special_tokens=True)
>>>  return generated[:generated.rfind(".")+1]
>>> prompt = "Gracie and Joe are choosing numbers on the complex plane. Joe chooses the point $1+2i$. Gracie chooses $-1+i$. How far apart are Gracie and Joe's points?"
>>> res = generate_text(prompt)
>>> res

Benchmark / Evaluation

Model	Average	ARC	HellaSwag	MMLU	TruthfulQA	Winogrande	GSM8k
Sharathhebbar24/math_gpt2_sft	28.503	22.87	30.41	25.06	37.62	51.54	0.68

{
    "all": {
        "acc": 0.25082189621988066,
        "acc_stderr": 0.030526589726831692,
        "acc_norm": 0.25112870356236633,
        "acc_norm_stderr": 0.03129390389566968,
        "mc1": 0.24112607099143207,
        "mc1_stderr": 0.014974827279752334,
        "mc2": 0.3762297840067963,
        "mc2_stderr": 0.01445991036363257
    },
    "harness|arc:challenge|25": {
        "acc": 0.20563139931740615,
        "acc_stderr": 0.01181074526074258,
        "acc_norm": 0.22866894197952217,
        "acc_norm_stderr": 0.012272853582540799
    },
    "harness|hellaswag|10": {
        "acc": 0.2884883489344752,
        "acc_stderr": 0.004521334761709224,
        "acc_norm": 0.30412268472415854,
        "acc_norm_stderr": 0.00459094683972719
    },
    "harness|hendrycksTest-abstract_algebra|5": {
        "acc": 0.19,
        "acc_stderr": 0.03942772444036625,
        "acc_norm": 0.19,
        "acc_norm_stderr": 0.03942772444036625
    },
    "harness|hendrycksTest-anatomy|5": {
        "acc": 0.2074074074074074,
        "acc_stderr": 0.03502553170678319,
        "acc_norm": 0.2074074074074074,
        "acc_norm_stderr": 0.03502553170678319
    },
    "harness|hendrycksTest-astronomy|5": {
        "acc": 0.17763157894736842,
        "acc_stderr": 0.031103182383123398,
        "acc_norm": 0.17763157894736842,
        "acc_norm_stderr": 0.031103182383123398
    },
    "harness|hendrycksTest-business_ethics|5": {
        "acc": 0.19,
        "acc_stderr": 0.03942772444036622,
        "acc_norm": 0.19,
        "acc_norm_stderr": 0.03942772444036622
    },
    "harness|hendrycksTest-clinical_knowledge|5": {
        "acc": 0.2188679245283019,
        "acc_stderr": 0.025447863825108618,
        "acc_norm": 0.2188679245283019,
        "acc_norm_stderr": 0.025447863825108618
    },
    "harness|hendrycksTest-college_biology|5": {
        "acc": 0.25,
        "acc_stderr": 0.03621034121889507,
        "acc_norm": 0.25,
        "acc_norm_stderr": 0.03621034121889507
    },
    "harness|hendrycksTest-college_chemistry|5": {
        "acc": 0.2,
        "acc_stderr": 0.04020151261036845,
        "acc_norm": 0.2,
        "acc_norm_stderr": 0.04020151261036845
    },
    "harness|hendrycksTest-college_computer_science|5": {
        "acc": 0.32,
        "acc_stderr": 0.046882617226215034,
        "acc_norm": 0.32,
        "acc_norm_stderr": 0.046882617226215034
    },
    "harness|hendrycksTest-college_mathematics|5": {
        "acc": 0.24,
        "acc_stderr": 0.042923469599092816,
        "acc_norm": 0.24,
        "acc_norm_stderr": 0.042923469599092816
    },
    "harness|hendrycksTest-college_medicine|5": {
        "acc": 0.21965317919075145,
        "acc_stderr": 0.031568093627031744,
        "acc_norm": 0.21965317919075145,
        "acc_norm_stderr": 0.031568093627031744
    },
    "harness|hendrycksTest-college_physics|5": {
        "acc": 0.23529411764705882,
        "acc_stderr": 0.04220773659171453,
        "acc_norm": 0.23529411764705882,
        "acc_norm_stderr": 0.04220773659171453
    },
    "harness|hendrycksTest-computer_security|5": {
        "acc": 0.23,
        "acc_stderr": 0.04229525846816505,
        "acc_norm": 0.23,
        "acc_norm_stderr": 0.04229525846816505
    },
    "harness|hendrycksTest-conceptual_physics|5": {
        "acc": 0.2680851063829787,
        "acc_stderr": 0.028957342788342347,
        "acc_norm": 0.2680851063829787,
        "acc_norm_stderr": 0.028957342788342347
    },
    "harness|hendrycksTest-econometrics|5": {
        "acc": 0.24561403508771928,
        "acc_stderr": 0.040493392977481404,
        "acc_norm": 0.24561403508771928,
        "acc_norm_stderr": 0.040493392977481404
    },
    "harness|hendrycksTest-electrical_engineering|5": {
        "acc": 0.2482758620689655,
        "acc_stderr": 0.036001056927277716,
        "acc_norm": 0.2482758620689655,
        "acc_norm_stderr": 0.036001056927277716
    },
    "harness|hendrycksTest-elementary_mathematics|5": {
        "acc": 0.24074074074074073,
        "acc_stderr": 0.0220190800122179,
        "acc_norm": 0.24074074074074073,
        "acc_norm_stderr": 0.0220190800122179
    },
    "harness|hendrycksTest-formal_logic|5": {
        "acc": 0.23015873015873015,
        "acc_stderr": 0.03764950879790605,
        "acc_norm": 0.23015873015873015,
        "acc_norm_stderr": 0.03764950879790605
    },
    "harness|hendrycksTest-global_facts|5": {
        "acc": 0.18,
        "acc_stderr": 0.038612291966536934,
        "acc_norm": 0.18,
        "acc_norm_stderr": 0.038612291966536934
    },
    "harness|hendrycksTest-high_school_biology|5": {
        "acc": 0.25483870967741934,
        "acc_stderr": 0.024790118459332208,
        "acc_norm": 0.25483870967741934,
        "acc_norm_stderr": 0.024790118459332208
    },
    "harness|hendrycksTest-high_school_chemistry|5": {
        "acc": 0.19704433497536947,
        "acc_stderr": 0.02798672466673622,
        "acc_norm": 0.19704433497536947,
        "acc_norm_stderr": 0.02798672466673622
    },
    "harness|hendrycksTest-high_school_computer_science|5": {
        "acc": 0.22,
        "acc_stderr": 0.041633319989322695,
        "acc_norm": 0.22,
        "acc_norm_stderr": 0.041633319989322695
    },
    "harness|hendrycksTest-high_school_european_history|5": {
        "acc": 0.19393939393939394,
        "acc_stderr": 0.0308741451365621,
        "acc_norm": 0.19393939393939394,
        "acc_norm_stderr": 0.0308741451365621
    },
    "harness|hendrycksTest-high_school_geography|5": {
        "acc": 0.3484848484848485,
        "acc_stderr": 0.033948539651564025,
        "acc_norm": 0.3484848484848485,
        "acc_norm_stderr": 0.033948539651564025
    },
    "harness|hendrycksTest-high_school_government_and_politics|5": {
        "acc": 0.32124352331606215,
        "acc_stderr": 0.033699508685490674,
        "acc_norm": 0.32124352331606215,
        "acc_norm_stderr": 0.033699508685490674
    },
    "harness|hendrycksTest-high_school_macroeconomics|5": {
        "acc": 0.23333333333333334,
        "acc_stderr": 0.021444547301560476,
        "acc_norm": 0.23333333333333334,
        "acc_norm_stderr": 0.021444547301560476
    },
    "harness|hendrycksTest-high_school_mathematics|5": {
        "acc": 0.2851851851851852,
        "acc_stderr": 0.027528599210340492,
        "acc_norm": 0.2851851851851852,
        "acc_norm_stderr": 0.027528599210340492
    },
    "harness|hendrycksTest-high_school_microeconomics|5": {
        "acc": 0.29831932773109243,
        "acc_stderr": 0.029719142876342856,
        "acc_norm": 0.29831932773109243,
        "acc_norm_stderr": 0.029719142876342856
    },
    "harness|hendrycksTest-high_school_physics|5": {
        "acc": 0.2781456953642384,
        "acc_stderr": 0.03658603262763744,
        "acc_norm": 0.2781456953642384,
        "acc_norm_stderr": 0.03658603262763744
    },
    "harness|hendrycksTest-high_school_psychology|5": {
        "acc": 0.26788990825688075,
        "acc_stderr": 0.018987462257978652,
        "acc_norm": 0.26788990825688075,
        "acc_norm_stderr": 0.018987462257978652
    },
    "harness|hendrycksTest-high_school_statistics|5": {
        "acc": 0.4351851851851852,
        "acc_stderr": 0.03381200005643525,
        "acc_norm": 0.4351851851851852,
        "acc_norm_stderr": 0.03381200005643525
    },
    "harness|hendrycksTest-high_school_us_history|5": {
        "acc": 0.2647058823529412,
        "acc_stderr": 0.0309645179269234,
        "acc_norm": 0.2647058823529412,
        "acc_norm_stderr": 0.0309645179269234
    },
    "harness|hendrycksTest-high_school_world_history|5": {
        "acc": 0.28270042194092826,
        "acc_stderr": 0.029312814153955927,
        "acc_norm": 0.28270042194092826,
        "acc_norm_stderr": 0.029312814153955927
    },
    "harness|hendrycksTest-human_aging|5": {
        "acc": 0.31390134529147984,
        "acc_stderr": 0.031146796482972465,
        "acc_norm": 0.31390134529147984,
        "acc_norm_stderr": 0.031146796482972465
    },
    "harness|hendrycksTest-human_sexuality|5": {
        "acc": 0.2595419847328244,
        "acc_stderr": 0.03844876139785271,
        "acc_norm": 0.2595419847328244,
        "acc_norm_stderr": 0.03844876139785271
    },
    "harness|hendrycksTest-international_law|5": {
        "acc": 0.2231404958677686,
        "acc_stderr": 0.03800754475228733,
        "acc_norm": 0.2231404958677686,
        "acc_norm_stderr": 0.03800754475228733
    },
    "harness|hendrycksTest-jurisprudence|5": {
        "acc": 0.25925925925925924,
        "acc_stderr": 0.042365112580946336,
        "acc_norm": 0.25925925925925924,
        "acc_norm_stderr": 0.042365112580946336
    },
    "harness|hendrycksTest-logical_fallacies|5": {
        "acc": 0.25153374233128833,
        "acc_stderr": 0.03408997886857529,
        "acc_norm": 0.25153374233128833,
        "acc_norm_stderr": 0.03408997886857529
    },
    "harness|hendrycksTest-machine_learning|5": {
        "acc": 0.29464285714285715,
        "acc_stderr": 0.043270409325787296,
        "acc_norm": 0.29464285714285715,
        "acc_norm_stderr": 0.043270409325787296
    },
    "harness|hendrycksTest-management|5": {
        "acc": 0.17475728155339806,
        "acc_stderr": 0.037601780060266224,
        "acc_norm": 0.17475728155339806,
        "acc_norm_stderr": 0.037601780060266224
    },
    "harness|hendrycksTest-marketing|5": {
        "acc": 0.20085470085470086,
        "acc_stderr": 0.026246772946890488,
        "acc_norm": 0.20085470085470086,
        "acc_norm_stderr": 0.026246772946890488
    },
    "harness|hendrycksTest-medical_genetics|5": {
        "acc": 0.3,
        "acc_stderr": 0.046056618647183814,
        "acc_norm": 0.3,
        "acc_norm_stderr": 0.046056618647183814
    },
    "harness|hendrycksTest-miscellaneous|5": {
        "acc": 0.23499361430395913,
        "acc_stderr": 0.01516202415227844,
        "acc_norm": 0.23499361430395913,
        "acc_norm_stderr": 0.01516202415227844
    },
    "harness|hendrycksTest-moral_disputes|5": {
        "acc": 0.23699421965317918,
        "acc_stderr": 0.02289408248992599,
        "acc_norm": 0.23699421965317918,
        "acc_norm_stderr": 0.02289408248992599
    },
    "harness|hendrycksTest-moral_scenarios|5": {
        "acc": 0.23798882681564246,
        "acc_stderr": 0.014242630070574915,
        "acc_norm": 0.23798882681564246,
        "acc_norm_stderr": 0.014242630070574915
    },
    "harness|hendrycksTest-nutrition|5": {
        "acc": 0.23202614379084968,
        "acc_stderr": 0.024170840879341005,
        "acc_norm": 0.23202614379084968,
        "acc_norm_stderr": 0.024170840879341005
    },
    "harness|hendrycksTest-philosophy|5": {
        "acc": 0.1864951768488746,
        "acc_stderr": 0.02212243977248077,
        "acc_norm": 0.1864951768488746,
        "acc_norm_stderr": 0.02212243977248077
    },
    "harness|hendrycksTest-prehistory|5": {
        "acc": 0.24074074074074073,
        "acc_stderr": 0.02378858355165854,
        "acc_norm": 0.24074074074074073,
        "acc_norm_stderr": 0.02378858355165854
    },
    "harness|hendrycksTest-professional_accounting|5": {
        "acc": 0.2695035460992908,
        "acc_stderr": 0.026469036818590627,
        "acc_norm": 0.2695035460992908,
        "acc_norm_stderr": 0.026469036818590627
    },
    "harness|hendrycksTest-professional_law|5": {
        "acc": 0.2529335071707953,
        "acc_stderr": 0.011102268713839989,
        "acc_norm": 0.2529335071707953,
        "acc_norm_stderr": 0.011102268713839989
    },
    "harness|hendrycksTest-professional_medicine|5": {
        "acc": 0.4411764705882353,
        "acc_stderr": 0.030161911930767102,
        "acc_norm": 0.4411764705882353,
        "acc_norm_stderr": 0.030161911930767102
    },
    "harness|hendrycksTest-professional_psychology|5": {
        "acc": 0.25,
        "acc_stderr": 0.01751781884501444,
        "acc_norm": 0.25,
        "acc_norm_stderr": 0.01751781884501444
    },
    "harness|hendrycksTest-public_relations|5": {
        "acc": 0.21818181818181817,
        "acc_stderr": 0.03955932861795833,
        "acc_norm": 0.21818181818181817,
        "acc_norm_stderr": 0.03955932861795833
    },
    "harness|hendrycksTest-security_studies|5": {
        "acc": 0.20408163265306123,
        "acc_stderr": 0.025801283475090506,
        "acc_norm": 0.20408163265306123,
        "acc_norm_stderr": 0.025801283475090506
    },
    "harness|hendrycksTest-sociology|5": {
        "acc": 0.24378109452736318,
        "acc_stderr": 0.03036049015401465,
        "acc_norm": 0.24378109452736318,
        "acc_norm_stderr": 0.03036049015401465
    },
    "harness|hendrycksTest-us_foreign_policy|5": {
        "acc": 0.24,
        "acc_stderr": 0.04292346959909281,
        "acc_norm": 0.24,
        "acc_norm_stderr": 0.04292346959909281
    },
    "harness|hendrycksTest-virology|5": {
        "acc": 0.22289156626506024,
        "acc_stderr": 0.03240004825594687,
        "acc_norm": 0.22289156626506024,
        "acc_norm_stderr": 0.03240004825594687
    },
    "harness|hendrycksTest-world_religions|5": {
        "acc": 0.3216374269005848,
        "acc_stderr": 0.03582529442573122,
        "acc_norm": 0.3216374269005848,
        "acc_norm_stderr": 0.03582529442573122
    },
    "harness|truthfulqa:mc|0": {
        "mc1": 0.24112607099143207,
        "mc1_stderr": 0.014974827279752334,
        "mc2": 0.3762297840067963,
        "mc2_stderr": 0.01445991036363257
    },
    "harness|winogrande|5": {
        "acc": 0.5153906866614049,
        "acc_stderr": 0.014045826789783668
    },
    "harness|gsm8k|5": {
        "acc": 0.006823351023502654,
        "acc_stderr": 0.0022675371022544823
    }
}

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	28.03
AI2 Reasoning Challenge (25-Shot)	22.87
HellaSwag (10-Shot)	30.41
MMLU (5-Shot)	25.06
TruthfulQA (0-shot)	37.62
Winogrande (5-shot)	51.54
GSM8k (5-shot)	0.68

Downloads last month: 226

Safetensors

Model size

0.1B params

Tensor type

F32

Model tree for Sharathhebbar24/math_gpt2_sft

Quantizations

1 model

Dataset used to train Sharathhebbar24/math_gpt2_sft

Space using Sharathhebbar24/math_gpt2_sft 1

Evaluation results

normalized accuracy on AI2 Reasoning Challenge (25-Shot)
test set Open LLM Leaderboard

22.870
normalized accuracy on HellaSwag (10-Shot)
validation set Open LLM Leaderboard

30.410
accuracy on MMLU (5-Shot)
test set Open LLM Leaderboard

25.060
mc2 on TruthfulQA (0-shot)
validation set Open LLM Leaderboard

37.620
accuracy on Winogrande (5-shot)
validation set Open LLM Leaderboard

51.540
accuracy on GSM8k (5-shot)
test set Open LLM Leaderboard

0.680