Sheikhaei

/

llama-3.2-1b-en-zh-translator

like 0

Hi!
If you share the compressed/optimized version of the model, I will run the test and report the scores to you.
Bests,
Sadegh

First, you should install openvino, openvino-genai, optimum-intel (I install them by conda and pip).

The compressed/optimized version of the model is generate by following command:

optimum-cli export openvino --model Sheikhaei/llama-3.2-1b-en-zh-translator --weight-format int4 --trust-remote-code ./llama-3.2-1b-en-zh-translator-ov-int4

And the example is:

import openvino_genai as ov_genai
import time

def streamer(subword):
    print(subword, end='', flush=True)
    # Return flag corresponds whether generation should be stopped.
    return ov_genai.StreamingStatus.RUNNING


config = ov_genai.GenerationConfig()
config.max_new_tokens = 1000

model_path = "D:\\tmp\\llama-3.2-1b-en-zh-translator-ov-int4\\"

start_time = time.time()
pipe = ov_genai.LLMPipeline(model_path, "CPU")

print(time.time() - start_time)
start_time = time.time()

prompt = """### English:
I am using openvino to compress and optimize the model，and try to calculate the BLEU score and COMET score.
### Mandarin:
"""

pipe.generate(prompt, config, streamer)

print(time.time() - start_time)