jbochi

/

madlad400-3b-mt

like 128

You need to install candle first (see https://github.com/huggingface/candle#get-started).

If you already have it installed, do a git pull . Support of MADLAD was added in https://github.com/huggingface/candle/pull/1285.

Please reopen if the error persists.

If you already have it installed, do a git pull .

I installed candle. But what should i pull with git?

My guess is that you are using an older version of candle. If you do a git pull, you should get the latest changes.

My guess is that you are using an older version of candle. If you do a git pull, you should get the latest changes.

I done these steps:

cargo new myapp
cd myapp
cargo add --git https://github.com/huggingface/candle.git candle-core
cargo build
cargo run --example quantized-t5 --release -- --model-id "jbochi/madlad400-3b-mt" --weight-file "model-q4k.gguf" --prompt "<2de> How are you, my friend?" --temperature 0

What else should i do? Have i missed something?

Can you please try this?

git clone https://github.com/huggingface/candle.git 
cd candle
cargo run --example quantized-t5 --release -- --model-id "jbochi/madlad400-3b-mt" --weight-file "model-q4k.gguf" --prompt "<2de> How are you, my friend?" --temperature 0

If you are using a mac, you can add --features accelerate after --release. If you have a GPU, you can add --features cuda.

Hey!
Is it possible to run same model "model-q4k.gguf" from local device, like post downloading the same, if yes how to achieve it, is their any cmd to load and run from folder.

Hey @MyNAAAm

candle checks if the weight file exists locally. It only downloads from the hub if it doesn't exist.

So you can do something like:

$ cargo run --example quantized-t5 --release -- --model-id "jbochi/madlad400-3b-mt" --weight-file "path/to/model-q4k.gguf" --config-file "path/to/config.gguf" --prompt "<2de> How are you, my friend?" --temperature 0

If it doesn't work, can you please open an issue in the candle repository?

Hello, thanks for these quantized versions. Can you please give hint on how to implement it into app where user input is asked for the text to be translated. I was working on offline app, where user inputs text and gets output in russian and quantized version is the best I have seen, as I have limited gpu memory. would highly apprecitate your help!

Can you please try this?

git clone https://github.com/huggingface/candle.git 
cd candle
cargo run --example quantized-t5 --release -- --model-id "jbochi/madlad400-3b-mt" --weight-file "model-q4k.gguf" --prompt "<2de> How are you, my friend?" --temperature 0

If you are using a mac, you can add --features accelerate after --release. If you have a GPU, you can add --features cuda.

The features cuda are not supported with the quantized-t5 example. (the t5 example does support it).

Can you please try this?
git clone https://github.com/huggingface/candle.git
cd candle
cargo run --example quantized-t5 --release -- --model-id "jbochi/madlad400-3b-mt" --weight-file "model-q4k.gguf" --prompt "<2de> How are you, my friend?" --temperature 0

This is working of course.

I am new to Rust and Cargo, but as I understand it, when the current directory is within the candle folder, Cargo will look for the example in the .../candle/examples folder, where it will indeed find and execute the example. In the original post, we are in the myapp folder and Cargo will search for the example in the .../myapp/examples folder, which is not existent or empty. I tried to place the candle quantized-t5 example in myapp/examples, but this results in loads of error messages, because many dependencies are not available there with "use xxx". Must be due to the Rust module/packaging system.