Experience with LASER

#1
by Venkman42 - opened

Hi there,

Just wanted to know how you experienced the process of making this.
Was it a big hassle?
How computational intensive is it to apply the LASER Technique?
Do you think it's possible on a consumer grade gpu/ gaming pc? Or does it require renting something like a A100?

Owner

Hi @Venkman42 , it wasn't too difficult but it requires a lot of VRAM so I had to use an A100. I tried playing with randomized SVD (https://github.com/mlabonne/laserRMT/tree/main) to lower the VRAM requirements, but the perplexity calculation is still too expensive to run on a T4. It requires more work to work on a consumer-grade GPU but I'm pretty sure it's possible. @fernandofernandes already lowered the bar compared to the original implementation of LASER.

Hi @Venkman42 , it wasn't too difficult but it requires a lot of VRAM so I had to use an A100. I tried playing with randomized SVD (https://github.com/mlabonne/laserRMT/tree/main) to lower the VRAM requirements, but the perplexity calculation is still too expensive to run on a T4. It requires more work to work on a consumer-grade GPU but I'm pretty sure it's possible. @fernandofernandes already lowered the bar compared to the original implementation of LASER.

Thanks for the insight :) looks like I'll have to have some patience and wait for other people to come out with laser versions ^^ would love to see a phi-2 dpo laser version ^^

what is the prompt template for this?

Sign up or log in to comment