metadata

title: Llama Nallm Test
emoji: 💬
colorFrom: yellow
colorTo: purple
sdk: gradio
sdk_version: 4.36.1
app_file: app.py
pinned: false
license: mit

A simple interface for testing out NA-LLM-qa model based on Llama-3-8B checkpoint.

Use

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
gradio app.py

Notes

The model is hosted with HuggingFace Inference Endpoint.
- The Endpoint may be paused due to inactivity. In that case, calling a signal will "wake up" the endpoint, but it would take around several minutes.
- For now, the endpoint is gated. Set appropriate hf_token with READ permission to the organization.
Input filtering
- The model performs unexpectedly for non-questions
- For this reason, a simple SVM-based filter is applied
  - The filter is a OneClassSVM trained with question sections of na-llm
  - The model, along with corresponding vectorizer, is saved in question_undetector.pkl as (vectorizer, model) object.
- The hosting machine should be powerful enough at least to run a simple SVM model