README.md

---
title: Llama Nallm Test
emoji: 💬
colorFrom: yellow
colorTo: purple
sdk: gradio
sdk_version: 4.36.1
app_file: app.py
pinned: false
license: mit
---

A simple interface for testing out NA-LLM-qa model based on Llama-3-8B checkpoint.

# Use

```sh
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
gradio app.py
```

# Notes

- The model is hosted with HuggingFace Inference Endpoint.
  - The Endpoint may be paused due to inactivity. In that case, calling a signal will "wake up" the endpoint, but it would take around several minutes.
  - For now, the endpoint is gated. Set appropriate `hf_token` with READ permission to the organization.
- Input filtering
  - The model performs unexpectedly for non-questions
  - For this reason, a simple SVM-based filter is applied
    - The filter is a `OneClassSVM` trained with question sections of na-llm
    - The model, along with corresponding vectorizer, is saved in `question_undetector.pkl` as `(vectorizer, model)` object.
  - The hosting machine should be powerful enough at least to run a simple SVM model