Spaces:
Runtime error
Runtime error
File size: 2,642 Bytes
54f8747 a240da9 e6fd86e a240da9 01ae0bb e364c3d 01ae0bb 29eecb6 a240da9 01ae0bb a240da9 e6fd86e a240da9 01ae0bb a240da9 66bc8ec 3d6a12e 29eecb6 1ec7d38 29eecb6 e364c3d 29eecb6 b38dea1 66bc8ec 2e4eb94 66bc8ec 29eecb6 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 |
---
title: Gistillery
emoji: 🏭
colorFrom: purple
colorTo: gray
sdk: docker
app_port: 7860
---
# Dump your knowledge, let AI refine it
## Installation
Create a Python environment with Python 3.10+. Install the requirements and the package:
``` sh
python -m pip install -r requirements.txt
python -m pip install .
```
For development, instead do:
``` sh
python -m pip install -r requirements.txt
python -m pip install -r requirements-dev.txt
python -m pip install -e .
```
## Starting
### Preparing environemnt
Set an environemnt variable called "HF_HUB_TOKEN" with your Hugging Face token or create a `.env` file with that env var.
### Running the app
Run the `start.sh` script. This may require a `chmod +x start.sh` if not already executable.
```sh
./start.sh
```
Instead, you can also run each part individually. For this, in one terminal, start the background worker:
```sh
python src/gistillery/worker.py
```
In another terminal, start the web server:
```sh
uvicorn src.gistillery.webservice:app --reload --port 8080
```
For example requests, check `requests.org`.
A very simple web interface is available via gradio. To start it, run:
```sh
python demo.py
```
and navigate to the indicated URL (usually http://127.0.0.1:7860).
## Docker
To run everything with Docker, first build the image:
```sh
docker build -t gistillery:latest .
```
Next run the container:
```sh
docker run -p 7860:7860 -p 8080:8080 -v $HOME/.cache/huggingface/hub:/home/user/.cache/huggingface/hub gistillery:latest
```
Note that the Hugging Face cache folder is mounted as a docker volume to make use of potentially available local model cache instead of downloading the transformers models each time the container is started. To prevent that, remove the `-v ...` parameter. The database used for storing the results is ephemeral and will be deleted when the docker container is stopped. The backend server is also exposed directly via port 8080 to enable DB backups (see below).
### Backup
To download a backup of the backend DB, visit `localhost:8080/backup`. If you wish to start the app based on a backup, set the `DB_FILE_NAME` environment variable to the name of the backup.
## Checks
### Running tests
```sh
python -m pytest tests/
```
### Other
```sh
mypy src/
black src/ && black tests/ && black demo.py
ruff src/ && ruff tests/ && ruff demo.py
```
## TODOs
### Tools
i. Reading pdf in general
i. Reading arxiv
i. Generating text from youtube videos using whisper
### Deployment
i. Make DB location configurable, mountable when running in docker (otherwise, it will be deleted each time the container is stopped).
|