|
---
|
|
license: mit
|
|
---
|
|
|
|
# Tau LLM Unity ML Agents Project
|
|
|
|
Welcome to the Tau LLM Unity ML Agents Project repository! This project focuses on training reinforcement learning agents using Unity ML-Agents and the PPO algorithm. Our goal is to optimize the performance of the agents through various configurations and training runs.
|
|
|
|
## Project Overview
|
|
|
|
This repository contains the code and configurations for training agents in a Unity environment using the Proximal Policy Optimization (PPO) algorithm. The agents are designed to learn and adapt to their environment, improving their performance over time.
|
|
|
|
### Key Features
|
|
|
|
- **Reinforcement Learning**: Utilizes the PPO algorithm for training agents.
|
|
- **Unity ML-Agents**: Integrates with Unity ML-Agents for a seamless training experience.
|
|
- **Custom Reward Functions**: Implements gradient-based reward functions for nuanced feedback.
|
|
- **Memory Networks**: Incorporates memory networks to handle temporal dependencies.
|
|
- **TensorBoard Integration**: Monitors training progress and performance using TensorBoard.
|
|
|
|
## Configuration
|
|
|
|
Below is the configuration used for training the agents:
|
|
|
|
```yaml
|
|
behaviors:
|
|
TauAgent:
|
|
trainer_type: ppo
|
|
hyperparameters:
|
|
batch_size: 256
|
|
buffer_size: 4096
|
|
learning_rate: 0.00003
|
|
beta: 0.005
|
|
epsilon: 0.2
|
|
lambd: 0.95
|
|
num_epoch: 10
|
|
learning_rate_schedule: linear
|
|
network_settings:
|
|
normalize: true
|
|
hidden_units: 256
|
|
num_layers: 4
|
|
vis_encode_type: simple
|
|
memory:
|
|
memory_size: 256
|
|
sequence_length: 256
|
|
num_layers: 4
|
|
reward_signals:
|
|
extrinsic:
|
|
gamma: 0.99
|
|
strength: 1.0
|
|
curiosity:
|
|
gamma: 0.995
|
|
strength: 0.1
|
|
network_settings:
|
|
normalize: true
|
|
hidden_units: 256
|
|
num_layers: 4
|
|
learning_rate: 0.00003
|
|
keep_checkpoints: 10
|
|
checkpoint_interval: 100000
|
|
threaded: true
|
|
max_steps: 3000000
|
|
time_horizon: 256
|
|
summary_freq: 10000
|
|
```
|
|
|
|
## Model Naming Convention
|
|
|
|
The models in this repository follow the naming convention `Tau_<series>_<max_steps>`. This helps in easily identifying the series and the number of training steps for each model.
|
|
|
|
## Getting Started
|
|
|
|
### Prerequisites
|
|
|
|
- Unity 6
|
|
- Unity ML-Agents Toolkit
|
|
- Python 3.10.11
|
|
- PyTorch
|
|
- Transformers
|
|
|
|
### Installation
|
|
|
|
1. Clone the repository:
|
|
```bash
|
|
git clone https://github.com/p3nGu1nZz/Tau.git
|
|
cd tau\MLAgentsProject
|
|
```
|
|
|
|
2. Install the required Python packages:
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
3. Open the Unity project:
|
|
- Launch Unity Hub and open the project folder.
|
|
|
|
### Training the Agent
|
|
|
|
To start training the agent, run the following command:
|
|
```bash
|
|
mlagents-learn .\config\tau_agent_ppo_c.yaml --run-id=tau_agent_ppo_A0 --env .\Build --torch-device cuda --timeout-wait 300 --force
|
|
```
|
|
Note: The preferred way to run a build is by creating a new build into the `Build` directory which is referenced by the above command.
|
|
|
|
### Monitoring Training
|
|
|
|
You can monitor the training progress using TensorBoard:
|
|
```bash
|
|
tensorboard --logdir results
|
|
```
|
|
|
|
## Results
|
|
|
|
The training results, including the average reward and cumulative reward, can be visualized using TensorBoard. The graphs below show the performance of the agent over time:
|
|
|
|
|
|
|
|
|
|
## Citation
|
|
|
|
If you use this project in your research, please cite it as follows:
|
|
|
|
```bibtex
|
|
@misc{Tau,
|
|
author = {K. Rawson},
|
|
title = {Tau LLM Unity ML Agents Project},
|
|
year = {2024},
|
|
publisher = {GitHub},
|
|
journal = {GitHub repository},
|
|
howpublished = {\url{https://github.com/p3nGu1nZz/Tau}},
|
|
}
|
|
```
|
|
|
|
## License
|
|
|
|
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
|
|
|
|
## Acknowledgments
|
|
|
|
- Unity ML-Agents Toolkit
|
|
- TensorFlow and PyTorch communities
|
|
- Hugging Face for hosting the model repository |
