---
language: en
library_name: pytorch
license: apache-2.0
pipeline_tag: reinforcement-learning
tags:
- reinforcement-learning
- Generative Model
- GenerativeRL
- LunarLanderContinuous-v2
benchmark_name: Box2d
task_name: LunarLanderContinuous-v2
model-index:
- name: QGPO
  results:
  - task:
      type: reinforcement-learning
      name: reinforcement-learning
    dataset:
      name: LunarLanderContinuous-v2
      type: LunarLanderContinuous-v2
    metrics:
    - type: mean_reward
      value: '200.0'
      name: mean_reward
      verified: false
---

# Play **LunarLanderContinuous-v2** with **QGPO** Policy

## Model Description
<!-- Provide a longer summary of what this model is. -->

This implementation applies **QGPO** to the Box2d **LunarLanderContinuous-v2** environment using [GenerativeRL](https://github.com/opendilab/di-engine).



## Model Usage
### Install the Dependencies
<details close>
<summary>(Click for Details)</summary>

```shell
# install GenerativeRL with huggingface support
pip3 install GenerativeRL[huggingface]
# install environment dependencies if needed
pip3 install gym[box2d]==0.23.1
```
</details>

### Download Model from Huggingface and Run the Model

<details close>
<summary>(Click for Details)</summary>

```shell
# running with trained model
python3 -u run.py
```
**run.py**
```python
import gym

from grl.algorithms.qgpo import QGPOAlgorithm
from grl.datasets import QGPOCustomizedTensorDictDataset

from grl.utils.huggingface import pull_model_from_hub


def qgpo_pipeline():

    policy_state_dict, config = pull_model_from_hub(
        repo_id="zjowowen/LunarLanderContinuous-v2-QGPO",
    )

    qgpo = QGPOAlgorithm(
        config,
        dataset=QGPOCustomizedTensorDictDataset(
            numpy_data_path="./data.npz",
            action_augment_num=config.train.parameter.action_augment_num,
        ),
    )

    qgpo.model.load_state_dict(policy_state_dict)

    # ---------------------------------------
    # Customized train code ↓
    # ---------------------------------------
    # qgpo.train()
    # ---------------------------------------
    # Customized train code ↑
    # ---------------------------------------

    # ---------------------------------------
    # Customized deploy code ↓
    # ---------------------------------------
    agent = qgpo.deploy()
    env = gym.make(config.deploy.env.env_id)
    observation = env.reset()
    images = [env.render(mode="rgb_array")]
    for _ in range(config.deploy.num_deploy_steps):
        observation, reward, done, _ = env.step(agent.act(observation))
        image = env.render(mode="rgb_array")
        images.append(image)
    # save images into mp4 files
    import imageio.v3 as imageio
    import numpy as np

    images = np.array(images)
    imageio.imwrite("replay.mp4", images, fps=30, quality=8)
    # ---------------------------------------
    # Customized deploy code ↑
    # ---------------------------------------


if __name__ == "__main__":

    qgpo_pipeline()

```
</details>

## Model Training

### Train the Model and Push to Huggingface_hub

<details close>
<summary>(Click for Details)</summary>

```shell
#Training Your Own Agent
python3 -u train.py
```
**train.py**
```python
import gym

from grl.algorithms.qgpo import QGPOAlgorithm
from grl.datasets import QGPOCustomizedTensorDictDataset
from grl.utils.log import log
from grl_pipelines.diffusion_model.configurations.lunarlander_continuous_qgpo import (
    config,
)


def qgpo_pipeline(config):

    qgpo = QGPOAlgorithm(
        config,
        dataset=QGPOCustomizedTensorDictDataset(
            numpy_data_path="./data.npz",
            action_augment_num=config.train.parameter.action_augment_num,
        ),
    )

    # ---------------------------------------
    # Customized train code ↓
    # ---------------------------------------
    qgpo.train()
    # ---------------------------------------
    # Customized train code ↑
    # ---------------------------------------

    # ---------------------------------------
    # Customized deploy code ↓
    # ---------------------------------------
    agent = qgpo.deploy()
    env = gym.make(config.deploy.env.env_id)
    observation = env.reset()
    for _ in range(config.deploy.num_deploy_steps):
        env.render()
        observation, reward, done, _ = env.step(agent.act(observation))
    # ---------------------------------------
    # Customized deploy code ↑
    # ---------------------------------------


if __name__ == "__main__":
    log.info("config: \n{}".format(config))
    qgpo_pipeline(config)

```
</details>

**Configuration**
<details close>
<summary>(Click for Details)</summary>


```python
{'train': {'project': 'LunarLanderContinuous-v2-QGPO-VPSDE', 'device': 'cuda', 'wandb': {'project': 'IQL-LunarLanderContinuous-v2-QGPO-VPSDE'}, 'simulator': {'type': 'GymEnvSimulator', 'args': {'env_id': 'LunarLanderContinuous-v2'}}, 'model': {'QGPOPolicy': {'device': 'cuda', 'critic': {'device': 'cuda', 'q_alpha': 1.0, 'DoubleQNetwork': {'backbone': {'type': 'ConcatenateMLP', 'args': {'hidden_sizes': [10, 256, 256], 'output_size': 1, 'activation': 'relu'}}}}, 'diffusion_model': {'device': 'cuda', 'x_size': 2, 'alpha': 1.0, 'solver': {'type': 'DPMSolver', 'args': {'order': 2, 'device': 'cuda', 'steps': 17}}, 'path': {'type': 'linear_vp_sde', 'beta_0': 0.1, 'beta_1': 20.0}, 'reverse_path': {'type': 'linear_vp_sde', 'beta_0': 0.1, 'beta_1': 20.0}, 'model': {'type': 'noise_function', 'args': {'t_encoder': {'type': 'GaussianFourierProjectionTimeEncoder', 'args': {'embed_dim': 32, 'scale': 30.0}}, 'backbone': {'type': 'TemporalSpatialResidualNet', 'args': {'hidden_sizes': [512, 256, 128], 'output_dim': 2, 't_dim': 32, 'condition_dim': 8, 'condition_hidden_dim': 32, 't_condition_hidden_dim': 128}}}}, 'energy_guidance': {'t_encoder': {'type': 'GaussianFourierProjectionTimeEncoder', 'args': {'embed_dim': 32, 'scale': 30.0}}, 'backbone': {'type': 'ConcatenateMLP', 'args': {'hidden_sizes': [42, 256, 256], 'output_size': 1, 'activation': 'silu'}}}}}}, 'parameter': {'behaviour_policy': {'batch_size': 1024, 'learning_rate': 0.0001, 'epochs': 500}, 'action_augment_num': 16, 'fake_data_t_span': None, 'energy_guided_policy': {'batch_size': 256}, 'critic': {'stop_training_epochs': 500, 'learning_rate': 0.0001, 'discount_factor': 0.99, 'update_momentum': 0.005}, 'energy_guidance': {'epochs': 1000, 'learning_rate': 0.0001}, 'evaluation': {'evaluation_interval': 50, 'guidance_scale': [0.0, 1.0, 2.0]}, 'checkpoint_path': './LunarLanderContinuous-v2-QGPO'}}, 'deploy': {'device': 'cuda', 'env': {'env_id': 'LunarLanderContinuous-v2', 'seed': 0}, 'num_deploy_steps': 1000, 't_span': None}}
```

```json
{
    "train": {
        "project": "LunarLanderContinuous-v2-QGPO-VPSDE",
        "device": "cuda",
        "wandb": {
            "project": "IQL-LunarLanderContinuous-v2-QGPO-VPSDE"
        },
        "simulator": {
            "type": "GymEnvSimulator",
            "args": {
                "env_id": "LunarLanderContinuous-v2"
            }
        },
        "model": {
            "QGPOPolicy": {
                "device": "cuda",
                "critic": {
                    "device": "cuda",
                    "q_alpha": 1.0,
                    "DoubleQNetwork": {
                        "backbone": {
                            "type": "ConcatenateMLP",
                            "args": {
                                "hidden_sizes": [
                                    10,
                                    256,
                                    256
                                ],
                                "output_size": 1,
                                "activation": "relu"
                            }
                        }
                    }
                },
                "diffusion_model": {
                    "device": "cuda",
                    "x_size": 2,
                    "alpha": 1.0,
                    "solver": {
                        "type": "DPMSolver",
                        "args": {
                            "order": 2,
                            "device": "cuda",
                            "steps": 17
                        }
                    },
                    "path": {
                        "type": "linear_vp_sde",
                        "beta_0": 0.1,
                        "beta_1": 20.0
                    },
                    "reverse_path": {
                        "type": "linear_vp_sde",
                        "beta_0": 0.1,
                        "beta_1": 20.0
                    },
                    "model": {
                        "type": "noise_function",
                        "args": {
                            "t_encoder": {
                                "type": "GaussianFourierProjectionTimeEncoder",
                                "args": {
                                    "embed_dim": 32,
                                    "scale": 30.0
                                }
                            },
                            "backbone": {
                                "type": "TemporalSpatialResidualNet",
                                "args": {
                                    "hidden_sizes": [
                                        512,
                                        256,
                                        128
                                    ],
                                    "output_dim": 2,
                                    "t_dim": 32,
                                    "condition_dim": 8,
                                    "condition_hidden_dim": 32,
                                    "t_condition_hidden_dim": 128
                                }
                            }
                        }
                    },
                    "energy_guidance": {
                        "t_encoder": {
                            "type": "GaussianFourierProjectionTimeEncoder",
                            "args": {
                                "embed_dim": 32,
                                "scale": 30.0
                            }
                        },
                        "backbone": {
                            "type": "ConcatenateMLP",
                            "args": {
                                "hidden_sizes": [
                                    42,
                                    256,
                                    256
                                ],
                                "output_size": 1,
                                "activation": "silu"
                            }
                        }
                    }
                }
            }
        },
        "parameter": {
            "behaviour_policy": {
                "batch_size": 1024,
                "learning_rate": 0.0001,
                "epochs": 500
            },
            "action_augment_num": 16,
            "fake_data_t_span": null,
            "energy_guided_policy": {
                "batch_size": 256
            },
            "critic": {
                "stop_training_epochs": 500,
                "learning_rate": 0.0001,
                "discount_factor": 0.99,
                "update_momentum": 0.005
            },
            "energy_guidance": {
                "epochs": 1000,
                "learning_rate": 0.0001
            },
            "evaluation": {
                "evaluation_interval": 50,
                "guidance_scale": [
                    0.0,
                    1.0,
                    2.0
                ]
            },
            "checkpoint_path": "./LunarLanderContinuous-v2-QGPO"
        }
    },
    "deploy": {
        "device": "cuda",
        "env": {
            "env_id": "LunarLanderContinuous-v2",
            "seed": 0
        },
        "num_deploy_steps": 1000,
        "t_span": null
    }
}
```

</details>

**Training Procedure** 
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
- **Weights & Biases (wandb):** [monitor link](https://wandb.ai/zjowowen/IQL-LunarLanderContinuous-v2-QGPO-VPSDE)

## Model Information
<!-- Provide the basic links for the model. -->
- **Github Repository:** [repo link](https://github.com/opendilab/GenerativeRL/)
- **Doc**: [Algorithm link](https://opendilab.github.io/GenerativeRL/)
- **Configuration:** [config link](https://huggingface.co/OpenDILabCommunity/LunarLanderContinuous-v2-QGPO/blob/main/policy_config.json)
- **Demo:** [video](https://huggingface.co/OpenDILabCommunity/LunarLanderContinuous-v2-QGPO/blob/main/replay.mp4)
<!-- Provide the size information for the model. -->
- **Parameters total size:** 8799.79 KB
- **Last Update Date:** 2024-12-04

## Environments
<!-- Address questions around what environment the model is intended to be trained and deployed at, including the necessary information needed to be provided for future users. -->
- **Benchmark:** Box2d
- **Task:** LunarLanderContinuous-v2
- **Gym version:** 0.23.1
- **GenerativeRL version:** v0.0.1
- **PyTorch version:** 2.4.1+cu121
- **Doc**: [Environments link](https://www.gymlibrary.dev/environments/box2d/lunar_lander/)