AnnaMats's picture
Second Push
05c9ac2
๏ปฟ# ML-Agents ์šฉ ๋„์ปค ์‚ฌ์šฉ๋ฒ•
๋„์ปค๋ฅผ ์‚ฌ์šฉํ•ด ์ถ”๋ก ๊ณผ ํ•™์Šต์„ ํ•˜๊ณ ์žํ•˜๋Š” Windows์™€ Mac ์‚ฌ์šฉ์ž๋ฅผ ์œ„ํ•œ ์†”๋ฃจ์…˜์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.
์ด๊ฒƒ์€ Python๊ณผ TensorFlow ์„ค์น˜๋ฅผ ํ”ผํ•˜๊ณ ์ž ํ•˜๋Š” ๋ถ„์—๊ฒŒ ๋งค๋ ฅ์ ์ธ ์˜ต์…˜์ด ๋  ๊ฒƒ์ž…๋‹ˆ๋‹ค. ํ˜„์žฌ ์„ค์ •์€ TensorFlow์™€ Unity๊ฐ€ _CPU๋ฅผ ํ†ตํ•ด์„œ๋งŒ_
๊ณ„์‚ฐํ•˜๋„๋ก ํ•ฉ๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ ๋„์ปค ์‹œ๋ฎฌ๋ ˆ์ด์…˜์€ GPU๋ฅผ ์‚ฌ์šฉํ•˜์ง€ ์•Š๊ณ  ์‹œ๊ฐ์  ๋ Œ๋”๋ง์„ ์œ„ํ•ด [`Xvfb`](https://en.wikipedia.org/wiki/Xvfb)๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.
`Xvfb`๋Š” `ML-Agents`(๋˜๋Š” ๋‹ค๋ฅธ ์‘์šฉ ํ”„๋กœ๊ทธ๋žจ)๊ฐ€ ๊ฐ€์ƒ์œผ๋กœ ๋ Œ๋”๋ง์„ ํ•  ์ˆ˜ ์žˆ๊ฒŒํ•˜๋Š” ์œ ํ‹ธ๋ฆฌํ‹ฐ ์ž…๋‹ˆ๋‹ค. ์ฆ‰, `ML-Agents`๋ฅผ ์‹คํ–‰ํ•˜๋Š” ๊ธฐ๊ณ„๊ฐ€ GPU๋ฅผ ๊ฐ€์ง€๊ณ  ์žˆ๊ฑฐ๋‚˜
๋””์Šคํ”Œ๋ ˆ์ด๋ฅผ ๊ฐ€์ง€๊ณ  ์žˆ๋‹ค๊ณ  ๊ฐ€์ •ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ์ด๊ฒƒ์€ ์นด๋ฉ”๋ผ ๊ธฐ๋ฐ˜์˜ ์‹œ๊ฐ์  ๊ด€์ฐฐ ์š”์†Œ๊ฐ€ ํฌํ•จ๋œ ํ™˜๊ฒฝ์€ ๋”์šฑ ๋Š๋ ค์งˆ ์ˆ˜๋„ ์žˆ์Œ์„ ์˜๋ฏธํ•ฉ๋‹ˆ๋‹ค.
## ์š”๊ตฌ์‚ฌํ•ญ
- ์œ ๋‹ˆํ‹ฐ _Linux Build Support_ ์ปดํฌ๋„ŒํŠธ
- [๋„์ปค](https://www.docker.com)
## ์„ค์น˜
- ์œ ๋‹ˆํ‹ฐ ์ธ์Šคํ†จ๋Ÿฌ๋ฅผ [๋‹ค์šด๋กœ๋“œ](https://unity3d.com/kr/get-unity/download)ํ•˜๊ณ  _Linux Build Support_ ์ปดํฌ๋„ŒํŠธ๋ฅผ ์ถ”๊ฐ€ํ•˜์‹ญ์‹œ์˜ค.
- ๋„์ปค๊ฐ€ ์„ค์น˜๋˜์–ด ์žˆ์ง€ ์•Š๋‹ค๋ฉด [๋‹ค์šด๋กœ๋“œ](https://www.docker.com/community-edition#/download)ํ•˜๊ณ  ์„ค์น˜ ํ•˜์‹ญ์‹œ์˜ค.
- ํ˜ธ์ŠคํŠธ ๋จธ์‹ ๊ณผ ๋ถ„๋ฆฌ๋œ ํ™˜๊ฒฝ์—์„œ ๋„์ปค๋ฅผ ์‹คํ–‰ํ•˜๊ธฐ ๋•Œ๋ฌธ์—, ํ˜ธ์ŠคํŠธ ๋จธ์‹ ์•ˆ์— ๋งˆ์šดํŠธ๋œ ๋””๋ ‰ํ† ๋ฆฌ๋Š” ํŠธ๋ ˆ์ด๋„ˆ ํ™˜๊ฒฝ ์„ค์ • ํŒŒ์ผ,
์œ ๋‹ˆํ‹ฐ ์‹คํ–‰ ํŒŒ์ผ, ์ปค๋ฆฌํ˜๋Ÿผ ํŒŒ์ผ๊ณผ TensorFlow ๊ทธ๋ž˜ํ”„์™€ ๊ฐ™์€ ๋ฐ์ดํ„ฐ๋ฅผ ๊ณต์œ ํ•˜๊ธฐ์œ„ํ•ด ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค.
์ด๋ฅผ ์œ„ํ•ด, ํŽธ์˜์ƒ ๋น„์–ด์žˆ๋Š” `unity-volume` ๋””๋ ‰ํ† ๋ฆฌ๋ฅผ ์ €์žฅ์†Œ์˜ ๋ฃจํŠธ์— ๋งŒ๋“ค์—ˆ์œผ๋‚˜, ๋‹ค๋ฅธ ๋””๋ ‰ํ† ๋ฆฌ์˜ ์‚ฌ์šฉ์€ ์ž์œ ๋กญ๊ฒŒ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
์ด ๊ฐ€์ด๋“œ์˜ ๋‚˜๋จธ์ง€ ๋ถ€๋ถ„์—์„œ๋Š” `unity-volume` ๋””๋ ‰ํ† ๋ฆฌ๊ฐ€ ์‚ฌ์šฉ๋œ๋‹ค๊ณ  ๊ฐ€์ •ํ•˜๊ณ  ์ง„ํ–‰๋ฉ๋‹ˆ๋‹ค.
## ์‚ฌ์šฉ๋ฒ•
ML-Agents ์šฉ ๋„์ปค ์‚ฌ์šฉ์—๋Š” ์„ธ ๋‹จ๊ณ„๊ฐ€ ํฌํ•จ๋ฉ๋‹ˆ๋‹ค.: ํŠน์ • ํ”Œ๋ž˜๊ทธ๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ์œ ๋‹ˆํ‹ฐ ํ™˜๊ฒฝ ๋นŒ๋“œ, ๋„์ปค ์ปจํ…Œ์ด๋„ˆ ๋นŒ๋“œ
๋งˆ์ง€๋ง‰์œผ๋กœ, ์ปจํ…Œ์ด๋„ˆ ์‹คํ–‰. ๋งŒ์•ฝ ML-Agents ์šฉ ์œ ๋‹ˆํ‹ฐ ํ™˜๊ฒฝ ๋นŒ๋“œ์— ์ต์ˆ™ํ•˜์ง€ ์•Š๋‹ค๋ฉด, [3D ๋ฐธ๋Ÿฐ์Šค ๋ณผ ์˜ˆ์ œ์™€ ํ•จ๊ป˜ ์‹œ์ž‘ํ•˜๊ธฐ](Getting-Started-with-Balance-Ball.md) ๊ฐ€์ด๋“œ๋ฅผ ๋จผ์ € ์ฝ์œผ์‹ญ์‹œ์˜ค.
### ํ™˜๊ฒฝ ๋นŒ๋“œ (์˜ต์…˜)
_ํ•™์Šต์„ ์œ„ํ•ด ์—๋””ํ„ฐ ์‚ฌ์šฉ์„ ์›ํ•œ๋‹ค๋ฉด ์ด ๋‹จ๊ณ„๋ฅผ ๊ฑด๋„ˆ๋›ธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค._
๋„์ปค๋Š” ์ผ๋ฐ˜์ ์œผ๋กœ ํ˜ธ์ŠคํŠธ ๋จธ์‹ ๊ณผ (๋ฆฌ๋ˆ…์Šค) ์ปค๋„์„ ๊ณต์œ ํ•˜๋Š” ์ปจํ…Œ์ด๋„ˆ๋ฅผ ์‹คํ–‰ํ•˜๊ธฐ ๋•Œ๋ฌธ์—,
์œ ๋‹ˆํ‹ฐ ํ™˜๊ฒฝ์€ ๋ฆฌ๋ˆ…์Šค ํ”Œ๋žซํผ์ด ๊ตฌ์ถ•๋˜์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์œ ๋‹ˆํ‹ฐ ํ™˜๊ฒฝ์„ ๋นŒ๋“œํ•  ๋•Œ, ๋นŒ๋“œ ์„ธํŒ… ์ฐฝ(Build Settings window)์—์„œ
๋‹ค์Œ ์˜ต์…˜์„ ์„ ํƒํ•ด ์ฃผ์‹ญ์‹œ์˜ค:
- ํƒ€๊ฒŸ ํ”Œ๋žซํผ์„ `๋ฆฌ๋ˆ…์Šค`๋กœ ์„ค์ • (Set the _Target Platform_ to `Linux`)
- _์•„ํ‚คํ…์ฒ˜_๋ฅผ `x86_64'๋กœ ์„ค์ • (Set the _Architecture_ to `x86_64`)
- ํ™˜๊ฒฝ์—์„œ ์‹œ๊ฐ์ ์ธ ๊ด€์ฐฐ์„ ํ•„์š”๋กœ ํ•˜์ง€์•Š๋Š”๋‹ค๋ฉด, `headless` ์˜ต์…˜์„ ์„ ํƒํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค (์•„๋ž˜ ์‚ฌ์ง„ ์ฐธ์กฐ).
`๋นŒ๋“œ` (Build)๋ฅผ ํด๋ฆญํ•˜๊ณ , ํ™˜๊ฒฝ ์ด๋ฆ„์„ ์„ ํƒํ•˜๊ณ  (์˜ˆ์‹œ: `3DBall`) ์ถœ๋ ฅ ๋””๋ ˆํ† ๋ฆฌ๋ฅผ `unity-volume`์œผ๋กœ ์„ค์ •ํ•˜์‹ญ์‹œ์˜ค.
๋นŒ๋“œ ํ›„์—, ํŒŒ์ผ `<ํ™˜๊ฒฝ ์ด๋ฆ„>.x86_64` ์™€ ํ•˜์œ„๋””๋ ‰ํ† ๋ฆฌ `<ํ™˜๊ฒฝ ์ด๋ฆ„>_Data/` ๊ฐ€ `unity-volume` ์— ์ƒ์„ฑ ๋˜์–ด์žˆ๋Š”์ง€ ํ™•์ธํ•˜์‹ญ์‹œ์˜ค.
![๋„์ปค๋ฅผ ์œ„ํ•œ ๋นŒ๋“œ ์„ค์ •](images/docker_build_settings.png)
### ๋„์ปค ์ปจํ…Œ์ด๋„ˆ ๋นŒ๋“œ
์ฒซ ๋ฒˆ์งธ, ๋„์ปค ๋จธ์‹ ์ด ์‹œ์Šคํ…œ์—์„œ ์ž‘๋™ํ•˜๋Š”์ง€ ํ™•์ธํ•˜์‹ญ์‹œ์˜ค. ์ €์žฅ์†Œ์˜ ์ตœ์ƒ๋‹จ์—์„œ ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ํ˜ธ์ถœํ•˜์—ฌ
๋„์ปค ์ปจํ…Œ์ด๋„ˆ๋ฅผ ๋นŒ๋“œํ•˜์‹ญ์‹œ์˜ค:
```sh
docker build -t <image-name> .
```
`<image-name>`์„ ๋„์ปค ์ด๋ฏธ์ง€ ์ด๋ฆ„์œผ๋กœ ๋ฐ”๊พธ์‹ญ์‹œ์˜ค, ์˜ˆ์‹œ: `balance.ball.v0.1`.
### ๋„์ปค ์ปจํ…Œ์ด๋„ˆ ์‹คํ–‰
์ €์žฅ์†Œ์˜ ์ตœ์ƒ๋‹จ์—์„œ ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ํ˜ธ์ถœํ•˜์—ฌ ๋„์ปค ์ปจํ…Œ์ด๋„ˆ๋ฅผ ์‹คํ–‰ํ•˜์‹ญ์‹œ์˜ค:
```sh
docker run --name <container-name> \
--mount type=bind,source="$(pwd)"/unity-volume,target=/unity-volume \
-p 5005:5005 \
<image-name>:latest \
--docker-target-name=unity-volume \
<trainer-config-file> \
--env=<environment-name> \
--train \
--run-id=<run-id>
```
์ธ์ˆ˜(argument) ๊ฐ’ ์ •๋ณด:
- `<container-name>` ์€ ์ปจํ…Œ์ด๋„ˆ๋ฅผ ๊ตฌ๋ถ„ํ•˜๊ธฐ์œ„ํ•ด ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค (์ปจํ…Œ์ด๋„ˆ๋ฅผ ์ธํ„ฐ๋ŸฝํŠธํ•˜๊ฑฐ๋‚˜ ์ข…๋ฃŒ์‹œํ‚ฌ ๋•Œ).
์ด๊ฒƒ์€ ์„ ํƒ์‚ฌํ•ญ์ด๋ฉฐ ์„ค์ •ํ•˜์ง€ ์•Š์•˜์„ ๊ฒฝ์šฐ ๋„์ปค๋Š” ๋žœ๋คํ•œ ์ด๋ฆ„์„ ์ƒ์„ฑํ•ฉ๋‹ˆ๋‹ค. _๋„์ปค ์ด๋ฏธ์ง€๋ฅผ ์‹คํ–‰ํ•  ๋•Œ๋งˆ๋‹ค
๊ณ ์œ ํ•œ ์ด๋ฆ„์„ ๊ฐ€์ ธ์•ผํ•จ์— ์œ ์˜ํ•˜์‹ญ์‹œ์˜ค._
- `<image-name>` ์ปจํ…Œ์ด๋„ˆ๋ฅผ ๋นŒ๋“œํ•  ๋•Œ ์‚ฌ์šฉํ•  image name์„ ์ฐธ์กฐํ•ฉ๋‹ˆ๋‹ค.
- `<environment-name>` __(์˜ต์…˜)__: ๋ฆฌ๋ˆ…์Šค ์‹คํ–‰ํŒŒ์ผ๊ณผ ํ•จ๊ป˜ ํ•™์Šต์„ ํ•  ๊ฒฝ์šฐ, ์ธ์ˆ˜ ๊ฐ’์ด ์‹คํ–‰ํŒŒ์ผ์˜ ์ด๋ฆ„์ด ๋œ๋‹ค.
์—๋””ํ„ฐ์—์„œ ํ•™์Šต์„ ํ•  ๊ฒฝ์šฐ, `<environment-name>` ์ธ์ˆ˜๋ฅผ ์ „๋‹ฌํ•˜์ง€ ๋ง๊ณ  ์œ ๋‹ˆํ‹ฐ์—์„œ _"Start training by pressing
the Play button in the Unity Editor"_ ๋ฉ”์„ธ์ง€๊ฐ€ ํ™”๋ฉด์— ํ‘œ์‹œ๋  ๋•Œ :arrow_forward: ๋ฒ„ํŠผ์„ ๋ˆ„๋ฅด์‹ญ์‹œ์˜ค.
- `source`: ์œ ๋‹ˆํ‹ฐ ์‹คํ–‰ํŒŒ์ผ์„ ์ €์žฅํ•  ํ˜ธ์ŠคํŠธ ์šด์˜์ฒด์ œ์˜ ๊ฒฝ๋กœ๋ฅผ ์ฐธ์กฐํ•ฉ๋‹ˆ๋‹ค.
- `target`: ๋„์ปค๊ฐ€`source` ๊ฒฝ๋กœ์— ์ด ์ด๋ฆ„์„ ๊ฐ€์ง„ ๋””์Šคํฌ๋กœ ๋งˆ์šดํŠธํ•˜๋„๋ก ํ•ฉ๋‹ˆ๋‹ค.
- `docker-target-name`: ML-Agents ํŒŒ์ด์ฌ ํŒจํ‚ค์ง€์—๊ฒŒ ์œ ๋‹ˆํ‹ฐ ์‹คํ–‰ํŒŒ์ผ์„ ์ฝ๊ณ  ๊ทธ๋ž˜ํ”„๋ฅผ ์ €์žฅํ•  ์ˆ˜ ์žˆ๋Š” ๋””์Šคํฌ์˜ ์ด๋ฆ„์„ ์•Œ๋ ค์ค€๋‹ค.
**๊ทธ๋Ÿฌ๋ฏ€๋กœ `target`๊ณผ ๋™์ผํ•œ ๊ฐ’์„ ๊ฐ€์ ธ์•ผ ํ•ฉ๋‹ˆ๋‹ค.**
- `trainer-config-file`, `train`, `run-id`: ML-Agents ์ธ์ž๋“ค์€ `mlagents-learn`๋กœ ์ „๋‹ฌ๋ฉ๋‹ˆ๋‹ค. ํŠธ๋ ˆ์ด๋„ˆ ์„ค์ • ํŒŒ์ผ์˜ ์ด๋ฆ„ `trainer-config-file`,
์•Œ๊ณ ๋ฆฌ์ฆ˜์„ ํ•™์Šตํ•˜๋Š” `train`, ๊ทธ๋ฆฌ๊ณ  ๊ฐ ์‹คํ—˜์— ๊ณ ์œ ํ•œ ์‹๋ณ„์ž๋ฅผ ํƒœ๊น…ํ•˜๋Š”๋ฐ ์‚ฌ์šฉ๋˜๋Š” `run-id`.
์ปจํ…Œ์ด๋„ˆ๊ฐ€ ํŒŒ์ผ์— ์ ‘๊ทผํ•  ์ˆ˜ ์žˆ๋„๋ก trainer-config ํŒŒ์ผ์„ `unity-volume` ์•ˆ์— ๋‘˜ ๊ฒƒ์„ ๊ถŒ์žฅํ•ฉ๋‹ˆ๋‹ค.
`3DBall` ํ™˜๊ฒฝ ์‹คํ–‰ํŒŒ์ผ์„ ํ•™์Šตํ•˜๊ธฐ ์œ„ํ•ด ๋‹ค์Œ ๋ช…๋ น์–ด๊ฐ€ ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค:
```sh
docker run --name 3DBallContainer.first.trial \
--mount type=bind,source="$(pwd)"/unity-volume,target=/unity-volume \
-p 5005:5005 \
balance.ball.v0.1:latest 3DBall \
--docker-target-name=unity-volume \
trainer_config.yaml \
--env=3DBall
--train \
--run-id=3dball_first_trial
```
๋„์ปค ๋งˆ์šดํŠธ์— ๋Œ€ํ•œ ์„ธ๋ถ€ ์‚ฌํ•ญ์€ ๋„์ปค์˜ [์ด ๋ฌธ์„œ](https://docs.docker.com/storage/bind-mounts/)๋ฅผ ์ฐธ๊ณ ํ•ด ์ฃผ์‹ญ์‹œ์˜ค.
**์ฐธ๊ณ ** ๋„์ปค๋ฅผ ์‚ฌ์šฉํ•ด ์‹œ๊ฐ์ ์ธ ๊ด€์ฐฐ์„ ํฌํ•จํ•œ ํ™˜๊ฒฝ์„ ํ•™์Šตํ•  ๊ฒฝ์šฐ, ์ฝ˜ํ…Œ์ด๋„ˆ๋ฅผ ์œ„ํ•ด ํ• ๋‹นํ•œ ๋„์ปค์˜ ๋””ํดํŠธ ๋ฉ”๋ชจ๋ฆฌ๋ฅผ ๋Š˜๋ ค์•ผํ•  ๊ฒƒ์ž…๋‹ˆ๋‹ค.
์˜ˆ๋ฅผ ๋“ค์–ด, [์—ฌ๊ธฐ](https://docs.docker.com/docker-for-mac/#advanced) Mac ์‚ฌ์šฉ์ž๋ฅผ ์œ„ํ•œ ๋„์ปค ์ง€์‹œ์‚ฌํ•ญ์„ ๋ด์ฃผ์‹ญ์‹œ์˜ค.
### ์ปจํ…Œ์ด๋„ˆ ์ค‘์ง€ ๋ฐ ์ƒํƒœ ์ €์žฅ
ํ•™์Šต ์ง„ํ–‰ ์ƒํ™ฉ์— ๋งŒ์กฑํ–ˆ์„ ๊ฒฝ์šฐ, ์ƒํƒœ๋ฅผ ์ €์žฅํ•˜๋Š” ๋™์•ˆ `Ctrl+C` or `โŒ˜+C` (Mac) ํ‚ค๋ฅผ ์‚ฌ์šฉํ•˜๊ฑฐ๋‚˜ ๋‹ค์Œ ๋ช…๋ น์–ด๋ฅผ ํ†ตํ•ด ๋„์ปค ์ปจํ…Œ์ด๋„ˆ๋ฅผ ์ค‘์ง€ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:
```sh
docker kill --signal=SIGINT <container-name>
```
`<container-name>` ์€ `docker run` ๋ช…๋ น์–ด์— ์ง€์ •๋œ ์ปจํ…Œ์ด๋„ˆ ์ด๋ฆ„์ž…๋‹ˆ๋‹ค. ์ง€์ •ํ•˜์ง€ ์•Š์œผ๋ฉด ๋ฌด์ž‘์œ„๋กœ ์ƒ์„ฑ๋˜๋ฉฐ`docker container ls`๋ฅผ ํ†ตํ•ด ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
## ํ•œ๊ธ€ ๋ฒˆ์—ญ
ํ•ด๋‹น ๋ฌธ์„œ์˜ ํ•œ๊ธ€ ๋ฒˆ์—ญ์€ [์žฅํ˜„์ค€ (Hyeonjun Jang)]([https://github.com/janghyeonjun](https://github.com/janghyeonjun))์— ์˜ํ•ด ์ง„ํ–‰๋˜์—ˆ์Šต๋‹ˆ๋‹ค. ๋‚ด์šฉ์ƒ ์˜ค๋ฅ˜๋‚˜ ์˜คํƒˆ์ž๊ฐ€ ์žˆ๋Š” ๊ฒฝ์šฐ totok682@naver.com ์œผ๋กœ ์—ฐ๋ฝ์ฃผ์‹œ๋ฉด ๊ฐ์‚ฌ๋“œ๋ฆฌ๊ฒ ์Šต๋‹ˆ๋‹ค.