File size: 5,055 Bytes
e29a01b 5e96b6e e7b61d1 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 |
---
license: mit
language:
- en
tags:
- robotics
- motion planning
---
# Neural MP
Neural MP is a machine learning-based motion planning system for robotic manipulation tasks. It combines neural networks trained on large-scale simulated data with lightweight optimization techniques to generate efficient, collision-free trajectories. Neural MP is designed to generalize across diverse environments and obstacle configurations, making it suitable for both simulated and real-world robotic applications. This repository contains the implementation, data generation tools, and evaluation scripts for Neural MP.
All Neural MP checkpoints, as well as our [training codebase](https://github.com/mihdalal/neuralmotionplanner) are released under an MIT License.
For full details, please read our paper(coming soon) and see [our project page](https://mihdalal.github.io/neuralmotionplanner/).
## Model Summary
- **Developed by:** The Neural MP team consisting of researchers from Carnegie Mellon University.
- **Language(s) (NLP):** en
- **License:** MIT
- **Pretraining Dataset:** Coming soon
- **Repository:** [https://github.com/mihdalal/neuralmotionplanner](https://github.com/mihdalal/neuralmotionplanner)
- **Paper:** Coming soon
- **Project Page & Videos:** [https://mihdalal.github.io/neuralmotionplanner/](https://mihdalal.github.io/neuralmotionplanner/)
## Installation
Please check [here](https://github.com/mihdalal/neural_mp?tab=readme-ov-file#installation-instructions) for detailed instructions
## Usage
Neural MP model takes in 3D point cloud and start & goal angles of the Franka robot as input, and predict 7-DoF delta joint actions. We provide a wrapper class [NeuralMP](https://github.com/mihdalal/neural_mp/blob/master/neural_mp/real_utils/neural_motion_planner.py) for inference and deploy our model in the real world.
Here's an deployment example with the Manimo Franka control library:
Note: using Manimo is not required, you may use other Franka control libraries by creating a wrapper class which inherits from FrankaRealEnv (check [franka_real_env.py](https://github.com/mihdalal/neural_mp/blob/master/neural_mp/envs/franka_real_env.py))
```python
import argparse
import numpy as np
from neural_mp.envs.franka_real_env import FrankaRealEnvManimo
from neural_mp.real_utils.neural_motion_planner import NeuralMP
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--mdl_url",
type=str,
default="mihdalal/NeuralMP",
help="hugging face url to load the neural_mp model",
)
parser.add_argument(
"--cache-name",
type=str,
default="scene1_single_blcok",
help="Specify the scene cache file with pcd and rgb data",
)
parser.add_argument(
"--use-cache",
action="store_true",
help=("If set, will use pre-stored point clouds"),
)
parser.add_argument(
"--debug-combined-pcd",
action="store_true",
help=("If set, will show visualization of the combined pcd"),
)
parser.add_argument(
"--denoise-pcd",
action="store_true",
help=("If set, will apply denoising to the pcds"),
)
parser.add_argument(
"--train-mode", action="store_true", help=("If set, will eval with policy in training mode")
)
parser.add_argument(
"--tto", action="store_true", help=("If set, will apply test time optimization")
)
parser.add_argument(
"--in-hand", action="store_true", help=("If set, will enable in hand mode for eval")
)
parser.add_argument(
"--in-hand-params",
nargs="+",
type=float,
default=[0.1, 0.1, 0.1, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 1.0],
help="Specify the bounding box of the in hand object. 10 params in total [size(xyz), pos(xyz), ori(xyzw)] 3+3+4.",
)
args = parser.parse_args()
env = FrankaRealEnvManimo()
neural_mp = NeuralMP(
env=env,
model_url=args.mdl_url,
train_mode=args.train_mode,
in_hand=args.in_hand,
in_hand_params=args.in_hand_params,
visualize=True,
)
points, colors = neural_mp.get_scene_pcd(
use_cache=args.use_cache,
cache_name=args.cache_name,
debug_combined_pcd=args.debug_combined_pcd,
denoise=args.denoise_pcd,
)
# specify start and goal configurations
start_config = np.array([-0.538, 0.628, -0.061, -1.750, 0.126, 2.418, 1.610])
goal_config = np.array([1.067, 0.847, -0.591, -1.627, 0.623, 2.295, 2.580])
if args.tto:
trajectory = neural_mp.motion_plan_with_tto(
start_config=start_config,
goal_config=goal_config,
points=points,
colors=colors,
)
else:
trajectory = neural_mp.motion_plan(
start_config=start_config,
goal_config=goal_config,
points=points,
colors=colors,
)
success, joint_error = neural_mp.execute_motion_plan(trajectory, speed=0.2)
```
|