README.md

---
license: apache-2.0
---
# VFIMamba: Video Frame Interpolation with State Space Models
This is the official checkpoint library for [VFIMamba: Video Frame Interpolation with State Space Models](https://arxiv.org/abs/2407.02315). 
Please refer to [this repository](https://github.com/MCG-NJU/VFIMamba) for our code.

## Model Description
VFIMamba is the first approach to adapt the SSM model to the video frame interpolation task. 
1. We devise the Mixed-SSM Block (MSB) for efficient inter-frame modeling using S6. 
2. We explore various rearrangement methods to convert two frames into a sequence, discovering that interleaved rearrangement is more suitable for VFI tasks. 
3. We propose a curriculum learning strategy to further leverage the potential of the S6 model.

Experimental results demonstrate that VFIMamba achieves the state-of-the-art performance across various datasets, in particular highlighting the potential of the SSM model for VFI tasks with high resolution.

## Usage
We provide two models, an efficient version (VFIMamba-S) and a stronger one (VFIMamba). You can choose what you need by specifying the parameter model.

### Manually Load
Please refer to [the instruction here](https://github.com/MCG-NJU/VFIMamba/tree/main?tab=readme-ov-file#sunglassesplay-with-demos) for manually loading the checkpoints and a more customized experience. 
```bash
python demo_2x.py  --model **model[VFIMamba_S/VFIMamba]**      # for 2x interpolation
python demo_Nx.py --n 8 --model **model[VFIMamba_S/VFIMamba]** # for 8x interpolation
```


### Hugging Face Demo
For Hugging Face demo, please refer to [the code here](https://github.com/MCG-NJU/VFIMamba/blob/main/hf_demo_2x.py).
```bash
python hf_demo_2x.py --model **model[VFIMamba_S/VFIMamba]**      # for 2x interpolation
```


## Citation
If you think this project is helpful in your research or for application, please feel free to leave a star⭐️ and cite our paper:
```
@misc{zhang2024vfimambavideoframeinterpolation,
      title={VFIMamba: Video Frame Interpolation with State Space Models}, 
      author={Guozhen Zhang and Chunxu Liu and Yutao Cui and Xiaotong Zhao and Kai Ma and Limin Wang},
      year={2024},
      eprint={2407.02315},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2407.02315}, 
}
```