VFIMamba
VFIMamba
zgzaacm commited on
Commit
401cb04
1 Parent(s): d16fcfd

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +47 -3
README.md CHANGED
@@ -1,3 +1,47 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ ---
4
+ # VFIMamba: Video Frame Interpolation with State Space Models
5
+ This is the official checkpoint library for [VFIMamba: Video Frame Interpolation with State Space Models](https://arxiv.org/abs/2407.02315).
6
+ Please refer to [this repository](https://github.com/MCG-NJU/VFIMamba) for our code.
7
+
8
+ ## Model Description
9
+ VFIMamba is the first approach to adapt the SSM model to the video frame interpolation task.
10
+ 1. We devise the Mixed-SSM Block (MSB) for efficient inter-frame modeling using S6.
11
+ 2. We explore various rearrangement methods to convert two frames into a sequence, discovering that interleaved rearrangement is more suitable for VFI tasks.
12
+ 3. We propose a curriculum learning strategy to further leverage the potential of the S6 model.
13
+
14
+ Experimental results demonstrate that VFIMamba achieves the state-of-the-art performance across various datasets, in particular highlighting the potential of the SSM model for VFI tasks with high resolution.
15
+
16
+ ## Usage
17
+ We provide two models, an efficient version (VFIMamba-S) and a stronger one (VFIMamba). You can choose what you need by specifying the parameter model.
18
+
19
+ ### Manually Load
20
+ Please refer to [the instruction here](https://github.com/MCG-NJU/VFIMamba/tree/main?tab=readme-ov-file#sunglassesplay-with-demos) for manually loading the checkpoints and a more customized experience.
21
+ ```bash
22
+ python demo_2x.py --model **model[VFIMamba_S/VFIMamba]** # for 2x interpolation
23
+ python demo_Nx.py --n 8 --model **model[VFIMamba_S/VFIMamba]** # for 8x interpolation
24
+ ```
25
+
26
+
27
+ ### Hugging Face Demo
28
+ For Hugging Face demo, please refer to [the code here](https://github.com/MCG-NJU/VFIMamba/blob/main/hf_demo_2x.py).
29
+ ```bash
30
+ python hf_demo_2x.py --model **model[VFIMamba_S/VFIMamba]** # for 2x interpolation
31
+ ```
32
+
33
+
34
+ ## Citation
35
+ If you think this project is helpful in your research or for application, please feel free to leave a star⭐️ and cite our paper:
36
+ ```
37
+ @misc{zhang2024vfimambavideoframeinterpolation,
38
+ title={VFIMamba: Video Frame Interpolation with State Space Models},
39
+ author={Guozhen Zhang and Chunxu Liu and Yutao Cui and Xiaotong Zhao and Kai Ma and Limin Wang},
40
+ year={2024},
41
+ eprint={2407.02315},
42
+ archivePrefix={arXiv},
43
+ primaryClass={cs.CV},
44
+ url={https://arxiv.org/abs/2407.02315},
45
+ }
46
+ ```
47
+