update

README.md

@@ -28,8 +28,9 @@ inference: false
 
 ## Model Introduction
 
-CogVideoX is an open-source video generation model that shares the same origins as [清影](https://chatglm.cn/video).
-The table below provides a list of the video generation models we currently offer, along with their basic information.
+CogVideoX is an open-source version of the video generation model originating from [QingYing](https://chatglm.cn/video).
+The table below displays the list of video generation models we currently offer, along with their foundational
+information.
 
 <table style="border-collapse: collapse; width: 100%;">
   <tr>
@@ -38,41 +39,41 @@ The table below provides a list of the video generation models we currently offe
     <th style="text-align: center;">CogVideoX-5B (Current Repository)</th>
   </tr>
   <tr>
-    <td style="text-align: center;">Model Introduction</td>
-    <td style="text-align: center;">An entry-level model with good compatibility. Low cost for running and secondary development.</td>
-    <td style="text-align: center;">A larger model with higher video generation quality and better visual effects.</td>
-  </tr>
-  <tr>
-    <td style="text-align: center;">Inference Precision</td>
-    <td style="text-align: center;">FP16, FP32<br><b>NOT support BF16</b> </td>
-    <td style="text-align: center;">BF16, FP32<br><b>NOT support FP16</b> </td>
+    <td style="text-align: center;">Model Description</td>
+    <td style="text-align: center;">Entry-level model with compatibility and low cost for running and secondary development.</td>
+    <td style="text-align: center;">Larger model with higher video generation quality and better visual effects.</td>
   </tr>
   <tr>
     <td style="text-align: center;">Inference Speed<br>(Step = 50)</td>
     <td style="text-align: center;">FP16: ~90* s</td>
-    <td style="text-align: center;">BF16: ~200* s</td>
+    <td style="text-align: center;">BF16: ~180* s</td>
+  </tr>
+  <tr>
+    <td style="text-align: center;">Inference Precision</td>
+    <td style="text-align: center;"><b>FP16*(recommended)</b>, BF16, FP32, INT8, no support for INT4</td>
+    <td style="text-align: center;"><b>BF16(recommended)</b>, FP16, FP32, INT8, no support for INT4</td>
   </tr>
   <tr>
-    <td style="text-align: center;">Single GPU Memory Consumption</td>
-    <td style="text-align: center;">18GB using <a href="https://github.com/THUDM/SwissArmyTransformer">SAT</a><br><b>12GB* using diffusers</b><br></td>
-    <td style="text-align: center;">26GB using <a href="https://github.com/THUDM/SwissArmyTransformer">SAT</a><br><b>21GB* using diffusers</b><br></td>
+    <td style="text-align: center;">Single GPU Memory Usage<br></td>
+    <td style="text-align: center;">FP16: 18GB using <a href="https://github.com/THUDM/SwissArmyTransformer">SAT</a> / <b>12.5GB* using diffusers</b><br><b>INT8: 7.8GB* using diffusers</b></td>
+    <td style="text-align: center;">BF16: 26GB using <a href="https://github.com/THUDM/SwissArmyTransformer">SAT</a> / <b>20.7GB* using diffusers</b><br><b>INT8: 11.4GB* using diffusers</b></td>
   </tr>
   <tr>
-    <td style="text-align: center;">Multi-GPU Inference Memory Consumption</td>
-    <td style="text-align: center;"><b>10GB* using diffusers</b><br></td>
-    <td style="text-align: center;"><b>15GB* using diffusers</b><br></td>
+    <td style="text-align: center;">Multi-GPU Memory Usage</td>
+    <td style="text-align: center;"><b>FP16: 10GB* using diffusers</b><br></td>
+    <td style="text-align: center;"><b>BF16: 15GB* using diffusers</b><br></td>
   </tr>
   <tr>
-    <td style="text-align: center;">Fine-Tuning Memory Consumption (Per GPU)</td>
-    <td style="text-align: center;">47 GB (bs=1, LORA)<br>61 GB (bs=2, LORA)<br>62GB (bs=1, SFT)</td>
-    <td style="text-align: center;">63 GB (bs=1, LORA)<br>80 GB (bs=2, LORA)<br>75GB (bs=1, SFT)<br></td>
+    <td style="text-align: center;">Fine-tuning Memory Usage (per GPU)</td>
+    <td style="text-align: center;">47 GB (bs=1, LORA)<br> 61 GB (bs=2, LORA)<br> 62GB (bs=1, SFT)</td>
+    <td style="text-align: center;">63 GB (bs=1, LORA)<br> 80 GB (bs=2, LORA)<br> 75GB (bs=1, SFT)<br></td>
   </tr>
   <tr>
     <td style="text-align: center;">Prompt Language</td>
     <td colspan="2" style="text-align: center;">English*</td>
   </tr>
   <tr>
-    <td style="text-align: center;">Maximum Prompt Length</td>
+    <td style="text-align: center;">Max Prompt Length</td>
     <td colspan="2" style="text-align: center;">226 Tokens</td>
   </tr>
   <tr>
@@ -81,13 +82,13 @@ The table below provides a list of the video generation models we currently offe
   </tr>
   <tr>
     <td style="text-align: center;">Frame Rate</td>
-    <td colspan="2" style="text-align: center;">8 frames per second</td>
+    <td colspan="2" style="text-align: center;">8 frames / second </td>
   </tr>
   <tr>
     <td style="text-align: center;">Video Resolution</td>
-    <td colspan="2" style="text-align: center;">720 x 480, does not support other resolutions (including fine-tuning)</td>
+    <td colspan="2" style="text-align: center;">720 * 480, no support for other resolutions (including fine-tuning)</td>
   </tr>
-  <tr>
+    <tr>
     <td style="text-align: center;">Positional Encoding</td>
     <td style="text-align: center;">3d_sincos_pos_embed</td>
     <td style="text-align: center;">3d_rope_pos_embed<br></td>
@@ -96,17 +97,21 @@ The table below provides a list of the video generation models we currently offe
 
 **Data Explanation**
 
-+ When testing with the diffusers library, the `enable_model_cpu_offload()` and `pipe.vae.enable_tiling()` options were
-  enabled. This configuration was not tested on non-**NVIDIA A100 / H100** devices, but it should generally work on all
-  **NVIDIA Ampere architecture** and above. Disabling these optimizations will significantly increase memory usage, with
-  peak usage approximately 3 times the values shown in the table.
-+ For multi-GPU inference, `enable_model_cpu_offload()` must be disabled.
-+ Inference speed tests used the above memory optimization options. Without these optimizations, inference speed
-  increases by around 10%.
-+ The model supports only English input. For other languages, translation to English is recommended during large model
-  processing.
-
-+ **Note** Using [SAT](https://github.com/THUDM/SwissArmyTransformer)  for inference and fine-tuning of SAT version
++ When testing with the diffusers library, the `enable_model_cpu_offload()` option and `pipe.vae.enable_tiling()`
+  optimization were enabled. This solution has not been tested on devices other than **NVIDIA A100 / H100**. Typically,
+  this solution is adaptable to all devices above the **NVIDIA Ampere architecture**. If the optimization is disabled,
+  memory usage will increase significantly, with peak memory being about 3 times the table value.
++ The CogVideoX-2B model was trained using `FP16` precision, so it is recommended to use `FP16` for inference.
++ For multi-GPU inference, the `enable_model_cpu_offload()` optimization needs to be disabled.
++ Using the INT8 model will lead to reduced inference speed. This is done to allow low-memory GPUs to perform inference
+  while maintaining minimal video quality loss, though the inference speed will be significantly reduced.
++ Inference speed tests also used the memory optimization mentioned above. Without memory optimization, inference speed
+  increases by approximately 10%. Only the `diffusers` version of the model supports quantization.
++ The model only supports English input; other languages can be translated to English for refinement by large models.
+
+**Note**
+
++ Using [SAT](https://github.com/THUDM/SwissArmyTransformer)  for inference and fine-tuning of SAT version
   models. Feel free to visit our GitHub for more information.
 
 ## Quick Start 🤗
@@ -119,13 +124,16 @@ optimizations and conversions to get a better experience.**
 1. Install the required dependencies
 
 ```shell
-pip install --upgrade opencv-python transformers diffusers
+# diffusers>=0.30.1
+# transformers>=0.44.0
+# accelerate>=0.33.0 (suggest install from source)
+# imageio-ffmpeg>=0.5.1
+pip install --upgrade transformers accelerate diffusers imageio-ffmpeg 
 ```
 
 2. Run the code
 
 ```python
-import gc
 import torch
 from diffusers import CogVideoXPipeline
 from diffusers.utils import export_to_video
@@ -138,11 +146,6 @@ pipe = CogVideoXPipeline.from_pretrained(
 )
 
 pipe.enable_model_cpu_offload()
-
-gc.collect()
-torch.cuda.empty_cache()
-torch.cuda.reset_accumulated_memory_stats()
-torch.cuda.reset_peak_memory_stats()
 pipe.vae.enable_tiling()
 
 video = pipe(
@@ -157,9 +160,6 @@ video = pipe(
 export_to_video(video, "output.mp4", fps=8)
 ```
 
-If the generated model appears “all green” and not viewable in the default MAC player, it is a normal phenomenon (due to
-OpenCV saving video issues). Simply use a different player to view the video.
-
 ## Explore the Model
 
 Welcome to our [github](https://github.com/THUDM/CogVideo), where you will find:
@@ -169,6 +169,7 @@ Welcome to our [github](https://github.com/THUDM/CogVideo), where you will find:
 3. Reasoning and fine-tuning of SAT version models, and even pre-release.
 4. Project update log dynamics, more interactive opportunities.
 5. CogVideoX toolchain to help you better use the model.
+6. INT8 model inference code support.
 
 ## Model License
 
@@ -184,3 +185,5 @@ This model is released under the [CogVideoX LICENSE](LICENSE).
   year={2024}
 }
 ```
+
+

README_zh.md

@@ -6,7 +6,7 @@
   </div>
   <p align="center">
   <a href="https://huggingface.co/THUDM/CogVideoX-5b/blob/main/README.md">📄 Read in English</a> | 
-  <a href="https://huggingface.co/spaces/THUDM/CogVideoX-5B">🤗 Huggingface Space</a> |  
+  <a href="https://huggingface.co/spaces/THUDM/CogVideoX-5B">🤗 Huggingface Space</a> |
   <a href="https://github.com/THUDM/CogVideo">🌐 Github </a> | 
   <a href="https://arxiv.org/pdf/2408.06072">📜 arxiv </a>
 </p>
@@ -28,21 +28,21 @@ CogVideoX是 [清影](https://chatglm.cn/video) 同源的开源版本视频生
     <td style="text-align: center;">入门级模型，兼顾兼容性。运行，二次开发成本低。</td>
     <td style="text-align: center;">视频生成质量更高，视觉效果更好的更大尺寸模型。</td>
   </tr>
-  <tr>
-    <td style="text-align: center;">推理精度</td>
-    <td style="text-align: center;">FP16, FP32<br><b>不支持 BF16</b> </td>
-    <td style="text-align: center;">BF16, FP32<br><b>不支持 FP16</b> </td>
-  </tr>
   <tr>
     <td style="text-align: center;">推理速度<br>(Step = 50)</td>
     <td style="text-align: center;">FP16: ~90* s</td>
-    <td style="text-align: center;">BF16: ~200* s</td>
+    <td style="text-align: center;">BF16: ~180* s</td>
   </tr>
   <tr>
     <td style="text-align: center;">单GPU显存消耗<br></td>
     <td style="text-align: center;">18GB using <a href="https://github.com/THUDM/SwissArmyTransformer">SAT</a><br><b>12GB* using diffusers</b><br></td>
     <td style="text-align: center;">26GB using <a href="https://github.com/THUDM/SwissArmyTransformer">SAT</a><br><b>21GB* using diffusers</b><br></td>
   </tr>
+  <tr>
+    <td style="text-align: center;">单GPU量化推理显存消耗(diffusers)<br></td>
+    <td style="text-align: center;"><b>INT8: 7.8GB* </b><br>INT4: 不支持</td>
+    <td style="text-align: center;"><b>INT8: 11.5GB* </b><br>INT4: 不支持</td>
+  </tr>
   <tr>
     <td style="text-align: center;">多GPU推理显存消耗</td>
     <td style="text-align: center;"><b>10GB* using diffusers</b><br></td>
@@ -83,10 +83,11 @@ CogVideoX是 [清影](https://chatglm.cn/video) 同源的开源版本视频生
 **数据解释**
 
 + 使用 diffusers 库进行测试时，启用了 `enable_model_cpu_offload()` 选项 和 `pipe.vae.enable_tiling()` 优化，该方案未测试在非
-   **NVIDIA A100 / H100** 外的实际显存占用，通常，该方案可以适配于所有 **NVIDIA 安培架构**
+   **NVIDIA A100 / H100** 外的设备上的实际显存 / 内存占用。通常，该方案可以适配于所有 **NVIDIA 安培架构**
    以上的设备。若关闭优化，显存占用会成倍增加，峰值显存约为表格的3倍。
 + 多GPU推理时，需要关闭 `enable_model_cpu_offload()` 优化。
-+ 推理速度测试同样采用了上述显存优化方案，不采用显存优化的情况下，推理速度提升约10%。 
++ 使用 INT8 模型会导致推理速度降低，此举是为了满足显存较低的显卡能正常推理并保持较少的视频质量损失，推理速度大幅降低。
++ 推理速度测试同样采用了上述显存优化方案，不采用显存优化的情况下，推理速度提升约10%。 只有`diffusers`版本模型支持量化。  
 + 模型仅支持英语输入，其他语言可以通过大模型润色时翻译为英语。
 
 **提醒**
@@ -102,13 +103,16 @@ CogVideoX是 [清影](https://chatglm.cn/video) 同源的开源版本视频生
 1. 安装对应的依赖
 
 ```shell
-pip install --upgrade opencv-python transformers accelerate diffusers
+# diffusers>=0.30.1
+# transformers>=0.44.0
+# accelerate>=0.33.0 (suggest install from source)
+# imageio-ffmpeg>=0.5.1
+pip install --upgrade transformers accelerate diffusers imageio-ffmpeg 
 ```
 
-2. 运行代码
+2. 运行代码 (BF16 / FP16)
 
 ```python
-import gc
 import torch
 from diffusers import CogVideoXPipeline
 from diffusers.utils import export_to_video
@@ -121,11 +125,6 @@ pipe = CogVideoXPipeline.from_pretrained(
 )
 
 pipe.enable_model_cpu_offload()
-
-gc.collect()
-torch.cuda.empty_cache()
-torch.cuda.reset_accumulated_memory_stats()
-torch.cuda.reset_peak_memory_stats()
 pipe.vae.enable_tiling()
 
 video = pipe(
@@ -140,8 +139,6 @@ video = pipe(
 export_to_video(video, "output.mp4", fps=8)
 ```
 
-如果您生成的模型在 MAC 默认播放器上表现为 "全绿" 无法正常观看，属于正常现象 (OpenCV保存视频问题)，仅需更换一个播放器观看。
-
 ## 深入研究
 
 欢迎进入我们的 [github](https://github.com/THUDM/CogVideo)，你将获得：
@@ -151,6 +148,7 @@ export_to_video(video, "output.mp4", fps=8)
 3. SAT版本模型进行推理和微调，甚至预发布。
 4. 项目更新日志动态，更多互动机会。
 5. CogVideoX 工具链，帮助您更好的使用模型。
+6. INT8 模型推理代码。
 
 ## 模型协议