一站式视频生成工具MotionAgent

 阿里巴巴达摩院旗下魔搭社区（ModelScope）宣布开源一站式视频生成工具MotionAgent。

MotionAgent 集成了大语言模型、文本生成图片模型、视频生成模型和音乐生成模型。

使用 MotionAgent 可以通过输入创意和描述，让 AI 帮助生成剧本、剧照、视频和音乐。

其中，通义千问 -7B 模型可以帮助生成高质量的剧本内容，文本生成图片模型可以生成剧照，而核心模型 I2VGen-XL 可以将剧照转化为视频，最后音乐生成模型可以生成想要的音乐。

通过 MotionAgent，用户可以将自己的创意轻松变成现实。

项目地址：https://github.com/modelscope/motionagent

介绍

MotionAgent是一个能将用户创造的剧本生成视频的深度学习模型工具。用户通过我们提供的工具组合，进行剧本创作、剧照生成、图片/视频生成、背景音乐谱写等工作。

MotionAgent的模型由ModelScope开源模型社区提供支持。

功能特性

• 剧本生成（Script Generation）
  • 用户指定故事主题和背景，即可生成剧本
  • 剧本生成模型基于LLM（如Qwen-7B-Chat），可生成多种风格的剧本
• 剧照生成（Movie still Generation）
  • 通过输入一幕剧本，即可生成对应的剧照场景图片
• 视频生成（Video Generation）
  • 图生视频
  • 支持高分辨率视频生成
• 音乐生成（Music Generation）
  • 自定义风格的背景音乐

  -----------------------------------

  MotionAgent is your AI assistent to convert ideas into motion pictures.  

  如果您熟悉中文，可以阅读中文版本的README。

  MotionAgent is a deep learning model tool that can generate videos from user-created scripts. Users can create scripts, generate movie stills, generate images/videos, and compose background music through our provided toolset.

  The model of MotionAgent is powered by the open-source model community ModelScope.

  Features

• Script Generation
  • Users can generate scripts by specifying the story theme and background
  • The script generation model is based on LLM (such as Qwen-7B-Chat), which can generate scripts of various styles
• Movie still Generation
  • Generate corresponding movie still scene images
• Video Generation
  • Generate videos from images
  • Support high-resolution video generation
• Music Generation
  • Custom style background music

Quick Start

Compatibility Verification

Verified environments:

• python3.8
• torch2.0.1
• CUDA11.7
• OS: Ubuntu 20.04
• Nvidia-A100 40G

Resource Requirements

• GPU memory: 36GB
• Disk: It is recommended to reserve more than 50GB of storage space

Installation Guide

conda virtual environment

Use the conda virtual environment, refer to Anaconda to manage your dependencies, after installation, execute the following commands:

conda create -n motion_agent python=3.8
conda activate motion_agent

GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/modelscope/motionagent.git --depth 1
cd motionagent

# Install dependencies
pip3 install -r requirements.txt

# Run the application
python3 app.py

# Note: MotionAgent currently supports single-card GPU, if your environment has multiple cards, please use the following command
# CUDA_VISIBLE_DEVICES=0 python3 app.py
# Note: If you are using the Modelscope community Notebook or if your disk memory is less than 100GB, please turn on the clear_cache switch. Each run will result in re-downloading the model, causing a significant decrease in speed. Please be patient and wait.
# python3 app.py --clear_cache

# Finally, click on the URL generated in the log to access the page.

Model List

[1] Qwen-7B-Chat： Model | Space

[2] SDXL 1.0：Model | Space

[3] I2VGen-XL： Model | Space

[4] MusicGen： Model | Space

More Information

• ModelScope library

  ModelScope Library is a model ecosystem repository hosted on github, belonging to the Damo Academy Moda project.

• Contribute models to ModelScope

  from https://github.com/modelscope/motionagent