Workflow Ai Generation

Workflow — AI Generation

What: Synthesize new media with open-source, commercial-safe AI models. Strict license filter: Apache-2 / MIT / BSD / GPL. NC / research-only models are always-dropped.

Skills used

media-tts-ai, media-whisper, media-sd, media-svd, media-musicgen, media-lipsync, media-depth, media-ocr-ai, media-tag, media-demucs, media-denoise-ai.

Tool matrix

TTS (media-tts-ai)

Model	License	Best for
Kokoro	Apache-2	general default
OpenVoice	MIT	voice cloning
CosyVoice	Apache-2	Chinese
Chatterbox	MIT	expressive
Piper	MIT	embedded / offline
StyleTTS2	MIT	single-voice quality
Parler	Apache-2	style prompting
Bark	MIT	creative / SFX
Orpheus	Apache-2	modern expressive

DROPPED: XTTS-v2 (CPML NC), F5-TTS (research).

Image (media-sd)

Model	License	Best for
FLUX-schnell	Apache-2	default (4-step distilled)
Kolors	Apache-2	bilingual EN/ZH
Sana	Apache-2	fast 4K
ComfyUI	GPL-3	node-graph workflows

DROPPED: FLUX-dev (NC), SDXL / SD3 base (restrictive).

Video (media-svd)

Model	License	Best for
LTX-Video	Apache-2	fastest
CogVideoX	Apache-2	high quality (slower)
Mochi	Apache-2	cinematic
Wan	Apache-2	versatile

DROPPED: Stable Video Diffusion (NC research).

Music / SFX (media-musicgen)

Model	License	Best for
Riffusion	Apache-2	spectrogram-diffusion
YuE	Apache-2	long-form structured

DROPPED: Meta MusicGen (CC-BY-NC).

Lipsync (media-lipsync)

Model	License
LivePortrait	MIT
LatentSync	Apache-2

DROPPED: Wav2Lip (research), SadTalker (NC).

OCR (media-ocr-ai)

Model	License	Best for
PaddleOCR	Apache-2	Latin + Chinese default
Tesseract 5	Apache-2	fastest classical
TrOCR	MIT	handwriting
EasyOCR	Apache-2	multilingual

DROPPED: Surya (commercial restriction).

Tagging / captioning (media-tag)

CLIP (MIT), SigLIP (Apache-2), BLIP-2 (BSD), LLaVA (Apache-2 but needs Llama-2 backbone — check license).

Stem separation (media-demucs)

htdemucs 4-stem, htdemucs_6s 6-stem (adds guitar + piano).

Speech-to-text (media-whisper)

whisper.cpp (MIT fastest CPU), faster-whisper (MIT CUDA-accelerated).

Example composite workflows

Explainer video from script

script → TTS voiceover (media-tts-ai Kokoro) → slide images (media-sd FLUX-schnell) → B-roll clips (media-svd LTX-Video) → host animation (media-lipsync LivePortrait) → background music (media-musicgen Riffusion) → assemble (ffmpeg-cut-concat) → mix (ffmpeg-audio-filter + sidechain ducking) → loudness normalize (media-ffmpeg-normalize) → auto-burn subtitles (media-whisper + ffmpeg-subtitles) → transcode H.264 → YouTube upload (media-cloud-upload).

Multilingual voice clone

30 s clean reference → DeepFilterNet denoise → OpenVoice clone to EN/ES/FR/JP → per-track denoise → mux with localized subs.

Book cover + trailer

FLUX-schnell cover variations → LLaVA auto-describe for alt-text → Mochi cinematic trailer → StyleTTS2 narrator → YuE cinematic score → assemble.

Automated podcast chapter thumbnails

scenedetect chapter boundaries → extract frame per chapter → BLIP-2 caption → Kolors generate thumbnail from caption → embed chapter metadata.

Digital human with cloned voice

OpenVoice clone → LivePortrait drives portrait → FLUX-schnell branded background → chromakey + RVM matte composite → Riffusion audio bed → transcode + upload.

Gotchas

• License discipline. Always check the skill's references/LICENSES.md BEFORE adopting a new model. HuggingFace license field is NOT authoritative. Pin model weights to specific commit hashes.
• GPL-3 in ComfyUI / RVM — requires source distribution if modified/redistributed. Use dynamically / in pipeline, not bundled into proprietary product.
• All Layer 9 skills need GPU for reasonable throughput (10–50× slower on CPU).
• TTS sample rates vary: Kokoro 24 kHz, Piper 22.05 kHz, StyleTTS2 24 kHz, OpenVoice 24 kHz. Resample ALL to 48 kHz before mixing.
• Voice cloning needs CLEAN reference. Noisy sample → noisy clone. DeepFilterNet the reference first.
• FLUX-schnell is 4-step distilled. Do NOT push --steps above 4–8 — wastes compute, no quality gain.
• ComfyUI workflow JSONs pin specific node versions. Use ComfyUI Manager to install matching nodes.
• LTX-Video prompt adherence is phrasing-sensitive. "cat running" ≠ "running cat".
• CogVideoX-5b needs ~20 GB VRAM. 2b variant runs on 8 GB at lower quality.
• Riffusion produces 5.11-second clips natively. Chain with crossfade for longer, or use YuE for structured long-form.
• LivePortrait expects clean frontal portrait. Angled faces, glasses, occluded mouths degrade output.
• LivePortrait outputs 512×512 by default. Upscale with media-upscale for larger.
• Whisper large-v3 is 3 GB. Test with base.en (140 MB) first — quality gap to medium (1.5 GB) is small for clean audio.
• Whisper hallucinates on silence. Trim leading/trailing with silenceremove.
• Whisper word-level timestamps require --word_timestamps True (faster-whisper) or --max-len 1 --split-on-word (whisper.cpp).
• Whisper --language auto is fragile on accented speech. Specify language explicitly.
• Diarization is external to Whisper. Use pyannote.audio (MIT) or simple-diarizer.
• OCR selection: PaddleOCR for Latin+Chinese, TrOCR for handwriting, Tesseract for speed on clean docs.
• CLIP / SigLIP / BLIP-2 need fixed-resolution inputs (224/336/384/448). The tagctl.py script resizes internally.
• LLaVA needs an LLM backbone (Vicuna / Llama-2 7B+). Llama-2 community license has revenue caps.

Related

• workflow-ai-enhancement — for enhancing EXISTING footage (not generating new).
• workflow-podcast-pipeline — for podcast-specific AI workflows.
• workflow-analysis-quality — VMAF + QC on AI-generated output.