1

I need to generate videos that are several hours long from CSV files. I have a working solution, but it's unusably slow and am looking for alternative ideas.

What I'm currently doing:

  • using an image library to generate individual frames
  • use ffmpeg to concatenate all the frames into a video

Ffmpeg is surprisingly the bottleneck here. I can parallelize the frame generation and get ~200fps on my macbook. The concat piece, however, churns along at a blazing 4-5fps

Any ideas? The videos are just data visualization: tickers, vector maps, etc. I don't know how to cut out the middleman step of millions of .pngs

None
  • 1
  • 1
  • Perhaps you can parallelize the video generation as well? Pass groups of images to the video generating process to create partial videos, which you combine when all parts are ready. – Rik D Oct 01 '20 at 14:01
  • 1
    Have you considered using ffmpeg's library to directly generate the video? – user253751 Oct 01 '20 at 14:36
  • ffmpeg already uses multi cores, it peaks around 70% CPU usage, but optimizing that won't help too much. As far as using their library, it's... Less than user friendly to say the least. I'd basically need to build wrappers around everything just to keep my sanity, and It would be nearly undebuggable as I'd end up with 1mb text commands. I'm hoping there's another solution! – None Oct 01 '20 at 16:20
  • Huh? ffmpeg's library isn't based on text commands. It's the same library that every video player and recorder uses to play and record video. https://en.wikipedia.org/wiki/Libavcodec – user253751 Oct 01 '20 at 18:13
  • A shot in the dark, but three things: 1) what's the storage type on your MB? 2) do you have enough RAM? 3) change the compression algorithm used by ffmpeg. – jaskij Oct 01 '20 at 19:18
  • Are all the millions of images different, of do they contain batches of exactly the same image? Like when you want to display a map for 10 seconds, do you have (10 * fps) images which are exactly the same? – Rik D Oct 01 '20 at 20:09
  • They're all slightly different, as in it's an animated gps track. So 99% the same from one frame to the next – None Oct 02 '20 at 16:34
  • Did you try something different than ffmepg? The accepted answer to this [superuser Q&A](https://superuser.com/questions/249101/how-can-i-combine-30-000-images-into-a-timelapse-movie) suggests AviDemux. – Doc Brown Oct 02 '20 at 19:52
  • Can you generate short video clips directly? If so, you could then use something like `ffmpeg -i "concat:video1.ts|video2.ts" -c copy out.ts` – James Youngman Oct 03 '20 at 22:50
  • @JamesYoungman - That's the exact same problem though as each short clip still generates painfully slow – None Oct 04 '20 at 23:10

0 Answers0