How to optimize multicore encoding

Issues involving a Windows version of FFmpeg. Do not post general usage questions here.

Re: How to optimize multicore encoding

Postby rogerdpack » Sun Aug 05, 2012 4:55 am

Thanks for doing those.
I actually wasn't sugesting to use
(http://ffmpeg.zeranoe.com/builds/win32/ ... -static.7z), but to use

http://x32.elijst.nl/FFmpeg-20120622.7z

which are compiled with a different threading library.
Actually, I've had some good results using newer builds. I'll get you yet another one to try on Monday :P
-roger-
rogerdpack
 
Posts: 1454
Joined: Fri Aug 05, 2011 9:56 pm

Re: How to optimize multicore encoding

Postby rogerdpack » Mon Aug 06, 2012 3:25 pm

Ok here's a latest build from my script [1]:
http://rogerdpack.t28.net/incoming/ffmp ... 2a1f52b.7z

it should be similar to the elijist.nl one but it's a little newer build so who knows might make a difference.
GL.
-r
rogerdpack
 
Posts: 1454
Joined: Fri Aug 05, 2011 9:56 pm

Re: How to optimize multicore encoding

Postby marcjn » Wed Aug 08, 2012 4:07 pm

Somehow, the link http://rogerdpack.t28.net/incoming/ffmp%20...%202a1f52b.7z doesn't work. Do you have another one?
Thanks
marcjn
 
Posts: 43
Joined: Tue Jun 21, 2011 12:36 am

Re: How to optimize multicore encoding

Postby rogerdpack » Wed Aug 08, 2012 4:17 pm

oops I moved it
here's a later version, anyway:
http://rogerdpack.t28.net/incoming/ffmp ... 449a446.7z
rogerdpack
 
Posts: 1454
Joined: Fri Aug 05, 2011 9:56 pm

Re: How to optimize multicore encoding

Postby marcjn » Wed Aug 08, 2012 6:27 pm

Thanks, I got the file.

I'll be glad to try your new build, in the mean time, here are some results from Zeranoe builds.

These were executed on a Win 7 HP laptop with a CPU Intel Core2 Quad Q9000 @ 2.00GHz. The clip had 1474 frames at 23.976 fps, with an avisynth script feeding ffmpeg, primarily to facilitate the control of the video quality by running ssim to adjust bitrate for xvid avi and crf for x264 mp4 encoding. With this simple avisynth script, I didn't notice any impact of avisynth on the ffmpeg performance. That could be very different with heavy processing in avisynth, prior to ffmpeg encoding.

I first used my "old" ffmpeg build (version N-36193-gf514695, built on Dec 26 2011 17:50:37 with gcc 4.6.2) and then a more recent one (ffmpeg version N-41843-gc17808c, built on Jun 22 2012 17:40:07 with gcc 4.7.1). Encoding parameters were adjusted for an SSIM quality of around 84-85 for the various codecs. I used only one pass.

In the 12/26/11 version, the XVID encoding with 'mpeg4' codec is the fastest, with a medium CPU usage, although all processors are not working at the same load. The threads option should be set at 4 or 6 and CPU utilizion then rises to ~67%.
XVID encoding with the libxvid codec is very slow (~20% of the encoding speed of xvid mpeg4 or x264 mp4). The '-threads' parameter is critical, and should be set at 4 or 6 for maximum speed. Quality (measured by SSIM) is slightly better than the mpeg4 codec.
X264 encoding is generally as fast as XVID/mpeg4, with a better compression capability for a given SSIM (as expected). All processors are evenly utilized. Threads should be set at 0 or 6 for best speed, but then CPU usage gets into the 90%+ range.
ffmpeg-N36193.jpg
From Zeranoe's static build dated 20111226

The more recent ffmpeg version (6/22/2012) produced very similar results for X.264 and XVI/mpeg4, but XVID encoding with libxvid showed markingly different results. Encoding speed was much higher: around 22 fps vs. 6-12 fps, very low CPU usage (consistently around 26%). The load is not well distributed among the four processors and the '-threads' option has no effect whatsoever.
ffmpeg-N41843.jpg
From Zeranoe's static build dated 20120622

Let me know if something is not clear.
marcjn
 
Posts: 43
Joined: Tue Jun 21, 2011 12:36 am

Re: How to optimize multicore encoding

Postby marcjn » Wed Aug 08, 2012 8:01 pm

Just noticed that your last version doesn't have the libxvid codec.
marcjn
 
Posts: 43
Joined: Tue Jun 21, 2011 12:36 am

Re: How to optimize multicore encoding

Postby rogerdpack » Wed Aug 08, 2012 9:29 pm

The latest seem to have xvid:

ffmpeg-20120804-git-f857465-win32-static\bin\ffmpeg.exe" -codecs | grep xvid
EV libxvid libxvidcore MPEG-4 part 2

What clip were you using? a movie? anime?

What code is XVID encoding with 'mpeg4' codec? Is it libxvid or mpeg4? What command lines did you use?

(also as a note, around dec 25 the default threads became "-threads 0" which means autodetect, which, I think, means 1.5x the number of cores. It might also be interesting to try it with -threads 1 since I've had it still use all my cores with that, for some reason, with libx264--it must be using threads underneath).

I also wonder what happened with libxvid that it appears to use just 1 core but is now faster? Weird. Might want to ask the ffmpeg-dev about that one.

Also in your graphs, I assume "processors used" is when run with -threads 6? Also if that is the specific processor numbers, I'd probably just say "2" instead of "1,4" or do I misunderstand that column?

You may also want to try it with slice-based threads in there, as well.

Thank you for the results. I guess one interesting thiing (to me) is that the mpeg4 codec takes like half the cpu, but is almost as fast (fps) wise, and produces reasonably sized files. I wonder why it refuses to use more than 2 cores...If it could use all 4 then it appears like it would be much faster than libx264.

You might also want to experiment with different libx264 preset/tune/profiles and see if one fits the bill better for you.

Thanks much.
-roger-
rogerdpack
 
Posts: 1454
Joined: Fri Aug 05, 2011 9:56 pm

Re: How to optimize multicore encoding

Postby marcjn » Thu Aug 09, 2012 12:33 am

I am using a movie clip recorded from TV, and processed via an avisynth script rmm.avs (making it 23.976 fps progressive, cropping, and some denoising):
Code: Select all
video="C:/Users/JN_normal/Documents/_Videos/_Video_logs/mm-tests/check3/rmm.d2v"
MPEG2Source(video)
tfm(flags=3, d2v=video)
tdecimate()
crop (4, 102, 712, 272)
Removegrain(mode=2)
vid = last
audio = NicMPG123Source("rmm.mp2")
Audiodub(vid, audio)
Here is the ffmpeg command I am using for XVID/mpeg4:
Code: Select all
ffmpeg -i rmm.avs -threads 4 -vcodec mpeg4 -vtag XVID -b:v 883K -aspect 2.326 -g 240 -trellis 2 -mbd rd -flags +mv4+aic -y -f avi -an  rmm-1ff.avi
For XVID/libxvid, I just replace the mpeg4 codec
Code: Select all
ffmpeg -i rmm.avs -threads 4 -vcodec libxvid -vtag XVID -b:v 883K -aspect 2.326 -g 240 -trellis 2 -mbd rd -flags +mv4+aic -y -f avi -an  rmm-1ff.avi
For x.264:
Code: Select all
ffmpeg -i rmm.avs -threads 2 -c:v libx264 -profile:v main -preset:v medium -level 3.1 -x264opts crf=21.241  -aspect 2.326 -y -f mp4  -an rmm-1ff.mp4

'Processors used' is just an approximative description of the Windows Task Manager Performance diagram, when one or two processors are showing higher usage than the others. It describes the diagram for each of the 'threads n' configurations. When I put '3, 2', it means that processors #2 and #3 are working more than the others. When it is just 2 (diagram below), it means that processor #2 seems to be the only one working.
Win Task Mgr.jpg
This is somewhat subjective, but it always means that there is no even load distribution among processors. When that happens, I indicate 'All' in that column. Also, it seems that the same ffmpeg command doesn't always load the same processor(s).

I agree with your comments on the mpeg4 codec. It is still not as good as x.264, but it looks like if it could use all the processors better, it would be a winner.
As far as experimenting with different libx264 presets, I had done that, but it looks like I have to redo the work, as the old ffmpeg presets don't work any longer, and I have to play with the x.264 presets now.

More results with your build to come.
P.S-1. How can I use ' slice-based threads' ?
P.S-2. I assumed the build didn't support libxvid because it rejected my libxvid command above, and libxvid is not listed in 'ffmpeg -codecs'. But could it be hidden somewhere?
Thanks.
marcjn
 
Posts: 43
Joined: Tue Jun 21, 2011 12:36 am

Re: How to optimize multicore encoding

Postby rogerdpack » Thu Aug 09, 2012 12:59 am

slices:
http://ffmpeg.org/ffmpeg.html
-slices
and
-thread_type (I had to look it up, I've never used it...)
You can see which codecs support sliced threading with $ffmpeg -codecs
in later versions --one of the columns is sliced.

libxvid:

$ ffmpeg-20120730-git-a85b4a5-win32-static\bin\ffmpeg.exe"
-codecs | grep libxvid
...
EV libxvid libxvidcore MPEG-4 part 2

I wouldn't suggest using builds from August (yet) as they seem to have a broken pthreads...actually I would almost suggest just laying low and waiting until the threading issue is resolved otherwise your results might still be inaccurate... :)

Also throw libvpx in the mix why not? :P
-r
rogerdpack
 
Posts: 1454
Joined: Fri Aug 05, 2011 9:56 pm

Re: How to optimize multicore encoding

Postby marcjn » Thu Aug 09, 2012 3:29 am

Thanks for the information on slices. I checked the list of codecs, and realized that mpeg4 supports both frame-based and slice-based multi-threaded encoding and decoding :P (while libx264 and libvpx support neither). I just need to try it.

The archive I used from your link is ffmpeg-distro-static-3865ec2acea976da695b9d4c8375d1af1449a446.7z. It contains only the following directories: distros > ffmpeg-distro-static-3865ec2acea976da695b9d4c8375d1af1449a446 and just two files: ffmpeg-32.exe and ffmpeg-x86_64.exe.
I used ffmpeg-32, and the list of codecs from 'ffmpeg-32.exe -codec' doesn't contain libxvid. I wonder if we are talking about the same build :?:
In any case, not a big deal, I was looking at libxvid more as a reference than to really use it.
Last edited by marcjn on Thu Aug 09, 2012 3:37 am, edited 1 time in total.
marcjn
 
Posts: 43
Joined: Tue Jun 21, 2011 12:36 am

PreviousNext

Return to Usage Issues

Who is online

Users browsing this forum: No registered users and 3 guests