We're seeing a hang attempting to decode H264 video using DXVA2 (aka pixel format PIX_FMT_DXVA2_VLD). This is in the recently released Plex Media Center (http://www.plexapp.com). The only workaround we've found is to use a single thread when decoding.
It would be great if you could help us diagnose the issue. We suspect an ffmpeg bug, but it's possible that we're doing something wrong as well.
In our repro case, we call avcodec_thread_init with 8 threads, then avcodec_open, then obviously avcodec_decode_video. If we don't call avcodec_thread_init and we leave the context's default of 1 thread, then no hang occurs and decoding proceeds smoothly.
The hang itself occurs on the second or third call to avcodec_decode_video. Unfortunately, because of the difficulty of debugging ffmpeg code on Windows, I don't have good stack traces to provide. All I can see is avcodec waiting on what appear to be pthreads condition variables. None of our related threads are hung, and avcodec_decode_video is not returning, so my guess would be a deadlock in ffmpeg.
Here's a sample thread in cdb - not very useful, obviously:
24 Id: 3bc8.15e4 Suspend: 1 Teb: fff50000 Unfrozen
1681f728 76eb0bdd ntdll_779a0000!NtWaitForMultipleObjects+0x15
1681f7c4 75f71a2c KERNELBASE!WaitForMultipleObjectsEx+0x100
1681f80c 75f74208 KERNEL32!WaitForMultipleObjectsExImplementation+0xe0
1681f828 6e0c5173 KERNEL32!WaitForMultipleObjects+0x18
WARNING: Stack unwind information not available. Following frames may be wrong.
1681f858 6e0c5582 libpthread_2!pthreadCancelableTimedWait+0x47
1681f8d8 6e0c66c3 libpthread_2!sem_timedwait+0x15e
1681f928 661a1740 libpthread_2!pthread_cond_wait+0xeb
1681f974 6e0c5121 avcodec_53!av_parser_close+0x71ac
1681f9a4 76861287 libpthread_2!pthread_exit+0xf5
1681f9dc 76861328 msvcrt!_endthreadex+0x44
1681f9e4 75f7339a msvcrt!_endthreadex+0xce
1681f9f0 779d9ed2 KERNEL32!BaseThreadInitThunk+0xe
1681fa30 779d9ea5 ntdll_779a0000!__RtlUserThreadStart+0x70
1681fa48 00000000 ntdll_779a0000!_RtlUserThreadStart+0x1b
More context: we're using ffmpeg library bits built from current ffmpeg sources, using MSYS/MinGW. We have some customizations, but they are relatively minor and unrelated to this area.
The following is our configure command line:
$ cat configure-windows.sh
./configure --disable-static --disable-debug --enable-runtime-cpudetect --disable-mmx --disable-mmx2 --arch=i386 --disable-muxers --enable-muxer=spdif --enable-muxer=adts --disable-encoders --enable-encoder=ac3 --enable-encoder=aac --disable-libvorbis --disable-devices --disable-ffplay --disable-ffserver --disable-ffmpeg --enable-shared --disable-decoder=mpeg_xvmc --enable-postproc --enable-gpl --enable-protocol=http --enable-pthreads --enable-runtime-cpudetect --enable-memalign-hack --enable-dxva2 --enable-decoder=h264_dxva2 --enable-hwaccel=h264_dxva2 --extra-cflags=-Isysinc
Is there any other data I can provide that might help to debug this?