Syncing Multiple Audio Streams With One Video Stream

Questions involving a Windows version of FFmpeg.
Post Reply
roderrooder
Posts: 6
Joined: Tue Jan 02, 2018 10:41 am

Syncing Multiple Audio Streams With One Video Stream

Post by roderrooder » Tue Jan 02, 2018 10:52 am

I'll try and keep this as simple as possible, and thanks for any help.

I have two PC's one that I'm gaming on, one that I'm recording on. I also have two RME audio cards (one in each respective PC) hooked up to each other to share audio between the PC's. My main display is hooked up to a displayport splitter that splits to my recording PC's capture card and I'm also routing my game audio from the sound card in my gaming PC to the capture PC's audio card.

I'm running discord on the capture PC and have my game audio, discord audio, and my personal mic all on separate input channels. I have the capture card video and the game audio channel output to the same MP4 file and then have my mic audio and the discord audio output to two separate WAV files. This way I can choose to omit the voices from the video so only game audio can be heard and or I can mix the volumes independently after recording. This way I can make sure the discord, my voice, or the game audio isn't overwhelming in the final product. I'm also taking advantage of segmenting so I can continuously record keeping the last 130 minutes of everything at any given time.

Everything seemed to be working great, until my capture card dropped a few frames over the course of 3 hours. The resulting footage had out of sync audio and video, the discord audio and my mic were also now out of sync with the game footage. After searching around for a very long time it seemed the solution was -vsync 1 but it didn't seem to work, here was my code for that:

Code: Select all

    ffmpeg -guess_layout_max 0 -y -f dshow -video_size 3440x1440 -rtbufsize 2147M -pixel_format nv12 -r 100.00 -vsync 1 ^
    -i video="Video (00 Pro Capture HDMI 4K+)":audio="Analog (1+2) (RME Fireface UC)" -map 0:0 -map 0:1 ^
    -codec:v h264_nvenc -pix_fmt nv12 -b:v 250M -maxrate 250M -bufsize 250M -b:a 320k -ac 2 ^
    -segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\PC%02d.mp4 ^
    -guess_layout_max 0 -f dshow -rtbufsize 100M -i audio="Analog (3+4) (RME Fireface UC)" -map 1:0 -b:a 320k -ac 2 ^
    -segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\Voices\Theirs\TheirsPC%02d.wav ^
    -guess_layout_max 0 -f dshow -rtbufsize 100M -i audio="Analog (5+6) (RME Fireface UC)" -map 2:0 -b:a 320k -ac 2 ^
    -segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\Voices\Mine\MinePC%02d.wav
That's all one line of code, carrots are line breaks. But after playing for a while and dropping some frames the problems persisted. I was under the impression that -vsync 1 forced the desired frame rate, duplicating frames to make sure the audio stayed in-sync. So I thought maybe I had it in the wrong place and tried this:

Code: Select all

ffmpeg -guess_layout_max 0 -y -f dshow -video_size 3440x1440 -rtbufsize 2147M -pixel_format nv12 -r 100.00 ^
-i video="Video (00 Pro Capture HDMI 4K+)":audio="Analog (1+2) (RME Fireface UC)" -vsync 1 -map 0:0,0:1 -map 0:1 ^
-codec:v h264_nvenc -pix_fmt nv12 -b:v 250M -maxrate 250M -bufsize 250M -b:a 320k -ac 2 ^
-segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\PC%02d.mp4 ^
-guess_layout_max 0 -f dshow -rtbufsize 100M -i audio="Analog (3+4) (RME Fireface UC)" -map 1:0 -b:a 320k -ac 2 ^
-segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\Voices\Theirs\TheirsPC%02d.wav ^
-guess_layout_max 0 -f dshow -rtbufsize 100M -i audio="Analog (5+6) (RME Fireface UC)" -map 2:0 -b:a 320k -ac 2 ^
-segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\Voices\Mine\MinePC%02d.wav
I actually have yet to drop a frame since then... is this my code causing this? With vsync working correctly would it no-longer display dropped frames in the console? Am I mapping correctly in conjunction with vsync? It almost seems like -map 0:0,0:1 would be syncing the audio to the video but what I would really like is if I drop frames just take the last successful frame and duplicate it. That way (theoretically) the other two audio channels would stay in-sync too... I'm just trying to make sure that I don't screw up an important recording before it happens.

Thanks for any help.

roderrooder
Posts: 6
Joined: Tue Jan 02, 2018 10:41 am

Re: Syncing Multiple Audio Streams With One Video Stream

Post by roderrooder » Tue Jan 02, 2018 5:53 pm

Quick clarification, my frames are being dropped by the capture card do to an "overloaded buffer" but even when drastically lowering the bitrate I still get the error every few hours and drop frames. Sounds like -vsync can't fix frames dropped by the capture card... Just a quick update after reading for a few more hours.

Honestly open to any kind of way to sync all this up.

roderrooder
Posts: 6
Joined: Tue Jan 02, 2018 10:41 am

Re: Syncing Multiple Audio Streams With One Video Stream

Post by roderrooder » Fri Jan 12, 2018 9:50 pm

Just thought I'd let you guys know that I got this working.

I needed to use -framerate instead of -r to get it to duplicate dropped frames, the reason I wasn't using -framerate in the first place is because it would spam the message "past duration too large" and the video wouldn't play back after exiting the recording. I also used an audio filter to delay the other channels so they they would start at the the same time as the video. I had to use -af "adelay=200|200" instead of -filter_complex "[1:0] adelay=200|200 [1:0]" for some reason. Ended up being cleaner anyways and it worked but I don't understand why the -filter_complex wasn't working an the -af was. I went ahead and applied -vsync 1 and -async 1 but I'm not quite sure if that is default and therefore redundant? Lastly I used -preset llhp (low latency high performance) which seemed to decrease the amount of dropped frames I get, this could also be untrue? Nonetheless the result - everything stays in-sync even if I drop frames and all tracks "start" at the same time:

Code: Select all

ffmpeg -guess_layout_max 0 -y -f dshow -video_size 3440x1440 -rtbufsize 2147.48M -pixel_format nv12 -framerate 200 ^
-i video="Video (00 Pro Capture HDMI 4K+)":audio="SPDIF/ADAT (1+2) (RME Fireface UC)" -map 0:0,0:1 -map 0:1 ^
-preset: llhp -codec:v h264_nvenc -pix_fmt nv12 -b:v 250M -maxrate:v 250M -minrate:v 250M -bufsize:v 250M -b:a 320k ^
-ac 2 -r 100 -async 1 -vsync 1 -segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\PC%02d.mp4 ^
-guess_layout_max 0 -f dshow -rtbufsize 2000M -i audio="Analog (3+4) (RME Fireface UC)" -map 1:0 -b:a 320k -ac 2 ^
-af "adelay=200|200" -segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\Voices\Theirs\TPC%02d.wav ^
-guess_layout_max 0 -f dshow -rtbufsize 2000M -i audio="Analog (5+6) (RME Fireface UC)" -map 2:0 -b:a 320k -ac 2 ^
-af "adelay=825|825" -segment_time 600 -segment_wrap 9 -f segment C:\Users\djcim\Videos\PC\Voices\Mine\MPC%02d.wav
I was able to get the rid of the "past duration too large" warning by increasing the input framerate to 200, I have absolutely no idea why this would fix that message. Put -r 100 on the output to make sure it came out as 100FPS which is what I actually wanted the framerate to be. Does anyone know why -framerate 200 would get rid of the "past duration too large" warning message when my screen is 100Hz? Is it the amount of frames that are stored in the buffer and not necessarily the frames per second? I am thoroughly confused and can't find answers anywhere.

If anyone can answer any of my questions before and after the block of code I would greatly appreciate it. But to be honest if no one cares to speak up or doesn't know, all is well. It's finally working as I desired, even though I'm confused.

Post Reply