Syncing Video and Audio

General JUCE discussion
1

I’m using the QuickTimeMovieComponent for mp4 video playback. For reasons I won’t get into here, I have the audio for each video in a separate mp3 file, and am using AudioFormatReader/AudioTransportSource to playback the audio file. Most of the time it works perfect. But I’ve had a few users complain about “clicking” in the audio, and I’ve noticed occasional glitches in the audio when the video first starts playing back.

Currently, whenever the user hits play, it calls the QuickTimeMovieComponent’s play() method, and the AudioTransportSource’s start() method. I was worried about the two getting out of sync, so I have a background thread checking the video and audio file’s time every second. If they ever get off by more than 1/10 of a second, it corrects the video’s time to match the audio’s. I don’t think this is causing the audio glitch, because it doesn’t change the audio’s time, only the videos, so the audio should just be playing back smoothly. And I’ve never noticed the video get “corrected” so I think they’re playing back as expected. Two questions:

  1. Is there any reason why one file would get out of sync from the other? Or can I rely on them both being played back without a problem?
  2. Any obvious reason for the janky audio?

Video/Audio loading:

void VideoComponent::setupMovie(String vidPath, String demoAudioPath)
{
    File curVid = vidPath;
    movieStream = curVid.createInputStream();
    videoPlayer->loadMovie (movieStream, false); 
 
    File audioFile = demoAudioPath;
    
    if (audioFile.exists())
    {
        transportSource->stop();
        transportSource->setSource (0);
        currentAudioFileSource = 0;
        
        AudioFormatManager formatManager;
        formatManager.registerBasicFormats();
        
        AudioFormatReader* reader = formatManager.createReaderFor (audioFile);
        
        if (reader != 0)
        {
            currentAudioFileSource = new AudioFormatReaderSource (reader, true);
            transportSource->setSource (currentAudioFileSource, 0, nullptr, reader->sampleRate);
        }
    }
}

Playback control:

if (videoPlayer->isPlaying()) 
{
     videoPlayer->stop();
     transportSource->stop();
} 
else 
{
     videoPlayer->play();
     transportSource->start();
}

Sync checker:

if (fabs(videoPlayer->getPosition() - transportSource->getCurrentPosition()) > 0.1)
            videoPlayer->setPosition(transportSource->getCurrentPosition());
2

Clicking could be a few things, typical ones being;

  • Clicking is part of the audio file already
  • Buffer size is too low

Sync issues will definitely arise when using some compressed audio formats, like MP3; they don’t all have sample-accurate seeking. This is a reason to use timecode (ie: LTC/SMPTE).

3

K. I added a buffer, we’ll see if that helps:

transportSource->setSource (currentAudioFileSource, 44100, tSlice, reader->sampleRate);

I’m not sure how I would incorporate timecode though. I don’t see any support for timecode in the AudioTransportSource and QuickTimeMovieComponent classes. Any ideas there?

4

Your best bet for implementing timecode is somehow using an AudioPlayHead along with your movie component, and perhaps custom version of the AudioTransportSource?

5

K. Thanks for the tips. If I run into sync issues I know where to start.