Video sync


#1

A friend is building a video DAW which allows you to overdub videos.
He built it using swift and is coming up against sync problems.
He is a great musician and pretty fussy about the timing. If a video is out of sync by a few milliseconds he is going to notice. I thought C++ and JUCE might be a good way to try and improve the timing.
At this point I was thinking of trying to set something which captured a video from the macbook camera and played it back and allowed you to overdub. Possibly a midi click of some sort.
I have a feeling that with any operating system and threads there is probably a limit to what can be achieved … I assume in any video recording frames are going to get dropped with scheduling …
Anyone here have an opinion or solved these problems in the past with JUCE ?
I have a sense I have bitten off more than I can chew but JUCE is so well made it seems like a great environment to try…
Sean


#2

I think JUCE only has VideoComponent for dealing with video and that has very limited capabilities. You are not going to be doing a video editor/processor of any kind with that.

Your friend should likely be able to use the Apple video APIs and Swift just fine. Swift is native compiled code like C++. Apple is also gearing all their development support towards Swift now. It likely isn’t even a good idea anymore to do a Mac-only application with C++…(Of course if there’s any chance the application needs to work in Windows/Linux/Android in the future, C++ with something like JUCE is still an attractive option.)

Your friend’s app probably just has some misuse of the Apple APIs in the current code if the stuff is not working correctly. I take it he is using the new AV Foundation stuff, not the obsolete QuickTime things? (Testing with release builds too, of course and not debug builds? :slight_smile: )


#3

Thanks yes that makes sense. I don’t know much about SWIFT but it does seem to have thread handling, pointers and direct access to the memory so it should be able to solve those problems. I would think that if you stuck to a mac in the beginning and only used the Mac camera you should be able to get it to work well in Apple’s language! Thanks for your input you probably saved me hours of time …
Sean


#4

I am working in the same field. The timing is “ok”, but never sample accurate, because the VideoComponent sends directly to CoreAudio (aka driver), so it runs independent from your audio processing (and you have no option to alter the sound from the video).
At the ADC I spoke to a guy from Apple, and he made me aware, that the AVFoundation classes have the option to intercept the audio stream from the AVVideoComponent. But currently it is differently implemented, and I don’t know if it’s straight forward to change that, so it plays nicely in a JUCE processing chain…

However, I started an OS-independent video implementation github: filmstro_ffmpeg as JUCE module, where the audio is provided as a PositionableAudioSource and the video is sync’ed to that.
I realised problems with edge cases like e.g. old AVI (divx) files, so I don’t use it in production yet, but I hope it will get there…

There is a nice player in the package as example how to use it…


#5

I’ve not written any video code… but wouldn’t you use a CMAudioClock ?

https://developer.apple.com/documentation/coremedia/1618913-cmaudioclockcreate

Rail


#6

Good point, thanks @Rail_Jon_Rogut. The AVPlayer uses one already, so using VideoComponent::getPlayPosition should be accurate for the samples, just you are converting a int sample position to a double and back. But should be good enough for 99.9% of the cases… maybe just don’t issue a jump for less that 5 samples off…