I am having trouble understanding how the return value of Plugin::getLatencySeconds() is used within Tracktion Engine to make sure my tracks are all aligned.
I have an Internal Plugin that returns a good looking value for Plugin::getLatencySeconds() and the value is read within PluginNode::prepareToPlay() where automationAdjustmentTime holds the value of -0.17066666666666666. After this I lose track of what’s happening, and any track that includes my plugin has noticeable latency in playback, and the head aches start .
So, can anyone explain how we go from returning a latency value to getting expected playback? Thanks!
Does your plugin actually delay the input by that amount?
The short answer is that it’s pretty complicated and I can’t really explain it in a post on here.
I did give a whole talk about it a few years ago through:
But you shouldn’t have to know all that. The tl;dr is if your plugin delays the input, you need to report that and the graph will create latency nodes at all the sum points to balance it.
Hey Dave, thanks for the quick reply. I’ll give the video a watch for my own curiosity, thanks.
The plugin in question is actually the te::PitchShiftPlugin. When you say input I’m a bit confused, but that might just be more evidence that my understanding is incomplete. Input gets a lot of different definitions in audio development
In the problem context I have two tracks with the te::SamplerPlugin attached to each and only one with the te::PitchShiftPlugin. Both tracks have a single hi-hat sample loaded, and when I press play to move the transport I hear the pitch shifted one later. I’m unsure where to continue my investigation at this point, but clearly some point between the plugin returning its latency value and the nodes being summed something is going wrong.
There’s info as well in your ADC talk from last year which went up on YouTube a few days ago (which of course you know but I’m just pointing that out in case you didn’t know it’s been uploaded in the last few days to the ADC channel). I’ve only watched the 2023 video, so just pointing that out in case there is additional info in it not covered in the one you’ve linked that you’d like to reference.
@Dave Having watched your talk, is there a way to get Tracktion Engine to show it’s full graph at any given time? It seems like looking to see that the latency nodes have been added correctly would be a sensible place to continue my search. I should expect to see my “F nodes” to compensate for the latency of the pitch shifter which would my version of the “C Node” shown here.
Yes, but it’s an internal function used for testing, not one that’s polished for public use
Look at tracktion_graph/tracktion_TestUtilities.h and the createGraphDescription (Node&)
In Waveform we actually have this function we can trigger to dump the state to disk (requires macOS and graphvis/dot installed. Creating the NodeGraph is important as that does the transformations.
static void writePlaybackGraphToDesktop()
{
#if JUCE_MAC
if (auto e = getEditIfOnEditScreen())
{
// Detach the Edit from the DeviceManager first
auto& dm = e->engine.getDeviceManager();
tracktion::graph::PlayHead playHead;
tracktion::PlayHeadState playHeadState { playHead };
tracktion::ProcessState processState { playHeadState, e->tempoSequence };
tracktion::CreateNodeParams cnp { processState, dm.getSampleRate(), dm.getBlockSize() };
EditPlaybackContext epc (e->getTransport());
std::atomic<double> time;
if (auto n = createNodeForEdit (epc, time, cnp))
{
auto graph = tracktion::node_player_utils::prepareToPlay (std::move (n), {}, cnp.sampleRate, cnp.blockSize, {}, {}, true);
const std::string s = tracktion::test_utilities::createGraphDescription (*graph->rootNode);
juce::TemporaryFile sourceFile;
sourceFile.getFile().replaceWithText (s);
const auto destFile = File::getSpecialLocation (File::userDesktopDirectory)
.getNonexistentChildFile ("edit_graph", ".svg");
auto cmd = juce::String ("cat {src} | dot -Tsvg > {dest}")
.replace ("{src}", sourceFile.getFile().getFullPathName().quoted())
.replace ("{dest}", destFile.getFullPathName().quoted());
std::system (cmd.toRawUTF8());
}
}
#endif
}
You might also want to test with the LatencyPlugin. There is a chance that the latency reported by Rubberband doesn’t actually match up with what it’s applying…
Running this test with Rubberband currently, the offset looks to be about 0.032 seconds, a very different number than the 0.1706 being returned by getLatencySeconds(). This is the render of the normal track, and the copied normal track with pitch shifting applied.
(Here I’ve crudely panned the two tracks to make for an easier comparison).
So it seems as though getLatencySeconds() isn’t behaving as expected, or there’s latency being added elsewhere that’s not being accounted for, or some other mystery 3rd thing. I’ll get on using your writePlaybackGraphToDesktop() in my project…
I’m fairly sure that the PDC is working correctly.
It’s much more likely that the latency reported by Rubberband isn’t what we’re reporting from the TimeStretcher. Are you able to double check that?
It’s been a long time since I used the Rubberband stretcher.
And the effect was to delay the playback of the pitch shifted samples by a further 3 seconds, when based on our discussion I would have expected them to arrive very early. Something like expectedLateness = 3 secondsAdded - 0.032 secondsAlreadyLate - 0.176 secondsFromFunctionReturn (bearing in mind that secondsAlreadyLate’s source has yet to be explained).
Could it matter that the PitchShiftPlugin is inside of a RackInstancePlugin?
No, PDC doesn’t work like that. If you’re reporting 3s of latency (in the PitchShiftPlugin) but not applying it, that 3 seconds will get applied to all the other paths. So the path with PitchShiftPlugin will end up being 3s ahead.
I think I need to write a test that measures the latency of the various time stretchers.
The idea would be to just create a clip with a single transient sample and render it through the pitch shift plugin (with no pitch shift). The rendered output should have the transients in the same place. This is basically my test from above.
FYI I’ve started on this test, I just haven’t got to the stage of committing it yet but here are the prelim results:
I created an audio file with a single sample of 0.5f at 1s
Then I added this as a clip to the first two tracks of an Edit
I then added a PitchShiftPlugin to track 1 (using soundTouchBetter mode)
Then I played it back and captured the result
The following is the source file (as expected) and the output beneath. Bear in mind this is audio output not a render so you would expect things to be shifted in time by the amount of the latency
The latency measured to the first sample is 0.173s
The latency measured to the second sample is 0.186s
The latency reported by the plugin is 0.186s which matches the second spike
So the plugin is actually introducing 0.013s more latency than reporting
I’ve not had a chance to test with Elastique or Rubberband as they’re not set up in the main Engine repo for licensing reasons. I’ll need to check if the behaviour is similar there.
But from the above I think I can conclude that the PDC system is working correctly but the latency introduced by the PitchShiftPlugin is not the same as the plugin is reporting.
Next I’ll need to see if this is a bug in the TimeStretch implementation or the PitchShiftPlugin.
I know at this point you’ve more or less rules out the graph being the source of the problem, but for completeness I got my graph visualised, and it looks fine (I think).
Yeah, I think the code is correct but there seems to be something funny going on with the pitch shifter latency. I’ve added some tests here:
Which time the delay between an impulse and how long it takes to come out of the pitch shifter and compares it to the reported delay. This seemed to be spot on for Rubber Band but when I commit it to CI it fails so I might need to look a little closer.
I’m still trying to figure out why adding rubber band breaks a ton of our tests (it’s like maths just isn’t working in the machine as ~400 unrelated tests now all fail).
But I did get as far as adding Rubber Band to Waveform to be used in the Pitch Shift Plugin and everything seems to work there as expected, no changes required:
pitch_shift_rubberband.mp4.txt (639.7 KB)
(This is an mp4, I don’t know why I can’t add that here, it should just play in the browser but if you download it and remove the .txt extension it should play).
So I’m struggling to see what exactly isn’t working for you?
As far as I can tell:
The Rubber Band TimeStretcher correctly compensates for the latency it introduces
PitchShiftPlugin adds some internal latency to buffer the TimeStretcher and correctly reports this
All the PDC handling in the Engine works correctly so tracks will align correctly when played back
.
