FR: Add timing info to VBlankListener callbacks

Feature Requests
1

To anyone using VBlankAttachments to do any kind of animations, read on, as this one’s for you!

VBlank attachments were a huge step up to creating smooth moving components in a JUCE gui. There remains one fatal flaw, however. I propose a solution to fix this and make this mechanism really solid.

The issue:

The VBlankListener ListenerList in the component peer simply calls into each listener in a sequential fashion. It is then up to us, the users, to provide the callback lambda which is called to each listener.

If we want to do any movement, each object which is registered as a listener (usually via the VBlankAttachment class, will need to handle any time based logic from it’s callback. A solid solution is how games do it. You determine the time between this callback and the last, and update your physics based on the elapsed time. Where this falls apart, is each listener is called sequentially. So whoever gets called last now has a later timestamp, and the elapsed time is greater than the first listener to be called.

If you have several listeners, each updating paths, with variable amounts of time being used depending on the work to be done in each callback, this can result in two components of the same type falling out of sync. The one earlier in the listener list does not have enough elapsed time to make a step in the animation, but the later one does.

While this discrepancy evens out on the whole, and one won’t get ahead of the other, it does create little 1 frame glitches, which is noticeable.

If you know what I’m talking about, please back me up in the comments.

I have a quick and dirty fix, which I hope the JUCE team will be able to implement in one way or another:

The Solution
Pass the current timestamp through the function chain from the root of the VBlank itself. I am on windows here, so the relevent code is in juce_Windowing_windows.cpp line 1947, which I have changed to the following:

    void onVBlank() override
    {
        auto time = Time::getMillisecondCounterHiRes();
        vBlankListeners.call ([time] (auto& l) { l.onVBlank(time); });
        dispatchDeferredRepaints();

        if (renderContext != nullptr)
            renderContext->onVBlank();
    }

Now every single listener is given the same timestamp, and calculations can be made from a common timestamp, without having functions from earlier in the list effecting the timestamp in the ones called later.

It has resolved the issue for me here, and I think this is the correct way for this to work.

Vblank details
2

As a test, I have 2 waveforms updating via a vBlankAttachment.

I am outputting the delta time from 2 objects. The following lists show a then b, repeated.

Using the stock JUCE config, and handling delta time in the object itself:

DeltaTime: 6.9226
DeltaTime: 6.9288
DeltaTime: 8.5207
DeltaTime: 8.6311
DeltaTime: 7.0463
DeltaTime: 6.9349
DeltaTime: 5.2633
DeltaTime: 5.2638
DeltaTime: 6.9275
DeltaTime: 7.0218
DeltaTime: 6.9603
DeltaTime: 6.8767
DeltaTime: 8.3606
DeltaTime: 8.3405
DeltaTime: 5.6355
DeltaTime: 5.7426
DeltaTime: 7.2746
DeltaTime: 7.2266

With my ‘fix’:

DeltaTime: 1.4945
DeltaTime: 1.4945
DeltaTime: 6.9254
DeltaTime: 6.9254
DeltaTime: 7.0063
DeltaTime: 7.0063
DeltaTime: 6.8405
DeltaTime: 6.8405
DeltaTime: 6.9504
DeltaTime: 6.9504
DeltaTime: 6.9741
DeltaTime: 6.9741
DeltaTime: 7.5741
DeltaTime: 7.5741
DeltaTime: 6.3133
DeltaTime: 6.3133
DeltaTime: 7.75
DeltaTime: 7.75
DeltaTime: 6.1118
DeltaTime: 6.1118

In the former, you can see there are little fluctuations, which can add up and have the occasional update land a frame late.

2 Likes
3

In addition to better synchronisation between components, having a timestamp would also allow to use the correct frame output time that can be retrieved on the macOS callback. That would get rid of the fluctuations you are seeing and make the painting work with the time the frame will be displayed in the future.

macOS pseudocode:

// in init.. determine mach timeFactor
mach_timebase_info_data_t timebase_info;
mach_timebase_info(&timebase_info);
auto timeFactor = double(timebase_info.numer) / double(timebase_info.denom);

static CVReturn displayLinkCallback(CVDisplayLinkRef /*displayLink*/,
    const CVTimeStamp * /*now*/,
    const CVTimeStamp *outputTime,
    CVOptionFlags /*flagsIn*/,
    CVOptionFlags * /*flagsOut*/,
    void *context) {

    auto display_fps = double(outputTime->videoTimeScale) / double(outputTime->videoRefreshPeriod);
    auto frametime_ms = double(outputTime->hostTime) * timeFactor * 1e-6;
   
    // frametime_ms and display_fps could now be fed to the lambda callbacks for better timing
    
     return kCVReturnSuccess;
}
2 Likes
4

Interesting idea, I feel like it would be good work to bundle in with this FR: FR: Callback or other mechanism for exposing Component debugging/timing

1 Like
5

I’m not sure timing paint calls for D2D is that useful, as draw calls might be batched and only executed much later, during presentation.

6

That’s a good point. I’ve been seeing lots of JUCE_SCOPED_TRACE_EVENT_FRAME (which map to windows etws) peppered in the d2d drawing code, and was curious if that resulted in usable execution timings (I know nothing about how it works). My strategy so far has been to optimize paint calls on macOS and sanity check on d2d.

7

Yes I fear you may be right. The vblank happens just before a frame is rendered to screen, but it’s hard to say what the time between that callback and the actual render may be. However, if you know what the refresh rate is, you can assume that time for each callback and step the animation in that increment. I like that the apple hook has the frame time in the future, that’s a much better way of doing things. But in order to be system agnostic, perhaps the best we can hope for is the current time and the displays refresh rate. Then we can assume the step size and run some extra logic to ensure things don’t fall too far out of the expected range?

8

As this is a somewhat exploratory thing on my part, I’m just going to keep weighing in on things I find as I find them. I enjoy the discussion and hope this leads to something.

I have found obtaining the refresh rate of the monitor is a key component to having stable motion. Delta time is not enough in isolation, as if you are moving positions at a speed which does not line up to a frame, things will still not look right. By broadcasting the frame rate to any component that needs it, you can adapt your movement speed to the nearest number that works with frame rate, then you get even steps and it looks smooth.

On top of this, simply skipping every n frames based on your real refresh rate and your target frame rate is not good. Frame times are never guaranteed, and frames can be dropped before you get the callback. What I am doing currently is this:


    auto currentTime = juce::Time::getMillisecondCounterHiRes();
    auto deltaTime = currentTime - lastFrameTime;

    auto typicalFrameTime = 1000.0 / screenRefreshRate;
    auto targetFrameTime = 1000.0 / actualFPS;

    if (deltaTime < targetFrameTime - (typicalFrameTime / 2.0))
	{
        return;
	}

    lastFrameTime = currentTime;

This will consider the current frame as valid unless the next expected frame will be closer to the target.

I think I’m a few days out from something really solid, but so far I have created an intermediary ‘master’ VBlankAttachment which lived in my top level editor, then ‘slaves’ which will find this parent and add themselves as listeners to it.

The master get’s the VBlank call we are all currently used to, and then passes down a call to it’s listeners, but this time it injects the delta time if it considers the frame a valid frame based on the target fps.

I also have another callback which will notify of a change to display fps, so any listeners know to update their movement logic to suit the new incoming intervals.

9

Latest observation:

Skipping frames does nothing for the ‘allocated time’ in a VBlank callback.

When we artificially limit the frame rate, we are still bound by the display refresh rate for time allocated to this callback.

At native 60fps, we have 16.6666666667ms per frame.
At native 120fps, we have 8.33333333333ms per frame.
At native 120fps, but limited to 60fps, we still have 8.33333333333ms per frame. We just skip every other frame.

This means our update time could be very limited, on high fps monitors, even if we artificially limit our update rate. It’s very easy to run over.

I think this means the ‘master’ updater needs to be able to issue update signals as well as paint signals. This way we can use the skipped frames to do work.

1 Like
10

This FR seems resolved by commits added by @attila to develop a couple weeks ago.

Support was also added to AnimatorUpdater — but can someone ELI5 the benefits to me — is it only relevant for physics/game based stuff? The change makes it so that all components will get the same timestamp in the callbacks… which lets components synchronize and therefore coordinate animation / fire events at more precise times? I’m missing an example use case, I think. Does it always involve also tracking time in each component?

11

With this change it’s no longer necessary to track (more like measure) time in each component.

This avoids the pitfall where each component would execute an update function in the VBlankListener callback, and if some of those took non-negligible amount of time, then it would cause subsequent callbacks in the chain to observe time at a later point. This could cause times tracked by each component to drift apart, and cause things that should move together, also visibly drift apart.

12

I am not sure if it’s mentioned anywhere, but after my today tests it looks like controling frame rate should be implemented inside juce::Component::paint(), but not in VBlankListener::onVBlank().

Because repaint() called in onVBlank() cause paint() execution happens asynchronously. And today I’ve just found out that time duration in onVBlank() can be varoius on each callback and not executed as evenly as it is in paint().

At least on Macbook Pro M1 I experienced it.

13

Nice addition @attila!

The only thing left to make this a-okay in my book is the issue of limiting the update rate globally in one place.

I use the below on windows to set a ‘target framerate’ which returns the nearest multiple of the hardware refresh rate, so it looks smooth. I then do the logic at the top level and decide whether or not to call the listeners down the chain. Happy to send through all my working source if it helps you, but I suspect you’ll have more edge cases than I’ve considered.

struct NativeWindowHelpers
{
	static std::optional<int> getNativeFPS(juce::Component* component);
	static int nearestSuitableTargetFPS(int current, int target);
};

std::optional<int> NativeWindowHelpers::getNativeFPS(juce::Component* component)
{
#ifdef JUCE_WINDOWS
	auto peer = component->getPeer();
	if (peer == nullptr)
		return std::nullopt;

    windows::HWND hwnd = static_cast<windows::HWND>(peer->getNativeHandle());
	//hwnd = nullptr;
	if (hwnd == nullptr)
		return std::nullopt;

    windows::HDC hdc = windows::GetDC(nullptr);

	if (hdc == nullptr)
		return std::nullopt;

	int refreshRate = windows::GetDeviceCaps(hdc, VREFRESH);

	if (refreshRate > 0)
		return refreshRate;
	else
		return std::nullopt;

#else //macOS

	return std::nullopt;

#endif // JUCE_WINDOWS

}

int NativeWindowHelpers::nearestSuitableTargetFPS(int current, int target)
{
    int closestDiv = 1;
    double minDifference = std::numeric_limits<double>::max();

    for (int i = 1; i <= std::sqrt(current); ++i) {
        if (current % i == 0) {
            int division1 = i;
            int division2 = current / i;

            double diff1 = std::abs(division1 - target);
            double diff2 = std::abs(division2 - target);

            if (diff1 < minDifference) {
                minDifference = diff1;
                closestDiv = division1;
            }

            if (diff2 < minDifference) {
                minDifference = diff2;
                closestDiv = division2;
            }
        }
    }

    return closestDiv;
}
1 Like
14

What is exactly the timeStamp which is provided by the VBlankAttachment?

On macOS, if within a VBlankAttachment callback I compare the provided timeStamp to the current time (Time::getMillisecondCounterHiRes() / 1000.0), the difference can be negative if there’s a heavy load, so the timeStamp seems to indicate a time in the past.

On the other hand, I see that the linux implementation of the VBlankAttachment simply provides Time::getMillisecondCounterHiRes() / 1000.0 as the timeStamp, so I wouldn’t have expected the values to be so different.

[edit:]

So on some platforms, the timestamp corresponds to the predicted frame output time. If the timestamp lies in the past the message thread is lagging and the frame won’t be drawn so callback should be discarded, right?

15

Apart from Linux, these are timestamps provided by the OS vblank mechanism. You cannot compare these values with Time::getMillisecondCounterHiRes() and expect anything other than both getting incremented at roughly the same speed. For drawing purposes the vblank provided value will more closely track the frame times, where Time::getMillisecondCounterHiRes() will have processing and load based jitter.

The only thing you can expect from these timestamps is that they will increment at a very steady rate in sync with the vblank events. If framedrops occur, you should receive fewer VBlank callbacks, but whenever you do receive one, you should construct a frame using the provided timestamp and not discard any frames on your end.

1 Like
16

My first try in order to limit the framerate would be a slightly different approach.

Instead of calculating an appropriate framerate I think I would set a minimum frame time instead.

This could be done with a custom VBlankAttachment from user code. As a rough sketch

struct LimitedVBlankAttachment
{
    LimitedVBlankAttachment (double minFrameTimeSecIn, Component* c, std::function<void (double)> callbackIn)
        : minFrameTimeSec { minFrameTimeSecIn },
          callback { std::move (callbackIn) },
          attachment { c, [this] (auto t) { onVBlank (t); } }
    {
    }

    void onVBlank (double timestamp)
    {
        if (timestamp - lastTimeStamp < minFrameTimeSec)
            return;

        lastTimeStamp = timestamp;
        callback (timestamp);
    }

    double minFrameTimeSec;
    std::function<void (double)> callback;
    VBlankAttachment attachment;
    double lastTimeStamp{};
};

You could then use this everywhere in place of a VBlankAttachment.

LimitedVBlankAttachment { 1.0 / 60.0 - 0.01, this, [this] (auto x) { update(); } };
2 Likes
17

Thanks for the explanation!

18

@attila What was the rationale behind adding the timestamp to the vblank chain, opposed to delta time? Aside from the slight annoyance of having to cache the last time and do the math in any component that uses it, is there a benefit to this I’m not seeing?

19

A small typo: in VBlankAttachment, timestampMs should be replaced by timestampSec :

void onVBlank (double timestampMs) override;

void VBlankAttachment::onVBlank (double timestampMs)
{
    NullCheckedInvocation::invoke (callback, timestampMs);
}
20

The timestamps allow to achieve perfect sync on supported OSes (currently macOS). macOS produces timestamps for the time a frame will be displayed giving the most precision possible. These timestamps also correctly deal with variable refresh-rates and multiple screens and are the best source of timing info available for frames on the screen.

The same kind of timing information btw. would also be possible when using the Direct2D renderer, but at this time it is not implemented.

1 Like