A proposal to improve repainting of Juce components


#1

Greetings, Persons of Orange.

I’ve been whinging :wink: for a few months about an issue I had about repaint rectangles on Juce (which I mistakenly believed was only happening on OS/X). I’ve finally diagnosed it, and I have a potential solution that should speed up repaints on quite a few Juce clients while maintaining complete backward compatibility.

The Problem

Consider the following near-trivial Juce app - an app that is 1000x1000 pixels in size and has the following components:

[list]Window - a top-level Component (say, a ResizeableWindow)
Content - a content Component contained in Window.
Top - an opaque Component with bounds [0, 0, 1000, 500], contained in Content.
Bottom - an opaque Component with bounds [0, 500, 1000, 500], contained in Content.
[/list]

And let us suppose that none of these Components implement Component::paintOverChildren - which apparently is the most common case.

Now, suppose we rapidly animate just the individual pixels at global coordinates (0, 0) and (999, 999). What happens?

In a given update cycle, we dirty (0, 0) in Top and (999, 999) in Bottom. These propagate up to Window, the dirty rectangle is computed as [0, 0, 1000, 1000] and this clip rectangle propagates back down to force a complete redisplay of Top and Bottom - 1,000,000 pixels.

But because there are no implementations of Component::paintOverChildren in any of these components, once, say, Top has written pixel (0, 0), it is guaranteed that no other component will write over (0, 0). In fact, we only needed to update 2 pixels…

An easy solution

A simple fix would be to add a flag wontPaintOverChildren with a setter, and test to see if this flag were set before dirtying the parent. It’s backward compatible, adds on bit flag, and, for all components that do NOT use that feature, a single bit test… tiny!

The code is below, and in fact I’m going to go off and tweak Juce to work this way, stay tuned!


void Component::setWontPaintOverChildren (const bool w)
{
    flags.wontPaintOverChildren = w;
}

bool Component::mightPaintOverChildren() const {
    return !flags.wontPaintOverChildren || (parentComponent && parentComponent->mightPaintOverChildren());
}

void Component::repaintParent()
{
      if (parentComponent != nullptr && parentComponent->mightPaintOverChildren())
          parentComponent->internalRepaint (ComponentHelpers::convertToParentSpace (*this, getLocalBounds()));
}

void Component::internalRepaintUnchecked (const Rectangle<int>& area, const bool isEntireComponent)
{
  // ...
  
            if (parentComponent != nullptr && parentComponent->mightPaintOverChildren())
                parentComponent->internalRepaint (ComponentHelpers::convertToParentSpace (*this, area));

#2

This code was a bust. :frowning:

I tried several variations. If I run the code as written, my animated components don’t even animate; if I revert repaintParent() to its original code, it all works - but I still get the update issue.

There must be a good solution for it… the problem is pretty obvious, as stated.


#3

Don’t use paint over component.


#4

I never use paintOverChildren - is that what you mean?

I’m working on a fix for my problem at least… I’m going to use a CachedComponentImage which takes invalidate requests and keeps separate dirty rectangles for its children - stay tuned!


#5

Yeah, sorry about that.

I’m still not understanding the problem here. Many months ago when I wrote my own scrolling waveform draw code, I traced through Juce calls and in fact, Juce “RectangeList” clips out children when the parent draws, if the child has setOpaque(true). If a parent is fully obscured it never gets the call to paint().

I do recall, that Jules mentioned something about not being able to get the exact list of dirty rectangles from the operating system under Mac OS X. Jules can you clarify?

I think that if you’re drawing a waveform and doing a costly computation (such as applying a low pass then downsampling for display purposes) you would certainly want to cache the result. Repeating the calculations every time you get called to paint wastes cycles.


#6

I didn’t mean adding an image cache - I meant adding a “fake” cache that simply handles the invalidation rectangles a little more gracefully but delegates all the drawing to what’s there.

(I do have an image cache, but there’s gingerbread on top of it, which might be where the cost is coming from. I might have to either move the cache up a little and include the extra stuff, or have a SECOND image cache… gah.)

However, I’m still redrawing the entire screen every refresh, when I only need to redraw 10 vertical pixels of it each time - about 100:1 ratio. Even with a better cache, that will be fairly consumptive.

Many months ago when I wrote my own scrolling waveform draw code, I traced through Juce calls and in fact, Juce “RectangeList” clips out children when the parent draws, if the child has setOpaque(true). If a parent is fully obscured it never gets the call to paint().

That’s absolutely true, and I’ve also observed this but that wouldn’t prevent this issue (unless you mean repaint…)

I have a clock on the far right (which doesn’t impinge on the Waveform) and a cursor on the far left (which does). If only the clock changes, it only dirties a small area which doesn’t affect the Waveform, and then there are never any calls to Waveform::paint, as you say. But if the clock and the cursor change, both of them dirty the top level component at the left and right sides, resulting in a huge repaint rectangle.


#7

This should ONLY be happening on OS X. The Windows implementation of heavyweight peer breaks down the update region received from the system, into a RectangleList:

juce_win32_windowing.cpp:

class WindowClassHolder...
...
  if (regionType == COMPLEXREGION && ! transparent)
  {
...
    contextClip.addWithoutMerging (Rectangle<int> (cx - x, rects->top - y, rects->right - cx, rects->bottom - rects->top).getIntersection (clipBounds));
...
  }
...
...

The loop extracts rectangles from the HRGN and adds them to the clip for the GraphicsContext. Therefore, in your paint() you can call GraphicsContext::getClipBounds(), GraphicsContext::getClipRegion(), and GraphicsContext::clipRegionIntersects() to optimize the waveform drawing.

However, on OS X I remember Jules mentioning that the operating system does not provide a way to extract the list of dirty rectangles (Jules?). Therefore on OS X the update is sub-optimal.

I suspect that because this information is simply not available, you will have to come up with an appropriate work around. There’s no way to “make it work right.” You might need to rearrange your interface and put in some caching to get it to work.


#8

Good detective work guys. I’ve been scratching my head over a repainting scrolling waveform as well. Caching the image and moving it, plus marking certain areas as dirty and repainting just those has been my workaround as well, but with little real success so far.

Of course - the other way to make it faster would be to use an OpenGL Component.

Interested to see what you come up with!


#9

Ah!

That makes sense. I’d initially found it only happened on OS/X but going through the code I’d recently decided that it should be happening on Windows.

I’m experimenting with a workaround - I’ll let you know!

(And I’m going to move all my decorations into my image cache, it shouldn’t be too hard…)


#10

Yes, there are at least a couple of threads about this if you want more details…


#11

I have to say that in the light of morning, I’m a little baffled why OS/X would be implicated considering that all the components involved are light-weight components.

Since none of the components have peers except the top one, from my reading of the code I don’t quite see how the brain-deadness of OS/X would be implicated?!

If all the repaints to light-weight components are forced to go through the brain-dead top-level component when they don’t have to, it seems like a feature deficiency that could be fixed.

Anyway, I’m about to find out for sure, as I have my new layout manager which keeps separate dirty rectangles for each of its contained children. I’ll keep you posted.


#12

Because the operating system generates the “repaint invalid areas” signal, and provides Juce with the region to update.

Imagine two disjoint opaque child Components with (w,h)=(1,1) and coodinates (x,y) = (0,0) and (2,2) respectively (in other words, non overlapping children).

Now imagine that you request a repaint of both components. Under both Mac OS X and Windows, the resulting update region consists of two disjoint single pixels.

On Windows, Juce is able to convert the update region into a RectangleList, “subtract out” the bounds of all of the opaque children, and recognize that the parent of these children does not need to update (since the inverse of the update region intersected with the union of the opaque childrens’ bounds is an empty RectangleList).

All this magic happens in Component::paintComponentAndChildren()

juce_Component.cpp

void Component::paintComponentAndChildren (Graphics& g)
{
    const Rectangle<int> clipBounds (g.getClipBounds());

    if (flags.dontClipGraphicsFlag)
    {
        paint (g);
    }
    else
    {
        g.saveState();
        ComponentHelpers::clipObscuredRegions (*this, g, clipBounds, Point<int>());

        if (! g.isClipEmpty())
            paint (g);

        g.restoreState();
    }

    for (int i = 0; i < childComponentList.size(); ++i)
    {
        Component& child = *childComponentList.getUnchecked (i);

        if (child.isVisible())
        {
            if (child.affineTransform != nullptr)
            {
                g.saveState();
                g.addTransform (*child.affineTransform);

                if ((child.flags.dontClipGraphicsFlag && ! g.isClipEmpty()) || g.reduceClipRegion (child.getBounds()))
                    child.paintWithinParentContext (g);

                g.restoreState();
            }
            else if (clipBounds.intersects (child.getBounds()))
            {
                g.saveState();

                if (child.flags.dontClipGraphicsFlag)
                {
                    child.paintWithinParentContext (g);
                }
                else if (g.reduceClipRegion (child.getBounds()))
                {
                    bool nothingClipped = true;

                    for (int j = i + 1; j < childComponentList.size(); ++j)
                    {
                        const Component& sibling = *childComponentList.getUnchecked (j);

                        if (sibling.flags.opaqueFlag && sibling.isVisible() && sibling.affineTransform == nullptr)
                        {
                            nothingClipped = false;
                            g.excludeClipRegion (sibling.getBounds());
                        }
                    }

                    if (nothingClipped || ! g.isClipEmpty())
                        child.paintWithinParentContext (g);
                }

                g.restoreState();
            }
        }
    }

    g.saveState();
    paintOverChildren (g);
    g.restoreState();
}

Since this code is shared on all platforms (juce_Component.cpp is platform independent), the feature should in theory work everywhere.

Unfortunately, the desired optimal behavior of Component::paintComponentAndChildren() for the case you mentioned (which by the way, already perfectly implements your suggestion in A proposal to improve repainting of Juce components) depends on the clip region being set correctly in the Graphics.

The reason the repaints are ‘forced’ to go through the parent is because Mac OS X provides only a bounding box and not a region to indicate the area requiring update. In the original example, with two sibling components having (t,l,w,h) coordinates of (0,0,1,1) and (2,2,1,1) the resulting bounding box for both platforms is (0,0,3,3). But on Windows, we can extract from the HRGN, the two disjoint rectangles. On Mac OS X we have only the larger, inefficient rectangle which forces additional drawing.

You will have created a duplicate of the code in Component::paintComponentAndChildren(). Hopefully you used the existing juce::RectangleList code which makes it easy to compose regions, instead of reinventing the wheel. If you follow this path to its logical conclusion you will discover what Jules has discovered about OS X - that you do not receive a rich description of the area requiring update.

I suggest, two alternative courses of action:

  1. Change your code to draw faster, by using image caching and reworking your UI

  2. Research the topic of getting the update region on Mac OS X to see if this can be fixed in Juce heavyweight peer for OS X.

I’m no Mac expert but I think the code of interest is here (Jules?)

juce_mac_NSViewComponentPeer.mm

void NSViewComponentPeer::drawRect (NSRect r)
{
    if (r.size.width < 1.0f || r.size.height < 1.0f)
        return;

    CGContextRef cg = (CGContextRef) [[NSGraphicsContext currentContext] graphicsPort];

    if (! component->isOpaque())
        CGContextClearRect (cg, CGContextGetClipBoundingBox (cg));

   #if USE_COREGRAPHICS_RENDERING
    if (usingCoreGraphics)
    {
        CoreGraphicsContext context (cg, (float) [view frame].size.height);

        insideDrawRect = true;
        handlePaint (context);
        insideDrawRect = false;
    }
    else
   #endif
    {
        const int xOffset = -roundToInt (r.origin.x);
        const int yOffset = -roundToInt ([view frame].size.height - (r.origin.y + r.size.height));
        const int clipW = (int) (r.size.width  + 0.5f);
        const int clipH = (int) (r.size.height + 0.5f);

        RectangleList clip;
        getClipRects (clip, view, xOffset, yOffset, clipW, clipH);

        if (! clip.isEmpty())
        {
            Image temp (getComponent()->isOpaque() ? Image::RGB : Image::ARGB,
                        clipW, clipH, ! getComponent()->isOpaque());

            {
                ScopedPointer<LowLevelGraphicsContext> context (component->getLookAndFeel()
                                                                    .createGraphicsContext (temp, Point<int> (xOffset, yOffset), clip));

                insideDrawRect = true;
                handlePaint (*context);
                insideDrawRect = false;
            }

            CGColorSpaceRef colourSpace = CGColorSpaceCreateDeviceRGB();
            CGImageRef image = juce_createCoreGraphicsImage (temp, false, colourSpace, false);
            CGColorSpaceRelease (colourSpace);
            CGContextDrawImage (cg, CGRectMake (r.origin.x, r.origin.y, clipW, clipH), image);
            CGImageRelease (image);
        }
    }
}

Note that when USE_COREGRAPHICS_RENDERING is set to 1, the resulting graphics context has only a boring rectangle for the bounds, regardless of the complexity of the clip region. This goes back to what I was saying that you can’t extract the clip when using Core Graphics.

Why don’t you try not using CoreGraphics to render?

Dunno what to say about iOS, it seems CoreGraphics is the only rendering choice from looking at juce_ios_UIViewComponentPeer.mm.


#13

Because the operating system generates the “repaint invalid areas” signal, and provides Juce with the region to update.

If I move a lightweight Component in front of another light-weight Component, then in some sense, why is the operating system even implicated at all? These light-weight Components are nothing that the operating system knows about - they are a fiction that Juce handles. Perhaps we can avoid this bad feedback loop somehow? Let’s go through the steps that actually happen.

I move my cursor component in front of another one - somewhere else, in another component, my clock updates. Within the light Juce components, two small, independent repaint events occur. These propagate correctly up through Juce to the top level.

I think we’re all agreed up to here as to what happens. Now let’s go slowly…

At this point, Juce MUST tell the OS about these two dirty areas, or they simply won’t be repainted. True?

And then the OS, which is getting these two dirty areas in one “clock cycle”, crudely glues them together and slams them back as a huge rectangle. Yes?

Is there some way to say, “Do not glue this repaint together with any other repaints”? Doubtful but…?

Is there some way to detect that this huge update is in fact only in response to Juce’s repaints and filter it out?

Apparently there’s no way to get a better repaint region from the OS under any circumstances? Why isn’t this a crucial issue for Mac game programmers?

And as to the other workarounds, I am doing all or most of these, and today I’m going to move all my rendered parts into the cache (even though my timing experiments show that I should only expect a marginal improvement). And it’s unlikely I’m going to start writing Objective C, a language I know nothing about, to fix a moderate-priority bug on one platform.

But when it comes down to it, I want you to be able to zoom waaay in and watch the cursor fly by - and this is being stymied by the fact that I’m rendering almost 100 times as much area as I need to. There’s a limit to how much I can get around this factor of 100 in my own code…

Assuming we reach an impasse here, I might reformulate this problem and present it to the people at StackOverflow, who have been very helpful on obscure issues like this in the past.


#14

When Juce wants to repaint a component, it tells the operating system that the bounding rectangle for the component is dirty. If Juce does this two or more times for disjoint rectangles, before relinquishing control in its event loop, then we have a complex region requiring an update instead of a simple region.

With considerable effort it would be possible to add a sideband RectangleList to the Juce ComponentPeer which tracks dirty rectangles generated via repaint(). The problem comes when rectangles become invalid from outside the application. Such as when another window is dragged over your window. Or even with the same application. For example, you pop a modal dialog box over your main window and the user moves it.

Yes and no. From the perspective of Juce, the rectangles are combined to form a larger rectangle. However, the Core Graphics context (operating system one, not the Juce one) has its clip set to the region formed from the two rectangles, rather than the simple bounding box. The problem is that there is no way to extract these rectangles

Try calling ComponentPeer::performAnyPendingRepaintsNow() right after you call repaint() on your clock widget.

It’s not practical to do that cleanly.

For almost any game, the entire screen is redrawn for every frame, so non-rectangular update regions are a non-issue.

Also, consider a scrolling waveform. The whole thing redraws with every frame. How would you handle that? Clearly, the code that draws the waveform needs to be fast enough to handle being able to be fully redrawn in every frame. I think this is the approach you should be taking. This way, the update region is not an issue.

I’m with you on this one. But it might be worth spending 20 or 30 minutes reviewing the CoreGraphics APIs and see if there are any recent changes to support extraction of clipping rectangles. You could try posting in an Apple developer forum, or asking your question in #macdev on EFFNet IRC. At the very least, if you come up with some new information, it might be enough for Jules to implement a fix. As this could improve the performance of any Juce app for OS X, I believe it would have value.

You need to improve your waveform drawing code so it can fully repaint in every frame with no appreciable slowdown.


#15

All right, this has been incredibly valuable!

I think my strategy is the following:
[list]
[] Get all my rendering into my cache.[/]
[] Experiment with ComponentPeer::performAnyPendingRepaintsNow().[/]
[] Write this problem up and post it in wider circles.[/][/list]

Thanks for so much useful information and the sounding board…


#16

#macdev guys are saying that AppKit will merge the disjoint rectangles into a single rectangle anyway, under most cases.

Try #define USE_COREGRAPHICS_RENDERING 0 in AppConfig.h and see how that treats you.

I’m looking at the juce_mac_CoreGraphicsContext.mm and juce_mac_NSViewComponentPeer.mm and to be honest it looks like it already does as much as it can. It is possible to extract the rectangles from the CGContextRef. This is being done in getClipRects() for the software renderer. For the Core Graphics renderer its not necessary since Core Graphics is doing the clipping and its already set.

If the juce::Component optimization for opaque children is not working it means that AppKit is indeed merging the disjoint rectangles.

This all affirms what I said earlier, that you need to make your waveform drawing code FAST - it’s not a problem with the system.


#17

FWIW, at least oneunderlying issue is that OS X must combine certain rectangles for core graphics rendering. Because Quartz 2D is a sub pixel rendering engine, not combining certain types of rectangles (ex. overlapping) would result in visible artifacts. In my experience, the combinatorial logic is usually pretty solid. Look at the Juce Demo. The live o-scope display on the audio recording page invalidates and draws can be traced and look very logical (and also seem to have good performance).

In general, even in non Juce Mac/iOS development, the only times that I generally have run into trouble is when things overlap. For example, I can stick a dummy control on the Demo audio page and use a timer to update nonsense to it, and performance still looks good. But if I place it partially under the live audio component there is a pretty big penalty in extra rendering.


#18

Thanks guys, lots of excellent descriptions on here!

Just to summarise: CoreGraphics actually does a great job of handling the regions internally. But they don’t provide an API function that can find out whether a given rectangle intersects its (complex) clip region. So all I can do is to get the overall bounding rectangle of the entire clip region, and see if my rectangle intersects that, which is obviously going to give some false positives.

I did have a look in the 10.7 SDKs to see if there’s anything new in there, but they still don’t seem to have added anything that does this.


#19

I posted the question to StackOverflow:

How to determine if a rectangle interscts the current CoreGraphics clipping region?

Fingers crossed!


#20

[quote=“jules”]Just to summarise: CoreGraphics actually does a great job of handling the regions internally. But they don’t provide an API function that can find out whether a given rectangle intersects its (complex) clip region. So all I can do is to get the overall bounding rectangle of the entire clip region, and see if my rectangle intersects that, which is obviously going to give some false positives.
[/quote]

Maybe this flu/cold thing still has me befuddled, but now I’m intrigued. Just to make sure I understand, your headache is not basic Cocoa drawing using NSView’s getRectsBeingDrawn:count: (or needsToDrawRect:), but when the CGContext has a complex clipping path set, and you are stuck calling CGContextGetClipBoundingBox for the current context, correct?

What I don’t understand is how this translates to a big performance hit. This is still presumably in the drawRect for a given component Peer instance. The drawing itself should still be clipped. So unless the component is doing something computationally intensive in response to paint, it’s hard for me to see how this translates into the huge performance drags I’m seeing reported.