[Solved] Better Repaint Debugging

(Edit) – This was a feature request but was later discovered to be impossible.

Here are some steps for better debugging:

  1. The JUCE debug repaint IS showing you the dirty regions the UI has decided to update for your app, it is NOT showing you the full rectangle the OS told you to draw.
  2. You can see the rectangle only by using Quartz Debug Tool which you can get from apple developer downloads

Between the two you can see a) the rectangle which is repainting in your app, anything inside or touching this rectangle repaints. b) anything which flashes with repaint debugging on actually was redisplayed to the the screen.

If your UI code is fast and you’re getting inexplicable frame drops, it COULD be from the repaint throttling happening in the NSComponentPeer – although it’s not clear why apple is dropping your dirty regions from repainting, and there’s no real transparency to figure it (that I’ve found)

More info on this below

Hey! I’ve been migrating from OpenGL to CoreGraphics because of rendering issues on M1 which appear to be more solid with CoreGraphics – the issue is that I’m still having some lag in certain view of my UI.

Upon further investigation I’m finding that CoreGraphics is passing a massive clip region to top component regardless of my “dirty” component bounds and regions, and it appears to be due to how rectangles are merged etc etc – I know this has been discussed at length – but it would be great if there was some way to debug the true clip region that goes to the OS on our views, this way we could much more easily debug the true repaint strategy that’s occurring on the OS level.

Even with a great repaint strategy – the way everything gets merged seems to just fail at the highest layer, and it doesn’t appear: JUCE_COREGRAPHICS_RENDER_WITH_MULTIPLE_PAINT_CALLS does anything to improve the massive clip region which is produced.

I think I need to also just a better understanding on how the view stuff happens, is there a reason we group all the clips into a single handlePaint on the highest level – it seems it would be much faster to just have more paint calls with smaller regions?

A bit more info using this setup as an example:

-- REPAINT CALLED --
X: 1027
Y: 286
W: 34
H: 126
DEFERRED REGION NUM RECTS: 1
-- REPAINT CALLED --
X: 370
Y: 286
W: 367
H: 142
DEFERRED REGION NUM RECTS: 2
-- REPAINT CALLED --
X: 46
Y: 286
W: 34
H: 126
DEFERRED REGION NUM RECTS: 3
---CLEARED DEFERRED PAINTS---
----handlePaintClipBounds-----
46
286
1015
142

^ Would it be fair to assume that with: JUCE_COREGRAPHICS_RENDER_WITH_MULTIPLE_PAINT_CALLS – it would mean this would result in 3 handlePaints with different regions? – I think another part that might have me confused is that the frame rate between the different elements in this single large render call are different – so even though it appears to be handling all their paints at once in the code – it’s not what we see on screen?

So I suppose this is half request for better visualization and half me trying to figure out what the heck is up 0.o – I’ve never seen just part of my UI have a lower frame rate than the others

There’s a difficulty here because (at least last time I checked) even though JUCE is called to repaint a large rectangle only the actual dirty areas are then updated by Apple, so making a true repaint debugging mechanism seems impossible with Core Graphics :frowning:

1 Like

I think you have to use the Quartz Debug tool to see the actual areas that are repainted by CoreGraphics.

Will it shows the areas that JUCE emitted paint calls for? Those that weren’t in the actual non-unified dirty rects?

Thanks for the tip – I just downloaded this I hadn’t heard of it – I’ll give it a shot today. I’m not sure what can explain the frame drops I’m seeing as it doesn’t appear to be from the repaint throttling, but more from this massive clip region, although I can’t work out how / why.

When I get some more info I’ll report back as I’m pretty much decided on getting out of OpenGL

The confusing thing about that @yairadix is I don’t see any sort of “dirty needs repaint” type flags like that – it looks like children would determine if they need to repaint based on the clip region, however the repaint debugging does flash in the components paint method. Do you know where that dirty area is tracking internally around the repaints? Having trouble spotting that.

Thanks for that @dave96 – was exactly what I was looking for – And I think I’ve found either a bug or something strange. The behavior I see is:

  • Repaint called on component
  • CoreGraphics Context Passed into paint routines which cover said component
  • Component is repainted
  • Component is not updated on screen

I’m trying to figure out how exactly the repaint debugging functions since it’s on the ComponentPeer level and not the components themselves when their paint is called:

    {
        // enabling this code will fill all areas that get repainted with a colour overlay, to show
        // clearly when things are being repainted.
        g.restoreState();

        static Random rng;

        g.fillAll (Colour ((uint8) rng.nextInt (255),
                           (uint8) rng.nextInt (255),
                           (uint8) rng.nextInt (255),
                           (uint8) 0x50));
    }

It seems like g.fillAll is magically being called with a clipped region which is different than the original graphics context – which means there’s some sort of clipping which occurs and persists as the rendering moves from parent to children – which is making me believe that for some reason my child is rendering and then not being included in the final clip context which actually updates and re-renders for some reason even though it’s had repaint called?

This is what I described before.
JUCE is told to repaint a big rectangle while actually only the true dirty regions get updated (sent to the GPU to the appropriate texture), and this is why the current repaint debugging is misleading because JUCE’s coloring of regions doesn’t show on screen.

That’s correct. Unfortunately there is no way to work around this when rendering into the CGContext provided by macOS.

We have an alternative rendering approach in development. In general it’s slower to render the same region, and isn’t compatible with CORE_GRAPHICS_RENDER_ASYNC, but it will avoid CoreGraphics consolidating dirty regions. This will be a big net gain in some situations.

3 Likes

Thanks for more info guys – but I still think there’s something that not functioning as expected in my app – If I could get a little more info I think it can track it down, but it still feels like there is some magic happening I don’t understand.

So in the NSViewComponentPeer – we’re allowing the context to be be created via CoreGraphics which correctly matches the NSRect r passed into the function, although I don’t see how the low level graphics object is able to deduce that as it actually appears to not utilize r in it’s construction – which makes me believe there’s some sort of internal clip region history management or something? I’m talking line 921:

            const auto height = getComponent().getHeight();
            CGContextConcatCTM (cg, CGAffineTransformMake (1, 0, 0, -1, 0, height));
            CoreGraphicsContext context (cg, (float) height);
            handlePaint (context);

^ how does the context here result in the same r as passed into draw rect? When I check the context clip region it’s correct.

Then handle paint happens and it looks like to me, components are drawn based on this large clip region – not any sort of true dirty regions which is totally fine, my graphics are really fast! I don’t think I have a rendering bottleneck, all my paint functions are sub milliseconds, or a 0 when I check the difference of the current time in ms at the start and end of the function.

With repaint debugging on – at the end of the function we fill the current clip region of the graphics context:

    if (JUCE_IS_REPAINT_DEBUGGING_ACTIVE)
   #endif
    {
        // enabling this code will fill all areas that get repainted with a colour overlay, to show
        // clearly when things are being repainted.
        g.restoreState();

        static Random rng;

        g.fillAll (Colour ((uint8) rng.nextInt (255),
                           (uint8) rng.nextInt (255),
                           (uint8) rng.nextInt (255),
                           (uint8) 0x50));
    }

^^ here is where I’m confused – Again – the clip region of the graphic context matches the large rect passed in from the top level – however – when we call fillAll on the graphics context, only the bounds of the component which had repaint actually called flashes – so I’m confused, is the ComponentPeer updating only a single component per render call regardless of the dirty region? How is it that the flashing section here doesn’t match the bounds of the clip region?

What’s happening in my app is I’ve called repaint, repaint is called, and just one component, in between two other components, is not updated on the screen, although there doesn’t appear to be any frames dropped as everything around the slowed down area is rendering fast with no issues.

Any insight into the magic there? I’d like to figure out why that region isn’t updating as it doesn’t appear to be a performance or throttling issue.

Best,

J

Are you sure that both r and the context clip region are ‘correct’?

I think you’ll find that you will have previously marked multiple smaller regions as dirty in setNeedsDisplayRectangles, and it is only these regions where CoreGraphics will actually update what is showing on the screen. Both r and the context clip region will be a larger rect that encloses the smaller ones that actually get rendered.

1 Like

Thanks @t0m – yeah it actually doesn’t appear to be related to the context itself at all, all that stuff seems fine – the confusing part at this point is that I see repaint was called on the component I request repaint on, I see it included in the clip region from the OS – I see it flashing in quartz debug, but I do not see it flashing with repaint debugging on.

So I’m just trying to work out the little bit of magic here which is how the repaint debug rects works, when the clip region of the context, isn’t the same as the square I see flash during the component peer repaint.

If each paint of the component peer is flashing a single rect – doesn’t that indicate that only a single component is being repainted in our JUCE layer regardless of the clip region? i.e. if this massive clip region is coming in as the context clip bounds, how does only the single components region reflash.

Realizing now that fillRect – might have some sort of complex shape inside of it with cutouts of the various components which were painted?

(sorry for the edits)

There is no magic in the repaint debugging.

Given that getRectsBeingDrawn no longer works, querying the context for the clip region (or r) is the only way we can get information about clipping from the operating system. So JUCE’s clip region is based on that and we use that clip region to select which components need to be repainted.

However, the actual clip region in the context may be smaller in area than that obtained from CGContextGetClipBoundingBox (or r). If you attempt to render anything that is outside of the context’s internal clip region, but still within JUCE’s clip region, it will not be displayed in the context.

1 Like

Thanks @t0m I get it now – we call fill all on the context, that’s the full rect, but the OS is still only displaying it’s internal rects it’s tracking as dirty which its not allowing us access to querying thus the draw for each individual rect doesn’t work. – Also, it’s impossible to display this region from within JUCE.

I think the conclusion I’ve come to is I’m still experiencing:

// In 10.11 changes were made to the way the OS handles repaint regions, and it seems that it can
// no longer be trusted to coalesce all the regions, or to even remember them all without losing
// a few when there's a lot of activity.

I think I’ll try some things like synchronizing when repaint is called on all of the components as a group, and potentially even coalescing their rectangles myself and then calling repaint on the the full group to ensure that all of them get into the clip region together and guarantee they’re painted as a group and not dropped.

Interesting stuff! Thanks for the patience and explanations!

As mentioned above QuartzDebug is probably the best way to see what is actually being drawn to the screen, this should allow you to see the coalesced rectangle, and give you some idea of the components that are likely being repainted. I’ve never found the JUCE paint debugging tools to be overly helpful and normally misleading. The is all based on my experience, and it has been a while since I’ve had to do this.

One thing I recommend doing is making sure every component has one of two jobs. Either, it has child components, or, it draws. In other words only leaf node components should implement the paint method.

For example avoid components that draw a background and have child components. Instead create a separate background component that is a sibling to the other child components.

Once you’ve done this, for all components that don’t change often and the drawing is more expensive than drawing an image, call setBufferedToImage (True) on them.

Note, If you call setBufferedToImage (True) on a component that does need to repaint often (say a meter), it may make things worse. This is because the image will need to be redrawn and then drawn to the screen, making it more expensive than just drawing it straight to the screen.

However, if something doesn’t change often (like a background), but it is constantly being redrawn (due to a coalesced rectangle), then setBufferedToImage (True) will mean most of the time it can just redraw the image, thus avoiding calls to the components paint method, as this only needs to be called to repaint the image.

Unfortunately if a child component has to repaint, even if the parent is set to buffer to an image, its image will always be redrawn before being painted to screen. This is unintuitive and it means calling setBufferedToImage (true) can actually make things worse! I can’t remember the exact reasoning why it behaves this way, but this is the reason I say only allow leaf node components to draw. Essentially if you stick to this rule it will be easier to make optimisations using setBufferedToImage.

Other optimisations that may help.

  • Always call setOpaque (true) for components that have no transparency. IME this is quite rare, but if it’s true for you it could prevent some paint methods being called.

  • Call setPaintingIsUnclipped (true) on all components that don’t draw outside their bounds. IME this can (and should) apply to almost all components. If you have a lot of components this could at least reduce the cost of repainting.

3 Likes

Yep!

That all sounds correct – except there can be some strange stuff with painting unclipped and affine transforms which I haven’t really tested.

For me – what I’ve currently found is that my graphics are fast & render no problem, CPU usage is absolutely tiny, but I’m still getting dropped frames calling repaints at 30FPS.

For me – it’s the throttling, although it doesn’t make sense at all that it doesn’t work, and it’s very difficult to pinpoint why it’s a problem, cause in theory it looks fine:

    static bool shouldThrottleRepaint()
    {
        return areAnyWindowsInLiveResize() || ! JUCEApplication::isStandaloneApp();
    }

In core graphics, repaints are throttled when you’re not running in standalone. I do most of my development in standalone so it was driving me nuts to figure out why my plugins were running differently in the plugin wrappers.

Of course all the repaint regions are fine, and JUCE is doing everything in a way which makes sense, but for some reason when the repaint regions are grouped and sent as a big blast at 30FPS, vs just letting the UI handle more: setNeedsDisplayInRect – it doesn’t display all the regions which were sent in the deferred draws every frame – and it seems it’s impossible to figure out why.

SO AN INTERESTING NOTE IN 2022 – I started using the OpenGL renderer like 8 years ago and it was massively faster at doing animations than whatever the default render setup at the time was. However now – CoreGraphics with all of the optimizations you mentioned above is much faster than OpenGL for me, on both my Intel & M1 machines. (Forget about OpenGL renderer on the new M1s)

So – In my opinion we really need is to have some flags around whether to throttle the repaints – this has been discussed other places on the forum but never implemented.

I’m going to do my own manual QA to see if these logic rendering problems etc are issues for us, and of course it’s all weighing issues. With the throttling on by default – EVERY user experiences repaint issues regardless of DAW with my app in order to avoid issues only some customers would experience – so it’s really just that you need to weigh the problem. I would rather have all users experience smooth graphics by default, with the potential of us running too much paint and fighting with the DAW for some users, and then have a backup plan for them, rather than slow graphics for everyone.

Parameter automation was one reason why the UI repaints use shouldThrottleRepaint().
The host may sends hundreds of setParameter() calls in one second. In the past this triggered to many repaints and overloaded the UI thread. Not sure if this is still a problem with parameter attachments.

1 Like

We just changed JUCE code to always throttle and added a way to set the repaint rate. Right now we throttle repaint calls to 20hz. We save a lot of CPU cycles that way. The JUCE_COREGRAPHICS_DRAW_ASYNC is a very interesting flag as well as it tells the the NSView to draw to a backing layer on a different thread, thus freeing the JUCE message thread from it.

Yeah this was my initial solution as well so I had the same behavior everywhere, but I really don’t have any other explanation than with the throttling and a lot of animations, certain rects just don’t get updated, and it’s not any sort of CPU issue :man_shrugging:t3:

Alright – sorry to continue bring this thread back to life but I’ve finally found where I’ve gone wrong, how to retain the repaint throttling, and how to keep smooth graphics without worrying about dropped frames – and it may help others.

It turns out in my case, what’s happened is I’ve actually optimized a bit too far. For example, I have an audio analyzer running, which renders a path of the current freq plot. Now that’s a complex path and I only want to render it when it changes – so I run a timer, and in the timer check if the buffer has changed – if it’s changed, then I render the new path, simple enough.

What happens is with throttling and coalescing in CoreGraphics, it seems as though the OS just simply drops a rect here and there if you pass too many at once – in a “smart” repaint strategy, this causes more than just a dropped frame, cause you’re only repainting when things change. If it keeps happening it looks like you have horrible frame rate, but that’s not the case at all, you just have some unlucky rects being dropped.

So what’s the fix? Paths are much cheaper on CoreGraphics, just render them every frame!

If a rect gets dropped, who cares, it just renders it again next frame and looks great.

2 Likes