Direct2D preview branch

[matt]

Hello @fuo-

Thanks for digging into this and for testing the new renderer. Please see below.

• setBufferedToImage(true) seems to cause strange behaviors, like component popping in and out of view. I do have component buffered to image that are child of other that are also buffered to image, so that might be the cause. I did not go to the bottom of it.

Could you please provide a specific code example?

• g.getInternalContext().getPhysicalPixelScaleFactor() does report the display scale and ignores any affinetransform scale.

It’s not entirely clear to me what the correct behavior is for that function; that may nor may not be right.

• Image getPixelAt and setPixelAt methods are super slow (like 1ms per call), while Image::BitmapData getPixelColour and setPixelColour remain fast. This seems to be due to initialiseBitmapData, which is called when a new BitmaData is created for each and every get/setPixelAt call.

I recommend not using getPixelAt or setPixelAt with GPU-cached images; if you need to directly edit the image data, use a BitmapData instead. getPixelAt and setPixelAt is forcing the Image to be mapped back to the CPU and unmapped with each call. I don’t think there is a good way to speed that up.

I need to put together a more detailed post on this topic.

• fillRect with a width or height smaller than 1.0f do tend to disappear at times with AffineTransform scale smaller than 1 (popping in and out of view when scaling the UI). This is also independent of the display scale ratio, as even with a 200% display scale any rectangle with a dimensions smaller than 1.0f will face this problem, even if they are effectively almost 2 pixels thick.

Interesting; I’ll try that out.

This can be solved by using drawLine, but it is less efficient, and the doc explicitly recommends against it :

That should work. Please note that many of those recommendations in the are specific to the software renderer and may not apply to the new renderer.

Matt

[matt]

OK, that makes sense. DPI scaling for VST child windows has been troublesome.

Matt

[caustik]

I have a general question about this Direct2D work. If you’re currently developing plug-ins which are heavily based on OpenGL (3.2), how much interop is there between the two? Can you utilize OpenGL render targets and shaders while also taking advantage of the Direct2D stuff?

[fuo]

Thank you for your response!

It’s not entirely clear to me what the correct behavior is for that function; that may nor may not be right.

AFAIK all the other renderers do include AffineTransform scale in the calculation.

I recommend not using getPixelAt or setPixelAt with GPU-cached images

In that case I think it should be made clear, possibly with a jassert, that these methods are to be avoided with the Direct2D renderer.
I do use them to prepare a few images during the initialization of my plugin.
The overhead seemed okay with the existing renderers and the process felt instantaneous, so I did not look twice. When I switched to the Direct2D renderer it simply stalled, making it feel like it was simply hanging with high CPU and GPU consumption and no window shown. I had to kill the process manually. Now it appears I might have just wait for around 25 minutes instead

I don’t pretend that using these methods is good practice, but they are there, with no warning in the documentation, and they do work well enough with the existing renderers for non-critical tasks. They are certainly used in many plugins out there, and this might cause issues when switching to the Direct2D renderer if things are not made explicit enough, or a workaround is found.

[matt]

You can nest a child window using OpenGL inside of a parent window that is painting with Direct2D. Otherwise, I don’t think there’s much interoperability.’

Matt

[matt]

I think the question of how getPhysicalPixelScaleFactor should function will shake out in the integration with JUCE 8.

I agree that an assert for getPixelAt and setPixelAt would be a good idea.

Matt

[fuo]

I hope backward compatibility will win over semantic here, as this would be a rather silent and difficult to track modification.
The affine transform scale is also much easier, faster and more precise to get for the renderer as it has access to the transform matrix, compared to having to go through the whole hierarchy with juce::Component::getApproximateScaleFactorForComponent(this)

[matt]

Hi @fuo-

I was able to reproduce your problem and I tried modifying getPhysicalPixelScaleFactor per your recommendation. However - in my test code, the path thickness was rendered incorrectly even at 100% DPI scaling. For example:

    Path p;
    p.addCentredArc(20.0f, 20.0f, 10.0f, 10.0f, 0.0f, 0.1f, 5.0f, true);
    g.setColour(juce::Colours::red);
    g.addTransform(AffineTransform::scale(2.0f, 2.0f));
    g.strokePath(p, PathStrokeType{ 10.0f, juce::PathStrokeType::mitered }, AffineTransform::scale(2.0f, 2.0f));

Software renderer at 100% DPI scaling:

Direct2D renderer at 100% DPI scaling:

I changed Direct2DGraphicsContext::drawPath to set the device context transform before drawing the path with this result:

I just pushed that change along with another commit adding an assert checking for mapping small image areas. Please give that a try.

Matt

[anthony-nicholls]

@fuo

I hope backward compatibility will win over semantic here, as this would be a rather silent and difficult to track modification.

Backwards compatibility is an important consideration for the JUCE team however there will be times that we need to make breaking changes, when we do it’s not done lightly. If we knowingly make a breaking change then we’ll always do what we can to warn users, that may be with compile-time warnings, jasserts, an entry in the breaking changes document, or general documentation. In this case it may well just be a small oversight within this colossus chunk of work. It’s worth highlighting that right now the Direct2D is still on a preview branch for this very reason, it allows us to address any of these issues before it becomes widely used so we really appreciate users such as yourself checking the branch out reporting back with your experience.

@matt I just wanted to take a moment to say thanks for the continued hard work. The Direct2D renderer is clearly going to be a huge benefit to almost all JUCE customers going forward and you’ve continued to take on a lot of feedback, bugs reports, and investigations over the last few months all of which have contributed to improving on the already amazing results you’ve achieved

[matt]

Wow, Anthony, thank you. That means a lot to me.

I really appreciate everyone taking the time to test the renderer, provide feedback, and hunt bugs.

Matt

[matt]

The JUCE team is now in the process of reviewing and integrating Direct2D, which to me is very welcome news.

I anticipate that as part of this process they’ll be considering how to handle issues such as DPI scaling and backwards compatibility. Scaling has proven especially tricky; JUCE and Direct2D are mostly a close one-to-one match except for certain aspects of scaling bitmaps. I’m more than happy to put my pencil down and leave the decisions to the team.

Matt

[matt]

Also - thank you @fuo for identifying the path scaling issue and for your detective work. That was very helpful.

Matt

[evanberard]

I am working on a plugin on Mac with heavy graphics calculations and it took a huge amount of optimization to make it buttery smooth. However when I brought it to windows, the UI was so choppy it was unusable.

Switching to the JUCE direct 2d renderer basically made it buttery smooth on windows as well. The difference is insane. Amazing work. I’m also running into a scaling issue when zooming a component but I’m looking forward to seeing that resolved!

[matt]

Hi @evanberard -

Thanks for trying it out; very glad to hear about your results.
Could you please send more details about your scaling issue?

Matt

[remaincalm]

still working great here - thanks again matt, excellent work

[fuo]

Hi @matt

That is working great, thank you!
I really want to keep using this branch as it performs so well, and I can get rid of all those pesky setBufferedToImage calls

If getPhysicalPixelScaleFactor is reworked to only return the display scale factor, then it would be good to have another method to get the transform scale factor (or the whole display + transform) directly from the graphic context instead of having to call juce::Component::getApproximateScaleFactorForComponent

Here is an example of the few differences I could spot. I don’t know how to switch to the software renderer so I chose the openGL one as a point of reference, as it is very close to the software one.
MainComponent.h (7.3 KB)

You can rescale the window to change the AffineTransfrom scale factor, and switch between direct2D and openGL renderers by clicking the button on top.

[matt]

Hi @fuo - I’m looking into getPhysicalPixelScaleFactor some more. I’ll check out your example.

Here’s how to select the renderer:
Home · mattgonzalez/JUCEDirect2DTest Wiki · GitHub)-Renderer-selection

mattgonzalez/JUCEDirect2DTest/blob/pips/PIPs/Select Renderer.h

Matt

[matt]

Hi @fuo-

Excellent example code! I converted it to switch between the software renderer & D2D.

The 1 pixel lines were due to floating-point rounding; that’s sorted out. Same with your “remove from int” rectangle seams.

The “remove from float” rectangle seams are also due to float rounding; for me, they look the same with the SW renderer and with the D2D renderer. So I’m not sure that’s actually a bug.

Good catch on the joint style; that was an easy one to fix.

I’ve seen the shifted gradients before, especially when applying transforms to gradients. Those are still eluding me; I don’t fully understand some of the code in the software renderer that positions the gradients (GradientPixelIterators::Linear).

Matt

[matt]

@fuo

If getPhysicalPixelScaleFactor is reworked to only return the display scale factor, then it would be good to have another method to get the transform scale factor (or the whole display + transform) directly from the graphic context instead of having to call juce::Component::getApproximateScaleFactorForComponent

Could you please modify your example to demonstrate the issue with getPhysicalPixelScaleFactor?

Thanks-

Matt

[fuo]

The 1 pixel lines were due to floating-point rounding; that’s sorted out. Same with your “remove from int” rectangle seams.

Great! I think this will solve a lot of the visual issues I have with the plugin I am working on.

The “remove from float” rectangle seams are also due to float rounding; for me, they look the same with the SW renderer and with the D2D renderer. So I’m not sure that’s actually a bug.

Indeed, that was just a point of comparison with the “int” one.
That said, if this can be “fixed” then that is probably something that should be discussed for JUCE 8, as this is a constant source of issues when dealing with float rectangles (not only using the removeFrom methods) with different scales: how they do not cover the space in a complementary manner, and also how they do not always line up properly with int rectangles, even when the positions and sizes are identical.

Could you please modify your example to demonstrate the issue with getPhysicalPixelScaleFactor?

You can add this to TestComponent::paint :

		auto scaleTestBounds = bounds.removeFromTop(60);

		{ // scale
			auto physicalPixelScale = g.getInternalContext().getPhysicalPixelScaleFactor();
			auto displayScale = juce::Desktop::getInstance().getDisplays().getDisplayForRect(getScreenBounds())->scale;
			auto affineTransformScale = juce::Component::getApproximateScaleFactorForComponent(this);

			g.setColour(juce::Colours::black);

			g.drawText("display scale: " + juce::String(displayScale), scaleTestBounds.removeFromTop(15).reduced(20, 0), juce::Justification::centredLeft);
			g.drawText("affineTransform scale: " +  juce::String(affineTransformScale), scaleTestBounds.removeFromTop(15).reduced(20, 0), juce::Justification::centredLeft);
			g.drawText("final scale: " + juce::String(displayScale * affineTransformScale), scaleTestBounds.removeFromTop(15).reduced(20, 0), juce::Justification::centredLeft);
			g.drawText("context scale: " + juce::String(physicalPixelScale), scaleTestBounds.removeFromTop(15).reduced(20, 0), juce::Justification::centredLeft);
		}

Now for some context, I do need the final scale so I know if I need to regenerate costly assets (eg circular gradients).