Something like drawing to a frame buffer? [SOLVED]

Hello,

So I want to play with different algorithms I have found on the web: fluid dynamics, particle systems, etc. Things that will process a 2D array, and then write out the results.

There was a Graphics::setPixel() but in this thread Jules claims it was only calling fillRect (x, y, 1, 1) anyway.

So I look in the documentation for the Graphics class, and it says

If you have a lot of rectangles to draw, it may be more efficient to create a RectangleList and use this method [fillRectList()] than to call fillRect() multiple times.

So armed with that information: I have a single component of setSize(200, 200); and put this into the paint method:

void paint(Graphics& g) override

{
    RectangleList<int> dots;

    for (int i = 0; i < cols-1; ++i)
        for (int j = 0; j < rows-1; ++j)
        {
            dots.add(i,j,1,1);
        }

    g.fillRectList(dots);
}

I call repaint() at 1Hz and my CPU goes into meltdown. App freezes.

I try adding the OpenGL module, adding an OpenGLContext context; to my component and setting context.attachTo (*this); in the constructor, but that makes no difference.

Am I doing anything wrong?

What’s the generally accepted right way of doing what I want to be doing?

I’ve attached the full source for completeness, but there really isn’t any more to it.

MainComponent.h (1.9 KB)

Maybe heavy reallocation during your nested loops? Y not use the image class for that? I suppose that’s better made for the job

Aha! That seems to have brought things back more into line!

Can you explain the allocation comment?

I’ve never used the Image class before. Does this look like correct usage? (ignore the -1 loop)

void paint(Graphics& g) override
{

    Image image{Image::RGB, cols, rows, true};

    for (int i = 0; i < cols-1; ++i)
        for (int j = 0; j < rows-1; ++j)
        {
            image.setPixelAt(i,j,Colour(0,0,0));
        }

    g.drawImageAt(image, 0, 0, false);
}

You should rather use Image::BitmapData::setPixelColour method.

2 Likes

And your cache will appreciate to swap the x and y loop, since the image is lined up in rows (having x as inner loop).

But using the BitmapData instead of setPixelAt() as @MBO pointed out, will make the biggest difference according to my experience.

But drawing into an image, just to draw it on the screen is really no improvement, it is overhead:

  • allocating memory for the temp image
  • copying the whole image in g.drawImageAt() (best case, worst case rescaling due to logical pixels)

So if you want to draw on the screen, best use the primitives directly on the component like fillEllipse, drawRect(). That is already the optimised way.

If you have a complicated drawing method, but you need to draw the same image multiple times, you can switch on setBufferedToImage() for your component, that will do that buffering in the background for you.

But I think for that kind of generated scenes you will benefit from learning OpenGL. Having said that, you can get quite far without, if you are mindful.

1 Like

Adding a Rectangle to the Rectangle list will require memory to store that Rectangle, and by default your Rectangle list will only allocate memory for so many Rectangles (let’s assume 8). When exceeding the space (9th Rectangle to add), the Rectangle list will allocate some more memory (let’s assume 16), copy the first 8 to that new memory location, free the previous memory. And that will again happen when you add the 17th Rectangle. Even if it would only store pointers, adding 40.000 Rectangles might need quite a lot of reallocation.

As you know the number of Rectangles beforehand, you might want to call ensureStorageAllocated(), this should also speed up things. JUCE: RectangleList< ValueType > Class Template Reference

Also your Rectangles don’t overlap, right? JUCE: RectangleList< ValueType > Class Template Reference This just adds the Rectangles without checking overlaps.

You should rather use Image::BitmapData::setPixelColour method.

@MBO, @daniel, could you show me how to use this? I can’t understand from the documentation alone.

And your cache will appreciate to swap the x and y loop, since the image is lined up in rows (having x as inner loop).

Thanks for the tip, although upon looking at the code I supplied again, i used variable names cols and rows (incorrectly), but i and j are lined up in row major - shout out to our own Timur for his Want fast c++? Know your hardware talk. - so dispite inaccurate naming, its actually iterating through optimally, right? My bad for naming it wrong.

Btw on that topic, should I just prefer a contiguous 1D array over 2D anyway? And access it with something like array[(x * width )+ y] and do away with this double array nonsense?


So if you want to draw on the screen, best use the primitives directly on the component like fillEllipse , drawRect() . That is already the optimised way.

But this is where I started, and it was creating crazy CPU load, no?


Thanks for your input @danielrudrich, very informative.

Based on your tips; here’s my paint function:

void paint(Graphics& g) override
{
    RectangleList<int> dots;
    dots.ensureStorageAllocated(cols*rows);

    for (int i = 0; i < rows-1; ++i)
        for (int j = 0; j < cols-1; ++j)
        {
            Rectangle<int> rect{i,j,1,1};
            dots.addWithoutMerging(rect);
        }

    g.fillRectList(dots);
}

This will yield around an 80% CPU load. (debug build) Definitely an improvement from the previously non-functional! But the previous method of using Image hits my CPU at about 67%.

Now if only I can figure out Image::BitmapData::setPixelColour :thinking:

It works like:

Image::BitmapData data (image, Image::BitmapData::writeOnly);

for (int y = 0; y < rows-1; ++y)
    for (int x = 0; x < cols-1; ++x)
        data.setPixelColour (x, y, Colour (r, g, b));

You get even more benefit, if you write into the data directly, but then you need to know the underlying format, and where the individual colour components are located. The reason for that is, that Colour is sometimes doing more work than you would expect it to do, e.g. colour space conversions.

But to repeat myself, prefer to use drawing primitives, that will be passed on to the backend if possible. You cannot optimise in user code better than what the backend does potentially using hardware acceleration.

By simply attaching an OpenGLContext to your component, the g.fillRectangle (rect.toFloat()); will be executed on the hardware.

class FastComponent : public Component
{
public:
    FastComponent()
    {
        context.attachTo (*this);
    }
    ~FastComponent()
    {
        context.detach();
    }
    void paint (Graphics& g)
    {
        g.fillAll (Colours::black);
        g.setColour (Colours::green);

        for (int i= 0; i < 1000; ++i)
            // draw your particle
    }
private:
    OpenGLContext context;
};
1 Like

From what I’ve seen, using OpenGLContext that way doesn’t necessarily ease the CPU load much, and can even make it worse. Lots of stuff related to the drawing will still be happening on the CPU and additionally the CPU will have to wait to send the OpenGL commands into the GPU.

It will probably not be as fast and streamlined than writing OpenGL code yourself, but I would claim it is definitely the better option than rasterise on the CPU into an image and draw that, what is what this thread is proposing.

Having said that, I didn’t benchmark myself, so take it with a grain of salt.

From my experience it’s platform dependent and also what is drawn. I did a quick test on macOS (mac mini 2012 i7 2.6GHz). I was filling a 500x500 rectangle with a solid colour using Image::setPixelAt, BitmapData::setPixelColour and - just for comparison - fillRect (i, j, 1, 1) with and without OpenGLContext attached. So, from fastest to slowest:

BitmapData::setPixelColour + OpenGLContext = 1ms
Image::setPixelAt + OpenGLContext = 2ms
BitmapData::setPixelColour without OpenGLContext = 4ms
Image::setPixelAt without OpenGLContext = 5ms
fillRect (i, j, 1, 1) + OpenGLContext = 15ms
fillRect (i, j, 1, 1) without OpenGLContext = 58ms

Times measured with Time::getMillisecondCounterHiRes().

I’m really surprised that Image::setPixelAt is not so slow as one could expect. Of course drawing pixels using fillRect does not make any sense - I just wanted to see what is a difference.

2 Likes

and the winner would obviously be

fillRect (0, 0, 500, 500);

I still don’t get the goal of the thread, since this is something you would never do by hand…

Exactly, it’s performed immediately.

Well, if you have a relatively small area to fill with some pixels, IMO it’s ok to use just BitmapData::setPixelColour or write directly to BitmapData::data…

@Daniel, you keep saying this, but my intention is to do particle systems, kernel/convolution matrices, fluid dynamics etc. and for that, a single fillRect of 500 by 500 obviously will not do! I need to process every pixel individually.

Thank you for your help @daniel @MBO with Image::BitmapData::setPixelColour it is indeed the fastest! Would you mind explaining a little bit the connection between BitmapData and Image (indeed feel free to make a comparison with audio if you like, i.e. is it something like what an AudioBuffer is to a AudioProcessor::processBlock()?)


@Xenakios is indeed right, every time I have enabled OpenGLContext (on an Aspire 5740 running Lubuntu 18.10) in the past, it has worsened the CPU load, However now with the newest method (Image::BitmapData) it has gone from 1 CPU core running at 50% to two cores running at 42% each. I’m not sure what is better!? - Especially if I aim to port this to android. My results are akin to @MBO however, differing with the OpenGLContext versions.

Again if anyone could enlighten me about Image::BitmapData that would be greatly appreciated. Many thanks for your help. I’m happy to call this thread solved! :smiley:

Image::BitmapData gives you direct access to the pixels in your image. Pixel components are written in a buffer byte after byte. So you can even get a pointer to this buffer Image::BitmapData::data and change for example only one colour component of a certain pixel (or its alpha channel). But remember that there are many pixel formats and JUCE supports directly three of them:
https://docs.juce.com/master/classImage.html#ab47b5746d2df286ae6f8da6af5463c01

If you need to draw your image once or refresh it from time to time it’s IMHO ok to use this method. If you need to draw and change your image on the fly etc. it is definitely better to render it using OpenGL (or OpenGLES on Android).

I know it’s a completely simple operation, once you understand how the pixel format is laid out, but that brings me to this more abstract point … I don’t know how pixel formats are laid out, nor where to go to find out.

Do you know of a nice resource I could consume to learn this? Perhaps maybe the equivalent of Tmur’s C++ in the Audio Industry but for graphics.

Luckily JUCE only supports 3 formats, see Image::PixelFormat

The enum in RGBA suggests the natural order:

enum { indexA = 3, indexR = 0, indexG = 1, indexB = 2 }

but when trying it turns out, it is the reverse order: BGRA (if I am not mistaken)

In reality, there are plenty more image formats:

  • Packed Pixel vs. Planar: each pixel is written “interleaved” or planar: all red values, then all green values, and so on
  • YCbCr: similar to the analogue video signal, it starts with a brightness, followed by colour information in blue difference to Y and red difference to Y (leaves green out, because that is inherently included) - it’s a bit like M/S stereo
  • Some formats will use a different number of bits for each component, see Chroma Subsampling
  • and probably many more in the quest for the best compression…

For the juce draw operations you can assume those three formats only, at the cost of only 8 bit colour values (sold as “millions of colours”) vs. in photography and print some would wish for 16 bit per channel (sold as “billions of colours”).

Hope that helps, if you have questions in a specific direction, feel free to ask.

Btw. apologies, I was focussing with my previous posts to the particle system idea, where drawing primitives would be the preferred. But for image processing you are right, you need this kind of approach.

Thanks Daniel, this is great, but I want a complete understanding (and more than I can reasonably ask you to answer for me) i.e. from your post (and clicking through the documentation could tell me) there is an enum for this, but no idea how to use it.

I’m just guessing that Image::BitmapData.data (is it accessed with a .?) is a pointer to a contiguous array that based on what you wrote, with the RGBA pixel format probably holds data like this: {B,G,R,A,B,G,R,A,B,G,R,A,B,G,R,A} etc. so to modify every alpha pixel in the image, I would do something like

for (auto& i : data)
    data[(i+1)*3] *= 0.5f;

But I’m just completely stabbing in the dark here. And even if you do provide the answer for me, I still won’t be able to do anything more than what you have (graciously) provided me.

I need to learn how to fish, rather than be given one. I need something like learnopengl.com but for basic image processing in c++

I understand you. I studied image processing 20 years ago, so I don’t have the latest resources at hand, I’m sorry. And in terms of OpenGL/Vulcan so much has changed, my knowledge from back then is no longer useful since the invention of shader language, that happened meanwhile. So I need to learn too…

With your assumption you are totally correct, it is a raw array of the single components.

  • But they are unsigned 8 bit values rather than float.
    That means, either you convert each pixel to float and back multiplying by 255.0 and take care of overflow, since float can exceed 1.0, then the pixel would flip, or you stay in char values if possible.
  • often the data is accessed in individual lines, so the data can be memory aligned (i.e. start at an even address, based to I think 16). That’s why the BitmapData has a “getLinePointer()”

The loop looks slightly different, you would rather use the old style loop, because you are rather interested in the index than elements, and you want to process the individual components in each go.
Your example could be written:

for (int y=0; y < bitmapData.getHeight(); ++y)
{
    auto* p = bitmapData.getLinePointer (y);
    for (int i = 0; i < bitmapData.getWidth(); ++i)
    {
         *p = jlimit (0.0f, 255.0f, *p * 0.5f); ++p; // blue
         *p = jlimit (0.0f, 255.0f, *p * 0.5f); ++p; // green
         *p = jlimit (0.0f, 255.0f, *p * 0.5f); ++p; // red
         ++p; // alpha
    }
}

I am currently writing some effects, including colour lookup tables or a sobel filter, I can link it here later (it’s done but needs cleanup, was done in a hurry)

1 Like

UNBELIEVABLE!

That snippet of code, along with your advice was so helpful to me.

For the record, i think bitmapData.getWidth() and bitmapData.getHeight() functions don’t exist. There is however getBounds() that returns a rectangle, that probably is what you were looking for, but since I already keep track of the columns and rows this wasn’t a problem.

Here’s the best bit. I was able to put your advice from this thread together with your advice from the other thread I started about mixing colours in a Colour object, and first time, without any compiler errors, I was able to do the most advanced bit of pointer wizardry I have ever done… Behold this snippet of (probably very ugly) code (that will likely make me cringe in a couple of years)

for (int i = 1; i < rows-1; ++i)
    {
        for (int j = 1; j < cols-1; ++j)
        {
            uint8* c1 = buffer1Data.getPixelPointer(i-1,j);
            uint8* c2 = buffer1Data.getPixelPointer(i+1,j);
            uint8* c3 = buffer1Data.getPixelPointer(i,j-1);
            uint8* c4 = buffer1Data.getPixelPointer(i,j+1);

            uint8 newR = (*(c1++) + (*c2++) + *(c3++) + *(c4++)) / 4;
            uint8 newG = (*(c1++) + (*c2++) + *(c3++) + *(c4++)) / 4;
            uint8 newB = (*(c1) + (*c2) + *(c3) + *(c4)) / 4;

            auto* writeOut = buffer2Data.getPixelPointer(i, j);
            writeOut[0] = newR;
            writeOut[1] = newG;
            writeOut[2] = newB;

            // swap buffers
            std::swap(buffer1, buffer2);
        }
    }

I’m sure there’s something wrong there, for example, I think I probably want to swap the buffer after drawing the image to screen, but whatever… I did pointer magic and the compiler didn’t complain! :smiley:


edit: yup, that buffer swap was in the wrong place! :wink: