Drawing Signed Char Data efficiently via Image class

Hey all,

I am attempting to draw an image via JUCE at a high FPS. Currently, I do it via the paint() call using:

if (m_hasUnRenderedDrawData)
{
Image myImage(Image::PixelFormat::RGB, Image.Width, Image.Height, true);

  for (int i = 0; i < Image.Width*Image.Height; i++)
  {
  	myImage.setPixelAt(i%Image.Width, i / Image.Width, juce::Colour(Image.Data[i*3], Image.Data[i*3 + 1], Image.Data[i*3 + 2]));
  }

  g.drawImageAt(myImage, 0, 0);

  m_hasUnRenderedDrawData = false;

}

However, although this works, this obviously takes a painful amount of time to constantly draw pixel by pixel. Is there any way to dump this data efficiently and effectively into the Image class to be able to draw it?

Ouch! Yes, that’s definitely not going to be efficient!

Best way to draw into an image is to create a Graphics object with the image, and just use that to draw with normal 2D vector operations.

If you really need to poke individual pixels for some reason (and think carefully about whether this before deciding you do), then that’s what the Image::BitmapData class is for - you can find examples of how to use it if you grep the codebase.

Your paint method looks muddled anyway - the way you’ve written it makes the Image look redundant to me. I guess you’re trying to cache some data to an image and have got your logic a bit confused. But if that’s true, and all you’re really doing is trying to cache something as an image, why not use Component::setBufferedToImage instead, and avoid having to create the Image yourself.

Jules,

Much appreciated.

I have a background render which generates that signed char data and then I need it displayed on the screen. The generation of new data (signed char[]) occurs at 30fps - so I’d like to display it as fast as it comes in. When the data comes in, I trigger the m_hasUnRenderedDrawData flag, so I only paint if it changes. The same data block is always written to, so same pointer always (in example above/below, Image.Data is a pointer to the image data and will never change location).

Taking what you said, it doesn’t appear that placing an image into a graphics object and then drawing the graphics object will make much of a difference, correct?

I looked at using Image::BitmapData but as you said, that’s if I’m going to do a pixel-by-pixel type of operation when the reality is that I just want to assign the Image::BitmapData data pointer (uint8) to point to the signed char buffer that I already am using. Placing the Image as a member, m_imgFrame, Doing something like:

Image::BitmapData temp(m_imgFrame, Image::BitmapData::ReadWriteMode::readWrite);
temp.data = (uint8*)Image.Data;

Is what I would hope but it’s not actually going through - am I just doing something stupid?

You can’t provide your own bitmap data for the Image class (well… technically yes you can, but that’s a much more complicated process). But you can just get an Image::BitmapData object and use that to quickly copy into the image on your background thread.

Jules,

Can you explain that process? I can’t see via the API on how to perform that operation and looking through the code examples isn’t exactly 1-1 with what I’m expecting.

For me to generate the BitmapData object, I already need the image in place - which means the image data is already there? If I can’t reassign the BitmapData.data member like I attempted to above, how can I use the BitmapData class to overwrite the Image data?

Just copy your own data to the bitmapdata using a mem copy.

No, you can’t just replace the pointer! (Think about it: The image could be using openGL or some other native format!)

The docs for that class are pretty clear:

/** Retrieves a section of an image as raw pixel data, so it can be read or written to.

    You should only use this class as a last resort - messing about with the internals of
    an image is only recommended for people who really know what they're doing!

    A BitmapData object should be used as a temporary, stack-based object. Don't keep one
    hanging around while the image is being used elsewhere.

    Depending on the way the image class is implemented, this may create a temporary buffer
    which is copied back to the image when the object is deleted, or it may just get a pointer
    directly into the image's raw data.

    You can use the stride and data values in this class directly, but don't alter them!
    The actual format of the pixel data depends on the image's format - see Image::getFormat(),
    and the PixelRGB, PixelARGB and PixelAlpha classes for more info.
*/

…not sure what you’re struggling with?

Yes, you do copy your own data to it, but don’t use memcpy!

You need to take into account pixel format, pixel stride and line-stride, so unless you handle every edge-case yourself, the safest method is to just use Image::BitmapData::setPixelColour to write to it, which is probably fast enough for most use-cases. (And if you have LTO then the compiler may even be able to inline that method and hoist its loop invariants which would make it pretty much optimal code anyway)

Jules,

So if I hear you right, for every frame update I receive:

Image::BitmapData temp(m_imgFrame, Image::BitmapData::ReadWriteMode::readWrite);
	for (int i = 0; i < ImageData.Width*ImageData.Height; i++)
	{
		temp.setPixelColour(i%ImageData.Width, i / ImageData.Width, juce::Colour(ImageData.Data[i * 3], ImageData.Data[i * 3 + 1], ImageData.Data[i * 3 + 2]));
	}

	m_hasUnRenderedDrawData = true;
	repaint();

while in paint():

if (m_hasUnRenderedDrawData)
{
g.drawImageAt(m_imgFrame, 0, 0);
m_hasUnRenderedDrawData = false;
}

Doing this, I see a speed increase but still not to what I would consider real-time. If this matches what you were expecting, then we might have to do that really complicated route you had mentioned to directly place my data into the object?

(That’s a very strange way to write an image copying loop, it’s traditional to use two nested loops for x and y!)

Have you actually profiled a release build and shown this to be the bottleneck? Because if not, there’s really no point in trying to guess how to optimise it.

The complicated route is to write your own ImagePixelData class, but it won’t necessarily be any better, it depends what’s actually taking the time, and profiling is the only way to find out.

Jules,

I know - I’m a little out of the ordinary ;].

I just profiled it and you were right - the above operation takes roughly 15-30ms so that definitely isn’t my bottleneck. I went ahead and moved my function calls to an AnimatedAppComponent class and trigger it periodically at 10fps and that appears to work - generally speaking. Now, I am wondering why I had to do this as opposed to triggering the paint() call separately from mouseDrag() which should be working even better and faster…

If I use the setFramesPerSecond(10) - it works, if I set it to 15fps, it doesn’t - just appears frozen. I would hope that I can have a more versatile solution - one that repaints as quickly?

Again, I think you’d need to profile to see what’s going on, but just shuffling a few pixels around with the CPU shouldn’t be a problem as long as you’re not doing anything silly. Are you doing all this on a background thread?

I was doing it through an AnimatedAppComponent for the above by setting the FPS there - but I think I got it. I think what is occurring is that my render drops when it receives calls too quickly - as in, if I call to update the image data and then call it again before it finishes, as in the case where I am doing mousedrag, then it “hangs” and therefore appears as though it’s occurring really slowly when in fact, it is just dropping frames.

I’m still investigating this but I think that’s the main part - that MouseDrag() can call itself multiple times before even the first function call finishes - I could do some mutex work here but who knows.