transformedImageRender() offset

valley · September 24, 2008, 7:47pm

Jules, transformedImageRender() seems to have an offset bug.

When scaling images using bilinear interpolation, the image is clearly transposed by one source pixel up and to the left. You can see this in the JUCE demo by flipping between low and high quality resampling.

I’m following the code through from Graphics::drawImage(), and I’m still getting lost working through the transformations, so I don’t know if you’re trying to get the offset right, and missing it further back, or whether you’re ignoring the offset entirely. If I can get that figured out, I’ll follow up.

Anyway, sidestepping the issue that transformedImageRender() doesn’t do boundary reflections or any other edge correction, the bodge fix which correctly aligns images resized using medium and low quality resampling is just to change:

929: double sx = srcX;
930: double sy = srcY;

to:

double sx = srcX - 0.5;
double sy = srcY - 0.5;

Since this may impact negatively (thunk) on other affine image transforms, I’d hesitate to leave it at that. I’m going to try and take a closer look at the JUCE code when I don’t have people pestering me with questions. So I’ll try and get a fix in, but you’ll have a much better grasp of this than I, so you’ll probably beat me to it.

jules · September 25, 2008, 4:49pm

I wonder why that’s the case… I haven’t time to pore over those routines at the moment, but if you think your bodge is sensible, let me know - if it works, then I can’t see a reason not to use it.

valley · October 1, 2008, 1:17am

OK, I’m about 90% sure that you need to subtract 0.5 from sx or sy in the bilinear scaling code.

Applying this offset definitely locks the resample into phase with the nearest neighbour code above. I’ve done a number of checks to make sure that the nearest neighbour interpolation is in fact producing truth, and it’s solid. The affine transforms are correct AFAICT, so I think you just forgot to correct for the kernel size in the bilinear code. I’m going to say go ahead and submit those offsets to the code base.

jules · October 2, 2008, 10:33am

Cool, I’m happy to trust you on that one!

photon · October 7, 2008, 12:20am

I agree with valley that there is an offset problem when scaling an image bilinear, but I don’t agree with that solution.

But first there is an other problem to deal with. I found this one while finding a better solution for the offset problem.

In LowLevelGraphicsSoftwareRenderer::clippedBlendImageWarping (where transformedImageRender is used) the intersection of the transformed image bounds and the clipping rectangle is calculated. But what’s the reason for adding 1 to the width and the height of the transformed image bounds. This leads to incorrect results, e.g. if I draw an 100x100px image scaled-to-fit with drawImageWithin to an 300x300px rectangle I get and 301x301px result drawn.

So I suggest to remove both +1 there, if no one can explain why they are there:

[code] Path imageBounds;
imageBounds.addRectangle ((float) srcClipX, (float) srcClipY, (float) srcClipW, (float) srcClipH);
imageBounds.applyTransform (transform);
float imX, imY, imW, imH;
imageBounds.getBounds (imX, imY, imW, imH);

    if (Rectangle::intersectRectangles (destClipX, destClipY, destClipW, destClipH,
                                        (int) floorf (imX),
                                        (int) floorf (imY),

                                      1 + roundDoubleToInt (imW),

                                      1 + roundDoubleToInt (imH)))

                                      roundDoubleToInt (imW),

                                      roundDoubleToInt (imH)))
  {
      const uint8 alpha = (uint8) jlimit (0, 0xff, roundDoubleToInt (opacity * 256.0f));

      float srcX1 = (float) destClipX;[/code]

Back to the offset problem. Subtracting a fixed value doesn’t work correct, e.g. it breaks the drawing of the tray icon on Linux by shifting the icon image 1px down and right leading to a black border on the top and left.

To solve the problem the offset must be variable and dependent on the source and destination sizes. While testing I noticed that removing the two floor calls in the nearest neighbourhood path and also applying the initial offset gives better results.

[code]template <class DestPixelType, class SrcPixelType>
static void transformedImageRender (Image& destImage,
[…]
DestPixelType*,
SrcPixelType*) throw() // forced by a compiler bug to include dummy
// parameters of the templated classes to
// make it use the correct instance of this function…
{
int destStride, destPixelStride;
uint8* const destPixels = destImage.lockPixelDataReadWrite (destClipX, destClipY, destClipW, destClipH, destStride, destPixelStride);

int srcStride, srcPixelStride;
const uint8* const srcPixels = sourceImage.lockPixelDataReadOnly (srcClipX, srcClipY, srcClipWidth, srcClipHeight, srcStride, srcPixelStride);

const double srcXoffset = (1.0 - (double) srcClipWidth / destClipW) / 2.0;
const double srcYoffset = (1.0 - (double) srcClipHeight / destClipH) / 2.0;

if (quality == Graphics::lowResamplingQuality) // nearest-neighbour…
{
for (int y = 0; y < destClipH; ++y)
{

```
      double sx = srcX;
```
```
      double sy = srcY;
```

```
      double sx = srcX - srcXoffset;
```

      double sy = srcY - srcYoffset;

      DestPixelType* dest = (DestPixelType*) (destPixels + destStride * y);

      for (int x = 0; x < destClipW; ++x)
      {

          const int ix = roundDoubleToInt (floor (sx)) - srcClipX;

          const int ix = roundDoubleToInt (sx) - srcClipX;

          if (((unsigned int) ix) < (unsigned int) srcClipWidth)
          {

              const int iy = roundDoubleToInt (floor (sy)) - srcClipY;

              const int iy = roundDoubleToInt (sy) - srcClipY;

              if (((unsigned int) iy) < (unsigned int) srcClipHeight)
  [...]

}
else
{
jassert (quality == Graphics::mediumResamplingQuality); // (only bilinear is implemented, so that’s what you’ll get here…)

  for (int y = 0; y < destClipH; ++y)
  {

```
      double sx = srcX - 0.5;
```
```
      double sy = srcY - 0.5;
```

```
      double sx = srcX - srcXoffset;
```

      double sy = srcY - srcYoffset;
      DestPixelType* dest = (DestPixelType*) (destPixels + destStride * y);

      for (int x = 0; x < destClipW; ++x)
      {

[…]
}[/code]

These images show the difference (they’re small but important):

valley · October 7, 2008, 1:53am

I don’t have time right now to dig into the image size thing now, but I’ll get on that first thing tomorrow. I always specify the image size I want, and never use drawImageWithin(), so I wouldn’t have noticed that failing, sorry.

I’ve tested your fix though, and I’m not sure I agree with what you’ve done here. AFAICT you’ve added a fixed - 0.5 source pixel offset to the destination. I’m getting this value from some zoom co-ordinate mapping code of my own, but you can see it to some degree by looking at the behaviour of the edge effect. If left untreated, edge effect should be visible to equal degree on all four corners of a fully scaled image. With bilinear, for example, the kernel at positions 0, and n-1 will exceed the image boundaries by one pixel, so both extents of the image will contain a row/column that is interpolated with a none existent pixel. Since JUCE is not doing mirroring, the effect is a kind of halo for the image border. Normally, such an effect would never happen with nearest neighbour. Currently, with your fix in place, I’m getting edge effects on the bottom right using nearest neighbour, and about 25/75 with bilinear.

It’s hard for me to measure pixel positioning with bilinear simply because the smoothing effect makes it hard to determine exact boundaries. But visually, I’m almost convinced that I’m seeing a fractional offset between my image shown in nearest neighbour, and bilinear. Given the figures above, I’d say that there is a 0.25 pixel offset between the two types, plus the overall 0.5 pixel offset between what my pointer is telling me about the image co-ordinate space, and what the grey-level is telling me about the pixel under my mouse.

I was worried that a brute force offset would have repercussions, but, whilst more evolved, I don’t think your fix is correct.

photon · October 7, 2008, 1:07pm

The drawImageWithin method also specifies the destination size explicitly, you give it an image (defining the source size) and a destination rectangle to draw it in.

Yes, and I think that’s the correct thing to do.

I’m aware of that and it’s working with my fix as I expect it (see later explantion).

Did you apply the size fix too, if you miss that one things will look wrong.

Can you explain why you subtract fixed 0.5 from the source pixel position? This seems odd, because consider a situation where you scale an image up 8x so a source pixel is 0.125x0.125 measured in destination pixel size. With your fixed -0.5 you offset by 4 whole source pixels.

Okay let me explain step by step what I do. I use an upscaling by 2, so nearest neighbour interpolation isn’t at a disadvantage. I also will only apply the changes to the bilinear path and no touch the nearest neighbour path (but I could apply them, for the example case the don’t make a difference):

The offset problem with bilinear is clearly visible, the borders aren’t equal but should be.

With your fix, the borders are equal but the resulting image is one pixel too big. You can notice the size difference by comparing to the nearest neighbour interpolated image.

Fixing the destination size (removing the +1 in clippedBlendImageWarping) gives the correct size but borders aren’t equal any more.

Adding a fixed -0.5 source pixel offset to the destination (like my fix does) instead of your fixed -0.5 destination pixel offset corrects this.

The “edge effect” is also there: the left most pixel has a colour of (255,65,65) because it was mixed with transparent black outside the image before it was rendered onto the white background. The pixel to its right is full red (255,0,0) as the red 2px border in the original image. The effect is the same for all edges, as I expect it, because the edges of the source image are all the same and the scale factor in both directions is the same.

Can you explain to me where you see these offsets in the last image (SVN r580 + size and offset), because I can’t see them.

valley · October 7, 2008, 5:30pm

The drawImageWithin method also specifies the destination size explicitly, you give it an image (defining the source size) and a destination rectangle to draw it in.
[/quote]

Well that’s odd. I didn’t notice image sizes coming back as anything other than what I’d asked for. Of course, the renderer may be trying to draw one pixel beyond the image extents, and I wouldn’t necessarily spot that. I’ll have to walk through the code again, to see what’s going on there.

It depends. I need pixels to be exactly where I expect them to be on the image. Currently that’s not how things are.

I did, yes.

Specifically because the translation when switching between nearest neighbour, and bilinear was half a source pixel. Since I could overlay a grid on an image that was resized using nearest neighbour and see that the pixel boundaries exactly aligned to what I would expect, I took the nearest neighbour algorithm to be providing truth.

No, I’m still offsetting by 0.5 of a source pixel. It doesn’t matter what the scale factor is, the window that I’m copying from is being offset by half a pixel. I.E, instead of making x=11 tween over 8 destination pixels with x=10 and x=12, I’m just moving down to x=10.5 and tweening through the range x=9.5 … x=11.5.

Specifically, I want my four pixel kernel to be centered on the pixel that I’m interpolating.

Yup. I need to look into whether I’m seeing that, and if so why.

[quote]

Can you explain to me where you see these offsets in the last image (SVN r580 + size and offset), because I can’t see them.[/quote]

I wouldn’t expect to see them there, since a 2 times magnification will not show a quarter pixel effect clearly. At four or eight times, which is the scaling I’m working at, I can see it.

I need to do some code for a co-worker, but I’ll get to this in a few hours.

photon · October 7, 2008, 7:30pm

Okay, now I understand why your fix should be the right one.

I tried some other things and am confused now. I also scaled some images with Gimp in bilinear mode and I noticed one thing: If I take a 100x100px image and scale it to 800x100px there is now interpolation in the vertical direction with Gimp (as I would expect), but in JUCE there is interpolation in the vertical direction if and only if your fix is applied.

I’ll investigate more.

photon · October 8, 2008, 9:11pm

After I did another series of test images I’m now convinced that valleys fix is correct, but incomplete. If the image is only scaled in one direction the -0.5 offset leads to extra bluring where no filtering at all should happen, but this is easy to fix, only add the -0.5 offset if the direction is scaled.

I compared the different fixes with the results Gimp produces with bilinear filtering and with valleys fix JUCE and Gimp produces exactly the same results, expect the extra bluring mentioned above and the fact that JUCE does not edge extension but assumes transparent black around the source.

In the 8x compare you can clearly see the initial offset problem in SVN r579 and before and the slightly wrong results my fix (SVN r580 + fixes1) produces. You can also see the extra bluring where no filtering at all should happen (SVN r580):

Here are the rest of the test images I did:

http://majestic42.net/matthias/juce/compare_4x.png
http://majestic42.net/matthias/juce/compare_2x.png
http://majestic42.net/matthias/juce/compare_1.5x.png
http://majestic42.net/matthias/juce/compare_1.1x.png
http://majestic42.net/matthias/juce/compare_1.01x.png

SVN r580 + fixes2 was made using the following changes in juce_LowLevelGraphicsSoftwareRenderer.cpp. The size fix:

[code] Path imageBounds;
imageBounds.addRectangle ((float) srcClipX, (float) srcClipY, (float) srcClipW, (float) srcClipH);
imageBounds.applyTransform (transform);
float imX, imY, imW, imH;
imageBounds.getBounds (imX, imY, imW, imH);

    if (Rectangle::intersectRectangles (destClipX, destClipY, destClipW, destClipH,
                                        (int) floorf (imX),
                                        (int) floorf (imY),

                                      1 + roundDoubleToInt (imW),

                                      1 + roundDoubleToInt (imH)))

                                      roundDoubleToInt (imW),

                                      roundDoubleToInt (imH)))
  {
      const uint8 alpha = (uint8) jlimit (0, 0xff, roundDoubleToInt (opacity * 256.0f));

      float srcX1 = (float) destClipX;[/code]

valleys -0.5 offset fix and the edge extension fix:

[code]template <class DestPixelType, class SrcPixelType>
static void transformedImageRender (Image& destImage,
[…]
SrcPixelType*) throw() // forced by a compiler bug to include dummy
// parameters of the templated classes to
// make it use the correct instance of this function…
{
int destStride, destPixelStride;
uint8* const destPixels = destImage.lockPixelDataReadWrite (destClipX, destClipY, destClipW, destClipH, destStride, destPixelStride);

int srcStride, srcPixelStride;
const uint8* const srcPixels = sourceImage.lockPixelDataReadOnly (srcClipX, srcClipY, srcClipWidth, srcClipHeight, srcStride, srcPixelStride);


if (quality == Graphics::lowResamplingQuality) // nearest-neighbour..
{
    [...]
}
else
{
    jassert (quality == Graphics::mediumResamplingQuality); // (only bilinear is implemented, so that's what you'll get here..)

    for (int y = 0; y < destClipH; ++y)
    {

```
      double sx = srcX - 0.5;
```
```
      double sy = srcY - 0.5;
```

      double sx = srcX - (srcClipWidth == destClipW ? 0.0 : 0.5);

      double sy = srcY - (srcClipHeight == destClipH ? 0.0 : 0.5);

      DestPixelType* dest = (DestPixelType*) (destPixels + destStride * y);

      for (int x = 0; x < destClipW; ++x)
      {
          const double fx = floor (sx);
          const double fy = floor (sy);
          const int ix = roundDoubleToInt (fx) - srcClipX;
          const int iy = roundDoubleToInt (fy) - srcClipY;

          if (ix < srcClipWidth && iy < srcClipHeight)
          {
              PixelARGB p1 (0), p2 (0), p3 (0), p4 (0);

              const SrcPixelType* src = (const SrcPixelType*) (srcPixels + srcStride * iy + srcPixelStride * ix);

```
              if (iy >= 0)
```
```
              {
```
```
                  if (ix >= 0)
```
```
                      p1.set (src[0]);
```

                  if (((unsigned int) (ix + 1)) < (unsigned int) srcClipWidth)

```
                      p2.set (src[1]);
```
```
              }
```

              if (((unsigned int) (iy + 1)) < (unsigned int) srcClipHeight)

```
              {
```

                  src = (const SrcPixelType*) (((const uint8*) src) + srcStride);

```
                  if (ix >= 0)
```
```
                      p3.set (src[0]);
```

                  if (((unsigned int) (ix + 1)) < (unsigned int) srcClipWidth)

```
                      p4.set (src[1]);
```
```
              }
```

              const SrcPixelType* temp = src;

```
              if (iy < 0)
```

                  temp = (const SrcPixelType*) (((const uint8*) temp) + srcStride);

```
              if (ix < 0)
```
```
                  p1.set (temp[1]);
```
```
              else
```
```
                  p1.set (temp[0]);
```

              if (((unsigned int) (ix + 1)) < (unsigned int) srcClipWidth)

```
                  p2.set (temp[1]);
```
```
              else
```
```
                  p2 = p1;
```

              if (((unsigned int) (iy + 1)) < (unsigned int) srcClipHeight)

```
              {
```

                  src = (const SrcPixelType*) (((const uint8*) src) + srcStride);

```
                  if (ix < 0)
```
```
                      p3.set (src[1]);
```
```
                  else
```
```
                      p3.set (src[0]);
```

                  if (((unsigned int) (ix + 1)) < (unsigned int) srcClipWidth)

```
                      p4.set (src[1]);
```
```
                  else
```
```
                      p4 = p3;
```
```
              }
```
```
              else
```
```
              {
```
```
                  p3 = p1;
```
```
                  p4 = p2;
```

              }

              const int dx = roundDoubleToInt ((sx - fx) * 255.0);
              p1.tween (p2, dx);
              p3.tween (p4, dx);
              p1.tween (p3, roundDoubleToInt ((sy - fy) * 255.0));

              if (p1.getAlpha() > 0)
                  dest->blend (p1, alpha);
          }

[…]
}[/code]

With this fixes JUCE produces the same results as Gimp for bilinear filtering.

jules · October 10, 2008, 3:41pm

Well, that looks like a nice piece of work, but surely it’ll crash or fail if the image is transformed with a rotation? It looks to me like you’re neglecting the cases where the source point falls way outside the required area, or even outside the source image?

photon · October 10, 2008, 10:30pm

You’re right, my suggested changes for edge extension will fail in such cases, I’ll try to come up with a better solution. Edge extension in such non-trivial cases seem to be hard.

jules, can you explain why there is this +1 in size:

[code] if (Rectangle::intersectRectangles (destClipX, destClipY, destClipW, destClipH,
(int) floorf (imX),
(int) floorf (imY),

                                      1 + roundDoubleToInt (imW),
                                      1 + roundDoubleToInt (imH)))

    {
        const uint8 alpha = (uint8) jlimit (0, 0xff, roundDoubleToInt (opacity * 256.0f));[/code]

Topic		Replies	Views
Direct2D preview branch Direct2D	68	3845	April 18, 2024
Strange behaviour General JUCE discussion	10	435	May 12, 2017
Massive performance decrease in Software Rendering General JUCE discussion	43	1506	March 6, 2013
DrawableButton Image Quality General JUCE discussion	13	1002	February 2, 2009
Understanding image blit in juce_RenderingHelpers General JUCE discussion	4	468	December 2, 2021

transformedImageRender() offset

Purchase

Discover

Learn

Support

About

Events

transformedImageRender() offset

Related topics

Purchase

Discover

Learn

Support

About

Events