# Unicode Rendering Bug (Mac & Windows)

**URL:** <https://forum.juce.com/t/unicode-rendering-bug-mac-windows/50138>\
**Category:** General JUCE discussion\
**Created:** [February 11, 2022, 4:01pm UTC](https://forum.juce.com/t/unicode-rendering-bug-mac-windows/50138 "2022-02-11T16:01:37Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![benstaton](https://avatars.discourse-cdn.com/v4/letter/b/9fc29f/32.png) [@benstaton](https://forum.juce.com/u/benstaton)\
**Post date:** [February 11, 2022, 4:01pm UTC](https://forum.juce.com/t/unicode-rendering-bug-mac-windows/50138/1 "2022-02-11T16:01:37Z")

</div>

Hi all,

I’m using `juce::AttributedString` to render Unicode text (UTF-8) containing fancy characters such as emojis.

It works, but it stops short of rendering the full string. Here’s some test code taken from a component `paint()` method:

```auto
        juce::AttributedString attrString;
        attrString.setText( L"Abc🍕😎🏈Def" ); // I know - don't embed actual Unicode chars in source code*.
        attrString.setFont( juce::Font( 20.f ) );
        attrString.setJustification( juce::Justification::centred );
        attrString.setWordWrap( juce::AttributedString::WordWrap::none );
        attrString.setColour( juce::Colours::white );
        attrString.draw( g, getLocalBounds().toFloat() );

```

This is the result on Windows:

![image](https://us1.discourse-cdn.com/flex026/uploads/juce/original/2X/1/1613d32c455a51b9032058c5e2888a77492b7915.png)

And Mac:

![image](https://us1.discourse-cdn.com/flex026/uploads/juce/original/2X/e/e71c2dabc4950aeff4e6d09572a270fb992cf818.png)

Note that it chopped the last 3 characters off on Windows, and the last 3 characters are the wrong font/color on Mac.

The _first_ bug is in `juce::AttributedString::setText (const& String newText)`:

```
auto newLength = newText.length();

```

In this example, `newLength` is set to 9 which seems correct, right? There are 9 characters in total.

Not quite. It should actually be 18! Why? Because it needs the size _in bytes_ not _characters_. And since each emoji is represented by 4 bytes in UTF-8, the total for 6 normal characters (6 bytes) plus 3 emojis (3 \* 4 = 12 bytes) is 18.

Strictly speaking, I’m not 100% sure if it’s _bytes_ or something more ‘Unicodey’ like ‘code points’ or whatever, but they might be equivalent with UTF-8 anyway.

Regardless, changing it to this gets it working _ **on Mac only** _:

```
const auto newLength = (int) newText.getNumBytesAsUTF8(); // Returns 18 in this example.

```

![image](https://us1.discourse-cdn.com/flex026/uploads/juce/original/2X/7/72112ff04339a060446141f4e9d814349d253ec3.png)

Unfortunately, an additional fix is required on Windows:

`juce::DirectWriteLayout::setupLayout()` passes the wrong string length to IDWriteFactory::CreateTextLayout() (see juce\_DirectWriteLayout.cpp line 370 in JUCE 6.1.5). Instead of using `String::length()` it should pass the length in UTF-16 characters (Unicode code points?), which is 12 in this example (6 normal characters plus 3 x 2 bytes for the emojis). For example:

```
    const std::wstring wstr (text.getText().toWideCharPointer());
    const auto textLen = (UINT32)wstr.length();

    hr = directWriteFactory.CreateTextLayout (wstr.c_str(), textLen, dwTextFormat,
                                              maxWidth, maxHeight, textLayout.resetAndGetPointerAddress());

```

And voila!

![image](https://us1.discourse-cdn.com/flex026/uploads/juce/original/2X/e/ec307ddf7f0d3bd4d827fa01a4213a2b83af1ad0.png)

There may be more to it than this - I’m not a Unicode expert, nor am I intimately familiar with the JUCE text rendering code - but it’s working well here.

Would be nice to see it properly fixed in JUCE, followed by an update for `juce::TextEditor` so we can let users actually _edit_ Unicode text in 2022!

Many thanks,  
Ben

---

<div class="post-metadata">

**Author:** ![benstaton](https://avatars.discourse-cdn.com/v4/letter/b/9fc29f/32.png) [@benstaton](https://forum.juce.com/u/benstaton)\
**Post date:** [February 14, 2022, 8:57am UTC](https://forum.juce.com/t/unicode-rendering-bug-mac-windows/50138/2 "2022-02-14T08:57:26Z")

</div>

It occurred to me I’ve only been testing _single line_ strings. There will probably be complications with multi-line strings.

Edit: Confirmed. I forgot to mention I also changed an assert at the top of `AttributedString::draw()` from this…

`jassert (text.length() == getLength (attributes));`

…to this…

`jassert ((int) text.getNumBytesAsUTF8() == getLength (attributes));`

Which worked nicely until I tried it with a 2 line string. Looks like `getNumBytesAsUTF8()` includes “\n” (as you would expect), but `getLength (attributes)` doesn’t. Obviously, it’s just an assert and it still renders fine. I’m tempted to disable the assert for now.

---

<div class="post-metadata">

**Author:** ![reuk](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.juce.com/reuk/32/21494_2.png) [@reuk](https://forum.juce.com/u/reuk)\
**Post date:** [February 23, 2022, 5:03pm UTC](https://forum.juce.com/t/unicode-rendering-bug-mac-windows/50138/3 "2022-02-23T17:03:49Z")

</div>

Thanks for reporting this issue.

The root of the problem was that the macOS and Windows built-in formatted string facilities both use UTF-16 strings internally. Attribute positions are specified as offsets into an array of 16-bit words. Because UTF-16 is a variable-width encoding, codepoint N inside the array doesn’t necessarily exist at index N. An additional conversion is required to translate the character index to an offset in the 16-bit buffer.

I imagine the problem has been overlooked for a long time because many commonly-used glyphs can be represented using a single UTF-16 word, so the indices happened to line up.

I’ve implemented a fix for the problem here:

> <https://github.com/juce-framework/JUCE/commit/e27fb35996c7b17e86f9961c93343ca861f47dba>
>
> CFAttributedString ranges must be given in terms of 16-bit word offsets,
> rather …than codepoints.

* * *

> [@benstaton](#):
>
> ```auto
> juce::AttributedString attrString;
> attrString.setText( L"Abc🍕😎🏈Def" ); // I know - don't embed actual Unicode chars in source code*.
> 
> ```

I know that you know not to do this, but it’s really not _ever_ a good idea, not even in testing code! I’d recommend using the UTF-8 String-Literal helper tool in the Projucer (in the Tools menu) to convert unicode strings into a format suitable for inclusion in a C++ source file.

---

<div class="post-metadata">

**Author:** ![benstaton](https://avatars.discourse-cdn.com/v4/letter/b/9fc29f/32.png) [@benstaton](https://forum.juce.com/u/benstaton)\
**Post date:** [February 23, 2022, 6:40pm UTC](https://forum.juce.com/t/unicode-rendering-bug-mac-windows/50138/4 "2022-02-23T18:40:35Z")

</div>

Thanks for looking into it @reuk ! Very much appreciated.

Glad you were able to home in on a more thorough fix (albeit with some internal issues still up for discussion).

And yes, the embedded emojis were just a super quick convenience, since I could force both VS and Xcode to save/load the .cpp file with appropriate encoding. But you’re absolutely right, of course, that the Projucer helper is the way to go under normal circumstances.

Thanks again,  
Ben

---

<div class="post-metadata">

**Author:** ![reuk](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.juce.com/reuk/32/21494_2.png) [@reuk](https://forum.juce.com/u/reuk)\
**Post date:** [February 23, 2022, 6:41pm UTC](https://forum.juce.com/t/unicode-rendering-bug-mac-windows/50138/5 "2022-02-23T18:41:50Z")

</div>

> [@benstaton](#):
>
> albeit with some internal issues still up for discussion

Could you elaborate on this please? Are you seeing problems after updating to the newest develop?

---

<div class="post-metadata">

**Author:** ![benstaton](https://avatars.discourse-cdn.com/v4/letter/b/9fc29f/32.png) [@benstaton](https://forum.juce.com/u/benstaton)\
**Post date:** [February 23, 2022, 6:46pm UTC](https://forum.juce.com/t/unicode-rendering-bug-mac-windows/50138/6 "2022-02-23T18:46:14Z")

</div>

> [@reuk](#):
>
> An additional conversion is required to translate the character index to an offset in the 16-bit buffer.

Oh, my bad!!! I just misread what you said about the additional conversion that translates the character index to an offset in the 16-bit buffer. I read it as “an additional _conversation_ is required…”, as in the fix isn’t quite 100% finalized yet!

But, of course, it is! Thanks again 🙂

---

<div class="post-metadata">

**Author:** ![reuk](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.juce.com/reuk/32/21494_2.png) [@reuk](https://forum.juce.com/u/reuk)\
**Post date:** [February 23, 2022, 6:47pm UTC](https://forum.juce.com/t/unicode-rendering-bug-mac-windows/50138/7 "2022-02-23T18:47:05Z")

</div>

Cool, thanks for clarifying!