# Strangeness with UTF8!

**URL:** <https://forum.juce.com/t/strangeness-with-utf8/9125>\
**Category:** General JUCE discussion\
**Created:** [August 19, 2012, 9:42pm UTC](https://forum.juce.com/t/strangeness-with-utf8/9125 "2012-08-19T21:42:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomswirly](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.juce.com/tomswirly/32/236_2.png) [@tomswirly](https://forum.juce.com/u/tomswirly)\
**Post date:** [August 19, 2012, 9:42pm UTC](https://forum.juce.com/t/strangeness-with-utf8/9125/1 "2012-08-19T21:42:43Z")

</div>

Hello, Jucers.

I have an interesting issue regarding UTF-8 where I feel that the behavior of Juce has changed, and is now incorrect.

It used to be that if CharPointer\_UTF8::isValidString(p, size) returned false, you were guaranteed that String::fromUTF8(p, size) would fail - so when I convert from strings, (e.g. from files) I always check CharPointer\_UTF8::isValidString()

I added a lot of translated text to my program, and a lot of it seems to fail CharPointer\_UTF8::isValidString() - yet the string looks like a perfect good UTF-8 string. And now, if I ignore CharPointer\_UTF8::isValidString(), then I get a correct string!

What’s going on here?

`inline String str(const string& s) {
  const char* p = s.c_str();
  size_t size = s.size();
  bool valid = CharPointer_UTF8::isValidString(p, size);
  if (!valid) {
    LOG(ERROR) << "Badly encoded string |" << s << "| " << s.size();
    LOG(ERROR) << s[0] << ", " << s[1];
    valid = true; // HACK - IGNORE THE FACT THAT THIS STRING IS BAD!
  }
  return valid ? String::fromUTF8(p, size) : "(badly encoded string)";
}
`

with result:

`I0819 23:36:21.582591 2956513280 Instance.cpp:267] registered
E0819 23:36:21.731360 2696910144 Juce.h:101] Badly encoded string |Öffnen Sie den letzten| 23
E0819 23:36:21.731408 2696910144 Juce.h:102] \303, \226
`

---

<div class="post-metadata">

**Author:** ![buck](https://avatars.discourse-cdn.com/v4/letter/b/c5a1d2/32.png) [@buck](https://forum.juce.com/u/buck)\
**Post date:** [August 20, 2012, 1:01am UTC](https://forum.juce.com/t/strangeness-with-utf8/9125/2 "2012-08-20T01:01:03Z")

</div>

In juce\_CharPointer\_UTF8.h, looking at the definition of CharPointer\_UTF8::isValidString(), I think ‘dataToTest’ needs to be bumped before the “extra values” are checked for validity. In other words change

[code] n &= mask;

```
while (--numExtraValues >= 0)
{[/code]

```

to

` n &= mask;
    dataToTest++;
    while (--numExtraValues >= 0)
    {`

Which should make your simple testcase work (it worked for me).

Also, the Wikipedia article on UTF-8 implies that about half of the 4-byte sequences are valid, so you might want to relax the previous check

`if (bit <= 0x10)
    return false;`  
to

`if (bit <= 0x8)
    return false;`

Although then you may get false positives instead of false negatives. What you really need in the case of (bit == 0x10) is to sanity-check that the actual decoded Unicode character is \<= 0x10ffff, which I am currently too lazy to write the code for.  
Hope this helps.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://forum.juce.com/u/jules)\
**Post date:** [August 20, 2012, 8:04am UTC](https://forum.juce.com/t/strangeness-with-utf8/9125/3 "2012-08-20T08:04:30Z")

</div>

Wow, looking at that method again, it’s actually a real piece of crap - there are several stupid bugs in it apart from the ones you mentioned. Quite embarrassing really, I was obviously having a bad day when I wrote it! Thanks for the heads-up, I’ll kick some shape into it today!

---

<div class="post-metadata">

**Author:** ![tomswirly](https://sea2.discourse-cdn.com/flex026/user_avatar/forum.juce.com/tomswirly/32/236_2.png) [@tomswirly](https://forum.juce.com/u/tomswirly)\
**Post date:** [August 20, 2012, 8:23am UTC](https://forum.juce.com/t/strangeness-with-utf8/9125/4 "2012-08-20T08:23:45Z")

</div>

Well done, buck!

And don’t get too down, Jules - every diamond codebase has a little dreck somewhere in its interstices…
