Value-safety question (ValueTree to XML and back)


#1

If i store a double, into a value tree, save it as xml to disc, restore it, recreate ValueTree from Xml, can i be sure the the double has exact the same value as before?


#2

TBH that’s something that I’ve been meaning to test, as it depends on the double->string->double conversions. It’s probably ok, and will certainly be accurate to within a very very small fraction, but I’m not sure if it’ll be bit-for-bit identical or not.


#3

No it doesn’t.
You can’t have a textual representation of some double value, so there is always a loss of precision when converting to/from text.
Think of 1/10, try to store this in a double, it’s an infinite precision number (like 1/3 is for our base 10 mind). When you convert it, you’ll loose some precision.
Value use String(double) constructor, which in turns ensure 9 digits precision after the decimal place.


#4

yes, thanks, ok than i have feature request, an option that all doubles/floats will saved in a binary representation, and not as decimal-number, or another technique which has binary correctness. Its very important for my app, to use exact the same values (and not a little more or less).

-like hexadecimal #a4b6d6fa5f4e2a8e with a leading “#” to be different
-or save Sign,Exponent,Significand as sperate Numbers like 654654654*10^-3 (not sure if this helps??)


#5

No, that’s not true.

There’s a very simple and well-known algorithm to do it with 100% accuracy - basically, you simply keep printing digits until the string you’ve generated would result in the original value when it gets read back in. Had a link to an article about this somewhere, but can’t find it now.


#6

1/10 results in the base 2 number 0.0101010101010101010101010101010101010 etc…
There is no possibility you’ll get a finite representation in text for such a number. (In decimal, it’s easy it’s 0.1, but bases conversion is not mathematically exact with discrete number representation)

At worst, you’ll hit the DBL_EPSILON limit when printing it, but good luck doing any valid computation on such small number (like the classical:
number = number - floor(number);
number = number * 10; // number is not number * 10, but (number +/- DBL_EPSILON) * 10, you’re loosing precision here).

Anyway, the current code doesn’t do this, it’s limited to 9 digits by defaults in the String code.

Just a remark to the original poster, if ValueTree were improved to store an int64 (like I asked in the other thread), then you would simply reinterpret_cast your double to int64 and back, and it’s done, bit exact representation.


#7

There doesn’t need to be - the whole point is not to create an exact representation for it, but to create a representation that can be re-loaded as the original number, and that is possible without anything becoming infinite. This was all proven by some clever academic folks whose names I’ve forgotten, but I read a paper about it once, and I think most double conversion libraries use their technique.


#8

I made a test, the result is interesting

DBG("float 0.1f = String "+String(0.1f)+" conversion "+(String(0.1f).getFloatValue()==0.1f ? String("ok") : String("wrong")));
DBG("double 0.1 = String "+String(0.1)+ " conversion "+(String(0.1).getFloatValue()==0.1   ? String("ok") : String("wrong")));

Output

float 0.1f = String 0.100000001 conversion ok
double 0.1 = String 0.1 conversion wrong

#9

chkn, you’re not understanding the issue. You could get good results for both case, if you used an higher digit precision in the String(double, int digits) constructor.
However, there is no way to know beforehand the number of digits required, and worst, if you hit the processor FPU limit, you’re fucked and you could enter an infinite loop, if the algorithm were made to minimize conversion error, because of the missing precision.

@Jules, you probably refer to this document.
However, the guys from PHP and Java both implemented this algorithm, but it enters an infinite loop as Intel’s FPU processor doesn’t give good results when dealing with DBL_EPSILON/DBL_MIN.
Please refer to this article.

Anyway, you could waste a lot of time trying to reinvent the weel by implementing the algorithm yourself, and hitting the same bugs as PHP / Java, or simply reuse the official strtod function for this, but in the end:

  1. Either you care about the double precision, and you better store it binary wise (so you can actually save NaN and +/-INF numbers).
  2. Either you don’t care about the complete double precision, but more about the speed, and you better use the current code (which only save 9 decimal digits of precision).

#10

my example refers to my intial post, what happens if i store a floating-point into value-tree and store it in a xml and restore it (and there i have no influence in the number of digits to use) .


#11

and there was a mistake in my example (used getFloatValue() instead of getDoubleValue()), it seems to be anything correct

DBG("float 0.1f = String "+String(0.1f)+" conversion "+(String(0.1f).getFloatValue()==0.1f ? String("ok") : String("wrong"))); DBG("double 0.1 = String "+String(0.1)+ " conversion "+(String(0.1).getDoubleValue()==0.1 ? String("ok") : String("wrong")));

float 0.1f = String 0.100000001 conversion ok double 0.1 = String 0.1 conversion ok


#12

In your second example, you’re storing 0.1 in float, but read it back as 0.1000001f. This is not correct.

If your code read:

    DBG("float 0.1f = String "+String(0.1f)+" conversion "+(String(0.1f).getFloatValue()==0.1f ? String("ok") : String("wrong")));
    DBG("double 0.1 = String "+String(0.1)+ " conversion "+(String(0.1, 16).getDoubleValue()==0.1   ? String("ok") : String("wrong")));

You’d get 16 digits of precision, and, probably, the conversion will succeed in both case.
The real question is how many digits of precision are required, provided the code does select the best decimal representation of the IEEE number.

If I were you I would try the extrema number like “2.2250738585072012e-308” as it’s most likely to fail.


#13

yes, but back to the primary topic, jules, it would be nice to have an option when converting (to xml) to have an option to store floats & doubles in a binary representation.

yes (as i could do it always myself), but i think something like described above should be added to the library, because saving data as it is (without changing a bit) is a general request…


#14

I think it might be worthwhile to separate the serialisation of ValueTrees out into a base ValueTreeSerialiser (of which the current binary and xml behaviours could be subclasses). It’d probably mean exposing a little more of the innards, but perhaps exposure could be minimised by making the serialiser a friend of ValueTree and giving it functions to access any guts it might need. That could also make it possible to override only certain parts of a serialisation process.

might not be as simple as the top-of-the-head idea sees it though :slight_smile:


#15

yes, then it would be nice to have other serializers like for JSON (or simply any other textual/binary format which has a tree-structure)


#16

well, i mean there’s nothing stopping anyone from writing their own such things :slight_smile: but it’d certainly be nice if there were a standard base to work from


#17

heh, thinking about it, a generic ‘base’ would just consist of a ‘serialise’ function :smiley: how it does it depends entirely on how the data is to be stored. Still, a ValueTreeXmlSerialiser which can be customised is something I actually need at the moment, so I’m bashing one together. I’ll post it here when it’s done.


#18

well, I’ve just knocked this up: ValueTreeSerialiser.zip

It’s pretty rubbish, and I’ve barely tested any of it, but it does what I need it to do. It’s going to be slower than the normal createXml() from a ValueTree, but I don’t think it would make anyone actually cry.

There are a bunch of places you can override to poke in your own behaviour. There are comments in the header explaining what does what. It’s pretty straightforward. And crap. But it works. The end.