TextDiff unit test failure

#1

Our CI server recorded a failure in the TextDiff unit test. Looking over the test, it seems that it uses a random seed, it uses the Random constructor with no arguments. As you can imagine, we have no way of reproducing the test case since we don't know the seed. But there was definitely a failure.

So just a heads up, there exists a combination of inputs for which the TextDiff unit test fails. Any JUCE unit test that uses a Random seed should log the seed used in its output so the test can be reproduced later. Unit tests which use random seeds are practically useless since an intermittent failure cannot be reproduced easily.

In my opinion we should do the following:

1) Change "setSeedRandomly" to "generateRandomSeed" as a static member function so the caller can intercept the generated seed

2) Modify all the unit tests to call generateRandomSeed manually, log the seed, and then construct the Random object with the seed

3) Change the no-argument constructor of Random to use generateRandomSeed instead of setSeedRandomly.

At a minimum, these classes should log the seeds used in the corresponding unit tests:

 

AbstractFifo

JSON

Random

Uuid

MemoryInputStream

String

TextDiff

 

#2

Hmm.. Thanks for the heads-up, I'll run a long test and see if I can get it to happen again.

And yes, good idea about logging the seeds - in fact, a good plan would probably be to put a shared Random object into the base class, so that all the unit tests are based on a single overall seed which gets logged.

#3

Here’s the output from the TextDiff unit test:

They seem to differ in the 7th char.

NOTE When I post the actual text with the missing characters from the code page, the forum actually generates an error. Probably it is vulnerable to a SQL injection attack.

#4

Jules what exactly is the meaning of your change to UnitTest? You're copying the Random in the test runner by value? So all unit tests start out with the same Random state?

 

#5

Yes, there's one Random which all the tests can share, so there's only one seed to log, but it's different for each run.

#6

I just stumbled over the same unit test failure. 

#5837 failed: Expected value: bbacacaccbbcabcbbaaccaaccccbcabcbcacb, Actual value: bbacacaccbbcabcbbaaccaaccccbcabcbcacb

The failure seems to be in a non-printable character which does not really help to pin down the problem.

#7

Patrick is this reproducible? If you can find the random seed that makes this happen every time that would be awesome.