Is JUCE's test suite supposed to be passing?

General JUCE discussion
1

I just tried running the UnitTestRunner on vanilla JUCE develop (ebe954f176f532) and 7 different tests suites are failing:

  • juce_AudioDeviceManager.cpp: AudioDeviceManagerTests
  • juce_WavAudioFormat.cpp: WaveAudioFormatTests
  • juce_VST3PluginFormat_test.cpp: VST3PluginFormatTests
  • juce_AudioProcessorGraph.cpp: AudioProcessorGraphTests
  • juce_JSONSerialisation_test.cpp: JSONSerialisationTest
  • juce_CIDevice.cpp: DeviceTests
  • juce_CISubscriptionManager.cpp: SubscriptionTests

When I disable those tests the suite passes.

(Iā€™m using Xcode 15.2 in macOS 13.5)

If it were just one or two I would provide specific info about their failures but Iā€™m wondering if Iā€™m missing something basic. Is this supposed to be a regression test suite that passes, where they are regularly checked in a CI or something? Do the tests fail just for me?

Btw: also fail via the JUCE DemoRunner in UnitTestsDemo

2

Bisected the first of these breakages to 2022-02-07 c1a3cc28fd33 where WaveAudioFormatTests fails in the added assertion jassert (ISRC.length() == 12);,

Screenshot 2024-03-25 at 12.02.29
Screenshot 2024-03-25 at 12.02.292938Ɨ1624 666 KB

3

If itā€™s just jasserts then unfortunately itā€™s expected. There are a couple of different cases from memory

  1. jassserts regarding running on a message thread, this depends where you run the test for example if you run it from he DemoRunner all is fine
  2. Some jasserts that are there to warn a user that for example that a host might be misbehaving but there are tests covering the behaviour in that circumstance
  3. The not you highlighted which from memory the test passes what is really an illegal code according the standard, the test in this instance should probably just be updated

I have an internal MR that Iā€™ve been working on/off for a while that tries to address all of these but probably wonā€™t make it in for a little while.

1 Like
4

I see, if I click resume on each failing jassert then after ~30 of these I do eventually get ā€œAll tests completed successfullyā€.

So the tests are meant to be run without assertions?

5

At this moment in time, yes. That being said Iā€™m keen to change that, hence not just pushing a commit that turns the assertions of for the UnitTestRunner.

1 Like
6

Hi Anthony, whatā€™s your general strategy for testing that asserts are being hit when they should be? Iā€™ve just recently been using a little ā€˜assertIfNotInTest()ā€™ sort of function instead of jassert but itā€™s not very prettyā€¦

1 Like
7

Over the years I have attempted to test assertions are hit and there are ways you can kind of do it but Iā€™ve come to the conclusion that you shouldnā€™t test that asserts are hit. An assertion is a useful catch when things have gone wrong (or rather about to go wrong) normally testing some kind of precondition, but they should basically be outside of the documented behaviour (our contract). If itā€™s outside the contract I donā€™t think a test is a good idea, in a way the assertion is instead of a test. Tests should be testing how it behaves when used within the contract, not outside of it. I think if you do want that kind of behaviour an exception or returning an error/optional/expected type would be a better way to deal with it as then you can test it and as a user of the class/function you can respond to the error programmatically.

From memory John Lakos covers this topic quite well and in some depth here https://youtu.be/1QhtXRMp3Hg thereā€™s also lots of books that go into this I recall the Pragmatic Programmer having a good chapter on it. Maybe the assertions chapter in Code Complete.

IMO Ideally tests should never trigger an assertion as they should meet all the pre-conditions otherwise they arenā€™t best documenting how to use whatever is being tested. There are occasions however when this is a little painful but this is where techniques such as dependency injection can normally help in order to allow mocking other parts of a system.

2 Likes
8

Thanks for the video recommendation, Anthony, it was really interesting. It seems that they do build their asserts in a way that lets them be tested, using a few different categories of assert. I could see the benefit to users and those maintaining forks if JUCE had such a system, but it would be quite a big change.

Iā€™ll go off and try to write narrower contracts nowā€¦