UnitTestRunner output

Is it possible to customize or change the text in UnitTestRunner::endTest to something more specific like “1 test passed” or “Name of Test Group passed”?

“All tests completed successfully” leads to some ambiguity, especially when the test process is interrupted for some reason. Context in this particular case is pluginval, which uses the test runner (I’m working on validating the JUCE plugin examples in CI).

What exactly is the ambiguity? We have the name of the test, followed by a report that everything was successful. I don’t see why a duplication of the name would be helpful, nor the number of tests within the group.

Hey Tom,

Starting test: something...
All tests completed

It’s not a biggie, it’s just a bit confusing. It’s a singular “test” that concludes with a plural “All tests” — so it sounds like “all” my tests pass, when it was just this one particular one.

If the process/log is interrupted before a failing test, the last message about success/failure is “All tests completed” which isn’t true. At the end of the test run, we see “All Tests completed” which is finally true for the entire “suite.”

Would be nice to customize to:

Starting test: something...
Test passed

OK, that makes sense.

You need to check the return code of a process to work out if it was interrupted. If the return code indicates an error then you know something has gone wrong. It would be the the responsibility of the person launching the process to print a message saying that the process was interrupted and any other information. After an interruption you cannot know what exactly what state the unit test runner was in, so you cannot rely upon the logging either.

You need to check the return code of a process to work out if it was interrupted.

Yes, we are doing this in CI. I’ll check to make sure the pluginval GUI log is clear as well. pluginval inserts some extra text about timing at the end of the process, so maybe I can put something more conclusive there. It sort of looked like I could subclass the runner to override expect to customize pass/fail, but thought I’d ask first.

1 Like

Thanks, that’s perfect and will help clarify pluginval output a lot, I think!