Pluginval can set an AudioParameterBools to odd values

General JUCE discussion
21

But I’m not sure that it does? Maybe I’ve misunderstood the problem?

I thought the problem was this:

  • Host tries to set a discrete parameter a “random” normalised float value
  • Plugin then “maps” that float value to a discrete step
  • Host then asks for the same parameter value as a normalised float again
  • The value we sent and the value we get back are not the same and a failure is logged

Now, it seems like what we’re doing in your approach is to guess the “map” the plugin uses internally so we only send it values that when it applies it’s “map”, does not modify the value. Thus we get the same value back as we sent and no error is logged.

The problem is that if we guess the “map” incorrectly (incorrect round mode, truncation instead of round etc.) then we still get errors the rest of the time.

The only way I can see to solve this is if we can ask the parameter to “map” the value itself, set that mapped value and then ask for the value back. The test is that the change in value is applied and correctly restored, not if it’s mapped correctly.

If we want to remove all false negatives (i.e. test failures) then the only current way I can see to do that with the existing APIs is to just omit discrete parameters from the test?

Maybe bools will work if we only sent 0 and 1 as we know what these should come back as. Equally we could send 0.0f and 1.0f to try and get the discrete min/max step?

22

Well, we can only write a test, when we know what the outcome should be.

The NormalisableRange of a RangeBasedParameter is applied afterwards, so a non-linear mapping should not be our concern.

The information we have from a parameter is getNumSteps() and the isDiscrete() flag.
The range is fixd 0 to 1, because it is normalised.

With no additional information, we hve to assume (which the host would do as well), that it is appropriate to send values, that snap to equidistant numbers, 0 being the smallest and 1 being the highest.

If you want to write a test, that tests any arbitrary number, you would indeed need to know that particular mapping. So this is impossible. But we don’t know, so applying occams razor we send values we expect a host to send.

Otherwise not testing at all is an option, but IMHO not the preferable.

23

I think we might be talking cross purposes a bit here.
There’s a few different layers going on so I’ll try to address them all.

Firstly, Can I clarify what tests are actually failing here?
AFAICT it’s only PluginStateTestRestoration?

I just re-read the first few posts here at it seems the problem isn’t that the plugin is doing some arbitrary snapping, it’s that it’s not doing any snapping. As far as I can tell this is the order of events:

  • Some test runs in pluginval setting plugin parameter values to random values (totally acceptable behaviour, see below)
  • That test ends and the PluginStateTestRestoration starts by requesting the plugin’s parameter values
  • Rather than returning snapped values, the plugin returns the unsnapped, random value
  • pluginval then sets a bunch of random parameter values (again, totally acceptable behaviour)
  • pluginval tries to restore the original state
  • pluginval then requests all the parameter values again and compares these to the values at the start of the test

TBH, this all sounds perfectly reasonable to me. This test seems to be highlighting some asymmetry in how the plugin behaves when a) a parameter value is set and b) complete state is set. It seems like the failing plugins aren’t snapping individual parameter set calls, but is snapping parameter values in response to a whole-state change.

It’s that difference that I don’t understand. Why would plugins behave like that? Sounds a bit dodgy to me?

I would have thought that if you set a parameter value either from an individual param call or the whole state, it should snap parameters the same way as these are the legal values in the plugin. For example, if a binary parameter stores its state as true/false, it can’t be anything other than those two choices. When you convert that to float, it must be 0.0f or 1.0f.

The change requested in this PR seems to work around this problem by only setting certain parameter values but again, this isn’t really what this test is testing, it’s testing if a plugin restores its state exactly back to what it was before.

Note that this test is a level 6 test so goes above the “lowest common denominator for conformance” level 5.

I don’t think there’s anything to do with Occam’s razor here, we’re not inventing anything new?
And I’m not considering non-linear mappings, that fits in to the same box as above. As far as I’m aware, plugins would do this internally, they’d just have a different snapping function so should apply that on state restoration or plugin value setting.

Yes, but I think as I described above, it’s pretty clear what this test is testing and it’s an odd asymmetry that it’s flagging.

I think this misses the point of pluginval which isn’t to create a host that does the most sensible thing, in fact the opposite, it should do anything within the boundaries of the plugin APIs. It is perfectly acceptable in any of the major plugin APIs to send any normalised float value to any parameter. What the parameter then does with this is up to the plugin. It just needs to behave consistently.

As far as I can remember, there is no test in pluginval that sets a random parameter value and expects to be able to read that same value back is there?

Finally, auval was mentioned here and IIRC this was the main influence of this test. I think it does something very similar with its -strict option.

Are plugins that are failing pluginval still passing auval -strict?

I can’t remember the exact testing auval does but it might be same as pluginval or it might be the more strict one I described above but pluginval doesn’t actually test?

Finally, yes, applying the snapping before setting parameter values might well fix this, but I think it would then be testing something slightly different and we might be masking other issues that are currently being highlighted.

These kinds of grey areas of behaviour are what makes writing plugins/hosts so difficult and I think pluginval is at least mostly in the right here for testing something that should work if plugins at least apply internal snapping consistently?

Finally, an example of when a host might not do any snapping to parameter changes it sends could be with VST2.4. This doesn’t have any real API notion of discrete/stepped parameters. You only get that if you support the external VSTXML format which certainly isn’t required from either the host or plugin side (juce only added it a few years ago).

If you don’t have that VSTXML, you just have to send any float values (imagine an automation ramp). The plugin will then have to have some kind of “snapped” notion which it either applies to its internal data representation or not. Either way, it should behave the same from a parameter value change or a state change I think.

24

The TL;DR of the above is this:

Can someone tell me where this is coming from? Is it juce or individual plugin behaviour?

25

I admit, that I don’t fully understand what leads to the error.

  • I have no problem when running strictness 5
  • When running strictness 10 I get this output:
22: -----------------------------------------------------------------
22: Starting tests in: pluginval / Plugin state restoration...
22: !!! Test 14 failed: Filter Type not restored on setStateInformation -- Expected value  within 0.1 of: 0.666667, Actual value: 0
22: 
22: Time taken to run test: 137 ms
22: FAILED!!  1 test failed, out of a total of 17
22: -----------------------------------------------------------------
  • The FilterType is an AudioParameterChoice, save and restoring is APVTS with replaceState, it happens with a vanilla project only adding this one parameter

However:

  • It fails on our CI, but I was unable to get it to fail on my local machine
  • The CI runs the binary from tracktion/pluginval release 1.0.3 on windows and macOS self hosted runners

I didn’t understand the failures, because where is originalState coming from? I didn’t see where the state was modified before originalState was retrieved.

So what I tried was to limit the test to the ones we know how the plugin has to handle it rather than skipping the discrete parameter alltogether. I am fully aware, that this leaves out scenarios which I believed we would not know what the expected answer was. But I could be wrong with that. You have certainly more experience with that topic.

I genuinely don’t care how it is resolved, I just tried to be helpful.
I am happy with any solution so that our juce plugins succeed strictness 10.

26

FWI here is my case (seems to be related to this post :innocent:). I am using pluginval 1.0.3 locally on macOS 14.5. And pluginval only fails:

  • on the test Plugin state restoration
  • on parameters that can be modified by another meta parameter
  • at a high stricness level (e.g., 10)
  • when the plugin is AU (VST3 passes all tests)

Here is the output (Dynamic Bypass** is a bool):

Starting tests in: pluginval / Plugin state restoration...
!!! Test 219 failed: Dynamic Bypass01 not restored on setStateInformation -- Expected value  within 0.1 of: 0, Actual value: 0.253267
!!! Test 220 failed: Dynamic Bypass02 not restored on setStateInformation -- Expected value  within 0.1 of: 0, Actual value: 0.487158
!!! Test 223 failed: Dynamic Bypass05 not restored on setStateInformation -- Expected value  within 0.1 of: 0, Actual value: 0.406759
!!! Test 229 failed: Dynamic Bypass11 not restored on setStateInformation -- Expected value  within 0.1 of: 0, Actual value: 0.253966
27

I think this test is highlighting a genuine bug somewhere, possibly in the juce parameter handling. It’s highlighting an inconsistency in how plugin values are stored/returned which is violating the state invariants.

The value the parameters have at the start of the test is theoretically random, they’ll be left with whatever the previous test set them to but it’s highlighting that they’re not being properly validated and snapped internally.

This is actually one of the reasons for running the tests multiple times and in random orders, to catch this kind of thing.

1 Like
28

But you think this is likely a bug inside JUCE somewhere, and not an issue with plugin code?

29

Honestly I’m not sure. Is everyone having this problem using APVTS? Or is it wider than that?

I’m not sure if the juce team would even consider it a “bug”. Remember tests on levels 6-10 are more like “good practice warnings that indicate possible problem areas”.

1 Like
30

I’ve encountered this issue in plugins not using the APVTS.

31

I think there’s definitely an inconsistency in that both AudioParameterInt and AudioParameterChoice implement getValue and setValue as

float getValue() const
    { return convertTo0to1 (value); }
void setValue (float newValue)
    { value = convertFrom0to1 (newValue); valueChanged (get()); }

but AudioParameterBool just does

float getValue() const         { return value; }
void setValue (float newValue) { value = newValue; valueChanged (get()); }

Maybe this would fix it:

void setValue (float newValue)
    { value = (newValue >= 0.5f ? 1.0f : 0.0f); valueChanged (get()); }

This was already discussed here, btw.

(edit) And the fix was already proposed here, in this thread… I think the confusion was caused by

@dave96 mentions in the other thread that it might be required by auval that a parameter set to some odd value should return that odd value.

which doesn’t make sense to me (it already doesn’t happen for any type other than Bool), and is also contradicted by Dave himself here

As far as I can remember, there is no test in pluginval that sets a random parameter value and expects to be able to read that same value back is there?

2 Likes
32

This is only a contradiction in terms of different tools, auval vs pluginval. And like I said, I can’t remember exactly what auval tests for, maybe it doesn’t do this test and if so that’s good, pluginval doesn’t either and that’s consistent.

Can I clarify if it is only AudioParameterBool’s that are causing this failure?
If so, yes I think this would fix it:

Or even using the range member and the same body as the AudioParameterInt/AudioParameterChoice. E.g. (not tested):

float getValue() const
    { return range.convertTo0to1 (value); }
void setValue (float newValue)
    { value = range.convertFrom0to1 (newValue); valueChanged (get()); }

Does making that local change fix the problem for everyone here? If so, maybe we can propose this to the juce team?

Question on pluginval
33

Yep so far I’m pretty sure I’ve seen it only with AudioParameterBools, and the change above fixes it.

More precisely: if the change is made to the plugin’s underlying JUCE, then it successfully fixes the problem with the AU plugin, but it still occurs (I think more rarely) with VST3s. When the fix is also applied to pluginval’s underlying JUCE, it fully fixes the problem for VST3s as well.

34

FWIW I had the same issue with AudioParameterChoice.

35

That’s interesting…

That must mean that some plugin values have been set without going via the AudioParameterChoice::setValue?

Do you know where that happens?

36

I don’t want to lose momentum on this but as I’ve not experienced it myself can someone do me a favour and create a minimal example of a failure? If you can do it as one of the juce example demos I can create a fork of juce and a branch of pluginval to use that fork. Then I can add the changes discussed to the forked juce and see if that does fix the problem.

Once we have some evidence and a proposed fix I can PR that back to juce for consideration.

It does look like we’ll need an example with both bool and choice parameters in though after this comment:

If someone can do that I’ll take it from there.

Thanks.

1 Like
37

Had this on my backlog for a while and decided to have a go finally. And it’s interesting! It looks like we have two quite different bugs here.

First of all, I created a minimal example:
CMakeLists.txt (1.3 KB)
PluginProcessor.cpp (4.3 KB)

It just has three bool and two choice parameters. Originally the sequence was bools first, then choices (will be important later on). The uploaded version has the choices first.

First the Audio Unit plugin:

  • I’m running pluginval (current develop, unmodified JUCE) with strictness 10 and 10 repeats, as the error doesn’t always occur.
  • With current JUCE develop, the state restore test fails occasionally, and all three bool parameters are affected. The choice parameters are not affected.
  • With the proposed change (modifying setValue() so it can only set to 0 or 1), the problem disappears completely. The change here is only applied to the plugin’s JUCE, not pluginval’s!

Now the interesting part, the VST3 plugin:

  • Again pluginval with strictness 10 and 10 repeats
  • The state restore test fails much more rarely, and interestingly it only fails for the first bool parameter.
  • Making the proposed change to the plugin’s JUCE makes no difference
  • Making the proposed change to pluginval’s JUCE makes no difference either!
  • Putting the choice parameters first results in the test failing for the first choice parameter only!
  • Adding a float parameter at the beginning of the parameter list makes the errors go away.
  • Giving the float parameter an interval so it has e.g. 5 steps does not trigger the error (after all I would have expected that, but also in the log I see that for the stepped float parameter, 2147483647 steps are reported, which is wrong and probably yet another bug?).

So it’s definitely (at least) two problems we’re dealing with here, and the second one doesn’t seem related to the first at all.

I’m confused and hungry now.

4 Likes
Pluginval clarification
38

curious if there has been any resolution, im running into this on VST3 tests for:

  • AudioParameterChoice parameters
  • AudioParameterBool parameters
  • parameters modified by other meta parameters
39

FWIW I’m seeing this too - maybe one in ten runs, always on bool parameters. Curious if there’s been any progress. Maybe I’ll just set the random seed to something that always passes, ugh?