[solved] Has anyone been able to build the RTAS SDK with __cdecl (yes, the doc says it is possible)

As per subject, I have a plug-in that perfectly builds with Visual Studio 2013 also in RTAS format using JUCE 4.2.2.

The problem is that, for reasons that are too complicated to be explained here (maybe in another topic), I am facing a blocking issue that would be easily resolved if I could use the cdecl calling convention everywhere, in contrast to having to use stdcall for building the PT SDK libraries and the JUCE RTAS wrapper cpp files only.

The PT SDK documentation clearly states that this should be possible:

Starting with the 6.1 Plug-in SDK, you can use a calling convention other than __stdcall. However, your plug-in project and the plug-in library project must both be set to the same calling convention. Also starting with the 6.1 version, you can set your structure alignment to any setting larger than 2 bytes. However, the alignment for all projects in the workspace or solution must be the same.

Of course I have done all of that, setting cdecl and and alignment of 8 bytes throughout all the projects in my solution (and also for those specific RTAS wrapper cpp files).

Surprisingly enough, even with all those changes the project builds, without linking errors, but the resulting plug-in does not appear in Pro Tools at all, and when I close Pro Tools I get an error message stating that my plug-in has accessed a forbidden location in heap space (thus I understand that my plug-in IS being probed by Pro Tools somehow)

Please also note that this is not a PACE signing/wrapping issue, because if I revert the calling convention changes back to stdcall, the resulting binary (which is wrapped in a post-build step of the project) is listed just fine in Pro Tools.

I know this is a shot in the dark, but maybe someone has suggestions in this area…

Can you describe the blocking issue you have? I’m curious as to what __cdecl would offer to your specific problem.

Sorry for the delay, more urgent tasks have diverted my attention until now.

I suggest to anyone who is going to read this to grab a cup of coffee and some concentration: I fear this won’t be neither easy nor short.

Now, I will write this while redoing the whole thing, as a further attempt to get some inspiration for a viable solution.

So, I am working on this cross-platform plug-in mainly on a Mac, and occasionally build it on Windows for testing/debugging before releases.

During one of these dev sessions on Windows, the plug-in started to consistently hit a jassert located in the LeakedObjectDetector destructor.

The DBG message associated with the jassert is:

*** Dangling pointer deletion! Class: KeyPress

This puzzled me because none of the code that was recently modified in the project has anything to do with KeyPress instances at all.

I spent an entire day hunting for KeyPress mismatched allocations (and I use RAII a lot, so that was a strange symptom anyway), but I could find literally nothing wrong.

Out of desperation, to see if I was going mad, I tried the following code:

    KeyPress p;     // (1) an instance of KeyPress is created here
}                   // (2) and it is destroyed upon exit from this curly

This literally cannot trigger any leak-detection-related issue, right, RIGHT?

When stepping over line (2), I get that dreaded assertion.

Ok, so… something is seriously wrong with what’s happening here, and it is not due to actual mismatched allocations, double deletions or memory leaks, becasue the code above is certainly correct but still it hits the assert.

In addition, if really there were a double deletion somewhere in the code, that should trigger the assertion also on the Mac build, which shares exactly the same code. But on Mac it runs just fine without assertions.

Out of ideas on how to investigate this, I check if the assertion is consistently hit in all the targets of the Windows build, which are:

  • 64-bit stand-alone app target -> no assertions hit
  • 32-bit stand-alone app target -> no assertions hit
  • 64-bit polymorphic plug-in target (VST2, VST3, AAX) -> no assertions hit
  • 32-bit polymorphic plug-in target (VST2, VST3, AAX, RTAS) -> assertion is hit only in this config!

So, it only happens in the 32-bit plug-in build. What makes that different from all the others? RTAS

To confirm my suspicions, if I disable the RTAS build in the offending target with:

#define JucePlugin_Build_RTAS 0

then rebuild and run, no assertions are hit with RTAS disabled.

Ok… so I have the culprit, or at least a strong clue, but RTAS support is mandatory for this project and disabling it is not an option.

Furthermore, still it is unexplained what RTAS has to do with that absurd jassert regarding KeyPress, so I went further.

For a fact, I know that the assert is being hit when the LeakCounter object for the KeyPress class is decremented past 0, going into the negative.

The LeakCounter instance for a given class is a singleton, obtained in all places where it is needed by calling the static method LeakedObjectDetector::getCounter().

Then, to follow what is happening to the LeakCounter for the KeyPress class, I added the printouts shown below:

static LeakCounter& getCounter() noexcept
    static LeakCounter counter;

    // ***** start code added by me
    if (strcmp (getLeakedObjectClassName(), "KeyPress") == 0)
        DBG (String::formatted ("counter object: %p, value: %d", &counter, counter.numObjects.get()));
    // ***** end code added by me

    return counter;

And then I watched what was being printed when running this seemingly harmless code, previously mentioned:

    KeyPress p;     // (1) an instance of KeyPress is created here
}                   // (2) and it is destroyed upon exit from this curly

This is the result:

counter object: 0DE6CC70, value: 0  // printed when stepping over line (1)
counter object: 0DE6BE90, value: 0  // printed when stepping over line (2)

Oh well, this means that:

  • When the KeyPress instance p is created at line (1), the LeakCounter instance which is incremented has address 0DE6CC70

  • When the KeyPress instance p is destroyed at line (2), the LeakCounter instance that is decremented has an address, 0DE6BE90, which is different from the one that was incremented in the previous line, which means that it is not the same LeakCounter object, but another one.

This ultimately results in line (2) trying to decrement a LeakCounter instance whose value is 0 already (as shown by the printout), and that in turns triggers the jassert that I am hitting.

The problem is clearly that the two lines (1) and (2) above are using two distinct LeakCounter instances, instead of the same for both as it would be expected.

But still, it is also true that both lines (1) and (2) are also getting their LeakCounter invoking that same getCounter() method where I have added my printouts, otherwise I wouldn’t be getting two lines printed to the output because of my DBGs.

The code of getCounter() is seen above, and it is quite simple. Without my printouts the original code is:

static LeakCounter& getCounter() noexcept
    static LeakCounter counter;
    return counter;

This is a common and well known pattern, for creating and returning singletons: the function constructs the static variable upon its first invokation, and then returns a reference to it for that and all following invokations.

So, how is it possible that the same getCounter() function is called, but it yields two different instances of LeakCounter in return?

That’s not possible, unless the getCounter() function being called is not really the same in both places…

As an additional check, I add to my DBG printouts the actual function pointer of the getCounter() being called, like this:

static LeakCounter& getCounter() noexcept
    static LeakCounter counter;

    if (strcmp (getLeakedObjectClassName(), "KeyPress") == 0)
        DBG (String::formatted ("function pointer %p, counter object: %p, value: %d", getCounter, &counter, counter.numObjects.get ()));

    return counter;

And the result is this:

function pointer 1017C030, counter object: 115ECC70, value: 0
function pointer 100C02A0, counter object: 115EBE90, value: 0

Oook, so this means that:

  • line (1) calls a getCounter() function, whose address is 1017C030, and obtains in return a LeakCounter object whose address is 115ECC70.

  • line (2) calls another getCounter() function, whose address is 100C02A0, and obtains in return a different LeakCounter object, whose address is 115EBE90

This somehow makes some sense: each of the two getCounter() functions (1017C030 and 100C02A0) creates and returns its static LeakCounter object (115ECC70 and 115EBE90, respectively).

But why do those two getCounter() functions even exist?
Only one should be generated by the LeakedObjectDetector class template, for any given argument class (KeyPress in this case)

So, let’s search the addresses of those funcion pointers inside the .map file generated by the linker, to see if it is possible to understand what they correspond to:

  • 1017C030 is matched in the following line:

    ?getCounter@?$LeakedObjectDetector@VKeyPress@juce@@@juce@@CAAAVLeakCounter@12@XZ 1017c030 f i juce_audio_processors.obj

    which makes sense, because that seems a legit decorated name for a function whose full name is LeakedObjectDetector<KeyPress>::getCounter().

  • 100C02A0 is matched in the following line:

    ?getCounter@?$LeakedObjectDetector@VKeyPress@juce@@@juce@@CGAAVLeakCounter@12@XZ 100c02a0 f i juce_audio_plugin_client_RTAS_1.obj

    the decorated name here looks surprisingly similar to the one above, with the only notable difference of one character (CGAAVLeakCounter instead of CAAAVLeakCounter).

This finding confirms my suspicions: the second function pointer corresponds to another LeakedObjectDetector<KeyPress>::getCounter(), different from the first one.

Ok, but what exactly makes them different?

Thanks to the undname.exe tool, provided with Visual Studio, I can get to know what exactly each of those decorated names mean:

the first corresponds to:

static LeakedObjectDetector<KeyPress>::LeakCounter& __cdecl LeakedObjectDetector<KeyPress>::getCounter()

while the second means:

static LeakedObjectDetector<KeyPress>::LeakCounter& __stdcall LeakedObjectDetector<KeyPress>::getCounter()

The difference is now clear: they are two copies of the same function, differing only because if their calling convention (cdecl and stdcall respectively).

Unsurprisingly, the one which uses stdcall comes from juce_audio_plugin_client_RTAS_1.obj, which makes sense because the RTAS plug-in wrapper is in fact built with that calling convention.

To confirm what I have found, if I replace the stdcall calling convention in every place where it is used (also in the PT SDK projects that build the libraries needed for RTAS), I can build the polymorphic 32-bit plug-in just fine, and it will load correctly (as a VST), showing the correct printouts:

function pointer 0CA2F8F0, counter object: 0DF5AE90, value: 0
function pointer 0CA2F8F0, counter object: 0DF5AE90, value: 1

but unfortunately, doing so results in Pro Tools 10 not seeing it as a RTAS plug-in any more (as I described in more detail in my first post).

And this, gentlemen, is the story of why I am searching for a way to build the PT SDK with cdecl calling convention, which the SDK documentation says it is entirely possible, but that I am somehow unable to put in practice.



But wait?! Why in the world does the destruction of a KeyPress instance in my simple code (which is in a cpp file built using cdecl) end up calling a getCounter() function which is defined in some other RTAS module, and that uses an entirely different calling convention???


1 Like

@yfede Did you try looking at the direction of JUCE calling convention?
maybe there’s a JUCE_CALLTYPE bug somewhere? (which is set in appropriate to what it should)
(just a suggestion / wonder)

Thanks for the suggestion, will look at that too

I’ve made it!!! :tada::confetti_ball:

No more stdcall in the way, and no more need to set it for those RTAS cpp files too!

And, more important, no more double copies of static functions due to the two calling conventions, so the compiler doesn’t get confused any more and that dreaded assertion is finally solved

Care to share how you fixed it in case anyone else comes looking?

Also, the reason you’re only getting bugs is that x64 has one uniform calling convention (per platform), contrary to x32.

yes, I am aware of this, but thank you for adding to that.
This means that even if RTAS were available as a format in the 64-bit build, the problem wouldn’t be there anyway in that case. I omitted this particular corollary from my explanation above because it was long enough already :slight_smile:

Yeah sure, I got a little carried away with the happiness for having solved this and I forgot to post my solution:

As anybody that has access to the PT SDK already knows, the PlugInLibrary project that is found there has two targets (for debug builds, the same is true for release builds): “Debug” and “Debug_stdcall+2byte”. The latter is the one to be used to build the library that correctly links with JUCE projects and yields working plug-ins.

As the name suggests, that one is the target that uses the “stdcall” calling convention, so for my attempts at using “cdecl” I built the lib using the regular “Debug” target instead (which, I checked, actually uses “cdecl”). The library built and linked just fine in that case, but the plug-in was not recognized (behavior already explained in my first post).

Then I realized that the “Debug_stdcall+2byte” cfg does not only set the calling convention, but it also sets the “struct member alignment” setting to “2 byte”, as the name suggests.

In consideration of that, I added to the PlugInLibrary project another target named “Debug_2byte”, copied from the “stdcall+2byte” one but with “calling convention” reverted back to the “cdecl” default, then used it to build a new .lib file named “PlugInLib_2byte.lib”.

I then set the “calling convention” to “cdecl” also for the RTAS cpp files in the JUCE project, changed the Visual Studio project to link against the newly built “PluginLib_2byte.lib” and… it worked.
It built, it linked and the resulting RTAS plug-in was correctly seen and instantiated by Pro Tools 10 on first attempt.