Deploying an ONNX model

Has anybody experience with deploying an ONNX model in an audio plugin to mac and win?

From my current understanding my first option would be to use onnx runtime as a dynamic library.
Using a DL in a plugin seems painful based on the posts here, since I would have to save the DL in the common DL folders on mac and windows, which then requires a DL with a custom name to avoid conficts, which then requires me to build onnxruntime myself with a custom name.

My second option would be to use ORT builder by the one and only @olilarkin, but I might be interested in GPU inference in the future and Im kind of worried about another dependency which might not stay up to date and the large windows binary size that is mentioned in its readme.

Edit: another company that makes really great plugins seems to store the onnxruntime DL in a custom location. Maybe they are using juce::DynamicLibrary to load it?

You can easily enable GPU execution providers in a fork of OrtBuilder. Also the large windows .lib is solved with the included patch file to disable LTO.

1 Like

here is some info about how you can link it statically in a cmake project: Static library usage · Issue #8555 · microsoft/onnxruntime · GitHub

and here someone is publishing non-customised static libs:

1 Like

We took the dynamic library path back when we integrated onnxruntime in one of our EQs a few years ago. ORT builder did not exist back then and our attempts to build a static library version of the onnx runtime failed especially due to clashes with the static linked protobuf dependency all our plugins have which clashed with the protofbuf version integrated in onnx runtime, which was of course incompatible with the version we used… This was such a big pain that we didn’t revisit that topic again since then and successfully sticked to the dynamic linked library approach.

We compile onnx runtime with a few custom tweaks ourselves and prefix the names of the resulting dynamic libraries with our company name and add a version number to it. Over time we updated to a newer version of onnx runtime to be able to access newer features in newer plugins. So far there were no problems running multiple older and newer plugins in the same host in parallel which both each link against their own onnx runtime version.

Not sure if you maybe mean us with that, but on macOS we definitively do that :wink: This is relatively easy to accomplish by modifying the install name path of the dylib and then indeed put it in a custom location. I’d need to scan through some build scripts to find out the exact command for that. On Windows we didn’t find a similar solution so we ended up installing our libraries in System32. What we didn’t know at that time was the Linker support for delay-loaded DLLs | Microsoft Learn approach which makes loading DLLs from a custom location also possible on Windows. juce::DynamicLibrary is an option if you load a bunch of function symbols from a library exposing a pure C interface. However with a complex C++ library like onnx runtime this is not really a suitable option. As you might know, C++ symbol names are mangled following a platform specific pattern which makes it hard to load such symbols manually from a library on the one hand and if you should succeed it’s really difficult to handle member classes and member functions correctly. So no, I wouldn’t suggest exploring that route :wink:

That all being said, depending on the complexity of your projects I’d definitively suggest to consider static linking as it avoids some hassle – while of course increasing the binary size of your plugins a bit.

Hope that helps a bit :slight_smile:

2 Likes

Thanks to both of you for the detailled answers!
I started going down the static route, but (in this short time) I was not able to make ort-builder work for my plugin project.
VST3 compiles, but on AU or AAX I get undefined symbols:

  onnx::propagateElemTypeFromInputToOutput(onnx::InferenceContext&, unsigned long, unsigned long), referenced from:
      onnxruntime::contrib::EmbedLayerNormalizationShapeInference(onnx::InferenceContext&) in libonnxruntime.a[arm64][4](shape_inference_functions.cc.o)
      onnxruntime::contrib::EmbedLayerNormalizationShapeInference(onnx::InferenceContext&) in libonnxruntime.a[arm64][4](shape_inference_functions.cc.o)
      onnxruntime::contrib::EmbedLayerNormalizationShapeInference(onnx::InferenceContext&) in libonnxruntime.a[arm64][4](shape_inference_functions.cc.o)
      onnxruntime::contrib::AttentionTypeAndShapeInference(onnx::InferenceContext&, int) in libonnxruntime.a[arm64][4](shape_inference_functions.cc.o)
      onnxruntime::contrib::AttentionTypeAndShapeInference(onnx::InferenceContext&, int) in libonnxruntime.a[arm64][4](shape_inference_functions.cc.o)
      onnxruntime::contrib::AttentionTypeAndShapeInference(onnx::InferenceContext&, int) in libonnxruntime.a[arm64][4](shape_inference_functions.cc.o)
      onnxruntime::contrib::AttentionTypeAndShapeInference(onnx::InferenceContext&, int) in libonnxruntime.a[arm64][4](shape_inference_functions.cc.o)
      ...

But using the non-customized static libs you suggested works for all formats, so thanks again for that :slight_smile:
Using the ort-builder version reduces the binary size of the VST3 file by 15 mb.

I was, smart limit and smart comp 2 rule!

1 Like

cant see why AU or AAX would be any different to VST3 w.r.t linking the static library… are you shure it was getting linked to those targets?