OK after more digging and searching for potential memory limitations, I stumbled over this post:
According to Core Audio engineering the memory limit imposed for AU Extensions on iOS is 300 MB for 32-bit devices and 360 MB on 64-bit devices. These are performance considerations as you can expect from iOS devices having limited shared resources. [source]
This would explain the crashs (I am still dramatically underwhelmed by the level of error reporting on iOS).
This problem might also be related to this forum post:
and +1 for a virtual method AudioProcessor::didReceiveMemoryWarning