Ohh, I did some experiments and replaced AudioBuffer<T> with my custom float “c array” and got it working well in GarageBand.
I do not know why this is so, perhaps GarageBand has something to do with memory allocation in the heap. The load even dropped by 5%