Glitch and Synchronizing audio - multiple instances, same plugin

I am working on a plugin that requires audio buffers from each instance of a single plugin to be shared between other instances (similar in some ways to sidechaining).
I am currently using global buffers to shuffle around the data and have locking in the audio callbacks. I am trying to avoid unnecessary high latency IPC mechanisms since this is all within same plugin.

What I find is that it generally works ok, however when the Host becomes very busy, or when there is a bunch of UI stuff needing attention in the host - the audio will glitch and stay glitchy until the Host stops play and then is restarted.
My guess is that the order each instance is called to fetch its buffer is changing when the host gets busy, or perhaps the audio callback in the plug is being interrupted by higher priority host action.

Does anyone have any suggestions to what may be the cause of the glitching, and how I may be able to solve it?
Ie; ways to ensure order and sync of multiple instances?