Could it be possible to generate an unique key for each instance of the plugin, and then fetch related data with that key in a global shared instance? It might be clunky to achieve thread safety and such, but after all it is just a map of few pointers.
Note that I’m not familiar with plugin dev thus it can be a totally dumb approach.
Note that if you can avoid that it is better. You could just pass the adress of the stuff into the constructor of your objects to store them as a (reference) (usually named owner) member. In that case life-time must be extra double checked. But IMHO most of time the overall main class remains from start to end, thus is easy to manage.