I am also experienced VST3 resizing weirdness in Reaper (64bit, OS X).
In my case I have observed that if either
setResizeLimits is present in the editor’s constructor, then the width and height defined in
setSize are ignored, and the plugin GUI is initially scaled to fill Reaper’s FX window.
setFixedAspectRatio is also present then resizing Reaper’s FX window by dragging its edges will cause the plugin GUI to be resized, which is not the case with VST2 plugins (I believe the plugin should only resize when Juce’s resizable corner component is dragged). The GUI will “jump” around the screen as the FX window is resized, and might even be positioned off screen.
I have observed this behaviour in VST3 versions of commercial Juce plugins by other developers.
Here is example code that will reproduce the problem:
setResizeLimits(600, 400, 1200, 800);
setSize (600, 400);