When I try things like this I start returning true for every setup in isBusesLayoutSupported.
The version that supports only your desired layout should look like that:
bool isBusesLayoutSupported (const BusesLayout& layout) override
if (layout.getChannelSet (true, 0) == AudioChannelSet::stereo() &&
layout.getChannelSet (false, 0) == AudioChannelSet::stereo() &&
layout.getChannelSet (false, 1) == AudioChannelSet::mono())
But chances are, the plugin will not show at all, if that specific setup is not supported by the host.
@Xenakios knows a lot about Reaper, let’s hope he chimes in…
EDIT: added missing return false