That dialogue probably selects output channels for the first output bus.
To make it work you can either make it a single 6-channel output bus or augment the audio setup window to support configuring multiple busses.
Remember getting it to work on multi-channel audio by selecting 1+2, 3+4 and 5+6 all together.
And enabling 5.1 channel format in isBusesLayoutSupported.