I’m on macOS 13.4 (Apple Silicon).
Attaching and detaching a USB audio device works fine unless it’s the selected input device. If so I get errors like below, another audio device gets selected and it doesn’t work. When selecting another audio device in the Audio Settings dialog things recover.
What I would like to happen is that when I remove the selected audio device, another audio device gets selected and it works.
Is this a bug or should I handle it in some way? 
2023-05-23 20:43:47.707885+0200 Gaudiamus[72522:834955] HALC_ProxyIOContext.cpp:1487 HALC_ProxyIOContext::IOWorkLoop: failed to send the final message to the server, Error: 0x10000003
2023-05-23 20:43:47.729383+0200 Gaudiamus[72522:835175] HALC_ProxyIOContext.cpp:1070 HALC_ProxyIOContext::IOWorkLoop: the server failed to start, Error: 0x6E6F7065
2023-05-23 20:43:47.729488+0200 Gaudiamus[72522:835175] HALSystem.cpp:2145 AudioObjectPropertiesChanged: no such object
2023-05-23 20:43:47.729544+0200 Gaudiamus[72522:833869] [AMCP] 5135 HALC_ProxyIOContext.cpp:816 HALC_ProxyIOContext::_StartIO(): Start failed - StartAndWaitForState returned error 1852797029
2023-05-23 20:43:47.801962+0200 Gaudiamus[72522:831251] AudioHardware-mac-imp.cpp:409 AudioObjectHasProperty: no object with given ID 135
2023-05-23 20:52:08.162644+0200 Gaudiamus[72522:831251] HALC_ShellObject.mm:229 HALC_ShellObject::HasProperty: call to the proxy failed, Error: 268435459 (\^P)
2023-05-23 20:52:08.168846+0200 Gaudiamus[72522:843583] HALC_ProxyIOContext.cpp:1070 HALC_ProxyIOContext::IOWorkLoop: the server failed to start, Error: 0x6E6F7065
2023-05-23 20:52:08.168945+0200 Gaudiamus[72522:843583] HALSystem.cpp:2145 AudioObjectPropertiesChanged: no such object
2023-05-23 20:52:08.168995+0200 Gaudiamus[72522:843316] [AMCP] 96039 HALC_ProxyIOContext.cpp:816 HALC_ProxyIOContext::_StartIO(): Start failed - StartAndWaitForState returned error 1852797029
2023-05-23 20:52:08.266711+0200 Gaudiamus[72522:831251] AudioHardware-mac-imp.cpp:409 AudioObjectHasProperty: no object with given ID 165