How to embed VST3 plugin editor inside current application window instead of opening a new window

I have used the plugin example to get my app to the point where I can open VST3 plugins inside a new window. However, I would like to just embed the editor component inside the current application window, not open a new window. I also would not like the plugin editor to steal keyboard focus. Is this possible to do? The app is linux only and the plugins would be VST3 only. There would also only ever be 1 plugin being shown at a time

See this screenshot from Reaper which shows the editor inside the FX window for an example of what I would like to accomplish:
image

Edit:

So I have made some sort of hack-like progress. I definitely do not feel like this is the way to do it but it gives me hope. I have a plugin view component which I kind of modeled after the juce ViewPort. This plugin view has a weak reference to a content component, along with a setViewedComponent method which sets this weak reference to the component that is to be displayed. When I want to show the plugin editor I open the window using plugin->showWindowExplicitly(). Then I use the setViewedComponent method on my plugin view and pass in the child component at index 1 of the plugin window (its index 1 since my plugin is resizable and index 0 seems to be just the resizable content component or something, not the actual editor)

// plugin is a tracktion_engine:::Plugin*
 plugin->showWindowExplicitly();
 pluginView->setViewedComponent(plugin->windowState->pluginWindow->getChildComponent(1));

This produces the following:

The black window is the plugin window, while the grey window is my actual app. The editor is in my app window, and resizing the app works as expected. Once I close the external plugin window the editor disappears though. Also it seems the new window steals keyboard focus away which I don’t want.

If you based your project on the PluginWindow in PluginWindow.h then change its inheritance from using DocumentWindow to Component. Then you should be able to add that component to your plugin hierarchy.

I’m not sure of anyway to stop the plugin from stealing keyboard keystrokes once it has focus. You might be able to steal focus back to your app.

@g-mon I have tried that as well but it seems the component is empty as I just get an empty screen when I add it to the component hierarchy using addAndMakeVisible. The component is being added to the hierarchy as the number of children increases by 1, but nothing seems to be there.

Here is how I attempt to add a component (the screen has a list box on it that displays the plugin names, when you select an item this code runs:) Note: editor is a juce::Component* member variable

  int selectedRow = listBox.getSelectedRow();
                if (selectedRow != -1)
                {
                    //currentTrackView->showPlugin(listModel->getPluginList()[selectedRow]);
                    listModel->getPluginList()[selectedRow]->showWindowExplicitly();
                    editor = listModel->getPluginList()[selectedRow]->windowState->pluginWindow.get();
                    listBox.setVisible(false);
                    DBG("before child component num children: " + juce::String(getNumChildComponents()));
                    addAndMakeVisible(editor);
                    DBG("after adding child component num children: " + juce::String(getNumChildComponents()));
                    editor->setBounds(getLocalBounds());
                    editor->toFront(true);
                }

I inherited from component and removed the window related code:

#pragma once
#include <tracktion_engine/tracktion_engine.h>
bool isDPIAware (tracktion_engine::Plugin&)
{
    // You should keep a DB of if plugins are DPI aware or not and recall that value
    // here. You should let the user toggle the value if the plugin appears tiny
    return true;
}

//==============================================================================
class PluginEditor : public juce::Component
{
public:
    virtual bool allowWindowResizing() = 0;
    virtual juce::ComponentBoundsConstrainer* getBoundsConstrainer() = 0;
};

//==============================================================================
struct AudioProcessorEditorContentComp : public PluginEditor
{
    AudioProcessorEditorContentComp (tracktion_engine::ExternalPlugin& plug) : plugin (plug)
    {
        JUCE_AUTORELEASEPOOL
        {
            if (auto pi = plugin.getAudioPluginInstance())
            {
                editor.reset (pi->createEditorIfNeeded());

                if (editor == nullptr)
                    editor = std::make_unique<juce::GenericAudioProcessorEditor> (*pi);

                DBG("editor children: " + juce::String(editor->getNumChildComponents()));
                DBG("editor name: " + juce::String(editor->getName()));
                DBG("adding the editor to the content component");
                addAndMakeVisible (*editor);
                DBG("editor is visible: " + juce::String(float(editor->isVisible())));
            }
        }

        resizeToFitEditor (true);
    }

    bool allowWindowResizing() override { return false; }

    juce::ComponentBoundsConstrainer* getBoundsConstrainer() override
    {
        if (editor == nullptr || allowWindowResizing())
            return {};

        return editor->getConstrainer();
    }

    void resized() override
    {
        if (editor != nullptr)
            editor->setBounds (getLocalBounds());
    }

    void childBoundsChanged (Component* c) override
    {
        if (c == editor.get())
        {
            plugin.edit.pluginChanged (plugin);
            resizeToFitEditor();
        }
    }

    void resizeToFitEditor (bool force = false)
    {
        if (force || ! allowWindowResizing())
            setSize (juce::jmax (8, editor != nullptr ? editor->getWidth() : 0), juce::jmax (8, editor != nullptr ? editor->getHeight() : 0));
    }

    tracktion_engine::ExternalPlugin& plugin;
    std::unique_ptr<juce::AudioProcessorEditor> editor;

    AudioProcessorEditorContentComp() = delete;
    JUCE_DECLARE_NON_COPYABLE_WITH_LEAK_DETECTOR (AudioProcessorEditorContentComp)
};

//=============================================================================
class PluginWindow : public juce::Component
{
public:
    PluginWindow (tracktion_engine::Plugin&);
    ~PluginWindow() override;

    static std::unique_ptr<Component> create (tracktion_engine::Plugin&);

    void show();

    void setEditor (std::unique_ptr<PluginEditor>);
    PluginEditor* getEditor() const         { return editor.get(); }

    void recreateEditor();
    void recreateEditorAsync();

private:

    std::unique_ptr<PluginEditor> createContentComp();

    std::unique_ptr<PluginEditor> editor;

    tracktion_engine::Plugin& plugin;
    tracktion_engine::PluginWindowState& windowState;
};

//==============================================================================
#if JUCE_LINUX
constexpr bool shouldAddPluginWindowToDesktop = false;
#else
constexpr bool shouldAddPluginWindowToDesktop = true;
#endif

PluginWindow::PluginWindow (tracktion_engine::Plugin& plug)
        : plugin (plug), windowState (*plug.windowState)
{


    auto position = plugin.windowState->lastWindowBounds.getPosition();
    setBounds (getLocalBounds() + position);

    recreateEditor();

}

PluginWindow::~PluginWindow()
{
    plugin.edit.flushPluginStateIfNeeded (plugin);
    setEditor (nullptr);
}

void PluginWindow::show()
{
    setVisible (true);
    toFront (false);
}

void PluginWindow::setEditor (std::unique_ptr<PluginEditor> newEditor)
{
    JUCE_AUTORELEASEPOOL
    {

        editor.reset();

        if (newEditor != nullptr)
        {
            editor = std::move (newEditor);
        }

    }
}

std::unique_ptr<juce::Component> PluginWindow::create (tracktion_engine::Plugin& plugin)
{
    if (auto externalPlugin = dynamic_cast<tracktion_engine::ExternalPlugin*> (&plugin))
        if (externalPlugin->getAudioPluginInstance() == nullptr)
            return nullptr;

    std::unique_ptr<PluginWindow> w;

    {
        struct Blocker : public Component { void inputAttemptWhenModal() override {} };

        Blocker blocker;
        blocker.enterModalState (false);

#if JUCE_WINDOWS && JUCE_WIN_PER_MONITOR_DPI_AWARE
        if (! isDPIAware (plugin))
        {
            ScopedDPIAwarenessDisabler disableDPIAwareness;
            w = std::make_unique<PluginWindow> (plugin);
        }
        else
#endif
        {
            w = std::make_unique<PluginWindow> (plugin);
        }
    }

    if (w == nullptr || w->getEditor() == nullptr)
        return {};

    w->show();

    return w;
}

std::unique_ptr<PluginEditor> PluginWindow::createContentComp()
{
    if (auto ex = dynamic_cast<tracktion_engine::ExternalPlugin*> (&plugin))
        return std::make_unique<AudioProcessorEditorContentComp> (*ex);

    return nullptr;
}

void PluginWindow::recreateEditor()
{
    setEditor (nullptr);
    setEditor (createContentComp());
}

void PluginWindow::recreateEditorAsync()
{
    setEditor (nullptr);

    juce::Timer::callAfterDelay (50, [this, sp = SafePointer<Component> (this)]
    {
        if (sp != nullptr)
            recreateEditor();
    });
}

//==============================================================================
class ExtendedUIBehaviour : public tracktion_engine::UIBehaviour
{
public:
    ExtendedUIBehaviour() = default;

    std::unique_ptr<juce::Component> createPluginWindow (tracktion_engine::PluginWindowState& pws) override
    {
        if (auto ws = dynamic_cast<tracktion_engine::Plugin::WindowState*> (&pws))
            return PluginWindow::create (ws->plugin);

        return {};
    }

    void recreatePluginWindowContentAsync (tracktion_engine::Plugin& p) override
    {
        if (auto* w = dynamic_cast<PluginWindow*> (p.windowState->pluginWindow.get()))
            return w->recreateEditorAsync();

        UIBehaviour::recreatePluginWindowContentAsync (p);
    }
};

I ended up ditching all the window stuff and just using createEditorIfNeeded directly and it seems to work (still cant gain back keyboard focus even after calling grabKeyboard focus)

std::unique_ptr<juce::Component> createPluginWindow (tracktion_engine::PluginWindowState& pws) override
    {

        std::unique_ptr<juce::AudioProcessorEditor> editor;
        if (auto ws = dynamic_cast<tracktion_engine::Plugin::WindowState*> (&pws))
        {

            if (auto externalPlugin = dynamic_cast<tracktion_engine::ExternalPlugin*> (&(ws->plugin)))
            {

                if (auto pi = externalPlugin->getAudioPluginInstance())
                {
                    editor.reset(pi->createEditorIfNeeded());

                    if (editor == nullptr)
                    {
                        DBG("using generic editor");
                        editor = std::make_unique<juce::GenericAudioProcessorEditor> (*pi);
                    }

                    return editor;
                }

            }

        }

        return {};
    }

Is this a bad idea?

That’s fine. Some plugins can’t resize, you you’ll need to figure out how you’ll deal with that in your UI.

Do you have a component in your UI that will accept focus? Text Editor or similar?

the parent of my list box accepts focus, and it actually doesnt seem to be a focus issue because after adding the plugin editor to my application UI window, I checked using hasKeyBoardFocus and the plugin editor does not have focus, and the parent of my listbox does indeed have focus. However, my application command manager keyboard commands are not working despite the list box parent (which is an application command target) having focus and the commands were working prior to adding the editor.