Issues when calling XSetTransientForHint with windows returned by Component::getWindowHandle()

Audio Plugins
1

First off, I’m new to juce and very new to X11, so apologies in advance if my post doesn’t make sense

I’ve been trying to get a hosted plugin’s native editor window to stick on top of the host’s window (I named the function that should do this set_component_native_owning_window). I read that this can be accomplished with XSetTransientForHint, so I tried it, but the function fails every time, returning code 1. I tried using an error handler with XSetErrorHandler, but the handler callback never gets called, so I don’t know what exactly the error was
After that, I tried calling XStoreName with both windows, and it failed both times. So that leads me to believe that this is a general issue with X11 functions, not something specific to XSetTransientForHint
From there I did some poking around in juce’s files to see how it was calling X11 functions, and I discovered that juce doesn’t use the functions in Xlib.h directly. Instead it seems to load them dynamically as function pointers from libX11.so.6 in juce_XSymbols_linux.cpp and then call them like this: juce::X11Symbols::getInstance()->Xfunc().
At that point I started thinking that juce was just using a different version of X11 internally, and that these errors were the result of ABI incompatibilities, so I edited juce’s files to load XSetTransientForHint from the .so (they weren’t loaded by default), and edited my code to use juce::X11Symbols::getInstance() for all of my X11 calls
And the exact same error still occurred!
Honestly I’m at a loss. I must be doing something fundamentally wrong here. Can anyone that’s familiar with X11 help me out here?

My code:


#include "X11/Xlib.h"


int handler(Display * d, XErrorEvent * e) {
    return 0; // the handler does nothing, I just set a breakpoint here to see if it gets called. It never does
}

void set_handler() { // I call this in my editor's constructor
    XSetErrorHandler(handler);
}

void set_component_native_owning_window(juce::Component& to_be_owned, juce::Component& to_be_owner) {

    auto display = XOpenDisplay(nullptr); // open the display referenced by the DISPLAY environment variable

    if(display) {
        auto to_be_owned_window = reinterpret_cast<Window>(to_be_owned.getWindowHandle()), // I inspected bit in gdb, and determined that the window handles have different values, so that isn't the issue
             to_be_owner_window = reinterpret_cast<Window>(to_be_owner.getWindowHandle());

        if(int ret = XSetTransientForHint(display, to_be_owner_window,
                                                   to_be_owned_window)) { // XSetTransientForHint failed. This happens every time

            auto str = get_x_error(ret); // I inspect this variable from gdb
        }
    }
    else { // XOpenDisplay failed. Doesn't happen in my testing

    }
}

Miscellaneous information:

  • distro is ubuntu
  • DAW is reaper
  • compiler is g++
  • all function shown are called from the message thread
2

Its a bit of a mistake to alter JUCE’ code after ignoring the errors you encountered with XSetTransientForHint and XSetErrorHandler, because you’re just adding complexity to a very complex situation, already.

Better to learn to use juce::X11Symbols->someFunc()

You seem to have swapped the ID’s in your XSetTransientForHint() call, btw. In your code, you’re passing to_be_owner_window as the transient_window and to_be_owned_window as the parent_window, which is the reverse of what you intend. You want the to_be_owned window to be transient for the to_be_owner window.

XSetTransientForHint depends on the window manager supporting the hint - some window managers ignore it. Did you determine this isn’t the case with your WM? Try different window managers, because XSetTransientForHint is not globally respected by all wm’s…

Are you calling the X11 functions on the UI thread, for sure?

Your error handler might not be working because X11 errors are asynchronous, and you need to call XSync to ensure errors are flushed and the handler is invoked. Maybe JUCE is doing this somewhere, maybe you have issues in the thread/event-loop setup which preclude this from happening.

Lastly: why doesn’t JUCE’ own setAlwaysOnTop() work for you? Maybe it also is using techniques ignored by your window manager …

Try this fixed code:

#include <juce_gui_basics/juce_gui_basics.h>
#include <X11/Xlib.h>
#include <iostream>

// Error handler for X11 errors
static int xErrorHandler(Display* d, XErrorEvent* e) {
    char error_msg[256];
    // Use JUCE's X11Symbols to get error text
    juce::X11Symbols::getInstance()->XGetErrorText(d, e->error_code, error_msg, sizeof(error_msg));
    std::cerr << "X11 Error: " << error_msg << " (code: " << e->error_code
              << ", resource: " << e->resourceid << ")" << std::endl;
    return 0;
}

// Set the error handler (call in editor's constructor)
void set_handler() {
    juce::X11Symbols::getInstance()->XSetErrorHandler(xErrorHandler);
}

void set_component_native_owning_window(juce::Component& to_be_owned, juce::Component& to_be_owner) {
    // Use JUCE's display to ensure consistency
    Display* display = juce::X11Symbols::getInstance()->XOpenDisplay(nullptr);
    if (!display) {
        std::cerr << "Failed to open X11 display" << std::endl;
        return;
    }

    // Get window handles from JUCE components
    Window to_be_owned_window = reinterpret_cast<Window>(to_be_owned.getWindowHandle());
    Window to_be_owner_window = reinterpret_cast<Window>(to_be_owner.getWindowHandle());

    // Validate window handles
    XWindowAttributes attrs;
    if (!juce::X11Symbols::getInstance()->XGetWindowAttributes(display, to_be_owned_window, &attrs)) {
        std::cerr << "Invalid to_be_owned window: " << to_be_owned_window << std::endl;
        juce::X11Symbols::getInstance()->XCloseDisplay(display);
        return;
    }
    if (!juce::X11Symbols::getInstance()->XGetWindowAttributes(display, to_be_owner_window, &attrs)) {
        std::cerr << "Invalid to_be_owner window: " << to_be_owner_window << std::endl;
        juce::X11Symbols::getInstance()->XCloseDisplay(display);
        return;
    }

    // Set the to_be_owned window as transient for the to_be_owner window
    Status result = juce::X11Symbols::getInstance()->XSetTransientForHint(display, to_be_owned_window, to_be_owner_window);
    // Flush errors to ensure the handler is called
    juce::X11Symbols::getInstance()->XSync(display, False);

    if (result == 0) {
        std::cerr << "XSetTransientForHint failed" << std::endl;
    } else {
        std::cout << "XSetTransientForHint succeeded" << std::endl;
    }

    // Clean up
    juce::X11Symbols::getInstance()->XCloseDisplay(display);
}
1 Like
3

Thanks for the response!

I tested your code and the error handler is getting called now! It look like a BadWindow error is occurring. XSetTransientForHint also succeeds, but after comparing your code and my code, I realized that the failure check in my original post was incorrect (XSetTransientForHint returns nonzero on success, not on failure), so that means XSetTransientForHint was succeeding the whole time. The behavior of the windows is weird though, probably because the parent window is bad. The owned window sticks on top of everything, not just the owner window, and the contents are completely gray

I have to step away from my computer for a bit, but I’ll test the code more and post a detailed response once I’m back and get everything figured out

Thanks again!