OpenGlContext and Threads


#1

Hello All,

In order to test my conception, I have basic application with a main component. This component is an OpenGLRenderer. It contains a context :

[code]MainComponent::MainComponent ()
{
openGLContext.setComponentPaintingEnabled (true);
openGLContext.attachTo(*this);
openGLContext.setSwapInterval(1);

setSize (600, 300);

}[/code]

A thread is running and I’d like updating the openglcontext of the maincomponent from this thread.

[code]void MyOwnThread::run ()
{
int i = 0;
while(!threadShouldExit())
{
i++;
displayComponent.openGLContext.makeActive(); ////////---->always fails
if(i == 0)
{
::OpenGLHelpers::clear(Colours::aqua);

    }
    else if(i == 1)
    {
        ::OpenGLHelpers::clear(Colours::red);
        
    }
    else if(i == 2)
    {
        ::OpenGLHelpers::clear(Colours::pink);
        
        i = -1;
    }
    displayComponent.openGLContext.triggerRepaint();
    sleep(1000);
}

}[/code]

Where am i wrong? (I am a beginner in opengl applications…)

Thanks


#2

Technically it’s possible to use your own thread, but it won’t give you any advantages that I can think of, and would be very difficult to set up correctly. Certainly wouldn’t do it that way myself.

Just use the thread that juce provides - have a look at way the GL demo works in the juce demo.


#3

Hi,

I’m a bit hijacking this thread, but I have a similar issue. I can’t use the thread Juce provides because I need to control its creations and its members, and most importantly its behaviour.
I can workaround the creation & members issue (except that I can’t assert if it’s the same thread each time that’s calling renderOGL, so it might break if you happen you change it).
However I don’t want the thread to call me at regular interval, I want to be able to make it wait for a specific amount of time (I don’t want 60fps in my application).

So, if I send you a patch that allows a subclass of the OGLRenderer to use its own thread, would you integrate it ?
(making the createOGLRendererThread virtual, and exposing the OGLThread code)

I would like to add that it was possible in the previous version of Juce, and it’s not more possible, so I’m struck into porting it to Juce 2.0.x


#4

[quote=“X-Ryl669”]Hi,

I’m a bit hijacking this thread, but I have a similar issue. I can’t use the thread Juce provides because I need to control its creations and its members, and most importantly its behaviour.
I can workaround the creation & members issue (except that I can’t assert if it’s the same thread each time that’s calling renderOGL, so it might break if you happen you change it).
However I don’t want the thread to call me at regular interval, I want to be able to make it wait for a specific amount of time (I don’t want 60fps in my application).

So, if I send you a patch that allows a subclass of the OGLRenderer to use its own thread, would you integrate it ?
(making the createOGLRendererThread virtual, and exposing the OGLThread code)

I would like to add that it was possible in the previous version of Juce, and it’s not more possible, so I’m struck into porting it to Juce 2.0.x[/quote]

I’d be reluctant to take that kind of suggestion, because on some platforms (i.e. android), there’s no way to run your own thread, you must use the OS’s thread, so I designed my architecture around coping with that situation.

Also, I’m really struggling to see why it would make any difference… You say you want to control its creation and members, but in the juce GL model you create your own object that inherits from OpenGLRenderer, and are in total control of its lifetime and members. Why would it be any different if you inherit from Thread instead?


#5

The thread I’m talking about in my architecture is a PresentationThread which is in charge of uploading video sample to the GPU, and displaying them.
It has to handle synchronization very very precisely (no more than 1ms jitter).
It’s using OGL when it’s the only option (linux, ahem), but DX on Win32 (and the framebuffer in some specific cases).
When it starts, it does some work which is not related to OGL (like measuring the clock, allocating video sample buffers, preparing subtitles…). Putting this initialization code inside the newOGLContextCreated() is more a hack than what I was expecting, since I need this code once, not each time the context is re-created. Yet, it works…

The behaviour itself is more of an issue, because the Presentation thread, upon displaying a sample, computes the time to wait before checking the video sample queue again.
I can’t deal with a the “unknown” time that the background thread will call me, as it would break all my nice synchronization code.
I’ve thought about sleeping in the renderOGL() method, but that does not work because you additionally sleep inside the background thread itself, and you’re doing some time-consuming code too that’s not constant.

So, I’m struck.
How do I solve this ?


#6

I’ve actually been doing a very similar thing in a side-project, playing video from ffmpeg via GL, and it’s working very nicely on the existing architecture - we’ve got a window containing multiple hi-def streams playing smoothly in overlapping lightweight juce components (all sitting inside a top-level component using the juce GL component rendering stuff). We even use a plain old juce DropShadower to give each video window a shadow which overlaps the others. Didn’t require any hackery or even any direct calls to openGL functions, the data all just gets sent via the OpenGL image format, and rendered using a normal Graphics::drawImage call!

Saying you need 1ms accuracy sounds like you might be doing it the wrong way - we handled the video by letting the GL thread run as per usual, so that it’ll call renderOpenGL() once per vertical blank. Each time renderOpenGL is called, we simply render the correct frame, based on the current time of the audio clock. Since the video you’re drawing can only be seen once per vertical blank, which is proabably every 16 ms, I can’t really see why you’d care about when you do the rendering within that time-window?


#7

Because I’m playing > 60Hz (my source sensor is a CMOSIS CMV4000, 270fps), and because I’ve a time-ratio (for slow motion & high motion).
BTW, I’ll need to have a look to the OpenGL image format, since I’m currently doing YUV->RGB conversion with shaders on the GFX.

I don’t care about the vertical sync at all (the monitors are so limited anyway) as I’m doing triple buffering, so I don’t base the video refresh on whatever monitor is connected (what about 90Hz, 120Hz monitor ?)

Anyway, if I could at least control the sleep time of the background thread, that would help me a lot.


#8

You already can! If you call setSwapInterval(0), it’ll spin the render loop as fast as you want, and you can sleep yourself if you want to…

…BUT having just done something like this myself, it sounds like you’re doing it all wrong! Our video player does varispeed and handles any frame rate - we’ve even testing it varispeeding through high-frame rate output from a Viper camera. It doesn’t matter how fast your render loop is going - if you’re using setSwapInterval (1) (the default), it’ll render once per output frame, i.e. as fast as your actual graphics card can get the frames out… If you render any more often than that, it’s a waste of effort because those frames are never seen!


#9

Not by you. The algorithm does handle it however.
Basically what you mean is:

renderOGL()
{
    while (isPlaying)
    {
        Frame * frame = Stack::popFrame();
        currentClock = audioClock.getClock();
        if (frame.timestamp < currentClock && frame.timestamp + MaximumAllowedLateness > currentClock)
        {  
            // Frame is too late, but not by much, let's display it anyway 
            renderFrame();
            return;
        }

        if (frame.timestamp < currentClock) 
        {   
              // frame too late anyway, let's delete it
              delete frame; continue; 
        }
        
        if (frame.timestamp > currentClock)
        {
              // Let's burn CPU here, to wait for the right time
              wait(frame.timestamp - currentClock); // Will call sleep if the delay is large, or busy loop else
              renderFrame(frame);
              return;
        }
}

The issue with such logic, is that you’re always lagging and you can’t cope with synchronization (since there is no feedback to the decode loop possible).
So as soon as you’re not able to deal with the number of frame your decoder gives you (for example, if the monitor/BG OGL thread is slower than the decoder).
If I knew when I would be called again (hence the next “ideal” timestamp to be displayed), I’m telling the decoder not to decode the stream anymore, until a frame with such a timestamp (or a bit higher) happens. Then I can synchronize, else I can’t do properly, I will simply fill the memory with useless samples anyway, or block the decoder thread while filling the stack (which is even worse, because I’m inducing jitter when the decoder will have to resume).


#10

No, that’s not even close to what we’re doing!

It’s more of a one-liner like this:

[code]void renderOpenGL()
{
currentFrame = cache.fetchFrameAt (audioStream.getCurrentTime());

// that's it.. Nothing else happens in here because the frame gets rendered by the normal 
// component paint method which will be invoked immediately after this. The only reason the code
// above doesn't also happen in the paint method is because if done here, it avoids the MM lock

}[/code]


#11

Fallacy.
You’re hiding the logic inside your cache.fetchFrameAt().

What happens to the frames you’re not displaying because too late ?
if you display them, well, it’s wrong, and it’ll show a slowed down motion.
If you drop them, it’s a waste of effort to decode them, so you better tell the decoder not to decode the next “currentTime - lastFrameTimestamp” timeperiod of frames to cope with your delay.
And to be mathematically correct, you need to take into account the next time you’ll be called (“nextCallTime - lastFrameTimestamp”), else you’ll still be late when you’re called back.
BUT, in the current code, you don’t know when the next time you’ll be called (unless you’re doing your own wait).

Unless you actually don’t care about it (let the CPU burn 3x more resources for nothing), the idea of a circular buffer is a waste of resources, and I wonder how it worked in your case.


#12

Honestly, no fallacies involved!

Whether you believe it or not, I’ve got an app right here that’ll happily play multiple hi-def streams with varispeed (including audio scrubbing), playing backwards, syncing to random audio tracks, etc etc…, all coming from ffmpeg. When it runs out of bandwidth, the frame-rate drops but still stays in time, and the audio still works. The design’s super-clean, no need for low-level hackery or fancy GL tricks! And our playout graph will also handle edits, effects, etc., although we’ve not added any of them yet.

FYI the reason for this side-project is that a friend of mine from the video industry observed that everybody gets video playback wrong, and suggested that we try to build an engine that does it properly. I’m not going to reveal too many secrets about it because we’ll probably end up licensing it as a library for pro video.

But anyway, the TL;DR is that you definitely don’t need your own thread to play video correctly!


#13

Ok, so let’s shift to another dimension where you can duplicate your daytime at will, you’d finish the “geniouser” (© Lewis Carroll) Projucer, enter a revolutionary video player, find out the question to the answer of the life, the universe and everything, and conquer the world.
In local time, we would all wonder how you did it.

Anyway, since I’m attached to my local 4D origin, I’ll try to deal with the -no- swap interval system and come back later…
BTW, I think you’re using an idea that’s used in Live555 (local “scheduler” with smart tasks), but I never succeeded getting this thing working in all cases.


#14

[quote=“X-Ryl669”]Ok, so let’s shift to another dimension where you can duplicate your daytime at will, you’d finish the “geniouser” (© Lewis Carroll) Projucer, enter a revolutionary video player, find out the question to the answer of the life, the universe and everything, and conquer the world.
In local time, we would all wonder how you did it.

Anyway, since I’m attached to my local 4D origin, I’ll try to deal with the -no- swap interval system and come back later…
BTW, I think you’re using an idea that’s used in Live555 (local “scheduler” with smart tasks), but I never succeeded getting this thing working in all cases.[/quote]

Thanks, and good luck! TBH the video stuff is really interesting, I wish I had more time to work on that…


#15

[quote=“X-Ryl669”]Fallacy.
Unless you actually don’t care about it (let the CPU burn 3x more resources for nothing), the idea of a circular buffer is a waste of resources, and I wonder how it worked in your case.[/quote]

Just so you know, for all all your panache, it’s you that’s doing things differently than everyone else, not Jules. Rendering frames that no-one will ever see is silly. Every other video player system and framework tries it’s best to render every frame that can be output (and no more) and to pick the exact right video frame for that time.

I suspect your problem is that you’re trying to do decoding work in the render thread. Its more normal to use two or more threads, so playback and presentation are separate.

Bruce


#16

Bruce, I think you overlooked the conversation.
I’m using a rendering thread, and the issue I have with the current scheme is that I can’t feedback the presentation delay back to the rendering thread, to obviously avoid decoding useless frames (like I used to do with Juce 1.x).
And until I’ve decoded a given number of frame (at least the vertical sync of the monitor), you can’t detect it anyway, so you must react after your presentation thread starts yelling that’s it’s full.

Check the pseudo code in my initial post, you’ll see what’s I’m talking about.


#17

Not sure what you mean. I see some ‘like this?’ code, but I’m not sure what you mean. What’s your pseudo code? I’m curious what you’re doing.

Bruce


#18

I meant the pseudo code from this post
http://www.rawmaterialsoftware.com/viewtopic.php?f=2&t=10997#p62415

The TL;DR is that is you pop your decoded frame inside the rendering thread, and the rendering thread is lagging from the decoding thread, you need to tell the decoding thread at “which” point you’re going to synchronize, so the decoding thread can just parse the next frame (avoid decoding them) until the expected frame is there. Not knowing when you’ll be called next is an issue, since you can only underestimate this delay, so you ends up always being late of at least one v-sync period in the rendering thread. If you overestimate, you can have skips with are really really bad.

In my former code, I computed this time precisely, and I had no single wasted decoding frame. In the current code, I couldn’t.


#19

Well, I don’t decode to order - I have the right range of frames. But I upload the next frame I expect to use, just by looking at what frame time I expect to render next (now + 1).

I then ‘order’ that frame, and another thread tries to find it, and put it into a pre-mapped buffer object and reports back (or gets a closer frame than the last I got and sends that).

I think the only case your code is useful is when there’s a large disparity between display and movie frame rates, and the movie is intra-frame only?

But I’m not certain why you can’t do the same calculations in the render thread Juce pops you into? Get current time, extrapolate frame #, guess next frame # and order decoding?

Bruce

[quote=“X-Ryl669”]I meant the pseudo code from this post
http://www.rawmaterialsoftware.com/viewtopic.php?f=2&t=10997#p62415

The TL;DR is that is you pop your decoded frame inside the rendering thread, and the rendering thread is lagging from the decoding thread, you need to tell the decoding thread at “which” point you’re going to synchronize, so the decoding thread can just parse the next frame (avoid decoding them) until the expected frame is there. Not knowing when you’ll be called next is an issue, since you can only underestimate this delay, so you ends up always being late of at least one v-sync period in the rendering thread. If you overestimate, you can have skips with are really really bad.

In my former code, I computed this time precisely, and I had no single wasted decoding frame. In the current code, I couldn’t.[/quote]


#20

Can someone point me in the right direction on how to take the AVFrame object after it is decoded and convert it to a Juce Image type so that I can render it in the paint method (I would imagine this is probably only a few lines of code)?  I know that it needs to be scaled, but once I have the scaled AVFrame (probably from YUV to RGB -> PIX_FMT_RGB24), I am stuck on how to display this. I was looking into the Image::BitmapData, but that doesn't seem to work directly with the raw image data from ffmpeg.

Your help is greatly appreciated. 

Thanks,

Darren