I need to stream Images over network. It’s a stream of Images sent with constant rate (imagine you want to grab/capture image data from some video device like a webcam and send it over network to a client app). I have noticed the Image::writeImageToStream is very slow. By “very slow” I mean I cannot get any better than ca. 5 fps (my images are like 400x400). I have already limited that issue to the Image::writeImageToStream as if I get rid of if (and will be sending some constant block of memory every time, although I left generating/capturing images untouched so that I know it’s not the capturing part consuming my CPU), then 25fps and higher is not a problem.
I have tried both PNGImageFormat and JPEGImageFormat. My PC is Win10 with some 4x2.8Ghz Intel i7 not doing any other CPU consuming tasks when testing this. I’ve also tried replacing libpng with latest v1.6.37 from http://www.libpng.org/pub/png/libpng.html. The above ca. 5 fps was when in Debug mode. In Release (where some code optimizations occur etc.) I got slightly better results and can get ca. 10 fps, but it’s still far beyond what I would expect:) Especially that I’m talking about some relatively small images 400x400 (RGB) without any frame processing.
Any recommendations on how/if I could get higher fps? I’m happy to use different methods if you know any.
Let me answer your question, though I thought it’s not important as I have already narrowed my issue to the Image::writeImageToStream CPU consumption.
By the way, why do you think I’m doing all that stuff on the UI thread? This has nothing to do with the UI. Please imagine I have a non-UI console app only. The image is not rendered anywhere. By “FPS” I only meant how quickly I can repeat the Image::writeImageToStream.
There’s a separate thread where I’m doing basically 3 things in a loop:
capture image data into Image object (this consumes almost no CPU and doesn’t block at all)
write captured image data to stream
send the stream (currently I’m using a StreamingSocket and sending data)
I can “comment” the above steps no 1. and 3. and still cannot get 5 (in Debug) / 10 (in Release) loop executions per second. On the other hand when I leave 1. and 3. and “comment out” step 2. only (so that I’m sending same constant, but non-empty memoryblock everytime), I can easily have more than 30 loops executions per second (even in Debug mode).
To find the answer to your question, you would need to measure. The great thing about JUCE is you have the source code, so dig in and see where the time is going. I took a quick look and the PNG version, and it essentially copying each line of image data into a temporary buffer, and handing that to the PNG encoder. And somewhere in the PNG encoder code, it is streaming the data to the output stream… so, is it the encoding or the streaming?
Well, PNG encoding is slow almost by design, so I would not expect spectacular results. When comes to JPEG it really depends on a quality factor and how detailed are images. Try to change it below 90% and measure execution times then.
Thank You guys for all the advices so far. While I have tested JPEG already, it’s not the best option currently, as I am also trying to optimize network traffic and send frame “difference” only (as I have slowly changing images - below 10% of image area changes frame by frame and sometimes image is unchanged for a few minutes) - so only lossless image formats are of my interest at this stage. Results are really good so far - single frame (in PNG format) is usually below 7kb (as the “difference” frame is almost black if there is not much change so after compression in PNG its size is quite low) if there is only a small change in an image (compared to the previous frame).
By the way - calculating frame “difference” doesn’t consume much of CPU as I have already tried running it without actually calculating the difference.
With JPEG (even having 100% quality it’s still lossy) I would have to change my concept and send “complete” images/frame (instead of frame “differences”). I will try that out just by curiosity - network traffic will be much higher, but - as MBO is suggesting - I might save some CPU and be able to encode and send images more frequently. I will post results of my experiments here.
I’m thinking how they did it with apps like Skype? Some smart/fast custom image encoders? GPU processing? Do they simply send images in some lossy format?
There are many video compression codecs which use such method of compression: there is one so called base frame and after that a certain number of frames containing differences. Video streaming services mostly use one or other such codecs (H264 etc.)
Webcams and industrial cameras very often stream separate JPEG frames (MJPEG). Mostly they have a hardware encoder inside so encoding does not take too much time. Decoding is very fast and can be done even on a slow computer. In certain applications (security and medical) MJPEG is even preferred from a legal point of view as differential frames can be questioned when comes to a legal battle.
You should not have any problems with streaming such images via network - I have been writing such applications for years and streaming jpegs or other data blocks via sockets directly or using InterprocessConnection works ok.
They don’t encode images, but video. The encoding makes use of the previous frame, so it can be much faster. Plus the frames tend to be smaller, because they use P and I frames (Picture = has full data information) and I (Intermediate, is just a diff to the previous frame). This together with a lot of other trickery. And the file size matters, because moving bytes is often the bottleneck.
For encoding video, a library like ffmpeg makes much more sense. Unfortunately ffmpeg is no fun library, because many people write on it, which means, there are often different approaches, hard to wrap ones head around.
I created a video engine, which encapsulates ffmpeg in juce, but it’s in an early state. But you will find many building blocks there, e.g. how to convert JUCE Images into AVFrames for consumption in encoders and back:
Thank You guys, appreciate all you advices. At this stage I’m having acceptable results with the JPEGImageFormat writer and qith quality of 0.2 I can get 12 “FPS” with total CPU consumption of 10% (in Release mode) on my dev PC. Higher rates are achivable as well so I’m happy to close the topic.
This has nothing to do with the UI. Please imagine I have a non-UI console app only. The image is not rendered anywhere. By “FPS” I only meant how quickly I can repeat the Image::writeImageToStream.