Hi all, I was wondering if anyone has any tips for rendering waveforms in a DAW environment. I know there are a lot of DAW (or similar) developers out there and was hoping someone would be able to share some tips for drawing a large number of waveforms quickly, at different zoom levels and with constantly changing bounds i.e. scrolling with the transport.
At the moment I am using the AudioThumbnail class and caching these at a resolution of 64 samples per thumb sample. Then, my waveform is drawn as part of a component in a viewport timeline. To speed up the drawing I am clipping the bounds of the drawing time to Component::getVisibleArea() so obviously only the visible section of the waveform is drawn.
This works fairly well when the user is just scrolling around or zooming in and out but slows down when there are lots of visible thumbnails on the screen or when the window is constantly moving. Obviously this has to do with the fact that the all the thumbnail lines are drawn each time the window moves by as little as a pixel.
Now, what I’ve done in the past is to cache whole downscaled waveforms to images and then rescale sections of them when necessary but that doesn’t really fit well with this sort of application. I also need to be able to re-draw the waveforms at any zoom level and at the moment this crawls to a standstill when having to re-read from the actual file.
Here some ideas I have had:
Keep all the drawing off the main thread. I imagine this would work by when the zoom changes, a TimeSliceThread is used to draw the waveform into an image, notifying the main thread once it has drawn the required period so it can repaint itself.
This will certainly make sure that the UI remains responsive but could lead to periods of blank waveforms while the rendering thread catches up with the viewed position. This wouldn’t work well for example if only the end section of a waveform is being viewed at a very high, constantly changing zoom level as the rendering thread would have to keep re-rendering the image from the beginning.
This could also lead to very high memory consumption as extremely large images would be required at high zoom levels e.g. 3 pixels per sample would lead to 396900 wide image for a standard 3 min, 44100KHz file.
This could be improved by a selective render pattern, i.e. render the viewed part of the waveform first and then fill in the gaps around as the user is likely to scroll close by rather than jump around. However, this starts to get very complicated and in my experience you get inconsistencies when drawing using the AudioThumbnail class in sections as the low-res boundaries changes.
A compromise would be to render only the visible section on a background thread and repaint once that has been drawn. That should reduce the image size and amount of time to render the section but I am still worried that it won’t be fast enough to keep up with the transport.
I think this details some of the problems and I know there is a solution out there as Reaper for example does a fantastic job of quickly rendering, very zoomed in waveforms whilst following the transport cursor.
Any tricks, tips or general mockery of my current thinking are welcome.
The following link may be of some help. It’s Mac focused but the Juce Graphics system pretty much resembles Quartz/CoreGraphics anyway so that shouldn’t be an issue.
Hi sonic, thanks for the reply, I’ll take a closer look at that article soon.
Glad you’re liking dRowAudio, the waveform classes in there are pretty damm fast as they’re tripple buffered to images at various stages which makes drawing them extremely quick. The problem is that they were designed for a relatively small window of sizes so don’t scale to extreme zooming but I never needed them for that (they were originally for a DJ application).
Of course anything I learn will hopefully find a place in the module. I think for this though I’m going to have to go the route of background caching sections of the thumbnail to Images (say three times the visible width) and keep a reserve image ready to render the next block, swapping them over when necessary. Fiddly but the best thing I can think of right now.
Hi @dave96
Sorry to resurrect an old thread, but I’m kind of having the same issues using the Juce AudioThumbnail.
What route did you take to improve it?
I’m also seeing some weaknesses in AudioThumbnail.
For example, it seems that it recomputes the cache way too much. If you look at AudioThumbnail::CachedWindow::refillCache, it will go ahead and invalidate the cache if this condition is not satisfied
That seems weird to me.
What if I paint 1000px, and then want to repaint 10px in the same visible area? it will flush the cache and recompute a 10px cache. And so on, ending in infinite recaching when moving things over a waveform for example. Am I missing something?
Interesting. For me, AudioThumbnail::CachedWindow::refillCache is a big bottleneck when zooming in and out. What’s puzzling is that it’s smooth or slow depending on the zoom level, so it feels like there are “zoom zones” in which refillCache takes a lot of resources.
Of course I’m talking about when having 30 tracks on screen with 1hr long clips.
When trying this in Waveform Free, it’s fine, so it seems that I’m doing something wrong.
I’ll have to get to the bottom of that one…
The AudioThumbnail has two levels of detail which you can control using the samples per blocks sourceSamplesPerThumbnailSample.
If you need to zoom out that far maybe a higher number works better?
IIRC it will switch to the original automatically as soon as one cached sample block spans over multiple pixels.
Maybe it could be improved with multiple levels of detail like progressive image compressions use.
Or simply a few more layers with numbers like 1024, 262144 etc.
AudioThumbnail has a virtual AudioThumbnailBase, but I am not sure if it is possible without altering the AudioThumbnailCache…
Sadly this is not really related to the performance issue I’m seeing. If you set this value to 1, it essentially will save every single sample of the file, which takes a lot of space and time, defeating the purpose of the thumbnail. The larger the value, the lower res you get. This does not impact the cache issue that much.
Well, sadly this is not entirely true. Yes AudioThumbnail will take care of painting only what is needed, but if you look at the if statement, it goes into refillCache with the entire area. So you certainly want to call drawChannel only with the visible area, otherwise you could perform an extremely expensive recache operation depending on your zoom level and file size. So @baramgb is right, but I’m already doing that, and the performance is not great still. Again I’m talking about zooming in and out a lot with 30+ waveforms of 1hr long files. So quite an extreme use case.
Agreed. I hope you are not doing that, because that is not what AudioThumbnail is for.
That is interesting, thanks. I didn’t look into that function. Maybe the order of taking the clipRect could be reversed, so the invisible part will not trigger an invalidate the cache?
No I’m not doing the 1 sample thing, although I won’t lie I tried just to see how it performed
I tried to move the clipping before the call to refillCache but it’s not that trivial because you also need to change the start and end time, and that point in the stack you have no clue how. Simply computing a ratio is not robust since there could be tempo automation etc on the client side
I don’t have much to add here other than saying that I’m interested in this discussion surrounding optimization or even visual improvments to make it look less chunky - with a sort-of daw product, I have many thumbnails flying around all over the place and they are by far and large the cause of my performance issues when scrolling around the sequencer.
I haven’t solved the cache perf issue yet, but I hacked into juce so it fills a path instead of drawing rectangles. That way it looks smoother.
Also, I implemented sample accurate drawing when zoomed in far enough. If I go past a threshold, instead of calling drawChannels I manually go to the audio file, read samples, and draw a path.
