I’ve always wanted a sound editor to analyze a sound and plot its features (pitch,loudness, spectral centroid, zcr…) like subtracks of the sound in a way similar too how cubase unfold plugin parameters. http://www.harmony-central.com/Newp/2003/Cubase-SL-20-lg.jpg
Then these feature track could be used to control vst plugin automation or to specify loop points for samplers…
I’m willing to contribute to this. I haven’t much experience in interface design tho. but I can write the DSP part.
[quote=“mdsp”]I had another idea for something that is really missing IMHO in audioeditors:
“matching pursuit decomposition on time frequency dictionaries”
to have an idea of what I mean: guimauve[/quote]
:shock: Woah, I just had a play with guimwave - I don’t think I understand the maths, but I love the way you can decompose a sound into it’s component sine waves and then select which ones to play back. It would be so cool if you could then move the blobs about and resize them etc. If this project gets going again, I’ll definitely help out if it’s going to have this kind of thing…
(as it is, I may have to modify guimwave to read/write to .wav files - I really like this program)
yes, I have lots of ideas related to DSP.
IMO the best way to do it, would be plugin mechanism inside the Editor (though I’ve never designed a plugin based app before).
the plugins I’m thinking about are mostly related to analysis (FFT,LPC,MFCC,zero crossing,pitch,enveloppe,…) and return either scalar of vector data for each audioframe.
If needed the plugin could provide a custom view of the data, otherwise the host would have a generical representation (say curves for scalar and spectrogram-like for vector data).
Based on this layer it’s also possible to imagine plugins using these results.
However for now it’s really too amitious IMO.
But I see some sample applications that would be interresting:
Say we have the pitch of the audio. a second plugin could use it to find ideal loop points within the data and then store it inside the wav or aiff chunk fro later use by the sampler of you choice.
If scripting is available (like in virtualDub ) it could be very easy to batch process lots of file.
With pitch, energy, and zero-crossing rate, it would be possible to segment voice in phonemes, in a way quite similar to what does recycle for drums.
while I’m speaking about drums, once hits are segemented, they could be automatically classified in categories (say BD, SD, HH,…) and then mapped in a soundfont.
I know this is far and complex, just dreamin’ out loud
and it implies that there will be dependencies between plugins .
It sounds to me like like the plugin idea could fit into valley’s idea of ‘proprietary’ plugins for the editor. To be honest, I think the (wave display) plugin api needs to be specified more thoroughly before I’d be willing to start work on the program (I don’t want to start by hard-coding something that should really be a plugin). There’s bound to be certain things you’ll only realise it needs as you start actually coding, but I don’t think it would be too hard to come up with a basic framework for what the plugins should look like (code-wise).
That said, I’ve started to have some ideas of my own about this project, and am quite excited about what it could become with a bit of work. What I’d like to see is a fully-customisable (/skin-able) interface, where you could make it emulate your favourite editor (personally that’s an ancient version of SoundForge XP I’ve been using for years now).
Anway, I’d like to help out, but I think a little organisation is needed first (imo). Also, what about making it a sourceforge project?
unless Rob picked up the ball and ran a long way with it, AFAIK, it is still very much in teh design and planning stage right now.
I’m expecting to be finished with a major project that has been borrowing my free time in a few weeks. I’d like to come back to this all if there are people interested.
well i’m not sure how well this would work in an audio editor… maybe it should just an app all by itself? plus we could go ahead and get started on it without having to wait on the audio editor.
ok… so if nobody else is going to i guess i’ll start coding…
the only thing is i don’t know much about fft, how do you want me to present the data to you and what kind of data will i get back out? should i display the data as an image with time on the x axis and frequency on the y?
I’ve joined the FEAPI “Feature Extractrion Plugin API” team http://feapi.sourceforge.net/
which is closely related to the initial idea I posted here.
It’s a very simple API for feeding a plugin with audio data and getting timestamped results out of it.
if you want to what to know what’s going on, the better is to check the mailing list and do a
svn co https://feapi.svn.sourceforge.net/svnroot/feapi/trunk/FEAPI feapi
I’ve made the Mac port, as well as a MaxMSP host and a plugin proxy helper class. We should make a release quite soon for both Windows and OSX with 2 hosts examples (command line an maxmsp) and 5 plugins (max, rms, Spectral centroid/flatness/flux/rollof, loudness, zerocrossings).
