Amazon S3

General JUCE discussion
1

Last week I had to support basic object operations with Amazon’s ‘Simple Storage Server’ (S3). I used the REST interface, and it was still a surprising amount of work. Not huge, but a lot of bits and pieces. A MIME compatible Base64 encoder/decoder class, SHA1 and HMAC_SHA1 classes, etc. As far as I could tell, Amazon doesn’t have an official C/C++ SDK, though they offer an SDK for iOS and Android. And the user submissions all seemed to be wrappers for curl.

I’m not sure how popular Amazon Web Services are, but if any of the above is of general interest, I’ll flesh out the classes to be more generic and post them.

2

That’s a cool feature jfitzpat! (that smells like a JUCE Module…)

I’m curious to know a few things:

  • Did you use the standard URL class or the Libcurl framework?
  • For the Base64, did you use the (none-standand) Juce version, or wrote your own?

My relationship with S3 has always been using the web console… so it’s nice to know that there are non-lazy people around here…

3

I actually modified URL, which is a bit part of the reason I would need to clean up a bit (stand alone class instead of hack?). I needed headers and wanted PUT (using POST to upload files is weird and problematic). You could easily use libcurl instead. The hard part is generating the URL and proper temporary credentials. After that, the actual HTTP stuff is straightforward.

For that purpose, the non standard Base64 wouldn’t do. I did a Base64 that does ‘mime’ style encoding (pad to % 4 == 0, use = for the pads at the end). Signature is build up a credential string, HMAC_SHA-1 hash it using the credential secret, Convert the result to Base64, URL encode it for use as a parameter (which it is).

Personally, I’d rather have not dealt with S3 at all, but manually using the console was just not supportable/sustainable, so I had to do a publishing tool.

4

That would be awesome if you made your work available somehow. That could be useful for a lot of people I’m sure…

5

OK, I’ll do a quick clean up of the top, say invoke curl or something, then figure out how to cleanly rework the bottom so it is a reusable JUCE module.

6

OK, here is a super simple stripped down S3 that invokes curl. The SHA-1 class is weird and terrible, I snagged it from an old project. Frankly, I think someone snagged it for that too (different style than the rest of that project), but there were no credits to pass on. The Base64, HMAC_SHA1, and AmazonS3 are all new, use as you wish.

Usage would be something like:

    AmazonS3 s3 ("MYCREDENTIALS", "myprivatekey");
    S3Object obj ("TheS3Bucket", "TheObjectID");
    
    // Fetch "TheObjectID" from TheS3Bucket and store it in ~/test
    // If it fails, print the result part of the returned header
    if (! s3.getObject (obj, File (CharPointer_UTF8 ("~/test"))))
        Logger::outputDebugString (obj.getResult());

    // Put ~/test in TheS3Bucket as TheObjectID
    // If it fails, again print the result from the HTTP header
    if (! s3.putObject (obj, File (CharPointer_UTF8 ("~/test"))))
        Logger::outputDebugString (obj.getResult());

I’ve only tested this curl version on Mac, though it is coded to use .\curl.exe on Windows. It also uses https, so if your Curl isn’t setup for secure credentials, you’ll want to change AmazonS3::createURL to use http instead. When I get a chance, I’ll clean up the URL extensions and post a non curl version. Though, like I said above, creating the stupid credentialed URLs is the hard part.