Implementing a virtual file system

General JUCE discussion
1

It’s been my geeky dream to use a virtual file system on my next project, for which it’s very likely I will be using Juce. The idea is that other file systems can be ‘embedded’ in the main file system, so that they can be ideally accessed via a standard File() object. Examples for file system types would include zip files, pasteboard contents, music libraries, cloud storage, ftp access etc. Some of these would be quite specific to my own needs, others would be more generally useful (eg. to access assets from the Android APK in the same way as other files, or (maybe in future) to access user documents on Windows 8 Store apps which use a non-standard API).

Here’s one example of an existing library : http://freecode.com/projects/physicsfs. The idea could be taken pretty far (eg. file systems containing other file systems) but a fairly simple API such as this would deliver what I need:

// -------------------- Initialisation -----------------------------

// Mount Android resources to the path '/Assets' in the main filesystem
fileSystem.mount("/Assets", new FileSystemAndroidAPK());

// Mount a zip file to the path '/Archives' in the main filesystem
fileSystem.mount("/Archives", new FileSystemZip("SomeZipFile.zip"));

// Mount Dropbox storage folder, and add some sort of listener to respond to errors, supply sign-in credentials etc. (this idea obviously needs some fleshing out)
FileSystemDropbox::Listener dropBoxListener;
fileSystem.mount("Cloud Storage/Dropbox", new FileSystemDropbox(dropBoxListener));


// -------------------- Usage -----------------------------

// Read a wav file from the Android APK
InputStream* pWavFileFromAndroidAssets = File("Assets/MySoundFile.wav").createInputStream();

// Obtain a file from a zip archive - the zip file header is loaded lazily on demand, and is unloaded if it hasn't been used for n seconds
InputStream* pFileFromZipArchive = File("Archives/SomeZipFile.zip/SomeFile.dat").createInputStream();

// Get a file from Dropbox - we may need a listener which needs to take some sort of action if sign-in is needed, etc, or at least return a meaningful error code if not signed in so the application can take appropriate action
InputStream* pFileFromDropbox = File("Cloud Storage/Dopbox/SomeFileFromTheCloud.txt").createInputStream();

// Iterate all files in the mounted zip archive
DirectoryIterator* pZipFileIterator =  File("Archives/SomeZipFile.zip").createDirectoryIterator(DirectoryIterator::Flag_Recursive, "*.*");

I can think of at least a couple of possible implementations:

  1. Heavily modify the File and DirectoryIterator code. Advantages : all code can use the virtual file system via the File() class. Disadvantages : Merging in future Juce changes to my local copy.
  2. Make my own File style class (eg. VirtualFile) which sits outside the core Juce library. Advantages : No modification to Juce. Disadvantages : Only application code can use the virtual file system, but any library code which accepts Input/OutputStreams can still work.

Anyone fancy chipping in with their thoughts?

2

I find it easier to just use SQLite for application storage. The benefit is that you get atomic writes and journaled rollback, and more structured access to the data (keys for example).

3

I suppose I’m trying to fit two requirements really:

  1. A unified way to present data from different sources to the user. For example, in one scenario I’d like to present two side-by-side directory lists that the user can drag files between. A user should be able to navigate to (eg.) the iOS pasteboard in one list, and their (eg.) Dropbox account in another list and then drag their stuff over.

  2. A nice coherent way to access streams from virtually anywhere in code using a path/URL.

The first requirement suggests that this could be implemented high up as a UI abstraction (which was my original plan) but somewhere the second requirement still has to be met, so it starts to make sense to build it in at a lower level.

SQLite looks interesting but I can’t really see how it fits.

4

SQLite fits in very well in a way that lots of developers actually use databases for file storage to generate cross-platform unification of storage,
and the additional benefits of transactions, optionaly transparant encryption, no need for a container format etc.

Virtual file systems may look quite simple, but they can really become complicated, at the latest when different semantics come into play
(think OSX and the differing behaviour of drag and drop operations on files and folders in contrast to Windows and Linux)
Those buggers can really eat up your brain, and your valuable time, too.

5

SQLite may help with project data/file formats, but I already have all that nailed and it’s not the problem. For example, on iOS I want to unify the way that both I (as a developer) and users can access files from sources such as Dropbox, FTP, audio pasteboard, iTunes file sharing etc. I can see how SQLite may help with storing read-only resources, but I’m happier using conventional files for this for many reasons.

On Android, it would be nice to just access a read only resource from the APK using standard path notation.

After having recently ported an app from iOS/Windows/OSX/Playbook/Android to Windows 8 Store and finding that it supports fopen/fclose/fread for some files, but needs a different asynchronous file API for accessing files outside the sandbox, a virtual file system would have been a Godsend.

1 Like