Syncable database solutions

General JUCE discussion
1

Hi all

I've been looking around for a while for syncable database solutions, i.e. store data locally and then replicate it to a server when there is a network connection available. The end purpose is so that a user can edit data on one device and then see it on another once they've both been synced. This brings up the issue of conflicts e.g. if a user edits the same data on different devices while both offline and then both go online. 

The best solution I've found so far is Couchbase Lite (http://developer.couchbase.com/mobile/get-started/couchbase-lite-overview/index.html), which is a multi-platform library that covers all of that ground. The drawback is that requires cross-language code as each platform has a different package (i.e. Android Java, iOS/OSX obj-C, Windows C#...) - there is no C++ library. 

Before I start write all the bridge code, I wanted to see if there are any C/C++ options that would get me there. A possible contender I've found is LevelDB (http://leveldb.org) which is pretty bare bones, and although there are a lot of addons in Node.js including https://github.com/hij1nx/level-replicator, I couldn't find anything in C++. 

Anyone know of any other solutions? 

 

2

Hi, adamski, this is exactly what I want for my current project. I've been struggling on it for many days. Unquestionable, we're in a net-time, people're using various digital-devices every day. There's no reason not concern our apps with data synchronization.

From what I know, JUCE lacks few necessary functions, such as data base and net data transfer etc. so I have to to use ValueTrees as data models, they read from and writes to local files, then using cURL upload to, download from and manage them on remote server. The troublesome things I'm facing also are data synchronization and conflicts. What I did were:

1. Which is master, and which is subordinate.

2. Using time-stamp for every local file and remote file.

3. Using a temp-file to record and compare time-stamp and each operation of local, such as add, delete, modify the content or some attachment files (path). Whenver any edit operation occurred, it'll auto sync after a while. If sync successful, the data will be written to local files permanently and remove the matching records form the temp-file. If not, the actual data will move to and remain in another file which named "needSync".

This is inconvenient, low efficiency, unsafety and instability. I need a better way to handle it eagerly. I'll pay close attention to this thread, if you find something new, could you please post them here?

Thanks!

3

loopfine, your solution sounds a little clunky if you dont mind me saying so. I was initially looking at TouchDB and CouchDB, as they have replication and conflict resolution protocols built in. TouchDB has now become Couchbase Lite, and Couchbase have a suite of tools - Couchbase Lite that stores local data, Sync Gateway that provides an HTTP interface to Couchbase Server (which IMO is a little overkill for my needs) but the whole thing is designed for storing and accessing user's data with different access rights e.g. a user can have public or private data in different 'buckets', so that you could have data that users share with all other users of the app, or choose to share with certain other users. 

It all seems very well thought out with decent documentation, and at this point I think the work of bridging each platform language to C++ is something I can live with considering that all the hard work of replication, synchronisation and conflict management has been done. 

I will report back once I've got it all working with JUCE (at least with one platform...), unless I find an alternative C/C++ solution. 

4

Thanks for your reply, adamski.

No, I don't mind anybody says that and I'm willingly accept any criticism and irony if it could help me a lot. I'm not a software engineer, not a programmer, not even an amateur, I write codes just for fun, learning English base on some interesting thing and spend some boring time. I didn't know any computer language, can't say the simplest English sentence before I heard of 'JUCE'. Even worse, I rarely passed any maths examine since I was in kindergarten. So, anyone who can programs has the very right to laugh at me with some prideful superiority :)

I've read the introduction from Couchbase Lite's web site. It looks fine despite some of them I can't understand. Now I have few stupid questions, could you please waste some time enlighten me?

1. Will the non-Couchbase-documents be stored in the database? For example: audio and video clips, image files, other type files etc., I called them attachments in my app. Maybe hundreds of thousands of and even more. Or the database only store the index, all the actual data be saved seperately?

2. Using this database, do I need to do some program and extra settings on the remote server? Is the server  provided by Couchbase or my own, or any FTP server will do?

3. I'm curious of the data sync mechanisms, especially the real-time sync. For exmaple: QQ, wechat, apple memo etc., you type some letters on one device, nearly the same time, the very letters appearing on another device, so magic! If I don't use other libraries (except cURL), can I accomplish this by JUCE?

Sorry for waste your time read these stupid words. I'm so frustrated for my horrible English...

5

1. Yes it supports attachments: http://developer.couchbase.com/mobile/develop/guides/couchbase-lite/native-api/attachment/index.html

2. You either need to run Sync Gateway (which provides the HTTP interface) and Couchbase Server, which you would install and run on your server, or you can just point Couchbase Lite to a CouchDB instance, which has its own HTTP interface. However with CouchDB doing per-user data is much more difficult. Couchbase do not provide any hosted database service. But if you don't need per user data or other features of Couchbase Server you could use one of the many hosted CouchDB services e.g. Iris Couch, or Cloudant. 

3. I have no idea, that sounds like websockets - I'm sure its possible with JUCE but not something I have experience with outside Node.js

No time wasted! 

6

Thanks very much! 

I'll do more research and study on these fields.

Wish all the best for your work on this database.

7

I was curious whether you took a look at Dropbox's API for enabling Dropbox syncing within an app and if so, what you thought of it?  I could understand why in some cases on would want to handle the data oneself, but there are other cases where you just want to give users an easy way to keep their devices in sync and my impression was that Dropbox's API provided that functionality.

8

Have you looked at this:

https://www.ohmstudio.com/flip ?

Admittedly I've never dealt with it, it's just something I noted down some time ago because it looked interesting - your post reminded me of it.

9

Thanks for the suggestions guys.

I have had a look at Dropbox API in the past and decided I needed more control.  Might take a closer look again. 

Ohmforce Flip looks very interesting too, maybe a bit overkill for my needs but great to know its there - I love what those guys do, been using their plugins since they first released Ohmboyz :)

I realised I can use Couchbase Lite in almost the same way as a local CouchDB instance via HTTP calls, i.e. do everything via HTTP requests meaning no need for different bridge code for each platform, just start up the instance and make calls to localhost on the necessary port via the REST API. yes

1 Like