Experiments with serialization


#1

I’ve got a project on the go at the moment where the data is stored in native types. On save and load the data is transferred to or from a ValueTree, which is stored on disk as XML. It works well, but each class has a serialize and deserialize method, the contents of which need to match. There’s too much duplication and scope for error.

Anyway - someone showed me the boost serialize class. And I needed an excuse to play with more complex template stuff as well … not my usual area.

So here’s what some demo classes look like:

class Award
{
public:
	void saveload(ValueTreeArchiver::Archiver & ar)
	{
		ar.copy(name, "name");
	}

	String name;
};

class Person
{
public:
	void saveload(ValueTreeArchiver::Archiver & ar)
	{
		ar.copy(name, "name");
		ar.copy(age, "age");
		ar.copy(height, "height");
		ar.copy(awards, "awards");
	}

	String name; 
	int age{ 0 };
	double height{ 0.0 };
	std::vector<Award> awards;
};

And here’s a basic test which passes:

Person p;
	p.name = "Fred";
	p.age = 69;
	p.height = 6.0; 
	p.awards.push_back({ "Greatest Man Alive" });
	p.awards.push_back({ "Ate the most Onions" });

	/* Saving ... */
	ValueTree savedData{ "person" };
	ValueTreeArchiver::Archiver archiver; 

	archiver.save(p, savedData); // < this is all there is to saving

	DBG(savedData.toXmlString());

	/* Loading ... */

	Person q;
	archiver.restore(q, savedData);

	/* Now q is identical to p.  Let's test that:- */
	jassert(q.awards[0].name == "Greatest Man Alive"); 
	jassert(q.age == 69);

I think that’s pretty successful and a lot simpler than having a separate save and load method. Also the handling of the vector is 100 times easier.

And here’s the horrible template shit:

namespace ValueTreeArchiver
{
	enum MethodType
	{
		direct,
		container
	};

	template<typename T> struct DirectlySupported { static const MethodType value = container; };
	template<> struct DirectlySupported<float> { static const MethodType value = direct; };
	template<> struct DirectlySupported<String> { static const MethodType value = direct; };
	template<> struct DirectlySupported<double> { static const MethodType value = direct; };
	template<> struct DirectlySupported<int> { static const MethodType value = direct; };
	template<> struct DirectlySupported<bool> { static const MethodType value = direct; };

	/** Default method uses valuetree directly */
	template <MethodType t>
	struct Method
	{
		template <typename T>
		static void save(T & value, const Identifier & identifier, ValueTree & tree)
		{
			tree.setProperty(identifier, value, nullptr);
		}

		template <typename T>
		static void load(T & value, const Identifier & identifier, ValueTree & tree)
		{
			value = tree[identifier];
		}
	};

	/** If we aren't a simple class assume we are something handled by a container ..  */
	template <>
	struct Method<container>
	{
		template <typename T>
		static void save(T& value, const Identifier& identifier, ValueTree& tree);

		template <typename T>
		static void load(T& value, const Identifier& identifier, ValueTree& tree);
	};

	class Archiver
	{
	public:
		template <typename T>
		void copy(T & value, const Identifier & identifier)
		{
			if (isSaving)
				Method<DirectlySupported<T>::value>::save(value, identifier, t);
			else
				Method<DirectlySupported<T>::value>::load(value, identifier, t);
		}

		template <class C>
		void restore(C & baseObject, const ValueTree & savedData)
		{
			t = savedData;
			isSaving = false;
			baseObject.saveload(*this);
		}

		template <class C>
		void save(C & baseObject, const ValueTree & treeToWriteTo)
		{
			t = treeToWriteTo;
			isSaving = true;
			baseObject.saveload(*this);
		}

		bool isSaving{ false };
		ValueTree t;
	};

	template <typename T>
	void Method<container>::save(T& value, const Identifier& identifier, ValueTree& tree)
	{
		ValueTree container{ identifier };
		tree.addChild(container, -1, nullptr);

		for (auto & v : value)
		{
			Archiver archiver;

			ValueTree n{ "node" };
			archiver.save(v, n);
			container.addChild(n, -1, nullptr);
		}
	}

	template <typename T>
	void Method<container>::load(T& value, const Identifier& identifier, ValueTree& tree)
	{
		auto parent = tree.getChildWithName(identifier);

		auto numChildren = parent.getNumChildren();

		for (int i = 0; i < numChildren; ++i)
		{
			auto n = parent.getChild(i);

			Archiver archiver;

			typename T::value_type newObject;
			archiver.restore(newObject, n);

			value.push_back(std::move(newObject));
		}
	}

}

Any thoughts and feedback much appreciated. I think I’d like to special case things like OwnedArray, and I’m not really happy with how it defaults to assuming a C++11 container. It works here, but there’s probably some fancy way of detecting containers using type traits I need to look up.


#2

Merging save & load may seem nice in very simple examples, but it gets worse quickly in my experience - for instance, loading serialized data from a different version, maybe even incompatible state possibly requires logic and/or parsing (but only for the deserialization part). Serializing to a different version number is also a very nice feature, so it probably needs some versioning system as well (with possible separate versions for each node).

I designed a serialization system as well, and one of the most useful features (I found) was that each serialized structures could be nested with any amount of associative key/serialized structure value pairs, so you basically have an associative tree container of state and nested trees, that serializes itself seamlessly. Thus, there is one unified interface without using intermediate separate structures (valuetrees, valuetreearchivers and xml strings).

It decouples the serialized structure and the serialization methods to be less limiting. If you ever change some class’ simple property to an entire nested page (or whatever), that class’ serialization would still work as expected from a top-level view without changing anything (increases encapsulation), because the page structure can be embedded directly in the class serialization.

Not sure about how your system works with multiple things that need to be serialized collectively (for instance, a class that has a person and an award).


#3

Agreed - I’m probably preparing to shoot myself in the foot later. It will never stay this clean.

I’m struggling to follow this.

RIght now it only copes if the person and award were in some kind of container. But what I want to do is have it detect (via some template magic) when it’s a container, and if that fails then try calling the loadsave method on the class, which will result in a tree structure (which is probably what you are describing?).

At the moment the generated XML looks like this:

<person name="Fred" age="69" height="6">
  <awards>
    <node name="Greatest Man Alive"/>
    <node name="Ate the most Onions"/>
  </awards>
</person>

If I made the improvements then a class, let’s call it ‘parent’ containing a person and an award would look like this

<parent>
 <person name="" age=""/>
 <award name="Ate the most Onions"/>
</parent>

#4

But, ValueTree has already a serialization feature: writeToStream(…)

Just provide a model to access your data in aValueTree, instead of save the data in the class directly.


#5

[quote=“bazrush, post:3, topic:20230”]
Agreed - I’m probably preparing to shoot myself in the foot later. It will never stay this clean.
[/quote]Well it’s better than starting out dirty :slight_smile:

Perhaps looking at this design makes my concept clearer… it also shows how to handle generic templated containers based on iterators:
http://pastebin.com/eSQGP9hu


#6

That’s what I’ve done in a lot of cases. But sometimes either for legacy of performance reasons it ends up not being like that. And besides it was half a lazy Sunday delve into templates :wink:


#7

If you really want to go down a rabbit hole, look at boost::fusion for serialization. It will happily serialise POD structs to xml and vice versa. I’ve been using it for reading old binary formats like wav. See this (warning very ugly code) for an example of this.


#8

Of course I want to go down a rabbit hole. What a silly question! :slight_smile: It’s going to have to wait for the weekend though, otherwise no actual work-that-might-pay-bills will get done…