Live Xml Update, makes system hang after 30 mins


#1

Hi Jules,

In my application I’m getting huge data from multiple wireless devices on my serial port and that I’m logging in and Xml file.
This wireless data (packets) would keep coming for hours together.

I was using xmlElement’s writeToFile method after every 1000 packets, and these packets come at 25 Hz, so every 40 second
I log the data to the file. In about 30 mins the file becomes so heavy that it hangs the application. The problem is if I
deleteAllChildElements from the xmlElement after every 40 seconds then I’m unable to append new data to this file.

I also tried using XmlDocument’s getDocumentElement so that I could try and append the data to the end of the opened
file like as follows

	i64temp = i64totalDumpedPacketCount;

	//Open Xml head of the log
	juce::XmlElement* pLogElement = tempLog.getDocumentElement(true);

	//Getting the first child element
	juce::XmlElement* child      = _pXMLElement->getFirstChildElement();
	juce::XmlElement* nextChild;

	//Adding the child elements in the loop
	do
	{
		nextChild = child->getNextElement();
		//Removing the child from xml element
		_pXMLElement->removeChildElement (child,false);

		//Adding the child to the log at the end
		pLogElement->insertChildElement (child,i64temp++);

		//Equating the next child to the child for next iteration
		child = nextChild;

	}while (child != 0);

	pLogElement->writeToFile( _sXmlFile ,"" );         
	pLogElement->deleteAllChildElements();

This also is not appending data.

Jules is there any method by which I can append the data to the end of xml file without parsing the entire file.
Or can you suggest me some thing else which gives me the right direction.

Thanks in advance.


#2

[quote]Jules is there any method by which I can append the data to the end of xml file without parsing the entire file.
Or can you suggest me some thing else which gives me the right direction.[/quote]

It’s just not the sort of thing that XML is designed for - it’d be vastly more efficient to simply append to a file than to insert new data into it, even if you didn’t need to parse the whole file to find the point at which you should insert it… And if the XML is so big that you can’t handle it to write the file, then how are you going to actually use the data afterwards? If you’ve a choice, I’d suggest using a text file instead.

Or if it really has to be XML, why not log to multiple files? As soon as one hits a certain size, open a new one.

The only reasonably efficient way to do it in one file would be to write a hack where you read the file as text, search backwards for the end of the last element, and then hackily insert new xml data into the file. Very nasty. But any kind of proper XML parser is going to have to read the whole file, which means the insertions will become slower as the size grows, there’s no way around that.


#3

I guess you are right I have to move to text file eventually… in one of your posts

http://www.rawmaterialsoftware.com/viewtopic.php?f=2&t=5740&p=31280#p31280

you said…

If I have to just append data to end of the file not using the hack way… is it possible to do it as you have mentioned…
Do I have to use writeToStream and then writeToFile…? I’ll try any ways…

Thank you


#4

[quote]If I have to just append data to end of the file not using the hack way… is it possible to do it as you have mentioned…
Do I have to use writeToStream and then writeToFile…? I’ll try any ways…[/quote]

Of course it’s possible, but if you re-parse or re-write the whole file each time, then it’ll become exponentially slower as the file grows! That’s fine for small files, but if you expect this to scale, then no chance!


#5

Yes I tried that and I’m writing the text file already.
Thank you.