Creating Large XML documents

Hello friends. I'm wondering if anyone has solved a problem that I'm looking at. I'd like to try to exchange an XML file between clients that could get pretty large at times. I anticipate fifty thousand or so elements on a standard day and as many as ten times that amount during peak periods. I worry about memory consumption trying to create such a beast before writing out to the OS. If anyone can provide any insight, I'd be very appreciative.

Thanks.

pD

I too have this question.

I posted it a few days ago and nobody has answered me. It's here:
http://forums.devshed.com/t156981/s.html

Hopefully they will answer you.

-- Matt

There is no reason why the XML document has to be held all in memory. You could open a file and create it a line at a time, writing each line to the file as you go.

Similarly when reading an XML file, SAX parsers are designed to read and parse it an element at a time, so are not limited by memory either. DOM parsers on the other hand generally DO read the whole document and create an in-memory representation, so may have problems with huge files.

If you are concerned with passing large files around you can compress it once you have written it to disk, using a standard compression format such as zip, gzip etc. XML files contain a LOT of redundant information so can compress down to a fraction of their original size.

Dave - The Developers' Coach

As we are doing this for other people (read: customers, clients), we have to make sure it's perfect and wont have any "side-effects" or we may lose the client. It's very important we get it right.

My company's client wanted to develop a HUGE knowledge store about species of trees, parts, usage, etc etc and they have over 15,000 species to be included at 1st version. I am assigned to write the XML file to load the information into Flash and I was given 2 months to write just all that in XML (that's about 250 species per day). But I really do not think writing 15,000 different trees together with their sub information is practical on 1 single XML file. I read somewhere, too large a XML can freeze the user's pc. If this happens, then we have lost the client.

These are to be packed into a CD to be sold and the user is supposed to put the CD in and the Flash loads, and then search whatever trees he wants.

One more thing though, how to you write the search algorithm to search for the thing the user wants?

Good question, and I would like to know the answer myself.
Our company has to send a HUGE amount of statistical info off to regulatory agencies on a regular basis. Currently I think this is still done with old-time cobol batch programs writing tapes, mostly; perhaps some via ftp; dunno for sure about that last one as I work on the web side of stuff. Would be interesting to hear from folks who's companies have to do similar reporting functions and have moved to newer technologies such as XML/Web services...

fv