Dealing with Very Large Text Files...

As a course of my work I commonly deal with parsing data output (stored in ASCII text files). This output can commonly be hundreds of megabytes if not larger... At times, I want to inspect the data, but I often find even the most powerful of systems is no match for a 2 GB txt file. I know things would be much easier if I just had this all in a database, but we're just starting that integration now.

Heres my question, what is the best program to view and tinker with these VERY large text files? Usually I like to stay in win2k, but linux is also a viable option. Simplicity is good, though functionality and ease of use are also considered. WordPad has its limits, anything above 700 MB or so seems to lock up my machine pretty well. I don't even need to see the entire thing, maybe just a preview of say the first X number of lines... Something like a notepad that could open a 2GB file would be great...

Would buying more RAM or adding and additional processor (already running at 700 MHz) aid me in this persuit?

On a side note, what would be an optimal system configuration to parse such files? Is CPU or RAM more important when doing this? My system usually does fine when parsing these files, even the very big ones (the parsing scripts are a mixed bag: some PHP, others Perl, still others VB; whatever seems easiest or whoever has time to program them ). It does take a while, and sometimes the data comes out in an incorrect format, so its nice to preview the raw data to make adjustments before I parse it incorrectly by mistake...

Thanks,

Zach Sniezko

RAM is almost always beneficial. At the current prices, you should be able to get 1 GB of RAM for under $300 (two 512 MB PC133 SDRAM DIMMs).

However, take a look at UltraEdit. I've never had to edit 500 MB files, but I have had to open 2-5 MB files before and UltraEdit didn't even flinch. You can download an evaluation copy off their site.

It's also a great editor for coding -- it has many features I can no longer live without.

Hi. I used to do computational chemistry (in a past life ) which would generate very large "scratch" files. These files would range in size from 2 - 30 Gigs.

Linux (at least Red Hat) does not like files which are larger than this size. Fortunately we were able to split the file into multiple 2 GIG chunks.

My point is, Linux may not be as viable an alternative as you would like.

Cheers,
Tim

Linux would be fine ... 2GB is a long way from 30GB

Linux has a command called head. The head command will take a file and give you just the first bit you ask for.
"head thefile -c 2000000 > afile" will create afile with the forst 2MB from thefile. You can then open afile in an editor easily.

Thanks for the help!

If you want to stay on Win2k like you said you may want to check out thegun.exe. I've been using it for quite a while and find it to be quite practical. The author claims that it has no max. file size limitations, the only limitation being your system's memory. I haven't been able to test this claim but I have loaded pretty large files and it is very fast.

It is completely free and the download is only 6k. You can get it here:

http://www.pbq.com.au/home/hutch/thegun.htm

I looked into it further and found the following (excerpt from their site):

Disk based text editing - up to 2GB file size, minimum RAM used even for multi-megabyte files.