[VB.NET] Getting an Out Of Memory exception when opening large files.

I'm working on a program that will take a file, grab the hex values for a certain number of bytes, and put them into a string. I dug around a bit and found this line of code somewhere:

rtbInp.Text = String.Join(" ", IO.File.ReadAllBytes(<insert file name>).Select(Function(b) b.ToString("X2")).ToArray())

Turns out, that worked perfectly, as it took each byte in the file, turned it into hex, and put it into the text box. My only alteration was to put it into a variable instead so I could later use SubString to grab specific values. Then I hit one minor problem: files larger than about 150MB. It wouldn't be often that I'd have files this large, but when I attempt to use one that big, I get "Exception of type 'System.OutOfMemoryException' was thrown". I've done some searching and have seen a couple sites that talk about file paging, but when looking at the code, it just confuses the heck out of me.

Basically, I'm looking for one of two things: a way to read the entire file, regardless of size, and avoid the out of memory exception, or a way to take the line of code above, but only have it read in a certain number of bytes (300 would be the most I'd need) while keeping the same functionality.

Any help would be greatly appreciated.

Rather than using File.ReadAllBytes(), you'll need to open a streamreader and read the file in a few bytes at time... say 1024 byte chunks.

Each time your read a chunk perform your hex conversion, and then append the results to a StringBuilder object. Do no use string concatenation.

The other is to remember you are representing each byte of that large file with two characters, and each character is itself two bytes because of unicode, so a 150MB file on disk will use 600MB of RAM when you read it in. Are you sure you won't be better off leaving this file on disk and seeking to only the parts you need?

Well, I initially had code to read in a character at a time which almost worked, but I think I read it in wrong or something. I think I had it get the ASCII value of each character, which gave major issues when that value hit higher than 127, since I was going from ASCII to Hex (yeah, I question my logic on that one), ASCII values above 127 didn't have Hex values that I saw, so I kind of screwed myself.

I'll take a look at it down the road, but in the meantime I did find a small workaround: I used a Try...Catch statement to catch the error, and if the user opens the file in a Hex Editor and deletes any data after a certain byte, it works just fine (only the first fifty or sixty bytes are actually needed).

If only the first 50 or 60 bytes are needed, then for the love of God why are you reading in the entire file?!?

As stated, I at first had a way to read in each character, but I don't think it hit me until I changed the code that I was reading in the file/converting the characters wrong. I was reading the file character by character, converting each character to Hex based on the ASCII value of the character I read in, and that's where the problems began. Anything with an ASCII value higher than 127 gave my program an issue. That's when I dug around on the net and found the line of code in the first post that I used, then actually tried to combine the two ideas, and just got stuck on how to do it.

Bottom line, I'm reading in the entire file because I can't seem to find a good way to only read in the first fifty bytes or so while converting each character to Hex and not giving me issues. I've written several programs for myself in my spare time, but this is my first attempt at reading a file and converting the characters from one type to another.

Something like this maybe?



Adjust the byte array length and count parameter of the .Read as appropriate

If you don't want the dashes then add .Replace("-", string.empty) to the end of the Conversion line.

Anywho,

Cheers!
Twyn

Surprisingly, I figured it out. I did some searching online and found the following:



That actually worked perfectly. I do thank everyone for the suggestions though.