November 23rd, 2012, 11:10 AM
I have an odd question.
I have windows 7, and I made a text file read/write database for a chatbot to use. I have a small issue that is located in the red box.
I know that this is because of the sectors the NT File System uses. I just don't want a 4 MB real size to become 300 MB of useless space.
Could I increase the performance by moving this "Database" comprised of thousands (and later maybe millions) of small files by moving it over to a flash drive, on a different file system (like EXT2) or something like a solid state harddrive.
Also, any tools that are optimized to fix problems exclusively like this would be appreciated.
A secondary note, I could put the entire database into one file, but python would have to read that file each time the "database" would need to be called (which is after every input). I plan for the "database" to be very large, and I would like to keep loading times small.
November 23rd, 2012, 02:46 PM
The difference between the size and size on disk is due to the block size used by the file system. Virtually all file systems are block based like NTFS; most, including NTFS, allow you to choose the block size when you format the disk. So if you were to create a new partition or use a different storage device and reformat it with a smaller block size it would reduce the amount of unused space consumed by files that are small.
Placing the files on a flash drive or ssd would probably increase performance due to the decreased seek time. However, on a USB 2 connection the actual data transfer speed would be slower. For small files this matters less than it does for large files.
Using an actual database engine solves this problem.