I have an odd question.

I have windows 7, and I made a text file read/write database for a chatbot to use. I have a small issue that is located in the red box.



I know that this is because of the sectors the NT File System uses. I just don't want a 4 MB real size to become 300 MB of useless space.

Could I increase the performance by moving this "Database" comprised of thousands (and later maybe millions) of small files by moving it over to a flash drive, on a different file system (like EXT2) or something like a solid state harddrive.

Also, any tools that are optimized to fix problems exclusively like this would be appreciated.

A secondary note, I could put the entire database into one file, but python would have to read that file each time the "database" would need to be called (which is after every input). I plan for the "database" to be very large, and I would like to keep loading times small.

The difference between the size and size on disk is due to the block size used by the file system. Virtually all file systems are block based like NTFS; most, including NTFS, allow you to choose the block size when you format the disk. So if you were to create a new partition or use a different storage device and reformat it with a smaller block size it would reduce the amount of unused space consumed by files that are small.

Placing the files on a flash drive or ssd would probably increase performance due to the decreased seek time. However, on a USB 2 connection the actual data transfer speed would be slower. For small files this matters less than it does for large files.


Using an actual database engine solves this problem.