1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2012
    Rep Power

    I have an odd question.

    I have windows 7, and I made a text file read/write database for a chatbot to use. I have a small issue that is located in the red box.

    I know that this is because of the sectors the NT File System uses. I just don't want a 4 MB real size to become 300 MB of useless space.

    Could I increase the performance by moving this "Database" comprised of thousands (and later maybe millions) of small files by moving it over to a flash drive, on a different file system (like EXT2) or something like a solid state harddrive.

    Also, any tools that are optimized to fix problems exclusively like this would be appreciated.

    A secondary note, I could put the entire database into one file, but python would have to read that file each time the "database" would need to be called (which is after every input). I plan for the "database" to be very large, and I would like to keep loading times small.
  2. #2
  3. No Profile Picture
    Lost in code
    Devshed Supreme Being (6500+ posts)

    Join Date
    Dec 2004
    Rep Power
    The difference between the size and size on disk is due to the block size used by the file system. Virtually all file systems are block based like NTFS; most, including NTFS, allow you to choose the block size when you format the disk. So if you were to create a new partition or use a different storage device and reformat it with a smaller block size it would reduce the amount of unused space consumed by files that are small.

    Placing the files on a flash drive or ssd would probably increase performance due to the decreased seek time. However, on a USB 2 connection the actual data transfer speed would be slower. For small files this matters less than it does for large files.

    Also, any tools that are optimized to fix problems exclusively like this would be appreciated.
    Using an actual database engine solves this problem.

    Originally Posted by Spad
    Ah USB, the only rectangular connector where you have to make 3 attempts before you get it the right way around

IMN logo majestic logo threadwatch logo seochat tools logo