#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2003
    Location
    Austin, Texas
    Posts
    132
    Rep Power
    14

    Too many files hurts performance?


    Note: This was originally posted in the MySQL forum http://forums.devshed.com/t106677/s.html

    For every record in my database I am storing two files named after the records primary key. When a user submits a query, the files are called for each record that matches his query. I will be having several million records and the files will range from 5kb to 2mg.

    This is working fine on a small scale (5,000 records), but I read this from http://lists.mysql.com/java/900
    One of the things to watch for is that you can't put too many files, more than a few hundreds, in the same directory on Unix or performance will suffer.
    So I am trying a directory structure using the first four digits of the primary key. Ex: 1234 would go into /1/2/3/4/ and 12345 would go into /2/3/4/5/. I am currently using Windows XP but plan to use FreeBSD 4.X or 5.X.

    How does FreeBSD handle several files in a directory (like 10,000-100,000)? Am I better off breaking the files up by digits, or is this directory structure inefficient?

    Thank you,
    Brandon
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2001
    Posts
    4
    Rep Power
    0
    It's not about directory alone but partition, if you run out of inode, creating more subdirs and relocate files under it doesn't change your inode capacity.
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2003
    Location
    Austin, Texas
    Posts
    132
    Rep Power
    14
    I'm relatively new to Unix and have researched Inode since FreeBSD's post, but I still am a little unsure of the inodes default setting and how to set it larger.

    Iím using FreeBSDís default partition. I have a pretty minimal install, only using the command line interface and just a couple programs (MySQL, Java SDK). I plan to add anywhere from 500 million to 1 billion files. Will the default partition for a 100gig hard drive have a large enough inode capacity?

    Also, does it matter how many files are in a given directory (assuming itís under the inode limit) when locating a given file? I am splitting them up as mentioned earlier so there wonít be too many files in one directory, but if it doesnít matter then I will put all the files in one.

    Thanks again,
    Brandon
    Last edited by ace2600; January 5th, 2004 at 07:36 PM.
  6. #4
  7. Banned ;)
    Devshed Supreme Being (6500+ posts)

    Join Date
    Nov 2001
    Location
    Woodland Hills, Los Angeles County, California, USA
    Posts
    9,782
    Rep Power
    4300
    >> but I still am a little unsure of the inodes default setting and how to set it larger.
    See the -i option for newfs. Online man page at: http://www.freebsd.org/cgi/man.cgi?q...SE&format=html
    Up the Irons
    What Would Jimi Do? Smash amps. Burn guitar. Take the groupies home.
    "Death Before Dishonour, my Friends!!" - Bruce D ickinson, Iron Maiden Aug 20, 2005 @ OzzFest
    Down with Sharon Osbourne

    "I wouldn't hire a butcher to fix my car. I also wouldn't hire a marketing firm to build my website." - Nilpo
  8. #5
  9. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2003
    Location
    Austin, Texas
    Posts
    132
    Rep Power
    14
    Thank you Scorpions4Ever, I had no idea what command to search for. As for my second question:

    "does it matter how many files are in a given directory (assuming itís under the inode limit) when locating a given file? I am splitting them up as mentioned earlier so there wonít be too many files in one directory, but if it doesnít matter then I will put all the files in one."

IMN logo majestic logo threadwatch logo seochat tools logo