#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    May 2000
    Posts
    179
    Rep Power
    14

    net::ftp 2000 image files


    Hi all -

    How would perl handle a job putting about 2000+ files using the net::ftp module? (I don't have any number on the actual file sizes).

    I want to run it as a scheduled job every so often. I have experience using net::ftp, but not with so many files.

    Are there any traps to avoid when implementing this kind of program?

    Thanks
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed God 1st Plane (5500 - 5999 posts)

    Join Date
    Oct 2000
    Location
    Back in the real world.
    Posts
    5,966
    Rep Power
    190
    this will take quite some time since ftp takes several handshakes till transmission starts... regarding ping times, this can become:
    1 ping takes 50ms
    estimate 5 handshake-items, this makes quarter a second per transfer PLUS your actual data.
    1/4*2000 = 500 seconds only for handshaking.

    i have no experience using this module but log your sessions, then you can find out about the number of handshakes...

    do they all go to the same server or different ones?

    anyway,
    send one after the other. donīt use threads or multiple processes with that many files.

    some servers kill your script if it runs for too long though..

    I want to run it as a scheduled job every so often
    check at the beginning if the last job has finished or donīt start. you will overload your server soon.

    ... any other suggestions?
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    May 2000
    Posts
    179
    Rep Power
    14
    All the files are going to the same server.

    I was thinking about using multiple processes, because of the server possibly killing a single long executing process. Though,I don't know how I could use multiple processes creatively or effectively for this type of application.

    Why did you suggest not using multiple process?

    Very helpful feedback.
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed God 1st Plane (5500 - 5999 posts)

    Join Date
    Oct 2000
    Location
    Back in the real world.
    Posts
    5,966
    Rep Power
    190
    if you want to use several processes, dont spawn one for each transfer!! this is what i meant.
    if you are not on an ultra-sparc 10k, more than a few 10 processes at the same time will kill your CPU and Network power.
  8. #5
  9. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    May 2000
    Posts
    179
    Rep Power
    14
    if you want to use several processes, dont spawn one for each transfer!! this is what i meant.
    That would be way crazy. LOL. But it would probably be fun to see how the CPU reacts ...

    Anyway, how would somebody use multiple processes to tackle this? I was thinking about splitting the file transfers into 2, 3, or 4 optional processes. But I'm still in the learning stages with multi-process programming. And I'm learning it on windows, which doesn't help since I'm more comfortably with *nix.

    Windows technology involved
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed God 1st Plane (5500 - 5999 posts)

    Join Date
    Oct 2000
    Location
    Back in the real world.
    Posts
    5,966
    Rep Power
    190
    it is not that much different. i am no perl pro, but fork() should be available in perl...
    be careful with parent/child processes. you need to "disown" somewhere so the forkīed process wonīt be killed if the parent process is...
  12. #7
  13. 11
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Jul 2001
    Location
    Lynn, MA
    Posts
    4,635
    Rep Power
    82
    What kind of boxes are to transferring from/to? You might be better served with rsync, which can work over SSH.

    Rsync is a very smart client- it will only upload files that have changed, and even then only the PARTS of the files that have changed. It's very, very fast as a consequence.

    If you're transferring to a linux/*nix box running SSH, you can most definitely use rsync and cron to do this.
  14. #8
  15. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    May 2000
    Posts
    179
    Rep Power
    14
    transferring from windows 2000 to windows 2000.

    fork() is available in Perl. It doesn't, unfortunately, work quite the same way for both windows and *nix.

    NET::FTP might not be the right solution.
  16. #9
  17. No Profile Picture
    Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Aug 2000
    Location
    Indiana
    Posts
    614
    Rep Power
    15
    Tou guys are talking about stuff way to much ;p. Unless it takes 300+ seconds to transfer one image (in which case you've got bigger problems then the ones discussed here) a typical ftp server wont time you out. Just split up the images and run multiple copied of the script at once (since fork( ) doesn't work in windows), or use threads to have one script do multiple things at once (you might be able to do an interesting thing with setting up a 'queue' for threads to read from -- just read all the files you need to upload into an array (the filenames of course, not the file content itself) and have each thread grab/upload that file, then grab the next one).

    Also, if you're worried about getting timed out of the server -- Net::FTP will let you know if you can't upload a file for some reason:
    Code:
    unless($ftp->put('myfile.jpg')) {
        # something went wrong, it will tell you in $!
        # perhaps re-establish your connection with the ftp server
    }
    Jon Coulter
    ledjon@ledjon.com
  18. #10
  19. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    May 2000
    Posts
    179
    Rep Power
    14
    I greatly appreciate the feedback.

    I like JonLed's suggestion. I read about and tested fork() on windows 2000 with perl > 5.6 and it works, although, not like it's documented in Lincoln Stein's "Network Programming". That's another threaded discussion.

    So, trying the queque thing and forking the job sounds cool.
  20. #11
  21. 11
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Jul 2001
    Location
    Lynn, MA
    Posts
    4,635
    Rep Power
    82
    You haven't begun to live until you've known the true joy of rsync via SSH for transferring files. . . Look into it next time you need to do something like this on *nix.

    But, given that you're on win32, it would be a slightly larger pain the butt to accomplish, though it would be possible.

IMN logo majestic logo threadwatch logo seochat tools logo