#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    May 2012
    Posts
    2
    Rep Power
    0

    Unix script help


    Hi, guys! I'm new here and really need help. :3

    I have to make a Unix script that creates an archive of a target directory on a remote computer, it is like kind of to create a "backup" on a remote computer (in the same LAN).

    The script must be started on the local computer, take the desired directory as an argument and create a number of archives, each no bigger in size than 2GB. Each file must be placed only in one archive file, as opposed to half of it in archive1 and the other half in archive2. The archives must be transfered along the net by using "ssh" with generated keys (to avoid password request every time). The directory on the remote computer can either be given as an argument or be built-in.
    The script must also create a simple database (some form of txt file), listing which archive contains which files.

    The problem is - I have forgotten almost everything about this.
    I am really sorry, but I really need help, so if anyone can tell me what to do I will be very grateful.


    -------------
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Regular (2000 - 2499 posts)

    Join Date
    Mar 2006
    Posts
    2,442
    Rep Power
    1688
    And if a single file is larger than 2GB?
    This is quite a nasty issue. It can be solved but only by parsing a list of the file contents and adding up file sizes - which involves a fair amount of processing and potentially disk space on the target server.
    I have done something similar (at a higher level), load balancing backups across X number of backup streams to ensure each was around the same size. The way I'd do this, without putting too much thought into it, would be to use the output of a du command (specifying the option to return information at the file level) and building up a set of files containing the names of the files to be included for each archive. If du fails to give the full and/or relative path name you'll need to something with, probably, the find command.
    The moon on the one hand, the dawn on the other:
    The moon is my sister, the dawn is my brother.
    The moon on my left and the dawn on my right.
    My brother, good morning: my sister, good night.
    -- Hilaire Belloc
  4. #3
  5. Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Sep 2006
    Posts
    844
    Rep Power
    387

    Cool


    Originally Posted by Durarara
    Hi, guys! I'm new here and really need help. :3

    I have to make a Unix script that creates an archive of a target directory on a remote computer, it is like kind of to create a "backup" on a remote computer (in the same LAN).

    The script must ... Blah, blah, blah....
    You really must try to do your homework assignment on your own, otherwise you will not learn anything.

    Here are some commands you may want to consider using:

    scp (or ssh), tar, gzip, zip, ls, split,...
  6. #4
  7. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    May 2012
    Posts
    2
    Rep Power
    0
    Originally Posted by LKBrwn_DBA
    You really must try to do your homework assignment on your own, otherwise you will not learn anything.

    Here are some commands you may want to consider using:

    scp (or ssh), tar, gzip, zip, ls, split,...
    Oh, no - this isn't my homework assignment, this is from a course I attended two years back. I did it then but have since forgotten how scrips worked with UNIX and wanted to re-do it. And since I have literally forgotten everything about writing scripts for UNIX I need teh help.

    P.S.
    There will not be any files larger than 2 GB, it's just a simple script, no need to worry about that.


    -----------------
  8. #5
  9. No Profile Picture
    Contributing User
    Devshed Regular (2000 - 2499 posts)

    Join Date
    Mar 2006
    Posts
    2,442
    Rep Power
    1688
    You will have the need to use the various openSSH commands: ssh to fire off the script on the remote server, and, probably, scp to transfer the archive across. Leaving all that aside, it is then a 'simple' task to list files with their sizes (I'd suggest du -ak) and do 'something' with the output - either directly via a pipe, or to a temporary file, and then process that.
    In your 'control loop' you'd add up cumulative sizes (the -k will ensure all values are in units of 1K so your addition will be simple) and when the current rolling total is less that your cut-off point arrange for the current listed file to be in the current archive, else zeroise counts, close/finish current archive and start a new one. Rinse and repeat and remember to ignore entries for directories!

    Something like:
    Code:
    X=Arch_Ctrl
    cat /dev/null > $X
    T=0
    C=$(( 2 * 1024 * 1024 * 1024 ))
    A=1
    archive=Arch_$A
    echo $archive $(date "+%c") >> $X
    du -ak | while read s n
    do
      if [ -f "$n" ]
      then
        T=$(( T + s ))
        if [ $T -gt $C ]
        then
    # over limit, new archive
          A=$(( A + 1 ))
          archive=Arch_$A
          echo $archive $(date "+%c") >> $X
          T=s
        fi
    # archive the file
      fi
    done
    The moon on the one hand, the dawn on the other:
    The moon is my sister, the dawn is my brother.
    The moon on my left and the dawn on my right.
    My brother, good morning: my sister, good night.
    -- Hilaire Belloc

IMN logo majestic logo threadwatch logo seochat tools logo