#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2012
    Location
    Baltimore, MD
    Posts
    37
    Rep Power
    6

    FTP Script Help!!


    I am trying to write a script that allows me to check a directory for files that i will sftp to another server.

    When the file is created it creates a .ready file that is supposed to signal a file is fully created and ready to sftp. for example a file named 10312014.13.07.40.27711113 would have trailing 10312014.13.07.40.27711113.ready file as well.

    i dont know how to properly check for 10312014.13.07.40.27711113.ready and if it's there sftp 10312014.13.07.40.27711113.

    below is what I have so far, it's just a shell of an idea, I'm stuck and any help would be greatly appreciated.

    Thanks!

    Code:
    #!/bin/bash
    source /opt/foobar/$1/config/usrconfig.sh
    source  /opt/foobar/$ENV/config/$2
    
    ### foobar environment variables ###
    export FTPHOST="sftpserv.foobar.com"                                    # server to ftp to
    export FTPUSR="usmssh"                                              # ftp user id
    export FTPPASS=""                                      # ftp user password
    export PATTERN=                                      # pattern for file name to ftp
    export USING_MARKER=no                                          # using a marker file???
    #export TDEMARKER="X"                                    # pattern for marker file if no files to ftp
    export FTPDIR="prod/TDE/in/"  # directory to ftp to on ftp server
    export DAEMONLOG=/opt/foobar/$ENV/log/TDEdeamon.log
    export DAEMONSLEEP=10
    export CURRDATE=`date +%Y%m%d`
    export READDIR=/opt/foobar/$ENV/data/TDE/output
    export TEMP=/opt/foobar/$ENV/tmp
    export TDE_BACKUP=$APP_ROOT/data/TDE/backup/
    
    cd $READDIR
    if [ -f  $READDIR ]
       then
         echo " $CURRDATE files found to send!"
       else
         echo "no files found in $READDIR to send!"
         exit
    fi
    sleep 2
    
    
    for i in `ls $READDIR`
     do
            cp $i $TDE_BACKUP
            echo $i
    
    
            if [ -f $i.ready ]
            then
             rm *.ready
    
    			/usr/bin/sftp $FTPUSR@$FTPHOST << E_O_F
    			cd $FTPDIR
    			put  $i
    			ls -l $PATTERN*
    			quit
    E_O_F
            else
            echo "No files are ready"
            fi
    done
    .
  2. #2
  3. Maddening Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    16,452
    Rep Power
    9645
    Naturally there's a few ways you could accomplish this, but my tactic would be to construct a big "string" of commands to send to sftp. Goes like
    1. echo the cd command
    2. Use find to get all *.ready files
    3. Pipe into sed to remove the .ready extension and to prepend a "put"
    4. echo the ls and quit commands
    5. Pipe the whole thing into sftp
    Code:
    (echo cd $FTPDIR; find $READDIR -depth -maxdepth 1 -name '*.ready' | sed -r 's/(.*)\.ready/put $1/'; echo ls -l $PATTERN*; echo quit) | /usr/bin/sftp $FTPUSR@$FTPHOST
    You can test by not piping into sftp and you should get output like
    Code:
    cd <ftp dir>
    put /opt/foobar/<env>/data/TDE/output/10312014.13.07.40.27711113
    put /opt/foobar/<env>/data/TDE/output/<files...>
    ls -l <pattern>
    quit
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2012
    Location
    Baltimore, MD
    Posts
    37
    Rep Power
    6
    Thanks so much for your help! your input helped me get over my first hurdle, however another one arose that I am having issues with, and I am sure it has to do with how I coded this thing up.

    I need to send multiple files, not just one. I thought I could send multiple like this, but I cant, so I played with creating a batch file, but I am clearly not doing this right.

    Please see below code.

    What am I doing wrong now? Thanks in advanced!


    Code:
    #!/bin/bash -x
    source /opt/foobar/$1/config/usrconfig.sh
    
    ### foobar environment variables ###
    export FTPHOST="sftpserv.foobar.com"                                    # server to ftp to
    export FTPUSR="foouser"                                              # ftp user id
    export FTPPASS="foo123"                                      # ftp user password
    export PATTERN=                                      # pattern for file name to ftp
    export USING_MARKER=no                                          # using a marker file???
    #export TDEMARKER="X"                                    # pattern for marker file if no files to ftp
    export FTPDIR="/home/foo"  # directory to ftp to on ftp server
    export DAEMONLOG=/home/foobaruser/TDEdeamon.log
    export DAEMONSLEEP=10
    export CURRDATE=`date +%d-%m-%Y-%H-%M`
    export READDIR=/home/foobaruser/TDE
    #export TEMP=/opt/foobar/$ENV/tmp
    export TDE_BACKUP=/home/foobaruser/TDE/backup
    
    cd $READDIR
    if [ -a  $READDIR ]
       then
         echo " $CURRDATE files found to send in $READDIR!" >>$DAEMONLOG 2>&1
       else
         echo "no files found in $READDIR to send!" >>$DAEMONLOG 2>&1
         exit
    fi
    sleep 2
    
    touch tde_batch
    for i in *.ready
     do {
     FILE="${i%.ready}"
     [ -e "$FILE" ] || continue
      echo " Going to put ${FILE} in $FTPDIR directory" >>$DAEMONLOG 2>&1
      echo "put ${FILE}" >> tde_batch
      }
      echo "quit" >> tde_batch
      
      sftp -b tde_batch $FTPUSR@$FTPHOST  
      
            cp $i $TDE_BACKUP
                cd $TDE_BACKUP
                rm *.ready
    done
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2012
    Location
    Baltimore, MD
    Posts
    37
    Rep Power
    6
    I ran the script with a -x below is the output.

    It looks like it's running fine until it finds the first file, it echo's the file, writes it to the batch file, then echoes the quit to the batch file, I want to traverse, echo every single file into the batch file first, then echo the quit, then let the batch file sftp.

    My logic is clearly wrong, some proper directive would be appreciated!

    Thanks! I hope this is better information to help with. Also I see the permission denied error, but I assume that batch needs ssh keys shared and not a password, in the production env, the keys are already shared and that wont be an issue

    Code:
    + source /opt/foobar/or_foo/config/usrconfig.sh
    + export FTPHOST=foobar01.foobar.com
    + FTPHOST=foobar01.foobar.com
    + export FTPUSR=foouser
    + FTPUSR=foouser
    + export FTPPASS=fooy123
    + FTPPASS=fooy123
    + export PATTERN=
    + PATTERN=
    + export USING_MARKER=no
    + USING_MARKER=no
    + export FTPDIR=/home/foouser
    + FTPDIR=/home/foouser
    + export DAEMONLOG=/home/foobaruser/TDEdeamon.log
    + DAEMONLOG=/home/foobaruser/TDEdeamon.log
    + export DAEMONSLEEP=10
    + DAEMONSLEEP=10
    ++ date +%d-%m-%Y-%H-%M
    + export CURRDATE=03-11-2014-10-07
    + CURRDATE=03-11-2014-10-07
    + export READDIR=/home/foobaruser/TDE
    + READDIR=/home/foobaruser/TDE
    + export TDE_BACKUP=/home/foobaruser/TDE/backup
    + TDE_BACKUP=/home/foobaruser/TDE/backup
    + cd /home/foobaruser/TDE
    + '[' -a /home/foobaruser/TDE ']'
    + echo ' 03-11-2014-10-07 files found to send in /home/foobaruser/TDE!'
    + sleep 2
    + touch tde_batch
    + for i in '*.ready'
    + FILE=10312014.10.36.51.52667612
    + '[' -e 10312014.10.36.51.52667612 ']'
    + echo ' Going to put 10312014.10.36.51.52667612 in /home/foouser directory'
    + echo 'put 10312014.10.36.51.52667612'
    + echo quit
    + sftp -b tde_batch foouser@foobar01.foobar.com
    Permission denied (gssapi-keyex,gssapi-with-mic,publickey,password,keyboard-interactive).
    Couldn't read packet: Connection reset by peer
    + cp 10312014.10.36.51.52667612.ready /home/foobaruser/TDE/backup
    + cd /home/foobaruser/TDE/backup
    + rm 10312014.10.36.51.52667612.ready
    + for i in '*.ready'
    + FILE=10312014.11.15.41.64934033
    + '[' -e 10312014.11.15.41.64934033 ']'
    + continue
    + for i in '*.ready'
    + FILE=10312014.11.22.53.52444830
    + '[' -e 10312014.11.22.53.52444830 ']'
    + continue
    + for i in '*.ready'
    + FILE=10312014.12.34.26.49837220
    + '[' -e 10312014.12.34.26.49837220 ']'
    + continue
    + for i in '*.ready'
    + FILE=10312014.12.36.16.5815654
    + '[' -e 10312014.12.36.16.5815654 ']'
    + continue
    + for i in '*.ready'
    + FILE=10312014.12.47.03.24215656
    + '[' -e 10312014.12.47.03.24215656 ']'
    + continue
    + for i in '*.ready'
    + FILE=10312014.12.49.21.40757399
    + '[' -e 10312014.12.49.21.40757399 ']'
    + continue
    + for i in '*.ready'
    + FILE=10312014.13.01.41.84187558
    + '[' -e 10312014.13.01.41.84187558 ']'
    + continue
    + for i in '*.ready'
    + FILE=10312014.13.07.18.23453400
    + '[' -e 10312014.13.07.18.23453400 ']'
    + continue
    + for i in '*.ready'
    + FILE=10312014.13.07.40.27711113
    + '[' -e 10312014.13.07.40.27711113 ']'
    + continue
    + for i in '*.ready'
    + FILE=10312014.13.26.27.08714191
    + '[' -e 10312014.13.26.27.08714191 ']'
    + continue
    
    
    
    [foobaruser@foobar02 TDE]$ cat tde_batch
    put 10312014.10.36.51.52667612
    quit
  8. #5
  9. No Profile Picture
    Contributing User
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2006
    Posts
    2,635
    Rep Power
    1811
    That's because in your loop you echo a quit command into the batch file for each file name you loop over. In fact you have a fully process each file set of commands output for each file. If you are going to do it that way then, before the loop echo any connect commands into the script you are going to pass to sftp. Inside the loop echo any sftp commands (put, etc.), then after the loop put any tidy up commands, such as the quit. One additional useful thing is that you should be able to get sftp to run a command for you (not sure what that would be, but in normal ftp it is the ! character). So you should be able to do something like echo "! echo "processing filename $i..." >> tde_batch" along with the cp and rm commands so they can all be done 'in-line'
    The moon on the one hand, the dawn on the other:
    The moon is my sister, the dawn is my brother.
    The moon on my left and the dawn on my right.
    My brother, good morning: my sister, good night.
    -- Hilaire Belloc
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2012
    Location
    Baltimore, MD
    Posts
    37
    Rep Power
    6
    Originally Posted by SimonJM
    That's because in your loop you echo a quit command into the batch file for each file name you loop over. In fact you have a fully process each file set of commands output for each file. If you are going to do it that way then, before the loop echo any connect commands into the script you are going to pass to sftp. Inside the loop echo any sftp commands (put, etc.), then after the loop put any tidy up commands, such as the quit. One additional useful thing is that you should be able to get sftp to run a command for you (not sure what that would be, but in normal ftp it is the ! character). So you should be able to do something like echo "! echo "processing filename $i..." >> tde_batch" along with the cp and rm commands so they can all be done 'in-line'
    Please excuse my ignorance, but I don't completely understand what you're suggesting I do with the loop, How do I make whats in the { } keep looping before exiting and then echoing the quit
  12. #7
  13. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2012
    Location
    Baltimore, MD
    Posts
    37
    Rep Power
    6
    I got it to work finally.

    Below is finished product.

    Thanks for all involved!

    Code:
    #!/bin/bash -x
    source /opt/foobar/$1/config/usrconfig.sh
    
    ### foobar environment variables ###
    export FTPHOST="sftpserv.foobar.com"                                    # server to ftp to
    export FTPUSR="foouser"                                              # ftp user id
    export FTPPASS="foo123"                                      # ftp user password
    export PATTERN=                                      # pattern for file name to ftp
    export USING_MARKER=no                                          # using a marker file???
    #export TDEMARKER="X"                                    # pattern for marker file if no files to ftp
    export FTPDIR="/home/foo"  # directory to ftp to on ftp server
    export DAEMONLOG=/home/foobaruser/TDEdeamon.log
    export DAEMONSLEEP=10
    export CURRDATE=`date +%d-%m-%Y-%H-%M`
    export READDIR=/home/foobaruser/TDE
    #export TEMP=/opt/foobar/$ENV/tmp
    export TDE_BACKUP=/home/foobaruser/TDE/backup
    
    cd $READDIR
    if [ -a  $READDIR ]
       then
         echo " $CURRDATE files found to send in $READDIR!" >>$DAEMONLOG 2>&1
       else
         echo "no files found in $READDIR to send!" >>$DAEMONLOG 2>&1
         exit
    fi
    sleep 2
    
    touch tde_batch
    cd $READDIR
    find $READDIR -type f -name '*.ready' | while read file
    do
      real_file="${file%.ready}"
      echo "put $real_file $FTPDIR" >> tde_batch
      cp $real_file $TDE_BACKUP
    done
    echo "quit" >> tde_batch
    sftp -b tde_batch "$FTPUSR@$FTPHOST"
    cp tde_batch $TDE_BACKUP/tde_batch.$CURRDATE
    rm tde_batch  
    rm *
    exit 0
  14. #8
  15. No Profile Picture
    Contributing User
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2006
    Posts
    2,635
    Rep Power
    1811
    That's the general idea, yes. There's a couple of things that give me a little twitch ... that final rm * is one of them, but I can see the idea/need behind it. But when you link that to the cd commands (you do the same cd twice, by the way) with no check that the target directory exists I ... twitch a little
    The moon on the one hand, the dawn on the other:
    The moon is my sister, the dawn is my brother.
    The moon on my left and the dawn on my right.
    My brother, good morning: my sister, good night.
    -- Hilaire Belloc

IMN logo majestic logo threadwatch logo seochat tools logo