SFTP inbound files: Seen before closed?

Hello,

A customer uses SFTP to put (i.e. "push") files to our
External Linux server (Legacy Linux 7.2).

Our Internal Linux server constantly loops and scans customer
directories on that External server, looking for inbound files.

When an inbound file is found, it is copied via ssh rsync from
the External server to the Internal server, where it gets
decrypted.

Problem: Internal server is recognizing files written by SFTP
before those files arrive in their entirety. This results in
decryption errors on the Internal server when an incomplete
file is copied to there.

This problem does not occur if a customer stores their file
with FTP. The Internal server does not detect an FTP'd file
until it is written in its entirety.

Is this because FTP stores files more quickly than SFTP ?

Or is it because files stored by FTP are not created in the
directory until they are closed? SFTP'd files seem to be available in their directory as soon as they start to be written, ahead of when they complete and are closed.

Can the behavior of SFTP be changed by a CONFIG file
entry? If so, what is that entry and where is it located?

Have suggested our SFTP customer store their files in a
/TMP directory, then move to the scanned directory once
SFTP completes the put and displays the command prompt.

The customer is resisting, need a convincing explanation.

Thank you.

More info from original poster:

SFTP does its own transparent encryption.

In additon to that, customer encrypts with PGP.

So, Internal server must decrypt with PGP after the
SFTP transparent decryption takes place.

If you do not have a primo answer, I will settle for a bs answer.

SFTP files are visible in the directory before they are
fully written.

This is not the case with FTP.

Therefore an remote site cannot simply to transition to
SFTP withou wreaking major havoc on the support
implemented on the receiving host, which formerly
used FTP.

The allure of SFTP is the SSH security on the userid
and password. So what? We have never been hacked.

Hasn't anyone else had to deal with this issue?!?!?!

Fountain_spray

I assume your concern is that you have another process/daemon that looks for the file's existance, and then starts some process. It gets confused when the file is visible but not all there.

If so, here is a simple work arround, not elegant, but should work. Assume you want to send 'bigdatafile' which is big enough to take some time, more than a meg or two.

Change your daemon to look for 'doorbell' file.
then have your client first sftp 'bigdatafile' file, than then send a tiny file called 'doorbell'

I'd kinda agree with fishtoprecords suggestion. I've previously got around similiar issue by first sending a <filename>.start file, then sending <filename> followed by <filename>.complete - obviously <filename>.start and <filename>.complete are just empty marker files. In your case, you probably just need to check for the presence of <filename>.complete files.

Hello fishtoprecords and atlantisstorm,

Thank you both for venturing a solution.

Our application dispatches an independent task for each file
that gets found by our directory-watch "scanner" process
(written in c++). Everything else is in Perl, with some bash.

Our scanner runs continuously and without delay.

So, lets say bigfile is found. Our dispatcher will create a
task and call the top-level processing "wrapper" program.
I can have the wrapper program read in a control file
that will only be set if a prior doorbell inbound file was
retrieved. If not set, wrapper shall terminate further
processing of the task for the assumed incomplete bigfile.

When a doorbell file enters our dispatching process, we
write to a control file. The next time our scanner
presents bigfile to dispatcher, dispatcher kicks off wrapper.

Wrapper shall read the doorbell control file to see if the
doorbell has "rung". If so, then reset the doorbell control
file and process bigfile.

That said, our customer has sworn to do nothing special for
us. That is, they will not so much as write a doorbell file.

So, a variation on the doorbell concept would be to check
the customer-ID. If this file is from the SFTP push customer,
then store the size of the file into a control file. Check that
control file the next time a task is created for that file
(instead of a doorbell file). Compare the sizes, and if equal,
then process. If not equal, then store the new size,
terminate that task, and await the next dispatcher task
for that file.

Here is my quick and dirty solution:

Use CRONtab to activate a process that handles recycling the
scanner.

Change the default scanner to leave out the inbound folder
of our SFTP push customer from continuous scanning.

Then, once a day, turn off scanning for all other customer
inbound folders, and turn on scanning for the inbound folder
of our SFTP push customer.

That way, unless the SFTP push customer is sending a file
at that moment, any files that have accumulated in their
inbound folder will have been fully written and closed by then.

After say, 1 minute, turn off scanning for the SFTP push
customer inbound folder and turn on scanning for all other
customers.

This may cause its own set of problems, as not all processes
will have been killed between each recycling of the scanner
process.

But this is the only way I can address the issue as framed
by our customer without having to beg or write alot of code.

fountain_spray

Have you tried to see what the behaviour is using FTPS (over SSL, rather than SSH)? That may be an alternative way to bring encryption into it, and it should behave more like normal FTP.

Hello Ishnid,

Great suggestion, but as our Legacy Linux 7.2 from
RedHat does not contain SSL support, therefore
we do not have FTPS available.

Besides, our customer decided on SSH/SFTP so
we are obliged to support that.

Fountain_spray

Hello folks,

The best solution is to use either of these commands to
determine if the file is still being written to:

/sbin/fuser -a -u -v /pathname/filename

/usr/sbin/lsof -u userid

Thanks all.

Fountain_spray