I have a folder on my Mule ESB server which several sources can point to (one SCP, one SFTP, along with others). Whenever I detect a file, I want to move it to another directory. The problem is, I'm moving the partially-completed files causing me to lose the data.
I've tried using the File's "fileage" attribute, but it doesn't seem to work reliably. I'm trying to keep latency down as much as possible, since most files will be <10kb, but some will need to be as large as 100mb.
Is there any way to know if a file is a partial, versus a complete? I know WinSCP uses the .filepart extension, but that's just for one application, and even then that extension can be changed (or completely removed) in WinSCP's preferences.
I solved my own problem.
With a bit more testing it turns out the problem is at the OS level. On RedHat the "Last Modified" time stamp is only updated every ~1000ms, creating too large a latency for the 4kb files. On Windows it's even worse: the "last modified" is only set when the transfer starts, then reset when it completes. The MuleSoft team is technically correct to pull fileAge from this time stamp, but the OS isn't updating it often enough to work. They should use file size instead. I'll be submitting a work-around ticket.