Job Runs Twice

We have a production job that checks for the arrival of 2 files (a trigger file and a data file). It checks every 1 minute for the existence of the files and also monitors file size for 15 seconds for size stability. After it finds the files, it processes the data file (usually takes about 2 minutes) and then moves and renames both files to an archive directory after the run.

This has been working fine, but for some reason last night after the job completed it started up again about 20 seconds later. This time it failed because the data file was not there. When I check the log, the second job starts because it says the 2 files exist in the directory even though they are gone by the time it starts.

I was able to replicate this behavior several more times this morning, and watched the files being moved to the archive directory prior to the 2nd job "finding" them and starting. I decided to change the amount of time that it monitors file size from 15 seconds to 5 seconds and that fixed the problem, it didn't run a second time.

Any idea what is going on?

There is an issue with the job setup. The job is set to check for files every minute. It takes about 2 minutes to run and only then it moves the files. The scheduler ignores semaphore files while the job is running, but once it completes (and files aren't moved yet) , the scheduler starts checking for files again. Because of the concurrency issues, it may happen that the scheduler will find both files again before the job is done with moving them after job is run, and as a result it will spawn a new instance of the same job, which should fail because file are being moved.

There are 2 solutions.
1. Set file check interval greater than job run duration. the interval should also have some cushion to accommodate chances of unexpected delays.
2. Move files to a staging directory on job start and then process them from there. This is a better and more reliable solution.