File Move Issues

Right now, a program that we did not write and cannot edit creates a file that one of our scripts copies. Our script waits at regular intervals looking in the directory for the file, and when it finds the file, it copies the file to another location. The problem is that the file being retrieved for copying, yesterday, was 64MB. Our script noticed that the file was present and attempted to copy the file while it was still being written. This caused the script to abort. I've looked at your documentation but didn't find what I was looking for. Do you have a way for a script to be certain that a file is completely written before attempting to manipulate it?

My initial thoughts:
1. I could increase the interval to be so high that any file would have enough time to be written. This is definitely not preferred because most files (all files until yesterday) are so small that this is not an issue, and we don't want the scripts to be dramatically slowed down by needless waiting.

2. I considered changing the program that creates the file to create it with a different extension than what the script expects. Once the program has finished creating the file, it could then rename it to have the extension that the script is expecting. This would ensure that the script would always pickup a completed file. This will not work, though, because I cannot edit the program that creates the file.

Any thoughts?

It all depends on how the file is being written. If the program witting the file, does it in shared read/write mode, there is no way to be certain that a file is completely written before doing anything to that file. However, if the file is opened in a shared read mode (because you can copy it) but exclusive write mode, you can indirectly find out if the file is being used before you copy it. You can try to open file for write/append access and if that attempt fails, you can be certain the file is being used by some other process.

Another commonly used solution is monitoring file size stability. If the program that created the file, keeps writing to this file without long pauses you can have JAL or VBScript job get the current file size, wait for several seconds and then get the file size again. If both sizes match, it can start file copying operation, otherwise wait for a few seconds and get the size again. Repeat that until two consecutive size samples match. I believe there is a ready to use JAL example or template available in 24x7 help just for this type of the file processing.

And finally we have developed FileCopy and LineCopy utilities that can be used to copy files incrementally or partially (from end or start or middle). Sometimes when dealing with large text files you may want to access or copy certain parts of the file before the entire file is ready For more info and to download them please visit http://www.softtreetech.com/24x7/script.htm

Great! Thank you for your help.

We decided to try using the file size checking, but I ran into a problem. Here's the code with comments explaining the problem:

Code:

dim dest_file, string, "C:\\Documents and Settings\\Joshua Youngblood\\Desktop\\BjyTest.txt"
dim found_file, boolean
dim file_size, number, 0
dim files_match, boolean
dim message, string
dim prev_file_size, number, 0
dim source_file, string, "F:\\Home\\Joshua Youngblood\\BjyTest.txt"

look_for_source_file:

logaddmessageex "INFO", 100070, "MyTest", "Looking for source file..."

// I start copying a large file into the source directory. This statement, as expected,
// finds the file, even though the file is only partially copied at this point.
fileexists source_file, found_file
if found_file, get_file_size, wait_for_file
get_file_size:
// Because the file was found, this code is executed; but the script dies here with this
// error: "File not found". It appears that "filesize" thinks that a file being written doesn't exist.
// Is it supposed to behave this way? I still didn't find the example of how to do this in the samples,
// but this seems simple enough.
filesize source_file, file_size

concatex "Previous Size: ", prev_file_size, "; Current Size: ", file_size, message
logaddmessageex "INFO", 100070, "MyTest", message

isequal file_size, prev_file_size, files_match
if files_match, copy_file, wait_for_file_to_be_written
copy_file:
   filecopy source_file, dest_file
   goto end_file_size_check
wait_for_file_to_be_written:
   set prev_file_size, file_size
   wait 2
   goto get_file_size
end_file_size_check:

goto end_file_copy
wait_for_file:
wait 5
goto look_for_source_file
end_file_copy:

I have something that works, now.

Code:

// Variables
dim dest_file, string, "C:\\Documents and Settings\\Joshua Youngblood\\Desktop\\BjyTest.txt"
dim file_size, number, 0
dim file_is_written, boolean
dim last_error, string
dim source_file, string, "F:\\Home\\Joshua Youngblood\\BjyTest.txt"

// Ensure that the script doesn't die, if the file isn't
// finished being written.
get_source_file_availability:
logaddmessageex "INFO", 100070, "MyTest", "Looking for source file..."

onerrorresumenext
filesize source_file, file_size
onerrorstop

getlasterror last_error

isequal last_error, "", file_is_written
if file_is_written, copy_file, wait_for_file_availability
copy_file:
logaddmessageex "INFO", 100070, "MyTest", "Copying the source file to the destination..."
filecopy source_file, dest_file
goto end_file_check
wait_for_file_availability:
logaddmessageex "INFO", 100070, "MyTest", "Waiting for entire source file to become available..."
wait 5
set last_error, ""
goto get_source_file_availability
end_file_check:

I would like to point out another quirk that I discovered while trying to find a solution. In the above code, I am using an "if" statement. I originally tried using a "choosecase" to see if "last_error" was an empty string. For some reason, when I did that, the script died with a file not found error. It happens even if I put an "onerrorresumenext" before the choosecase. I say this so that SoftTree can take a look at it and so that others attempting to accomplish the same thing will use an "if" instead of a "choosecase".

I'm not sure why you need getlasterror and error control. In the event this job is running, you alredy know the file is there. Why to make it more complicated. Is this job triggered from some other job? Can it be converted to a regular file-watch job and have 24x7 check for you the file size stability using buil-in job properties? In the latest case all you need is JAL's filecopy of "cmd /C copy" for a program type job

You wrote: "In the event this job is running, you alredy know the file is there."
Answer: I don't know the file is there. This code is doing exactly what I was attempting to do in the previous code that I posted using "fileexists" and "filesize". Because "fileexists" is useless if "filesize" still thinks the file doesn't exist, I'm using "filesize" to check for file existence and completion. "filesize" dies with a "File Not Found" error if the file is in the process of being written or if it doesn't exist. This gives me both checks in one. Essentially, if "filesize" doesn't generate an error, I know that the file exists AND is finished being written. At that point, I can copy it.

You wrote: "Why to make it more complicated"
Answer: I agree that this is more complicated to comprehend for other programmers, but when the JAL commands don't behave as expected, I have to improvise. I would prefer to use the first code that I posted even though it's longer.

You wrote: "Is this job triggered from some other job?"
Answer: No. It's manually kicked off by our operator. We have some programs that create reports, but the program that puts the file in the source directory is not written by us. The script (not the little prototype that I've posted), which is about 600 lines and kicks off about 5 or 6 other programs, is simply waiting for the source file that gets created by one of our reports, even though our report doesn't put the file in the source directory. The script currently checks for the existence of the file using JAL's "fileexists". It correctly noticed that the large source file exists, but the source file was still being written. Well, the script was programmed to assume that if it found the file, then it could copy it. The next thing done was a JAL "filecopy". The script died with a "File Not Found" error because the file was still in the process of being written. So I just needed to think of a way to be sure that the file was completely written before copying.

You wrote: "Can it be converted to a regular file-watch job and have 24x7 check for you the file size stability using buil-in job properties? In the latest case all you need is JAL's filecopy of "cmd /C copy" for a program type job"
Answer: I'm not entirely sure what you mean. Although, I just read your documentation again and found something that might help me clarify my code. Using FileDate and FileTime, I think that I could check source file's modified date (instead of file size) after I know that it exists at regular intervals; and when several seconds has passed without a modification, I can assume the file is no longer being written.

I see the issue now. The FileSize causes errors when if the file is still in use. I thought this file was available for reading, not locked.

If this is the case you should be simple able to run a loop querying FileSize, sort of a dummy query

Code:

// Variables
Dim dest_file, string, "C:\\Documents and Settings\\Joshua Youngblood\\Desktop\\BjyTest.txt"
Dim file_size, number, 0
Dim source_file, string, "F:\\Home\\Joshua Youngblood\\BjyTest.txt"

// set error handler
OnErrorGoTo WAIT

CHECK_AGAIN:
// do dummy check for file size
FileSize source_file, file_size
OnErrorStop

// copy file and exit
LogAddMessageEx "INFO", 100070, "MyTest", "Copying the source file to the destination..."
FileCopy source_file, dest_file
Exit

// wait for 10 seconds and check file availability again
WAIT:
LogAddMessageEx "INFO", 100070, "MyTest", "Waiting for entire source file to become available..."
Wait 10
GoTo CHECK_AGAIN

I'm sorry that I didn't communicate my problem clearly, earlier. Thank you for your help.

I'm sorry if I my post made you think that you "didn't communicate my problem clearly". I only stated that my original assumption was wrong. Because the file could be copied ina middle of the processing, I assumed that you could do almost anything to that file.

In the regard to file watch jobs, in I case I didn't express this reference clearly, there is a built-in file-watch future in 24x7 that can be used to monitor file sizes before kicking a job. You will see it if you choose "when certain files exist" schedule type. In the next step in the job properties wizard you can choose to have 24x7 watch for file size stability after it finds the required files. The nice thing about this feature is that 24x7 can handle multiple file names at once as well as it can handle file masks as well as dynamic macro variables. For example, if a job requires presence of files with TXT extension and CSV extension created today in folder A, a watch condition can be specified as
C:\A\someprefix@T"mmddyyyy"*.txt, C:\A\someprefix@T"mmddyyyy"*.csv - this goes to the semaphore file name job property

and then you can specify that files should have their size stable before 24x7 start running the job.