file processing

Hi All,

I have files that get copied to a director through out the day. files are unique in names and not static. Some times it could be one file and other times multiple files that get copied to this directory.

I need to set up an automated process that will execute a script I have to deal with these files. The job will need to check directory for files and then execute script files exist. I need this job to make sure files are done writing to directory before the script runs. I am not sure how to do this if the files are not static in name and I never know how many files to expect and any give time. I would think file watcher would be out of the question. Would monitor director changes be the was to go? Concern it running script before a file or multiple files finish writing to directory. I would all need this job to run all day an execute as it detects file(s) have hit directory. Can anyone give me some suggestions or examples for set up such a process? I would have to set up about 20 of these jobs to monitor 20 different directories and execute 20 different scripts found in directory I am monitoring.

Thanks in advance

Hi,

You can setup a file watch job (whenever files exists schedule) for the target directory using file mask. You can then enable file size check option. This option will make the scheduler to check file sizes don't change before launching the job. I think that should do what you want.

In my example lets say at 1 am I get three files and than 3 am I only get one. How will the size checker know to make sure all 3 files have finished writing to folder before the job kicks off? Thanks for reply!!!

How do you know when you have the complete file set? Do you always wait for 3 files, or for a file with a certain name pattern or type? Please provide formal description of the business rule.

I never know how many files can get sent to directory first instance could be 3 files, second instance 5 files, third instance 1 file.

I never now and files names are not static. other than extentions *HLD or *TRN

Thanks

MM

Then how do you know when the job is supposed to run?

Exactly,

This is what I am trying to figure out how to have a job execute a script when a file or multiple files hits a director.

Let's start from the beginning and forget about file names or types. Are you trying to figure out when file write operations complete and files are ready, or when all *required* files exist in the target directory?

If it is the first case, you can set the scheduler to monitor for file size consistency, this is a built-in feature, for while watch jobs, watch or files then monitor their size is stable, before running the job.

If it is the second case, you will likely need to figure out first which files are required for the processing, in other words, what represents a complete file set. You can then write some script to check for content or other file attributes indicating the complete set is available. For example, many data processing programs write some sort of manifest file or transaction control file after they are done with the data files. The presence of such control file indicates readiness of the complete data/file set.