JAl job gets hung up in queue after it is finished

Having issues with a schedule job (jal), when testing the job (debug) it runs just fine but when it is run normally the job gets hung in the queue when it is finished. I can see that it is finished by: 1) the files have been moved and 2) it is no longer showing up in the task manager. It also cannot be “killed”; the only way to get it out of the queue is to stop the 24x7 service.
Version 5.1.395

CODE: Archive Files By Date

Dim no_more Boolean
Dim complete Boolean
Dim rtn Boolean
Dim fertn Boolean
Dim ctdate String
Dim formdate String
Dim source_files String
Dim full_file_name String
Dim file_list String
Dim new_folder String
Dim file_name String
Dim archivefolder String

// Source Folder include double back slashed at the end of the path
Dim source_folder String "C:\\test\\"

SEARCH:
// Search the source folder for files
ConcatEx source_folder, "*.*", source_files
Dir source_files, file_list
IsEqual file_list, "", no_more
IfThen no_more, COPY

COPY:
// Get file list from folder and get file date
LoopUntil no_more, DONE
GetToken ",", file_list, file_name
IsEqual file_name, "", complete
IfThen complete, DONE
ConcatEx source_folder, file_name, full_file_name
FileDate full_file_name, ctdate
Format(ctdate, "yyyy-mm-dd", formdate)

BACKUPLOGS:
// Check for the backup folder and create it if missing
ConcatEx source_folder, formdate, archivefolder
IsDir archivefolder, rtn
If rtn, DIR_OK, DIR_MAKE

DIR_OK:
GoTo COPYFILE

DIR_MAKE:
DirCreate archivefolder

COPYFILE:
ConcatEx (archivefolder, "\\", file_name, new_folder)
Filemove full_file_name, new_folder

DONE:
Exit

Do you see JAL.EXE process in the task manager after the job completes?

If you set timeout property for the job, is the scheduler able to clear the job from its queue after the specified timeout time?

No the jal.exe process in task manager is no longer there when the job completes.

In order to use the “time out” function the job has to be run in “detached process” which is not our standard. Plus the job will vary depending on the amount of files to be moved. It takes about 25 minutes to move four thousand files.

For testing purposes I did as requested and used a different queue, the job “finished” in 26 minutes and the time out was set for 30 minutes, and after 33 minutes the job was still showings as RUNNING in the job monitor.

Correction:
Version 5.1 Build 405
JVM Version 1.6.0_17-b04

Generally speaking all jobs should run detached there only very few exceptions to that rule.

Back to the issue, please enable tracing in scheduler options, and rerun the job. After job completion review debug.log file and let us know if you see any errors or exceptions there.

Here is what I found in the debug.log

2013-09-27 12:23:23,278 [Job #400 - ArchiveFilesByDate] DEBUG com.softtreetech.jscheduler.business.runner.ProgramJobRunner - waitForProcess(): end
2013-09-27 12:23:23,278 [Job #400 - ArchiveFilesByDate] DEBUG com.softtreetech.jscheduler.business.runner.AbstractJobRunner - isFailed(...) : exit code: 0
2013-09-27 12:23:23,278 [Job #400 - ArchiveFilesByDate] DEBUG com.softtreetech.jscheduler.business.runner.AbstractJobRunner - isFailed(...) : Enumeration found [0]
2013-09-27 12:23:23,278 [Job #400 - ArchiveFilesByDate] DEBUG com.softtreetech.jscheduler.business.runner.AbstractJobRunner - Writing output file E:\24x7\Output\400\400_20130927122323.out
2013-09-27 12:23:23,278 [Job #400 - ArchiveFilesByDate] DEBUG com.softtreetech.jscheduler.business.runner.AbstractJobRunner - isFailed(...) : output:
2013-09-27 12:23:23,278 [Job #400 - ArchiveFilesByDate] DEBUG com.softtreetech.jscheduler.business.runner.ProgramJobRunner - killProcess start
2013-09-27 12:23:23,278 [Job #400 - ArchiveFilesByDate] DEBUG com.softtreetech.jscheduler.business.runner.ProgramJobRunner - runJob(): end
2013-09-27 12:23:23,278 [Thread-38638] INFO com.softtreetech.jscheduler.business.runner.AbstractJobRunner$TimeoutVerifier - run():

2013-09-27 12:23:23,278 [Job #400 - ArchiveFilesByDate] DEBUG com.softtreetech.jscheduler.business.runner.JALScriptJobRunner - Deleting workfile E:\24x7\Queue\48085.tmp
2013-09-27 12:23:23,278 [Job #400 - ArchiveFilesByDate] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptCommand - Start: execFileExists
2013-09-27 12:23:23,278 [Job #400 - ArchiveFilesByDate] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptCommand - File: E:\24x7\Queue\48085.tmp.err
2013-09-27 12:23:23,278 [Job #400 - ArchiveFilesByDate] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptCommand - Start: execFileExists
2013-09-27 12:23:23,278 [Job #400 - ArchiveFilesByDate] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptCommand - File: E:\24x7\Queue\48085.tmp.out

No smoking gun as fur as I can tell.

Please try one more thing for me, disable output capturing option in the job, rerun it and see if that helps.

Disabled the capture, rebooted the server and let the job run as scheduled, still hangs up in the queue.

Well, we know now it's not a logging conflict, the job process must be leaving some resources open.
Can you please add to the end of your JAL job scrip the following commands instead of Exit

Dim pid
JobProcessID pid
ProcessKill pid

The first run ended with an error: An error occurred while executing automation script Line 52: Invalid datatype.

The Dim statement needs a Datatype, I used datatype “Number”, same results; 1) the files have been moved and 2) it is no longer showing up in the task manager and the JOB shows as RUNNING in the Queue.

Sorry I forgot to add the data type.

Does the job have any notifications and/or other on finish actions

Yes,
Send e-mail message:
Job Start
Job Finish
Job Error

I get the e-mail on the start of the job and on an error but not on the finish.

Let's temporarily disable them to ensure that is not a factor.

Disabled the all the e-mail notifications – same results.
Created a new job – copied the JAL from notepad – same results.

Just to be certain, other JAL jobs work fine, this is the only JAL job having difficulties, correct?