Jobs hung in queue

I have jobs that hang in the queue after they have finished running. Is there a way to clean these jobs out? Besides restarting the service/program? I have the jobs running detached/synchronous and each has there own queue (only 8 jobs). They finish for the day and then sit in the queue holding up the same job for the next day.

Thanks for any insight.

Do you use any "on job finish" notification actions for these jobs. If yes, what do you do?

I use a on error notification, send email via smtp. There are email notices within the script. Those get processed without any problems. I can run these in debug mode and they process without error.

Included is a script that hangs. If you see something out of the ordinary let me know.

//*********** Script Start**************//
OnErrorResumeNext

Dim vProcess, number
Dim vFileId, number
Dim vFile, String
Dim vFileDate, String
Dim vDate, date
Dim vTime, time
Dim vDateTime, datetime
Dim vPath, string
Dim vAppend, boolean
Dim vLogNote, string
Dim vNot_Found, boolean
Dim vLAERProc, number
Dim vEXCELProc, number
Dim vProcTF, boolean

//set datetime for log file string
Now vTime
Today vDate
DateTime( vDate, vTime, vDateTime )
Format( vDate, "mmddyyyy", vFileDate )

//Create string for logfile access
ConcatEX( "\\\server_name\\24x7\\logs\\emarg\\emargjob", vFileDate, ".log", vFile )

// Look for running processes from LAER.exe
STEP1:
IsTaskRunning( "LAER.exe", vProcTF )
IfThen( vProcTF, FIND_LAER )

IsTaskRunning( "EXCEL.EXE", vProcTF )
IfThen(vProcTF, FIND_EXCEL )

//Append No Process found, continue with TCN.exe
FileOpen( vFile, "LineMode", "write", "TRUE", vFileId )
ConcatEx( vdatetime, " LAER.exe & Excel.exe process was not found, starting TCN.exe" , vLogNote )
FileWrite( vFileId, vLogNote )
FileClose( vFileId )

SUB_MAIN:
//Create log file for emarg jobs, and note start of TCN.exe
FileOpen( vFile, "LineMode", "write", "TRUE", vFileId )
ConcatEx( vdatetime, " starting TCN.exe on EMARG-EC-1" , vLogNote )
FileWrite( vFileId, vLogNote )
FileClose( vFileId )

//Start TCN.exe
Run( "C:\\TCN\\TCN.exe", "C:\\TCN\\", vprocess )

//Set time & date for job finish, create good flag file
Now vTime
Today vDate
DateTime( vDate, vTime, vDateTime )

//Append to log file Date, time and actions from program.
FileOpen( vFile, "LineMode", "write", "TRUE", vFileId )
ConcatEx( vdatetime, " TCN.exe on EMARG-EC-1 has ended without error" , vLogNote )
FileWrite( vFileId, vLogNote )
FileClose( vFileId )

//FileOpen( "\\\server_name\\24x7\\logs\\TCNgood.flg", "LineMode", "write", "TRUE", vFileId )
//FileWrite( vFileId, "0" )
//FileClose( vFileId )

ConCatex(vLogNote, ", please check log.", vLogNote )

MailSend "email@emailadress.com", "", &
"email-address", &
"INFORMATION <EMARG> **TCN.exe**", &
vLogNote
Exit

Goto END_SUB

// FIND AND EXTERMINATE PROCESSES
FIND_LAER:
IsTaskRunning( "LAER.exe", vProcTF )
IfThen( vProcTF, KILL_LAER)

Goto STEP1

FIND_EXCEL:
IsTaskRunning( "EXCEL.EXE", vProcTF )
IfThen( vProcTF, KILL_EXCEL)

Goto STEP1

// EXTERMINATE LAER
KILL_LAER:
ProcessGetID( "LAER.exe", vLAERProc )
ProcessKill( vLAERProc )

//set datetime for log file string
Now vTime
Today vDate
DateTime( vDate, vTime, vDateTime )
Set vAppend = true

//Append action of process kill for LAER.exe to Log File, create Bad flg file

FileOpen( vFile, "LineMode", "write", "TRUE", vFileId )
ConcatEx( vdatetime, " LAER.exe Process was terminated, starting TCN.exe" , vLogNote )
FileWrite( vFileId, vLogNote )
FileClose( vFileId )

//Create Bad flg file
//FileOpen( "\\\server_name\\24x7\\logs\\LISAUtilBAD.flg", "LineMode", "write", "TRUE", vFileId )
//FileWrite( vFileId, "1" )
//FileClose( vFileId )

MailSend "email@emailaddress.com", "", &
"email-address", &
"RED-ALERT <EMARG> **LAER.exe**", &
"THE PROCESS HAS BEEN TERMINATED AFTER ITS ALOTTED RUN TIME. Please contact system admin."

Goto STEP1

KILL_EXCEL:
//EXTERMINATE EXCEL
IsTaskRunning( "EXCEL.EXE", vProcTF )

LoopWhile( vProcTF, LOGUPDATE )
ProcessGetID( "EXCEL.EXE", vEXCELProc )
ProcessKill( vEXCELProc )
IsTaskRunning( "EXCEL.EXE", vProcTF )

LOGUPDATE:
//set datetime for log file string
Now vTime
Today vDate
DateTime( vDate, vTime, vDateTime )
Set vAppend = true

//Append action of process kill for SFTPUtil.exe to Log File, create Bad flg file

FileOpen( vFile, "LineMode", "write", "TRUE", vFileId )
ConcatEx( vdatetime, " EXCEL.exe Process was terminated." , vLogNote )
FileWrite( vFileId, vLogNote )
FileClose( vFileId )

//Send alert email to job master
MailSend "email@emailaddress.com", "", &
"email-address", &
"RED-ALERT <EMARG> **EXCEL.EXE**", &
"THE PROCESS HAS BEEN TERMINATED AFTER ITS ALOTTED RUN TIME FOLLOWING LAER.EXE. Please contact system admin."

Goto STEP1

END_SUB:
Exit

Last edited by DLUTER on Mon Jun 18, 2007 4:38 pm; edited 1 time in total

I would like to suggest the following

1. Get Process Explorer from Microsoft/Sysinternals http://www.microsoft.com/technet/sysinternals/utilities/ProcessExplorer.mspx
installed on the system

2. Next time when the job hangs, run this program and try to in the task list process with name 24x7.exe whose command line contains /JOB parameter following by the number for the hung job.

3. If you find such process, please check what files, handles and resources are being used by that process. Please let us know what you get there. The purpose of doing this check is to find which resources are not being released and/or files not closed that cause the job process to hang.

Thanks, these jobs run at night, so i'll have something tomorrow morning for you. (PST)

I've included screen shots of the queue and process explorer. I don't see that 24x7 is hung on any files or anything, if you need something else, trace log or something let me know i'll turn that on and let'er rip for a bit.

So the process with id 29640 is not in the task list. Right?

Didn't see that, I believe it was tied into the service (srvany.exe) that I setup to run it as a service. It was held on wscript.exe. I didn't have that in the screen shot, my apologies. I've since cleared the queue and the jobs have finished. I'll double check this over the next few days and post what I find. as a side note, i've split thos 3 jobs up to use different queues also.

Thanks for your help.
David

If you have a separate test environment, I suggest to try the following.

Set the script job to run every minute all day and monitoring it using process explorer. If you click on 24x7.exe process and then use View / Show Lower Pane menu you can see the resources in use by the process. Check if you see any differences before the job runs, when it runs and after and especially after it hangs. There must be some resource such as file, handle, thread, etc... that makes the job to hang and in turn makes the queue to wait for job completion forever

Well I did that on the test & prod box and it my job never hung. I'll be checking this for the next few days, hopefully i'll have something by monday. Thanks for the help.