24x7 Not Executing Scheduled Tasks

We have 24x7 deployed on Windows environments. There are scheduled jobs that run at various frequencies (daily, hourly, every 10 minutes and so on). Some jobs are scripted directly in JAL while some are scheduled by using the wizard. Generally, 24x7 has functioned properly, but there are instances when jobs do not run.

Sometimes we are able to explicitly determine that a job has not run as scheduled. However, there are instances when we have determined that a scheduled job has not run through indirect means such as viewing the 24x7 job log (if the entries have not already been purged) or further investigation.

The 24x7 Windows Service tends to be in a running state, which leads us to the impression that 24x7 is “stuck” somewhere.

How can we troubleshoot or debug 24x7 when we observe a scheduled job has not run?

Are there ways we could monitor 24x7 job activity (preferably outside the system where 24x7 is hosted) to explicitly determine that a job has not run at a scheduled time?

Could this be achieved from within the 24x7 framework and architecture?

Last edited by rockenfeller on Fri Oct 24, 2008 2:47 pm; edited 1 time in total

While it should be possible to extrapolate missing runs by comparing schedules and logs, I don't think that would be the best option, you wouldn't know if the job didn't run at all, or ran late for some reason( perhaps the associated job queue was busy running other job).

From my point of view the best option would be to figure out and fix the cause of the "missing" job runs. I'm pretty sure this situation is not like a job is running, running, then suddenly skipping 1 or 2 runs and then running again. From your issue description, it is most likely that job queues are not used properly. Please check whether all jobs are set to run in the same default queue [default] or they are distributed across multiple queues. Also check all jobs especially database and script jobs are set to run detached. You want that to protect the scheduling engine from job anomalies and resource leaks. And lastly, it is a good idea to enable job tracing option (Tools/Options menu; Log tab, Trace Enabled option) and job performance statistics (same place) while you are troubleshooting the issue. This will provide you with a trace file of each script job run and also make 24x7 to write performance files for each job run. After a little while you can use the Queue Utilization report in the Reports menu to see whether the queues are configured properly and jobs don't compete for the same queue and don't wait in queues for long times.