Dummy JavaScript job occasionally fails

On 24x7 Multi-platform 4.1 255 I have a number of tiny 'start' and 'finish' jobs to delineate blocks of processing.

This particular one only watches for a semaphore file and runs when it's found (I can later look through the log for the start and finish jobs to see when that set of jobs began and ended).

Seeing as I don't want the job to actually run anything I've made it a JavaScript job which executes:

	Code:
	// dummy job

Job 242 usually runs fine every hour, but every few days it's been exiting with code -1.

What's going on?

The log shows:

Code:

16-May-2008 02:11:25 AM 2 O2MfZQ+yPxYL/FuwWawsTALECuI= 242 Warehouse Hourly Finish Job started.
16-May-2008 02:11:27 AM 2 O2MfZQ+yPxYL/FuwWawsTALECuI= 258 Warehouse Mon-Sat Fri Choose Path Running on day of week 5 (0=Sun ... 6=Sat)
16-May-2008 02:11:27 AM 3 O2MfZQ+yPxYL/FuwWawsTALECuI= 242 Warehouse Hourly Finish Job completed with exit code -1. This exit code does not satisfy job exit code condition. Job failed.
16-May-2008 02:11:27 AM 2 O2MfZQ+yPxYL/FuwWawsTALECuI= 258 Warehouse Mon-Sat Fri Choose Path Creating Fri trigger
16-May-2008 02:11:27 AM 1 O2MfZQ+yPxYL/FuwWawsTALECuI= 242 Warehouse Hourly Finish Job has been disabled!
16-May-2008 02:11:28 AM 2 O2MfZQ+yPxYL/FuwWawsTALECuI= 258 Warehouse Mon-Sat Fri Choose Path Job finished.
16-May-2008 02:11:28 AM 2 O2MfZQ+yPxYL/FuwWawsTALECuI= 193 Warehouse Mon-Sat Reports HealthStat_Panel Job started.
16-May-2008 02:11:29 AM 2 O2MfZQ+yPxYL/FuwWawsTALECuI= 242 Warehouse Hourly Finish Error message sent.

The debug log reports:

Code:

2008-05-16 02:11:25,796 [Thread-437883] DEBUG com.softtreetech.jscheduler.business.runner.AbstractJobRunner$TimeoutVerifier - run(): start
2008-05-16 02:11:25,796 [Thread-437883] DEBUG com.softtreetech.jscheduler.business.runner.AbstractJobRunner$TimeoutVerifier - run(): timeout check not required
2008-05-16 02:11:26,218 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - runJob (JobPropertiesWorkFile):
2008-05-16 02:11:26,218 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - runJob(): jid=10621; workfile created
2008-05-16 02:11:26,218 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - runJob(): start
2008-05-16 02:11:26,218 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - execProcess(): command line [C:\Program Files\Java\j2re1.4.2_16\bin\java.exe -jar jscheduler.jar /JOBFILE 10621]; work directory [C:\Program Files\24x7_Scheduler\]
2008-05-16 02:11:26,218 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - waitForProcess(): start
2008-05-16 02:11:26,218 [Thread-437885] DEBUG com.softtreetech.jscheduler.business.runner.AbstractJobRunner$TimeoutVerifier - run(): start
2008-05-16 02:11:26,218 [Thread-437885] DEBUG com.softtreetech.jscheduler.business.runner.AbstractJobRunner$TimeoutVerifier - run(): timeout check not required
2008-05-16 02:11:27,203 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - waitForProcess(): end
2008-05-16 02:11:27,203 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.runner.AbstractJobRunner - isFailed(...) : exit code -1
2008-05-16 02:11:27,203 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - killProcess start
2008-05-16 02:11:27,203 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - stopExecutionImpl(): jid=10621; workfile deleted
2008-05-16 02:11:27,203 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - runJob(): exit code: -1
2008-05-16 02:11:27,203 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - runJob(): failed
2008-05-16 02:11:27,218 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.log.LoggerImpl - found asynchronous log message
2008-05-16 02:11:27,218 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.log.LoggerImpl - asynchronous log message inserted
2008-05-16 02:11:27,640 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.A.if - disableJob(...) : Job with id [242] is removed.
2008-05-16 02:11:27,656 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.db.JobDbImpl - updateJob(JobProperties) : start
2008-05-16 02:11:27,656 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.db.drivers.file.FileJobDbStorage - update(...) : start
2008-05-16 02:11:27,656 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.db.drivers.file.FileJobDbStorage - update(...) : end
2008-05-16 02:11:27,656 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.db.JobDbImpl - updateJob(JobProperties) : end
2008-05-16 02:11:27,656 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.log.LoggerImpl - found asynchronous log message
2008-05-16 02:11:27,656 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.log.LoggerImpl - asynchronous log message inserted
2008-05-16 02:11:27,781 [Job #258 - Warehouse Mon-Sat Fri Choose Path] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - waitForProcess(): end
2008-05-16 02:11:27,781 [Job #258 - Warehouse Mon-Sat Fri Choose Path] DEBUG com.softtreetech.jscheduler.business.runner.AbstractJobRunner - isFailed(...) : exit code 0
2008-05-16 02:11:27,781 [Job #258 - Warehouse Mon-Sat Fri Choose Path] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - killProcess start
2008-05-16 02:11:27,781 [Job #258 - Warehouse Mon-Sat Fri Choose Path] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - stopExecutionImpl(): jid=10619; workfile deleted
2008-05-16 02:11:27,781 [Job #258 - Warehouse Mon-Sat Fri Choose Path] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - runJob(): exit code: 0
2008-05-16 02:11:27,781 [Job #258 - Warehouse Mon-Sat Fri Choose Path] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - runJob(): end
2008-05-16 02:11:28,078 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.J - mail.smtp.auth=true
2008-05-16 02:11:28,078 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.J - sendEmailImpl(): Date: Fri, 16 May 2008 02:11:28 +1200 (NZST)
2008-05-16 02:11:28,078 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.J - sendEmailImpl(): From: healthstat@xxx
2008-05-16 02:11:28,078 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.J - sendEmailImpl(): To: tom@xxx
2008-05-16 02:11:28,078 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.J - sendEmailImpl(): Subject: Message from 24x7 Scheduler
2008-05-16 02:11:28,078 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.J - sendEmailImpl(): X-Mailer: 24x7 Scheduler
2008-05-16 02:11:28,078 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.J - sendEmailImpl(): account & password information is specified
2008-05-16 02:11:28,500 [Job #258 - Warehouse Mon-Sat Fri Choose Path] DEBUG com.softtreetech.jscheduler.business.queue.JobQueue - QUEUE_UNLOCKED
2008-05-16 02:11:28,984 [Job #193 - Warehouse Mon-Sat Reports HealthStat_Panel] DEBUG com.softtreetech.jscheduler.business.runner.ProgramJobRunner - runJob(): start
2008-05-16 02:11:29,468 [Job #193 - Warehouse Mon-Sat Reports HealthStat_Panel] DEBUG com.softtreetech.jscheduler.business.runner.ProgramJobRunner - execProcess(): command line [sasbatch.bat C:\DOCUME~1\ALLUSE~1\DOCUME~1\SASPRO~1\Management\sasprogs\HealthStat_Panel.sas /a] in work directory [null]
2008-05-16 02:11:29,468 [Job #193 - Warehouse Mon-Sat Reports HealthStat_Panel] DEBUG com.softtreetech.jscheduler.business.runner.ProgramJobRunner - waitForProcess(): start
2008-05-16 02:11:29,468 [Thread-437890] DEBUG com.softtreetech.jscheduler.business.runner.AbstractJobRunner$TimeoutVerifier - run(): start
2008-05-16 02:11:29,468 [Thread-437890] DEBUG com.softtreetech.jscheduler.business.runner.AbstractJobRunner$TimeoutVerifier - run(): timeout check not required
2008-05-16 02:11:30,312 [Job #242 - Warehouse Hourly Finish] DEBUG com.softtreetech.jscheduler.business.queue.JobQueue - QUEUE_UNLOCKED
2008-05-16 02:11:30,375 [Job #242 - Warehouse Hourly Finish] ERROR com.softtreetech.jscheduler.business.runner.JobExecutorImpl - Job errors: Job completed with exit code -1. This exit code does not satisfy job exit code condition. Job failed.

Can you check in the log if it always occurs at the time when some other job is writing to the job log?

16-May-2008 02:11:27 AM 2 O2MfZQ+yPxYL/FuwWawsTALECuI= 258 Warehouse Mon-Sat Fri Choose Path Running on day of week 5 (0=Sun ... 6=Sat)
16-May-2008 02:11:27 AM 3 O2MfZQ+yPxYL/FuwWawsTALECuI= 242 Warehouse Hourly Finish Job completed with exit code -1. This exit code does not satisfy job exit code condition. Job failed.

Yes, I'd say you're correct (the jobs are in different queues).

Interesting. I'm not saying that there is a problem with having 2 jobs running at the same time, I ma just suspecting that there could be some resource sharing conflict., in other words 2 jobs are competing for some supposedly shared resource and #242 is falling behind. Is that always #258 job that collides with #242 or different jobs?

The previous time it was a different job:

#25 ran 2008-05-10T02:10:08 / 2008-05-10T02:10:11 RC=0
#242 ran 2008-05-10T02:10:09 / 2008-05-10T02:10:10 RC=-1

I still don't understand what they could have in common or where they could compete. Is this the only job failing? Do you run other JavaScript jobs?

It only happens infrequently, though I will take more note of it in the future. I don't recall other jobs failing. We do run a number of other JavaScript jobs.

Cheers