Job completed with exit code 255 ?

24x7 MP 4.4b317
JVM 1.5.0_22
Red Hat Linux 5.5

I have been experiencing a number of 24x7 Jobs failing with the above error message. The same job will work most of the time, but occasionally fail with the exit code 255. There does not seem to be any pattern as to which job will fail and when.
A sample from the debug.log file for a job #455 is below;

2010-10-06 10:00:00,152 [Job #455 - VIVALDI_REFDATA_SEND] DEBUG com.softtreetech.jscheduler.business.log.SysLogAdaptor - Syslog: invoked
2010-10-06 10:00:00,219 [Job #455 - VIVALDI_REFDATA_SEND] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - runJob (JobPropertiesWorkFile):
2010-10-06 10:00:00,220 [Job #455 - VIVALDI_REFDATA_SEND] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - runJob(): jid=185704; workfile created
2010-10-06 10:00:00,220 [Job #455 - VIVALDI_REFDATA_SEND] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - runJob(): start
2010-10-06 10:00:00,220 [Job #455 - VIVALDI_REFDATA_SEND] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - execProcess(): command line [/usr/java/jre1.5.0_22/bin/java -jar jscheduler.jar /JOBFILE 185704]; work directory [/usr/local/24x7_Scheduler/]
2010-10-06 10:00:00,233 [Job #455 - VIVALDI_REFDATA_SEND] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - waitForProcess(): start
2010-10-06 10:00:00,748 [Job #455 - VIVALDI_REFDATA_SEND] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - waitForProcess(): end
2010-10-06 10:00:00,748 [Job #455 - VIVALDI_REFDATA_SEND] DEBUG com.softtreetech.jscheduler.business.runner.AbstractJobRunner - isFailed(...) : exit code 255
2010-10-06 10:00:00,748 [Job #455 - VIVALDI_REFDATA_SEND] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - killProcess start
2010-10-06 10:00:00,748 [Job #455 - VIVALDI_REFDATA_SEND] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - stopExecutionImpl(): jid=185704; workfile deleted
2010-10-06 10:00:00,748 [Job #455 - VIVALDI_REFDATA_SEND] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - runJob(): exit code: 255
2010-10-06 10:00:00,748 [Job #455 - VIVALDI_REFDATA_SEND] DEBUG com.softtreetech.jscheduler.business.runner.DetachedJobRunner - runJob(): failed
2010-10-06 10:00:00,749 [Job #455 - VIVALDI_REFDATA_SEND] DEBUG com.softtreetech.jscheduler.business.log.SysLogAdaptor - Syslog: invoked
2010-10-06 10:00:00,809 [Job #455 - VIVALDI_REFDATA_SEND] DEBUG com.softtreetech.jscheduler.business.queue.JobQueue - QUEUE_UNLOCKED
2010-10-06 10:00:00,880 [Job #455 - VIVALDI_REFDATA_SEND] ERROR com.softtreetech.jscheduler.business.runner.JobExecutorImpl - Job errors: Job completed with exit code 255. This exit code does not satisfy job exit code condition. Job failed.
2010-10-06 10:00:06,931 [Thread-445] INFO com.softtreetech.jscheduler.business.runner.AbstractJobRunner$TimeoutVerifier - run():
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner$TimeoutVerifier.run(Unknown Source)
2010-10-06 10:00:06,931 [Thread-445] DEBUG com.softtreetech.jscheduler.business.runner.AbstractJobRunner$TimeoutVerifier - run(): ended

Please tell us a bit more about the job setup. What kind of a job is that?

I have cut/pasted the details from the Job's properties window (as seen in web console) below.
NOTE: Some fields [in particular the semaphore file field] contain data that is not realated to the job. When in "edit" mode, the correct details are shown, but when in "properties" mode you seem to have some filed filled with details of other jobs (might be a cached field ?)

General
Job No 455
Job Name VIVALDI_REFDATA_SEND
Status Enabled
Description Send Cardclear Ref Data to Accelya (ES) for Vivaldi.
Job Definition
Type Script
Script //
// Job Specific values to be defined here.
//
var filepath = "/scheduler/scripts";
var filename = "vivaldi_ftp.ksh";
//
// Calculate the correct value for the logfiles suffix from the current date/time
//
now = new Date();
logsuffix = ""
var suffdate = '@T"yyyymmdd"';
var suffhour = now.getHours();
var suffmin = now.getMinutes();
var suffsec = now.getSeconds();
if (suffhour <= 9) {suffhour = "0" + suffhour};
if (suffmin <= 9) {suffmin = "0" + suffmin};
if (suffsec <= 9) {suffsec = "0" + suffsec};
logsuffix = logsuffix.concat(suffdate,suffhour,suffmin,suffsec);
//
// Build the string to be passed to the runAndWait command.
//
var emailtarget = "emails here";
var jobname = '@V"job_name"';
var jobnum = '@V"job_id"';
var logdir = '@T"yyyymmdd"';
var logfilename = jobname + "." + logsuffix + ".txt";
var logfile = "/scheduler/logs/" + logdir + "/" + logfilename;
var controlscript = "/scheduler/scripts/24x7Jobman";
//
// Now issue the runAndWait command
//
Scheduler.logAddMessage('INFO',jobnum,jobname,'Running Job = ' + jobname + ', output to ' + logfile);
cmdstring = controlscript + " " + filepath + "/" + filename + " >> " + logfile;
timeout = 0;
var rwi = Process.runAndWait(cmdstring,0);
//
// Now Save the output from the previous command to logfile.
//
var strOutput = rwi.getOutput();
if ((logfile!=null) && (logfile!="")) {
File.save(logfile,strOutput); // Save the output
}
//
// Do we need to email the Job Output ?
//
if ((emailtarget!=null) && (emailtarget!="")) {
var logfilefound = File.exists(logfile);
if (logfilefound) {
emailsubject = "24x7 : " + logfilename;
emailmessage = "Output from 24X7 Job : " + jobname;
emailattach = logfile;
Mail.sendWithAttachment("email", "password", emailtarget, emailsubject, emailmessage, emailattach);
}
}
Script Type JS
Host [local]
Asynchronous No
Detached Yes
Queue CARDCLEAR_OUTPUTS
Priority Normal
Timeout (minutes) 0
Logging Enabled Yes
Run As
Schedule and Triggers
Schedule Type Run all day; Repeat every 01:00 hours:minutes
Start Time 09/08/2010
End Time 31/12/2019
Semaphore file /data/outbound/ba/out/pgp.bacsp_????????_????
Semaphore file rule Delete before execution
Email Account
Email subject & message
Polling Interval (minutes)
Skip Late Job No
Skip Holidays No
Slide Holidays No
Job Calendar [default]
Notification Events
Start No
Finish No
Not Found No
Execution Error No
Late Job No
Notification Actions and Properties
Send Email No
Account
Recipient
Execute SQL Command No
Database profile
SQL Command(s)
Create Semaphore Files No
Semaphore File(s) @V"event"0004900050.txt
Run External Program No
Command line
Timeout (minutes)
Run another job No
Job ID
Error Handling
Max. Delay (minutes) 10
Exit Code Condition [any]
Disable on Error No
Ignore Errors No
Retry after Error No
Number of Retries 1
Retry Interval (seconds) 01:00

For security reasons and to prevent web spam bots from picking your emails, I removed them from the posted script.

I cannot tell what exit code 255 means, that could be a reaction to some process signal, and exit code fro mexternal process/function passed through… But since this is a JavaScript job and not an external process it should be fairly easy to troubleshoot the cause of 255. Please set the job to run non-detached, I mean if the job detached property is set, unset it. Enable the scheduler tracing option in Tools/Options/Log if it is not already enabled. Wait for the job to fail again. After the failure review debug.log file in the scheduler directory. There should be some error or exception message pointing to a specific issue. Please note that the debug log is rotating. Don't wait too long after the error as it can be overwritten by trace data from other jobs and activities.

Hope this helps.

I have done as requested (non-detached), and set the trace mode on.

I have detected a similar failure (not error code 255 - but it might be) and have e-mailed the debug.log file to you.

I think the debug log indicates some process concurrency issues related to preferences.xml file reading/writing operations. Somehow the file is not being locked during updates and your email sending jobs are attempting to read email settings from this configuration file while it is still being written. That normally shouldn't happen. While it is possible to check for file state before sending email messages, it might be just easier to tweak some job properties to prevent the entire issue. Try playing with synchronous/asynchronous properties and detached/non-detached. I think the right combination of these 2 settings should fix this issue. You may need to do it for multiple jobs running concurrently, not just for the one that is failing.