process hasn't exited

We have two types of jobs that have recently stated receiving:

process hasn't exited

and the job fails.

Please advise.

debug log:

2013-02-04 14:40:13,271 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.runner.RemoteJobRunner - runJob
com.softtreetech.jscheduler.common.SchedException: process hasn't exited
at com.softtreetech.jscheduler.business.agent.remote.RemoteAgentImpl.executeJob(Unknown Source)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
at com.softtreetech.jscheduler.business.agent.remote.RemoteAgentImpl_Stub.executeJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.RemoteJobRunner.runJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.runJobIgnoringErrorsIfNeeded(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
2013-02-04 14:40:13,272 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.log.SysLogAdaptor - Syslog: invoked
2013-02-04 14:40:13,468 [Job #1144 - TOL_1 PMT Load Driver Northside] ERROR com.softtreetech.jscheduler.business.log.LoggerImpl - Error generating HTML report files
java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
at com.softtreetech.jscheduler.business.log.HtmlLogger.copyFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportsFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportFiles(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.generateReports(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
2013-02-04 14:40:13,469 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.db.JobDbImpl - updateJob(JobProperties) : start
2013-02-04 14:40:13,470 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.db.drivers.file.FileJobDbStorage - update(...) : start
2013-02-04 14:40:13,470 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.db.drivers.file.FileJobDbStorage - update(...) : end
2013-02-04 14:40:13,470 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.db.JobDbImpl - updateJob(JobProperties) : end
2013-02-04 14:40:13,471 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.log.SysLogAdaptor - Syslog: invoked
2013-02-04 14:40:13,661 [Job #1144 - TOL_1 PMT Load Driver Northside] ERROR com.softtreetech.jscheduler.business.log.LoggerImpl - Error generating HTML report files
java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
at com.softtreetech.jscheduler.business.log.HtmlLogger.copyFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportsFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportFiles(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.generateReports(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.db.JobDbImpl.disableJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.disableJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.processFailedJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.onFails(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
2013-02-04 14:40:13,662 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - mail.smtp.auth=true
2013-02-04 14:40:13,664 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): Date: Mon, 4 Feb 2013 14:40:13 -0500 (EST)
2013-02-04 14:40:13,664 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): From: job.scheduler@ipayx.com
2013-02-04 14:40:13,665 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): To: alerts@ipayx.com, operations@ipayx.com, 5673953932@vtext.com
2013-02-04 14:40:13,665 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): Subject: Message from 24x7 Scheduler - job "TOL_1 PMT Load Driver Northside"
event
2013-02-04 14:40:13,665 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): X-Mailer: 24x7 Scheduler
2013-02-04 14:40:13,665 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): account & password information is specified
2013-02-04 14:40:13,937 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl - mail sent
2013-02-04 14:40:13,937 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.log.SysLogAdaptor - Syslog: invoked
2013-02-04 14:40:14,135 [Job #1144 - TOL_1 PMT Load Driver Northside] ERROR com.softtreetech.jscheduler.business.log.LoggerImpl - Error generating HTML report files
java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
at com.softtreetech.jscheduler.business.log.HtmlLogger.copyFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportsFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportFiles(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.generateReports(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.NotificationProcessorImpl.processOnFailEvent(Unknown Source)
at com.softtreetech.jscheduler.business.NotificationProcessorImpl.onFails(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.onFails(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
2013-02-04 14:40:14,136 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.queue.JobQueue - QUEUE_UNLOCKED
2013-02-04 14:40:14,329 [Job #1144 - TOL_1 PMT Load Driver Northside] ERROR com.softtreetech.jscheduler.business.runner.JobExecutorImpl - Job errors: Remote job failed. process hasn't exited
2013-02-04 14:40:14,474 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.business.event.JobSchedulerImpl - Number of pending jobs: 163
2013-02-04 14:40:17,610 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - show() : start
2013-02-04 14:40:17,862 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - show() : end
2013-02-04 14:40:54,633 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - onFinish() : start
2013-02-04 14:40:54,633 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - [Thu Jan 01 07:15:00 EST 1970]
2013-02-04 14:40:54,634 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - [Thu Jan 01 07:15:00 EST 1970, Thu Jan 01 11:05:00 EST 1970]
2013-02-04 14:40:54,634 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - [Thu Jan 01 07:15:00 EST 1970, Thu Jan 01 11:05:00 EST 1970, Thu Jan 01 14:05:00 EST 1970]
2013-02-04 14:40:54,634 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - [Thu Jan 01 07:15:00 EST 1970, Thu Jan 01 11:05:00 EST 1970, Thu Jan 01 14:05:00 EST 1970, Thu Jan 01 21:59:00 EST 1970]
2013-02-04 14:40:54,647 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - onFinish() : end
2013-02-04 14:40:54,647 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.db.JobDbImpl - updateJob(JobProperties) : start
2013-02-04 14:40:54,647 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.db.drivers.file.FileJobDbStorage - update(...) : start
2013-02-04 14:40:54,648 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.db.drivers.file.FileJobDbStorage - update(...) : end
2013-02-04 14:40:54,649 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.db.JobDbImpl - updateJob(JobProperties) : end

scheduler log:

4-Feb-2013 02:40:13 PM 3 5ou5AYjUgNmragQXMn/7WhhcR5A= 1144 TOL_1 PMT Load Driver Northside Remote job failed. process hasn't exited
4-Feb-2013 02:40:13 PM 1 5ou5AYjUgNmragQXMn/7WhhcR5A= 1144 TOL_1 PMT Load Driver Northside Job has been disabled!
4-Feb-2013 02:40:13 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 1144 TOL_1 PMT Load Driver Northside Error message sent.

Please take a look what you get in logs on the agent side of the processing? Are there any other clues?

I did not find anything around that time frame in the agent scheduler.log.

4-Feb-2013 11:47:04 PM 2 null 433 TOL_1 Multi PMT Process Job completed with exit code 0, exit code condition satisfied.
4-Feb-2013 11:47:04 PM 2 null 433 TOL_1 Multi PMT Process Remote job finished.
5-Feb-2013 12:02:01 AM 2 null 700 TOL_1 Update Office Cycles CGI Remote job started.
5-Feb-2013 12:02:01 AM 2 null 700 TOL_1 Update Office Cycles CGI Job started.

Are you sure you have checked the correct agent?
In your log fragment I don't see any references to "TOL_1 PMT Load Driver Northside"
Nor I see any activities around 2013-02-04 14:40:13. It seems like you are looking in a wrong log file.

What kind of command line are you starting from the job?

Lastly, your scheduler settings are mis configured. You enabled HTML logs feature and as a destination for HTML files specified invalid directory. As a result, every time you run a job, that leads to an exception "Error generating HTML report files. java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory"

Yes, I was on the correct agent, notice the references to other the TOL1 jobs, however, I did not proof my message, sorry about that I messed up, I was in the wrong part of the day

4-Feb-2013 02:20:05 PM 2 null 978 Tol_1 ReportTable Extract Remote job finished.
4-Feb-2013 02:30:08 PM 2 null 1144 TOL_1 PMT Load Driver Northside Remote job started.
4-Feb-2013 02:30:08 PM 2 null 1144 TOL_1 PMT Load Driver Northside Job started.
4-Feb-2013 02:37:01 PM 2 null 398 Tol_1 Monitor Files Remote job started.
4-Feb-2013 02:37:01 PM 2 null 398 Tol_1 Monitor Files Job started.

here is the command starting the job:
export HOSTNAME=earchive.ipayx.net;/home/einvoice/cgi-php/ei_bat_pmtload_driver.php 15

the file being processed was 445mb

will change the configuration

I suggest adding in front of the command csh-c

I added csh-c in front of the word export. csh-c was the first thing in the command line. The job ran but did not do anything.

csh-c export HOSTNAME=earchive.ipayx.net;/home/einvoice/cgi-php/ei_bat_pmtload_driver.php 15

I have restarted the tol1 agent as jobs started receiving communications errors and stated the agent was not found.

I removed the csh-c as, resubmitted the job. The job ran normally and processed the 145 mb file.

Almost appears related to the Agent causing this trouble.

since restarting the agent, we have not received any "exited" errors on this job.

Thank you for the status update.

Good to know that the issue is resolved for now, although we still don't know what it occurred in the first place.