![SoftTree Technologies](templates/GreenTech/images/logo.gif) |
SoftTree Technologies
Technical Support Forums
|
|
Author |
Message |
owensjim
Joined: 08 Aug 2011 Posts: 49
|
|
process hasn't exited |
|
We have two types of jobs that have recently stated receiving:
process hasn't exited
and the job fails.
Please advise.
debug log:
2013-02-04 14:40:13,271 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.runner.RemoteJobRunner - runJob
com.softtreetech.jscheduler.common.SchedException: process hasn't exited
at com.softtreetech.jscheduler.business.agent.remote.RemoteAgentImpl.executeJob(Unknown Source)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
at com.softtreetech.jscheduler.business.agent.remote.RemoteAgentImpl_Stub.executeJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.RemoteJobRunner.runJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.runJobIgnoringErrorsIfNeeded(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
2013-02-04 14:40:13,272 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.log.SysLogAdaptor - Syslog: invoked
2013-02-04 14:40:13,468 [Job #1144 - TOL_1 PMT Load Driver Northside] ERROR com.softtreetech.jscheduler.business.log.LoggerImpl - Error generating HTML report files
java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
at com.softtreetech.jscheduler.business.log.HtmlLogger.copyFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportsFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportFiles(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.generateReports(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
2013-02-04 14:40:13,469 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.db.JobDbImpl - updateJob(JobProperties) : start
2013-02-04 14:40:13,470 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.db.drivers.file.FileJobDbStorage - update(...) : start
2013-02-04 14:40:13,470 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.db.drivers.file.FileJobDbStorage - update(...) : end
2013-02-04 14:40:13,470 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.db.JobDbImpl - updateJob(JobProperties) : end
2013-02-04 14:40:13,471 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.log.SysLogAdaptor - Syslog: invoked
2013-02-04 14:40:13,661 [Job #1144 - TOL_1 PMT Load Driver Northside] ERROR com.softtreetech.jscheduler.business.log.LoggerImpl - Error generating HTML report files
java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
at com.softtreetech.jscheduler.business.log.HtmlLogger.copyFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportsFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportFiles(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.generateReports(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.db.JobDbImpl.disableJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.disableJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.processFailedJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.onFails(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
2013-02-04 14:40:13,662 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - mail.smtp.auth=true
2013-02-04 14:40:13,664 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): Date: Mon, 4 Feb 2013 14:40:13 -0500 (EST)
2013-02-04 14:40:13,664 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): From: job.scheduler@ipayx.com
2013-02-04 14:40:13,665 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): To: alerts@ipayx.com, operations@ipayx.com, 5673953932@vtext.com
2013-02-04 14:40:13,665 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): Subject: Message from 24x7 Scheduler - job "TOL_1 PMT Load Driver Northside"
event
2013-02-04 14:40:13,665 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): X-Mailer: 24x7 Scheduler
2013-02-04 14:40:13,665 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): account & password information is specified
2013-02-04 14:40:13,937 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl - mail sent
2013-02-04 14:40:13,937 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.log.SysLogAdaptor - Syslog: invoked
2013-02-04 14:40:14,135 [Job #1144 - TOL_1 PMT Load Driver Northside] ERROR com.softtreetech.jscheduler.business.log.LoggerImpl - Error generating HTML report files
java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
at com.softtreetech.jscheduler.business.log.HtmlLogger.copyFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportsFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportFiles(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.generateReports(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.NotificationProcessorImpl.processOnFailEvent(Unknown Source)
at com.softtreetech.jscheduler.business.NotificationProcessorImpl.onFails(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.onFails(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
2013-02-04 14:40:14,136 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.queue.JobQueue - QUEUE_UNLOCKED
2013-02-04 14:40:14,329 [Job #1144 - TOL_1 PMT Load Driver Northside] ERROR com.softtreetech.jscheduler.business.runner.JobExecutorImpl - Job errors: Remote job failed. process hasn't exited
2013-02-04 14:40:14,474 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.business.event.JobSchedulerImpl - Number of pending jobs: 163
2013-02-04 14:40:17,610 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - show() : start
2013-02-04 14:40:17,862 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - show() : end
2013-02-04 14:40:54,633 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - onFinish() : start
2013-02-04 14:40:54,633 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - [Thu Jan 01 07:15:00 EST 1970]
2013-02-04 14:40:54,634 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - [Thu Jan 01 07:15:00 EST 1970, Thu Jan 01 11:05:00 EST 1970]
2013-02-04 14:40:54,634 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - [Thu Jan 01 07:15:00 EST 1970, Thu Jan 01 11:05:00 EST 1970, Thu Jan 01 14:05:00 EST 1970]
2013-02-04 14:40:54,634 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - [Thu Jan 01 07:15:00 EST 1970, Thu Jan 01 11:05:00 EST 1970, Thu Jan 01 14:05:00 EST 1970, Thu Jan 01 21:59:00 EST 1970]
2013-02-04 14:40:54,647 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - onFinish() : end
2013-02-04 14:40:54,647 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.db.JobDbImpl - updateJob(JobProperties) : start
2013-02-04 14:40:54,647 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.db.drivers.file.FileJobDbStorage - update(...) : start
2013-02-04 14:40:54,648 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.db.drivers.file.FileJobDbStorage - update(...) : end
2013-02-04 14:40:54,649 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.db.JobDbImpl - updateJob(JobProperties) : end
scheduler log:
4-Feb-2013 02:40:13 PM 3 5ou5AYjUgNmragQXMn/7WhhcR5A= 1144 TOL_1 PMT Load Driver Northside Remote job failed. process hasn't exited
4-Feb-2013 02:40:13 PM 1 5ou5AYjUgNmragQXMn/7WhhcR5A= 1144 TOL_1 PMT Load Driver Northside Job has been disabled!
4-Feb-2013 02:40:13 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 1144 TOL_1 PMT Load Driver Northside Error message sent.
|
|
Mon Feb 04, 2013 6:42 pm |
|
![](templates/GreenTech/images/spacer.gif) |
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7855
|
|
|
|
Please take a look what you get in logs on the agent side of the processing? Are there any other clues?
|
|
Mon Feb 04, 2013 10:06 pm |
|
![](templates/GreenTech/images/spacer.gif) |
owensjim
Joined: 08 Aug 2011 Posts: 49
|
|
|
|
I did not find anything around that time frame in the agent scheduler.log.
4-Feb-2013 11:47:04 PM 2 null 433 TOL_1 Multi PMT Process Job completed with exit code 0, exit code condition satisfied.
4-Feb-2013 11:47:04 PM 2 null 433 TOL_1 Multi PMT Process Remote job finished.
5-Feb-2013 12:02:01 AM 2 null 700 TOL_1 Update Office Cycles CGI Remote job started.
5-Feb-2013 12:02:01 AM 2 null 700 TOL_1 Update Office Cycles CGI Job started.
|
|
Tue Feb 05, 2013 4:02 pm |
|
![](templates/GreenTech/images/spacer.gif) |
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7855
|
|
|
|
Are you sure you have checked the correct agent?
In your log fragment I don't see any references to "TOL_1 PMT Load Driver Northside"
Nor I see any activities around 2013-02-04 14:40:13. It seems like you are looking in a wrong log file.
What kind of command line are you starting from the job?
Lastly, your scheduler settings are mis configured. You enabled HTML logs feature and as a destination for HTML files specified invalid directory. As a result, every time you run a job, that leads to an exception "Error generating HTML report files. java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory"
|
|
Tue Feb 05, 2013 10:50 pm |
|
![](templates/GreenTech/images/spacer.gif) |
owensjim
Joined: 08 Aug 2011 Posts: 49
|
|
|
|
Yes, I was on the correct agent, notice the references to other the TOL1 jobs, however, I did not proof my message, sorry about that I messed up, I was in the wrong part of the day
4-Feb-2013 02:20:05 PM 2 null 978 Tol_1 ReportTable Extract Remote job finished.
4-Feb-2013 02:30:08 PM 2 null 1144 TOL_1 PMT Load Driver Northside Remote job started.
4-Feb-2013 02:30:08 PM 2 null 1144 TOL_1 PMT Load Driver Northside Job started.
4-Feb-2013 02:37:01 PM 2 null 398 Tol_1 Monitor Files Remote job started.
4-Feb-2013 02:37:01 PM 2 null 398 Tol_1 Monitor Files Job started.
here is the command starting the job:
export HOSTNAME=earchive.ipayx.net;/home/einvoice/cgi-php/ei_bat_pmtload_driver.php 15
the file being processed was 445mb
will change the configuration
|
|
Wed Feb 06, 2013 10:40 am |
|
![](templates/GreenTech/images/spacer.gif) |
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7855
|
|
|
|
I suggest adding in front of the command csh-c
|
|
Wed Feb 06, 2013 11:23 am |
|
![](templates/GreenTech/images/spacer.gif) |
owensjim
Joined: 08 Aug 2011 Posts: 49
|
|
|
|
I added csh-c in front of the word export. csh-c was the first thing in the command line. The job ran but did not do anything.
csh-c export HOSTNAME=earchive.ipayx.net;/home/einvoice/cgi-php/ei_bat_pmtload_driver.php 15
|
|
Thu Feb 07, 2013 11:38 am |
|
![](templates/GreenTech/images/spacer.gif) |
owensjim
Joined: 08 Aug 2011 Posts: 49
|
|
|
|
I have restarted the tol1 agent as jobs started receiving communications errors and stated the agent was not found.
I removed the csh-c as, resubmitted the job. The job ran normally and processed the 145 mb file.
Almost appears related to the Agent causing this trouble.
|
|
Thu Feb 07, 2013 4:48 pm |
|
![](templates/GreenTech/images/spacer.gif) |
owensjim
Joined: 08 Aug 2011 Posts: 49
|
|
|
|
since restarting the agent, we have not received any "exited" errors on this job.
|
|
Mon Feb 11, 2013 12:03 pm |
|
![](templates/GreenTech/images/spacer.gif) |
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7855
|
|
|
|
Thank you for the status update.
Good to know that the issue is resolved for now, although we still don't know what it occurred in the first place.
|
|
Tue Feb 12, 2013 12:01 am |
|
![](templates/GreenTech/images/spacer.gif) |
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|
|