SoftTree Technologies SoftTree Technologies
Technical Support Forums
RegisterSearchFAQMemberlistUsergroupsLog in
process hasn't exited

 
Reply to topic    SoftTree Technologies Forum Index » 24x7 Scheduler, Event Server, Automation Suite View previous topic
View next topic
process hasn't exited
Author Message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post process hasn't exited Reply with quote
We have two types of jobs that have recently stated receiving:

process hasn't exited

and the job fails.

Please advise.

debug log:

2013-02-04 14:40:13,271 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.runner.RemoteJobRunner - runJob
com.softtreetech.jscheduler.common.SchedException: process hasn't exited
at com.softtreetech.jscheduler.business.agent.remote.RemoteAgentImpl.executeJob(Unknown Source)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
at com.softtreetech.jscheduler.business.agent.remote.RemoteAgentImpl_Stub.executeJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.RemoteJobRunner.runJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.runJobIgnoringErrorsIfNeeded(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
2013-02-04 14:40:13,272 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.log.SysLogAdaptor - Syslog: invoked
2013-02-04 14:40:13,468 [Job #1144 - TOL_1 PMT Load Driver Northside] ERROR com.softtreetech.jscheduler.business.log.LoggerImpl - Error generating HTML report files
java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
at com.softtreetech.jscheduler.business.log.HtmlLogger.copyFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportsFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportFiles(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.generateReports(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
2013-02-04 14:40:13,469 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.db.JobDbImpl - updateJob(JobProperties) : start
2013-02-04 14:40:13,470 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.db.drivers.file.FileJobDbStorage - update(...) : start
2013-02-04 14:40:13,470 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.db.drivers.file.FileJobDbStorage - update(...) : end
2013-02-04 14:40:13,470 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.db.JobDbImpl - updateJob(JobProperties) : end
2013-02-04 14:40:13,471 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.log.SysLogAdaptor - Syslog: invoked
2013-02-04 14:40:13,661 [Job #1144 - TOL_1 PMT Load Driver Northside] ERROR com.softtreetech.jscheduler.business.log.LoggerImpl - Error generating HTML report files
java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
at com.softtreetech.jscheduler.business.log.HtmlLogger.copyFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportsFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportFiles(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.generateReports(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.db.JobDbImpl.disableJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.disableJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.processFailedJob(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.onFails(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
2013-02-04 14:40:13,662 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - mail.smtp.auth=true
2013-02-04 14:40:13,664 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): Date: Mon, 4 Feb 2013 14:40:13 -0500 (EST)
2013-02-04 14:40:13,664 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): From: job.scheduler@ipayx.com
2013-02-04 14:40:13,665 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): To: alerts@ipayx.com, operations@ipayx.com, 5673953932@vtext.com
2013-02-04 14:40:13,665 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): Subject: Message from 24x7 Scheduler - job "TOL_1 PMT Load Driver Northside"
event
2013-02-04 14:40:13,665 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): X-Mailer: 24x7 Scheduler
2013-02-04 14:40:13,665 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl(): account & password information is specified
2013-02-04 14:40:13,937 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.Mailer - sendEmailImpl - mail sent
2013-02-04 14:40:13,937 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.log.SysLogAdaptor - Syslog: invoked
2013-02-04 14:40:14,135 [Job #1144 - TOL_1 PMT Load Driver Northside] ERROR com.softtreetech.jscheduler.business.log.LoggerImpl - Error generating HTML report files
java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
at com.softtreetech.jscheduler.business.log.HtmlLogger.copyFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportsFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportFiles(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.generateReports(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.NotificationProcessorImpl.processOnFailEvent(Unknown Source)
at com.softtreetech.jscheduler.business.NotificationProcessorImpl.onFails(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.onFails(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
2013-02-04 14:40:14,136 [Job #1144 - TOL_1 PMT Load Driver Northside] DEBUG com.softtreetech.jscheduler.business.queue.JobQueue - QUEUE_UNLOCKED
2013-02-04 14:40:14,329 [Job #1144 - TOL_1 PMT Load Driver Northside] ERROR com.softtreetech.jscheduler.business.runner.JobExecutorImpl - Job errors: Remote job failed. process hasn't exited
2013-02-04 14:40:14,474 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.business.event.JobSchedulerImpl - Number of pending jobs: 163
2013-02-04 14:40:17,610 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - show() : start
2013-02-04 14:40:17,862 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - show() : end
2013-02-04 14:40:54,633 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - onFinish() : start
2013-02-04 14:40:54,633 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - [Thu Jan 01 07:15:00 EST 1970]
2013-02-04 14:40:54,634 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - [Thu Jan 01 07:15:00 EST 1970, Thu Jan 01 11:05:00 EST 1970]
2013-02-04 14:40:54,634 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - [Thu Jan 01 07:15:00 EST 1970, Thu Jan 01 11:05:00 EST 1970, Thu Jan 01 14:05:00 EST 1970]
2013-02-04 14:40:54,634 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - [Thu Jan 01 07:15:00 EST 1970, Thu Jan 01 11:05:00 EST 1970, Thu Jan 01 14:05:00 EST 1970, Thu Jan 01 21:59:00 EST 1970]
2013-02-04 14:40:54,647 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.ui.dialogs.jobwizard.JobWizard - onFinish() : end
2013-02-04 14:40:54,647 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.db.JobDbImpl - updateJob(JobProperties) : start
2013-02-04 14:40:54,647 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.db.drivers.file.FileJobDbStorage - update(...) : start
2013-02-04 14:40:54,648 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.db.drivers.file.FileJobDbStorage - update(...) : end
2013-02-04 14:40:54,649 [AWT-EventQueue-0] DEBUG com.softtreetech.jscheduler.db.JobDbImpl - updateJob(JobProperties) : end




scheduler log:

4-Feb-2013 02:40:13 PM 3 5ou5AYjUgNmragQXMn/7WhhcR5A= 1144 TOL_1 PMT Load Driver Northside Remote job failed. process hasn't exited
4-Feb-2013 02:40:13 PM 1 5ou5AYjUgNmragQXMn/7WhhcR5A= 1144 TOL_1 PMT Load Driver Northside Job has been disabled!
4-Feb-2013 02:40:13 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 1144 TOL_1 PMT Load Driver Northside Error message sent.
Mon Feb 04, 2013 6:42 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7841

Post Reply with quote
Please take a look what you get in logs on the agent side of the processing? Are there any other clues?
Mon Feb 04, 2013 10:06 pm View user's profile Send private message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post Reply with quote
I did not find anything around that time frame in the agent scheduler.log.

4-Feb-2013 11:47:04 PM 2 null 433 TOL_1 Multi PMT Process Job completed with exit code 0, exit code condition satisfied.
4-Feb-2013 11:47:04 PM 2 null 433 TOL_1 Multi PMT Process Remote job finished.
5-Feb-2013 12:02:01 AM 2 null 700 TOL_1 Update Office Cycles CGI Remote job started.
5-Feb-2013 12:02:01 AM 2 null 700 TOL_1 Update Office Cycles CGI Job started.
Tue Feb 05, 2013 4:02 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7841

Post Reply with quote
Are you sure you have checked the correct agent?
In your log fragment I don't see any references to "TOL_1 PMT Load Driver Northside"
Nor I see any activities around 2013-02-04 14:40:13. It seems like you are looking in a wrong log file.

What kind of command line are you starting from the job?

Lastly, your scheduler settings are mis configured. You enabled HTML logs feature and as a destination for HTML files specified invalid directory. As a result, every time you run a job, that leads to an exception "Error generating HTML report files. java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory"
Tue Feb 05, 2013 10:50 pm View user's profile Send private message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post Reply with quote
Yes, I was on the correct agent, notice the references to other the TOL1 jobs, however, I did not proof my message, sorry about that I messed up, I was in the wrong part of the day

4-Feb-2013 02:20:05 PM 2 null 978 Tol_1 ReportTable Extract Remote job finished.
4-Feb-2013 02:30:08 PM 2 null 1144 TOL_1 PMT Load Driver Northside Remote job started.
4-Feb-2013 02:30:08 PM 2 null 1144 TOL_1 PMT Load Driver Northside Job started.
4-Feb-2013 02:37:01 PM 2 null 398 Tol_1 Monitor Files Remote job started.
4-Feb-2013 02:37:01 PM 2 null 398 Tol_1 Monitor Files Job started.


here is the command starting the job:
export HOSTNAME=earchive.ipayx.net;/home/einvoice/cgi-php/ei_bat_pmtload_driver.php 15

the file being processed was 445mb

will change the configuration
Wed Feb 06, 2013 10:40 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7841

Post Reply with quote
I suggest adding in front of the command csh-c
Wed Feb 06, 2013 11:23 am View user's profile Send private message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post Reply with quote
I added csh-c in front of the word export. csh-c was the first thing in the command line. The job ran but did not do anything.

csh-c export HOSTNAME=earchive.ipayx.net;/home/einvoice/cgi-php/ei_bat_pmtload_driver.php 15
Thu Feb 07, 2013 11:38 am View user's profile Send private message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post Reply with quote
I have restarted the tol1 agent as jobs started receiving communications errors and stated the agent was not found.

I removed the csh-c as, resubmitted the job. The job ran normally and processed the 145 mb file.

Almost appears related to the Agent causing this trouble.
Thu Feb 07, 2013 4:48 pm View user's profile Send private message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post Reply with quote
since restarting the agent, we have not received any "exited" errors on this job.
Mon Feb 11, 2013 12:03 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7841

Post Reply with quote
Thank you for the status update.

Good to know that the issue is resolved for now, although we still don't know what it occurred in the first place.
Tue Feb 12, 2013 12:01 am View user's profile Send private message
Display posts from previous:    
Reply to topic    SoftTree Technologies Forum Index » 24x7 Scheduler, Event Server, Automation Suite All times are GMT - 4 Hours
Page 1 of 1

 
Jump to: 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


 

 

Powered by phpBB © 2001, 2005 phpBB Group
Design by Freestyle XL / Flowers Online.