Author |
Message |
owensjim
Joined: 08 Aug 2011 Posts: 49
|
|
job logging |
|
Logging appears to be intermitent. Logging stopped on the 18th then started again on the 21st.
When selecting the "job log" tab of an individual job, the job log information is not updated in some instances. If I manually run a job, the job log information is updated for that job and show in the log file for that job.
I have other jobs that are logging properly both on the individual job log tab and in the overal log file.
The first example shows logging for today, the second shows where logging stopped between the 18 that the 21st
Any recommendations?
|
|
Mon Nov 21, 2011 12:36 pm |
|
|
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7903
|
|
|
|
Please check the actual log files or database logs whichever you use.
Screen updates aren't guaranteed to be real time or accurate.
Thanks
|
|
Mon Nov 21, 2011 3:21 pm |
|
|
owensjim
Joined: 08 Aug 2011 Posts: 49
|
|
|
|
OK, I need some assistance.
Here is the scenario. We are running job scheduler 4.4 on a VM machine. That machine was on a network that was encountering network trouble and caused jobs to fail. On Nov 17, the VM machine was moved to a co-location as a VM machine where the job scheduler 4.1 machine is running. Since that time the job log information for 4.4 is all over the place. Some jobs are logging properly while other do not see to be logging. We have syslog information written to disk so I can see that jobs are processing as they should.
I cannot use job scheduler to do any type of research. This is our production environment.
See the attached example:
As you can see the job log tab is blank for this job. Now I have a gap in logging. Note the missing information from 11-21 06:03 pm through 11-22 04:16. According to our systems guys nothing was going on that should have caused this gap in the log.
Restart job scheduler?
|
|
Tue Nov 22, 2011 1:18 pm |
|
|
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7903
|
|
|
|
There are few things at your disposal that can improve the logging issue.
1. Graphical scheduler mode is designed for use as a development IDE and for troubleshooting configuration issues in graphical mode. Normally the scheduler process should run in the background as a demon or a service.
2. Use 24x7 web-based management console to work with the scheduler remotely; you will need just a web browser and you can see graphical job logs. You can also use the command line console to query logs.
3. Enable HTML Status Logs and/or database logs features to make the scheduler write to persistent log files or better to a database table or to syslog if you wish. This will provide you with more consistent log data, no log truncation and rotation will take place. You will have the complete history of all events.
|
|
Tue Nov 22, 2011 11:44 pm |
|
|
owensjim
Joined: 08 Aug 2011 Posts: 49
|
|
|
|
Is there a readme or some type of directions that I can give the systems guys to enable the 24x7 web-based management console to work with the scheduler? I asked them to do this and they came back with "need more information" also for the "command line" console.
I believe we have "enable HTML Status Logs" on. See the attachments.
|
|
Mon Nov 28, 2011 3:25 pm |
|
|
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7903
|
|
|
|
The documentation is installed with the product. Look for PDF files in the product home directory, their names are kind self-descriptive
|
|
Tue Nov 29, 2011 12:20 am |
|
|
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7903
|
|
|
Tue Nov 29, 2011 12:24 am |
|
|
owensjim
Joined: 08 Aug 2011 Posts: 49
|
|
|
|
Thanks, I found the PDF.
After job scheduler restart, we still are experiencing issues. We are missing a period of logging again today.
28-Nov-2011 08:02:31 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 184 FTP 4 Ftp Driver Remote job finished.
28-Nov-2011 08:02:32 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 184 FTP 4 Ftp Driver Finish message sent.
28-Nov-2011 08:03:15 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 185 FTP 4 PostFtp Gpg Crypt Semaphore file(s) /home/jscheduler/24x7_Scheduler/FINISH0018400185.txt is found. Submitting job to the queue.
29-Nov-2011 09:10:00 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Remote job started.
29-Nov-2011 09:10:00 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Agent "FTP2" contacted.
29-Nov-2011 09:10:02 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Remote job finished.
I copied this from the scheduler.log.
|
|
Tue Nov 29, 2011 10:23 am |
|
|
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7903
|
|
|
|
Since you have syslog enabled, can you check if the missing log messages are available in syslog? If yes, roughly how many of them are missing?
|
|
Tue Nov 29, 2011 11:59 am |
|
|
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7903
|
|
|
|
Also, are they present in the generated HTML report files?
|
|
Tue Nov 29, 2011 12:01 pm |
|
|
owensjim
Joined: 08 Aug 2011 Posts: 49
|
|
|
|
We are using a tool called Log Analyzer, the SYSLOG information is formatted and dumped into the log analyzer table. The log analyzer report matches what I saw in the scheduler.log file.
Today 09:10:00
Message =2011-11-29 09:10:00,908 [INFO] #012HOST: jscheduler-master.ipayx.net#012JOB ID: 479#012JOB NAME: FTP 2 Mqueue Filter#012Agent "FTP2" contacted.
Today 09:10:00
Messagetype Syslog
Message =2011-11-29 09:10:00,046 [INFO] #012HOST: jscheduler-master.ipayx.net#012JOB ID: 479#012JOB NAME: FTP 2 Mqueue Filter#012Remote job started.
Today 08:10:01
pam_unix(cron:session): session closed for user root
Details for Syslogmessage with ID '77650089'
Date Today 08:10:01
Facility SECURITY
Severity INFO
Host jscheduler-master
Syslogtag CRON[5797]:
ProcessID
Messagetype Syslog
Message pam_unix(cron:session): session closed for user root
There are no entries for job scheduler logging between 20:03 PM yesterday and today at 09:10 AM.
Yesterday 20:10:02
pam_mysql - SELECT returned no result.
Details for Syslogmessage with ID '76772943'
Date Yesterday 20:10:02
Facility SECURITY
Severity ERR
Host jscheduler-master
Syslogtag CRON[26057]:
ProcessID
Messagetype Syslog
Message pam_mysql - SELECT returned no result.
Yesterday 20:03:15
=2011-11-28 20:03:15,527 [INFO] #012HOST: jscheduler-master.ipayx.net#012JOB I ...
Details for Syslogmessage with ID '76762123'
Date Yesterday 20:03:15
Facility USER
Severity INFO
Host jscheduler-master
Syslogtag user:
ProcessID
Messagetype Syslog
Message =2011-11-28 20:03:15,527 [INFO] #012HOST: jscheduler-master.ipayx.net#012JOB ID: 185#012JOB NAME: FTP 4 PostFtp Gpg Crypt#012Semaphore file(s) /home/jscheduler/24x7_Scheduler/FINISH0018400185.txt is found. Submitting job to the queue.
Yesterday 20:02:32
=2011-11-28 20:02:32,329 [INFO] #012HOST: jscheduler-master.ipayx.net#012JOB I ...
Details for Syslogmessage with ID '76761858'
Date Yesterday 20:02:32
Facility USER
Severity INFO
Host jscheduler-master
Syslogtag user:
ProcessID
Messagetype Syslog
Message =2011-11-28 20:02:32,329 [INFO] #012HOST: jscheduler-master.ipayx.net#012JOB ID: 184#012JOB NAME: FTP 4 Ftp Driver#012Finish message sent.
I do not have any HTML reports.
We have the following error occuring so I do not have any HTML reports, I will get the systems guy to correct the problem.
2011-11-29 13:50:02,669 [Job #479 - FTP 2 Mqueue Filter] ERROR com.softtreetech.jscheduler.business.log.LoggerImpl - Error generating HTML report files
java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at com.softtreetech.jscheduler.business.log.HtmlLogger.copyFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportsFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportFiles(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.generateReports(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)
|
|
Tue Nov 29, 2011 3:08 pm |
|
|
owensjim
Joined: 08 Aug 2011 Posts: 49
|
|
|
|
Hi again,
I am still having missing log entries but did notice the "Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException" continued for pages, I could not see where this message stoped, but logging started again at 3:10 see the following at the same time as the logging stopped:
1-Dec-2011 01:55:33 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 186 FTP 4 Gpg Crypt Finish message sent.
1-Dec-2011 01:55:41 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 185 FTP 4 PostFtp Gpg Crypt Semaphore file(s) /home/jscheduler/24x7_Scheduler/FINISH0018400185.txt is found. Submitting job to the queue.
1-Dec-2011 03:10:00 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Remote job started.
1-Dec-2011 03:10:00 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Agent "FTP2" contacted.
1-Dec-2011 01:55:28 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 186 FTP 4 Gpg Crypt Agent "FTP4" contacted.
1-Dec-2011 01:55:33 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 186 FTP 4 Gpg Crypt Remote job finished.
1-Dec-2011 01:55:33 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 186 FTP 4 Gpg Crypt Finish message sent.
1-Dec-2011 01:55:41 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 185 FTP 4 PostFtp Gpg Crypt Semaphore file(s) /home/jscheduler/24x7_Scheduler/FINISH0018400185.txt is found. Submitting job to the queue.
Then noticed this:
2011-12-01 13:55:34,386 [Job #186 - FTP 4 Gpg Crypt] ERROR com.softtreetech.jscheduler.business.log.LoggerImpl - Error generating HTML report files
java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at com.softtreetech.jscheduler.business.log.HtmlLogger.copyFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportsFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportFiles(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.generateReports(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.NotificationProcessorImpl.processEvent(Unknown Source)
at com.softtreetech.jscheduler.business.NotificationProcessorImpl.onFinish(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.onFinish(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
30-Nov-2011 10:32:03 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 186 FTP 4 Gpg Crypt Remote job finished.
30-Nov-2011 10:32:04 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 186 FTP 4 Gpg Crypt Finish message sent.
30-Nov-2011 10:32:26 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 184 FTP 4 Ftp Driver Semaphore file(s) /home/jscheduler/24x7_Scheduler/FINISH0018600184.txt is found. Submitting job to the queue.
1-Dec-2011 09:17:02 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 544 CTS_LI Devel Daily Reports Remote job started.
1-Dec-2011 09:17:03 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 544 CTS_LI Devel Daily Reports Agent "CTS-LI Devel" contacted.
1-Dec-2011 09:17:30 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 544 CTS_LI Devel Daily Reports Remote job finished.
30-Nov-2011 06:32:19 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 186 FTP 4 Gpg Crypt Finish message sent.
30-Nov-2011 06:32:26 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 184 FTP 4 Ftp Driver Semaphore file(s) /home/jscheduler/24x7_Scheduler/FINISH0018600184.txt is found. Submitting job to the queue.
30-Nov-2011 09:05:00 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Remote job started.
30-Nov-2011 09:05:00 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Agent "FTP2" contacted.
30-Nov-2011 09:05:01 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Remote job finished.
|
|
Thu Dec 01, 2011 4:36 pm |
|
|
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7903
|
|
|
|
Hi. This could be caused by a bug, I mean AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
Please send your current debug.log to supportATsofttreetech.com before it gets overwriten by more recent entries.
|
|
Thu Dec 01, 2011 5:19 pm |
|
|
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7903
|
|
|
|
by the way, the first exception is valid, you specified non existing output directory for HTML status reports. Please create /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs or fix 24x7 cvongiuration and point it to an existing directory.
java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
|
|
Thu Dec 01, 2011 6:24 pm |
|
|
owensjim
Joined: 08 Aug 2011 Posts: 49
|
|
|
|
How can I change the debug log so that it will retain more infomation?
|
|
Fri Dec 02, 2011 11:39 am |
|
|
|