SoftTree Technologies SoftTree Technologies
Technical Support Forums
RegisterSearchFAQMemberlistUsergroupsLog in
job logging

 
Reply to topic    SoftTree Technologies Forum Index » 24x7 Scheduler, Event Server, Automation Suite View previous topic
View next topic
job logging
Author Message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post job logging Reply with quote
Logging appears to be intermitent. Logging stopped on the 18th then started again on the 21st.

When selecting the "job log" tab of an individual job, the job log information is not updated in some instances. If I manually run a job, the job log information is updated for that job and show in the log file for that job.

I have other jobs that are logging properly both on the individual job log tab and in the overal log file.

The first example shows logging for today, the second shows where logging stopped between the 18 that the 21st





Any recommendations?
Mon Nov 21, 2011 12:36 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7838

Post Reply with quote
Please check the actual log files or database logs whichever you use.

Screen updates aren't guaranteed to be real time or accurate.

Thanks
Mon Nov 21, 2011 3:21 pm View user's profile Send private message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post Reply with quote
OK, I need some assistance.

Here is the scenario. We are running job scheduler 4.4 on a VM machine. That machine was on a network that was encountering network trouble and caused jobs to fail. On Nov 17, the VM machine was moved to a co-location as a VM machine where the job scheduler 4.1 machine is running. Since that time the job log information for 4.4 is all over the place. Some jobs are logging properly while other do not see to be logging. We have syslog information written to disk so I can see that jobs are processing as they should.

I cannot use job scheduler to do any type of research. This is our production environment.

See the attached example:


As you can see the job log tab is blank for this job. Now I have a gap in logging. Note the missing information from 11-21 06:03 pm through 11-22 04:16. According to our systems guys nothing was going on that should have caused this gap in the log.

Restart job scheduler?
Tue Nov 22, 2011 1:18 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7838

Post Reply with quote
There are few things at your disposal that can improve the logging issue.

1. Graphical scheduler mode is designed for use as a development IDE and for troubleshooting configuration issues in graphical mode. Normally the scheduler process should run in the background as a demon or a service.
2. Use 24x7 web-based management console to work with the scheduler remotely; you will need just a web browser and you can see graphical job logs. You can also use the command line console to query logs.
3. Enable HTML Status Logs and/or database logs features to make the scheduler write to persistent log files or better to a database table or to syslog if you wish. This will provide you with more consistent log data, no log truncation and rotation will take place. You will have the complete history of all events.
Tue Nov 22, 2011 11:44 pm View user's profile Send private message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post Reply with quote
Is there a readme or some type of directions that I can give the systems guys to enable the 24x7 web-based management console to work with the scheduler? I asked them to do this and they came back with "need more information" also for the "command line" console.

I believe we have "enable HTML Status Logs" on. See the attachments.






Mon Nov 28, 2011 3:25 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7838

Post Reply with quote
The documentation is installed with the product. Look for PDF files in the product home directory, their names are kind self-descriptive
Tue Nov 29, 2011 12:20 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7838

Post Reply with quote
just in case, here are links to on-line versions

24x7 Web-based Interface Installation and Running with Apache Tomcat
http://www.softtreetech.com/24x7/24x7Tomcat.pdf
24x7 Web-based Interface Reference
http://www.softtreetech.com/24x7/24x7web_gui.pdf
Tue Nov 29, 2011 12:24 am View user's profile Send private message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post Reply with quote
Thanks, I found the PDF.

After job scheduler restart, we still are experiencing issues. We are missing a period of logging again today.

28-Nov-2011 08:02:31 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 184 FTP 4 Ftp Driver Remote job finished.
28-Nov-2011 08:02:32 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 184 FTP 4 Ftp Driver Finish message sent.
28-Nov-2011 08:03:15 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 185 FTP 4 PostFtp Gpg Crypt Semaphore file(s) /home/jscheduler/24x7_Scheduler/FINISH0018400185.txt is found. Submitting job to the queue.
29-Nov-2011 09:10:00 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Remote job started.
29-Nov-2011 09:10:00 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Agent "FTP2" contacted.
29-Nov-2011 09:10:02 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Remote job finished.


I copied this from the scheduler.log.

Tue Nov 29, 2011 10:23 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7838

Post Reply with quote
Since you have syslog enabled, can you check if the missing log messages are available in syslog? If yes, roughly how many of them are missing?
Tue Nov 29, 2011 11:59 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7838

Post Reply with quote
Also, are they present in the generated HTML report files?
Tue Nov 29, 2011 12:01 pm View user's profile Send private message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post Reply with quote
We are using a tool called Log Analyzer, the SYSLOG information is formatted and dumped into the log analyzer table. The log analyzer report matches what I saw in the scheduler.log file.



Today 09:10:00
Message =2011-11-29 09:10:00,908 [INFO] #012HOST: jscheduler-master.ipayx.net#012JOB ID: 479#012JOB NAME: FTP 2 Mqueue Filter#012Agent "FTP2" contacted.
Today 09:10:00
Messagetype Syslog
Message =2011-11-29 09:10:00,046 [INFO] #012HOST: jscheduler-master.ipayx.net#012JOB ID: 479#012JOB NAME: FTP 2 Mqueue Filter#012Remote job started.
Today 08:10:01
pam_unix(cron:session): session closed for user root
Details for Syslogmessage with ID '77650089'
Date Today 08:10:01
Facility SECURITY
Severity INFO
Host jscheduler-master
Syslogtag CRON[5797]:
ProcessID
Messagetype Syslog
Message pam_unix(cron:session): session closed for user root


There are no entries for job scheduler logging between 20:03 PM yesterday and today at 09:10 AM
.


Yesterday 20:10:02

pam_mysql - SELECT returned no result.
Details for Syslogmessage with ID '76772943'
Date Yesterday 20:10:02
Facility SECURITY
Severity ERR
Host jscheduler-master
Syslogtag CRON[26057]:
ProcessID
Messagetype Syslog
Message pam_mysql - SELECT returned no result.

Yesterday 20:03:15

=2011-11-28 20:03:15,527 [INFO] #012HOST: jscheduler-master.ipayx.net#012JOB I ...
Details for Syslogmessage with ID '76762123'
Date Yesterday 20:03:15
Facility USER
Severity INFO
Host jscheduler-master
Syslogtag user:
ProcessID
Messagetype Syslog
Message =2011-11-28 20:03:15,527 [INFO] #012HOST: jscheduler-master.ipayx.net#012JOB ID: 185#012JOB NAME: FTP 4 PostFtp Gpg Crypt#012Semaphore file(s) /home/jscheduler/24x7_Scheduler/FINISH0018400185.txt is found. Submitting job to the queue.

Yesterday 20:02:32

=2011-11-28 20:02:32,329 [INFO] #012HOST: jscheduler-master.ipayx.net#012JOB I ...
Details for Syslogmessage with ID '76761858'
Date Yesterday 20:02:32
Facility USER
Severity INFO
Host jscheduler-master
Syslogtag user:
ProcessID
Messagetype Syslog
Message =2011-11-28 20:02:32,329 [INFO] #012HOST: jscheduler-master.ipayx.net#012JOB ID: 184#012JOB NAME: FTP 4 Ftp Driver#012Finish message sent.





I do not have any HTML reports.
We have the following error occuring so I do not have any HTML reports, I will get the systems guy to correct the problem.

2011-11-29 13:50:02,669 [Job #479 - FTP 2 Mqueue Filter] ERROR com.softtreetech.jscheduler.business.log.LoggerImpl - Error generating HTML report files
java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at com.softtreetech.jscheduler.business.log.HtmlLogger.copyFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportsFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportFiles(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.generateReports(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)

Tue Nov 29, 2011 3:08 pm View user's profile Send private message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post Reply with quote
Hi again,

I am still having missing log entries but did notice the "Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException" continued for pages, I could not see where this message stoped, but logging started again at 3:10 see the following at the same time as the logging stopped:


1-Dec-2011 01:55:33 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 186 FTP 4 Gpg Crypt Finish message sent.
1-Dec-2011 01:55:41 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 185 FTP 4 PostFtp Gpg Crypt Semaphore file(s) /home/jscheduler/24x7_Scheduler/FINISH0018400185.txt is found. Submitting job to the queue.
1-Dec-2011 03:10:00 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Remote job started.
1-Dec-2011 03:10:00 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Agent "FTP2" contacted.


1-Dec-2011 01:55:28 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 186 FTP 4 Gpg Crypt Agent "FTP4" contacted.
1-Dec-2011 01:55:33 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 186 FTP 4 Gpg Crypt Remote job finished.
1-Dec-2011 01:55:33 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 186 FTP 4 Gpg Crypt Finish message sent.
1-Dec-2011 01:55:41 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 185 FTP 4 PostFtp Gpg Crypt Semaphore file(s) /home/jscheduler/24x7_Scheduler/FINISH0018400185.txt is found. Submitting job to the queue.

Then noticed this:

2011-12-01 13:55:34,386 [Job #186 - FTP 4 Gpg Crypt] ERROR com.softtreetech.jscheduler.business.log.LoggerImpl - Error generating HTML report files
java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at com.softtreetech.jscheduler.business.log.HtmlLogger.copyFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportsFile(Unknown Source)
at com.softtreetech.jscheduler.business.log.HtmlLogger.generateReportFiles(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.generateReports(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.log.LoggerImpl.addLogRecord(Unknown Source)
at com.softtreetech.jscheduler.business.NotificationProcessorImpl.processEvent(Unknown Source)
at com.softtreetech.jscheduler.business.NotificationProcessorImpl.onFinish(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.onFinish(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.startExecution(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException


30-Nov-2011 10:32:03 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 186 FTP 4 Gpg Crypt Remote job finished.
30-Nov-2011 10:32:04 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 186 FTP 4 Gpg Crypt Finish message sent.
30-Nov-2011 10:32:26 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 184 FTP 4 Ftp Driver Semaphore file(s) /home/jscheduler/24x7_Scheduler/FINISH0018600184.txt is found. Submitting job to the queue.
1-Dec-2011 09:17:02 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 544 CTS_LI Devel Daily Reports Remote job started.
1-Dec-2011 09:17:03 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 544 CTS_LI Devel Daily Reports Agent "CTS-LI Devel" contacted.
1-Dec-2011 09:17:30 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 544 CTS_LI Devel Daily Reports Remote job finished.

30-Nov-2011 06:32:19 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 186 FTP 4 Gpg Crypt Finish message sent.
30-Nov-2011 06:32:26 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 184 FTP 4 Ftp Driver Semaphore file(s) /home/jscheduler/24x7_Scheduler/FINISH0018600184.txt is found. Submitting job to the queue.
30-Nov-2011 09:05:00 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Remote job started.
30-Nov-2011 09:05:00 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Agent "FTP2" contacted.
30-Nov-2011 09:05:01 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Remote job finished.
Thu Dec 01, 2011 4:36 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7838

Post Reply with quote
Hi. This could be caused by a bug, I mean AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException
Please send your current debug.log to supportATsofttreetech.com before it gets overwriten by more recent entries.
Thu Dec 01, 2011 5:19 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7838

Post Reply with quote
by the way, the first exception is valid, you specified non existing output directory for HTML status reports. Please create /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs or fix 24x7 cvongiuration and point it to an existing directory.

java.io.FileNotFoundException: /home/jscheduler/24x7_Scheduler/24x7_reports/joblogs/reports.htm (No such file or directory)
Thu Dec 01, 2011 6:24 pm View user's profile Send private message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post Reply with quote
How can I change the debug log so that it will retain more infomation?
Fri Dec 02, 2011 11:39 am View user's profile Send private message
Display posts from previous:    
Reply to topic    SoftTree Technologies Forum Index » 24x7 Scheduler, Event Server, Automation Suite All times are GMT - 4 Hours
Page 1 of 1

 
Jump to: 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


 

 

Powered by phpBB © 2001, 2005 phpBB Group
Design by Freestyle XL / Flowers Online.