SoftTree Technologies SoftTree Technologies
Technical Support Forums
RegisterSearchFAQMemberlistUsergroupsLog in
24X7 logging

 
Reply to topic    SoftTree Technologies Forum Index » 24x7 Scheduler, Event Server, Automation Suite View previous topic
View next topic
24X7 logging
Author Message
gbablg



Joined: 24 Mar 2011
Posts: 3
Country: United Kingdom

Post 24X7 logging Reply with quote
All,

First things first, its my first post to this forum so hello to everyone.

My organisation is running 24x7 version 3.6.5 in order to operate a fairly complex process that consists of about 120 individual jobs. A substantial part of this operates overnight when less human attention/intervention is forthcoming. The process has been (for the most part) reliable/stable for the past 6 months. However, from time to time we experience problems of jobs stalling (no 24X7 error messages are generated) and the queue filling up. Ultimately, if not caught early enough this causes a crucial part of the process to fail.

To prevent this in the future we are trying to ultilise of our existing monitoring software which has a probe that can inspect log files (for strings etc) and send alerts. I realise 24X7 is currently logging to schedule.log and have the following questions:

I can find any way of configuring what is logged and when. Is there something I'm missing? Do newer versions support this?
Can the log files be generated on a daily basis? i.e. a new one for each day, e.g. scheduleDDMMYY.log

Thanks in advance for your input.

Alex
Thu Mar 24, 2011 3:41 pm View user's profile Send private message
gbablg



Joined: 24 Mar 2011
Posts: 3
Country: United Kingdom

Post Reply with quote
As a quick follow up.

I have also tried to make use of the html log generation which does to create the logs by day. However, it seems the 'message' column of the table appears to truncate the data contained in the schedule.log

Example:

log file entry:
23/03/2011 01:36:21:331 0 632 0131-E5M: Open p_upload.xls file Semaphore file(s) @V"env:ZN_TASKS_P_UPLOAD"\\p_upload_open.txt found. Event is ignored because the job is already queued.

html representation:
Date/Time Agent Job No Job Name Message
23/03/2011 01:36:21:331 SERVER_NAME 632 0131-E5M: Open p_upload.xls file Semaphore file(s) @V

As you can see the message column is missing quite a lot of the information. Is this a known bug?

Thanks,
Alex
Fri Mar 25, 2011 4:00 am View user's profile Send private message
SysOpJ



Joined: 20 Aug 2010
Posts: 95

Post Reply with quote
Logging is kind of broad subject, so I'll get to some of that in a minute.

For your issue specifically, take a look at any recently created jobs or jobs recently moved to different queues, set to run detached, or run asynchronously. Usually this kind of hang-up points to resource contention.

There's not really a global configuration for what is logged and when beyond the basic logging, or the trace logging, barring the configuration values for purging or limiting the number of lines. However, you can log on a job-specific basis by adding LogAddMessageEx calls to your jobs (which would put the messages in the normal places, i.e., the default log, the Windows system event log (if configured to do so), and the generated HTML logs (I'll have to check that truncation issue). Or you could use LogAddMessage just to put messages on the Windows system event log.

If you log to the Windows system event log, you could also use the LogSearch call instead of your external service (although I understand that you may prefer an external monitor.) There are also a number of other JDL calls for managing the event log.

Also, if you turn tracing on, you'll have a log file written for each script type job.

To roll over the log file you could create a job using FileSize and FileCopy (using @T to get a time-related file name). I do believe the Multi-Platform version has a preferences setting to do this automatically by file size, although I'm not sure if the preference does daily rollover. You'll also be responsible for cleaning those up yourself using something like FileDir (http://www.softtreetech.com/24x7/archive/46.htm).

You could also monitor the number of jobs in the queues and those running using QueueJobList ... this might help identify the culprit.
Fri Mar 25, 2011 7:08 am View user's profile Send private message
gbablg



Joined: 24 Mar 2011
Posts: 3
Country: United Kingdom

Post Reply with quote
Thanks for the swift response. We've actually made no changes to our configuration for the past 4 months. Up until 19th March we've had very few issues with the scheduler, its been very reliable. Last week Sun - Sun was fairly bad with 4 failures, 2 of which were due to jobs hanging.

Last night, a scheduled job set to run at 2031 Sun - Thur, hung. The job is very basic, it executes a cmd file which creates 5 text files in various local locations. The cmd window had closed and the 5 text files were successfully created, however 24x7 still showed the job as running. Consequently, four other (event driven) jobs that are triggered by the existance of these text files were queued but could not run. Thankfully our monitoring software detected the presence of the 'already queued' string in the log file and alerted the on call person.

Have you any idea why the 2031 job was still 'running' despite the cmd file successfully running and terminating? Furthermore, since this job has no JAL scripting, would the debug log provide any valuable insights? If not, where might we begin?

On another point; A lot of our jobs are put in a background queue. I notice that in the Queue directory there is no 'background' folder. Where do .Q or TMP files go for the background jobs that go into a queue?

One final thing; We realise we're running an old version of 24x7, 3.6.5. Can you confirm whether you no longer officially support this version? If its still support are there any plans for its end of end of life? Based on my (limited!) description of the issues we face, do you think upgrading the version will help to prevent them?

Thanks,
Alex
Mon Mar 28, 2011 6:39 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 6712

Post Reply with quote
Please check if anything interesting turned up in the debug.log and 24x7.err log files
Tue Mar 29, 2011 1:04 am View user's profile Send private message
Display posts from previous:    
Reply to topic    SoftTree Technologies Forum Index » 24x7 Scheduler, Event Server, Automation Suite All times are GMT - 4 Hours
Page 1 of 1

 
Jump to: 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


 

 

Powered by phpBB © 2001, 2005 phpBB Group
Design by Freestyle XL / Flowers Online.