SoftTree Technologies SoftTree Technologies
Technical Support Forums
RegisterSearchFAQMemberlistUsergroupsLog in
24x7 Scheduler stops dispatching jobs

 
Reply to topic    SoftTree Technologies Forum Index » 24x7 Scheduler, Event Server, Automation Suite View previous topic
View next topic
24x7 Scheduler stops dispatching jobs
Author Message
gregm



Joined: 19 Jan 2005
Posts: 6

Post 24x7 Scheduler stops dispatching jobs Reply with quote

We use v3.3.8. Run it as a service on 150 Win2K and Win2K3 servers.
Frequently the scheduler stops dispatching jobs, as evidenced
by SCHEDULE.LOG not being updated. The 24x7 service is still running though.
No errors logged in SCHEDULE.LOG and no events logged in the event log.
Stopping and restarting the service gets things going again.
The problem doesn't have much consistency. May occur within minutes after the service
is started; may not occur for days. BUT, we encounter this at
least twice a week on one of our 150 servers.
Also, our servers are on a weekly reboot schedule. No evidence of memory leaks.

Any ideas? Any recommended troubleshooting steps?

I am evaluating upgrading to v3.4.24, but got to get it thru regression testing, etc.

Thank you.

Tue Jan 11, 2005 7:02 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7970

Post Re: 24x7 Scheduler stops dispatching jobs Reply with quote

I think the best path is to find if this problem is linked to specific jobs. Another "quick" try is to set all jobs to run detached. This can be done using a script job. If you have 24x7 running in the Master mode (e.g. as server) then you can have a job that remotely updates jobs on all servers. Definitely more options are available in 3.4.24 then in 3.3.8.

Because you are running it on a large number of servers it makes sense to maintain single global log. Having single log for all servers you can faster find if there are any job dependencies or common job-related or server-related problems. There are many ways how global logs can be created. For example, you can setup jobs to write start/finish/error events to a centralized database using job notification actions. The drawback is you may need to install database client software on all servers and modify all jobs. Or you can periodically ship schedule.log files to a central server where you would then run command line job using HTMLGen.exe utility that comes with 24x7. This utility can produce global HTML log files listing job events for all servers.

: We use v3.3.8. Run it as a service on 150 Win2K and Win2K3 servers.
: Frequently the scheduler stops dispatching jobs, as evidenced
: by SCHEDULE.LOG not being updated. The 24x7 service is still running though.
: No errors logged in SCHEDULE.LOG and no events logged in the event log.
: Stopping and restarting the service gets things going again.
: The problem doesn't have much consistency. May occur within minutes after the
: service
: is started; may not occur for days. BUT, we encounter this at
: least twice a week on one of our 150 servers.
: Also, our servers are on a weekly reboot schedule. No evidence of memory
: leaks.

: Any ideas? Any recommended troubleshooting steps?

: I am evaluating upgrading to v3.4.24, but got to get it thru regression
: testing, etc.

: Thank you.

Tue Jan 11, 2005 7:20 pm View user's profile Send private message
Robert



Joined: 07 Feb 2002
Posts: 30

Post Re: 24x7 Scheduler stops dispatching jobs Reply with quote

I came to this forum looking post(s) pertaining to this problem,
and I'm having the exact same problem! I am running 24x7 v3.4.17
on Windows Server 2000, Service Pack 4, running as a service. Sure
enough, periodically, 24x7 stops dispatching jobs. I had our ops
guys monitor the service and notify me via email when the service
stops. The service does not stop since I get no email notifications,
but no jobs get dispatched. Very bad! (This happened again last
night!)

FYI, I have a job set up in 24x7 that writes a text file once every
three minutes to a folder. That way I can see gaps in the files when
24x7 is messing up. My symptoms are exactly like the previous post.
There is no pattern I can pick up as to when it will fail, but it
will do so at least once or twice a week. I am running 24x7 only on
ONE computer however, unlike the person in the previous post. I
definitely think there's a bug in 24x7. If 24x7 is not run as a
service, it works as expected, but that's not an option on this server.

Any ideas?

: We use v3.3.8. Run it as a service on 150 Win2K and Win2K3 servers.
: Frequently the scheduler stops dispatching jobs, as evidenced
: by SCHEDULE.LOG not being updated. The 24x7 service is still running though.
: No errors logged in SCHEDULE.LOG and no events logged in the event log.
: Stopping and restarting the service gets things going again.
: The problem doesn't have much consistency. May occur within minutes after the
: service
: is started; may not occur for days. BUT, we encounter this at
: least twice a week on one of our 150 servers.
: Also, our servers are on a weekly reboot schedule. No evidence of memory
: leaks.

: Any ideas? Any recommended troubleshooting steps?

: I am evaluating upgrading to v3.4.24, but got to get it thru regression
: testing, etc.

: Thank you.

Wed Jan 19, 2005 11:47 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7970

Post Re: 24x7 Scheduler stops dispatching jobs Reply with quote

I have no idea why this is happening, but I have a solution to this problem. Assign different job streams (e.g. applications) to different job queues and set all jobs that don't use global variables to run detached.

: I came to this forum looking post(s) pertaining to this problem,
: and I'm having the exact same problem! I am running 24x7 v3.4.17
: on Windows Server 2000, Service Pack 4, running as a service. Sure
: enough, periodically, 24x7 stops dispatching jobs. I had our ops
: guys monitor the service and notify me via email when the service
: stops. The service does not stop since I get no email notifications,
: but no jobs get dispatched. Very bad! (This happened again last
: night!)

: FYI, I have a job set up in 24x7 that writes a text file once every
: three minutes to a folder. That way I can see gaps in the files when
: 24x7 is messing up. My symptoms are exactly like the previous post.
: There is no pattern I can pick up as to when it will fail, but it
: will do so at least once or twice a week. I am running 24x7 only on
: ONE computer however, unlike the person in the previous post. I
: definitely think there's a bug in 24x7. If 24x7 is not run as a
: service, it works as expected, but that's not an option on this server.

: Any ideas?

Wed Jan 19, 2005 2:44 pm View user's profile Send private message
gregm



Joined: 19 Jan 2005
Posts: 6

Post Re: 24x7 Scheduler stops dispatching jobs Reply with quote

I found a couple jobs not running DETACHED (a surprise to me).
I will ensure all my jobs run DETACHED and see what results I get.
Thank you.

: I think the best path is to find if this problem is linked to specific jobs.
: Another "quick" try is to set all jobs to run detached. This can
: be done using a script job. If you have 24x7 running in the Master mode
: (e.g. as server) then you can have a job that remotely updates jobs on all
: servers. Definitely more options are available in 3.4.24 then in 3.3.8.

: Because you are running it on a large number of servers it makes sense to
: maintain single global log. Having single log for all servers you can
: faster find if there are any job dependencies or common job-related or
: server-related problems. There are many ways how global logs can be
: created. For example, you can setup jobs to write start/finish/error
: events to a centralized database using job notification actions. The
: drawback is you may need to install database client software on all
: servers and modify all jobs. Or you can periodically ship schedule.log
: files to a central server where you would then run command line job using
: HTMLGen.exe utility that comes with 24x7. This utility can produce global
: HTML log files listing job events for all servers.

Mon Jan 24, 2005 5:09 pm View user's profile Send private message
Display posts from previous:    
Reply to topic    SoftTree Technologies Forum Index » 24x7 Scheduler, Event Server, Automation Suite All times are GMT - 4 Hours
Page 1 of 1

 
Jump to: 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


 

 

Powered by phpBB © 2001, 2005 phpBB Group
Design by Freestyle XL / Flowers Online.