SoftTree Technologies SoftTree Technologies
Technical Support Forums
RegisterSearchFAQMemberlistUsergroupsLog in
semaphore files created but jobs would not start

 
Reply to topic    SoftTree Technologies Forum Index » 24x7 Scheduler, Event Server, Automation Suite View previous topic
View next topic
semaphore files created but jobs would not start
Author Message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post semaphore files created but jobs would not start Reply with quote
Jobs would not start that should have been triggered by creation of a semaphore file.

jobscheduler stopped processing semaphore triggered jobs. No processing error, no indication of any troubles.

Job scheduler continued to process jobs but none of the triggered jobs would start. We had no processing errors during that time.

I had systems check the jscheduler server for any issues. We restarted job scheduler and the semaphore triggered started processing.
Tue Mar 05, 2013 6:26 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7839

Post Reply with quote
I have heard about this issue before. It could happen if the system runs out of file handles and scheduler is simply unable to check for semaphore files existence and to open new files. The root cause of that effect is not very clear and known to happen only on certain types of Linux systems. As a workaround I suggest scheduling a weekly scheduler service restart.
Tue Mar 05, 2013 10:57 pm View user's profile Send private message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post Reply with quote
Unfortunately, I did not restart our other job scheduler environment.

Job Scheduler encountered an issue today where the "watch" for semaphore files quit working. Jobs that were not triggered by semeaphore files continued to process. Those jobs that should have started by creation of a semaphore file waited. Job scheduler was restarted and the jobs triggered by semaphore files started processing.

This is the first time that I have seen this type of problem in our Linux VM machines. I do know that some DNS changes were made on Saturday and I am working with systems to see what other changes were made.
Wed Mar 06, 2013 11:53 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7839

Post Reply with quote
I think I didn't state clearly the cause. What I meant ot say is "...if the system runs out of file handles available to the scheduler process...." In a nut shell, the handles aren't released for a long time and new ones get opened faster then the old ones are released.... Where exactly handles get stuck is not clear and which middle-ware is to blame for that is unclear too. The overall effect is handle leaking at a process level until all resources get exhausted. That's why restarting the entire process clears the resource issue and the scheduler can start anew. If restarting the scheduler is not a viable option for you, consider replacing semaphore file based job dependencies with "run Another Job" actions on Job Finish event.
Wed Mar 06, 2013 1:36 pm View user's profile Send private message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post Reply with quote
You suggested a work around:

If restarting the scheduler is not a viable option for you, consider replacing semaphore file based job dependencies with "run Another Job" actions on Job Finish event. I have been directed to change all of our jobs to this format. How do you set up a "job Another job" that has two and three semaphore requirements for it to start.

I have been changing the schedule to "no scheduler" on those jobs that are being executed from "run Another job", is that correct?
Tue Mar 12, 2013 5:33 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7839

Post Reply with quote
I think so.

Just for clarity, if job A uses "Run Another Job" action to trigger job B and job B never needs to start on its own, then yes, job B schedule should be set to "No schedule"
Tue Mar 12, 2013 8:59 pm View user's profile Send private message
owensjim



Joined: 08 Aug 2011
Posts: 49

Post Reply with quote
OK, that is correct B would never have to start on its own.

What about the jobs that I have set up that require two semaphore files. How can I set them up "with run another job"?

Job A creates finish001 at 0100
job B creates finish002 at 0200

job C cannot start until both Job A and B have created their semeaphore file.

How do I set up the same type scenario using "run another Job"
Wed Mar 13, 2013 10:31 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7839

Post Reply with quote
Unfortunately, i case of forks and joins like job C dependent on A and B, you would need to use the regular job dependencies.
Wed Mar 13, 2013 10:10 pm View user's profile Send private message
Display posts from previous:    
Reply to topic    SoftTree Technologies Forum Index » 24x7 Scheduler, Event Server, Automation Suite All times are GMT - 4 Hours
Page 1 of 1

 
Jump to: 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


 

 

Powered by phpBB © 2001, 2005 phpBB Group
Design by Freestyle XL / Flowers Online.