SoftTree Technologies SoftTree Technologies
Technical Support Forums
RegisterSearchFAQMemberlistUsergroupsLog in
How to run jobs in parallel in the same queue
Goto page 1, 2  Next
 
Reply to topic    SoftTree Technologies Forum Index » 24x7 Scheduler, Event Server, Automation Suite View previous topic
View next topic
How to run jobs in parallel in the same queue
Author Message
laiko



Joined: 11 Feb 2010
Posts: 43
Country: India

Post How to run jobs in parallel in the same queue Reply with quote
Hi, (it's me again :) )
I have a single job which 'on finish' should trigger 5 jobs in parallel. When I run them (non-Asynchronous and Detached - which are the default), they dont really run in parallel. One waits for the other to complete before it runs. I am running unix shell programs (if that info helps).
Tue Mar 09, 2010 10:46 am View user's profile Send private message
laiko



Joined: 11 Feb 2010
Posts: 43
Country: India

Post Reply with quote
Please help me out on this. The jobs are very slow since they are running in sequence.
Wed Mar 10, 2010 3:48 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7952

Post Reply with quote
You should assign them to different job queue. They will then run concurrently (in parallel)
Wed Mar 10, 2010 5:28 pm View user's profile Send private message
laiko



Joined: 11 Feb 2010
Posts: 43
Country: India

Post Reply with quote
The Dependency monitor becomes convoluted (less readability) as the number of my dependent jobs increase. So I am depending on the Queue to group the related (dependent) jobs together. I think putting them in different queues is a bad idea. Is there a work-around or any configuration set-up to allow parallel execution and grouping of jobs at the same time? Can you please suggest a best way to implement this?
Again, thanks in advance.
Thu Mar 11, 2010 3:37 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7952

Post Reply with quote
Putting them into different queues is a good idea indeed. This is how it is supposed to work in case you need to run concurrent jobs. Queues are for job run-time control and organization, priorities, , load balancing, etc….

You can also use asynchronous job mode to run jobs in the background, win which case they start and free their queue right away, but you will likely pay for that design later. From my point of view running a bunch of asynchronous jobs = chaos.
Thu Mar 11, 2010 9:57 am View user's profile Send private message
laiko



Joined: 11 Feb 2010
Posts: 43
Country: India

Post Reply with quote
Not a good news... we can have even 30 jobs running at the same time... any idea how we can work around it?
Fri Mar 12, 2010 7:21 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7952

Post Reply with quote
I'm sorry, I didn't get your point. Workaround what?

I suggested to use multiple queues for running job concurrently. I didn't say you cannot run them concurrently. I also said that you can run jobs concurrently by using asynchronous job mode, and I said that that would be likely a bad idea. Which of that is a bad news?
Fri Mar 12, 2010 9:50 am View user's profile Send private message
laiko



Joined: 11 Feb 2010
Posts: 43
Country: India

Post Reply with quote
Sorry for the misunderstanding. The thing is that the number of parallel jobs that we can run are unpredictable. It can be 2 to even 30 at the same time. Apart from that, the elapse time varies depending on the data volume. It is difficult to assign which should go to which queue at which time. It maybe that Queue1 and Queue2 jobs are already done, but Queue3 still has 5 pending jobs. It would be good if there is dynamic balancing such that those pending jobs automatically move to idle queues.
Mon Mar 15, 2010 12:38 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7952

Post Reply with quote
If you say that there are no job dependencies, and they can all can run at any time and pretty much in any order, and they all take a while to run, then just set them to run asynchronous. That's it. If they are quick jobs like taking only several seconds or several minutes to run, then I don't get what's the issue with queuing them?
Mon Mar 15, 2010 6:47 am View user's profile Send private message
laiko



Joined: 11 Feb 2010
Posts: 43
Country: India

Post Reply with quote
We have a datawarehouse for which we perform daily loads for various businesses, such as BusinessA, BusinessB.. etc. Say, we have 10 Businesses and each has different rules and the number jobs within each varies, plus the elapse times are unpredictable but definitely not in seconds. 30 minutes on the average, can be 2 hrs or even 6. To give you a picture of a couple of jobs, I attached an illustration. These are very simple. We have businesses whose jobs will make your eyes roll. My concern here is the unpredictability of the jobs elapsed times will make it difficult to pre-assign them to the queues. It can happen that one day, all queue loads are balanced, then the next day, half of the queues are idle and half are busy. I am looking for dynamic load balancing, ie, 24x7 can distribute and run the jobs automatically to balance the loads across queues. With this, my daily load can complete faster.


Tue Mar 16, 2010 12:19 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7952

Post Reply with quote
Ok, that sounds good. For this situation I suggest the following method:

For each business, create 3 queues. Each business is independent and that is fine, that what job queues are for. They allow physical job separation by business, application, etc….

The overall job/queue setup can vary for different businesses, but for the moment let's assume they are all similar. So I suggest using 1st queue for all jobs that can run for 15+ minutes and set them all asynchronous so that they don't wait for each other; using 2nd queue for all jobs that must run in a certain sequence or just may not run concurrently with jobs of the same "class" (by class I mean some business categorization of your jobs that have special requirements); using 3rd queue for everything else, like ad-hoc jobs, maintenance jobs, etc...

Lastly, choosing right names for queues and jobs could make a big difference from the system management point of view. For example, you can name queues like "Q for Bus. A DataWare Loads", "Q for Bus. A Control Jobs", "Q for Bus. A Ad-hoc", etc… Of course you can come up with better names. The idea here is to organize the processing in a way that would simplify system management, and easy troubleshooting in case of operational problems, and also will make each business processes sort of independent from processes run for other businesses.


Last edited by SysOp on Tue Mar 16, 2010 9:25 pm; edited 1 time in total
Tue Mar 16, 2010 12:59 pm View user's profile Send private message
barefootguru



Joined: 10 Aug 2007
Posts: 195

Post Reply with quote
It's worth running some tests to check you don't get a speed hit for running too many jobs in parallel. e.g. 3 jobs might run for 10 minutes each, so 30 minutes in series, but 40 minutes in total if running at the same time.

We use 24x7 for running a SAS data warehouse. Because SAS is so disk intensive the overall run time is improved by scheduling jobs serially. I ended up with two queues for the ETL jobs: one for the big slow jobs (which often has long waits associated with it) and one for quick jobs which are time critical.

Cheers
Tue Mar 16, 2010 3:33 pm View user's profile Send private message
laiko



Joined: 11 Feb 2010
Posts: 43
Country: India

Post Reply with quote
By the way, am just wondering why is this so:

SysOp:
Quote:
you can run jobs concurrently by using asynchronous job mode, and I said that that would be likely a bad idea


I tried the Asynchronous mode and the jobs got stuck in Running mode. What happens when the mode is asynchronous that make jobs hang?
Wed Mar 17, 2010 1:14 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7952

Post Reply with quote
Did the job actually start doing anything?
Wed Mar 17, 2010 3:28 pm View user's profile Send private message
laiko



Joined: 11 Feb 2010
Posts: 43
Country: India

Post Reply with quote
No, they were not running. No process was active but the monitor showed they're in running status.
Mon Mar 22, 2010 1:46 am View user's profile Send private message
Display posts from previous:    
Reply to topic    SoftTree Technologies Forum Index » 24x7 Scheduler, Event Server, Automation Suite All times are GMT - 4 Hours
Goto page 1, 2  Next
Page 1 of 2

 
Jump to: 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


 

 

Powered by phpBB © 2001, 2005 phpBB Group
Design by Freestyle XL / Flowers Online.