General 24x7 questions

I have finally setup my new 24x7 network but have a few licence and technical questions I hope you can answer for me please. I have attached an image of the layout.

1) Can you give me any idea how much bandwidth will be used by the remote agent when it talks to the 24x7 Scheduler via the Internet?

2) In the Agent settings should I use Async or Sync?

3) Does the 24x7 Scheduler open up the TCP connection to the Remote Agent or the other way around. What I am trying to work out is where I need to open up the TCP port on the firewalls. Ideally I would prefer to open the port say 10095 on the Remote Agent firewall (Firewall B in the diagram) rather than our main firewall (Firewall A).

4) Does the 24x7 Scheduler keep a constant connection to Remote Agent or does it connect as and when it is needed?

5) I am confused as to what licences I have and what I need to purchase. The 24x7 Master server uses external programs (ones I have written) to access the 24x7 via API calls whereas the Remote Agents simply run applications that they have been told to run by the 24x7 Master server. I believe I currently have a main license for the 24x7 Master plus another license to be able to access the Master via the API's. Is this correct? Do I have any other licenses? What do I need to purchase for the Agents so that they can run applications only they will not ever be schedulers just dumb agents.

6) When I create a job via the api to run on a Remote Agent will the job stay flagged as Running if I do a Queue query via the API until it has finished? The reason I has is that I have say 10 jobs that are dynamically created the first 9 I push to each of the 3 servers to spread the load but I also have a 10th job that must be run on the Master server which does some tidy up work but can only be run when the other 9 jobs are finished. Each job needs to run one at a time on each server so in effect I need to be running 3 jobs at the same time spread across the 3 servers and then when all 9 jobs are done the tidy up job will have code in it to poll the job queue to ensure that their are no jobs still running and then it will start its tidy up. I am just not sure how best to do this, should I create my 9 jobs as sync so that I have 3 jobs running at the same time (1 job on each server) and then have the tidy up job running as Async so that it will start immediately after it is scheduled?

Thanks in advance for your help.

Kind regards,

Nathan

	Quote:
	1) Can you give me any idea how much bandwidth will be used by the remote agent when it talks to the 24x7 Scheduler via the Internet?

This of course depends on how often they need to talk and how the agent is used. In general, if you only use it to run remote jobs (no file replication), the requirements are pretty low. To run a job, the scheduler connects to the agent, asks for remote user authentication, which only requires a few dozen bytes, then transfers compressed job definition, which is typically only a few Kbytes, and then waits for the agent to return some status code back with no real traffic.

	Quote:
	2) In the Agent settings should I use Async or Sync?

Please use Sync. Async is a legacy option that only complicates the communication protocol (adds some sort of 'server push' layer)

	Quote:
	3) Does the 24x7 Scheduler open up the TCP connection to the Remote Agent or the other way around. What I am trying to work out is where I need to open up the TCP port on the firewalls. Ideally I would prefer to open the port say 10095 on the Remote Agent firewall (Firewall B in the diagram) rather than our main firewall (Firewall A).

The scheduler opens up the TCP connection. In this sense, agent is a server, scheduler is a client. The firewall should allow connection from scheduler to agent. However, you can theoretically create jobs that make the agent to talk back, in other words have a job that goes to the agent, checks some business conditions and depending on the outcome makes a call back to the scheduler to get some global variable to run something else. Please see help topics like RemoteDir, FileTransfer, Set/GetRemoteVariable… If use such features in the code running on the agent side, you would need to open firewall in both directions.

	Quote:
	4) Does the 24x7 Scheduler keep a constant connection to Remote Agent or does it connect as and when it is needed?

Only when it is needed, connect – run job/process – disconnect.

Quote:

5) I am confused as to what licences I have and what I need to purchase. The 24x7 Master server uses external programs (ones I have written) to access the 24x7 via API calls whereas the Remote Agents simply run applications that they have been told to run by the 24x7 Master server. I believe I currently have a main license for the 24x7 Master plus another license to be able to access the Master via the API's. Is this correct? Do I have any other licenses? What do I need to purchase for the Agents so that they can run applications only they will not ever be schedulers just dumb agents.

The licensing structure is flat and simple, for every system you need a separate license. Or you can get a site license for unlimited use in one geographical location (office, campus, etc…)

Quote:

6) When I create a job via the api to run on a Remote Agent will the job stay flagged as Running if I do a Queue query via the API until it has finished? The reason I has is that I have say 10 jobs that are dynamically created the first 9 I push to each of the 3 servers to spread the load but I also have a 10th job that must be run on the Master server which does some tidy up work but can only be run when the other 9 jobs are finished. Each job needs to run one at a time on each server so in effect I need to be running 3 jobs at the same time spread across the 3 servers and then when all 9 jobs are done the tidy up job will have code in it to poll the job queue to ensure that their are no jobs still running and then it will start its tidy up. I am just not sure how best to do this, should I create my 9 jobs as sync so that I have 3 jobs running at the same time (1 job on each server) and then have the tidy up job running as Async so that it will start immediately after it is scheduled?.

It depends on how you run the job. In most cases you will see it as running. Please keep in mind that the job is created in the scheduler, queued in the scheduler, but physically executed on the agent. So that the API for creating jobs and monitoring queues should connect to the scheduler, not the agent. From your description of the requirement, it sounds like you don't need that complication with the queue monitoring. There is an easy way – setup 3 job queues, 3 remote agent profiles, and 3 "runner" jobs to run synchronous, that mode is import . It doesn't matter how you assign them to agents and queues. Using the API (or "control" job) dynamically modify and run each "runner" job using different queues and agents.

Here is a pseudo code demonstrating this idea.

JobModify Job1, "QUEUE", "Queue1"
JobModify Job1, "AGENT_NAME", "Agent1"
QueueJob Job1

JobModify Job1, "QUEUE", "Queue2"
JobModify Job1, "AGENT_NAME", "Agent2"
QueueJob Job2

JobModify Job1, "QUEUE", "Queue3"
JobModify Job1, "AGENT_NAME", "Agent3"
QueueJob Job2

JobModify Job2, "QUEUE", "Queue1"
JobModify Job2, "AGENT_NAME", "Agent1"
QueueJob Job2
…
repeat for second and third jobs, queuing it for each server ina different queue.

The queues will take care of the rest and ensure that on each agent jobs run sequentially, while each agent runs them concurently.

Thanks for the prompt and very detailed reply.

I am still a little confused by point 5 regarding licensing. When I set up my first 24x7 server I purchased a license which enabled me to create and schedule jobs but then I started to write my own code to schedule jobs dynamically. I was told I need to purchase an additional license so that the API could talk to the 24x7 scheduler. So in effect I had to purchase two licenses. Am I right in understanding that each remote agent will only need 1 license because the API must always talk to the scheduler?

Regarding point 6 are you saying that queues are handled in sync so it you had 3 jobs spread across 3 queues (all with the same start time/date) they would all start at the same time?

Thanks again.

5) Yes, you got it correctly, the API is like a client for the scheduler. You can run this client locally or remotely to the scheduler, for example, you can have it connect from your workstation to server running the scheduler.

6) Each queue is kind of independent. So if you place 3 jobs into 3 separate queues at the same time and each queue is free at that time, all 3 jobs start running simultaneously. Place and additional job into each queue (3 jobs), and these jobs won't start until the first job in the queue is done. This scenario of course is correct for synchronous job only. Asynchronous jobs start in queues and wait for their start based on queue usage, but after start don't use the queue and run in the background

Job sequencing across queues can be also controlled using job dependencies.

Thats great, thanks.

I would love to be able to use job dependencies but I cannot see any way to access this from the API when jobs are created using the JDL.

Nathan

There are 2 types of job dependencies

1: The simple 'Run another job' dependency is based on the notification action of type Run Job. Using the API you can modify MSG_ACTIONS, MSG_JOB and MSG_JOB_ID properties of the predecessor job to make it trigger something else. Please see JDL Properties topic in the help for more info.

2: The 'semaphore' dependency is based on so called "file-watch" schedule type, in the GUI it is called 'when certain files exist' Basically, you set the predecessor job to create some semaphore file, and set the dependent job to watch for that file or multiple files if more complicated dependencies are needed. A job can create different files on different events and therefore trigger different jobs. It can also create multiple files to trigger several other jobs simultaneously. A job can be set to watch for multiple dependencies as well, so that it can start only if all predecessor jobs complete successfully. Look for FILE, MSG_FILE, and MSG_FILE_NAME in the same help topic.

Other MSG_xxx properties may be needed to properly setup which events trigger dependent jobs.

But the bottom line it is surely possible to handle using the API.

Wow so is it possible to create a job using the JDL that will run a .exe and when the .exe closes a semaphore file is created? Also where is the semaphore file created when using remote agents (on the scheduler server or the remote agent server)?

Thanks

On the scheduler.

The scheduler takes care of job time and event watching and also processing of "notification" actions. Agents run jobs only. This allows using dependencies to control process workflow across multiple agents.

The job that willl use multiple semaphore file to trigger the running of a .exe will be created using a JDL I guess using the SCHEDULE_TYPE="F" but which parameter do I use to specify the semaphore files and how do I specify multiple files that must exist for the job to run?

Thanks

I have created the following JDL file which looks fine when I run it and view the properties of the job in the GUI but the semaphore file is not getting created at the end of the job but I cannot see why? Thanks in advance for your help.

ACCOUNT=AAA@AAAA.co.uk
ASYNC=N
COMMAND="C:\Program Files\AAAAAA\SSSSSS.exe"
DESCRIPTION=Description - 13:41:04 16/07/2009
START_IN=C:\Program Files\Web Content Extractor\
HOST=
QUEUE=MASTER
BACKUP_HOST=master
BACKUP_AGENT=master
NAME=CS34
DISABLE_ON_ERROR=Y
DISABLED=N
DETACHED=Y
DELETE_RULE=D
IGNORE_ERRORS=N
JOB_TYPE=P
MSG_ACCOUNT=AAA@AAAA.co.uk
MSG_EMAIL=Y
MSG_FINISH=N
MSG_RECIPIENT=bbbb@dffff.com
MSG_FILE=Y
MSG_FILE_NAME=C:\Semaphore\CS34.chk
MSG_START=N
MSG_ERROR=Y
MSG_ACTIONS=EE,FF
MSG_PASSWORD=123
NUMBER_OF_RETRIES=1
MSG_PASSWORD=123
PRIORITY=-1
REBOOT=N
SCHEDULE_TYPE=O
START_DATE=16/07/2009
START_TIME=13:41:04
WINDOW=I
TIMEOUT=3

I think the problem is caused by properties with invalid values, for example, PRIORITY=-1, HOST=
Please specify only the properties that you need, and don't refer to properties for which you are can use default values

It also looks like you want to have MSG_FINISH=Y - you want the semaphore created on finish and email sent on error. Am I right?

I can understand what you are saying about the HOST but am confused by the PRIORITY as I set this to -1 (low) on my scheduler server and 1 (high) on my dedicated 24x7 remote agents which gives me the best of both worlds so not sure why you think this might be wrong? You are right in that I want the email sent on error and the semaphore send on job finishing.

Your were right in that I should have used the MSG_FINISH=Y, thanks for all your help. I have also just got a remote agent running the other side of the world being controlled from a master scheduler using TCP and it is so cool and the amount of data being sent is very very small no noticible delay. Thanks again.

I'm glad you’ve got it working. Please disregard my comment about PRIORITY, I’ve got confused, thought that valid values were 0, 1, 2 etc… and negative 1 was an invalid number.