SoftTree Technologies SoftTree Technologies
Technical Support Forums
RegisterSearchFAQMemberlistUsergroupsLog in
24x7 Scheduler Use case?

 
Reply to topic    SoftTree Technologies Forum Index » 24x7 Scheduler, Event Server, Automation Suite View previous topic
View next topic
24x7 Scheduler Use case?
Author Message
gemisigo



Joined: 11 Mar 2010
Posts: 2102

Post 24x7 Scheduler Use case? Reply with quote
Original post : http://www.softtreetech.com/support/phpBB2/viewtopic.php?p=27524#27524

SysOp wrote:
Thank you for the big picture. Can we dive a bit more into technical details?

Yes, we can, though some of the details are not available yet. Also, I'm not familiar with this agent mechanism yet. I imagine the concept is that there are remote agents running on remote servers that can be connected using the GUI (running on a machine at our company). Jobs are created/copied/added locall and then propagated to those remote agents, which will run them according to schedules assigned (regardless of GUI connected or not) and the results can be monitored through the GUI (at a later time, if necessary). If these are not true the rest below might not apply.

SysOp wrote:

What results? where are they stored? If that is a SQL maintenance job, it's fairly easy to check with a simple SQL query pulling data fro msdb database, for example, query msdb.dbo.sysmaintplan_log and msdb.dbo.sysmaintplan_logdetail. I'm pretty sure you know what I'm talking about. Is that what you mean, or something else?

As for the SQL, the maintenance jobs are created as a mixture of pure SQL Server Agent scheduled jobs and Maintenance Plans. Those consists of regular database consistency checks, index reorganizations and rebuilds, statistics updates, full and transaction logs, log and history cleanups, updates to daily data, etc. Some of these were not created by us (though they should have been) and therefore they completely lack uniformity both in naming and scripting conventions and operation. I know we can get their results by querying remote msdb but it would be much more useful to be able to check, modify AND copy jobs to another server(s) without having to touch the database itself. Copying maintenance plan is a nightmare anyway. Most of them are already substituted by corresponding T-SQL statements run by SQL Server Agent, but modifying anything in them requires manually propagating the changes to other servers.

SysOp wrote:

What kind of OS maintenance tasks are you talking about? Do they log their statuses anywhere? report errors in windows event logs? other log files, If yes, you can use 24x7 to automate log scanning for errors or error statuses, group them bring to one central place. It can be done using agents, or WMI or something else, whatever works. Actually anything can be automated as long as it's clear what to run, what to check, where to look for results, how to read these results.

OS maintenance is somewhat more "read-only" involving much less modification. It consists of checking the cluster status, disk space, file sizes, application conditions and reading their logs. Most of them are simple scans of different log files.

SysOp wrote:

MySQL scheduled tasks can be checked too, mysql.information_schema.events table, etc...

While the on-board units cannot be connected directly through VPN, anything running on the main server can reach them. If the remote agent is running on that server it might query them. Occasional failure in network connection might be a problem, but it's definitely worth a try.

I have to read the docs a few times to get a better understanding how this thing works and what it is capable of. I'll be smarter tomorrow.
Thu Apr 26, 2012 7:39 pm View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7841

Post Reply with quote
Your understanding of the working of centralized scheduler and agents is more or less correct. What you described is a star schema. The topology can be more complex if required. For example, agents can talk to agents, push back to schedulers, there could be multiple schedulers talking to same agents, it's pretty flexible. The best option is not to run the scheduler in GUI mode on the remote system, run it as a service and use a web based 24x7 interface, which you can access from anywhere, even from your phone if security permits. If you devices can connect to the central scheduler, you can install 24x7 instances from there and make local jobs that in case of detecting processing errors, will push them to the central scheduler.

Here is a hint that might help you to figure out how to setup remote checking for statuses. To make a job run and check something remotely, for example, check content of a log file, and return the success/failure status back, create a script type job, batch or JavaScript, of VBScript, or JAL (all of them are supported by 24x7) , if it finds an error message in the monitored log, make the job fail, for example, return some unsuccessful error code or raise an error, and that would make the agent to return the job failure status back to the scheduler. You can them configure the job in case of an error to send an email to somebody alerting them about the error, or trigger another "fix it" job. Of course if the job doesn't find errors on the remote system, it would complete successfully and no emails would be sent.

To check some status in a database using a SQL job, use compound SQL statement or stored procedure with code like
IF EXISTS (SELECT some condition here…)
BEGIN
RAISERROR …some error message here
END

By the way, in case you need to check something in a database, there is no need to run it remotely as long as you can connect to the database from the central scheduler, that would be simpler and more reliable (less moving parts)


If you have any questions, please don't hesitate to ask. 24x7 is a very flexible and feature rich system with lots of functions and options. Figuring out all ins and outs may take a while.


Last edited by SysOp on Fri Apr 27, 2012 8:28 am; edited 1 time in total
Thu Apr 26, 2012 9:38 pm View user's profile Send private message
gemisigo



Joined: 11 Mar 2010
Posts: 2102

Post Reply with quote
SysOp wrote:
Your understanding of the working of centralized scheduler and agents is more or less correct. What you described is a star schema. The topology can be more complex if required. For example, agents can talk to agents, push back to schedulers, there could be multiple schedulers talking to same agents, it's pretty flexible. The best option is not to run the scheduler in GUI mode on the remote system, run it as a service and use a web based 24x7 interface, which you can access from anywhere, even from your phone if security permits. If you devices can connect to the central scheduler, you can install 24x7 instances from there and make local jobs that in case of detecting processing errors, will push them to the central scheduler.

--
Actually, the structure is like a three-level tree, where our server is the root (and where the central GUI should reside), deployed company servers are at next level and the vehicles are the leaves. Also, there's no way putting an agent on each of those vehicles as there are more than 1k of them, but if there can be an agent on the deployed server and it could run jobs on the on-board units, that would be awesome. Just to see clearly the terms: I suppose our server would be the "central scheduler" in this case with deployed servers running "remote agents", right?

Basically what we need is something that runs the jobs on deployed servers locally without the need to be connected to central scheduler but push the results back to it when there is connection and that jobs can be maintained from our server. Sending a warning/email from the agents while central scheduler is down would be nice. I guess that's feasible (based on what you said). I don't know, is a hybrid of using both web based interface and central scheduler attainable?

SysOp wrote:

By the way, in case you need to check something in a database, there is no need to run it remotely, as long as you can connect to the database from the central scheduler, you can do the same check from there

--
Yes, casual checks can be run that way but the whole management system must carry on even when the central scheduler is failing for whatever reason. Jobs must not fail because of that.

SysOp wrote:

If you have any questions, please don't hesitate to ask. 24x7 is a very flexible and feature rich system with lots of functions and options. Figuring out all ins and outs may take a while.

--
Many questions are open yet, e. g. how to configure agents to run in clustered environment? Is it possible at all? What happens when the agent cannot connect to an on-board unit (network failures are frequent because of poor network coverage, on-board unit disabled, etc.)?
Fri Apr 27, 2012 3:58 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7841

Post Reply with quote
Quote:
Just to see clearly the terms: I suppose our server would be the "central scheduler" in this case with deployed servers running "remote agents", right?

So far it fits your description of the task. with one consideration. If the volume of jobs per server is like in the dozens, you can setup them on the central scheduler and have the scheduler periodically execute them remotely using agents. If the volume is much higher, you don't want one scheduler to be running zillions of jobs and have one point of contention. You can run 'local" scheduler on each deployed server and make their jobs push job statuses to the central system, several ways to do it, so you can see all job statuses in one places. As you can see, there is a trade off, load balancing and failure management vs. managing job in multiple places.

Quote:
Yes, casual checks can be run that way but the whole management system must carry on even when the central scheduler is failing for whatever reason. Jobs must not fail because of that.

This means that you require the second scenario, multiple "local" schedulers pushing job statuses to one central place. By the way, that central place could be a table in some central repository database. the central scheduler could run a periodic job monitoring data in that table and triggering alerts or other chained events across multiple systems.

Quote:
how to configure agents to run in clustered environment? Is it possible at all?

Sure, install 24x7 on each node, install 24x7 Windows service (see output of 24x7srv.exe /?). Use windows cluster admin to configure service failover. Note that by default agents listen on port 1096. You will need to configure cluster to failover that port from a failing node to an active node.

Quote:
What happens when the agent cannot connect to an on-board unit (network failures are frequent because of poor network coverage, on-board unit disabled, etc.)?

It's up to you. Depends on how you want to handle it. For example, one of the options is to set the job to retry after a failure after certain period of time before giving up and reporting a complete failure. You can use the existing job properties to set number of retries and wait time between retries.

Hope this helps.
Fri Apr 27, 2012 8:43 am View user's profile Send private message
gemisigo



Joined: 11 Mar 2010
Posts: 2102

Post Reply with quote
Splendid!

SysOp wrote:

This means that you require the second scenario, multiple "local" schedulers pushing job statuses to one central place. By the way, that central place could be a table in some central repository database. the central scheduler could run a periodic job monitoring data in that table and triggering alerts or other chained events across multiple systems.

Okay, I finally understood what "pushing data" means. I can set up jobs that try to send arbitrary data (e.g. results of other jobs or whatever) from deployed servers to the central scheduler, including error handling for missing connections (retry x times, after y minutes/hours passed, etc.). Meaning that you're not doing that instead of me (as I initially thought) but you provide a tool that makes it easy to do. Very clever and versatile. Though it won't work in this case because the connection is kinda "one-way" (the deployed server does not see the central scheduler) but at least I got the principle. Never mind, we can use the agent to do its job, that's one step forward. As far as I understand, agents store job outcomes in their standalone, local databases. I suppose there's a way for the central scheduler to query those somehow (so to say, "pull" instead of "push"), isn't there? If there's we can always check results when central scheduler is back online since they won't be lost.

SysOp wrote:

Sure, install 24x7 on each node, install 24x7 Windows service (see output of 24x7srv.exe /?). Use windows cluster admin to configure service failover. Note that by default agents listen on port 1096. You will need to configure cluster to failover that port from a failing node to an active node.

Ehm... It's a bit more complicated than that and at the moment I've no idea how to solve that. You see, normally the communication application (that collects data from on-boad units, besides doing a bunch of other things) and SQL Server are running on separate nodes. Therefore we would need an agent running on both nodes and only running the jobs for the appropriate service (or both in case one of them fails over to the other node). Or I can imagine another scenario with the agent running on one node but executing jobs for both. But I'm very far from that yet.

Also, in this case licensing would be per running agents + the central scheduler, am I correct?

I'll spend the rest of the week reading the manual, trying different features and collecting questions about them.

Your help is highly appreciated, thank you very much.
Fri Apr 27, 2012 9:30 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7841

Post Reply with quote
Quote:
As far as I understand, agents store job outcomes in their standalone, local databases. I suppose there's a way for the central scheduler to query those somehow (so to say, "pull" instead of "push"), isn't there? If there's we can always check results when central scheduler is back online since they won't be lost.

Actually not the case. To make things easier, 24x7 code is somewhat "componentized" if I may use that term. Agent and scheduler are different run-time modes in which different components get activated. In agent mode, 24x7 doesn't load job definitions, doesn't activate scheduling engine and some other components. Basically it doesn't run its own local jobs, it simply waits for instructions sent from remote schedulers (and other agents, the latest is a rare case, it's possible theoretically but used very rarely). In job run-time (job schedule on and picked by the scheduler), the scheduler connects to the agent specified in job properties (if its a remote job) and tells the agent what it wants the agent to do, transferring relevant parts of the job definition to the agent, it waits for the job completion, gets the completion status and errors back, and then executes notification actions attached to the job if there are any (send email, trigger other job, create semaphore file, execute db command, etc...)

Quote:
You see, normally the communication application (that collects data from on-boad units, besides doing a bunch of other things) and SQL Server are running on separate nodes. Therefore we would need an agent running on both nodes and only running the jobs for the appropriate service (or both in case one of them fails over to the other node). Or I can imagine another scenario with the agent running on one node but executing jobs for both. But I'm very far from that yet.

Can you map a common file share for both nodes so that the required files are node independent (or just copy them fro each node to a common share periodically) or add a "case" logic like "get node name, if node_A, read files from \\node_B\some_folder\some_file, otherwise read files from \\node_A\some_folder\some_file" or something similar?
another option is to use the cluster to configure drive mapping with failover, so that the mapping appears constant to each node? That subject is a bit beyond job scheduling. i'm not an expert in that area. But just in case you want to do some intelligent file mirroring, 24x7 supports built-in functions for file synchronization across multiple local/shared folders as well as across multiple systems via file scheduler-agent supported file replication (job-ftp-server file replication)
Fri Apr 27, 2012 10:35 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7841

Post Reply with quote
One more thing. A scheduler can be configured to act both as a scheduler and as an agent, so that it runs its own jobs as well as jobs sent from other schedulers, that's what the distributed server mode is for. That mode is implicit in agent mode, but not in scheduler mode. See Tools/Options menu, Network tab.
Fri Apr 27, 2012 10:39 am View user's profile Send private message
gemisigo



Joined: 11 Mar 2010
Posts: 2102

Post Reply with quote
Meaning that agent mode requires continuous connection with the scheduler? I'm sorry, I should have known that if I read the user guide first and asked questions later. That's exactly what we want to avoid. But this distributed server mode seems interesting. I still haven't read the manual, but I'm certainly going to do it this week as this thing is getting urgent. No further questions till then, I really should stop wasting your time.
Thu May 03, 2012 3:22 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7841

Post Reply with quote
Connections are handled much like database client connections to database servers. Open connection, run processes, wait for completion, close connection.

Here is an example of how it works for multiple concurrent jobs.

Perhaps we have 3 remote jobs A, B, C all setup on scheduler #1 for execution by agent #2. Job A starts at 9am and runs for 10 minutes. Job B starts at 9:05am and runs for 10 minutes. Job C starts at 9:06 and runs for 1 second.

1. At 9am the scheduler opens first connection to the agent and instructs the agent what to do, passing definition of job A. The connection remains open for the duration of job A (10 min)
2. At 9:05am the scheduler opens second connection to the agent and instructs the agent what to do, passing definition of job B. The connection remains open for the duration of job B (10 min)
3. At 9:06am the scheduler opens third connection to the agent and instructs the agent what to do, passing definition of job C. The connection remains open for the duration of job C (1 second). At this moment, there are 3 independent connections
4. At 9:06:01am job C completes, and the third connection gets closed. 2 connections are still left active for jobs A and B.
5. At 9:10am job A completes, and the first connection gets closed. 1 connection is still left active for job B.
6. At 9:15am job B completes, and the second connection gets closed. At this point no connections are open between the scheduler and the agent.
Thu May 03, 2012 10:14 am View user's profile Send private message
SysOp
Site Admin


Joined: 26 Nov 2006
Posts: 7841

Post Reply with quote
And one more thing. A remote job can be designed to start external process and quit not waiting for the started process to complete. In this case the job completes immediately and its connection is closed with the job. A second job can be setup to connect to the same agent later and check the outcome of the process started by the previous job, for example, check some log files or database data.. As you can see, no connections are required in between.
Thu May 03, 2012 10:17 am View user's profile Send private message
Display posts from previous:    
Reply to topic    SoftTree Technologies Forum Index » 24x7 Scheduler, Event Server, Automation Suite All times are GMT - 4 Hours
Page 1 of 1

 
Jump to: 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


 

 

Powered by phpBB © 2001, 2005 phpBB Group
Design by Freestyle XL / Flowers Online.