Scheduler Failure

Over the weekend we had a problem with 24x7 scheduler (Windows V3.4.24) and I'm trying to determine what caused the failure. The scheduler is running as a service, and the box was re-booted at 4:00pm. It appears 24x7 re-started fine. It ran the first scheduled job at 7pm fine. The next job runs off a trigger file, it started and submitted job to the queue but never shows it finishing in 24x7. After that nothing ran, until I restarted the service the next day. I checked the application log for the job that started and it finished successfully in a few minutes but that was never shown in the 24x7 log.

An additional note on this, we run in distributed server mode using port 1096. After the re-boot, apparently port 1096 was used by something else on the server because I could no longer communicate using that port. I had to switch to port 1097 to use the remote control agent.

Any ideas what would cause this, the only unusual thing was the addition of the re-boot at 4pm? Here is the log for that time period:

10/11/2008 08:30:00 0 217 0 Obsolete Rollups in Planning Job started.
10/11/2008 08:30:01 0 217 0 Obsolete Rollups in Planning Job finished.
10/11/2008 16:02:52 0 0 0 24x7 Scheduler [24x7 service] 24x7 Scheduler starting...
10/11/2008 19:00:00 0 207 0 EDWP MDM_SAP_DATA Update Job started.
10/11/2008 19:13:36 0 207 0 EDWP MDM_SAP_DATA Update Job finished.
10/11/2008 20:06:51 0 208 0 SAP to MDM Update Job started.
10/11/2008 20:06:51 0 208 0 SAP to MDM Update Semaphore file(s) \\hounafas003\etl_essbase01\MDM\Imports\SAPImport_complete.txt,\\hounafas003\etl_essbase01\MDM\Imports\SAPImport.txt found. Submitting job to the queue.
10/12/2008 21:50:29 0 0 0 24x7 Scheduler 24x7 Scheduler starting...
10/12/2008 21:51:05 0 0 0 24x7 Scheduler 24x7 Scheduler terminating...
10/12/2008 21:51:50 0 0 0 24x7 Scheduler 24x7 Scheduler starting...
10/12/2008 21:51:54 0 0 0 24x7 Scheduler 24x7 Scheduler terminating...
10/12/2008 21:52:25 0 0 0 24x7 Scheduler [24x7 service] 24x7 Scheduler starting...
10/12/2008 21:52:26 0 0 0 24x7 Scheduler [24x7 service] 24x7 Scheduler terminating...
10/12/2008 21:52:26:207 1 0 24x7 Scheduler Error starting listener on port: 1096 Distributed communications error.
10/12/2008 21:52:39 0 0 0 24x7 Scheduler 24x7 Scheduler starting...
10/12/2008 21:52:41 0 0 0 24x7 Scheduler 24x7 Scheduler terminating...
10/12/2008 21:52:50 0 0 0 24x7 Scheduler [24x7 service] 24x7 Scheduler starting...
10/12/2008 21:52:51 0 0 0 24x7 Scheduler [24x7 service] 24x7 Scheduler terminating...
10/12/2008 21:52:51:223 1 0 24x7 Scheduler Error starting listener on port: 1096 Distributed communications error.
10/12/2008 21:52:59 0 0 0 24x7 Scheduler 24x7 Scheduler starting...
10/12/2008 21:53:18 0 0 0 24x7 Scheduler 24x7 Scheduler terminating...
10/12/2008 21:53:19 0 0 0 24x7 Scheduler 24x7 Scheduler starting...
10/12/2008 21:53:26 0 0 0 24x7 Scheduler 24x7 Scheduler terminating...
10/12/2008 21:53:54 0 0 0 24x7 Scheduler [24x7 service] 24x7 Scheduler starting...
10/12/2008 21:53:55 0 208 0 SAP to MDM Update Job started.
10/12/2008 21:54:15 0 208 0 SAP to MDM Update Finish message sent.
10/12/2008 21:54:15 0 208 0 SAP to MDM Update Job finished.

Thanks,
Jeff

Why are there so many "scheduler starting" "scheduler terminating" messages? Are you trying to run it in the GUI mode while the service is running?
About the job in question, is that job set to run asynchronous?

Yes, When I couldn't connect via the remote agent due to port change, I connected via the GUI. I think I tried a couple of times before I figured out how to change the listening port. I also had trouble getting the service to stop and re-start at first. That is part of those messages.

No, the job is not set to run asynchronous mode.

Couple more specific questions...
What type of jobs are EDWP MDM_SAP_DATA? (external batch/program, internal script, etc...) and SAP to MDM Update? If these jobs are scripts, are they currently set to run detached?

They are both JAL script jobs, neither is set to "detached". Both jobs call other batch processes from within them using the RunAndWait command.

It seems like there was a problem with "SAP to MDM Update" job not releasing all resources and making it to get stuck in the queue on completion. As a result, it blocked all other jobs assigned to the same queue.

Suggestions:
1. Set all jobs to run detached to avoid such issues. There are only very few cases when a script job needs to run non-detached.
2. Upgrade to 3.4.33. This will give you more control over jobs and allow killing a stuck job without restarting the scheduler and affecting other jobs. This upgrade is free.
3. When running 24x7 as a service, you shouldn't be using the GUI version on the same system. You should be using either 24x7 Remote Control or the 24x7 web interface to manage the scheduler on-line, without service restarts and port conflicts.

Follow up questions to your suggestions:

1. If I change these to detached jobs, will they still "wait" in the queue for any jobs in front of them to complete first?

2. We normally don't use the GUI when running as a service, we use the remote control agent. However, as mentioned, if the listening port changes I believe you have to use the GUI to correct. You mention using the web interface, where is the documentation for that and is it available in version 3.4.24 or do we have to upgrade?

Thanks for your help!
Jeff

1. Yes this will. The detached mode doesn't affect how jobs are treated by the queue. This affects access to the scheduler memory and internal structures. Detached processes are run as separate system processes and isolated from the scheduler.

2. Yes and in that case after change the service should be restarted to pickup new settings.

For the documentation, see Windows Start Menu -> Programs -> 24x7 Automation 3 -> Documentation -> 24x7 Web-based Management Console Reference
It is also available on the web, http://www.softtree.com/24x7/24x7-documents.shtml

Per your suggestion, I decided to upgrade to 3.4.33 on our test server. I downloaded the upgrade files and ran the install. I got an error while it was trying to replace the file logcat.dll. I ran a process explorer session and it appears this file is locked by the wmiprvse.exe process. I aborted the install and it said it rolled it back but now 24x7 won't run at all. I re-booted and tried again, but got the same results. What now?

Thanks,
Jeff

Please close the Event Viewer and rerun the upgrade. It is the Event Viewer that is using and locking that DLL.

I don't have the event viewer running in the foreground, I did a re-boot and still get that error. Is there some other way to shut it down?

You can try renaming the file and then restarting the system. Whatever is using it, after restart won't be able to load and lock it.

PS. This resource DLL is not used by 24x7. It is there for external applications reading data from Windows event logs (or monitoring event logs)

I ended up stopping the wmiprvse.exe process which had the file locked and I was able to successfully install the new version. At the end of the install it wanted to re-start the server so I did. Now when I try to run 24x7 I am getting the error "Error starting listener on port:1098 Distributed communications error." If I click on ok it shuts down. It never gives me a chance to change the port.

I am not running this version as a service, how chan I change the port it is using. Evidently something else takes that port some times on a re-start.

I think I found my own answer, I edited the registry and changed the syncport value there to 1097 and it started fine.

Thanks for the status update.