One scheduler instance, some jobs run twice

We are using 24x7 in graphical mode. We can see only one javaw.exe in task manager when listing processes from all users.
Still some jobs are run twice seconds apart.
For example, one job is scheduled to run every hour. At 2:00 it ran only once. At 3:00 and 4:00 it ran twice each time. Again at 5:00 it ran only once. No specific pattern.
It was set as an asynchronous job. Read the forum and tried it as synchronous job, still runs twice.
Now I have set the property to only run one instance of the job and do nothing if there is another instance in the queue and here is the log:
16-Jun-2014 05:00:00 PM 2 lZTQDB0IurGyQU7VObRKXo6uwbE= 89 JOB_A Job started.
16-Jun-2014 05:00:00 PM 2 lZTQDB0IurGyQU7VObRKXo6uwbE= 95 Job_B Job started.
16-Jun-2014 05:00:00 PM 2 lZTQDB0IurGyQU7VObRKXo6uwbE= 2 Job_C Job started.
16-Jun-2014 05:00:01 PM 2 lZTQDB0IurGyQU7VObRKXo6uwbE= 62 Job_D Job started.
16-Jun-2014 05:00:01 PM 2 lZTQDB0IurGyQU7VObRKXo6uwbE= 61 Job_E Job started.
16-Jun-2014 05:00:01 PM 1 lZTQDB0IurGyQU7VObRKXo6uwbE= 89 JOB_A An instance of job "JOB_A" is already running or queued. Job settings do not allow adding more instances.
16-Jun-2014 05:00:02 PM 2 lZTQDB0IurGyQU7VObRKXo6uwbE= 2 JOB_C Job started.

JOB_A and JOB_C ran twice!

Any idea what is the problem with my scheduler?
I'm using version V5.1 Build 402

It's not mentioned in this forum but we're running 5.1.403. I remember some scheduling issues in the previous version.

Having said that, every month or so the queues in this version get stuck so jobs pile up, so about to try the 4.2 series.

	Quote:
	Still some jobs are run twice seconds apart

How often do you see this happening? Every day? Same time or random times?

It happens every day but not on same times. Yesterday it ran twice at 7:00 and 10:00 only. Today it ran twice at 7:00 and 8:00.

Is it always on a hour? Is this issue recurring for just one job or multiple jobs?

It is happening for multiple jobs.
For example one is scheduled to run every hour, another one is scheduled to run at 9:10, 9:40, ..., 4:10, 4:40.

Do you have 2 instances of the scheduler running concurrently by any chance? For example one running as a service or daemon process and another one started by someone in GUI mode using remote session or something?

I'm now seeing this under 5.2.417. Here we have 2 fast jobs which are scheduled to run every 15 minutes:

Code:

20-Jun-2014 08:30:00 AM 2 O2MfZQ+yPxYL/FuwWawsTALECuI= 827 Management Misc WMITracker Job started.
20-Jun-2014 08:30:01 AM 1 O2MfZQ+yPxYL/FuwWawsTALECuI= 827 Management Misc WMITracker An instance of job "Management Misc WMITracker" is already running or queued. Job settings do not allow adding more instances.
20-Jun-2014 08:30:01 AM 1 O2MfZQ+yPxYL/FuwWawsTALECuI= 565 Management Misc ScratchTracker An instance of job "Management Misc ScratchTracker" is already running or queued. Job settings do not allow adding more instances.
20-Jun-2014 08:30:03 AM 2 O2MfZQ+yPxYL/FuwWawsTALECuI= 827 Management Misc WMITracker Job completed with exit code 0, exit code condition satisfied.
20-Jun-2014 08:30:03 AM 2 O2MfZQ+yPxYL/FuwWawsTALECuI= 565 Management Misc ScratchTracker Job started.
20-Jun-2014 08:30:05 AM 2 O2MfZQ+yPxYL/FuwWawsTALECuI= 565 Management Misc ScratchTracker Job completed with exit code 0, exit code condition satisfied.

But it is not running them twice?

Correct, so maybe only bogus warning message.

They don't look bogus to me, it's just that the queue controls stop the job from running multiple instances concurrently. Does this issue affect all jobs? Do you see any pattern there?

	Quote:
	Do you have 2 instances of the scheduler running concurrently by any chance?

No as I mentioned in my original question we are only running one instance in graphical mode.

How do you explain the random behavior if we were running two instances?

	Quote:
	They don't look bogus to me

Why is the scheduler triggering the job twice in first place? It first triggers the job seconds apart from the first trigger then stops it because only one instance is allowed. I don't get why two triggers?[/quote]

Don't know that. Something is causing that. If I knew the answer or it was obvious, I wouldn't ask you all these questions. Let's try figuring out the root cause.
is there a way to monitor system performance using performance monitor or other utility to find out if duplicate job runs align with high CPU usage? How many CPU cores do you have on that system?

We only have one CPU on that server.

It has 90%-100% cpu usage at the time of multiple runs, which I assume is reasonable as there are at least 3 jobs starting to run at that time. They will all increase CPU usage.