Left files in Queue directory

Hi I got a notification that the queue files were backed up.
I looked at the debug.log file, but there was nothing there to see.

The job that hung this time started at 10:53 yesterday.

There were no entries around that time in the debug file.

I will send you log tomorrow but there was nothing in there around the time the queue got locked up.

Most interesting is that I hit the delete all in the queue monitor and it deleted every job except the one that was hung.

Here's an error I saw in the putty session:

srv_etl@pbdwjs31:/opt/24x7_Scheduler_prod/Queue/small_jobs> java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(Unknown Source)
at java.util.AbstractList$Itr.remove(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.notifyJobFinished(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.ÃO0000(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.Ã00000(Unknown Source)
at com.softtreetech.jscheduler.business.runner.AbstractJobRunner.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl.execute(Unknown Source)
at com.softtreetech.jscheduler.business.runner.JobExecutorImpl$1.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Another thing I noticed I don't know if this is the answer or not...
About a week ago I noticed this issue.

So I deleted the queue and then created another one and then re-assigned all of the jobs to that queue.

As soon as I did this, I noticed none of the jobs that are assigned to this queue are hanging.

Now the jobs that are hanging are in a different queue.

I doubt, that new queue issue is related directly. The exception your saw in the putty session is important and it refers to some concurrency issue. I bet that exception has caused the job not to start leaving it in limbo – not started and reported success/error to the scheduler. As a result on the scheduler side it got stuck in the queue and the queue making the queue wait forever for the success/error status.

To resolve this concurrency issue, please make jobs in that queue to run synchronously.

I will check all of the jobs, but I believe they are already running synchronously.

Could it be the issue is that 2 jobs are kicking off at the same time (very possible) and they are both fighting to grab the queue? The one that loses get locked up somehow?

Yes, it is sure possible, but the issue is not the queue, they compete on the agent side and that's where the issue is

Hi there I am still getting the concurrent error and then my queues back up. Earlier in the post you said:

"ConcurrentModificationException means that something is being removed from some collection or queue by one internal process while another internal processes is iterating through collection elements and validating their statuses. Something must be going on very fast or very slow or the collection is very very large and takes long time to check statuses of collection items. My guess, the job was set to run asynchronous and was deleted manually while still running. When the connection ended, some internal processes had a reference to the background job queue still open, didn't reopen that reference and tried to delete that job again. I don't think this is a critical issue, or at least I don't think it has any direct side effects.

As a side note I think this exception should not happen. Some internal method is not synchronized allowing other processes to modify/remove elements from the collection while it is still working on it – some mutex is missing somewhere. I am going to submit this issue to the support tracking system for further review."

Now I don't know if anything ever came out of this but can you tell me if any issue was resolved when you submitted this to the support tracking system?

I have the agent running in debug mode, but it deletes the entries very fast so by the time I realize this happens the entries are already removed.

Please advise.

Thanks.

Hi,

	Quote:
	Some internal method is not synchronized allowing other processes to modify/remove elements from the collection

This statement is generally correct.

For your information, in our tracking system this issue id is 24X7MP_1163. We are working on a new maintenance release for 24x7 and a fix for this issue should be available in that release - targeting second week of August.

Good to hear. Thanks for the update.