Author |
Message |
soumitrabajpai
Joined: 23 Feb 2007 Posts: 8
|
|
Queue Full Error |
|
Hi,
We have a detached job running in a queue. There are no other jobs in the queue. Today we received an error "Queue is Full" and it backed up all the other queues
Any ideas?
Thank you,
Best Regards,
Soumitra
|
|
Fri Mar 16, 2007 10:27 am |
|
 |
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7966
|
|
|
|
Did you run out of disk space on the server?
Do you have a stagnant job in the queue that is blocking all subsequent jobs?
|
|
Fri Mar 16, 2007 10:36 am |
|
 |
soumitrabajpai
Joined: 23 Feb 2007 Posts: 8
|
|
|
|
No I guess disk space is not an issue, since we have another job monitoring all servers for disk spaces etc.
Stagnant Job -- not sure.
This is the only job that runs on its own queue. Like in this queue we have only one job. Since it runs once every minute endlessly we made it detached.
Also it executes a simple SP and returns retsults, like a monitor.
I mean an error like DB Connection timeout or failed wouldn't surprise me, but queue full when there is only one job is a bit unexpected.
Thank you,
Best Regards,
Soumitra
|
|
Fri Mar 16, 2007 10:43 am |
|
 |
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7966
|
|
|
|
Please check how many files you got in the queue directory [24x7 home]/Queue/[queue name] and if there are any, delete them all.
|
|
Fri Mar 16, 2007 11:40 am |
|
 |
soumitrabajpai
Joined: 23 Feb 2007 Posts: 8
|
|
|
|
I can see some 4500 file in the folder. I have cleaned the folder the job is working fine. Thank you.
Is there any reason for the files to get accumulated in the queue folder.
Any ways of preventing this in the future?
Thank you,
Best Regards,
Soumitra
|
|
Fri Mar 16, 2007 11:52 am |
|
 |
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7966
|
|
|
|
Stagnant job. A job that got stuck and blocked everything after that.
|
|
Fri Mar 16, 2007 1:45 pm |
|
 |
soumitrabajpai
Joined: 23 Feb 2007 Posts: 8
|
|
|
|
Hi,
Our jobs failed again today. Is there any way to find out which is the stagnant job causing the jobs to queue up.
Please find the Q file and the tmp file for your reference.
Also is there any way to decode the .tmp file?
[Q File Dump]
-------------------------------
03/16/2007 23:11:00 944 0 0 289 OPREM_1 (Process Operation Reminders) OperationReminders 1082 477 C:\Program Files\24x7 Automation 3\Queue\OperationReminders\jdqDFC2.tmp
[.tmp File Dump]
-------------------------------
xROo0!tAM!K
vSJ UT9!^㸲=u!qs; '~~~}X,~F{'DJt6#+3F NYn6s15)NxO䒠u_,
<o>21qf GhN1QZ.B$SDD `lɲqNeABB(%c
~^5iPueXGM
n2q=hYR}vlo:cVQqkpݭνMhD.G^|qx.}x遷wVߠmen@|]r]'|߁X_i1)
Thank you,
Best Regards,
Soumitra
Please note I have attachded the dump in the post only because I couldn't find any link to upload text file on the forum
|
|
Mon Mar 19, 2007 9:35 am |
|
 |
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7966
|
|
|
|
This binary data is encrypted and doesn't help at all. Why don't you simply open the graphical Queue Monitor? It will tell you which jobs are queued, which are running and in what state. The first job in the queue with the running status and time long ago would be the job that is blocking the queue.
|
|
Mon Mar 19, 2007 9:55 am |
|
 |
soumitrabajpai
Joined: 23 Feb 2007 Posts: 8
|
|
|
|
Hi,
Thanks for the reply. We did check the queue monitor and were able to free up the resources.
There is a problem here.
1. The Queue monitor is not available on the Remote Control. Therefore we have to log on to the production servers to access it from the Master Scheduler.
2. To get to the Master 24X7 scheduler we need to first stop the 24X7 service, which defeats our purpose.
a) Since terminating the queues causes many jobs in their various stages of prcocessing to be terminated. We have to then figure out which jobs to rerun
and from where.
b) Restarting 24X7 could mean that the jobs are cleared because of the restart and might clear the stagnant job on the queue.
Again we need some mitigation steps to prevent this from happening and better idea how to diagonse rogue jobs that cause the stagnation.
Best Regards,
Soumitra
|
|
Mon Mar 19, 2007 10:51 am |
|
 |
SysOp
Site Admin
Joined: 26 Nov 2006 Posts: 7966
|
|
|
|
If you want, you can upgrade your license to one that supports 24x7 web-based management console. The web-based console includes graphical queue monitor and it can be used with all versions of 24x7 Scheduler regardless of whether they are run in a service or graphical mode. The web server hosting 24x7 web-based console can be run on any computer on your network. Most of the web servers can be used. If you don't have one you can have Tomcat installed and configured in minutes.
With multi-platform version of the scheduler, you can also use 24x7 command-line based console. That kind of console also can print list of queued jobs and their states.
|
|
Mon Mar 19, 2007 11:32 am |
|
 |
|