Error unmarshaling return header

We have started receiving the following errors, this is from the scheduler log, I have the debug log turned off:

25-Mar-2012 06:10:06 PM 3 5ou5AYjUgNmragQXMn/7WhhcR5A= 180 FTP 2 Gpg Crypt 1 Remote job failed. Error unmarshaling return header; nested exception is: java.net.SocketException: Connection reset
25-Mar-2012 06:10:07 PM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 180 FTP 2 Gpg Crypt 1 Error message sent.

27-Mar-2012 02:30:07 AM 3 5ou5AYjUgNmragQXMn/7WhhcR5A= 272 Tol_2 CC Settle Dates Remote job failed. Error unmarshaling return header; nested exception is: java.net.SocketException: Connection reset
27-Mar-2012 02:30:08 AM 1 5ou5AYjUgNmragQXMn/7WhhcR5A= 272 Tol_2 CC Settle Dates Job has been disabled!

27-Mar-2012 04:55:00 AM 2 5ou5AYjUgNmragQXMn/7WhhcR5A= 479 FTP 2 Mqueue Filter Agent "FTP2" contacted.
27-Mar-2012 04:55:01 AM 3 5ou5AYjUgNmragQXMn/7WhhcR5A= 356 Tol_5 B2B PMT Load Driver Remote job failed. Error unmarshaling return header; nested exception is: java.net.SocketException: Connection reset

27-Mar-2012 08:27:14 AM 3 5ou5AYjUgNmragQXMn/7WhhcR5A= 257 Tol_2 Daily Reports Remote job failed. Error unmarshaling return header; nested exception is: java.net.SocketException: Connection reset
27-Mar-2012 08:27:15 AM 1 5ou5AYjUgNmragQXMn/7WhhcR5A= 257 Tol_2 Daily Reports Job has been disabled!

We restarted Scheduler.

Can you give me more information on this?

Hi,

There appears to be some sort of network issue causing aborted socket connections between the scheduler and the agent. The errors say:
Remote job failed. Error unmarshaling return header; nested exception is: java.net.SocketException: Connection reset

Please try restarting the affected 24x7 agent and check if that resolves the issue. If it doesn't help, you may need to reboot the entire agent server. If that doesn't help, try restarting the scheduler, and if that doesn't help, check with network admins what they can do to troubleshoot the connectivity issue.

Last edited by SysOp on Wed Mar 28, 2012 1:03 pm; edited 1 time in total

The error occured on three different servers. FTP2, TOL2, and TOL5.

Prior to opening this thead we restarted scheduler.

We have not received any unmarshaling return header messages since that time which was about 1000.

Hi. What type of system do you use to run the scheduler? Is that a Windows system or *nix? Which version?

we are running a combination of linux machines

scheduler is running on:
jim.owens@jscheduler-master:/$ uname -r
2.6.18-2-pve
jim.owens@jscheduler-master:/$ uname -a
Linux jscheduler-master 2.6.18-2-pve #1 SMP Mon Feb 1 10:45:26 CET 2010 x86_64 GNU/Linux

agent machines having the unmarshaling issues:
Linux tol2.ipayxepay.net 2.6.9-5.ELsmp #1 SMP Wed Jan 5 19:30:39 EST 2005 i686 i686 i386 GNU/Linux
[jim.owens@tol2 ~]$ uname -r
2.6.9-5.ELsmp

Linux tol5a.ipayxepay.net 2.6.18-8.el5 #1 SMP Fri Jan 26 14:15:14 EST 2007 x86_64 x86_64 x86_64 GNU/Linux
[jim.owens@tol5a ~]$ uname -r
2.6.18-8.el5

Thank you. I goggled for that configuration and found some pointers to bugs in JRE 1.5 for Linux. For example, http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4972794
If this error keep recurring often, consider updating JRE on the scheduler system to a more recent version, or alternatively migrating to 24x7 v5 which uses local JRE 1.6, in which that bug is supposed to be fixed.

I also highly recommend comparing existing JRE versions on the scheduler and agent machines and making sure they are the same, including the minor version number. If they are not, please update them to make them the same.