java.lang.OutOfMemoryError: PermGen space

Dear Support Team,

I've got the following entry/error in debug.log:

2014-08-06 10:03:00,155 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.JavaScriptJobRunner - runJob(): starting JavaScript
2014-08-06 10:03:00,155 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.JavaScriptJobRunner - runJob(): job not using user authentication
2014-08-06 10:03:00,155 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptThreadImpl - evaluateScript(): starting jsthread
2014-08-06 10:03:00,155 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptThreadImpl - evaluateScript(): waiting
2014-08-06 10:03:00,155 [JavaScript] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptThreadImpl - jsthread started
2014-08-06 10:03:00,155 [JavaScript] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptRunner - runScript: JavaScriptRunner
2014-08-06 10:03:00,155 [JavaScript] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptRunner - Connecting to database...
2014-08-06 10:03:00,155 [JavaScript] DEBUG com.softtreetech.jscheduler.business.sql.SqlConnector - loadDriver() driverPath=./drivers/ora/ojdbc14_g.jar driverClass=oracle.jdbc.OracleDriver
2014-08-06 10:03:00,155 [JavaScript] DEBUG com.softtreetech.jscheduler.business.sql.SqlConnector - parsed urls: 1
2014-08-06 10:03:00,155 [JavaScript] DEBUG com.softtreetech.jscheduler.business.sql.SqlConnector - url #0 file:/C:/Program Files/24x7/drivers/ora/ojdbc14_g.jar
2014-08-06 10:03:00,155 [JavaScript] DEBUG com.softtreetech.jscheduler.business.sql.SqlConnector - java.net.URLClassLoader@1846044
2014-08-06 10:03:00,420 [JavaScript] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptThreadImpl - java.lang.OutOfMemoryError: PermGen space
2014-08-06 10:03:00,420 [JavaScript] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptThreadImpl - jsthread stopped
2014-08-06 10:03:00,420 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptThreadImpl - evaluateScript(): script terminated with exception: java.lang.OutOfMemoryError: PermGen space
2014-08-06 10:03:00,420 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.JavaScriptJobRunner - runJob(): script evaluated
2014-08-06 10:03:00,420 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.JavaScriptJobRunner - runJob(): stopping local js thread
2014-08-06 10:03:00,420 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptThreadImpl - stopJScript()
2014-08-06 10:03:00,420 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.JavaScriptJobRunner - runJob(): job finished
2014-08-06 10:03:00,420 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.queue.JobQueue - QUEUE_UNLOCKED

running the following (schedule polling 1min) script:
( i'm looking in a table to get schedulerId entered by other programs and want to execute them)
This works fine, but after a while the scheduler hangs up, i cant close the tool anymore and have to kill it (Java (TM) Platform SE Binary) via taskmanager)

var job_id=0;
var internal_id=0;
var ret_val_str="";
var sql="";

Database.connect("DWH");

sql = "select ref.TAB_KEY from STG01.Z_DWH_SCHEDULER_STEUERUNG s, ( select min(SOURCE_SERVER || '|' || SOURCE_JOBID || '|' || TARGET_SERVER || '|' || TARGET_JOBID || '|' || to_char(JOB_INSERT_TIME,'yyyymmddHH24MISS')) as TAB_KEY from STG01.Z_DWH_SCHEDULER_STEUERUNG where TARGET_SERVER = 'bdopis1' and JOB_STATUS = 1 ) ref where ref.TAB_KEY = (SOURCE_SERVER || '|' || SOURCE_JOBID || '|' || TARGET_SERVER || '|' || TARGET_JOBID || '|' || to_char(JOB_INSERT_TIME,'yyyymmddHH24MISS'))";
var tab_key = Database.retrieve(sql);

if (tab_key != "")
{
sql = "select TARGET_JOBID from STG01.Z_DWH_SCHEDULER_STEUERUNG s where '" + tab_key + "' = (SOURCE_SERVER || '|' || SOURCE_JOBID || '|' || TARGET_SERVER || '|' || TARGET_JOBID || '|' || to_char(JOB_INSERT_TIME,'yyyymmddHH24MISS'))";
ret_val_str = Database.retrieve(sql);
job_id = ret_val_str;

sql = "update STG01.Z_DWH_SCHEDULER_STEUERUNG set JOB_STATUS=2, JOB_EXECUTE_TIME=sysdate where '" + tab_key + "' = (SOURCE_SERVER || '|' || SOURCE_JOBID || '|' || TARGET_SERVER || '|' || TARGET_JOBID || '|' || to_char(JOB_INSERT_TIME,'yyyymmddHH24MISS'))";
ret_val = Database.execute(sql);
sql = "commit";
ret_val = Database.execute(sql);

var internal_id = Scheduler.runJob(job_id);
}

Database.disconnect();

Thank you
Dominik

How many records are returned by the database query?
Do you see jdbc.log in the 24x7 directory? If yes, is there any additional information in that related to that exception?

Just one record with one value (jobid to start) or nothing.

in jdbc.log we can often find (before crash) something like that:

Wed Aug 06 14:51:00 CEST 2014
Job: 24x7 Scheduler
----------------------------------
registerDriver: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@324b85]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@fb0b0]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@a85268]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@83b18f]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@12b9f79]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@e7ccd7]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@161401f]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@137fea]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@113e371]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@1b6956f]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@ede19e]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@161b0bc]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@11641cf]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@1313e0e]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@16d079a]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@811e18]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@1346812]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@1a06ec2]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@111c3f0]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@e3f02a]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@1ac1adf]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@199049]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@90fd20]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@4ba66]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@d8355]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@4e4fc4]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@13a0067]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@6e056e]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@b1404f]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@148f7e0]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@8c1852]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@37086a]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@1a5ee7c]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@e51510]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@174e4b3]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@16d3d51]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@1afe17b]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@fef26a]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@6bc5aa]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@9b1ac]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@15660ef]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@23a559]
skipping: driver[className=oracle.jdbc.driver.OracleDriver,oracle.jdbc.driver.OracleDriver@6412da]
DriverManager.deregisterDriver: oracle.jdbc.OracleDriver@a998c1
couldn't find driver to unload

Scheduler hang up again at 14:57 (at this same time is no log in jdbc) , this time the last message in debug.log was:

2014-08-06 14:57:00,006 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.JavaScriptJobRunner - runJob(): starting JavaScript
2014-08-06 14:57:00,006 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.JavaScriptJobRunner - runJob(): job not using user authentication
2014-08-06 14:57:00,006 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptThreadImpl - evaluateScript(): starting jsthread
2014-08-06 14:57:00,006 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptThreadImpl - evaluateScript(): waiting
2014-08-06 14:57:00,006 [JavaScript] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptThreadImpl - jsthread started
2014-08-06 14:57:00,006 [JavaScript] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptRunner - runScript: JavaScriptRunner
2014-08-06 14:57:00,006 [JavaScript] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptRunner - Connecting to database...
2014-08-06 14:57:00,006 [JavaScript] DEBUG com.softtreetech.jscheduler.business.sql.SqlConnector - loadDriver() driverPath=./drivers/ora/ojdbc14_g.jar driverClass=oracle.jdbc.OracleDriver
2014-08-06 14:57:00,006 [JavaScript] DEBUG com.softtreetech.jscheduler.business.sql.SqlConnector - parsed urls: 1
2014-08-06 14:57:00,021 [JavaScript] DEBUG com.softtreetech.jscheduler.business.sql.SqlConnector - url #0 file:/C:/Program Files/24x7/drivers/ora/ojdbc14_g.jar
2014-08-06 14:57:00,021 [JavaScript] DEBUG com.softtreetech.jscheduler.business.sql.SqlConnector - java.net.URLClassLoader@19dd520
2014-08-06 14:57:00,583 [JavaScript] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptThreadImpl - java.lang.OutOfMemoryError: PermGen space
2014-08-06 14:57:00,583 [JavaScript] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptThreadImpl - jsthread stopped
2014-08-06 14:57:00,583 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptThreadImpl - evaluateScript(): script terminated with exception: java.lang.OutOfMemoryError: PermGen space
2014-08-06 14:57:00,583 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.JavaScriptJobRunner - runJob(): script evaluated
2014-08-06 14:57:00,583 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.JavaScriptJobRunner - runJob(): stopping local js thread
2014-08-06 14:57:00,583 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.javascript.JavaScriptThreadImpl - stopJScript()
2014-08-06 14:57:00,583 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.runner.JavaScriptJobRunner - runJob(): job finished
2014-08-06 14:57:00,599 [Job #45 - MSI Remote Control - einfach Pollen] DEBUG com.softtreetech.jscheduler.business.queue.JobQueue - QUEUE_UNLOCKED

Please try modifying master.bat file or master.sh, whichever you use to start the scheduler and the line starting with Java, add at the end -XX:MaxPermSize=256m

This should increase the maximum allowed size of permanent heap memory.

if you run it as a service, the same would need to be added to the system registry as service parameters.

Hi,

i'll try it - thank you.

The job is just a Mini-(polling)Job whith no big data or other memory catching elements. So in my opinion it looks like a memory leak.
Is the parameter a solution or just a workaround, which extens the time until the problem will appear again?

Thank you
Dominik

Sorry, but I still have the same Problem.

I still have the same problem.
After 2-3h the 24x7 Scheduler hang up an has to be kill by taskmanager.

May it be the jdbc entry:
DriverManager.deregisterDriver: oracle.jdbc.OracleDriver@1a8e0bd
couldn't find driver to unload

??

I can't understand it. Driver is loading, SQL is executing. But the driver is not unloading.

Thank you
Dominik

I suggest 2 step approach
1. Setting a timeout for that job, so that the scheduler can clear it from the queue and it won't block other jobs
2. Trying to figure out the route cause leading to the out of memory error. Can you reproduce the issue if you comment out var internal_id = Scheduler.runJob(job_id); line?

I've googled a little bit and at the moment i think it's a problem between your 24x7 scheduler as an 32bit application and our 64bit (oracle) odbc driver.
What do you think abaout?

As far as I can see from your log, it doesn't use ODBC, it connects to your server directly using Oracle JDBC driver supplied with the scheduler.

One of the available options is to try upgrading to a more recent driver version, but it would make sense after we establish the issue is in the driver or between the driver and the scheduler. That's why I asked you to comment out "runJob" to exclude other factors. If after commenting "runJob" the issue is reproducible, then updating the driver would be next logical step, otherwise, it's something else.

	Quote:
	That's why I asked you to comment out "runJob" to exclude other factors.

I did it, but still have the problem

Thank you. This indicates that most likely the issue is caused by database operations. I suggest installing latest version of 24x7. latest versions are packaged with a different version of Oracle JDBC driver. Please install new version into a different folder, do not override your existing installation. Recreate DWH database connection settings. Recreate that particular job keeping runJob line commented out so as not to have the job to depend on any other job. Please try running it using the new version and see if that resolves the issue.