Web Based Console issues and suggestions

So, we've recently purchased the web based management console to manage our multi-platform edition 24x7 schedulers. I've run into a few things that raise questions and some features that aren't there that I'd like to suggest be implemented.

I'm using the jsp addition at the moment. I'm attaching to the latest multi-platform edition of the scheduler.

1. I'm having some issues with the web interface being stable. More often than not, it seems, the login screen will come up fine. But after filling in the login information and pressing the login button the application appears to just hang. The frame layout is built in the browser. I can see where the top tool bar should be. I can see where the job list should be. And I can see the frames for the individual job tool bar and the job information area. But nothing ever gets populated. I've let the browser sit over night before and it just never loads. I've restarted both the scheduler and the web application many times. I've used different accounts to login. Nothing seems to help the issue. And then every once in awhile it will juts magically work and the app loads successfully. Are there any known issues that might be related to this? Something in my configuration?

I'm running the console under BEA Weblogic 8.1 using java 1.4.2_05. It's an old environment. I've attempted to move to java 1.5 and 1.6. But neither of these are compatible with the version of weblogic I'm using. Any known issues relating to java versions?

I did not see anything in the configuration documentation that talked about memory requirements. I started the application off with 128MB allocated to it. I've since bumped that up to 512MB to see if it would make a difference. What is the recommendation from SoftTree for this setting?

2. Once I am able to successfully get the console to come up I am having issues with editing jobs. All jobs come up with the same error when I try to edit them. The first error states "Original template for this job is not available. Would you like to edit this job with the Job Wizard?". After selecting OK to go ahead and edit the job anyways I receive a second error that simply says 'Bad format string "'. The double quote is part of the error message. This happens with all jobs that I have tried. We do not typically use the blank job template to create jobs and do not use any custom templates. This error has been seen on jobs migrated from older schedulers, new jobs created on the current scheduler and even the sample jobs that come with the scheduler by default. I can create a new job from the web console and as soon as it's created I try to edit it. Same errors.

3. Suggestion -- There's no way that I've seen to edit the users/groups from within the web console. The only way I see to do it is to interactivly launch the scheduler on the server and modify the user list from the GUI interface. This is cumbersome do to the fact that my schedulers run as windows services. This forces me to shutdown the service and run the scheduler interactivly for maintenance. Exactly what I was trying to avoid by implementing the web console.

4. Suggestion -- It would appear that the only way to allow access to the scheduler via the web console is on a per user basis. It would be nice to be able to manage this by groups instead. My goal is to allow our developers access to the scheduler environments via the web console. With an ever changing work force this adds unnecessary work having to individually add them to each scheduler. Not sure how easy that would be. But there it is.

5. Suggestion -- The ability to import jobs from remote schedulers does not appear to be available via the web console. We run the scheduler in a staged environment where jobs are deployed to a development environment, moved to a staging environment and finally into production. Iimporting the jobs into each new environment would be the best way to handle this. But it looks like this is another task that will require using the interactive GUI of the scheduler on the server itself.

And lastly. One minor thing not related to the web console. The documentation lists four default security groups which are provided with the scheduler. The documentation also mentions the ability to modify the permissions associated with these groups. I see how to do this on the windows edition of the scheduler but not the multi-platform edition. Am I just missing it? Also, is there an ability to create additional groups other than the default four?

And one more thing to add to this. I've had one of my browser sessions hung up while typing this (related to problem #1) and happened to have the logs open. There have been a couple of errors generated in the logs. It almost looks like it's bombing on loading some css files. But that doesn't make a lot of sense consider sometimes it works. The error messages are as follows:

####<Jul 16, 2009 12:35:07 PM EDT> <Debug> <HTTP> <AUSLYNCWS07> <24x7WebConsole> <ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <BEA-101147> <HttpServer(24645424,null default ctx,24x7WebConsole) Found no context for "/css/main.css". This request does not match the context path for any installed Web applications, and there is no default Web application configured.>
####<Jul 16, 2009 12:35:07 PM EDT> <Debug> <HTTP> <AUSLYNCWS07> <24x7WebConsole> <ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <BEA-101147> <HttpServer(24645424,null default ctx,24x7WebConsole) Found no context for "/css/main.css". This request does not match the context path for any installed Web applications, and there is no default Web application configured.>
####<Jul 16, 2009 12:54:11 PM EDT> <Error> <WebLogicServer> <AUSLYNCWS07> <24x7WebConsole> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '23' for queue: 'weblogic.kernel.Default' has been busy for "743" seconds working on the request "Http Request: /jwi24x7/menu.jsp", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
####<Jul 16, 2009 12:54:11 PM EDT> <Error> <WebLogicServer> <AUSLYNCWS07> <24x7WebConsole> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '24' for queue: 'weblogic.kernel.Default' has been busy for "743" seconds working on the request "Http Request: /jwi24x7/job_tree.jsp", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
####<Jul 16, 2009 12:58:34 PM EDT> <Debug> <HTTP> <AUSLYNCWS07> <24x7WebConsole> <ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <BEA-101147> <HttpServer(24645424,null default ctx,24x7WebConsole) Found no context for "/css/main.css". This request does not match the context path for any installed Web applications, and there is no default Web application configured.>
####<Jul 16, 2009 12:58:47 PM EDT> <Debug> <HTTP> <AUSLYNCWS07> <24x7WebConsole> <ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <BEA-101147> <HttpServer(24645424,null default ctx,24x7WebConsole) Found no context for "/css/main.css". This request does not match the context path for any installed Web applications, and there is no default Web application configured.>
####<Jul 16, 2009 1:04:11 PM EDT> <Error> <WebLogicServer> <AUSLYNCWS07> <24x7WebConsole> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '23' for queue: 'weblogic.kernel.Default' has been busy for "1,343" seconds working on the request "Http Request: /jwi24x7/menu.jsp", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
####<Jul 16, 2009 1:04:11 PM EDT> <Error> <WebLogicServer> <AUSLYNCWS07> <24x7WebConsole> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '24' for queue: 'weblogic.kernel.Default' has been busy for "1,343" seconds working on the request "Http Request: /jwi24x7/job_tree.jsp", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>

Thanks for your valuable feedback.

Please note that issue #1 is caused by a known bug in Tomcat/JSP engine for which a workaround is available. I'll try to find it for you and post it here.

There is no recommended memory use – it depends on the web server type, Java version, number of concurrent users, and some other factors. In general, default web server settings should be ok to support several concurrent users accessing the console. In case you need to handle tens or hundreds or concurrent users, or more users, you would surely need to bump up the memory given to web server.

Issue #2 - this sounds to me as a bug, I mean the 'Bad format string '

Let’s discuss this separately or if you want, you can open a support case. It sounds like the issue is data related so we would need to get a copy of the job definition that is causing the error.

Thanks for the information. Hopefully the fix for issue #1 is not Tomcat specific. As I mentioned before, I am currently running the application under Weblogic. But if the fix can be applied to Weblogic as well then it's all good. Good to know that Tomcat can be an issue, as well. I was getting ready to throw up a linux server with Tomcat on it just to see if it would behave better under that.

Issue #2 can be reproduced with a newly installed scheduler where no jobs other than the provided sample jobs are present. Would you like me to start a new thread for this issue?

Here is what I found for the workaround I mentioned in the previous reply. You can modify code of login.jsp file and add file delete command for temporary work files generated by your web server. This login.jsp can be found in the web console deployment directory. After the second <% tag (5th line) insert "delete" code as in the following example. This example is for Apache Tomcat web server running on Windows, so you would also need to correct the delete command for your environment and file location.

	Code:
	try { Runtime.getRuntime().exec ("cmd /C del /F /S /Q \"D:\\Program Files\\Apache Software Foundation\\Tomcat 5.0\\work\\Catalina\\localhost\\24x7\\.\""); } catch(Exception ex) { /* nothing we can do here, just ignore */ }

Alright, so far I'm just continuing to have problems with this version of the console.

I attempted to adapt your suggested fix to our weblogic environment without success. From what I've been able to figure out it appears weblogic keeps it's equivalent working files open and locked such that they cannot be deleted while the app is running. So no luck there.

I have been able to get the app running under two other platforms running tomcat and was able to get the login.jsp page to delete the contents of the work directory each time the page is loaded. This appeared to work at first. But sadly I'm stuck right back where I was. I have not been able to successfully log into a scheduler since yesterday afternoon. I let the app sit idle all night lastnight hoping whatever it was stuck doing would clear up if given some time. No joy.

Right now I have the console running under the following environments:

Environment #1

Environment #2

Environment #3

All three environments exhibit the same problem. After providing server/username/password and selecting login. The page builds four frames and then just sits there attempting to load the content. Nothing else happens. Restarting the app, tomcat, weblogic have no impact.

I have verified that the files under the work directory are definitely being deleted. I placed a test file there and loaded the login.jsp page. The file gets deleted.

I've run jconsole against the java process running under tomcat to monitor resource usage. Everything appears normal there. Heap/Memory/Thread utilization looks normal.

Is there a known good environment that the console has been tested and verified on? Could it be something to do with the schedulers I'm attaching to? All the scheduler environments I'm attaching to are running the Multi-Platform edition under Windows 2008.

I'm about ready to ditch this thing and attempt the ASP version. Any other suggestions?

This is not caused by the scheduler. Some kind of internal locking occurs in the web server JVM. It might be waiting for some resources it didn't deallocate clearly during the previous session. I'm speculating as the cause is not exactly known.

When do you see this happening? When you login, then logout, and login again? Can you reproduce it if you login, logout and close browser, open new web browser and login again?

Typically I can only ever get one login to connect successfully if I can get one to connect at all. Yesterday, after implementing the previously suggested fix of clearing the work directory I was able to log in twice. Once each to two different schedulers. The first two logins were using IE6. I attempted a third connection using firefox and it hung. I have not been able to successfully log in since then.

I've have been able to produce the problem with a clean startup. One successful login. Logout. A seocnd login attempt from the same browser hangs.

Closing all browsers and starting fresh does not help. I shutdown my workstation lastnight and powered it on fresh this morning. I still cannot log in. I have restarted the tomcat service several times. This does not help. I stlil cannot log in.

About the only thing I haven't done is restart the server itself. Although that seems a bit extreme.

Hi What would be the command to execute on Linux for issue #1 as you had listed above?

Thanks.

You mean the adaptation of the jsp code that was provided? Tomcat under linux didn't seem to like calling the command directly so I put it into a script file and called that instead. I created this script file

/usr/share/tomcat5/bin/cleanup:

	Code:
	#!/bin/bash rm -rf /var/cache/tomcat5/work/Catalina/localhost/jwi24x7/*

And added this to the login.jsp page in place of the above suggested code:

	Code:
	try { Runtime.getRuntime().exec ("/usr/share/tomcat5/bin/cleanup"); } catch(Exception ex) { /* nothing we can do here, just ignore */ }

Seems to work good.

Thanks!

Several people have confirmed that the latest version of the console has somewhat fixed that problem for them. Some get an error message about broken communication after logout, some don't, but in the end they can login again without closing the browser. Please check if the latest version works for you, and let us know what you get if doesn't work for you.

Here is the link to brief release notes for the latest version http://www.softtreetech.com/support/phpBB2/viewtopic.php?t=22962

I'd agree with the "somewhat" portion of the fix. I did try the new version of the jsp web interface and found that I had more success with logging in. However, it does still go into the mode of locking up on login. Just not quite as frequently.

Well, the developers have been suffering through this random logon issue with the JSP scheduler for awhile. It's a little painful. But some features are missing from the ASP version. So, they've lived with it. We recently upgraded them to IE8 and for some reason the problem seemed to get worse. So I took another crack at trying to figure out what was going on. I think I may have stumbled across the problem. Not sure what the final fix might be. But this work around is effective for me at the moment.

The hang appears to be happening when the job_tree.asp is being built. I'm not sure if there are multiple things trying to iterate jobs at the same time in the underlying code when building the initial page. But it appears to be causing some sort of lock situation that causes the page to never load completely. So, here's what I did. I added a sleep statement to the job_tree.asp page.

The sleep statement was added at line 64. So this:

	Code:
	try { folders = jobResources.getJobsTreeStructure(user, session.getAttribute ("sched_type").equals("cscheduler")); }

becomes this:

	Code:
	try { Thread.sleep(1000); folders = jobResources.getJobsTreeStructure(user, session.getAttribute ("sched_type").equals("cscheduler")); }

Ever since this change, I can log in successfully every time.

Just out of curiosity, since no comment has been received, can this issue be confirmed? And will it be addressed in a later version of the web front-end?

Here is what may actually help. If you are running web console using Apache Tomcat web server, please try 2 configuration changes

1. Allow web server to use more memory for applications then the ridiculously low default setup for heap size
On Unix\Linux systems, in Tomcat's [Tomcat home]/bin/startup.sh add in the beginning
export CATALINA_OPTS="-Xms256m -Xmx512m"
on Windows systems, in Tomcat's [Tomcat home]\bin\startup.bat add in the beginning
set CATALINA_OPTS=-Xms256m -Xmx512m

2. Change request/response timeout value for endpoints
On Unix\Linux systems, in Tomcat's [Tomcat home]/conf/server.xml find line beginning with connectionTimeout and change the value to
connectionTimeout="2000000000"
on Windows systems, in Tomcat's [Tomcat home]\conf\server.xml find line beginning with connectionTimeout and change the value to
connectionTimeout="2000000000"

Restart Tomcat after the changes.