Event log errors

Hi, in the event log of my 24x7 server I have 40 or 50 instances of

Reporting queued error: faulting application 24X7.EXE, version 3.4.26.0, faulting module PBVM70.DLL, version 7.0.3.10095, fault address 0x001db9e5.

Ever seen this before?

Yes, this looks familiar. Yet there is no simple and exact explanation or solution. It is a generic error indicating that the process crashed in PBVM70.DLL. Likely one of the jobs is generating exceptions causing the process to crash. Try to correlate timing of these events with job runs to find out which job is generating these.

It doesn't look like it's caused by a particular job, the error seems to come up after the server has been rebooted

Maybe during reboot? While the scheduler is shutting down and trying to forcedly abort running jobs or queues?

could be although it doesn't happen every time the server is rebooted. There weren't any running jobs as such although several will have been file-watching.

That one seems to have gone away; how bout this one though?

Event Type: Information
Event Source: Application Popup
Event Category: None
Event ID: 26
Date: 16/07/2007
Time: 4:04:28 p.m.
User: N/A
Computer:
Description:
Application popup: 24x7.exe - Application Error : The instruction at "0x039c704d" referenced memory at "0x02064aa8". The memory could not be "read".

Click on OK to terminate the program

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

This is is the same kind. Does it also happen when the system is rebooted?

Because you get a message box in this case, you can probably see which job(s) has just finished, started or in a middle of processing? You can check schedule.log for this.

Schedule.log does show jobs running when the error comes up although it's not the same job each time and schedule.log displays no error.

It's not on a predictable timeframe but doesn't come up when jobs aren't running. 24x7 is running as a service under a user account so I can't see what the popup was.

Are these jobs running detached?

I would like to find out whether the error occurs in a job process during process termination or some other phase or in the scheduler itself. That's why I am asking to reconcile time of the event log message with the job timing and figure out which job(s) are running.

If you got these jobs running detached (which is highly recommended) you should be able to see in the schedule.log the job completion status and from there determine which job(s) cause this error. The next step would be to figure out what they are doing that leads to the error condition (for example, database connection closing, file access, email functions, etc…)

None of the jobs are running detached, there is one asynch job but again it's not the same every time.

Can they be set to run detached?

Here's some more information.

After this error, in task manager with show processes from all users checked, an additional 24x7.exe process shows. When I stop the 24x7 service, the extra process remains. The service is running as an account, so it can't interact with the desktop and can't show the message. The event log seems to tell us what it says though, and it's always the same source memory address, and usually the same referenced memory address.

So all jobs should be set to detached? As I said it does not happen on a particular job or series of jobs, it's an assortment. If it's not an issue for the program then it's not an issue for me.

Also, the action of all our current jobs is to run external perl scripts using runandwait and then use jobrun to run the next job.

When the error happens it's quite bizarre; the schedule.log has no idea that the job(s) that were running at the time, there's just no trace. Odder still is the fact that all the perl scripts do their own logging to a file which simply does not happen. The oddest thing of all is that the file actions in the perl script still happen as if everything was fine.

Windows runs services just like all other programs, just on a different desktop not shown to the user. If services display any messages, print to the console or if fatal errors occur and Windows itself attempts to show these messages on the desktop, you cannot see them because the entire desktop is not visible.

The described error type occurs usually in a third-party dll pbvm70.dll for which we have no source code, so knowing the address of the error is not helpful as we still cannot debug the process. However, if we can find what leads to this error, we can surely suggest how to avoid it.

Please set jobs to run detached. Not only this should make the system more stable because a single job failure cannot kill the entire system, but it can also help to eliminate many points of job process contentions. Please be careful with changing jobs to detached. Script jobs using global variables, if run detached will have no access to the global shared memory. Such jobs can be modified to use shared files or system registry as a place to store shared settings. Other then global variables, nothing else is different for detached jobs.