Month: June 2012

11.1.2.1 FR Service Crashes

I have a client that uses Financial Reporting a lot.  When their system has a heavy load, the FR service tends to run out of memory.  We have adjusted the Java heap setting up as high as we can, but after awhile, they still get an “out of memory” error in the FianancialReporting0sysout.log. 

They know how to stop and start services using the built-in Oracle Start/Stop EPM System scripts.  Unfortunately, we have found that when FR gets an “out of memory” condition, it no longer responds at all.  When the Oracle script signals for a shutdown, the service still does not stop. 

When this happens, the only way to recover FR is to manually stop the service in Windows Services or to do a reboot of the server. 

The moral of the story is to not always trust the Start/Stop EPM System scripts when you have errors on your EPM applications.

Oracle HTTP Server timeouts

I’ve been to several clients that have large calculations, whether they are very complicated or maybe it’s just a large cube that has a less than optimal calculation (i.e. CALC ALL;).  One thing that was driving me crazy with Essbase Admininstration Services (EAS) was that when I was running one of these slow calculations, I would get an error after about 5 minutes that looked like an application crash.  When these faux crashes happen, the calculation clearly is still executing on the applicaiton based on the sessions information in EAS and the fact that the ESSSVR.exe for that application is still using CPU on the server.

Sure, you can run the calculations in the background to get by or execute them via MaxL, but sometimes you just want to run a process interactively.  I finally dug in and did some searching on timeouts in the EPM system. 

At one of my clients, we were trying to reload data to a cube that was just migrated to 11.1.2.1.  There were 8 – 2 GB files that we were loading, so it was taking awhile.  Each time, the job would error out because of a timeout and cause the Essbase server to crash.  My client was getting irritated, and I couldn’t figure out what was going on.  I had similar issues at another client with a calculation, so I started wondering if there was a timeout somewhere. 

That’s when I realized that I have been using the OHS port for EAS all along (http://foundationServer:19000/easconsole/easconsole.jnlp).  This happened in the thick client and thin client consoles with the 19000 port, but does not happen with the 10080 port.

I did some searching and found that business rules can timeout if they run more than 5 minutes from Workspace, which is the same timeout value I had with my EAS data load.   

So, since I have OHS here, I decided to give it a shot.  I edited the mod_wl_ohs.conf file in NotePad and under EAS and EAS Console, I set the timeout value to 14400 (4 hours).  I then stopped and started OHS, and when I tested it – it worked like a charm.

 Here’s what I did:

 Oracle HTTP Server (OHS)

  1. Edit the following file:
    E:\Oracle\Middleware\user_projects\epmsystem1\httpConfig\ohs\config\OHS\ohs_component\mod_wl_ohs.conf
  2. Add (or edit, if already present) the following parameters:
    WLIOTimeoutSecs 14400
    WLSocketTimeoutSecs 14400
    to the sections for EAS and EASConsole. Each section begins with an XML tag, for example:

<LocationMatch ^/eas/>

</LocationMatch>

<LocationMatch ^/easconsole/>

</LocationMatch>

 
Add the new “WLIOTimeoutSecs 14400” and “WLSocketTimeoutSecs 14400” properties as new lines within the tags, for example:

<LocationMatch ^/eas/>
     SetHandler weblogic-handler
     WeblogicCluster easserver01:10080
     WLIOTimeoutSecs 14400
     WLSocketTimeoutSecs 14400
</LocationMatch>

<LocationMatch ^/easconsole/>
     SetHandler weblogic-handler
     WeblogicCluster easserver01:10080
     WLIOTimeoutSecs 14400
     WLSocketTimeoutSecs 14400
</LocationMatch>

  1. Restart the Oracle HTTP Server after the modifications are complete.