Life of OBIEE Catalog object permission – A niche Subject “ACL” cache

Access Control List(ACL) is very common terminology and not confined precisely within OBIEE. Several software use ACL to define security object permissions.An ACL specifies which users or system processes are granted access to objects, as well as what operations are allowed on given objects. We have seen ACL when we edit Objects permission or open Catalog object using Catalog Manager. I have noticed “preserve ACL” in OBIEE 10g Catalog Manager if you want to apply permission across objects. Also noticed ACL again on 11g while we embarked a very strange/weird and complicated issue.

We have OBIEE 11g .7 version 140415 running on production since launch past 5 months. Recently couple of customer raised concern about page visibility for different kind of subscription. We have customer could use multiple subscription which can be switched from a portal and based on that switch specific page should have to be visible. However we have experience a issue when a user logged in and seeing Dashboard A and logout and relogin again with different subscription Dashboard A still visible while Dashboard B should have to be visible.

Initially how you attack this problem ? I know what you are thinking of but we have very complicated system with several layers before you land to dashboard page. We have One portal which redirects to IDP (performing TCping Authentication) ,then control goes on Horizontal Cluster nodes via Load Balancer and then control goes to Analytics. So there are 3 way channels before you are landing to OBIEE page. On top of this we have deprecated 11g security system running initialization block. I know this is damn bad and Oracle recommended but we have to live with the crap because Oracle said this can still be functional. So in summary we have no Application Roles / Groups setup on EM and Console and all object based permission defined under Catalog is legacy Catalog Group permission.

So there are various aspect and point we need to carefully verify before we attack this problem. I started debugging this issue from Catalog Object level permission first , nothing seems weird but did little bit tweak here and there and it didn’t resolve the issue.

We thought it could be SSO Ping Auth issue which is bypassing the RPD security and not getting GROUPS variable populated right way. So we added WEBGROUPS and ROLES system variables in RPD and enabled all level of trace on EM and Console on oracle bi security objects including the most granular level of LOGLevel trace in instanceconfig.xml file.

We have found after logout of initial session and login new session is not picking up the permission correctly and it retains the old permission on object.

There are security masking logic on Read , Write and All permissions in each level of Catalog object while can be traceable in sawlog file which clearly depicts the issue ! But unfortunately we don’t have smoking gun yet to resolve the problem.

A search in OTN and googling around doesn’t seem to be matching to right use case ! this was a frustrating issue which bother for 4 days until actual fix has been found and Finally we have isolated our environment from SSO , Cluster , Load balancer everything to segregate the root cause of the issue. We came to know its OBIEE 11g issue for sure. We have tried several areas of tweak to fix this issue mostly on file system level , cleaning cache , clean volatileuserdata, removing cacheduserinfo, removing that user content and re-instate, GUID refresh and nothing works and we were almost out of ideas what to do next 😦 😦

FINALLY there is an idea triggered on my mind if there is any configuration can be done on refresh level which could potentially refresh dashboard object including the permission could resolve the issue. And I realize only Instanceconfig.xml is a place where you can declare on such parameter. So in search of some parameter I found below:


This will reload catalog object permission on 1 minutes interval . So potentially this could fix my issue.

And it Voila… it fixed my problem … Now right user getting correct permissions on multiple subscription switch. But still its not end of the game ! WHY ? Yes, we have seen catalog object refreshing in 1 minutes interval but my problem reoccur if multiple login/logout happens within 1 minute of interval …. Sad !! So this is not going to be a solution. Also reload catalog object frequently is not an good idea as SAW server does a hard IO work each time this happens to crawl across catalog and refresh object permission. Also there is no instanceconfig.xml parameter exist to lowered it down to seconds level  😦

Adding this parameter is as good as “Reload server metadata trigged” by Admin from analytics Admin link.  So I was in search how to invoke this automatically rather than required on demand.

Investing more energy on that reveals what is this object permission and seems closely related to ACL and we came to know below details how OBIEE handle the users in Catalog:

The catalog is designed to scale to thousands of concurrent users. To achieve this scaling, the catalog adheres to the following guidelines:

  • The average user typically only reads from the catalog and rarely, if ever, writes to it. In Release 11g, each user is constantly and automatically updating his or her Most Recently Used file, but each user’s “read” operations still far outweigh the user’s “writes” operations. Therefore, the read-to-write ratio is typically at least 100 to 1.
  • While a locking mechanism guarantees that only one user can write to an object at a time, it is rare for multiple users to attempt to write simultaneously to the same object. A feature called “lazy locking” allows users to continue reading an object even when another user is updating that object.
  • Modern file systems cache “small” files directly inside the directory record, such that reading any information on a directory simultaneously loads all small files directly into the operating system’s memory cache. Therefore, it is good practice to keep files in the catalog “small,” especially the frequently “read” .atr metadata files. When these metadata files remain small, then all the .atr files in a directory are loaded into memory with one physical hard disk read. Every file that exceeds the “small” threshold adds another physical hard disk read, which can cause a 100% degradation for each large file. In other words, use care when considering storing arbitrary “Properties” in .atr files.
  • Reading an object’s .atr metadata file using NFS is far slower than reading it directly from a local disk. For this reason, Presentation Services additionally caches all .atr files internally. This cache can become briefly “stale” when another node in the cluster writes data to the file that is newer than the data that is cached by the current node. Therefore, all nodes are refreshed according to the MaxAgeMinutes element in the instanceconfig.xml, whose default for a cluster is 5 minutes. This default setting commonly achieves the best trade-off between the possibility of stale data and the known performance impact. (The default for an environment without clusters is 60 minutes.)

Ours are pretty similar configuration of having millions of .atr file (as we have very large user base in production) in NAS storage continuously accessed by Sawserver catalog crawler.

Looking much on ACL on OTN reveals not same but similar kind of issue and I came to know about bunch of very important config parameters listed at the end of this blog.

Now I have full fledged smoking gun which could potentially resolve my issue which is below:



So I applied in instanceconfig.xml to see the results and unfortunately sawserver was not started as this is unrecognized tag ! When I see OTN carefully and I found this can be added only 140715 version on wards and not less . So we have asked Oracle to provide patch for 140415. After following up Oracle with Sev 1 issue with several escalation we finally got the fix . We applied it , add this params in instanceconfig and it resolves the issue…. 🙂 🙂  Oracle provided the fix with 3 files , one xsd and 2 saw binary files (.so files) . So the long running nightmare is over !!! 🙂 I am sooo happy…

So we know NOW what is ACL Cache and how it could potentially behave weird and cause lot of trouble to our life and we know what should be the way to get rid of the problem around it. See below some of the other important reference on Cache under instanceconfig.xml file. Recommend is if you Disable ACL cache which Oracle confirmed has no performance impact.


Use those judiciously if you know what is that for and later don’t burn your hands … 😉







<!– this cache is NOT related to the security stack It is only for the account info in the webcat itself –>

<!– this cache is NOT related to the security stack It is only for the account info in the webcat itself –>

<!– This CatalogAttributes cache contains all webcat objects’ metadata (the owner, permissions, properties, attributes etc).
The internal permissions have a their own cache as to who is allowed to access the webcat object.

So to cleanup the internal permissions cache, this CatalogAttributescache must also be cleaned. –>






OBIEE 12c(OBIEE Installation guide on Linux Platform

Oracle Business Intelligence Enterprise Edition 12c – OBIEE released on OTN 26th Oct 2015. As soon as I get all the downloadable contents this is my first attempt to install the software.

We know Oracle has made bunch of amendments / enhancements to 12c on top of very successful 11g release. I am going to blog couple of new 12c features pretty soon in terms of UI as well as config changes.

So lets first follow the step by step process to install the software(64 bit) in my Oracle Linux 6.0 OS .

Step 1 – Download and Procure the required files

Download Oracle Business Intelligence 12c from :

1Unlike 11g the installation files are little different and need to installed as per mentioned order. As I am using the server version I have downloaded below files :

2So you have to download and Install JDK first. For me, I have downloaded jdk-8u66-linux-x64.rpm package.


So it looks like First JDK , then Weblogic and then OBIEE main installer and no separate RCU required to be installed. Also notice the files are little bit different with .jar for Weblogic Infrastructure and .bin for OBIEE installation.

Step 2 – Install JDK

Should be simple install command:

sudo yum localinstall jdk-8u66-linux-x64.rpm

Post install verify the version:


Step 3 – Install below Prerequisite packages


Should be simple sudo yum install command

Step 4 – Install Weblogic (Fusion Middleware) Infrastructure

My Linux server has 48 GB Memory with 140GB storage available to perform the installation. It is the simple java command to Launch the setup.

5So this opens the Oracle Inventory window: Select default Inventory location :

Middleware-Install-01Middleware-Install-02Middleware-Install-03 Middleware-Install-04 Middleware-Install-05 Middleware-Install-06 Middleware-Install-07 Middleware-Install-08 Middleware-Install-09 Middleware-Install-10Middleware-Install-11

Step 5 – Install OBIEE 12c Software

Execute the binary content as below:


Step 6 – Configure RCU, Middleware, OBIEE

In this step we have to run some config setup file to configure RCU , Middleware domain .

I would prefer to install RCU separately (using ……\Middleware\Oracle_Home\oracle_common\bin\rcu.bat)before you come to this step because RCU creation , DB type validations etc can results several errors which you want to be fixed before Middleware config started.

Config01 Config02 Config03

In Below steps if you see error called Domain “bi” already exists then remove the content from below and click Next.


You can experience this issue only if you aborted any previous session of the config steps.

Config04STOP here and check if you have 12c Database or at-least 11g R2 ( installed where you are trying to create RCU. If not then it will throw you error and Installer will abort.

RCU2Also check the Database Character Set otherwise you will get warning to change data-set to AL32UTF8 like below:

issue 1For me it was WE8MSWIN1252 in Oracle Db 12c and I have executed below steps to changed it to acceptable CharacterSet.



— if the above fails:

Also If your 12c Database configured with multi tenancy Container Database RCU can’t be created and it will give below error:

issue3I have resolved that error using below steps: fortunately for me during Oracle Db 12c installation I have clicked option ‘pdborcl’ as Pluggable database.

  1. Connect as sys user as sysdba
  2. then execute  : ALTER SESSION SET container = pdborcl ;
  3. select name, open_mode from v$pdbs;   — it should return pdborcl
  4. Alter pluggable database all open;
  5. Add below in TNSNAMES.ora

(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
(SERVICE_NAME = pdborcl)

If everything works great follow next steps of Configuration:


Config06 Config07 Config08 Config09 Config10As mentioned before If you encounter issue with RCU installation just isolate this configuration step from it and trigger RCU script from :


This will prompt a RCU Config steps and you can debug any issue here like SYSDBA privilege, Connect String issue, Port issue, password issue etc etc. Here is sample screen.


Assuming the installation and configuration steps completed. Notice there are change of ports in 12c comparing 11g:

WebLogic Console
Fusion Middleware Control
Visal Analyzer
BI Publisher
BI Analytics

Step 7 – Testing post Installation / UI

This is Analytics:

analytics01analytics03This is visual Analyzer:analytics02 Some of the Visual Analyzer interfaces:

analytics04 analytics05 analytics06

OBIEE 11g – Horizontal Clustering

Hola … I was thinking about to post this thread from long time (since last 5 months) as I have been extensively involved in a OBIEE 11g platform build from scratch, playing with 3 nodes OBIEE installation , Vertical / horizontal clustering et all.

Now the game is over and I want to detail out each and every steps in details and the hurdles I have faced during step by step setup execution.

Honestly before writing this blog thread I have searched across web and googling around several sites to get consolidated Clustering steps and I never found it. This is nothing mean to wrong to other blogger who are posting excellent stuff in their blogs since ages but I feel each post on Horizontal Cluster lacks some information on details.

Idea is to perform 3 nodes horizontal clustering with one OBIEE instance running on each of the nodes. OBIEE version to be installed in with bundle patch version of p18283508. which will make final version to : (Build 140402.1431 64-bit)

There are few restrictions in terms of browser support for this version with IE11. See my previous post for the hack on IE11 to get this work done perfectly.

A little background from OBIEE 10g version just to understand why we made decision to go to OBIEE 11g.

  1. OBIEE 10g lacks features of 11g
  2. Platform wise 11g is more robust with enterprise architecture
  3. In 10g we had faced severe issue with memory in 32 bit architecture and in vertical clustering. The idea of vertical clustering is a crappy design with several architectural flaws which we want to get rid of in 11g
  4. We had 2 OBIEE 10g nodes with 12 instances running on each node is a nightmare to manage and control. Moreover it doesn’t use shared cache with max 4 GB memory address limitation in each sawserver and nqserver process which is a support pain to replenish instances with memory issue/leaks.

Okay so lets start a high level setup details:

  1. Install OBIEE on Node 1 – Primary Node
  2. Install OBIEE on Node 2 – Secondary Node
  3. Install OBIEE on Node 3 – Third Node
  4. Apply OBIEE Bundle patches on Node 1 , then Node 2 and then Node 3
  5. Apply customization , if required and ported from 10g
  6. Perform config chances in EM – Failover and Scaleout to add additional 2 nodes
  7. Perform config chances in EM – Failover
  8. Do Integration check and deploy the required RPD/Catalog on Shared NAS/NFS mount

I will explain the entire things in step by step. In my case it is 2 node cluster installation but for 3rd node the process is exactly same and no difference except extra host to be select on EM.

In above customization steps it is NOT required to have the below OS level file changes present as pre-requisite and you can do it later.

* soft nofile 131072
* hard nofile 131072
* soft nproc 131072
* hard nproc 131072

I assumed that you have right packages downloaded from OTN and right patches before continue.

In below steps I am planning to install OBIEE through Unix script using command line parameters so that no GUI is required. I have created my own response file to pass through certain command line steps which will install OBIEE RCU schema first, OBIEE11g and WLS and then perform some 10g ported customization and finally deploy respective catalog/rpd which is updated from 10g.

The scope for this thread is to define the Steps required to Horizontal cluster on multi-nodes. To save all Logs into a target file use command : script obiee_installation_log.txt  and then start command line execution. Finally once execution done type : exit. this will save entire log from buffer to target file and you can review it to see in details.


Execute weblogic command line params to install RCU. This is similar to what we have in GUI. A command line example is below which will drop existing RCU.

rcu -silent -dropRepository -compInfoXMLLocation /u00/media/ofm_rcu_linux_11/rcuHome/rcu/config/ComponentInfo.xml -storageXMLLocation /u00/media/ofm_rcu_linux_11/rcuHome/rcu/config/Storage.xml -databaseType ORACLE -connectString <Replace with database connect string as db:port:instance> -dbuser <SYDBA Username> -dbrole sysdba -schemaPrefix <SchemaPrefix> -component MDS -component BIPLATFORM -f < /u00/media/ofm_rcu_linux_11/rcuHome/rcuPassword.txt

A command line example is below which will create RCU.

rcu -silent -createRepository -compInfoXMLLocation /u00/media/ofm_rcu_linux_11/rcuHome/rcu/config/ComponentInfo.xml -storageXMLLocation /u00/media/ofm_rcu_linux_11/rcuHome/rcu/config/Storage.xml -databaseType ORACLE -connectString <Replace with database connect string as db:port:instance> -dbuser <SYDBA Username> -dbrole sysdba -schemaPrefix <SchemaPrefix> -component MDS -component BIPLATFORM -f < /u00/media/ofm_rcu_linux_11/rcuHome/rcuPassword.txt

Note that rcuPassword.txt could be any file which will act as source file for storing Sysdba User/Password and passed through command line.

N.B:- If you see any error like this that means you have trouble creating RCU and you have to perform clean-up this error before this:

9-28-2015 10-55-58 AM

The reason because RCU schema pre-exist and didn’t clean-up well. You have to find the schema name from below table acts as a RCU version history keeper and then delete those records/commit before proceed.

select * from System.SCHEMA_VERSION_REGISTRY$

2. Execute OBIEE installer with response file. A sample example below: This will be similar like doing Enterprise installation using UI.

cd /u00/media/bishiphome/Disk1
echo inventory_loc=$HOME/oraInventory > $HOME/oraInst.loc
echo inst_group=${OS_GROUP} >> $HOME/oraInst.loc
mkdir $HOME/oraInventory

./runInstaller -silent -response /u00/response_file -invPtrLoc $HOME/oraInst.loc -waitforcompletion

This will unset ORACLE_HOME (which is mandatory) , then creating Inventory Location with OS group and then continue installation with response file called response_file

If you are using response_file then for primary node installation below are important parameters you need to change in response_file. You can find the details of standard response for for OBIEE 11g installation in Oracle doc or just google it





So my OBIEE installation on Primary node completed and I used to get below issue whenever I ran command line / GUI:

[CONFIG] FAILED:Executing: opmnctl start coreapplication_obisch1
[CONFIG]:Modifying BI Configuration Files
[CONFIG] SUCCESS:Modifying BI Configuration Files
Configuration:BI Configuration failed
[CONFIG] Failed.
[ACTION]: BI Configuration

The installation of Oracle AS Common Toplevel Component, Oracle Business Intelligence Shiphome failed.

To avoid this I used to do a tweak and intercept of correcting the opmn.xml  while 40% of the installation is over and once you get this file created. This file located at directory :  /u00/app/Middleware/instances/instance1/config/OPMN/opmn.

Or else you can just keep watching on below during Installation and then make the changes on opmn.xml file:

[CONFIG]:Creating Instance
[CONFIG] SUCCESS:Creating Instance

Just take the backup of existing opmn.xml and then Change its contents…

<process-type id=”OracleBISchedulerComponent” module-id=”CUSTOM”>
– <module-data>
– <category id=”start-parameters”>
<data id=”start-executable” value=”$ORACLE_HOME/bifoundation/server/bin/” />
– <!– enable console log to be able to see process startup error –> f(clean);
<data id=”no-stdio” value=”false” />
– <category id=”stop-parameters”>
<data id=”stop-executable” value=”integrator” />
– <category id=”ping-parameters”>
<data id=”ping-type” value=”integrator” />
– <category id=”ready-parameters”>
<data id=”use-ping-for-ready” value=”true” />
<start timeout=”600″ retry=”1″ />
<stop timeout=”120″ />
<restart timeout=”720″ retry=”1″ />

<process-type id=”OracleBISchedulerComponent” module-id=”CUSTOM”>
<category id=”start-parameters”>
<data id=”start-executable” value=”$ORACLE_HOME/bifoundation/server/bin/” />
<!– enable console log to be able to see process startup error
<data id=”no-stdio” value=”false” />
<category id=”ping-parameters”>
<data id=”ping-url” value=”/”/>
<category id=”restart-parameters”>
<data id=”reverseping-timeout” value=”345″/>
<data id=”no-reverseping-failed-ping-limit” value=”3″/>
<data id=”reverseping-failed-ping-limit” value=”6″/>
<start timeout=”300″ retry=”3″/>
<stop timeout=”300″/>
<restart timeout=”300″ retry=”3″/>
<ping timeout=”60″ interval=”600″/>

This will make sure Scheduler start-up will be fine and no issue to Proceed the BI Configurations. This is just a hack a no reason why Oracle haven’t done this default in there installation package. You might need to do it across all secondary nodes.

The idea is you should be getting below to confirm entire Installation is done successfully.

[CONFIG] SUCCESS:Modifying BI Configuration Files
Configuration:BI Configuration completed successfully
The installation of Oracle AS Common Toplevel Component, Oracle Business Intelligence Shiphome completed successfully.


Now we have to scale-out to Node 2 (secondary) and then 3rd node.

For this in Node 2 we need to make sure:

  1. we will not re-execute RCU commands as we already have RCU in place during primary node installation
  2. we will make changes in response file and make sure we put DOMAIN_HOSTNAME= <primary node Ip/host> and SCALEOUT_BISYSTEM=true
  3. I would prefer to use INSTANCE_NAME=instance2 to identify it is 2nd instance in cluster
  4. All activities will continue here like installing OBIEE / Weblogic , configuration except Admin server installation.
  5. Each additional node in cluster will act like Managed server and its system components and only Primary server will act like Admin and Managed server both
  6. Node 2 installation would be pretty much faster than Node 1. In my experience Primary node takes 30-40 minutes while all secondary nodes will take ~15 minutes to do complete install. Anyway it again vary from system to system based on capacity but I did a tweak on Java memory parameters for faster start-up/shutdown and Installation on OS level.
  7. Performance enhancement for faster start-up/Installation.

Make changes in all OBIEE Node’s . Need root access. No reboot required

  1. Edit or create /etc/sysconfig/rngd to contain:

# Add extra options here

EXTRAOPTIONS=”-r /dev/urandom”

  1. Then “service rngd start”.
  2. If that works, then “chkconfig rngd on” ( to start it at boot ).
  3. Add this on .bash_profile in Unix as below:

export JAVA_OPTIONS=””

So finally in Node 2 the output should be look like this:

9-28-2015 8-17-49 PM


Repeat STEP 2 for Node 3 . It must be similar.


  1. Apply Patches on NODE 1 using Opatch
  2. Apply Patches on NODE 2 using Opatch
  3. Apply Patch on NODE 3

N.B:- You must be thinking why I am not doing performing Installation and patch in one node completely and then proceed to next node. It will not going to work during Scale-out phase of other nodes. The reason because Once Primary node is upgraded with latest bundle patch and you are trying to perform scale-out a version mismatch occurs  for secondary nodes and it is not being able to access the primary nodes module. In this case you will be getting below error:

[2015-09-28T17:57:31.608-04:00] [as] [ERROR] [] [] [tid: 38] [ecid: 0000L0LxUqL3b6G5uzd9iX1M2RR700000T,0] ERROR: Instance creation failed.[[
Caused by:


Lets do a quick sanity check first for Node 1.

top -u orabi should throw below running process on Node 1:

Process-Primary Node

top -u orabi should throw below running process on Node 2.

Process-Secondary Node

We can see Node 1 and 2 added in cluster. this means scale-out is successful and EM recognizes both node.

Also you see some of the processes are down on Node 2 from EM which is fine. Now we will see what additional steps we need to do.



If you are deploying RPD and Catalog it is recommended to deploy it now into share path (which you must have to do if you want 3 nodes to share the same RPD and Catalog).

  • So Go to EM-) Deployment-)Repository. Add new RPD and Catalog after “Lock and Edit”.
  • Apply and Activate changes.
  • For us we have common NAS/NFS mount shared and accessible from 3 OBIEE nodes. It looks like below: (Catalog final name is obfuscated due to security reason)

9-28-2015 4-17-15 PM

Now “Lock and Edit” and Go to EM-) Availability -) Failover -) Make secondary Host like below and Apply, Activate Changes.


Now perform “Restart All” . This will make all the processes up and running across all nodes. But note, we haven’t yet created OBIEE Managed server system components on secondary nodes.

Go to Capacity management -) Scalability -) Add one components on each secondary nodes. This is nothing but vertical clustering on horizontal cluster. If you add more than one components it will create more than one instance on single host.


Note that, more than one system components means more power and it will run more than 1 instance of sawserver, nqserver,Cluster Controller, nqScheduler and Javahost process. Do it if you need it else not required and single instance in each node with 64 bit architecture is enough capable to handle 700-1000 concurrent user request (considering OBIEE performance parameters have been correctly applied)

Hit “Apply” and “Activate Changes” . It will take some time to create additional processes and start them up on secondary nodes. Once this is done successfully then see the list of running process on secondary node by typing top -u orabi in unix session. And observe below that instance2 system components creates under below location on 2nd node.


After successful connections Failover EM screen will be like below:


I have faced several issues with EM not recognizing the new set of components and processes during Restart. In such cases do Individual component restart or try using opmnctl commands to restart opmn managed components else Bounce Primary node at once and then bounce secondary node once. This should resolve most of the problem otherwise it is bigger issue and something wrong during cluster setup process.

Now “Restart All” from EM and see all green means Horizontal Cluster setup process completed successfully 😀 😀

All Green


If you have any customization carried forward from 10g upgrade this is the right time to do that in each nodes one by one starting from Primary. Follow the steps to stop services on Primary and then make customization changes and then start on primary and then stop on secondary , do customization and then start services. This is standard process step by step.

My Observations:

  • Apparently RPD is share under Shared Path but physically RPD located under repository path below under each Nodes and when the opmnctl process starts up it loads the RPD from this physical path. So the concept of Shared RPD applicable when you open RPD in online mode and made some changes the change is reflected online and this location acts as temporary staging location and after changes in RPD become it sync with clustered nodes and propagated the changes.

For Primary: /u00/app/Middleware/instances/instance1/bifoundation/OracleBIServerComponent/coreapplication_obis1/repository

For Secondary:


  • Even in Horizontal cluster mechanism Cache has been create in individual nodes. Global Cache is a concept which is applicable during Cache seeding only and nowhere else.
  • If you want to do Vertical clustering on top of Horizontal its easy from EM.
  • After Clustering you can access Node 1 using its HostName . If Host1 is down you can still access Node 1 as WLS internal clustering/Load balance will automatically route the request to 2nd Node by the help of Clustered Controller module. If 2nd node also not found it will redirect request to 3rd Node.
  • The idea of WLS Horizonal clustering is High availability which tells even if Admin Server is down in Node 1 , clustered managed servers/nodes still can work and serve user query without having any downtime.
  • Don’t try to do Customization on STEP 7 before STEP 6 (EM and Failover changes plus adding instances on secondary nodes) because you might have some customization which require that instances directory to be created first inside Secondary nodes.
  • Entire activities above can be done using Unix scripts only. For e.g, even the Horizontal cluster/ Failover/ scalability can be done using WLS scripting and invoking python script from inside shell script. For me its easy to use EM UI to do this but certainly end to end steps possible through scripts.
  • You don’t need to have a successful RPD connections to DB exist while doing this as in those steps services will be bounced several times. You just need a basic placeholder RPD
  • You don’t need the tnsnames to be in ../MiddleWareHome/Oracle_BI1/network/admin path as long as you are using IP:port syntax in RPD connection pool. Else you might think about updating tnsnames.ora on that location to get connectivity with RPD if RPD use DB connect string
  • Note, if you have 3rd, 4th nodes to be added in horizontal cluster you can’t have Scheduler process and ClusterController process in system components for any additional nodes beyond 2nd nodes. This is because failover and cluster controller process can be only present in two nodes/hosts/servers and you have to manage the deployment such a way that both primary and secondary node shouldn’t go down at same time. So apart from Primary and 1 Secondary node Failover will not be available with high availability for Scheduler and ClusterController
  • Scheduler will run only on Node 1(Primary) and Node 2 (Secondary) but it will not run on 3rd Node (Secondary) but scheduler query can run on Node 3.

OBIEE 11g ( support with IE11- XML hack

We know that by now I am writing this blog OBIEE already supported with IE11 (compatibility mode). When I am talking about it is very specific to latest bundle patch version (anything released after (Build 140402.1431 64-bit)

This could be older post but might be helpful who are still with (recognized as most stable so far) and had older patch of 140415 .We are running on 140415 and without IE 11 support . We are in product company having global presence across world. So for us it is not luxury rather a very necessary to think of how we can get support with IE 11 as we can’y say our customer to use different Browser version and they could have enterprise restriction and in our case several customers have Integration of BI dashboard pages in their own application under iFrame. So it is kind of enterprise wide challenge. Also we can’t release part of application in 11g for some users and rest in 10g. It is again maintenance and communication overhead.

I read that IE11 support is not in Oracle Certification matrix and Oracle said they stopped releasing back-port patch only for IE11 with So responsibility is due on us (BI team) how can we can system hack to get this compatibility with less changes in client side.
I took the challenge to find some tweaks to get this system hack. See how in step by step I approached to solution/fix/hack.

What I see if you open OBIEE .7 version in IE11 you will get Presentation service error in browser saying the browser version not supported . If you add that URL in compatibility mode IE11 will save this as URL request come with domain name and save in that list. And immediately you can see IE11 no more complaining about it . But its not end of story . After login you will find several things are broken like Links , Navigation , View selector , Prompts doesn’t work properly .

If you want to see the issue you can put below string in IE11 URL header . This will tell what kind of compatibility your browser is using. For IE11 you can clearly see no “compatible” header will return in User Agent string .

javascript:alert(navigator.userAgent) – below is example from IE 10 browser .


The reason behind it is IE11 run the browser in default Document Mode as (Edge) . If you click F12 Developers tool in IE11 you will see Emulation tab . And here you will find Document Mode and User Agent .

Compatibility will not tweak on Document mode . So the challenge is, you have to pass the string related to Browser compatibility which will override client browser settings and forced into Emulation mode you specified.

I start reading how Browser works with this compatibility and what kind of X-UA string responsible to do the browser tweak.

I found a tiny OBIEE config file pass this Emulation string with application URL called iecommon.xml file .

You can find this under : <MiddleWare_Home>/Oracle_BI1/bifoundation/web/display/featuretables/iecommon.xml

Only things is you have to change below lines. See 100 was by default , you have to change it IE=10.

After making this change you need to bounce BI System components.

<feature name=”requiredMetaTags” xsi:type=”xsd:string”>&lt;meta http-equiv=”X-UA-Compatible” content=”IE=100″ &gt;</feature>

Simple huh ? Just removing one zero (0) . Now  paste  “javascript:alert(navigator.userAgent)” (without double quotes) in IE URL header now and you can see the strings returned as IE 10 compatible. Means it did the job  !

Now if you presss F12 in IE11 and go to Emulation tab you will find Document header has been forced to change IE 10(default) and its no more Edge. So as soon as you hit OBIEE .7 URL in IE11 the server side parameter will forced them to run in Document mode IE10 .

Now try pulling your IE 11 reports (Note that we still need browser based compatibility settings and that is one time so should be fine for users) and you can clearly see the difference. No issues with Prompts, Navigation , Action link , Drill down , View selector , Alerts , Answers . Its voila ….

This steps is not declared officially and Oracle never said this works but its a system hack/tweak which perfectly work for us. So please use it on own risk.

Enjoy …till next time 🙂

OBIEE 11g Services Start-up Guide for Unix Platform

Below are the step by step guide to do clean start-up and shut-down of OBIEE 11g services. As we all know 11g is beast compared to its ancestor so we need to be careful doing the things in the right way.

Here I have demonstrated the steps how to do that in Linux/Unix platform and the idea is to kick-off below commands from the user having admin roles and in my case dba groups .This user is same I have used to do installation. If you have any other local user better to avoid it for services start-up as this could cause several locks into Java OPSS security files and will be causing problem to start Admin and Managed server of weblogic in bootstrap process.

In below steps I am running all the processes in nohup mode in background of unix session so that it will continue to run even if you have exit from the current running session.

1) Start-up Admin Server:

[orabi@obiappl11g-xxx ~]$ nohup sh /u00/app/MiddlewareHome/user_projects/domains/bifoundation_domain/bin/ > wls_start.log &

Find the below lines in log files which will make sure clear start of Admin Server.
<Apr 28, 2014 10:05:56 PM CDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING>
<Apr 28, 2014 10:05:56 PM CDT> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode>

2)  Start-up Node Manager:

[orabi@obiappl11g-xxx ~]$ nohup sh /u00/app/MiddlewareHome/wlserver_10.3/server/bin/ > nodem_start.log &

Find the below lines in log files to confirm start-up:

<Apr 28, 2014 10:07:03 PM> <INFO> <Secure socket listener started on port 9556>
Apr 28, 2014 10:07:03 PM weblogic.nodemanager.server.SSLListener run
INFO: Secure socket listener started on port 9556

3) Start-up Managed Server:

[orabi@obiappl11g-xxx ~]$ nohup sh /u00/app/MiddlewareHome/user_projects/domains/bifoundation_domain/bin/ bi_server1 http://obiappl11g-xxx:7001 > start_bi_server1.log &

Look for below lines in log files to confirm Managed Server start-up:

WebLogic Managed Server “bi_server1” for domain “bifoundation_domain” running in Production Mode>
<Apr 28, 2014 10:53:14 PM CDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING>
<Apr 28, 2014 10:53:14 PM CDT> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode>

Note: In above steps http://obiappl11g-xxx:7001 is my Admin URL where  obiappl11g-xxx is server name.

Make sure you have added WLS_USER and WLS_PWD in below file before start-up so you don’t need to pass that in command line parameters:


You can start-up this from WLS Console or Fusion Middleware Control also.

4) Start OPMN managed services:

[orabi@obiappl11g-xxx ~]$ /u00/app/MiddlewareHome/instances/instance1/bin/opmnctl startall

enter and wait until command prompt comeback

Check the status:

[orabi@obiappl11g-xxx ~]$ /u00/app/MiddlewareHome/instances/instance1/bin/opmnctl status

It should be look like this:

Processes in Instance: instance1
ias-component                    | process-type       |     pid | status
coreapplication_obiccs1          | OracleBIClusterCo~ |    5467 | Alive
coreapplication_obisch1          | OracleBIScheduler~ |    5801 | Alive
coreapplication_obijh1           | OracleBIJavaHostC~ |    5465 | Alive
coreapplication_obips1           | OracleBIPresentat~ |    5463 | Alive
coreapplication_obis1            | OracleBIServerCom~ |    5464 | Alive

Now Voila…. After all services started up in good health 🙂



Note: You can see background processes from top -u serid> command:

[orabi@obiappl11g-xxx ~]$ top -u orabi

 5065 orabi     20   0 5983m 1.1g  42m S  1.0  2.3   2:20.89 java
 7117 orabi     20   0 19452 1352  948 R  0.7  0.0   0:00.19 top
 2643 orabi     20   0 6100m 1.2g  48m S  0.3  2.6   3:01.12 java
 2841 orabi     20   0 2095m 129m  19m S  0.3  0.3   0:05.34 java
 5464 orabi     20   0 4538m 121m  47m S  0.3  0.3   0:04.26 nqsserver
 5465 orabi     20   0 3980m 232m  16m S  0.3  0.5   0:07.90 java
 2448 orabi     20   0  105m 1936 1492 S  0.0  0.0   0:00.24 bash
 2578 orabi     20   0  103m 1432 1120 S  0.0  0.0   0:00.03 sh
 2809 orabi     20   0  103m 1364 1108 S  0.0  0.0   0:00.01 sh
 5011 orabi     20   0  103m 1264 1092 S  0.0  0.0   0:00.00 sh
 5012 orabi     20   0  103m 1436 1120 S  0.0  0.0   0:00.02 startWebLogic.s
 5440 orabi     20   0 68328 8424 5900 S  0.0  0.0   0:00.00 opmn
 5441 orabi     20   0 1757m  17m  10m S  0.0  0.0   0:06.68 opmn
 5463 orabi     20   0 4172m 271m  72m S  0.0  0.6   0:03.49 sawserver
 5467 orabi     20   0 1481m  24m  15m S  0.0  0.1   0:00.70 nqsclustercontr
 5801 orabi     20   0 1909m  78m  51m S  0.0  0.2   0:01.66 nqscheduler

or else you can see them from running job list:

[orabi@obiappl11g-xxx ~]$ jobs
[1]   Running                 nohup sh /u00/app/MiddlewareHome/user_projects/domains/bifoundation_domain/bin/> wls_start.log &
[2]-  Running                 nohup sh /u00/app/MiddlewareHome/wlserver_10.3/server/bin/ > nodem_start.log &
[3]+  Running                 nohup sh /u00/app/MiddlewareHome/user_projects/domains/bifoundation_domain/bin/ bi_server1 http://obiappl11g-t1.gain.tcprod.local:7001 > start_bi_server1.log &

Make sure you follow the same rule during shutdown…

Leave it for you guys to explore how to do that 😉 😛

2012 in review – My Blog …Wish you all Happy New Year !

The stats helper monkeys prepared a 2012 annual report for this blog.

Here’s an excerpt:

About 55,000 tourists visit Liechtenstein every year. This blog was viewed about 210,000 times in 2012. If it were Liechtenstein, it would take about 4 years for that many people to see it. Your blog had more visits than a small country in Europe!

Click here to see the complete report.