Customizing Apache Superset UI – Config , Theme Changes


Lets talk something different than Traditional BI today which is about Apache Superset , A modern UI framework based on Flask and React JS.

As of my writing I am on Superset 0.99.0a version : Assuming I have my virtualenv , node module everything running and I have superset installer synced from docker Hub or cloned from Github I am straight going to the customization of different pieces of UI.

I am touching basics of the installation quickly : and my local folder where Superset exist is ‘ssdev’

pip install virtualenv
virtualenv venvdxp — create the env called ssdev
. ./venvdxp/bin/activate or source venvdxp/bin/activate — Activating virtualenv ssdev

Create your own Superset Dev env directory:
mkdir ssdev
cd ssdev
sudo git clone https://github.com/apache/incubator-superset
or
sudo git clone https://github.com/pauldx/incubator-superset
cd incubator-superset
(venv) dpaul@<HOST> ~/ssdev/incubator-superset $
sudo pip install -r requirements.txt
cd ~/superset_dxp
sudo chmod -R 777 incubator-superset/
cd incubator-superset/
sudo pip install -e .
fabmanager create-admin –app superset (note you can’t have multiple account in same name and same emailid in 1 venv )
superset db upgrade
superset init or superset-init
superset load_examples
nohup superset runserver -d -p 8080 &
tail -f nohup.out

The version I am working on below is :

cat $HOME/ssdev/incubator-superset/superset/assets/version_info.json
{“GIT_SHA”: “0509d7aefd71ee21b417c315c8f29cb51bb6b48c”, “version”: “0.999.0dev”}

Create Custom Folder for UI in Superset to make custom change localized here : In my case my custom changes placed under TC folder :
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cd ~/ssdev/incubator-superset/superset/assets/stylesheets/less/
mkdir TC
cp -r cosmo/* TC/
cd ~/ssdev/incubator-superset/superset/assets/stylesheets/less/TC
mv cosmoTheme.js TCTheme.js
 
~/ssdev/incubator-superset/superset/assets/stylesheets/less/index.less
Comment below and Add TC config :
//@import “./cosmo/variables.less”;
//@import “./cosmo/bootswatch.less”;
 
// TC Branding Config
@import “./TC/variables.less”;
@import “./TC/bootswatch.less”;
 
~/ssdev/incubator-superset/superset/assets/stylesheets/superset.less
Comment below and Add TC Config:
@import ‘./less/index.less’;
//@import “./less/cosmo/variables.less”;
 
// TC Branding Config change
@import “./less/TC/variables.less”;
Changing All UI Fonts to “Lato” Font:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Replace below to change all font to Lato: (Note Lato file must be copied/ deployed to the Superset Unix/ Linux env in ~/.fonts)
and then :
$fc-cache -fv
$fc-list
to load the fonts.
if fc-cache is not available in docker then use :  apt-get install fontconfig
Now change below on  :  ~/ssdev/incubator-superset/superset/assets/stylesheets/less/TC/variables.less
 
 
//@font-family-sans-serif:  Helvetica, Arial;
 
@font-family-sans-serif:  Lato;
 
//@font-family-serif:       Georgia, “Times New Roman”, Times, serif;
 
@font-family-serif:       Lato ;
//** Default monospace fonts for `<code>`, `<kbd>`, and `<pre>`.
//@font-family-monospace:   Menlo, Monaco, Consolas, “Courier New”, monospace;
 
@font-family-monospace:   Lato;
Changing Logo:
~~~~~~~~~~~~~~
Copy new logo here: ~/ssdev/incubator-superset/superset/assets/images/TClogo.png
 
Edit ~/ssdev/incubator-superset/superset/config.py
Change the Logo name there:
APP_ICON = ‘/static/assets/images/TClogo.png’
Shaping the Top Nav Bar(height) plus bgcolor and font color:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~/ssdev/incubator-superset/superset/assets/stylesheets/less/TC/variables.less
 
Replace below:
//@navbar-height:                    50px;
@navbar-height:                      30px;
//TC Branding colors
@tcbrand-blue:         #0093e0;
//@navbar-inverse-bg:                         #fff;
@navbar-inverse-bg:                         @tcbrand-blue;
//@navbar-inverse-link-color:                 @gray-dark;
@navbar-inverse-link-color:                 #ffffff;
Navbar onhover underline color:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~/ssdev/incubator-superset/superset/assets/stylesheets/less/TC/bootswatch.less
 
.navbar-inverse .navbar-nav > li.active > a {
//  border-bottom: 3px solid @brand-primary;
border-bottom: 3px solid #F38A00;
}
.navbar-nav > li > a {
//  padding-top: 18px;
// padding-top: 8px;
}
Changing the Caret color:
~~~~~~~~~~~~~~~~~~~~~~~
~/ssdev/incubator-superset/superset/assets/stylesheets/less/TC/bootswatch.less
 
.caret {
  border: none;
  color:  #ffffff;
}
.caret:hover {
  color:  #ffffff;
}
Changing The filterbox visualizations:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~/ssdev/incubator-superset/superset/assets/src/visualizations/filter_box.jsx
 
const backgroundImage = (
                ‘linear-gradient(to right, lightgrey, ‘ +
              //  `lightgrey ${perc}%, rgba(0,0,0,0) ${perc}%`
                `lightgrey ${0}%, rgba(0,0,0,0) ${0}%`
              );
 
<DateFilterControl
              name={since}
              label={t(‘Start Date’)}
              description={t(‘Select starting date’)}
              onChange={this.changeFilter.bind(this, since)}
              value={this.state.selectedValues[since]}
            />
 
 
              name={until}
              label={t(‘End Date’)}
              description={t(‘Select end date’)}
              onChange={this.changeFilter.bind(this, until)}
              value={this.state.selectedValues[until]}
            />
Background blue onhover color:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~/ssdev/incubator-superset/superset/assets/stylesheets/react-select/select.less
 
.VirtualizedSelectFocusedOption {
//  background-color: rgba(0, 0, 0, 0.1);
background-color: #ffffff ;
color: #0093e0;
}
 
.VirtualizedSelectFocusedOption:hover {
  cursor: pointer;
background-color: #0093e0 ;
color: #ffffff;
}
Changing the Brand Primary color from SS default green to TC Blue:
Will change Dashboard links to blue , Fav color to blue and export link color to blue:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~/ssdev/incubator-superset/superset/assets/stylesheets/less/TC/variables.less
 
//@brand-primary:         #00A699;
@brand-primary:         #0093E0;
Changes the style of the Input fields:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~/ssdev/incubator-superset/superset/assets/stylesheets/less/TC/variables.less
 
//** Text color for “s
//@input-color:                    @text-color;
@input-color:                    #0093e0;
 
//@input-bg-disabled:              @gray-lighter;
@input-bg-disabled:             #0093e0 ;
 
//@input-border-focus:             #66afe9;
@input-border-focus:             #0093e0;
Button shape change:
~~~~~~~~~~~~~~~~~~~~
~/ssdev/incubator-superset/superset/assets/stylesheets/less/TC/variables.less
 
// Allows for customizing button radius independently from global border radius
//@btn-border-radius-base:         @border-radius-base;
//@btn-border-radius-large:        @border-radius-large;
//@btn-border-radius-small:        @border-radius-small;
 
@btn-border-radius-base:         30px;
@btn-border-radius-large:        30px;
@btn-border-radius-small:        30px;
Removing the table border color:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~/ssdev/incubator-superset/superset/assets/stylesheets/less/TC/variables.less
 
//** Border color for table and cell borders.
//@table-border-color:            #ddd;
@table-border-color:            #fff;
Dropdown menu changes:
~~~~~~~~~~~~~~~~~~~~~~~~~
~/ssdev/incubator-superset/superset/assets/stylesheets/less/TC/variables.less
 
//@dropdown-border:                rgba(0,0,0,.15);
@dropdown-border:                #0093e0;
 
//@dropdown-link-color:            @gray-dark;
@dropdown-link-color:            #495057;
Fix custom logo alignment issue:
$ASSET/node_modules/bootstrap/less/navbar.less
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
(Not available in older SS version) And add margin on below section at :
 
 
.navbar-brand {
  float: left;
  padding: @navbar-padding-vertical @navbar-padding-horizontal;
  font-size: @font-size-large;
  line-height: @line-height-computed;
  height: @navbar-height;
  &:hover,
  &:focus {
    text-decoration: none;
  }
  > img {
    display: block;
    margin-top: -7px ;
  }
Else change in here :
Add new entry after navbar section as below :
~/ssdev/incubator-superset/superset/assets/stylesheets/less/TC/bootswatch.less
.navbar-brand > img {
  display: block;
margin-top: -7px !important;
height: 34px !important;
}
Changing Filter control text:
~~~~~~~~~~~~~~~~~~~~~~~~~~~
~/ssdev/incubator-superset/superset/assets/src/explore/components/controls/DateFilterControl.jsx
 
              bsSize=”small”
              className=”float-left ok”
              bsStyle=”primary”
              onClick={this.close.bind(this)}
            >
             Apply
            
              className=”float-right”
            >
                bsSize=”small”
                className=”now”
                onClick={this.setValueAndClose.bind(this, ‘now’)}
              >
               Today
              
                bsSize=”small”
                className=”clear”
                onClick={this.setValueAndClose.bind(this, ”)}
              >
               Reset
              
Removing Right side navbar menu items : (Profile / Languages)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~/ssdev/incubator-superset/superset/templates/appbuilder/navbar_right.html
Commented out this section:
<!–li class=”dropdown”>
    <a class=”dropdown-toggle” data-toggle=”dropdown” href=”javascript:void(0)”>
       
 
       
        
    </a>
    <ul class=”dropdown-menu” id=”language-picker”>
      <li class=”dropdown”>
        {% for lang in languages %}
            {% if lang != locale %}
                <a tabindex=”-1″ href=”{{appbuilder.get_url_for_locale(lang)}}”>
 
                     – {{languages[lang].get(‘name’)}}
                  
                </a>
            {% endif %}
        {% endfor %}
      </li>
    </ul>
</li–!>
 
Add below for Dropdown after : {% if not current_user.is_anonymous() %}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<li class=”dropdown”>
      <a
        class=”dropdown-toggle”
        data-toggle=”dropdown”
        title=”{{g.user.get_full_name()}}”
        href=”javascript:void(0)”
      >
 
<SPAN STYLE=”font-family: Lato”>Dashboards</SPAN>
  &nbsp;<b class=”caret”></b>
      </a>
        <ul class=”dropdown-menu”>
             <li>
<a
         href=”/superset/dashboard/world_health” title=”Pricing Analysis” target=”_self”>
        <SPAN STYLE=”font-family: Lato”>Pricing</SPAN>
          </a>
</li>
            <li>
<a
         href=”/superset/dashboard/births/” title=”Supply Analysis” target=”_self”>
        <SPAN STYLE=”font-family: Lato”>Supply</SPAN>
          </a>
</li>
        </ul>
    </li>

Add links as tabs in Menu:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

add below after : {% include ‘appbuilder/navbar_right.html’ %}
~/ssdev/incubator-superset/superset/templates/appbuilder/navbar.html
      <li >
          <a
         href=”/superset/dashboard/world_health” title=”WorldBank” target=”_self”>
        <SPAN STYLE=”font-family: Lato”>Pricing</SPAN>
          </a>
        </li>
        <li >
          <a
         href=”/superset/dashboard/births/” title=”Birth” target=”_self”>
        <SPAN STYLE=”font-family: Lato”>Supply</SPAN>
          </a>
       </li>

Change ~/ssdev/incubator-superset/superset/templates/appbuilder/navbar.html

<img
          width=”146″ height=”43″ src=”{{ appbuilder.app_icon }}”
          alt=”{{ appbuilder.app_name }}”
        />

To remove Right side navbar menu items : (Version info /Superset’s Github / Documentation  )

Remove all text below:
          <a href=”/static/assets/version_info.json” title=”Version info”>
            <i class=”fa fa-code-fork”></i> &nbsp;
          </a>
        </li>
        <li>

 

          <a href=”http://google.com” title=”Google” target=”_blank”>
            <i class=”fa fa-github”></i> &nbsp;
          </a>
        </li>
        <li>
          <a href=”https://superset.incubator.apache.org” title=”Documentation” target=”_blank”>
            <i class=”fa fa-book”></i> &nbsp;
          </a>
        </li>
Superset default Menu items text can be changed by editing below file content:
./incubator-superset/superset/views/core.py
 
Changing the Dashboard in place Button  look and feel to be brand button:
.button
{
background-color:    #0093e0;
border-radius:     3rem;
border-color:     #0093e0;
}
Add TC brand color palettes for all visualizations:
~/ssdev/incubator-superset/superset/assets/src/modules/colors.js
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
Add TC new color palette
In superset/assets/src/modules/colors.js, add this part of code in line 43:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
export const tcColors = [
  ‘#F38A00’,
  ‘#FFBF69’,
  ‘#0093E0’,
  ‘#98D2EB’,
  ‘#4D7EA8’,
  ‘#9EB5C9’,
  ‘#3E4958’,
  ‘#6A7E95’,
  ‘#6D6E70’,
  ‘#DBDBDB’,
];
And once that part is added, add this in line 104:
tcColors,
Resulting:
 
export const ALL_COLOR_SCHEMES = {
  bnbColors,
  d3Category10,
  d3Category20,
  d3Category20b,
  d3Category20c,
  googleCategory10c,
  googleCategory20c,
  lyftColors,
  tcColors,
};
To change Heatmap control to add TC color Visualizations:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~/ssdev/incubator-superset/superset/assets/src/modules/colors.js
 
Add below in line 129 :
 
tcbrand_blue: [
    ‘#C1EAFF’,
    ‘#94DAFF’,
    ‘#6BCCFF’,
    ‘#47C0FF’,
    ‘#0093E0’,
  ],
Add below highlight:
~/ssdev/incubator-superset/superset/assets/src/explore/controls.jsx
 
  linear_color_scheme: {
    type: ‘ColorSchemeControl’,
    label: t(‘Linear Color Scheme’),
    choices: [
      [‘fire’, ‘fire’],
      [‘white_black’, ‘white/black’],
      [‘black_white’, ‘black/white’],
      [‘tcbrand_blue’, ‘tcbrand_blue’],
      [‘dark_blue’, ‘light/dark blue’],
      [‘pink_grey’, ‘pink/white/grey’],
      [‘greens’, ‘greens’],
      [‘purples’, ‘purples’],
      [‘oranges’, ‘oranges’],
      [‘blue_white_yellow’, ‘blue/white/yellow’],
      [‘red_yellow_blue’, ‘red/yellowish/blue’],
      [‘brown_white_green’, ‘brown/white/green’],
      [‘purple_white_green’, ‘purple/white/green’],
    ],
    default: ‘tcbrand_blue’,
And change the slice view parameter default to : tcbrand_blue or we can change Default in control.jsx 
Adding new SuperSet menu item:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
Add below in #2752 Line
~/ssdev/incubator-superset/superset/views/core.py
 
appbuilder.add_link(
    ‘NewLink1’,
    label=_(‘NewLink1’),
    href=’/superset/dashboard/newlink1′,
#    category_icon=’fa-flask’,
#    icon=’fa-flask’,
    category=’Dashboard’,
    category_label=__(‘Dashboard’),
)
 
 
appbuilder.add_link(
    ‘NewLink2’,
    label=_(‘NewLink2’),
    href=’/superset/dashboard/newlink2′,
#    category_icon=’fa-flask’,
#    icon=’fa-flask’,
    category=’Dashboard’,
    category_label=__(‘Dashboard’),
)
 
appbuilder.add_link(
    ‘Demand360’,
    label=_(‘Demand360’),
    href='<place any URL here>’,
#    category_icon=’fa-flask’,
#    icon=’fa-flask’,
    category=’HT360′,
    category_label=__(‘HT360’),
)
I would only do this if you mind the extra space between container body with navbar:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ASSET/stylesheets/superset.less
 
div.navbar {
  z-index: 999;
margin-bottom: 10px;
}
Adding lato font:
~~~~~~~~~~~~~~~~~~~~
$ASSET/stylesheets/superset.less
 
Add below :
 
body {
margin: 0px !important;
font-family: ‘Lato’, sans-serif !important;
}
Removing Select[xxx] in dropdown :
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~/ssdev/incubator-superset/superset/assets/src/visualizations/filter_box.jsx
 
Remove space[%s] :
<OnPasteSelect
            placeholder={t(‘Select [%s]’, filter)}
Customize free form text to ‘Custom Range’ :
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ASSET/src/explore/components/controls/DateFilterControl.jsx
 
<PopoverSection
            title=”Free form”
            isSelected={this.state.type === ‘free’}
            onSelect={this.setType.bind(this, ‘free’)}
Title is not in Lato Font :
~~~~~~~~~~~~~~~~~~~~~~
$ASSET/stylesheets/superset.less
 
.editable-title input {
    padding: 2px 6px 3px 6px;
    font-family: Lato ;
}
 
.editable-title input[type=”button”] {
    border-color: transparent;
    background: transparent;
    white-space: normal;
    text-align: left;
font-family: Lato ;
 
}
Dashboard and header component change:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ASSET/src/dashboard/components/Header.jsx
$ASSET/src/dashboard/components/Dashboard.jsx
./src/components/EditableTitle.jsx
./src/explore/components/ControlHeader.jsx
./src/dashboard/components/SaveModal.jsx
Also CSS list of changes in individual dashboards is possible from Superset dashboard pages itself
 


java.net.SocketTimeoutException in OBIEE/Weblogic : Read timed out


I can continue write a long thread about this issue that we have faced in regard of this SocketTimeoutException. However this time I will keep in concise.
Here is the issue : two of the prompts in our Production environment suddenly broken which was working fine for past 3-5 months. In between we have made bundle patch upgrade on our 11.1.1.7 env , we did network/firewall level changes so we were not sure why those two prompts started behaving weirdly. Also we found we had similar kind of prompt exist elsewhere which was not facing the vulnerabilities.

By broken, I mean when user click on the prompt they are keep getting “Please wait” appearing in the drop-down menu and the drop-down value never displayed. And this only happens when you are accessing application through web / outside your corporate network and through external DNS. The prompt works fine in internal VPN network.
Since we have thousands of login happen each day, the Production OBIEE environment crashes after couple of days because of this issue. By crashes we mean OBIEE/EM?Console login hangs , you will not find any process went down in opmnctl status -l , all alive , EM / Console even doesn’t show anything red(if you able to login here) determining its a crash .

Pretty interesting ….. !!! Huh ?

When we inspect the console log of browser we have spotted after 2-3 minutes the browser throws below error :

“Internal Server error” Ref :
500 Internal Server Error
ERROR Codes for Reference #3.5e7f1cb8.1547495087.4c779890

The above was Akamai reference number. And for this issue there is nothing to do with Akamai edge or origin server and its cache issue. We have checked Akamai has no issue with cache content

As soon we receive below error the ADR incident log for obiee , bi_Serverx.log and biserver_diagnostic.log throws below :

This appears in Incident log (change the path to be exact for your environment)
+++++++++++++++++++++++++++++
/u00/app/Middleware/user_projects/domains/bifoundation_domain/servers/bi_server1/

adr/diag/ofm/bifoundation_domain/bi_server1/incident/incdir_539
Incident Id: 7997
Incident Source: SYSTEM
Create Time: Tue Jan 15 12:11:11 EST 2019
Problem Key: DFW-99998 [java.net.SocketTimeoutException][oracle.security.jps.ee.http.JpsAbsFilter$1.run][analytics]
ECID: e8398b29c4083075:10e4302f:1685274d016:-8000-0000000000008333
Application Name: analytics
User Name: <WLS Kernel>
Error Message Id: DFW-99998

Context Values
————–
DFW_SERVER_NAME : bi_server1
DFW_DOMAIN_NAME : bifoundation_domain
DFW_USER_NAME : <WLS Kernel>
DFW_APP_NAME : analytics
Description
———–
Incident detected using watch rule “UncheckedException”:
Watch time: Jan 15, 2019 12:11:11 PM EST
Watch ServerName: bi_server1
Watch RuleType: Log
Watch Rule: (SEVERITY = ‘Error’) AND ((MSGID = ‘WL-101020’) OR (MSGID = ‘WL-101017’) OR (MSGID = ‘WL-000802’) OR (MSGID = ‘BEA-101020’) OR (MSGID = ‘BEA-101017’) OR (MSGID = ‘BEA-000802′))
Watch DomainName: bifoundation_domain
Watch Data:
DATE : Jan 15, 2019 12:11:11 PM EST
SERVER : bi_server1
MESSAGE : [ServletContext@756918633[app:analytics module:analytics path:/analytics spec-version:2.5 version:11.1.1]] Root cause of ServletException.
java.net.SocketTimeoutException: read is alrady timed out
——————–

THIS APPEARS IN BI_SERVERx.log
/u00/app/Middleware/user_projects/domains/bifoundation_domain/servers/bi_server2/logs/bi_server2.log
++++++++++++++++++++++++++++++
[WARNING:7] [WL-320068] [Diagnostics] [host: <hostname>] [nwaddr: [10.30.xx.xxx] [tid: [ACTIVE].ExecuteThread: ’71’ for queue: ‘weblogic.kernel.Default (self-tuning)’] [userId: <WLS Kernel>] [LOG_FILE: /u00/app/Middleware/user_projects/domains/bifoundation_domain/servers/bi_server2/logs/bi_server2.log] Watch ‘UncheckedException’ with severity ‘Notice’ on server ‘bi_server2’ has triggered at Jan 23, 2019 9:36:00 AM EST. Notification details: [[
WatchRuleType: Log
WatchRule: (SEVERITY = ‘Error’) AND ((MSGID = ‘WL-101020’) OR (MSGID = ‘WL-101017’) OR (MSGID = ‘WL-000802’) OR (MSGID = ‘BEA-101020’) OR (MSGID = ‘BEA-101017’) OR (MSGID = ‘BEA-000802′))
WatchData: DATE = Jan 23, 2019 9:36:00 AM EST SERVER = bi_server2 MESSAGE = [ServletContext@1881523291[app:analytics module:analytics path:/analytics spec-version:2.5 version:11.1.1]] Root cause of ServletException.
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:128)
at weblogic.servlet.internal.PostInputStream.read(PostInputStream.java:196)
at java.io.InputStream.read(InputStream.java:82)
at weblogic.servlet.internal.ServletInputStreamImpl.read(ServletInputStreamImpl.java:222)
at com.siebel.analytics.utils.InputStreamWithLimit.read(InputStreamWithLimit.java:39)
at com.siebel.analytics.utils.IOUtils.copyStreams(IOUtils.java:66)
at com.siebel.analytics.web.j2eeutils.SAWHttpUtils.getRequestBytes(SAWHttpUtils.java:39)
at com.siebel.analytics.web.integration.LoadBalancerHTTPFilter.determineServerFromRequest(LoadBalancerHTTPFilter.java:293)
at com.siebel.analytics.web.integration.LoadBalancerHTTPFilter.determineServerFromRequest(LoadBalancerHTTPFilter.java:243)
at com.siebel.analytics.web.integration.LoadBalancerHTTPFilter.doFilter(LoadBalancerHTTPFilter.java:175)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:324)
at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:460)
at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:171)
at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:163)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3748)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3714)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2283)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2182)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1499)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
SUBSYSTEM = HTTP USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: ’38’ for queue: ‘weblogic.kernel.Default (self-tuning)’ MSGID = WL-101017 MACHINE = <hostname> TXID = CONTEXTID = 0000MXv8etiFKAw_wDCCyW1SGacg003qNu TIMESTAMP = 1548254160834
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 30000
]]

++++++++++++++++++++++++++++

As you are seeing there are two variations of the log :

1) java.net.SocketTimeoutException: Read timed out
2) java.net.SocketTimeoutException: read is alrady timed out (note: the typo is not me and it is writing by the OBIEE product itself ! )

Essentially both are same and incident log says this is your problem key : “Problem Key: DFW-99998 [java.net.SocketTimeoutException][oracle.security.jps.ee.http.JpsAbsFilter$1.run][analytics]”

When we started investigating we have seen the prompt code is not special and we have used regular SQL results with SELECT statement and UNION statement.
Most weird part of if we use union all (as lowercase) it works and if we use FETCH ONLY 650001 ROWS clause at the end of the select statement of prompt logical query it works absolutely fine. Again these are the interesting facts which we can contradict later but these are our findings.

Here is the solution :

After going through several layers of network trace , Akamai , IDP Ping SSO , External / Internal DNS check, Checking the packets transfer across firewall , tracing the various logs we are seeing that when you click prompt the request headers even not passing through presentation server session log so its being stuck somewhere in the network and network is not able to process the request . Finally we came to know that we had security layer incorporated by Secureworks (that is a Intrusion Prevention System) which intercept the OBIEE browser POST request header and when it sees the SQL Injections operation (by SELECT or UNION ) via its Intrusion Prevention rule it blocks the inflow of traffic and that stops passing the request to other network layers and cause Weblogic server to go in unknown state. We had to allow those SQL rules across network as white listed traffic to stop this “SocketTimeoutException”.
As soon as we did this those 2 prompts started working fine but the “SocketTimeoutException” didn’t completely goes away . However the volume of this Exception reduced by 100x, 200x and we have seen no sign of crashing the OBIEE platform.

So in Summary :

Root Cause:

The Intrusion Prevention System misidentifying application traffic as malicious.

 Contributing factors:

  • OBIEE uses raw SQL statements for normal application functionality causing the IPS to misidentify the traffic as malicious.
  • OBIEE products do not utilize TLS.
  • Specific filter in the OBIEE, using those filters generate the socket timeout exception which create cascading effect having the product to become unresponsive
  • The server encountered an internal error or mis-configuration and was unable to complete the request.

Unfortunately, because of this Network issue, being in application team, we had to loads of hours to do root cause analysis before we figure this out. Facts of life for developers !!!