Troubleshooting your BAM Applications

Exclusive offer: get 50% off this eBook here
Oracle BAM 11gR1 Handbook

Oracle BAM 11gR1 Handbook — Save 50%

Your essential BAM sidekick for monitoring, alerting, and application best practices with Oracle Business Activity Monitoring with this book and ebook

$20.99    $10.50
by Pete Wang | November 2012 | SOA Oracle

In a typical business flow in BAM, business data moves from various upstream data sources to Oracle BAM, which internally processes and pushes the data to IE browsers for report rendering. As different technologies and BAM Server components are involved, it could be time-consuming and challenging to identify and solve the problem if anything goes wrong in this flow.

In this article by Pete Wang , author of Oracle BAM 11gR1 Handbook, we will cover:

  • Methodologies for troubleshooting Oracle BAM
  • Troubleshooting Active Data processing
  • Troubleshooting BAM HA issues

(For more resources related to this topic, see here.)

Understanding BAM logging and troubleshooting methodologies

In many cases, enabling BAM logging is the prerequisite for troubleshooting BAM issues. It is critical to set up the correct loggers to appropriate levels (for example, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, and FINEST), so that you can collect the information to identify the actual problem, and determine the root cause.

Apart from logging, it is also important to have the proven methodologies in place, so that you can follow these methods/procedures to conduct your troubleshooting practice.

Understanding BAM logging concepts

BAM provides a set of loggers, which are associated with BAM Java packages/ classes. Like Java, these loggers are named by following the dot convention, and are organized hierarchically.

Let's take a look at an example. oracle.bam.adc is the root logger for the Active Data Cache. All the loggers for the sub-modules within the Active Data Cache should be named after oracle.bam.adc, and therefore become descendants of the root logger. For instance, oracle.bam.adc.security, which is the logger that tracks Active Data Cache security logs, is the child logger of oracle.bam.adc.

The logging level for the descendant/child (for example, oracle.bam.adc.security)inherits from the ancestor/parent (for example, oracle.bam.adc adc) by default, unless its logging level is explicitly specified. Thus, you should be careful when setting a root or parent logger to a low level (for example, TRACE:32 ), which may produce a large amount of log entries in the log file

The following table lists the major root-level loggers for troubleshooting key BAM components:

Logger name

Description

oracle.bam.adapter

This is the logger for troubleshooting the BAM

Adapter issues

oracle.bam.adc

This is the logger for troubleshooting the BAM

Active Data Cache operations, such as data

persistence, ADC APIs, Active Data processing

with ADC, and so on

orable.bam.common

This is the logger for debugging BAM common

components, for example, BAM Security or BAM

Messaging Framework

oracle.bam.ems

This is the logger for debugging BAM Enterprise

Message Sources (EMS)

oracle.bam.eventengine

This is the logger for debugging the Event Engine

oracle.bam.reportcache

This is the logger for debugging the Report Cache

oracle.bam.web

This is the logger for debugging the BAM web

applications, which include the Report Server

oracle.bam.webservices

This is the logger for debugging the BAM web

services interface

Enabling logging for BAM

To set up loggers for BAM, perform the following steps:

  1. Log in to Enterprise Manager 11g Fusion Middleware Control.
  2. Click on OracleBamServer(bam_server1) from the left pane, and select BAM Server | Logs | Log Configuration.

  3. Expand oracle.bam to set the BAM loggers. To ensure that the log-level changes are persistent, check the checkbox for Persist log level state across component restarts.
  4. Click on Apply.

The logs are written to the <server_name>-diagnostic.log file in the <mserver_domain_dir>/servers/<server_name>/logs directory. By default, the log file follows the size-based rotation policy, and the rotational size is 10M. You can change the default behavior by editing the log file configuration as follows:

  1. Log in to Enterprise Manager 11g Fusion Middleware Control.
  2. Click on OracleBamServer(bam_server1) from the left pane, and select BAM Server | Logs | Log Configuration.
  3. In the Log Configurations screen, click on the Log Files tab.
  4. Select odl-handler, and then click on Edit Configuration
  5. Edit the log file configuration, and click on OK.

Setting BAM loggers to appropriate values

Similar to what most Fusion Middleware components do, Oracle BAM uses the lo g levels specified in the Oracle Diagnostic Logging (ODL) standard to control the log details in the diagnostic log file.

Similar to the log levels in Java, ODL log levels and their descriptions are listed in the following table:

Log level (Java)

Log level (ODL)

Description

SEVERE+100

INCIDENT_

ERROR:1

This log level enables the BAM Server that reports critical issues or fatal errors.

SEVERE

ERROR:1

This log level enables the BAM Server components to report issues (system errors, exceptions, or malfunctions) that may prevent the system from working properly.

WARNING

WARNING:1

This log level enables the BAM Server components to report events or conditions that should be reviewed and may require actions.

INFO

NOTIFICATION:1

This is the default setting for all the BAM loggers. This log level is used to capture the lifecycle events of BAM Server components and the key messages for notification purposes. For example, if you need to verify the cache location or its running status for the BAM Report Cache, you can set the report cache logger to this log level.

CONFIG

NOTIFICATION:16

This log level enables the BAM Server components to write more detailed configuration information.

FINE

TRACE:1

This log level enables the BAM Server components to write the debug information to the log file. To  troubleshoot the BAM server components, you may start with this log level, and increase to FINER or FINEST, if needed.

FINER

TRACE:16

This log level enables the BAM Server components to write a fairly detailed debug information to the log file.

FINEST

TRACE:32

This log level enables the BAM Server components to write a highly detailed debug information.

Inherited from parent

 

Specify this value if you want a specific logger to inherit the log level from its parent logger.

The default setting for all BAM loggers is NOTIFICATION:1. For troubleshooting purposes, it is recommended to set the appropriate loggers to TRACE:1, TRACE:16, or TRACE:32.

The logging configuration is persisted in the following location on your BAM host: <domain_dir>/config/fmwconfig/ servers/<server_name>/logging.xml. Theoretically, you can edit this file to modify BAM loggers. However, it is not recommended to do so unless you have a good understanding of the configuration file. If you have multiple BAM instances in your environment, you can easily duplicate the logging configuration by copying the logging.xml file to all BAM instances, rather than making changes through EM.

Introducing the methodologies for troubleshooting BAM

Oracle BAM utilizes different technologies such as EMS, BAM Adapter, Web services, and ODI to integrate different enterprise information systems. Business data received from various data sources are then pushed all the way from the Active Data Cache, through the Report Cache and the Report Server, to Web browsers for rendering reports in real-time. Due to the complexity of the system and various technologies involved, it is critical to use the right troubleshooting methodologies to analyze and resolve issues.

The following are the basic rules of thumb for troubleshooting your BAM applications:

  • Understand the BAM key terminologies and architecture
  • Identify the problem
  • Set up BAM loggers
  • Collect the information and debugging

Understanding the key terminologies and the BAM architecture

Understanding the key terminologies and the BAM architecture is a prerequisite for troubleshooting the BAM applications. The key terminologies for BAM include Active Data, Active Data Cache, Report Cache, Report Server, ChangeList, Data Objects, Report, and so on.

Identifying the problem

Different issues may require different techniques to troubleshoot. For example, for a report design issue (for example, calculated fields do not show correct values), you should focus on the building blocks for the report design, instead of enabling loggings for the BAM server, which does not provide any help at all.

A BAM issue typically falls into the following categories:

  • Report design and report loading (static rendering)
  • Report rendering with Active Data (Active Data processing)
  • Issues with the key BAM Server components (Active Data Cache, securities, Report Cache, and Event Engine)
  • BAM Web applications (Active Viewer, Active Studio, Architect, Administrator, and Report Server)
  • Issues with BAM Integration Framework (EMS, Web services APIs, SOA Suite integration, and ODI integration)

To fully identify a problem, you need to first understand the problem description and the category to which the issue belongs. Then, you can gather relevant information, such as the time frame of when the issue happened, the BAM Server release information and patch level, BAM deployment topologies (single node or HA), and so on.

Setting up BAM loggers

Setting up BAM loggers with appropriate logging levels is the key for troubleshooting BAM issues.

BAM loggers can be set up to the following levels (Java logging): SEVERE, WARNING, INFO, CONFIG, FINE, FINER, and FINEST. In a normal situation, all the BAM loggers are set to INFO, by default. In the case of debugging, it is recommended to increase the level to FINER or FINEST.

Loggers contain hierarchies. You need to be careful when setting up root level loggers to FINE, FINER, or FINEST. Suppose that you want to troubleshoot a login issue with the BAM start page using the root logger oracle.bam.adc, which is set to FINEST. In this case, all the descendants that inherit from it have the same logging level. As a result, a large amount of unused log entries are produced, which is not helpful for troubleshooting, and can also impact the overall performance. Therefore, you should set up the corresponding child logger (oracle.bam.adc.security) without enabling the root logger.Collecting information and debugging.

Once the problem is identified and BAM loggers are set up appropriately, it is time to collect the logs to analyze the problem.

The following table lists the files that can be used to troubleshoot BAM issues:

Name

Description

<server_name>.out

This is the standard output file redirected by the Node Manager. If the server is started using the  tartManagedWebLogic.sh script directly, you need to refer to the standard output (either the command prompt output or another redirected file specified). This file is located in the following directory: <mserver_domain_dir>/servers/<server_name>/logs.

<mserver_domain_dir> refers to the domain home directory for the Managed Server for BAM; <server_name> refers to the name of the Managed Server for BAM, for example, WLS_BAM1.

Use this file to collect the server starting information and standard output.

<server_name>.log

This log provides information for the WebLogic Server that hosts BAM. This file is located in the following directory:

<mserver_domain_dir>/servers/<server_name>/logs.

<server_name>-diagnostic.log

Unlike the <server_name>.log file, this log file keeps a track of BAM-specific logs produced by BAM loggers.

The location of this file is as follows: <mserver_domain_dir>/servers/<server_name>/logs.

Debugging actually becomes easier once you have all this relevant information in place.

Oracle BAM 11gR1 Handbook Your essential BAM sidekick for monitoring, alerting, and application best practices with Oracle Business Activity Monitoring with this book and ebook
Published: April 2012
eBook Price: $20.99
Book Price: $34.99
See more
Select your format and quantity:

Troubleshooting the BAM applications

In this section, you will learn how to troubleshoot the BAM Active Data processing issues and the issues around HA.

Troubleshooting the Active Data processing issues

BAM uses the pushed-based mechanism to process Active Data. As illustrated in the following diagram, the Active Data Cache produces Active Data based on the ViewSet, and data changes in associated Data Objects. After that, the Active Data is sent to the Report Cache and then the Report Server for processing:

Theoretically, Active Data processing could fail in any step of the fl ow (Active Data Cache, Report Cache, Report Server, network, or JavaScript issues at the client side).

Troubleshooting Report Server issues

At the BAM Server side, Active Data is originally produced in the Active Data Cache, and then pushed forward to the Report Cache and the Report Server. The Report Server is the component that maintains the persistent HTTP connections between the client and BAM Server, and pushes Active Data payloads (ChangeLists) asynchronously to the client.

Since Active Data processing in the Report Server is the last step in the processing chain at the server side, the confirmation of a successful Active Data processing in the Report Server implies that Active Data processing in the Report Cache and the Active Data Cache are also successful. Thus, it is best practice to start diagnosing the Report Server first, and then move on to the Report Cache and the Active Data Cache for further analysis.

To diagnose the Active Data issues in the Report Server, enable the loggers as shown in the following table:

Logger name

Description

oracle.bam.web.

reportserver.activedata.

ActiveDataPage

(ActiveDataPage)

This logger tracks the push events in the Report Server, and can be used to determine whether the Report Server has successfully pushed the Active Data to the client or not.

oracle.bam.web.

reportserver.activedata.

ActiveDataServlet

(ActiveDataServlet)

This logger records asynchronous HTTP request/response-related information, which is useful to debug persistent connections issues between the client and the BAM Server.

oracle.bam.web.

reportserver.activedata.

ActiveDataViewSet

(ActiveDataViewSet)

This logger keeps a track of the ChangeLists processing details in the Report Server.

Note that the recommended logging level for debugging is TRACE:32 (FINEST).

Enabling ActiveDataPage

To troubleshoot the Active Data processing issues in the Report Server, the first thing to do is set up the appropriate logging level for the ActiveDataPage logger. Once this logger is set to TRACE:32 (FINEST), you can then search the <server_name>-diagnostic.log file to look for the flushAllActiveData or ChangeList keywords.

The following is an example of the logs produced by the ActiveDataPage logger:

In this example, the ChangeList XML payload contains the index, the viewsetID attributes, and the Record of the Data Object (Employee Data Object in the /Samples folder), which are passed to the client (Web browser) by the Report Server through the invocation of the ActiveDataPage.flushAllActiveData method. Thus, you can utilize the ActiveDataPage logger and the generated logs to determine whether the Report Server has asynchronously pushed the Active Data to the client successfully.

Enabling ActiveDataViewSet

You should set up the logging level for the ActiveDataViewSet logger to TRACE:32 (FINEST), in order to view the ChangeList processing information in detail. The log entries typically include the following information:

  • processing index: This is the index of theChangeList being currently processed
  • last index: This is the index of the previously processed ChangeList
  • strViewSetID: This is the ID of the ViewSet that produces the ChangeList
  • strActiveDataSessionID: This is the ID of the ActiveDataSession that is the session context for processing the Active Data
  • strReportID: This is the ID of the BAM report that is the receipt of the Active Data
  • writing response script to client: These are the scripts that are passed to the client for processing the Active Data
  • processed index: This is the index of the ChangeList that has been processed

The following is an excerpt of the logs produced by ActiveDataViewSet, and you can rely on these logs to collect more information regarding the processing details of the Active Data:

inside ActiveDataViewSet.onMessage()...
processing index: 1
last index: 0
strViewSetID = b5bdcb24408df9fbce6c167133c5167bac-76fe
strActiveDataSessionID = b5bdcb24408df9fbce6c167133c5167bac_7701
strReportID = b5bdcb24408df9fbce6c167133c5167bac_7701
writing response script to client: [[

]]
processed index: 1

Enabling ActiveDataServlet

The corresponding ActiveDataServlet java class is the asynchronous Java Servlet that actually pushes the Active Data to the client through the previously established persistent connection when the Active Data arrives in the Report Server.

To debug the ActiveDataServlet related issues, set the ActiveDataServlet logger to TRACE:32 (FINEST), and then search the ActiveDataServlet keywords to look for the information.

The following is an example of the logs produced by the ActiveDataServlet servlet:

doRequest: Method = POST; URL = http://bamhost1:9001/OracleBAM/
ActiveDataServlet; Query = null; Session ID = nls4PB8CpvJ3L
KkqvJ1LGG2pp1qMBnTYhTyPtXV3n2xnDF2Qr8n5!1135941767!1325513922380; Is
Session Valid = true; Is Session From Cookie = true; Is Session From
URL = false

Right after a new report/dashboard is rendered for the first time, it sends an HTTP POST request to the ActiveDataServlet, which registers the client (report/dashboard) to ensure it gets called back when the Active Data arrives at the Report Server in the future. Therefore, the previous message typically appears at this time frame, and can be used to troubleshoot the client/server communication issues.

Troubleshooting the Active Data Cache and the Report Cache

The following are the scenarios in which you should consider turning on loggers to troubleshoot Active Data processing in the Active Data Cache or the Report Cache:

  • The ActiveDataPage flushAllActiveData method is never called to push the Active Data to the client.
  • The ActiveDataPage flushAllActiveData method is invoked, however, the parameters passed to this method are incorrect. For example, the XML payload for the ChangeList is incorrect.
  • The error messages in the log file indicate that the errors happen in one of the oracle.bam.adc.* or oracle.bam.reportcache.* modules.

The following table lists the common loggers for the Active Data Cache and the Report Cache:

Logger name

Description

oracle.bam.adc.

kernel.viewsets.

PushViewset

(PushViewset)

This logger tracks the ViewSet and ChangeList related activities, and can be used to verify the push event for ChangeLists.

oracle.bam.adc.

kernel.datasets.

Dataset (Dataset)

This logger provides you Data Object-related information, for example, the data changes made to the Data Object. You can use this logger to determine if the Data Object operations (Insert, Update, Upsert, and Delete) are successful or not.

oracle.bam.reportcache

This logger keeps a track of the activities in the Report Cache, which can be used to determine the ChangeList and the ViewSet caching issues.

The following are the excerpts of the logs produced by the PushViewset logger:

[109] Timer task 1: Pushed changelist 1 for viewset 7fcce3dcef0c2343-
677b43b123bfcc5665-7fd0.
[109] Timer task 1: Completed.

As you can see, these logs indicate that changelist 1 for viewset 7fcce3dcef0c2343-677b43b123bfcc5665-7fd0 has been pushed by the Active Data Cache to the Report Cache for further processing.

Troubleshooting the client-side issues

BAM utilizes the asynchronous JavaScript and XML (AJAX) to achieve asynchronous client/server communications and client-side Active Data processing. With AJAX, your BAM report/dashboard can send the initial data to the BAM Server and receive the Active Data asynchronously without having to reload the whole UI.

Using AJAX allows you to build your BAM applications in an effective and efficient way. However, troubleshooting the AJAX aspect of the code is challenging. In this section, you will learn how to debug client-side issues, specifically the JavaScripts for processing the Active Data.

Enabling client-side logging

ActiveData.js is the JavaScript that handles Active Data processing at the client side. The following are the scenarios in which you should review client-side logging (logs produced by the ActiveData.js JavaScript):

  • It is verified that the BAM Server has successfully processed the Active Data, and pushed them to Web browser(s). However, the changes seen on the server side are not reflected in your BAM report/dashboard.
  • BAM report views stop refreshing while showing the Reconnecting… message in one or more views.

To enable the AciveData.js logging, perform the following steps:

  1. Open a command prompt on a machine (for example, bamhost1) that hosts Oracle BAM. Change the directory to the following:

    <domain_dir>/servers/<server_name>/tmp/_WL_user/oraclebam_
    11.1.1.

    <domain_dir> represents the WebLogic Server domain directory for BAM. <server_name> refers to the name of the Managed Server for BAM.

  2. Locate the ActiveData.js file under the current directory. On Unix/Linux, use the find command as follows:

    bamhost1> find . -name ActiveData.js

  3. The find command returns multiple occurrences of ActiveData.js as shown in the next screenshot. The highlighted location that contains both the activedata directory and a build number (for example, 13846) is the one you should edit:
  4. Make a backup of the following file, and then edit it in a text editor:

    /eow4lx/war/13846/reportserver/scripts/activedata/ActiveData.js.

  5. Replace all occurrences of the string // LOG: with an empty string "".
  6. Open an Internet Explorer (IE) web browser on your client host.
  7. In the menu bar, select Tools | Internet Options.
  8. In the General tab, click on the Delete... button to clear Temporary Internet Files and Cookies.
  9. Click on the Security tab, and then select the Local intranet zone.
  10. Select the Custom level... button.
  11. In the ActiveX controls and plug-ins section, select Enable for every item.
  12. Click on the Internet zone, and repeat steps 10 to 11.
  13. Click on OK to save all the changes.

With the JavaScript logging enabled, the client-side Active Data processing details are logged in the ActiveData<timestamp>.txt file in the C:\Temp directory. Search the ProcessActiveData or the ChangeList keywords in the log file, and you should get a result similar to the following:

This information is similar to what you saw earlier in the server log file. The ChangeList XML document is the payload, which is to be processed by the ActiveData.js JavaScript. Once it is processed successfully, the client JavaScript should make the changes accordingly in the report view.

Oracle BAM 11gR1 Handbook Your essential BAM sidekick for monitoring, alerting, and application best practices with Oracle Business Activity Monitoring with this book and ebook
Published: April 2012
eBook Price: $20.99
Book Price: $34.99
See more
Select your format and quantity:

Case study

Suppose that you have designed a BAM report/dashboard that contains one or more tab views, and the following issue occurred in one of your environments.

You found that some report views stopped loading or refreshing once the Reconnecting message appeared in one or more views. The initial rendering of the report worked as expected.

Now let's take a look at a case study that showcases how we use the troubleshooting methodologies to diagnose this issue.

Identifying the problem

To accurately and effectively describe the problem, you are required to collect the relevant information from the system. The following is an example of the type of information that you may collect:

  • Hardware and software specifications (for example, machine specifications, operating systems, JVMs, Application Server, BAM release information, patching levels, and so on)
  • BAM deployment topology (a single node environment or an HA environment)
  • The detailed steps to reproduce the problem if applicable
  • The BAM Server standard output file (for example, <server_name>.out) and diagnostic log file (for example, <server_name>-diagnostic.log)

Once you have gathered the required information and identified the problem, you can start diagnosing the issue. The common mistake that most people often make is that they attempt to try different solutions, and hope that they will work without actually understanding the issue and determining possible causes. Our experiences tell us that having a thorough understanding of your environment and the problem itself is the key to diagnosing and troubleshooting problems. For example, you need to first attempt to find out information, such as when did the issue happen?, in what context did the issue occur?, and so on, before jumping to implement any possible solutions.

Diagnosing the problem

To diagnose a complex issue, you need to figure out a plan that can help you to narrow down the problem, and detect the root cause.

When troubleshooting this particular issue, it is critical to understand why the Reconnecting... message is displayed in the report views. To ensure the continuous Active Data processing at the client side, the Web browser must free up its memory consumption and other resources, by executing a function from the ActiveData.js JavaScript. As a result, the Web browser will attempt to reconnect the BAM Report Server, and then you will observe the Reconnecting... message, which is the indicator of such a reconnecting event. Inside the code of the JavaScript, the iActiveDataScriptsCleanupFactor parameter is introduced to specify how frequently this event will be triggered.

After you understand the reconnecting behavior, it is time for us to diagnose the real problem, which is why the Web browser stopped processing the Active Data. You need to enable logging for ActiveData.js, and then inspect the log file (for example, C:\temp\ActiveData<timestamp>.txt) at the client side.

Search the MonitorMemoryLeaked keyword in the log file, and you may end up finding the following message:


ActiveData: MonitorMemoryLeaked: g_iMemoryLeakThresholdChars =
1048576this.m_iMemoryLeaked = 1049100

In this message there are two variables, which are explained as follows:

  • g_iMemoryLeakThresholdChars: This variable represents a threshold value defined in the iActiveDataScriptsCleanupFactor parameter for Oracle BAM. The default value for iActiveDataScriptsCleanupFactor is 1048576 (bytes), and you can view and change this parameter in BAMWebConfig. xml, which is located in the <bam_domain_dir>/config/fmwconfig/ servers/<server_name>/applications/oracle-bam_11.1.1/config directory.
  • m_iMemoryLeaked: This variable represents the total characters (in bytes) of all ChangeList payloads that have been processed in ActiveData.js.

The MonitorMemoryLeaked message is written to the log file only when m_iMemoryLeaked reaches the threshold value (1048576 bytes by default) specified in the iActiveDataScriptsCleanupFactor parameter. In the meantime, ActiveData.js attempts to free up memory consumed by IE, close the previous connection, and then open a new connection to the BAM Report Server.

The default value for iActiveDataScriptsCleanupFactor is 1048576 bytes, which is not sufficient in most cases, especially in the production environment. You can set this parameter to a large value (for example, 10 MBs), and you should notice that the Reconnecting... scenario does not happen that often after that

Solution

Tuning iActiveDataScriptsCleanupFactor does not completely resolve the issue; however, it does help to narrow down the possible causes, which should be related to the newly established HTTP connection, or the transmission of the messages through the connection. In the end, it is identified that the issue is caused by JavaScript being executed in the same IFRAME name, which is shared by two or more tab views. Applying Patch 12419288 on top of the 11.1.1.4 release should fix the problem. If you are on 11.1.1.6 or any later releases, the issue is already fixed, and no patches are needed.

In summary, this case demonstrates how to use the methodology to troubleshoot a complex issue, and the importance of enabling JavaScript logging in the scenario of client-side Active Data processing.

Troubleshooting BAM HA issues

In a BAM HA environment, the common issues you might encounter are as follows: one Managed Server for BAM cannot start up, the BAM start page or report cannot be accessed using the load balancer URL, the BAM Server is down and does not fail over to another node, and so on. These issues are BAM HA configuration-related.

The majority of BAM HA issues that we have seen are configuration-related, and are typically caused by inappropriate configuration in the WebLogic Server domainlevel or BAM components-level.

The following are the guidelines to troubleshoot such HA issues:

  • Understanding the BAM HA topology
  • Inspecting log files for the Managed Server for BAM
  • Setting up appropriate loggers

Understanding the topology is critical for troubleshooting HA issues. In a typical BAM HA environment, you need to review the WebLogic domain topology, and collect the following key information:

  • Collecting BAM cluster information, such as the cluster address, cluster messaging mode (Unicast or Multicast), servers that are assigned to the cluster (for example, WLS_BAM1 and WLS_BAM2), and so on
  • Identifying the WLS Server instance (for example, WLS_BAM1 or WLS_BAM2) on which the BAM Server components are running
  • Verifying BAM deployment

Inspecting log files is a must for troubleshooting BAM issues in an HA environment.

The following table lists the files that can be used to troubleshoot these issues:

 

Name

Description

<server_name>.out

This file can be found in the following directory:

<mserver_domain_dir>/servers/<server_name>/logs.

<mserver_domain_dir> refers to the domain home directory for the Managed Server for BAM, for example, ORACLE_BASE/admin/<domain_name>/mserver/<domain_name>.

<server_name> refers to the name of the Managed Server for BAM, for example, WLS_BAM1.

<server_name>-diagnostic.log

The location of this file is as follows: <mserver_domain_dir>/servers/<server_name>/logs.

Use this log file to troubleshoot issues with BAM server components (for example, securities, ADC, and so on)

There are no specific loggers for debugging HA issues, especially configuration issues. However, some BAM components, such as Active Data Cache or Messaging Framework, may throw exceptions at runtime. To debug such BAM-specific issues, enabling loggers will help to identify the problem.

Case study

In this case study, you will learn how to troubleshoot a typical BAM HA startup issue.

Identifying the problem

In an HA environment that contains both the Managed Server for BAM and the Managed Server for SOA, if the Managed Server for BAM is started before the Managed Server for SOA, an authentication error happens when trying to log in to the BAM start page. The login issue gets resolved by restarting the Managed Server for BAM.

Diagnosing the problem

To analyze this problem, you need to first review the HA topology. In this case, the Weblogic Server domain contains the Manager Servers for both BAM and SOA.

After this, you need to review the <server_name>-diagnostic.log file, which shows the following snippets:

Caused by: java.security.AccessControlException:
access denied
(oracle.security.jps.service.policystore.
PolicyStoreAccessPermission
Context:APPLICATION Context Name:soa-infra Actions:
getApplicationPolicy)
at java.security.AccessControlContext.checkPermission
(AccessControlConte
xt.java:323)

The underlined code indicates that the issue is related to the policy store configuration. By default, BAM stores its policies and application roles in the filebased policy store, which is system-jazn-data.xml in the <server_domain_dir>/config/fmwconfig directory. Note that <mserver_domain_dir> refers to the domain directory for the Managed Server, and is different than the <aserver_domain_dir>, which represents the domain directory for the Administration Server.

When the BAM server is started, it initiates the default policies and application roles in system-jazn-data.xml if they do not exist in the policy store. When SOA server gets started, it appends the SOA-related policies and application roles to the same file. However, in an HA environment, these BAM security information may get overwritten by SOA, or vice versa.

What happens in this scenario is that there are two system-jazn-data.xml files located in the <mserver_domain_dir>/config/fmwconfig directory and the <aserver_domain_dir>/config/fmwconfig directory, respectively. When BAM is started from the <mserver_domain_dir> location, these BAM security details are only written to the file in the <mserver_domain_dir>/config/fmwconfig directory. As the original system-jazn-data.xml file resided in <aserver_domain_dir>, is not updated, it overwrites the same file in <mserver_domain_dir> when the Managed Server for SOA is started.

Solution

Once the root cause is determined, it is time to work out the solution.

The ideal solution is to use a centralized policy store, which can be either OID or Oracle Database-based.

If you don't want to use a centralized policy store to store the policies and the application roles, you need to perform the following steps to solve this problem:

  1. Use the WebLogic Server script (for example, startManagedWebLogic.sh) in the <aserver_domain_dir> directory to start the Managed Server for SOA and the Managed Server for BAM separately, which ensures that < server_domain_dir>/config/fmwconfig/system-jazn-data.xml contains the policies and the application roles for both SOA and BAM.
  2. Stop both the Managed Servers.
  3. Delete the the oracle-bam_11.1.1 directory from the following location: <aserver_domain_dir>/config/fmwconfig/servers/<server_name>/applications.
  4. Start both the Managed Servers from the <mserver_domain_dir> directory. This ensures that the system-jazn-policy.xml file is copied from the <aserver_domain_dir>/config/fmwconfig directory to the <mserver_domain_dir>/config/fmwconfig directory.

Oracle BAM has three configuration files, which are BAMCommonConfig.xml, BAMWebConfig.xml, and BAMServerConfig.xml, located in the <domain_dir>/config/fmwconfig/servers/<server_name>/applications/oracle-bam_11.1.1/config directory. As it is recommended to create two separate domain directories for the Administration Server and the Managed Server for BAM, you need to ensure that such configuration files only exist in the <mserver_domain_dir> directory.

Summary

It is obvious that troubleshooting the BAM applications varies from one system to another. However, the troubleshooting methodologies and guidelines remain the same. So, in the article, we focused more on methodologies and troubleshooting guidelines, so that you can apply the techniques used in the case study to real-world scenarios.

Resources for Article :

 


Further resources on this subject:


About the Author :


Pete Wang

Pete Wang is part of the Oracle SOA team that focuses on providing best-in-class technical expertise and services around SOA technologies, to global key opportunities and strategic accounts. Pete’s specialized areas are SOA, JavaEE, Integration, Memory based real-time business event analytics and monitoring solutions (BAM/CEP).

Pete is the Global Technical Lead for BAM, which is a strategic role to lead product the readiness program and knowledge management, actively engage in customer escalation and advanced resolution to ensure the success of customers. Prior to that, Pete took Sales Consulting roles at both Oracle and BEA Systems.

Pete has 12 years experience in the design and development of SOA/JavaEE applications, and specializes in designing and troubleshooting large-scale and mission-critical systems built with various middleware technologies. Pete demonstrated strong technical and soft skills by successfully executing POCs, architecture review, benchmarks, performance tuning and support for key pre-sales and post-sales opportunities and clients.

Books From Packt


Oracle SOA Infrastructure Implementation Certification Handbook (1Z0-451)
Oracle SOA Infrastructure Implementation Certification Handbook (1Z0-451)

Oracle BPM Suite 11g Developer's cookbook
Oracle BPM Suite 11g Developer's cookbook

Getting Started with Oracle BPM Suite 11gR1 – A Hands-On Tutorial
Getting Started with Oracle BPM Suite 11gR1 – A Hands-On Tutorial

Getting Started With Oracle SOA Suite 11g R1 – A Hands-On Tutorial
Getting Started With Oracle SOA Suite 11g R1 – A Hands-On Tutorial

Oracle SOA Suite Developer's Guide
Oracle SOA Suite Developer's Guide

Oracle Fusion Middleware Patterns
Oracle Fusion Middleware Patterns

Business Process Driven SOA using BPMN and BPEL
Business Process Driven SOA using BPMN and BPEL

The Business Analyst's Guide to Oracle Hyperion Interactive Reporting 11
The Business Analyst's Guide to Oracle Hyperion Interactive Reporting 11


No votes yet

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
9
s
z
g
i
7
Enter the code without spaces and pay attention to upper/lower case.
Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software