Oracle GoldenGate 11g: Configuration for High Availability

Share this page via Facebook, Twitter or LinkedIn.

Your message has been sent.

How would you like to send this article:

Save this article to your account for easy access.

by John P Jeffries | February 2011 | Enterprise Articles Oracle

High Availability (HA) has become an important factor in computer system design in recent years. Systems can't afford to be down, not even for a minute, as they may be mission critical, life supporting, regulatory, or the financial impact may be too great to bear. Oracle has played a major role in developing a number of HA solutions, one of which is Real Application Clusters (RAC). Oracle Streams is heavily integrated with RAC out of the box and requires no additional configuration. This is not true for GoldenGate, where the Manager process has to be made "RAC aware".

In this article by John P. Jeffries, author of Oracle GoldenGate Implementer's Guide, we learn how to configure GoldenGate in a RAC environment and explore the various components that effectively enable HA for data replication and integration.

Oracle GoldenGate 11g Implementer's guide

Design, install, and configure high-performance data replication solutions using Oracle GoldenGate

The very first book on GoldenGate, focused on design and performance tuning in enterprise-wide environments
Exhaustive coverage and analysis of all aspects of the GoldenGate software implementation, including design, installation, and advanced configuration
Migrate your data replication solution from Oracle Streams to GoldenGate
Design a GoldenGate solution that meets all the functional and non-functional requirements of your system
Written in a simple illustrative manner, providing step-by-step guidance with discussion points
Goes way beyond the manual, appealing to Solution Architects, System Administrators and Database Administrators

Read more about this book

(For more resources on Oracle, see here.)

This includes the following discussion points:

Shared storage options
Configuring clusterware for GoldenGate
GoldenGate on Exadata
Failover

We also touch upon the new features available in Oracle 11g Release 2, including the Database Machine, that provides a "HA solution in a box".

GoldenGate on RAC

A number of architectural options are available to Oracle RAC, particularly surrounding storage. Since Oracle 11g Release 2, these options have grown, making it possible to configure the whole RAC environment using Oracle software, whereas in earlier versions, third party clusterware and storage solutions had to be used. Let's start by looking at the importance of shared storage.

Shared storage

The secret to RAC is "share everything" and this also applies to GoldenGate. RAC relies on shared storage in order to support a single database having multiple instances, residing on individual nodes. Therefore, as a minimum the GoldenGate checkpoint and trail files must be on the shared storage so all Oracle instances can "see" them. Should a node fail, a surviving node can "take the reins" and continue the data replication without interruption.

Since Oracle 11g Release 2, in addition to ASM, the shared storage can be an ACFS or a DBFS.

Automatic Storage Management Cluster File System (ACFS)

ACFS is Oracle's multi-platform, scalable file system, and storage management technology that extends ASM functionality to support files maintained outside of the Oracle Database. This lends itself perfectly to supporting the required GoldenGate files. However, any Oracle files that could be stored in regular ASM diskgroups are not supported by ACFS. This includes the OCR and Voting files that are fundamental to RAC.

Database File System (DBFS)

Another Oracle solution to the shared filesystem is DBFS, which creates a standard file system interface on top of files and directories that are actually stored as SecureFile LOBs in database tables. DBFS is similar to Network File System (NFS) in that it provides a shared network file system that "looks like" a local file system.

On Linux, you need a DBFS client that has a mount interface that utilizes the Filesystem in User Space (FUSE) kernel module, providing a file-system mount point to access the files stored in the database.

This mechanism is also ideal for sharing GoldenGate files among the RAC nodes. It also supports the Oracle Cluster Registry (OCR) and Voting files, plus Oracle homes.

DBFS requires an Oracle Database 11gR2 (or higher) database. You can use DBFS to store GoldenGate recovery related files for lower releases of the Oracle Database, but you will need to create a separate Oracle Database 11gR2 (or higher) database to host the file system.

Configuring Clusterware for GoldenGate

Oracle Clusterware will ensure that GoldenGate can tolerate server failures by moving processing to another available server in the cluster. It can support the management of a third party application in a clustered environment. This capability will be used to register and relocate the GoldenGate Manager process.

Once the GoldenGate software has been installed across the cluster and a script to start, check, and stop GoldenGate has been written and placed on the shared storage (so it is accessible to all nodes), the GoldenGate Manager process can be registered in the cluster. Clusterware commands can then be used to create, register and set privileges on the virtual IP address (VIP) and the GoldenGate application using standard Oracle Clusterware commands.

The Virtual IP

The VIP is a key component of Oracle Clusterware that can dynamically relocate the IP address to another server in the cluster, allowing connections to failover to a surviving node. The VIP provides faster failovers compared to the TCP/IP timeout based failovers on a server's actual IP address. On Linux this can take up to 30 minutes using the default kernel settings!

The prerequisites are as follows:

The VIP must be a fixed IP address on the public subnet.
The interconnect must use a private non-routable IP address, ideally over Gigabit Ethernet.

Use a VIP to access the GoldenGate Manager process to isolate access to the Manager process from the physical server. Remote data pump processes must also be configured to use the VIP to contact the GoldenGate Manager.

The following diagram illustrates the RAC architecture for 2 nodes (rac1 and rac2) supporting 2 Oracle instances (oltp1 and oltp2). The VIPs are 11.12.1.6 and 11.12.1.8 respectively, in this example:

The user community or application servers connect to either instance via the VIP and a load balancing database service, that has been configured on the database and in the client's SQL*Net tnsnames.ora file or JDBC connect string.

The following example shows a typical tnsnames entry for a load balancing service. Load balancing is the default and does not need to be explicitly configured. Hostnames can replace the IP addresses in the tnsnames.ora file as long as they are mapped to the relevant VIP in the client's system hosts file.

OLTP =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = 11.12.1.6)(PORT = 1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 11.12.1.8)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = oltp)
    )
  )

This is the recommended approach for scalability and performance and is known as active-active. Another HA solution is the active-passive configuration, where users connect to one instance only leaving the passive instance available for node failover.

The term active-active or active-passive in this context relates to 2-node RAC environments and is not to be confused with the GoldenGate topology of the same name.

On Linux systems, the database server hostname will typically have the following format in the /etc/hosts file.

For Public VIP: <hostname>-vip

For Private Interconnect: <hostname>-pri

The following is an example hosts file for a RAC node:

127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

#Virtual IP Public Address
11.12.1.6 rac1-vip rac1-vip
11.12.1.8 rac2-vip rac2-vip

#Private Address
192.168.1.33 rac1-pri rac1-pri
192.168.1.34 rac2-pri rac2-pri

Creating a GoldenGate application

The following steps guide you through the process of configuring GoldenGate on RAC. This example is for an Oracle 11g Release 1 RAC environment:

Install GoldenGate as the Oracle user on each node in the cluster or on a shared mount point that is visible from all nodes. If installing the GoldenGate home on each node, ensure the checkpoint and trails files are on the shared filesystem.
Ensure the GoldenGate Manager process is configured to use the AUTOSTART and AUTORESTART parameters, allowing GoldenGate to start the Extract and Replicat processes as soon as the Manager starts.
Configure a VIP for the GoldenGate application as the Oracle user from 1 node.
```
<CLUSTERWARE_HOME>/bin/crs_profile -create ggsvip \
-t application \
-a <CLUSTERWARE_HOME>/bin/usrvip \
-o oi=bond1,ov=11.12.1.6,on=255.255.255.0
```
CLUSTERWARE_HOME is the oracle home in which Oracle Clusterware is installed. E.g. /u01/app/oracle/product/11.1.0/crs
ggsvip is the name of the application VIP that you will create.
oi=bond1 is the public interface in this example.
ov=11.12.1.6 is the virtual IP address in this example.
on=255.255.255.0 is the subnet mask. This should be the same subnet mask for the public IP address.
Next, register the VIP in the Oracle Cluster Registry (OCR) as the Oracle user.
```
<CLUSTERWARE_HOME>/bin/crs_register ggsvip
```
Set the ownership of the VIP to the root user who assigns the IP address. Execute the following command as the root user:
```
<CLUSTERWARE_HOME>/bin/crs_setperm ggsvip -o root
```
Set read and execute permissions for the Oracle user. Execute the following command as the root user:
```
<CLUSTERWARE_HOME>/bin/crs_setperm ggsvip -u user:oracle:r-x
```
As the Oracle user, start the VIP.
```
<CLUSTERWARE_HOME>/bin/crs_start ggsvip
```

To verify the the VIP is running, execute the following command then ping the IP address from a different node in the cluster.

<CLUSTERWARE_HOME>/bin/crs_stat ggsvip -t
Name 	Type 	Target 	State 	Host
------     -------   ------   -------   ------
ggsvip 	application ONLINE 	ONLINE 	rac1

ping -c3 11.12.1.6
64 bytes from 11.12.1.6: icmp_seq=1 ttl=64 time=0.096 ms
64 bytes from 11.12.1.6: icmp_seq=2 ttl=64 time=0.122 ms
64 bytes from 11.12.1.6: icmp_seq=3 ttl=64 time=0.141 ms
--- 11.12.1.6 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.082/0.114/0.144/0.025 ms

Oracle Clusterware supports the use of "Action" scripts within its configuration, allowing bespoke scripts to be executed automatically during failover. Create a Linux shell script named ggs_action.sh that accepts 3 arguments: start, stop or check. Place the script in the <CLUSTERWARE_HOME>/crs/public directory on each node or if you have installed GoldenGate on a shared mount point, copy it there.
- Ensure that start and stop: returns 0 if successful, 1 if unsuccessful.
- check: returns 0 if GoldenGate is running, 1 if it is not running.
As the Oracle user, make sure the script is executable.
```
chmod 754 ggs_action.sh
```
To check the GoldenGate processes are running, ensure the action script has the following commands. The following example can be expanded to include checks for Extract and Replicat processes:
- First check the Linux process ID (PID) the GoldenGate Manager process is configured to use.
```
GGS_HOME=/mnt/oracle/ggs 	# Oracle GoldenGate
home
pid=`cut -f8 ${GGS_HOME}/dirpcs/MGR.pcm`
```
- Then, compare this value (in variable $pid) with the actual PID the Manager process is using. The following example will return the correct PID of the Manager process if it is running.
```
ps -e |grep ${pid} |grep mgr |cut -d " " -f2
```

The code to start and stop a GoldenGate process is simply a call to ggsci.

ggsci_command=$1
ggsci_output=`${GGS_HOME}/ggsci << EOF
${ggsci_command}
exit
EOF`

Create a profile for the GoldenGate application as the Oracle user from 1 node.
```
<CLUSTERWARE_HOME>/bin/crs_profile \
-create goldengate_app \
-t application \
-r ggsvip \
-a <CLUSTERWARE_HOME>/crs/public/ggs_action.sh \
-o ci=10
```
CLUSTERWARE_HOME is the Oracle home in which Oracle Clusterware is installed. For example: /u01/app/oracle/product/11.1.0/crs
-create goldengate_app the application name is goldengate_app.
-r specifies the required resources that must be running for the application to start. In this example, the dependency is the VIP ggsvip must be running before Oracle GoldenGate starts.
-a specifies the action script. For example: <CLUSTERWARE_HOME>/crs/public/ggs_action.sh
-o specifies options. In this example the only option is the Check Interval which is set to 10 seconds.
Next, register the application in the Oracle Cluster Registry (OCR) as the oracle user.
```
<CLUSTERWARE_HOME>/bin/crs_register goldengate_app
```
Now start the Goldengate application as the Oracle user.
```
<CLUSTERWARE_HOME>/bin/crs_start goldengate_app
```

Check that the application is running.

<CLUSTERWARE_HOME>/bin/crs_stat goldengate_app -t
Name Type Target State Host
------ ------ -------- ----- ----
goldengate_app application ONLINE ONLINE rac1

You can also stop GoldenGate from Oracle Clusterware by executing the following command as the oracle user:
```
CLUSTERWARE_HOME/bin/crs_stop goldengate_app
```

Oracle has published a White Paper on "Oracle GoldenGate high availability with Oracle Clusterware". To view the Action script mentioned in this article, refer to the document, which can be downloaded in PDF format from the Oracle Website at the following URL:
http://www.oracle.com/technetwork/middleware/goldengate/overview/ha-goldengate-whitepaper-128197.pdf

Oracle GoldenGate 11g Implementer's guide

Design, install, and configure high-performance data replication solutions with this Oracle GoldenGate book and eBook

Published: February 2011
eBook Price: $32.99
Book Price: $54.99
See more

Select your format and quantity:

Read more about this book

(For more resources on Oracle, see here.)

Increasing system resilience

Implementing Oracle RAC is a step toward high availability. For a RAC environment to be totally resilient to outages, all single points of failure must be removed from all elements. For example, the network infrastructure, storage solution, and power supply. To facilitate this, the recommendation is as follows:

Dual fibre channels to shared storage (SAN)
RAID disk subsystem (striped and mirrored)
Mirrored OCR and Voting disks
Bonded network on high speed interconnect via two physically connected switches
Bonded network on public network (VIP) via two physically connected switches
Dual power supply to each node, switch, and storage solution via UPS

When using ASM as your storage manager, Oracle recommends configuring a redundant diskgroup. However, if you have a RAID disk subsystem, you can configure ASM to use external redundancy. It is also best practice to "stripe on stripe" where ASM stripes the data across the LUNs, thus reducing I/O contention and increasing performance.

GoldenGate on Exadata

Oracle Sun Database Machine features a number of Oracle 11gR2 database servers configured for RAC offering highly parallel processing on Exadata storage and high speed Infiniband network interfaces. The environment also supports GoldenGate, enabling real-time data integration.

Depending on the machine size, the database servers and storage cells can be configured in a number of ways to provide more than one RAC cluster. If your source and target database reside in the same Database Machine, there is no need to configure a Data Pump process. Data can be transmitted directly from Extract to Replicat at very high speeds.

Configuration

Configuring GoldenGate on Exadata is a similar process to 11gR1 RAC environments except we use DBFS as the shared mount point supporting the persistent GoldenGate files.

The GoldenGate Manager must only run on one node in a RAC cluster. To prevent Extract and Replicat processes being started concurrently, mount DBFS on a single RAC node will deny access to the checkpoint files from other nodes. Ensure the mount point detail is written to the node's /etc/fstab file.

For example:

/sbin/mount.dbfs#/@DBConnectString /mnt/oracle/dbfs fuse rw,user,
noauto 0 0

If the GoldenGate home is not on the shared storage, ensure that GoldenGate is installed on each node in the cluster and that the parameter files exist in the local subdirectories. Checkpoint and trail files must reside on the shared storage.

Parameter files may also reside on the shared storage. However, in this case the Oracle environment variables must be set in the Oracle user profile on each node. This removes the need to set the Oracle environment in the GoldenGate Extract and Replicat parameter files. Furthermore, should you wish to make changes to your GoldenGate configuration, this can be done in one place without having to copy parameter files to each node.

Creating the Extract process

The following example steps you through the creation of an Extract process on the source database using DBFS to store the trail files:

Ensure the DBFS file system is already mounted. As the root user create the dirchk and dirdat GoldenGate subdirectories on top of the mount point (/mnt/oracle/dbfs). For example:
```
mkdir /mnt/oracle/dbfs/ggs/dirchk
mkdir /mnt/oracle/dbfs/ggs/dirdat
```
As root, grant the oracle user read-write privileges to the ggs directory.
```
chown -R oracle:oinstall /mnt/oracle/dbfs/ggs
```
As the oracle user, remove the GoldenGate subdirectories, dirchk and dirdat, from the GoldenGate home on each node in the cluster.
```
rmdir $GGS_HOME/dirchk
rmdir $GGS_HOME/dirdat
```
Now create symbolic links to the newly created DBFS mount point directories on each node in the cluster.
```
cd $GGS_HOME
ln -s /mnt/oracle/dbfs/ggs/dirchk dirchk
ln -s /mnt/oracle/dbfs/ggs/dirdat dirdat
```
Now that the DBFS mount point and GoldenGate subdirectories have been created on the shared filesystem, we can create the Extract process. Firstly, create an Extract parameter file using EDIT PARAMS command in ggsci.

The ASM connect details allow GoldenGate to access the online redo logs for all database instances (threads). The symlinks simplify the Extract parameter files by not having to specify the full path to the shared trail file directory.

In the following example, Extract parameter file snippet, the Oracle SID is set by the SETENV parameter. For local parameter files, ensure the Oracle SID is set appropriately for the node you are configuring.

EXTRACT EOLTP01
SETENV (ORACLE_SID=OLTP1)
USERID ggs_admin, PASSWORD ggs_admin
EXTTRAIL ./dirdat/aa
TRANLOGOPTIONS ASMUSER SYS@ASM, ASMPASSWORD Password1

Then create the Extract process from GGSCI.

GGSCI (rac1) 1> add extract EOLTP01, tranlog, begin now, threads 4
EXTRACT added.

Finally create the EXTTRAIL from GGSCI.

GGSCI (rac1) 2> add exttrail ./dirdat/aa, extract EOLTP01,
megabytes 500
EXTTRAIL added.

Creating the Replicat process

Now that the DBFS mount point and GoldenGate subdirectories have been created on the shared filesystem, we can create the Replicat process.

Firstly, create a Replicat parameter file using EDIT PARAMS command in GGSCI. The discard directory need not be on the shared DBFS filesystem as the files are not required for GoldenGate process recovery.
```
REPLICAT ROLAP01
SETENV (ORACLE_SID=OLAP1)
USERID ggs_admin, PASSWORD ggs_admin
DISCARDFILE ./dirrpt/rolap01.dsc, PURGE
```
Now create the Replicat process. Note the Replicat uses the same EXTTRAIL as the source.
```
GGSCI (dbolap1) 1> add replicat ROLAP01, exttrail ./dirdat/aa
REPLICAT added.
```

We have looked at some of the Oracle 11g Release 2 new features, including the Exadata Database Machine. Now let's look at failover, ensuring availability.

Failover

By default VIPs and database services automatically failover to a surviving instance in the case of a crash or node eviction. The VIP will automatically failback to its "home" node once the failed database instance restarts. VIPs and database services can also be manually relocated for maintenance reasons using the Oracle Server Control utility's srvctl relocate command, allowing a node to be taken off-line without affecting user connections to the database.

Automatic failover

What does this mean to the GoldenGate Manager process running on one node in a clustered environment?

To help answer this question, Oracle Clusterware can also be installed on other servers apart from the database servers to form a single cluster. For example, you can use four database servers and two additional Oracle GoldenGate servers in the single cluster. The Oracle Database would run on the four database servers and GoldenGate would run on one of the two remaining servers, with failover to its dedicated twin. Because the Goldengate Manager and Data Pump processes (if any) are configured to use the VIP, the failover will be automatic.

You may however, wish to install GoldenGate on every database server in a cluster with no dedicated GoldenGate servers. In this configuration, the automatic failover using the VIP is still supported.

Manual failover

The following sections discuss the various methods of manual failover.

Relocating a service

For maintenance reasons, it is sometimes necessary to relocate a database service from one RAC instance to another. The following example shows how to relocate the database service using srvctl:

srvctl status service -d OLTP -s ACTIVE_SRV
Service ACTIVE_SRV is running on instance(s) OLTP2
srvctl relocate service -d OLTP -s ACTIVE_SRV -i OLTP2 -t OLTP1
$ srvctl status service -d OLTP -s ACTIVE_SRV
Service ACTIVE_SRV is running on instance(s) OLTP1

Relocating a VIP

When GoldenGate is running, you may want to move GoldenGate to run on a different server, again for maintenance reasons. The following Oracle Clusterware command executed by the Oracle user allows you to do this. Use the crs_relocate program with the force option to move the VIP as well.

<CLUSTERWARE_HOME>/bin/crs_relocate -f goldengate_app
Attempting to stop `goldengate_app` on member `rac2`
Stop of `goldengate_app` on member `rac2` succeeded.
Attempting to stop `ggatevip` on member `rac2`
Stop of `ggatevip` on member `rac2` succeeded.
Attempting to start `ggatevip` on member `rac1`
Start of `ggatevip` on member `rac1` succeeded.
Attempting to start `goldengate_app` on member `rac1`
Start of `goldengate_app` on member `rac1` succeeded.

This is exactly what happens automatically when a node crashes or gets evicted from the cluster. The GoldenGate Manager process is restarted on a surviving node where its configuration auto-starts the Extract and Replicat processes.

Summary

Oracle RAC is one of the most popular database configurations, first introduced in Oracle 9i, superseding Oracle Parallel Server. GoldenGate has recently played a major role in the Oracle 11g Release 2 RAC environment with the advent of Exadata and the Database Machine. In fact both are commonly sold together to provide a robust OLTP and OLAP solution in the same Oracle-Sun equipment rack.

In this article, we learnt how to configure GoldenGate on Oracle RAC, leveraging HA through Clusterware configuration techniques, and explored the new shared storage solutions available in Oracle 11g Release 2. We discovered the importance of automatic relocation and startup of the GoldenGate Manager and processes on the new instance to restore data replication to downstream systems, all in the name of HA.

Further resources on this subject:

Setting Up Oracle Order Management [Article]
An Overview of Oracle Advanced Pricing [Article]
Oracle Siebel CRM 8 Developer's Handbook: RAW [Book]
Oracle WebLogic Server 11gR2: Administration Essentials: RAW [Book]
The Business Analyst's Guide to Oracle Hyperion Interactive Reporting 11 [Book]

Oracle GoldenGate 11g Implementer's guide

Design, install, and configure high-performance data replication solutions with this Oracle GoldenGate book and eBook

Published: February 2011
eBook Price: $32.99
Book Price: $54.99
See more

Select your format and quantity:

About the Author :

John P Jeffries

Originally from a development background, John P Jeffries has worked for Oracle Corporation since the Siebel acquisition in 2005. His time at Siebel was spent developing ETL solutions for Data Warehouses, later moving to Oracle Consulting and then Oracle Advanced Customer Services as a Senior Principal Consultant, earning the internal title of "The UK's Data Replication Expert". Now a freelance consultant, he can be found onsite in many of the world's most respected financial institutions in London, consulting on Oracle GoldenGate, Streams, and Active Data Guard. With over 15 years of Oracle experience, and an OCP since Oracle 8i, he has extensive knowledge of Oracle databases, including Enterprise Linux and RAC, coupled with the ability to design and build high-performance distributed database systems. He has trained internal and external clients in Data Warehousing and Data Replication techniques, and continues to share his knowledge and experience through his own Website: http://www.oracle11ggotchas.com.

Read Packt's recent Q&A with John for further information