5 Making Applications Highly Available Using Oracle Clusterware

Application Resource Profiles

Attributes are defined by name=value entries in profile files and these entries can be in any order in the file. The following are some of the primary attributes of an application profile:

• Resources that are required by an application which are defined by settings for the REQUIRED_RESOURCES parameter. Oracle Clusterware relocates or stops an application if a required resource becomes unavailable. Required resources are defined for each node.

• Rules for choosing the node on which to start or restart an application are defined by settings for the PLACEMENT parameter. The application must be accessible by the nodes that you nominate for placement.

• A list of nodes to use in order of preference when Oracle Clusterware starts or fails over an application which is defined by settings for the HOSTING_MEMBERS parameter. This list is used if the placement policy defined by the PLACEMENT parameter is favored or restricted.

• The filenames of application profiles must be in the form resource_name.cap where resource_name is the name that you or the system assigns to an application and cap is the file suffix. The Oracle Clusterware commands in profiles refer to applications by name, such as resource_name, but not by the full filename.

Using Entry Points to Manage Resources Using Oracle Clusterware

You can use entry points to specify how to start a resource, stop a resource, and check a resource. You can implement entry points in various ways, such as by using shell or Perl scripts, C++ functions, or Java functions. Oracle Clusterware has the following entry points:

• START—The start (online) entry point brings the resource online.

• STOP—The stop (offline) entry point takes the resource offline.

• CHECK—The check (monitor) entry point monitors the health of a resource.

Using the crs_profile Command to Create An Application Resource Profile

To create an action profile, you use the crs_profile command. Example 5-1 uses the crs_profile command to create an application profile for the postman action script that is used to monitor email.

$ crs_profile -create postman -t application -B /opt/email/bin/crs_postman \
-d "Email Application" -r network1 -l application2 \
-a postman.scr -o ci=5,ft=2,fi=12,ra=2

The contents of the application profile file that the example creates are as follows:

NAME=postman
TYPE=application
ACTION_SCRIPT=/oracle/crs/script/postman.scr
ACTIVE_PLACEMENT=0
AUTO_START=always
CHECK_INTERVAL=5
DESCRIPTION=email app
FAILOVER_DELAY=0
FAILURE_INTERVAL=12
FAILURE_THRESHOLD=2
HOSTING_MEMBERS=
OPTIONAL_RESOURCES=application2
PLACEMENT=balanced
REQUIRED_RESOURCES=network1
RESTART_ATTEMPTS=2
SCRIPT_TIMEOUT=60

A good example of an action script is the xclock script, which is a simple action script that is a default binary on all Linux and UNIX platforms. Example 5-2 shows the contents of the xclock action script.

#!/bin/bash
# start/stop/check script for xclock example
# To test this change BIN_DIR to the directory where xclock is based
# and set the DISPLAY variable to a server within your network.
 
 
BIN_DIR=/usr/X11R6/bin
LOG_DIR=/tmp
BIN_NAME=xclock
DISPLAY=yourhost.domain.com:0.0
export DISPLAY
 
if [ ! -d $BIN_DIR ]
then
        echo "start failed"
        exit 2
fi
 
PID1=`ps -ef | grep $BIN_NAME | grep -v grep | grep -v xclock_app | awk '{ print $2 }'`
 
case $1 in
'start')
        if [ "$PID1" != "" ]
        then
           status_p1="running"
        else
           if [ -x $BIN_DIR/$BIN_NAME  ]
           then
             umask 002
             ${BIN_DIR}/${BIN_NAME} & 2>${LOG_DIR}/${BIN_NAME}.log
             status_p1="started"
           else
             echo `basename $0`": $BIN_NAME: Executable not found"
           fi
        fi
 
         echo "$BIN_NAME: $status_p1"
        ;;
 
'stop')
        if [ "${PID1}" != "" ]
        then
           kill -9 ${PID1} && echo "$BIN_NAME daemon killed"
        else
           echo "$BIN_NAME: no running Prozess!"
        fi
        ;;
'check')
        if [ "$PID1" != "" ]
        then
           echo "running"
           exit 0
        else
           echo "not running"
           exit 1
        fi
        ;;
*)
        echo "Usage: "`basename $0`" {start|stop|check}"
        ;;
esac 

Oracle Clusterware Required Resources List

Oracle Clusterware uses the required resources list, with the placement policy and hosting nodes list, to determine the cluster nodes that are eligible to host an application. Required resources must be ONLINE on the nodes on which the application is running or started.

The failure of a required resource on a hosting node causes Oracle Clusterware to attempt to restart the application on the current node. If RESTART_ATTEMPTS is not set to 0, and if the application cannot start on the current node, then Oracle Clusterware attempts to fail the application over to another node that provides the required resource. Alternatively, Oracle Clusterware stops the application if there is no suitable node. In this case, Oracle Clusterware posts a not restarting event notification.

You can also use required resource lists to start, stop, and relocate groups of interdependent applications when you use the crs_start, crs_stop, or crs_relocate commands with the force (-f) option. In other words, you can configure a set of resources to have other required resources. For example you can configure resources A, B, and C where A and B depend on C. Then if you stop resource C with -force option, this action will stop all three resources. The same is true for the crs_relocate command.

In addition, using the -force option can relocate an online dependency, if necessary, to enable the starting of a resource. For instance, assume that resource VIP_A that has a primary node assignment of node A is running instead on node B. If you perform a crs_start on the instance resource for node A, then this operation will fail. This is because the VIP_A resource is required for the instance on node A and the VIP_A resource is online. However, the resource is on a node where the resource cannot run. Performing a crs_start -f on instance B, however, forces the VIP_A resource to relocate and then start the instance.

crs_profile –create network1 -t application \
-a CRS_home/bin/usrvip \
-o oi=eth0,ov=138.3.83.78,on=255.255.240.0

crs_register network1

crs_setperm network1 –o root

crs_setperm network1 –u user:oracle:r-x

crs_start network1

Application Placement Policies

The placement policy specifies how Oracle Clusterware selects a node on which to start an application and where to relocate the application after a node failure. Only cluster nodes on which all of the required resources are available, as listed in an application's profile, are eligible to be considered as hosting nodes for an application. Oracle Clusterware supports the following placement policies:

• balanced—Oracle Clusterware favors starting or restarting the application on the node that is currently running the fewest resources. A placement that is based on optional resources is considered first. Next, the host with the fewest resources running is chosen. If no node is favored by these criteria, then any available node is chosen.

• favored—Oracle Clusterware refers to the list of nodes in the HOSTING_MEMBERS attribute of the application profile. Only cluster nodes that are in this list and that satisfy the resource requirements are eligible for placement consideration. Placement due to optional resources is considered first. If no node is eligible based on optional resources, then the order of the hosting nodes determines which node runs the application. If none of the nodes in the hosting node list are available, then Oracle Clusterware places the application on any available node. This node may or may not be included in the HOSTING_MEMBERS list.

• restricted—Similar to favored except that if none of the nodes on the hosting list are available, then Oracle Clusterware does not start or restart the application. A restricted placement policy ensures that the application never runs on a node that is not on the list, even if you manually relocate it to that node.

You must specify hosting nodes in the HOSTING_MEMBERS attribute to use a favored or restricted placement policy. Do not specify hosting nodes in the HOSTING_MEMBERS attribute with a balanced placement policy. Otherwise, the application will not validate and you cannot register it. If ACTIVE_PLACEMENT is set to 1, then the placement of the application is reevaluated whenever you add a node to the cluster or if the cluster node restarts. This enables Oracle Clusterware to relocate applications to a preferred node after the node recovers from a failure. To have Oracle Clusterware relocate an application to a preferred node at a time other than when the node rejoins the cluster, use the REBALANCE attribute to specify a time at which placement can be reevaluated.

Optional Resources in Placement Decisions

Oracle Clusterware uses optional resources to choose a hosting node based on the number of optional resources that are in an ONLINE state on the hosting node. If each node has an equal number of optional resources in an ONLINE state, then Oracle Clusterware considers the order of the optional resources as follows:

• Oracle Clusterware compares the state of the optional resources on each node starting at the first resource that you list in the application profile and then proceeds through the list.

• For each consecutive resource in your list, if the resource is ONLINE on one node, then any node that does not have the resource ONLINE is not considered.

• Oracle Clusterware evaluates each resource in the list in this manner until only one node is available to host the resource.

• The maximum number of optional resources is 58.

If this algorithm results in multiple preferred nodes, then the resource is placed on one of these nodes chosen according to its placement policy.

How Oracle Clusterware Runs Action Programs

This section describes how Oracle Clusterware runs action programs. The first argument to an action program is the command start, stop, or check depending on which action Oracle Clusterware is running. The second argument is the Oracle Clusterware resource name of the application. This enables a script to determine which instance of the resource that Oracle Clusterware is starting, stopping, or checking.An action program can retrieve any of its Oracle Clusterware resource attributes from the environment by using $_CAA_attribute_name. For example, $_CAA_NAME contains the application name, the second argument to the script, and $_CAA_HOSTING_MEMBERS contains its HOSTING_MEMBERS attribute.

User Defined Attributes

Oracle Clusterware supports user-defined attributes in Oracle Clusterware applications, which are attributes having names that contain USR. User-defined attributes are stored as part of the Oracle Clusterware application profile for the application. You can reference them in an action program using $_USR_attributename. To add a user-defined attribute, add it to the file CRS_home/crs/template/application.tdf using the following syntax:

#
# an example user-defined attribute
#
#!===========================
attribute: USR_EXAMPLE
type: string
switch: -o example
default:
required: no

The attribute parameter contains the name of the new attribute. The type parameter defines the type of the user-defined attribute and can be one of the following:

• string

• boolean

• integer—a numeric attribute

• positive_integer—a numeric attribute that must be positive

• name string

• name_list—a comma-delimited list of names

The switch parameter describes how the attribute is specified for the crs_profile command. Set the required field to no for user-defined attributes.

Windows crsuser Program

This section describes the Windows crsuser program. The syntax for the crsuser command is:

crsuser add [domain\]username

For example, on a Windows system you could issue the following command as an operating system user that is part of ORA_DBA group and the Local Administrator group:

C:\> crsuser add oracledomain\oracluster

Provide the user's Windows password. This creates the OracleCRSToken_user service that Oracle Clusterware needs to start the Oracle Clusterware resources under the given user ID (when they are not running as the LocalSystem account). You can also use the crsuser command: remove [domain\]username to remove a token service and crsuser list to list a registered users.

Registering Application Resources

Each application that you manage with Oracle Clusterware must have an application profile and the profile must be registered in the OCR. Use the crs_register command to register applications in the OCR. For example, enter the following command to register the mail monitoring application from the previous example:

# crs_register postman

If you modify a profile, then update the OCR by running the crs_register -u command again.

Starting Application Resources

To start an application resource that is registered with Oracle Clusterware, use the crs_start command. For example:

# crs_start postman

The following text is an example of the command output:

Attempting to start `postman` on node `rac1`
Start of `postman` on node `rac1` succeeded.

The application now runs on the node named rac1.

The command waits for the amount of time specified by the setting for the SCRIPT_TIMEOUT parameter to receive a notification of success or failure from the action program each time the action program is called. Application resources can be started if they have stopped due to exceeding their failure threshold values. You must register a resource with crs_register before you can start it.

To start and stop the resources, use the crs_start and crs_stop commands. Manual starts or stops outside of Oracle Clusterware can invalidate the resource status. In addition, Oracle Clusterware may attempt to restart a resource on which you perform a manual stop operation.

All required resources must be online on the node where you start the resource. If the resources that the REQUIRED_RESOURCES parameter identifies are offline, then the command crs_start resource_name will start the required resources before starting the resource.

Running the crs_start command on a resource sets the resource target value to ONLINE. Oracle Clusterware attempts to change the state to match the target by running the action program with the start parameter. When a resource is running, both the target state and current state are ONLINE.

Starting an Application on an Unavailable Node

When starting an application on a cluster node that is unavailable, crs_start can give indeterminate results. In this scenario, the start section of the action program is run, but the cluster node fails before notification of the start is displayed on the command line. The crs_start command returns a failure with the error Remote start for resource_name failed on node node_name. The application is actually ONLINE but fails over to another node making the application appear as though it were started on the incorrect node.

If a cluster node fails while you are starting a resource on that node, then check the state of the resource on the cluster by using the crs_stat command to determine the state of that resource.

Relocating Applications and Application Resources

Use the crs_relocate command to relocate applications and application resources. For example, to relocate the mail monitoring application to the node known as rac2, enter the following command:

# crs_relocate postman -c rac2

Each time that the action program is called, the crs_relocate command waits for the duration identified by the value for the SCRIPT_TIMEOUT parameter to receive notification of success or failure from the action program. A relocation attempt fails if:

• The application has required resources that are ONLINE on the initial node

• Applications that require the specified resource are ONLINE on the initial node

To relocate an application and its required resources, use the -f option with the crs_relocate command. Oracle Clusterware relocates or starts all resources that are required by the application regardless of their state.

Stopping Applications and Application Resources

To stop applications and application resources, use the crs_stop command. Immediately after the crs_stop command completes, the application status converts to OFFLINE. Because Oracle Clusterware always attempts to match a resource's state to its target, the Oracle Clusterware subsystem stops the application. The following example stops the mail application from the example:

# crs_stop postman

The following text is an example of the command output:

Attempting to stop `postman` on node `rac1`
Stop of `postman` on node `rac1` succeeded.

You cannot stop an application if the application is a required resource for another online application unless you use the force (-f) option. If you use the crs_stop -f resource_name command on an application that is required by other resources and if those resources are online, then Oracle Clusterware stops the application. In addition, all of the resources that require that application that are online are also stopped.

Managing Automatic Oracle Clusterware Resource Operations for Action Scripts

The following section explains additional information for controlling how Oracle Clusterware manages restarts. You can prevent Oracle Clusterware from automatically restarting a resource by setting several action program attributes. You can also control how Oracle Clusterware manages the restart counters for your action programs. In addition, you can customize the timeout values for the start, stop, and check actions that Oracle Clusterware performs on action scripts. These topics are described under the following headings:

• Preventing Automatic Restarts

• Automatically Manage Restart Attempts Counter for Resources

• Implications of Restart and Timeout Features for Previous Releases

Unregistering Applications and Application Resources

To unregister an application or an application resource, use the crs_unregister command. You cannot unregister an application or resource that is ONLINE or required by another resource. The following example unregisters the mail application:

# crs_unregister postman

The unregistration process frees space on the OCR. Additionally, run the crs_unregister command as a clean-up step when a resource is no longer managed by Oracle Clusterware. Generally, you should unregister all permanently stopped applications.

Displaying Clusterware Application and Application Resource Status Information

To display status information about applications and resources that are on cluster nodes, use the crs_stat command. The following example displays the status information for the postman application:

# crs_stat postman
NAME=postman
TYPE=application
TARGET=ONLINE
STATE=ONLINE on rac2

Enter the following command to view information about all applications and resources in tabular format:

# crs_stat -t

The following text is an example of the command output:

Name         Type        Target      State     Host
----------------------------------------------------------------
cluster_lockd application ONLINE    ONLINE     rac2
dhcp          application OFFLINE   OFFLINE

Enter the following command to determine:

• How many times an application resource has been restarted

• How many times an application resource has failed within the failure interval

• The maximum number of times that an application or resource can be restarted or fail

• The target state of the application or resource and the normal status information

# crs_stat -v

To view verbose content in tabular format, enter the following command:

# crs_stat -v -t

The following text is an example of the command output:

Name          Type       R/RA  F/FT   Target    State    Host
----------------------------------------------------------------------
cluster_lockd application 0/30  0/0   ONLINE    ONLINE   rac2
dhcp          application 0/1   0/0   OFFLINE   OFFLINE
named         application 0/1   0/0   OFFLINE   OFFLINE
network1      application 0/1   0/0   ONLINE    ONLINE   rac1

Enter the following command to view the application profile information that is stored in the OCR:

# crs_stat -p

The following text is an example of the command output:

NAME=cluster_lockd
TYPE=application
ACTION_SCRIPT=cluster_lockd.scr
ACTIVE_PLACEMENT=0
AUTO_START=restore
CHECK_INTERVAL=5
DESCRIPTION=Cluster lockd/statd
FAILOVER_DELAY=30
FAILURE_INTERVAL=60
FAILURE_THRESHOLD=1
REBALANCE=
HOSTING_MEMBERS=
OPTIONAL_RESOURCES=
PLACEMENT=balanced
REQUIRED_RESOURCES=
RESTART_ATTEMPTS=2
SCRIPT_TIMEOUT=60 ...

See the crs_stat command for more information.