9 Adding and Deleting Oracle RAC from Nodes on Linux and UNIX Systems

Prerequisite Steps for Extending Oracle RAC to Target Nodes

The following steps describe how to set up target nodes to be part of your cluster:

Step 1   Make physical connections

Connect the target nodes' hardware to the network infrastructure of your cluster. This includes establishing electrical connections, configuring network interconnects, configuring shared disk subsystem connections, and so on. See your hardware vendor documentation for details about this step.

Step 2   Install the operating system

Install a cloned image of the operating system that matches the operating system on the other nodes in your cluster. This includes installing required service patches and drivers. See your hardware vendor documentation for details about this process.

Step 3   Create Oracle users

Create Oracle users.

As root user, create the Oracle users and groups using the same user ID and group ID as on the existing nodes.

Step 4   Verify the installation

Verify the installation with the Cluster Verification Utility (CVU) by performing the following steps:

1. From the /bin directory in the CRS_home on the existing nodes, run the CVU command to verify your installation at the post hardware installation stage as shown in the following example, where node_list is a comma-delimited list of nodes you want in your cluster:

   cluvfy stage -post hwos -n node_list|all [-verbose]
   

   Note:

   You can only use the all option with the -n argument if you have set the CV_NODELIST variable to represent the list of nodes on which you want to perform the CVU operation.

   You can also use this CVU command to:

   • Verify the node is reachable, for example, to all of the nodes from the local node.

   • Verify user equivalence to all given nodes the local node, node connectivity among all of the given nodes, accessibility to shared storage from all of the given nodes, and so on.

   See Also:

   "Using the Cluster Verification Utility" section in the Oracle Clusterware Administration and Deployment Guide

2. From the /bin directory in the CRS_home on the existing nodes, run the CVU command to obtain a detailed comparison of the properties of the reference node with all of the other nodes that are part of your current cluster environment where ref_node is a node in your existing cluster against which you want CVU to compare, for example, the target nodes that you specify with the comma-delimited list in node_list for the -n option, orainventory_group is the name of the Oracle inventory group, and osdba_group is the name of the OSDBA group:

   cluvfy comp peer [ -refnode ref_node ] -n node_list 
   [ -orainv orainventory_group ] [ -osdba osdba_group ] [-verbose]
   

   For the reference node, select a node from your existing cluster nodes against which you want CVU to compare, for example, the target nodes that you specify with the -n option.

   Note:

   For all of the add node and delete node procedures for Linux and UNIX systems, temporary directories such as /tmp, $TEMP, or $TMP, should not be shared directories. If your temporary directories are shared, then set your temporary environment variable, such as $TEMP, to a nonshared location on a local node. In addition, use a directory that exists on all of the nodes.

Step 5   Check the installation

To verify that your installation is configured correctly, perform the following steps:

1. Ensure that the target nodes can access the private interconnect. This interconnect must be properly configured before you can complete the procedures described in this chapter.

2. If you are not using a cluster file system, then determine the location on which your cluster software was installed on the existing node. Make sure that you have at least 250 MB of free space on the same location on each of the target nodes to install Oracle Clusterware. In addition, ensure you have enough free space on each target node to install the Oracle binaries.

3. Ensure that the OCR and the voting disk are accessible by the target nodes using the same path. In addition, the OCR and voting disk devices must have the same permissions as on the existing nodes.

4. Verify user equivalence to and from an existing node to the target nodes using rsh or SSH on Linux and UNIX systems, or on Window systems make sure that you can run the following command from all of the existing nodes of your cluster where the hostname is the public network name of the target node:

   NET USE \\hostname\C$
   

   You have the required administrative privileges on each node if the operating system responds with:

   Command completed successfully.
   

After completing the procedures in this section, your target nodes are connected to the cluster and configured with the required software to make them visible to Oracle Clusterware.

Extend Oracle Clusterware to Target Nodes

Extend an existing Oracle Clusterware home to the target nodes following the instructions in Oracle Clusterware Administration and Deployment Guide.

If you are using Oracle Clusterware without vendor clusterware, then you can add and delete Oracle Clusterware on nodes without stopping the existing nodes. If you are using Oracle Clusterware with vendor clusterware, then you can add nodes on some Linux and UNIX systems without stopping the existing nodes if your clusterware supports this. See your vendor-specific clusterware documentation for more information.

Configure Shared Storage on Target Nodes

Use the information in this section to configure shared storage so that the target nodes can access the Oracle software, and so that the existing nodes can access the target nodes and instances. Then, use the procedure described in "Add the Oracle Real Application Clusters Database Homes to Target Nodes".

To extend an existing Oracle RAC database to the target nodes, configure the same type of storage on the target nodes as you are using on the existing nodes in the Oracle RAC environment:

• Automatic Storage Management (ASM)

  You do not need to install ASM on the added node because the ASM instance is created implicitly upon node addition. If you are using ASM for storage, then make sure that the target nodes can access the ASM disks with the same permissions as the existing nodes.

  See Also:

  "Extending ASM to Nodes Running Single-Instance or Oracle RAC Databases" for instructions about adding a new ASM instance to a node running either a single-instance database or an Oracle RAC database instance in a cluster

• Oracle Cluster File System (OCFS)

  If you are using Oracle Cluster File Systems, then make sure that the target nodes can access the cluster file systems in the same way that the other nodes access them.

  Run the following command to verify your cluster file system and obtain detailed output where nodelist includes both the preexisting nodes and the target nodes and file system is the name of the file system that you used for the Oracle Cluster File System:

  cluvfy comp cfs -n nodelist -f file system [-verbose]
  

  See Also:

  Oracle Clusterware Administration and Deployment Guide for more information about enabling and using the CVU, and your platform-specific Oracle Clusterware installation guide for more information about the Oracle Cluster File System

• Vendor cluster file systems

  If your cluster database uses vendor cluster file systems, then configure the target nodes to use the vendor cluster file systems. See the vendor clusterware documentation for the preinstallation steps for your Linux or UNIX platform.

• Raw device storage

  If your cluster database uses raw devices, then prepare the raw devices on the target nodes, as follows:

  To prepare raw device storage on the target nodes, you need at least two new disk partitions to accommodate the redo logs for each new instance. Make these disk partitions the same size as the redo log partitions that you configured for the existing nodes' instances. Also create an additional logical partition for the undo tablespace for automatic undo management.

  On applicable operating systems, you can create symbolic links to your raw devices. Optionally, you can create a raw device mapping file and set the DBCA_RAW_CONFIG environment variable so that it points to the raw device mapping file. Use your vendor-supplied tools to configure the required raw storage.

  See Also:

  Your platform-specific Oracle Real Application Clusters installation guide for procedures about using DBCA to create and delete Oracle RAC databases

  Run the following command to verify that the prepared raw device storage is accessible from all of the configured cluster nodes where node_list includes both the pre-existing nodes and the newly added nodes and storageID_list is a comma-delimited list of storage identifiers:

  cluvfy comp ssa [ -n node_list ] [ -s storageID_list ] [-verbose]
  

  See Also:

  Oracle Clusterware Administration and Deployment Guide for more information about enabling and using the CVU

Add the Oracle Real Application Clusters Database Homes to Target Nodes

You can add the Oracle RAC database home to target nodes using either of the following methods:

• Extending the Database Home to Target Nodes Using OUI in Interactive Mode

• Extending the Database Home to Target Nodes Using OUI in Silent Mode

Add ASM and Oracle RAC Database Instances to Target Nodes

You can use either Enterprise Manager or DBCA to add Oracle RAC database instances to the target nodes. To add a database instance to a target node with Enterprise Manager, see the Oracle Database 2 Day + Real Application Clusters Guide for complete information.

This section describes using DBCA to add Oracle RAC database instances under the following topics:

• Using DBCA in Interactive Mode to Add ASM and Database Instances to Target Nodes

• Using DBCA in Silent Mode to Add ASM and Database Instances to Target Nodes

These tools guide you through the following tasks:

• Creating and starting an ASM instance (if the existing instances were using ASM) on each target node

• Creating a new database instance on each target node

• Creating and configuring high availability components

• Creating the Oracle Net configuration

• Starting the new instance

• Creating and starting services if you entered services information on the Services Configuration page

After adding the instances to the target nodes, you should perform any necessary service configuration procedures, as described in Chapter 4.

dbca -silent -addInstance -nodeList node -gdbName gdbname [-instanceName instname]
 -sysDBAUserName sysdba -sysDBAPassword password

Step 1: Delete Instances from Oracle Real Application Clusters Databases

The procedures in this section explain how to use DBCA in interactive or silent mode, to delete an instance from an Oracle RAC database. To delete a database instance from a target node with Enterprise Manager, see the Oracle Database 2 Day + Real Application Clusters Guide.

This section includes the following topics:

• Using DBCA in Interactive Mode to Delete Instances from Nodes

• Using DBCA in Silent Mode to Delete Instances from Nodes

dbca -silent -deleteInstance [-nodeList node] -gdbName gdbname -instanceName
 instname -sysDBAUserName sysdba -sysDBAPassword password

Step 2: Delete Nodes from the Cluster

Once you have deleted the instance, you can begin the process of deleting the node from the cluster. You accomplish this by running scripts on the node you want to delete to remove the Oracle Clusterware installation and you run scripts on the remaining nodes to update the node list.

The following steps assume that the node to be removed (node2 in this discussion) is still functioning. Before beginning these procedures, ensure that the $ORACLE_HOME environment variable is set correctly on the existing nodes.

Use the following procedures to delete nodes from Oracle clusters on Linux or UNIX systems:

1. Stop the node applications on the node you are deleting.

   As the root user, issue the srvctl stop nodeapps command to stop and remove the nodeapps on the node you are deleting. For example:

   # srvctl stop nodeapps -n nodename
   

2. Remove the listener from the node.

   If this is the Oracle home from which the node-specific listener named LISTENER_nodename runs, then use NETCA to remove this listener. If necessary, re-create this listener in another home. Invoke NETCA and proceed as follows:

   1. Choose Cluster Configuration.

   2. Select only the node you are removing and click Next.

   3. Choose Listener Configuration and click Next.

   4. Choose Delete and delete any listeners configured on the node you are removing.

   See Also:

   Oracle Database Net Services Administrator's Guide for more information about NETCA

3. Depending on whether you have a shared or nonshared Oracle home, complete one of the following two procedures:

   • For a shared home, run the following command on each of the nodes that are to be deleted:

     runInstaller -detachHome -local ORACLE_HOME=Oracle_home
     

   • For a nonshared home, on each node that you are deleting, perform the following two steps:

     • Run the following command:

       runInstaller -updateNodeList ORACLE_HOME=Oracle_home
       CLUSTER_NODES="" –local 
       

       The runInstaller command is located in the directory Oracle_home/oui/bin. Using this command does not launch an installer GUI.

     • Deinstall the Oracle home from the node that you are deleting by running the following command from the Oracle_home/oui/bin directory:

       ./runInstaller -deinstall -silent "REMOVE_HOMES={Oracle_home}" -local
       

4. Verify that all database resources are running on each node that will not be deleted.

   1. Issue the crs_stat command from the CRS_HOME/bin directory to verify the other nodes. The following example shows the statistics for the database db_name on the node node2:

      NAME=ora.db_name.db
      TYPE=application
      TARGET=ONLINE
      STATE=ONLINE on node2
      

   2. Ensure that the database resource is not running on a node you are deleting. Issue the crs_relocate command from the CRS_HOME/bin directory to perform this as shown in the example:

      crs_relocate ora.db_name.db
      

5. Remove the node applications on the node you are deleting.

   As the root user, issue the following command:

   # srvctl remove nodeapps -n nodename
   

6. Update the node list on the remaining nodes in the cluster.

   1. Issue the following command to set the display environment:

      DISPLAY=ipaddress:0.0; export DISPLAY
      

   2. Define the database Oracle homes ($ORACLE_HOME) in the Oracle inventory for the nodes that are still in the cluster. If there is no $ORACLE_HOME, you can skip this step.

      As the ORACLE user, run the installer with the updateNodeList option on any remaining nodes in the cluster, and include a comma-delimited list of nodes that remain in the cluster:

      $ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=Oracle_Home
       CLUSTER_NODES=node1,node3,node4
      

7. Stop Oracle Clusterware on the node you are deleting.

   As the root user on the node that you are deleting, run the rootdelete.sh script in the CRS_home/install/ directory to disable the Oracle Clusterware applications that are on the node. For example:

   # cd CRS_home/install
   # ./rootdelete.sh local nosharedvar nosharedhome
   

   Only issue this command once and use the nosharedhome argument if you are using a local file system. The nosharedvar option assumes the OCR.LOC file is not on a shared file system. The default for this command is sharedhome that prevents you from updating the permissions of local files such that they can be removed by the oracle user. If the ocr.loc file is on a shared file system, then issue the CRS_home/install/rootdelete.sh remote sharedvar command.

   If you are deleting more than one node from your cluster, then repeat this step on each node that you are deleting.

8. Delete the node and remove it from the OCR.

   As the root user on any node that you are not deleting:

   1. To determine the node number of any node, issue the olsnodes -n command from the CRS_home/bin directory. For example:

      # olsnodes -n
      node1 1
      node2 2
      

   2. Run the rootdeletenode.sh script from the CRS_home/install/ directory. The rootdeletenode.sh script calls the clscfg -delete script deletes the node from the Oracle cluster and updates the OCR.

      The following example removes only one node, node2:

      # cd CRS_home/install
      # ./rootdeletenode.sh node2,2
      

      To delete only one node, enter the node name and number of the node that you want to delete with the command CRS_home/install/rootdelete.sh node1,node1-number. To delete multiple nodes, issue the command CRS_home/install/rootdelete.sh node1,node1-number,node2,node2-number,... nodeN,nodeN-number, where node1 through nodeN is a list of the nodes that you want to delete, and node1-number through nodeN-number represents the node number.

      If you do not perform this step, the olsnodes command will continue to display the deleted node as a part of the cluster.

   3. Confirm that the node has been deleted by issuing the olsnodes command:

      ./olsnodes -n
      node1 1
      

9. Define the CRS home in the Oracle inventory for the nodes that are still in the cluster.

   As the ORACLE user, perform the following steps:

   1. Issue the following command to set up the display environment:

      DISPLAY=ipaddress:0.0; export DISPLAY
      

   2. Issue the runInstaller command from the CRS home and specify a comma-delimited list of nodes that remain in the cluster:

      From the CRS home run the installer with the updateNodeList option on any remaining nodes in the cluster, and include a comma-delimited list of nodes that remain in the cluster:

      CRS_home/oui/bin/runInstaller -updateNodeList ORACLE_HOME=CRS_home 
      CLUSTER_NODES=node1, node3, node4 CRS=TRUE
      

10. Delete the Oracle home and the CRS home from the deleted node.

    Once the node updates are done, you must manually delete the Oracle home and CRS home from the node that you have deleted. Note: if either of these home directories is located on a shared file system, then skip this step.

    1. In the Oracle home directory, issue the following command:

       $ORACLE_HOME: rm -rf *
       

    2. In the CRS home directory, issue the following command:

       $CRS_HOME : rm -rf *
       

11. Ensure that all initialization scripts and soft links are removed from the deleted node.

    For example, as the root user on a Linux system, issue the following commands:

    rm -f /etc/init.d/init.cssd
    rm -f /etc/init.d/init.crs 
    rm -f /etc/init.d/init.crsd 
    rm -f /etc/init.d/init.evmd 
    rm -f /etc/rc2.d/K96init.crs
    rm -f /etc/rc2.d/S96init.crs
    rm -f /etc/rc3.d/K96init.crs
    rm -f /etc/rc3.d/S96init.crs
    rm -f /etc/rc5.d/K96init.crs
    rm -f /etc/rc5.d/S96init.crs
    rm -Rf /etc/oracle/scls_scr
    

    Optionally, you can also remove the /etc/oracle directory, the /etc/oratab file, and the Oracle inventory from the deleted node.

12. Optionally, remove additional Oracle homes, ASM homes, or Enterprise Manager homes (if used), from the Oracle inventory on all of the remaining nodes.

    On all remaining nodes, run the installer to update the node list. The following example assumes that you have deleted node2 from the cluster:

    runInstaller -updateNodeList -local \ORACLE_HOME=$ORACLE_HOME 
    CLUSTER_NODES=node1,node3,node4
    

13. Verify that you have deleted the node from the cluster.

    Run the following command to verify that the node is no longer a member of the cluster and to verify that the Oracle Clusterware components have been removed from this node:

    cluvfy comp crs -n all [-verbose]
    

    The response from this command should not contain any information about the node that you deleted; the deleted node should no longer have the Oracle Clusterware components on it.

    See Also:

    Oracle Clusterware Administration and Deployment Guide for more information about enabling and using the CVU