2
Configuring Shared Disks for Real Application Clusters

This chapter describes how to configure shared disk subsystems to deploy Real Application Clusters in environments that do not support (or for which you do not want to use) a cluster file system (CFS). If your system supports CFS and you do not want to use raw devices as described in this chapter, then proceed to Chapter 3. The topics in this chapter include:

• Configuring Shared Disk Subsystems for Real Application Clusters
• Configuring Logical Devices for Real Application Clusters

Configuring Shared Disk Subsystems for Real Application Clusters

Real Application Clusters requires that each instance share access to unformatted devices on a shared disk subsystem unless you are using a cluster file system for database files. The instances in Real Application Clusters write data onto the shared files to update the control file, server parameter file, each datafile, and each redo log file. The number and type of raw devices required depend on several factors as described in the following sections.

You must configure at least one device for configuration information as described under the following heading, "The Configuration Raw Device". If you select one of the preconfigured database options on the Installer's Database Configuration screen, or if you use the Database Configuration Assistant (DBCA) interactively, then you must also configure additional raw devices as described under the heading "Additional Raw Devices Required by the DBCA".

The Configuration Raw Device

You must create at least one shared raw device as an information repository for the database server configuration. This device is referred to as the Server Management (SRVM) configuration device and it is a component of Real Application Clusters. Enterprise Manager (EM) uses SRVM to perform Real Application Clusters-specific operations.

The SRVM components include the server configuration file or raw device, the Global Services Daemon (GSD) that manipulates the device, and the Server Control (SRVCTL) utility and interfaces to the configuration. SRVM also provides facilities for the DBCA, the Oracle Universal Installer (OUI), and the Network Configuration Assistant (NetCA) to enable these tools to operate on multiple nodes.

You must create a shared raw device for SRVM:

• If this installation is the first Oracle9i Server installation on your cluster
• To migrate from a pre-release 1 (9.0.1) Oracle9i Real Application Clusters database to Oracle9i Release 2 (9.2)
• To manually create your database

Oracle Enterprise Manager (EM), the Database Configuration Assistant (DBCA), the Server Control (SRVCTL) utility, and other management tools use this raw device to store configuration information about:

• The instances in your Real Application Clusters database
• The node that each instance runs on
• The Oracle home within which the database runs
• SPFILE for each cluster database

  Note: In Oracle8i and releases prior to it, Oracle stored this information in either the db_name.conf file on UNIX platforms, or in the Registry on Windows platforms.

The configuration procedures that you use for this raw device depend on your operating system:

• On UNIX, configure a shared raw device for the Server Management (SRVM) configuration; the OUI prompts you for the name of this shared file.
• On Windows NT and Windows 2000, configure a Cluster Manager (CM) voting disk/SRVM Configuration Disk. Label this raw partition as srvcfg by creating an Object Link Manager (OLM) symbolic link to a raw partition that is at least 100MB in size.

  Note: On Windows NT and Windows 2000, SRVM uses the srvcfg raw partition as the configuration repository and as the Cluster Manager voting disk.

Configure additional raw devices as described in the next section if you plan to use the DBCA to create your database.

Additional Raw Devices Required by the DBCA

To use the Database Configuration Assistant, you must configure raw devices as described in this section. These devices are in addition to the configuration device mentioned in the previous section. Create these devices before running the OUI to install the Oracle Enterprise Edition software. The DBCA cannot create a Real Application Clusters database unless you have properly configured the following raw devices:

• Seven raw devices for seven tablespace datafiles
• At least two raw devices for control files
• One raw device for each instance for its own tablespace for automatic undo management, or one rollback segment tablespace to use manual undo management
• At least two raw devices for redo log files for each instance
• One raw device for the server parameter file and five additional raw devices, one for each of the following features: Context, Oracle9i interMedia, OLAP (Online Analytical Processing), XML DB (Extensible Markup Language), and ODM (Oracle Data Mining)

  See Also: The Oracle9i Database Installation Guide for Windows for additional information about raw device requirements for Windows NT and Windows 2000

  Note: Each instance has its own redo log files, but all instances in a cluster share the control files and datafiles. In addition, each instance's log files must be readable by all other instances for recovery.

Planning Your Raw Device Creation Strategy

Before installing the Oracle Enterprise Edition software, create enough partitions of specific sizes to support your database and also leave a few spare partitions of the same size for future expansion. For example, if you have space on your shared disk array, select a limited set of standard partition sizes for your entire database. Partition sizes of 50MB, 100MB, 500MB, and 1GB are suitable for most databases. Also create a few very small and a few very large spare partitions that are, for example, 1MB and perhaps 5GB or greater in size. Based on your plans for using each partition, determine the placement of these spare partitions by combining different sizes on one disk, or by segmenting each disk into same-sized partitions.

DBCA Database Configuration Options

The DBCA has four database templates, General Purpose, Transaction Processing, Data Warehouse, and New Database. Chapter 3 describes these configuration templates in more detail. The first three templates contain preconfigured database options. The New Database template does not contain preconfigured options and is fully customizable.

If you use the General Purpose, Transaction Processing, or Data Warehouse configuration, then you must create specific tablespaces using the minimum sizes as described under the heading "Recommended Tablespace and File Capacities".

You can customize these tablespaces if you select the Customized database configuration type and the New Database template on the Universal Installer's Database Configuration screen. You can also customize these tablespaces after installing the Oracle software but before creating the database by invoking the DBCA and choosing New Database template on the Template Selection page. You can also specify the datafile names and their sizes.

To use the Customized configuration option, ensure that the raw volumes on UNIX, or raw partitions on Windows NT and Windows 2000, have enough space to accommodate the customized sizes.

Special Considerations for SunClusters: Using Veritas Volume Manager

Use Veritas Volume Manager to create your volumes and then change the permissions and ownerships on the volumes to the correct user such as oracle user. Also make sure that all the volumes on your disks have the same names and the same permissions on each node. More information about Veritas Volume Manager is available at:

http://www.veritas.com

Make sure that the parent directory of the raw device is owned by root and that this directory does not have write permission for any user other than root.

Recommended Tablespace and File Capacities

Use the tablespace size requirements in Table 2-1 for the General Purpose, Transaction Processing, and Data Warehouse configuration types. These requirements apply to UNIX, Windows NT, and Windows 2000 platforms. If you use the New Database configuration type, then use these recommended sizes as guidelines.

Table 2-1  Minimum Tablespace Sizes

Create a Raw Device For	With File Size
SYSTEM tablespace	420MB
Server Parameter File	5MB
USERS tablespace	120MB
TEMP tablespace	120MB
UNDOTBS1 tablespace	320MB
UNDOTBS2 tablespace	320MB
The DBCA uses automatic undo management by default and you should have one undo tablespace for each instance.
Note: Undo tablespace raw device datafiles for all the preconfigured database templates should be at least 320MB in size.
EXAMPLE tablespace	160MB
CWMLITE tablespace	100MB
XDB tablespace	50MB
ODM tablespace	280MB
INDX tablespace	70MB
TOOLS tablespace	12MB
DRSYS tablespace	250MB
At least two control files	110MB each
At least two redo log files for each instance	120MB (for each file)
srvcfg for the SRVM Configuration Repository	100MB

Some operating systems require additional space for the partition sizes in Table 2-1. Refer to your operating system-specific documentation for the exact raw partition size requirements.

If you do not use the DBCA and instead create your database manually, then the number of raw devices you create depends on the number of instances and database options that you install.

Configuring Logical Devices for Real Application Clusters

The configuration of raw devices is operating system-specific as described in the following sections:

• Configuring Raw Volumes for Real Application Clusters on UNIX
• Configuring Logical Drives on Windows NT and Windows 2000

Refer to the section that corresponds to your operating system.

Configuring Raw Volumes for Real Application Clusters on UNIX

Use the following procedures to configure raw volumes on UNIX:

1. Obtain root privileges.

   See Also: Oracle9i Administrator's Reference for your UNIX operating system

2. Oracle Corporation recommends that you create the datafiles, control files, and redo log files using the file naming format in Table 2-2. Although you can use any file naming scheme, the format in Table 2-2 simplifies administration. This format uses file names that identify the database with a db_name entry and that also identify the raw volume type using object names.

   Table 2-2  Recommended Names for Oracle Database Files for Two-Instance Configurations

   Database Object	Example Raw Volume
   SYSTEM tablespace	db_name_raw_system_420m
   Server Parameter File	db_name_raw_spfile_5m
   USERS tablespace	db_name_raw_users_120m
   TEMP tablespace	db_name_raw_temp_120m
   UNDOTBS1 tablespace	db_name_raw_undotbs1_320m
   UNDOTBS2 tablespace	db_name_raw_undotbs2_320m
   EXAMPLE tablespace	db_name_raw_example_160m
   CWMLITE (OLAP) tablespace	db_name_raw_cwmlite_100m
   XDB tablespace	db_name_raw_xdb_50m
   ODM (Oracle Data Mining) tablespace	db_name_raw_odm_280m
   INDX tablespace	db_name_raw_indx_70m
   TOOLS tablespace	db_name_raw_tools_12m
   DRSYS tablespace	db_name_raw_drsys_250m
   controlfile1	db_name_raw_controlfile1_110m
   controlfile2	db_name_raw_controlfile2_110m
   Two redo log files for each instance	db_name_raw_thread_lognumber_120m
   Where thread is the thread ID of the instance and number is the log number, 1 or 2, of the instance.

   Note: If you do not use automatic undo management, then substitute the undotbs1 and undotbs2 entries with the following entry to accommodate the RBS tablespace raw volume: db_name_raw_rbs_640m

The following step is recommended for all configuration types. If you select the New Database configuration type or Customized installation on the OUI, then you can enter the raw device names on the DBCA Database Storage screen.

3. On the node from which you plan to run the OUI, create an ASCII file identifying the database object-to-raw-device mapping as shown in Table 2-3. The DBCA uses this mapping file during database creation. When creating the ASCII file content for the objects, name them using the format:

   database_object=raw_device_file_path
   
   

Where database_object represents a particular database object and raw_device_file_path is the path of the datafile for that object.

Table 2-3  UNIX ASCII File Contents for the Oracle Universal Installer

Database Object	Used For
system	SYSTEM tablespace datafile
spfile	Server parameter file
users	USERS tablespace datafile
temp	TEMP tablespace datafile
undotbs1	Undo tablespace 1 datafile
undotbs2	Undo tablespace 2 datafile
example	EXAMPLE tablespace datafile
cwmlite	CWMLITE (OLAP) tablespace datafile
xdb	XDB tablespace datafile
odm	Oracle Data Mining (ODM) tablespace datafile
indx	INDX tablespace datafile
tools	TOOLS tablespace datafile
drsys	DRSYS tablespace datafile
control1	Control file 1 tablespace datafile
control2	Control file 2 tablespace datafile
redo1_1	First redo log file for the first instance
redo1_2	Second redo log file for the first instance
redo2_1	First redo log file for the second instance
redo2_2	Second redo log file for the second instance

Note: You must create at least two redo log file entries for each instance. If you do not use automatic undo management, then substitute the entries for undotbs1 and undotbs2 with an entry for rbs to use manual undo management.

When you create the ASCII file, separate each database object from its path with an equal (=) sign as shown in Example 2-1:

Example 2-1 Example UNIX ASCII File for a Two-Instance General Purpose Real Application Clusters Database Configuration

system=/dev/vx/rdsk/oracle_dg/clustdb_raw_system_420m
spfile=/dev/vx/rdsk/oracle_dg/clustdb_raw_spfile_5m
users=/dev/vx/rdsk/oracle_dg/clustdb_raw_users_120m
temp=/dev/vx/rdsk/oracle_dg/clustdb_raw_temp_120m
undotbs1=/dev/vx/rdsk/oracle_dg/clustdb_raw_undotbs1_320m
undotbs2=/dev/vx/rdsk/oracle_dg/clustdb_raw_undotbs2_320m
example=/dev/vx/rdsk/oracle_dg/clustdb_raw_example_160m 
cwmlite=/dev/vx/rdsk/oracle_dg/clustdb_raw_cwmlite_100m 
xdb=/dev/vx/rdsk/oracle_dg/clustdb_raw_xdb_50m 
odm=/dev/vx/rdsk/oracle_dg/clustdb_raw_odm_280m 
indx=/dev/vx/rdsk/oracle_dg/clustdb_raw_indx_70m
tools=/dev/vx/rdsk/oracle_dg/clustdb_raw_tools_12m
drsys=/dev/vx/rdsk/oracle_dg/clustdb_raw_dr_250m
control1=/dev/vx/rdsk/oracle_dg/clustdb_raw_controlfile1_110m
control2=/dev/vx/rdsk/oracle_dg/clustdb_raw_controlfile2_110m
redo1_1=/dev/vx/rdsk/oracle_dg/clustdb_raw_log11_120m
redo1_2=/dev/vx/rdsk/oracle_dg/clustdb_raw_log12_120m
redo2_1=/dev/vx/rdsk/oracle_dg/clustdb_raw_log21_120m
redo2_2=/dev/vx/rdsk/oracle_dg/clustdb_raw_log22_120m

Note: If you are not using automatic undo management, then substitute the entries for undotbs1 and undotbs2 with: rbs=/dev/vx/rdsk/oracle_dg/clustdb_raw_rbs_640m

To enable Oracle to determine the raw device volume names, you must specify that Oracle use the ASCII file described in this section. Do this by setting the following environment variable where filename is the complete path name of the ASCII file that contains the entries from Example 2-1:

setenv DBCA_RAW_CONFIG filename

UNIX Preinstallation Steps

After configuring the raw volumes, perform the following steps prior to installation as root user:

1. Make sure you have an osdba group defined in the /etc/group file on all nodes of your cluster. The group names for the osdba and osoper group and their group number must be identical on all nodes of your UNIX cluster that will be part of the Real Application Clusters database. The default UNIX group name for the osdba and osoper groups is dba.
2. Create an oracle user account on each node so that the oracle user:
   1. Is a member of the osdba group
   2. Is used only to install and update Oracle software
   3. Has write permissions on remote directories
3. Create a mount point directory on each node to serve as the top of your Oracle software directory structure so that:
   1. The name of the mount point on each node is identical to that on the initial node
   2. The oracle account has read, write, and execute privileges
4. On the node from which you will run the OUI, set up user equivalence by adding entries for all nodes in the cluster, including the local node, to the .rhosts file of the oracle account, or to the /etc/hosts.equiv file.
5. Exit the root account when you are done.
6. As oracle user, check for user equivalence for the oracle user by performing a remote login (rlogin) to each node in the cluster. If you are prompted for a password, then you have not given the oracle user identical attributes on all nodes. You must correct this because the OUI cannot use the rsh or rcp commands to copy Oracle products to the remote nodes' directories without user equivalence.

   Note: UNIX clusters also require an environment setup similar to single-instance Oracle environments. For these instructions and other operating system-specific Real Application Clusters preinstallation instructions, see the Oracle9i Installation Guide for your UNIX operating system.

After configuring your raw volumes, proceed to Chapter 3 to install the Oracle9i Enterprise Edition software and to configure your Oracle9i Real Application Clusters database.

Configuring Logical Drives on Windows NT and Windows 2000

If you do not use a cluster file system for datafiles, then the datafiles, control files, and redo log files must reside on unformatted raw devices on Windows NT and Windows 2000 platforms. On Windows, these are more commonly referred to as logical drives that reside within extended partitions. The extended partitions point to raw space on the disks. To configure the logical drives, create multiple logical partitions using Windows NT Disk Administrator or Windows 2000 Disk Management.

Before creating the logical partitions, first create extended partitions that refer to the raw space on the disk. Then create multiple logical partitions within the extended partitions and assign symbolic link names to them using the Object Link Manager (OLM).

See Also: Oracle9i Database Installation Guide for Windows for further information about configuring logical drives for Real Application Clusters on Windows NT and Windows 2000

The DBCA General Purpose, Transaction Processing, and Data Warehouse database configuration types require the symbolic link names shown in the left-hand column of Table 2-4 for a two-instance Real Application Clusters database. Oracle uses these names to map the tablespaces as shown in the sample file in Example 2-1.

Table 2-4  Symbolic Link Names for Preconfigured Database Types

Symbolic Link Name...	Used for...
db_name_system	SYSTEM tablespace
db_name_spfile	Server parameter file
db_name_users	USERS tablespace
db_name_temp	TEMP tablespace
db_name_undotbs1	First UNDO tablespace
db_name_undotbs2	Second UNDO tablespace
db_name_example	EXAMPLE tablespace
db_name_cwmlite	CWMLITE tablespace
db_name_xdb	XDB tablespace
db_name_odm	ODM tablespace
db_name_indx	INDX tablespace
db_name_tools	TOOLS tablespace
db_name_drsys	DRSYS tablespace
db_name_control1	First control file
db_name_control2	Second control file
srvcfg	SRVM Configuration Repository
db_name_redo_thread_number Where thread is the thread ID of the instance and number is the log number (1 or 2) for the instance.	Redo Log Files Each instance must have at least two redo log files. If the database name is db, then the link names for the first instance's redo log files should be: db_redo1_1 db_redo1_2 The link names for the second instance's redo log files should be: db_redo2_1 db_redo2_2

Note: If you do not use automatic undo management, then to accommodate a rollback segment tablespace, instead of using db_name_undotbs1 and db_name_undotbs2, create this symbolic link: db_name_rbs

To install the Oracle operating system-dependent clusterware, run the Oracle Cluster Setup Wizard. The Cluster Setup Wizard installs the Object Link Manager with which you create persistent symbolic links to the logical drives.

If you do not install the Oracle OSD clusterware, then copy the Object Link Manager software from the preinstall directory on the Oracle CD-ROM. Refer to the online document, the Oracle9i Database Installation Guide for Windows for further procedures for completing the configuration using OLM.

After configuring your logical drives, proceed to Chapter 3 to install the Oracle9i Enterprise Edition software and to configure your Real Application Clusters database.