Adding Servers to a Cluster

3.7 Adding Servers to a Cluster

This procedure describes how to add servers to a cluster.

For adding nodes to an Oracle VM cluster, refer to Expanding an Oracle VM RAC Cluster on Exadata in Oracle Exadata Database Machine Maintenance Guide.

Caution:

If Oracle Clusterware manages additional services that are not yet installed on the new nodes, such as Oracle GoldenGate, then note the following:

It may be necessary to stop those services on the existing node before running the addNode.sh script.
It is necessary to create any users and groups on the new database servers that run these additional services.
It may be necessary to disable those services from auto-start so that Oracle Clusterware does not try to start the services on the new nodes.

Note:

To prevent problems with transferring files between existing and new nodes, you need to set up SSH equivalence. See Step 4 in Expanding an Oracle VM Oracle RAC Cluster on Exadata in for details.

Ensure the /etc/oracle/cell/network-config/*.ora files are correct and consistent on all database servers. The cellip.ora file all database server should include the older and newer database servers and storage servers.

Ensure the ORACLE_BASE and diag destination directories have been created on the Oracle Grid Infrastructure destination home.

The following is an example for Oracle Grid Infrastructure 11g:

# dcli -g /root/new_group_files/dbs_group -l root mkdir -p   \
  /u01/app/11.2.0/grid  /u01/app/oraInventory /u01/app/grid/diag

# dcli -g /root/new_group_files/dbs_group -l root chown -R grid:oinstall \
  /u01/app/11.2.0 /u01/app/oraInventory /u01/app/grid

# dcli -g /root/new_group_files/dbs_group -l root chmod -R 770   \
  /u01/app/oraInventory 

# dcli -g /root/new_group_files/dbs_group -l root chmod -R 755   \
  /u01/app/11.2.0  /u01/app/11.2.0/grid

The following is an example for Oracle Grid Infrastructure 12c:

# cd /
# rm -rf /u01/app/*
# mkdir -p /u01/app/12.1.0.2/grid
# mkdir -p /u01/app/oracle/product/12.1.0.2/dbhome_1
# chown -R oracle:oinstall /u01

Ensure the inventory directory and Grid home directories have been created and have the proper permissions. The directories should be owned by the Grid user and the OINSTALL group. The inventory directory should have 770 permission, and the Oracle Grid Infrastructure home directories should have 755.

If you are running Oracle Grid Infrastructure 12c or later:
- Make sure oraInventory does not exist inside /u01/app.
- Make sure /etc/oraInst.loc does not exist.
Create users and groups on the new nodes with the same user identifiers and group identifiers as on the existing nodes.

Note:
If Oracle Exadata Deployment Assistant (OEDA) was used earlier, then these users and groups should have been created. Check that they do exist, and have the correct UID and GID values.
Log in as the Grid user on an existing host.
Verify the Oracle Cluster Registry (OCR) backup exists.
```
ocrconfig -showbackup 
```
Verify that the additional database servers are ready to be added to the cluster using commands similar to following:
```
$ cluvfy stage -post hwos -n \
  dm02db01,dm02db02,dm02db03,dm02db04,dm02db05,dm02db06,dm02db07,dm02db08 \
  -verbose

$ cluvfy comp peer -refnode dm01db01 -n \
  dm02db01,dm02db02,dm02db03,dm02db04,dm02db05,dm02db06,dm02db07,dm02db08 \
  -orainv oinstall -osdba dba | grep -B 3 -A 2 mismatched

$ cluvfy stage -pre nodeadd -n \
  dm02db01,dm02db02,dm02db03,dm02db04,dm02db05,dm02db06,dm02db07,dm02db08 \
  -verbose -fixup -fixupdir /home/grid_owner_name/fixup.d
```
In the preceding commands, grid_owner_name is the name of the Oracle Grid Infrastructure software owner, dm02db01 through db02db08 are the new database servers, and refnode is an existing database server.
Note:
- The second and third commands do not display output if the commands complete correctly.
- An error about a voting disk, similar to the following, may be displayed:
```
 ERROR: 
 PRVF-5449 : Check of Voting Disk location "o/192.168.73.102/ \
 DATA_CD_00_dm01cel07(o/192.168.73.102/DATA_CD_00_dm01cel07)" \
 failed on the following nodes:
 Check failed on nodes: 
         dm01db01
         dm01db01:No such file or directory
 …
 PRVF-5431 : Oracle Cluster Voting Disk configuration check
```
  If such an error occurs:
  
  - If you are running Oracle Grid Infrastructure 11g, set the environment variable as follows:
```
$ export IGNORE_PREADDNODE_CHECKS=Y
```
  Setting the environment variable does not prevent the error when running the cluvfy command, but it does allow the addNode.sh script to complete successfully.
  
  - If you are running Oracle Grid Infrastructure 12c or later, use the following addnode parameters: -ignoreSysPrereqs -ignorePrereq
  
  In Oracle Grid Infrastructure 12c and later, addnode does not use the IGNORE_PREADDNODE_CHECKS environment variable.
- If a database server was installed with a certain image and subsequently patched to a later image, then some operating system libraries may be older than the version expected by the cluvfy command. This causes the cluvfy command and possibly the addNode.sh script to fail.
  
  It is permissible to have an earlier version as long as the difference in versions is minor. For example, glibc-common-2.5-81.el5_8.2 versus glibc-common-2.5-49. The versions are different, but both are at version 2.5, so the difference is minor, and it is permissible for them to differ.
  
  Set the environment variable IGNORE_PREADDNODE_CHECKS=Y before running the addNode.sh script or use the addnode parameters -ignoreSysPrereqs -ignorePrereq with the addNode.sh script to workaround this problem.
Ensure that all directories inside the Oracle Grid Infrastructure home on the existing server have their executable bits set. Run the following commands as the root user.
```
find /u01/app/11.2.0/grid  -type d  -user root  ! -perm /u+x  !     \
-perm /g+x ! -perm o+x

find /u01/app/11.2.0/grid  -type d  -user grid_owner_name  ! -perm /u+x  !   \
-perm /g+x ! -perm o+x
```
In the preceding commands, grid_owner_name is the name of the Oracle Grid Infrastructure software owner, and /u01/app/11.2.0/grid is the Oracle Grid Infrastructure home directory.

If any directories are listed, then ensure the group and others permissions are +x. The Grid_home/network/admin/samples, $GI_HOME/crf/admin/run/crfmond, and Grid_home/crf/admin/run/crflogd directories may need the +x permissions set.

If you are running Oracle Grid Infrastructure 12c or later, run commands similar to the following:
```
# chmod -R u+x /u01/app/12.1.0.2/grid/gpnp/gpnp_bcp*

# chmod -R o+rx /u01/app/12.1.0.2/grid/gpnp/gpnp_bcp*

# chmod o+r /u01/app/12.1.0.2/grid/bin/oradaemonagent /u01/app/12.1.0.2/grid/srvm/admin/logging.properties

# chmod a+r /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/*O

# chmod a+r /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/*0

# chown -f gi_owner_name:dba /u01/app/12.1.0.2/grid/OPatch/ocm/bin/emocmrsp
```
The Grid_home/network/admin/samples directory needs the +x permission:
```
chmod -R a+x /u01/app/12.1.0.2/grid/network/admin/samples
```

Run the following command. It is assumed that the Oracle Grid Infrastructure home is owned by the Grid user.

$ dcli -g old_db_nodes -l root chown -f grid_owner_name:dba \
  /u01/app/11.2.0/grid/OPatch/ocm/bin/emocmrsp

This step is needed only if you are running Oracle Grid Infrastructure 11g. In Oracle Grid Infrastructure 12c, no response file is needed because the values are specified on the command line.

Create a response file, add-cluster-nodes.rsp, as the Grid user to add the new servers similar to the following:
```
RESPONSEFILE_VERSION=2.2.1.0.0

CLUSTER_NEW_NODES={dm02db01,dm02db02,   \
dm02db03,dm02db04,dm02db05,dm02db06,dm02db07,dm02db08}

CLUSTER_NEW_VIRTUAL_HOSTNAMES={dm0201-vip,dm0202-vip,dm0203-vip,dm0204-vip,  \
dm0205-vip,dm0206-vip,dm0207-vip,dm0208-vip}
```
In the preceding file, the host names dm02db01 through db02db08 are the new nodes being added to the cluster.

Note:
The lines listing the server names should appear on one continuous line. They are wrapped in the documentation due to page limitations.
Ensure most of the files in the Grid_home/rdbms/audit and Grid_home/log/diag/* directories have been moved or deleted before extending a cluster.
Refer to My Oracle Support note 744213.1 if the installer runs out of memory. The note describes how to edit the Grid_home/oui/ora-param.ini file, and change the JRE_MEMORY_OPTIONS parameter to -Xms512m-Xmx2048m.

Add the new servers by running the addNode.sh script from an existing server as the Grid user.

If you are running Oracle Grid Infrastructure 11g:

$ cd Grid_home/oui/bin
$ ./addNode.sh -silent -responseFile /path/to/add-cluster-nodes.rsp

If you are running Oracle Grid Infrastructure 12c or later, run the addnode.sh command with the CLUSTER_NEW_NODES and CLUSTER_NEW_VIRTUAL_HOSTNAMES parameters. The syntax is:

$ ./addnode.sh -silent "CLUSTER_NEW_NODES={comma_delimited_new_nodes}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={comma_delimited_new_node_vips}"

For example:

$ cd Grid_home/addnode/

$ ./addnode.sh -silent "CLUSTER_NEW_NODES={dm02db01,dm02db02,dm02db03,dm02db04,dm02db05,
dm02db06,dm02db07,dm02db08}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={dm02db01-vip,dm02db02-vip,
dm02db03-vip,dm02db04-vip,dm02db05-vip,dm02db06-vip,dm02db07-vip,dm02db08-vip}" 
-ignoreSysPrereqs -ignorePrereq

Verify the grid disks are visible from each of the new database servers.
```
$ Grid_home/grid/bin/kfod disks=all dscvgroup=true
```
Run the orainstRoot.sh script as the root user when prompted using the dcli utility.
```
$ dcli -g new_db_nodes -l root \
  /u01/app/oraInventory/orainstRoot.sh
```
Disable HAIP on the new servers.

Before running the root.sh script, on each new server, set the HAIP_UNSUPPORTED environment variable to TRUE.
```
$ export HAIP_UNSUPPORTED=true
```
Run the Grid_home/root.sh script on each server sequentially. This simplifies the process, and ensures that any issues can be clearly identified and addressed.

Note:
The node identifier is set in order of the nodes where the root.sh script is run. Typically, the script is run from the lowest numbered node name to the highest.
Check the log file from the root.sh script and verify there are no problems on the server before proceeding to the next server. If there are problems, then resolve them before continuing.

Check the status of the cluster after adding the servers.

$ cluvfy stage -post nodeadd -n \
  dm02db01,dm02db02,dm02db03,dm02db04,dm02db05,dm02db06,dm02db07,dm02db08 \
  -verbose

Check that all servers have been added and have basic services running.
```
crsctl stat res -t
```
Note:

It may be necessary to mount disk groups on the new servers. The following commands must be run as the oracle user.
```
$ srvctl start diskgroup -g data
$ srvctl start diskgroup -g reco
```
If you are running Oracle Grid Infrastructure releases 11.2.0.2 and later, then perform the following steps:
1. Manually add the CLUSTER_INTERCONNECTS parameter to the SPFILE for each Oracle ASM instance.
```
ALTER SYSTEM SET cluster_interconnects = '192.168.10.x'       \
sid='+ASMx' scope=spfile
```
2. Restart the cluster on each new server.
3. Verify the parameters were set correctly.

Parent topic: Configuring the New Hardware