clouddba:-DBA blog on Oracle,Oracle cloud,DevOps,PostgreSQL and Other Databases

Introduction

This document will help to guide you for installation of Oracle 12cR1 RAC grid infrastructure on Virtual Box which is very popular because of shared storage simulation feature.

Below is overall requirement in nut-shell.

Hostname:-rac2

SCAN name should be resolved by following 3 IP.

 

Download Software

Download the following software.

• Rhel 7.4
• VirtualBox (5.0.4)
• Oracle 12c Release 1 (12.1.0.2) Software (64 bit)

Depending on your version of VirtualBox and  Linux, there may be some slight variation in how the screen shots look.

 

VirtualBox Network Setup

We need to make sure a host-only network is configured and check/modify the IP range for that network. This will be the public network for our RAC installation.

• Start VirtualBox from the menu.

• Select the “File > Preferences” menu option.

• Click “Network” in the left pane and click the “Host-only Networks” tab.

• Click the “Adds new host-only network” button on the right size of the screen. Depending on the host OS, a network called “vboxnet0” or “VirtualBox Host-Only Ethernet Adapter” will be created.

• Click the “Edits selected host-only network.” button on the right size of the screen.

• If you want to use a different subnet for your public addresses you can change the network details here. Just make sure the subnet you choose doesn’t match any real subnets on your network. I’ve decided to stick with the default, which for me is “192.168.56.X”.

 

Virtual Machine Setup

Now we must define the two virtual RAC nodes. We can save time by defining one VM, then cloning it when it is installed.

Start VirtualBox and click the “New” button on the toolbar. Enter the name “rac1”, OS “Linux” and Version “Red Hat (64 bit)”, then click the “Next” button.

Please select to allocate 4 GB RAM.

Next section, please select the following option

Please select “vdi” option

Please select first option “Dynamically Allocated”

Please select prefer location to create the disk and select 30GB size

 

Now press “create” button and your virtual machine is ready.

The “rac1” VM will appear on the left hand pane. Scroll down the details on the right and click on the “Network” link.

Make sure “Adapter 1” is enabled, set to “NAT”, then click on the “Adapter 2” tab.

Make sure “Adapter 2” is enabled, set to “Host-only Adapter”, then click on the “Adapter 3” tab.

Now in adapter 3,please select following:-

Move “Hard Disk” to the top of the boot order and uncheck the “Floppy” option, then click the “OK” button.

Guest Operating System Installation

With the new VM highlighted, click the “Start” button on the toolbar. On the “Select start-up disk” screen, choose the relevant Linux ISO image and click the “Start” button.

Please select following option to start installation:-

Please select prefer language as “English”

Now in the customization screen, Please select “Installation Destination” to reconfigure disk configuration.

Please select following option to configure disk partitions yourself.

Please select LVM and add new partition using “+” button below:-

The following is example to create and mention partition name and size.

The customized partition will now look like below:-

Now you need to “Accept Changes”

Once you press “Done” on upper side of screen,You will again come to customization menu.You need to select “Software Selection” to customize installation.Other wise it will be minimum installation without GUI.

Please select below option :-

Again you need to press “Done” button and now you need to re-configure network setting: –

First virtual ethernet setting should be default but enabled with automatic DHCP.

For second ethernet card, you need to specify following IP and gateway. This is public ip.

for third network interface, only put IP do not use gateway. This is private ip.

Now change hostname to “rac1” as below:-

Please select root password:-

The installation will continue now:-

 

It will take almost 30 minutes to finish installation.

After installation, please accept the license and perform post installation configuration.

 

Please configure yum to configure package installation

Please mount RHEL disk and copy all it’s content to any folder. For my case, I am copying to /home/debasis/SOFT  folder.

Please configure yum repository by creating following file.Login as root user.

Go to /etc/yum.repos.d

[root@rac1 yum.repos.d]# cat rhel.repo

[rhel.repo]
 gpgcheck = 0
 enabled = 1
 baseurl = file:///home/debasis/SOFT
 name = rhel-repo

 

[root@rac1 yum.repos.d]# yum clean all

Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-
 : manager

This system is not registered with an entitlement server. You can use subscription-manager to register.
 Cleaning repos: rhel.repo
 Cleaning up everything

Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos

[root@rac1 yum.repos.d]# yum repolist

Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-
 : manager

This system is not registered with an entitlement server. You can use subscription-manager to register.
 rhel.repo | 4.1 kB 00:00
 (1/2): rhel.repo/group_gz | 137 kB 00:00
 (2/2): rhel.repo/primary_db | 4.0 MB 00:00
 repo id repo name status
 rhel.repo rhel-repo 4,986
 repolist: 4,986

 

 

VM Guest installation for mouse and keyboard integration.(This step may create problem in RHEL 7.4.I have seen blank screen after installation of guest edition. You may ignore this step.)

 

Please install following package to correctly install vm guest edition.

yum install kernel-devel*

yum install gcc*

Please click Run button.

Configure SSH now

Now you can ping 192.168.56.101 but can not ssh You need to configure ssh . You need to login as root.

enable  PermitRootLogin to yes in  /etc/ssh/sshd_config.

[root@rac1 yum.repos.d]# service sshd restart

Redirecting to /bin/systemctl restart sshd.service

 

Install following package

 

yum install -y binutils.x86_64 glibc.x86_64 libgcc.x86_64 libstdc++.x86_64 libaio.x86_64 \
libXext.x86_64 libXtst.x86_64 libX11.x86_64 libXau.x86_64 ksh.x86_64 libxcb.x86_64 libXi.x86_64 \
make.x86_64 sysstat.x86_64 compat-libcap1.x86_64 compat-libstdc++- 33.x86_64 gcc.x86_64 gcc-c++.x86_64 \
glibc-devel.x86_64 libstdc++-devel.x86_64 libaio-devel.x86_64 cloog-ppl.x86_64 cpp.x86_64 \
glibc-headers.x86_64 kernel-headers.x86_64 mpfr.x86_64 ppl.x86_64 xterm.x86_64 xorg-x11- xauth \
xclock X11 xorg-x11- server-utils xorg-x11- xauth xclock X11 xorg-x11- server-utils psmisc.x86_64

 

Now all kernel parameters need to be added in /etc/sysctl.conf

 

fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500

Run the following command to change the current kernel parameters.

/sbin/sysctl -p

Add the following lines to a file called “/etc/security/limits.d/oracle-rdbms-server-12cR1-reinstall.conf” file.

oracle   soft   nofile    1024

oracle   hard   nofile    65536

oracle   soft   nproc    16384

oracle   hard   nproc    16384

oracle   soft   stack    10240

oracle   hard   stack    32768

oracle   hard   memlock    134217728

oracle   soft   memlock    134217728

grid soft nofile 1024

grid hard nofile 65536

grid soft nproc 16384

grid hard nproc 16384

grid soft stack 10240

grid hard stack 32768

grid hard memlock 134217728

grid soft memlock 134217728

Please create below groups and user GRID and ORACLE.

 

groupadd -g 54321 oinstall

groupadd -g 54322 dba

groupadd -g 54323 oper

groupadd -g 54325 asmdba

groupadd -g 54328 asmadmin

groupadd -g 54329 asmoper

useradd -u 54322 -g oinstall -G asmadmin,asmdba,asmoper,dba -d /home/grid -m -c "User for oracle grid infra" -s /bin/bash grid

useradd -u 54321 -g oinstall -G dba,asmdba,oper,asmadmin -d /home/oracle -m -c "User for Oracle Database Software" -s /bin/bash oracle

Please modify below files to make compatible for RAC installation

/etc/selinux/config

SELINUX= permissive

/etc/systemd/logind.conf

RemoveIPC=no

/etc/sysconfig/network

NOZEROCONF=yes

Disable firewall services

systemctl stop firewalld.service

systemctl disable firewalld.service

Please add following entry in /etc/pam.d/login

session required /lib64/security/pam_limits.so

session required pam_limits.so

Please create following directory and change the permissions

mkdir -p /u01/oemcc

mkdir -p /u01/app/grid

mkdir -p /u01/app/product

mkdir -p /u01/app/oraInventory

mkdir -p /u01/app/oracle

chown -R oracle:oinstall /u01

chown -R grid:oinstall /u01/app/oraInventory

chown -R grid:oinstall /u01/app/grid

chmod -R 775 /u01/

chmod -R 770 /u01/app/oraInventory

chown -R oracle:oinstall /u01

chown -R grid:oinstall /u01/app/oraInventory

chown -R grid:oinstall /u01/app/grid

chmod -R 775 /u01/

chmod -R 770 /u01/app/oraInventory

 

Please add/modify .bash_profile under oracle user in /home/oracle

 

# .bash_profile

# Get the aliases and functions
 if [ -f ~/.bashrc ]; then
 . ~/.bashrc
 fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

export ORACLE_BASE=/u01/app/oracle
 ORACLE_HOME=$ORACLE_BASE/product/12c/db_1
 LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
 LD_LIBRARY_PATH_64=$ORACLE_HOME/lib
 export ORACLE_BASE ORACLE_HOME LD_LIBRARY_PATH LD_LIBRARY_PATH_64

TNS_ADMIN=$ORACLE_HOME/network/admin
 LIB_PATH=$ORACLE_HOME/lib
 SQLPATH=$ORACLE_HOME/lib
 CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
 export ORACLE_PROCDIR ORACLE_PATH ORACLE_PROC ORA_NLS10 NLS_LANG TNS_ADMIN LIB_PATH SQLPATH CLASSPATH

PATH=$PATH:$ORACLE_HOME/bin:/usr/ccs/bin:/usr/bin:/etc:/usr/openwin/bin:/usr/sbin:/usr/ucb/bin:/usr/local/bin/:/usr/sfw/bin
 export PATH
 ORACLE_SID=ORCL1
 export ORACLE_SID

Please add/modify .bash_profile under grid user in /home/grid

 

# .bash_profile

# Get the aliases and functions
 if [ -f ~/.bashrc ]; then
 . ~/.bashrc
 fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

export ORACLE_BASE=/u01/app/grid
 ORACLE_HOME=$ORACLE_BASE/product/grid
 LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
 LD_LIBRARY_PATH_64=$ORACLE_HOME/lib
 export ORACLE_BASE ORACLE_HOME LD_LIBRARY_PATH LD_LIBRARY_PATH_64

TNS_ADMIN=$ORACLE_HOME/network/admin
 LIB_PATH=$ORACLE_HOME/lib
 SQLPATH=$ORACLE_HOME/lib
 CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
 export ORACLE_PROCDIR ORACLE_PATH ORACLE_PROC ORA_NLS10 NLS_LANG TNS_ADMIN LIB_PATH SQLPATH CLASSPATH

PATH=$PATH:$ORACLE_HOME/bin:/usr/ccs/bin:/usr/bin:/etc:/usr/openwin/bin:/usr/sbin:/usr/ucb/bin:/usr/local/bin/:/usr/sfw/bin
 export PATH
 ORACLE_SID=+ASM1
 export ORACLE_SID

 

Please add following entry in /etc/hosts

/etc/hosts

 

Now you need to configure shared storage which is requirement for ASM disks to be available from both node.The node needs to be stopped during this activity.

Open command prompt from your laptop after coming out from Linux node.You may choose either D: or C: to store raw ASM disk.

mkdir D:\VirtualBox\ol7-121-ra
cd:
cd C:\VirtualBox\ol7-121-rac

"c:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm1.vdi --size 5120 --format VDI --variant Fixed

"c:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm2.vdi --size 5120 --format VDI --variant Fixed

"c:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm3.vdi --size 5120 --format VDI --variant Fixed

"c:\Program Files\Oracle\VirtualBox\VBoxManage" createhd --filename asm4.vdi --size 5120 --format VDI --variant Fixed

"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach rac1 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm1.vdi --mtype shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach rac1 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm2.vdi --mtype shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach rac1 --storagectl "SATA" --port 3 --device 0 --type hdd --medium asm3.vdi --mtype shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach rac1 --storagectl "SATA" --port 4 --device 0 --type hdd --medium asm4.vdi --mtype shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd asm1.vdi --type shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd asm2.vdi --type shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd asm3.vdi --type shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" modifyhd asm4.vdi --type shareable

[root@rac1 dev]# fdisk /dev/sdb

Welcome to fdisk (util-linux 2.23.2).

 

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.

 

Device does not contain a recognized partition table

Building a new DOS disklabel with disk identifier 0x184caae5.

 

Command (m for help): n

Partition type:

p   primary (0 primary, 0 extended, 4 free)

e   extended

Select (default p): p

Partition number (1-4, default 1):

First sector (2048-10485759, default 2048):

Using default value 2048

Last sector, +sectors or +size{K,M,G} (2048-10485759, default 10485759):

Using default value 10485759

Partition 1 of type Linux and of size 5 GiB is set

 

Command (m for help): w

The partition table has been altered!

 

Calling ioctl() to re-read partition table.

Syncing disks.

 

Please add following entry to enable SCSI

Add the following to the “/etc/scsi_id.config” file to configure SCSI devices as trusted. Create the file if it doesn’t already exist.

options=-g

 

Please identify SCSI ID using following command for each individual disk of ASM which is added and shared.

[root@rac1 dev]# /usr/lib/udev/scsi_id -g -u -d /dev/sdb

1ATA_VBOX_HARDDISK_VB7a401c0f-c528452b

[root@rac1 dev]# /usr/lib/udev/scsi_id -g -u -d /dev/sdc

1ATA_VBOX_HARDDISK_VB7a276b42-ba7c6f30

[root@rac1 dev]# /usr/lib/udev/scsi_id -g -u -d /dev/sdd

1ATA_VBOX_HARDDISK_VBf3184451-5fe33178

[root@rac1 dev]# /usr/lib/udev/scsi_id -g -u -d /dev/sde

1ATA_VBOX_HARDDISK_VB8548ed60-cf05012b

Please create a file 99-ASM.rules under /etc/udev/rules.d and make following entry. RESULT should be followed by SCSI ID retrieved from previous step

KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB96eadf26-399d1fed", SYMLINK+="oracleasm/asm-disk1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB6233f8cc-70f59a49", SYMLINK+="oracleasm/asm-disk2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VBa903d07b-739bbeee", SYMLINK+="oracleasm/asm-disk3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VBba6aa7c3-7b993224", SYMLINK+="oracleasm/asm-disk4", OWNER="grid", GROUP="asmadmin", MODE="0660"

Please restart udev

# /sbin/udevadm control –reload-rules

Now you need to use partprobe to refresh new partitions

[root@rac1 dev]# /sbin/partprobe /dev/sdb1

[root@rac1 dev]# /sbin/partprobe /dev/sdc1

[root@rac1 dev]# /sbin/partprobe /dev/sdd1

[root@rac1 dev]# /sbin/partprobe /dev/sde1

Now you need to clone the RAC1 virtual machine to RAC2 using below steps

 

Remove all ASM storage device from storage option from VM Virtual box and share the ASM disk already created and attached with RAC1.

Please stop both virtual box RAC1 and RAC2 and run following command from command  prompt from your host laptop/desktop.

"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach ora_linux7_12c_n2 --storagectl "SATA" --port 1 --device 0 --type hdd --medium asm1.vdi --mtype shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach ora_linux7_12c_n2 --storagectl "SATA" --port 2 --device 0 --type hdd --medium asm2.vdi --mtype shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach ora_linux7_12c_n2 --storagectl "SATA" --port 3 --device 0 --type hdd --medium asm3.vdi --mtype shareable

"c:\Program Files\Oracle\VirtualBox\VBoxManage" storageattach ora_linux7_12c_n2 --storagectl "SATA" --port 4 --device 0 --type hdd --medium asm4.vdi --mtype shareable

Please re-initialize all IP in cloned node after logging into cloned VM (Say RAC2)

 

Hostname needs  change now to rac2:-

hostnamectl set-hostname rac2

Please share the local folder where you have kept all the installable files or you can ftp to rac1

Open mobaxterm to fetch GUI in local.

Please unzip the grid installation binaries under /u01/app/grid.

Now you need to go to /u01/app/grid/grid

Now please wait and your runInstaller will open the GUI. Select the “Install and Configure Oracle Grid Infrastructure for a Cluster” option, then click the “Next” button.

Accept the “Configure a Standard cluster” option by clicking the “Next” button.

Select the “Advance Installation” option, then click the “Next” button.

You can select your own language preference.

Please provide cluster name,scan name and port. Scan name should be in /etc/hosts.

Enter the details of the second node in the cluster, then click the “OK” button.

Click the “SSH Connectivity…” button and enter the password for the “grid” user. Click the “Setup” button to configure SSH connectivity, and the “Test” button to test it once it is complete. Once the test is complete, click the “Next” button.

Please select public and private network interface which you defined earlier for corresponding interface name.

ASM configuration should start now.

Please select 2 disks for OCR,Voting disk and management DB.

Allocation Unit Size 4 MB (For any Diskgroup you may create)

Please set the password for ASMSNMP

Please use following option.

Please select the group as below

Please select ORACLE_BASE and ORACLE_HOME

Please select oracle inventory as following

Please do not select automatic execution of root.sh rather I would prefer to run manually.

The validation check will result as below:-

Please install the following package downloded manually in both rac1 and rac2 node.

//drive.google.com/open?id=0B8-3R318F39hNl9BOEVGekVuU0U

yum install compat-libstdc++-33-3.2.3-71.el7.x86_64.rpm

Please run the verification again.

After installation progress,you will get following screen where it has been instructed to run the following script.

[root@rac1 ~]# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

 

Please execute root.sh for rac1

[root@rac1 ~]# /u01/app/grid/product/grid/root.sh

Please execute root.sh for rac2

Please proceed with next steps. It may take another 15 minutes to 30 minutes to complete.

 

Please check over-all status using below command:-

[root@rac1 ~]# . oraenv

[root@rac1 ~]# crsctl stat res -t

---

Now I will proceed for Oracle 12 database binary installation:-

Please invoke runInstaller after extracting binaries for database

[oracle@rac1 ~]$ ls -ltr
total 2625048
drwxr-xr-x. 7 oracle oinstall 136 Jul 11 2014 database
-rw-r–r–. 1 oracle oinstall 1014527110 Feb 16 2016 p17694377_121020_Linux-x86-64_2of8.zip
-rw-r–r–. 1 oracle oinstall 1673517582 Feb 16 2016 p17694377_121020_Linux-x86-64_1of8.zip

 

 

For testing purpose I am selecting this option.But for production please select 2nd option to install database first and then create database using DBCA

Please select Server class

Please select 2nd option

Please select admin managed database

Please select both node of the RAC

Please select SSH connectivity to establish SSH password less connection between 2 nodes for binary file transfer

/u01/app/product/ora12c/db_1

Please ignore the warning and proceed.