Installation of Oracle 12c RAC Flex cluster infrastructure on Virtual Box
Introduction
This document will help to guide you for installation of Oracle 12cR1 RAC Flex cluster infrastructure on Virtual Box which is very popular because of shared storage simulation feature.
Below is overall requirement in nut-shell.
We need 3 servers for now.I will install RAC on First 2 servers and 3rd server will act as DNS server.My desktop RAM is 16 GB.
Hostname:-rac1.example.com
Hostname:-rac2.example.com
Hostname:-dns.example.com
Physical IP:192.168.50.113
SCAN name should be resolved by GNS.The name would be :-rac-scan.gns.rac.example.com
Virtual IP should be assigned from DHCP and DNS.
GNS name:-gns.rac.example.com
Download Software
Download the following software.
Depending on your version of VirtualBox and Linux, there may be some slight variation in how the screen shots look.
VirtualBox Network Setup
We need to make sure a host-only network is configured and check/modify the IP range for that network. This will be the public network for our RAC installation.
-
Start VirtualBox from the menu.
-
Select the “File > Preferences” menu option.
-
Click “Network” in the left pane and click the “Host-only Networks” tab.
-
Click the “Adds new host-only network” button on the right size of the screen. Depending on the host OS, a network called “vboxnet0” or “VirtualBox Host-Only Ethernet Adapter” will be created.
-
Click the “Edits selected host-only network.” button on the right size of the screen.
-
If you want to use a different subnet for your public addresses you can change the network details here. Just make sure the subnet you choose doesn’t match any real subnets on your network. I’ve decided to choose “192.168.50.X”.
-
Please go to DHCP server tab and do not select DHCP as we are building seperate DHCP server
Virtual Machine Setup
Now we must define the two virtual RAC nodes. We can save time by defining one VM, then cloning it when it is installed.
Start VirtualBox and click the “New” button on the toolbar. Enter the name “rac1”, OS “Linux” and Version “Oracle (64 bit)”, then click the “Next” button.
Please select to allocate 4 GB RAM.
Next section, please select the following option
Please select “vdi” option
Please select first option “Dynamically Allocated”
Please select prefer location to create the disk and select 30GB size
Now press “create” button and your virtual machine is ready.
The “rac1” VM will appear on the left hand pane. Scroll down the details on the right and click on the “Network” link.
Make sure “Adapter 1” is enabled, set to “NAT”, then click on the “Adapter 2” tab.
Make sure “Adapter 2” is enabled, set to “Host-only Adapter”, then click on the “Adapter 3” tab.
Now in adapter 3,please select following:-
Move “Hard Disk” to the top of the boot order and uncheck the “Floppy” option, then click the “OK” button.
Guest Operating System Installation in all the VM .
Please use any standard note to install Oracle Linux 6.
You can choose 100GB mount point for /u01 in RAC1 and RAC2 .You can choose 30 GB root mount point for DNS server.
Configure YUM repository in all the VM
With the new VM highlighted, click the “Start” button on the toolbar. On the “Select start-up disk” screen, choose the relevant Linux ISO image and click the “Start” button.y folder. For my case, I am copying to /home/debasis/SOFT folder.
Please configure yum repository by creating following file.Login as root user.
Go to /etc/yum.repos.d
[root@rac1 yum.repos.d]# cat rhel.repo
[rhel.repo]
gpgcheck = 0
enabled = 1
baseurl = file:///home/debasis/SOFT
name = rhel-repo
[root@rac1 yum.repos.d]# yum clean all
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-
: manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Cleaning repos: rhel.repo
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
[root@rac1 yum.repos.d]# yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-
: manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
rhel.repo | 4.1 kB 00:00
(1/2): rhel.repo/group_gz | 137 kB 00:00
(2/2): rhel.repo/primary_db | 4.0 MB 00:00
repo id repo name status
rhel.repo rhel-repo 4,986
repolist: 4,986
This part will help to configure DHCP and DNS for NAME server DNS( 192.168.50.113)
Please change /etc/sysconfig/network-scripts/ifcfg-eth1 parameter BOOTPROTO=dhcp
configure DHCP server:-
DHCP server will be used to provide dynamic IP for VIP and SCAN IP. There has to be a package which provides the service, and sure enough this is called dhcp. The dhcp client normally is installed already which makes it easy to avoid confusion.
Step 1:-Install DHCP server
yum install dhcp*
Step 2:-Configure DHCP server
vi /etc/dhcp/dhcpd.conf
Please add following entry
I am explaining major parameters
domain-name-servers is name or dns server ip (192.168.50.113)
router is default gateway (192.168.50.1)
range is between 192.168.50.117 and 192.168.50.130.So all DHCP IP will be assigned from that range.
Configure NAME server:-
DNS for GNS
We need to configure DNS first and then we have to delegate the subdomain to GNS, which involves a little bit of work on the DNS server. Also, we need to put a virtual IP address for GNS into DNS, but that address must not be in use. I also created a dedicated server for DNS and bind, on 192.168.50.113. The configuration changes I made are as follows:
Step 1:-Install bind packages
yum install bind*
Step 2:-Make configuration change in /etc/named.conf
options
{
/* make named use port 53 for the source of all queries, to allow
* firewalls to block all ports except 53:
*/
query-source port 53;
query-source-v6 port 53;
// Put files that named is allowed to write in the data/ directory:
directory “/var/named”; // the default
dump-file “data/cache_dump.db”;
statistics-file “data/named_stats.txt”;
memstatistics-file “data/named_mem_stats.txt”;
allow-transfer {“none”;};
zone-statistics yes;
};
logging
{
channel default_debug {
file “data/named.run”;
severity dynamic;
};
};
zone “example.com” IN {
type master;
file “/var/named/example.com.db”;
allow-transfer { 192.168.50.113; };
};
zone “50.168.192.in-addr.arpa” IN {
type master;
file “/var/named/192.168.50.db”;
};
// named.rfc1912.zones here….
Step 3:=Now you need to create example.com.db and 192.168.50.db under /var/named
[root@dns dhcp]#vi /var/named/example.com.db
$TTL 86400
$ORIGIN example.com.
@ 1D IN SOA dns.example.com. hostmaster.example.com. (
2002022401 ; serial
3H ; refresh
15 ; retry
1w ; expire
3h ; minimum
)
; main domain name servers
IN NS dns
dns IN A 192.168.50.113
; A record for mail server above
mail IN A 192.168.1.113
rac1 IN A 192.168.50.110
rac2 IN A 192.168.50.111
example.com. A 192.168.50.113
; sub-domain definitions
$ORIGIN rac.example.com.
@ IN NS gns.rac.example.com.
IN NS dns.example.com.
; sub-domain address records for name server only – glue record
gns IN A 192.168.50.115 ; ‘glue’ record
[root@dns dhcp]#vi /var/named/192.168.50.db
$TTL 86400 ; 24 hours could have been written as 24h or 1d
@ 1D IN SOA dns.example.com. hostmaster.example.com. (
2002022401 ; serial
3H ; refresh
15 ; retry
1w ; expire
3h ; minimum
)
IN NS dns.example.com.
dns IN A 192.168.56.113
110 IN PTR rac1.example.com.
111 IN PTR rac2.example.com.
115 IN PTR gns.rac.example.com.
Step 3:-Restart the named server
service named restart
Reference:-
DNS and DHCP Setup Example for Grid Infrastructure GNS (Doc ID 946452.1)
Please stop network manager for all servers so that /etc/resolv.conf will not be updated automatically.
service NetworkManager stop
chkconfig NetworkManager off
Please make entry in /etc/resolv.conf in all servers so that it connects to named server
options attempts: 2
options timeout: 1
search example.com rac.example.com
nameserver 192.168.50.113
VM Guest installation for mouse and keyboard integration.You can do for all VM.
Please install following package to correctly install vm guest edition.
yum install kernel-devel*
yum install gcc*
Please click Run button.
Configure SSH now in ALL VM
Now you can ping but can not ssh You need to configure ssh . You need to login as root.
enable PermitRootLogin to yes in /etc/ssh/sshd_config.
[root@rac1 yum.repos.d]# service sshd restart
Redirecting to /bin/systemctl restart sshd.service
Do this step for all servers
Install following package in RAC1
yum install -y binutils.x86_64 glibc.x86_64 libgcc.x86_64 libstdc++.x86_64 libaio.x86_64 \
libXext.x86_64 libXtst.x86_64 libX11.x86_64 libXau.x86_64 ksh.x86_64 libxcb.x86_64 libXi.x86_64 \
make.x86_64 sysstat.x86_64 compat-libcap1.x86_64 compat-libstdc++- 33.x86_64 gcc.x86_64 gcc-c++.x86_64 \
glibc-devel.x86_64 libstdc++-devel.x86_64 libaio-devel.x86_64 cloog-ppl.x86_64 cpp.x86_64 \
glibc-headers.x86_64 kernel-headers.x86_64 mpfr.x86_64 ppl.x86_64 xterm.x86_64 xorg-x11- xauth \
xclock X11 xorg-x11- server-utils xorg-x11- xauth xclock X11 xorg-x11- server-utils psmisc.x86_64
Now all kernel parameters need to be added in /etc/sysctl.conf in RAC1
Run the following command to change the current kernel parameters.
/sbin/sysctl -p
Add the following lines to a file called “/etc/security/limits.d/oracle-rdbms-server-12cR1-preinstall.conf” file in RAC1.
Please create below groups and user GRID and ORACLE in RAC1.
groupadd -g 54321 oinstall
groupadd -g 54322 dba
groupadd -g 54323 oper
groupadd -g 54325 asmdba
groupadd -g 54328 asmadmin
groupadd -g 54329 asmoper
useradd -u 54322 -g oinstall -G asmadmin,asmdba,asmoper,dba -d /home/grid -m -c “User for Oracle Grid Infrastructure Software” -s /bin/bash grid
useradd -u 54321 -g oinstall -G dba,asmdba,oper,asmadmin -d /home/oracle -m -c “User for Oracle Database Software” -s /bin/bash oracle
Please modify below files to make compatible for RAC installation in RAC1
/etc/selinux/config
SELINUX= permissive
/etc/systemd/logind.conf
RemoveIPC=no
/etc/sysconfig/network
NOZEROCONF=yes
Disable firewall services for all VM
service iptables stop
chkconfig iptables off
Please add following entry in /etc/pam.d/login in RAC1
session required /lib64/security/pam_limits.so
session required pam_limits.so
Please create following directory and change the permissions in RAC1
mkdir -p /u01/oemcc
mkdir -p /u01/app/grid
mkdir -p /u01/app/product
mkdir -p /u01/app/oraInventory
mkdir -p /u01/app/oracle
chown -R oracle:oinstall /u01
chown -R grid:oinstall /u01/app/oraInventory
chown -R grid:oinstall /u01/app/grid
chmod -R 775 /u01/
chmod -R 770 /u01/app/oraInventory
chown -R oracle:oinstall /u01
chown -R grid:oinstall /u01/app/oraInventory
chown -R grid:oinstall /u01/app/grid
chmod -R 775 /u01/
chmod -R 770 /u01/app/oraInventory
Please add/modify .bash_profile under oracle user in /home/oracle in RAC1
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
export ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=$ORACLE_BASE/product/12c/db_1
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
LD_LIBRARY_PATH_64=$ORACLE_HOME/lib
export ORACLE_BASE ORACLE_HOME LD_LIBRARY_PATH LD_LIBRARY_PATH_64
TNS_ADMIN=$ORACLE_HOME/network/admin
LIB_PATH=$ORACLE_HOME/lib
SQLPATH=$ORACLE_HOME/lib
CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
export ORACLE_PROCDIR ORACLE_PATH ORACLE_PROC ORA_NLS10 NLS_LANG TNS_ADMIN LIB_PATH SQLPATH CLASSPATH
PATH=$PATH:$ORACLE_HOME/bin:/usr/ccs/bin:/usr/bin:/etc:/usr/openwin/bin:/usr/sbin:/usr/ucb/bin:/usr/local/bin/:/usr/sfw/bin
export PATH
ORACLE_SID=ORCL1
export ORACLE_SID
Please add/modify .bash_profile under grid user in /home/grid in RAC1
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
export ORACLE_BASE=/u01/app/grid
ORACLE_HOME=$ORACLE_BASE/product/grid
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
LD_LIBRARY_PATH_64=$ORACLE_HOME/lib
export ORACLE_BASE ORACLE_HOME LD_LIBRARY_PATH LD_LIBRARY_PATH_64
TNS_ADMIN=$ORACLE_HOME/network/admin
LIB_PATH=$ORACLE_HOME/lib
SQLPATH=$ORACLE_HOME/lib
CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
export ORACLE_PROCDIR ORACLE_PATH ORACLE_PROC ORA_NLS10 NLS_LANG TNS_ADMIN LIB_PATH SQLPATH CLASSPATH
PATH=$PATH:$ORACLE_HOME/bin:/usr/ccs/bin:/usr/bin:/etc:/usr/openwin/bin:/usr/sbin:/usr/ucb/bin:/usr/local/bin/:/usr/sfw/bin
export PATH
ORACLE_SID=+ASM1
export ORACLE_SID
Please add following entry in /etc/hosts in RAC1
/etc/hosts
Now you need to configure shared storage which is requirement for ASM disks to be available from both RAC node.
Open command prompt from your laptop after coming out from Linux node.You may choose either D: or C: to store raw ASM disk.
[root@rac1 dev]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x184caae5.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-10485759, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-10485759, default 10485759):
Using default value 10485759
Partition 1 of type Linux and of size 5 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Please add following entry to enable SCSI
Add the following to the “/etc/scsi_id.config” file to configure SCSI devices as trusted. Create the file if it doesn’t already exist.
options=-g
Please identify SCSI ID using following command for each individual disk of ASM which is added and shared.
[root@rac1 dev]# /usr/lib/udev/scsi_id -g -u -d /dev/sdb
1ATA_VBOX_HARDDISK_VB7a401c0f-c528452b
Please create a file 99-ASM.rules under /etc/udev/rules.d and make following entry. RESULT should be followed by SCSI ID retrieved from previous step
KERNEL==”sd?1″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -d /dev/$parent”, RESULT==”1ATA_VBOX_HARDDISK_VB1bb0c812-29a5f87c”, NAME=”asm-disk1″, OWNER=”oracle”, GROUP=”dba”, MODE=”0660″
Please restart udev
# /sbin/udevadm control –reload-rules # /sbin/start_udev
Now you need to use partprobe to refresh new partitions
[root@rac1 dev]# /sbin/partprobe /dev/sdb1
Now you need to clone the RAC1 virtual machine to RAC2 using below steps
Remove all ASM storage device from storage option from VM Virtual box and share the ASM disk already created and attached with RAC1.
Please stop both virtual box RAC1 and RAC2 and run following command from command prompt from your host laptop/desktop.
“c:\Program Files\Oracle\VirtualBox\VBoxManage” storageattach ora_linux6_12c_n2 –storagectl “SATA” –port 1 –device 0 –type hdd –medium asm1.vdi –mtype shareable
Please re-initialize all IP in cloned node after logging into cloned VM (Say RAC2)
Log in to the “rac2” virtual machine as the “root” user so we can reconfigure the network settings to match the following.
- hostname: rac2.example.com
- eth0: DHCP (*Not* Connect Automatically)
- eth1: IP=192.168.50.110, Subnet=255.255.255.0, Gateway=192.168.50.1, DNS=192.168.50.113, Search=example.com (Connect Automatically)
- eth2: IP=192.168.1.102, Subnet=255.255.255.0, Gateway=<blank>, DNS=<blank>, Search=<blank> (Connect Automatically)
Amend the hostname in the “/etc/sysconfig/network” file.
NETWORKING=yes HOSTNAME=rac2.example.com
Check the MAC address of each of the available network connections. Don’t worry that they are listed as “eth3” to “eth5”. These are dynamically created connections because the MAC address of the “eth0” to “eth2” connections are incorrect.
# ifconfig -a | grep eth eth3 Link encap:Ethernet HWaddr 08:00:27:43:41:74 eth4 Link encap:Ethernet HWaddr 08:00:27:4B:4F:0F eth5 Link encap:Ethernet HWaddr 08:00:27:E8:70:17 #
Edit the “/etc/sysconfig/network-scripts/ifcfg-eth0”, amending only the HWADDR setting as follows and deleting the UUID entry. Note, the HWADDR value comes from the “eth3” interface displayed above.Please change the HWADDR and interface name to “eth0”
HWADDR=08:00:27:43:41:74
Edit the “/etc/sysconfig/network-scripts/ifcfg-eth1”, amending only the IPADDR and HWADDR settings as follows and deleting the UUID entry. Note, the HWADDR value comes from the “eth4” interface displayed above.Please change the HWADDR and interface name to “eth1”
HWADDR=08:00:27:4B:4F:0F IPADDR=192.168.50.111
Edit the “/etc/sysconfig/network-scripts/ifcfg-eth2”, amending only the IPADDR and HWADDR settings as follows and deleting the UUID entry. Note, the HWADDR value comes from the “eth5” interface displayed above.Please change the HWADDR and interface name to “eth2”
HWADDR=08:00:27:E8:70:17 IPADDR=192.168.1.102
Also please check rules related to network under /etc/udev/rules.d/90-persistent.rules file and change
interface name accordingly
Please share the local folder where you have kept all the installable files or you can ftp to rac1
Open mobaxterm to fetch GUI in local.
Please unzip the grid installation binaries under /u01/app/grid.
Now you need to go to /u01/app/grid/grid
Now please wait and your runInstaller will open the GUI. Select the “Install and Configure Oracle Grid Infrastructure for a Cluster” option, then click the “Next” button.
Accept the “Configure a Flexcluster” option by clicking the “Next” button.
Select the “Advance Installation” option, then click the “Next” button.
You can select your own language preference.
Please provide cluster name,scan name and port. GNS IP should be 192.168.50.115 (This is configured in /var/named/example.com.db in DNS server)
Enter the details of the second node in the cluster, then click the “OK” button.
Click the “SSH Connectivity…” button and enter the password for the “grid” user. Click the “Setup” button to configure SSH connectivity, and the “Test” button to test it once it is complete. Once the test is complete, click the “Next” button.
Please select public and private network interface which you defined earlier for corresponding interface name.
ASM configuration should start now.
Please select 2 disks for OCR,Voting disk and management DB.
Allocation Unit Size 4 MB (For any Diskgroup you may create)
Please set the password for ASMSNMP
Please use following option.
Please select the group as below
Please select ORACLE_BASE and ORACLE_HOME
Please select oracle inventory as following
Please do not select automatic execution of root.sh rather I would prefer to run manually.
The validation check will result as below:-
Please install the following package downloded manually in both rac1 and rac2 node.
https://drive.google.com/open?id=0B8-3R318F39hNl9BOEVGekVuU0U
yum install compat-libstdc++-33-3.2.3-71.el7.x86_64.rpm
Please run the verification again.
After installation progress,you will get following screen where it has been instructed to run the following script.
[root@rac1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
Please execute root.sh for rac1
[root@rac1 ~]# /u01/app/grid/product/grid/root.sh
Please proceed with next steps. It may take another 15 minutes to 30 minutes to complete.
Please check over-all status using below command:-
[root@rac1 ~]# . oraenv
[root@rac1 ~]# crsctl stat res -t
