Friday, 20 April 2018

Exadata Database Machine Health Check Using Exachk Utility


Oracle provides "Exachk" utility to conduct a comprehensive Exadata Health Check on Exadata Database Machine to validate hardware, firmware and configuration.

Exachk Utility is available for Oracle engineered systems such as Exadata (V2 and above), Exalogic, Exalytics, SuperCluster, MiniCluster, ZDLRA & Big Data. Exachk utility performs the following checks:
  • Configuration checks for Compute nodes, Storage cells and InfiniBand Switches
  • Grid Infrastructure, Database and ASM and Operating System software checks
  • MAA Scorecard which conducts an automatic MAA Review
  • Exadata Software Planner, Software prechecks, Exadata and Database Critical Issue alerts

It is recommended to execute the latest exachk at the following situation:
  • Monthly
  • Before any planned maintenance activity
  • Immediately after completion of planned maintenance activity
  • Immediately after an outage or incident

Steps to Exadata Health Exachk Using Exachk Utility

  • Download latest Exachk utility from the MOS note. As of writing the latest Exachk verion available is "12.2.0.1.4_20171212"
Oracle Exadata Database Machine exachk or HealthCheck (Doc ID 1070954.1)

Note: It is recommended to use latest Exachk to perform Exadata Health Check





  • As root user, create 'Exachk' directory on compute node 1 as follows
[root@dm01db01 ~]# cd /root
[root@dm01db01 ~]# mkdir Exachk


  • Using Winscp Copy the Downloaded Exachk utility from your desktop/laptop to the Exadata compute node 1 under /root/Exachk





  • As root user, Login to Exadata Compute node 1 and unzip the Exachk utility
[root@dm01db01 ~]# cd /root/Exachk/

[root@dm01db01 Exachk]# ls -ltr

total 112576
-rw-r--r-- 1 root root 115158363 Apr 10 05:11 exachk.zip

[root@dm01db01 Exachk]# unzip exachk.zip






  • Ensure that the SSH is setup across all Compute nodes, Storage cells and Ibswitches
[root@dm01db01 Exachk]# dcli -g ~/all_group -l root 'uptime'


To Setup SSH across the cluster, use the following command:

[root@dm01db01 ~]# cd /opt/oracle.SupportTools/
[root@dm01db01 oracle.SupportTools]# ./setup_ssh_eq.sh ~/all_group root welcome1

  • As root user, Execute the Exachk utility
[root@dm01db01 ~]# cd /root/Exachk/

[root@dm01db01 Exachk]# ls -ltr


[root@dm01db01 Exachk]# ./exachk





Depending on the Exadata Cluster Size and number of databases it may take several minutes to complete Exachk execution.

  • Using Winscp, copy the Exachk zip file and/or html file to your desktop/laptop to review
  • Open the html file, review it and take necessary action if necessary
  • Under the table of contents there are different component listed. Look out for the CRITICAL and FAIL components.
Click on the 'view' hyperlink for more details and the recommendation to fix the problem.



MAA Scorecard



Conclusion

In this article we have learned to perform complete Exadata Stack Health Check using Exachk utility. Exachk Utility is available for Oracle engineered systems such as Exadata (V2 and above), Exalogi, Exalytics, SuperCluster, MiniCluster, ZDLRA & Big Data.

Saturday, 14 April 2018

Exadata - Configure Compute Node and Storage Cell SMTP Email Notification

On Exadata Database Machine you can configure the following Compute Node and Storage Cell attributes to setup the database server and Storage cells to send notifications about alerts.
  • smtpServer
  • smtpFrom
  • smtpFromAddr
  • smtpToAddr
  • snmpSubscriber
  • notificationMethod
  • notificationPolicy

In this article we will demonstrate how to setup the database server and Storage cells to send notifications about alerts

Compute Nodes:


Configure Compute nodes SMTP email notification for alerts. This can be accomplished using dbmcli alter dbserver command


# Compute node 1


DBMCLI>alter dbserver smtpFrom='Exadata - dm01db01'

DBMCLI>alter dbserver smtpFromAddr='dbmadmin@dm01db01.netsoftmate.com'
DBMCLI>alter dbserver smtpToAddr='oradba@netsoftmate.com'
DBMCLI>alter dbserver smtpServer='smtp.server'
DBMCLI>alter dbserver snmpSubscriber=((host=192.168.10.1,port=162,community=public,type=ASR))
DBMCLI>alter dbserver notificationPolicy='critical,warning,clear'
DBMCLI>alter dbserver notificationMethod='mail,snmp'
DBMCLI>alter dbserver validate mail


Or you can use the following command


DBMCLI>alter dbserver smtpFrom='Exadata - dm01db01', smtpFromAddr='dbmadmin@dm01db01.netsoftmate.com', smtpToAddr='oradba@netsoftmate.com', smtpServer='smtp.server', snmpSubscriber='host=192.168.10.1,port=162,community=public,type=ASR', notificationPolicy='critical,warning,clear', notificationMethod='mail,snmp'


DBMCLI>alter dbserver validate mail


*** Repeat the above step for all the Compute nodes in the cluster.


# verify


# dcli -g ~/dbs_group -l root "dbmcli -e 'list dbserver detail' | grep smtpFrom"

# dcli -g ~/dbs_group -l root "dbmcli -e 'list dbserver detail' | grep smtpFromAddr"
# dcli -g ~/dbs_group -l root "dbmcli -e 'list dbserver detail' | grep smtpToAddr"
# dcli -g ~/dbs_group -l root "dbmcli -e 'list dbserver detail' | grep smtpServer"
# dcli -g ~/dbs_group -l root "dbmcli -e 'list dbserver detail' | grep notificationMethod"


or use the following command


# dcli -g ~/dbs_group -l root "dbmcli -e 'list dbserver detail' | egrep '(smtpFrom|smtpFromAddr|smtpToAddr|smtpServer|notificationMethod)'"




Storage Cells:

Configure Compute nodes SMTP email notification for alerts. This can be accomplished using cellcli alter cell command

# Storage Cell 01


CELLCLI>alter cell smtpFrom='Exadata - dm01cel01'

CELLCLI>alter cell smtpFromAddr='celladmin@dm01cel01.netsoftmate.com'
CELLCLI>alter cell smtpToAddr='oradba@netsoftmate.com'
CELLCLI>alter cell smtpServer='smtp.server'
CELLCLI>alter cell snmpSubscriber=((host=192.168.10.1,port=162,community=public,type=ASR))
CELLCLI>alter dbserver notificationPolicy='critical,warning,clear'
CELLCLI>alter cell notificationMethod='mail,snmp'
CELLCLI>alter cell validate mail


or you can also use the following command


CELLCLI>alter cell smtpFrom='Exadata - dm01cel01', smtpFromAddr='celladmin@dm01cel01.netsoftmate.com', smtpToAddr='oradba@netsoftmate.com', smtpServer='smtp.server', notificationMethod='mail,snmp'


CELLCLI>alter cell validate mail

# Verify

# dcli -g ~/cell_group -l root "dbmcli -e 'list cell detail' | grep smtpFrom"

# dcli -g ~/cell_group -l root "dbmcli -e 'list cell detail' | grep smtpFromAddr"
# dcli -g ~/cell_group -l root "dbmcli -e 'list cell detail' | grep smtpToAddr"
# dcli -g ~/cell_group -l root "dbmcli -e 'list cell detail' | grep smtpServer"
# dcli -g ~/cell_group -l root "dbmcli -e 'list cell detail' | grep notificationMethod"


or you can use the following command


# dcli -g ~/cell_group -l root "cellcli -e 'list cell detail' | egrep '(smtpFrom|smtpFromAddr|smtpToAddr|smtpServer|notificationMethod)'"


*** Repeat the above step for all the Storage Cells in the cluster.



Conclusion

In this article we have learned how to Configure Compute nodes and Storage Cell SMTP email notification for alerts.



Friday, 13 April 2018

Exadata - Replace Failed Internal USB Drive on Exadata Storage Cell

While working on Exadata Storage cell patching, the patching failed due to failed internal USB drive on a storage cell.

Oracle uses internal USB drive to backup Exadata Storage cell automatically. We don't have to backup Storage cell manually.

In this article I will demonstrate how to replace a failed USB drive an Exadata Storage cell

  • You will receive an automated smtp alert (if configured) similar to below.



  • You can also use the following command to check for USB drive failure

[root@dm01cel01 ~]# cellcli -e list alerthistory
         1_1     2018-04-10T18:25:42-05:00       warning         "Internal USB status is not present.  Affected USB Slots : 0"

  • You can also use the following ILOM command to check for USB drive failure

[root@dm01cel01 ~]# ssh dm01cel01-ilom
Password: *******


Oracle(R) Integrated Lights Out Manager



Version 3.2.10.22.a r121524



Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.


Warning: HTTPS certificate is set to factory default.



Hostname: dm01cel01-ilom



-> show /SYS/MB/USB0


  • Open an SR with Oracle if an ASR is already generated
  • Upload sundgia.sh and ILOM Snapshot to the SR for investigation
  • Oracle confirms the that USB drive is faulted
  • Oracle opens a Field task
  • Oracle dispatch team contacts the SR owner with the hardware dispatch details
  • Confirm the Hardware replacement schedule over email and/or SR
  • Schedule the Hardware replacement
  • Oracle FE arrives at the data center with the new USB drive
  • Shutdown the storage cell by following the steps from the MOS below

Steps to shut down or reboot an Exadata storage cell without affecting ASM (Doc ID 1188080.1)
  • Oracle FE replaces the faulty USB drive and power up the storage cell
  • Confirm that the USB drive is good

-> show /SYS/MB/USB0


 /SYS/MB/USB0

    Targets:


    Properties:

        type = USB Port
        fault_state = OK
        clear_fault_action = (none)


    Commands:

        cd
        set
        show


->



[root@dm01cel01 ~]# cellcli -e list alerthistory

         1_2     2018-04-11T02:45:49-05:00       clear           "Internal USB status is back to normal.  Affected USB Slots : 0"

  • You will receive an automated smtp alert (if configured) similar to below that the USB status is back to normal




Conclusion

In this article we have learned how to replace a faulty USB drive in Exadata Storage cell. Oracle uses USB drive to backup Exadata Storage cell automatically. We don't have to backup Storage cell manually.

Sunday, 18 March 2018

Patching Oracle Database Appliance (ODA) Stack


Oracle Database Appliance stack patching is very simple and straight forward. The ODA patching process is fully automated and single patch contains updates for all system components such as OS, Firmware, BIOS, ILOM, Storage, Grid Infrastructure and Database. ODA patches are released every quarter and one should always consult the MOS note 888888.1 for recommended patches.

Oracle Database Appliance stack Patching consists of following major steps:

  1. Patching ODA Nodes (Includes Grid Infrastructure Patching)
  2. Patching Storage Shelve(s)
  3. Patching Database


In this article we will demonstrate the steps for patching entire Oracle Database Appliance stack to 12.2.1.2.0.

Steps for Patching Oracle Database Appliance (ODA) Stack




Step 1: ODA Nodes Patching

  • As root user, note down the current version of the ODA on both nodes
# oakcli show version -detail
  • As root user, Verify the shared disks status on both nodes. There shouldn't be any bad disk.

# oakcli show disk
# oakcli show diskgroup

  • As root user, Validate the ODA infrastructure on both nodes
# oakcli validate -d
  • Ensure that sufficient space is available in the following file sytems on both nodes
# df -h -P / /opt /u01 /boot
  • Download and stage the ODA patches on the both nodes
# mkdir -p /u01/app/oracle/software/ODA
  • Ensure root user equivalence is working on both nodes

From node 1
ssh 192.168.16.25 hostname
 


From node 2
ssh 192.168.16.24 hostname
 

  • Copy software to node 2
# mkdir -p /u01/app/oracle/software/ODA
# scp /u01/app/oracle/software/ODA/p* 192.168.16.25:/u01/app/oracle/software/ODA/
  • Unpack software on both nodes

# /opt/oracle/oak/bin/oakcli unpack –package /u01/app/oracle/software/ODA/p27119652_122120_Linux-x86-64_1of3.zip
# /opt/oracle/oak/bin/oakcli unpack –package /u01/app/oracle/software/ODA/p27119652_122120_Linux-x86-64_2of3.zip
# /opt/oracle/oak/bin/oakcli unpack –package /u01/app/oracle/software/ODA/p27119652_122120_Linux-x86-64_3of3.zip

  • Stop databases
$ srvctl stop database -d <name>
  • Umount the NFS mounts if any
# umount /nsf
  • Verify the patch
# oakcli update -patch 12.2.1.2.0 -verify
  • Run from the patching from ILOM CLI

ssh odanode1-ilom
enter the root password when prompted
-> start /SP/console
enter y and hit return to confirm
# oakcli update -patch 12.2.1.2.0 -server -local 

Note: Node will be rebooted as part of patching

  • Ensure that CRS is Up and Running
# $GRID_HOME/bin/crsctl stat res -t
  • Perform patching post checks
# oakcli update -patch 12.2.1.2.0 -verify
  • Check the inventory looks as expected:
# oakcli show version -detail
  • Check GI patches

$ $GRID_HOME/OPatch/opatch lspatches


*****Repeat the above steps on the second node before continuing...*******




Step 2: Storage patching



Once both ODA nodes are patched successfully, we can continue with the storage patching. 
 

  • As root user on node 1 only, execute the following command:
# oakcli update -patch 12.2.1.2.0 -storage
  • Post Storage Patching, ensure that CRS is up and running on both nodes
# $GRID_HOME/bin/crsctl stat res -t
  • Verify that the patching sucessful on both nodes.
# oakcli show version -detail



Step 3: Database Patching:



Once both ODA nodes and Storage Patching completed successfully, we can now patch databases.
 

  • As root user on execute the following command:
# /opt/oracle/oak/bin/oakcli update -patch  12.2.1.2.0  --database --local
  • Verify the versions of the databsae homes

# oakcli show databases -detail 
# oakcli show dbhomes -detail


Conclusion
 

In this article we have learned to patch an Oracle Database Appliance. Oracle Database Appliance stack patching is very simple and straight forward. The ODA patching process is fully automated and single patch contains updates for all system components. The Patching is also known as One-Button patching process as it is fully automated with just few steps.



Saturday, 17 March 2018

Oracle DBCS : Create Virtual Image Database Deployment - Part 4


In previous articles Oracle DBCS: Create Virtual Image Database Deployment series we have learned how to:

Oracle DBCS : Create Virtual Image Database Deployment - Part 1
https://netsoftmate.blogspot.in/2018/02/oracle-dbcs-create-virtual-image-database-deployment.html

Oracle DBCS : Create Virtual Image Database Deployment - Part 2
https://netsoftmate.blogspot.in/2018/03/oracle-dbcs-create-virtual-image-database-deployment-part2.html

Oracle DBCS : Create Virtual Image Database Deployment - Part 3
https://netsoftmate.blogspot.in/2018/03/oracle-dbcs-create-virtual-image-database-deployment-part3.html

In this article we will learn the final step on how to Create a Database in Virtual Image Database Deployment.

Prerequisites

  • Create Virtual Image Database Deployment
  • Create Storage Volumes for Oracle Database Software and Database Files
  • Install Oracle Database Software

Step to Create a Database in Virtual Image Database Deployment:

  • Get the IP address of the Compute node you want to connect from Oracle Database Cloud Service Console. Here my Deployment Service Name is "NSM-DBaaS-VM" and the IP address is 144.21.72.128

  • On the left pane, expand "SSH" and select "Auth". On the right pane, click on "Browse" button. Select the Private Key that matches the Public Key for your Deployment. Click "Open"

  • Enter login as "opc" user. You will be connected without prompting for the password. Switch to the "root" user and "oracle user". Verify no database is running currently.

  • Login as oracle user and set Oracle Home, Oracle Base and PATH variable. Make sure Oracle Executable dbca is set in PATH correctly.

  • Start dbca in silent mode by providing the values on the command line as shown below:

  • Connect to the database and verify the status



Conclusion

In this article we have learned how to create a database in Virtual Image Database Deployment.

Oracle DBCS : Create Virtual Image Database Deployment - Part 3


In previous articles we have learned how to create Virtual Image deployment and Scale Up Storage Using Oracle Database Cloud Service and Create one storage volume for the Oracle Database software and one storage volume for all database files, and prepare them for use. 

Oracle DBCS : Create Virtual Image Database Deployment - Part 1
https://netsoftmate.blogspot.in/2018/02/oracle-dbcs-create-virtual-image-database-deployment.html

Oracle DBCS : Create Virtual Image Database Deployment - Part 2


https://netsoftmate.blogspot.in/2018/03/oracle-dbcs-create-virtual-image-database-deployment-part2.html

In this article we will learn how to Install Oracle Database Software

Prerequisites

  • Create Virtual Image Database Deployment
  • Create Storage Volumes for Oracle Database Software and Database Files

Steps to Install Oracle Database Software in Virtual Image Database Deployment.

  • Get the IP address of the Compute node you want to connect from Oracle Database Cloud Service Console. Here my Deployment Service Name is "NSM-DBaaS-VM" and the IP address is 144.21.72.128

  • On the left pane, expand "SSH" and select "Auth". On the right pane, click on "Browse" button. Select the Private Key that matches the Public Key for your Deployment. Click "Open"

  • Enter login as "opc" user. You will be connected without prompting for the password. Switch to the "root" user and "oracle user"

  • Login as oracle user, switch directory to /u01 and look for the zip file containing the Oracle Database software by displaying the contents of the /scratch/db directory. Extract the zip file into the current directory /u01

  • Ensure that the Oracle software is extracted correctly

  • Switch to root user and execute the set-up scripts as follows. Exit the session


Conclusion



In this article we have learned how to Install Oracle Database Software on a Virtual Image Database Deployment. Installing Oracle Software in VM Image deployment is very easy, you just need to locate the correct Oracle Software version and extract them in the correct directory.

Oracle DBCS : Create Virtual Image Database Deployment - Part 2

In previous article "Oracle DBCS : Create Virtual Image Database Deployment - Part 1" we have learned how to Create Virtual Image Database Deployment using Create Service Wizard.
https://netsoftmate.blogspot.in/2018/02/oracle-dbcs-create-virtual-image-database-deployment.html

In this article we will Scale Up Storage Using Oracle Database Cloud Service and Create one storage volume for the Oracle Database software and one storage volume for all database files, and prepare them for use. This will be the part 2 and continuation of the previous article.


Prerequisites
Create Virtual Image Database Deployment


Steps to Scale up Storage using Oracle Database Cloud Service Console in Virtual Image Database Deployment.


  • Open a web browser and enter the URL you received in the Welcome email to login to Oracle Cloud Account
https://myservices-xxxxx-xxxxxxxxxxef4b21bb7ee3b2cf4123d1.console.oraclecloud.com/mycloud/faces/dashboard.jspx


  • Enter your username and password


  • On the home page, Click "Menu" under "Database" Cloud Service as shown below


  • Click "Open Service Console"


  • Currently the Storage is 32GB. Let's Scale Up the Storage. Click on the Instance Name


  • Click on the "Menu" icon and Select "Scale Up/Down"

  • I am adding addition 30GB storage. Click "Yes, Scale Up/Down Service"Here I am adding addition 30GB storage. Click "Yes, Scale Up/Down Service"


  • A message printed on the screen "Service scale up/down request is accepted". The Instance status changed to "Service Maintenance"


  • After sometime we can see that the Storage is now Scale up to 62GB. Click on Instance to add more Storage


  • Click on the "Menu" icon and Select "Scale Up/Down"


  • This time I am adding addition 50GB storage. Click "Yes, Scale Up/Down Service"


  • A message printed on the screen "Service scale up/down request is accepted". The Instance status changed to "Service Maintenance"


  • After sometime we can see that the Storage is now Scale up to 112GB.


  • Get the IP address of the Compute node you want to connect from Oracle Database Cloud Service Console. Here my Deployment Service Name is "NSM-DBaaS-VM" and the IP address is 144.21.72.128


  • On the left pane, expand "SSH" and select "Auth". On the right pane, click on "Browse" button. Select the Private Key that matches the Public Key for your Deployment. Click "Open"


  • Enter login as "opc" user. You will be connected without prompting for the password. Switch to the "root" user


  • Display the list of block devices, the two volumes created are xvdc and xvdd


  • First format the volume for the Oracle Database software and mount it as /u01 as shown below




  • Now format the volume for the database files, and mount it as /u02 as shown below




  • Verify the mount points and display the block devices


  • Update the /etc/fstab file so new mount points get mounted automatically whenever the VM is rebooted



Conclusion

In this article we have learned how to Scale Up Storage using Oracle Database Cloud Service console and create storage volumes for Oracle Database Software and Database files.