LATEST TOPICS

Oracle 12c RAC: Clusterware logs are now centralized

When dealing with a Oracle Real Application Cluster (RAC) issue, a DBA generally needs to jump around multiple directories to find the relevant logs. Those days are now history as Oracle has centralized all the Clusterware (RAC) log files in a common location starting with Oracle version 12c.

Troubleshooting a RAC issue is now made easier as we can easily locate all the required log/trace files under a centralized location. We need not jump around different directories to find the Clusterware related log files. We can now free up a tiny slot of memory in our brain as it doesn’t need to remember the different log locations for a Oracle Clusterware setup, which we used to do for pre 12c Oracle RAC databases.

Further, Oracle has changed the naming conventions for the Clusterware logs starting with Oracle 12.1.0.2. All the Clusterware log files are now termed as TRACE files and ends with .trc extension (except the Clusterware alert log) rather than the .log extension which was the case in prior releases.

Starting with Oracle 12c, the Clusterware log/trace files are stored in the Automatic Diagnostic Repository (ADR) under the ADR_HOME location $ADR_BASE/diag/crs/`hostname`/crs. We can confirm that from ADRCI command line as shown below

#---//
#---// Finding ADR_HOME for Clusterware logs //---
#---//
myrac01 {/home/oracle}: adrci

ADRCI: Release 12.1.0.2.0 - Production on Wed Jan 13 04:01:00 2016

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

ADR base = "/app/oracle"
adrci> show homes
ADR Homes:
diag/tnslsnr/myrac01/mgmtlsnr
diag/rdbms/_mgmtdb/-MGMTDB
diag/crs/myrac01/crs

Let’s quickly walk through the new centralized Clusterware log location to identify the related Clusterware log/trace files.

Oracle stores all the Clusterware diagnostic log/trace files under trace directory of Clusterware ADR_HOME.

#---//
#---// Locating Clusterware centralized log repository //----
#---// 
myrac01 {/home/oracle}: ls -ld $ADR_BASE/diag/crs/`hostname`/crs/trace
drwxrwxr-x 2 oracle oinstall 876544 Jan 13 04:02 /app/oracle/diag/crs/myrac01/crs/trace

#---//
#---// moving to Clusterware centralized log repository //----
#---// 
myrac01 {/home/oracle}: cd $ADR_BASE/diag/crs/`hostname`/crs/trace

Clusterware alert log file

#---//
#---// Clusterware alert log file //----
#---//
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt alert.log
-rw-rw---- 1 oracle oinstall 1024107 Jan 13 04:04 alert.log

Cluster Ready Services (CRS) daemon/agent trace files

#---//
#---// crs daemon and agent trace files //----
#---// 
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt crs* | tail -6
-rw-rw---- 1 root   oinstall  1340371 Jan 13 04:03 crsd.trm
-rw-rw---- 1 root   oinstall  7238701 Jan 13 04:03 crsd.trc		
-rw-rw---- 1 oracle oinstall   739563 Jan 13 04:04 crsd_oraagent_oracle.trm
-rw-rw---- 1 oracle oinstall  4371314 Jan 13 04:04 crsd_oraagent_oracle.trc		
-rw-rw---- 1 root   oinstall   650733 Jan 13 04:04 crsd_orarootagent_root.trm
-rw-rw---- 1 root   oinstall  3565651 Jan 13 04:04 crsd_orarootagent_root.trc	

Cluster Synchronization Services (CSS) daemon trace files

#---//
#---// css daemon trace files //----
#---// 
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt ocssd* | tail -2
-rw-rw---- 1 oracle oinstall   443750 Jan 13 04:04 ocssd.trm
-rw-rw---- 1 oracle oinstall  2868877 Jan 13 04:04 ocssd.trc

High Availability Service (HAS) daemon/agent trace files

#---//
#---// has daemon and agent trace files //----
#---// 
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt ohasd* | tail -6
-rw-rw---- 1 root   oinstall   783540 Jan 13 04:04 ohasd_orarootagent_root.trm
-rw-rw---- 1 root   oinstall  4805435 Jan 13 04:04 ohasd_orarootagent_root.trc
-rw-rw---- 1 root   oinstall   211308 Jan 13 04:04 ohasd.trm
-rw-rw---- 1 root   oinstall  1225033 Jan 13 04:04 ohasd.trc
-rw-rw---- 1 oracle oinstall  1382492 Jan 13 04:04 ohasd_oraagent_oracle.trm
-rw-rw---- 1 oracle oinstall  6877368 Jan 13 04:04 ohasd_oraagent_oracle.trc

Event Management (EVM) daemon trace files.

#---//
#---// evm daemon trace files //----
#---// 
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt evm* | tail -2
-rw-rw---- 1 oracle oinstall  1562665 Jan 13 04:04 evmd.trm
-rw-rw---- 1 oracle oinstall 10052660 Jan 13 04:04 evmd.trc

Grid Interprocess Communication (GIPC) daemon trace files

#---//
#---// gipc daemon trace files //----
#---// 
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt gip* | tail -2
-rw-rw---- 1 oracle oinstall   763055 Jan 13 04:04 gipcd.trm
-rw-rw---- 1 oracle oinstall  5507597 Jan 13 04:04 gipcd.trc

Cluster System Monitor daemon trace files

#---//
#---// sysmon daemon trace files //----
#---// 
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt osysmon* | tail -2
-rw-rw---- 1 root oinstall  557926 Jan 13 03:46 osysmond.trm
-rw-rw---- 1 root oinstall 2867760 Jan 13 03:46 osysmond.trc

Cluster Logger Service daemon trace files

#---//
#---// logger daemon trace files //----
#---// 
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt ologgerd* | tail -2
-rw-rw---- 1 root oinstall   410752 Jan 13 03:48 ologgerd.trm
-rw-rw---- 1 root oinstall  2742627 Jan 13 03:48 ologgerd.trc

Apart from the Clusterware service (daemon) trace files, the diagnostic location also contains the client utility trace files as shown below

OCRCHECK utility trace files

#---//
#---// ocrcheck client trace files //----
#---// 
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt ocrcheck* | tail -2
-rw-rw---- 1 oracle oinstall 137 Jan 13 01:30 ocrcheck_31338.trm
-rw-rw---- 1 oracle oinstall 310 Jan 13 01:30 ocrcheck_31338.trc

OCRCONFIG utility trace files.

#---//
#---// ocrconfig client trace files //----
#---// 
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt ocrconfig* | tail -2
-rw-rw---- 1 oracle oinstall 137 Jan 13 01:30 ocrconfig_30866.trm
-rw-rw---- 1 oracle oinstall 312 Jan 13 01:30 ocrconfig_30866.trc

CRSCTL utility trace files.

#---//
#---// crsctl client trace files //----
#---// 
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt crsctl* | tail -2
-rw-rw---- 1 oracle oinstall  175 Jan 13 01:31 crsctl_1103.trm
-rw-rw---- 1 oracle oinstall  729 Jan 13 01:31 crsctl_1103.trc

Cluster Monitor utility (oclumon) trace files

#---//
#---// oclumon client trace files //----
#---// 
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt oclumon* | tail -2
-rw-rw---- 1 oracle oinstall 166 Dec 17 03:54 oclumon_2175.trm
-rw-rw---- 1 oracle oinstall 387 Dec 17 03:54 oclumon_2175.trc

Grid Plug and Play utility (GPNP) trace files

#---//
#---// gpnp client trace files //----
#---// 
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt gpn* | tail -2
-rw-rw---- 1 oracle oinstall  205 Jan 13 01:31 gpnptool_664.trm
-rw-rw---- 1 oracle oinstall  608 Jan 13 01:31 gpnptool_664.trc

As we can observe, it is now very easy to locate all the Clusterware trace/log files under a single centralized log repository. This will indeed help DBAs to save their precious time while troubleshooting Clusterware related issues.

2 Comments
  1. Fouef
    • Gary Cates
Clef two-factor authentication
%d bloggers like this:
Visit Us On LinkedinVisit Us On TwitterVisit Us On Google PlusCheck Our Feed