LATEST TOPICS

ORA-16535: Oracle Restart or Oracle Clusterware prevented completion of broker operation

 

Recently, while doing Oracle database switchover through DG broker, I have encountered with the following errors.

Just as note, I was deaing with Oracle Database Version: 11.2.0.3

 

Performing switchover NOW, please wait... 

Error: ORA-16535: Oracle Restart or Oracle Clusterware prevented completion of broker operation

Since, I knew and crosschecked the broker configuration was working fine prior to the switchover, I did not even bother to review the broker configuration.

As a initial step to the troubleshooting, looked into the broker logs and noticed the following message

08/27/2014 22:01:31
Data Guard unable to notify Oracle Clusterware
Canceling SWITCHOVER
Command SWITCHOVER TO [db_name] completed with error ORA-16535

 

As the error was clearly suggesting, something went wrong with the Clusterware rather than the broker;

16535, 00000, "Oracle Restart or Oracle Clusterware prevented completion of broker operation"
// *Cause: The broker could not complete the operation because Oracle Restart
// (for single instance databases) or Oracle Clusterware (for RAC
// databases) could not stop services or instances on this database.
// *Action: Check the Data Guard broker log file for more information and
// stop the appropriate services or instances using SRVCTL. Then
// retry the broker operation.

 

I started to review the Clusterware logs.

In crsd.log , following entries were found

2014-08-27 22:01:30.220: [ CRSPE][3703461632]{1:43442:45252} CRS-2673: Attempting to stop 'ora.[db_name].[service_name].svc' on '[cluster_node_name]'

2014-08-27 22:01:30.220: [ AGFW][3713967872]{1:43442:45252} Agfw Proxy Server forwarding the message: RESOURCE_STOP[ora.[db_name].[service_name].svc 3 1] ID 4099:311849 to the agent /opt/grid/11.2.0.4.0/bin/oraagent_oracle
2014-08-27 22:01:30.220: [UiServer][3701360384]{1:43442:45252} Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2673: Attempting to stop 'ora.[db_name].[service_name].svc' on '[cluster_node_name]']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.[db_name].[service_name].svc 3 1]
WAIT:
TextMessage[0]
]
2014-08-27 22:01:30.503: [ AGFW][3713967872]{1:43442:45252} Received the reply to the message: RESOURCE_STOP[ora.[db_name].[service_name].svc 3 1] ID 4099:311850 from the agent /opt/grid/11.2.0.4.0/bin/oraagent_oracle
2014-08-27 22:01:30.504: [ AGFW][3713967872]{1:43442:45252} Agfw Proxy Server sending the reply to PE for message:RESOURCE_STOP[ora.[db_name].[service_name].svc 3 1] ID 4099:311849
2014-08-27 22:01:30.504: [ CRSPE][3703461632]{1:43442:45252} Received reply to action [Stop] message ID: 311849
2014-08-27 22:01:30.504: [ AGFW][3713967872]{1:43442:45252} Received the reply to the message: RESOURCE_STOP[ora.[db_name].[service_name].svc 3 1] ID 4099:311850 from the agent /opt/grid/11.2.0.4.0/bin/oraagent_oracle
 2014-08-27 22:01:30.504: [ CRSPE][3703461632]{1:43442:45252} Got agent-specific msg: CRS-5017: The resource action "ora.[db_name].[service_name].svc stop" encountered the following error:
 ORA-06550: line 1, column 33:
 PLS-00553: character set name is not recognized
 ORA-06550: line 0, column 0:
 PL/SQL: Compilation unit analysis terminated
 . For details refer to "(:CLSN00108:)" in "/opt/grid/11.2.0.4.0/log/[cluster_node_name]/agent/crsd/oraagent_oracle/oraagent_oracle.log".
2014-08-27 22:01:30.505: [UiServer][3701360384]{1:43442:45252} Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-5017: The resource action "ora.[db_name].[service_name].svc stop" encountered the following error:
ORA-06550: line 1, column 33:
PLS-00553: character set name is not recognized
ORA-06550: line 0, column 0:
PL/SQL: Compilation unit analysis terminated
. For details refer to "(:CLSN00108:)" in "/opt/grid/11.2.0.4.0/log/[cluster_node_name]/agent/crsd/oraagent_oracle/oraagent_oracle.log".
]
MSGTYPE:
TextMessage[1]
OBJID:
TextMessage[ora.[db_name].[service_name].svc 3 1]
WAIT:
TextMessage[0]
]

2014-08-27 22:01:30.505: [ AGFW][3713967872]{1:43442:45252} Received the reply to the message: RESOURCE_STOP[ora.[db_name].[service_name].svc 3 1] ID 4099:311850 from the agent /opt/grid/11.2.0.4.0/bin/oraagent_oracle
2014-08-27 22:01:30.506: [ AGFW][3713967872]{1:43442:45252} Agfw Proxy Server sending the last reply to PE for message:RESOURCE_STOP[ora.[db_name].[service_name].svc 3 1] ID 4099:311849
2014-08-27 22:01:30.508: [ CRSPE][3703461632]{1:43442:45252} Received reply to action [Stop] message ID: 311849
2014-08-27 22:01:30.508: [ CRSPE][3703461632]{1:43442:45252} RI [ora.[db_name].[service_name].svc 3 1] new internal state: [STABLE] old value: [STOPPING]
2014-08-27 22:01:30.508: [ CRSPE][3703461632]{1:43442:45252} CRS-2675: Stop of 'ora.[db_name].[service_name].svc' on '[cluster_node_name]' failed

2014-08-27 22:01:30.509: [ CRSRPT][3701360384]{1:43442:45252} Published to EVM CRS_ACTION_FAILURE for ora.[db_name].[service_name].svc
2014-08-27 22:01:30.509: [UiServer][3701360384]{1:43442:45252} Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2675: Stop of 'ora.[db_name].[service_name].svc' on '[cluster_node_name]' failed]
MSGTYPE:
TextMessage[1]
OBJID:
TextMessage[ora.[db_name].[service_name].svc 3 1]
WAIT:
TextMessage[0]
]
2014-08-27 22:01:30.509: [ CRSPE][3703461632]{1:43442:45252} Sequencer for [ora.[db_name].[service_name].svc 3 1] has completed with error: CRS-0216: Could not stop resource 'ora.[db_name].[service_name].svc'.

2014-08-27 22:01:30.514: [ CRSPE][3703461632]{1:43442:45252} Received reply to action [Stop] message ID: 311847
2014-08-27 22:01:30.514: [ CRSPE][3703461632]{1:43442:45252} Got agent-specific msg: CRS-5017: The resource action "ora.[db_name].[service_name].svc stop" encountered the following error:
ORA-06550: line 1, column 33:
PLS-00553: character set name is not recognized
ORA-06550: line 0, column 0:
PL/SQL: Compilation unit analysis terminated
. For details refer to "(:CLSN00108:)" in "/opt/grid/11.2.0.4.0/log/[cluster_node_name]/agent/crsd/oraagent_oracle/oraagent_oracle.log".

By going through the logs, it was clear that CRS was not able to stop the database service as part of the database switchover process and was inturn resulting in the failure of broker operation.

As a workaround, I had stopped the database service ora.[db_name].[service_name].svc manually

srvctl stop service -d [db_name] -s [service_name]

Re-issued switchover with DG broker and everything went fine this time.

 

NOTE: This issues seems to be due to the Oracle Bug 14748254 ( Doc ID 1681525.1 ) and has an alternative fix to this issue.

 

 

%d bloggers like this:
Visit Us On LinkedinVisit Us On TwitterVisit Us On Google PlusCheck Our Feed