LATEST TOPICS

Shareplex: Fixing startup issue caused by shared memory segments

While trying to start Shareplex instance, it is failing with following errors.

##---
##--- starting Shareplex instance ---##
##---
[oracle@mylab-01 log]$ $SP_BIN/sp_cop -u$SP_COP_TPORT &
[1] 406
[oracle@mylab-01 log]$

*******************************************************
* SharePlex for Oracle Startup
* Copyright 2014 Dell, Inc.
* ALL RIGHTS RESERVED.
* Protected by U.S. Patents: 7,461,103 and 7,065,538
* Version: 8.6.2.43-m64-oracle120
* VarDir : /shareplex/pdb131_2103
* Port   : 2103
*******************************************************
can't setup shared memory statistics capability - exiting

Let’s check, if we can find anything from the event log.

##---
##--- errors logged in event_log file ---##
##---
Error    2015-12-05 23:10:18.627346 406 3377944400 Cop: Error cleaning up previous shared memory segment 134381585  [module shs]
Error    2015-12-05 23:10:18.627468 406 3377944400 Cop: Cannot delete because there are users attached  [module shs]
Notice   2015-12-05 23:10:18.627571 406 3377944400 Cop: Check if SharePlex processes are running and kill them if necessary  [module shs]
Notice   2015-12-05 23:10:18.627592 406 3377944400 Cop: Remove shared-memory segments, owned by the SharePlex user, with a key value ending in '8cbf'  [module shs]
Notice   2015-12-05 23:10:18.627616 406 3377944400 Cop: Remove semaphores, owned by the SharePlex user, with key values of 0x00008cbf or 0x00108cbf  [module shs]
Error    2015-12-05 23:10:18.627632 406 3377944400 Cop: can't setup shared memory statistics capability - exiting  [module cop]

As per the event_log, Shareplex is not able to clean up the shared memory segments and looks like there are some stale processes attached to those shared memory segments. It is asking us to check if there are any Shareplex process running against the same port and kill those processes if required. Further, it is asking us to remove specific shared memory segments (key: 8cbf) and semaphores (Keys: 0x00008cbf and 0x00108cbf) owned by the Shareplex user.

Let’s first check, if there are any stale Shareplex processes running on the system.

##---
##--- Shareplex processes running on the system ---##
##---
[oracle@mylab-01 log]$ ps -ef | grep sp_
oracle   12143     1  0 16:25 pts/2    00:00:00 /shareplex/splex_8.6.2_12/.app-modules/sp_xport -u2103
oracle   12144     1  0 16:25 pts/2    00:00:00 /shareplex/splex_8.6.2_12/.app-modules/sp_opst_mt -u2103
oracle   12145     1  0 16:25 pts/2    00:00:00 /shareplex/splex_8.6.2_12/.app-modules/sp_ordr -u2103
oracle   12252     1  0 16:29 pts/2    00:00:00 /shareplex/splex_8.6.2_12/.app-modules/sp_mport 0xc0a8e60c+PI+mylab-02+sp_mport+0xc0a8e60d R -u2103
oracle   12256     1  0 16:29 pts/2    00:00:03 /shareplex/splex_8.6.2_12/.app-modules/sp_ocap -u2103
oracle   14152 12087  0 16:40 pts/2    00:00:00 grep sp_

As we can see there are Shareplex processes running on this system. However, all of these processes are in stale state as the associated parent process (sp_cop) is not running. There is a possibility that Shareplex was terminated abruptly which left the child processes in a stale state. These stale processes are now preventing Shareplex to startup the instance against the same port. We need need to terminate these stale processes to be able to startup the Shareplex instance on the same port.

Let’s terminate the stale Shareplex processes.

##---
##--- Terminating stale Shareplex processes ---##
##---
[oracle@mylab-01 log]$ kill -9 12143 12144 12145 12252 12256

##---
##--- Validating no stale processes exist ---##
##---
[oracle@mylab-01 log]$ ps -ef | grep sp_
oracle   14252 12087  0 16:41 pts/2    00:00:00 grep sp_

In most cases terminating the stale processes will also release the shared memory segments and semaphores associated with the stale processes. However, if the shared memory segments and semaphores are not released automatically, we need to do that manually.

We can use the ipcs command to identify the shared memory segments and semaphores associated with the stale processes. To find the shared memory segments and semaphores, we will match the memory segments and semaphores with the key found in the event_log. In our case the key is 8cbf for shared memory segments and 0x00008cbf, 0x00108cbf for semaphores.

Let’s check the status of shared memory segments and semaphores to find out if there are any segments/semaphores matching the given keys.

##---
##--- checking shared memory segments and semaphores ---##
##---
[oracle@mylab-01 log]$ ipcs -smaa

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x00000000 3538944    root      644        80         2
0x00000000 3571713    root      644        16384      2
0x00000000 3604482    root      644        280        2
0x00000000 131694605  oracle    640        2932736    94
0x00000000 131727374  oracle    640        796917760  47
0x00000000 131760143  oracle    640        5455872    47
0x8abc60fc 131792912  oracle    640        16384      47
0x00008cbf 134381585  oracle    644        33554432   3		#--> matching segment
0x00108cbf 134414354  oracle    644        139264     3		#--> matching segment
0x00308cbf 134447123  oracle    644        8388608    2		#--> matching segment
0x00408cbf 134479892  oracle    644        33554432   1		#--> matching segment
0x00508cbf 134512661  oracle    644        8388608    0		#--> matching segment

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0xc7a6f808 2162689    oracle    640        126
0xc7a6f809 2195458    oracle    640        126
0xc7a6f80a 2228227    oracle    640        126
0xc7a6f80b 2260996    oracle    640        126
0xc7a6f80c 2293765    oracle    640        126
0x00008cbf 2326534    oracle    666        12	#--> matching semaphore
0x00108cbf 2359303    oracle    666        26	#--> matching semaphore

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

As we can see there are shared memory segments (shmid: 134381585,134414354,134447123,134479892,134512661) matching with the given key which needs to be released. Additionally, we can also find the semaphores (semid: 2326534,2359303) matching with the given key which also needs to be released.

Let’s release the identified shared memory segments and semaphores associated with the stale processes. We can use the command ipcrm with a flag -m shmid to release shared memory segments and with a flag -s semid to release the semaphores as shown below.

##---
##--- releasing shared memory segments with 'ipcrm -m' ---##
##---
[oracle@mylab-01 log]$ ipcrm -m 134381585
[oracle@mylab-01 log]$ ipcrm -m 134414354
[oracle@mylab-01 log]$ ipcrm -m 134447123
[oracle@mylab-01 log]$ ipcrm -m 134479892
[oracle@mylab-01 log]$ ipcrm -m 134512661

##---
##--- releasing semaphores with 'ipcrm -s' ---##
##---
[oracle@mylab-01 log]$ ipcrm -s 2326534
[oracle@mylab-01 log]$ ipcrm -s 2359303

Lets validate if the memory segments and semaphores are released.

##---
##--- checking shared memory segments and semaphores ---##
##---
[oracle@mylab-01 log]$ ipcs -smaa

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x00000000 3538944    root      644        80         2
0x00000000 3571713    root      644        16384      2
0x00000000 3604482    root      644        280        2
0x00000000 131694605  oracle    640        2932736    88
0x00000000 131727374  oracle    640        796917760  44
0x00000000 131760143  oracle    640        5455872    44
0x8abc60fc 131792912  oracle    640        16384      44
0x00000000 134381585  oracle    644        33554432   1          dest
0x00000000 134414354  oracle    644        139264     1          dest
0x00000000 134447123  oracle    644        8388608    1          dest

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0xc7a6f808 2162689    oracle    640        126
0xc7a6f809 2195458    oracle    640        126
0xc7a6f80a 2228227    oracle    640        126
0xc7a6f80b 2260996    oracle    640        126
0xc7a6f80c 2293765    oracle    640        126

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

As we can observe, there are no more memory segments or semaphores matching with the key found in the event_log, which means those segments and semaphores are now released. Now, we should be able to start up the Shareplex instance without any issues.

##---
##--- starting up Shareplex instance ---##
##---
[oracle@mylab-01 log]$ $SP_BIN/sp_cop -u$SP_COP_TPORT &
[1] 1399
[oracle@mylab-01 log]$

*******************************************************
* SharePlex for Oracle Startup
* Copyright 2014 Dell, Inc.
* ALL RIGHTS RESERVED.
* Protected by U.S. Patents: 7,461,103 and 7,065,538
* Version: 8.6.2.43-m64-oracle120
* VarDir : /shareplex/pdb131_2103
* Port   : 2103
*******************************************************


##---
##--- validate Shareplex is up and running fine ---##
##---
[oracle@mylab-01 log]$ $SP_BIN/sp_ctrl


*******************************************************
* SharePlex for Oracle Command Utility
* Copyright 2014 Dell, Inc.
* ALL RIGHTS RESERVED.
* Protected by U.S. Patents: 7,461,103 and 7,065,538
*******************************************************

sp_ctrl (mylab-01:2103)> show

Process    Source                               Target                 State                   PID
---------- ------------------------------------ ---------------------- -------------------- ------
Capture    o.orppdb13_1                                                Running                1400
Read       o.orppdb13_1                                                Running                1403
Import     mylab-02                             mylab-01               Running                1405
Post       o.mypdb_01-mylab-02                  o.orppdb13_1           Running                1402
Export     mylab-01                             mylab-02               Running                1404

As we can see, we are now able to start up the Shareplex instance without any issue and all the Shareplex processes are up and running fine.

No Responses
%d bloggers like this:
Visit Us On LinkedinVisit Us On TwitterVisit Us On Google PlusCheck Our Feed