查看: 8620|回复: 20
求助 oracle Rac 节点2起不来！！！！！
论坛徽章:0
节点2起不来，VIP已经漂移到节点1 ，请大家帮忙看看，如何解决！非常感谢！
IP配置如下：
172.16.1.2& &&&tradedb1
172.16.1.4& &&&tradedb1-vip
172.16.3.2& &&&tradedb1-priv
172.16.1.3& &&&tradedb2
172.16.1.5& &&&tradedb2-vip
172.16.3.3& &&&tradedb2-priv
172.16.1.8& &&&tradedb-cluster-scan
论坛徽章:0
alerttradedb2.log 如下：
16:19:36.820
[mdnsd(11416)]CRS-5602:mDNS service stopping by request.
16:19:39.516
[gpnpd(11426)]CRS-2329:GPNPD on node tradedb2 shutdown.
[client(49903)]CRS-10001:02-Sep-15 16:19 ACFS-9459: ADVM/ACFS is not supported on this OS version: 'centos-release-6-5.el6.centos.11.1.x86_64
[client(49905)]CRS-10001:02-Sep-15 16:19 ACFS-9201: Not Supported
16:21:08.090
[client(50763)]CRS-2101:The OLR was formatted using version 3.
16:21:42.131
[ohasd(51095)]CRS-2112:The OLR service started on node tradedb2.
16:21:42.143
[ohasd(51095)]CRS-1301:Oracle High Availability Service started on node tradedb2.
[client(51322)]CRS-10001:02-Sep-15 16:21 ACFS-9459: ADVM/ACFS is not supported on this OS version: 'centos-release-6-5.el6.centos.11.1.x86_64
[client(51324)]CRS-10001:02-Sep-15 16:21 ACFS-9201: Not Supported
[client(51478)]CRS-10001:02-Sep-15 16:21 ACFS-9459: ADVM/ACFS is not supported on this OS version: 'centos-release-6-5.el6.centos.11.1.x86_64
16:22:00.358
[gpnpd(51650)]CRS-2328:GPNPD started on node tradedb2.
16:22:02.621
[cssd(51707)]CRS-1713:CSSD daemon is started in exclusive mode
16:22:04.492
[ohasd(51095)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
16:22:12.324
[cssd(51707)]CRS-1707:Lease acquisition for node tradedb2 number 2 completed
16:22:13.613
[cssd(51707)]CRS-1605:CSSD voting file is online: /dev/asm-ocr01; details in /oracle/product/grid/11.2.0/log/tradedb2/cssd/ocssd.log.
[cssd(51707)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node tradedb1 details at (:CSSNM00006 in /oracle/product/grid/11.2.0/log/tradedb2/cssd/ocssd.log
16:22:15.684
[ohasd(51095)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'tradedb2'.
16:22:27.283
[gpnpd(51650)]CRS-2329:GPNPD on node tradedb2 shutdown.
16:22:29.630
[mdnsd(51640)]CRS-5602:mDNS service stopping by request.
16:22:49.286
[gpnpd(51877)]CRS-2328:GPNPD started on node tradedb2.
16:22:51.957
[cssd(51939)]CRS-1713:CSSD daemon is started in clustered mode
16:22:53.835
[ohasd(51095)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
16:23:01.629
[cssd(51939)]CRS-1707:Lease acquisition for node tradedb2 number 2 completed
16:23:02.906
[cssd(51939)]CRS-1605:CSSD voting file is online: /dev/asm-ocr01; details in /oracle/product/grid/11.2.0/log/tradedb2/cssd/ocssd.log.
16:32:51.017
[/oracle/product/grid/11.2.0/bin/cssdagent(51915)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113 {0:0:148} in /oracle/product/grid/11.2.0/log/tradedb2/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
16:32:51.018
[cssd(51939)]CRS-1656:The CSS daemon is terminating
Details at (:CSSSC00012 in /oracle/product/grid/11.2.0/log/tradedb2/cssd/ocssd.log
16:32:51.018
[cssd(51939)]CRS-1603:CSSD on node tradedb2 shutdown by user.
16:32:56.182
[ohasd(51095)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'tradedb2'.
论坛徽章:55
这个日志看看
/oracle/product/grid/11.2.0/log/tradedb2/cssd/ocssd.log
论坛徽章:0
16:22:02.620: [& & CSSD][]clsu_load_ENV_levels: Module = CSSD, LogLevel = 2, TraceLevel = 0
16:22:02.620: [& & CSSD][]clsu_load_ENV_levels: Module = GIPCNM, LogLevel = 2, TraceLevel = 0
16:22:02.621: [& & CSSD][]clsu_load_ENV_levels: Module = GIPCGM, LogLevel = 2, TraceLevel = 0
16:22:02.621: [& & CSSD][]clsu_load_ENV_levels: Module = GIPCCM, LogLevel = 2, TraceLevel = 0
16:22:02.621: [& & CSSD][]clsu_load_ENV_levels: Module = CLSF, LogLevel = 0, TraceLevel = 0
16:22:02.621: [& & CSSD][]clsu_load_ENV_levels: Module = SKGFD, LogLevel = 0, TraceLevel = 0
16:22:02.621: [& & CSSD][]clsu_load_ENV_levels: Module = GPNP, LogLevel = 1, TraceLevel = 0
16:22:02.621: [& & CSSD][]clsu_load_ENV_levels: Module = OLR, LogLevel = 0, TraceLevel = 0
[& & CSSD][]clsugetconf : Configuration type [4].
16:22:02.621: [& & CSSD][]clssscmain: Starting CSS daemon, version 11.2.0.3.0, in (exclusive) mode with uniqueness value
16:22:02.622: [& & CSSD][]clssscmain: Environment is production
16:22:02.622: [& & CSSD][]clssscmain: Core file size limit extended
16:22:02.626: [& & CSSD][]clssscmain: GIPCHA down 0
16:22:02.628: [& & CSSD][]clssscGetParameterOLR: OLR fetch for parameter logsize (8) failed with rc 21
16:22:02.628: [& & CSSD][]clssscExtendLimits: The current soft limit for file descriptors is 65536, hard limit is 65536
16:22:02.628: [& & CSSD][]clssscExtendLimits: The current soft limit for locked memory is , hard limit is
16:22:02.629: [& & CSSD][]clssscGetParameterOLR: OLR fetch for parameter priority (15) failed with rc 21
16:22:02.629: [& & CSSD][]clssscSetPrivEnv: Setting priority to 4
16:22:02.641: [& & CSSD][]clssscSetPrivEnv: Can't access local IPMI device--no device configured or driver missing/incompatible. IPMI support may be available with static IP configuration.
16:22:02.641: [& & CSSD][]clssscmain: Running as user grid
16:22:02.643: [& & CSSD][]clssscmain: RT queue setting is at default value
16:22:02.644: [& & CSSD][]clssscGetParameterOLR: OLR fetch for parameter auth rep (9) failed with rc 21
16:22:02.644: [& & CSSD][]clssgmInitCMInfoMin: clsmonJoined set via eXclusive mode
16:22:02.645: [& & CSSD][]clssscGetParameterOLR: OLR fetch for parameter diagwait (14) failed with rc 21
[&&clsdmt][]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=tradedb2DBG_CSSD))
16:22:02.650: [&&clsdmt][]PID for the Process [51707], connkey 4
16:22:02.650: [& & CSSD][]clssnmInitNMInfoMin: Initializing first-reconfig to (0)
16:22:02.650: [& & CSSD][]clssscmain: initgminfo done
16:22:02.652: [& & CSSD][]clssgmclientlsnr: Spawned
16:22:02.652: [& & CSSD][]clssgmEvtInformation: reqtype (13) cmProc ((nil)) client ((nil))
16:22:02.652: [& & CSSD][]clssgmEvtInformation: reqtype (13) req (0x7f)
16:22:02.652: [& & CSSD][]clssnmQueueNotification: type (13) 0x7f
16:22:02.653: [& & CSSD][]clssgmclientlsnr: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_tradedb2_)(GIPCID=))
16:22:02.653: [& & GPNP][]clsgpnp_Init: [at clsgpnp0.c:585] '/oracle/product/grid/11.2.0' in effect as GPnP home base.
16:22:02.653: [& & GPNP][]clsgpnp_Init: [at clsgpnp0.c:619] GPnP pid=51707, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=3
16:22:02.662: [& & GPNP][]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399] Using FS Wallet Location : /oracle/product/grid/11.2.0/gpnp/tradedb2/wallets/peer/
[& &CLWAL][]clsw_Initialize: OLR initlevel [70000]
16:22:02.692: [& & GPNP][]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url &ipc://GPNPD_tradedb2& disco && [f=0 claimed- host: cname: seq: auth:]
16:22:02.702: [& & GPNP][]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri &http://www.grid-pnp.org/2005/12/gpnp-errors#&]
16:22:03.562: [& & CSSD][]clssscSelect: cookie accept request 0x15f7650
16:22:03.562: [& & CSSD][]clssgmAllocProc: (0x7f) allocated
16:22:03.563: [& & CSSD][]clssgmClientConnectMsg: properties of cmProc 0x7f - 1,2,3,4,5
16:22:03.563: [& & CSSD][]clssgmClientConnectMsg: Connect from con(0x13d) proc(0x7f) pid(51662) version 11:2:1:4, properties: 1,2,3,4,5
16:22:03.563: [& & CSSD][]clssgmClientConnectMsg: The CSSD monitor is process (0x7f), number 1
16:22:03.563: [& & CSSD][]clssgmEvtInformation: reqtype (11) cmProc (0x7f) client ((nil))
16:22:03.563: [& & CSSD][]clssgmEvtInformation: reqtype (11) req (0x7f)
16:22:03.563: [& & CSSD][]clssnmQueueNotification: type (11) 0x7f
16:22:03.599: [& & CSSD][]clssscSelect: cookie accept request 0x15f7650
16:22:03.599: [& & CSSD][]clssgmAllocProc: (0x7f) allocated
16:22:03.599: [& & CSSD][]clssgmClientConnectMsg: properties of cmProc 0x7f - 1,2,3,4,5
16:22:03.599: [& & CSSD][]clssgmClientConnectMsg: Connect from con(0x19a) proc(0x7f) pid(51682) version 11:2:1:4, properties: 1,2,3,4,5
16:22:03.599: [& & CSSD][]clssgmClientConnectMsg: The CSSD agent is process (0x7f), number 2
16:22:03.600: [& & CSSD][]clssgmEvtInformation: reqtype (11) cmProc (0x7f) client ((nil))
16:22:03.600: [& & CSSD][]clssgmEvtInformation: reqtype (11) req (0x7f)
16:22:03.600: [& & CSSD][]clssnmQueueNotification: type (11) 0x7f
16:22:04.710: [& & GPNP][]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri &http://www.grid-pnp.org/2005/12/gpnp-errors#&]
16:22:06.720: [& & GPNP][]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote &ipc://GPNPD_tradedb2& disco &&
16:22:06.721: [& & CSSD][]clssscGetParameterProfile: profile fetch failed for parameter ocrid (4) with return code 5
16:22:06.721: [& & CSSD][]clssscmain: OCRID is 0
16:22:06.721: [& & CSSD][]clssscmain: Cluster GUID is f4cf11bfba
16:22:06.721: [& & CSSD][]clssnmNotifyReq: type (12)
16:22:06.722: [& & CSSD][]clssscGetParameterOLR: OLR fetch for parameter node number hint (6) failed with rc 21
16:22:06.722: [& & CSSD][]clssnmReadDiscoveryProfile: voting file discovery string(/dev/asm*)
16:22:06.722: [& & CSSD][]clssnkInit: NK generic layer initializing.
16:22:06.726: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:06.726: [& & CLSF][]Allocated CLSF context
16:22:06.726: [& & CSSD][]clssnmvDDiscThread: using discovery string /dev/asm* for initial discovery
16:22:06.726: [& &SKGFD][]Discovery with str:/dev/asm*:
16:22:06.726: [& &SKGFD][]UFS discovery with :/dev/asm*:
16:22:06.749: [& &SKGFD][]Fetching UFS disk :/dev/asm-disk01:
16:22:06.749: [& &SKGFD][]Fetching UFS disk :/dev/asm-vote01:
16:22:06.749: [& &SKGFD][]Fetching UFS disk :/dev/asm-ocr01:
16:22:06.749: [& &SKGFD][]Fetching UFS disk :/dev/asm-disk02:
16:22:06.749: [& &SKGFD][]OSS discovery with :/dev/asm*:
16:22:06.750: [& &SKGFD][]Handle 0x7fd0 from lib :UFS:: for disk :/dev/asm-disk01:
16:22:06.750: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-vote01:
16:22:06.750: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:06.750: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-disk02:
16:22:06.750: [& &SKGFD][]Lib :UFS:: closing handle 0x7fd0 for disk :/dev/asm-disk01:
16:22:06.750: [& &SKGFD][]Lib :UFS:: closing handle 0x7f for disk :/dev/asm-vote01:
16:22:06.750: [& &SKGFD][]Lib :UFS:: closing handle 0x7f for disk :/dev/asm-ocr01:
16:22:06.751: [& &SKGFD][]Lib :UFS:: closing handle 0x7f for disk :/dev/asm-disk02:
16:22:06.751: [& & CSSD][]clssnmvDiskVerify: discovered a potential voting file
16:22:06.751: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:06.752: [& & CSSD][]clssnmvDiskVerify: Successful discovery for disk /dev/asm-ocr01, UID 4ce283ae-a5154ffa-bf056a5d-8a19acfe, Pending CIN 0::0, Committed CIN 0::0
16:22:06.752: [& &SKGFD][]Lib :UFS:: closing handle 0x7f for disk :/dev/asm-ocr01:
16:22:06.752: [& & CSSD][]clssnmvDiskVerify: Successful discovery of 1 disks
16:22:06.752: [& & CSSD][]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
16:22:06.752: [& & CSSD][]clssnmCompleteVFDiscovery: Completing voting file discovery
16:22:06.752: [& & CSSD][]clssnmvDiskStateChange: state from discovered to pending disk /dev/asm-ocr01
16:22:06.752: [& & CSSD][]clssnmvDiskStateChange: state from pending to configured disk /dev/asm-ocr01
16:22:06.752: [& & CSSD][]clssnmCompleteVFDiscovery: Committed configuration for CIN 0::0
16:22:06.752: [& & CSSD][]&&misscount& && && & 30& & reboot latency& && &3
16:22:06.753: [& & CSSD][]&&long I/O timeout&&200& & short I/O timeout&&27
16:22:06.753: [& & CSSD][]&&diagnostic wait& &&&0&&active version 11.2.0.3.0
16:22:06.753: [& & CSSD][]&&Listing unique IDs for 1 voting files:
16:22:06.753: [& & CSSD][]& & voting file 1: 4ce283ae-a5154ffa-bf056a5d-8a19acfe
16:22:06.753: [& & CSSD][]clssnmOpenGIPCEndp: opening cluster listener on gipc://tradedb2:nm_tradedb-cluster
16:22:06.753: [GIPCHGEN][] gipchaInternalRegister: Initializing HA GIPC
16:22:06.753: [GIPCHGEN][] gipchaNodeCreate: adding new node 0x181acb0 { host '', haName 'CSS_tradedb-cluster', srcLuid 07b7d79d-, dstLuid 00000 numInf 0, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [0 : 0], createTime , sentRegister 0, localMonitor 0, flags 0x1 }
16:22:06.753: [& & GPNP][]clsgpnp_Init: [at clsgpnp0.c:585] '/oracle/product/grid/11.2.0' in effect as GPnP home base.
16:22:06.753: [& & GPNP][]clsgpnp_Init: [at clsgpnp0.c:619] GPnP pid=51707, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=0
16:22:06.758: [& & GIPC][] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 769], original from [clsssc.c : 970]
16:22:06.759: [& & GPNP][]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399] Using FS Wallet Location : /oracle/product/grid/11.2.0/gpnp/tradedb2/wallets/peer/
[& &CLWAL][]clsw_Initialize: OLR initlevel [30000]
16:22:06.776: [& & GPNP][]clsgpnp_getCachedProfileEx: [at clsgpnp.c:613] Result: (26) CLSGPNP_NO_PROFILE. Can't get offline GPnP service profile: local gpnpd is up and running. Use getProfile instead.
16:22:06.776: [& & GPNP][]clsgpnp_getCachedProfileEx: [at clsgpnp.c:623] Result: (26) CLSGPNP_NO_PROFILE. Failed to get offline GPnP service profile.
16:22:06.776: [& & GPNP][]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url &ipc://GPNPD_tradedb2& disco && [f=1 claimed- host: cname: seq: auth:]
16:22:06.787: [& & GPNP][]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote &ipc://GPNPD_tradedb2& disco &&
16:22:06.787: [GIPCXCPT][] gipchaInternalReadGpnp: No network info configured in GPNP, using defaults, ret gipcretFail (1)
16:22:06.989: [GIPCHGEN][] gipchaInternalReadGpnp: configuring default multicast addresses
16:22:06.989: [GIPCHGEN][] gipchaInternalReadGpnp: configuring default bootstrap communications modes
16:22:06.989: [GIPCHGEN][] gipchaInternalReadGpnp: configuring bootstrap communications using:&&broadcast and multicast
16:22:06.989: [GIPCHGEN][] gipchaInternalReadGpnp: mcast address[&&0 ]&&224.0.0.251
16:22:06.989: [GIPCHGEN][] gipchaInternalReadGpnp: mcast address[&&1 ]&&230.0.1.0
16:22:06.990: [GIPCHTHR][] gipchaWorkerThread: starting worker thread hctx 0x15a] { gipchaContext : host 'tradedb2', name 'CSS_tradedb-cluster', luid '07b7d79d-', numNode 0, numInf 0, usrFlags 0x0, flags 0x62 }
16:22:06.993: [GIPCHDEM][] gipchaDaemonThread: starting daemon thread hctx 0x15a] { gipchaContext : host 'tradedb2', name 'CSS_tradedb-cluster', luid '07b7d79d-', numNode 0, numInf 0, usrFlags 0x0, flags 0x62 }
16:22:06.993: [& & GPNP][]clsgpnp_Init: [at clsgpnp0.c:585] '/oracle/product/grid/11.2.0' in effect as GPnP home base.
16:22:06.993: [& & GPNP][]clsgpnp_Init: [at clsgpnp0.c:619] GPnP pid=51707, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=0
16:22:07.001: [& & GIPC][] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 769], original from [clsssc.c : 970]
16:22:07.002: [& & GPNP][]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399] Using FS Wallet Location : /oracle/product/grid/11.2.0/gpnp/tradedb2/wallets/peer/
[& &CLWAL][]clsw_Initialize: OLR initlevel [30000]
16:22:07.029: [& & GPNP][]clsgpnp_getCachedProfileEx: [at clsgpnp.c:613] Result: (26) CLSGPNP_NO_PROFILE. Can't get offline GPnP service profile: local gpnpd is up and running. Use getProfile instead.
16:22:07.029: [& & GPNP][]clsgpnp_getCachedProfileEx: [at clsgpnp.c:623] Result: (26) CLSGPNP_NO_PROFILE. Failed to get offline GPnP service profile.
16:22:07.029: [& & GPNP][]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url &ipc://GPNPD_tradedb2& disco && [f=1 claimed- host: cname: seq: auth:]
16:22:07.040: [& & GPNP][]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote &ipc://GPNPD_tradedb2& disco &&
16:22:07.040: [ GIPCLIB][] gipclibGetClusterGuid: retrieved cluster guid f4cf11bfba
16:22:07.243: [& & GPNP][] clsgpnp_Init: [at clsgpnp0.c:585] '/oracle/product/grid/11.2.0' in effect as GPnP home base.
16:22:07.243: [& & GPNP][] clsgpnp_Init: [at clsgpnp0.c:619] GPnP pid=51707, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=2003
16:22:07.251: [& & GPNP][] clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399] Using FS Wallet Location : /oracle/product/grid/11.2.0/gpnp/tradedb2/wallets/peer/
[& &CLWAL][]clsw_Initialize: OLR initlevel [70000]
16:22:07.279: [& & GPNP][] clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url &ipc://GPNPD_tradedb2& disco && [f=3 claimed- host: cname: seq: auth:]
16:22:07.291: [& & GPNP][] clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote &ipc://GPNPD_tradedb2& disco &&
16:22:07.291: [ CLSINET][] Returning NETDATA: 1 interfaces
16:22:07.291: [ CLSINET][] # 0 Interface 'eth1',ip='172.16.3.3',mac='40-a8-f0-1e-88-8d',mask='255.255.255.0',net='172.16.3.0',use='cluster_interconnect'
16:22:07.291: [GIPCHGEN][] gipchaNodeAddInterface: adding interface information for inf 0x7f { host '', haName 'CSS_tradedb-cluster', local (nil), ip '172.16.3.3', subnet '172.16.3.0', mask '255.255.255.0', mac '40-a8-f0-1e-88-8d', ifname 'eth1', numRef 0, numFail 0, idxBoot 0, flags 0x1841 }
16:22:07.292: [GIPCHTHR][] gipchaWorkerCreateInterface: created local interface for node 'tradedb2', haName 'CSS_tradedb-cluster', inf 'udp://172.16.3.3:60277'
16:22:07.292: [GIPCHTHR][] gipchaWorkerCreateInterface: created local bootstrap multicast interface for node 'tradedb2', haName 'CSS_tradedb-cluster', inf 'mcast://224.0.0.251:.3.3'
16:22:07.294: [GIPCHTHR][] gipchaWorkerCreateInterface: created local bootstrap multicast interface for node 'tradedb2', haName 'CSS_tradedb-cluster', inf 'mcast://230.0.1.0:.3.3'
16:22:07.295: [GIPCHTHR][] gipchaWorkerCreateInterface: created local bootstrap broadcast interface for node 'tradedb2', haName 'CSS_tradedb-cluster', inf 'udp://172.16.3.255:42424'
16:22:07.295: [& & CSSD][]clssnmOpenGIPCEndp: listening on gipcha://tradedb2:nm2_tradedb-cluster
16:22:07.297: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:07.297: [& & CLSF][]Allocated CLSF context
16:22:07.297: [& &SKGFD][]Handle 0x181e6b0 from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:07.298: [& & CSSD][]clssnmlalloccxhyname tradedb2
16:22:07.299: [& & CSSD][]clssnmvDiskAvailabilityChange: voting file /dev/asm-ocr01 now online
16:22:07.299: [& & CSSD][]clssnmlgetfileslot: found expired slot 2 for host tradedb2 leasename tradedb2
16:22:07.301: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:07.302: [& & CLSF][]Allocated CLSF context
16:22:07.302: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:07.302: [& &SKGFD][]Lib :UFS:: closing handle 0x7f for disk :/dev/asm-ocr01:
16:22:08.304: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:08.304: [& & CLSF][]Allocated CLSF context
16:22:08.304: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:08.305: [& &SKGFD][]Lib :UFS:: closing handle 0x7f for disk :/dev/asm-ocr01:
16:22:09.306: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:09.306: [& & CLSF][]Allocated CLSF context
16:22:09.306: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:09.307: [& &SKGFD][]Lib :UFS:: closing handle 0x7f for disk :/dev/asm-ocr01:
16:22:10.308: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:10.308: [& & CLSF][]Allocated CLSF context
16:22:10.309: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:10.309: [& &SKGFD][]Lib :UFS:: closing handle 0x7f for disk :/dev/asm-ocr01:
16:22:11.310: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:11.310: [& & CLSF][]Allocated CLSF context
16:22:11.311: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:11.311: [& &SKGFD][]Lib :UFS:: closing handle 0x7f for disk :/dev/asm-ocr01:
16:22:12.297: [ CLSINET][] Returning NETDATA: 1 interfaces
16:22:12.297: [ CLSINET][] # 0 Interface 'eth1',ip='172.16.3.3',mac='40-a8-f0-1e-88-8d',mask='255.255.255.0',net='172.16.3.0',use='cluster_interconnect'
16:22:12.297: [GIPCHGEN][] gipchaNodeAddInterface: updating interface information for host '', haName 'CSS_tradedb-cluster', oldIp '172.16.3.3', newIp '172.16.3.3:60277'
16:22:12.311: [& & CSSD][]clssnmlfmtlease: uniqueness , gipc addr gipcha://tradedb2:nm2_tradedb-cluster
16:22:12.312: [& & CSSD][]clssnmvStatusBlkInit: myinfo nodename tradedb2, uniqueness
16:22:12.324: [& & CSSD][]clssnmlgetslot:lease acquisition for node tradedb2/slot 2 completed in 5020 msecs
16:22:12.326: [& & CSSD][]clssnmvDHBValidateNCopy: node 1, , has a disk HB, but no network HB, DHB has rcfg , wrtcnt, , LATS , lastSeqNo 0, uniqueness , timestamp /
16:22:12.326: [& & CSSD][]clssnmvDHBValidateNCopy: node 2, , has a disk HB, but no network HB, DHB has rcfg , wrtcnt, 1917661, LATS , lastSeqNo 0, uniqueness , timestamp /
16:22:12.338: [& &SKGFD][]Lib :UFS:: closing handle 0x181e6b0 for disk :/dev/asm-ocr01:
16:22:12.338: [& & CSSD][]clssnmInitNodeDB: Initializing with OCR id 0
16:22:12.631: [& & CSSD][]clssscSelect: cookie accept request 0x15f7650
16:22:12.631: [& & CSSD][]clssgmAllocProc: (0x7fd0) allocated
16:22:12.632: [& & CSSD][]clssgmClientConnectMsg: properties of cmProc 0x7fd0 - 1,2,3,4,5
16:22:12.632: [& & CSSD][]clssgmClientConnectMsg: Connect from con(0x41c) proc(0x7fd0) pid(51664) version 11:2:1:4, properties: 1,2,3,4,5
16:22:12.632: [& & CSSD][]clssgmClientConnectMsg: msg flags 0x0000
16:22:12.632: [& & CSSD][]clssgmExecuteClientRequest(): type(37) size(80) only connect and exit messages are allowed before lease acquisition proc(0x7fd0) client((nil))
16:22:12.633: [& & CSSD][]clssgmDeadProc: proc 0x7fd0
16:22:12.633: [& & CSSD][]clssgmDestroyProc: cleaning up proc(0x7fd0) con(0x41c) skgpid&&ospid 51664 with 0 clients, refcount 0
16:22:12.633: [& & CSSD][]clssgmDiscEndpcl: gipcDestroy 0x41c
16:22:12.636: [& & CSSD][]clssscSelect: cookie accept request 0x15f7650
16:22:12.636: [& & CSSD][]clssgmAllocProc: (0x7f241405ece0) allocated
16:22:12.637: [& & CSSD][]clssgmClientConnectMsg: properties of cmProc 0x7f241405ece0 - 1,2,3,4,5
16:22:12.637: [& & CSSD][]clssgmClientConnectMsg: Connect from con(0x480) proc(0x7f241405ece0) pid(51664) version 11:2:1:4, properties: 1,2,3,4,5
16:22:12.637: [& & CSSD][]clssgmClientConnectMsg: msg flags 0x0000
16:22:13.338: [& & CSSD][]clssnm_skgxnmon: Compatible vendor clusterware not in use
16:22:13.338: [& & CSSD][]clssnmNotifyReq: type (20)
16:22:13.338: [& & CSSD][]clssnmReadNodeInfo: dynamic endp HA gipcha://tradedb1:nm2_tradedb-cluster len 37 ver 11.2.0.3.0
16:22:13.338: [& & CSSD][]clssnmReadNodeInfo: NM endp for node 1 (tradedb1) - gipcha://tradedb1:nm2_tradedb-cluster
16:22:13.338: [& & CSSD][]clssnmReadNodeInfo: added node 1 (tradedb1) to cluster, lease uniqueness
16:22:13.338: [& & CSSD][]clssnmReadNodeInfo: dynamic endp HA gipcha://tradedb2:nm2_tradedb-cluster len 37 ver 11.2.0.3.0
16:22:13.340: [& & GPNP][] clsgpnp_Init: [at clsgpnp0.c:585] '/oracle/product/grid/11.2.0' in effect as GPnP home base.
16:22:13.340: [& & GPNP][] clsgpnp_Init: [at clsgpnp0.c:619] GPnP pid=51707, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=2003
16:22:13.348: [& & GPNP][] clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399] Using FS Wallet Location : /oracle/product/grid/11.2.0/gpnp/tradedb2/wallets/peer/
[& &CLWAL][]clsw_Initialize: OLR initlevel [70000]
16:22:13.378: [& & GPNP][] clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url &ipc://GPNPD_tradedb2& disco && [f=3 claimed- host: cname: seq: auth:]
16:22:13.390: [& & GPNP][] clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote &ipc://GPNPD_tradedb2& disco &&
16:22:13.592: [& & CSSD][]clssnmReadNodeInfo: NM endp for node 2 (tradedb2) - gipcha://tradedb2:nm2_tradedb-cluster
16:22:13.592: [& & CSSD][]clssnmReadNodeInfo: added node 2 (tradedb2) to cluster, lease uniqueness
16:22:13.592: [& & CSSD][][INFO]clssnmInitNodeDB: no vendor clusterware, IPMI allowed
[& &CLWAL][]clsw_Initialize: OLR initlevel [70000]
16:22:13.602: [& & CSSD][]clssnmReadWallet: OpenWallet: CLSW-9: The cluster wallet to be operated on does not exist., returned 9
16:22:13.602: [& & CSSD][]clssnmSetupCookie: clssnmReadWallet fails - Node not configured for NodeKill
16:22:13.602: [& & CSSD][]clssnmInitNodeDB: failure 1 reading node kill
16:22:13.602: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:13.602: [& & CLSF][]Allocated CLSF context
16:22:13.602: [& & CSSD][]clssnmvDiskOpen: Opening /dev/asm-ocr01
16:22:13.603: [& &SKGFD][]Handle 0x181daf0 from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:13.604: [& & CSSD][]clssnmvStatusBlkInit: myinfo nodename tradedb2, uniqueness
16:22:13.614: [& &SKGFD][]Lib :UFS:: closing handle 0x181daf0 for disk :/dev/asm-ocr01:
16:22:13.616: [& & CSSD][]clssscmain: clssnmvInit completed for node tradedb2, number 2, in cluster tradedb-cluster
16:22:13.617: [& & CSSD][]clssgmDeathChkThread: Spawned
16:22:13.616: [& & CSSD][]clssnmvWorkerThread: spawned for disk /dev/asm-ocr01
16:22:13.619: [& & CSSD][]clssnmNotifyReq: type (11)
16:22:13.619: [& & CSSD][]clssnmNotifyReq: found 1 for type (11) 0x7f
16:22:13.619: [& & CSSD][]clssnmCompleteGMReq: Completed request type 11 with status 1
16:22:13.619: [& & CSSD][]clssgmDoneQEle: re-queueing req 0x7f status 1
16:22:13.619: [& & CSSD][]clssnmNotifyReq: found 1 for type (11) 0x7f
16:22:13.619: [& & CSSD][]clssnmCompleteGMReq: Completed request type 11 with status 1
16:22:13.619: [& & CSSD][]clssgmDoneQEle: re-queueing req 0x7f status 1
16:22:13.619: [& & CSSD][]clssnmNotifyReq: type (13)
16:22:13.619: [& & CSSD][]clssgmCheckReqNMCompletion: Completing request type 11 for proc (0x7f), operation status 1, client status 0
16:22:13.619: [& & CSSD][]clssnmNotifyReq: found 1 for type (13) 0x7f
16:22:13.619: [& & CSSD][]clssnmCompleteGMReq: Completed request type 13 with status 1
16:22:13.619: [& & CSSD][]clssgmDoneQEle: re-queueing req 0x7f status 1
16:22:13.619: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:13.619: [& & CLSF][]Allocated CLSF context
16:22:13.619: [& & CSSD][]clssnmvDiskPingThread: spawned for disk /dev/asm-ocr01
16:22:13.620: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:13.620: [& & CSSD][]clssgmclientOpenEndp: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_tradedb-cluster_2)(GIPCID=))
16:22:13.620: [& & CSSD][]clssgmPeerListener: Spawned for node
16:22:13.622: [& & CSSD][]clssgmPeerListener: physical hostname tradedb2 privname 172.16.3.3
16:22:13.622: [& & CSSD][]clssgmPeerListener: gipc addr gipc://tradedb2:gm_tradedb-cluster
16:22:13.622: [& & CSSD][]clssgmPeerListener: gipc addr gipcha://tradedb2:gm2_tradedb-cluster
16:22:13.622: [& & CSSD][]clssgmclientOpenEndp: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_tradedb2_tradedb-cluster)(GIPCID=))
16:22:13.622: [& & CSSD][]clssgmCheckReqNMCompletion: Completing request type 11 for proc (0x7f), operation status 1, client status 0
16:22:13.623: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:13.623: [& & CLSF][]Allocated CLSF context
16:22:13.623: [& & CSSD][]clssgmPeerListener: listening on&&gipcha://tradedb2:gm2_tradedb-cluster
16:22:13.623: [& & CSSD][]clssnmvKillBlockThread: spawned for disk /dev/asm-ocr01 initial sleep interval (1000)ms
16:22:13.623: [& &SKGFD][]Handle 0x7f23f8036690 from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:13.624: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:13.624: [& & CLSF][]Allocated CLSF context
16:22:13.624: [& & CSSD][]clssgmEvtInformation: reqtype (22) cmProc (0x7f) client ((nil))
16:22:13.624: [& & CSSD][]clssgmEvtInformation: reqtype (22) req (0x7f)
16:22:13.624: [& & CSSD][]clssnmQueueNotification: type (22) 0x7f
16:22:13.625: [& & CSSD][]clssgmEvtInformation: reqtype (22) cmProc (0x7f) client ((nil))
16:22:13.625: [& & CSSD][]clssgmEvtInformation: reqtype (22) req (0x7f)
16:22:13.625: [& & CSSD][]clssnmQueueNotification: type (22) 0x7f
16:22:13.625: [& &SKGFD][]Handle 0x7f23f0086090 from lib :UFS:: for disk :/dev/asm-ocr01:
论坛徽章:0
16:22:13.625: [& & CSSD][]clssnmvDHBValidateNcopy: Saving DHB uniqueness of
for node tradedb1, number 1
16:22:13.625: [& & CSSD][]clssnmvDHBValidateNCopy: node 1, tradedb1, has a disk HB, but no network HB, DHB has rcfg , wrtcnt, , LATS , lastSeqNo , uniqueness , timestamp /
16:22:13.624: [& & CSSD][]clssnmPollingThread: Spawned, poll interval 1000
16:22:13.627: [& & CSSD][]clssnmRcfgMgrThread: Spawned
16:22:13.626: [& & CSSD][]clssnmSendingThread: Spawned
16:22:13.629: [& & CSSD][]clssnmClusterListener: Spawned
16:22:14.623: [& & CSSD][]clssgmWaitOnEventValue: after CmInfo State&&val 3, eval 1 waited 0
16:22:14.626: [& & CSSD][]clssnmvDHBValidateNcopy: Setting LATS valid due to second DHB seen for node tradedb1, number 1
16:22:14.626: [& & CSSD][]clssnmvDHBValidateNcopy: Copying unique
to node structure for node tradedb1, number 1; previous unique value was 0
16:22:14.627: [& & CSSD][](:CSSNM00006clssnmvDHBValidateNcopy: An active CSS found on node tradedb1, number 1, when we are exclusive mode
16:22:14.627: [& & CSSD][]clssgmSendMsgToAgent: Sending msg id 4402, size 43, product CRS, facility CRS, to agent (0x7f)
16:22:15.623: [& & CSSD][]clssgmWaitOnEventValue: after CmInfo State&&val 3, eval 1 waited 0
16:22:51.956: [& & CSSD][]clsu_load_ENV_levels: Module = CSSD, LogLevel = 2, TraceLevel = 0
16:22:51.956: [& & CSSD][]clsu_load_ENV_levels: Module = GIPCNM, LogLevel = 2, TraceLevel = 0
16:22:51.957: [& & CSSD][]clsu_load_ENV_levels: Module = GIPCGM, LogLevel = 2, TraceLevel = 0
16:22:51.957: [& & CSSD][]clsu_load_ENV_levels: Module = GIPCCM, LogLevel = 2, TraceLevel = 0
16:22:51.957: [& & CSSD][]clsu_load_ENV_levels: Module = CLSF, LogLevel = 0, TraceLevel = 0
16:22:51.957: [& & CSSD][]clsu_load_ENV_levels: Module = SKGFD, LogLevel = 0, TraceLevel = 0
16:22:51.957: [& & CSSD][]clsu_load_ENV_levels: Module = GPNP, LogLevel = 1, TraceLevel = 0
16:22:51.957: [& & CSSD][]clsu_load_ENV_levels: Module = OLR, LogLevel = 0, TraceLevel = 0
[& & CSSD][]clsugetconf : Configuration type [4].
16:22:51.957: [& & CSSD][]clssscmain: Starting CSS daemon, version 11.2.0.3.0, in (clustered) mode with uniqueness value
16:22:51.958: [& & CSSD][]clssscmain: Environment is production
16:22:51.958: [& & CSSD][]clssscmain: Core file size limit extended
16:22:51.963: [& & CSSD][]clssscmain: GIPCHA down 0
16:22:51.965: [& & CSSD][]clssscGetParameterOLR: OLR fetch for parameter logsize (8) failed with rc 21
16:22:51.965: [& & CSSD][]clssscExtendLimits: The current soft limit for file descriptors is 65536, hard limit is 65536
16:22:51.965: [& & CSSD][]clssscExtendLimits: The current soft limit for locked memory is , hard limit is
16:22:51.966: [& & CSSD][]clssscGetParameterOLR: OLR fetch for parameter priority (15) failed with rc 21
16:22:51.966: [& & CSSD][]clssscSetPrivEnv: Setting priority to 4
16:22:51.977: [& & CSSD][]clssscSetPrivEnv: Can't access local IPMI device--no device configured or driver missing/incompatible. IPMI support may be available with static IP configuration.
16:22:51.978: [& & CSSD][]clssscmain: Running as user grid
16:22:51.979: [& & CSSD][]clssscmain: RT queue setting is at default value
[&&clsdmt][]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=tradedb2DBG_CSSD))
16:22:51.983: [&&clsdmt][]PID for the Process [51939], connkey 4
16:22:51.984: [& & CSSD][]clssscGetParameterOLR: OLR fetch for parameter auth rep (9) failed with rc 21
16:22:51.984: [& & CSSD][]clssscGetParameterOLR: OLR fetch for parameter diagwait (14) failed with rc 21
16:22:51.986: [& & CSSD][]clssnmInitNMInfoMin: Initializing first-reconfig to (0)
16:22:51.987: [& & CSSD][]clssscmain: initgminfo done
16:22:51.987: [& & CSSD][]clssgmclientlsnr: Spawned
16:22:51.987: [& & CSSD][]clssgmEvtInformation: reqtype (13) cmProc ((nil)) client ((nil))
16:22:51.987: [& & CSSD][]clssgmEvtInformation: reqtype (13) req (0x7f)
16:22:51.987: [& & CSSD][]clssnmQueueNotification: type (13) 0x7f
16:22:51.989: [& & CSSD][]clssgmclientlsnr: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_tradedb2_)(GIPCID=))
16:22:51.989: [& & GPNP][]clsgpnp_Init: [at clsgpnp0.c:585] '/oracle/product/grid/11.2.0' in effect as GPnP home base.
16:22:51.989: [& & GPNP][]clsgpnp_Init: [at clsgpnp0.c:619] GPnP pid=51939, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=3
16:22:51.998: [& & GPNP][]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399] Using FS Wallet Location : /oracle/product/grid/11.2.0/gpnp/tradedb2/wallets/peer/
[& &CLWAL][]clsw_Initialize: OLR initlevel [70000]
16:22:52.025: [& & GPNP][]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url &ipc://GPNPD_tradedb2& disco && [f=0 claimed- host: cname: seq: auth:]
16:22:52.032: [& & GPNP][]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri &http://www.grid-pnp.org/2005/12/gpnp-errors#&]
16:22:52.914: [& & CSSD][]clssscSelect: cookie accept request 0x17cc720
16:22:52.914: [& & CSSD][]clssgmAllocProc: (0x7f) allocated
16:22:52.915: [& & CSSD][]clssgmClientConnectMsg: properties of cmProc 0x7f - 1,2,3,4,5
16:22:52.915: [& & CSSD][]clssgmClientConnectMsg: Connect from con(0x140) proc(0x7f) pid(51894) version 11:2:1:4, properties: 1,2,3,4,5
16:22:52.915: [& & CSSD][]clssgmClientConnectMsg: The CSSD monitor is process (0x7f), number 1
16:22:52.920: [& & CSSD][]clssgmEvtInformation: reqtype (11) cmProc (0x7f) client ((nil))
16:22:52.920: [& & CSSD][]clssgmEvtInformation: reqtype (11) req (0x7f)
16:22:52.920: [& & CSSD][]clssnmQueueNotification: type (11) 0x7f
16:22:52.934: [& & CSSD][]clssscSelect: cookie accept request 0x17cc720
16:22:52.934: [& & CSSD][]clssgmAllocProc: (0x7f) allocated
16:22:52.935: [& & CSSD][]clssgmClientConnectMsg: properties of cmProc 0x7f - 1,2,3,4,5
16:22:52.935: [& & CSSD][]clssgmClientConnectMsg: Connect from con(0x197) proc(0x7f) pid(51915) version 11:2:1:4, properties: 1,2,3,4,5
16:22:52.935: [& & CSSD][]clssgmClientConnectMsg: The CSSD agent is process (0x7f), number 2
16:22:52.940: [& & CSSD][]clssgmEvtInformation: reqtype (11) cmProc (0x7f) client ((nil))
16:22:52.940: [& & CSSD][]clssgmEvtInformation: reqtype (11) req (0x7f)
16:22:52.940: [& & CSSD][]clssnmQueueNotification: type (11) 0x7f
16:22:54.039: [& & GPNP][]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri &http://www.grid-pnp.org/2005/12/gpnp-errors#&]
16:22:56.049: [& & GPNP][]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote &ipc://GPNPD_tradedb2& disco &&
16:22:56.049: [& & CSSD][]clssscGetParameterProfile: profile fetch failed for parameter ocrid (4) with return code 5
16:22:56.049: [& & CSSD][]clssscmain: OCRID is 0
16:22:56.049: [& & CSSD][]clssscmain: Cluster GUID is f4cf11bfba
16:22:56.049: [& & CSSD][]clssnmNotifyReq: type (12)
16:22:56.050: [& & CSSD][]clssscmain: last used node number 2
16:22:56.050: [& & CSSD][]clssnmReadDiscoveryProfile: voting file discovery string(/dev/asm*)
16:22:56.050: [& & CSSD][]clssnkInit: NK generic layer initializing.
16:22:56.052: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:56.052: [& & CLSF][]Allocated CLSF context
16:22:56.052: [& & CSSD][]clssnmvDDiscThread: using discovery string /dev/asm* for initial discovery
16:22:56.053: [& &SKGFD][]Discovery with str:/dev/asm*:
16:22:56.053: [& &SKGFD][]UFS discovery with :/dev/asm*:
16:22:56.055: [& &SKGFD][]Fetching UFS disk :/dev/asm-disk01:
16:22:56.055: [& &SKGFD][]Fetching UFS disk :/dev/asm-vote01:
16:22:56.055: [& &SKGFD][]Fetching UFS disk :/dev/asm-ocr01:
16:22:56.055: [& &SKGFD][]Fetching UFS disk :/dev/asm-disk02:
16:22:56.055: [& &SKGFD][]OSS discovery with :/dev/asm*:
16:22:56.056: [& &SKGFD][]Handle 0x7f784c08f6d0 from lib :UFS:: for disk :/dev/asm-disk01:
16:22:56.056: [& &SKGFD][]Handle 0x7f784c090070 from lib :UFS:: for disk :/dev/asm-vote01:
16:22:56.056: [& &SKGFD][]Handle 0x7f784c090c80 from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:56.057: [& &SKGFD][]Handle 0x7f784c091890 from lib :UFS:: for disk :/dev/asm-disk02:
16:22:56.057: [& &SKGFD][]Lib :UFS:: closing handle 0x7f784c08f6d0 for disk :/dev/asm-disk01:
16:22:56.057: [& &SKGFD][]Lib :UFS:: closing handle 0x7f784c090070 for disk :/dev/asm-vote01:
16:22:56.057: [& &SKGFD][]Lib :UFS:: closing handle 0x7f784c090c80 for disk :/dev/asm-ocr01:
16:22:56.057: [& &SKGFD][]Lib :UFS:: closing handle 0x7f784c091890 for disk :/dev/asm-disk02:
16:22:56.057: [& & CSSD][]clssnmvDiskVerify: discovered a potential voting file
16:22:56.057: [& &SKGFD][]Handle 0x7f784c08e690 from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:56.059: [& & CSSD][]clssnmvDiskVerify: Successful discovery for disk /dev/asm-ocr01, UID 4ce283ae-a5154ffa-bf056a5d-8a19acfe, Pending CIN 0::0, Committed CIN 0::0
16:22:56.059: [& &SKGFD][]Lib :UFS:: closing handle 0x7f784c08e690 for disk :/dev/asm-ocr01:
16:22:56.059: [& & CSSD][]clssnmvDiskVerify: Successful discovery of 1 disks
16:22:56.059: [& & CSSD][]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
16:22:56.059: [& & CSSD][]clssnmCompleteVFDiscovery: Completing voting file discovery
16:22:56.059: [& & CSSD][]clssnmvDiskStateChange: state from discovered to pending disk /dev/asm-ocr01
16:22:56.059: [& & CSSD][]clssnmvDiskStateChange: state from pending to configured disk /dev/asm-ocr01
16:22:56.059: [& & CSSD][]clssnmCompleteVFDiscovery: Committed configuration for CIN 0::0
16:22:56.059: [& & CSSD][]&&misscount& && && & 30& & reboot latency& && &3
16:22:56.059: [& & CSSD][]&&long I/O timeout&&200& & short I/O timeout&&27
16:22:56.060: [& & CSSD][]&&diagnostic wait& &&&0&&active version 11.2.0.3.0
16:22:56.060: [& & CSSD][]&&Listing unique IDs for 1 voting files:
16:22:56.060: [& & CSSD][]& & voting file 1: 4ce283ae-a5154ffa-bf056a5d-8a19acfe
16:22:56.060: [& & CSSD][]clssnmOpenGIPCEndp: opening cluster listener on gipc://tradedb2:nm_tradedb-cluster
16:22:56.060: [GIPCHGEN][] gipchaInternalRegister: Initializing HA GIPC
16:22:56.060: [GIPCHGEN][] gipchaNodeCreate: adding new node 0x19f2360 { host '', haName 'CSS_tradedb-cluster', srcLuid d273f824-, dstLuid 00000 numInf 0, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [0 : 0], createTime , sentRegister 0, localMonitor 0, flags 0x1 }
16:22:56.060: [& & GPNP][]clsgpnp_Init: [at clsgpnp0.c:585] '/oracle/product/grid/11.2.0' in effect as GPnP home base.
16:22:56.060: [& & GPNP][]clsgpnp_Init: [at clsgpnp0.c:619] GPnP pid=51939, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=0
16:22:56.068: [& & GIPC][] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 769], original from [clsssc.c : 970]
16:22:56.069: [& & GPNP][]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399] Using FS Wallet Location : /oracle/product/grid/11.2.0/gpnp/tradedb2/wallets/peer/
[& &CLWAL][]clsw_Initialize: OLR initlevel [30000]
16:22:56.096: [& & GPNP][]clsgpnp_getCachedProfileEx: [at clsgpnp.c:613] Result: (26) CLSGPNP_NO_PROFILE. Can't get offline GPnP service profile: local gpnpd is up and running. Use getProfile instead.
16:22:56.096: [& & GPNP][]clsgpnp_getCachedProfileEx: [at clsgpnp.c:623] Result: (26) CLSGPNP_NO_PROFILE. Failed to get offline GPnP service profile.
16:22:56.097: [& & GPNP][]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url &ipc://GPNPD_tradedb2& disco && [f=1 claimed- host: cname: seq: auth:]
16:22:56.108: [& & GPNP][]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote &ipc://GPNPD_tradedb2& disco &&
16:22:56.109: [GIPCXCPT][] gipchaInternalReadGpnp: No network info configured in GPNP, using defaults, ret gipcretFail (1)
16:22:56.310: [GIPCHGEN][] gipchaInternalReadGpnp: configuring default multicast addresses
16:22:56.310: [GIPCHGEN][] gipchaInternalReadGpnp: configuring default bootstrap communications modes
16:22:56.310: [GIPCHGEN][] gipchaInternalReadGpnp: configuring bootstrap communications using:&&broadcast and multicast
16:22:56.310: [GIPCHGEN][] gipchaInternalReadGpnp: mcast address[&&0 ]&&224.0.0.251
16:22:56.310: [GIPCHGEN][] gipchaInternalReadGpnp: mcast address[&&1 ]&&230.0.1.0
16:22:56.312: [GIPCHTHR][] gipchaWorkerThread: starting worker thread hctx 0x10] { gipchaContext : host 'tradedb2', name 'CSS_tradedb-cluster', luid 'd273f824-', numNode 0, numInf 0, usrFlags 0x0, flags 0x62 }
16:22:56.314: [GIPCHDEM][] gipchaDaemonThread: starting daemon thread hctx 0x10] { gipchaContext : host 'tradedb2', name 'CSS_tradedb-cluster', luid 'd273f824-', numNode 0, numInf 0, usrFlags 0x0, flags 0x62 }
16:22:56.314: [& & GPNP][]clsgpnp_Init: [at clsgpnp0.c:585] '/oracle/product/grid/11.2.0' in effect as GPnP home base.
16:22:56.314: [& & GPNP][]clsgpnp_Init: [at clsgpnp0.c:619] GPnP pid=51939, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=0
16:22:56.322: [& & GIPC][] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 769], original from [clsssc.c : 970]
16:22:56.323: [& & GPNP][]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399] Using FS Wallet Location : /oracle/product/grid/11.2.0/gpnp/tradedb2/wallets/peer/
[& &CLWAL][]clsw_Initialize: OLR initlevel [30000]
16:22:56.350: [& & GPNP][]clsgpnp_getCachedProfileEx: [at clsgpnp.c:613] Result: (26) CLSGPNP_NO_PROFILE. Can't get offline GPnP service profile: local gpnpd is up and running. Use getProfile instead.
16:22:56.350: [& & GPNP][]clsgpnp_getCachedProfileEx: [at clsgpnp.c:623] Result: (26) CLSGPNP_NO_PROFILE. Failed to get offline GPnP service profile.
16:22:56.350: [& & GPNP][]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url &ipc://GPNPD_tradedb2& disco && [f=1 claimed- host: cname: seq: auth:]
16:22:56.359: [& & GPNP][]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote &ipc://GPNPD_tradedb2& disco &&
16:22:56.359: [ GIPCLIB][] gipclibGetClusterGuid: retrieved cluster guid f4cf11bfba
16:22:56.562: [& & GPNP][] clsgpnp_Init: [at clsgpnp0.c:585] '/oracle/product/grid/11.2.0' in effect as GPnP home base.
16:22:56.562: [& & GPNP][] clsgpnp_Init: [at clsgpnp0.c:619] GPnP pid=51939, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=2003
16:22:56.570: [& & GPNP][] clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399] Using FS Wallet Location : /oracle/product/grid/11.2.0/gpnp/tradedb2/wallets/peer/
[& &CLWAL][]clsw_Initialize: OLR initlevel [70000]
16:22:56.598: [& & GPNP][] clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url &ipc://GPNPD_tradedb2& disco && [f=3 claimed- host: cname: seq: auth:]
16:22:56.607: [& & GPNP][] clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote &ipc://GPNPD_tradedb2& disco &&
16:22:56.607: [ CLSINET][] Returning NETDATA: 1 interfaces
16:22:56.607: [ CLSINET][] # 0 Interface 'eth1',ip='172.16.3.3',mac='40-a8-f0-1e-88-8d',mask='255.255.255.0',net='172.16.3.0',use='cluster_interconnect'
16:22:56.607: [GIPCHGEN][] gipchaNodeAddInterface: adding interface information for inf 0x7f { host '', haName 'CSS_tradedb-cluster', local (nil), ip '172.16.3.3', subnet '172.16.3.0', mask '255.255.255.0', mac '40-a8-f0-1e-88-8d', ifname 'eth1', numRef 0, numFail 0, idxBoot 0, flags 0x1841 }
16:22:56.608: [GIPCHTHR][] gipchaWorkerCreateInterface: created local interface for node 'tradedb2', haName 'CSS_tradedb-cluster', inf 'udp://172.16.3.3:49152'
16:22:56.608: [GIPCHTHR][] gipchaWorkerCreateInterface: created local bootstrap multicast interface for node 'tradedb2', haName 'CSS_tradedb-cluster', inf 'mcast://224.0.0.251:.3.3'
16:22:56.608: [GIPCHTHR][] gipchaWorkerCreateInterface: created local bootstrap multicast interface for node 'tradedb2', haName 'CSS_tradedb-cluster', inf 'mcast://230.0.1.0:.3.3'
16:22:56.608: [GIPCHTHR][] gipchaWorkerCreateInterface: created local bootstrap broadcast interface for node 'tradedb2', haName 'CSS_tradedb-cluster', inf 'udp://172.16.3.255:42424'
16:22:56.609: [& & CSSD][]clssnmOpenGIPCEndp: listening on gipcha://tradedb2:nm2_tradedb-cluster
16:22:56.610: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:56.610: [& & CLSF][]Allocated CLSF context
16:22:56.611: [& &SKGFD][]Handle 0x17eb6d0 from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:56.611: [& & CSSD][]clssnmlalloccxhyname tradedb2
16:22:56.612: [& & CSSD][]clssnmvDiskAvailabilityChange: voting file /dev/asm-ocr01 now online
16:22:56.612: [& & CSSD][]clssnmlgetfileslot: found expired slot 2 for host tradedb2 leasename tradedb2
16:22:56.615: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:56.615: [& & CLSF][]Allocated CLSF context
16:22:56.616: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:56.616: [& &SKGFD][]Lib :UFS:: closing handle 0x7f for disk :/dev/asm-ocr01:
16:22:57.617: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:57.617: [& & CLSF][]Allocated CLSF context
16:22:57.618: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:57.618: [& &SKGFD][]Lib :UFS:: closing handle 0x7f for disk :/dev/asm-ocr01:
16:22:58.619: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:58.620: [& & CLSF][]Allocated CLSF context
16:22:58.620: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:58.621: [& &SKGFD][]Lib :UFS:: closing handle 0x7f for disk :/dev/asm-ocr01:
16:22:59.622: [& &SKGFD][]NOTE: No asm libraries found in the system
16:22:59.622: [& & CLSF][]Allocated CLSF context
16:22:59.622: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-ocr01:
16:22:59.623: [& &SKGFD][]Lib :UFS:: closing handle 0x7f for disk :/dev/asm-ocr01:
16:23:00.624: [& &SKGFD][]NOTE: No asm libraries found in the system
16:23:00.624: [& & CLSF][]Allocated CLSF context
16:23:00.625: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-ocr01:
16:23:00.625: [& &SKGFD][]Lib :UFS:: closing handle 0x7f for disk :/dev/asm-ocr01:
16:23:01.611: [ CLSINET][] Returning NETDATA: 1 interfaces
16:23:01.611: [ CLSINET][] # 0 Interface 'eth1',ip='172.16.3.3',mac='40-a8-f0-1e-88-8d',mask='255.255.255.0',net='172.16.3.0',use='cluster_interconnect'
16:23:01.611: [GIPCHGEN][] gipchaNodeAddInterface: updating interface information for host '', haName 'CSS_tradedb-cluster', oldIp '172.16.3.3', newIp '172.16.3.3:49152'
16:23:01.625: [& & CSSD][]clssnmlfmtlease: uniqueness , gipc addr gipcha://tradedb2:nm2_tradedb-cluster
16:23:01.626: [& & CSSD][]clssnmvStatusBlkInit: myinfo nodename tradedb2, uniqueness
16:23:01.629: [& & CSSD][]clssnmlgetslot:lease acquisition for node tradedb2/slot 2 completed in 5010 msecs
16:23:01.641: [& & CSSD][]clssnmvDHBValidateNCopy: node 1, , has a disk HB, but no network HB, DHB has rcfg , wrtcnt, , LATS , lastSeqNo 0, uniqueness , timestamp /
16:23:01.641: [& & CSSD][]clssnmvDHBValidateNCopy: node 2, , has a disk HB, but no network HB, DHB has rcfg , wrtcnt, 1917668, LATS , lastSeqNo 0, uniqueness , timestamp /
16:23:01.642: [& &SKGFD][]Lib :UFS:: closing handle 0x17eb6d0 for disk :/dev/asm-ocr01:
16:23:01.643: [& & CSSD][]clssnmInitNodeDB: Initializing with OCR id 0
16:23:01.985: [& & CSSD][]clssscSelect: cookie accept request 0x17cc720
16:23:01.985: [& & CSSD][]clssgmAllocProc: (0x7f) allocated
16:23:01.986: [& & CSSD][]clssgmClientConnectMsg: properties of cmProc 0x7f - 1,2,3,4,5
16:23:01.986: [& & CSSD][]clssgmClientConnectMsg: Connect from con(0x419) proc(0x7f) pid(51896) version 11:2:1:4, properties: 1,2,3,4,5
16:23:01.986: [& & CSSD][]clssgmClientConnectMsg: msg flags 0x0000
16:23:01.986: [& & CSSD][]clssgmExecuteClientRequest(): type(37) size(80) only connect and exit messages are allowed before lease acquisition proc(0x7f) client((nil))
16:23:01.987: [& & CSSD][]clssgmDeadProc: proc 0x7f
16:23:01.987: [& & CSSD][]clssgmDestroyProc: cleaning up proc(0x7f) con(0x419) skgpid&&ospid 51896 with 0 clients, refcount 0
16:23:01.987: [& & CSSD][]clssgmDiscEndpcl: gipcDestroy 0x419
16:23:01.990: [& & CSSD][]clssscSelect: cookie accept request 0x17cc720
16:23:01.991: [& & CSSD][]clssgmAllocProc: (0x7f) allocated
16:23:01.991: [& & CSSD][]clssgmClientConnectMsg: properties of cmProc 0x7f - 1,2,3,4,5
16:23:01.991: [& & CSSD][]clssgmClientConnectMsg: Connect from con(0x47d) proc(0x7f) pid(51896) version 11:2:1:4, properties: 1,2,3,4,5
16:23:01.991: [& & CSSD][]clssgmClientConnectMsg: msg flags 0x0000
16:23:02.643: [& & CSSD][]clssnm_skgxnmon: Compatible vendor clusterware not in use
16:23:02.643: [& & CSSD][]clssnmNotifyReq: type (20)
16:23:02.643: [& & CSSD][]clssnmReadNodeInfo: dynamic endp HA gipcha://tradedb1:nm2_tradedb-cluster len 37 ver 11.2.0.3.0
16:23:02.643: [& & CSSD][]clssnmReadNodeInfo: NM endp for node 1 (tradedb1) - gipcha://tradedb1:nm2_tradedb-cluster
16:23:02.643: [& & CSSD][]clssnmReadNodeInfo: added node 1 (tradedb1) to cluster, lease uniqueness
16:23:02.643: [& & CSSD][]clssnmReadNodeInfo: dynamic endp HA gipcha://tradedb2:nm2_tradedb-cluster len 37 ver 11.2.0.3.0
16:23:02.645: [& & GPNP][] clsgpnp_Init: [at clsgpnp0.c:585] '/oracle/product/grid/11.2.0' in effect as GPnP home base.
16:23:02.645: [& & GPNP][] clsgpnp_Init: [at clsgpnp0.c:619] GPnP pid=51939, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=2003
16:23:02.654: [& & GPNP][] clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399] Using FS Wallet Location : /oracle/product/grid/11.2.0/gpnp/tradedb2/wallets/peer/
[& &CLWAL][]clsw_Initialize: OLR initlevel [70000]
16:23:02.679: [& & GPNP][] clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url &ipc://GPNPD_tradedb2& disco && [f=3 claimed- host: cname: seq: auth:]
16:23:02.691: [& & GPNP][] clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote &ipc://GPNPD_tradedb2& disco &&
16:23:02.893: [& & CSSD][]clssnmReadNodeInfo: NM endp for node 2 (tradedb2) - gipcha://tradedb2:nm2_tradedb-cluster
16:23:02.893: [& & CSSD][]clssnmReadNodeInfo: added node 2 (tradedb2) to cluster, lease uniqueness
16:23:02.893: [& & CSSD][][INFO]clssnmInitNodeDB: no vendor clusterware, IPMI allowed
[& &CLWAL][]clsw_Initialize: OLR initlevel [70000]
16:23:02.902: [& & CSSD][]clssnmReadWallet: OpenWallet: CLSW-9: The cluster wallet to be operated on does not exist., returned 9
16:23:02.902: [& & CSSD][]clssnmSetupCookie: clssnmReadWallet fails - Node not configured for NodeKill
16:23:02.902: [& & CSSD][]clssnmInitNodeDB: failure 1 reading node kill
16:23:02.903: [& &SKGFD][]NOTE: No asm libraries found in the system
16:23:02.903: [& & CLSF][]Allocated CLSF context
16:23:02.903: [& & CSSD][]clssnmvDiskOpen: Opening /dev/asm-ocr01
16:23:02.904: [& &SKGFD][]Handle 0x1937690 from lib :UFS:: for disk :/dev/asm-ocr01:
16:23:02.904: [& & CSSD][]clssnmvStatusBlkInit: myinfo nodename tradedb2, uniqueness
16:23:02.906: [& &SKGFD][]Lib :UFS:: closing handle 0x1937690 for disk :/dev/asm-ocr01:
16:23:02.909: [& & CSSD][]clssscmain: clssnmvInit completed for node tradedb2, number 2, in cluster tradedb-cluster
16:23:02.910: [& & CSSD][]clssgmDeathChkThread: Spawned
16:23:02.912: [& & CSSD][]clssnmNotifyReq: type (11)
16:23:02.909: [& & CSSD][]clssnmvWorkerThread: spawned for disk /dev/asm-ocr01
16:23:02.912: [& & CSSD][]clssnmNotifyReq: found 1 for type (11) 0x7f
16:23:02.912: [& & CSSD][]clssnmCompleteGMReq: Completed request type 11 with status 1
16:23:02.912: [& & CSSD][]clssgmDoneQEle: re-queueing req 0x7f status 1
16:23:02.913: [& & CSSD][]clssnmNotifyReq: found 1 for type (11) 0x7f
16:23:02.913: [& & CSSD][]clssnmCompleteGMReq: Completed request type 11 with status 1
16:23:02.913: [& & CSSD][]clssgmDoneQEle: re-queueing req 0x7f status 1
16:23:02.913: [& & CSSD][]clssgmCheckReqNMCompletion: Completing request type 11 for proc (0x7f), operation status 1, client status 0
16:23:02.913: [& & CSSD][]clssnmNotifyReq: type (13)
16:23:02.913: [& & CSSD][]clssnmNotifyReq: found 1 for type (13) 0x7f
16:23:02.913: [& & CSSD][]clssgmCheckReqNMCompletion: Completing request type 11 for proc (0x7f), operation status 1, client status 0
16:23:02.913: [& & CSSD][]clssnmCompleteGMReq: Completed request type 13 with status 1
16:23:02.913: [& & CSSD][]clssgmDoneQEle: re-queueing req 0x7f status 1
16:23:02.914: [& & CSSD][]clssgmclientOpenEndp: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_tradedb-cluster_2)(GIPCID=))
16:23:02.915: [& & CSSD][]clssgmPeerListener: Spawned for node
16:23:02.916: [& & CSSD][]clssgmPeerListener: physical hostname tradedb2 privname 172.16.3.3
16:23:02.916: [& & CSSD][]clssgmPeerListener: gipc addr gipc://tradedb2:gm_tradedb-cluster
16:23:02.916: [& &SKGFD][]NOTE: No asm libraries found in the system
16:23:02.916: [& & CSSD][]clssgmPeerListener: gipc addr gipcha://tradedb2:gm2_tradedb-cluster
16:23:02.916: [& & CLSF][]Allocated CLSF context
16:23:02.916: [& & CSSD][]clssnmvDiskPingThread: spawned for disk /dev/asm-ocr01
16:23:02.916: [& & CSSD][]clssgmclientOpenEndp: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_tradedb2_tradedb-cluster)(GIPCID=))
16:23:02.916: [& &SKGFD][]Handle 0x7f from lib :UFS:: for disk :/dev/asm-ocr01:
16:23:02.917: [& &SKGFD][]NOTE: No asm libraries found in the system
16:23:02.917: [& & CLSF][]Allocated CLSF context
16:23:02.917: [& & CSSD][]clssgmPeerListener: listening on&&gipcha://tradedb2:gm2_tradedb-cluster
16:23:02.917: [& &SKGFD][]Handle 0x7f782c03d9c0 from lib :UFS:: for disk :/dev/asm-ocr01:
认证徽章论坛徽章:22
检查一下 tradedb2 节点的votedisk& &的权限 在tradedb2 grid 用户下执行 crsctl query css votedisk
论坛徽章:0
[root@tradedb2 ~]# /oracle/product/grid/11.2.0/bin/crsctl query css votedisk
##&&STATE& & File Universal Id& && && && && & File Name Disk group
--&&-----& & -----------------& && && && && & --------- ---------
1. ONLINE& &4ce283aea5154ffabf056a5d8a19acfe (/dev/asm-ocr01) [OCR]
Located 1 voting disk(s).
论坛徽章:0
[grid@tradedb2 ~]$ /oracle/product/grid/11.2.0/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME& && && &&&TARGET&&STATE& && &&&SERVER& && && && && && & STATE_DETAILS& && &
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
& && &1& && &&&OFFLINE OFFLINE& && && && && && && && && && && && && && && && && &
ora.cluster_interconnect.haip
& && &1& && &&&OFFLINE OFFLINE& && && && && && && && && && && && && && && && && &
& && &1& && &&&ONLINE&&ONLINE& && & tradedb2& && && && && && && && && && && && &
& && &1& && &&&OFFLINE OFFLINE& && && && && && && && && && && && && && && && && &
& && &1& && &&&ONLINE&&OFFLINE& && && && && && && && && && & STARTING& && && && &
ora.cssdmonitor
& && &1& && &&&ONLINE&&ONLINE& && & tradedb2& && && && && && && && && && && && &
& && &1& && &&&OFFLINE OFFLINE& && && && && && && && && && && && && && && && && &
ora.diskmon
& && &1& && &&&OFFLINE OFFLINE& && && && && && && && && && && && && && && && && &
& && &1& && &&&OFFLINE OFFLINE& && && && && && && && && && && && && && && && && &
& && &1& && &&&ONLINE&&ONLINE& && & tradedb2& && && && && && && && && && && && &
& && &1& && &&&ONLINE&&ONLINE& && & tradedb2& && && && && && && && && && && && &
& && &1& && &&&ONLINE&&ONLINE& && & tradedb2& && && && && && && && && && && &
招聘 : 论坛徽章:25
16:23:02.906: [& &SKGFD][]Lib :UFS:: closing handle 0x1937690 for disk :/dev/asm-ocr01:
16:23:02.917: [& &SKGFD][]NOTE: No asm libraries found in the system
crs磁盘损坏，Lib :UFS:: closing handle 0x1937690 for disk :/dev/asm-ocr01:
& && && && && && & NOTE: No asm libraries found in the system
crs有自动备份，查看ocrconfig –showbackup
进行恢复crs即可， ocrconfig -restore /grid/crs_home/product/11.2.0/cdata/rac-cluster/week.ocr
处理过程如下：
1. 创建新的crs磁盘，或者使用已经mount的data盘，
2. 停止所有集群：crsctl stop has -f
3. 以nocrs方式启动：crsctl start crs -excl -nocrs
4. 2个节点修改/etc/oracle/ocr.loc文件，将损坏的orc修改成新的，或data.
5. restore
& & ocrconfig -restore /grid/crs_home/product/11.2.0/cdata/rac-cluster/week.ocr
6. 检查ocr文件是否换成新的.
& &&&ocrcheck
7. 创建crs，把crs信息保存在新创建的crs盘里面.
& &&&7.1 alter system set asm_diskstring = '/dev/asm*';
& &&&7.2 create spfile='+DATA' 或者 create spfile='+新的crs'
& &&&7.3&&crsctl replace votedisk +DATA 或者&&crsctl replace votedisk +新的crs
8. 停止集群
crsctl stop has –f
9. 2个节点启动集群
crsctl start crs
成功启动集群.
论坛徽章:0
ocssd.log反复报如下错误 ：
group name CRF-, member number -1, flags 0x4000e00
15:08:46.055: [& & CSSD][]clssgmDiscEndpcl: gipcDestroy 0x4f2b
15:08:46.056: [& & CSSD][]clssgmDeadProc: proc 0x7f821c082250
15:08:46.056: [& & CSSD][]clssgmDestroyProc: cleaning up proc(0x7f821c082250) con(0x4efc) skgpid&&ospid 54988 with 0 clients, refcount 0
15:08:46.056: [& & CSSD][]clssgmDiscEndpcl: gipcDestroy 0x4efc
15:08:46.091: [& & CSSD][]clssgmWaitOnEventValue: after CmInfo State&&val 3, eval 1 waited 0
15:08:46.319: [& & CSSD][]clssnmvDHBValidateNCopy: node 1, tradedb1, has a disk HB, but no network HB, DHB has rcfg , wrtcnt, , LATS , lastSeqNo , uniqueness , timestamp /
15:08:47.091: [& & CSSD][]clssgmWaitOnEventValue: after CmInfo State&&val 3, eval 1 waited 0
15:08:47.319: [& & CSSD][]clssnmvDHBValidateNCopy: node 1, tradedb1, has a disk HB, but no network HB, DHB has rcfg , wrtcnt, , LATS , lastSeqNo , uniqueness , timestamp /
15:08:48.091: [& & CSSD][]clssgmWaitOnEventValue: after CmInfo State&&val 3, eval 1 waited 0
15:08:48.320: [& & CSSD][]clssnmvDHBValidateNCopy: node 1, tradedb1, has a disk HB, but no network HB, DHB has rcfg , wrtcnt, , LATS , lastSeqNo , uniqueness , timestamp /
15:08:48.584: [& & CSSD][]clssnmSendingThread: sending join msg to all nodes
15:08:48.584: [& & CSSD][]clssnmSendingThread: sent 5 join msgs to all nodes
15:08:49.092: [& & CSSD][]clssgmWaitOnEventValue: after CmInfo State&&val 3, eval 1 waited 0
15:08:49.321: [& & CSSD][]clssnmvDHBValidateNCopy: node 1, tradedb1, has a disk HB, but no network HB, DHB has rcfg , wrtcnt, , LATS , lastSeqNo , uniqueness , timestamp /
15:08:50.092: [& & CSSD][]clssgmWaitOnEventValue: after CmInfo State&&val 3, eval 1 waited 0
15:08:50.321: [& & CSSD][]clssnmvDHBValidateNCopy: node 1, tradedb1, has a disk HB, but no network HB, DHB has rcfg , wrtcnt, , LATS , lastSeqNo , uniqueness , timestamp /
15:08:51.060: [& & CSSD][]clssscSelect: cookie accept request 0x79f650
15:08:51.060: [& & CSSD][]clssgmAllocProc: (0x7f821c057070) allocated
15:08:51.060: [& & CSSD][]clssgmClientConnectMsg: properties of cmProc 0x7f821c057070 - 1,2,3,4,5
15:08:51.060: [& & CSSD][]clssgmClientConnectMsg: Connect from con(0x4fc4) proc(0x7f821c057070) pid(54988) version 11:2:1:4, properties: 1,2,3,4,5
15:08:51.060: [& & CSSD][]clssgmClientConnectMsg: msg flags 0x0000
15:08:51.062: [& & CSSD][]clssscSelect: cookie accept request 0x7f821c057070
15:08:51.062: [& & CSSD][]clssscevtypSHRCON: getting client with cmproc 0x7f821c057070
15:08:51.062: [& & CSSD][]clssgmRegisterClient: proc(4/0x7f821c057070), client(1/0x7f821c02f780)
15:08:51.062: [& & CSSD][]clssgmJoinGrock: global grock CRF- new client 0x7f821c02f780 with con 0x7f, requested num -1, flags 0x4000e00
15:08:51.062: [& & CSSD][]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been rece group name CRF-, member number -1, flags 0x4000e00
15:08:51.062: [& & CSSD][]clssgmDiscEndpcl: gipcDestroy 0x4ff3
15:08:51.063: [& & CSSD][]clssgmDeadProc: proc 0x7f821c057070
15:08:51.063: [& & CSSD][]clssgmDestroyProc: cleaning up proc(0x7f821c057070) con(0x4fc4) skgpid&&ospid 54988 with 0 clients, refcount 0
15:08:51.063: [& & CSSD][]clssgmDiscEndpcl: gipcDestroy 0x4fc4
15:08:51.092: [& & CSSD][]clssgmWaitOnEventValue: after CmInfo State&&val 3, eval 1 waited 0
15:08:51.322: [& & CSSD][]clssnmvDHBValidateNCopy: node 1, tradedb1, has a disk HB, but no network HB, DHB has rcfg , wrtcnt, , LATS , lastSeqNo , uniqueness , timestamp /
15:08:52.092: [& & CSSD][]clssgmWaitOnEventValue: after CmInfo State&&val 3, eval 1 waited 0
15:08:52.323: [& & CSSD][]clssnmvDHBValidateNCopy: node 1, tradedb1, has a disk HB, but no network HB, DHB has rcfg , wrtcnt, , LATS , lastSeqNo , uniqueness , timestamp /
15:08:52.572: [& & CSSD][]clssnmRcfgMgrThread: Local Join
15:08:52.572: [& & CSSD][]clssnmLocalJoinEvent: begin on node(2), waittime 193000
15:08:52.572: [& & CSSD][]clssnmLocalJoinEvent: set curtime () for my node
15:08:52.572: [& & CSSD][]clssnmLocalJoinEvent: scanning 32 nodes
15:08:52.572: [& & CSSD][]clssnmLocalJoinEvent: Node tradedb1, number 1, is in an existing cluster with disk state 3
15:08:52.572: [& & CSSD][]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
15:08:53.092: [& & CSSD][]clssgmWaitOnEventValue: after CmInfo State&&val 3, eval 1 waited 0
15:08:53.323: [& & CSSD][]clssnmvDHBValidateNCopy: node 1, tradedb1, has a disk HB, but no network HB, DHB has rcfg , wrtcnt, , LATS , lastSeqNo , uniqueness , timestamp /
15:08:53.585: [& & CSSD][]clssnmSendingThread: sending join msg to all nodes
15:08:53.585: [& & CSSD][]clssnmSendingThread: sent 5 join msgs to all nodes
15:08:54.092: [& & CSSD][]clssgmWaitOnEventValue: after CmInfo State&&val 3, eval 1 waited 0
15:08:54.324: [& & CSSD][]clssnmvDHBValidateNCopy: node 1, tradedb1, has a disk HB, but no network HB, DHB has rcfg , wrtcnt, , LATS , lastSeqNo , uniqueness , timestamp /
15:08:55.093: [& & CSSD][]clssgmWaitOnEventValue: after CmInfo State&&val 3, eval 1 waited 0
15:08:55.324: [& & CSSD][]clssnmvDHBValidateNCopy: node 1, tradedb1, has a disk HB, but no network HB, DHB has rcfg , wrtcnt, , LATS , lastSeqNo , uniqueness , timestamp /
15:08:56.067: [& & CSSD][]clssscSelect: cookie accept request 0x79f650
15:08:56.067: [& & CSSD][]clssgmAllocProc: (0x7f821c057070) allocated
15:08:56.067: [& & CSSD][]clssgmClientConnectMsg: properties of cmProc 0x7f821c057070 - 1,2,3,4,5
15:08:56.067: [& & CSSD][]clssgmClientConnectMsg: Connect from con(0x508c) proc(0x7f821c057070) pid(54988) version 11:2:1:4, properties: 1,2,3,4,5
15:08:56.067: [& & CSSD][]clssgmClientConnectMsg: msg flags 0x0000
15:08:56.068: [& & CSSD][]clssscSelect: cookie accept request 0x7f821c057070
15:08:56.068: [& & CSSD][]clssscevtypSHRCON: getting client with cmproc 0x7f821c057070
15:08:56.068: [& & CSSD][]clssgmRegisterClient: proc(4/0x7f821c057070), client(1/0x7f821c02fc50)
15:08:56.068: [& & CSSD][]clssgmJoinGrock: global grock CRF- new client 0x7f821c02fc50 with con 0x7fbb, requested num -1, flags 0x4000e00
15:08:56.068: [& & CSSD][]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been rece group name CRF-, member number -1, flags 0x4000e00
itpub.net All Right Reserved. 北京盛拓优讯信息技术有限公司版权所有　　　　
　北京市公安局海淀分局网监中心备案编号：10　广播电视节目制作经营许可证：编号（京）字第1149号