GI – TonieLook's Data Blog

Symptoms:

opatchauto resume hit PROC-00053.

[root@hkx6e3db02 ~]# export ORACLE_HOME=/u01/app/19.4.0/grid
[root@hkx6e3db02 ~]# /u01/app/19.4.0/grid/OPatch/opatchauto resume
...
Cluster registry server error. Server error stack -
PROC-00053: The configured patch level, release patch level and the patch level on the other active cluster nodes are not all the same.
- end of server error stackCRS-2883: Resource 'ora.crsd' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-41053: checking Oracle Grid Infrastructure for file permission issues
PRVH-0116 : Path "/u01/app/19.9.0/grid/crs/install/cmdllroot.sh" with permissions "rw-r--r--" does not have execute permissions for the owner, file's group, and others on node "hkx6e3db02".
PRVG-2032 : Group of file "/u01/app/19.9.0/grid/crs/install/cmdllroot.sh" did not match the expected value on node "hkx6e3db02". [Expected = "oinstall(1001)" ; Found = "root(0)"]
CRS-4000: Command Start failed, or completed with errors.
2021/01/26 16:48:20 CLSRSC-117: Failed to start Oracle Clusterware stack

Check GI alert log: failed to start CRSD: CRS-8503 [clssgsgrpstat()+952].

[grid@hkx6e3db02 ~]$ tail -f /u01/app/grid/diag/crs/hkx6e3db02/crs/trace/alert.log
...
2021-01-26 16:48:10.132 [OCTSSD(348843)]CRS-2401: The Cluster Time Synchronization Service started on host hkx6e3db02.
2021-01-26 16:48:11.975 [CRSD(352902)]CRS-8500: Oracle Clusterware CRSD process is starting with operating system process ID 352902
2021-01-26 16:48:16.161 [CRSD(352902)]CRS-8503: Oracle Clusterware process CRSD with operating system process ID 352902 experienced fatal signal or exception code 11.
2021-01-26T16:48:16.188364+08:00
Errors in file /u01/app/grid/diag/crs/hkx6e3db02/crs/trace/crsd.trc  (incident=529):
CRS-8503 [clssgsgrpstat()+952] [Signal/Exception: 11] [Instruction Addr: 0x7f06ca1501cb] [Memory Addr: 0x18] [] [] [] [] [] [] [] []
Incident details in: /u01/app/grid/diag/crs/hkx6e3db02/crs/incident/incdir_529/crsd_i529.trc
...

Check crs releasepatch not the same with the other node:

[root@hkx6e3db02 ~]# crsctl query crs releasepatch
Oracle Clusterware release patch level is [482815673] and the complete list of patches [29391849 29401763 29517242 29517247 29585399 29834717 29850993 29851014 30177140 ] have been applied on the local node. The release patch string is [19.4.0.0.0].
[root@hkx6e3db01 ~]# crsctl query crs releasepatch
Oracle Clusterware release patch level is [869426834] and the complete list of patches [29517242 29517247 29585399 31771877 31772784 31773437 31780966 ] have been applied on the local node. The release patch string is [19.9.0.0.0].

Solution:

Shutdown crs from the other node temporally during the patching, CRSD able to startup and proceed.

[root@hkx6e3db01 ~]# crsctl stop crs
[root@hkx6e3db02 ~]# /u01/app/19.4.0/grid/OPatch/opatchauto resume

GI alert log:

2021-01-26 17:06:41.341 [OCTSSD(327708)]CRS-2401: The Cluster Time Synchronization Service started on host hkx6e3db02.
2021-01-26 17:07:04.940 [CRSD(335160)]CRS-8500: Oracle Clusterware CRSD process is starting with operating system process ID 335160
2021-01-26 17:07:07.790 [CRSD(335160)]CRS-1012: The OCR service started on node hkx6e3db02.
2021-01-26 17:07:07.839 [CRSD(335160)]CRS-1201: CRSD started on node hkx6e3db02.

Reference:

Oracle RAC RU rolling patching from 19.3 to 19.9.0.0.201020(Patch 31750108) – Install/Configuration – 한국오라클사용자그룹 (koreaoug.org)

[root@hkx6e3db02 ~]# export ORACLE_HOME=/u01/app/19.4.0/grid [root@hkx6e3db02 ~]# /u01/app/19.4.0/grid/OPatch/opatchauto resume ... MGTCA-1109 : failed to determine Oracle Base location PRCT-1011 : Failed to run "orabase". Detailed error: 2021/01/26 17:57:23 CLSRSC-180: An error occurred while executing the command '/u01/app/19.9.0/grid/bin/mgmtca applysql' After fixing the cause of failure Run opatchauto resume ] OPATCHAUTO-68061: The orchestration engine failed. OPATCHAUTO-68061: The orchestration engine failed with return code 1 OPATCHAUTO-68061: Check the log for more details. OPatchAuto failed.

[grid@hkx6e3db02 ~]$ mgmtca oracle.cluster.mgmtcommon.MgmtDBException: MGTCA-1109 : failed to determine Oracle Base location at oracle.cluster.mgmtcommon.MgmtUtil.getCrsHomeOracleBase(MgmtUtil.java:518) at oracle.cluster.mgmtca.Mgmtca.main(Mgmtca.java:340) Caused by: PRCT-1011 : Failed to run "orabase". Detailed error: at oracle.cluster.deployment.ClusterwareInfo.getCRSHomeOracleBase(ClusterwareInfo.java:6486) at oracle.cluster.mgmtcommon.MgmtUtil.getCrsHomeOracleBase(MgmtUtil.java:513) ...

[grid@hkx6e3db02 ~]$ which orabase /u01/app/19.9.0/grid/bin/orabase [grid@hkx6e3db02 ~]$ orabase /u01/app/grid [root@hkx6e3db02 ~]# which orabase /u01/app/19.9.0/grid/bin/orabase [root@hkx6e3db02 ~]# orabase /u01/app/19.4.0/grid

cat /u01/app/19.9.0/grid/install/orabasetab #orabasetab file is used to track Oracle Home associated with Oracle Base #/u01/app/19.4.0/grid:/u01/app/grid:OraGI19Home1:N: /u01/app/19.9.0/grid:/u01/app/grid:OraHome1:N:

Tag: GI

GI patch to 19.9 PROC-00053: The configured patch level, release patch level and the patch level on the other active cluster nodes are not all the same.

GI 19c opatchauto resume: MGTCA-1109 : failed to determine Oracle Base location. PRCT-1011 : Failed to run “orabase”.