srvctl start database error: PRCR-1079 CRS-5017 “ORA-01017: invalid username/password; logon denied” sqlplus / as sysdba error: “ORA-12547: TNS:lost contact” issue caused by /var/tmp/.oracle

During Oracle DB 19.9 patching, hit 2 speared issue for opatch & srvctl, at the end all relate to DB login issue with /var/tmp/.oracle socket locks.

dataptch error for ORA-12547:

$ ./datapatch -verbose
SQL Patching tool version 19.9.0.0.0 Production on Fri Jan 29 18:44:56 2021
Copyright (c) 2012, 2020, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_20567_2021_01_29_18_44_56/sqlpatch_invocation.log

Connecting to database...
Error: prereq checks failed!
Database connect failed with: ORA-12547: TNS:lost contact (DBD ERROR: OCIServerAttach)

Please refer to MOS Note 1609718.1 and/or the invocation log
...

srvctl error with ORA-01017:

$ srvctl start database -db c19_dwhtest_x6e3
PRCR-1079 : Failed to start resource ora.c19_dwhtest_x6e3.db
CRS-5017: The resource action "ora.c19_dwhtest_x6e3.db start" encountered the following error: 
ORA-01017: invalid username/password; logon denied
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/hkx6e3db01/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.c19_dwhtest_x6e3.db' on 'hkx6e3db01' failed
CRS-2632: There are no more servers to try to place resource 'ora.c19_dwhtest_x6e3.db' on that would satisfy its placement policy

Solution:

Remove /var/tmp/.oracle folder to release socket locks. Exit and re-login to oracle session, startup DB then proceed the previous steps.

CAUTION: this action would terminate all the running oracle DB instances in the host. equal to kill pmon process or shutdown abort. Suggest run the command in oracle account, to prevent it messing grid/root processes.

$ rm -rf /var/tmp/.oracle/

Reference:

/var/tmp/.oracle — oracle-tech

Linux Account temporary locked by pam_tally2 after wrong password: ssh error “Permission denied, please try again.”

While login into some linux host with wrong password, the OS account might get temporarily locked per pam_tally2 configuration. The ssh will return “Permission denied” even later input the correct password, until the timeout period freed.

$ ssh oracle@172.23.5.221
oracle@172.23.5.221's password: 
Permission denied, please try again.
oracle@172.23.5.221's password: 
Permission denied, please try again.
oracle@172.23.5.221's password: 
Permission denied (publickey,password).

Solution:

While gained access into the host, execute below pam_tally2 command to reset the temporarily lock:

# pam_tally2 -u oracle -r
Login           Failures Latest failure     From
oracle              1    02/25/21 12:23:56  xxx.net

Modify the sshd file for changing the lock_time (seconds) if neccessary.

# cat /etc/pam.d/sshd
#%PAM-1.0
auth	   required	pam_sepermit.so
auth       substack     password-auth
auth       include      postlogin
auth       required     pam_tally2.so deny=5 onerr=fail lock_time=600 serialize
...

Reference:

Exadata: Exadata account locked, pam_tally2 and host_access_control (exadata-dba.blogspot.com)