ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE

This document describes how to fix "ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE" error in RAC database.

Error:
 

Solution:

Convert the database into Non-RAC and start the database in upgrade mode.

SQL> startup nomount
ORACLE instance started.

Total System Global Area 1140850688 bytes
Fixed Size            2923584 bytes
Variable Size          486540224 bytes
Database Buffers      637534208 bytes
Redo Buffers           13852672 bytes
SQL>

SQL> alter system set cluster_database=false scope=spfile;

System altered.

SQL>

Bounce database to take effect.

SQL> show parameter cluster

NAME                     TYPE     VALUE
------------------------------------ ----------- ------------------------------
cluster_database             boolean     TRUE
cluster_database_instances         integer     2
cluster_interconnects             string
SQL> shut immediate;
ORA-01507: database not mounted


ORACLE instance shut down.
SQL>

Start the database i
SQL> startup upgrade
ORACLE instance started.

Total System Global Area 1140850688 bytes
Fixed Size            2923584 bytes
Variable Size          486540224 bytes
Database Buffers      637534208 bytes
Redo Buffers           13852672 bytes
Database mounted.
Database opened.
SQL>

JAN 2020 PSU patch for 12c(12.1.0.2) Grid Home

This document describes how to apply January 2020 PSU patches on Grid Home.

Below are the steps to complete PSU patch on GRID Home

1. Download PSU patch
2. Upgrade opatch to latest version
3. Analyze the patches (prepatch analysis)
4. Apply patches on Grid home
5. Post patch steps

1. Download PSU patches:

Use below url and download database and GI PSU.

https://www.oracle.com/security-alerts/cpujan2020.html

Note: In my case I have downloaded combo OJVM patches for Grid (30464119) and DB (30463684).


2. Upgrade opatch to latest version 

Take backup of OPatch folder which is under $GRID_HOME

Unzip the patch under $GRID_HOME

grid@server1 grid_home]$ unzip p6880880_180000_Linux-x86-64.zip


3. Analyze the patches (prepatch analysis)

SCP downloaded GI patch to server and unzip it. Verify the readme.txt file for any pre-requisite steps and analyze it for conflicts. Proceed for patching if no conflicts come. We have to raise an SR for merge patch if we get conflicts.

In my case, Non-shared oracle homes and grid homes across the 2 nodes. So we are applying the patch in rolling passion.  No ACFS file system available.

Note: We have to unmount ACFS file system if it is available in our environment before patching and mount it after apply the patch.

Run analyze command as root user and find conflicts if any. Output looks like below.

[root@server1 ~]# /u01/app/grid/12102/grid_home/OPatch/opatchauto apply /u02/software/19C_Upgrade/JAN20_CPU_12102/GI/30463691/30464119 -analyze -oh /u01/app/grid/12102/grid_home

OPatchauto session is initiated at Sat May  2 09:13:28 2020

System initialization log file is /u01/app/grid/12102/grid_home/cfgtoollogs/opatchautodb/systemconfig2020-05-02_09-13-36AM.log.

Session log file is /u01/app/grid/12102/grid_home/cfgtoollogs/opatchauto/opatchauto2020-05-02_09-13-50AM.log
The id for this session is CZ2S

Executing OPatch prereq operations to verify patch applicability on home /u01/app/grid/12102/grid_home
Patch applicability verified successfully on home /u01/app/grid/12102/grid_home

OPatchAuto successful.

--------------------------------Summary--------------------------------

Analysis for applying patches has completed successfully:

Host:server1
CRS Home:/u01/app/grid/12102/grid_home
Version:12.1.0.2.0


==Following patches were SUCCESSFULLY analyzed to be applied:

Patch: /u02/software/19C_Upgrade/JAN20_CPU_12102/GI/30463691/30464119/26983807
Log: /u01/app/grid/12102/grid_home/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-02_09-14-22AM_1.log

Patch: /u02/software/19C_Upgrade/JAN20_CPU_12102/GI/30463691/30464119/30304402
Log: /u01/app/grid/12102/grid_home/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-02_09-14-22AM_1.log

Patch: /u02/software/19C_Upgrade/JAN20_CPU_12102/GI/30463691/30464119/30304434
Log: /u01/app/grid/12102/grid_home/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-02_09-14-22AM_1.log

Patch: /u02/software/19C_Upgrade/JAN20_CPU_12102/GI/30463691/30464119/30340202
Log: /u01/app/grid/12102/grid_home/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-02_09-14-22AM_1.log


OPatchauto session completed at Sat May  2 09:15:19 2020
Time taken to complete the session 1 minute, 51 seconds
[root@server1 ~]#


4. Apply patch on Grid Home

Shutdown the  database. Cluster services should be up and running while applying grid patch. Apply the patch as root user.

[root@server1 ~]# /u01/app/grid/12102/grid_home/OPatch/opatchauto apply /u02/software/19C_Upgrade/JAN20_CPU_12102/GI/30463691/30464119 -oh /u01/app/grid/12102/grid_home

OPatchauto session is initiated at Sat May  2 09:55:24 2020

System initialization log file is /u01/app/grid/12102/grid_home/cfgtoollogs/opatchautodb/systemconfig2020-05-02_09-55-33AM.log.

Session log file is /u01/app/grid/12102/grid_home/cfgtoollogs/opatchauto/opatchauto2020-05-02_09-55-54AM.log
The id for this session is QHK5

Executing OPatch prereq operations to verify patch applicability on home /u01/app/grid/12102/grid_home
Patch applicability verified successfully on home /u01/app/grid/12102/grid_home


Bringing down CRS service on home /u01/app/grid/12102/grid_home
Prepatch operation log file location: /u01/app/grid/12102/grid_home/cfgtoollogs/crsconfig/crspatch_server1_2020-05-02_09-57-32AM.log
CRS service brought down successfully on home /u01/app/grid/12102/grid_home


Start applying binary patch on home /u01/app/grid/12102/grid_home
Binary patch applied successfully on home /u01/app/grid/12102/grid_home


Starting CRS service on home /u01/app/grid/12102/grid_home
Postpatch operation log file location: /u01/app/grid/12102/grid_home/cfgtoollogs/crsconfig/crspatch_server1_2020-05-02_10-29-36AM.log
CRS service started successfully on home /u01/app/grid/12102/grid_home

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:server1
CRS Home:/u01/app/grid/12102/grid_home
Version:12.1.0.2.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u02/software/19C_Upgrade/JAN20_CPU_12102/GI/30463691/30464119/26983807
Log: /u01/app/grid/12102/grid_home/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-02_10-02-07AM_1.log

Patch: /u02/software/19C_Upgrade/JAN20_CPU_12102/GI/30463691/30464119/30304402
Log: /u01/app/grid/12102/grid_home/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-02_10-02-07AM_1.log

Patch: /u02/software/19C_Upgrade/JAN20_CPU_12102/GI/30463691/30464119/30304434
Log: /u01/app/grid/12102/grid_home/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-02_10-02-07AM_1.log

Patch: /u02/software/19C_Upgrade/JAN20_CPU_12102/GI/30463691/30464119/30340202
Log: /u01/app/grid/12102/grid_home/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-02_10-02-07AM_1.log


OPatchauto session completed at Sat May  2 10:45:56 2020
Time taken to complete the session 50 minutes, 32 seconds
[root@server1 ~]#

5. Post patch steps

Verify Readme.txt for any post patch installation steps and proceed.

OPATCHAUTO-72043: Patch collection failed JAN 2020 GRID PSU patch

This document describes the issue which we had while applying JAN 2020 GRID PSU and how to resolve it.


Error:

[root@server1 ~]# /u01/app/grid/12102/grid_home/OPatch/opatchauto apply /u02/software/19C_Upgrade/JAN20_CPU_12102/GI/30463691/30464119 -analyze -oh /u01/app/grid/12102/grid_home

OPatchauto session is initiated at Sat May  2 08:57:01 2020

System initialization log file is /u01/app/grid/12102/grid_home/cfgtoollogs/opatchautodb/systemconfig2020-05-02_08-58-31AM.log.

Session log file is /u01/app/grid/12102/grid_home/cfgtoollogs/opatchauto/opatchauto2020-05-02_08-59-24AM.log
The id for this session is VSF5
OPATCHAUTO-72043: Patch collection failed.
OPATCHAUTO-72043: Failed to create bundle patch object.
OPATCHAUTO-72043: Please verify the patch supplied.
OPatchAuto failed.

OPatchauto session completed at Sat May  2 09:00:18 2020
Time taken to complete the session 3 minutes, 25 seconds

 opatchauto failed with error code 42
[root@server1 ~]# 

Error Details:

2020-05-02 09:08:36,646 SEVERE [1] com.oracle.glcm.patch.auto.OPatchAuto - OPatchAuto failed.
com.oracle.glcm.patch.auto.OPatchAutoException: OPATCHAUTO-72043: Patch collection failed.
OPATCHAUTO-72043: Failed to create bundle patch object.
OPATCHAUTO-72043: Please verify the patch supplied.
at com.oracle.glcm.patch.auto.db.integration.model.plan.PatchingProcessInitializer.initializePatchPackageBag(PatchingProcessInitializer.java:97)
at com.oracle.glcm.patch.auto.db.integration.model.plan.PatchingProcessInitializer.processInit(PatchingProcessInitializer.java:66)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBCommonSupport.initializePatchData(DBCommonSupport.java:126)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBBaseProductSupport.initializePatchData(DBBaseProductSupport.java:403)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBCommonSupport.loadTopology(DBCommonSupport.java:163)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBBaseProductSupport.loadTopology(DBBaseProductSupport.java:190)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBProductSupport.loadTopology(DBProductSupport.java:69)
at com.oracle.glcm.patch.auto.OPatchAuto.loadTopology(OPatchAuto.java:1732)
at com.oracle.glcm.patch.auto.OPatchAuto.prepareOrchestration(OPatchAuto.java:730)
at com.oracle.glcm.patch.auto.OPatchAuto.orchestrate(OPatchAuto.java:397)
at com.oracle.glcm.patch.auto.OPatchAuto.orchestrate(OPatchAuto.java:344)
at com.oracle.glcm.patch.auto.OPatchAuto.main(OPatchAuto.java:212)
Caused by: com.oracle.glcm.patch.auto.db.integration.model.productsupport.patch.PatchCollectionException:
at com.oracle.glcm.patch.auto.db.framework.core.patch.OPatchSingletonPatchValidatorAndGenerator.generate(OPatchSingletonPatchValidatorAndGenerator.java:112)
at com.oracle.glcm.patch.auto.db.framework.core.patch.CompositePatchValidatorAndGenerator.generate(CompositePatchValidatorAndGenerator.java:164)
at oracle.dbsysmodel.patchsdk.PatchFactory.getInstance(PatchFactory.java:246)
at com.oracle.glcm.patch.auto.db.framework.core.patch.impl.PatchPackageFactoryImpl.getPatchPackageFromDir(PatchPackageFactoryImpl.java:75)
at com.oracle.glcm.patch.auto.db.product.patch.BundlePatchObject.prcoessPatch(BundlePatchObject.java:131)
at com.oracle.glcm.patch.auto.db.product.patch.BundlePatchObject.createPatchBagForPatchLocation(BundlePatchObject.java:124)
at com.oracle.glcm.patch.auto.db.product.patch.BundlePatchObject.<init>(BundlePatchObject.java:77)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.patch.PatchInformationInitializer.createPatch(PatchInformationInitializer.java:43)

at com.oracle.glcm.patch.auto.db.integration.model.plan.PatchingProcessInitializer.initializePatchPackageBag(PatchingProcessInitializer.java:80)


Solution:

This error came due to insufficient privileges on patch downloaded directory. Issue got resolved after giving 775 privileges on patch directory.


[root@server1 19C_Upgrade]# chmod -R 775 JAN20_CPU_12102/GI/30463691/

ORA-02085: database link GOLD_TO_SILVER connects to SILVER

This post describes how to resolve "ORA-02085: database link GOLD_TO_SILVER connects to SILVER" issue.


Error:

SQL> select count(*) from dual@GOLD_TO_SILVER;
select count(*) from dual@GOLD_TO_SILVER
                          *
ERROR at line 1:
ORA-02085: database link GOLD_TO_SILVER connects to SILVER


SQL>

Solution:

This error may occurs when global_names set to "TRUE"

SQL> show parameter global_name

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

global_names                         boolean     TRUE

SQL> show parameter db_name

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_name                              string      GOLD

SQL> alter system set global_names=FALSE scope=both;

System altered.

SQL> show parameter global_name

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

global_names                         boolean     FALSE

SQL> select count(*) from dual@GOLD_TO_SILVER;

  COUNT(*)

----------

         1

SQL>

Thanks for going through this post..................

TNS-01151: Missing listener name, GOLD, in LISTENER.ORA

This document describes how to resolve "TNS-01151: Missing listener name, GOLD, in LISTENER.ORA" error.

Error:

[oracle@ebsdb GOLD_ebsdb]$ lsnrctl start GOLD
LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 03-OCT-2019 14:03:58
Copyright (c) 1991, 2014, Oracle.  All rights reserved.
Starting /u01/app/GOLD/12.1.0//bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Log messages written to /u01/app/GOLD/12.1.0/log/diag/tnslsnr/ebsdb/gold/alert/log.xml
TNS-01151: Missing listener name, GOLD, in LISTENER.ORA

Listener failed to start. See the error message(s) above...
[oracle@ebsdb GOLD_ebsdb]$

This error occurs when service name or SID is incorrect or missing in listener.ora file which is under $TNS_ADMIN($ORACLE_HOME/network/admin).

                                                               (OR)

Incorrect environment settings of $TNS_ADMIN

Solution:

In my case listener.ora having correct entries. But got error due to incorrect settings of TNS_ADMIN.

listener.ora:

GOLD =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = ebsdb.apps.com)(PORT = 1521))
       )
     )

SID_LIST_GOLD =
  (SID_LIST =
    (SID_DESC =
      (ORACLE_HOME= /u01/app/GOLD/12.1.0)
      (SID_NAME = GOLD)
        )
     )

STARTUP_WAIT_TIME_GOLD = 0
CONNECT_TIMEOUT_GOLD = 10
TRACE_LEVEL_GOLD = OFF

LOG_DIRECTORY_GOLD = /u01/app/GOLD/12.1.0/network/admin
LOG_FILE_GOLD = GOLD
TRACE_DIRECTORY_GOLD = /u01/app/GOLD/12.1.0/network/admin
TRACE_FILE_GOLD = GOLD
ADMIN_RESTRICTIONS_GOLD = ON
SUBSCRIBE_FOR_NODE_DOWN_EVENT_GOLD = OFF

Set TNS_ADMIN to correct path to fix the issue.

Ex:

$export TNS_ADMIN= /u01/app/GOLD/12.1.0/network/admin





Thanks for going to this post......

SEVERE: [FATAL] [INS-10102] Installer initialization failed.

Getting below error while installing EBS R12.2.0 in Linux server.

Click on warning button to see the error.

Installer logs having below error message. We can find error logs in inventory location. In my case "/u01/app/oraInventory/logs". 

Error 1:
SEVERE: [FATAL] [INS-10102] Installer initialization failed.
   CAUSE: An unexpected error occured while initializing the Installer.
   ACTION: Contact Oracle Support Services or refer logs
   SUMMARY:
       - [INS-21003] Installer has detected that an invalid inventory pointer location file was specified..
Refer associated stacktrace #oracle.install.commons.util.exception.DefaultErrorAdvisor:37
INFO: Advice is ABORT
SEVERE: Unconditional Exit
INFO: Adding ExitStatus FAILURE to the exit status set
INFO: Finding the most appropriate exit status for the current application
INFO: Exit Status is -1
INFO: Shutdown Oracle Database 12c Release 1 Installer
[root@ebsdb logs]# 

Error 2:

Found below error in "/tmp/OraInstall2019-09-26_02-55-57PM/oraInstall2019-09-26_02-55-57PM.err"

---# Begin Stacktrace #---------------------------
ID: oracle.install.commons.util.exception.DefaultErrorAdvisor:37
oracle.install.commons.base.driver.common.InstallerException: [INS-10102] Installer initialization failed.
        at oracle.install.commons.base.driver.common.Installer.run(Installer.java:324)
        at oracle.install.ivw.common.util.OracleInstaller.run(OracleInstaller.java:106)
        at oracle.install.ivw.db.driver.DBInstaller.run(DBInstaller.java:142)
        at oracle.install.commons.util.Application.startup(Application.java:921)
        at oracle.install.commons.flow.FlowApplication.startup(FlowApplication.java:181)
        at oracle.install.commons.flow.FlowApplication.startup(FlowApplication.java:198)
        at oracle.install.commons.base.driver.common.Installer.startup(Installer.java:355)
        at oracle.install.ivw.db.driver.DBInstaller.startup(DBInstaller.java:130)
        at oracle.install.ivw.db.driver.DBInstaller.main(DBInstaller.java:161)
Caused by: oracle.install.commons.base.driver.common.SetupDriverException: [INS-21003] Installer has detected that an invalid inventory pointer location file was specified.
        at oracle.install.driver.oui.OUIInstallDriver.load(OUIInstallDriver.java:393)
        at oracle.install.ivw.db.driver.DBSetupDriver.load(DBSetupDriver.java:225)
        at oracle.install.commons.base.driver.common.Installer.run(Installer.java:306)
        ... 8 more

---# End Stacktrace #-----------------------------


We will get below error if we ignore this error and proceed with the installation.

Solution:

Error is with some permissions and found group details are not present in "/etc/oraInst.loc".

[root@ebsdb logs]# cat /etc/oraInst.loc
inventory_loc=/u01/app/oraInventory
[root@ebsdb logs]#

Added group details and after adding "/etc/oraInst.loc" file looks like below.

[root@ebsdb logs]# cat /etc/oraInst.loc
inventory_loc=/u01/app/oraInventory
inst_group=dba
[root@ebsdb logs]#

Thanks for reviewing this post...............

Decrypt Weblogic Admin password

This post describes how to decrypt weblogic password using python script.

Assumption: We have boot.properties file which contains encrypted password.

Step 1: Get encrypted password from boot.properties file.

Go to below location and cat boot.properties to get encrypted weblogic password.

cd $DOMAIN_HOME/servers/Adminserver/security

[oracle@server1 AdminServer]$ cd /u01/app/Middleware/user_projects/domains/base_domain/servers/AdminServer/security/

[oracle@server1 security]$ cat boot.properties
# Generated by Configuration Wizard on Thu Jul 11 13:11:09 IST 2019
username={AES}3Ryhu7KAt1t74cqx/UFhcwhq675U5gxfiChMAqgdCW8=
password={AES}s+0t9pgYnGBXRtLxtzp0grC7qbQDjobTZlLc/Tm/AwA=


Step 2: Create python script in the server with below parameters.

from weblogic.security.internal import *
from weblogic.security.internal.encryption import *

passwd = "<encrypted password from boot.properties file"
secPath = "security folder path under DOMAIN_HOME"
encService = SerializedSystemIni.getEncryptionService(secPath)
coeService = ClearOrEncryptedService(encService)
print "Password is : " + coeService.decrypt(passwd)


[oracle@server1 security]$ vi DecryptPassword.py

from weblogic.security.internal import *

from weblogic.security.internal.encryption import *

passwd = "{AES}s+0t9pgYnGBXRtLxtzp0grC7qbQDjobTZlLc/Tm/AwA="

secPath = "/u01/app/Middleware/user_projects/domains/base_domain/security"

encService = SerializedSystemIni.getEncryptionService(secPath)

coeService = ClearOrEncryptedService(encService)

print "Password is : " + coeService.decrypt(passwd)

:wq!

Step 3: Get decrypted password using wlst script.

execute below command.

[oracle@server1 security]$ /u01/app/Middleware/oracle_common/common/bin/wlst.sh DecryptPassword.py

 

Output: 

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0

Initializing WebLogic Scripting Tool (WLST) ...

Jython scans all the jar files it can find at first startup. Depending on the system, this process may take a few minutes to complete, and WLST may not return a prompt right away.

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Password is : welcome123

[oracle@server1 security]$

Thanks for your patience to view this post................