Thursday, October 22, 2015

Install ORACLE PSU in an Active Data Guard Environment with FAST START FAILOVER monitored by Data Guard Broker Observer

This procedure will describe installing the 11.2.0.3.15 Oracle PSU on top of 11.2.0.3 with no other PSU / CPU patch installed. 


Patch p21150891_112030_Linux-x86-64.zip contains the Oracle PSU 11.2.0.3.15 (20760997) and the 11.2.0.3.5 (21068553) JVM PSU.

The procedure applies for both, but here is about the installation of the Oracle PSU 11.2.0.3.15 (20760997).

Only one off patch has been applied to both Primary & Standby Oracle Homes.

Latest opatch version has been installed in both Oracle Homes.

It can be used as a reference for 10gR2 up to 11gR2 (11.2.0.4) for any PSU , CPU or JVM patch.

ATTENTION : Its not about Standby First Installabale Patch Set , even the 11.2.0.3.15 is one!

Lets start:


1. Configured Environment


- Primary DB : apfpl
- Standby DB : apfpl2
- Active Data Guard (Phisical Standby in READ ONLY mode) 
- Data Guard Broker is used to manage the configuration - (Configured for Maximum Performance)
- Fast Start Failover has been enabled and Observer has been started on a seperate machine from the DB servers


2. PSU 11.2.0.3.15 Apply Procedure

A. Download & Unzip Patch on the DB servers and check for conflicts ith other patches

$ cd 21150891/20760997/
$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./ 

!! If opatch mentions conflict remove the one off patches. !!


a. On Primary DB Server 

$ $ORACLE_HOME/OPatch/opatch rollback -id 12919564 -connectString apfpl:sys:apfpl 

b. On Standby DB Server 

$ $ORACLE_HOME/OPatch/opatch rollback -id 12919564 -connectString apfpl:sys:apfpl


B. Disable Active Data Guard usign Data Guard Broker



a. Stop redo apply on Standby DB (MRP Process)


$ dgmgrl
DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys@apfpl
Password:
Connected.
DGMGRL> show configuration;

Configuration - DG_APFPL_FSF

  Protection Mode: MaxPerformance
  Databases:
    apfpl  - Primary database
    apfpl2 - (*) Physical standby database

Fast-Start Failover: ENABLED

Configuration Status:
SUCCESS

DGMGRL> edit database apfpl2 set state = 'apply-off';
Succeeded.
DGMGRL> show configuration;

Configuration - DG_APFPL_FSF

  Protection Mode: MaxPerformance
  Databases:
    apfpl  - Primary database
      Warning: ORA-16829: fast-start failover configuration is lagging

    apfpl2 - (*) Physical standby database
      Warning: ORA-16829: fast-start failover configuration is lagging

Fast-Start Failover: ENABLED

Configuration Status:
WARNING

DGMGRL>


b. Shutdown the physical standby database and start it to mount state

$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Oct 22 10:45:16 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL>shutdown immediate
...
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>startup mount
....
Database mounted
SQL> exit;


c. Connect to DG Broker again and enable redo apply on the Standby

$ dgmgrl
DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys@apfpl
Password:
Connected.
DGMGRL> edit database apfpl2 set state = 'apply-on';

!! Wait a bit and then !!

DGMGRL> show configuration

Configuration - DG_APFPL_FSF

  Protection Mode: MaxPerformance
  Databases:
    apfpl  - Primary database
    apfpl2 - (*) Physical standby database

Fast-Start Failover: ENABLED

Configuration Status:
SUCCESS
DGMGRL>exit;


C. On the observer machine connect with dgmgrl  



a. Stop observer & disable Fast Start Failover

$ dgmgrl
DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys@apfpl
Password:
Connected.
DGMGRL> stop observer;
Done.
DGMGRL> show configuration;

Configuration - DG_APFPL_FSF

  Protection Mode: MaxPerformance
  Databases:
    apfpl  - Primary database
      Warning: ORA-16819: fast-start failover observer not started

    apfpl2 - (*) Physical standby database
      Warning: ORA-16819: fast-start failover observer not started

Fast-Start Failover: ENABLED

Configuration Status:
WARNING
DGMGRL> disable fast_start failover;
Disabled.
DGMGRL> show configuration;

Configuration - DG_APFPL_FSF

  Protection Mode: MaxPerformance
  Databases:
    apfpl  - Primary database
    apfpl2 - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS


b. Disable Primary Site Redo Log transport to Standby DB

DGMGRL> edit database apfpl set state='TRANSPORT-OFF';
Succeeded.
DGMGRL> show database apfpl;

Database - apfpl

  Role:            PRIMARY
  Intended State:  TRANSPORT-OFF
  Instance(s):
    apfpl

Database Status:
SUCCESS
DGMGRL> exit


D. Apply Patch on Standby Database Server 



a. Stop All Oracle Processes on Standby Database Server

$ export ORACLE_SID=apfpl
$ lsnrctl stop
The command completed successfully
$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Oct 22 10:45:16 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL>shutdown immediate
...
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>exit;


b. Install PSU on Standby Server Oracle Home

$ cd 21150891/20760997
$ $ORACLE_HOME/OPatch/opatch apply
....
OPatch succeeded.

!! If opatch completes with warnings , please check in Oracle Support site.    !!
!! In most of cases is ok to proceed, otherwise follow Oracle Support Advice. !!

c. Start Listener and Standby Database in mount state

$ lsnrctl start
The command completed successfully
$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Oct 22 10:45:16 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> startup mount
Database mounted.

E. Apply Patch on Primary Database Server 

a. Stop All Oracle Processes on Primary Database Server

$ export ORACLE_SID=apfpl
$ lsnrctl stop
The command completed successfully
$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Oct 22 10:45:16 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL>shutdown immediate
...
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>exit;


b. Install PSU on Primary Server Oracle Home

$ cd 21150891/20760997
$ $ORACLE_HOME/OPatch/opatch apply
....
OPatch succeeded.


F. Start Primary in upgrade state and run the upgrade script

$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Oct 22 10:45:16 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> startup upgrade
Database opened.
SQL> @?/rdbms/admin/catbundle.sql psu apply

!! Check the log report in the end of the script for errros !!

SQL> shutdown immediate
Database Closeed
SQL>startup
Database opened
SQL>@?/rdbms/admin/utlrp
SQL>exit;


G. On the Observer Machine re-enable Primary Site archiving to Standby Site

$ dgmgrl
DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys@apfpl
Password:
Connected.
DGMGRL> edit database apfpl set state='TRANSPORT-ON';
Succeeded.

!! Wait for Standby Database to Re-sync with Primary by applying the upgrade redo logs !!
!! Check the Standby alert log until you see the following                                                         !!

.......
Media Recovery Waiting for thread 1 sequence 33 (in transit)
Recovery of Online Redo Log: Thread 1 Group 4 Seq 33 Reading mem 0
  Mem# 0: /oraslow/oradata/apfpl2/apfdbou2/standby_redo01.log


H. Re-enable Active Data Guard & Fast Failover of the configuration



a. Stop redo apply on Standby

DGMGRL> edit database apfpl2 set state = 'apply-off';
Succeeded.
DGMGRL>show configuration


b. Open the physical standby database in read-only mode 

$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Oct 22 10:45:16 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> alter database open read only;
Database opened.


c. Re-enable redo apply on Standby

DGMGRL> edit database apfpl2 set state = 'apply-on';
Succeeded.


d. Enable Fast Start Failover

DGMGRL> enable fast_start failover;
Enabled.


e. Start the Observer 

DGMGRL> start observer;
Started.

f. Check the configuration

DGMGRL> show configuration verbose;

Configuration - DG_APFPL_FSF

  Protection Mode: MaxPerformance
  Databases:
    apfpl  - Primary database
    apfpl2 - (*) Physical standby database

  (*) Fast-Start Failover target

  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'

Fast-Start Failover: ENABLED

  Threshold:        30 seconds
  Target:           apfpl2
  Observer:         s-eacea-rh-mgmtdbo01-p
  Lag Limit:        30 seconds
  Shutdown Primary: TRUE
  Auto-reinstate:   TRUE

Configuration Status:
SUCCESS
DGMGRL>exit;


YOU ARE DONE !!


No comments:

Post a Comment