Thursday, October 23, 2014

Active-passive High Availability Solution for Oracle Database without Data Guard

Here is a very useful solution to create an Oracle Standby DB without using Data Guard (DG).
Oracle DG  does not supported in 10g & 11g Standard  Editions.

Part1 : http://wadhahdaouehi.tn/2014/10/part1-active-passive-high-availability-solution-for-oracle-database-without-dataguard/

Part2: http://wadhahdaouehi.tn/2014/10/part-2-active-passive-high-availability-solution-for-oracle-database-without-data-guard/


Wednesday, August 6, 2014

How to open a Physical Standby as Primary in an (Active) Data Guard Configuration

Here are the steps to open a Physical Standby as a new independent Database by removing (Active) Data Guard & Data Guard Broker (if exists) Configuration


1. Check the status of the DG COnfiguration

Log on to primary and standby and run in SQLPLUS:

On Primary :

SQL> SELECT OPEN_MODE,PROTECTION_MODE,DATABASE_ROLE FROM V$DATABASE;

OPEN_MODE            PROTECTION_MODE      DATABASE_ROLE
-------------------- -------------------- ----------------
READ WRITE           MAXIMUM PERFORMANCE  PRIMARY


On Standby :

SQL> SELECT OPEN_MODE,PROTECTION_MODE,DATABASE_ROLE FROM V$DATABASE;

OPEN_MODE            PROTECTION_MODE      DATABASE_ROLE
-------------------- -------------------- ----------------
READ WRITE           MAXIMUM PERFORMANCE  PRIMARY


Protection mode should be MAXIMUM PERFORMANCE as above.

If not you have to set it on Primary 

SQL> ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE PERFORMANCE;

If you use DG Broker , use DGMGRL login to primary DB and set it as follows:

$ dgmgrl
DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys@myprimary
Password:
Connected.

DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MAXPERFORMANCE;

Now you will have to remove the DG Broker configuration:

DGMGRL> show configuration verbose;

Configuration - DG_FSF

  Protection Mode: MaxPerformance
  Databases:
    myprimary - Primary database
    mystandby - (*) Physical standby database

  (*) Fast-Start Failover target

  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'

Fast-Start Failover: ENABLED

  Threshold:        30 seconds
  Target:           eoletbou2
  Observer:         s-eacea-rh-mgmtdbo01-p
  Lag Limit:        30 seconds
  Shutdown Primary: TRUE
  Auto-reinstate:   TRUE

Configuration Status:
SUCCESS

DGMGRL>

Stop Observer
DGMGRL> stop observer;

Disable Fast Start Failover (Active Dataguard)
DGMGRL> DISABLE FAST_START FAILOVER;
Disabled.

Remove the DG configuration
DGMGRL> REMOVE CONFIGURATION;
Removed configuration

DGMGRL>exit;


2. On the primary and standby database set dg_broker_start=false:


SQL> alter system set dg_broker_start=false scope=both;

System altered.

SQL> show parameter dg_broker

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
dg_broker_config_file1               string      /oraslow/app/oracle/product/11
                                                 .2.0/dbhome_2/dbs/dr1myprimary.dat
dg_broker_config_file2               string      /oraslow/app/oracle/product/11
                                                 .2.0/dbhome_2/dbs/dr2myprimary
                                                 .dat
dg_broker_start                      boolean     FALSE


3. On both system rename the metadata files shown by the previous command

SQL> ! mv /oraslow/app/oracle/product/11.2.0/dbhome_2/dbs/dr1myprimary.dat.dat /oraslow/app/oracle/product/11.2.0/dbhome_2/dbs/dr1myprimary.back

SQL> ! mv /oraslow/app/oracle/product/11.2.0/dbhome_2/dbs/dr2myprimary.dat /oraslow/app/oracle/product/11.2.0/dbhome_2/dbs/dr2myprimary.back

4. Check the values for the following prameters in both primary and stanby and unset/reset if needed.

ATTENTION : Some of them have been unset when previously removed the DG Broker configuration

show parameter LOG_ARCHIVE_CONFIG 
show parameter DB_FILE_NAME_CONVERT
show parameter LOG_FILE_NAME_CONVERT
show parameter LOG_ARCHIVE_DEST_n pointing to the Standby Database and valid for STANDBY_LOGFILES
show parameter LOG_ARCHIVE_DEST_STATE_n
show parameter DG_BROKER_START
show parameter DG_BROKER_CONFIG_FILE1
show parameter DG_BROKER_CONFIG_FILE2
show parameter STANDBY_ARCHIVE_DEST
show parameter STANDBY_FILE_MANAGEMENT
show parameter FAL_SERVER
show parameter FAL_CLIENT

5. Drop Standby Redologs from the Primary Database

SQL> SELECT GROUP# FROM V$STANDBY_LOG;

SQL>  SELECT GROUP# FROM V$STANDBY_LOG;

    GROUP#
----------
         4
         5
         6
         7

SQL> ALTER DATABASE DROP STANDBY LOGFILE GROUP 4;

Database altered.

SQL> ALTER DATABASE DROP STANDBY LOGFILE GROUP 5;

Database altered.

SQL> ALTER DATABASE DROP STANDBY LOGFILE GROUP 6;

Database altered.

SQL> ALTER DATABASE DROP STANDBY LOGFILE GROUP 7;

Database altered.

6. Shutdown standby and open it in mount state in order to disable managed recovery and activate it (Make Primary)

SQL> shutdown immediate;
SQL> startup mount;

-- Check the mode and the changed parameters

SQL> SELECT OPEN_MODE,PROTECTION_MODE,DATABASE_ROLE FROM V$DATABASE;

OPEN_MODE            PROTECTION_MODE      DATABASE_ROLE
-------------------- -------------------- ----------------
MOUNTED              MAXIMUM PERFORMANCE  PHYSICAL STANDBY


SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH;

Database altered.

SQL> ALTER DATABASE ACTIVATE PHYSICAL STANDBY DATABASE;

Database altered.


SQL> SELECT OPEN_MODE,PROTECTION_MODE,DATABASE_ROLE FROM V$DATABASE;

OPEN_MODE            PROTECTION_MODE      DATABASE_ROLE
-------------------- -------------------- ----------------
MOUNTED              MAXIMUM PERFORMANCE  PRIMARY

SQL> ALTER DATABASE OPEN;

Database altered.

SQL> SELECT OPEN_MODE,PROTECTION_MODE,DATABASE_ROLE FROM V$DATABASE;

OPEN_MODE            PROTECTION_MODE      DATABASE_ROLE
-------------------- -------------------- ----------------
READ WRITE           MAXIMUM PERFORMANCE  PRIMARY

7. Former Physical Standby is now and independent Primary DB 



REFERENCES

https://blogs.oracle.com/AlejandroVargas/resource/How-to-open-the-standby-when-the-primary-is-lost.pdf

Oracle Support Documents : 733794.1 , 975614.1, 261336.1

Monday, July 28, 2014

RMAN-03009: failure of delete command ... ORA-19606: Cannot copy or restore to snapshot control file

This error happen when a snapshot controlfile becomes obsolete, but rman is using it and cannot delete it. The problem that comes out of it is that all other obsolete backups are not deleted also since rman tries first to delete the controlfile copy.

Solution :

Take a note of the cuurent location of the snapshot controlfile and set a new location :

RMAN>show all;

RMAN configuration parameters for database with db_unique_name mydb;

.....

CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/myloc/mysnapc.f'; # OLD

RMAN>

RMAN>CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/mylocation/mysnapc.f'; # Configure new location


RMAN>host 'rm /myloc/mysnapc.f'; # delete old file from disk 

RMAN> corsscheck copy; # Mark the deleted file expired

RMAN> delete expired copy; # Delete the expired file from rman catalog/db control file

RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/myloc/mysnapc.f'; # Restore to old location

RMAN>exit;

REFERENCES: Oracle Support Doc ID 1215493.1

Wednesday, July 16, 2014

Converting a Physical Standby to Snapshot Standby using Data Guard Broker (11gR2)

The following steps show how easy is to use Data Guard Broker (dgmgrl) to manage states of the database in a Data Guard (DG) configuration. In the next example the configuration is : 

ACTIVE DATA GUARD CONFIGURATION - Maximum Performance
Primary Database : prmdb
Physical Standby Database : phstdb -- Read Only with Apply
Fast-Start Failover: Enabled

Use the following procedure to convert phstdb to Snapshot Standby :

1. Connect to DG Broker and check the active configuration :

$ dgmgrl
DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys@prmdb
Password:
Connected.
DGMGRL> show configuration verbose;

Configuration - DG_CONF_FSF

  Protection Mode: MaxPerformance
  Databases:
    prmdb - Primary database
    phstdb - (*) Physical standby database

  (*) Fast-Start Failover target

  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'

Fast-Start Failover: ENABLED

  Threshold:        30 seconds
  Target:           phstdb
  Observer:         observer.host
  Lag Limit:        30 seconds
  Shutdown Primary: TRUE
  Auto-reinstate:   TRUE

Configuration Status:
SUCCESS

2. Disable Fast Start Failover otherwise you will get : ORA-16668: operation cannot be performed on the fast-start failover target standby database

DGMGRL> disable FAST_START failover;
Disabled.
DGMGRL> show configuration verbose;

Configuration - DG_CONF_FSF

  Protection Mode: MaxPerformance
  Databases:
    prmdb - Primary database
    phstdb - Physical standby database

  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

3. Issue the convert to snapshot standby statement;

DGMGRL> convert database phstdb to snapshot standby;
Converting database "phstdb" to a Snapshot Standby database, please wait...
Database "phstdb" converted successfully

DGMGRL> show configuration verbose;

Configuration - DG_CONF_FSF

  Protection Mode: MaxPerformance
  Databases:
    prmdb - Primary database
    phstdb - Snapshot standby database

  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

You are done and you can check it with sqlplus 

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE

From this point and forward you can treat phstdb as a normal Oracle Database.

You can go back and converting it to Physical Standby at any time as long as you have the space in FRA to keep the flashback logs required. 

The space is needed because going back is accomplished by creating a guaranteed restore point thus you need the space to keep the flashback logs.

You must also take in mind the space needed for the archive logs coming from the Primary (Log Shipping is not disabled) and for the new Archive logs produced by the Snapshot Standby.

After returning to the initial state the latest are deleted.

Now in order to return to the previous state do the following :

1. From the same DGMGRL session (or a new one) issue the convert to physical standby command:

DGMGRL> convert database phstdb to physical standby;
Converting database "phstdb" to a Physical Standby database, please wait...
Operation requires shutdown of instance "phstdb" on database "phstdb"
Shutting down instance "phstdb"...
Database closed.
Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "phstdb" on database "phstdb"
Starting instance "phstdb"...
ORACLE instance started.
Database mounted.
Continuing to convert database "phstdb" ...
Operation requires shutdown of instance "phstdb" on database "phstdb"
Shutting down instance "phstdb"...
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "phstdb" on database "phstdb"
Starting instance "phstdb"...
ORACLE instance started.
Database mounted.
Database "phstdb" converted successfully

2. Enable the Fast Start failover and check the active configuration

DGMGRL> enable FAST_START FAILOVER;
Enabled.
DGMGRL> show configuration verbose;

Configuration - DG_CONF_FSF

  Protection Mode: MaxPerformance
  Databases:
    prmdb - Primary database
    phstdb - (*) Physical standby database

  (*) Fast-Start Failover target

  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'

Fast-Start Failover: ENABLED

  Threshold:        30 seconds
  Target:           phstdb
  Observer:         observer.host
  Lag Limit:        30 seconds
  Shutdown Primary: TRUE
  Auto-reinstate:   TRUE

Configuration Status:
SUCCESS

3. Using SQL Plus you can check that you are back to the initial configuration:

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ ONLY WITH APPLY


REFERENCES
Oracle® Data Guard Broker 11g Release 2 (11.2) -- 6.11 Scenario 10: Converting a Physical Standby to a Snapshot Standby


Wednesday, May 14, 2014

11gR2 Database Duplicate from Active Database to new Host

Here is a procedure to duplicate an 11gR2 database to a new host with the same O/S and DB version (in the example is 11.2.0.3) but to a different location using common area for all DB files (OMF).

The procedure is for Linux x86_64 and assuming the Oracle Software has been installed and Listener is running and that the directories used in the init.ora have been created and oracle user has full privileges on them.

Everything is executed from the new host


1. Create a custom initaux.ora or copy from the old host but the following parameters are required:

*.control_files='/oradata/aux/control01.ctl','/oradata/aux/control02.ctl'
*.db_block_size=8192
*.db_name='aux'
*.db_create_file_dest=/oradata/aux  # Instead of using DB_FILE_NAME_CONVERT , use OMF !!
*.db_recovery_file_dest='/oradata/aux/fra'
*.db_recovery_file_dest_size=10G
*.diagnostic_dest='/oradata'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=auxXDB)'
*.log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST'
*.memory_target=2G
*.remote_login_passwordfile='EXCLUSIVE'

Copy the file to $ORACLE_HOME/dbs/initaux.ora 

2. Add static entry to the listener

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = aux
      (ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_1)
      (SID_NAME = aux)
    )

Reload listener configuration: 
$ lsnrctl reload

3. Add TNSNAMES entries for TARGET and AUX and RMAN_CATALOG (if exists) in the new host tnsnames.ora

# If target usesd catalog from rman backups
RMAN_CATALOG=
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = rman_catalog_host)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = rman_catalog)
    )
  )

TARGET =
  (DESCRIPTION =
    (address = (protocol = TCP)(host = target_host)(port = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = target)
    )
  )

AUX =
  (DESCRIPTION =
    (address = (protocol = TCP)(host = aux_host)(port = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = aux)
    )
  )

!!! ATTENTION !!! 
AUX & TARGET TNSNAMES ALIAS entry must also created on the old host TNSNAMES.ORA other wise RMAN session will fail with :

ORA-17629: Cannot connect to the remote database server
ORA-17627: ORA-12154: TNS:could not resolve the connect identifier specified
ORA-17629: Cannot connect to the remote database server

during copying the control file from TARGET in the first steps of the memory script execution.

4. Create a new password file for AUX to match the production one

$ rm $ORACLE_HOME/dbs/orapwaux
$ orapwd file=orapwaux password=pass

5. Start the auxiliary instance

$ export ORACLE_SID=aux 
$ sqlplus / as sysdba
SQL> startup nomount
SQL>exit;

6. Start RMAN and execute the CLONE creation script

!!! ATTENTION !!! 
Always use password when invoking RMAN otherwise session may fail

$ rman target sys@target/pass auxiliary sys@aux/pass catalog rman_catalog_user@rman_catalog/pass
RMAN>run {
allocate channel c1 type disk;
allocate channel c2 type disk;
allocate auxiliary channel aux1 type disk;
allocate auxiliary channel aux2 type disk;
sql 'alter system switch logfile';
duplicate target database to aux from active database;
}

6. You have finsihed

After RMAN session is completed then AUX is up and running.

contents of Memory Script:
{
   Alter clone database open resetlogs;
}
executing Memory Script

database opened
Executing: alter database flashback on
Finished Duplicate Db at 14-MAY-14
released channel: c1
released channel: c2
released channel: aux1
released channel: aux2

RMAN>exit;

You can now create a new spfile in the default location and modify all the parameters you want by editing the pfile created in the following commands.

$ sqlplus / as sysdba
SQL>create pfile='/mylocation/mypfile' from memory;
SQL>shutdown immediate;
SQL>create spfile from pfile='/mylocation/mypfile';
SQL>startup



Wednesday, April 9, 2014

OMS 12c: "The page has expired. Click OK to continue"

I have been using OMS 12c for long time and i always see the following  message after some time of not actively using console: “The page has expired. Click OK to continue”

This is very annoying to me specially when i use the DB Performance Monitoring page to monitor a db performance over time and when i try to check i got this message. 

Here is how to fix this issue :

Go to OMS_INSTALLATION_HOME/bin

./emctl set property -name oracle.sysman.eml.maxInactiveTime -value -1 -sysman_pwd PASSWORD

The previous command Sets the property maxInactiveTime=-1 which disables this functionality. 
You can put any value you want there and reflects to minutes.

Restart OMS 12c stack

./emctl stop oms
./emctl start oms


REFERENCES:

Oracle Enterprise Manager Cloud Control Advanced Installation and Configuration Guide 12c Release 1 (12.1.0.1) -- Part Number E24089-09 
  • 11 Performing Additional Configuration Tasks
    • Modifying the Default Login Timeout Value


Thursday, February 27, 2014

ORA-08104 after a session has been killed during an online index creation/rebuild

Killing a session that is creating/rebuilding an index online rebuild your session was killed and you trying to drop the index or re-creat/rebuild you get 

ORA-08104: this index object ##### is being online built or rebuilt

In order to resolve this issue you have to use function dbms_repair.online_index_clean as sysdba. This is a fix that is installed with patch for BUG:3805539 and if you get an error calling the function , then please install it.

SQL>set serveroutput on

SQL>declare
  isclean boolean;
begin
  isclean := false;

  while isclean=false
  loop
    isclean := dbms_repair.online_index_clean(object_id=>dbms_repair.all_index_id, wait_for_lock=>dbms_repair.lock_wait
    );
    dbms_lock.sleep(10);

  end loop;

exception

when others then
  raise;

end;
/

Parameter input:

object_id : 
    ALL_INDEX_ID -> cleanup all index that qualify
    [OBJECT_ID] -> cleanup the specified index by using the object_id

wait_for_lock :
     LOCK_WAIT -> retry getting DML locks on underlying table
     LOCK_NOWAIT -> do not retry getting DML locks on underlying table

Returns:
     TRUE -> specified indexe(s) were cleaned up successfully
     FALSE -> one or more indexes failed cleaned up but its possible that some may have been cleaned up successfully.

REFERENCES
Oracle Support Doc ID 375856.1 & 1378173.1





Thursday, February 20, 2014

How to avoid a performance problem commonly caused by Advanced Queuing usage in a RAC environment

A very nice article from to IOUG to investigate and avoid perfromance bottlenecks in an Oracle RAC having investigation ,analysis and resolution, starting from the top events such and latch: cache buffer chains & GC Buffer Waits of a singel instance on a 5 Node RAC.

PDF : http://www.ioug.org/d/do/3355

Alternative link: http://www.ioug.org/p/bl/et/blogid=40&blogaid=241

Wednesday, February 19, 2014

ORA-29902 DRG-10502 Utilizing Text Indexes on Standby in Read Only Mode (Active DataGuard)

Several Oracle Text indexexes were dropped and re-created in the Primary DB. 

The BO Reports that utilize the indexes on the Standby in an Active Data Guard configuration reported the follwoing error when the SQL uses CONTAINS :

ORA-29902: error in executing ODCIIndexStart() routine
ORA-20000: Oracle Text error:
DRG-50857: oracle error in drdmdcntORA-20000: Oracle Text error:DRG-10502: index 1174 does not existORA-06512: at "CTXSYS.DRUE", line 160ORA-06512: at "CTXSYS.DRVXMD", line 148ORA-06512: at line 1


The problem arises beacuse the SQL uses cached execution plan is used in the Standby DB.

The only workaround is to fulsh the shared pool in the Standby DB.

SQL>alter system flush shared_pool;

References : Oracle Spport Doc ID 1570093.1




Friday, February 14, 2014

expdp/impdp fails reports ORA-21780 at the end of export

Recently i had an expdp that reported the following:

ORA-39097: Data Pump job encountered unexpected error -21780
ORA-39065: unexpected master process exception in DISPATCH
ORA-21780: Maximum number of object durations exceeded.

After searching Oracle Support (see below) the issue is not affecting the data and the consistency of the dump file thus data were successfully exported.

The side effect is that it leaves an orphaned data pump job in dba_datapump_jobs.

Check http://agstamy.blogspot.be/2012/10/cleanup-orphaned-datapump-jobs.html of how to clean orphaned data pump jobs.

The issue is raised after changing the PARALELL workers of the running datapump job.

The Oracle Support document states that this happens only if lowering the PARALLEL workers of a running job, but the referenced bug reference and workaround state DO NOT MODIFY PARALLEL while the job is running.

REFERENCES:

Oracle Support Doc ID 1357811.1

Versions Affected : Version 10.1.0.2 to 11.2.0.3

BUG:10198387 - DATA PUMP EXPORT REPORTS ORA-21780 WHEN PARALLELISM (PARALLEL=N) IS MODIFIED

WORKAROUND

Don't change the number of parallel processes while Data Pump Export is running.

FIXED :

11.2.0.4
12.1

Wednesday, February 12, 2014

Slow Materialized View Fast Refreshes

A very good article about the mechanisms of MVIEW Fast Refresh, specially when fast refresh is occuring but becoming slow.

http://www.dba-oracle.com/t_materialized_view_fast_refresh_performance.htm


Tuesday, February 4, 2014

OMS12c Agent has stopped monitoring COLL_DISABLED

Agent has stopped communicating with OMS and collection has been halted.

Steps to workaround and fix this issue :

1. get the status of the agent

./emctl status agent

Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
.
.
Total Megabytes of XML files uploaded so far : 2.39
Number of XML files pending upload           : 5,067
Size of XML files pending upload(MB)         : 4.83
Available disk space on upload filesystem    : 79.38%
Collection Status                            : [COLLECTIONS_HALTED(
 UPLOAD SYSTEM Threshold - unable to purge files in upload system)]

2. If the error is the above then as workaround re-secure the agent and problem will be fixed.

./emctl stop agent

./emctl secure agent

Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Agent is already stopped...   Done.
Securing agent...   Started.
Enter Agent Registration Password :
EMD gensudoprops completed successfully
Securing agent...   Successful.
./emctl start agent


Issue should be fixed.


REFERENCES
Oracle Support DOC ID : 1618514.1