EBS ADOP Woes: Tackling ORA-20001 in Cleanup Phase

 

This blog aims to support DBAs who encounter issues during the EBS application R12.2 ADOP phase=cleanup, specifically when it fails with the ORA-20001 error message: 'Error while calling ad_zd.cleanupORA-01555: snapshot too old.' The steps provided here offer guidance in identifying and addressing the challenges associated with the ADOP phase=Cleanup problem.

 

Below Step:

Issue:

We encountered the following issue during the application of the adpatch ADOP patching cycle when executing the adop phase=cleanup

 

[applmgr@r122 ~]$ adop phase=cleanup

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Please wait. Validating credentials…

RUN file system context file: /u01/applmgr/UAT_TEST/fs2/inst/apps/UAT_TEST_r122/appl/admin/UAT_TEST_r122.xml

PATCH file system context file: /u01/applmgr/UAT_TEST/fs1/inst/apps/UAT_TEST_r122/appl/admin/UAT_TEST_r122.xml

************* Start of session *************

version: 12.2.0

started at: Wed May 24 2021 03:34:53

APPL_TOP is set to /u01/applmgr/UAT_TEST/fs2/EBSapps/appl

[STATEMENT] Using 4 workers (Default: 4, Recommended maximum limit: 62)

Cleanup is not done in earlier session

[START 2021/05/24 03:35:27] adzdoptl.pl run

ADOP Session ID: 4

Phase: cleanup

Log file: /u01/applmgr/UAT_TEST/fs_ne/EBSapps/log/adop/4/adop_20210524_033421.log

[START 2021/05/24 03:35:48] cleanup phase

[EVENT] [START 2021/05/24 03:35:52] Performing Cleanup steps

[EVENT] [START 2021/05/24 03:35:57] Running CLEANUP ddls in ddl handler table

Calling: adpatch options=hotpatch,nocompiledb interactive=no console=no workers=4 restart=no abandon=yes defaultsfile=/u01/applmgr/UAT_TEST/fs2/EBSapps/appl/admin/UAT_TEST/adalldefaults.txt patchtop=/u01/applmgr/UAT_TEST/fs2/EBSapps/appl/ad/12.0.0/patch/115/driver logfile=cleanup.log driver=ucleanup.drv

ADPATCH Log directory: /u01/applmgr/UAT_TEST/fs_ne/EBSapps/log/adop/4/cleanup_20210524_033421/UAT_TEST_r122/log

[EVENT] [END 2021/05/24 03:39:43] Running CLEANUP ddls in ddl handler table

[EVENT] Cleaning up ABORT DDL from DDL Handler Table

[START 2021/05/24 03:39:54] Generating All DDL Report

[EVENT] Report: /u01/applmgr/UAT_TEST/fs2/EBSapps/appl/ad/12.0.0/sql/ADZDALLDDLS.sql

[EVENT] Output: /u01/applmgr/UAT_TEST/fs_ne/EBSapps/log/adop/4/cleanup_20210524_033421/UAT_TEST_r122/adzdallddls_20210524_033957.out

[END 2021/05/24 03:39:58] Generating All DDL Report

[EVENT] Calling cleanup in QUICK mode

[WARNING] Cleanup may take a while. Please wait.

[ERROR] Failed to execute sql statement :

declare

result varchar2(10);

begin

ad_zd.cleanup(‘QUICK’);

exception

when others then

raise_application_error(-20001,’Error while calling ad_zd.cleanup’ || sqlerrm);

end;

[ERROR] SQLPLUS error: buffer=

SQL*Plus: Release 10.1.0.5.0 – UAT_TESTuction on Wed May 24 03:40:01 2021

Copyright (c) 1982, 2005, applmgr. All rights reserved.

SQL> SQL> Connected.

SQL> declare

ERROR at line 1:

ORA-20001: Error while calling ad_zd.cleanupORA-01555: snapshot too old:

rollback segment number 2 with name “_SYSSMU2_735814084$” too small

ORA-06512: at line 7

Disconnected from applmgr Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit UAT_TESTuction

With the Partitioning, OLAP, Data Mining and Real Application Testing options

[UNEXPECTED]Error occurred while calling cleanup plsql API

[UNEXPECTED]Cleanup phase completed with errors/warnings. Please check logfiles

Log file: /u01/applmgr/UAT_TEST/fs_ne/EBSapps/log/adop/4/adop_20210524_033421.log

adop exiting with status = 1 (Fail)

[applmgr@r122 ~]$

 

Solution:

STEP:1

To address this issue, we conducted a thorough check and discovered that the undo tablespace was completely filled. Subsequently, we resolved the problem by extending the undo tablespace using the following steps

 

SQL> SELECT df.tablespace_name TABLESPACE,

df.total_space_mb TOTAL_SPACE_GB,

(df.total_space_mb - fs.free_space_mb) USED_SPACE_GB, fs.free_space_mb FREE_SPACE_GB,

ROUND(100 * (fs.free_space / df.total_space),2) PCT_FREE

FROM (SELECT tablespace_name, SUM(bytes) TOTAL_SPACE,TRUNC((SUM(bytes) / 1048576/1024),2) TOTAL_SPACE_MB

FROM dba_data_files GROUP BY tablespace_name) df,

(SELECT tablespace_name, SUM(bytes) FREE_SPACE,TRUNC((SUM(bytes) / 1048576/1024),2) FREE_SPACE_MB

FROM dba_free_space GROUP BY tablespace_name) fs

WHERE df.tablespace_name = fs.tablespace_name(+) ORDER BY fs.tablespace_name;

 

TABLESPACE_NAME  TOTAL_GB      USED_GB    FREE_MB  PCT_FREE

APPS_UNDOTS1       10            9.99             0.1    0.01

 

SQL> ALTER DATABASE DATAFILE ‘/u01/applmgr/UAT_TEST/data/undo01.dbf’ RESIZE 15G;

Database altered.

SQL>

STEP:3

Then again After increasing the tablespace,we have executed the adop phase=cleanup then it got completed with out any issues.

 

[applmgr@r122 ~]$ adop phase=cleanup

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Please wait. Validating credentials…

RUN file system context file: /u01/applmgr/UAT_TEST/fs2/inst/apps/UAT_TEST_r122/appl/admi

n/UAT_TEST_r122.xml

PATCH file system context file: /u01/applmgr/UAT_TEST/fs1/inst/apps/UAT_TEST_r122/appl/ad

min/UAT_TEST_r122.xml

************* Start of session *************

version: 12.2.0

started at: Fri Feb 28 2021 04:11:18

APPL_TOP is set to /u01/applmgr/UAT_TEST/fs2/EBSapps/appl

[STATEMENT] Using 4 workers (Default: 4, Recommended maximum limit: 59)

Cleanup is not done in earlier session

[START 2021/02/28 04:11:45] adzdoptl.pl run

ADOP Session ID: 4

Phase: cleanup

Log file: /u01/applmgr/UAT_TEST/fs_ne/EBSapps/log/adop/4/adop_20210228_041049.log

[START 2021/02/28 04:12:03] cleanup phase

[EVENT] [START 2021/02/28 04:12:06] Performing Cleanup steps

[EVENT] [START 2021/02/28 04:12:11] Running CLEANUP ddls in ddl handler table

Calling: adpatch options=hotpatch,nocompiledb interactive=no console=no workers=4 restart=no abandon=yes defaultsfile=/u01/applmgr/UAT_TEST/fs2/EBSapps/appl/admin/UAT_TEST/adalldefaults.txt patchtop=/u01/applmgr/UAT_TEST/fs2/EBSapps/appl/ad/12.0.0/patch/115/driver logfile=cleanup.log driver=ucleanup.drv

ADPATCH Log directory: /u01/applmgr/UAT_TEST/fs_ne/EBSapps/log/adop/4/cleanup_20210228_041049/UAT_TEST_r122/log

[EVENT] [END 2021/02/28 04:13:26] Running CLEANUP ddls in ddl handler table

[EVENT] Cleaning up ABORT DDL from DDL Handler Table

[START 2021/02/28 04:13:32] Generating All DDL Report

[EVENT] Report: /u01/applmgr/UAT_TEST/fs2/EBSapps/appl/ad/12.0.0/sql/ADZDALLDDLS.sql

[EVENT] Output: /u01/applmgr/UAT_TEST/fs_ne/EBSapps/log/adop/4/cleanup_20210228_041049/UAT_TEST_r122/adzdallddls_20210228_041335.out

[END 2021/02/28 04:13:36] Generating All DDL Report

[EVENT] Calling cleanup in QUICK mode

[WARNING] Cleanup may take a while. Please wait.

[EVENT] [END 2021/02/28 04:57:56] Performing Cleanup steps

[END 2021/02/28 04:58:05] cleanup phase

[START 2021/02/28 04:58:11] Generating AD_ZD_LOGS Report

[EVENT] Report: /u01/applmgr/UAT_TEST/fs2/EBSapps/appl/ad/12.0.0/sql/ADZDSHOWLOG.sql

[EVENT] Output: /u01/applmgr/UAT_TEST/fs_ne/EBSapps/log/adop/4/cleanup_20210228_041049/UAT_TEST_r122/adzdshowlog.out

[END 2021/02/28 04:58:24] Generating AD_ZD_LOGS Report

[END 2021/02/28 04:58:25] adzdoptl.pl run

adop phase=cleanup – Completed Successfully

Log file: /u01/applmgr/UAT_TEST/fs_ne/EBSapps/log/adop/4/adop_20210228_041049.log

adop exiting with status = 0 (Success)

[applmgr@r122 ~]$

 

Thanks for visiting a blog, hope you would have fixed the issue you have faced using above steps.

Oracle EBS R12.2 – ADOP Phases and Parameters Explained

Oracle E-Business Suite R12.2 uses Online Patching powered by ADOP (Application DBA Online Patching).

Understanding its phases and parameters is essential for smooth patching with minimal downtime.

This blog provides a clear explanation of each ADOP phase, along with commonly used parameters.

1) PREPARE Phase

The prepare phase starts a new patching cycle by preparing the patch file system for usage.

Syntax:

adop phase=prepare

2) APPLY Phase

The apply phase applies patches to the patch file system, allowing online patching with minimal downtime.

Syntax:

adop phase=apply patches=<patch_number>

Optional Parameters During APPLY Phase

Instead of specifying parameters on the command line, you can use an input file.

adop phase=apply input_file=<full_path_to_file>

Input file may contain:

workers=<number>

patches=<patch1>.drv, <patch2>.drv

phase=<phase>

patchtop=<path>

merge=<yes/no>

defaultsfile=<file>

abandon=<yes/no>

restart=<yes/no>

workers=<number>

Note: Always specify the full path of the input file.

Important APPLY Parameters

a) restart

Used to resume a previously failed patching session.

adop phase=apply patches=<patch> restart=yes

b) abandon

Starts the patch apply from scratch.

adop phase=apply patches=<patch> abandon=yes

c) apply_mode

Used to run the patch in downtime mode (patch applied on run FS).

adop phase=apply patches=<patch> apply_mode=downtime

d) apply = yes/no

Run in test mode by specifying:

apply=no

e) analytics

Enables additional analysis reports during patching.

adop phase=apply analytics=yes

This runs multiple SQL scripts and generates outputs under:

$EBS_HOME/fs_ne/EBSapps/log/adop/<adop_session>/

Generated reports include:

ADZDCMPED.sql – Differences between Run and Patch editions

ADZDSHOWED.sql – List of editions

ADZDSHOWOBJS.sql – Editioned object summary

ADZDSHOWSM.sql – Seed data manager report

3) FINALIZE Phase

The finalize phase performs required steps to prepare the system for cutover, including compilation of invalid objects.

Syntax:

adop phase=finalize

Modes:

finalize_mode=(full|quick)

4) CUTOVER Phase

During cutover, a new run file system is created from the patch file system and becomes active.

Syntax:

adop phase=cutover

Optional Parameters During CUTOVER

a) mtrestart

Prevents automatic restart of application services.

adop phase=cutover mtrestart=no

b) cm_wait

Specify how long ADOP waits for active concurrent requests before shutting down ICM.

Example:

adop phase=cutover cm_wait=30

(Default: waits indefinitely)

5) CLEANUP Phase

Used to clean patching artifacts and logs after cutover.

Syntax:

adop phase=cleanup cleanup_mode=(full|standard|quick)

Default is standard.

6) FS_CLONE Phase

Synchronizes the patch file system with the run file system.

Syntax:

adop phase=fs_clone

Note: Prepare phase automatically performs fs_clone if missed during the last patching cycle.

Optional FS_CLONE Parameters

a) force

Restart fs_clone from scratch.

adop phase=fs_clone force=yes

b) Patch File System Backup Count

s_fs_backup_count=<number>

Default = 0 (no backup)

Determines how many patch file system backups to retain.

7) ABORT Phase

Used to abort the current patching cycle.

Syntax:

adop phase=abort

Note: Can only be executed before cutover.

Conclusion

ADOP is a powerful tool that enables Oracle EBS R12.2 to perform online patching with minimal service downtime.

Understanding its phases and parameters is essential for efficient and error-free patching.

Resolving MRP Failure with ORA-10567 in Oracle Data Guard (Standby Database)

Introduction

In an Oracle Data Guard environment, the Managed Recovery Process (MRP) plays a critical role in applying redo logs from the primary database to the standby database. Any disruption in MRP can directly impact data synchronization and disaster recovery readiness.

This blog explains a real-time issue where the MRP process terminated unexpectedly with ORA-10567 and ORA-00600 errors, along with the root cause analysis and step-by-step recovery solution.

---

Real-World Scenario

During routine monitoring of a physical standby database, it was observed that the MRP process had stopped unexpectedly. As a result, redo logs were no longer being applied, causing a gap between the primary and standby databases.

Further investigation of the alert log revealed internal errors related to block corruption. Since this was a production standby environment, immediate action was required to restore synchronization and avoid potential data protection risks.

---

Issue Description

The following errors were observed in the standby database alert log:

ORA-00600: internal error code, arguments: [3020], [7], [768706], [30128834], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 7, block# 768706, file offset is 2002272256 bytes)
ORA-10564: tablespace TBS02
ORA-01110: data file 7: 'F:\APP\ADMINISTRATOR\ORADATA\TBS\TBS02.DBF'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK'

---

Root Cause Analysis

The error clearly indicates a data block corruption in the standby database datafile.

• ORA-10567 → Redo does not match the data block
• ORA-00600 [3020] → Internal error triggered due to inconsistency
• Datafile 7 (TBS02) contains corrupted block

During redo apply, MRP attempts to apply changes to a corrupted block, which causes recovery to fail and terminate.

---

Solution

To resolve this issue, the corrupted datafile must be restored from the primary database and applied on the standby.

---

Step 1: Take Backup from Primary Database

Connect to RMAN on the primary database:

RMAN> rman target /

Take backup of the affected datafile:

RMAN> backup device type disk format 'F:\rman_backup_21\%U' datafile 7;

After the backup completes:

• Copy the backup pieces to the standby server

---

Step 2: Restore Datafile on Standby

On the standby database:

Start and mount the standby:

SQL> startup nomount;
SQL> alter database mount standby database;

Connect to RMAN:

RMAN> rman target /

Catalog the backup location:

RMAN> catalog start with 'F:\rman_backup_21\' noprompt;

Restore the corrupted datafile:

RMAN> restore datafile 7;

Exit RMAN:

RMAN> exit;

---

Step 3: Restart MRP

SQL> recover managed standby database disconnect from session;

---

Validation

After recovery:

• Verify MRP is running
• Check alert log for errors
• Monitor redo apply lag

---

Key Takeaways

• ORA-10567 indicates redo inconsistency due to block corruption
• MRP stops when encountering corrupted blocks
• Restoring affected datafile from primary resolves the issue
• Quick identification is critical to avoid data lag

Resolving “Missing PATCH FS CONTEXT FILE” Error During ADOP fs_clone in Oracle EBS

Introduction

During Oracle E-Business Suite (R12.2) patching, the ADOP (Online Patching) cycle performs multiple validation checks to ensure the environment is consistent before proceeding. One of the critical phases is fs_clone, which synchronizes the run and patch file systems.

This blog explains a real-time issue where the fs_clone phase failed due to a missing PATCH FS CONTEXT FILE, along with the root cause and resolution steps.

---

Real-World Scenario

During a production patching activity, the adop prepare and apply phases completed successfully, and the team proceeded with the fs_clone phase.

However, the process failed with unexpected Perl errors. There were no clear database errors or blocking sessions, and system resources were normal. Initial analysis suggested a script-level issue, but further investigation revealed that the problem was related to configuration metadata.

The root cause was identified as a missing PATCH file system context entry in the database, which caused the validation step to fail.

This is a typical case where:

• The issue is not clearly visible in logs
• The failure is caused by missing configuration data rather than runtime errors

---

Issue Description

While executing:

adop phase=fs_clone

The following errors were observed:

Use of uninitialized value $result in split at
.../ADOPValidationUtils.pm line 1294

No such file or directory at
.../ADOPValidationUtils.pm line 230

---

Root Cause Analysis

The failure occurred because the PATCH file system context entry was missing in the table:

FND_OAM_CONTEXT_FILES

This table stores context file details for both run and patch file systems.

Verification Query

SQL> SELECT DISTINCT(PATH)
FROM FND_OAM_CONTEXT_FILES
WHERE NAME NOT IN ('TEMPLATE', 'METADATA', 'config.txt')
AND CTX_TYPE = 'A'
AND (STATUS IS NULL OR UPPER(STATUS) IN ('S', 'F'))
AND EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'patch';

If this query returns no rows, it confirms that the patch context file is missing.

Note: Do not proceed with the solution if this query returns data.

---

Solution

Step 1: Verify Environment

echo $FILE_EDITION

Expected output:

Patch

echo $CONTEXT_FILE

---

Step 2: Upload Context File

$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile='/d03/inst/apps/clone/appl/admin/cloneapp.xml' logfile=/tmp/patchctxupload.log

You will be prompted to enter the APPS password.

---

Step 3: Re-verify

Run the verification query again to confirm the entry exists.

---

Step 4: Re-run fs_clone

adop phase=fs_clone

The process should now complete successfully.

---

Key Takeaways

• Not all errors are clearly visible in logs
• Configuration inconsistencies can cause patch failures
• FND_OAM_CONTEXT_FILES plays a key role in validation
• Proper verification is essential before applying fixes

---

Best Practices

• Validate context file entries before patching
• Ensure run and patch file systems are synchronized
• Monitor ADOP logs carefully
• Maintain proper documentation
• Perform pre-check validations

---

Conclusion

The “Missing PATCH FS CONTEXT FILE” issue highlights how configuration gaps can interrupt Oracle EBS patching. By identifying the missing entry and correcting it using the appropriate utility, the issue can be resolved efficiently.

Understanding these types of hidden configuration dependencies helps DBAs ensure smooth patching cycles and maintain system stability.