Skip to main content

How can we help you?

Druva Documentation

Troubleshooting Oracle DTC issues

Phoenix Editions: File:/tick.png Business File:/cross.png Enterprise File:/tick.png Elite

 

Failed to restore the database to an alternate server

You might encounter the following issues while restoring the database to an alternate server:

Issue

You get the following error message:

ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 11: Resource temporarily unavailable


Cause

A database is already running on the restore location and using the data file.

Resolution

Perform either of the following actions:

  1. Use different location for restore.

  2. Shutdown the database running on the restore location. This will overwrite the existing database.

  3. Drop the database running on the restore location. This will remove the files used by the existing database.

Issue

You get the following error message:

ORA-01261: Parameter db_create_file_dest destination string cannot be translated\nORA-01262: Stat failed on a file destination directory\nLinux-x86_64 Error: 2: No such file or directory\nSQL> Disconnected\n"

Cause

You get this error when the spfile is restored, but the paths of the following files do not exist on the target server:

  • control_files

  • db_recovery_file_dest

  • archive_log_dest

  • Audit_file_dest

  • db_create_file_dest

Resolution

Create the required directory structure on the target server and make sure the oracle dba user has access to it.

Issue

You get the following error message:

ORA-27125: unable to create shared memory segment\nLinux-x86_64 Error: 28: No space left on device\

Cause

You get this error message when you want to restore the spfile along with the database, and the SGA or PGA memory available to start the database is insufficient.

Resolution

Free up the required memory or reduce the SGA and PGA memory in the spfile of the database.

Issue

You get the following error message:

ORA-01017: invalid username/password; logon denied

Cause

You get this error when either wallet is not configured or configured incorrectly on the target server. Additionally, if you provide incorrect credentials of the target database, the restore fails with this error.

Resolution

Make sure that you configure the wallet correctly on the target server and provide correct database credentials.

Issue

You get the following error message:

NID-00111: Oracle error reported from target database while executing

begin       dbms_backup_restore.nidprocessdf(:fno, :istemp, :skipped, :idchged,                                        :nmchged);    end; 

ORA-20000: File /home/oracle/oracle_base/oradata/CDB01/datafile/o1_mf_temp_knf4c73o_.tmp has wrong dbid or dbname, remove or restore the offending file.

Cause

When you retrigger restore on an already restored database, the name of the database being restored gets updated in the temp file. However, the temp file name remains the same as the earlier restore. This causes conflict, and the restore fails.

Resolution

Delete the temp file and retrigger restore.

Issue

You get the following error message:

ORA-27072: File I/O error

Linux-x86_64 Error: 28: No space left on device

Cause

The available memory is not sufficient to restore the database.

Resolution

Make sure enough space is present on the target server for restoring the database.

Issue

You get the following error message:

Failed to authenticate database credentials [Failed to fetch authentication details]

Cause

The Oracle Home and Oracle Base provided is incorrect or does not exist on the target server.

Resolution

Provide correct Oracle Home and Oracle Base and retrigger restore.

Issue

You get the following error message:

RMAN-03002: failure of recover command at 10/12/2022 05:41:06

ORA-19698: /home/oracle/oracle_base/oradata/NCDB_19C/onlinelog/o1_mf_1_kklyyd23_.log is from different database: id=3039599149, db_name=CLN1

Cause

When you restore a database, the log file contains logs of that database. When you retrigger restore of the same database, the log file contains the logs of the earlier restore, and the name of the log file also remains the same as that of the earlier restore. This causes conflict, and the restore fails.

Resolution

Drop the database that uses the log files using the following commands and retrigger restore:

shutdown abort;

startup mount exclusive restrict;

drop database;

exit

Issue

You get the following error message:

ORA-19504: failed to create file "+DATA/MHRACDB/DATAFILE/tbspace1.dbf"

ORA-17502: ksfdcre:3 Failed to create file +DATA/MHRACDB/DATAFILE/tbspace1.dbf

ORA-15001: diskgroup "DATA" does not exist or is not mounted

Cause

You get this error if the control file and spfile created in the earlier restore are not deleted and the database is running on the ASM.

Resolution

Perform the cleanup activity by deleting the old control file and spfile of the database created in the last restore. You can find these files in $ORACLE_HOME/dbs/ with the names cntrl<dbname>.dbf, init<dbname>.ora, and spfile<dbname>.ora respectively.
To restore the database without performing the cleanup activity, configure ASM on the target server with the same configurations as that of the source database.

Issue

You get the following error message:
level=debug ts=2022-11-15T21:08:05.997485224+05:30 filename=cmd.go:162 message="stdout: cmd: RMAN-00571: ===========================================================" Layer=OracleApiUtil

level=debug ts=2022-11-15T21:08:05.997514404+05:30 filename=cmd.go:162 message="stdout: cmd: RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============" Layer=OracleApiUtil

level=debug ts=2022-11-15T21:08:05.997529214+05:30 filename=cmd.go:162 message="stdout: cmd: RMAN-00571: ===========================================================" Layer=OracleApiUtil

level=debug ts=2022-11-15T21:08:05.99754723+05:30 filename=cmd.go:162 message="stdout: cmd: RMAN-03002: failure of sql statement command at 11/15/2022 21:08:05" Layer=OracleApiUtil

level=debug ts=2022-11-15T21:08:05.997561437+05:30 filename=cmd.go:162 message="stdout: cmd: ORA-01103: database name 'ORCLDB' in control file is not 'ORCL_DB'" Layer=OracleApiUtil

Cause

You get this error if the SID and the restored database name are different, and spfile is not selected for restore.

Resolution

Select the Restore SP file field and retrigger restore.

Unable to perform large Oracle DTC restore

Issue

The PhoenixIOServer process crashes leading to the failure of a database restore. The /var/log/messages file shows the following messages:

eb-db0 kernel: Out of memory: Kill process 4495 (PhoenixIOServer) scroe 259 or sacrifice child
eb-db0 kernel: Killed process 4495 (PhoenixIOServer) total-vm:7833136kB, anon-rss:6477996kB, file-res:0kB, shmem-rss:0kB
eb-db0 kernel: oom_reaper: reaped process 4495 (PhoenixIOServer), now anon-ree:0kB, file-rss:0kB, shmem-rss:0kB

Cause

One of the reasons could be the PhoenixIOServer process is taking up a lot of memory and is getting killed by the OOM (Out-Of-Memory) reaper.

Resolution

  1. Disable the OOM reaper by running the following command:
    sudo -s sysctl -w vm.oom-kill = 0
    For more information about enabling or disabling OOM reaper, see How to Adjust Linux Out-Of-Memory Killer Settings

  2. Reduce the number of worker threads that are used for the restore by setting the value of the DATAMOVER_WORKERS attribute in the /etc/PhoenixOracle/Phoenix.yml file and restart the PhoenixOracle service. 

Reducing the number of worker threads can impact restore time.

Failure of the FULL backup job

Issue

The FULL backup job fails with the following error:

===========================================================

=============== ERROR MESSAGE STACK FOLLOWS ===============

===========================================================

failure of backup command at 05/20/2022 19:30:17

expected archived log not found, loss of archived log compromises recoverability

error identifying file /oracle/ora12cdb/product/12.1.0/db_1/dbs/arch/1_82426_1074616154.dbf

unable to obtain file status

8Linux-x86_64 Error: 2: No such file or directory

Cause

The backup job might fail due to missing archive logs.

Resolution

  1. Run the crosscheck archivelog all command through RMAN and delete the expired archive log files by running the following command:

delete expired archivelog all;

  1. Retrigger FULL backup.

Failed to discover database 

The discovery of databases hosted on the Oracle servers that are registered with Druva might fail due to various reasons. See the following table for possible causes and their respective resolutions:

Cause

Resolution

The PhoenixOracle service is not running.

Start the PhoenixOracle service and try again.

The Oracle database instance is offline.

Make sure the database is up and running.

The database name is unavailable in the DBA registry.

Add the database name entry in the DBA registry located at /etc/oratab.

Database authentication is not defined.

Assign database authentication.

Failure of RMAN backup 

RMAN backup jobs might fail due to various reasons. See the following table for possible causes and their respective resolutions:

Cause

Resolution

The user specified does not have the SYSDBA privileges to perform RMAN backups

Assign the SYSDBA privileges to the user and try again.

The database is not in the ARCHIVELOG mode.

Enable the Perform offline backup if database is in NOARCHIVELOG mode RMAN setting in the backup policy.

The PhoenixOracle service is not running.

Start the PhoenixOracle service and try again.

Failure of offline RMAN backup

Issue

Offline RMAN backup might fail with the following error:
TNS:listener does not currently know of service requested in connect descriptor

Cause

When the database is in the NOARCHIVELOG mode, the database is shut down and brought to the mount state. When the database is shut down, the TNS is turned offline and therefore TNS does not work in the NOARCHIVELOG mode.

Resolution

Add the listener entry to the static list so that even if the database is down, it can connect. The same is applicable for restore operations as well.

For a standalone database, add the following entry in the listener.ora file located at $ORACLE_HOME/network/admin:
SID_LIST_LISTENER=

  (SID_LIST=

    (SID_DESC=

      (ORACLE_HOME=/u01/app/oracle/product/12.2.0.1/db_1)

      (SID_NAME=ORCL))

  )

For a RAC database, if the database name is RAC, and the instance id is RAC2, the following entry has to be added in the listener.ora file for each node:
SID_LIST_LISTENER=

  (SID_LIST=

    (SID_DESC=

      (GLOBAL_DBNAME=RAC)

      (ORACLE_HOME=/u01/app/oracle/product/12.2.0.1/db_1)

      (SID_NAME=RAC2))

  )

Failure of Snapshot restore

Restore databases from snapshots might fail due to the following reasons:

Cause

Resolution

Insufficient space on the Oracle server host on which the snapshot restore is requested.

Make sure the Oracle server host has the required space.

RMAN does not have enough permissions to read downloaded backup data.

Check the RMAN logs and give the required permissions to the backup folder.

An invalid path is specified in the Restore Location field on the Restore Target page.

Make sure you provide the correct path.

Oracle incremental backups get converted to full backups

Issue

When an incremental backup is triggered, the Hybrid Workloads agent requests information about databases that are updated. Druva backs up the changed data and creates a snapshot in your storage. During the incremental backup, the changed blocks (delta) from the data files are backed up, but the log files are skipped. 

See the following table for the possible causes of incremental backups getting converted to full backups and their respective workarounds:

Cause Resolution
The incarnation of the database, which is being backed up changes. Perform a full backup to initiate a new archive log chain with the latest incarnation.
The archived log sequences are missing after the last backup. Perform a full backup to initiate a new archive log chain.
An incremental backup of a database for which a full backup is not complete is attempted. Perform a full backup to create a baseline for the next incremental backup.
Inconsistencies between the uploaded database files and their metadata are detected. This might happen if the backup information is modified or deleted from the control file externally, maybe via any script, and not by the Phoenix agent.  Trigger new backup from the Phoenix Management Console.
If a full backup is executed by the Hybrid Workloads agent and the second full backup is being executed by a third-party backup tool or an Oracle Server. Then the Hybrid Workloads agent can convert the next incremental backup to a full backup.  Trigger new backup from the Phoenix Management Console.
Data corruption occurs during the last backup of a database. Perform a full backup.