Oracle DTC backup fails with INTERNAL65535
Problem description
-
Oracle backup fails immediately with the error - INTERNAL65535 on the console.
-
Connecting to database will give you the error - ORA-01090: shutdown in progress – connection is not permitted.
Cause
Your /var/log/audit filesystem is full Or does not have any inodes available.
Traceback
ORA-09817: Write to audit file failed.
Linux-x86_64 Error: 28: No space left on device
main_oracle_service.log
stack="failed to parse results: \nORACLE_BASE=/ora/app/oracle\nORACLE_HOME=/ora/app/oracle/product/12.1.0.2/dbhome\nERROR:\nORA-09817:
Write to audit file failed.\nLinux-x86_64 Error: 28: No space left on device\nAdditional information: 12\nORA-01075:
you are currently logged on\n\n\n\\ngoroutine 11 [running]:\nruntime/debug.Stack(0xc000408f30, 0xc0001dcd80,
0x10c)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9f\nphoenix/oraclesbt/client/pkg/logutil.(*loggerImpl).
Resolution
Note - Please get the help of the Oracle admin.
1. Identify the Oracle mount which has no space left.
$ df -Ph <Mount_point>
2. Go inside the volume ($ cd <mount_point>),then use the commands to locate the largest files.
$ find . -ls | sort -nrk7 | head -10
$ du -S . | sort -nr | head -10
3. Can check the below location/files for clearing the spaces.
-
audit file location in init.ora file
-
any old unwanted audit files.
-
any old trace files.
-
any large Oracle Net log files.
-
Moving, compressing, or deleting old archive redo log files.
-
Removing old installation files or binaries
-
If you have datafiles with ample free space, consider resizing them to smaller sizes.