ORA-17502: ksfdcre:4 Failed to create file +FRA

A weekend is about the start and got a call and one of our database is having issue. User is unable to connect and all their activity seems hung.

ISSUE : Instance Terminated and make Node2 inaccessible due to Oracle Sees FRA is Full.
But FRA is not Full ~ 2TB is available and db_recovery_file_dest_size=2.7TB. +FRA is ~18% Utilized only.

What i Did? Drop all existing restore point and restarted all node.

So i check the alert log on node 1 database.

 
*************************************
RVWR encountered an error when writing flashback database logs.
See error stack in alert log.
Since a Guarantee Restore Point was created,
RVWR cannot force to turn off flashback.
Therefore RVWR will terminate this instance.
*************************************
Fri Sep 16 05:17:28 2016
Errors in file /optware/orabase/grid/diag/rdbms/ckyclu3/CXXXLU31/trace/CXXXLU31_rvwr_41392.trc:
ORA-38701: Flashback database log 9 seq 5 thread 1: "+FRA"
ORA-17502: ksfdcre:4 Failed to create file +FRA
ORA-15041: diskgroup "FRA" space exhausted
USER (ospid: 41392): terminating the instance due to error 38701
Fri Sep 16 05:17:28 2016
System state dump requested by (instance=1, osid=41392 (RVWR)), summary=[abnormal instance termination].
System State dumped to trace file /optware/orabase/grid/diag/rdbms/ckyclu3/CXXXLU31/trace/CXXXLU31_diag_50122_20160916051728.trc
Fri Sep 16 05:17:30 2016
License high water mark = 81
Fri Sep 16 05:17:33 2016
Instance terminated by USER, pid = 41392
Fri Sep 16 05:17:34 2016
USER (ospid: 1359): terminating the instance
Fri Sep 16 05:17:34 2016
Instance terminated by USER, pid = 1359
Fri Sep 16 05:17:35 2016 

Continue reading

ORA-06512: at line 1 SQL BEGIN sys.dbms_xstream_gg.SET_FOO_TRIGGER_SESSION_CONTXT (fire=>TRUE); END;.

Just another day n the office where our Development was refresh from production.

Development DB is configured to have GG multi bidirectional replication.
I started the Manager, extract and pump without any issue but when starting the replicat i’m getting below error.

2016-05-09 13:34:41  ERROR   OGG-01668  Oracle GoldenGate Delivery for Oracle, ckn2ure2.prm:  PROCESS ABENDING.
2016-05-09 13:40:18  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): start CKL2BRE2.
2016-05-09 13:40:18  INFO    OGG-00963  Oracle GoldenGate Manager for Oracle, mgr.prm:  Command received from GGSCI on host gcxxxx2s:37774 (START REPLICAT CKL2BRE2 ).
2016-05-09 13:40:18  INFO    OGG-00975  Oracle GoldenGate Manager for Oracle, mgr.prm:  REPLICAT CKL2BRE2 starting.
2016-05-09 13:40:19  INFO    OGG-00995  Oracle GoldenGate Delivery for Oracle, ckl2bre2.prm:  REPLICAT CKL2BRE2 starting.
2016-05-09 13:40:19  INFO    OGG-03035  Oracle GoldenGate Delivery for Oracle, ckl2bre2.prm:  Operating system character set identified as UTF-8. Locale: en_US, LC_ALL:.
2016-05-09 13:40:19  ERROR   OGG-00868  Oracle GoldenGate Delivery for Oracle, ckl2bre2.prm:  ORA-26944: User "GGS_ADMIN" attempted to invoke a procedure without proper permissions.
ORA-06512: at "SYS.DBMS_LOGREP_UTIL", line 197
ORA-06512: at "SYS.DBMS_LOGREP_UTIL", line 230
ORA-06512: at "SYS.DBMS_XSTREAM_GG", line 9
ORA-06512: at line 1 SQL BEGIN sys.dbms_xstream_gg.SET_FOO_TRIGGER_SESSION_CONTXT (fire=>TRUE); END;.
2016-05-09 13:40:19  ERROR   OGG-01668  Oracle GoldenGate Delivery for Oracle, ckl2bre2.prm:  PROCESS ABENDING.
oracle@gcxxxx2s[CXXXESIT2] /gg111/112/11.2/11g =>

Continue reading

ORA-17628: Oracle error 19505 returned by remote Oracle server

I have colleague of mine handed over a failed RMAN duplication issue…
after fixing the issue i started again the active duplicate.

but when its almost done… i got below error

 
input datafile file number=00023 name=+DATA/cxxxxsit/datafile/oxxx_data.357.873341887
RMAN-03009: failure of backup command on c8 channel at 03/25/2016 06:57:03
ORA-17628: Oracle error 19505 returned by remote Oracle server
continuing other job steps, job failed will not be re-run

checking the alert log… ASM was exhausted. seems the old files from previous failed Active Duplication was not remove.
my mistake.. 😦

lessons learned… before doing any activity.. do the pre checks… 🙂
and below note is very helpful.

 
Known issues on ORA-17628: Oracle error 19505 returned by remote Oracle server Reported in Rman Active Duplicate (Doc ID 1401333.1)

Thanks!

ORA-00245: control file backup failed; target is likely on a local file system

Basically one day at work, my colleague handed over an issue related to RMAN.

 
Starting backup at 14-FEB-2016 06:02:44
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1444 instance=CXXXEU22 device type=DISK channel ORA_DISK_1: starting compressed full datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 02/14/2016 06:02:46
ORA-00245: control file backup failed; target is likely on a local file system
 Continue reading 

ORA-00904: “UTL_RAW”.”CAST_FROM_NUMBER”: invalid identifier

Recently i’d been testing the backup and recovery of our tape backup.
after i enable the cluster database parameter to true, i’m getting below on error on the alert log of my node database.

Emon ping encountered error 12801
Errors in file /optware/oracle/diag/rdbms/NCXX2PT/NCXX2PT1/trace/NCXX2PT1_q00n_27470.trc:
ORA-12801: error signaled in parallel query server PZ99, instance ncxxdsgtu22:NCXX2PT2 (2)
ORA-00904: "UTL_RAW"."CAST_FROM_NUMBER": invalid identifier
Fri Jan 15 04:17:33 2016

Continue reading

ORA-00600: internal error code, arguments: [kkqvmRmViewFromLst1]

Our database is running 11.2.0.4 and suddenly we got the issue.
the error occurred during refresh of materialized view, its been running for a while and suddenly its having issue during refresh.

here is the error on the alert log.

 
Errors in file /opt/app/oracle/diag/rdbms/cxxxa/CXXXA/trace/CXXXA_ora_1686.trc (incident=1985798):
ORA-00600: internal error code, arguments: [kkqvmRmViewFromLst1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /opt/app/oracle/diag/rdbms/cxxxa/CXXXA/incident/incdir_1985798/CXXXA_ora_1686_i1985798.trc
Sat Oct 11 14:12:41 2014 

as per oracle its a known bug on 11.2.0.4 and not patch available for 11.2.0.4 they just given a work around.

 
1. create a view on the select statement on materialized view
2. create/refresh the materialized view and using the view created earlier.

for now it is working for us. here’s a note from oracle support:

 
Based on the symptoms and call stack trace, your issue seems similar to Bug 17345064 : XF12.1.0.2APPLY - TRC - KKQVMRMVIEWFROMLST, which is in status 11, still being worked by Oracle Development team. 

thanks for visiting..

ORA-00600: internal error code, arguments: [13013], [5001], [482]

Encountered Ora 600 error out of nowhere… DB is old and no issues for a year. but suddenly it aborted with below error.
DB : 10.2.0.3

 
Errors in file /opt/app/oracle/admin/PBTOQ/bdump/pbtoq_smon_27517.trc:
ORA-00600: internal error code, arguments: [13013], [5001], [482], [4198088], [114], [4198088], [3], []
Wed Jul  3 21:34:42 2013
Errors in file /opt/app/oracle/admin/PBTOQ/bdump/pbtoq_pmon_27505.trc:
ORA-00474: SMON process terminated with error
Wed Jul  3 21:34:42 2013
PMON: terminating instance due to error 474

Continue reading